building a renderfarm with CUPS

This project was inspired by this article by Patrick Wagstrom. In it the unix printing software LPR was used to queue mp3 files into a playlist on a server. Printing software has to be good at queing as it must wait for the current page to finish printing, before the next job can continue.

The original LPR software has been superseded by CUPS – the Common Unix Printing System. CUPS is superior to LPR because it provide a more standard printing interface (different LPR distributions have different options) and is better at networking. CUPS also comes pre-installed on most Linux (and Mac OS X) distributions.

The original article used LPR input filters to launch the mp3 files. This can also be done with CUPS, but there are also alternatives which make it far easier. Initially I found this article by Tony Lawrence about trying to print to a script using CUPS – this lead me on to discover CUPS backend filters. A backend filter handles passing the print file to a printer via a specific protocol.

Creating the renderman backend filter

A backend filter can be written in any language. It just needs to provide an interface that CUPS can understand:

  • It must print a standard string when passed no parameters – this is used to interrogate the filter.
  • It should process a file passed in as the 6th parameter when run. If there are 5 parameters, then it should process standard input. Any other combination should fail. The other parameters in order are:

job-id user title copies options [job-file]

This simple filter was written in sh shell code and was based off of this code by Kirk Haderlie. It would be possible to use Perl or Python – it would certainly make it prettier! – but this is a very simple script, and it reduces the dependencies.

The renderer to use is passed in as the $DEVICE_URI environment variable from the printer. Alternatively we could base this off the NAME variable by having symlinks named aqsis, prman etc which point to the backend filter. Whichever way would work, but I chose to pass it from the $DEVICE_URI so as not to polute the cups backend directory too much.

The backend filter assumes that the renderer has been installed in /usr/local/bin and accepts the -Progress option. If you want to try this with a different renderer, either add to the PATH variable, or make sure that PATH is set in the environment the backend filter is run in.

#!/bin/sh
# install into /usr/lib/cups/backend/renderman
# Simon Bunker
# June 2008
 
# name of the backend filter
NAME=`basename ${0}`
 
# strip renderman: from $DEVICE_URI
RENDERER=${DEVICE_URI#renderman://}
 
# add /usr/local/bin/ to path
PATH=${PATH}:/usr/local/bin/
 
# check number of arguments (embodied in $#):
case $# in
0) echo "file renderman \"Unknown\" \"Renderman\""
exit 0
;;
5) INPUT=$6
break
;;
6) INPUT=$6
break
;;
*) echo "Wrong number of arguments"
echo "Usage: $NAME job-id user title copies options [file]"
exit 1
;;
esac
 
# pipe RIB file to renderer on stdin.
# pipe output to stderr to log in CUPS.
cat $INPUT | $RENDERER -Progress 1>&2

download CUPS renderman backend filter

We then copy the renderman backend filter to /usr/lib/cups/backend/ and restart the cups daemon using:

sudo /etc/init.d/cupsys restart

or:

sudo killall -HUP cupsd

Adding the render nodes

We can easily add a new renderfarm node by adding a new printer using the lpadmin command line interface:

lpadmin -p farm01 -E -v renderman://aqsis -m raw

lpadmin -p farm02 -E -v renderman://aqsis -m raw

The options are:

  • -p sets the printer name.
  • -E enable the printer. This must be after the printer name option, otherwise it just enables encryption.
  • -v sets this printer to use the renderman backend script. The options after the :// tell the backend which renderer to use.
  • -m setting this to raw means that the input text is unprocessed. Otherwise we could be sending postscript into the renderer.

We can then put our node into a printer class (called farm in this case). When printing to this class, CUPS will pick the next available node to do it’s processing – like most render farms.

lpadmin p farm01 -p farm02 -c farm

By default the pool will be stopped and not accepting jobs. To change this run:

cupsenable farm

accept farm

You can view your new printers using the command:

lpstat -v

Alternatively we can then set up a printer to use as our renderfarm slot using the CUPS web interface at http://localhost:631/. This can also be used to view print jobs and change printer options.

Setting up Permissions

By default filters get run as the lp user. This user has very few priviledges – and usually cannot write to most directories. To resolve this you need to add the User option to the cupsd.conf configuration file and then restart cupsd. I just set this to my username, as I wanted to write to my home directory and it got me up and running quickly. However this should probably have a dedicated user with proper group permissions if you are rendering over a network, and need better security.

You can also use lpadmin to limit submission to certain groups with the -u option.

Installing a RenderMan renderer

Before we can render anything, we need a renderer to render with. For this I decided to use Aqsis – mainly because it is free and a debian package was available.

It turned out that the debain package is out of date, so I had to compile from source. This is probably because Ubuntu tends to be a bit “bleeding edge”. Luckily apt-get/synaptic makes installing all the necessary dependencies (of which there are many) far easier.

I also tried installing Pixie, but I don’t think it likes the latest GCC, and I could not get it to work.

Submitting a job

Once we have a renderer and render nodes set up, we just need some RIB files to render.

The following Python script has a -g mode which generates a series of test RIB files (a teapot turntable) using cg kit. I installed the “light” version as we only want to use the RI interface, and it avoids having to install a lot of dependencies.

#!/usr/bin/python
# Simon Bunker
# June 2008
 
import sys, os
from optparse import OptionParser
 
from cgkit.ri import *
 
def isRIB(x): return x.endswith(".rib")
 
# only handles horizontal fill, square pixels
def getScreenWindow(width, height):
    aspectRatio = float(height) / float(width);
    RiScreenWindow(-1, 1, -aspectRatio, aspectRatio)
 
def ribgen(basepath, width, height):
    for frame in range(1,360):
        filename = "teapot.%04d" % frame
        RiBegin(basepath+"/"+filename+".rib")
        RiFrameBegin(frame)
        RiFormat(width, height,1.0)
        getScreenWindow(width, height)
        RiProjection(RI_PERSPECTIVE, fov=45)
        RiDisplay(basepath+"/"+filename+".tif", RI_FILE, RI_RGB)
        RiShadingRate(1)
        RiPixelSamples(2, 2)
 
        RiWorldBegin()
        RiTransform ([ 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 10, 1 ])
        RiTranslate(0,-1, 2)
        RiRotate(-20, 1, 0, 0)
 
        # distant global light
        RiTransformBegin
        RiConcatTransform ([ 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1 ])
        RiLightSource("distantlight", intensity=1.2, lightcolor=(1, 1, 1))
        RiTransformEnd
        # ambient light
        RiTransformBegin
        RiConcatTransform ([ 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1 ])
        RiLightSource("ambientlight", intensity=0.504043, lightcolor=(1, 1, 1 ))
        RiTransformEnd
 
        RiAttributeBegin()
        RiColor([1,0,0])
        RiSurface("plastic")
        RiRotate(frame, 0, 1, 0)
        RiRotate(-90, 1, 0, 0)
        RiGeometry("teapot")
        RiAttributeEnd()
 
        RiWorldEnd()
        RiFrameEnd()
        RiEnd()
 
def render(basepath):
    for ribfile in filter( isRIB, sorted(os.listdir(basepath))):
        os.system("/usr/bin/lp "+os.path.abspath(ribfile))
 
if __name__ == "__main__":
    basepath = os.getcwd()
 
    parser = OptionParser()
    parser.add_option("-g", "--generate", action="store_true", dest="ribgen", help="Generate teapot turntable RIB files.")
    (opts, args) = parser.parse_args()
 
    if opts.ribgen is True:
        ribgen(basepath, 320, 240)
    else:
        render(basepath)

download Python CUPS wrapper script

We can then submit a job to our queue using the lp or lpr command eg:

lp -d farm test.rib

If we set the renderman printer to be the default printer, then we don’t even need the -d argument. You can do this using:

lpadmin -d farm

The Python script will automatically pick up any RIB files in the current directory and send them to lp when run without any flags.

It can also be useful to set the LogLevel to debug in /etc/cups/cupsd.conf – you can then see the output from the backend filter (from stderr). This can give you more information about what has gone wrong if you aren’t getting any rendered image. The default info level will not give you any useful debugging information.

We can use the standard lp tools such as lpstat and lpq to view the state of our jobs on the farm, and lpmove and lprm (lprm – removes all jobs). This is where the CUPS web interface comes in very useful:

cups renderfarm jobs monitor

Viewing the results

I used djv to view the rendered images and memcoder to generate this movie (with the help from this page) and ffmpeg to encode it into Flash:

Get the Flash Player to see the wordTube Media Player.

Possible Extensions

We can extend the filter in a couple of ways – firstly by picking up options passed in from the -o flag. Useful ones would be to set the frames to render, the resolution etc.

We could create a script or Makefile to deal with dependencies. However this will still appear as just one job – and we will not be able to restart just a part of the job.

WE can pass feedback back to CUPS to tell it the percentage completed so far. I am not entirely sure how to do this, but I think this might be possible with a port monitor script – see this thread on the Apple printing mailing list, however I cannot find any information about how to write one. Or possibly by using a wrapper filter.

Conclusions

Using CUPS as a renderfarm works incredibly well, however if you are running a real renderfarm, then use a proper queuing system such as Alfred or Sun’s Grid Engine!

Here are some of the advantages and disadvantages of using CUPS:

Advantages

  • Already has commands for submitting, viewing and manipulating the job queue
  • You can easily set up a pool which will allocate jobs to slots as they become available
  • Can be used on/by remote machines easily
  • Adding extra job slots just means adding a new printer
  • Jobs can easily be paused or delayed
  • Can submit from Windows using SAMBA
  • Has built in web administration using http://localhost:163

Disadvantages

  • Has no way of adding job interdependance. eg you want the final job to wait whilst shadow maps jobs are rendered. This is a huge limitation.
  • Probably doesn’t scale to several thousand processors well (cannot test this one)
  • You could end up with a lot of scrap paper if you use the wrong queue!
[Slashdot] [Digg] [Reddit] [del.icio.us] [Facebook] [Technorati] [Google] [StumbleUpon]