MaxScript, Backburner and Dependencies!

Hi All,

I have today finally made progress with a problem I’ve been contending with, on and off, for a few weeks now! The problem is of using MaxScript and Backburner with dependencies. Just submitting Max renders to Backburner using Maxscript isn’t a problem, though for some unutterable reason dependencies are not supported. There are a handful of options out there to try and solve this problem, namely…

– Sending the job to backburner suspended, without any dependencies, then setting the dependencies by editing the (unused) <DependsOn> XML tag in the backburner job folder, before archiving and unarchiving the job (so that the XML file gets re-read). This is problematic as the only way to archive and unarchive a job (also not a function available via MaxScript!) is via Telnet, either through Python or dotNET. I managed to very vaguely get this working using dotNET but Telnet is not the most elegant of things, especially when in an automated system.

– Setting post-render scripts that perform a certain task when a render is complete. The problem here is that the post-render script is called every time a render server finishes its job – which might not necessarily be at the end of the actual job, of course. Whilst this is potentially surmountable by checking to see what frame was rendered in the script, it also meant that each render node needed a licensed copy of Max if it were to open and close files.

– Using cmdjob.exe, the command-line backburner queue magician. This is what I have ended up using.

The solution is actually relatively elegant now. I can’t post actual code as this is for a paid job, but the basic process is this:┬ácmdjob.exe can send jobs to backburner which have dependencies. So what you do is have a cmdjob task launch your actual render tasks. Instead of submitting scenes directly from Maxscript to Backburner, have cmdjob.exe run on Backburner, call up a copy of max and then run a script on it, with the script containing instructions to send a render job to backburner. Because this job is not sent until the cmdjob is executed, and because cmdjob’s can have dependencies, you can effectively have dependencies through maxscript.

Which is easier said than done! So here is a more detailed approach to the process:

– The user loads a scene they want to render.

– When they run the script, a series of other scripts are generated and saved alongside the .max file. These scripts contain all the instructions needed to alter the scene as per the users wish (for example, there may be 3 scripts – one simulates and saves out a particle sequence, one loads this sequence and pre-calculates a GI solution, the third loads both the generated particles and the pre-calculated GI solution and renders the final frames.) and then submit it to Backburner as netrender, before closing the instance of Max.

– Instead of simply sending the first job to the render farm via backburner in MaxScript, a commandline call is made to cmdjob.exe (more info here) to load a copy of max and run a certain script – in this case, the first one that we just generated. The crucial thing to know here is that cmdjob.exe jobs CAN be set to be dependent on other jobs.

– So the cmdjob.exe job is sent to backburner, and is picked up by a render node. This machine needs to have a licensed copy of Max and for that reason I recommend a special machine dedicated to these sorts of tasks. They don’t need a fast processor or fancy graphics card, but they do need a lot of RAM as they will be opening all your max scenes.

– This machine opens max and runs the script that was generated. This script basically contains all the options needed to be useful (such as turning on any particle generators, setting output paths for precalculated GI etc), then sends itself to Backburner via netrender (Job A). Cruicially, it then also submits the NEXT cmdjob.exe task to backburner, dependent on the job it’s just sent (Job B, dependent on Job A). It then closes that instance of Max, and backburner sets the task as finished.

– Next, Backburner finds itself with two new tasks – Job A and Job B from above. Job B is dependent on Job A. So Job A is set off to render, and when it’s finished, Job B begins – and the same process as the previous step begins all over again. This time it loads up the next script, submits that render (Job C), and submits another cmdjob (Job D), again, depedent on the one it just sent.

– This process continues ad-infinitum.

There are a lot of complications here. Do you want to hard-code the whole process of what submits what? My solution was the have the very original script run by the user generate all the scripts (.ms files prefixed with numbers indicating their order) to be used by the cmdjobs, and one of the last lines of each of those scripts was to “fileIn” another script (“fileIn” being the scripting equivalent of XRefing). This script deletes the .ms that called it and looks in the folder to see if there are any more. If there is, it launches a new cmdjob running the first script alphabetically. Thus, when that script runs and submits its next job, it will again delete the script which called it and look for the next. This way, I can have almost infinite scripts all daisy-chain off one another.

This took me a while to work out (thanks in no small part to a few frustrating Max bugs!) but it’s working quite well, with the added benefit of allowing the user to submit jobs to backburner that could previously not have been sent there. If you have any questions about the process, please feel free to email me at dan-grover@dan-grover.com and I’ll try and help as best I can!

Thanks,

Dan