Maya Hates Me: Mental Ray Runs out of Memory

I am by no means an expert, but I hope this helps.  If anyone DOES happen to be an expert, please let me know where I’m wrong, and please offer whatever advice you have.

If you’ve been using Mental Ray for any length of time, you’ll probably be familiar with this message: “Got 8 satellite CPUs… Rendering with Mental Ray… Mental Ray is out of memory… Mental Ray is out of memory… Cannot allocate 223000 bytes… Go eat bleach and die, because Mental Ray hates you”

Well, maybe not the bit about bleach.  But the Mental Ray Memory Grinder is a true annoyance.  The problem is flushing.  No, really, that’s the term that programmers use, and the analogy is pretty apt.  The way that Mental Ray is set up by default is like having a toilet that doesn’t flush.  The poo (in this case your scene and textures (no offense)) just keeps piling up until the smell can kill household pets.  Then Mental Ray commits seppuku and you’re left with a cryptic message and an excuse for Todd Fechter that just won’t fly.  Keep reading to find out how to make the damned thing work.

Mental Ray’s render process has several steps:
•    Load Raw Scene Data
•    Load Geometry Data
•    Tessellate Geometry
•    Occlude Geometry BSP tree
•    Load and Apply Textures
•    Output Render Data

Each of these processes takes up a certain chunk of RAM on top of what Maya, Mental Ray, and all the other programs currently running require.  If it gets past the amount that XP or Vista can handle, then Mental Ray takes a nosedive.  By default, all of those processes are non-flushable, which means that Mental Ray can’t flush the memory toilet before pinching off another loaf of RAM.  Alright, the metaphor is getting old, I know.  In any case, to fix this, you just need to make sure that Maya knows how and when you want to flush certain kinds of data.

SOLUTION:  The Mental Ray Settings Dance of Doom

STEP ONE: Close other open programs!

It has recently come to my attention that there are idiots in this world trying to run multiple copies of Maya while they’re rendering.  This is not good.  Close it.  Also, close down ALL OTHER PROGRAMS, especially that RAM whore known as Mozilla.  Please.  Just do it.

STEP TWO: Batch Render

Batch rendering instead of regular rendering skips the onscreen output step — this can be a double-edged sword, because it allows you to continue working in the background.  If you attempt to do something memory-intensive, while you are rendering, you can sabotoge your own attempt to make the render happen.

STEP THREE: Set your BSP size

BSP stands for Binary Space Partition, which is all I know about it in a technical sense.  It has something to do with Maya’s raytracing algorithms and the way that the scene is broken up to render.  In order to make this process flushable, you have to set your preferences for BSP.  In Maya 2008 and 2009, this option is hidden away in a node called miDefaultOptions.  In order to get to it, click in the script editor at the bottom of the window and type this in, with this capitalization.

Make sure to capitalize the D and O

Hit CTRL-A to open the attribute editor, and you will get a new window with options you can’t find in the regular render globals window.
The option we are looking for is under the Mental Ray Globals tab.

Set the Acceleration Method to Large BSP.  This setting is specifically for large scenes, and will break up the render into smaller BSP chunks, enabling flushing for the BSP step of the process.  Don’t ask me why, because I really don’t know.  In 2008 SP2 and above, you can also use BSP2, which is the default for Maya 2010.  Large BSP will slow down your render a bit, but it might be worth it.

Set Acceleration to Large BSP

STEP FOUR:  create .MAP files

If you still can’t render after the BSP tree is set, convert your texture files to .MAP . This file format is memory-mappable, which means that Mental Ray can cut it up and use only bits and pieces in RAM as they’re needed. Also, it can flush those parts as soon as they’re done being used.

To convert your textures to .MAP, you need a batch file called “convertfile.bat” You can find one version on my website here:  http://www.whetstonevfx.com/tools/convert_to_MAP.bat, but there are many versions available online.  To use it, just copy the file to the directory where you store your textures, and you can click and drag up to ten files at a time onto the batch file. It will automatically copy and convert your textures into .MAP using Maya’s command-line process called imf_copy.  If you’re working with a ridiculous number of texture files, and don’t want to re-import all the .map files, you can rename them to be .jpg, .tga, or whatever.  This tricks Maya into loading the right file, but still uses the .map.  Unfortunately, this means you can’t open the file in photoshop while it’s still named incorrectly.

Using MAP files will make your render take longer.  However, this is often preferable to not being able to render at all.

STEP FIVE: Instance your geometry

If your file has a large number of repeated geometry, such as a warehouse with stacks of the exact same crate — instance your geometry instead of creating straight duplication.  To do this, click on Duplicate Special instead of Duplicate ( Ctrl – Shift – D instead of  Ctrl – D ) and choose “instance” in the dialog box.  Instanced geometry is flushable.

STEP SIX: Set your Memory Limit

If, after all this, you still can’t get the file to render, you can manually reduce the amount of memory that Mental Ray is allowed to use.

In the render tab, go to the render menu. Next to the options labeled “Render Current Frame” and “Batch Render” there are little boxes. In the menu that comes up, uncheck the “auto memory limit” and click “calculate” Then, ignore this number and input something lower. 1024 or 1512 seem to work pretty well for me on a machine with 3 or more gigs of RAM.

Option Box!

Some versions of Maya don't have the "calculate" button

This number sets how much memory Mental Ray should take up before initiating the flushing process. If it’s set too high, you still won’t flush before you crash. If you’re working on a particularly low-memory machine, this might be your first step.

STEP SEVEN: Render from the Command Line

When you install Maya, it edits your environment variables, and installs its own set of command-line instructions you can run from DOS. It also edits your command shell structure.

So, provided you have Maya set up properly, you can create a text document known as a batch file to render from the command line.  Open Notepad, and write this:

render -r mr yourscene.mb;

then, save the text document as a .bat. Make sure you choose as the file type “All Files”.  Put this file in the same directory as the scene you want to render, and simply double-click.  Your file should batch render to wherever you set the render path, without ever opening Maya — leaving you with at least a few hundred megabytes of RAM to play with.  There are many tutorials out there about setting up command-line rendering.

STEP EIGHT: Consult Google

There are a number of other ways you can optimize rendering with Mental Ray. I’ll probably be posting more tips as my blog continues, but for now, try looking up the following things: baking occlusion maps, baking final gather calculation, portal lights for final gather, and setting up render tiles. I’m also hoping to start playing with BackBurner for Maya, see what that old Max standby can do for us.  Good luck!