Damn it, Maya. Just… Damn it.

Maya Hates Me: Mental Ray Runs out of Memory

I am by no means an expert, but I hope this helps.  If anyone DOES happen to be an expert, please let me know where I’m wrong, and please offer whatever advice you have.

If you’ve been using Mental Ray for any length of time, you’ll probably be familiar with this message: “Got 8 satellite CPUs… Rendering with Mental Ray… Mental Ray is out of memory… Mental Ray is out of memory… Cannot allocate 223000 bytes… Go eat bleach and die, because Mental Ray hates you”

Well, maybe not the bit about bleach.  But the Mental Ray Memory Grinder is a true annoyance.  The problem is flushing.  No, really, that’s the term that programmers use, and the analogy is pretty apt.  The way that Mental Ray is set up by default is like having a toilet that doesn’t flush.  The poo (in this case your scene and textures (no offense)) just keeps piling up until the smell can kill household pets.  Then Mental Ray commits seppuku and you’re left with a cryptic message and an excuse for Todd Fechter that just won’t fly.  Keep reading to find out how to make the damned thing work.

Mental Ray’s render process has several steps:
•    Load Raw Scene Data
•    Load Geometry Data
•    Tessellate Geometry
•    Occlude Geometry BSP tree
•    Load and Apply Textures
•    Output Render Data

Each of these processes takes up a certain chunk of RAM on top of what Maya, Mental Ray, and all the other programs currently running require.  If it gets past the amount that XP or Vista can handle, then Mental Ray takes a nosedive.  By default, all of those processes are non-flushable, which means that Mental Ray can’t flush the memory toilet before pinching off another loaf of RAM.  Alright, the metaphor is getting old, I know.  In any case, to fix this, you just need to make sure that Maya knows how and when you want to flush certain kinds of data.

SOLUTION:  The Mental Ray Settings Dance of Doom

STEP ONE: Close other open programs!

It has recently come to my attention that there are idiots in this world trying to run multiple copies of Maya while they’re rendering.  This is not good.  Close it.  Also, close down ALL OTHER PROGRAMS, especially that RAM whore known as Mozilla.  Please.  Just do it.

STEP TWO: Batch Render

Batch rendering instead of regular rendering skips the onscreen output step — this can be a double-edged sword, because it allows you to continue working in the background.  If you attempt to do something memory-intensive, while you are rendering, you can sabotoge your own attempt to make the render happen.

STEP THREE: Set your BSP size

BSP stands for Binary Space Partition, which is all I know about it in a technical sense.  It has something to do with Maya’s raytracing algorithms and the way that the scene is broken up to render.  In order to make this process flushable, you have to set your preferences for BSP.  In Maya 2008 and 2009, this option is hidden away in a node called miDefaultOptions.  In order to get to it, click in the script editor at the bottom of the window and type this in, with this capitalization.

Make sure to capitalize the D and O

Make sure to capitalize the D and O

Hit CTRL-A to open the attribute editor, and you will get a new window with options you can’t find in the regular render globals window.
The option we are looking for is under the Mental Ray Globals tab.

Set the Acceleration Method to Large BSP.  This setting is specifically for large scenes, and will break up the render into smaller BSP chunks, enabling flushing for the BSP step of the process.  Don’t ask me why, because I really don’t know.  In 2008 SP2 and above, you can also use BSP2, which is the default for Maya 2010.  Large BSP will slow down your render a bit, but it might be worth it.

Set Acceleration to Large BSP

Set Acceleration to Large BSP

STEP FOUR:  create .MAP files

If you still can’t render after the BSP tree is set, convert your texture files to .MAP . This file format is memory-mappable, which means that Mental Ray can cut it up and use only bits and pieces in RAM as they’re needed. Also, it can flush those parts as soon as they’re done being used.

To convert your textures to .MAP, you need a batch file called “convertfile.bat” You can find one version on my website here:  http://www.whetstonevfx.com/tools/convert_to_MAP.bat, but there are many versions available online.  To use it, just copy the file to the directory where you store your textures, and you can click and drag up to ten files at a time onto the batch file. It will automatically copy and convert your textures into .MAP using Maya’s command-line process called imf_copy.  If you’re working with a ridiculous number of texture files, and don’t want to re-import all the .map files, you can rename them to be .jpg, .tga, or whatever.  This tricks Maya into loading the right file, but still uses the .map.  Unfortunately, this means you can’t open the file in photoshop while it’s still named incorrectly.

Using MAP files will make your render take longer.  However, this is often preferable to not being able to render at all.

STEP FIVE: Instance your geometry

If your file has a large number of repeated geometry, such as a warehouse with stacks of the exact same crate — instance your geometry instead of creating straight duplication.  To do this, click on Duplicate Special instead of Duplicate ( Ctrl – Shift – D instead of  Ctrl – D ) and choose “instance” in the dialog box.  Instanced geometry is flushable.

STEP SIX: Set your Memory Limit

If, after all this, you still can’t get the file to render, you can manually reduce the amount of memory that Mental Ray is allowed to use.

In the render tab, go to the render menu. Next to the options labeled “Render Current Frame” and “Batch Render” there are little boxes. In the menu that comes up, uncheck the “auto memory limit” and click “calculate” Then, ignore this number and input something lower. 1024 or 1512 seem to work pretty well for me on a machine with 3 or more gigs of RAM.

Option Box!

Option Box!

Some versions of Maya don't have the "calculate" button

Some versions of Maya don't have the "calculate" button

This number sets how much memory Mental Ray should take up before initiating the flushing process. If it’s set too high, you still won’t flush before you crash. If you’re working on a particularly low-memory machine, this might be your first step.

STEP SEVEN: Render from the Command Line

When you install Maya, it edits your environment variables, and installs its own set of command-line instructions you can run from DOS. It also edits your command shell structure.

So, provided you have Maya set up properly, you can create a text document known as a batch file to render from the command line.  Open Notepad, and write this:

render -r mr yourscene.mb;

then, save the text document as a .bat. Make sure you choose as the file type “All Files”.  Put this file in the same directory as the scene you want to render, and simply double-click.  Your file should batch render to wherever you set the render path, without ever opening Maya — leaving you with at least a few hundred megabytes of RAM to play with.  There are many tutorials out there about setting up command-line rendering.

STEP EIGHT: Consult Google

There are a number of other ways you can optimize rendering with Mental Ray. I’ll probably be posting more tips as my blog continues, but for now, try looking up the following things: baking occlusion maps, baking final gather calculation, portal lights for final gather, and setting up render tiles. I’m also hoping to start playing with BackBurner for Maya, see what that old Max standby can do for us.  Good luck!


10 responses

  1. Kelly L.

    omg this happened to me and I wanted to scream because I forgot to save my lighting setup. I’ll have to try this. Thanks!

    April 29, 2009 at 4:19 pm

  2. Thanks a lot!!! Mental ray hates me too and I hate mental ray back. I just had a question. I turned off Auto memory limit both from batch rendering options and from mental ray globals’ extra attributes. Then I changed the memory limit from mrglobals performance and memory settings. Even then when I hit render, it gives an error messege that mental rays is out of memory limit and sets t he memory limit back to 128 MB. I have 12 GB RAM!!!! Why is this happening? Is maya overriding mrglobals from somewhere?
    Please help more. Thanks.

    July 19, 2009 at 9:55 am

    • edhobbit

      Mental Ray suffers from some serious memory leaks; make sure you have the latest service pack installed from Autodesk first. Try rendering from the command line — there are some fantastic tutorials out there to help you out creating your batch render file. At the very least, on the command line it might tell you specifically what it is that’s eating your memory.

      July 31, 2009 at 9:55 pm

  3. Thanks for the scoop Ed. I’ve been all over CGTalk and Highend for the last several hours looking for some solid information. Even Autodesk and Mental Images are light on support for these issues with Mental Ray. You may have just saved a life.

    September 1, 2009 at 12:30 am

  4. Jorx

    32 bit OS is probably your culprit- only if you are running a 64 bit operating system will your software (maya, mental ray) be able to use all 16 gigs. Otherwise it’s going to be very limited!

    I’m having issues right now as well- restricted to using a 32 bit windows xp, and I can’t get it to render my displaced geometry (with mental ray approx nodes).

    October 9, 2009 at 9:28 am

  5. WIlly

    Very helpful tips. Thanks a lot man!!!

    October 18, 2009 at 5:58 am

  6. Hi Ed,

    I don’t know if you remember me… I was in the same Todd’s Lighting and Texturing class last semester.

    Anyways, this help this is great! Memory leaks was the major reason why I started rendering everything en Lightwave. I wish this information would have reached my eyes before. RenderMan is flawless in terms of memory allocation. Have you tried it?

    Felipe Ruiz

    March 15, 2010 at 2:48 am

  7. Nauz

    Thanks!!!!!!! I was going insane because I couldn’t find entalRayGlobals in maya 2008… Finally i was able to fix the memory limit, which was set to 2200 by maya 🙂

    June 20, 2010 at 2:35 am

  8. Eduardo

    Great info, solved me a lot of trouble. One simple question though, any ideas on why, when I have instanced geometry, it renders perfectly from the viewport, but if I batch render, none of the intances appear? It’s driving me crazy, and i’ve been looking for an answer all day long, thanks in advance

    September 18, 2010 at 4:04 pm

  9. Ben


    May 7, 2012 at 3:08 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s