Rendering problems

  • Jack

    Jack - 2012-02-17

    I recently started using AoI and find it is a wonderful program and my thanks to those that developed it.  I have been using computers since the 90's, (mostly as a casual user so my knowledge of the inner workings is limited), and this was my first foray into the 3d realm.  I tried a lot of programs, but it wasn't until I discovered AoI that I was able to actually put my idea's on the screen.  The interface is easy to use and relatively intuitive once you establish a 3d mindset, the results are fantastic, and the documentation and tutorials are great resources.  I've still got a long way to go but I'm very pleased to be able to have gotten as far as I have and I wanted to make sure I gave a pat on the back to all who have worked on it so you'd know your efforts are very much appreciated.

    The problem I'm having is in rendering movies, (or even images for movies), when using anti-aliasing. 

    I've got a short animated segment that is about a half second long.  I can render it without AA and it works just fine, but I've tried a myriad of different options when trying to render with AA and every time it locks up. 

    For example, I tried this morning and it got about 2/3rds of the way when it just quits working.  The render preview shows no increase in frames or time, or change in the image, and closing the preview window can be difficult.  I tried once with an output of jpeg frames instead of a .mov and it did the same thing.  And it isn't just "slow" and something I need to wait for; I started a run before I went to sleep last night and it locked up about 2-3 minutes into the rendering and that's how I found it this morning.

    This morning I tried something different, thinking that their may be something in the project that causes it to lock at a certain point.  So instead of starting at "0" in the timeline, I started at .25 for the beginning to ensure that the general area where it usually locks up is rendered relatively early in the process, and it worked, (but it did seem to slow down as it neared the end).

    So it appears I can render using AA if I break everything up into half second segments, but, for obvious reasons, I'd rather not do that…

    If anyone has any thoughts or idea's they would be greatly appreciated.  I originally started using AoI 2.9, but switched to the new beta release yesterday to see if that made a difference, (it didn't).  I'm running 32 bit windows 7 with a E7500 Core Due at 2.93 GHz and 3 GB ram, with an Nvidia GeForce 8400 GS.  Java updated yesterday, and I added the JME within the past couple of days, and the driver for the video card is also up to date.  That was all I could think of that might be effecting the situation.

    Thanks for any help you might have.

  • Luke S

    Luke S - 2012-02-17

    Glad you are enjoying AOI.

    First off, I'm not an expert,but I am one of the (rare?) windows users here.
    A few questions that might help people track this down:

    1: How are you launching aoi? Double-clicking the jar? The 'Start Art Of Illusion' windows launcher? Other?
            1a: How much memory is allocated to AOI? (if you're not sure, go help ->about Art Of Illusion - you will get a little splash screen that has some info including "using Xmb of Ymb allocated")

    2: How many frames are you trying to render? (Frames per second Times length of the animation)
         2a: What is their resolution? (Pixels wide X pixels high)

    3: Could you check for error messages? To do this: Open a file browser. (go look at your documents folder or something) navigate to where your AOI installation lives. (usually Program Files\artofillusion) In the folder view on the left Right-Click on the 'artofillusion' folder, and choose 'open command prompt here.' In the command prompt window type: java -jar ArtOfIllusion.jar   and press enter. This will start AOI. Open your file, and try to render as usual. When the render freezes, bring the command prompt window to the front. There will be a lot of text. Scroll through it, and if you find anything that starts something like 'java.lang._-something-_exception', etc, etc Tell us about it - It might just hold the key.

    Hope we can help.

  • Peter Eastman

    Peter Eastman - 2012-02-18

    By the way, how did the beta installer work for you?  We've had mixed reports from Windows 7 users, with some people saying it works fine and others having it hang during installation.  So far I've been unable to track down the problem, so I'm looking for data points of people who did or didn't have problems with it.


  • Jack

    Jack - 2012-02-18

    Thanks for the replies, I appreciate the help and I'll do the easy question first.

    I had no problem with the beta install.  There was no hang during the install and I started it right up without any problems and I've been using it ever since.

    1) I start AOI using the windows launcher.  I checked the memory and I'll include everything it says, (figure that won't hurt anything…)

    Java 1.6.0_31
    Windows 7 6.1
    15 MB of 23 allocated (989 max)
    open GL is available and enabled

    2) I've used either 30 frames per second, or 15, depending on what I was trying to do.  I did some that had as many as 120
    + frames, and some with less than 30.  Neither would make it very far when using AA; sometimes as low as single digits, never beyond about 12-15 frames.

    2-a) I've tried various things, usually the default 400 x 300.  I tried something ridiculously small too, (like, 20 x 20),  thinking that if it was a case of "too much information" in the larger size that might work.  It didn't; it hung at about the same point.  For the most part I've tried it with just the basic settings, omitting some options I believe I'd probably like to include like "soft shadows" under the thought that these would probably just make things worse.  I also tried reducing the antialias max rays from 16 to 4 thinking it might help, but it didn't seem to.

    3) It's been a long time since I've used DOS commands, and I couldn't get it to work the way you suggested, but I finally managed to get to where I needed to be, and then run it.  Since it may help, these are the settings I used for this run:

    400 x 300
    .5 seconds long
    30 fps
    1 image/frame
    medium AA
    no other options checked
    quality 90

    This time it only went to frame 2 of 15, 13 seconds in.

    My guess is that this is the relevant section, (but I do have a screen shot of the entire last portion if that might help, as there are 2 or 3 dozen lines below this).

    - creat datasink for : file blah blah blah my movie name

    Exception in thread "Raytracer main thread" java.lang.outofmemory error: Java heap space

    it then goes on to list a dozen or so statements similar to:

    at artofillusion.raytracer.ray.<init>(

    and the final 4 entries are similar to the final one:

    8:38:36 AM EST [thread-12 in widgetflasher.reset() ( Widgetflasher: reset. on=true

    Now I'm no rocket scientist, (unless we're talking model rockets with Estes engines but its been over 20 years since I've played with those.), but I've got a feeling that the statement "outofmemory" might be significant.  My thought process was that it might be a memory problem so I tried things like using task manager to get close out a couple of resident memory hogs, and through Windows I added virtual memory on my backup hard drive, but I didn't see any significant benefits from either. 

    But then again, I really don't know much about memory, (which may be abundantly clear from my musings above), which is why I'm here…

    So any suggestions are most welcome, and if there is anything more I can do to help pinpoint the problem just let me know and I'll see what I can do.

  • Peter Eastman

    Peter Eastman - 2012-02-18

    Yep, the out of memory error is indeed the problem.  Here are a couple of things to try.

    1. When you launch it from the command window, try using this command:

    java -Xmx1500m -jar ArtOfIllusion.jar

    That will tell it to reserve 1.5 GB of memory (instead of 1 GB, as the standard launcher does).  You may need to quit other programs: if it can't find that big a block of memory, it will just fail to launch.

    2. In the Advanced Rendering Options window, you'll find a checkbox that says, "Use Less Memory (slower)".  Try enabling that.


  • Jack

    Jack - 2012-02-19

    Thank you Peter, (and Luke for your previous help).  I'll give that a try and see what happens.  I did try the "use less memory" button once without any luck, but it may be I was just too low for it to make a difference.

    For a long term solution, would bumping my ram from 3 to 4 GB help?  I assume so, but rather than just spend the money on it I figured I'd ask first…

  • Jack

    Jack - 2012-02-19

    And the results…  I couldn't get it to run at 1500, so I tried stepping it down.  AT 1100 it ran, at 1150 it didn't.  And that was after I shut down everything I could.  Its amazing how many things run in the background, (and how many just keep popping up after you shut them down).

    I did a 1/2 second run with AA and it just barely made it through, which is better than before.  It did kind of hang at the end, with the rendering window refusing to close, and at that point task manager said it was at 1218k.   Eventually the window did close though and I shut it down normally.

    I know this isn't an AOI issue but since everyone here is LOADS smarter than me about computers I figured I'd at least ask if anyone had any other suggestions, such as:

    1) Is there any kind of memory management program that might help?  I've tried them before on my netbook that only has 1 GB and it seemed like it helped, but I don't think the processes involved were all that complex

    2) Should I consider setting up another operating system such as Linux to run it?  I have an external hard drive that has tons of space that I just use for storage.  If I was to do that can I export my existing files?  And would they retain textures?

    Of course if you think adding 1 GB of ram would probably solve everything that would probably be the easiest route to take, but I figured I should investigate all the possibilities.

    Thanks again for your help,

  • Peter Eastman

    Peter Eastman - 2012-02-19

    I don't know too much about memory management on Windows 7, so I'll let other people answer about that.  But approaching it from a different direction, there might be some simple ways to reduce how much memory AoI is using.  Here are a few things to look at:

    - What surface accuracy are you using?  That makes a big difference in memory use.
    - Are you using any photon maps?  They can take a lot of memory.
    - What is the Interactive Surface Error set to in the preferences window?  If that's too small it can waste memory.
    - Select Scene->Images and make sure you don't have any images stored in your scene that you aren't actually using.  Also, if you have any very large images, can they be replaced with smaller versions?
    - Can you post your scene?  I can take a look and see what's taking most of the memory.


  • Luke S

    Luke S - 2012-02-19

    I'll let Peter work on the AOI specific stuff  - But for some windows specifics:

    at that point task manager said it was at 1218k

    Do you mean that the process was using 1218k? Or did the system have 1218k free? If the second, you do have some real memory hogs somewhere! See If you can't turn them off, at least when using AOI.

    Of course if you think adding 1 GB of ram would probably solve everything that would probably be the easiest route to take, but I figured I should investigate all the possibilities.

    This might do wonders for your overall system performance, but the question is whether 32 bit Windows can address that much - I know there is a limit (that's one of the reasons that people are going to 64 bit) but I can't remember where it is.

    BTW - usually, make sure that your virtual memory is at lest twice the size of your real ram!

    This is kind of a weird problem, I have less ram than you and I have never run out of memory trying to render something.

    One thing to keep in mind: No matter how much memory the system has, 32 bit java programs can only use 1.5GB. Period.
    --- On that thought: try running the command line start with -Xmx1536m (exactly 1.5GB) just to see what happens.

    One other thing to try: Might help someone see if the problem is your system or if there is something weird happening in the renderer. -- download the advanced rendering plugin, and try rendering your scene through that. (tools -> advanced rendering)
    You can ignore the extra features for now - this plugin uses slightly different code to render, and saves its output to a file as it goes instead of waiting until it is all rendered.

  • Jack

    Jack - 2012-02-19

    Thanks again to you both.  While I am of course trying to fix my own problem, I'm also doing the best I can to post as much information as possible so that if someone else has similar problems in the future this page might be able to help them.  (That's my disclaimer if it seems I'm going offtrack here).

    First the answers to Peter's questions.  When I render I generally have been using the default settings, without any extra options.  So that would mean accuracy of .02, no photon maps, the interactive surface error is .05.  I did up the maximum level of undo to 10, and I do seem to recall some mention of the larger that is the more memory it would take up, but I really do like being able to go back that far…  besides, I don't have a clue if that might effect the rendering.  I'm unable to post the scene, but I think my problem is more in the "lack of memory" category than the "excessive amounts of data needed to be rendered" category.  I've seen many AOI animations that are far more complex than anything I'm doing, which I assume probably didn't have the same problems when they were rendered.

    And on to Luke…

    The AOI process was taking up 1218k.  I checked and the 32 bit windows can handle 4 GB, which is where I would be if I added 1GB.  I learned long ago how important ram is and since it's only about $35 I'll probably go ahead and do it no matter what.  It never hurts, and usually helps.

    I just checked my virtual memory which was set to "auto" and found the stated amount being used as 3GB, with the suggested being about 4.5.  So I switched from "auto" to "set" and made it 6 and 10. 

    I'm kind of amazed you've never had a memory problem before given the problems it is giving me.  Then again, until now I have never had one either.  I did try running it with 1536 to no avail.  I do have the advanced renderer and will try that again, (I used it at least once but I don't believe it helped). 

    I also went through this morning and did some "spring cleaning" in the startup settings.  Of 32 processes that ran on startup I eliminated slightly over half.  Since I didn't know what many of them were I just did an internet search on the file name and learned a lot about what processes were running.  Seems every little thing you use wants to phone home, and most don't need to.

    Unfortunately, after doing that I was in exactly the same place I started at: 1100 would run, 1200 wouldn't.  I've now rebooted after changing the virtual memory and with both that and removing the resident programs I STILL get this error when trying to run with the set 1500 in memory:

    Error occured during initialization of VM
    Could not reserve enough space for object heap
    Could not create the Java virtual machine

    So I'm beginning to wonder if perhaps I've got some other memory problem, in which case another module might not make any difference.  In my internet travels this morning I came across a few programs designed to test memory function, so I'll try to do that sometime today.  And, as always, any other help is greatly appreciated.


  • Jack

    Jack - 2012-02-19

    I just ran the Windows Memory Diagnostic tool and it found no errors.  Of course I don't have a clue if that is a decent test or not, but it was at least encouraging.

    But perhaps I'm on the correct path as I just did a 1/2 second animation using the Advanced Raytracer that appeared to work flawlessly.  This was a 400 x 300 animation, AA on medium, Soft shadows thrown in to give it a little extra to work on, all done with "use less memory" checked.  It didn't hang at all and took 2.25 minutes to render the 15 frames.

    I'm going to try some more complex things this afternoon and I'll report back as to how they worked.


  • Peter Eastman

    Peter Eastman - 2012-02-19

    I checked and the 32 bit windows can handle 4 GB, which is where I would be if I added 1GB.

    It's actually more complicated than that.  32 bit Windows can handle a 4 GB <i>virtual address space</i>.  The relation between virtual address space and physical memory is very indirect.  You can use swap space to supplement your physical memory, and different processes have different virtual address spaces.  On the other hand, of the 4 GB, only 2 are available to your application.  The other 2 are reserved by the OS for its own use.  That's why AoI on 32 bit Windows can never use more than that no matter how much physical memory you have.

    That's not meant to discourage you from adding more physical memory.  3 GB isn't much by modern standards, and as you say, it's cheap.  It will probably make your system faster.  Just don't assume it will let you render much larger scenes.

    I've seen many AOI animations that are far more complex than anything I'm doing

    It's possible there are things in your scene that take more memory than they really need.  It's also possible those other animations were rendered on Mac or Linux (both of which handle virtual memory better than Windows does), or on 64 bit OSes.


  • Luke S

    Luke S - 2012-02-19


    The AOI process was taking up 1218k.

    Are you sure it was k? not m?  also, are you sure it was the AOI Process? (should be labeled java.exe or javaw.exe) when I launch on windows, mine takes up at least 58 or 59 MB, including the space the JVM uses for itself.

    if the process is named jusched.exe, you were looking at the tiny program that keeps your java runtime up to date.

    I'm glad that the advanced renderer seems to work for you - Its meant to allow larger renders than the normal render engine can handle.

  • Jack

    Jack - 2012-02-20

    It's possible there are things in your scene that take more memory than they really need.

    I'd say its not only possible, but knowing that I really don't know what I'm doing I'd say it is probable.  That's why my first inclination when this started happening was I did something really wrong that was causing it to screw up.  But since it rendered OK without AA, and then I did a test where I started at the place where it usually failed and it rendered from there on, I was less inclined to believe it was something I did.

    Are you sure it was k? not m? also, are you sure it was the AOI Process?

    No and no.  LOL  I've worked with computers for many years, (I still have my first, an Apple II+, in my basement), and at one point I knew quite a bit on how everything works.  But it changes so fast and since I didn't really NEED to keep up with it, I didn't.  So as much as I'd rather just sit down and have everything work as it is supposed to, I don't really mind experiences such as this one because I did learn quite a bit.  But in the process of learning it's real easy to forget everything you tried, (just like with AOI, and I find myself needing to go back to the manual frequently, but less and less with time).

    I added quite a bit to my scene today, perhaps as much as double what was there before.  Using the advanced render option and low memory usage I did my usual 1/2 second movie to see what would happen.  It worked!  So I got really brave and did a 500 x 350 and 1.5 second run, and it completed the 45 frames in 9:40.  We'll see what happens, but appears all your tips have helped tremendously.  Thank you.

    I do have one quick AOI specific question that I could probably figure out, eventually, but I was hoping someone might know off the top of their head.  I have a single object in the scene that I would prefer to NOT cast a shadow.  Any way to do this?


  • Nik Trevallyn-Jones

    I have a single object in the scene that I would prefer to NOT cast a shadow. Any way to do this?

    The only thing I can think of would be the Effects Catcher plugin.

    I don't actually think you can get quite the feature you want with the current Effects Catcher, but I *think* the feature would be pretty easy to add.

    I'll have a look and post back later today (my time).


  • Nik Trevallyn-Jones

    … on this point:

    I did test using a material with the "casts shadows" checkbox disabled, but I couldn't find any way to make this checkbox have any effect.

    In my tests, the object with this material set always cast a shadow if the material was not transparent - regardless of the "casts shadows" checkbox; and the object with this material never cast a shadow if the material was transparent - again regardless of the setting of the "casts shadows" checkbox.

    In addition, if the object had a non-transparent texture, then the object cast a shadow regardless of the material used.

    I had hoped that I could set a non-transparent texture on the object and combine it with a material (transparent or otherwise) that had "casts shadows" disabled to achieve the effect of a visible object that cast no shadow.

    Is this possible, and I've just messed up my test somehow?


  • Jack

    Jack - 2012-02-20

    Thanks Nik.  I knew I had seen something about shadows but couldn't remember where it was but you helped me find it.  Unfortunately, I've had about as much luck as you in getting it to work.  I'll keep playing with it and I think I may have another way of getting the effect I want.

    For anyone curious about rendering, I added more to my scene today, and included some camera movements and invisibility to the animation.  Then I rendered about 145 frames for just shy of 5 seconds of video without encountering any problems.  So if anyone who is having the same problems while using Windows 7 stumbles across the page I'd suggest:
    1) open msconfig and then remove any programs from "startup that you don't need/use.  If you're not sure exactly what they are just put the file name into a search engine and you should get plenty of information.  Do the same with "Services".
    2) close out any other programs that are running before rendering.

    I think that's mostly what worked for me.

  • Bill French

    Bill French - 2012-02-28

    I have been having some of the same crash issues with "out of memory" errors, but get away from that using AOI 2.8.1 for the actual rendering. I use 2.9.1 for setting up the scene, and texturing !

    On the shadow problem - Try setting the light source to "shadowless" (in the properties box).  If you only want one object  out of many, to be shadowless - then you will have to resort to spotlights on each object - and make each spotlight either shadowless - or not.

    Hope this helps somewhat !

    -----> Bill

  • Bill French

    Bill French - 2012-03-01

    Hi guys -
    I think I might have tracked down where the "out of memory" problem lies !!

    I run both AOI 2.8.1 and 2.,9.1, and both were behaving pretty well except for an occasional crash when I had 1024 rays (in advanced).
    I go looking at the splash screen under the help menu, and noticed that 2.8.1 was reporting "windows 7.6.1 (x86) and openGL available and enabled.
    When I looked at the 2.9.1 splash screen it reported windows 7.6.1 (x86) and openGL NOT available but enabled ?????

    I decided to goto the SUN site, and discovered that you can have 32bit and 64bit Java at the same time IF 32 is installed first. I downloaded the 64 bit version, installed it and "Bink" both AOI versions are running perfectly now.

    The 2.8.1 splash reports the same info BUT the 2.9.1 version now reports Java 1.6.0_31 - windows 7.6.1 (amd64). Notice that the windows version is now reported at 64bit !! 

    Odd - but works now.

    ---> Bill


Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.

No, thanks