Bug:
When rendering multiple images (with PhotoVideoRendering Plug-in) of a large .sh3d file (about 2.5 GB), memory does not appear to be released after each image completes, resulting in an unsustainable increase of memory use.
For reference, after one image, memory use is about 55 GB, then after two it is about 104 GB, then after three it is about 160 GB; after which the program will invariably crash with the system reporting that it has run out of Application Memory (there seems to be a Virtual Memory ceiling of about 200 GB on macOS as there would be enough free space on the root partition for the Virtual Memory Backing Store).
For additional illustration, as an example, while rendering the second of two images, a snapshot of the exact apportionment of memory use is as follows:
Sweet Home 3D
Real Memory: 12.01 GB (varies throughout the render)
Memory: 103.38 GB
kernel_task
Real Memory: 13.10 GB (varies throughout the render)
Memory: 45.0 MB
Attached file shows these same values as reported by Activity Monitor.
Context:
I use SH3D on my MacBook Pro 16" Core i9 (2020 model) with 32 GB of RAM and macOS 10.15.7 Catalina. I have over 400 GB of free space on the root partition.
I use the Yafaray Plug-in (version 2.1) and the PhotoVideoRendering Plug-in (version 2.1). Render setting is set to Level 4, with Photon tracing, Lanczos and Shinydiffusemat.
I run the .jar version of SH3D (version 6.4.2) with 28 GB memory allocation (ie. $ java -Xmx28672m -jar ~/SweetHome3D-6.4.2.jar).
My java is up to date and the reported version is:
$ java -version
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
This could be some space taken by texture images which are not freed between each rendering to optimize the launch of YafaRay rendering process. Did you try to render different SH3D files or changing textures between renderings, or did it happen only with one unchanged project?
Hi Emmanuel,
The problem happens without any interaction with the program while the images are being generated.
Furthermore, there is no modification made to the textures in between the renderings.
As far as I can tell, this problem exists with all my large project files.
Also, I’ve done some more testing (which takes time with these very long renders), and I can confirm that the problem exists with all of the following situations:
—Manually rendering successive images with the native “Create photo with Yafaray” (v1.4.3) plug-in;
—Manually rendering successive images with the “Simple photo rendering” option of the “Photo-video rendering” (v2.1) plug-in;
—Batch rendering of photos with the “Multiple photo rendering” option of the “Photo-video rendering” (v2.1) plug-in — which is also the feature I normally use.
It certainly seems to me like something isn’t released from memory between renders; textures could be plausible culprits (but you would know best, of course).
Hope this helps.
I fixed two memory leaks in the version 1.4.4 of YafaRay Rendering plug-in.
Could you check it works better for you?
Hi Emmanuel,
Sorry for the short wait (I've had a few very busy days). So, I’ve finally been able to run a few renders and, unfortunately, it seems that the problem is still present in this latest version. Is there something else I could do to help you zero in on the source of the problem?
Best,
Ghyslain
Thanks for your feedback.
Just to be sure: can you confirm that you tried with the plug-in I posted i.e.Tools > Create photo with YafaRay menu item?
Yes, I can now confirm that the same problem remains when manually rendering the images one after the other with the Tools > Create photo with YafaRay command from the plug-in.
Sorry to insist, but are you sure you don't see any improvement?
It's possible that the used memory could never decrease because of the way the garbage collector works, but after a while it wouldn't increase anymore and above all, the program wouldn't crash anymore.
Hi Emmanuel,
It’s no trouble at all.
I can confirm that I tried with the Tools > Create photo with YafaRay version 1.4.4 downloaded from the link you provided above.
As far as I can tell, the problematic behaviour seems to be the same as with the previous version. In the case of this particular trial run, I launched four renders one after the other, changing only the camera position using Stored Points of View (3D View > Go to point of view). The elapsed time was a couple of hours for the renders to complete.
On first render, the memory use went up to about 60 GB, which was expected. On the second render, memory went up to about 100 GB or so. On the third, it reached about 150 GB. By the time the fourth render was launched, the Virtual Memory Backing Store (the swap file) was reported at 100 GB by Activity Monitor. It is my experience that under macOS, at least on my MacBook Pro, there seems to be a hard limit of 100 GB on the swap file (regardless of free space on the HD). I don’t know if this is a macOS bug or a JavaVM bug, or if it is indeed some kind of limit under macOS (I couldn’t find a definitive answer online, but apparently others have also reported this behaviour).
In any case, at that point the system typically starts to misbehave and I am invariably presented with the “You have run out of Application Memory” warning dialogue window. In this particular trial run, I did end up killing SH3D before it crashed, simply because the system sometimes becomes completely unresponsive once the swap file has become “full.” I am willing to try again and let it crash, just to confirm it, but I would be surprised if it didn’t (in any case the computer would most probably be left unresponsive if it didn’t).
Do let me know if there is something else I can do to help.
Ghyslain
Thanks for your patience. Really strange...
And, in the Libraries in use dialog box displayed when you click on the Libraries... button of the About dialog box, the YafaRay rendering plug-in line displays version 1.4.4 too, right?
Yes, the plug-in is reported as version 1.4.4 in About > Libraries
Would it be possible that you share a simple example which provokes your memory leak?
Sure. Is there an email address where I could send you a Google Drive link to download the file?
Please write to info@eteks.com
Hi Emmanuel,
Sorry for the long delay. I just wanted to give you an update. So, I had prepared a file for you and decided to run some more tests to avoid having you work on it needlessly. In my latest rounds of tests, I can definitely see the GC kick in where it didn’t before. As far as I can tell, this is an improvement that avoids many of the situations where I would see the application crash before.
I do still run into “Out of Application Memory” errors that either crash the program or leave it paused by the system with no other option but to Force Quit. However, I’ve come to believe that this is due to the file being too large or too complex, a problem that existed before the bugs that you have now fixed. For that reason, I assume the specific problem I was seeing is gone and that’s why I didn’t send you my file after all. I think we can consider this bug report closed.
Many thanks,
Ghyslain
Thanks for your patient attempt to reproduce this issue.
I know there can be also some out of memory errors in some other cases, probably in how some 3D models are released when not used anymore, but hopefully, the memory leaks in YafaRay renderer are fixed.