#242 melt eats more memory than available

[T2000]

Hello,

I am using kdenlive to create some vacation videos. These tend to be rather large in size, i.e. around 3 hours.

The problem that I had with melt 0.9.6 and 0.9.8 and that i still have with 6.0 is that while rendering, melt consumes all memory and starts swapping until it eventually finishes rendering.

At first I thought it might be this bug:

https://forum.kde.org/viewtopic.php?f=265&t=128869

But apparently it isnt, because I tried to render Video only in H264 with constant bit rate 1000, and even resized it from full hd to 640x384 so it should be small enough to fit into my 8GB of ram... The video itself is 2h45mins in length,

Turns out that it crashes because it runs out of memory. So I gave it like 12GB of swap as a file on my filesystem and then it successfully finishes after around 9-10 hours.

Same goes for audio, I tried to render it with mp3 128kbit constant rate, but it takes about 16 hours and needs my large 12GB swap file. Otherwise it runs out of memory and crashes.

BTW: I have an Intel(R) Core(TM) i7 CPU L 640 @ 2.13GHz

So what can I do to help you debug the problem? It obviously exists in the current 6.0 version :(

Regards,

Stefan

[Dan Dennedy]

This might not be a leak but rather a memory usage issue with a large complex project when rendered from within Kdenlive. It is more important to monitor the RAM usage of processes rather than total RAM usage and increasing swap. You do not want to rely upon swap to render as that will surely make it take a really long time.

I am running a test by composing a 3 hour, 7 minute long project consisting of 1095 shots on a timeline. I am using Shotcut to make this test, but the same concepts apply when using Kdenlive. I noticed the Shotcut process RAM usage increase up to a little over 4 GB while making this project, which is a 1080p30 project. The render resolution matters less than the project profile resolution. If I render this within Shotcut, I expect the background melt render process to use about 3 GB. I expect those two, along with the OS (including desktop environment) will consume most, if not all, of a 8 GB system. And this does not include other applications running.

One remedy for me is to first close other apps and restart Shotcut because after loading my project from a fresh start, it is only using 1.2 GB RAM (it has not had to do much processing of the project simply by loading it). Now, I can expect to have more free RAM for the render process. You can try that with Kdenlive. Another remedy with Kdenlive is to use the script option within its render dialog. Then, you can close Kdenlive to free its RAM, and render the project from the command line (terminal/console window).

I will add an update after monitoring my render test. In the meantime, you should try the second remedy above (render script) and monitor the RAM usage of the melt process. There is a third remedy that requires editing the MLT XML in the Kdenlive project file, but that is more advanced. Let us see the result of these tests.

[T2000]

Hi Dan,

thank you for your recommendations.

Though I am affraid I didn't mention that I already do your first option. I create the script for rendering with kdenlive and then close it, so I have its RAM free'ed. Unfortunately this doesn't help much as I said.

If you say that with your 3 hours test project you don't have any trouble in terms of memory usage, there must be a bug within my XML. BTW: my XML file is almost 4MB in size.

So what shall I look at inside the XML? Are there any tags which might be of trouble and that I shall edit/remove for testing? In terms of used effects I am only using composite, dissolve and fade to/from black and for audio I only use fade in/out, gain and volume (keyframeable).

I still think there might be a memory leak, because melt keeps reserving more and more memory up to when the process finishes. Actually shortly before it finishes it starts to free the RAM again. But before its only consuming more and more and will crash if it runs out of RAM. So for the purpose of having it at least finish the rendering, I gave it the swap. Of course I know that this is a bad workaround.

Regards,

Stefan

[Dan Dennedy]

I do not yet know anything specific that you should remove from your XML.

My test did not use more than 1.7 GB RAM for the melt process.
So, I need to make my test project closer to yours based on your description - before I get your project file and try to recreate the project with my own clips.
What version of Kdenlive are you using?
What do you use composite for?

Meanwhile, I would like you to test something, if you will, to see if there is a build or some library version problem. I would like you to try to render the project using the melt that comes with Shotcut because it bundles many of its own dependencies including ffmpeg. Download it from https://shotcut.org/download/. You do not need to install it; just extract it somewhere like home dir or desktop. Next, run something like:

/path/to/Shotcut/Shotcut.app/melt -verbose /path/to/your.kdenlive -consumer avformat:test.mp4

[T2000]

I am using the newest version of kdenlive 15.12.3. And the composite transition I use for picture in picture, while showing one in full screen I let another scale from 1x1 pixels in the upper left corner towards 1/4 of the screen size, keep it there for like 10-15seconds and then let it scale back to 1x1 and make it disappear.

In terms of other software versions I am running gentoo on ~amd64 setup, meaning all versions that are in Gentoo testing.

OK, I will replace the static compiled 6.0 melt from the meltframework homepage with the one from shotcut that you mentioned. Probably I'll have a try tonight and let you know afterwards.

[Dan Dennedy]

No, you cannot replace your existing installed melt with the one from Shotcut. It is not statically compiled. The melt I referred to is a wrapper/launch script that sets up the environment to refer to the libs in the Shotcut bundle. You just have to follow the directions, but I do not know how well the pre-compiled binary works on gentoo; we'll see.

[T2000]

Ok, hours later the experiment is done and failed.

while executing /path/to/Shotcut/Shotcut.app/melt -verbose /path/to/your.kdenlive -consumer avformat:test.mp4

on the console with my project file from kdenlive and the melt script from shotcut I got:

Lots of these lines before it started the actual rendering:

[avi @ 0xe541c00] parser not found for codec pcm_s16le, packets or times may be invalid.
[producer avformat] audio: total_streams 1 max_stream 1 total_channels 1 max_channels 1

And during the rendering the console was spammed with:

[swscaler @ 0x7f1d09d630e0] YUV color matrix differs for YUV->YUV, using intermediate RGB to convert
[filter avcolor_space] Invalid format glsl
[filter avcolor_space] Invalid format glsl
[swscaler @ 0x7f1d09d630e0] YUV color matrix differs for YUV->YUV, using intermediate RGB to convert

And when it finally finished it showed:

[libx264 @ 0x7f7ce00c7aa0] frame I:1222 Avg QP:18.88 size:527389
[libx264 @ 0x7f7ce00c7aa0] frame P:80105 Avg QP:22.23 size: 11607
[libx264 @ 0x7f7ce00c7aa0] frame B:208995 Avg QP:32.81 size: 415
[libx264 @ 0x7f7ce00c7aa0] consecutive B-frames: 3.5% 1.2% 0.7% 94.5%
[libx264 @ 0x7f7ce00c7aa0] mb I I16..4: 38.0% 27.3% 34.7%
[libx264 @ 0x7f7ce00c7aa0] mb P I16..4: 1.6% 0.5% 0.1% P16..4: 7.8% 0.8% 2.2% 0.0% 0.0% skip:87.0%
[libx264 @ 0x7f7ce00c7aa0] mb B I16..4: 0.0% 0.0% 0.0% B16..8: 1.2% 0.0% 0.0% direct: 0.7% skip:98.0% L0:43.4% L1:52.6% BI: 4.0%
[libx264 @ 0x7f7ce00c7aa0] 8x8 transform intra:26.7% inter:54.8%
[libx264 @ 0x7f7ce00c7aa0] coded y,uvDC,uvAC intra: 25.9% 60.9% 48.4% inter: 1.5% 2.9% 1.0%
[libx264 @ 0x7f7ce00c7aa0] i16 v,h,dc,p: 63% 27% 7% 3%
[libx264 @ 0x7f7ce00c7aa0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 26% 24% 43% 1% 1% 1% 1% 1% 2%
[libx264 @ 0x7f7ce00c7aa0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 32% 32% 17% 4% 3% 3% 3% 3% 4%
[libx264 @ 0x7f7ce00c7aa0] i8c dc,h,v,p: 39% 52% 7% 3%
[libx264 @ 0x7f7ce00c7aa0] Weighted P-Frames: Y:15.1% UV:14.7%
[libx264 @ 0x7f7ce00c7aa0] ref P L0: 76.0% 15.6% 7.9% 0.5% 0.1%
[libx264 @ 0x7f7ce00c7aa0] ref B L0: 92.2% 7.2% 0.6%
[libx264 @ 0x7f7ce00c7aa0] ref B L1: 95.0% 5.0%
[libx264 @ 0x7f7ce00c7aa0] kb/s:1373.10
[aac @ 0x7f7ce00c9220] Qavg: 968.858
Current Position: 290321
[filter avcolor_space] Invalid format glsl
[swscaler @ 0x7f7835c4a8c0] YUV color matrix differs for YUV->YUV, using intermediate RGB to convert
[filter avcolor_space] Invalid format glsl
[swscaler @ 0x7f783604d700] YUV color matrix differs for YUV->YUV, using intermediate RGB to convert

But while it was doing this, it was consuming more and more memory like before, for example:

Roughly after 2,5 hours of rendering top showed:

18210 stefan 20 0 11.414g 5.197g 8544 S 378.5 67.9 462:19.55 melt

Tasks: 196 total, 2 running, 194 sleeping, 0 stopped, 0 zombie
%Cpu(s): 62.9 us, 1.1 sy, 25.8 ni, 8.3 id, 0.9 wa, 0.7 hi, 0.2 si, 0.0 st
KiB Mem : 8029016 total, 54696 free, 5847476 used, 2126844 buff/cache
KiB Swap: 14128816 total, 12729988 free, 1398828 used. 2050416 avail Mem

And the memory percentage of 67.9 kept increasing until I went to bed. I think without the swap it would have run out of memory and crashed.

The resulting video can be started with vlc, BUT however I cannot watch it. I just contains lage green and blue parts/squares. I can see nothing! The audio sounds OK though.

So how do we tackle the XML file now? There must be something which is different from your 3 hour test project, if your rendering doesnt eat more memor thatn you have.

[Dan Dennedy]

Thanks for trying. The most interesting message in the log is "Invalid format glsl," which indicates the presence of a Movit GPU filter or transition. I was not testing with that, which is stil somewhat experimental.
Obviously, there is a lot different between your project and mine. The "remedy" involving editing the XML is only to alleviate a minor memory usage issue, but this appears more than normal/expected. In any case, if you want to try it on a copy of your file, add autoclose="1" to every instance of <playlist>.
Meanwhile, please provide your XML for me to review. If you have trouble attaching, prefer not to, or do not have a good way to host it yourself; you can use my MediaFire file-drop: http://www.mediafire.com/filedrop/filedrop_hosted.php?drop=a83739d2a8d404faf01e698b01cd04f8927f9ae22c948c04c50fd4368757f35a

Also, meanwhile, to workaround the issue, which might take a good while to debug, have you tried rendering the project in 3 or parts using the render timeline zone feature of Kdenlive? Or, if that does not work well, duplicate the project file for each part and edit each one down. If you reallly want to create a full 3 hour movie, you can render each to Lossless/HQ and then sequence them in another very simple Kdenlive project, which should not exhibit this problem.

[T2000]

I just tried to upload my project file to your mediafire upload. But once it hit 100% upload my firefox crashed O_o Hope it worked.

I also tried to anonymize it a bit but repacing the ressources with XYZ.MOV or music.mp3. Other than that its the original file.

What does autoclose="1" do on the playlist element? Will give it a try later, though.

Thanks for your help so far!

EDIT: Yes I was using the GPU effects, because the pan and zoom was only available this way, I could not select it without GPU (didnt show up in the menu list)

[Dan Dennedy]

I got the upload OK.
In MLT multitrack projects, <playlist> represents a track. Normally, MLT needs to keep an object in-memory for each clip to allow for random access such as when seeking in an app such as Kdenlive. However, for sequential processing such as when rendering this is not needed. autoclose tells MLT to remove the objects from memory when no longer needed. This is really only a minor remedy, however, because most clip-like objects are actually dual layer. The lower layer is always in a cache to only keep several in memory at any one time and to recreate on-demand as-needed. This is used not only to reduce memory usage but also to prevent hitting limits on open files and child threads permitted in a single process. So, actually, autoclose causes the upper, MLT-facing layer of the object to be discarded when done with it.</playlist>

[T2000]

I see, so autoclose actually trys to close a clip as fast as possible then? Because as it is on the playlist element I thought it will affect the track, so whenever a track is not used anymore it can be freed from ram.
But if it really discards clips from ram as fast as possible this might be of help. So I will try to let kdenlive create a new render script, then edit the playlist elements to contain autoclose=1 and then render it from the console.

BTW: I am using 2 video and 2 audio tracks for now and there will be a 3rd video track in the final video, which will contain a couple of title clips.

[Dan Dennedy]

I advise against you continuing to work on this project as it is. There is no guarantee that I will fix this. I might lose motivation; I am not exactly thrilled with the opportunity to audit someone's 4MB of XML.

[T2000]

That's fine Dan, I appreciate your effort though. I can still render it with tons of swap :(

But what worries me is that this is my second 3h vacation video project and I have been using mlt 0.9.6, 0.9.8 and now even 6.0 and ALL of them were eating more memory than available. Which is why there must be a bug somewhere... And I am willing to help you mlt guys to find it, so my future projects will benefit from it.

[T2000]

Seems like the autoclose="1" options helps! I got the whole project compiled last night. But because it was finished when I looked at it this morning, I couldnt check its memory usage :(

So I gave it another try today and let it render for 6 hours, where it "just" had eaten 36% memory. Before after 6 hours it was heavily swapping already.

But i still dont know if this option is intended by you melt guys. Because if it is, someone should forward this to the kdenlive people so they can adjust the melt script they generate to contain this autoclose feature.

[Dan Dennedy]

It is great that this worked so well for you! I did forward this to the kdenlive developers mailing list. I will still also try to make some time to investigate your project to see if something is causing a memory leak, but, of course this is slow and difficult especially with such a large project that takes long to test.