Massive memory leak in mlt-melt
Brought to you by:
ddennedy,
lilo_booter
Menu
▾
▴
#221 Massive memory leak in mlt-melt
I'm hitting a massive memory leak in mlt-melt when rendering a kdenlive project. I'm using the following packages from Fedora 20:
kdenlive-0.9.8-2.fc20.x86_64
mlt-0.9.2-1.fc20.x86_64
For some of the projects I was able to get away by just adding crazy amounts of swap space but in the end that's too slow so I moved the rendering script to a VM with 64GB memory where I was able to render my project for a while simply by throwing enough RAM at the bug to keep it well fed.
It appears that using slideshows with panning & zoom ("Ken Burns effect") makes the problem much worse. Also it seems that rendering for DVD (.VOB files) is making the issue much worse. My latest attempt now died the oom killer death after rendering just about 25% of the 92 minute long project; with top I was able to observe the process to grow at a rate of about 60MB/s.
Iow the bug is crippling even for well equipped machines, I'm seeing it both when rendering from kdenlive and from the script, both on a full blown desktop setup and the VM which is a minimal install on a VM with barely enough software to do the rendering.
I do not have Fedora 20 installed anywhere at the moment, and F19 does not
have this version of mlt package. Also, it is not exactly clear how to
reproduce. So, I would have a fair amount of work to do to attempt to
reproduce the problem, only to fail or to possibly say "Yes, seems to be a
problem with these packages or combination of versions of mlt and ffmpeg.
However, it is fixed already (or I have a new fix), but the only way to
give it to you is through a nightly build." So, please test now against a
nightly build:
http://builds.meltytech.com/kdenlive/kdenlive-fedora17-x86_64-20141207.tar.bz2
After extracting the archive, run the start-kdenlive launch script. It is
not a package and no installation is required. Most multimedia dependencies
such as ffmpeg and friends are included.
OK, do you think that if I load a decently long 1920x1080 @29.97 video, simply place that on the timeline and render it to DVD/PAL 16:9; that I should be able to reproduce the problem?
On Mon, Dec 08, 2014 at 05:28:18PM +0000, Dan Dennedy wrote:
I'm fairly convinced. I just did the following:
I download https://www.youtube.com/watch?v=Kkloy7YkN4s with youtube-dl.
This yielded a .webm file. This video is a 480x360@29.97Hz video of
about 11MB.
Then start kdenlive. The project's video profile is HD/1080p 29.97 fps.
I then added the webm file as a clip and pull that clip on the first video
track. Then I tried to render the video. I wanted an output format that
changed the frame rate but ideally some output format different from .VOB
which I had used previously to demonstrate that the issue was not specific
to a particular codec or container. I thus picked HDV/PAL 720 25p.
I allowed the mlt-melt process to grow to a size of about 10GB before
killing it.
Compared the project in which I originally encountered the issue this
experiment changes much everything except the projects video profile and
the frame rates of the input and output video material.
Ralf
OK, thanks, I will definitely look into it. In the meantime, you can workaround the problem by using a 25 fps project profile. The profile does not need to match the inputs. Rather, it should be chosen based on the highest resolution of your intended outputs. If you are wanting to output both 25 and 29.97 fps, then you should use a 29.97 project profile, but if you only intend to output @25, then you can use a 25 fps project profile. There are different approaches and places to the framerate conversion based upon whether the framerate is different than the source at the project or render profile.
This is fixed in MLT git commit b86e84e.
In several hours, you will be able to download a new nightly build to confirm it:
http://builds.meltytech.com/kdenlive/kdenlive-fedora17-x86_64-20141209.tar.bz2
On Tue, Dec 09, 2014 at 07:40:27AM +0000, Dan Dennedy wrote:
With this build I observed my test case to consume about 1.3GB VSIZE
and my original project of about 72 minutes renders in roughly 90 minutes
which is much faster than before. After about 80% of the time when I
had leave the melt process had inflated itself to about 4GB. These are
figure I absolutely can live with but the slow inflation suggests there
still might be another leak?
Thanks!
Ralf
On Tue, Dec 09, 2014 at 07:40:27AM +0000, Dan Dennedy wrote:
Using that build rendering to a DVD/PAL 16:9 VOB (the non-CBR) profile
resulted in a surprisingly small .VOB file of just 1.2GB even though I
cranked up the video quality and audio bandwidth settings to maximum.
Was wondering if that is because there is not a whole lot of action
in my project - it's a classic concert with fairly constant camera
position. For a length of 1:12:11,13 long project at 8Mbit/s + 192kbit/s
I was expecting around 3.8GB. Anyway, mplayer claims the video is
25.000 fps / 8000.0 kbps.
The worse is that the index of the file is pretty broken. When I play the
.VOB in mplayer, mplayer claims will claim in the OSD that the video is just
20:19 long, seeking often lead mplayer to exit but if I just allow it to
play along, it is will show a black screen and no sound after the end of
the actual content and play for something like 100 minutes.
Ralf
VSIZE is a bad way to measure memory utilization. RSS is a little better. And there are others: http://lwn.net/Articles/230975/
You might be best off using your desktop environment's system monitor tool. I think their developers try to use best practices to make best use of available metrics.
This ticket is not about Kdenlive's DVD render profiles. In any case, VOB files do not have an index. I do not know why you are surprised to get a much shorter file when you chose non-CBR. Why choose DVD and then use non-CBR and increase the bitrates? That is not going to be DVD compliant.