#199 Very big memory usage !!!

v2.5x
closed-wont-fix
Internals (76)
5
2014-08-18
2008-08-13
JPSDR
No

Hello.

Here the problem :
First, i must say that the problem occurs only
with video encoded with lagarith codec in YV12 mode.
The problem doesn't occur with video encoded with
lagarith codec in other mode (YUY2), and it neither
occurs with other video encoded in YV12 (DivX, H264).
Even if i found sometime the memory used a little high.

I've already notify this problem to the person
doing the lagarith codec, but it's hard to say which
is responsible. I suggest you get in touch with him
to see where the problem lies.

My PC : Dual core (note core2duo) with 2Go of
mermory under Windows XP SP3.

When i open an avisynth script under virtual dub,
with the only command inside is :
AviSource("File.avi")
if the avi file is encoded with lagarith codec in
YV12 mode, the memory used go up until around 900Mo !!!!
This occur with V2.5.8 RC2 and RC3.
I've tested with V2.5.7, the memory used go up
(only... ^_^) to around 450Mo...
Opening directly the file in VDub, not using an
avisynth script, memory used is only 45Mo.
You must have a long enough file to reproduce the
problem, as it take around 20s for the memory used
to stabilise and stop growing.

I realy think there is a problem somewhere !!!

Discussion

  • Ian Brabham

    Ian Brabham - 2008-08-13
    • assigned_to: nobody --> ianb1957
    • milestone: --> v2.5x
    • labels: --> Internals
    • status: open --> pending-accepted
     
  • Ian Brabham

    Ian Brabham - 2008-08-13

    Logged In: YES
    user_id=673887
    Originator: NO

    The numbers although a little high are more or less as expected, particularly the change from 2.5.7 to 2.5.8. A fresh boot of a 2GB clean XP could have MEMORYSTATUS.dwAvailPhys of 1.5GB or greater. 2.5.7 would take 25% of this for the Cache, say 384MB, plus the 45MB for VDub gives 429Mb, leaves 20Mb for other stuff. 2.5.8 would take 50% less 64MB, say 704MB. plus 45MB for VDub, gives 749Mb, yeah 151Mb for other stuff, a bit out, maybe ...dwAvailPhys was a bit higher.

    The output cache will hold upto 200 frame entries, so 704/200=3.5MB/frame, which fits for YV12 HD.

    I have already decided to revert to using the 2.5.7 default cache size of 25% MEMORYSTATUS.dwAvailPhys for the released 2.5.8, I am just waiting for some feedback from other testers on other issues before I put up a RC4 with this change.

    If you add a SetMemoryMax(384) to the start of your script, the 2.5.8 memory size results should then be very close to those of the 2.5.7. Also running several instances at once, or with some other memory gobbler, should see each successive instance use less memory as it only takes 50% of the 50% remaining of the 50% remaining of the ...

    The Avisynth cache only uses memory that nothing else is using, so the default behaviour is not normally a problem. If it is a problem, for simple scripts add a SetMemoryMax(50) or perhaps smaller to the start of your script.

    In 2.5.8 you can do X=SetMemoryMax(0) in your script to get the actual memory_max value being used. Use SubTitle() or WriteFileStart() to access the value of X.

     
  • JPSDR

    JPSDR - 2008-08-13

    Logged In: YES
    user_id=1035566
    Originator: YES

    Ok, but, i think there still is something odd.

    First, it's not HD but SD, it's 720x480,in all the cases (YV12 or YUY2).
    I don't do any process, what i do is :
    Opening the avisynth script with the only command : AviSource("File.avi").
    With VDub, i don't do any filter or anything. I only do the following :
    Opening the YV12 Lagarith file (or YUY2 Lag. or huffyuv), and saving it in huffyuv YUY2.

    When the input file is an YV12 Lagarith, memory used is around 900Mo.
    When the input file is an YUY2 Lagarith or huffyuv, memory used is around 200Mo.
    In all cases, it's always the "same" input file.

    You said that the buffer is around 200 frames. If I follow your lead, that means, for
    720x480 YUY2, that make around 140Mo. Nothing suspicious here.
    For 720x480 YV12, it's make around 100Mo for 200 frames. With 900Mo used, there is
    something wrong !!! The problem doesn't occur with others codec YV12 files.
    So, i insist, but i think there is something wrong. Get in touch with the guy who make
    lagarith, maybe there is some incorect data/information provided by the codec wich cause
    avisynth to explose memory, or maybe there is some tricky bug in avysinth, maybe etc...

     
  • JPSDR

    JPSDR - 2008-08-13
    • status: pending-accepted --> open-accepted
     
  • Ian Brabham

    Ian Brabham - 2008-08-14
    • status: open-accepted --> pending-remind
     
  • Ian Brabham

    Ian Brabham - 2008-08-14

    Logged In: YES
    user_id=673887
    Originator: NO

    Can you please tabulate the final process memory size for both 2.5.7 and 2.5.8-RC3 when rendering the same YV12 Lagarith file and a similar YUY2 Lagarith file with these 2 scripts.

    SetMemoryMax(256)
    AviSource("File.avi")

    SetMemoryMax(32)
    AviSource("File.avi")

     
  • JPSDR

    JPSDR - 2008-08-14

    Logged In: YES
    user_id=1035566
    Originator: YES

    Hello.

    I've made tests with a 3000 frames 720x480 file, encoded in lagarith YUY2 and YV12.
    Process time was around 20s-30s. Memory used is the information provided by windows
    task manager of the mermory used by Virtualdub.

    Avisynth 2.5.8-RC3 :
    YUY2 :
    SetMemoryMax(32) : Memory grow slowy, stabilise at 50Mo.
    SetMemoryMax(256 & 384) : Memory grow slowy, memory used 72Mo at the end of the process,
    but wasn't yet stabilised.
    YV12 :
    SetMemoryMax(32) : Memory used : 75Mo, instantaneous. Stabilised.
    SetMemoryMax(256) : Memory used : 347Mo, grow very quickly. Stabilised.
    SetMemoryMax(384) : Memory used : 444Mo, grow very quickly. Stabilised.

    Avisynth 2.5.7 :
    YUY2 :
    SetMemoryMax(32) : Memory used : 60Mo, instantaneous. Stabilised.
    SetMemoryMax(256 & 384) : Memory used : 96Mo, grow very quickly. Stabilised.
    YV12 :
    SetMemoryMax(32) : Memory used : 96Mo, grow very quickly. Stabilised.
    SetMemoryMax(256) : Memory used : 320Mo, grow very quickly. Stabilised.
    SetMemoryMax(384) : Memory used : 488Mo, grow very quickly. Stabilised.

    As i see it, different causes :
    - In YV12 only, Lagarith report some incorrect data to avisynth. Problem is only
    on Lagarith side.
    - Some tricky bug in avisynth, triggered by Lagarith in YV12 mode. Problem is only
    on avisynth side. It can't be a "big obvius bug", beacause problem doesn't occor with
    others YV12 source files (DivX,h264(ffdshow),dgmpeg).
    - A mix of the two previous things. Problem is coming from both sides.

    My very first guess swings more on the first choice, but i think it's realy on your
    side you can check/verify that.

     
  • JPSDR

    JPSDR - 2008-08-14
    • status: pending-remind --> open-remind
     
  • Ian Brabham

    Ian Brabham - 2008-08-15

    Logged In: YES
    user_id=673887
    Originator: NO

    And of course the obvious thing I should have asked earlier, which version of Lagarith are you running and what is the url you actually used to downloaded it from.

     
  • Ian Brabham

    Ian Brabham - 2008-08-15
    • status: open-remind --> pending-remind
     
  • JPSDR

    JPSDR - 2008-08-15
    • status: pending-remind --> open-remind
     
  • JPSDR

    JPSDR - 2008-08-15

    Logged In: YES
    user_id=1035566
    Originator: YES

    Hello.

    I'm using the latest Lagarith codec version : 1.3.17.
    Here the link : http://lags.leetcode.net/codec.html

     
  • Ian Brabham

    Ian Brabham - 2008-08-24

    Logged In: YES
    user_id=673887
    Originator: NO

    I have found no memory leaks either Avisynth or Lagarith with the simple script provided. The Avisynth cache does however grow to approximately the SetMemoryMax limit. The default limit is currently quite high and has been causing problems, the next release candidate will have a lower default.

    In the YUY2 case the cache heuristics notice no cache hits and start early buffer reuse. This stops adding new frames to the cache. This happens after about 75 to 100 events. Corresponding to between 49Mb and 66Mb of cache.

    YV12 sources, 720 pixel wide, have a 360 byte chroma plane rowsize. As this is NOT mod16 aligned, the internal AlignPlanar filter is added to the graph, to provide mod16 pitch aligned frames. The AlignPlanar filter does not use the cache heuristics, so no early release of frame buffers occurs.

    Relaxing the alignment constraints for chroma planes to mod8, i.e 720 pixels wide YV12, will not require the AlignPlanar filter to be included. This will then show similar behaviour to the YUY2 case for YV12 720 pixel wide sources.

    SetMemoryMax(1024)
    AviSource("Lag_720x480_YV12.avi", False, "YV12")
    SetPlanarLegacyAlignment(True)

    With the above example the cache starts early reuse mode after about 75 to 100 frames or about 37Mb to 50Mb. Without Legacy Alignment the cache will grow to 1024Mb if that much free memory was available. Mod4 width source will of course always exhibit the original behaviour.

    Under normal conditions it is intended that the cache track as many frames as will fit within the SetMemoryMax limit. There are some heuristics that limit this.

     
  • Ian Brabham

    Ian Brabham - 2008-08-24
    • status: open-remind --> pending-wont-fix
     
  • SourceForge Robot

    • status: pending-wont-fix --> closed-wont-fix
     
  • SourceForge Robot

    Logged In: YES
    user_id=1312539
    Originator: NO

    This Tracker item was closed automatically by the system. It was
    previously set to a Pending status, and the original submitter
    did not respond within 14 days (the time period specified by
    the administrator of this Tracker).

     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks