#199 Very big memory usage !!!

v2.5x
closed-wont-fix
Ian Brabham
Internals (76)
5
2014-08-18
2008-08-13
JPSDR
No

Hello.

Here the problem :
First, i must say that the problem occurs only
with video encoded with lagarith codec in YV12 mode.
The problem doesn't occur with video encoded with
lagarith codec in other mode (YUY2), and it neither
occurs with other video encoded in YV12 (DivX, H264).
Even if i found sometime the memory used a little high.

I've already notify this problem to the person
doing the lagarith codec, but it's hard to say which
is responsible. I suggest you get in touch with him
to see where the problem lies.

My PC : Dual core (note core2duo) with 2Go of
mermory under Windows XP SP3.

When i open an avisynth script under virtual dub,
with the only command inside is :
AviSource("File.avi")
if the avi file is encoded with lagarith codec in
YV12 mode, the memory used go up until around 900Mo !!!!
This occur with V2.5.8 RC2 and RC3.
I've tested with V2.5.7, the memory used go up
(only... ^_^) to around 450Mo...
Opening directly the file in VDub, not using an
avisynth script, memory used is only 45Mo.
You must have a long enough file to reproduce the
problem, as it take around 20s for the memory used
to stabilise and stop growing.

I realy think there is a problem somewhere !!!

Discussion

1 2 > >> (Page 1 of 2)
  • Ian Brabham
    Ian Brabham
    2008-08-13

    • assigned_to: nobody --> ianb1957
    • milestone: --> v2.5x
    • labels: --> Internals
    • status: open --> pending-accepted
     
  • Ian Brabham
    Ian Brabham
    2008-08-13

    Logged In: YES
    user_id=673887
    Originator: NO

    The numbers although a little high are more or less as expected, particularly the change from 2.5.7 to 2.5.8. A fresh boot of a 2GB clean XP could have MEMORYSTATUS.dwAvailPhys of 1.5GB or greater. 2.5.7 would take 25% of this for the Cache, say 384MB, plus the 45MB for VDub gives 429Mb, leaves 20Mb for other stuff. 2.5.8 would take 50% less 64MB, say 704MB. plus 45MB for VDub, gives 749Mb, yeah 151Mb for other stuff, a bit out, maybe ...dwAvailPhys was a bit higher.

    The output cache will hold upto 200 frame entries, so 704/200=3.5MB/frame, which fits for YV12 HD.

    I have already decided to revert to using the 2.5.7 default cache size of 25% MEMORYSTATUS.dwAvailPhys for the released 2.5.8, I am just waiting for some feedback from other testers on other issues before I put up a RC4 with this change.

    If you add a SetMemoryMax(384) to the start of your script, the 2.5.8 memory size results should then be very close to those of the 2.5.7. Also running several instances at once, or with some other memory gobbler, should see each successive instance use less memory as it only takes 50% of the 50% remaining of the 50% remaining of the ...

    The Avisynth cache only uses memory that nothing else is using, so the default behaviour is not normally a problem. If it is a problem, for simple scripts add a SetMemoryMax(50) or perhaps smaller to the start of your script.

    In 2.5.8 you can do X=SetMemoryMax(0) in your script to get the actual memory_max value being used. Use SubTitle() or WriteFileStart() to access the value of X.

     
  • JPSDR
    JPSDR
    2008-08-13

    Logged In: YES
    user_id=1035566
    Originator: YES

    Ok, but, i think there still is something odd.

    First, it's not HD but SD, it's 720x480,in all the cases (YV12 or YUY2).
    I don't do any process, what i do is :
    Opening the avisynth script with the only command : AviSource("File.avi").
    With VDub, i don't do any filter or anything. I only do the following :
    Opening the YV12 Lagarith file (or YUY2 Lag. or huffyuv), and saving it in huffyuv YUY2.

    When the input file is an YV12 Lagarith, memory used is around 900Mo.
    When the input file is an YUY2 Lagarith or huffyuv, memory used is around 200Mo.
    In all cases, it's always the "same" input file.

    You said that the buffer is around 200 frames. If I follow your lead, that means, for
    720x480 YUY2, that make around 140Mo. Nothing suspicious here.
    For 720x480 YV12, it's make around 100Mo for 200 frames. With 900Mo used, there is
    something wrong !!! The problem doesn't occur with others codec YV12 files.
    So, i insist, but i think there is something wrong. Get in touch with the guy who make
    lagarith, maybe there is some incorect data/information provided by the codec wich cause
    avisynth to explose memory, or maybe there is some tricky bug in avysinth, maybe etc...

     
  • JPSDR
    JPSDR
    2008-08-13

    • status: pending-accepted --> open-accepted
     
  • Ian Brabham
    Ian Brabham
    2008-08-14

    • status: open-accepted --> pending-remind
     
  • Ian Brabham
    Ian Brabham
    2008-08-14

    Logged In: YES
    user_id=673887
    Originator: NO

    Can you please tabulate the final process memory size for both 2.5.7 and 2.5.8-RC3 when rendering the same YV12 Lagarith file and a similar YUY2 Lagarith file with these 2 scripts.

    SetMemoryMax(256)
    AviSource("File.avi")

    SetMemoryMax(32)
    AviSource("File.avi")

     
  • JPSDR
    JPSDR
    2008-08-14

    Logged In: YES
    user_id=1035566
    Originator: YES

    Hello.

    I've made tests with a 3000 frames 720x480 file, encoded in lagarith YUY2 and YV12.
    Process time was around 20s-30s. Memory used is the information provided by windows
    task manager of the mermory used by Virtualdub.

    Avisynth 2.5.8-RC3 :
    YUY2 :
    SetMemoryMax(32) : Memory grow slowy, stabilise at 50Mo.
    SetMemoryMax(256 & 384) : Memory grow slowy, memory used 72Mo at the end of the process,
    but wasn't yet stabilised.
    YV12 :
    SetMemoryMax(32) : Memory used : 75Mo, instantaneous. Stabilised.
    SetMemoryMax(256) : Memory used : 347Mo, grow very quickly. Stabilised.
    SetMemoryMax(384) : Memory used : 444Mo, grow very quickly. Stabilised.

    Avisynth 2.5.7 :
    YUY2 :
    SetMemoryMax(32) : Memory used : 60Mo, instantaneous. Stabilised.
    SetMemoryMax(256 & 384) : Memory used : 96Mo, grow very quickly. Stabilised.
    YV12 :
    SetMemoryMax(32) : Memory used : 96Mo, grow very quickly. Stabilised.
    SetMemoryMax(256) : Memory used : 320Mo, grow very quickly. Stabilised.
    SetMemoryMax(384) : Memory used : 488Mo, grow very quickly. Stabilised.

    As i see it, different causes :
    - In YV12 only, Lagarith report some incorrect data to avisynth. Problem is only
    on Lagarith side.
    - Some tricky bug in avisynth, triggered by Lagarith in YV12 mode. Problem is only
    on avisynth side. It can't be a "big obvius bug", beacause problem doesn't occor with
    others YV12 source files (DivX,h264(ffdshow),dgmpeg).
    - A mix of the two previous things. Problem is coming from both sides.

    My very first guess swings more on the first choice, but i think it's realy on your
    side you can check/verify that.

     
  • JPSDR
    JPSDR
    2008-08-14

    • status: pending-remind --> open-remind
     
  • Ian Brabham
    Ian Brabham
    2008-08-15

    Logged In: YES
    user_id=673887
    Originator: NO

    And of course the obvious thing I should have asked earlier, which version of Lagarith are you running and what is the url you actually used to downloaded it from.

     
  • Ian Brabham
    Ian Brabham
    2008-08-15

    • status: open-remind --> pending-remind
     
1 2 > >> (Page 1 of 2)