Menu
▾
▴
#12 Improved AviSynth Buffering
Here's my patch for ffdshow's AviSynth filter that implements buffering to allow AviSynth access to more than just the current frame. It also allows an AviSynth script to adjust the number of frames, as long as this happens evenly distributed across the whole video.
I've added 4 new controls to the AviSynth page:
* Buffer back: number of previous frames to keep around, i.e. the number of frames AviSynth can look back.
* Buffer current: a top limit for the number of frames that will be buffered for a single output frame; the actual value for this is calculated by dividing the number of frames going in by the number of frames going out - but if your video is a million frames and you request only one, without a limit one million frames would be buffered, which would probably just crash the player... :)
* Buffer ahead: number of frames to buffer ahead before feeding anything to AviSynth, i.e. the number of frames AviSynth can look ahead.
* Apply pulldown: applies a 3:2 pulldown to the video according to the interlace flags that the frames coming in have. Should be activated for IVTC, bobbing and other stuff that drops or adds frames. Deactivate it if you want progressive frames straight from a DVD, unless it was flagged incorrectly.
If you get jerky video, that most of the time means the buffering values are to small.
For example, the following script needs to buffer 1 frame back and 10 ahead:
TFM(order=1)
TDecimate(mode=1,hybrid=1)
The default 10/10/10 is probably a good compromise; and keeping more back buffers only uses more memory but doesn't slow things down. Also, you should probably use something like SetMemoryMax(16) in your script to limit the amount of memory AviSynth uses.
I've also made a minor modification to the libmpeg2 codec to flag sequence starts and ends as fieldtype flags so the filter can flush what's left at the end of a sequence - without that, you'd lose several frames at the end of the video, which wouldn't be so bad, but static DVD menus that are only 1 frame long would never get displayed without this.
In other news, I'm probably blowing the AviSynth environment away a bit too often, but that was the only way I could think of to clear AviSynth's cache - with some modifications at least seeking could probably be made to avoid this.
Also, I probably should check out how to change the reported FPS value in the video headers, but it does work as long as the timestamps are correct...
np: Boards Of Canada - Everything You Do Is A Balloon (Skampler)
Patch against rev. 1007
Logged In: YES
user_id=975750
Originator: YES
File Added: Improved_AviSynth_Buffering_2.diff
Logged In: YES
user_id=975750
Originator: YES
File Added: Improved_AviSynth_Buffering_2.diff
Logged In: YES
user_id=975750
Originator: YES
File Added: Improved_AviSynth_Buffering_2.diff
Updated patch to fix problems with Overlay Renderer, still against rev. 1007
Logged In: YES
user_id=975750
Originator: YES
(Ack. Sorry about the attachment mess - using the back button after uploading a file probably isn't the way to go on SF...)
haruhiko_yamagata noticed that using the Overlay Renderer the graph wouldn't run when ffdshow started buffering without delivering the first frame, so I now just more or less pass the first frame through (that is, I'm requesting a far away frame from AviSynth with just one frame buffered; that will give about the same result as with the original, unpatched implementation) and that makes the graph tick even with the Overlay Renderer.
I've also made the filter reset AviSynth whenever it's disabled as it doesn't make much sense to work on frames that have been long gone when the filter is reactivated.
Logged In: YES
user_id=1276834
Originator: NO
bug with new code: Extra frames in beginning (duplicates of first frames)
Number of extra very first frames= (buffer ahead)+1
Try script:
showframenumber()
and load decoded ffdshow video in virtualDub.
first frames returns number 1071079 (not 0).
Logged In: YES
user_id=975750
Originator: YES
> first frames returns number 1071079 (not 0).
This is by design, and not a bug.
The filter works by buffering a number of frames before requesting frames from AviSynth, so AviSynth can access more than just the current frame.
The problem with buffering is that to buffer frames, I have to take frames coming in and not return a processed frame until I've buffered enough frames, but several other DirectShow filters (including the Overlay Mixer) won't change into running state unless they get a frame, which makes the graph hang in this case.
To prevent that, I'm requesting a frame from AviSynth after buffering the first frame (which means that no matter which frame AviSynth requests, it will always get the single buffered frame) that I pass onwards. This is done since the ffdshow AviSynth filter doesn't know what exactly the script does, so I can't just pass the source frame through.
Since there's several filters that will start calculating metrics for several of the following frames (like TDecimate), I can't just request frame 0 as that will mess up the metric calculations for the next few frames, so I'm requesting frame 10810800 , which is the frame beyond the last frame of the "faked" ffdshow video source. I guess AviSynth clamps this to 10810799, but that shouldn't really matter.
What matters is that the next frame requested will be number 1, which should tell any filter that the metrics it has calculated when producing the last frame are invalid and need to be recalculated. Sure, I could just restart AviSynth, but that takes more time and I don't think it's worth it.
> bug with new code: Extra frames in beginning (duplicates of first frames)
> Number of extra very first frames= (buffer ahead)+1
I still can't think of a reason why this would happen, but I'll have a look at it. Still, when just watching a video I think this would hardly be noticed - and if you're encoding video do yourself a favour and use AviSynth directly...
Logged In: YES
user_id=975750
Originator: YES
I just tried this:
Using "Info()" as the AviSynth script, I turned on the AviSynth filter and put ffdshow's "Grab" filter (which writes anything that gets passed into it into image files) right next to it. Then I set it to grab frames 0 to 49 and to write them to BMP files in c:\Temp
That resulted in 31 files in c:\Temp; they didn't have consecutive frame numbers (note: that's ffdshow's internal frame numbering, not mine) in the beginning, so ffdshow might for some reason skip a few frames in the beginning, but there wasn't a single duplicate image coming out of my AviSynth filter in there.
I'll try something similar with graphedit and the dump DirectShow filter when I get home. I've just noticed that ffdshow seems to skip every third frame in the beginning with this XviD AVI file whether my filter is used or not - maybe that's something to do with this, and why there were only 31 images captured.
I'll investigate this further...
Logged In: YES
user_id=975750
Originator: YES
Actually - if the frame size changes when playback starts or something else happens that causes the filter to reset (and thus simply dumping the buffered frames) I wouldn't be surprised that you get to see a first frame a few times in a row, since the AviSynth instance and the buffers are torn down and created anew in that case.
This also drops the frames that had been buffered by then as I need to know beforehand that we've hit the end of something to flush the buffered frames out - which currently only works with MPEG2 and libmpeg2 because I modified it to return an end-of-stream marker on the last image.
But I'd rather first concentrate on the filter working reliable over the course of a movie or series episode and then try to minimize the number of times the filter gets reset - personally I'd rather have a smooth viewing experience with the first and/or last few frames missing than having to put up with telecine judder.
Watching a DVD with TIVTC I noticed that things got jerky after a few minutes (or a few thousand frames) as if the buffered frames and the frames AviSynth was requesting went out of sync, so I'm going to fix that first.
Logged In: YES
user_id=1276834
Originator: NO
thanks for details.
May be one-frame bug somewhere.
I try to use HUFFYUV codec (enabled in ffdshow) for testing. Details:
I take some video, open it with avisynth script with INFO(),
end encode it to HUFFYUV format. So I have hardcoded framenumbers.
Now I open this huff video with ffdshow wit internal avisynth script BLUR(1) and pulldown disabled.
I see in VirtualDub or play in MediaPlayer.
(avisynth enabled in vfw or ds interface).
And I see 11 duplicates. Can you confirm?
I am interested in realtime capture with ffdshow MPEG2 in AVI codec from my analog TV tuner. I want use fast avisynth filters in ffdshow for preprocessing.
Logged In: YES
user_id=1276834
Originator: NO
thanks for details.
May be one-frame bug somewhere.
I try to use HUFFYUV codec (enabled in ffdshow) for testing. Details:
I take some video, open it with avisynth script with INFO(),
end encode it to HUFFYUV format. So I have hardcoded framenumbers.
Now I open this huff video with ffdshow wit internal avisynth script BLUR(1) and pulldown disabled.
I see in VirtualDub or play in MediaPlayer.
(avisynth enabled in vfw or ds interface).
And I see 11 duplicates. Can you confirm?
I am interested in realtime capture with ffdshow MPEG2 in AVI codec from my analog TV tuner. I want use fast avisynth filters in ffdshow for preprocessing.
Logged In: YES
user_id=975750
Originator: YES
I'm afraid I don't see any duplicates in MPC; I do see ffdshow's frame count in the OSD jump from 1 to 12, though - but that's normal, as the frame count is increased with every frame that's decoded.
I do see duplicates in VirtualDub, but those are definitely not generated by ffdshow itself - if you look at the code, my filter works by taking the first <buffer ahead> + <something> input frames and only sending a single frame further up the filter chain on the first request. Until it has buffered enough, it just returns without doing anything else.
Wherever the duplicates come from (probably a buffer in VirtualDub or the ffvfw DLL that isn't overwritten if no frame is produced), I don't know. But that's really out of the realm of my filter.
But then again, this kind of buffering really isn't meant for VfW - why not use an AviSynth script directly with DirectShowSource and a GRF file? (This is totally out of thin air, as I don't have anything I could capture with...)
Still, ffdshow is a DirectShow filter, after all, and the ffvfw part can only do so much to cover this up IMHO... of course, you can decrease the number of buffers if you don't use temporal filtering, but I guess that's what you're after.
Setting up an AVI Source -> ffdshow decoder -> ffdshow encoder -> AVI Mux -> File Writer chain in GraphEdit with a 100 frame video produced the way you wrote above produces a video with no duplicates at the beginning, but with 11 frames missing at the end since for HuffYUV there's currently no notification that the stream has ended as I added for MPEG2, but I guess that's not really a problem for TV captures as those usually are longer than they need to be.
In closing, I'm afraid the best solution I can offer to you would be chopping off the repeated frames after encoding, as I really can't do much against them when you're using VfW, and I' afraid I can't reproduce them using DirectShow....
Logged In: YES
user_id=1276834
Originator: NO
If I play my Huff avi with previously hard-coded framenumbers (info),
I see same many duplicates of frame 0 with direstshow in MPClassic .
I use buffers 10-10-90, disabled pulldown, script:
BLUR(1.0)
Even if I decrease number of buffers to 0-0-0 I still have 1 extra duplicate frame.
Logged In: YES
user_id=975750
Originator: YES
Having thought this over again, I can assure you that this is never going to work without repeated frames when you use VfW (i.e. the ffvfw codec) to encode a video and frames are being buffered - there's just nothing I can do about it, since that's how VfW (which is ancient, technology-wise) works:
The encoding application (which includes VirtualDub, I'm sorry to say) hands the VfW codec an image and immediately expects to get an encoded frame back that is then added to the output file. Since I don't return frames for a while after the first frame, the old contents of the output buffer are written to the file several times as there was no new frame to replace them.
You absolutely positively *need* to either use AviSynth via an AVS file and access your DirectShow source using DirectShowSource and a GraphEdit graph file (as there is no such 1:1 restriction there), or you need to use an encoding application that either directly supports using a DirectShow graph as input or that completely uses a DirectShow graph to encode your video - you can, for example, do that in GraphEdit.
I'll modify my filter so buffering can be turned off, but then it'll work exactly like the current AviSynth filter without my patch, so using any AviSynth filter that requests more than the current frame (like TIVTC, Decomb or temporal filters) just plain won't work correctly.
Like I said - as long as you're using a VfW codec and encoding application, there's nothing I can do; and that's my final answer, I'm afraid - the only way to make this even remotely work with VfW would be changing the encoding application to not store the repeated frames in the file.
Logged In: YES
user_id=1276834
Originator: NO
Thanks.
I usully capture with directshow application like virtualVCR, etc.
As I wrote, I see similar delay (duplicates) in MPC (Directshow) too.
I now discover that is is partially dependent on CPU load (and format).
I test sort (363 frames) avi videofile 768x576 with hardcoded framenumbers (info).
I test 10,10,90 frame buffer settings.
if i use avisynth scipt as "showframenumber()", its frame numbers are different by 1 from hardcoded at first play (but sometimes they are coincide at second repeat play).
OSD numbers are usually different by Ahead+1.
I wonder why 1 frame dif?
And why 0,0,0 settings produce 1 frame dif too at first play pass?
Probably it is a bug.
IMHO, this settings must reproduce old algo.
Generally, I agree, that some frames lag is permitted.
What about posting this topic to avisynth forum for more wide discussion?
Logged In: YES
user_id=975750
Originator: YES
File Added: Improved_AviSynth_Buffering_3.diff
Updated patch to fix an off-by-one error, make 0/1/0 do no buffering and return the source fieldtype, against rev. 1048
Logged In: YES
user_id=975750
Originator: YES
This latest patch should really do no buffering (but still some copying that could be avoided) at all if you use 0/1/0 and uncheck "Apply Pulldown". I tried it on a 100-frame XviD file that contained the output of "Info()" and added a second "Info()" over that while encoding, and except for the frame number of frame 0 (which, of course, was 10.8-million-and-a-bit) they were identical.
Of course, if you use a value bigger than 0 for the number of frames to buffer ahead you will get a result that's missing this number of frames at the end, as the filter isn't notified that it should flush the remaining frames - but there were no duplicates at the beginning of the file.
Also, no matter what you do - always use as little buffering as possible/neccessary: using 10/10/90 for normal playback in MPC makes VMR9 drop lots of frames - obviously there's only so much time it's willing to wait for a frame before it gets dropped. It shouldn't matter for encoding, though.
One change I forgot to mention was that I added some code to notify the image filters that the graph was stopped when encoding ends, as I need to reset everything if encoding should work more than once in a row.
Logged In: YES
user_id=1276834
Originator: NO
I still have 1 frame number difference with new build for first prayback MPC.
So, 0/1/0 fuferss is not equivalent to old versions.
Logged In: YES
user_id=975750
Originator: YES
Ummm... in case you're doing it the following way:
Open file
Click stop
Click pause
Hit cursor right to step forward a frame several times
This always gives me a duplicate first frame, no matter which file or codec I'm using - try opening your XviD source in MPC using the XviD decoder and you'll get the same...
Also, since audio and video are synced up using timestamps in DirectShow a single duplicate frame (which, other than the above, I just can't reproduce) is hardly a problem, so I honestly don't understand all this fuss...
Moreover, please try this: drag ffdshow's "Grab" filter right below the AviSynth filter. Set it up to capture the first few frames. If the resulting images contain repeated frames then my filter is actually producing them. If there isn't, they're caused by something else.
Logged In: YES
user_id=1276834
Originator: NO
Sorry for disturb.
No, I did not use frame step.
Here is my method used:
1. I created short xvid video clip with hardcoded framenumbers by avisynth info(), save as xvidn.avi. PAL 720x576, 25fps. Tried also 360x288. Tried also Huffyuv.
2. Setup ffdshow Avisynth options as buffers 0,1,0 (or 10,10,10) , pulldown OFF, script:
crop(24,0,0,0).info()
or similar, to show both old and new frame numbers.
3. Open xvidn.avi in MPC by clicking or via File-open menu.
I see video with two vareing numbers : old and new. New is lesser by 1. We can press Pause to see this too.
NOTE: If I pres Stop button, and than Play button in MPC, these frame numbers become equal!
So, they are differ only at first pass.
If I re-open video, I see different numbers again.
It is not dependent on video player (WMP6.4, WMP9).
May be it is related to very high processor load (100%) at start of play (than decrease to about 40 %.
May be you use have more fast system?
I also no tried your GRAB method.
here is results (for first pass):
frame 10810799 of 10810800 ( really old frame 0)
frame 10810799 of 10810800 ( realle old frame 1) interesting, frames with same number are NOT duplicates.
frame 1 of 10810800 ( really old frame 2)
...
But when I use non-patched ffdshow build, these (old and new) frame numbers are always equal.
Now I douwload FFDshow source from SVN, apply your patch and try look to source and create binary.
I still did not understand all.
seems, this line must be at little above place:
applyPulldown=!!oldcfg.applyPulldown;
(but it is not related probably).
Logged In: YES
user_id=975750
Originator: YES
> frame 10810799 of 10810800 ( really old frame 0)
> frame 10810799 of 10810800 ( realle old frame 1) interesting,
> frames with same number are NOT duplicates.
> frame 1 of 10810800 ( really old frame 2)
The frame number AviSynth reports is totally meaningless - it is reset to zero everytime the filter resets AviSynth and thus throws out anything it had buffered until that point, which happens on a number of occasions.
Some of those occasions are when the stream gets seeked, when it gets stopped and restarted, when the end of an MPEG2 segment is encountered and when any property of the input frames or the filter's configuration changes.
So unless the resulting frames are really duplicates everything is working as designed. Also, keep in mind that I meant this filter to be used while playing back something where dropping a few frames is better than playback lagging behind; i.e. if you seek in a DVD or stop it it doesn't make any sense to show the remaining frames when playback resumes.
The "!!oldcfg.applyPulldown" is just a run-off-the-mill trick to convert an integer into a bool value without the compiler throwing a warning, which is used throughout ffdshow's code.
Logged In: YES
user_id=1276834
Originator: NO
More info:
When set buffers to 10,10,10 and GRAB mode,
I have got a captured files (at first pass, same script):
grab0000.jpg (info frame number 10810799, original 0)
grab0001.jpg (info frame number 10810799, original 1)
grab0012.jpg (info frame number 1, original 2)
grab0013.jpg (info frame number 3, original 3)
grab0014.jpg (info frame number 3, original 3)
...
2. I see a code fragment:
input.numBuffers=numBuffers=buffersNeeded+bufferBack+(applyPulldown ? 1 : 0);
applyPulldown=!!oldcfg.applyPulldown;
Seems ApplyPulldown is not initialized in first line.
3. more comments later. Thanks.
Logged In: YES
user_id=975750
Originator: YES
> 2. I see a code fragment:
>
> input.numBuffers=numBuffers=buffersNeeded+bufferBack+
> (applyPulldown ? 1 : 0);
> applyPulldown=!!oldcfg.applyPulldown;
>
> Seems ApplyPulldown is not initialized in first line.
Eeek, you're right... that slipped through. But even if applyPulldown was 0 when it should have been 1 it only means that there would be one less back buffer if and only if applying the pulldown meant buffering 2 frames from 1 input frame because the ring buffer I'm using is one element too small; i.e. nothing that could cause a frame to be duplicated. Oh, and that an extra buffer gets allocated later on when applyPulldown is true is not a bug - that's an extra buffer just beyond the ring buffer I use for keeping the fields from the last frame.
Anyway, I'll add some debugging statements tomorrow afternoon and build a version that dumps which frames get buffered and which get pushed out by the filter via debug prints - I assume you know how to use Sysinternals' DebugView?
Some more questions:
Are you sure that you have the grab filter right behind the AviSynth filter in the filter chain? (You can drag filters up and down in the filter list)
Are all other filters deactivated? And even then some filters might get included in the filter queue even though they're deactivated (resize comes to mind, if the AR is being changed), and if some other filter is causing duplicates I can't do much. Oh, and be sure to turn off "drop frame on delay" in the Decoder Options as well as "Queue output samples" just to be on the safe side.
Have you set ffdshow to actually decode the XviD file? Or are you using the XviD decoder and have set ffdshow to process raw video?
Also, make sure the XviD file doesn't contain B frames as the extra hoops you have to jump through to reorder those _might_ cause some padding frames to be added at the start...
As for the 100% CPU usage at the start - that's the decoder running at full tilt to produce the frames that are being buffered, which is the desired effect. Can't say I notice much of it here with one core of an AMD Athlon 64 X2 4400+ - what are you using?
And as far as the filenames produced by the grab filter are concerned - that's the number of the decoded frame, which is getting incremented while frames are being buffered.