Thread: [Audacity-devel] [PATCH] FFmpeg on-demand based input
A free multi-track audio editor and recorder
Brought to you by:
aosiniao
From: Michael C. <mc...@gm...> - 2012-05-23 07:08:09
Attachments:
0001-restore-build-for-EXPERIMENTAL_OD_FFMPEG.patch
0002-integrate-OD-FFmpeg-to-work-with-multiple-formats-in.patch
0003-add-on-demand-preference-to-library-pane.patch
0004-Fix-warnings-for-OD-ffmpeg.patch
0005-fix-indentation.patch
0006-fix-case-where-DecodeFrame-was-not-returning-errors.patch
0007-change-flushing-case-for-OD-ffmpeg.patch
0008-lazy-evaluate-for-OD-ffmpeg-seek-test.patch
0009-OD-FFmpeg-do-not-attempt-to-correct-stream-lag-unles.patch
0010-OD-FFmpeg-fix-a-bug-with-demand-seeking-by-using-a-s.patch
OD-FFmpeg-11-squashed-commits.patch
|
Hi, Here is a set of patches that allows ffmpeg on demand importing, updated for the new ffmpeg versions. Since this is a sizable new feature I am attaching the patches. I will commit as 10 modular commits (patches included), but for testing convenience I am also attaching a combined patch that squashes all 10. I tested and debugged for a while. I don't think it's polished yet, but since the user can switch it on and off, it is lower risk if I get the OK to commit. It enables EXPERIMENTAL_OD_FFMPEG, but the default functionality is the non-OD import, which is selectable via the Library preference pane in the ffmpeg section. It also refactors the relevant code from the non OD method. There are no large system/structural changes that would affect other parts of the code. I am not sure if seeking should be enabled or not since ffmpeg is not always that accurate at seeking. I expected to find more problems with it, but after these fixes in my basic test cases it does the correct thing with my system ffmpeg. For testing the patch, I will leave it in. If we decide to disable it, this will simply become background linear loading with no on-demand updates, which is still useful. I tested on linux and mac on flv, wav, mp3 (vbr, cbr), m4a (aac), mpeg video (audio track). Michael |
From: Michael C. <mc...@gm...> - 2012-05-31 20:49:51
|
On Tue, May 22, 2012 at 9:07 PM, Michael Chinen <mc...@gm...> wrote: > Hi, Anyone get a chance to try it out? Again, if you are just testing the functionality, just apply the single OD-FFmpeg-11-squashed-commits.patch. I guess the functional review is more important now. The default functionality does not change from current ffmpeg support, so I think it's safe to commit, but if we should commit after 2.0.1 I can understand. I'll be moving again from june 10 and not have good availability for at least a week or so, so if the patch can be reviewed by then that would be ideal, otherwise we can plan on doing it after 2.0.1 if we still target end of june early july because that might be cutting it too close. But please don't let that stop you from reviewing early. Michael |
From: Vaughan J. <va...@au...> - 2012-06-08 04:01:53
|
Have not had time to test it, but I confirm it builds on Win with flag EXPERIMENTAL_OD_FFMPEG on or off. - V On 5/31/2012 1:49 PM, Michael Chinen wrote: > On Tue, May 22, 2012 at 9:07 PM, Michael Chinen <mc...@gm...> wrote: >> Hi, > > Anyone get a chance to try it out? > Again, if you are just testing the functionality, just apply the > single OD-FFmpeg-11-squashed-commits.patch. > I guess the functional review is more important now. > The default functionality does not change from current ffmpeg support, > so I think it's safe to commit, but if we should commit after 2.0.1 I > can understand. > I'll be moving again from june 10 and not have good availability for > at least a week or so, so if the patch can be reviewed by then that > would be ideal, otherwise we can plan on doing it after 2.0.1 if we > still target end of june early july because that might be cutting it > too close. But please don't let that stop you from reviewing early. > > Michael > |
From: Martyn S. <mar...@gm...> - 2012-05-31 23:14:31
|
Hi Michael I tried to apply the single patch here on win7 with Tortoise and all the hunks failed, but I don't see why. Initially TortoiseMerge didn't seem to like the place I'd put the patch (the root of Audacity, one dir above src etc) and scanned for a better place. Then it failed. I was only going to test if it compiled anyway, any win-specific issues, as I'm no expert tester of FFMPEG, or OD! If it's all under EXPERIMENTAL_OD_FLAC, why not commit anyway with that off, and ask for tests with that on? TTFN Martyn On 31/05/2012 21:49, Michael Chinen wrote: > On Tue, May 22, 2012 at 9:07 PM, Michael Chinen<mc...@gm...> wrote: >> Hi, > > Anyone get a chance to try it out? > Again, if you are just testing the functionality, just apply the > single OD-FFmpeg-11-squashed-commits.patch. > I guess the functional review is more important now. > The default functionality does not change from current ffmpeg support, > so I think it's safe to commit, but if we should commit after 2.0.1 I > can understand. > I'll be moving again from june 10 and not have good availability for > at least a week or so, so if the patch can be reviewed by then that > would be ideal, otherwise we can plan on doing it after 2.0.1 if we > still target end of june early july because that might be cutting it > too close. But please don't let that stop you from reviewing early. > > Michael > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > audacity-devel mailing list > aud...@li... > https://lists.sourceforge.net/lists/listinfo/audacity-devel |
From: Michael C. <mc...@gm...> - 2012-06-01 06:23:58
|
On Thu, May 31, 2012 at 1:14 PM, Martyn Shaw <mar...@gm...> wrote: > Hi Michael > > I tried to apply the single patch here on win7 with Tortoise and all > the hunks failed, but I don't see why. Initially TortoiseMerge didn't > seem to like the place I'd put the patch (the root of Audacity, one > dir above src etc) and scanned for a better place. Then it failed. > > I was only going to test if it compiled anyway, any win-specific > issues, as I'm no expert tester of FFMPEG, or OD! > > If it's all under EXPERIMENTAL_OD_FLAC, why not commit anyway with > that off, and ask for tests with that on? Good idea Martyn, I just did that, so it is in with EXPERIMENTAL_OD_FFMPEG off. This is untested on windows so please any windows people please check that it builds off, then with it on. You may need to do a clean when changing Experimental.h stuff (at least on linux you do). Even after enabling it in Experimental.h, you will need to enable it in the library prefs if you want to see it in action. Best, Michael |
From: Martyn S. <mar...@gm...> - 2012-06-02 18:45:22
|
Hi Michael On 01/06/2012 07:23, Michael Chinen wrote: > On Thu, May 31, 2012 at 1:14 PM, Martyn Shaw<mar...@gm...> wrote: >> Hi Michael >> >> I tried to apply the single patch here on win7 with Tortoise and all >> the hunks failed, but I don't see why. Initially TortoiseMerge didn't >> seem to like the place I'd put the patch (the root of Audacity, one >> dir above src etc) and scanned for a better place. Then it failed. >> >> I was only going to test if it compiled anyway, any win-specific >> issues, as I'm no expert tester of FFMPEG, or OD! >> >> If it's all under EXPERIMENTAL_OD_FLAC, why not commit anyway with >> that off, and ask for tests with that on? > > Good idea Martyn, I just did that, so it is in with EXPERIMENTAL_OD_FFMPEG off. > > This is untested on windows so please any windows people please check > that it builds off, then with it on. Builds with #define EXPERIMENTAL_OD_FFMPEG on and off. > You may need to do a clean when changing Experimental.h stuff (at > least on linux you do). VS detects that. > Even after enabling it in Experimental.h, you will need to enable it > in the library prefs if you want to see it in action. I have not tested further. TTFN Martyn > Best, > Michael > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > audacity-devel mailing list > aud...@li... > https://lists.sourceforge.net/lists/listinfo/audacity-devel |
From: Gale (A. Team) <ga...@au...> - 2012-06-09 08:56:05
|
Hi Michael I've compiled Unicode Release on Windows with #define EXPERIMENTAL_OD_FFMPEG on (uncommented). I haven't had time to do much testing but if others who don't build Audacity want to, please try: http://gaclrecords.org.uk/audacity-win-2.0.1-alpha-09june12-r11769-ffmpeg-od.zip . I have found one "issue" but it occurs with non-FFmpeg On-Demand as well. If I import a file with about half an hour or more of audio, quickly drag a selection at the end of the track, Amplify (using a "Y" shortcut for Amplify) then press ENTER, the Amplify has no effect (although I can see the waveform which has started to be drawn in the selection is nowhere near 0 dB). As you can see here, Amplify actually sees the audio before running the effect at 0 dB: http://gaclrecords.org.uk/images/on-demand-amplify.png . This is completely repeatable for me. Windows 7 64-bit, 2.4 GHz, 6 GB RAM, dual core. When I use CTRL + R to access Amplify (almost as) quickly, I cannot reproduce this issue at all, nor in 1.3.13 (but that does not have effects shortcuts). Generally (80% of the time) if I drag in the waveform and apply an effect (even with the menus) the FFmpeg-On-Demand waveform drawing stops at that point (though the effect completes). This is true for WAV, video files and AAC. If I On Demand-import WAV files of a comparable length using libsndfile then run an effect on a selection, the On-Demand waveform drawing always completes. If I repeatedly abort the stalled FFmpeg-On-Demand drawing by File > Close (not saving changes), then repeat another FFmpeg-On-Demand import and stall it by running an effect, Audacity eventually tends to crash when clicking in the waveform. I do not have "Play and/or record using RAM" enabled. I'll try and test more later. Gale -- View this message in context: http://audacity.238276.n2.nabble.com/PATCH-FFmpeg-on-demand-based-input-tp7555261p7555344.html Sent from the audacity-devel mailing list archive at Nabble.com. |
From: Vaughan J. <va...@au...> - 2012-06-11 22:09:58
|
Thanks for that, Gale. I haven't found time to test it, but good that others now have the opportunity. I think this should probably go into 2.0.1. - V On 6/9/2012 1:55 AM, Gale (Audacity Team) wrote: > Hi Michael > > I've compiled Unicode Release on Windows with > #define EXPERIMENTAL_OD_FFMPEG on (uncommented). > > I haven't had time to do much testing but if others who don't build > Audacity want to, please try: > http://gaclrecords.org.uk/audacity-win-2.0.1-alpha-09june12-r11769-ffmpeg-od.zip > . > > I have found one "issue" but it occurs with non-FFmpeg On-Demand as well. > If I import a file with about half an hour or more of audio, quickly drag a > selection at the end of the track, Amplify (using a "Y" shortcut for > Amplify) > then press ENTER, the Amplify has no effect (although I can see the waveform > which has started to be drawn in the selection is nowhere near 0 dB). > > As you can see here, Amplify actually sees the audio before running the > effect > at 0 dB: > http://gaclrecords.org.uk/images/on-demand-amplify.png . > > This is completely repeatable for me. Windows 7 64-bit, 2.4 GHz, 6 GB RAM, > dual core. > > When I use CTRL + R to access Amplify (almost as) quickly, I cannot > reproduce > this issue at all, nor in 1.3.13 (but that does not have effects shortcuts). > > Generally (80% of the time) if I drag in the waveform and apply an effect > (even with the menus) the FFmpeg-On-Demand waveform drawing stops at > that point (though the effect completes). This is true for WAV, video files > and > AAC. If I On Demand-import WAV files of a comparable length using libsndfile > then run an effect on a selection, the On-Demand waveform drawing always > completes. > > If I repeatedly abort the stalled FFmpeg-On-Demand drawing by File > Close > (not saving changes), then repeat another FFmpeg-On-Demand import and > stall it by running an effect, Audacity eventually tends to crash when > clicking in the waveform. I do not have "Play and/or record using RAM" > enabled. > > I'll try and test more later. > > > > Gale > > -- |
From: Gale (A. Team) <ga...@au...> - 2012-06-12 10:07:42
|
"Vaughan Johnson" wrote: > Thanks for that, Gale. I haven't found time to test it, but good that > others now have the opportunity. I think this should probably go into > 2.0.1. I tested again (but still only on Windows so far), this time using the Win-nightly. I'm afraid I have the same reservations about speed and stability. I still find that only in a minority of occasions can I seek or edit while waveform calculation is in progress, then have the calculation complete. For example I've been waiting five minutes now on 70% CPU for audio from an hour-long VOB file stored on the hard drive to complete OD-calculation, having tried to seek in a few places in the waveform. It's still quite easy to crash if I File > Close on a stalled FFmpeg On Demand import I was interacting with, then try to FFmpeg On Demand import again, and repeat those steps a few times. If I just let that hour-long VOB file complete OD import, I see wide variations in time taken at each attempt, from 35 seconds to 90 seconds, usually about 70 seconds. Normal FFmpeg import is a reliable 20 seconds. Same story with an hour long WAV file. On-demand using libsndfile is always about 15 seconds. FFmpeg on-demand varies from about 50 seconds to 90 seconds (if I leave it alone). Should we try it without seeking/interacting with the import? And given we are able to seek/interact, can you explain a bit more about what "background" on-demand loading means (as written in the Libraries Preferences)? It would be good to know for the Manual. Thanks! Gale -- View this message in context: http://audacity.238276.n2.nabble.com/PATCH-FFmpeg-on-demand-based-input-tp7555261p7555356.html Sent from the audacity-devel mailing list archive at Nabble.com. |
From: Michael C. <mc...@gm...> - 2012-06-12 12:39:13
Attachments:
ODFFMPEG_disable_seeking.patch
|
Thanks everyone for testing. On Tue, Jun 12, 2012 at 12:07 AM, Gale (Audacity Team) <ga...@au...> wrote: > "Vaughan Johnson" wrote: >> Thanks for that, Gale. I haven't found time to test it, but good that >> others now have the opportunity. I think this should probably go into >> 2.0.1. > I tested again (but still only on Windows so far), this time using the > Win-nightly. > > I'm afraid I have the same reservations about speed and stability. I still > find > that only in a minority of occasions can I seek or edit while waveform > calculation > is in progress, then have the calculation complete. For example I've been > waiting > five minutes now on 70% CPU for audio from an hour-long VOB file stored on > the > hard drive to complete OD-calculation, having tried to seek in a few places > in > the waveform. Ok, I didn't see this in linux or mac, but maybe windows is more sensitive. It's also possible the ffmpeg version is different. > > It's still quite easy to crash if I File > Close on a stalled FFmpeg On > Demand > import I was interacting with, then try to FFmpeg On Demand import again, > and repeat those steps a few times. Good catch, thanks. > > If I just let that hour-long VOB file complete OD import, I see wide > variations > in time taken at each attempt, from 35 seconds to 90 seconds, usually about > 70 seconds. Normal FFmpeg import is a reliable 20 seconds. Thanks, I need to look into this. > > Same story with an hour long WAV file. On-demand using libsndfile is always > about 15 seconds. FFmpeg on-demand varies from about 50 seconds to > 90 seconds (if I leave it alone). I expect it to be much slower here since ffmpeg is copy in vs od libsndfile is aliased. > > Should we try it without seeking/interacting with the import? And given > we are able to seek/interact, can you explain a bit more about what > "background" on-demand loading means (as written in the Libraries > Preferences)? It would be good to know for the Manual. This is normal on demand (loads in the background, but use can demand different parts to be loaded). I guess users won't know what is meant by just on-demand. Here is a patch that disables seeking, please apply it if that eliminates the crashes/waveform stalls. I will test more on mac/linux to see if I can get these issues. If not, maybe I'll add a subcheckbox for seeking after 2.0.1 Unfortunately I'm in the process of moving and don't have a machine that builds audacity, but I should have one soon. Michael |
From: Gale A. <ga...@au...> - 2012-06-15 09:40:26
|
Summary: Tested on Windows 7 x64 and Ubuntu (single core 1 GHz, 1 GB RAM). Disabling seeking means that FFmpeg OD imports are now: * consistently up to 20% faster than normal FFmpeg * consistently complete even if interacting with the import * free from crashes when aborting and re-importing. There is a downside with seeking disabled that if you apply an effect before the selection is computed, the selection will be permanently silenced for that import. ATM I would prefer to disable seeking for 2.0.1 rather than have it seemingly unstable and unsatisfactory on Windows (and generally on slow single core machines). More details below. | From Michael Chinen <mc...@gm...> | Tue, 12 Jun 2012 02:38:21 -1000 | Subject: [Audacity-devel] [PATCH] FFmpeg on-demand based input > Thanks everyone for testing. > > On Tue, Jun 12, 2012 at 12:07 AM, Gale (Audacity Team) > <ga...@au...> wrote: > > "Vaughan Johnson" wrote: > >> Thanks for that, Gale. I haven't found time to test it, but good that > >> others now have the opportunity. I think this should probably go into > >> 2.0.1. > > I tested again (but still only on Windows so far), this time using the > > Win-nightly. > > > > I'm afraid I have the same reservations about speed and stability. I still > > find > > that only in a minority of occasions can I seek or edit while waveform > > calculation > > is in progress, then have the calculation complete. For example I've been > > waiting > > five minutes now on 70% CPU for audio from an hour-long VOB file stored on > > the > > hard drive to complete OD-calculation, having tried to seek in a few places > > in > > the waveform. > > Ok, I didn't see this in linux or mac, but maybe windows is more > sensitive. It's also possible the ffmpeg version is different. I am using 0.6.2 from here: http://manual.audacityteam.org/help/manual/man/faq_installation_and_plug_ins.html#ffdown . > > It's still quite easy to crash if I File > Close on a stalled FFmpeg On > > Demand > > import I was interacting with, then try to FFmpeg On Demand import again, > > and repeat those steps a few times. > > Good catch, thanks. > > > > > If I just let that hour-long VOB file complete OD import, I see wide > > variations > > in time taken at each attempt, from 35 seconds to 90 seconds, usually about > > 70 seconds. Normal FFmpeg import is a reliable 20 seconds. > > Thanks, I need to look into this. > > > > > Same story with an hour long WAV file. On-demand using libsndfile is always > > about 15 seconds. FFmpeg on-demand varies from about 50 seconds to > > 90 seconds (if I leave it alone). > > I expect it to be much slower here since ffmpeg is copy in vs od > libsndfile is aliased. OK, I did not realise at the time that this was copy-in. > > Should we try it without seeking/interacting with the import? And given > > we are able to seek/interact, can you explain a bit more about what > > "background" on-demand loading means (as written in the Libraries > > Preferences)? It would be good to know for the Manual. > > This is normal on demand (loads in the background, but use can demand > different parts to be loaded). I guess users won't know what is meant > by just on-demand. Thanks. I agree "on-demand" will not mean much. I am not sure if "background" is the best word as the user will see the waveform as it is computed. Possibly "On-demand loading (faster)" might be better (analogous with "read directly (faster)" in the Import / Export Preferences). > Here is a patch that disables seeking, please apply it if that > eliminates the crashes/waveform stalls. > I will test more on mac/linux to see if I can get these issues. If > not, maybe I'll add a subcheckbox for seeking after 2.0.1 Thanks. Tested on Win 7 and Ubuntu 12.04. Definitely, FFmpeg-OD import times are now consistent. It's now about 20% faster to import audio files when letting FFmpeg-OD complete than when using normal FFmpeg. If importing audio from video it's a few % faster, I would say. If I try to play or edit where the waveform has not yet been computed, this slows down completion but does not stall it, and I don't see crashes on repeated aborts using File > Close then re-importing. There is a problem though if user drags a selection and applies an effect before the waveform has been computed. This will silence the audio or render it as dither noise. If I then undo the unwanted effect before the calculation has completed, OD is stalled permanently. This is also true when seeking is enabled (as in HEAD) and the effect does what is intended. Libsndfile OD doesn't display this problem - probably a P3? If I undo the unwanted effect after calculation has completed, the selection reverts to uncomputed state, as it does if I undo and redo import. So I see no way to get the audio back except by File > Close and repeat the import. I have since found that merely seeking then playing files over 30 minutes long often stalls FFmpeg OD in HEAD, not only on Win 7 dual core but also on Ubuntu 12.04 on a basic netbook (1 GHz, 1 GB RAM). So I guess disabling seeking is less bad than what we have in HEAD. Disabling seeking shows the intended speed improvement rather than slowdown, no crash risk and generally the calculation completes. Either left "as is" or disabling seeking, I think there would have to be a release note on the behaviour, but I have said in the Manual that you can edit or play when the waveform appears (which is going to be good advice either way). The Status Bar "Click to change task focal point" should probably not appear if we are disabling seeking. I have not had time to test reopening projects hat have not been fully imported. Tests welcomed. One other issue - another VOB I tried to import from my hard drive seems to have multiple streams, but OD FFmpeg import only imports the first stream. Normal FFmpeg import imports the complete audio. Possibly P4. Gale |
From: Michael C. <mc...@gm...> - 2012-06-17 13:36:09
|
On Fri, Jun 15, 2012 at 9:40 AM, Gale Andrews <ga...@au...> wrote: > > Summary: Tested on Windows 7 x64 and Ubuntu (single core > 1 GHz, 1 GB RAM). Disabling seeking means that FFmpeg OD > imports are now: > > * consistently up to 20% faster than normal FFmpeg > * consistently complete even if interacting with the import > * free from crashes when aborting and re-importing. > > There is a downside with seeking disabled that if you apply > an effect before the selection is computed, the selection > will be permanently silenced for that import. > > ATM I would prefer to disable seeking for 2.0.1 rather than > have it seemingly unstable and unsatisfactory on Windows > (and generally on slow single core machines). > > More details below. > [...] Ok, I'll commit it the patch as is for now (disables seeking). Thanks a lot for the detailed testing. I'll hopefully look at these in the next few weeks for submission after 2.0.1. Michael |