Issues on certain HDTV channels
Brought to you by:
xuming
Menu
▾
▴
Hi Andy,
I recently got access to HDTV (1080i h264) on my HTPC which I record with ARGUS TV which uses ezthumb to create thumbnails. I noticed that thumbnail creation on some of these HDTV recordings is very slow.
A test on a problematic HDTV file went as follows:
First it hangs for a long time (for about a third of the video duration) saying "Building (Fast)" and eventually generates a white thumbnail file with only the file info text and no thumbnails. Then it switches to scan mode and generates a good thumbnail file with the ".1.jpg" extension. So in total it generates 2 JPEGs: a bad one (FAST mode) and a good one (SCAN mode).
My questions are:
Notable:
This issue is channel specific meaning that on certain channels all recordings are problematic and on other channels all recordings are not problematic.
Another small issue: your latest version (2.1.10) didn't have the version info updated in the console.
Samples:
HDTV files are quite large so I created 2 sample files with limited duration. So instead of hanging for minutes on "Building (Fast)" it hangs for 20seconds but it still demonstrates the problem nicely.
http://dl.dropbox.com/u/14050315/ezthumb_samples_20130303.zip
Using Win7 x64.
Greetings,
Sjakko
Last edit: Xuming 2013-03-04
Hi Sjakko,
Thanks for your comments and sample. I think it's a known issue. I've reviewed the related code and found a comment here:
"20120724 a supplemental method to cope with DVB rips.
It had been observed in some DVB rips, maybe H.264 stream as well,
there are dodge i-frames which marked as i-frame in the packet.
However after decoding by ffmpeg, they turned to be P/B frames,
which confused ezthumb. The reason is unknown yet. The safe mode
can handle it without trouble because the safe mode decodes from
the first i-frame packets.
After weighed by balance, I think it may be a good idea to
restart the ezthumb in the safe mode if it found no thumbnail
was generated in the required process, better than nothing"
I have some thoughts to accelerate the process. However I'm fully occupied by 2.2.0 now so this fix would be put in 2.2.1. Before that I suggest using '-p safe' to enforce the scan mode.
Cheers
Andy
Hi Andy,
Thank you for your reply. I hope you can find a way to speed it up considerably, preferably to the speed of SDTV MPEG2 recordings which are lightning fast now, but maybe this is too ambitious. On DVB recordings there always will be some errors in the stream indeed but ARGUS TV still hopes to rely on ezthumb to generate a thumbnail quickly without hogging system resources too much, which could interfere with smooth video playback on a HTPC. I'm testing the '-p safe' parameter which is better (but not ideal) on these problematic recordings, but (as expected) worse on not-problematic recordings. I hope you can find a way.
Happy to hear any further fixes/suggestions. Suggestions for improvements on ARGUS TV's side are welcome too. As a user who's in frequent contact with the ARGUS TV developer I can let him know. My goal is for ARGUS TV and ezthumb to work well together and reliably generate thumbs as fast as possible. Ideally with 1 command for all scenarios, but I understand this may not be possible. In those cases it would be nice if the feedback comes from ezthumb so ARGUS TV can deal with the situation accordingly.
Thanks,
Sjakko
Last edit: Sjako 2013-03-05
Hi Sjakko,
I'm a bit curious to how ARGUS TV runs ezthumb? Maybe I can make a mode for it to optimize the speed and performance. For example, '-p argus' the ezthumb could check the frame type before falling into the safe mode.
Kind regards
Andy
ARGUS TV uses the following parameters. An example:
-x ~ --override on --time-from 00:14:59 --time-end 01:06:39 --gap-margin 0 --gap-shots 0 -g 2x2 --opt-time off --opt-info off "\ANNE-PC\TV\ARGUS Recordings\Horizon\Horizon_2013-03-05_di_00-20_BBC2\2013-03-05_di_00-20_Horizon_BBC2.ts"
Where --time-from, --time-end and off course the file's path are different for every recording. Optimized speed and performance sounds great. An automatic mode where it automatically chooses the fastest mode possible would be great. Not only ARGUS TV users could benefit from that. And if remotely possible we'd like to avoid the safe mode alltogether and we hope you find a way to do the FAST mode for even these "dodgy" files.
Thanks a lot!
Sjakko
Thanks Sjakko. I'll think of it how to improve the performance after the 2.2.0 release.
Is there an approximate ETA for a new release?
Hi Sjako,
Thanks for your concern. I haven't prepaired for a release. I was quite
busy in the past two months so I don't have time to work on ezthumb. By the
way, do you have any suggestion about ezthumb? I'll collect all feedbacks
and will put them into the future release.
Kind regards
Andy
On Sat, Apr 13, 2013 at 8:53 PM, Sjako sjako@users.sf.net wrote:
Hi Andy,
No problem, we're all busy sometimes.
I don't think I have any new suggestions you haven't heard of already. The most important thing for us right now is a reliable way to quickly generate thumbnails for DVB recordings, which can be dodgy/damaged. More info can be found in my previous posts in this thread.
Thank you!
Sjakko
Hi Sjakko,
I've just released a new version of ezthumb. I believe it can resolve your
previous mentioned problems. It supports thumbnail from multiple video
clips as well.
Kind regards
Andy
On Tue, Apr 16, 2013 at 6:54 PM, Sjako sjako@users.sf.net wrote:
Hi Andy,
It looks like you nailed the issues with HD recordings! I do however see new behavior on some SD recordings, for which thumbnail generation takes minutes (instead of seconds with v2.1.10). The behavior is as follows: ezthumb hangs for minutes (depending on file size I guess) without writing anything to the console and then decides to use the "fast" mode after which the process resumes as usual and generates a thumbnail in seconds.
Let me see if I can find such a file. It may be large though (>1GB). I hope this isn't a problem for you.
Another problem I noticed is that the file sizes ezthumb reports is way off. I think on all files.
And does ezthumb use a cache? The second run on the same file (sometimes?) seems faster.
Last edit: Sjako 2013-08-20
Hi Sjako,
I haven't reproduced the file size issue. It probably caused by the 32-bit
system. Could you paste the command line output of ezthumb to me please?
There are some debug information to help me to locate the performance issue.
Kind regards
Andy
On Wed, Aug 21, 2013 at 4:43 AM, Sjako sjako@users.sf.net wrote:
Hi Sjako,
Thanks for your feedback. Ezthumb 3.0 applied a probe method to decide how to process the video. It was newly written so it's still under experiment. I reckon it can be fixed soon once I receive a sample.
The file size issue is new one to me. I'll have a look at it tonight.
Ezthumb doesn't use cache. The cache effect comes from the 32-bit file system in your computer.
Kind regards
Andy
Hi Andy,
I decided to upload a full size sample.
DELETED
The logs are included. The file size issue is there too.
Please let me know when I can take the file offline.
Last edit: Sjako 2013-08-25
Thanks Sjako, I've got the sample file. You may delete it now.
On Fri, Aug 23, 2013 at 11:30 PM, Sjako sjako@users.sf.net wrote:
Hi Sjako,
I have done the investigation, many thanks to your feedback and the sample file.
In ezthumb 2.x, most issues were reported in two root causes, how to seek inside the media file and how to find the end of the stream. The first one means some video clips couldn't be sought freely by ffmpeg so ezthumb has to be manually restarted in scan mode. The second one means some video files, especially the mpeg rippes, were damaged so the actual length were not as showed in the head structure.
Apart from the binding mode, ezthumb 3.0 was mostly rewritten to cope these two problems. It will probe the media files to find the seekability and bitrates prior to take snapshots. This process only scan the first 30 seconds of the video file so it took less 1 second, very light overhead aganist the benifit.
In the problematic secenario by your sample file, ezthumb was fooled by the bitrates. It believe the video was around 3Mbit/sec, actually 5Mbit/sec overall. I can't find the coding error anywhere. The best guess so far I have is that video is variable bitrate. The advertisement part, longer than 30 seconds occasional, is 3MBps and the main program is 5Mbps.
Indeed the ezthumb has coded the plan B, the quick scan, to cope this scenario, or suppose to. After all the auto detect strategy is not accurate. However it failed in your test. The method is so poor that I decide to move to binary search. I have done a prototype to test the performance. It's quite outstanding, less than half second. I'll make an update release in one or two days.
By the way, if you're quite positive in the integity of the video file, you can always use "-d head" option to disable the auto-detect to cut down the overhead.
Kind regards
Andy
Hi Andy,
Sorry for my late response. Kinda busy you know.
I tested 3.0.1 and it solved the issue for the sample file nicely and for a couple of other files too. Great work!
I do however have a couple of problematic files remaining, some of which weren't problematic with 3.0.0. Because of their size I'll upload samples one at a time (no point in uploading 3 large sample files that all have the same root cause), but that means we may have to go back and forth a couple of times to iron most issues out.
Thanks for the detailed explanation and your continuing efforts to approach perfection.
Here's a next sample:
https://docs.google.com/file/d/0BxpGfIz7SUlfR1VZQWEtb2FWYTg/edit?usp=sharing
Sjakko
Thanks Sjakko. I've got the sample.
On Fri, Sep 13, 2013 at 4:00 PM, Sjako sjako@users.sf.net wrote:
Hi Sjako,
I've fixed the problem and released an update. There's only one line
change and I wish ffmpeg won't change that feature in future. This bug
doesn't share the root cause with previous version so it should be a
safe fix. Many thanks for your help and support. Otherwise I can't
locate and fix these bugs so soon.
Best regards
Andy
On Sat, Sep 14, 2013 at 9:53 PM, Andy Xuming xuminic@gmail.com wrote:
Hi Andy,
Great work! Many issues are solved now. I have another one for you (+/- 6 GB):
https://docs.google.com/file/d/0BxpGfIz7SUlfZnk0U1RiMTRZZnc/edit?usp=sharing
Sjakko
Hi Sjako,
Thanks for your test and sample file. This problem is the same to that
in 3.0.2. Apparently the previous fix didn't do job well, which
applied a simple trick trying to get the last frame, too simple to
cover all the situations. Please find the new released 3.0.3 which
fixed this matter. It's been tested by all your previous sample files,
so far so good.
Kind regards
Andy
On Thu, Sep 26, 2013 at 7:04 AM, Sjako sjako@users.sf.net wrote:
Hi Andy,
Again much better. I pulled my recording folder through ezthumb 3.0.3 and from the log I extracted only the time it took to generate each jpeg. This is to give you an idea of the normal generation times. Here's the list:
11.036 s
5.893 s
7.096 s
6.527 s
6.508 s
6.526 s
6.662 s
7.586 s
6.629 s
9.333 s
7.632 s
6.456 s
6.188 s
6.074 s
6.531 s
4.959 s
6.147 s
5.364 s
7.563 s
7.047 s
14.587 s
25.363 s
11.885 s
5.890 s
8.120 s
8.591 s
12.871 s
7.857 s
6.876 s
4.632 s
5.856 s
5.182 s
6.482 s
3.832 s
7.722 s
7.920 s
7.592 s
36.959 s
6.275 s
5.953 s
5.864 s
34.257 s
6.199 s
6.860 s
7.192 s
9.391 s
7.725 s
10.544 s
7.955 s
6.945 s
7.685 s
8.885 s
7.495 s
7.310 s
8.456 s
11.539 s
8.781 s
13.051 s
15.055 s
8.786 s
7.217 s
7.269 s
7.828 s
8.055 s
6.963 s
32.234 s
6.146 s
7.107 s
13.274 s
12.237 s
7.300 s
8.514 s
10.631 s
9.159 s
9.540 s
8.936 s
8.750 s
15.901 s
9.779 s
11.004 s
3.982 s
10.247 s
12.212 s
13.132 s
14.111 s
12.898 s
12.415 s
13.469 s
8.315 s
7.506 s
8.724 s
10.103 s
9.387 s
11.392 s
9.951 s
10.814 s
4.843 s
4.937 s
3.810 s
4.319 s
10.835 s
13.139 s
11.343 s
11.615 s
9.888 s
12.801 s
10.692 s
11.207 s
11.280 s
10.583 s
9.318 s
11.241 s
11.285 s
11.111 s
11.160 s
12.359 s
10.084 s
12.181 s
11.330 s
11.200 s
8.214 s
5.281 s
7.553 s
7.510 s
6.247 s
7.213 s
5.441 s
8.935 s
4.190 s
7.133 s
6.346 s
7.028 s
7.692 s
6.793 s
8.007 s
6.788 s
5.490 s
4.793 s
12.746 s
7.743 s
5.495 s
28.160 s
32.577 s
5.889 s
7.804 s
5.698 s
18.224 s
28.001 s
14.298 s
10.597 s
7.048 s
13.704 s
2.626 s
7.809 s
13.895 s
9.090 s
11.693 s
11.356 s
11.346 s
10.113 s
12.346 s
9.491 s
10.004 s
8.387 s
10.060 s
16.902 s
13.373 s
0.923 s
11.107 s
17.629 s
3.820 s
12.549 s
8.895 s
10.081 s
10.059 s
9.707 s
10.582 s
30.365 s
10.508 s
8.539 s
14.886 s
3.676 s
10.382 s
8.617 s
29.164 s
8.982 s
14.251 s
10.032 s
9.208 s
12.472 s
10.369 s
10.690 s
8.364 s
0.696 s
12.232 s
12.320 s
10.622 s
10.579 s
12.822 s
15.912 s
13.631 s
11.102 s
12.346 s
2.127 s
7.459 s
6.068 s
5.237 s
7.387 s
6.837 s
6.103 s
6.303 s
5.171 s
6.545 s
5.997 s
9.317 s
10.758 s
7.259 s
4.613 s
3.786 s
2.164 s
0.741 s
8.745 s
7.542 s
7.518 s
8.716 s
28.311 s
8.235 s
8.295 s
7.776 s
5.871 s
6.358 s
6.280 s
6.641 s
21.048 s
6.674 s
20.416 s
6.114 s
25.325 s
8.805 s
8.782 s
8.045 s
7.402 s
6.873 s
7.226 s
5.667 s
5.523 s
6.370 s
7.212 s
0.736 s
7.634 s
7.315 s
7.656 s
10.083 s
7.131 s
7.821 s
Maybe I'll send you one of the slower files so you can further optimize things.
If you like, I have the full logs here:
https://dl.dropboxusercontent.com/u/14050315/20131004%20ezthumb%203.0.3%20logs.7z
As promised: https://docs.google.com/file/d/0BxpGfIz7SUlfcVhzSzhjZm42MzA/edit?usp=sharing
But if you ask me the following issue has priority: https://sourceforge.net/p/ezthumb/discussion/general/thread/f30ec815/
I hope you can fix that one before the upcoming ARGUS TV 2.1.1 release.
Thanks Sjako. I've received the sample. It took 12 seconds in my
machine. In theory it should take less than 4 seconds. I roughly have
an idea of what had happened here. I'll see if I can safely improve
the performance.
On Sat, Oct 5, 2013 at 8:40 PM, Sjako sjako@users.sf.net wrote: