Near the end of a long queue multi-file stops working

[Ronald J Kienle]

I love this program, I use it every day, so much so that I have come to rely on it. I have been running in small batches of 30-100 files at a time lately for testing my settings etc.

The only problem I am having is I have it set to encode 2 files at a time, which it does so flawlessly....until it approaches the end of the queue where it all of a sudden only does one file at a time. Like right now, it is only running 1 encode with four files left in a 48 file queue?

The only solution I have found is to wait until a file finishes, abort the rest of the queue, then requeue the remaining files and restart the process. Not the end of the world but a little annoying since I like to leave this alone for a few days and come back to completed files and it's a lot slower doing one file at a time.

Does anyone have any idea what could be causing this issue?

Thanks.

[Abel]

Hi, what type of files and ffmpeg parameters have to tried?
Another point is which is the encoding priority being used.

[Ronald J Kienle]

I'm encoding TS files recorded from cable TV averaging 5-10 G in size.

Currently I am using the following settings:

-c:v hevc_nvenc "-vf "fieldmatch,bwdif=mode=0:deint=1,decimate,format=yuv420p10le"
-c:a copy -c:s copy -preset hq -rc constqp -rc-lookahead 60 -spatial_aq 1 -temporal_aq 1 -nonref_p 1 -cq 26

I leave the priority set to normal but I have nothing else running. Even if I change it to High it still only does one file at a time near the end of a long queue.

[Abel]

Just as a test, could you try setting encoding priority to Below normal, and see if there's any change.

[Ronald J Kienle]

I will do that the next time I run a big batch. Like I said, this only happens when the queue is rather large, like 40+

BTW, thanks for a great program. I love it. Now that I am more knowledgeable I have dumped MCEBuddy and FFmpeg Batch is my replacement and I use it every day.

[Ronald J Kienle]

Strange behavior. I only had a small queue of 11 files and I was ripping a blu-ray disc and running a script extracting CCs. Batch was running two files but then when one finished it did not start another. When I noticed I set the priority to below normal like suggested and the second file started. As soon as the other file finished once again Batch did not start the second file....so I switched priority to High and it then started the second file......I have no idea what this means??

[Abel]

Yes, weird, I have not been able to reproduce the issue yet. Which CPU/Video card are you using?

I guess that although you use video card for encoding, it uses some CPU resources, and in case of having other processes running to, for some reason I don't know the application can be starved of resources, so that's why increasing priority has come effect, but just a guess.

[Ronald J Kienle]

The CPU is an older i5 and the video card is a GeForce 1660 Super. It must be the wa Batch views resources because I only run two encodings at a time to keep the CPU cycles manageable and when doing so FFmpeg only uses about 30% of the CPU for each file encoding and the other two programs I had running were using very little CPU so the total load was under 80%

[Abel]

This guide may help you tune your parameters, GPU encoding works somewhat on its own resources and API:
https://developer.nvidia.com/blog/nvidia-ffmpeg-transcoding-guide/

I don't know if you have tried pre-input parameters like this, or tried increasing to 3 or 4 encodings at once, as your system resources seem to be manageable, to see if only one, or more encoding threads are stuck in the application.

Pre-input:
-hwaccel cuda -hwaccel_output_format cuda

[Ronald J Kienle]

This happened again today after not doing it for quite some time. What I have noticed is that with really large queues running two encodes at a time that for a while the program stays in order and the next file in the queue is started as soon as one completes....then for some reason it will jump down and skip several files in the queue sometimes 10 or so. Once this happens the encode that was still running then finishes the queue in order until it gets to the the file that the second encode jumped to then it jumps ahead of the other.....and the second encode runs in the order of the queue from the new spot it jumped to.

BUT....once it gets close to the end of the queue the first encode session no longer jumps, it just terminates and leaves the other to finish sometimes up to 10 files.

So, I don't know if this is relevant but once the queue is spiit then it appears at some point FFmpeg Batch seems to think that one of the running encodes has run out of files but it hasn't, it just no longer jumps ahead of the other.

Any thoughts Abel?

[Abel]

As you mention, there seems to be some sort of race condition by which the task partitioner does not keep order, or misses some files, depending on the ffmpeg task, or the numer of times multi-file encoding is run. I have never found a posible cause. Sometimes, just restarting the application solves the problem.

[Ronald J Kienle]

I am sharing for you info Abel, so at least you know what I am experiencing. What I usually do is when it gets down to one file I monitor it and shut it down and restart it and requeue the remainin files and start multi-file encoding once more. Not ideal but not the end of the world either.

BUT....that brings me to a question I have been meaning to ask for a while. Would it be possible in a future release to have Batch restart yet remain the queue as loaded, at least of unfinished files and auto start too. This way when I am out of town but running a queue of several hundred files a power outage doesn't stop my encoding. I'm trying to catch up on thousands of files and it is going to take many months so I run this program 24/7/7.

MCEBuddy does this, I am not sure if it is because it runs as a service or not but that is one of the only features I like about it anymore as I have stopped using it and have switched to Batch.

Thanks as always Abel and Happy New Year!