#4 Regain Processing Speed when possible

closed
nobody
None
5
2010-10-15
2010-08-09
Phil
No

I didn´t figure a better way to explain myself nor if this should had go in another place, anyway, I have noticed that while FV++ process a high number of files (say 300GB) the speed keep on dropping. Case scenario I start FV++ no other application taking cpu/ram, so it´s fast, but if I run another application FV++ speed drops and even if I close the other running application it doesn´t speed up. That´s particularly a trouble since I need to make use of some heavy utilities at a times and because of that FV++ turns incredible slow ~15 hours creating/verifying hashes.

Discussion

  • Tom Bramer

    Tom Bramer - 2010-09-05

    How many files are you hashing? Have you compared the performance to any other application doing the same thing on the same data set in the same situation? The only improvement that I can see that would not be system dependent (i.e. looking at the disk position of files) is to use multithreading to exploit the disk I/O scheduling in the operating system (by reading multiple files at once). The improvement would be more significant when the filesystem is highly fragmented, though it might hurt performance in other cases. Hashing many very small files will be much slower than hashing one large one.

     
  • Phil

    Phil - 2010-09-05

    I have around 19.000 files, that´s 300 GB of mainly audio files in my external hard drive. I haven´t used other checksum verifier tool apart from the foobar integrity verifier component and FreeFileSync compare mode that checks bit by bit,
    Both keep a constant speed and/or regain it when I stop a resource intensive task. What it puzzles me is that if the speed of FileVerifier ++ started at 15 mb/s and after some hours it dropped to 7mb/s, if I abort the job and
    select to verify the rest of the files the speed starts again at 15 mb/s. Isn´t it possible that FileVerifier could do that automatically - without having to manually stop and verifiy the left files to regain speed?
    Say at some interval. I haven´t tested it in other OS apart from my XP SP3 machines (in case it could be an OS dependent deal).

     
  • Tom Bramer

    Tom Bramer - 2010-09-19

    I think what you're experience has to do with how the program calculates the processing rate. The rate displayed is not an instantaneous rate, but the overall rate up to that point. For example, if the program has processed 10 GiB and has taken 10 minutes to do this, the rate is 17.06 MiB/s at that point. If you do something that causes a slowdown, then this will affect the overall rate that is displayed. It's not that the program is processing at the same rate after the slowdown condition is no longer present, but that the slowdown condition lowered the average rate so much that it would take quite some time for that number to come back up.

     
  • Phil

    Phil - 2010-09-27

    That makes sense except for the remaining time. For example a given task at the beginning had a remaining time of 5 hours, during the resource intensive task 9 hours, but after the intensive task is gone the estimated time is still 9 hours, when It should be 5 or less. If I stop the processing and restart it with the files left, the remaining time is 5 hours or less...

     
  • Tom Bramer

    Tom Bramer - 2010-10-02

    The program is also using the average read rate to calculate the remaining time. It does this as the read rate is varies greatly over the entire process due to filesystem fragmentation or having a large number of files scattered around on the disk. If the program didn't use this method, then the displayed value would fluctuate to the point where it would be useless in cases of a large number of files.

     
  • Phil

    Phil - 2010-10-15

    Alright then, thanks for explaining tjbramer :)

     
  • Phil

    Phil - 2010-10-15
    • status: open --> closed
     
  • Tom Bramer

    Tom Bramer - 2012-04-17

    An issue was found where the processing speed was actually dropping over time. One user has noticed significantly improved processing time in certain cases. You might want to try version 0.6.6.6050 to see if there is any difference for your case.

     

Log in to post a comment.