How do I handle large number of files?

  • jasoncd

    jasoncd - 2012-06-04

    I have two folders that I am storing as split zip files, with a volume size of 4 GB and no compression. Both are about 850 GB. One has 350K files and zips in about 12 hours. The other has 3,700K files and takes about 3 *days*. I'm running Windows Server 2008 R2, x64, with 8 GB RAM. Is there anyway I can speed this up or do I just need to use an alternative strategy. Thanks.

  • Gabriel Magana-Gonzalez

    That seems like way too long to compress… Instead of using no compressions, I have found that using the lowest compression setting is very often much faster: The CPU load is not great, and the reduction in I/O writes (because the data is compressed) results in a very noticeable speed up.  Also another way to speed up is to write to an archive that is in a different hard disk than the one where the files you are compressing are. This is another huge speed boost.

    I would also bring up the issue of data corruption: Is the data critical?  what happens if you have just a single archive go bad?  Would it be a problem if you cannot get to the rest of the files?  If this is the case, you could consider making  many smaller archives (so if one file gets corrupted you can still expand the other files), or switching compression technology (WinRAR can add redundant information so you can recover from data corruption).

  • jasoncd

    jasoncd - 2012-06-04

    I think I should have posted this in the Help forum instead…

    I am actually backing up to a directly connected NAS (Drobo). I can try using the lowest compression instead, but the problem seems to be the number of files, not bandwidth/speed. Also, in this case, the majority of the files are JPG images. I think another problem is the number of volumes created. Now that I think about it, I tried this earlier with a much larger volume size, and the time required was normal. Unfortunately Drobo's don't like large files…


Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

No, thanks