Problem when compressing high number of files

2014-08-01
2014-08-28
  • Gergely Orsi
    Gergely Orsi
    2014-08-01

    Dear All,

    I've encountered an error when trying to create an archive with 7-zip.
    I'm using 7-Zip 9.20 64bit (2010-11-18), under Windows 7 Professional 64bit, with 4GB DDR3 1600 RAM (dual mode) and an i5_3570 CPU. Windows handles swap.

    I'd like to compress 11 million (yes, 11.000.000 ) files with average file size of 230KB (2500 GB of data uncompressed).

    However 7-zip exits with an error around 30%, I used tar, as I wanted to know if 7-zip can handle this file count in the first place, and I thought that tar woud be the fastest method, as no compression is done on the data.

    7-zip error is something like "operation is interrupted because of OS limitations" or similar.

    Is there any way to compress this data with 7-zip.?

    Thank you

    Gergő

     
  • Igor Pavlov
    Igor Pavlov
    2014-08-02

    Try 7z format / fastest / (lzma2) and look also
    "RAM usage" in Task manager.

     
  • Gergely Orsi
    Gergely Orsi
    2014-08-05

    Dear Igor,

    Thank you for your answer. The server is now running the weekly incremental backup since Friday night (which is a full backup at this time, unfortunately). Nevertheless it will finish by no later than tomorrow morning, so I can test it tomorrow.

    Thank you,

    Gergő

     
  • Gergely Orsi
    Gergely Orsi
    2014-08-07

    Dear Igor,

    I did the things you suggested, however 7-zip still halted after 167GB of data (started yesterday at 10:40, halted today at 6:00)

    Please see the attached jpg.
    It has 3 parts.

    The top one is just before the error (an auto screenshot was taken every 15 minutes), it represents the average cpu load and memory usage. The mem usage was between 2.9 and 3,1 GB. The average CPU load was around 20% for the 4 cores.

    The middle one is when the error seems to occur, as you can see no CPU usage, but the mem is still occupied.

    The last one is when the error message appeared. No mem allocation, no CPU usage.

    The error message translated to english is: The requested operation can not be fulfilled, because of a limitation in the file system.

    Any suggestions?

    Thankx,

    Gergő

     
    Attachments
  • Igor Pavlov
    Igor Pavlov
    2014-08-08

    Maybe it's RAM size problem.
    You can try some of the following things:
    - add more RAM modules.
    - reduce the number of files to half (for test purposes)
    - try 7-Zip 9.34 alpha.

     
  • Gergely Orsi
    Gergely Orsi
    2014-08-27

    Dear Igor,

    Finally I got it working.

    I used 9.34 alpha with fastest compression setting and lmza2. I choosed to split the volume to 25GB (BD) chunks and it worked. The compression took nearly a week and ended up with 36 BD chunks (800GB, ratio was ~ 30%).
    Memory requirement was around 3GB for the compression and utilized all four cores for around 20% each, the total file count was exactly 11,494,988.

    Thanks for your help Igor.

    Keep up the good work.

    Best Regards,

    Gergő

     
    • Igor Pavlov
      Igor Pavlov
      2014-08-27

      Please make somes tests for your archive.
      Open archive and check that it shows your folders/files (11,494,988).
      Try also "Test archive" operation.

       
  • Gergely Orsi
    Gergely Orsi
    2014-08-27

    The resulting files were encrypted (256bit AES) and are under transfer to a remote location. They are about to arrive today late. Than I'll decode them (will take a few days) and test the archives (will take a few days again).

    I've checked the archives and the file count was right, the complete archive test should be completed on the remote server to avoid any data loss/corruption during the transfer and encryption/decryption.

     
  • Igor Pavlov
    Igor Pavlov
    2014-08-27

    "Open" operation is fast. So you can check it now.
    LZMA2 Decoding or Testing for 3.4 GHz is faster than 12 MB/s.
    So 800GB / 12 MB/s = ~18 hours.

     
  • Krishty
    Krishty
    2014-08-28

    "The requested operation can not be fulfilled, because of a limitation in the file system."

    I've seen this error with NTFS compression enabled. Windows does not reliably support NTFS compression for files >20 GB.