Menu ▾ ▴

#2038 7-Zip hangs on extracting >4GB zip file compressed with Mac

open
nobody
zip (5) mac (1) extraction (2) hang (1)
5
2017-03-24
2017-02-05
aONe
No

Discussion

  • Igor Pavlov

    Igor Pavlov - 2017-02-06

    what version of 7-zip?

     

  • aONe

    aONe - 2017-02-06

    Thanks for the early reply Igor, tested it with the latest stable version 16.04 (2016-10-04), the process window just hangs, need to finalize the process via the task administrator.

     

    Last edit: aONe 2017-02-06

  • aONe

    aONe - 2017-02-06

    Just testing it with WinRAR, it opens and extracts the file but also gives an error saying the file is corrupt. Gonna file a bug in the apple site also.

     

  • Igor Pavlov

    Igor Pavlov - 2017-02-06

    File is too big. It's difficult to download it.
    Write what software was used to create that zip file.
    Try same software to unpack that file.

     

    Last edit: Igor Pavlov 2017-02-06

  • aONe

    aONe - 2017-02-06

    Compressed with bundled macOS Sierra 10.12.3. The system extracts it without any problem.
    Just tried with WinZip 5 for Mac and fails too, it says "Fatal error: Error: central directory not found".
    Also unzip extracts it with a warning "4294967296 extra bytes at beginning or within zipfile" and some messages "file #1: bad zipfile offset (local header sig): 4294967296". But it extracts it.

    So definetly the system is creating corrupt files, gonna file a bug in Apple site.

    Anyway will be very helpfull if 7-zip can handle this corruption, at least as unzip or WinRAR.

    It's helpfull if I share with you some bytes from the file?

     

  • Igor Pavlov

    Igor Pavlov - 2017-02-06

    I have start of file.
    It uses local data descriptor.
    I'll try to download full file for more tests.

    What exact size in bytes of archive and size of big file inside archive?
    You can upload also last 16 KB of archive.
    Maybe it can help.

    Also you can create another example (1 file of about 100 MB) and check same problem with that archive.

     

  • aONe

    aONe - 2017-02-06

    The compressed file contains more than one file. Also if the resulting compressed file is less than 4GB there's no problem, so the 100MB test gives no problem.

    Attached the dump of the last 16KB.

    The file I've shared has this contents (before compression):

    machine:test aone$ ls -la
total 8492896
drwxr-xr-x@  2 aone  staff        442 Feb  6 13:34 .
drwxr-xr-x@ 12 aone  staff        748 Feb  6 13:34 ..
-rw-r--r--@  1 aone  staff       6148 Jan 30 16:39 .DS_Store
-rwxrwxrwx@  1 aone  staff  434831844 Jan 27 17:51 01.mov
-rwxrwxrwx@  1 aone  staff  434831844 Jan 27 17:51 02.mov
-rwxrwxrwx@  1 aone  staff  434831844 Jan 27 17:51 03.mov
-rwxrwxrwx@  1 aone  staff  434831844 Jan 27 17:51 04.mov
-rwxrwxrwx@  1 aone  staff  434831844 Jan 27 17:51 05.mov
-rwxrwxrwx@  1 aone  staff  434831844 Jan 27 17:51 06.mov
-rwxrwxrwx@  1 aone  staff  434831844 Jan 27 17:51 07.mov
-rwxrwxrwx@  1 aone  staff  434831844 Jan 27 17:51 08.mov
-rwxrwxrwx@  1 aone  staff  434831844 Jan 27 17:51 09.mov
-rwxrwxrwx@  1 aone  staff  434831844 Jan 27 17:51 10.mov

    zipinfo gives me that:

    machine:~ aone$ zipinfo test_Over4GB.zip 
Archive:  test_Over4GB.zip   4332052318   25
warning [test_Over4GB.zip]:  4294967296 extra bytes at beginning or within zipfile
  (attempting to process anyway)
drwxr-xr-x  2.1 unx        0 bx stor 30-Jan-17 16:44 test/
-rw-r--r--  2.1 unx     6148 bX defN 30-Jan-17 16:39 test/.DS_Store
drwxrwxr-x  2.1 unx        0 bx stor 30-Jan-17 16:44 __MACOSX/
drwxrwxr-x  2.1 unx        0 bx stor 30-Jan-17 16:44 __MACOSX/test/
-rw-r--r--  2.1 unx      120 bX defN 30-Jan-17 16:39 __MACOSX/test/._.DS_Store
-rwxrwxrwx  2.1 unx 434831844 bX defN 27-Jan-17 17:51 test/01.mov
-rw-rw-rw-  2.1 unx      271 bX defN 27-Jan-17 17:51 __MACOSX/test/._01.mov
-rwxrwxrwx  2.1 unx 434831844 bX defN 27-Jan-17 17:51 test/02.mov
-rw-rw-rw-  2.1 unx      271 bX defN 27-Jan-17 17:51 __MACOSX/test/._02.mov
-rwxrwxrwx  2.1 unx 434831844 bX defN 27-Jan-17 17:51 test/03.mov
-rw-rw-rw-  2.1 unx      271 bX defN 27-Jan-17 17:51 __MACOSX/test/._03.mov
-rwxrwxrwx  2.1 unx 434831844 bX defN 27-Jan-17 17:51 test/04.mov
-rw-rw-rw-  2.1 unx      271 bX defN 27-Jan-17 17:51 __MACOSX/test/._04.mov
-rwxrwxrwx  2.1 unx 434831844 bX defN 27-Jan-17 17:51 test/05.mov
-rw-rw-rw-  2.1 unx      271 bX defN 27-Jan-17 17:51 __MACOSX/test/._05.mov
-rwxrwxrwx  2.1 unx 434831844 bX defN 27-Jan-17 17:51 test/06.mov
-rw-rw-rw-  2.1 unx      271 bX defN 27-Jan-17 17:51 __MACOSX/test/._06.mov
-rwxrwxrwx  2.1 unx 434831844 bX defN 27-Jan-17 17:51 test/07.mov
-rw-rw-rw-  2.1 unx      271 bX defN 27-Jan-17 17:51 __MACOSX/test/._07.mov
-rwxrwxrwx  2.1 unx 434831844 bX defN 27-Jan-17 17:51 test/08.mov
-rw-rw-rw-  2.1 unx      271 bX defN 27-Jan-17 17:51 __MACOSX/test/._08.mov
-rwxrwxrwx  2.1 unx 434831844 bX defN 27-Jan-17 17:51 test/09.mov
-rw-rw-rw-  2.1 unx      271 bX defN 27-Jan-17 17:51 __MACOSX/test/._09.mov
-rwxrwxrwx  2.1 unx 434831844 bX defN 27-Jan-17 17:51 test/10.mov
-rw-rw-rw-  2.1 unx      271 bX defN 27-Jan-17 17:51 __MACOSX/test/._10.mov
25 files, 4348327418 bytes uncompressed, 4332048550 bytes compressed:  0.4%

    Let me know if this info helps.

     

    Last edit: aONe 2017-02-06
    16kb.dump

  • Igor Pavlov

    Igor Pavlov - 2017-02-06

    Did you wait for 1-10 minutes?
    You can track "IO read bytes" in Task manager.

    file is larger than 4 GB.
    Good zip program must use zip64 headers in that case.
    But this mac program probably doesn't use zip64 headers.
    so some values are truncated from 64-bits to 32-bits.
    That why 7-zip probably can't use central directory header.
    So it reads local headers instead.
    But local headers in that zip doesn't contain pack sizes also.
    So 7-zip needs to scan archive for data descriptors.
    It can be slow, but probably it must work.
    So try to wait and track file reading in Task manager.

     

  • aONe

    aONe - 2017-02-06

    Now I waited for longer, and about 10 minuts later the process failed: "can't open as a compressed file".

    Already reported to Apple the zip64 headers tip.

    Thanks a lot for your help Igor.

     

  • Igor Pavlov

    Igor Pavlov - 2017-02-06

    10 minutes is too slow.
    What values in Read Bytes in task manager?
    Maybe 7-zip tries to open that archive several times with different offset.
    I still need some file to test it.
    Please create special zip archve:
    You need to create big.zip archive larger than 4 GB, but all files in zip archive must be identical (only names are different).
    So you can compress that big.zip to small.7z with big LZMA2 dictionary.
    Dictionary size must be larger than file size in zip archive and solid block size is set to "solid".
    For example, file size 50 MB. Dictionary size = 64 MB.
    Final size of 7z archive will be smaller than 64 MB.
    So it will be simpler to upload / download that example archive.

     

  • Igor Pavlov

    Igor Pavlov - 2017-02-06

    Correction:
    Dictionary size must be larger than file size in zip archive, solid block size is set to "solid" and the number of CPU threads = 2.

     

    Last edit: Igor Pavlov 2017-02-06

  • aONe

    aONe - 2017-02-08

    Sorry for the delay, here it goes the file!

     
    big.7z

  • Igor Pavlov

    Igor Pavlov - 2017-02-09

    Thanks for that example!

    That zip archive creates many problems for 7-zip code.
    I'll try to change 7-zip code.
    So next version of 7-zip will work better for such archives.

    Please attach also another small zip archive example with several small files inside, maybe less than 1 MB total (uncompressed and compressed).

    It's intersting why that problem still exists in mac os (and how long).
    Did you see another reports that mac creates bad zip archives?
    Maybe you use some unusal way to create these archives?
    Is it problem for any version of mac os?
    How long macos supports zip archive creation?

     

    Last edit: Igor Pavlov 2017-02-09

  • aONe

    aONe - 2017-02-09

    Thanks a lot Igor. Hope to receive some support from Apple so they can fix the corrupt creation for newer OS.

    I've never seen a problem in the Mac zip creation, this is the first I know. Actually this get reported to me, but did not seen any other report about it.

    There's only one GUI integrated way to create zip files in Mac, no unusual actions, just right click the file and select "Compress...". I've been using Mac since 10.5 (2007 or so) and I think it always had this ability. I've tested from 10.9 (2013) to the actual version 10.12.3, and they produced the same corrupted file, so it affects at least the latest 5 years. I cant test 10.4-10.5 if you need it, just will take some time to install the older OSs.

    For the 1mb files sample, you want it to also be >4GB, right?

     

    Last edit: aONe 2017-02-09
    Thumbnail
    Screen Shot 2017-02-09 at 10.57.01.png

  • Igor Pavlov

    Igor Pavlov - 2017-02-09

    For the 1mb files sample, you want it to also be >4GB, right?

    No, I just need some good small zip example for macos.

     

  • aONe

    aONe - 2017-02-09

    There you go.

     
    Sparkle.framework.zip

  • Igor Pavlov

    Igor Pavlov - 2017-02-09

    Does zip in macOS support "Open" operation for zip archives?
    If so, please make the following test with your big.zip and measure time of operation execution.
    1) Open your big.zip with macOS zip program. Time of execution?
    2) Extract _320x180 copy 10.mp4 (it's first file in archive). Time of execution?
    3) Extract _320x180.mp4. (it's last file in archive, it has incorrect offset in central directory). Time of execution?

    Does it extracts any file correctly?

     

  • aONe

    aONe - 2017-02-09

    The bundled macOS operation does not support open, just extracts the entire file. I can't just give you the time to expand the entire file, if it is useful.

     

  • aONe

    aONe - 2017-02-09

    I think I can do it with unzip (Info-Zip) via command line, will it be usefull?

     

  • Igor Pavlov

    Igor Pavlov - 2017-02-09

    No need Info-Zip.
    The purpose of that test was to prove that macos zip has errors.
    "Open" operation for zip archives usually uses special code that reads file list from the end of archive (from central directory in zip). And if the program uses that code, then probably it will see error.

    If the program supports only "Extract" operation, it doesn't need to read list of files. It can read archive data sequentially. That data is correct in mac's zip. Errors are only at the end of archive (central directory), required for "Open" operation.

    7-Zip supports "Open" operation, so it always calls Open and Extract even if you call "Extract" only.

     

    Last edit: Igor Pavlov 2017-02-09

  • aONe

    aONe - 2017-02-10

    Thanks for the aclaration Igor. I had no reply from Apple yet, no surprise here tough.

     

  • Igor Pavlov

    Igor Pavlov - 2017-02-10

    Actually the answer from Apple is not required anymore.
    Situation is clear.
    They use old zip code without zip64 extension support.
    That code creates good archives if final size is smaller than 4 GB.
    If archive is larger than 4 GB, then some headers at the end of archive are not correct for 100%. But files data is correct still.
    So it's possible to extract all files from such archives.
    We just need to ignore bad headers.
    So I'll fix that code in 7-zip.

     

Log in to post a comment.

Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.