Header Error / Data After Payload for Windows Created .Zip Files Which Unzip...
A free file archiver for extremely high compression
Brought to you by:
ipavlov
Menu
▾
▴
6 Attachments
#2183 Header Error / Data After Payload for Windows Created .Zip Files Which Unzip Fine with Other Utilities
Using 7-zip to decompress .zip files created with Windows 10 via "Send to > Compressed (zipped) folder" fails with "Headers Error" error message for some files in many cases, resulting in 0 byte file for them, despite the fact that the files unzip just fine with other utilities like WinRAR and Windows (eg. when opening in File Explorer and copying outside of it). I've encountered this with a number of published zip file-based product releases produced via Windows for different products.
Many Others with Same Issues with 7-zip Only (Not Other Zip Utilities)
Also, it seems 7-zip reports the same error messages for .zip files produced via other programs, such as all zips produced from ArchiveStream and
XProtect Mobile Server where it's reported that the zip files unzip just fine, without any corrupted files, with other zip utilities like Windows Extract. There are also over 5,000 other mentions of encountering this kind of issue with 7-zip.
Specific Archive Issues Identified
In this case, the most problematic AST.accda file from AST32.zip unzips fine via both WinRAR and Windows File Explorer to a file with size 2,818,048 bytes with an actual CRC32 = BA808BE4, while the Properties for the file inside the archive show an expected unpacked size of 2,727,936 with CRC32 of 255BA07F in both 7-Zip and WinRAR. Therefore it seems like issue may be related to Windows zipping with wrong file size and CRC in some cases.
Configuration and Test Files
In my case, these errors occur with 32-bit and 64-bit versions of 18.06, 16.04, 9.20 with both 7zFM GUI and 7za command line, at least on Windows 10 Pro x64 (10.0.17763), with the attached AST32.zip file. I also provided a similar AST64.zip file which has similar contents produced via the same method near the same time but does not have this issue, in case may be useful for comparison.
Warning with Other Windows Zip Files
Also, "Warning: There are some data after the end of the payload data" (shown in attached screenshot) occurs with the LTM.zip (also attached here) also created by Windows > Send to, however, inconsistently, 7-zip treats that as a warning instead of an error and produces a non-zero sized file which is accurate (not corrupt) as seems to commonly be the case with such situations, resulting in inconsistent behavior within 7-zip for common File Size and CRC mismatch issues (which I discovered after quite a bit of debugging, due to lack of useful error messages), possibly depending on whether the file is at the end of the .zip archive or not, as well as inconsistent behavior with other zip utilities.
Lack of Useful Error Messages
Also, 7-Zip fails to provide any kind of useful error message, which I have provided suggestions to improve further below, even when using command line, when performing Test or Extract with command line (7za.exe), regardless of whether using switches like -bb3 and/or -bbo1 in an attempt to get more information, and even when using Test method and command line instead of GUI.
Vague Error Messages Received
Error Message from GUI (also shown in attached screenshot):
"Headers Error : ASST.accda"
Error Message from Command line:
Sub items Errors: 1 Archives with Errors: 1
Only the much older 7-Zip v9.20 provides slightly more information about the error, which appears to indicate use of an "Unsupported Method" for some of the files in the archive, while other files unzip fine, as shown in the console output below:
Extracting AST_Installation_Instructions.txt Extracting AST.accda Unsupported Method
However, both files show the compression method used as simply "Deflate" for each file, as seen via right click each file (AST.accda and then AST_Installation_Instructions.txt) in 7-zip File Manager > Properties > Method = Deflate, even though "Unsupported Method" error (in older 9.20 or more generic Headers Error in later versions) only occurs for one file, AST.accda in this case.
I also performed Test for each file and the archive with WinRAR 5.61 (64-bit), and no errors were indicated.
Other .zip archived created via Windows Send To commonly end up with other errors (also shown in attached screenshot), such as the following, when unzipping LTM.zip for example, created via the same Windows > Send To method:
Warnings: There are some data after the end of the payload data
And still others which were created at same time, via same method, but with almost the same contents ended up unzipped without any error in the latest versions of 7-Zip at least, with AST.accda in AST64.zip (as opposed to AST.accda, a slightly different file, in AST32.zip).
Problems with Current Handling of Metadata Mismatching
However, regardless of that, WinRAR, Windows File Explorer, and other zip utilities seem to unzip such zip files fine, without even an error message or warning, even when testing them. Considering that, I would suggest 7-zip at the very least unzip those files as well, instead of creating a 0 byte needlessly corrupted file as it does currently, together with a warning message. This is all the more important if Windows is producing such inaccurate file size and CRC file metadata in many case, and since other utilities unzip without issue. Also, if you are producing a file at all on disk, as is the case on error now, it should, I believe, at least be one that may be accurate in many cases instead of a 0 byte one that is always inaccurate. It is good that 7-zip produce a warning of some kind (which WinRAR and Windows fail to do), but without wording too strongly (see the suggested wording provided further below), since in many cases the file isn't corrupt, and definitely without resulting in a needlessly corrupt 0 byte file when that can be avoided.
Producing a 0 byte file can be especially problematic when, for example, I unzipped at one point, and at a later date tried to open the 0 byte AST.accda, which, unlike text files, isn't immediately apparent that it's empty, since, in this case, it resulted in an infinite loop of Microsoft Office (repair?) installation dialogs for some reason. Therefore, a non-zero byte file if much better result to produce instead of a 0 byte file, especially if you are going to produce a file on disk at all.
Possible Advanced Options / Switches
If desired, you could add CLI switches and/or advanced options (eg. under 7-Zip File Manager > Tools > Options > Settings) for whether to create 0-byte files, no files, fail to A) keep unzipped files (regardless of mismatch) vs. B) replace with 0-byte files (as is done now) vs. C) produce no file (so don't have 0 byte files that may cause problems as did in this case) or D) rollback entire unzip process (all or nothing). However, I don't think those advanced options are critical if you at least just default to keeping the unzipped file (or finish writing it out to disk) (as other zip utilities do, considering how common this case is without any corruption occurring).
Suggested Detailed Warning Messages to Show
Also, I would suggest providing error messages which are helpful in indicating what is wrong, such as the following, in this case:
Warning: 'AST.accda' unzipped file size and contents failed to match metadata: Unzipped to size of 2,818,048 bytes, which was larger than expected 2,727,936 bytes and to contents with CRC-32 hash of 'BA808BE4' instead of expected '255BA07F'. There is a possibility that 'AST.accda' is corrupt.
Or when only the size fails to match, showing a warning like the following (either "was larger than" or "was smaller than", depending on the size difference):
Warning: 'AST.accda' unzipped file size failed to match metadata: Unzipped to size of 2,818,048 bytes, which was larger than expected 2,727,936 bytes. There is a possibility that 'AST.accda' is corrupt.
Or when only the CRC fails to match, showing a warning like the following:
Warning: 'AST.accda' unzipped file contents failed to match metadata: Unzipped to contents with CRC-32 hash of 'BA808BE4' instead of expected '255BA07F'. There is a possibility that 'AST.accda' is corrupt.
While the full warning can be far more useful, if someone wants an abbreviated warning for command line (depending on switches used) you could have this be:
Size+CRC Mismatch CRC Mismatch Size Mismatch
However, this the above is far less useful and obvious for most end-users, so I would suggest to defaulting to full warning/error message, at least for GUI-based unzipping.
Summary
In summary, the most important thing is that 7-zip is unable to unzip files which all other utilities unzip fine (likely due to the number of cases where metadata is frequently incorrect while the files themselves are correct, such as for files produced via Windows > Sent to), unzipping the full and often accurate file in cases like this instead of producing a 0-byte file on disk. And the behavior (vague error + 0 byte resulting file vs. data at the end warning + accurate non-0 sized file) should be the same regardless of whether the file with inaccurate metadata is the first or last file in the archive (or whatever the cause of the warning message when zipping with Windows). Also, that behavior should be consistent with other zip utilities which handle such common conditions without issue. But, in addition to that, more helpful warning messages in this case and other cases, would be useful, especially when even debug mode console usage and Test method usage fails to produce any info useful for debugging what is wrong with an archive or providing an idea of how severe or likely potential corruption, if any, there may be.
Thanks for your help and consideration with this issue.
Appreciatively,
Dan
--
Dan Moorehead
PowerAccess | PowerExcel AI Add-ins for Microsoft Access | Excel
For reference, those .zip's were created via "Send to > Compressed (zipped) folder" with Windows 7 Pro SP1, from the developer who produced them, as opposed to my Windows 10 Pro x64 PC used for testing unzipping them.
1) It's not BUG of 7-Zip. It's BUG of software that was used to create these ZIP files.
And it's good that 7-Zip shows error messages. That way it's possible to know about the problems and developers can fix "bad" software that creates incorrect archives.
2) you write that "Send to > Compressed (zipped) folder" produces such archive. But could you create such archives in your system?
Maybe it's just problem of one user that have some incorrect system or incorrect zip program?
The problem is not that warnings are shown, as I agree that they should be, instead it's that 7-Zip needlessly produces corrupt 0-byte files when instead if should correctly unzip the data that is actually there, like all other major zip utilities do, in addition to warning about the possibility of corruption. Then the user can decide after opening the file whether it seems valid or not. There is no benefit to 7-Zip needlessly replacing often valid and not corrupted files with 0 byte placeholder files when already warning about the issue and when able to successfully decompress those files.
This is not an isolated incident, as I had linked to other bug reports for various other products which produce or release zips where users report that only with 7-Zip do such files produce corrupt 0-byte file results, with those zip files unzipping correctly with no corruption of data with all other zip utilities. I provided links showing over 5,000 different references to this issue occurring exclusively whem using 7-Zip to unzip indicating how common such .zip files with mismatched CRC and/or lengths are for .zips which otherwise can still be unzipped successfully without any actual data corruption, when using other utilities to do so.
Also I provided multiple zip files including those produced from different people and systems on same day, with some corrupt and some not, seemingly randomly. If it is a bug with specific zip utilities, than it affects a large number of users, since it was Windows built-in zipping that resulted in it, and for a random subset of zips produced and tested. However, I also linked to bug reports of the same issue occurring very frequently with other 3rd party zip libraries and utilities as well, where it's reported that 7-zip is the only utility which fails to unzip them correctly, producing corrupted (0-byte file) results unlike other utilities. Therefore it's this has been reproduced and reported with many users, systems and tools used to create zip files.
Additionally, besides it's behavior being inconsistent with all other zip utilities in failing to support such common case zips, 7-Zip even inconsistently handles the same kinds of length mismatch issues, where if if occurs with the last file entry (and the resulting size is larger instead of smaller than expected) it simply reports a warning (about data after the payload) but continues to unzip the file (as should always be the case) but if smaller than expected or not the last file entry, the 7-Zip instead reports an error message and needlessly produces an always 100% incorrect 0-byte file instead.
Also, I was reporting how the error message was extremely vague to the point of being useless ("Header Errors") and for some earlier versions even provided an inaccurate error message (reporting "Not Supported" method for basic Deflate zipped files) when instead I would suggest providing a more detailed warning using the text templates I had provided.
As you can see, I am all for very clearly warning the user about the potential corruption, and want this to be even morr clear than it is currently, while also providing more consistency in handling of similar kinds of header issues, and treating as a warning in all similar cases instead of errors I some and warnings in others, so that the user can determine whether a file is corrupt instead of 7-Zip always producing corrupt 0-byte results even when it doesn't have to.
I do think 7-Zip may provide an advantage over WinRAR and the like in that it actually is capable of reporting warnings in such cases, instead of silently ignoring as other utilities do, provided it can treat such cases as warnings always (instead of errors just in some cases depending on file entry position and whether file ended up being larger or smaller than expected) and/or always unzipping potentially useful results instead of always corrupt 0-byte results. The fact that all other major zip utilities silently ignore such issues seems to further indicate how common they are, emphasising the need to handle this common case, albeit while maintaining the 7-Zip advantage of actually warning about it too (ideally in more useful ways like with the more detailed warning messages I had suggested).
Thanks,
Dan
Please don't write such long messages.
Write simpler.
I suppose that these archives are rare cases.
So it doesn't matter how 7-Zip unpacks them while it shows error message. If there is "bad unpacking" for "bad" archives, developers of "bad" software faster will fix their software, that produces "bad" archives.
I don't think that Windows really produces such files.
Can you reproduce bad creation in your system?
If some version of Windows really produces such archives, I want to know full statistics - exact version of Windows and exact conditions.
If there is some proof that it's not rare case, I can think about ways to change 7-Zip.
Last edit: Igor Pavlov 2019-02-17
FYI - Google's Android plugin for Gradle produces debug APK files that yield this same error with 7zip. Other utilities, such as Java's jar command, Windows Explorer, and Ubuntu unzip will extract without error. Furthermore, the APKs are valid and work on Android devices.
Last edit: Jeremy 2019-05-31
Attached an example APK produced by a debug build from a default starter project in Android Studio. 7zip reports this:
Ok.
I'll fix it in next version.
Hi, could you please state in which version has been fixed this issue? I am experiencing this issue still in Z-Zip version 19.00
Thanks a lot in advance
I don't update old versions of 7-Zip.
So 7-Zip 20.02 alpha can contain the fix.
Hi there!
This is also happening to any .cab files containing drivers directly from Microsoft. I'm using 7-zip version 21.01 Alpha.
Examples:
http://download.windowsupdate.com/d/msdownload/update/driver/drvs/2019/07/db836048-b3e6-4325-96c4-c75ae25980da_91cca6441ff281049c84e37d54f2ccb792f169bb.cab
http://download.windowsupdate.com/d/msdownload/update/driver/drvs/2018/12/48c65969-f719-48df-a977-f0bd19d7b973_f381f6105420f209015c3ce39d76de97c93163c2.cab
It contains additional digital signature after cab archive.
7-Zip and cab format do not know about that additional data.