Activity for Robert Simpson
-
Robert Simpson posted a comment on discussion Help
Even on SSD, the CPU is many times faster than the fastest SSD drive.
-
Could this be why?
-
The .NET runtime team recently has incorporated zlib-ng into their codebase, and their System.IO.Compression ZipFile implementation is now using it -- which is where I got the 19 seconds from. https://github.com/dotnet/runtime/pull/102403
-
Consider using zlib-ng for deflate/inflate
-
I've also seen sporadic build errors with -j due to the target directory sometimes not getting created before the first ASMC output completes and tries and fails to write to the output location. Since I was batching my builds together into a shell script I just pre-created the directories beforehand in my script before calling make.
-
Is there an expected behavior for these two properties? It seems like every other handler in the project returns UInt64 for these two properties, except the LzhHandler, which returns UInt32.
-
I'm working, I just needed to add clang to my VS2022 components. I've actually written a build script that builds Format7zF for Windows x64 and Arm64, then drops into WSL and builds Linux x64/Arm64 as well as MacOS x64/Arm64. I changed the default build script to build .dylib for MacOS instead of .so files. Also, x64 MacOS didn't seem to use ASM=1, so I modified it for that and used https://github.com/gitGNU/objconv to convert the output from ASMC Linux x64 to Mach-O format so I could get MacOS x64...
-
Details with 24.05: Name: ffmpeg-N-115376-git-aff24c165-win32-nonfree-shared.tar Size: 7 733 248 Packed Size: 2 194 895 ------------------------: Size: 0 Packed Size: 0 Folders: 0 Files: 1 ------------------------: Path: \Downloads\ffmpeg-N-115376-git-aff24c165-win32-nonfree-shared.tar.zst Type: zstd Physical Size: 2 194 895 Streams: 1 Blocks: 60 Method: decoded: XXH64 wnd-desc-log-MAX:27 wnd-MAX:128MiB unknown-content-size ------------------------: ------------------------: Testing reveals: Data...
-
Guess I just needed to modify my VS installation to include clang. Leaving the post up.
-
Building on Windows for arm64 fails, it's trying to use clang-cl ... (Visual Studio 2022 Pro, Windows 11) D:\SourceCode\7z2405\CPP\7zip\Bundles\Format7zF>nmake.exe /NOLOGO platform=arm64 if not exist "arm64" mkdir "arm64" if not exist "arm64/asm" mkdir "arm64/asm" cl -DZ7_EXTERNAL_CODECS -nologo -c -Foarm64/ -Wall -WX -EHsc -Gy -GR- -GF -MT -GS- -Zc:wchar_t -Zc:throwingNew -MP4 -O1 -Yc"StdAfx.h" -Fparm64/a.pch StdAfx.cpp StdAfx.cpp clang-cl --target=arm64-pc-windows-msvc -nologo -c -Foarm64/ -Wall...
-
Building on Windows for arm64 fails, it's trying to use clang-cl ... D:\SourceCode\7z2405\CPP\7zip\Bundles\Format7zF>nmake.exe /NOLOGO platform=arm64 if not exist "arm64" mkdir "arm64" if not exist "arm64/asm" mkdir "arm64/asm" cl -DZ7_EXTERNAL_CODECS -nologo -c -Foarm64/ -Wall -WX -EHsc -Gy -GR- -GF -MT -GS- -Zc:wchar_t -Zc:throwingNew -MP4 -O1 -Yc"StdAfx.h" -Fparm64/a.pch StdAfx.cpp StdAfx.cpp clang-cl --target=arm64-pc-windows-msvc -nologo -c -Foarm64/ -Wall -WX ../../../../Asm/arm64/LzmaDecOpt.S...
-
It's not a crypto problem, it's a password Unicode encoding problem. That password was probably encoded as UTF-8 when the ZIP was made. In English (US) Windows, the default codepage is 1252, not UTF-8. So any attempt to pass in a unicode character in the password prompt is going to get mis-encoded and not decrypt the file. Which leads back to my suggestion that you use the command prompt after having modified it to default to the UTF-8 encoding.
-
What's the password look like? Have you tried 7z command-line in Windows? Have you tried switching the codepage on the console to UTF-8 and tried the command line? https://stackoverflow.com/questions/57131654/using-utf-8-encoding-chcp-65001-in-command-prompt-windows-powershell-window
-
7-zip data is backed entirely by stream-based interfaces that are not atomic and therefore can't be shared across threads. There are several ways to help solve the atomic reading problem, here's two: Rewrite IInStream so that Read and Seek are a single operation Add a new interface such as ICloneable to clone an IInStream so that each instance could have its own guaranteed seek position. Either way, every single handler would have to be revisited to support this, and it would require hundreds of...
-
Assuming 1gbit ethernet, it takes 90 minutes just to read 690gb end-to-end without any processing at all, just straight streaming the data. With 10 million small files, that will probably add about 30% overhead in file lookups, memory lists, etc. TBH I am not sure what the analyzing part is doing.
-
Not even sure what you're trying to accomplish extracting an executable? Also not sure where "super.img" file is or where it comes from.
-
Could not reproduce. Downloaded the zip, hit "Test" with 7z, tested almost instantly. Extracted to my Downloads directory in maybe 2 seconds or less. Windows 11, I9-13900K, 32gb RAM
-
Could not reproduce. Downloaded the zip, hit "Test" with 7z, tested almost instantly. Extracted to my Downloads directory in maybe 2 seconds or less.
-
ISetProperties
-
This probably goes without saying, but I assume you tested this in Linux? I had to add definitions for InterlockedDecrement in Threads.h and Threads.c for GCC. Incidentally, what's wrong with using the Interlocked atomic functions by default for the reference count? Are you concerned about performance implications?
-
Digging through the .NET source code, what I suspected is the case. .NET constructs a generic method proxy for GetStream() which returns back a void * which it then passes to ConvertToManaged The de-compiled auto-generated code in .NET looks like this: [global::System.Runtime.InteropServices.DynamicInterfaceCastableImplementationAttribute] file unsafe partial interface InterfaceImplementation : global::MyLibrary.Compression.SevenZ.Interfaces.IInArchiveGetStream { [global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.Interop.ComInterfaceGenerator",...
-
Digging through the .NET source code, what I suspected is the case. .NET constructs a generic method proxy for GetStream() which returns back a void * which it then passes to ConvertToManaged The de-compiled auto-generated code in .NET looks like this: ~~~ [global::System.Runtime.InteropServices.DynamicInterfaceCastableImplementationAttribute] file unsafe partial interface InterfaceImplementation : global::MyLibrary.Compression.SevenZ.Interfaces.IInArchiveGetStream { [global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.Interop.ComInterfaceGenerator",...
-
Digging through the .NET source code, what I suspected is the case. .NET constructs a generic method proxy for GetStream() which returns back a void * which it then passes to ConvertToManaged The de-compiled auto-generated code in .NET looks like this: file unsafe partial interface InterfaceImplementation { [global::System.Runtime.InteropServices.UnmanagedCallersOnlyAttribute(CallConvs = new[] { typeof(global::System.Runtime.CompilerServices.CallConvMemberFunction) })] internal static int ABI_GetStream(global::System.Runtime.InteropServices.ComWrappers.ComInterfaceDispatch*...
-
Unfortunately there's no way to explicitly release a com object from .net core that works on all platforms. The garbage collector always runs in a separate thread, so objects will always have their Release called from that thread. The only methods .NET core has to release a com object are tagged as Windows-only, and not supported from Linux or OSX: https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.marshal.finalreleasecomobject?view=net-8.0 https://learn.microsoft.com/en...
-
If Release is atomic, then the subsequent delete is also going to be atomic and only ever called once. So unless 7z is using thread-static member variables, deleting an object from another thread shouldn't be a problem I would think?
-
Technically it's not "working with", it's just calling Release from the other thread which triggers 7z to delete the object. So if you're saying 7z can't delete an object created by another thread, then yeah there's a serious problem using 7z from .NET. In the testing I've done with 7z, all I've needed to do is make AddRef/Release atomic. Once I did that, everything in .NET seems to work perfectly. The only call .NET will ever make to a 7z object in a different thread is Release.
-
Are you asking me how .NET's source code works when it interops with COM? That I'm not sure. All I know is that my code looks like this in .NET: [GeneratedComInterface] [Guid("23170F69-40C1-278A-0000-000600400000")] [InterfaceType(ComInterfaceType.InterfaceIsIUnknown)] internal partial interface IInArchiveGetStream { [PreserveSig] int GetStream(int index, out ISequentialInStream? stream); } [GeneratedComInterface] [Guid("23170F69-40C1-278A-0000-000300010000")] [InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]...
-
I don't have to compile the 7z code at all, I can use your binaries from the website and have the same result. The problem is that in .NET, the garbage collector runs in a separate thread. The thread that caused 7z to create an object is not the same thread that will call Release on that object. Because of that, there is a race condition where 7z can be executing AddRef at the same time it is executing Release on the same object. A sample scenario: 1. Main .NET thread A calls IInArchiveGetStream::GetStream()....
-
Looks like CHandlerImg is the last one I could find. Fortunately the fixes are super simple, like in HandlerCont.h: From this: Z7_COM_UNKNOWN_IMP_3( IInArchive, IInArchiveGetStream, IInStream) To this: Z7_COM_UNKNOWN_IMP_4( IInArchive, IInArchiveGetStream, IInStream, ISequentialInStream)
-
I'm finding more places where this happens. For example, CHandlerImg in HandlerCont.h implements IInStream but it doesn't support QueryInterface for ISequentialInStream
-
These are all the implementations I found that do not allow QueryInterface on ISequentialInStream: InStreamWithCRC.h - CInStreamWithCRC MultiStream.h - CMultiStream DmgHandler.cpp - CInStream NtfsHandler.cpp - CInStream XzHandler.cpp - CInStream ArchiveOpenCallback.cpp - CInFileStreamVol
-
NtfsHandler CInStream cannot query for interface
-
MyCom.h AddRef/Release is not atomic
-
+1 on source code -- can't do the testing I need to do without it.
-
I realize that MTDEC_THREADS_MAX is set to 32, so the 128-core scenario is not going to use all the cores, but it would be nice to control that maximum externally.
-
The biggest problem is that 7-zip may not always be the only thing running at the time. It would actually be nice to be able to tell 7z not to use the entire CPU and all the RAM :) Personally, I have 24/32 cores, but if I run 7z on an Ampere ARM64 CPU with 128 cores, things could get a little messy. For me it's not about the RAM usage, it's about hogging every core during decompression.
-
If you have VMWare, just run a Mac image, create a small volume, add some files to it, then delete a few files, and then add some more. Just enough to scramble the volume's btree a bit. If you want a real challenge, make a snapshot and then update some files. Then you'll have multiple INODE's with the same OID and different XID's. One of the fundamental problems is that the APFS filesystem isn't supposed to be read top-down from the start of the btree to the end. It's a jumpscotch thing. You start...
-
If you decide not to implement -- I am going to write it up, as I'll probably need it for what I'm doing. Would that need an ID number as a plugin?
-
The [toc].xml file contains the digital signature.
-
Even you decide not to implement -- I am going to write it up, as I'll probably need it for what I'm doing. Would that need an ID number as a plugin?
-
I encountered it after signing up as a developer for MacOS, and downloading their XCode15 application. The file downloaded in XIP format, and inside that was a Contents file in pbzx format. https://developer.apple.com/xcode/resources/ https://en.wikipedia.org/wiki/.XIP So far I can only find XIP files of Apple's XCode app, the rest are DMG on their website. Attached is 7-zip opening the XIP with Open Inside * mode And another screenshot of the Contents file extracted from 7-zip after "open inside"...
-
I encountered it after signing up as a developer for MacOS, and downloading their XCode15 application. The file downloaded in XIP format, and inside that was a Contents file in pbzx format. https://developer.apple.com/xcode/resources/ https://en.wikipedia.org/wiki/.XIP So far I can only find XIP files of Apple's XCode app, the rest are DMG on their website. Attached is 7-zip opening the XIP with Open Inside * mode
-
Igor, Apple's been using a different format for the last few years for the inside of their XAR files. It's basically an XZ file, but split into blocks. The file signature begins with 'pbzx', followed by a 64-bit flags. Loop while flags == 0x1000000, read the next 64-bit flags read the 64-bit length of the block The block itself begins with the XZ signature '\xFD7zXZ' and length is the total size of the block plus signature. Write out the decompressed LZMA stream Read an extra 2-bytes containing a...
-
Igor, Apple's been using a different format for the last few years for the inside of their XAR files. It's basically an XZ file, but split into blocks. The file signature begins with 'pbzx', followed by a 64-bit flags. Loop while flags == 0x1000000, read the next 64-bit flags read the 64-bit length of the block The block itself begins with the XZ signature '\xFD7zXZ' and length is the total size of the block plus signature. Write out the decompressed LZMA stream Loop until flags isn't 0x1000000 After...
-
Igor, Apple's been using a different format for the last few years for the inside of their XAR files. It's basically an XZ file, but split into blocks. The file signature begins with 'pbzx', followed by a 64-bit flags. Loop while flags == 0x1000000, read the next 64-bit flags read the 64-bit length of the block The block itself begins with the XZ signature '\xFD7zXZ' and length is the total size of the block plus signature. Write out the decompressed LZMA stream Loop until flags isn't 0x1000000 Currently...
-
I got it working. The hard link's path is a full path to the hard link, unlike a symlink with a relative path. One of the tricky issues was that sometimes the copy of the hardlink appears in a directory at a higher level than the original file, so it could appear first in searches, sorts, and so on. So, I had to change the code to move symlinks and hard links to the bottom of my indexed list of files so they always (numerically) appear last. During iteration 7-zip will start at index 0 and work its...
-
So, I got it working -- basically APFS treats hard links the same way as symlinks, with a relative path. I have to convert the relative path to an absolute path for 7-zip. ... and I also have to make sure symlinks and hardlinks come numerically after their original counterparts, otherwise 7-zip tries to extract the hardlink before it has extracted the original, depending on what order they appear.
-
Following the code in UI/Common/ArchiveExtractCallback, there's a very specific sequence it goes through to resolve a hard link, and it's not taking into account the current directory -- only the directory at the root of extraction. I could try passing in a formatted path like RAR does to force 7zFM through the RAR codepath, but even passing in kpidINode (true) in GetArchiveProperty doesn't get me anywhere. I can see there's a ReparsePoint/GetRawProps codepath for NTFS, but that's not where I'm getting...
-
Thanks, I'll have a look at that. I've completely rewritten the apfs handler, using a severely stripped-down version of the apfs-fuse github project. I'm not sure if the apfs-fuse license and the 7z license are compatible. Hard links are about the only thing left that aren't working atm. It's also got a bunch of duplicated code like crc32, aes, sha1/sha256 and a few other things that need to be merged in. I've replaced the zlib stuff with 7z's, but not the lzfn and lzfse code just yet.
-
I'm kindof wondering if I'm supposed to be implementing one of the RawProps interfaces or something non-obvious ...
-
I'm building out a new APFS handler, and I'm having an issue trying to sort out hard links for extraction. Screenshots attached. Basically if I extract the parent folder "bin", it reports an error for the hard link, which it's trying to resolve to the root folder of my extraction, and not the relative folder of "bin". I've tried it with the "./" prefix, I've tried without it (just the name), and have tried supplying a full path that matches the kpidPath of the original file, but nothing's working...
-
I found a reference implementation of APFS here: https://github.com/Paragon-Software-Group/paragon_apfs_sdk_ce It's pretty clean, and seems to work really well. If anything, it's answered some questions where the Apple APFS documentation is open to interpretation.
-
Still a hot mess. Attached are my code changes so far, which I'm sure still aren't right. Also included a log from running that APFS file using SHOW_DEBUG_INFO
-
Bah, still a hot mess. I'll be less spammy in the future.
-
grrr ... may be a red-herring ... still digging. I had a defect in my code where it wasn't asking for streams until it got to subarchives, and the APFS file has no subarchives.
-
Yes.
-
I've actually had a big breakthrough just now ... All this time I've been opening the file through the VMDK, and operating on the file within 7z through the VMDK handler. I extracted out the APFS file instead just a few minutes ago, and suddenly things are looking much better and I'm not getting the failures I was seeing before.
-
The APFS file I'm testing, it looks like a hot mess. I have DIR_REC entries before INODE's, FILE_EXTENT records and DSTREAM's with no INODE, INODE's with a private_id both less than, equal to, and greater than the current ID ... it's just crazy. I also have FILE_EXTENT records with valid length and type, but a position of 0.
-
So ... making some progress, but hit another barrier. This time, when decompressing resource forks. For method = 8, we hit the LzfseDecoder, and the decompression loop exits properly, but the final 7 extra byte check fails on line 244: On entry, unpackSize = 2396, packSize = 1622 On exiting the loop and checking afterward, unpackSize = 0 and packSize = 63 On exiting, the unpackSize remainder is different for each file, but consistently not 7. // LZVN encoder writes 7 additional zero bytes if (packSize...
-
What size APFS file could I construct that'd be acceptable? The APFS spec definitely talks about multiple transaction ID's assigned to a single OID, and to take the highest one under normal circumstances.
-
Around line 1939: /* Object map B-trees are sorted by object identifier and then by transaction identifier but it's possible to have identical Ids in map ? do we need to look transaction id ? and search key with largest transaction id? */ if (key.oid <= prev.oid) return false; <-------- this is returning false for index 3 prev = key; Keys.Add(key.oid); Vals.Add(val); We have two matching oid's in here with different values, which causes the loop to end prematurely, which is why volume index 3 in...
-
I've got a fairly substantial APFS file (compressed to 15gb in 7z) that by default isn't recognized by 7-zip as an APFS file. What's the best way to send you this file? I debugged it down to two places in the APFS handler code: The first starting at line 2823 (your comment about "is this possible" in the code? The answer is yes, it's possible. Uncommenting the continue gets me past this issue and onto the next issue. if (type == APFS_TYPE_DSTREAM_ID) { PRF(printf(" DSTREAM_ID")); if (pair.Key.Size()...
-
Incidentally, this is how I found the UEFI thing. I was trying to walk that code and implement the same sort of functionality in .NET.
-
There's about 1500 lines of mystical file-open voodoo in OpenArchive::OpenStream2 that does a huge amount of gymnastics to figure out which codec to use, where to look, what order the codecs go in, and ... well, generally a ton of "magic" with almost no code comments to describe the workflow. Is there any thought to moving all of that logic into its own exportable function in 7z.dll such that one could just call an entrypoint like OpenArchive, passing in callbacks and so forth, and getting back a...
-
Thanks
-
case kpidErrorFlags: { UInt32 v = 0; if (!_headersError) v |= kpv_ErrorFlags_HeadersError; if (v != 0) prop = v; break; } If you pass a non-UEFI file into this handler, and it can't identify it (it has the NArcInfoFlags::kFindSignature flag in the registration), checking kpidErrorFlags after an S_FALSE from Open() will return a Headers error instead of kpv_ErrorFlags_IsNotArc Shouldn't that code above read if (_headersError) instead? And potentially shouldn't there be a different variable to check...
-
case kpidErrorFlags: { UInt32 v = 0; if (!_headersError) v |= kpv_ErrorFlags_HeadersError; if (v != 0) prop = v; break; } If you pass a non-UEFI file into this handler, and it can't identify it (it has the NArcInfoFlags::kFindSignature flag in the registration), checking kpidErrorFlags after an S_FALSE from Open() will return a Headers error instead of kpv_ErrorFlags_IsNotArc Shouldn't that code above read if (_headersError) instead?
-
Add support for 16-bit 444 images (code in ticket)
1