I've run into an issue with several dmg files from Apple. An example file is this one (you need to log in to Apple first): http://adcdownload.apple.com/Developer_Tools/xcode_4.4.1/xcode_4.4.1_6938145.dmg
Extracting all files from this HFS container results in many files that are zero length but should not be. Those same files show as zero length in the HFSExplorer window before extraction. The fully extracted dmg file on my windows box is 906MB, while the same DMG opened on my Mac is over 3GB. An example of a problematic file is code.app/Contents/Info.plist. On the Mac this is a 14K file. Viewed in HFSExplorer and extracted with either the GUI or UnHFS it is a zero length file.
I have debugged this a little bit but I'm interested in help from someone with more background on HFS. The data in the JParted FSEntry object appears to show this as a zero length file, so I don't know if this problem is indicative of an update to the HFS format by Apple or just a bug in this code. I'll add a post below with additional information on what I see in the debugger, and I'm grateful for any help you can provide.
Here's what I see in the HFSCommonFSFile object of the Xcode.app/Contents/Info.plist file that is zero length but should not be:
entry HFSCommonFSFile (id=71)
attributes HFSCommonFSAttributes (id=78)
catalogFile CommonHFSCatalogFile$HFSPlusImplementation (id=81)
dataFork HFSCommonFSFork (id=58)
forkData CommonHFSForkData$HFSPlusImplementation (id=117)
hper HFSPlusForkData (id=124)
extents HFSPlusExtentRecord (id=127)
array HFSPlusExtentDescriptor (id=132)
 HFSPlusExtentDescriptor (id=136)
 HFSPlusExtentDescriptor (id=137)
 HFSPlusExtentDescriptor (id=138)
 HFSPlusExtentDescriptor (id=139)
 HFSPlusExtentDescriptor (id=140)
 HFSPlusExtentDescriptor (id=142)
 HFSPlusExtentDescriptor (id=143)
 HFSPlusExtentDescriptor (id=144)
parent HFSCommonFSFile (id=71)
parent HFSCommonFSFile (id=71)
type FSForkType (id=120)
fileRecord CommonHFSCatalogFileRecord$HFSPlusImplementation (id=88)
keyRecord CommonHFSCatalogFileRecord$HFSPlusImplementation (id=88)
parent HFSPlusFileSystemHandler (id=90)
parentFileSystem HFSPlusFileSystemHandler (id=90)
resourceFork HFSCommonFSFork (id=93)
parentDirPath String (id=74)
One immediate question is whether these files are compressed. HFS+ compression (added in Mac OS X 10.6) is not yet supported by HFSExplorer.
I'm currently downloading the disk image so I can check that out myself, but it takes a little while since I'm currently on a slow connection.
Thanks so much for your reply. That would certainly help explain it. The DMG appears to be compressed to a fair degree, since the file itself is ~1.8GB while the mounted image on a Mac is ~3.5GB.
Can you offer any info on the outlook for implementing support for HFS+ compression (and perhaps also the size of the effort)? Is it a standard compression algorithm for which one might be able to find an OSS implementation (and then just wire it into the codebase), or is it an Apple specific algorithm? If the latter, is it documented?
Depending on the size of the effort, I might be able to offer to contribute to the project.
Thanks again for your help,
I dug around a little after your comment and I can confirm your suspicion:
$ sudo hfsdebug-lite Info.plist
<Catalog B-Tree node = 1433 (sector 0x28b98)>
path = Xcode:/Xcode.app/Contents/Info.plist
# Catalog File Record
type = file
file ID = 27014
flags = 0000000010000110
. File has a thread record in the catalog.
. File has extended attributes.
reserved1 = 0
createDate = Thu Aug 2 20:59:48 2012
contentModDate = Thu Aug 2 21:24:36 2012
attributeModDate = Thu Aug 2 21:24:36 2012
accessDate = Thu Aug 2 21:02:40 2012
backupDate = 0
# BSD Info
ownerID = 0 (root)
groupID = 0 (wheel)
adminFlags = 00000000
ownerFlags = 00100000
fileMode = -rw-r--r--
linkCount = 1
textEncoding = 0
attrBlocks = 0
# Finder Info
fdType = 0
fdCreator = 0
fdFlags = 0000000000000000
fdLocation = (v = 0, h = 0)
opaque = 0
# Data Fork
logicalSize = 0 bytes
# Resource Fork
logicalSize = 7988 bytes
totalBlocks = 2
fork temperature = no HFC record in B-Tree
clumpSize = 0
extents = startBlock blockCount % of file
0x4443 0x2 100.00 %
2 allocation blocks in 1 extents total.
2.00 allocation blocks per extent on an average.
<Attributes B-Tree node = 2197 (sector 0xab58)>
# Attribute Key
keyLength = 46
pad = 0
fileID = 27014
startBlock = 0
attrNameLen = 17
attrName = com.apple.decmpfs
# Inline Data
recordType = 0x10
reserved = 0
reserved = 0
attrSize = 16 bytes
attrData = 66 70 6d 63 04 00 00 00 eb 37 00 00 00 00 00 00
f p m c 7
compression magic = cmpf
compression type = 4 (resource fork has compressed data)
uncompressed size = 14315 bytes
I'm still very interested in your thoughts on what I wrote above. Thanks again.
I found some C code here here that reads HFS+ compressed files and it looks like it's using standard zlib compression. I expect this could be supported in HFSExplorer using java.util.zip.Inflater. Still work to read and manage the extended attributes, but maybe not too bad.
It was quite a bit more work than I initially suspected and I had to step away for a couple of months for other work related issues, but I've completed an initial revision supporting HFSPlus Compression for UnHFS. There's still more work to be done to support it through the HFSExplorer GUI but the groundwork is there.
Can I send you a git patch for review?
That's great news! However I think merging this patch may prove a bit difficult because I've also done some work on this… although not completed yet. It's mostly related to proper handling of the Attributes file..
Anyhow, please send me the patch so I can evaluate how to integrate it.
Sounds good. If you need to refactor or throw away some of the code I wrote, it's not an issue for me at all. Thanks for writing this and open sourcing it. Nice tool, and greatly appreciated.
Just wanted to confirm that you got my e-mail with the patch attached. I sent it to your sourceforge e-mail address.
Yes, I have it. Sorry for being so slow but I have had little time to review it. Will look into it this weekend.
No rush at all - I just wanted to make sure it got to you via the sourceforge e-mail address.
Thanks for your time!