Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

Apple DMG files: lots of zero length files

Joe Toomey
2012-11-29
2013-05-08
  • Joe Toomey
    Joe Toomey
    2012-11-29

    I've run into an issue with several dmg files from Apple.  An example file is this one (you need to log in to Apple first):  http://adcdownload.apple.com/Developer_Tools/xcode_4.4.1/xcode_4.4.1_6938145.dmg

    Extracting all files from this HFS container results in many files that are zero length but should not be.  Those same files show as zero length in the HFSExplorer window before extraction.  The fully extracted dmg file on my windows box is 906MB, while the same DMG opened on my Mac is over 3GB.  An example of a problematic file is code.app/Contents/Info.plist.  On the Mac this is a 14K file.  Viewed in HFSExplorer and extracted with either the GUI or UnHFS it is a zero length file.

    I have debugged this a little bit but I'm interested in help from someone with more background on HFS.  The data in the JParted FSEntry object appears to show this as a zero length file, so I don't know if this problem is indicative of an update to the HFS format by Apple or just a bug in this code.  I'll add a post below with additional information on what I see in the debugger, and I'm grateful for any help you can provide.

    Thanks,
    -Joe

     
  • Joe Toomey
    Joe Toomey
    2012-11-29

    Here's what I see in the HFSCommonFSFile object of the Xcode.app/Contents/Info.plist file that is zero length but should not be:

    entry   HFSCommonFSFile  (id=71)    
        attributes  HFSCommonFSAttributes  (id=78)  
        catalogFile CommonHFSCatalogFile$HFSPlusImplementation  (id=81) 
        dataFork    HFSCommonFSFork  (id=58)    
            forkData    CommonHFSForkData$HFSPlusImplementation  (id=117)   
                hper    HFSPlusForkData  (id=124)   
                    clumpSize   (id=126)    
                        [0] 0   
                        [1] 0   
                        [2] 0   
                        [3] 0   
                    extents HFSPlusExtentRecord  (id=127)   
                        array   HFSPlusExtentDescriptor[8]  (id=132)    
                            [0] HFSPlusExtentDescriptor  (id=136)   
                                blockCount  (id=147)    
                                    [0] 0   
                                    [1] 0   
                                    [2] 0   
                                    [3] 0   
                                startBlock  (id=148)    
                                    [0] 0   
                                    [1] 0   
                                    [2] 0   
                                    [3] 0   
                            [1] HFSPlusExtentDescriptor  (id=137)   
                                blockCount  (id=150)    
                                startBlock  (id=151)    
                            [2] HFSPlusExtentDescriptor  (id=138)   
                                blockCount  (id=152)    
                                startBlock  (id=153)    
                            [3] HFSPlusExtentDescriptor  (id=139)   
                                blockCount  (id=154)    
                                startBlock  (id=155)    
                            [4] HFSPlusExtentDescriptor  (id=140)   
                                blockCount  (id=156)    
                                startBlock  (id=157)    
                            [5] HFSPlusExtentDescriptor  (id=142)   
                                blockCount  (id=158)    
                                startBlock  (id=159)    
                            [6] HFSPlusExtentDescriptor  (id=143)   
                                blockCount  (id=160)    
                                startBlock  (id=161)    
                            [7] HFSPlusExtentDescriptor  (id=144)   
                                blockCount  (id=162)    
                                startBlock  (id=163)    
                    logicalSize (id=129)    
                        [0] 0   
                        [1] 0   
                        [2] 0   
                        [3] 0   
                        [4] 0   
                        [5] 0   
                        [6] 0   
                        [7] 0   
                    totalBlocks (id=130)    
                        [0] 0   
                        [1] 0   
                        [2] 0   
                        [3] 0   
            parent  HFSCommonFSFile  (id=71)    
            parent  HFSCommonFSFile  (id=71)    
            type    FSForkType  (id=120)    
        fileRecord  CommonHFSCatalogFileRecord$HFSPlusImplementation  (id=88)   
        keyRecord   CommonHFSCatalogFileRecord$HFSPlusImplementation  (id=88)   
        parent  HFSPlusFileSystemHandler  (id=90)   
        parentFileSystem    HFSPlusFileSystemHandler  (id=90)   
        resourceFork    HFSCommonFSFork  (id=93)    
    parentDirPath   String[2]  (id=74)
    
     
  • Erik Larsson
    Erik Larsson
    2012-11-29

    One immediate question is whether these files are compressed. HFS+ compression (added in Mac OS X 10.6) is not yet supported by HFSExplorer.
    I'm currently downloading the disk image so I can check that out myself, but it takes a little while since I'm currently on a slow connection.

     
  • Joe Toomey
    Joe Toomey
    2012-11-29

    Thanks so much for your reply.  That would certainly help explain it.  The DMG appears to be compressed to a fair degree, since the file itself is ~1.8GB while the mounted image on a Mac is ~3.5GB.

    Can you offer any info on the outlook for implementing support for HFS+ compression (and perhaps also the size of the effort)?  Is it a standard compression algorithm for which one might be able to find an OSS implementation (and then just wire it into the codebase), or is it an Apple specific algorithm?  If the latter, is it documented?

    Depending on the size of the effort, I might be able to offer to contribute to the project.

    Thanks again for your help,
    -Joe

     
  • Joe Toomey
    Joe Toomey
    2012-11-29

    I dug around a little after your comment and I can confirm your suspicion:

    $ sudo hfsdebug-lite Info.plist 
    Password:
      <Catalog B-Tree node = 1433 (sector 0x28b98)>
      path                 = Xcode:/Xcode.app/Contents/Info.plist
    # Catalog File Record
      type                 = file
      file ID              = 27014
      flags                = 0000000010000110
                           . File has a thread record in the catalog.
                           . File has extended attributes.
      reserved1            = 0
      createDate           = Thu Aug  2 20:59:48 2012
      contentModDate       = Thu Aug  2 21:24:36 2012
      attributeModDate     = Thu Aug  2 21:24:36 2012
      accessDate           = Thu Aug  2 21:02:40 2012
      backupDate           = 0
      # BSD Info
      ownerID              = 0 (root)
      groupID              = 0 (wheel)
      adminFlags           = 00000000
      ownerFlags           = 00100000
      fileMode             = -rw-r--r--
      linkCount            = 1
      textEncoding         = 0
      attrBlocks           = 0
      # Finder Info
      fdType               = 0
      fdCreator            = 0
      fdFlags              = 0000000000000000
      fdLocation           = (v = 0, h = 0)
      opaque               = 0
      # Data Fork
      logicalSize          = 0 bytes
      # Resource Fork
      logicalSize          = 7988 bytes
      totalBlocks          = 2
      fork temperature     = no HFC record in B-Tree
      clumpSize            = 0
      extents              =   startBlock   blockCount      % of file
                                   0x4443          0x2       100.00 %
                             2 allocation blocks in 1 extents total.
                             2.00 allocation blocks per extent on an average.
    # Attributes
      <Attributes B-Tree node = 2197 (sector 0xab58)>
      # Attribute Key
      keyLength            = 46
      pad                  = 0
      fileID               = 27014
      startBlock           = 0
      attrNameLen          = 17
      attrName             = com.apple.decmpfs
      # Inline Data
      recordType           = 0x10
      reserved[0]          = 0
      reserved[1]          = 0
      attrSize             = 16 bytes
      attrData             = 66 70 6d 63 04 00 00 00 eb 37 00 00 00 00 00 00 
                              f  p  m  c                 7                   
                             
      compression magic    = cmpf
      compression type     = 4 (resource fork has compressed data)
      uncompressed size    = 14315 bytes
    

    I'm still very interested in your thoughts on what I wrote above.  Thanks again.
    -Joe

     
  • Joe Toomey
    Joe Toomey
    2012-11-29

    I found some C code here here that reads HFS+ compressed files and it looks like it's using standard zlib compression.  I expect this could be supported in HFSExplorer using java.util.zip.Inflater.  Still work to read and manage the extended attributes, but maybe not too bad.

    Thoughts?

     
  • Joe Toomey
    Joe Toomey
    2013-01-23

    It was quite a bit more work than I initially suspected and I had to step away for a couple of months for other work related issues, but I've completed an initial revision supporting HFSPlus Compression for UnHFS.  There's still more work to be done to support it through the HFSExplorer GUI but the groundwork is there. 

    Can I send you a git patch for review?

     
  • Erik Larsson
    Erik Larsson
    2013-01-24

    That's great news! However I think merging this patch may prove a bit difficult because I've also done some work on this… although not completed yet. It's mostly related to proper handling of the Attributes file..

    Anyhow, please send me the patch so I can evaluate how to integrate it.

     
  • Joe Toomey
    Joe Toomey
    2013-01-24

    Sounds good.  If you need to refactor or throw away some of the code I wrote, it's not an issue for me at all.  Thanks for writing this and open sourcing it.  Nice tool, and greatly appreciated.

     
  • Joe Toomey
    Joe Toomey
    2013-02-01

    Just wanted to confirm that you got my e-mail with the patch attached.  I sent it to your sourceforge e-mail address.

    Thanks!

     
  • Erik Larsson
    Erik Larsson
    2013-02-01

    Yes, I have it. Sorry for being so slow but I have had little time to review it. Will look into it this weekend.

     
  • Joe Toomey
    Joe Toomey
    2013-02-01

    No rush at all - I just wanted to make sure it got to you via the sourceforge e-mail address.

    Thanks for your time!