In -setAsCustomIconForDirectory:withCompatibility:, the Icon\r file is not set invisible as expected (in Mac OS 10.5.7, at least).
Note that the Icon\r file is always created anew by the code:
if( ![iconrPath getFSRef:&iconrFSRef createFileIfNecessary:YES] )
return NO;
since the file was previously deleted if it already existed. Therefore, the subsequent FSGetCatalogInfo() call should never fail unless something is really wrong, which means the FinderInfo structure will not be initialized as desired. It also seems to be necessary to call FSSetCatalogInfo() as FSCreateResFile() does not appear to update the FinderInfo on an existing file as the code seems to expect.
The attached version of IconFamily.m addresses these issues and makes the method work as expected in my application.
Peter Hosey has integrated this fix for the 0.9.4 release. Please confirm that 0.9.4 resolves this issue (our testing indicates that it has), and close this issue if so. Thanks for the patch!!