alloc / free in shared library

2011-11-04
2013-05-28
  • Igor Pavlov
    Igor Pavlov
    2011-11-04

    Question to p7zip:

    I think about the ways to transfer some data like BSTR in PROPVARIANT.
    But in that case DLL (.so) and EXE must use same alloc / free pool.
    How does it work in posix/linux?
    SysAllocString in MyWindows.cpp uses malloc / free.
    Is there any problem when you call malloc / free from different places - from main executable and shared library (.so file)?
    Maybe there are some another "global" free / alloc functions?

     
  • my p7zip
    my p7zip
    2011-11-05

    In posix/linux system, the EXE and its DLL (.so) uses the same "libc.so", so you always have one global alloc pool.

    > Is there any problem when you call malloc / free from different places - from main executable and shared library (.so file)?
    I don't think so.
    Only Windows has such pitfalls ….

     
  • Igor Pavlov
    Igor Pavlov
    2011-11-05

    Do linux apllications use libc always with dynamic linking?
    It can't be linked as static code?

    Do other linux programs use that scheme, when application (like 7-Zip) uses "free" function  for pointer that was allocated by some .so plugin?
    Maybe there are more preferable ways to transfer strings from plugin?

     
  • my p7zip
    my p7zip
    2011-11-05

    > Do Linux applications use libc always with dynamic linking?
    > It can't be linked as static code?
    You can statically link a Posix application.

    But, as plugins are "DLL", it's a bit strange to mix a plugin (dynamic) and a statically linked program…

    The developer of the program that supports plugins can impose the build procedure…

    I deliver binaries of p7zip. As I want these binaries work on several Linux, the binaries are dynamically linked with "libc.so" because the "locale" library (mbstow, wtombs, …) searches the "locale" files in different directories according to the distribution of Linux …

    p7zip like many programs for Linux are open source.
    Each distribution of Linux provides a "package" to install p7zip binaries compiled under this distribution.
    So, you don't need to provide a statically linked binary …
    Only closed source programs are statically linked binary.

    > Do other linux programs use that scheme, when application (like 7-Zip) uses "free" function  for pointer that was allocated by some .so plugin?
    I don’t know.

    > Maybe there are more preferable ways to transfer strings from plugin?
    I don’t know.
    If you look at glibc  (http://developer.gnome.org/glib/stable/glib-Strings.html),
    You can see a « C » API where all allocations/reallocations/free are inside the same « DLL ».
    Perhaps, you can develop a “SevenZipString.dll” that must be used by the program and its plugins to ensure that the alloc/free always use the same allocation pool ?
    Or as you said, use the Windows “variant” API.

    What about imposing the “multithreaded DLL” model for 7-zip and its plugins ?
    It ensures that 7-zip and is plugins will use the same allocation pool, Doesn’t it ?

     

  • Anonymous
    2011-11-07

    If I recall correctly, the issue on Windows arises if the DLL and EXE use different runtime heaps (which may or may not be the case, based on which CRT they are linked to, etc). The common practice is for the DLL to provide its own "ReleaseMyObj(PVOID)" method - this not only fixes the heap issue but also abstracts the library mem-management details. *nix libraries usually offer a similar function for the latter reason.

     
  • Igor Pavlov
    Igor Pavlov
    2011-11-07

    In Windows I use PROPVARIANT.
    But there is problem there still.
    I used Variant* functions instead of PropVariant* (maybe by speed reasons),
    like VariantClear instead of PropVariantClear.
    But VariantClear doesn't support VT_BLOB, that I need now to transfer data blocks.
    So I have 2 ways (for Windows code):
    1) Upgrade to PropVariantClear and VT_BLOB
    2) Use some other interface (maybe simple memory pointers without free functions).

     
  • my p7zip
    my p7zip
    2011-11-07

    I don't remember where, but I think you already used BSTR to transfert data, didn't you ?
    (or perhaps, a 8 bits signature ? )

    "BSTR" stores :
    - an array of  WCHAR (16 bits or 32 bits)
    - the size of the array.
    You can store any value in the array of WCHAR….
    So you can transfert binary data through "BSTR"

    > 1) Upgrade to PropVariantClear and VT_BLOB
    I think that you should avoid any functions only available for Windows.

    If you want to get data from plugin, you can use a "BufferStream" (something that hasthe "ISequentialWrite" interface).

    BufferStream * vf = new BufferStream;

    call_a_plugin_function(vf);

    /* do what you want with the data stored in "bf" */

    delete bf;

    and in call_a_plugin_function( ISequentialWrite * vf) :
    vf->reserve(1024); // optimisation to avoid several realloc
    vf->write(data1,dataSize1);

    vf->write(dataN,dataSizeN);
    // where dataSize1 + … + dataSizeN = 1024

    why do you want to transfert data using blocks instead of using "streaming" or "buffered streaming" ?
    With "streaming" you can use threads :
    - one thread to produce data
    - another thread to consume data.

     
  • Igor Pavlov
    Igor Pavlov
    2011-11-07

    Yes, I use BSTR and BSTR inside PropVariant.
    I want to trasfer NT (NTFS)-security information of file (about 100-300 bytes per file).
    7-Zip can parse that data and show some info in GUI column or -slt mode.
    But if plugin shows that new property in list of properties and then sends it as BSTR, 7-Zip (at least old versions) will think that it's string and will try to print it as string. I can fix new 7-Zip, but it's not good when old versions like old plugin for FAR will call new DLL.
    Note that 7-Zip already uses CLSID inside BSTR in some places. But I don't think that I want to use standard BSTR for these files data properties.

    BTW, are there any data blocks related to files in Linux (that can be stored in archive)?
    For example, NT-security block contains owner / group  and list of other users with permissions.

     
  • my p7zip
    my p7zip
    2011-11-08

    > BTW, are there any data blocks related to files in Linux (that can be stored in archive)?
    No

    But a file can have an Access Control List (ACL).
    see http://linux.die.net/man/3/acl_get_file

    ACL can be stored as a list of string.
    example (from http://man.yolinux.com/cgi-bin/man2html?cgi_command=getfacl)

      The output format of getfacl is as follows:
           1:  # file: somedir/
           2:  # owner: lisa
           3:  # group: staff
           4:  user::rwx
           5:  user:joe:rwx       #effective:r-x
           6:  group::rwx       #effective:r-x
           7:  group:cool:r-x
           8:  mask:r-x
           9:  other:r-x
          10:  default:user::rwx
          11:  default:user:joe:rwx       #effective:r-x
          12:  default:group::r-x
          13:  default:mask:r-x
          14:  default:other:--

    Files can also have extended attributes : http://en.wikipedia.org/wiki/Extended_file_attributes#Linux

    I don't plan to support Unix ACL nor extended attributes in 7-zip archive.

    If someone wants to store/restore "Unix" attributes, then he should use the tar.xz archive where the tar is created with the
    tar (or star ?) tool provided by the OS …

    If you begin to support extented windows stuff, you will have to support :
    - hard link
    - soft link
    ( http://msdn.microsoft.com/en-us/library/windows/desktop/aa365006(v=vs.85).aspx ) .

    Should 7-zip become a full windows backup program ?

    I think that 7-zip (and so p7zip) should stay lite and be only a files archiver ….

    A files archiver stores (and compresses) :
    - name
    - path (directories)
    - data
    - basic permissions
    of files.

     

  • Anonymous
    2011-11-09

    BTW, are there any data blocks related to files in Linux (that can be stored in archive)

    On HFS+ partitions, used by OS X, files can have arbitrary data associated with a file which is stored as extended attributes ( notably, resource forks are also stored this way ).

    The built-in archivers provided by Apple (ditto and Archive Utility) preserve these metadata:

        1. For zip (and tar archives in older versions), the attributes are serialized in a separate archived file named "._" (in the AppleDouble format).
           These are often isolated into a top-level "__MACOSX" dir. Eg: The attributes for alpha/beta/gamma.txt is stored under __MACOSX/alpha/beta/._gamma.txt
        2. For tar files created on newer versions of OS X which ship with a patched version of bsdtar, PAX extensions are used.

    The "._" files are deserialized during extraction using the copyfile function (you can also use it to serialize the extended attributes into an AppleDouble file).

    Such zip files are fairly common on OS X. Usually, the common side effects of using p7zip to extract such a file is that the user is either confused by the presence of the unexpected "__MACOSX" folder, or something is slightly off ( eg: icon previews or labels are missing )

    http://en.wikipedia.org/wiki/Resource_fork
    http://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_formats
    http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man3/copyfile.3.html

     
  • Igor Pavlov
    Igor Pavlov
    2011-11-09

    ethereon:
    Can you provide examples of such ZIP / TAR acrhives?

    In Windows most files use same only small set of unique security blocks. So NTFS just stores the link to table of security blocks to reduce size.

    Do most MAC files have same extended attributes or resource forks?

     
  • Igor Pavlov
    Igor Pavlov
    2011-11-09

    p7zip:

    I don't plan to support hardlinks. It looks difficult for me now.
    But I plan to support security info (it will be stored as file property like timesamps or winAttributes). Also I plan to support NT substreams, like (c:\file.txt:substream). And it will be stored as usual file.

     

  • Anonymous
    2011-11-09

    Here's an example file, created using the Archive Utility (Finder's "Compress" option) : http://dl.dropbox.com/u/5479813/xattr-sample.zip . I manually set the extended attributes for these.

    Usually, files on OS X don't have extended attributes. However, it is fairly common for users to "customize" files in Finder (OS X's equivalent of Windows Explorer). For instance, they may add a color label or change the icon. These are stored as extended attributes.

    In addition, OS X uses extended attributes for its "quarantine" mechanism. So, files downloaded from the internet automatically get an extended attribute.

    The use of resource forks has declined in newer versions of OS X. Files which store critical data in resource forks are rare these days. An example from not so long ago would be Safari's webloc files which used to store the bookmark info in the resource fork:

    $ xattr -l Apple.webloc 
    com.apple.ResourceFork:
    ...
    00000140  00 00 00 00 00 00 00 15 68 74 74 70 3A 2F 2F 77  |........http://w|
    00000150  77 77 2E 61 70 70 6C 65 2E 63 6F 6D 2F 00 00 00  |ww.apple.com/...|
    00000160  15 68 74 74 70 3A 2F 2F 77 77 77 2E 61 70 70 6C  |.http://www.appl|
    00000170  65 2E 63 6F 6D 2F 00 00 00 05 41 70 70 6C 65 00  |e.com/....Apple.|
    ...
    
     
  • Igor Pavlov
    Igor Pavlov
    2011-11-09

    Can you provide .TAR example also?

     

  • Anonymous
    2011-11-09

    http://dl.dropbox.com/u/5479813/xattr-sample.tar

    Created using the patched bsdtar 2.8.3 which ships with OS X 10.7.2.

     
  • Igor Pavlov
    Igor Pavlov
    2011-11-09

    I don't see any change in TAR header for that special file.
    The only difference is name prefix "._"

     

  • Anonymous
    2011-11-09

    Correction for post #10: The new patched versions of bsdtar also use the AppleDouble format.

    The difference is that the zip files isolate the attribute files into the __MACOSX dir, while the tar stores it along with the original file ( as can be seen in the sample zip and tar files ).

     
  • my p7zip
    my p7zip
    2011-11-09

    Remark :

    restoring "NT (NTFS)-security information" or "Unix ACL" means something only if the machine used to store these security information has the same "security domaine" or is the same as the machine used to restore these security information.

    For exemple: what do you plan to do if you want to restore security information for a user "user1" or a group "group1" that don't exist …

    So, storing and restoring security block  should be an option that is "off" by defaut.

    The MacOSX OS use tricks to store extended attributes on filesystem like FAT32 or NTFS: each file named "filename" has its extended attributes in a file "._filename"…

    the MacOSX "tar" uses this same trick to store extended attributes.
    the MacOSX "zip" uses another trick ( __MACOSX dir).

    For p7zip, I wanted to ignore this kind of files but I did nothing …

     
  • my p7zip
    my p7zip
    2011-11-09

    to ethereon :
    You develop several patches for p7zip 9.20.

    question 1 :  patch for p7zip which merges the AppleDouble extended attributes during extraction. It works for both zip and tar variants.
    Does this patch work for 7z archive ?

    question 2 :
    On what system/OS do you build and test your version of p7zip ?

     

  • Anonymous
    2011-11-10

    Does this patch work for 7z archive ?

    The patch is format-independent. It works for any archive that stores AppleDouble-encoded metadata in one of the two standard schemes.

    For example: If you re-archive the contents of xattr-sample.zip above (including ._* files) using 7z and then extract the 7z archive using the patched version, the extended attributes will be applied correctly.

    On what system/OS do you build and test your version of p7zip ?

    Only on Mac OS X 10.6 and 10.7. However, the patches which aren't OS X-specific are posix compliant.