#34 split developer package from runtime package

1.0
open
nobody
None
2014-10-12
2014-04-13
No

for libraries is it possibile to split developer files from runtime files?

for example in libxml2 package there are both developer files (in include directory) and runtime files (in bin directory)

as well as are made packages if i would redistribute runtime files i must redistribute the entire package; and there are a lot of unnecessary files

Discussion

  • Andrea Zagli

    Andrea Zagli - 2014-04-13

    not only for libreries: for example postgresql

     
  • Alexx83

    Alexx83 - 2014-04-13

    This is big work on splitting packages. I don't have time for it

     
  • Andrea Zagli

    Andrea Zagli - 2014-04-20

    can i help you in any way?

     
  • Ray Donnelly

    Ray Donnelly - 2014-04-20

    Why must you "redistribute the entire package"?

    You can delete any static libraries and header files from the MSYS2 packages that your own software depends on. If it's GPLed software it may be wise to also redistribute absolutely everything anyway including the package's directory from MSYS2-packages or MINGW-packages (i.e. the PKGBUILD and patch files). I am not a lawyer so this is not legal advice.

    If you don't want the end-user to see a lot of files, you could package up all these extra bits into an archive file contained in your main archive.

    While you could add split packaging to the PKGBUILD files you are having these problems with, it does add to Alexey's workload as he handles all package signing and uploading, so in the costs-benefits equation, I'd prefer we don't split them up without very good reason (e.g. a large percentage of developer-only data in a package - in terms of file size, not file count).

     
  • Alexx83

    Alexx83 - 2014-04-20

    The biggest problem with splitting packages into some parts is that pacman handle this situation by writing own function for every package. See for example mingw-w64-gcc PKGBUILD. And also I need write 2 wrappers for every this function to handle 32 and 64 bit packages for mingw. This is very big work to rewrite all PKGBUILD's.

     
  • rubenvb

    rubenvb - 2014-07-22

    This is not how pacman (eh, Arch) works, and this is a good thin, IMHO. The fact that MSYS2 installs everything including binaries and static libraries does not mean you as a developer must redistribute all that mess. Usually the DLL will do. No one is forcing you to redistribute the MSYS2 packages as a whole alongside your distributed binaries.

     
  • Andrea Zagli

    Andrea Zagli - 2014-07-23

    i didn't understand... if i want my software works i must distribute every thing it needs: if my software use libxml i must supply it

    in gnu/linux it is more easy thanks to packages software as dpkg, yum and pacman but in windows i must make a single package (installer) with everything inside (look at gimp, inkscape, scribus, etc.)

    i don't know arch, but debian and redhat/fedora split packages: it is useless to have developing files (include, etc.) in a system where i don't develop and where i will never compile something

     
  • Alexx83

    Alexx83 - 2014-07-23

    MSYS2 is created for develop not for distribution.
    If you need distribute some program using our prebuilded packages you need create installer yourself and decide what files it needed.

    Regards,
    Alexey.

     
  • rubenvb

    rubenvb - 2014-07-23

    What I meant is that nothing is preventing you from only including those files you need to distribute for your application to work. You can just include the necessary DLLs in your installer. This is not at all influenced by what the MSYS2 packages contain or not. If your installer needs the libxml DLL, take it from /mingw??/bin/ and include it (and only it) with your application. That's how the applications you name do things.

    And yes, this is how Arch (and thus its packaging system) works, in contrast to most other distros.

     
  • Andrea Zagli

    Andrea Zagli - 2014-07-23

    yes i known that i can choose files from myself but seeing that i'm not the developer of the DLL i don't know exactly what files i need

    what i'm doing right now (with msys, not msys2) is using ready-made binary zip files from some repositories (msys, opensuse, etc.), so and i can create the installer that i'm sure it works (without going to attempt)

    it is simpler

    and yes i understand that pacman doesn't make the think easy

     
  • rubenvb

    rubenvb - 2014-07-23

    As a developer you should know what libraries your application depends on.
    You can find out what shared libraries your app depends on by running the MSYS2/Cygwin ldd utility on your executable. Ignore all files it shows in C:\Windows and its subdirectories.
    If you know what packages you installed to develop your application, you can also find out what shared libraries they come with by running (this is an example for libxml2):

    pacman -Ql mingw-w64-x86_64-libxml2 | grep dll
    

    This lists the DLL files that are in the 64-bit libxml2 package. What this command returns can be used as input for a packaging script, although that will then depend on MSYS2's pacman. Note that some libraries may need some configuration files as well.

    As you can see, if you built your software using the MinGW-w64 libraries in MSYS2's repository, you don't need to go look for anything else, as it is already present on your system.

     
  • Andrea Zagli

    Andrea Zagli - 2014-07-23

    as you said: "Note that some libraries may need some configuration files as well"

    and i add: "and many other possible files"

    and not only dll

     
  • Ray Donnelly

    Ray Donnelly - 2014-07-23

    Generally speaking, for an application or library that follows FHS that we've ported to MSYS2, you should delete the "lib" and "include" folders and keep the rest ("bin", "share" etc).

    With regards to split packages, I am strongly opposed to making our packages and package building scripts more complicated for its developers or our normal users just to suit someone who wants to re-package some of our stuff with his or her own installer.

    While you are well within your legal right to do this (you can even use the qt-installer-framework that we provide and use our msys2-installer package as a reference if you want to); I would like to point out that software distribution via installers is the opposite to what MSYS2 is all about! Yes, we do also distribute an installer for MSYS2 itself, but I'd like this to be the installer that solves (or at least helps to solve) the installer-proliferation and software update problems that are endemic on Windows.

     
  • rubenvb

    rubenvb - 2014-07-23

    Note that your packaging can be fully automated:

    1. copy GCC runtime DLLs
    2. for each binary:
      • run ldd and find 3rd party dlls
      • compile list of packages using pacman -Qo somedll.dll
      • extract all packages from /var/cache/pacman/pkg to somewhere
      • delete lib and include directories from somewhere
      • copy binary
    3. zip everything into your installer.

    Honestly? It can't get any simpler than this :-) Someone could easily write a general shell script for this!

     
  • rubenvb

    rubenvb - 2014-07-23

    Oh, who am I kidding, example deployment script:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    #! /usr/bin/env sh
    
    if [ "$#" -ne 2 ];
    then
      printf "Usage: ./deploy.sh exe directory\n"
    fi
    
    list=$(ldd $1 | grep /mingw64 | sed 's/.dll.*/.dll/')
    for dll in $list;
    do
      pkg=`pacman -Qo $dll | sed 's/.* is owned by //' | tr ' ' '-'`
      pkglist="$pkglist $pkg"
    done
    # remove duplicates
    pkglist=`echo $pkglist | tr ' ' '\n' | sort | uniq`
    printf "$pkglist\n"
    
    mkdir -p "$2"
    
    for pkg in $pkglist
    do
      tmp=`mktemp -d`
      cd $tmp
      tar -xf /var/cache/pacman/pkg/$pkg-any.pkg.tar.xz
      # more fine-grained control is possible here
      cp -r $PWD/mingw64/bin $2
      cp -r $PWD/mingw64/share $2
    done
    

    Example use:

    ./deploy.sh /mingw64/bin/iconv.exe /testdir
    

    will place all the dependencies of iconv.exe into testdir.

    This is a bit rough around the edges, but gets the basic idea across. In the end, only you know what files you will need exactly. The iconv example is a great one that shows that a single package can contain a bunch of binaries you may or may not need.

    Hope I have helped you a bit. Otherwise let this be found by future generations!

     
  • Ray Donnelly

    Ray Donnelly - 2014-10-12

    Ruben, could you maybe contribute this to MSYS2 somehow?

    Though really, I am a bit conflicted about building external packages / installers from MSYS2 packages since you'd be contributing to DLL hell whereas by keeping your packages within the MSYS2 system you aren't.

     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks