MSYS2 / Tickets / #34 split developer package from runtime package

Andrea Zagli - 2014-04-13

not only for libreries: for example postgresql

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Alexx83 - 2014-04-13

This is big work on splitting packages. I don't have time for it

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrea Zagli - 2014-04-20

can i help you in any way?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ray Donnelly - 2014-04-20

Why must you "redistribute the entire package"?

You can delete any static libraries and header files from the MSYS2 packages that your own software depends on. If it's GPLed software it may be wise to also redistribute absolutely everything anyway including the package's directory from MSYS2-packages or MINGW-packages (i.e. the PKGBUILD and patch files). I am not a lawyer so this is not legal advice.

If you don't want the end-user to see a lot of files, you could package up all these extra bits into an archive file contained in your main archive.

While you could add split packaging to the PKGBUILD files you are having these problems with, it does add to Alexey's workload as he handles all package signing and uploading, so in the costs-benefits equation, I'd prefer we don't split them up without very good reason (e.g. a large percentage of developer-only data in a package - in terms of file size, not file count).

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Alexx83 - 2014-04-20

The biggest problem with splitting packages into some parts is that pacman handle this situation by writing own function for every package. See for example mingw-w64-gcc PKGBUILD. And also I need write 2 wrappers for every this function to handle 32 and 64 bit packages for mingw. This is very big work to rewrite all PKGBUILD's.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

rubenvb - 2014-07-22

This is not how pacman (eh, Arch) works, and this is a good thin, IMHO. The fact that MSYS2 installs everything including binaries and static libraries does not mean you as a developer must redistribute all that mess. Usually the DLL will do. No one is forcing you to redistribute the MSYS2 packages as a whole alongside your distributed binaries.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrea Zagli - 2014-07-23

i didn't understand... if i want my software works i must distribute every thing it needs: if my software use libxml i must supply it

in gnu/linux it is more easy thanks to packages software as dpkg, yum and pacman but in windows i must make a single package (installer) with everything inside (look at gimp, inkscape, scribus, etc.)

i don't know arch, but debian and redhat/fedora split packages: it is useless to have developing files (include, etc.) in a system where i don't develop and where i will never compile something

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Alexx83 - 2014-07-23

MSYS2 is created for develop not for distribution.
If you need distribute some program using our prebuilded packages you need create installer yourself and decide what files it needed.

Regards,
Alexey.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

rubenvb - 2014-07-23

What I meant is that nothing is preventing you from only including those files you need to distribute for your application to work. You can just include the necessary DLLs in your installer. This is not at all influenced by what the MSYS2 packages contain or not. If your installer needs the libxml DLL, take it from /mingw??/bin/ and include it (and only it) with your application. That's how the applications you name do things.

And yes, this is how Arch (and thus its packaging system) works, in contrast to most other distros.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrea Zagli - 2014-07-23

yes i known that i can choose files from myself but seeing that i'm not the developer of the DLL i don't know exactly what files i need

what i'm doing right now (with msys, not msys2) is using ready-made binary zip files from some repositories (msys, opensuse, etc.), so and i can create the installer that i'm sure it works (without going to attempt)

it is simpler

and yes i understand that pacman doesn't make the think easy

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

rubenvb - 2014-07-23

As a developer you should know what libraries your application depends on.
You can find out what shared libraries your app depends on by running the MSYS2/Cygwin ldd utility on your executable. Ignore all files it shows in C:\Windows and its subdirectories.
If you know what packages you installed to develop your application, you can also find out what shared libraries they come with by running (this is an example for libxml2):

pacman -Ql mingw-w64-x86_64-libxml2 | grep dll

This lists the DLL files that are in the 64-bit libxml2 package. What this command returns can be used as input for a packaging script, although that will then depend on MSYS2's pacman. Note that some libraries may need some configuration files as well.

As you can see, if you built your software using the MinGW-w64 libraries in MSYS2's repository, you don't need to go look for anything else, as it is already present on your system.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrea Zagli - 2014-07-23

as you said: "Note that some libraries may need some configuration files as well"

and i add: "and many other possible files"

and not only dll

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ray Donnelly - 2014-07-23

Generally speaking, for an application or library that follows FHS that we've ported to MSYS2, you should delete the "lib" and "include" folders and keep the rest ("bin", "share" etc).

With regards to split packages, I am strongly opposed to making our packages and package building scripts more complicated for its developers or our normal users just to suit someone who wants to re-package some of our stuff with his or her own installer.

While you are well within your legal right to do this (you can even use the qt-installer-framework that we provide and use our msys2-installer package as a reference if you want to); I would like to point out that software distribution via installers is the opposite to what MSYS2 is all about! Yes, we do also distribute an installer for MSYS2 itself, but I'd like this to be the installer that solves (or at least helps to solve) the installer-proliferation and software update problems that are endemic on Windows.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

rubenvb - 2014-07-23

Note that your packaging can be fully automated:

copy GCC runtime DLLs

for each binary:

run ldd and find 3rd party dlls

compile list of packages using pacman -Qo somedll.dll

extract all packages from /var/cache/pacman/pkg to somewhere

delete lib and include directories from somewhere

copy binary

zip everything into your installer.

Honestly? It can't get any simpler than this :-) Someone could easily write a general shell script for this!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Oh, who am I kidding, example deployment script:

#! /usr/bin/env sh

if [ "$#" -ne 2 ];
then
  printf "Usage: ./deploy.sh exe directory\n"
fi

list=$(ldd $1 | grep /mingw64 | sed 's/.dll.*/.dll/')
for dll in $list;
do
  pkg=`pacman -Qo $dll | sed 's/.* is owned by //' | tr ' ' '-'`
  pkglist="$pkglist $pkg"
done
# remove duplicates
pkglist=`echo $pkglist | tr ' ' '\n' | sort | uniq`
printf "$pkglist\n"

mkdir -p "$2"

for pkg in $pkglist
do
  tmp=`mktemp -d`
  cd $tmp
  tar -xf /var/cache/pacman/pkg/$pkg-any.pkg.tar.xz
  # more fine-grained control is possible here
  cp -r $PWD/mingw64/bin $2
  cp -r $PWD/mingw64/share $2
done

Example use:

./deploy.sh /mingw64/bin/iconv.exe /testdir

will place all the dependencies of iconv.exe into testdir.

This is a bit rough around the edges, but gets the basic idea across. In the end, only you know what files you will need exactly. The iconv example is a great one that shows that a single package can contain a bunch of binaries you may or may not need.

Hope I have helped you a bit. Otherwise let this be found by future generations!

Ray Donnelly - 2014-10-12

Ruben, could you maybe contribute this to MSYS2 somehow?

Though really, I am a bit conflicted about building external packages / installers from MSYS2 packages since you'd be contributing to DLL hell whereas by keeping your packages within the MSYS2 system you aren't.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Macek - 2017-02-05

status: open --> closed
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Macek - 2017-02-05

I added a link to this script from our wiki. Although split packages would be nice, there hasn't been anyone to do that work. I hope to someday split Qt due to its hugeness, but others will have to stay I guess. Closing. Further discussion is possible on other channels.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

split developer package from runtime package

Software Distribution and Building Platform for Windows

Milestone

Searches

Help

#34 split developer package from runtime package

Discussion