MSYS2 / Tickets / #94 Problem with shell scripts

Alexx83 - 2014-10-12

can you post your build script?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ralph engels - 2014-10-12

Sure.

build script for libvirt.

PYTHON_PREFIX=python-config.sh --prefix
PYTHON_INCLUDES="${PYTHON_PREFIX}/include/python2.7"
PYTHON_LIBS=python-config.sh --ldflags
ABSPATH=$(cd "$(dirname "$0")"; pwd)
BUILDDIR=${ABSPATH}/_build
SRCDIR=../../libvirt
INSTDIR=${ABSPATH}/_inst
export CPATH="/mingw32/i686-w64-mingw32/include:/mingw32/include:${PYTHON_INCLUDES}"
export LPATH="/mingw32/i686-w64-mingw32/lib:/mingw32/lib"
export CFLAGS="-O3 -fstack-protector-all -DUSE_MINGW_ANSI_STDIO=1 -DWINPTHREAD_STATIC"
export CXXFLAGS="-O3 -fstack-protector-all -DUSE_MINGW_ANSI_STDIO=1 -DWINPTHREAD_STATIC"
export LIBRARY_PATH="/mingw32/lib/gcc/i686-w64-mingw32/4.9.1"
export LDFLAGS="${PYTHON_LIBS} -Wl,--nxcompat -Wl,--dynamicbase -Wl,--warn-common -g"
rm -rf ${BUILDDIR} &&
mkdir ${BUILDDIR} &&
cd ${BUILDDIR} &&
${SRCDIR}/configure \ --enable-static \ --without-xen \ --without-qemu \ --without-openvz \ --without-lxc \ --without-vbox \ --without-xenapi \ --without-avahi \ --without-polkit \ --without-libvirtd \ --without-uml \ --with-phyp \ --with-esx \ --without-hyperv \ --without-vmware \ --without-parallels \ --without-netcf \ --without-audit \ --without-dtrace \ --enable-expensive-tests \ --build=i686-w64-mingw32 \ --prefix=${INSTDIR} && make && make install && echo "done"

its a bit messy but as an experiment i removed all the CFLAGS CPATH and all the other exports as well as the custom LDFLAGS leaving only

../libvirt/configure \ --enable-static \ --without-xen \ --without-qemu \ --without-openvz \ --without-lxc \ --without-vbox \ --without-xenapi \ --without-avahi \ --without-polkit \ --without-libvirtd \ --without-uml \ --with-phyp \ --with-esx \ --without-hyperv \ --without-vmware \ --without-parallels \ --without-netcf \ --without-audit \ --without-dtrace \ --enable-expensive-tests \ --build=i686-w64-mingw32 \

and it still fails. strangely enough the same works if i write it directly in the shell.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Alexx83 - 2014-10-12

I'm create PKGBUILD to build libvirt and push it to MINGW-packages repo. I don't see this error with it.
Will try your build script.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Alexx83 - 2014-10-12

I modify your script small and it work for me:

!/usr/bin/bash

PATH=/mingw32/bin:$PATH
PYTHON_PREFIX=$(python-config.sh --prefix)
PYTHON_INCLUDES="${PYTHON_PREFIX}/include/python2.7"
PYTHON_LIBS=$(python-config.sh --ldflags)
ABSPATH=$(cd "$(dirname "$0")"; pwd)
BUILDDIR=${ABSPATH}/_build
SRCDIR=../libvirt-1.2.9
INSTDIR=${ABSPATH}/_inst
export CPATH="/mingw32/i686-w64-mingw32/include:/mingw32/include:${PYTHON_INCLUDES}"
export LPATH="/mingw32/i686-w64-mingw32/lib:/mingw32/lib"
export CFLAGS="-O3 -fstack-protector-all -DUSE_MINGW_ANSI_STDIO=1 -DWINPTHREAD_STATIC"
export CXXFLAGS="-O3 -fstack-protector-all -DUSE_MINGW_ANSI_STDIO=1 -DWINPTHREAD_STATIC"
export LIBRARY_PATH="/mingw32/lib/gcc/i686-w64-mingw32/4.9.1"
export LDFLAGS="${PYTHON_LIBS} -Wl,--nxcompat -Wl,--dynamicbase -Wl,--warn-common -g"
rm -rf ${BUILDDIR}
mkdir ${BUILDDIR}
cd ${BUILDDIR}
${SRCDIR}/configure \ --prefix=${INSTDIR}
--build=i686-w64-mingw32 \ --host=i686-w64-mingw32 \ --target=i686-w64-mingw32 \ --enable-static \ --without-xen \ --without-qemu \ --without-openvz \ --without-lxc \ --without-vbox \ --without-xenapi \ --without-avahi \ --without-polkit \ --without-libvirtd \ --without-uml \ --with-phyp \ --with-esx \ --without-hyperv \ --without-vmware \ --without-parallels \ --without-netcf \ --without-audit \ --without-dtrace \ --enable-expensive-tests

make
make install
echo "done"

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Alexx83 - 2014-10-12

Also doesn't understand why you export
LIBRARY_PATH
and add standard paths to CPATH and LPATH

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ralph engels - 2014-10-12

LIBRARY_PATH was a leftover from another project just forgot to remove it.

Ah i see i had the includes the wrong way around whoops, strange though if i removed :${PYTHON_INCLUDES} from CPATH it started building again but not the first time, had to restart the PC so looks like some process hang and caused the build failure. works now atleast.

BTW. reason i use the weird way to set up PYTHON_INCLUDES is that the mingw version of python emits windows paths (even in msys or msys2 shells) and that causes libvirt to fail building the python extention. Not sure why but every version i tried does it so either theres no fix for it yet or its a bug.

Feel free to borrow my build settings but the -DWINPTHREAD_STATIC only works on a winpthread patched with TDM's patches for it. May also need the TDM mingw64 because he patched gcc to ignore the dll version of the lib.

Nice thing about it is that you end up with a libvirt with no winpthreads dependency.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ray Donnelly - 2014-10-12

the mingw version of python emits windows paths

What command did you use to get these paths?

Last edit: Ray Donnelly 2014-10-12

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ralph engels - 2014-10-12

no command but if say an autotool based projects tries to get the path from python then it allways returns a windows path unless patched to run sed on the outputted path.
libvirt does that with the python includes causing the build to fail.
so i tried overwriting the autotools generated windows path by exporting the var libvirt uses eg. PYTHON_INCLUDES but it did not work as expected so i guess the only fix left is either getting mingw-python to output posix paths when run from an msys shell or run sed on the path somewhere in makfile.am and then regenerating the makefiles.

P.s if you're going to build libvirt you need to patch out an undefine in gnulib/lib/pthread_sigmask.c

int
pthread_sigmask (int how, const sigset_t new_mask, sigset_t old_mask)
//#undef pthread_sigmask / this one breaks mingw64 builds because it is newer exported to the static library /

after that it builds :)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ray Donnelly - 2014-10-12

The problem is that mingw-w64-python* can be (and are) used outside of MSYS2 programs, for this reason python-config and the sysconfig module return mixed-Windows paths - with Drive letters but using Unix separators which MSYS2 programs (including Autotools) should be generally happy with. Figuring out why Autotools is not happy with the mixed-Windows path would be the first thing I'd investigate.

To set an autoconf var you'd normally emit VARIABLE_NAME="value" into a file called e.g. $PWD/config.site, then set env. var CONFIG_SITE=$PWD/config.site before calling configure, so the same line as the call to configure:
CONFIG_SITE=$PWD/config.site ../${pkgname}-${pkgver}/configure ...

We could attempt to determine if Python has been run from an MSYS2 program and dynamically change its behaviour, however the fact that it's been run for an MSYS2 program doesn't necessarily mean that the final consumer of its output will be an MSYS2 program - it could be one of mingw32-make, qbs or cmake. I suppose we could invent an env. var to allow switching this? But adding those isn't something I'd do lightly.

Last edit: Ray Donnelly 2014-10-12

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ralph engels - 2014-10-12

No not quite.

it would be ok if it returned c:/somedir/someotherdir but it returns c:\somedir\someotherdir also in the msys shell.

to your second comment that would be nice, as it is i have to use various tricks to use it in the msys shell, in a windows shell it works fine.

the site config thing i actually tried but it does not work trust me i tried everything besides hacking the python sources myself as i would rather not break it completely.

Another rather weird bug with the mingw python is that it installs python modules in mingw/lib not mingw/lib/pythonversion and scripts in mingw/scripts and not in mingw/bin, i can move them afterwards and they work but its a bit annoying. I guess that behaviour came from the patches reusing the files for the msvc builds for mingw.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ray Donnelly - 2014-10-12

As "it" is an unknown entity here I'll have to check what's going on.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ralph engels - 2014-10-14

The mingw python i use is a little older than the one from mingw-builds (2.7.3 vs 2.7.5) but mine supports pythonw so im not going to replace it unless the new one can also be coerced to build the python-win extention.

Im trying your trick with replacing the original python-config scripts with the shell
based ones. If it works consider this solved.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ray Donnelly - 2014-10-14

The mind boggles.

I asked you two days ago, in response to:

the mingw version of python emits windows paths

"What command did you use to get these paths?"

And at that point you didn't point out that you're using a mingw python from a couple of years ago?

You know MSYS2 is about packaging the latest stuff and fixing bugs in the latest stuff so that we've got a complete cohesive system where everything works together properly?

If you had determined that any of our mingw-w64-python* packages didn't provide pythonw, then that would be a bug to report. However the fact is that they do provide pythonw.

MSYS2 isn't intended to be a buffet system where you should pick and choose components to suit yourself and mix with random other stuff.

It's a complete distribution. If you are going to post bug reports to MSYS2 in future, please be explicit in that bug report about all the versions you are using of all the random stuff you've mixed it with.

Last edit: Ray Donnelly 2014-10-14

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ralph engels - 2014-10-14

I know that,
the bug report was only for the problem with shell scripts but that turned out to be a user error (mine) which i appologize for.

Note to self take a step back when something acts up and you might find that you made a stupid mistake :S.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ray Donnelly - 2014-10-14

Ok no problem.

.. or aren't using the latest stuff where there's a good chance the bugs you're working around are fixed.

Apologies for ranting at you.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ralph engels - 2014-10-14

Thats ok m8, you had no way of knowing and im not very good at explaining in english, im from denmark. Only reason i brought up python was because you seemed to have an interrest in providing libvirt with Msys2/mingw64 so i thought it better to make you aware of the problems i had with building it. Ill check out the newer builds of python and thanks.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Alexx83 - 2014-10-14

I'm not yet build libvirt successfully (trying 1.2.9 version). Will try fix it when have free time.
Really, python-2.7.3 is very old, since it we made a lot of improvements and fixes to python. Also we provide pythonw executables too.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ralph engels - 2014-10-14

Took your advise and yet again i owe you an appology,
your python works great, not sure why but last time i updated it from mingw-builds it still had that annoying bug where it installed things in the same place as the msvc build.
Not so with yours so in the future ill look here for builds.

Great work Alex :)

P.s not sure if you noticed earlier in this thread i posted the fix to build libvirt,
there is an #undef pthread_sigmask in libvirts gnulib/pthread_sigmask.c just below the
function. Remove that and libvirt builds.
The daemon does not build on windows so just disable it.

If you want to hack on the libvirt source you need rpcgen.exe and i have ported the patches from cygwin for building it on Msys2, if you want them can i upload them somewhere ?.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ray Donnelly - 2014-10-14

Sure, adding a PKGBUILD for rpcgen would seem like a good idea.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ray Donnelly - 2014-10-14

Btw, I just spotted that you mentioned upgrading to Python 2.7.5; out latest version is Python 2 is 2.7.8-7:

mingw-w64-x86_64-python2-2.7.8-7-any.pkg.tar.xz and mingw-w64-i686-python2-2.7.8-7-any.pkg.tar.xz

You get it via:

pacman -S mingw-w64-x86_64-python2

or

pacman -S mingw-w64-i686-python3

Last edit: Ray Donnelly 2014-10-14
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ralph engels - 2014-10-15

Aye allready got it :) works like a charm thanks.

Should i upload the patchset for rpcgen somewhere ? im not sure i know enough about making new PKGBUILD scripts yet but i can try if noone else got the time.

rpcgen is actually a two part build, so it would be two PKGBUILD scripts.
building rpcgen itself is easy though as it just uses a makefile.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

ralph engels - 2014-10-15

Ok uploaded it on my project page on sourceforge.

libtirpc with patches for Msys2.
http://sourceforge.net/projects/cbadvanced/files/libtirpc-0.2.4-msys2-src.7z/download
and rpcgen itself (depends on the above)
http://sourceforge.net/projects/cbadvanced/files/rpcgen-2-12-90.7z/download

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Alexx83 - 2014-10-15

will see today, thanks!

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Alexx83 - 2014-12-02

status: open --> closed

assigned_to: Alexx83
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Problem with shell scripts

Software Distribution and Building Platform for Windows

Milestone

Searches

Help

#94 Problem with shell scripts

Discussion

build script for libvirt.

!/usr/bin/bash