From: LRN <lr...@gm...> - 2011-03-17 18:25:06
|
http://lrn.no-ip.info/other/mingw/msys/perl/5.8.8-0RC1 Finally tamed extra vendor extensions - took the extensions source tarballs from Cygwin-perl 5.8.8-4 (which is why -src archive is so large now - have to drag these along; it might be possible to download them instead, but i'm not sure where), as well as some patches. Test logs are in the -doc package. Logs for extensions are also available - they show only some minor, IMO, glitches, except for Math-BigInt-Fastcalc. I think the error there is due to the lack of bcos() function. Can't find any references to it in any extension or in perl source. Found it in http://kobesearch.cpan.org/htdocs/Math-BigInt/Math/BigFloat.pm.html#bcos- , but that is from Math-BigInt 1.993, while 5.8.8 comes with 1.51; not sure why Cygwin shipped this particular version. After several tries i've started to use a hybrid packaging scheme that partially relies on find / -newer and partially on backing up /lib/perl5 and /share/man before installing perl to guarantee that only perl files will be there. Would it be a good idea to backup /bin as well? Release notes certainly require updates. This is all assuming that fclose() undefined behaviour is changed to match the expected undefined behaviour (either that, or find a way to fix perl; how's it going, Charles?). |
From: Charles W. <cwi...@us...> - 2011-03-17 19:27:05
|
On 3/17/2011 2:24 PM, LRN wrote: > http://lrn.no-ip.info/other/mingw/msys/perl/5.8.8-0RC1 Great! Thanks for all your effort. > Finally tamed extra vendor extensions - took the extensions source > tarballs from Cygwin-perl 5.8.8-4 (which is why -src archive is so large > now - have to drag these along; it might be possible to download them > instead, but i'm not sure where), as well as some patches. I think bundling the sources is the right thing to do, GPL wise (remember, since our msys-perl executable, and all XS extension DLLs, link to the msys DLL, they are "derived works" and thus the viral GPL of the msys DLL applies, even if perl and its extensions use the Artistic License). So...if /we/ ship a binary "BigMath.dll", /we/ need to also provide the sources for the BigMath extension, on /our/ website. 'Course, this doesn't apply to non-compiled extensions, but...it's just easier to treat all extensions the same. > Test logs are in the -doc package. Logs for extensions are also > available - they show only some minor, IMO, glitches, except for > Math-BigInt-Fastcalc. I think the error there is due to the lack of > bcos() function. Can't find any references to it in any extension or in > perl source. Found it in > http://kobesearch.cpan.org/htdocs/Math-BigInt/Math/BigFloat.pm.html#bcos- , > but that is from Math-BigInt 1.993, while 5.8.8 comes with 1.51; not > sure why Cygwin shipped this particular version. > > After several tries i've started to use a hybrid packaging scheme that > partially relies on find / -newer and partially on backing up /lib/perl5 > and /share/man before installing perl to guarantee that only perl files > will be there. Would it be a good idea to backup /bin as well? Yeah, for the earlier official msys-perl packages I did much the same thing, only entirely *manually*, and then generated explicit manifest files to use when creating the official installation tarballs. When I was managing a custom MinGW/MSys distro for my team at $DAYJOB, I used find / -newer and similar as well. :-) I'll take a close look at your scheme, but usually automated beats manual... > Release notes certainly require updates. Surely. Also integrating perlrebase (perlpeflags?) if you haven't already done so... > This is all assuming that fclose() undefined behaviour is changed to > match the expected undefined behaviour (either that, or find a way to > fix perl; how's it going, Charles?). I haven't been able to address it yet (family stuff, then when I finally decided to "start over" with my mingw/msys installation, using mingw-get-0.2-alpha-1 [yay! 'mingw-get update' actually 'removes'!] I ran into problems, so Keith and I have spent the last week chasing that. I should be able to look into the fclose() issue this weekend.) BTW, don't be in a huge hurry to get this 5.8.8 release "adopted" officially. I think we'll want a public shakedown period. I don't foresee that shakedown taking a HUGE amount of time, but even after the fclose/MSYS thing is fixed, and we've updated the release notes, etc, and we're all happy with the product, we'll still need to do a public shakedown. Maybe 2-3 weeks from that point? No more than a month, surely, unless serious issues are found. Anyway, the "public shakedown" would involve "real life" scenarios on various platforms (W7, Vista, XP, ...) to do "real world" stuff, such as * running autoreconf on packages and rebuilding them * using CPAN to install /other/ extensions locally + both perl-only, and + C-based XS ones + using gcc-4.5.2-latest One of my 'smoke tests' for perl in the past, has been to build autoconf and automake, and run THEIR test suites. That's about 12 hours of heavy perl usage with lots of simultaneous processes...if sync_with_child is still an extant issue, that WILL smoke it out. (Some of this, of course, can happen right now, using your packages if people want to get started. Hint hint.) Naturally, I'll want to use your build script & patches & such to rebuild your binary package(s). Repeatability is a requirement (what if you get hit by a bus? Or me?) -- especially for such an important and difficult-to-port item as perl. But HUGE props for persevering; thank you SO much for taking on this task! -- Chuck |
From: LRN <lr...@gm...> - 2011-03-17 20:51:49
|
On 17.03.2011 22:26, Charles Wilson wrote: >> After several tries i've started to use a hybrid packaging scheme that >> partially relies on find / -newer and partially on backing up /lib/perl5 >> and /share/man before installing perl to guarantee that only perl files >> will be there. Would it be a good idea to backup /bin as well? > Yeah, for the earlier official msys-perl packages I did much the same > thing, only entirely *manually*, and then generated explicit manifest > files to use when creating the official installation tarballs. > > When I was managing a custom MinGW/MSys distro for my team at $DAYJOB, I > used find / -newer and similar as well. :-) > > I'll take a close look at your scheme, but usually automated beats manual... The problem with -newer is that some files do not get their timestamps updated. If you had /lib/perl5/5.8.8/Attribute/Handler.pm before installing, it will retain its timestamp after `make install.perl' and avoid being picked up by -newer. > >> Release notes certainly require updates. > Surely. Also integrating perlrebase (perlpeflags?) if you haven't > already done so... I've used `rebasesome', it is included in the tarball as well. It's the same thing as rebaseall, but rebases user-specified directories (in this case - perl source directory) instead of /bin and /lib. Buildscript also contains a commented-out line that invokes my binrebaseall instead (actually, commented-out code makes ~50% of the buildscript). > Anyway, the "public shakedown" would involve "real life" scenarios on > various platforms (W7, Vista, XP, ...) to do "real world" stuff, such as > * running autoreconf on packages and rebuilding them > * using CPAN to install /other/ extensions locally > + both perl-only, and > + C-based XS ones > + using gcc-4.5.2-latest > One of my 'smoke tests' for perl in the past, has been to build autoconf > and automake, and run THEIR test suites. That's about 12 hours of heavy > perl usage with lots of simultaneous processes...if sync_with_child is > still an extant issue, that WILL smoke it out. > > (Some of this, of course, can happen right now, using your packages if > people want to get started. Hint hint.) There's a set of mingwPORT scripts that i've adapted for mingw-w64 native x86_64 toolchain. Autoconf/Automake are among them. With testcases. I think can try perl on these. Oh, by the way, i think i forgot to rebase extensions... /me re-checks the buildscript Yes, rebasing of the perl source directory is done right after make (extensions are built much later, and they are located in separate subdirectory). I think i'll just re-rebase /bin/msys-perl5_8.dll, the entire /lib/perl5/${VER} and /lib/perl5/vendor_perl/${VER} again before packing them (on the other hand, i can't be sure that rebase worked unless i run some tests on these libraries). Or it might be possible to rebase every extension separately (will have to figure a way to calculate new base address for each one correctly). Or maybe just make/test/install them as-is, then rebase all extensions en masse and re-install the rebased versions. On the other hand, the tests worked, and they often, AFAIU, require more than one extension to be loaded at the time, and if that worked, then their base addresses are probably fine already. Maybe --enable-auto-image-base does something good after all. |
From: LRN <lr...@gm...> - 2011-03-21 13:46:26
|
On 17.03.2011 22:26, Charles Wilson wrote: > One of my 'smoke tests' for perl in the past, has been to build autoconf > and automake, and run THEIR test suites. That's about 12 hours of heavy > perl usage with lots of simultaneous processes...if sync_with_child is > still an extant issue, that WILL smoke it out. Did you run both test suites simultaneously? |
From: Charles W. <cwi...@us...> - 2011-03-22 02:34:23
|
On 3/21/2011 9:45 AM, LRN wrote: > On 17.03.2011 22:26, Charles Wilson wrote: >> One of my 'smoke tests' for perl in the past, has been to build autoconf >> and automake, and run THEIR test suites. That's about 12 hours of heavy >> perl usage with lots of simultaneous processes...if sync_with_child is >> still an extant issue, that WILL smoke it out. > Did you run both test suites simultaneously? No, I didn't (geez, that would probably melt my CPU). The latest autoconf testsuite has some built-in parallelism. In addition, the thing you'd be "worried" about is that you have a chain of fork/exec -- perl -> sh -> perl -> etc... And that happens all the time (with a few 'm4' execs thrown in). -- Chuck |
From: LRN <lr...@gm...> - 2011-03-23 06:01:32
Attachments:
autoconfmake.logs.tar.xz
|
On 22.03.2011 5:33, Charles Wilson wrote: > On 3/21/2011 9:45 AM, LRN wrote: >> On 17.03.2011 22:26, Charles Wilson wrote: >>> One of my 'smoke tests' for perl in the past, has been to build autoconf >>> and automake, and run THEIR test suites. That's about 12 hours of heavy >>> perl usage with lots of simultaneous processes...if sync_with_child is >>> still an extant issue, that WILL smoke it out. >> Did you run both test suites simultaneously? > No, I didn't (geez, that would probably melt my CPU). The latest > autoconf testsuite has some built-in parallelism. In addition, the > thing you'd be "worried" about is that you have a chain of fork/exec -- > perl -> sh -> perl -> etc... And that happens all the time (with a few > 'm4' execs thrown in). > Ran both testsuites. Logs are attached. There was a slight mishap with the automake testsuite. It tried to run "./configure PYTHON=foo", and incidentally i've REALLY had a file /bin/foo, which incidentally consisted of 3 bytes: foo, which had a recursive effect akin to a fork bomb (spawning lots of sh.exe processes). Not sure where the `foo' file came from (it might be that it picked up my CPython that was in PATH on one of earlier runs). After i've removed it and started the testsuite again everything went fine (with certain errors that do not seem to be very troubling and are not related to perl): cond35: undefined reference to `yywrap' lex3: undefined reference to `yywrap' lzma: broken tar pipe |
From: Charles W. <cwi...@us...> - 2011-03-23 15:14:58
|
On 3/23/2011 2:00 AM, LRN wrote: > Ran both testsuites. Logs are attached. Thanks for doing this; excellent data. (Since I have a few additional patches, including a fix [1] for the EBADF issue, AND I plan to at least TRY to turn threading on as well as support for 64bit ints, I'll need to run the test suite again with my build. But...your data means that if my mods break something then we at least have a "known good" fall back). My earlier mingw32-autoconf-2.67 test (using msys-1.0.15 + perl-5.6.1_2-2) has a failure, but that was a bug in rm.exe and/or MSYS at the time. 381: AC_SYS_LONG_FILE_NAMES FAILED (acspecific.at:15) rm -f -r /usr/tmp/cfnnnnn coredumps. It was fixed here (http://bit.ly/ehMfWV) and the fix is in msys-1.0.16. [1] Really, this too just papers over the problem, but at least it doesn't require modifying MSYS, so it will "work" even on stock msys-1.0.16. In PerlIOStdio_close I simply check the return value from fclose() [aka PerlSIO_fclose], and if it is zero but errno is not, then I explicitly force errno to BE zero. The side effect of this is, IF some OTHER system call had previously caused errno to be non-zero, you'll never know because this will cover it up. However, I've run the full test suite and all the extensions with this change, and it seems to have no ill effects and "solves" the problem with the various *IO-Compress* tests. But, that's with your original configuration. Once I turn on 64bit int support and threading...I will need to run the entire test suite again. > There was a slight mishap with the automake testsuite. > cond35: undefined reference to `yywrap' > lex3: undefined reference to `yywrap' > lzma: broken tar pipe My earlier mingw32-automake-1.11.1 test (again, with msys-1.0.15 and perl-5.6.1_2-2) also failed FAIL: cond35.test FAIL: lex3.test Now, this is due to the fact that we would NEED to have: gcc -o foo.exe foo.o -lfl but the testcase actually does: gcc -o foo.exe foo.o The reason it does this is because msys-flex installs an *MSYS* version of libfl.a into /lib, but does not (yet) install a *MINGW* version in /mingw/lib where mingw gcc will find it. Since auto* is smart enough to figure out that there is no libfl to link, it doesn't add -lfl. I plan add a mingw32-libfl package or something similar to remedy this problem -- search the mailing list archives for the relevant discussion -- but haven't yet gotten around to it. So, this is a "bug" with our flex toolkit, not perl or automake. Now, here's an oddity for you: My exising mingw32-automake-1.11.1 *passes* the lzma test, but my existing msys-automake-1.11 fails it. The reason is as follows: When building mingw32- variants, $PATH puts /mingw/bin first. When building msys- variants, you have to be in an msys-dvlpr shell, so $PATH puts /bin first. We have two versions of lzma.exe -- /bin/lzma.exe is an msys application. /mingw/bin/lzma.exe is a native win32 application. So, even if they were of the same version, you might see a bug in one but not the other. As it happens, my build environment when I build automake-1.11.1-msys contains xz-4.999.9beta_20091209-1-msys-1.0.12-bin but my build environment when I built automake1.11-1.11.1-mingw32 contains xz-4.999.9beta_20100401-1-mingw32-bin So not only was one lzma.exe "msys" and one "native" -- but they were even of different versions! In the former case, I too saw the lzma test fail. But in the latter case, the lzma test passed. Unfortunately, the --version option reports only "4.999.9beta" so we need more detail. What does the following command report on your box, with regards to mingw32-xz-bin and msys-xz-bin? mingw-get-info installed You can get mingw-get-info from http://bit.ly/hm46DZ and install it into /mingw/bin (look at the 'attachments' section of that bug tracker ticket item). Note that mingw-get-info requires mingw-get-0.2-alpha or above. Here's what I *now* have: mingw32-xz-bin: Installed Version: xz-4.999.9beta_20100401-1-mingw32-bin.tar.bz2 Repository Version: xz-4.999.9beta_20100401-1-mingw32-bin.tar.bz2 msys-xz-bin: Installed Version: xz-4.999.9beta_20100401-1-msys-1.0.13-bin.tar.gz Repository Version: xz-4.999.9beta_20100401-1-msys-1.0.13-bin.tar.gz so we'll see what happens when I try to run these tests in my spiffy new environment with my spiffy new perl. > It trieakind to run "./configure PYTHON=foo", > picked up my CPython that was in PATH on one of earlier runs When I'm building something (msys, mingw, cygwin) that seems to misbehave due to picking some external tool I've installed in C:/Program Files -- such as "native" Python, or even MikTeX -- I usually modify the build script to explicitly set PATH to remove the offending installations. -- Chuck |
From: LRN <lr...@gm...> - 2011-03-23 16:38:05
|
On 23.03.2011 18:14, Charles Wilson wrote: >> lzma: broken tar pipe > Now, here's an oddity for you: > > My exising mingw32-automake-1.11.1 *passes* the lzma test, but my > existing msys-automake-1.11 fails it. The reason is as follows: > > When building mingw32- variants, $PATH puts /mingw/bin first. When > building msys- variants, you have to be in an msys-dvlpr shell, so $PATH > puts /bin first. > > We have two versions of lzma.exe -- /bin/lzma.exe is an msys > application. /mingw/bin/lzma.exe is a native win32 application. So, even > if they were of the same version, you might see a bug in one but not the > other. As it happens, my build environment when I build > automake-1.11.1-msys > contains xz-4.999.9beta_20091209-1-msys-1.0.12-bin > but my build environment when I built > automake1.11-1.11.1-mingw32 > contains xz-4.999.9beta_20100401-1-mingw32-bin > So not only was one lzma.exe "msys" and one "native" -- but they were > even of different versions! > > In the former case, I too saw the lzma test fail. But in the latter > case, the lzma test passed. > > Unfortunately, the --version option reports only "4.999.9beta" so we > need more detail. What does the following command report on your box, > with regards to mingw32-xz-bin and msys-xz-bin? > > mingw-get-info installed > > You can get mingw-get-info from http://bit.ly/hm46DZ and install it into > /mingw/bin (look at the 'attachments' section of that bug tracker ticket > item). Note that mingw-get-info requires mingw-get-0.2-alpha or above. > > Here's what I *now* have: > > mingw32-xz-bin: > Installed Version: xz-4.999.9beta_20100401-1-mingw32-bin.tar.bz2 > Repository Version: xz-4.999.9beta_20100401-1-mingw32-bin.tar.bz2 > msys-xz-bin: > Installed Version: xz-4.999.9beta_20100401-1-msys-1.0.13-bin.tar.gz > Repository Version: xz-4.999.9beta_20100401-1-msys-1.0.13-bin.tar.gz > > so we'll see what happens when I try to run these tests in my spiffy new > environment with my spiffy new perl. $ mingw-get-info installed | grep xz msys-xz-bin: Installed Version: xz-4.999.9beta_20100401-1-msys-1.0.13-bin.tar.gz Repository Version: xz-4.999.9beta_20100401-1-msys-1.0.13-bin.tar.gz > >> It tried to run "./configure PYTHON=foo", >> picked up my CPython that was in PATH on one of earlier runs > When I'm building something (msys, mingw, cygwin) that seems to > misbehave due to picking some external tool I've installed in C:/Program > Files -- such as "native" Python, or even MikTeX -- I usually modify the > build script to explicitly set PATH to remove the offending installations. Did that too, which is how i got past it, i think (a thing of note - while it was picking CPython, it *passed* some Python tests, so CPython is not beyond hope). It might be a good idea to update /etc/profile to NOT to include system PATH. I can't presume to know all Msys use-cases, but apart from canonical Windows tools (which come from %WINDIR% and %SYSTEMDIR%) a typical software building process only involves programs from /mingw/bin and /bin, so having external and potentially disturbing programs in PATH might not be good. Adding proper directories to PATH by hacking /etc/profile can be done if the developer requires so. OTOH hacking /etc/profile to PATH=$(echo $PATH | sed -e s';offendingpath;;') is equally possible (although sedding-out a part of PATH is naturally more difficult than adding something to it). |
From: Charles W. <cwi...@us...> - 2011-03-23 17:17:03
|
On 3/23/2011 12:37 PM, LRN wrote: > On 23.03.2011 18:14, Charles Wilson wrote: >> Here's what I *now* have: >> >> mingw32-xz-bin: >> Installed Version: xz-4.999.9beta_20100401-1-mingw32-bin.tar.bz2 >> Repository Version: xz-4.999.9beta_20100401-1-mingw32-bin.tar.bz2 >> msys-xz-bin: >> Installed Version: xz-4.999.9beta_20100401-1-msys-1.0.13-bin.tar.gz >> Repository Version: xz-4.999.9beta_20100401-1-msys-1.0.13-bin.tar.gz >> >> so we'll see what happens when I try to run these tests in my spiffy new >> environment with my spiffy new perl. > $ mingw-get-info installed | grep xz > msys-xz-bin: > Installed Version: xz-4.999.9beta_20100401-1-msys-1.0.13-bin.tar.gz > Repository Version: xz-4.999.9beta_20100401-1-msys-1.0.13-bin.tar.gz Ah, so (a) you were using the msys lzma.exe for the test, even though it was the mingw32-automake, and (b) we still see the error, even though you were using _20100409 and I was using, at the time I saw the error, _20091209. This tells me the bug is intrinsic xz as ported to msys, and is NOT related to any of the following: * which msys compiler was used (2.95.3 vs 3.4.5) * which patchlevel of xz was used * which version of msys is used (1.0.13 vs 1.0.16) Now, since the reported error is: lzma: (stdin): Not enough memory) and the command is that elicits the error is tardir=lzma-1.0 && /bin/sh /f/src/mingw32ports/automake-1.11.1/mingwPORT/bld/tests/lzma.dir/missing --run tar chof - "$tardir" | lzma -9 -c >lzma-1.0.tar.lzma It could be any number of problems related to how msys-lzma deals with large amounts of data coming in on stdin. Maybe msys-lzma tries to allocate too much memory, or msys itself has some problems, in this case, pushing lots of data thru a pipe. I'm not sure...and if this is the only extant case where the bug, whatever it is, causes a problem, I'm not too interested in tracking it down. But whatever the issue, it is surely not a *perl* problem. In *your* case, simply installing mingw32-xz will resolve the error when building *mingw32-*automake1.11-1.11.1. >> build script to explicitly set PATH to remove the offending installations. > Did that too, which is how i got past it, i think (a thing of note - > while it was picking CPython, it *passed* some Python tests, so CPython > is not beyond hope). Oh, sure, using win32 python from inside an MSYS shell will kinda work. All unix paths passed in on the cmdline will get converted, but any unix paths embedded inside .py scripts won't. Hence, some things will work, others won't. > It might be a good idea to update /etc/profile to NOT to include system > PATH. No, bad idea. If you drop C:/Windows/system32 and related paths, then bad things will happen. Worse, because of i18n and 64bit WoW, the *NAME* of that directory -- and all the similar ones we might care about -- is not constant. So we can't just do: export PATH=[mingw/msys stuff]:/c/Windows/system32 We *might* get away with using certain other vars, like %SYSTEMROOT% but that misses on the system32... besides there may be other $PATH entries we need but can't predict. Leave it for the end user. > I can't presume to know all Msys use-cases, but apart from > canonical Windows tools (which come from %WINDIR% and %SYSTEMDIR%) On my box, both %SYSTEMROOT% and %WINDIR% resolve to C:\Windows (i.e. neither contains the /system32 bit). > typical software building process only involves programs from /mingw/bin > and /bin, so having external and potentially disturbing programs in PATH > might not be good. But they MAY be good. That's up to the end user. I don't think we need to go explicitly removing elements from $PATH in the /etc/profile and msys.bat scripts if the end user has put them into the environment using the windows facilities. For instance, I don't want to have to edit /etc/profile on every machine I use, in order to ensure that ClearCase is in my $PATH. But, since that tool is installed in different locations on the different machines -- so I can't add a fixed value in my shared ~/.bashrc -- I'd have to manually change machine specific settings in each machine's /etc/profile. Or...since the rational tools installer modifies the system %PATH%...simply leave well enough alone, and it will Just Work(tm), on all my machines. OTOH, in a *specific build script* for a particular port -- as in the case here, with msys-build-perl or the mingw32-automake1.11 mingwPORT.sh -- it is probably ok to make localized PATH changes like that. -- Chuck |
From: Charles W. <cwi...@us...> - 2011-03-20 18:23:02
|
On 3/17/2011 2:24 PM, LRN wrote: > http://lrn.no-ip.info/other/mingw/msys/perl/5.8.8-0RC1 Is there something wrong with the server this weekend? I can't seem to reach it anymore. -- Chuck |
From: LRN <lr...@gm...> - 2011-03-20 18:45:53
|
On 20.03.2011 21:22, Charles Wilson wrote: > On 3/17/2011 2:24 PM, LRN wrote: >> http://lrn.no-ip.info/other/mingw/msys/perl/5.8.8-0RC1 > Is there something wrong with the server this weekend? I can't seem to > reach it anymore. > > -- > Chuck > nginx frontend have crashed. I've hacked ipmasq to forward directly to my web-server, should work fine now. |
From: Charles W. <cwi...@us...> - 2011-03-20 21:05:02
Attachments:
msys-build-perl
|
On 3/20/2011 2:44 PM, LRN wrote: > On 20.03.2011 21:22, Charles Wilson wrote: >> Is there something wrong with the server this weekend? I can't seem to >> reach it anymore. > nginx frontend have crashed. I've hacked ipmasq to forward directly to > my web-server, should work fine now. got it. Question about the bundled msys-build-perl script. There is a LOT of stuff commented out. Now, I sometimes comment out valid bits of msys-build-* while working, simply because I don't need to do certain things twice -- but the bits are still valid, and a pristine "unpack -src and run msys-build-*' still needs those bits. Is that the case here? or are the commented-out bits really not necessary in your port. I've attached a copy of the msys-build-perl script that is in your -src, just in case it may differ from the /current/ copy on your system and so that you don't have to re-unpack -src yourself... -- Chuck |
From: LRN <lr...@gm...> - 2011-03-20 21:16:33
|
On 21.03.2011 0:04, Charles Wilson wrote: > On 3/20/2011 2:44 PM, LRN wrote: >> On 20.03.2011 21:22, Charles Wilson wrote: >>> Is there something wrong with the server this weekend? I can't seem to >>> reach it anymore. >> nginx frontend have crashed. I've hacked ipmasq to forward directly to >> my web-server, should work fine now. > got it. Question about the bundled msys-build-perl script. There is a > LOT of stuff commented out. > > Now, I sometimes comment out valid bits of msys-build-* while working, > simply because I don't need to do certain things twice -- but the bits > are still valid, and a pristine "unpack -src and run msys-build-*' still > needs those bits. > > Is that the case here? > > or are the commented-out bits really not necessary in your port. I've > attached a copy of the msys-build-perl script that is in your -src, just > in case it may differ from the /current/ copy on your system and so that > you don't have to re-unpack -src yourself... No, i'm just a messy kind of person. And i don't have perl under any kind of version control system. Which is why i tend to comment out the code i don't need instead of removing it. |
From: Charles W. <cwi...@us...> - 2011-03-21 04:13:48
|
On 3/20/2011 5:16 PM, LRN wrote: > On 21.03.2011 0:04, Charles Wilson wrote: >> or are the commented-out bits really not necessary in your port. I've >> attached a copy of the msys-build-perl script that is in your -src, just >> in case it may differ from the /current/ copy on your system and so that >> you don't have to re-unpack -src yourself... > No, i'm just a messy kind of person. And i don't have perl under any > kind of version control system. Which is why i tend to comment out the > code i don't need instead of removing it. Okay, with a bit of finagling I have reproduced your build. I have the same results for the core perl testsuite: 60 tests and 270 subtests skipped. Failed 15/993 test scripts, 98.49% okay. 469/116899 subtests failed, 99.60% okay. I have *basically* the same test results for all the extension packages, except for: log-ext-Compress-Zlib-2.005.txt Failed 4/8 test scripts, 50.00% okay. 24/726 subtests failed, 96.69% okay. log-ext-IO-Compress-Bzip2-2.005.txt Failed 4/15 test scripts, 73.33% okay. 426/8338 subtests failed, 94.89% okay. log-ext-IO-Compress-Zlib-2.005.txt Failed 23/64 test scripts, 64.06% okay. 2141/38668 subtests failed, 94.46% okay. log-ext-IO-Zlib-1.05.txt Failed 6/10 test scripts, 40.00% okay. 7/123 subtests failed, 94.31% okay. I suspect this is because you are running a patched msys-1.0.dll that papers over the EBADF issue, and I'm running stock. Also, one interesting thing: my package contents differ. Part of this is because of this configuration difference: For some reason, on my build the search for -lcrypt and -lgdbm / -lgdbm_compat succeeded, whereas on your build it did not. So, my msys-perl5_8.dll has a dep on msys-crypt dll, AND my build actually built the ODBM_File NDBM_File GDBM_File extensions, whereas yours skipped those extensions. Hence, my packages include their implementation files, documentation, etc. So, ignoring THOSE extra files, my -bin package ALSO has the following extra files in bin/: ysh shasum scandeps.pl ptee ptardiff ptar pod2readme pod_cover lwp-rget lwp-request lwp-mirror lwp-download crc32 cpantest config_data My hunch is that you did so many install/reinstall cycles that your /bin already had "old" versions of these files, so ${newer} missed them...and since they weren't explicitly listed in MSYS-STUFF/msys-perl-bin.pkglist nor specifically excluded in MSYS-STUFF/msys-perl-noship.list then your package missed them and mine picked them up. FWIW, the cygwin perl-5.10.1-5 package contains all of those "extra" /bin files except for ysh scandeps.pl pod2readme so I'm going to add them (all) to msys-perl-bin.pkglist to ensure they get picked up in the future. While I think using your rebasesome tool is useful, and the best tool to use *during* the build, I'm also going to add Reini's perlrebase script to the -bin specifically for end users to use -- especially end users who further extend their perl via CPAN. Now, that leaves us with a few open issues: #1) EBADF. #2) the remaining issues I commented on in this email: http://thread.gmane.org/gmane.comp.gnu.mingw.user/35610/focus=35721 ========= Cwd (was vs now): ../ext/Cwd/t/cwd.t 1 256 33 54 163.64% 7-33 ../ext/Cwd/t/cwd.t 2 512 29 2 6.90% 22-23 Much better. Good enough? Yes. not ok 22 # Failed test in ../ext/Cwd/t/cwd.t at line 171. # '/usr/src/perl/perl-5.8.8/t/linktest' # doesn't match '(?-xism:t/_ptrslt_/_path_/_to_/_a_/_dir_$)' not ok 23 # Failed test in ../ext/Cwd/t/cwd.t at line 172. # '/usr/src/perl/perl-5.8.8/t/linktest' # doesn't match '(?-xism:t/_ptrslt_/_path_/_to_/_a_/_dir_$)' These test are inside a 'SKIP if platform doesn't have symlinks'. We *fake* symlinks, so we pretend to have them -- thus the tests aren't skipped. But they don't behave like *real* symlinks, so these tests fail. ========= Cwd/taint ../ext/Cwd/t/taint.t 8 2048 17 8 47.06% 2 4 6 8 10 12 14 16 ../ext/Cwd/t/taint.t 8 2048 17 8 47.06% 2 4 6 8 10 12 14 16 You know what? I don't care about taint. ========= NDBM, ODBM ../ext/NDBM_File/t/ndbm.t 77 1 1.30% 2 ../ext/ODBM_File/t/odbm.t 78 1 1.28% 2 Good enough. Due to the fact that we don't have real permissions. Probably could force to skip by modifying line 43 below... 43 if ($^O eq 'amigaos' || $^O eq 'os2' || $^O eq 'MSWin32' || $^O eq 'NetWare' || $^O eq 'MacOS') { 44 print "ok 2 # Skipped: different file permission semantics\n"; 45 } 46 else { 47 my ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime, 48 $blksize,$blocks) = stat($Dfile); 49 print (($mode & 0777) == 0640 ? "ok 2\n" : "not ok 2\n"); 50 } ========= socketpair ../ext/Socket/t/socketpair.t 23 5888 45 23 51.11% 22-35 37-45 ../ext/Socket/t/socketpair.t 23 5888 45 23 51.11% 22-35 37-45 Yeesh. # Failed test 'socketpair (LEFT, RIGHT, AF_UNIX, SOCK_DGRAM, PF_UNSPEC)' # in ../ext/Socket/t/socketpair.t at line 176. # $! = Operation not supported on transport endpoint So LEFT and RIGHT sockets are not created and after that all operations on LEFT and RIGHT fail (22 of them) Well, MSys supports AF_UNIX sockets, AND it supports UDP on those sockets. What it DOESN'T support is setting them up in that configuration using socketpair(): net.cc: 2225 /* Win32 supports AF_INET only, so ignore domain and protocol arguments */ 2226 extern "C" int 2227 socketpair (int, int type, int, int *sb) So...we should expect all of these tests to fail -- and should simply skip them. I'll patch the test. (FYI, modern cygwin has improved its socketpair implementation). ========= File::Copy ../lib/File/Copy.t 4 1024 60 4 6.67% 28-29 55-56 ../lib/File/Copy.t 4 1024 60 4 6.67% 28-29 55-56 OK, as you explained (symlinks). ========= File::Find ../lib/File/Find/t/find.t 199 197 98.99% 3-199 ../lib/File/Find/t/find.t 199 197 98.99% 3-199 Hm. wasn't this: rm -rf ${srcdir}/t/for_find prior to the test supposed to fix this? Ah...if we do the rm *immediately* before running *only* this test, then we get: ../lib/File/Find/t/find.t 199 62 31.16% 26-28 38-40 51-53 62- 64 80-82 109-113 170 184 186 188 200-227 Failed 1/1 test scripts, 0.00% okay. 6/199 subtests failed, 96.98% okay. The rest are all symlink problems. ========= File::Find (taint) ../lib/File/Find/t/taint.t 8 2048 45 11 24.44% 3 5 10 26-28 32 46-48 ../lib/File/Find/t/taint.t 8 2048 45 11 24.44% 3 5 10 Better. Besides, it's taint. I don't care about taint. ========= File::Spec ../lib/File/Spec/t/Spec.t 471 1 0.21% 468 ../lib/File/Spec/t/Spec.t 472 1 0.21% 468 Test 468 got: "///../../..//a//b/../c" (../lib/File/Spec/t/Spec.t at line 672 fail #385) # Expected: "/a/b/../c" (File::Spec::Epoc->canonpath('///../../..//./././a//b/.././c/././')) On win32 I bet this looks for the //server/share UNC host ".." I don't think this is worth fixing. ========= Net::Ping (1) ../lib/Net/Ping/t/110_icmp_inst.t 1 256 2 2 100.00% 2 ../lib/Net/Ping/t/110_icmp_inst.............icmp socket error - Not owner at ../lib/Net/Ping/t/110_icmp_inst .t line 27 You need to be Adminstrator on Vista+ to access 'raw' sockets, which are required for using the ICMP protocol. I wasn't, so it failed. ========= Net::Ping (2) ../lib/Net/Ping/t/450_service.t 26 2 7.69% 9 18 Failed test 9 in ../lib/Net/Ping/t/450_service.t at line 84 # ../lib/Net/Ping/t/450_service.t line 84 is: ok $p -> ping("127.0.0.1"); # Failed test 18 in ../lib/Net/Ping/t/450_service.t at line 143 # ../lib/Net/Ping/t/450_service.t line 143 is: ok $p -> ack(); Again, need to be Administrator. ========== Test ../lib/Test/t/multiline.t 2 1 50.00% 2 ../lib/Test/t/multiline.t 2 1 50.00% 2 This is bogus. BOTH of the two tests are marked as "todo" -- and both tests deliberately fail. Being marked todo, both of these errors shold be ignored and not counted -- yet, the "todo" marker only seems to work on the first test. So, we fail the second test -- as expected -- but that failure is counted against us -- which is not expected. Ignoring it... ========== filefind-taint lib/filefind-taint.t 45 43 95.56% 3-45 lib/filefind-taint.t 45 6 13.33% 26-28 46-48 Much better. But it's taint. I don't care about taint. Anyway, these are all symlink problems. ========== op/stat op/stat.t 17 4352 86 20 23.26% 9 32 78-86 op/stat.t 17 4352 86 20 23.26% 9 32 78-86 Again, expected. not ok 9 - hard link ctime != mtime not ok 32 - -l (symlink problem) # Can't symlink op/stat.t: File exists at op/stat.t line 431. # Looks like you planned 86 tests but ran 77. What happened here is that test #78 killed the parent perl, so 78-86 were not performed. This is what happens: 425 SKIP: { 426 skip "No lstat", 2 unless $Config{d_lstat}; 427 428 # bug id 20020124.004 429 # If we have d_lstat, we should have symlink() 430 my $linkname = 'dolzero'; 431 symlink $0, $linkname or die "# Can't symlink $0: $!"; Now, we DO have lstat (it's implemented as a passthru to fstat), so we don't skip. However, symlink fails...so we die. Should probably do this: skip "No lstat", 2 unless $Config{d_lstat}'; skip "No symlinks on msys", 2 if $Config{osname) ~= msys; ========== op/taint op/taint.t 1 256 238 476 200.00% 1-238 op/taint.t 9 2304 238 420 176.47% 1-3 5 31-238 Meh. I don't care about taint. ========== Also, we probably shouldn't skip the cygwin tests, since msys is cygwin, kinda: lib/cygwin..................................skipped all skipped: cygwin specific test So, I've got a few additional patches to apply to the test suite, and the EBADF thing to track down. -- Chuck |
From: Charles W. <cwi...@us...> - 2011-03-21 05:55:52
|
On 3/21/2011 12:13 AM, Charles Wilson wrote: > On 3/20/2011 5:16 PM, LRN wrote: >> On 21.03.2011 0:04, Charles Wilson wrote: >>> or are the commented-out bits really not necessary in your port. I've >>> attached a copy of the msys-build-perl script that is in your -src, just >>> in case it may differ from the /current/ copy on your system and so that >>> you don't have to re-unpack -src yourself... >> No, i'm just a messy kind of person. And i don't have perl under any >> kind of version control system. Which is why i tend to comment out the >> code i don't need instead of removing it. > > Okay, with a bit of finagling I have reproduced your build. I have the > same results for the core perl testsuite: Oh, and a few other notes: from the perl-config-summary: config_args='-de -Dvendorprefix=/usr -Dsiteprefix=/usr' hint=recommended, useposix=true, d_sigaction=define usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef We probably want thread support -- so I'll give it a try. But if it "breaks" stuff I won't waste any time on it. useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef I think we DO want use64bitint support, but not use64bitall. I don't think this is very likely to break anything. usemymalloc=y, bincompat5005=undef Interesting. Earlier builds of cygwin and msys perl did NOT define usemymalloc. However, it seems to work, and modern cygwin perl has usemymalloc as well. Compiler: cc='gcc', ccflags ='-DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -Wdeclaration-after-statement', optimize='-O3 -fno-unit-at-a-time -s -mtune=pentium', cppflags='-DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -Wdeclaration-after-statement' I wonder if I should add -DDEBUGGING. This would allow to use 'perl -D' to access perl's internal state (see perldoc perlrun). Obviously I would NOT combine this with -g; there's no use in bloating the distro perl with 20MB of symbols that aren't much help at -O3. And -O0...no. OTOH, -DDDEBUGGING only adds a few 100kB. -- Chuck |
From: LRN <lr...@gm...> - 2011-03-27 21:05:26
|
On 28.03.2011 0:50, Charles Wilson wrote: > On 3/23/2011 11:30 PM, Charles Wilson wrote: > > Finally, there were two substantive patches I had some concerns about > > 0019-MSys-QUESTIONABLE-Core-perl-build-system-mods.patch > 0040-MSys-TEST-QUESTIONABLE-op-tests.patch > > The second one...well, it copies msys-1.0.dll and msys-crypt-1.dll into > the test dir. I think that's a bad idea, but I'll test first to see if > we can skip it (Reini also mentioned that 'upstream perl' has recently > made a change to taint mode on win32/cygwin, so that /bin is left in > $PATH...some version of THAT change might be a better approach). It should be noted that i've rigged the buildscript to run tests with /bin/perl, thus it does not matter whether it copies msys*.dll into test dir or not (so culling this patch out is a good idea). > I also took the opportunity to adopt LRN's > suggestion to rebase the (newly compiled) perl extension DLLs as each > one is built and installed. (FYI: following modern cygwin perl's > suggestion, I'm using > 0x56000000 and going up by 0x20000 > LRN's original package used > 0x05200000 and went down by 0x40000 > ^^^ > very low! > For the rationale behind the 0x20000 minimum rebase interval, see this > thread at cygwin: > http://cygwin.com/ml/cygwin/2009-06/msg00266.html > Note that cygwin implemented a different solution, so they don't REALLY > need the 0x20000 'buffer'. However, msys hasn't adopted anything > similar, so we DO need a larger buffer than rebase.exe's default > 0x10000. OTOH, 0x40000 chews thru a LOT of address space really quickly... I've had troubles with 0x20000. But then, it might have been caused by moon phases or by my dirty msys installation. I'll try 0x56000000/0x20000 next time i build perl. > #2) Then, start "improving" (IMO) the configuration, one new option at a > time: > a) turn on 64bitint support > b) turn on threading support > c) compile with -DDEBUGGING (which is not "-g"; it simply > enables some additional command line switches for debugging > perl scripts). However, it creates an ABI change in the > libperl, so extension DLLs from 'regular' and "DEBUGGING" > perls can't be mixed; it's one or the other. Our perl-5.6.1 > had -DDEBUGGING. Some of the patches will have to be tweaked further. Namely, you expressed a desire to add -DDEBUGGING, and that should go into /hints/msys.sh (created by a patch), since you can't tell the configure script to add it (or i did not find a way to do so). |
From: Charles W. <cwi...@us...> - 2011-03-28 01:52:28
|
On 3/27/2011 5:05 PM, LRN wrote: > On 28.03.2011 0:50, Charles Wilson wrote: >> it copies msys-1.0.dll and msys-crypt-1.dll into >> the test dir. I think that's a bad idea, but I'll test first to see if >> we can skip it (Reini also mentioned that 'upstream perl' has recently >> made a change to taint mode on win32/cygwin, so that /bin is left in >> $PATH...some version of THAT change might be a better approach). > It should be noted that i've rigged the buildscript to run tests with > /bin/perl, thus it does not matter whether it copies msys*.dll into test > dir or not (so culling this patch out is a good idea). Yep, that's what I thought too. It looks like 5.6.1 copied the DLL into there too but I think it was a bad idea even back then. The *reason* was to avoid those nasty popup windows, that you have to manually dismiss, during the testsuite. But /that/ cure is worse than the disease... It looks like this change to t/op/taint.t does the trick: @@ -153,7 +153,7 @@ my $TEST = catfile(curdir(), 'TEST'); } } - $ENV{PATH} = ''; + $ENV{PATH} = ($Is_Cygwin || $Is_Msys) ? '/usr/bin' : ''; delete @ENV{@MoreEnv}; $ENV{TERM} = 'dumb'; It doesn't make us PASS any more of the taint tests than we were previously, but it does avoid the nasty popups AND allows to not copy the msys dll into the build dir. >> For the rationale behind the 0x20000 minimum rebase interval, see this >> thread at cygwin: >> http://cygwin.com/ml/cygwin/2009-06/msg00266.html >> Note that cygwin implemented a different solution, so they don't REALLY >> need the 0x20000 'buffer'. However, msys hasn't adopted anything >> similar, so we DO need a larger buffer than rebase.exe's default >> 0x10000. OTOH, 0x40000 chews thru a LOT of address space really quickly... > I've had troubles with 0x20000. But then, it might have been caused by > moon phases or by my dirty msys installation. > I'll try 0x56000000/0x20000 next time i build perl. Well, on the fourth Sunday of each month, assuming the moon is in the seventh house, and Jupiter aligns with Mars, then 0x20000 works fine... >> #2) Then, start "improving" (IMO) the configuration, one new option at a >> time: >> a) turn on 64bitint support >> b) turn on threading support >> c) compile with -DDEBUGGING (which is not "-g"; it simply >> enables some additional command line switches for debugging >> perl scripts). However, it creates an ABI change in the >> libperl, so extension DLLs from 'regular' and "DEBUGGING" >> perls can't be mixed; it's one or the other. Our perl-5.6.1 >> had -DDEBUGGING. > Some of the patches will have to be tweaked further. Namely, you > expressed a desire to add -DDEBUGGING, and that should go into > /hints/msys.sh (created by a patch), since you can't tell the configure > script to add it (or i did not find a way to do so). Yes, I had intended that some of these changes be represented by additional patches. You can cheat (kinda) for debugging: you Configure with -d, AND -Dusedebug. That turns on both -DDEBUGGING as well as -g. Then you strip the results. Since our hints/msys.sh already specifies: ldflags="$ldflags -s " ccdlflags="$ccdlflags -s " lddlflags="$lddlflags -s " the stripping bit is already covered, so I really just need to use -Dusedevel when running Configure. I'm also planning to, as on cygwin and with msys-perl-5.6.1, force the perl5/{,vendor_perl,site_perl}/$VER to be 5.8, instead of 5.8.8 (if we bother to release 5.8.9 it OUGHT to be binary compatible. So if anybody custom installs additional extensions via perl-5.8.8+CPAN, those site_perl extensions *ought* to continue working with any future 5.8.9 without rebuilding. -- Chuck |
From: LRN <lr...@gm...> - 2011-03-27 23:10:55
|
On 28.03.2011 0:50, Charles Wilson wrote: > On 3/23/2011 11:30 PM, Charles Wilson wrote: > b) turn on threading support perl with threading support hangs up in one of the compression (libbz2) tests, eating 100% CPU (50% on 2-core machine). |
From: Charles W. <cwi...@us...> - 2011-03-28 01:56:22
|
On 3/27/2011 7:09 PM, LRN wrote: > On 28.03.2011 0:50, Charles Wilson wrote: >> On 3/23/2011 11:30 PM, Charles Wilson wrote: >> b) turn on threading support > perl with threading support hangs up in one of the compression (libbz2) > tests, eating 100% CPU (50% on 2-core machine). Yep, I've decided to punt on threading. As it happens, there were LOTS of major threading improvements in cygwin, just weeks after Earnie forked MSYS way back then. Sometime last year I tried to port over those (old) pthread fixes into modern msys, but my dll kept going kersplat, so I gave up. I don't think msys supports pthreads well enough for the rigors of perl's threading needs, and that probably won't change until msys-2.0 (or somebody devotes more time to porting over the pthread changes than I was willing to, last year). (There's also the issue that -Dusethreads implies -Uusemymalloc. However, msys-perl-5.{6,8} appears to need -Dusemymalloc so there's another reason to punt on threads) -- Chuck |
From: Charles W. <cwi...@us...> - 2011-03-21 12:48:28
|
On 3/21/2011 6:00 AM, LRN wrote: > On 21.03.2011 7:13, Charles Wilson wrote: >> ========= Net::Ping (2) >> >> ../lib/Net/Ping/t/450_service.t 26 2 7.69% 9 18 >> >> Failed test 9 in ../lib/Net/Ping/t/450_service.t at line 84 >> # ../lib/Net/Ping/t/450_service.t line 84 is: ok $p -> ping("127.0.0.1"); >> # Failed test 18 in ../lib/Net/Ping/t/450_service.t at line 143 >> # ../lib/Net/Ping/t/450_service.t line 143 is: ok $p -> ack(); >> >> Again, need to be Administrator. > Nope, i have the permissions and it still fails. Hmm. You're both right and wrong. The Net::Ping (1) test (110_icmp_inst.t) passes all tests if run *as Administrator* -- however, merely having the same privileges as Administrator but not BEING Administrator is not enough, since your test results failed 110_icmp_inst.t. However, even running *as Administrator* I still fail the two tests in Net::Ping (2) (450_service.t). I wonder if it's a firewall issue. Turns out, "$p->ping" doesn't always mean 'send an ICMP ping packet'; in this particular case, it is actually using TCP (and something called the "syn" protocol). However, the test sets up various servers listening on high ports, and then sends a test packet to 127.0.0.1 to verify the server is listening... Windows firewalling is odd, and sometimes even 127.0.0.1 can get blocked. In any case, I notice that tests 9 and 18 are ignored (todo'ed) on hpux, and test 18 is ignored on (native) Win32. I think I'm just going to ignore this; it's not really part of our reason for having perl, to enable TCP/IP servers written in perl to work under msys. -- Chuck |
From: Charles W. <cwi...@us...> - 2011-03-24 03:31:16
Attachments:
msys-perl-5.8.8-0RC1b-patches.tar.xz
|
FYI, here's the patchset as it stands currently. There are 20: 0001-Import-interesting-stuff-from-perl-5.6.1-MSYS-1.0.11.patch 0002-Import-more-interesting-stuff-from-perl-5.6.1-MSYS-1.patch 0003-Import-test-changes-from-perl-5.6.1-MSYS-1.0.11-src..patch 0004-Import-more-test-changes-from-perl-5.6.1-MSYS-1.0.11.patch 0005-Fix-compilation-on-msysGit.patch 0006-Fix-compiling-if-you-have-paths-with-spaces-in-your-.patch 0007-Various-fixes-from-LRN-s-port.patch 0008-fix-issue-with-trying-to-link-with-lm.patch 0009-MallocCfg_ptr-is-really-EXT-not-extern.patch 0010-msys-pkg-lists.patch 0011-Adapted-from-cygwin-WIN32CORE-patch.patch 0012-XSTypes-capitalization-fix.patch 0013-MSys-cwd.patch 0014-Adapted-from-cygwin-Net-ping-t-patch.patch 0015-MSys-Skip-some-DBM-tests-file-perms-not-supported.patch 0016-MSys-Skip-tests-that-require-socketpair-UDP.patch 0017-cygwin-MSys-Slight-improvement-in-path-canonicalizat.patch 0018-MSys-Avoid-die-in-testsuite-due-to-symlink.patch 0019-MSys-Remove-5.6.1-packaging-files.patch 0020-MSys-Workaround-fclose-success-but-errno-EBADF-issue.patch Basically, I've taken LRN's patchset (which was 11 patches, some of which were identical to upstream msysgit perl-5.8.8's [1], and some were similar but modified, and some were completely unique) and teased out the following: split them into "msysgit patch" followed by "LRN's changes TO that patch", or "msysgit patch same as upstream", or "LRN's OTHER changes". That takes us to 0014. So, 0001 thru 0014 here, give the same thing as LRN's 0001-0010+0012 (he didn't have an 0011). Then 0015-0020 are my changes to fix a few more test suite failures and papering over the EBADF issue. [1] http://repo.or.cz/w/msysgit.git/tree/msys:/src/perl/patches I'm posting them in this form for "historical purposes", because my next step is going to be re-ordering and splitting/recombining in ways that will make the heritage of the resulting reconfigured patchset fairly unrecognizable [2]. However, the end result (if I don't mess up) should be a perl-5.8.8 source tree that is exactly identical to the one created by this patch sequence. Note that you have to start with the following, before applying these patches: 1) unpack perl-5.8.8.tar.gz 2) unpack perl-5.8.8-ext-Win32CORE.tar.bz2 right over the top of it [2] Why? Well, for one thing it annoys me to have one patch (0001-) add a bunch of 5.6.1 "packaging" files and an old out-of-date and inaccurate buildscript to the MSYS-STUFF/ directory, and then have 0019 turn around and remove them, while 0005 *modifies* the buildscript we don't use anymore and will remove 12 patches later. Stuff like that. Quick summary: 0001-Import-interesting-stuff-from-perl-5.6.1-MSYS-1.0.11.patch msysgit 0001 patch with the same description 0002-Import-more-interesting-stuff-from-perl-5.6.1-MSYS-1.patch LRN's additional changes. Together, 0001 and 0002 here are equivalent to LRN's 0001- patch in perl-5.8.8-0RC1 0003-Import-test-changes-from-perl-5.6.1-MSYS-1.0.11-src..patch msysgit 0002 patch with the same description 0004-Import-more-test-changes-from-perl-5.6.1-MSYS-1.0.11.patch LRN's additional changes. Together, 0003 and 0004 here are equivalent to LRN's 0002- patch in perl-5.8.8-0RC1 0005-Fix-compilation-on-msysGit.patch msysgit 0003 patch with the same description; same as LRN's 0003- patch in perl-5.8.8-0RC1 0006-Fix-compiling-if-you-have-paths-with-spaces-in-your-.patch msysgit 0004 patch with the same description 0007-Various-fixes-from-LRN-s-port.patch LRN's additional changes. Together, 0006 and 0007 here are equivalent to LRN's 0004- patch in perl-5.8.8-0RC1 0008-fix-issue-with-trying-to-link-with-lm.patch msysgit 0005 patch with the same description; same as LRN's 0005 patch in perl-5.8.8-0RC1 0009-MallocCfg_ptr-is-really-EXT-not-extern.patch msysgit 0006 patch with the same description; same as LRN's 0006 patch in perl-5.8.8-0RC1 0010-msys-pkg-lists.patch LRN's 0007 patch in perl-5.8.8-0RC1 0011-Adapted-from-cygwin-WIN32CORE-patch.patch LRN's 0008 patch in perl-5.8.8-0RC1 0012-XSTypes-capitalization-fix.patch LRN's 0009 patch in perl-5.8.8-0RC1 0013-MSys-cwd.patch LRN's 0010 patch in perl-5.8.8-0RC1 0014-Adapted-from-cygwin-Net-ping-t-patch.patch LRN's 0012 patch in perl-5.8.8-0RC1 0015-MSys-Skip-some-DBM-tests-file-perms-not-supported.patch 0016-MSys-Skip-tests-that-require-socketpair-UDP.patch 0017-cygwin-MSys-Slight-improvement-in-path-canonicalizat.patch 0018-MSys-Avoid-die-in-testsuite-due-to-symlink.patch 0019-MSys-Remove-5.6.1-packaging-files.patch 0020-MSys-Workaround-fclose-success-but-errno-EBADF-issue.patch My new changes... -- Chuck |
From: Charles W. <cwi...@us...> - 2011-03-27 20:50:19
Attachments:
msys-perl-5.8.8-0RC1c-patches.tar.xz
|
On 3/23/2011 11:30 PM, Charles Wilson wrote: > FYI, here's the patchset as it stands currently. There are 20: > > 0001-Import-interesting-stuff-from-perl-5.6.1-MSYS-1.0.11.patch > 0002-Import-more-interesting-stuff-from-perl-5.6.1-MSYS-1.patch > 0003-Import-test-changes-from-perl-5.6.1-MSYS-1.0.11-src..patch > 0004-Import-more-test-changes-from-perl-5.6.1-MSYS-1.0.11.patch > 0005-Fix-compilation-on-msysGit.patch > 0006-Fix-compiling-if-you-have-paths-with-spaces-in-your-.patch > 0007-Various-fixes-from-LRN-s-port.patch > 0008-fix-issue-with-trying-to-link-with-lm.patch > 0009-MallocCfg_ptr-is-really-EXT-not-extern.patch > 0010-msys-pkg-lists.patch > 0011-Adapted-from-cygwin-WIN32CORE-patch.patch > 0012-XSTypes-capitalization-fix.patch > 0013-MSys-cwd.patch > 0014-Adapted-from-cygwin-Net-ping-t-patch.patch > 0015-MSys-Skip-some-DBM-tests-file-perms-not-supported.patch > 0016-MSys-Skip-tests-that-require-socketpair-UDP.patch > 0017-cygwin-MSys-Slight-improvement-in-path-canonicalizat.patch > 0018-MSys-Avoid-die-in-testsuite-due-to-symlink.patch > 0019-MSys-Remove-5.6.1-packaging-files.patch > 0020-MSys-Workaround-fclose-success-but-errno-EBADF-issue.patch Phase 2: I split up and reorganized all the patches, leading to about 55 different (much smaller) ones. The end result is the same [1] as that with the preceding set of 20, which in turn is the same as LRN's original 0RC1 package + 0015-0020 above. [1] Well, there are three minor differences: (1) A few of the original 5.6.1 patches simply added or deleted whitespace. LRN's and the 20-patchset kept those silly changes, the new 55-patchset omits them. (2) In one place, an 'msys' case statement had a cut-n-paste 'Cygwin' comment; I changed that to 'MSys', and (3) the port of cygwin's net-ping-t patch made some changes to 500_ping_icmp.t, but not to the almost identical 110_icmp_inst.t. I made similar changes to 110_icmp_inst.t. So, the 'diffstat' between 20patchset and 55patchset is: lib/ExtUtils/t/Embed.t | 2 +- lib/Net/Ping/t/110_icmp_inst.t | 14 +++++++++++--- t/op/mkdir.t | 1 - t/op/pat.t | 1 - t/op/stat.t | 1 + util.c | 2 +- In any event, the test results between 20patchset and 55patchset are identical, and both are strict (but slight) improvements over LRN's 0RC1, so that's good: no regressions. So, here's the list: 0001-MSys-Core-perl-code-mods.patch 0002-MSys-Core-perl-build-system-mods.patch 0003-MSys-Core-lib-mods-CGI.patch 0004-MacOS-Core-lib-mods-Cwd.patch 0005-MSys-Core-lib-mods-ExtUtils-MakeMaker.patch 0006-MSys-Core-lib-mods-File.patch 0007-MSys-Core-lib-mods-ExtUtils-Embed.patch 0008-MSys-Core-lib-mods-Net.patch 0009-MSys-Core-lib-mods-Pod.patch 0010-MSys-Core-lib-mods-Test-Harness.patch 0011-MSys-Core-lib-mods-Perl-Debugger-perl5db.patch 0012-MSys-Extension-lib-mods-NDBM_File-ODBM_File.patch 0013-MSys-Extension-lib-mods-SDBM_File.patch 0014-MSys-Extension-lib-mods-POSIX.patch 0015-MSys-Extension-lib-mods-Time-HiRes.patch 0016-MSys-Documentation.patch 0017-Cygwin-Strip-after-linking.patch 0018-Cygwin-No-don-t-strip-after-linking.patch 0019-MSys-QUESTIONABLE-Core-perl-build-system-mods.patch 0020-MSys-Correct-environ-handling-bug.patch 0021-MacOS-TEST-set-HARNESS_NOTTY.patch 0022-MSys-TEST-op-tests.patch 0023-MacOS-TEST-io-tests.patch 0024-MSys-TEST-io-tests.patch 0025-MSys-TEST-lib-tests.patch 0026-MSys-TEST-run-tests.patch 0027-MSys-TEST-core-perl-mods.patch 0028-MSys-TEST-core-lib-mods-AnyDBM_File.patch 0029-MSys-TEST-core-lib-mods-ExtUtils.patch 0030-MSys-TEST-core-lib-mods-File-Spec.patch 0031-MSys-TEST-core-lib-mods-IPC.patch 0032-MSys-TEST-core-lib-mods-Net.patch 0033-MSys-TEST-core-lib-mods-User.patch 0034-MSys-TEST-Extension-lib-mods-B.patch 0035-MSys-TEST-Extension-lib-mods-Cwd.patch 0036-MSys-TEST-Extension-lib-mods-DB_File.patch 0037-MSys-TEST-Extension-lib-mods-File-Glob.patch 0038-MSys-TEST-Extension-lib-mods-IO.patch 0039-MSys-TEST-Extension-lib-mods-SDBM_File-GDBM_File.patch 0040-MSys-TEST-QUESTIONABLE-op-tests.patch 0041-MSys-Fix-compiling-if-you-have-paths-with-spaces-in-.patch 0042-MSys-fix-issue-with-trying-to-link-with-lm.patch 0043-MSys-MallocCfg_ptr-is-really-EXT-not-extern.patch 0044-MSys-Various-build-system-fixes-from-LRN.patch 0045-MSys-Incorporate-Win32CORE-extension-into-libperl.patch 0046-MSys-XSTypes-capitalization-fix.patch 0047-MSys-Use-custom-cwd-implementations.patch 0048-MSys-TEST-Extension-lib-mods-Net-Ping.patch 0049-MSys-TEST-Extension-lib-mods-NDBM_File-ODBM_File.patch 0050-MSys-TEST-Extension-lib-mods-Socket.patch 0051-MSys-TEST-Extension-lib-mods-Net-Ping-2.patch 0052-MSys-cygwin-Slight-improvement-in-path-canonicalizat.patch 0053-MSys-TEST-Avoid-die-in-testsuite-due-to-symlink.patch 0054-MSys-Workaround-fclose-success-but-errno-EBADF-issue.patch 0055-MSys-package-lists.patch All are included in the attached tarball. Now, here's the interesting bit: some of the patches were obviously useless for our purposes, having been included way back in the original 5.6.1 patchset "by mistake" it appears: 0004-MacOS-Core-lib-mods-Cwd.patch 0017-Cygwin-Strip-after-linking.patch 0018-Cygwin-No-don-t-strip-after-linking.patch 0021-MacOS-TEST-set-HARNESS_NOTTY.patch 0023-MacOS-TEST-io-tests.patch In addition, LRN's build/packaging procedure used only one of the five files created by 0055-MSys-package-lists.patch and I figured it was better to simply omit that patch, and carry along the msys-perl-noship.list file as a top-level element of the -src tarball, rather than in patch form. Finally, there were two substantive patches I had some concerns about 0019-MSys-QUESTIONABLE-Core-perl-build-system-mods.patch 0040-MSys-TEST-QUESTIONABLE-op-tests.patch The first one actually *undoes* some case-insensitive file system accomodations in the installperl script (patch 0002 above; part of the original "Import interesting things from perl-5.6.1" 0001 patch from msysgit) which keeps the installed .pod files from landing in the same directory as the .pm files for the Pod module. The second one...well, it copies msys-1.0.dll and msys-crypt-1.dll into the test dir. I think that's a bad idea, but I'll test first to see if we can skip it (Reini also mentioned that 'upstream perl' has recently made a change to taint mode on win32/cygwin, so that /bin is left in $PATH...some version of THAT change might be a better approach). So...I did a third round where I used patches 0001-0055, BUT I omitted the following: 0004-MacOS-Core-lib-mods-Cwd.patch 0017-Cygwin-Strip-after-linking.patch 0018-Cygwin-No-don-t-strip-after-linking.patch 0019-MSys-QUESTIONABLE-Core-perl-build-system-mods.patch 0021-MacOS-TEST-set-HARNESS_NOTTY.patch 0023-MacOS-TEST-io-tests.patch 0055-MSys-package-lists.patch (For this round, I continued to use: 0040-MSys-TEST-QUESTIONABLE-op-tests.patch) And...I got the same test results as before. I did have to change the build script somewhat; I also took the opportunity to adopt LRN's suggestion to rebase the (newly compiled) perl extension DLLs as each one is built and installed. (FYI: following modern cygwin perl's suggestion, I'm using 0x56000000 and going up by 0x20000 LRN's original package used 0x05200000 and went down by 0x40000 ^^^ very low! For the rationale behind the 0x20000 minimum rebase interval, see this thread at cygwin: http://cygwin.com/ml/cygwin/2009-06/msg00266.html Note that cygwin implemented a different solution, so they don't REALLY need the 0x20000 'buffer'. However, msys hasn't adopted anything similar, so we DO need a larger buffer than rebase.exe's default 0x10000. OTOH, 0x40000 chews thru a LOT of address space really quickly... Here's my plan going forward: #1) try again, this time ommitting also 0040. Verify that test results don't get any worse, except perhaps in taint mode which we don't really care about. #2) Then, start "improving" (IMO) the configuration, one new option at a time: a) turn on 64bitint support b) turn on threading support c) compile with -DDEBUGGING (which is not "-g"; it simply enables some additional command line switches for debugging perl scripts). However, it creates an ABI change in the libperl, so extension DLLs from 'regular' and "DEBUGGING" perls can't be mixed; it's one or the other. Our perl-5.6.1 had -DDEBUGGING. -- Chuck |
From: LRN <lr...@gm...> - 2011-03-21 10:01:27
|
On 21.03.2011 7:13, Charles Wilson wrote: > On 3/20/2011 5:16 PM, LRN wrote: >> On 21.03.2011 0:04, Charles Wilson wrote: >>> or are the commented-out bits really not necessary in your port. I've >>> attached a copy of the msys-build-perl script that is in your -src, just >>> in case it may differ from the /current/ copy on your system and so that >>> you don't have to re-unpack -src yourself... >> No, i'm just a messy kind of person. And i don't have perl under any >> kind of version control system. Which is why i tend to comment out the >> code i don't need instead of removing it. > I have *basically* the same test results for all the extension packages, > except for: > ... > I suspect this is because you are running a patched msys-1.0.dll that > papers over the EBADF issue, and I'm running stock. True. > Also, one interesting thing: my package contents differ. Part of this > is because of this configuration difference: For some reason, on my > build the search for -lcrypt and -lgdbm / -lgdbm_compat succeeded, > whereas on your build it did not. > > So, my msys-perl5_8.dll has a dep on msys-crypt dll, AND my build > actually built the > ODBM_File > NDBM_File > GDBM_File > extensions, whereas yours skipped those extensions. Hence, my packages > include their implementation files, documentation, etc. I do have msys-crypt, but not libcrypt.dll.a. I guess i've had to install msys-libgcrypt-dev or something. Same goes for gdbm. Should be added to build-time dependency list. > So, ignoring > THOSE extra files, my -bin package ALSO has the following extra files in > bin/: > > ysh shasum scandeps.pl ptee ptardiff ptar pod2readme pod_cover lwp-rget > lwp-request lwp-mirror lwp-download crc32 cpantest config_data > > My hunch is that you did so many install/reinstall cycles that your /bin > already had "old" versions of these files, so ${newer} missed them...and > since they weren't explicitly listed in > MSYS-STUFF/msys-perl-bin.pkglist > nor specifically excluded in > MSYS-STUFF/msys-perl-noship.list > then your package missed them and mine picked them up. Yeah, i have these files in bin, but they are 'old' (2 days older than the newest ones) and weren't picked up. > FWIW, the cygwin perl-5.10.1-5 package contains all of those "extra" > /bin files except for > ysh scandeps.pl pod2readme > so I'm going to add them (all) to msys-perl-bin.pkglist to ensure they > get picked up in the future. Any ideas on how to modify installperl to force it to update timestamps? It would be much easier than figuring out what gets installed and maintaining pkglists > ========= File::Find > > ../lib/File/Find/t/find.t 199 197 98.99% 3-199 > ../lib/File/Find/t/find.t 199 197 98.99% 3-199 > > Hm. wasn't this: > rm -rf ${srcdir}/t/for_find > prior to the test supposed to fix this? I've tried to modify the removal statements in these tests, replacing them with `rm -rf t/for_find` or similar. Still didn't work right. I'm not sure what's going on. > ========= Net::Ping (2) > > ../lib/Net/Ping/t/450_service.t 26 2 7.69% 9 18 > > Failed test 9 in ../lib/Net/Ping/t/450_service.t at line 84 > # ../lib/Net/Ping/t/450_service.t line 84 is: ok $p -> ping("127.0.0.1"); > # Failed test 18 in ../lib/Net/Ping/t/450_service.t at line 143 > # ../lib/Net/Ping/t/450_service.t line 143 is: ok $p -> ack(); > > Again, need to be Administrator. Nope, i have the permissions and it still fails. > Oh, and a few other notes: from the perl-config-summary: > > We probably want thread support -- so I'll give it a try. But if it > "breaks" stuff I won't waste any time on it. > > I think we DO want use64bitint support, but not use64bitall. I don't > think this is very likely to break anything. > > I wonder if I should add -DDEBUGGING. This would allow to use 'perl -D' > to access perl's internal state (see perldoc perlrun). Obviously I > would NOT combine this with -g; there's no use in bloating the distro > perl with 20MB of symbols that aren't much help at -O3. And -O0...no. > > OTOH, -DDDEBUGGING only adds a few 100kB. If it works - great, go on. |
From: LRN <lr...@gm...> - 2011-03-29 08:05:32
Attachments:
Install.pm.hack
|
On 21.03.2011 13:00, LRN wrote: > On 21.03.2011 7:13, Charles Wilson wrote: > >> So, ignoring >> THOSE extra files, my -bin package ALSO has the following extra files in >> bin/: >> >> ysh shasum scandeps.pl ptee ptardiff ptar pod2readme pod_cover lwp-rget >> lwp-request lwp-mirror lwp-download crc32 cpantest config_data >> >> My hunch is that you did so many install/reinstall cycles that your /bin >> already had "old" versions of these files, so ${newer} missed them...and >> since they weren't explicitly listed in >> MSYS-STUFF/msys-perl-bin.pkglist >> nor specifically excluded in >> MSYS-STUFF/msys-perl-noship.list >> then your package missed them and mine picked them up. > Yeah, i have these files in bin, but they are 'old' (2 days older than > the newest ones) and weren't picked up. >> FWIW, the cygwin perl-5.10.1-5 package contains all of those "extra" >> /bin files except for >> ysh scandeps.pl pod2readme >> so I'm going to add them (all) to msys-perl-bin.pkglist to ensure they >> get picked up in the future. > Any ideas on how to modify installperl to force it to update > timestamps? It would be much easier than figuring out what gets > installed and maintaining pkglists I think i have a solution. Kind of. First, for perl itself you have to use "-f" installflag to force it to update files. At least i hope it does. OTOH, from the look of it perl itself was never the problem, so this step might be optional. So, in my buildscript installation now looks like this: make -j1 install.perl INSTALLFLAGS=-f DESTDIR="" 2>&1 | tee ${logdir}/log-install.txt make -j1 install.man install.html DESTDIR="" 2>&1 | tee -a ${logdir}/log-install.txt since installman does not understand the "-f" flag. The real problem is in extensions. They are the ones providing "ysh", "scandeps.pl", "pod2readme" and other nice files that get installed into /bin and NOT updated on new installs. Bad news: there's no equivalent of "-f" for extensions (they are using ExtUtils::Install). But, maybe we can tell them to UNinstall themselves (removing the files if they are there) and then install them again, getting what we want - their files with new timestamps? More bad news: If you do `make uninstall' for an extension, you'll get this: "Uninstall is unsafe and deprecated, the uninstallation was not performed". Cool! So, what do we do? Well, the "kind of" solution i've found is to patch /usr/lib/perl5/5.8.8/ExtUtils/Install.pm (patch is attached) at the beginning of cpan_install() and unpatch it back at the end. With this and with the "-f" flag for perl it is possible to package perl+extensions without backing up lib/perl5/5.8.8, lib/perl5/vendor_perl/5.8.8 and share/man directories. |