From: Charles W. <cwi...@us...> - 2010-05-09 00:52:56
|
As discussed previously, I've been testing Dave Korn's fix for pseudo-relocs. The bug manifests when complex relocations are performed, and then the process forks. When cygwin (msys) forks, it copies over much of the address space from the forker to the forkee. So, only SOME of the relocations should be (re)performed in the forkee -- but without this fix or something like it ALL relocations are (re)performed in the forkee. This 'double relocation' causes segfaults. Dave's patch differs from what cygwin decided to do; over in cygwin-land they moved most of the pseudo-reloc machinery into the cygwin DLL itself, rather than linking the pseudo-reloc code statically into each and every EXE and DLL. However, we can't easily use their version, because...well, cygwin's startup code as of 1.7.x is very different from its 1.3.4 version, and since MSYS forked from cygwin-1.3.4 it still looks a lot like its predecessor, and not very much like current cygwin. Anyway, I've tested this patch in MSYS, and 1) it doesn't break anything -- all of my old "pseudo-reloc" tests, for both "v1" and "v2" relocs, still work 2) it allows me to create a working, DLL-based, auto-import/pseudo-reloc-v2-based, build of guile and autogen. Yay! I've also attached a one-liner fix for cygwin/net.cc; my earlier change to include/netdb.h created a signature conflict for getdomainname(). OK to apply? (FWIW, if Cesar releases a new DLL it should probably get version bumped to 1.0.15 -- if for no other reason than the net.cc change below.) 2010.05.07 Dave Korn <...> * lib/pseudo-reloc.c (memskip_t): New struct and typedef. (__write_memory): Accept an optional memskip_t argument and avoid writing to any memory ranges mentioned in the linked list. (do_pseudo_reloc): Accept an optional memskip_t argument and pass it through in all calls to __write_memory. (_pei386_runtime_relocator): When reapplying relocs in a forked child process, avoid doubly-relocating the .data and .bss sections that were copied from the parent. 2010.05.07 Charles Wilson <...> * net.cc (getdomainname): Align signature with declaration in include/netdb.h. -- Chuck |
From: Charles W. <cwi...@us...> - 2010-05-12 02:29:37
|
On 5/8/2010 8:52 PM, Charles Wilson wrote: > OK to apply? Ping? > 2010.05.07 Dave Korn <...> > > * lib/pseudo-reloc.c (memskip_t): New struct and typedef. > (__write_memory): Accept an optional memskip_t argument and > avoid writing to any memory ranges mentioned in the linked list. > (do_pseudo_reloc): Accept an optional memskip_t argument and > pass it through in all calls to __write_memory. > (_pei386_runtime_relocator): When reapplying relocs in a > forked child process, avoid doubly-relocating the .data and > .bss sections that were copied from the parent. > > 2010.05.07 Charles Wilson <...> > > * net.cc (getdomainname): Align signature with > declaration in include/netdb.h. |
From: Cesar S. <ces...@gm...> - 2010-05-12 09:40:04
|
On 11/5/2010 23:28, Charles Wilson wrote: > On 5/8/2010 8:52 PM, Charles Wilson wrote: >> OK to apply? > > Ping? > Thanks for the patch. I will take a look shortly, reading again the references you provided, make sure there are no obvious mistakes and run it through a few testcases. Thanks, Cesar |
From: Cesar S. <ces...@gm...> - 2010-05-13 03:50:09
|
On 12/5/2010 06:39, Cesar Strauss wrote: > On 11/5/2010 23:28, Charles Wilson wrote: >> On 5/8/2010 8:52 PM, Charles Wilson wrote: >>> OK to apply? >> >> Ping? >> > > Thanks for the patch. I will take a look shortly, reading again the > references you provided, make sure there are no obvious mistakes and run > it through a few testcases. > The patch seems OK. The comments really help to understand the issue. The resulting package builds and runs fine. The message given at [1] seems to indicate it is supposed to fix the testcases at [2]. Could you try them? They crash for me in MSYS both with and without this patch. [1] http://cygwin.com/ml/cygwin-patches/2010-q2/msg00004.html [2] http://cygwin.com/ml/cygwin/2010-04/msg00957.html Just a minor nit: > + /* In that case, we want to avoid applying any pseudo-relocs that we > + know will already have been copied, pre-applied, from the parent's > + .data and .bss sections. See the references to child_copy() in > + dcrt0.cc#child_info_fork::handle_fork() and fork.cc#frok::parent() > + for the details. The references given above don't apply to MSYS. There is no child_info_fork class in dcrt0.cc, nor a frok class in fork.cc. Please either update these references for MSYS, or leave them out altogether. OK to apply after correcting the text. Thanks, Cesar |
From: Charles W. <cwi...@us...> - 2010-05-13 03:56:55
|
On 5/12/2010 11:49 PM, Cesar Strauss wrote: > The patch seems OK. The comments really help to understand the issue. > The resulting package builds and runs fine. Thanks. > The message given at [1] seems to indicate it is supposed to fix the > testcases at [2]. Could you try them? They crash for me in MSYS both > with and without this patch. > [1] http://cygwin.com/ml/cygwin-patches/2010-q2/msg00004.html > [2] http://cygwin.com/ml/cygwin/2010-04/msg00957.html I saw the same behavior. AFAICT, our version of g++-3.4.5 is the problem. With g++-4.3.4, the only remaining issue with the test case was the one that is fixed by this patch. However, with our old version of g++, there are so many other things going wrong that just fixing the double-pseudo-reloc is, well, swamped by everything else. > Just a minor nit: > >> + /* In that case, we want to avoid applying any pseudo-relocs that we >> + know will already have been copied, pre-applied, from the parent's >> + .data and .bss sections. See the references to child_copy() in >> + dcrt0.cc#child_info_fork::handle_fork() and fork.cc#frok::parent() >> + for the details. > > The references given above don't apply to MSYS. There is no > child_info_fork class in dcrt0.cc, nor a frok class in fork.cc. Please > either update these references for MSYS, or leave them out altogether. Right, that code has been refactored, a LOT. The MSYS code does many of the same things, but in different locations. I'll update when I get a chance. > OK to apply after correcting the text. Thanks. -- Chuck |
From: Charles W. <cwi...@us...> - 2010-05-19 03:59:58
Attachments:
build-machinery-1.0.15.patch
|
On 5/12/2010 11:56 PM, Charles Wilson wrote: >> OK to apply after correcting the text. Updated and applied. If you'd like to bump the version to reflect the aggregate changes: * Add declarations of fchdir and getdomainname to sys/unistd.h * Add declarations of rcmd, rexec, and rresvport to netdb.h * Correct bug involving double evaluations of pseudo-relocations after fork(). Affected MSYS applications must be recompiled to take advantage of this fix. * Add --replace option to mount and umount scripts * Split -bin component into -bin and -ext. and release a new package...that'd be just peachy. (Where '-ext' is understood to still be part of the basic MSYS installation; it merely contains those components that have dependencies on tools -- like bash.exe -- that are *outside* the msysCORE package itself). -- Chuck |
From: Cesar S. <ces...@gm...> - 2010-06-05 01:12:13
|
On 19/5/2010 00:59, Charles Wilson wrote: > On 5/12/2010 11:56 PM, Charles Wilson wrote: > >>> OK to apply after correcting the text. > > Updated and applied. If you'd like to bump the version to reflect the > aggregate changes: Sure. > and release a new package...that'd be just peachy. (Where '-ext' is > understood to still be part of the basic MSYS installation; it merely > contains those components that have dependencies on tools -- like > bash.exe -- that are *outside* the msysCORE package itself). > Fine by me. Did we ever reach a definitive agreement on this? Cesar |
From: Charles W. <cwi...@us...> - 2010-06-05 13:59:01
|
On 6/4/2010 9:12 PM, Cesar Strauss wrote: >> and release a new package...that'd be just peachy. (Where '-ext' is >> understood to still be part of the basic MSYS installation; it merely >> contains those components that have dependencies on tools -- like >> bash.exe -- that are *outside* the msysCORE package itself). >> > > Fine by me. Did we ever reach a definitive agreement on this? Neither Earnie nor Keith ever definitively said "OK"; the thread kinda died. Earnie's only post in the relevant thread was in a subthread concerned with British vs. American spelling; the last word from Keith was: http://thread.gmane.org/gmane.comp.gnu.mingw.devel/3747/focus=3803 > I haven't forgotten about this, but I'd like to give it some further > thought, before replying formally. -- Chuck |
From: Earnie <ea...@us...> - 2010-06-07 11:45:39
|
Charles Wilson wrote: > On 6/4/2010 9:12 PM, Cesar Strauss wrote: >>> and release a new package...that'd be just peachy. (Where '-ext' is >>> understood to still be part of the basic MSYS installation; it merely >>> contains those components that have dependencies on tools -- like >>> bash.exe -- that are *outside* the msysCORE package itself). >>> >> >> Fine by me. Did we ever reach a definitive agreement on this? > > Neither Earnie nor Keith ever definitively said "OK"; the thread kinda > died. Earnie's only post in the relevant thread was in a subthread > concerned with British vs. American spelling; the last word from Keith was: > > http://thread.gmane.org/gmane.comp.gnu.mingw.devel/3747/focus=3803 >> I haven't forgotten about this, but I'd like to give it some further >> thought, before replying formally. > The topic was interesting but I don't have a desire to comment until Keith has responded especially since mingw-get is Keith's baby ATM. Earnie |
From: Keith M. <kei...@us...> - 2010-06-07 18:03:06
|
On Saturday 05 June 2010 14:58:35 Charles Wilson wrote: > On 6/4/2010 9:12 PM, Cesar Strauss wrote: > >> and release a new package...that'd be just peachy. (Where > >> '-ext' is understood to still be part of the basic MSYS > >> installation; it merely contains those components that have > >> dependencies on tools -- like bash.exe -- that are *outside* the > >> msysCORE package itself). > > > > Fine by me. Did we ever reach a definitive agreement on this? > > Neither Earnie nor Keith ever definitively said "OK"; the thread > kinda died. Earnie's only post in the relevant thread was in a > subthread concerned with British vs. American spelling; the last > word from Keith was: > > http://thread.gmane.org/gmane.comp.gnu.mingw.devel/3747/focus=3803 > > > I haven't forgotten about this, but I'd like to give it some > > further thought, before replying formally. Indeed. Although technically that was a different discussion thread, I still owe you that response. I haven't given it a great deal of further thought. Yes, to avoid circular dependencies, we *need* to subdivide the current msysCORE into *at least* two separate tarballs, and I think the division Chuck proposed in http://thread.gmane.org/gmane.comp.gnu.mingw.devel/3747/focus=3790 seems reasonable. (At first, I was inclined to favour a split into a -dll and a -bin pairing, but since that would leave the -dll with additional content, beyond a normal -dll payload, the proposed split into -bin and -ext may be a better naming choice; I've no strong preference either way, and will defer to whatever you, Cesar and Chuck, ultimately agree on). One additional split I *would* make: carve out *all* of the /share/doc content, into separate -lic (for licence specifics) and -doc (for the rest) packages. As a final comment: we may need to review the post-install scripting at a later date. Eventually, we would hope to be able to script it directly into the mingw-get manifest, and have mingw-get execute it directly, (with already built-in knowledge of the sysroot path for any attendant MinGW installation). For the time being, it will probably be best to leave it as it is, in the -ext tarball. -- Regards, Keith. |
From: Charles W. <cwi...@us...> - 2010-06-07 18:56:13
|
On Mon, 07 Jun 2010 15:10 +0100, "Keith Marshall" wrote: > I haven't given it a great deal of further thought. Yes, to avoid > circular dependencies, we *need* to subdivide the current msysCORE > into *at least* two separate tarballs, and I think the division Chuck > proposed in > > http://thread.gmane.org/gmane.comp.gnu.mingw.devel/3747/focus=3790 > > seems reasonable. (At first, I was inclined to favour a split into > a -dll and a -bin pairing, but since that would leave the -dll with > additional content, beyond a normal -dll payload, the proposed split > into -bin and -ext may be a better naming choice; I've no strong > preference either way, and will defer to whatever you, Cesar and > Chuck, ultimately agree on). > > One additional split I *would* make: carve out *all* of the /share/doc > content, into separate -lic (for licence specifics) and -doc (for the > rest) packages. Well, moving the /share/doc stuff to a "-doc" and "-lic" package means that the "bin" package would contain only: bin/msys-1.0.dll bin/msysmnt.exe bin/ps.exe etc/fstab.sample etc/inputrc.default etc/profile m.ico msys.ico which not all THAT different from "just the DLL". I usually try to take my cue from the cygwin project (they've got a TON of users, and have been around a lot longer than any part of MinGW/MSYS other than MinGW's gcc and binutils ports). Their "cygwin-bin" package contains the cygwin DLL, and many of the .exes that are compiled from the winsup/cygwin source repository -- even though most of their packages follow our own concept where DLLs are in tarballs by themselves, and everything else (from each product) is put in different tarballs. So, that's why I proposed this sort of split. For instance, if we ever imported a version of their cygcheck tool from their winsup repo into our msys repo, I'd imagine that .exe would be part of the -bin package above. Ditto their new ldd.exe (the latter is rather more likely that the former, IMO...) Sounds like we have enough info from most of the interested parties. I'll leave the final decision to Cesar. > As a final comment: we may need to review the post-install scripting > at a later date. Eventually, we would hope to be able to script it > directly into the mingw-get manifest, and have mingw-get execute it > directly, (with already built-in knowledge of the sysroot path for > any attendant MinGW installation). For the time being, it will > probably be best to leave it as it is, in the -ext tarball. Yes, since otherwise, as .sh requires bash or dash, we'd have a circular dependency. So...the GNU Way(tm) to add extensible scripting is to link in libguile. Not sure I'm a big fan of Scheme, but statically linking into mingw-get a mingw version of libguile wouldn't be terribly difficult (that's what it's for, after all). Neither would lua: {C snippet}======================== /* initialize Lua */ L = lua_open(); /* load Lua base libraries */ lua_baselibopen(L); <<< not sure you need to do this if you <<< statically link all the lua libs you want /* run the script */ lua_dofile(L, "do-me.lua"); /* cleanup Lua */ lua_close(L); {C snippet}======================== But unlike cygwin's setup (or, for that matter, RPM and DEB) I think any package scripting should be interpreted and executed directly by the install application (or an interpreter embedded in the application itself), and not shelled out to an external interpreter like bash -- whether that interpreter is guile/Scheme, lua, ruby, python, or perl. Or what the heck: adapt libsic from the GVV (and others) own autotool book. Mainly so that we can completely avoid the whole "modifying/replacing in-use-files" problem -- and the "but what if I'm just installing MinGW, and not MSYS, and have no bash?" problem. -- Chuck |
From: Keith M. <kei...@us...> - 2010-06-08 18:18:19
|
On Monday 07 June 2010 19:56:05 Charles Wilson wrote: > On Mon, 07 Jun 2010 15:10 +0100, "Keith Marshall" wrote: > > One additional split I *would* make: carve out *all* of the > > /share/doc content, into separate -lic (for licence specifics) > > and -doc (for the rest) packages. > > Well, moving the /share/doc stuff to a "-doc" and "-lic" package > means that the "bin" package would contain only: > > bin/msys-1.0.dll > bin/msysmnt.exe > bin/ps.exe > etc/fstab.sample > etc/inputrc.default > etc/profile > m.ico > msys.ico Is that a problem? I don't believe that it should be. Leaving the documentation and licensing details in there seems more so, since all other packages follow the principles we've laid out in the Package Identification HOWTO, and that advocates separating them out. > which not all THAT different from "just the DLL". No, it isn't. The apparent odd-balls are msysmnt.exe, (which is primarily a library hook for the mount/umount scripts anyway), ps.exe and etc/profile; the rest, as you previously pointed out, are just arbitrary data files, with no dependency on anything else, so it makes complete sense to put them as close to the origin of the dependency graph as possible. I fully concur with your reasoning for putting etc/profile in there. Although it seems that, being a shell script, it would depend on a shell package such as msys-bash, it is actually the other way round; it specifies the MSYS specific start-up configuration required *by* the shell itself, so it does more correctly belong here. Now, that just leaves ps.exe, which is a free-standing user-space tool; it would seem out of place in a -dll package, but it imposes no external dependencies, so again, it seems appropriate to put it at this level in the dependency graph. Then, because it would seem out of place in a -dll package, the scales are tipped toward the choice of -bin for this particular package. (Of course, we could split it further, and provide -dll, -bin and -ext packages, but that would seem to be overkill ... see below). > I usually try to > take my cue from the cygwin project (they've got a TON of users, > and have been around a lot longer than any part of MinGW/MSYS other > than MinGW's gcc and binutils ports). Their "cygwin-bin" package > contains the cygwin DLL, and many of the .exes that are compiled > from the winsup/cygwin source repository -- even though most of > their packages follow our own concept where DLLs are in tarballs by > themselves, and everything else (from each product) is put in > different tarballs. That's fine; we can always learn from the experiences of others, but we aren't obliged to always follow their lead. However, in this case it does make sense to do so; as in the case of cygwin1.dll, the MSYS dll serves no useful purpose, (or at least it shouldn't), outside of an MSYS installation. Thus, there would be little tangible benefit from separating msys-1.0.dll from the core -bin package; it doesn't fall into the category of "redistributable", in the sense that users may need to redistribute it as a component of their own free-standing applications; (they shouldn't be using it this way, so we don't need to make it easy for them to abuse it; therefore, I consider any such separation to be overkill, for the sake of a naming convention). > So, that's why I proposed this sort of split. For instance, if we > ever imported a version of their cygcheck tool from their winsup > repo into our msys repo, I'd imagine that .exe would be part of the > -bin package above. Ditto their new ldd.exe (the latter is rather > more likely that the former, IMO...) Likely so, but in any case, these would seem to fall into the same category as ps.exe anyway. > Sounds like we have enough info from most of the interested > parties. I'll leave the final decision to Cesar. I concur. > > As a final comment: we may need to review the post-install > > scripting at a later date. Eventually, we would hope to be able > > to script it directly into the mingw-get manifest, and have > > mingw-get execute it directly, (with already built-in knowledge > > of the sysroot path for any attendant MinGW installation). For > > the time being, it will probably be best to leave it as it is, in > > the -ext tarball. > > Yes, since otherwise, as .sh requires bash or dash, we'd have a > circular dependency. Exactly so. However, my point was more that, with mingw-get, we don't yet have any convenient mechanism to execute them, as a post-install requirement, beyond politely suggesting to the user that he/she may like to run them manually, to complete a manual installation. Once we get to the point where mingw-get can perform post-install actions, will we need to provide these free-standing scripts anyway? Maybe, for historical interest only? > So...the GNU Way(tm) to add extensible scripting is to link in > libguile. Not sure I'm a big fan of Scheme, but statically linking > into mingw-get a mingw version of libguile wouldn't be terribly > difficult (that's what it's for, after all). Neither would lua: ... Well, that's a discussion for a (much) later date, and a new thread. I'd planned to investigate lua, or tcl, when we get to that point; guile would be another possibility, as indeed would be a home brewed solution; (I've done that before, for an old MS-DOS application). > But unlike cygwin's setup (or, for that matter, RPM and DEB) I > think any package scripting should be interpreted and executed > directly by the install application (or an interpreter embedded in > the application itself), and not shelled out to an external > interpreter ... I agree. I have every intention to (ultimately) provide an embedded interpreter within mingw-get, without any external dependency. > whether that interpreter is guile/Scheme, > lua, ruby, python, or perl. Or what the heck: adapt libsic from the > GVV (and others) own autotool book. Whatever ... > Mainly so that we can completely avoid the whole > "modifying/replacing in-use-files" problem -- and the "but what if > I'm just installing MinGW, and not MSYS, and have no bash?" > problem. Exactly so. -- Regards, Keith. |
From: Charles W. <cwi...@us...> - 2010-06-08 20:05:27
|
On Tue, 08 Jun 2010 10:19 +0100, "Keith Marshall" wrote: > On Monday 07 June 2010 19:56:05 Charles Wilson wrote: > > On Mon, 07 Jun 2010 15:10 +0100, "Keith Marshall" wrote: > > > One additional split I *would* make: carve out *all* of the > > > /share/doc content, into separate -lic (for licence specifics) > > > and -doc (for the rest) packages. > > > > Well, moving the /share/doc stuff to a "-doc" and "-lic" package > > means that the "bin" package would contain only: > > > > bin/msys-1.0.dll > > bin/msysmnt.exe > > bin/ps.exe > > etc/fstab.sample > > etc/inputrc.default > > etc/profile > > m.ico > > msys.ico > > Is that a problem? I don't believe that it should be. No, I didn't mean to imply that it would be. > Leaving the > documentation and licensing details in there seems more so, since all > other packages follow the principles we've laid out in the Package > Identification HOWTO, and that advocates separating them out. > > > which not all THAT different from "just the DLL". > > No, it isn't. The apparent odd-balls are msysmnt.exe, (which is > primarily a library hook for the mount/umount scripts anyway), ps.exe > and etc/profile; the rest, as you previously pointed out, are just > arbitrary data files, with no dependency on anything else, so it > makes complete sense to put them as close to the origin of the > dependency graph as possible. Yup. > (Of course, we could split it > further, and provide -dll, -bin and -ext packages, but that would > seem to be overkill ... see below). Definitely overkill. > > I usually try to > > take my cue from the cygwin project > > That's fine; we can always learn from the experiences of others, but > we aren't obliged to always follow their lead. Of course. I usually looked at cygwin, debian, redhat, mandriva, and suse packages for clues when I was putting together the updated MSYS packages last August and again this year -- but didn't blindly follow any of them. > However, in this case > it does make sense to do so; as in the case of cygwin1.dll, the MSYS > dll serves no useful purpose, (or at least it shouldn't), outside of > an MSYS installation. Thus, there would be little tangible benefit > from separating msys-1.0.dll from the core -bin package; it doesn't > fall into the category of "redistributable", in the sense that users > may need to redistribute it as a component of their own free-standing > applications; Right. > (they shouldn't be using it this way, so we don't need > to make it easy for them to abuse it; therefore, I consider any such > separation to be overkill, for the sake of a naming convention). Ack. > > [cygwin: cygcheck.exe, ldd.exe utilities] > > Likely so, but in any case, these would seem to fall into the same > category as ps.exe anyway. Precisely. > > > [post-install scripting] > > > > Yes, since otherwise, as .sh requires bash or dash, we'd have a > > circular dependency. > > Exactly so. However, my point was more that, with mingw-get, we don't > yet have any convenient mechanism to execute them, as a post-install > requirement, beyond politely suggesting to the user that he/she may > like to run them manually, to complete a manual installation. True. > Once > we get to the point where mingw-get can perform post-install actions, > will we need to provide these free-standing scripts anyway? Maybe, > for historical interest only? Hmm. Looking at pi.sh, all it is really concerned with is: 1) ensuring /etc/fstab has an entry for /mingw, if MinGW is installed --> this is most likely a job for mingw-get to do explicitly itself, rather than delegating to ANY script. 2) checking for script vs. exe conflicts with regards to the following names: cmd rvi vi: prefer script, remove .exe ftp ln make awk echo egrep fgrep printf pwd ex rview rvim view: prefer .exe, rm script 3) Ensuring that /mingw/bin/make.exe doesn't exist, or renaming it to /mingw/bin/mingw32-make.exe if it does. #2 and #3 really should be handled by taking proper care in how we put together the various packages, now that we're no longer tied to the giant all-in-one installers. And #1 is a job for mingw-get. So, perhaps we can do away with pi.sh/pi.bat. Now, it is true that none of our other packages have postinstall scripts -- but that's simply because there was no facility for executing them. I could see where it'd be nice, instead of having /etc/foo.default, to use postinstall/preremove scripts to manage "real" config files -- e.g. on uninstall, remove-if-unchanged-from-default; on install, cp-from-default-if-not-yet-present. But...it's certainly not anything we need to worry about right away. > > [script interpreters...] > > Well, that's a discussion for a (much) later date, and a new thread. Yep. > I agree. I have every intention to (ultimately) provide an embedded > interpreter within mingw-get, without any external dependency. No hurry, tho. > > Mainly so that we can completely avoid the whole > > "modifying/replacing in-use-files" problem -- and the "but what if > > I'm just installing MinGW, and not MSYS, and have no bash?" > > problem. > > Exactly so. Ack. -- Chuck |