From: Erwin W. <wat...@xs...> - 2011-08-25 05:02:27
|
Hi, Would it be hard to create a mingw-coreutils (not msys) package? It's not a package required for MinGW, but I think such a package would be 'nice-to-have', just to have some basic Unix commands available in the Windows Command Prompt. There is UnxUtils (http://unxutils.sourceforge.net/) and GnuWin32 (http://gnuwin32.sourceforge.net/), but these project didn't update in the past years. UnxUtils last update was on March 2003, and GnuWin32 coreutils dates from April 2005. Since there is a more recent msys-coreutils, I was thinking it might be easy to create an up-to-date mingw-coreutils package. regards, -- Erwin Waterlander |
From: Fabian G. <fa...@gr...> - 2011-08-25 07:56:46
|
Hi Erwin, Am 25.08.2011 07:02, schrieb Erwin Waterlander: > There is UnxUtils (http://unxutils.sourceforge.net/) and GnuWin32 > (http://gnuwin32.sourceforge.net/), but these project didn't update in > the past years. UnxUtils last update was on March 2003, and GnuWin32 > coreutils dates from April 2005. these aren't "native" ports either. Unxutils depends on a library called "downhill" to provide POSIX API under Windows and GnuWin32 uses its own LibGW32C. Replace these with msys-1.0.dll and you have the msys-coreutils package - basically. You won't be able to compile coreutils under Windows without such a POSIX compatibility layer library. Though it would be interesting to statically link them against the msys-1.0 library to get stand-alone versions of the utils. But I have been told this is not possible by design... - Fabian |
From: Earnie <ea...@us...> - 2011-08-25 13:38:23
|
Fabian Greffrath wrote: > these aren't "native" ports either. Unxutils depends on a library > called "downhill" to provide POSIX API under Windows and GnuWin32 uses > its own LibGW32C. Replace these with msys-1.0.dll and you have the > msys-coreutils package - basically. I did not know this. > > You won't be able to compile coreutils under Windows without such a > POSIX compatibility layer library. Well, you can but with a lot of work porting to the Windows runtime. > Though it would be interesting to > statically link them against the msys-1.0 library to get stand-alone > versions of the utils. But I have been told this is not possible by > design... Static linking of the runtime is *not* possible by design because of the way fork is emulated. The DLL holds the common data between the parent and the child processes and the child process is initialized with the parents data before the threads are allowed to execute. You can't emulate POSIX on Windows without a common data being shared between two processes. -- Earnie -- http://www.for-my-kids.com |
From: Fabian G. <fa...@gr...> - 2011-08-25 13:51:40
|
Am 25.08.2011 15:38, schrieb Earnie: > I did not know this. But it's true. All programs (i.e. all that are not already natively ported to Win32 API by their respective upstream) in UnxUtils are linked against the Downhill library, which describes itself as "[...] a collection of Win32-compatible routines designed to emulate UNIX API calls. These routines allow UNIX code to run under Win32 OS's (Windows 95, Windows NT and Windows 3.1 with Win32s) with as little modification as possible, without a lot of ugly #ifdefs." [1] And LibGW32C for GnuWin32 "is an implementation of a small part of GLibC, just enough to compile most Unix programs on MS Windows. It contains functions for passwords, process id's groups, and strings. Most are interfaces to the MS-Windows Win32 API. Some are just dummy functions that do nothing." [2] > Static linking of the runtime is *not* possible by design because of the > way fork is emulated. The DLL holds the common data between the parent > and the child processes and the child process is initialized with the > parents data before the threads are allowed to execute. You can't > emulate POSIX on Windows without a common data being shared between two > processes. Both of the aforementioned libraries support static linking AFAICT. - Fabian [1] http://unxutils.cvs.sourceforge.net/viewvc/unxutils/unxutils/Downhill/DOC/README.TXT?revision=1.1.1.1 [2] http://gnuwin32.sourceforge.net/packages/libgw32c.htm |
From: Charles W. <cwi...@us...> - 2011-08-25 14:22:18
|
On 8/25/2011 9:53 AM, Fabian Greffrath wrote: > Both of the aforementioned libraries support static linking AFAICT. And they do not provide an implementation of fork(). Neither do they provide a FULL posix enivonment -- most of their function implementations just set an error and return a nonsense value, like NULL. So, chmod() or setsid() fail. So, things will compile...but lots of operations will fail. Maybe that's ok, if the parts that DO work are all you want. But, imo, what's the point of a setsid.exe or nohup.exe utility that simply does nothing, and exits with error code 1? (OTOH, "sort.exe" and "cat.exe" probably work just fine...) Let's put it this way: while I can see the desire for (some of) the coreutils utilities in a pure native version, like unxutils and gnuwin32 provide, that's not something MinGW.org itself should provide -- especially as those tools will just conflict with actual /fully/ operational ones in our msys environment. Plus, there already exist two -- one active, one defunct -- projects whose purpose DOES encompass that need: gnuwin32 itself (and unxutils). IF you are concerned that the gnuwin32 coreutils is too old, I suggest going to them and volunteering to help update their version. That way everybody benefits, and the right projects get to focus the right people on their core tasks... -- Chuck |
From: Fabian G. <fa...@gr...> - 2011-08-25 14:42:32
|
Am 25.08.2011 16:22, schrieb Charles Wilson: > And they do not provide an implementation of fork(). Neither do they > provide a FULL posix enivonment -- most of their function > implementations just set an error and return a nonsense value, like > NULL. So, chmod() or setsid() fail. I see your point. > So, things will compile...but lots of operations will fail. Maybe > that's ok, if the parts that DO work are all you want. But, imo, what's > the point of a setsid.exe or nohup.exe utility that simply does nothing, > and exits with error code 1? (OTOH, "sort.exe" and "cat.exe" probably > work just fine...) TBH, setsid and nohup are not among the utils that you typically want to have as a native Win32 port - sort and cat however are. But then, as LRN already pointed out, all those utils should at best be used in a real shell and this is what finally brings us back to MSYS. > Let's put it this way: while I can see the desire for (some of) the > coreutils utilities in a pure native version, like unxutils and gnuwin32 > provide, that's not something MinGW.org itself should provide -- > especially as those tools will just conflict with actual /fully/ > operational ones in our msys environment. Plus, there already exist two Agreed, I think it would be nice to have sort, cat, ls, mv, cp and the like as native Win32 ports and part of MinGW for usage in a cmd.exe environment; for those of us who simply got used to using them and always forget that ls is called dir elsewhere. I also think they would not conflict with the ones provided by MSYS but supplement them. Conflicts could be suppressed by MSYS' reordering of the PATH variable - which it already does. > -- one active, one defunct -- projects whose purpose DOES encompass that > need: gnuwin32 itself (and unxutils). IF you are concerned that the > gnuwin32 coreutils is too old, I suggest going to them and volunteering > to help update their version. That way everybody benefits, and the > right projects get to focus the right people on their core tasks... Sure, but I think people may find "mingw-get install mingw32-coreutils" more attractive than downloading separate Binaries and Dependencies tarballs from the GnuWin32 project and manually extracting them into an appropriate directory. ;) - Fabian |
From: Fabian G. <fa...@gr...> - 2011-08-26 11:41:22
|
Am 26.08.2011 09:51, schrieb Fabian Greffrath: > Yes, or maybe revive UnxUtils. I don't know if the downhill library is > more straightforward, but last time I tried to build something against > libgw32c I didn't succeed. Will try out today. I am done for today. Tried to get the Downhill library to compile with MingW. I am currently stuck in DH_SIG.C, where things are getting complicated (crazy pointer conversions between incompatible types). I really doubt it's worth the effort to dig further into this. But I got an idea: What if the "critical" parts of msys-1.0.dll, i.e. the ones that prevent static linking, get replaced by some stubs (like the ones from libgw32c or downhill) depending on whether a dynamic or static library is build. Something like: #ifndef __STATIC_MSYS__ pid_t fork(void) { // actual implementation } #else pid_t fork(void) { return 0; } #endif So if compiled with -D__STATIC_MSYS__ then msys-1.0.dll could be turned into a static, though less functional, library. - Fabian |
From: Erwin W. <wat...@xs...> - 2011-09-05 15:25:21
|
On 08/26/2011 01:42 PM, Fabian Greffrath wrote: > Am 26.08.2011 09:51, schrieb Fabian Greffrath: >> Yes, or maybe revive UnxUtils. I don't know if the downhill library is >> more straightforward, but last time I tried to build something against >> libgw32c I didn't succeed. Will try out today. > I am done for today. Tried to get the Downhill library to compile with > MingW. I am currently stuck in DH_SIG.C, where things are getting > complicated (crazy pointer conversions between incompatible types). I > really doubt it's worth the effort to dig further into this. Hi Fabian, Thanks for trying. Don't spend too much time on it. Erwin |
From: Earnie <ea...@us...> - 2011-08-26 12:15:13
|
Fabian Greffrath wrote: > Agreed, I think it would be nice to have sort, cat, ls, mv, cp and the > like as native Win32 ports and part of MinGW for usage in a cmd.exe > environment; for those of us who simply got used to using them and > always forget that ls is called dir elsewhere. Could .bat files be created to emulate these? Certainly for sort, cat, mv and cp. The output of dir could be modified to emulate ls but would take more than just calling an equivalent cmd shell command. > Sure, but I think people may find "mingw-get install > mingw32-coreutils" more attractive than downloading separate Binaries > and Dependencies tarballs from the GnuWin32 project and manually > extracting them into an appropriate directory. ;) Eventually you'll be able to give a different project host for the install but that means the project would need to have the meta data xml files correct. As Chuck already pointed out this is just confusing and would cause a lot of "why is blah not working like it should" to the list. I warn in the MSYS README that you will be on your own if you replace the MSYS binaries with native ones, i.e. we don't support it. The utilities I chose to include in MSYS were included with good reason and usually the reason is a native version just doesn't provide good results when used within bash. It is the reason we supply an MSYS version of perl as an example. Also, note that it requires more processing for MSYS to fork a native executable because it must translate the strings to convert the paths and it is already slow enough just forking the MSYS process. -- Earnie -- http://www.for-my-kids.com |
From: Charles W. <cwi...@us...> - 2011-08-25 14:10:25
|
On 8/25/2011 9:38 AM, Earnie wrote: > Fabian Greffrath wrote: >> these aren't "native" ports either. Unxutils depends on a library >> called "downhill" to provide POSIX API under Windows and GnuWin32 uses >> its own LibGW32C. Replace these with msys-1.0.dll and you have the >> msys-coreutils package - basically. > > I did not know this. Well, kinda. Both of these libraries are basically stubs. While the goal of cygwin and msys is to provide real posix support (e.g. select() actually works, in exactly the same way it would on linux), downhill and libgw32c simply say: ok, you can call select() and we'll just pass it thru to WaitForMultipleObjects, so you need to pass "select" a windows Handle and not an actual file descriptor. [*] [*] actually, this is the SORT of thing these libraries do, but I picked a bad example with select() -- in actual fact, libgw32c does NOT implement select: int __select (nfds, readfds, writefds, exceptfds, timeout) int nfds; fd_set *readfds; fd_set *writefds; fd_set *exceptfds; struct timeval *timeout; { __set_errno (ENOSYS); return -1; } stub_warning (select) weak_alias (__select, select) #include <stub-tag.h> Anyway, libgw32c provides semi-implementations for certain limited cases -- and usually, those limited cases are "enough" for the things unxutil and gnuwin32 care about. Here's the gnuwin32 implementation of getgrnam: #include <stdlib.h> #include <grp.h> struct group * getgrnam (const char *grp) { return NULL; } Others are "real", like mktemp(), some of the iconv functions, etc. >> You won't be able to compile coreutils under Windows without such a >> POSIX compatibility layer library. > > Well, you can but with a lot of work porting to the Windows runtime. That's basically what downhill and gw32c are: the barest stubs that implement *some* of the posix API in terms of win32 calls, but they don't attempt to provide the full POSIX experience. If it's "too hard" or "too ugly" -- see getgrnam above. And no way, no how, does gw32c attempt to emulate fork: int __fork () { __set_errno (ENOSYS); return -1; } libc_hidden_def (__fork) stub_warning (fork) weak_alias (__fork, fork) #include <stub-tag.h> -- Chuck |
From: Charles W. <cwi...@us...> - 2011-08-25 19:05:44
|
On 8/25/2011 10:43 AM, Fabian Greffrath wrote: > I also think they would not conflict with the ones provided by MSYS > but supplement them. Conflicts could be suppressed by MSYS' reordering > of the PATH variable - which it already does. Uhm, no. In a normal MSYS shell, (e.g. where uname -s reports MINGW32 and not MSYS), the $PATH puts /mingw/bin first. This is deliberate -- if you have both the native mingw gcc compiler and the msys compiler installed, you want to be sure that the mingw one in /mingw/bin is used, NOT the msys one in /usr/bin. So, you do NOT want to re-order your path "in msys" to put the msys /bin first...if you do, you'll use the "correct" msys coreutils, but the wrong compiler! Now, if you put "mingw-coreutils" 'cp.exe' into /mingw/bin and leave the PATH as is...you'll use mingw-cp instead of msys-cp in /usr/bin. And so will all of your configure scripts. And other shell scripts. Are you sure that mingw-cp will work the same as all of those things expect? (IF you're running under msys, at least the shell will handle unix-to-win32 path translation -- but you'll always be relying on its heuristics...even when doing a simple cp. That's a recipe for confusion.) No, mingw-coreutils will DEFINITELY conflict. We already have enough trouble with msys-cygutils-dos2unix vs mingw-cygutils-dos2unix, and those aren't NEARLY as commonly used as coreutils progs. (Ditto msys-autoconf vs mingw-autoconf, automake, libtool, gettext, ... let's not make the problem worse, K?) -- Chuck |
From: Fabian G. <fa...@gr...> - 2011-08-26 07:47:08
|
Am 25.08.2011 21:05, schrieb Charles Wilson: > Uhm, no. In a normal MSYS shell, (e.g. where uname -s reports MINGW32 > and not MSYS), the $PATH puts /mingw/bin first. This is deliberate -- > if you have both the native mingw gcc compiler and the msys compiler > installed, you want to be sure that the mingw one in /mingw/bin is used, > NOT the msys one in /usr/bin. Sorry, Charles, I mixed it up. You are right, I am wrong. ;) > No, mingw-coreutils will DEFINITELY conflict. We already have enough > trouble with msys-cygutils-dos2unix vs mingw-cygutils-dos2unix, and > those aren't NEARLY as commonly used as coreutils progs. (Ditto > msys-autoconf vs mingw-autoconf, automake, libtool, gettext, ... let's > not make the problem worse, K?) No, we indeed shouldn't make it more complicated. It is already confusing enough sometimes, as the above example makes clear. - Fabian |
From: Fabian G. <fa...@gr...> - 2011-08-26 11:41:16
|
Am 26.08.2011 09:51, schrieb Fabian Greffrath: > Yes, or maybe revive UnxUtils. I don't know if the downhill library is > more straightforward, but last time I tried to build something against > libgw32c I didn't succeed. Will try out today. I am done for today. Tried to get the Downhill library to compile with MingW. I am currently stuck in DH_SIG.C, where things are getting complicated (crazy pointer conversions between incompatible types). I really doubt it's worth the effort to dig further into this. But I got an idea: What if the "critical" parts of msys-1.0.dll, i.e. the ones that prevent static linking, get replaced by some stubs (like the ones from libgw32c or downhill) depending on whether a dynamic or static library is build. Something like: #ifndef __STATIC_MSYS__ pid_t fork(void) { // actual implementation } #else pid_t fork(void) { return 0; } #endif So if compiled with -D__STATIC_MSYS__ then msys-1.0.dll could be turned into a static, though less functional, library. - Fabian |
From: Fabian G. <fa...@gr...> - 2011-10-21 19:25:23
|
Hi all, Am Freitag, den 26.08.2011, 13:42 +0200 schrieb Fabian Greffrath: > But I got an idea: What if the "critical" parts of msys-1.0.dll, i.e. > the ones that prevent static linking, get replaced by some stubs (like > the ones from libgw32c or downhill) depending on whether a dynamic or > static library is build. alright, I admit that the idea to replace fork() with a stub is a stupid one. But if fork() is really the only implemented function (family) that prevents static linking, wouldn't it be possible to create a static library and a set of headers that simply do not provide these functions anymore? Upstream configure scripts would need to detect this situation and fall back to calling system() in appropriate situations instead. - Fabian |
From: Earnie <ea...@us...> - 2011-10-22 16:35:26
|
Fabian Greffrath wrote: > Hi all, > > Am Freitag, den 26.08.2011, 13:42 +0200 schrieb Fabian Greffrath: >> But I got an idea: What if the "critical" parts of msys-1.0.dll, i.e. >> the ones that prevent static linking, get replaced by some stubs (like >> the ones from libgw32c or downhill) depending on whether a dynamic or >> static library is build. > > alright, I admit that the idea to replace fork() with a stub is a stupid > one. But if fork() is really the only implemented function (family) that > prevents static linking, wouldn't it be possible to create a static > library and a set of headers that simply do not provide these functions > anymore? Upstream configure scripts would need to detect this situation > and fall back to calling system() in appropriate situations instead. All things are possible with time and commitment. Good luck in pursuing this dream. I for one don't foresee it coming to fruition beyond a fork maintained by you but I sure hope you prove me wrong. -- Earnie -- http://www.for-my-kids.com |
From: LRN <lr...@gm...> - 2011-08-25 07:57:42
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 25.08.2011 9:02, Erwin Waterlander wrote: > Hi, > > Would it be hard to create a mingw-coreutils (not msys) package? > > It's not a package required for MinGW, but I think such a package > would be 'nice-to-have', just to have some basic Unix commands > available in the Windows Command Prompt. > My personal opinion: Bullshit. Setting aside technical difficulties that might arise during the port, i think that core utils are not as useful as you'd expect them to be. Normal POSIX practice of using commandline utilities from the console ALSO requires a POSIX-compatible shell. Without such shell (and Windows CMD is not POSIX-compatible) you're reduced to using only basic pipelines, without any kind of shell processing. I'm not sure that this is very useful, even for users accustomed to core utils (for users not accustomed to these utils they are just a bunch of cryptic .exe files anyway). If you think of it, utils are (to some extent) crutches for the shell language, which is utterly incapable of doing many useful things by itself. Myself, i've discovered that modern scripting languages with rich standard libraries (such as Python) offer enough power to write equivalents of shell utilities in about 100 or 200 LOC (if you cut out all the functionality that makes them into swiss knives, and concentrate on your real needs). Which makes the idea of messing with binary utils even less compelling. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJOVgAFAAoJEOs4Jb6SI2Cwjq0IANe1P2GxBNx7ZgMJptUoQvna zqNOR19AIOzK+yPwfNG77ZG+546iK/lzOxIgOsbhp6sht8b21T6MbBk1tRnkdP2n wJPUJdW6Tq33817MfE89Hnlr4pS92/EPkq0VBkRkJB8qaRaSe1uMtzGuAesSt3la 88xJZGLITzqbOcx5FEHZBCa1id3tclYPO5P7q8ilpD3sFnG4E/LfD9d8t9K1F+LH XzFvX9aiyYmftARKKhpxXrRFEbyQk3RgDCROjgSoy6QC38DSBoZBcUQ83k+0Hdq7 Tc975W3QFUsC4+j1F7hWxKcSLKCJEVW5Qh0qZqN2ik7V1W3xd80Lv4f1iP/Gvwc= =uDX6 -----END PGP SIGNATURE----- |
From: Earnie <ea...@us...> - 2011-08-25 13:49:28
|
Erwin Waterlander wrote: > > Since there is a more recent msys-coreutils, I was thinking it might be > easy to create an up-to-date mingw-coreutils package. > "easy" is a term relative to the amount of time you have to port and debug. As has been stated already, coreutils is POSIX dependent in its current form. If you want to transform coreutils to be Windows dependent then I suggest you setup a code repository management system that you can easily control the changes of code between the master and the fork. I seriously doubt that the upstream maintainers would encumber the code with a bunch of if __windows__ statements. -- Earnie -- http://www.for-my-kids.com |
From: Mcgroder, J. <jam...@hp...> - 2011-08-25 14:00:20
|
On Thursday, August 25, 2011 9:49 AM Earnie wrote: [...] > I seriously doubt that the upstream maintainers > would encumber the code with a bunch of > if __windows__ statements. I certainly hope not... it would be a complete waste of time considering most of us started using MSYS to GET AWAY from DOS... -- Jim |
From: Erwin W. <wat...@xs...> - 2011-08-25 15:23:46
|
On 08/25/2011 04:43 PM, Fabian Greffrath wrote: > Am 25.08.2011 16:22, schrieb Charles Wilson: > >> Let's put it this way: while I can see the desire for (some of) the >> coreutils utilities in a pure native version, like unxutils and gnuwin32 >> provide, that's not something MinGW.org itself should provide -- >> especially as those tools will just conflict with actual /fully/ >> operational ones in our msys environment. Plus, there already exist two > Agreed, I think it would be nice to have sort, cat, ls, mv, cp and the > like as native Win32 ports and part of MinGW for usage in a cmd.exe > environment; for those of us who simply got used to using them and > always forget that ls is called dir elsewhere. > > Indeed. >> -- one active, one defunct -- projects whose purpose DOES encompass that >> need: gnuwin32 itself (and unxutils). IF you are concerned that the >> gnuwin32 coreutils is too old, I suggest going to them and volunteering >> to help update their version. That way everybody benefits, and the >> right projects get to focus the right people on their core tasks... > Sure, but I think people may find "mingw-get install > mingw32-coreutils" more attractive than downloading separate Binaries > and Dependencies tarballs from the GnuWin32 project and manually > extracting them into an appropriate directory. ;) > A simple zip file would do... I agree with Charles that helping GnuWin32 is the best way to go. If I only had the time... Erwin |
From: Fabian G. <fa...@gr...> - 2011-08-26 07:50:24
|
Am 25.08.2011 17:23, schrieb Erwin Waterlander: > Indeed. I have added MSYS' bin/ directory to my PATH variable in Windows and thus all the coreutils are available in an cmd.exe environment as well; and they feel quite natural. > A simple zip file would do... I agree with Charles that helping GnuWin32 > is the best way to go. If I only had the time... Yes, or maybe revive UnxUtils. I don't know if the downhill library is more straightforward, but last time I tried to build something against libgw32c I didn't succeed. Will try out today. - Fabian |