MinGW - Minimalist GNU for Windows

Keith Marshall wrote:
>
>    $ rebase -b 0x30000000 /path/to/msys-1.0.dll
>

It might be worth Cesar dropping the base address to the user address 
space by default.  I had put it in the system address space without any 
issue at the time of the fork but if it is causing issue now, it 
wouldn't hurt to move it.  If I remember correctly Cygwin bases the DLL 
at 0x60000000 and I used 0x70000000.

Earnie

On 5/26/2010 8:56 AM, Keith Marshall wrote:\
> you said:
>> BTW, just as cygwin-rebase can't rebase cygwin1.dll, my msys port of
>> rebase can't rebase msys-1.0.dll.
>
> I assume that remains true of the package you eventually released?

Correct. rebase.exe depends on msys.  Mostly, because in order to be
really useful, you need rebaseall, which is a shell script.  So, that
means you can't use rebaseall to rebase the shell interpreter (in this
case, dash) nor the runtime used by dash, regardless of whether
rebase.exe itself relies on that runtime.

>> Maybe we need a pure win32 version...
> 
> Johannes Schindelin responded to that, offering just such a version, 
> which he had provided for msys-git. 

Johannes' version:
version=2.4.2-1
url=http://www.tishler.net/jason/software/rebase
+ a small patch to increase the space allotted for each DLL
That is, this is just an old version of Jason Tishler's original software.

current msys version:
rebase-3.0.1-1
<cygwin mirror>/release/rebase/

(which...just happens to be Jason Tisher's most recent version, anyway).

I guess what you're really asking is, can our rebase.exe (and
peflags.exe) be built as a native win32 app.  I think it can -- but see
below.

> Given that Chris Sutcliffe's 
> original problem, (which I myself encountered, identically, when I 
> upgraded to the current msys-1.0.dll release), entails exactly that 
> requirement, (that msys-1.0.dll must be rebased), will your rebase 
> package suffice to address the issue?  (FWIW, msys-1.0.dll is the 
> ONLY shared object which I have needed to rebase to date).

Well, for some reason I had always thought that cygwin1.dll itself was
not "rebasable" -- that, even if you used some native tool to change its
ImageBaseAddress, it would die horribly for some esoteric reason
(hardcoded pointer values? dunno...).

So, I've never attempted to rebase either cygwin1.dll or msys-1.0.dll. I
did find one message on the cygwin list from last December, recommending
to use rebase(all) to rebase *a copy* of cygwin1.dll, and then use
native tools to move it into place.  This had something to do with
trying to outsmart a BLODA.

> Since msys-1.0.dll itself seems to be most consistently reported as 
> problematic,

Errr...huh?  In my experience, it has *never* been the msys DLL that
couldn't be loaded in the same location in the child as in the parent --
mostly because it is the FIRST dependency to be loaded, so there's
nothing with which it could conflict.  It's more likely to be "Cwd.dll"
or "POSIX.dll" (some perl extension modules loaded almost last), or
sometimes msys-intl-8.dll -- or any of the guile srfi dlls.

> (and if you haven't already done so), could your rebase 
> package be constructed around Johannes' rebase.exe implementation, so 
> that it could deliver a solution to this problem?

I'm not sure it IS a problem [1] but there's no need to use "Johannes'"
version; I'm already using a more recent version of the same upstream
source.  The real question is, can it be built as a native app, and
SHOULD it be built as a native app?

Downside: building as a native tool means that it wouldn't grok posix
paths -- which is what rebaseall feeds it [2].

[1] As a thought experiment, I wonder if your earlier troubles -- which
you solved by rebasing just msys-1.0.dll -- could ALSO have been solved
by rebasing *everything else* except msys-1.0.dll? [3] Not that such an
approach would be desired, when it's simpler(?) to just rebase one DLL.
(simpler, that is, until you start trying to use the same rebase.exe for
both "fix msys-1.0.dll" and "fix everything else via rebaseall" duties:
see [2]).

[2] rebaseall feeds to rebase a list of DLLs, but not on the command
line.  The list of to-be-rebased DLLs is put in a file, and that file is
passed to rebase via its -T option.  So, since rebaseall is running
under msys-dash, the path to the file-containing-the-list will be
automagically converted to win32, but the *contents* of that file list
will not be.  I think this breaks rebaseall (and peflagsall).

> Further, since in an earlier message in the same thread:
> http://thread.gmane.org/gmane.comp.gnu.mingw.msys/4772/focus=4726
> 
> you had suggested the tentative solution:
> 
>   $ rebase -b 0x30000000 /path/to/msys-1.0.dll

[3] Ah, well, if you read carefully I was suggesting TWO alternate
approaches:

a) use the (not yet uploaded at that time) msys-rebaseall + msys-dash to
rebase everything (except msys-1.0.dll), or

b) use the cygwin (e.g. non-msys) rebase.exe and rebase JUST
msys-1.0.dll. I didn't really think this would work, and was kinda
surprised that it did.

Since you didn't have access to msys-rebaseall/msys-dash, you did (b)
and it worked. But nobody who was having the problem tested (a).

> and since PRECISELY that worked for me, (and also for Chris, I 
> believe), would that particular base address not, perhaps be more 
> universally acceptable as default, than the current choice?  (I did 
> note your reasoning for there being no universally CORRECT address).

I dunno. Is there some accepted standard (<0x4000000 is .exe, >0x7000000
is system) or something?

Re: the rebase package, I suggest a compromise, based on the following
points:

1) rebaseall and peflagsall both explicitly set PATH=/bin, they will
always use the rebase.exe (peflags.exe) in msys' /bin directory -- even
if a native version exists in /mingw/bin.

2) I believe that neither rebaseall nor peflagsall will work
successfully with a native rebase.exe (peflags.exe).

3) An msys-rebase.exe (msys-peflags.exe) can't rebase (set flags on) the
msys-1.0.dll itself.

So, how about a separate mingw32-rebase package, based on the same
source code, but which provides only the executables and not the
scripts? I'll probably have to update the msys-rebase package's
documentation to mention "the other one", but that's not too hard.

--
Chuck

On 2010-05-26 23:32Z, Charles Wilson wrote:
> On 5/26/2010 8:56 AM, Keith Marshall wrote:\
>> you said:
>>> BTW, just as cygwin-rebase can't rebase cygwin1.dll, my msys port of
>>> rebase can't rebase msys-1.0.dll.
>>
>> I assume that remains true of the package you eventually released?
> 
> Correct. rebase.exe depends on msys.  Mostly, because in order to be
> really useful, you need rebaseall, which is a shell script.  So, that
> means you can't use rebaseall to rebase the shell interpreter (in this
> case, dash) nor the runtime used by dash, regardless of whether
> rebase.exe itself relies on that runtime.

Would it help to use a native shell to run that script?
Here's a fellow who has resurrected an old native port of zsh:
  http://zsh-nt.sourceforge.net/

On 5/26/2010 8:14 PM, Greg Chicares wrote:
> Would it help to use a native shell to run that script?
> Here's a fellow who has resurrected an old native port of zsh:
>   http://zsh-nt.sourceforge.net/

Errm...doubtful.  The script also uses sed, grep, sort etc.  By the time
you actually invoke rebase, they have all exited, but they need to be
available to the script. I wonder if the stdout/stdin pipes with MSYS
programs, marshalled by 'zsh-nt', would work as desired...

--
Chuck

Charles Wilson wrote:

>
>>> Maybe we need a pure win32 version...
>>

>
> I guess what you're really asking is, can our rebase.exe (and
> peflags.exe) be built as a native win32 app.  I think it can -- but see
> below.
>

Yea, I think that is what I've seen before, a rebase that is native MSVCRT.

>
> Errr...huh?  In my experience, it has *never* been the msys DLL that
> couldn't be loaded in the same location in the child as in the parent --
> mostly because it is the FIRST dependency to be loaded, so there's
> nothing with which it could conflict.  It's more likely to be "Cwd.dll"
> or "POSIX.dll" (some perl extension modules loaded almost last), or
> sometimes msys-intl-8.dll -- or any of the guile srfi dlls.
>

I've never had a problem either but their are those that have on the 
user list that the rebase of the MSYS dll has helped.  I think it comes 
down to which chip set brand is being used.

>> (and if you haven't already done so), could your rebase
>> package be constructed around Johannes' rebase.exe implementation, so
>> that it could deliver a solution to this problem?
>
> I'm not sure it IS a problem [1] but there's no need to use "Johannes'"
> version; I'm already using a more recent version of the same upstream
> source.  The real question is, can it be built as a native app, and
> SHOULD it be built as a native app?
>

And I think it can be a native app without issue.

> Downside: building as a native tool means that it wouldn't grok posix
> paths -- which is what rebaseall feeds it [2].
>

I'm not so sure that this would be an issue.

>
> [2] rebaseall feeds to rebase a list of DLLs, but not on the command
> line.  The list of to-be-rebased DLLs is put in a file, and that file is
> passed to rebase via its -T option.  So, since rebaseall is running
> under msys-dash, the path to the file-containing-the-list will be
> automagically converted to win32, but the *contents* of that file list
> will not be.  I think this breaks rebaseall (and peflagsall).
>

But rebaseall would read the list of dll from the file and spawn the 
native rebase so the path would change from POSIX to WIN32 before rebase 
sees the path.

>> Further, since in an earlier message in the same thread:
>> http://thread.gmane.org/gmane.comp.gnu.mingw.msys/4772/focus=4726
>>
>> you had suggested the tentative solution:
>>
>>    $ rebase -b 0x30000000 /path/to/msys-1.0.dll
>
> [3] Ah, well, if you read carefully I was suggesting TWO alternate
> approaches:
>
> a) use the (not yet uploaded at that time) msys-rebaseall + msys-dash to
> rebase everything (except msys-1.0.dll), or
>
> b) use the cygwin (e.g. non-msys) rebase.exe and rebase JUST
> msys-1.0.dll. I didn't really think this would work, and was kinda
> surprised that it did.
>
> Since you didn't have access to msys-rebaseall/msys-dash, you did (b)
> and it worked. But nobody who was having the problem tested (a).
>
>> and since PRECISELY that worked for me, (and also for Chris, I
>> believe), would that particular base address not, perhaps be more
>> universally acceptable as default, than the current choice?  (I did
>> note your reasoning for there being no universally CORRECT address).
>
> I dunno. Is there some accepted standard (<0x4000000 is .exe,>0x7000000
> is system) or something?
>

User space < 0x70000000 >= System space

> Re: the rebase package, I suggest a compromise, based on the following
> points:
>
> 1) rebaseall and peflagsall both explicitly set PATH=/bin, they will
> always use the rebase.exe (peflags.exe) in msys' /bin directory -- even
> if a native version exists in /mingw/bin.
>

I consider that a broken feature.  The assumption that the PATH should 
be set to /bin is just wrong and should be fixed.

> 2) I believe that neither rebaseall nor peflagsall will work
> successfully with a native rebase.exe (peflags.exe).
>

And I disagree as noted above.

> 3) An msys-rebase.exe (msys-peflags.exe) can't rebase (set flags on) the
> msys-1.0.dll itself.
>
> So, how about a separate mingw32-rebase package, based on the same
> source code, but which provides only the executables and not the
> scripts? I'll probably have to update the msys-rebase package's
> documentation to mention "the other one", but that's not too hard.
>

And fix rebaseall and peflagsall so that it isn't setting the PATH or at 
least provide a REBASE_PATH to be prepended to PATH.

Earnie

On 5/27/2010 7:49 AM, Earnie wrote:
> Charles Wilson wrote:
>> [2] rebaseall feeds to rebase a list of DLLs, but not on the command
>> line.  The list of to-be-rebased DLLs is put in a file, and that file is
>> passed to rebase via its -T option.  So, since rebaseall is running
>> under msys-dash, the path to the file-containing-the-list will be
>> automagically converted to win32, but the *contents* of that file list
>> will not be.  I think this breaks rebaseall (and peflagsall).
>>
> 
> But rebaseall would read the list of dll from the file and spawn the 
> native rebase so the path would change from POSIX to WIN32 before rebase 
> sees the path.

No, rebase is called only once, and given the path to the file that
contains the list of DLLs.  So, the path to the listfile would get
converted, and then *rebase* reads the contents.  So, MSYS never gets
the chance to convert the paths listed inside the listfile.

The reason for the behavior on the part of rebase (e.g. called once to
act on a list of dll's, instead of being called multiple times to
operate singly on each dll) is so that rebase can do all the
book-keeping.  It determines the memory footprint of each DLL, and
increments (decrements) the base address to use for the *next* DLL by
that amount.  Otherwise, rebase would have to somehow indicate to the
script what the DLL's footprint was, and the script would have to track
that and use different -b 0xNNNNNNNN options on each call.

>> I dunno. Is there some accepted standard (<0x4000000 is .exe,>0x7000000
>> is system) or something?
>>
> 
> User space < 0x70000000 >= System space

Thanks. See here: http://www.drdobbs.com/184416272

> Applications (.exe files) start at 0x00400000 for all compilers
> that I tested.
...
> The address range for an application that is not reserved by any
> version of Windows is from 0x00400000 to 0x80000000. The system DLLs
> for Windows are currently based in memory from 0x70000000 to
> 0x78000000 on the Intel processors and from 0x68000000 to 0x78000000
> on the MIPS processors. Other standard DLLs (for OLE support) are
> apparently in the range 0x50000000 to 0x5f000000. When selecting base
> addresses for DLLs, Microsoft suggests that you select them from the
> top of the allowed address range downwards, in order to avoid
> conflicts with memory allocated dynamically by the application (which
> is allocated from the bottom up).
...
> In conclusion, the most suitable address range for DLLs is from
> 0x60000000  through 0x6f000000. Microsoft, seeking portability where
> it cannot be achieved, proposes to reduce the range further to
> 0x60000000  through 0x68000000 in order to accommodate both Intel and
> MIPS processors. (Also note that Microsoft’s upper limit overlaps the
> reserved range of the MIPS processor.) Microsoft’s proposal continues
> with a “first letter” scheme for the selection of the base address,
> which I have summarized in Table 1.

(Unfortunatly, the link to "Table 1" is broken)

>> 1) rebaseall and peflagsall both explicitly set PATH=/bin, they will
>> always use the rebase.exe (peflags.exe) in msys' /bin directory -- even
>> if a native version exists in /mingw/bin.
>>
> 
> I consider that a broken feature.  The assumption that the PATH should 
> be set to /bin is just wrong and should be fixed.

So, you WANT rebaseall, a particular script provided by a particular
package, which expects specific behavior and options from an executable
that is provided by *that same package*, to use whatever random
executable that happens to have the same name which it might find in the
PATH?  Especially when that PATH is *not* the normal path most MSYS
users are familiar with after they launch a "normal" shell via msys.bat?

I'm sorry, Earnie, but that's...not wise. It's ASKING for trouble.

The rebase.exe provided by the Microsoft SDK doesn't accept the -T
option. rebaseall uses it. That's one reason why PATH is explicitly set
to /bin. Also, remember that the recommended sequence is:

launch a dash shell
  * dash doesn't read the bash startup files, so your PATH is
    probably not what you're used to, in "regular" bash. Plus,
    you haven't run msys.bat, so all the PATH shenanigans it performs
    don't happen either.
  * In my case -- and that of many cygwin users -- I put neither
    the cygwin bin/ nor the MinGW/MSYS bin/ in my overall Windows
    PATH.  I rely on the .startup-files to do that.

run rebaseall
  * which needs to "find" sed, grep, gawk -- and rebase.exe.  Since the
    PATH is in an unusual state, rebaseall explicitly sets it to /bin.
    I don't see much difference between setting PATH=/bin and explicitly
    invoking every tool as /bin/foo.

>> 2) I believe that neither rebaseall nor peflagsall will work
>> successfully with a native rebase.exe (peflags.exe).
>>
> 
> And I disagree as noted above.

Because you misunderstand the situation.  rebase is NOT called in a
loop, nor are the DLL paths themselves listed on the command line -- so
MSYS doesn't get to fixup each individual path to the DLLs to be
rebased. The only path that appears in the command to rebase is the path
to the file containing the list -- so that's the only path that gets
fixed up; NOT the paths contained inside that listfile.

Now, both the Microsoft and Tishler versions of rebase CAN be called
with a list of DLLs passed explicitly on the command line.  In this
case, with a native version of rebase invoked within an msys-dash shell,
since each DLL pathname appears on the command line then MSYS would get
a chance to do its thing.

However, rebaseall doesn't do it that way, for two reasons: (1) command
line length limitations.  It's real easy to create a command line that's
too long, when you're listing every .dll in an entire installation. (2)
quoting issues. If the pathnames appeared on the command line, they'd
all need to be escaped/quoted to protect against shell special
characters and word splitting.

>> 3) An msys-rebase.exe (msys-peflags.exe) can't rebase (set flags on) the
>> msys-1.0.dll itself.
>>
>> So, how about a separate mingw32-rebase package, based on the same
>> source code, but which provides only the executables and not the
>> scripts? I'll probably have to update the msys-rebase package's
>> documentation to mention "the other one", but that's not too hard.
>>
> 
> And fix rebaseall and peflagsall so that it isn't setting the PATH or at 
> least provide a REBASE_PATH to be prepended to PATH.

Look at the code in rebaseall first, and then tell me if you still think
it'd be just great to use a native rebase with rebaseall. If so, then
I'll add REBASE_PATH support.

Oddly, however, nobody in six years on cygwin has ever found it
necessary to use *some other rebase* than the one *shipped with the
rebaseall* script.

--
Chuck

On Thursday 27 May 2010 00:32:53 Charles Wilson wrote:
> I guess what you're really asking is, can our rebase.exe (and
> peflags.exe) be built as a native win32 app.

Yes.

> > Given that Chris Sutcliffe's
> > original problem, (which I myself encountered, identically, when
> > I upgraded to the current msys-1.0.dll release), entails exactly
> > that requirement, (that msys-1.0.dll must be rebased), will your
> > rebase package suffice to address the issue?  (FWIW, msys-1.0.dll
> > is the ONLY shared object which I have needed to rebase to date).
>
> Well, for some reason I had always thought that cygwin1.dll itself
> was not "rebasable" -- that, even if you used some native tool to
> change its ImageBaseAddress, it would die horribly for some
> esoteric reason (hardcoded pointer values? dunno...).

I don't know either; you likely know better than I.

> So, I've never attempted to rebase either cygwin1.dll or
> msys-1.0.dll.

Yet you did recommend doing just that, when Chris originally reported 
the problem on the MinGW-MSYS list...

> I did find one message on the cygwin list from last 
> December, recommending to use rebase(all) to rebase *a copy* of
> cygwin1.dll, and then use native tools to move it into place.  This
> had something to do with trying to outsmart a BLODA.
>
> > Since msys-1.0.dll itself seems to be most consistently reported
> > as problematic,
>
> Errr...huh?

Well, more accurately, it has been frequently reported, (not 
necessarily on our lists), that an MSYS shell cannot be started AT 
ALL; right from the outset:

  d:\path\to\msys\bin\sh.exe --login -i

dies with a "cannot reserve space for cygwin's heap" message.

> In my experience, it has *never* been the msys DLL 
> that couldn't be loaded in the same location in the child as in the
> parent -- mostly because it is the FIRST dependency to be loaded,
> so there's nothing with which it could conflict.

Well, the message above is distinct from the diagnostic you describe, 
as indicative of this failure mode; of course, that in no way implies 
that the two do not have a common underlying cause.  However, at the 
point where this error occurs, there would appear to be little more 
than msys-1.0.dll and bash.exe involved.  (This was before you had 
made msys-rebase available; I've no idea if it would have even been 
possible to start dash.exe, rather than bash.exe, at this point).

> It's more likely 
> to be "Cwd.dll" or "POSIX.dll" (some perl extension modules loaded
> almost last), or sometimes msys-intl-8.dll -- or any of the guile
> srfi dlls.

Hmm.  In my case, it's none of those; at the time when I encountered 
this problem I had installed ONLY Console2, msysCORE, msys-coreutils 
and msys-bash, so the only likely candidates, besides msys-1.0.dll 
itself, would be those DLLs distributed with Console-2.00.144; there 
were no other MSYS related DLLs even present, on any file system 
attached to the box.  (I didn't try it, at the time, but it may be 
instructive to repeat the exercise with Console2 removed from the 
equation).

-- 
Regards,
Keith.

Charles Wilson wrote:
> On 5/27/2010 7:49 AM, Earnie wrote:
>> Charles Wilson wrote:
>>> [2] rebaseall feeds to rebase a list of DLLs, but not on the command
>>> line.  The list of to-be-rebased DLLs is put in a file, and that file is
>>> passed to rebase via its -T option.  So, since rebaseall is running
>>> under msys-dash, the path to the file-containing-the-list will be
>>> automagically converted to win32, but the *contents* of that file list
>>> will not be.  I think this breaks rebaseall (and peflagsall).
>>>
>>
>> But rebaseall would read the list of dll from the file and spawn the
>> native rebase so the path would change from POSIX to WIN32 before rebase
>> sees the path.
>
> No, rebase is called only once, and given the path to the file that
> contains the list of DLLs.  So, the path to the listfile would get
> converted, and then *rebase* reads the contents.  So, MSYS never gets
> the chance to convert the paths listed inside the listfile.
>

Ah, yea, well in that scenario you are correct.

> The reason for the behavior on the part of rebase (e.g. called once to
> act on a list of dll's, instead of being called multiple times to
> operate singly on each dll) is so that rebase can do all the
> book-keeping.  It determines the memory footprint of each DLL, and
> increments (decrements) the base address to use for the *next* DLL by
> that amount.  Otherwise, rebase would have to somehow indicate to the
> script what the DLL's footprint was, and the script would have to track
> that and use different -b 0xNNNNNNNN options on each call.
>

Good to know.

>>> I dunno. Is there some accepted standard (<0x4000000 is .exe,>0x7000000
>>> is system) or something?
>>>
>>
>> User space<  0x70000000>= System space
>
> Thanks. See here: http://www.drdobbs.com/184416272
>

Ah, well the MIPS wasn't documented at the time I adjusted the DLL to 
0x70000000.  So based on that information the formula is more.

if (CPU_SET == MIPS)
then
   User space < 0x68000000 >= System space
else
   User space < 0x70000000 >= System space
endif

>> In conclusion, the most suitable address range for DLLs is from
>> 0x60000000  through 0x6f000000. Microsoft, seeking portability where
>> it cannot be achieved, proposes to reduce the range further to
>> 0x60000000  through 0x68000000 in order to accommodate both Intel and
>> MIPS processors. (Also note that Microsoft’s upper limit overlaps the
>> reserved range of the MIPS processor.) Microsoft’s proposal continues
>> with a “first letter” scheme for the selection of the base address,
>> which I have summarized in Table 1.

Which is probably why Cygwin used 0x60000000.

>>> 1) rebaseall and peflagsall both explicitly set PATH=/bin, they will
>>> always use the rebase.exe (peflags.exe) in msys' /bin directory -- even
>>> if a native version exists in /mingw/bin.
>>>
>>
>> I consider that a broken feature.  The assumption that the PATH should
>> be set to /bin is just wrong and should be fixed.
>
> So, you WANT rebaseall, a particular script provided by a particular
> package, which expects specific behavior and options from an executable
> that is provided by *that same package*, to use whatever random
> executable that happens to have the same name which it might find in the
> PATH?  Especially when that PATH is *not* the normal path most MSYS
> users are familiar with after they launch a "normal" shell via msys.bat?
>

But if I didn't install to / the rebase executable wouldn't be found in 
/bin anyway.

> I'm sorry, Earnie, but that's...not wise. It's ASKING for trouble.

In either scenario there are issues and assuming /bin isn't correct 
either.  Perhaps a better scenario would be to set PATH to the directory 
name of argv[0].

>
> The rebase.exe provided by the Microsoft SDK doesn't accept the -T
> option. rebaseall uses it. That's one reason why PATH is explicitly set
> to /bin. Also, remember that the recommended sequence is:
>

Ok, an understandable reason but I still think there are better ways to 
accomplish the goal.

> launch a dash shell
>    * dash doesn't read the bash startup files, so your PATH is
>      probably not what you're used to, in "regular" bash. Plus,
>      you haven't run msys.bat, so all the PATH shenanigans it performs
>      don't happen either.

So it doesn't read /etc/profile or ~/.profile?  I knew there was some 
reason I used bash instead of the default cygwin shell.

>    * In my case -- and that of many cygwin users -- I put neither
>      the cygwin bin/ nor the MinGW/MSYS bin/ in my overall Windows
>      PATH.  I rely on the .startup-files to do that.
>

Yea, that isn't recommended by me either.

> run rebaseall
>    * which needs to "find" sed, grep, gawk -- and rebase.exe.  Since the
>      PATH is in an unusual state, rebaseall explicitly sets it to /bin.
>      I don't see much difference between setting PATH=/bin and explicitly
>      invoking every tool as /bin/foo.

If the tool is installed at / yes, but if the tool is installed in 
/usr/local it will be an issue.

>
>>> 2) I believe that neither rebaseall nor peflagsall will work
>>> successfully with a native rebase.exe (peflags.exe).
>>>
>>
>> And I disagree as noted above.
>
> Because you misunderstand the situation.  rebase is NOT called in a
> loop, nor are the DLL paths themselves listed on the command line -- so
> MSYS doesn't get to fixup each individual path to the DLLs to be
> rebased. The only path that appears in the command to rebase is the path
> to the file containing the list -- so that's the only path that gets
> fixed up; NOT the paths contained inside that listfile.
>

Yes, true, I misunderstood.  I thought you meant that rebaseall was 
reading the file but it is passing the file to rebase.

> Now, both the Microsoft and Tishler versions of rebase CAN be called
> with a list of DLLs passed explicitly on the command line.  In this
> case, with a native version of rebase invoked within an msys-dash shell,
> since each DLL pathname appears on the command line then MSYS would get
> a chance to do its thing.
>

Yes.

> However, rebaseall doesn't do it that way, for two reasons: (1) command
> line length limitations.  It's real easy to create a command line that's
> too long, when you're listing every .dll in an entire installation. (2)
> quoting issues. If the pathnames appeared on the command line, they'd
> all need to be escaped/quoted to protect against shell special
> characters and word splitting.
>

Yea, that command line limit would be bad.

>>> 3) An msys-rebase.exe (msys-peflags.exe) can't rebase (set flags on) the
>>> msys-1.0.dll itself.
>>>
>>> So, how about a separate mingw32-rebase package, based on the same
>>> source code, but which provides only the executables and not the
>>> scripts? I'll probably have to update the msys-rebase package's
>>> documentation to mention "the other one", but that's not too hard.
>>>
>>
>> And fix rebaseall and peflagsall so that it isn't setting the PATH or at
>> least provide a REBASE_PATH to be prepended to PATH.
>
> Look at the code in rebaseall first, and then tell me if you still think
> it'd be just great to use a native rebase with rebaseall. If so, then
> I'll add REBASE_PATH support.
>
> Oddly, however, nobody in six years on cygwin has ever found it
> necessary to use *some other rebase* than the one *shipped with the
> rebaseall* script.
>

Ok, I'll take a look.  Maybe because most users would use the setup.exe 
program to install it and it would end up in the /bin directory is the 
reason no one has complained.

Earnie

On 2010-05-27 15:03Z, Charles Wilson wrote:
> On 5/27/2010 7:49 AM, Earnie wrote:
>> Charles Wilson wrote:
> 
>>> I dunno. Is there some accepted standard (<0x4000000 is .exe,>0x7000000
>>> is system) or something?
>> 
>> User space < 0x70000000 >= System space
> 
> Thanks. See here: http://www.drdobbs.com/184416272
[...]
>> Microsoft’s proposal continues
>> with a “first letter” scheme for the selection of the base address,
>> which I have summarized in Table 1.
>
> (Unfortunatly, the link to "Table 1" is broken)

Here's Table 1:
  http://i.cmpnet.com/ddj/windevnet/images/wdj0012b/0012bt1.gif
[I clicked "Print" on the menu right above the article's name in
the URL you gave--usually that helps with bad links.] The table
here seems to be the same thing:

http://www.gidforums.com/t-994.html

One possible scheme is to choose a base address based on the first
letter of the DLL name.
First letter Base address
A - C 0x60000000
D - F 0x61000000
G - I 0x62000000
J - L 0x63000000
M - O 0x64000000
P - R 0x65000000
S - U 0x66000000
V - X 0x67000000
Y - Z 0x68000000

On 5/27/2010 9:49 PM, Earnie wrote:
> Charles Wilson wrote:
>> So, you WANT rebaseall, a particular script provided by a particular
>> package, which expects specific behavior and options from an executable
>> that is provided by *that same package*, to use whatever random
>> executable that happens to have the same name which it might find in the
>> PATH?  Especially when that PATH is *not* the normal path most MSYS
>> users are familiar with after they launch a "normal" shell via msys.bat?
>>
> 
> But if I didn't install to / the rebase executable wouldn't be found in 
> /bin anyway.

Wait.  (a) Why wouldn't you, and (b) are the binary *msys* packages
uploaded to sourceware expected to work properly if they are NOT
installed in the expected directory?

I know that the msys-autotool packages will have a very hard time if
they are not installed into MSYS "/" -- since they will look explicitly
for their share/autoconf/*, share/automakeN.NN/*, and share/aclocal/
files under / == /usr.

Even the mingw-autotool packages will have trouble if they are installed
somewhere other than /mingw ('course, it doesn't matter where that
/mingw is physically located: C:\MinGW or D:\my-backup-mingw\)

I know that msys executables are no longer *required* to be installed
into the /bin where msys-1.0.dll lives, but...even on linux, if you
download an RPM that is supposed to be installed in /usr, and somehow
manage to install it in /usr/local -- you'll be very lucky if it works
at all.  If you want to install an RPM somewhere else, you're expected
to download the SRPM and rebuild.

>> I'm sorry, Earnie, but that's...not wise. It's ASKING for trouble.
> 
> In either scenario there are issues and assuming /bin isn't correct 
> either.  Perhaps a better scenario would be to set PATH to the directory 
> name of argv[0].

Suppose instead of ACTUALLY hardcoding PATH=/bin, the rebaseall script
were, in the source package, named rebaseall.in, and contained
"PATH=@bindir@".  That what, when rebuilt with a different --prefix, it
would work.

Your idea of "use the directory of $0" would be ok, too.

But...you're arguing about the way an upstream application is coded --
it's not as if I wrote rebaseall, or was proposing a patch for
msys-1.0.dll itself.

While we can certainly patch the rebase package for MSYS's purposes --
and I *have*, simply because I wanted to use "dash" instead of "ash",
since the former is more actively maintained upstream -- this seems to
be a departure from the way most MSYS packages are treated. I've just
been describing how the pre-existing, upstream package operates, and why
it does so...not how my spiffy brand-new written-just-for-msys tool is
coded.

>> The rebase.exe provided by the Microsoft SDK doesn't accept the -T
>> option. rebaseall uses it. That's one reason why PATH is explicitly set
>> to /bin. Also, remember that the recommended sequence is:
> 
> Ok, an understandable reason but I still think there are better ways to 
> accomplish the goal.

Again, this is an upstream behavior. Usually we don't *require* rewrites
of upstream code, unless explicitly *buggy* on msys. That isn't the case
here: if I installed rebase on cygwin into /usr/local, it'd not work
there either. (SO, unless there is a *requirement* that rebaseall work
if installed somewhere other than / == /usr, your suggestions are "gee
it'd be nice if", not "this is broken because".)

>> launch a dash shell
>>    * dash doesn't read the bash startup files, so your PATH is
>>      probably not what you're used to, in "regular" bash. Plus,
>>      you haven't run msys.bat, so all the PATH shenanigans it performs
>>      don't happen either.
> 
> So it doesn't read /etc/profile or ~/.profile?  I knew there was some 
> reason I used bash instead of the default cygwin shell.

dash reads both /etc/profile and ~/.profile for *login* shells. It
always reads the file whose path is specified in the environment
variable ENV.  (So, if no ENV is set and dash is NOT started as a login
shell, then *no* startup files are read at all).  However, so if
rebase's README said 'start dash as a login shell: dash -l' then that'd
be one thing, but it doesn't. It says:

    1. shutdown all MSYS processes
    2. start dash (do not use bash or rxvt)
    3. execute /bin/rebaseall (in the dash window)

However, I'm not actually sure that's in error.  Many times, people's
login scripts start resident programs; mine launches ssh-agent -- which
will break rebaseall because it expects, and verifies, that NO msys
processes are running except for dash itself.

> If the tool is installed at / yes, but if the tool is installed in 
> /usr/local it will be an issue.

So, two questions: (1) is it *required* that pre-compiled MSYS packages
must operate correctly if installed somewhere other than their
compiled-in --prefix? (2) is it *required* that packages which we
provide pre-compiled for MSYS, when RE-compiled from source and
installed using a different --prefix must operate correctly?

Right now, rebase violates both. Personally, I think #1 is a fool's
game.  Now, #2...is another question. Most well-behavior packages obey
it, but the upstream rebase package does not.  It can be patched to do
so, of course -- and THAT patch might actually be accepted upstream
(most of the other MSYS mods -- e.g. explicitly using dash rather than
ash -- wouldn't).

As far as a proposed patch, I could see either

PATH=${REBASE_PATH:-@...@}

and creating the real rebaseall script from rebaseall.in at configure, or

tp1=${0%/*}
tp2=${tp1:-.}
PATH=$(cd $tp2 && pwd)

to always use rebase.exe from the same directory in which rebaseall
itself is found. This would facilitate testing an uninstalled rebaseall
(*) -- except that it would not be able to "find" sed, grep, sort..., so
we'd need

PATH=$(cd $tp2 && pwd):@bindir@

anyway. (If somebody didn't install *our* sed, grep, and sort -bin
packages into /, the God help 'em).

(*) You still have to shutdown all other MSYS applications, exit your
normal shell, launch a new dash shell, then explicitly invoke
/path/to/my/builddir/rebaseall...so this isn't really all THAT helpful.

>> Oddly, however, nobody in six years on cygwin has ever found it
>> necessary to use *some other rebase* than the one *shipped with the
>> rebaseall* script.
> 
> Ok, I'll take a look.  Maybe because most users would use the setup.exe 
> program to install it and it would end up in the /bin directory is the 
> reason no one has complained.

Sure. Cygwin distributes precompiled binaries in a "install as directed,
or don't blame us, WJM" fashion.  Traditionally, *msys* packages are
provided the same way; in the days of msys-dtk-1.0.1 + updates, nobody
would've expected the coreutils or perl update "overlay" to work
correctly if it were unpacked in /usr/local/ rather than /.  It MIGHT
have worked -- but if it didn't the answer would have been "Don't Do That."

The core MinGW packages like gcc, binutils, etc, of course, are
different (I exclude the mingw autotool packages, for reasons that now
resemble a dead horse).

--
Chuck

On Wed, May 26, 2010 at 7:32 PM, Charles Wilson
<cwilso11@...> wrote:
> On 5/26/2010 8:56 AM, Keith Marshall wrote:\
>> you said:
>>> BTW, just as cygwin-rebase can't rebase cygwin1.dll, my msys port of
>>> rebase can't rebase msys-1.0.dll.
>>
>> I assume that remains true of the package you eventually released?
>
> Correct. rebase.exe depends on msys.  Mostly, because in order to be

Forgive my ignorance, but why can't you just compile rebase
statically?  Then it won't depend on the msys dll.

On 5/28/2010 11:35 AM, NightStrike wrote:
>> Correct. rebase.exe depends on msys.  Mostly, because in order to be
> 
> Forgive my ignorance, but why can't you just compile rebase
> statically?  Then it won't depend on the msys dll.

There is no way to link any executable statically to the msys (or
cygwin) runtime.  That is, if you need posix support, then you're gonna
use msys-1.0.dll or cygwin1.dll.  So, there really isn't a beast called
"a static msys executable".  You either have posix support and a DLL
dependency, or you compile as a native application.  Usually, if
somebody says "compile (msys/cygwin) foo.exe statically" they mean "link
statically to all msys/cygwin libraries EXCEPT msys-1.0.dll/cygwin1.dll
and any necessary windows system DLLs".

However, even for "native" apps you'd still end up linking against some
win32 system DLLs.  There really are very few programs on windows that
can be considered truly "statically linked" -- and I don't know exactly
what they'd be able to DO, without access to windows services via those
system DLLs.

--
Chuck

On 5/27/2010 6:31 AM, Keith Marshall wrote:
> On Thursday 27 May 2010 00:32:53 Charles Wilson wrote:
>> I guess what you're really asking is, can our rebase.exe (and
>> peflags.exe) be built as a native win32 app.
> 
> Yes.

It can. I've created

rebase-3.0.1_1-1-mingw32-bin.tar.lzma
rebase-3.0.1_1-1-mingw32-doc.tar.lzma
rebase-3.0.1_1-1-mingw32-lic.tar.lzma
rebase-3.0.1_1-1-mingw32-src.tar.lzma

where the -bin package contains only rebase.exe and peflags.exe, built
as native apps.  Since rebase.exe includes C++ code, I linked it using
-static-libgcc so that there would be no deps on any DLLs (other than
w32 system ones).

>> Well, for some reason I had always thought that cygwin1.dll itself
>> was not "rebasable" -- that, even if you used some native tool to
>> change its ImageBaseAddress, it would die horribly for some
>> esoteric reason (hardcoded pointer values? dunno...).
> 
> I don't know either; you likely know better than I.

Well, obviously I was wrong, tho -- since it actually worked (for you,
on the msys "version" of cygwin; and for the correspondent who
originally suggested, over on the cygwin lists, trying it on "real" cygwin.

>> So, I've never attempted to rebase either cygwin1.dll or
>> msys-1.0.dll.
> 
> Yet you did recommend doing just that, when Chris originally reported 
> the problem on the MinGW-MSYS list...

Right, because I saw that recommendation on the cygwin lists.  It never
would have occured to *me*, independently of that recommendation, to do
so -- since I had this mistaken notion about cygwin being non-rebasable.

>> Errr...huh?
> 
> Well, more accurately, it has been frequently reported, (not 
> necessarily on our lists), that an MSYS shell cannot be started AT 
> ALL; right from the outset:
> 
>   d:\path\to\msys\bin\sh.exe --login -i
> 
> dies with a "cannot reserve space for cygwin's heap" message.
...
> Well, the message above is distinct from the diagnostic you describe, 
> as indicative of this failure mode; of course, that in no way implies 
> that the two do not have a common underlying cause.

Hmm. yeah, that actually does sound more like a problem with either (a)
msys itself being loaded in "the wrong place", or (b) some other (win32
system?) dll being loaded "too close" to msys dll, since msys expects
its heap start address to be "just above" the dll itself. And, of
course, in the same location in both parent and child.  IIRC.

>> It's more likely 
>> to be "Cwd.dll" or "POSIX.dll" (some perl extension modules loaded
>> almost last), or sometimes msys-intl-8.dll -- or any of the guile
>> srfi dlls.
> 
> Hmm.  In my case, it's none of those; at the time when I encountered 
> this problem I had installed ONLY Console2, msysCORE, msys-coreutils 
> and msys-bash, so the only likely candidates, besides msys-1.0.dll 
> itself, would be those DLLs distributed with Console-2.00.144; there 
> were no other MSYS related DLLs even present, on any file system 
> attached to the box.  (I didn't try it, at the time, but it may be 
> instructive to repeat the exercise with Console2 removed from the 
> equation).

No, I think you're right. There doesn't seem to be much else loaded in
memory at that stage to cause the problem.

--
Chuck

One possible scheme is to choose a base address based on the first
letter of the DLL name.
First letter Base address
A - C 0x60000000
D - F 0x61000000
G - I 0x62000000
J - L 0x63000000
M - O 0x64000000
P - R 0x65000000
S - U 0x66000000
V - X 0x67000000
Y - Z 0x68000000


No need to reinvent the wheel here.  The --enable-auto-image-base that Mumut 
Khan contributed to binutils does it even better:

/* Use the output file to create a image base for relocatable DLLs.  */

static unsigned long
compute_dll_image_base (const char *ofile)
{
  unsigned long hash = strhash (ofile);
  return 0x61300000 + ((hash << 16) & 0x0FFC0000);
}
 where strhash is a function that hashes the dll filename .

Now the number is 0x61300000  rather than 0x60000000, because  0x61000000 is 
cygwin1-dll base address.

For reference, search  the cygwin-apps archives around June/July 2005 for a 
discussion or rebase  problems.

Danny

Charles Wilson wrote:
> On 5/27/2010 9:49 PM, Earnie wrote:
>> Charles Wilson wrote:
>>> So, you WANT rebaseall, a particular script provided by a particular
>>> package, which expects specific behavior and options from an executable
>>> that is provided by *that same package*, to use whatever random
>>> executable that happens to have the same name which it might find in the
>>> PATH?  Especially when that PATH is *not* the normal path most MSYS
>>> users are familiar with after they launch a "normal" shell via msys.bat?
>>>
>>
>> But if I didn't install to / the rebase executable wouldn't be found in
>> /bin anyway.
>
> Wait.  (a) Why wouldn't you, and (b) are the binary *msys* packages
> uploaded to sourceware expected to work properly if they are NOT
> installed in the expected directory?
>

Yes, they should work properly if not installed in the expected 
directory.  I know there are a few that expect to find their files by 
default in /usr/share but those are exceptions easily worked around.

> I know that the msys-autotool packages will have a very hard time if
> they are not installed into MSYS "/" -- since they will look explicitly
> for their share/autoconf/*, share/automakeN.NN/*, and share/aclocal/
> files under / == /usr.
>

IIRC there is an environment variable that can be set to control where 
these are located.

> Even the mingw-autotool packages will have trouble if they are installed
> somewhere other than /mingw ('course, it doesn't matter where that
> /mingw is physically located: C:\MinGW or D:\my-backup-mingw\)
>

Since autotool is mostly scripts I would not have thought to have a 
autotool-mingw package installed.  But again IIRC there is an 
environment variable to control where to look.

> I know that msys executables are no longer *required* to be installed
> into the /bin where msys-1.0.dll lives, but...even on linux, if you
> download an RPM that is supposed to be installed in /usr, and somehow
> manage to install it in /usr/local -- you'll be very lucky if it works
> at all.  If you want to install an RPM somewhere else, you're expected
> to download the SRPM and rebuild.
>

Ack, auto-packaging.  But I am very much a believer in build it your 
self which tends to overcome the obstacles of prepackaged binary 
releases.  I have though come to trust apt-get enough to use it on my 
Ubuntu installations.  I never liked RPM syntax enough to learn it.

>>> I'm sorry, Earnie, but that's...not wise. It's ASKING for trouble.
>>
>> In either scenario there are issues and assuming /bin isn't correct
>> either.  Perhaps a better scenario would be to set PATH to the directory
>> name of argv[0].
>
> Suppose instead of ACTUALLY hardcoding PATH=/bin, the rebaseall script
> were, in the source package, named rebaseall.in, and contained
> "PATH=@bindir@".  That what, when rebuilt with a different --prefix, it
> would work.
>

Yes, this would work up to a point.  That point being one of ``make 
install prefix=/my/depot/release/directory''.  I loathe DESTDIR and 
changing prefix at install time is not supposed to disturb the build 
process.

> Your idea of "use the directory of $0" would be ok, too.
>

Isn't this more like what other distributed package scripts do?

> But...you're arguing about the way an upstream application is coded --
> it's not as if I wrote rebaseall, or was proposing a patch for
> msys-1.0.dll itself.
>

Ack.

> While we can certainly patch the rebase package for MSYS's purposes --
> and I *have*, simply because I wanted to use "dash" instead of "ash",
> since the former is more actively maintained upstream -- this seems to
> be a departure from the way most MSYS packages are treated. I've just
> been describing how the pre-existing, upstream package operates, and why
> it does so...not how my spiffy brand-new written-just-for-msys tool is
> coded.
>

Ack.  I don't expect you to change it unless you had written it. :)

>>> The rebase.exe provided by the Microsoft SDK doesn't accept the -T
>>> option. rebaseall uses it. That's one reason why PATH is explicitly set
>>> to /bin. Also, remember that the recommended sequence is:
>>
>> Ok, an understandable reason but I still think there are better ways to
>> accomplish the goal.
>
> Again, this is an upstream behavior. Usually we don't *require* rewrites
> of upstream code, unless explicitly *buggy* on msys. That isn't the case
> here: if I installed rebase on cygwin into /usr/local, it'd not work
> there either. (SO, unless there is a *requirement* that rebaseall work
> if installed somewhere other than / == /usr, your suggestions are "gee
> it'd be nice if", not "this is broken because".)
>

Ack.

>>> launch a dash shell
>>>     * dash doesn't read the bash startup files, so your PATH is
>>>       probably not what you're used to, in "regular" bash. Plus,
>>>       you haven't run msys.bat, so all the PATH shenanigans it performs
>>>       don't happen either.
>>
>> So it doesn't read /etc/profile or ~/.profile?  I knew there was some
>> reason I used bash instead of the default cygwin shell.
>
> dash reads both /etc/profile and ~/.profile for *login* shells. It
> always reads the file whose path is specified in the environment
> variable ENV.  (So, if no ENV is set and dash is NOT started as a login
> shell, then *no* startup files are read at all).  However, so if
> rebase's README said 'start dash as a login shell: dash -l' then that'd
> be one thing, but it doesn't. It says:
>
>      1. shutdown all MSYS processes
>      2. start dash (do not use bash or rxvt)
>      3. execute /bin/rebaseall (in the dash window)
>
> However, I'm not actually sure that's in error.  Many times, people's
> login scripts start resident programs; mine launches ssh-agent -- which
> will break rebaseall because it expects, and verifies, that NO msys
> processes are running except for dash itself.
>

Ack.

>> If the tool is installed at / yes, but if the tool is installed in
>> /usr/local it will be an issue.
>
> So, two questions: (1) is it *required* that pre-compiled MSYS packages
> must operate correctly if installed somewhere other than their
> compiled-in --prefix? (2) is it *required* that packages which we
> provide pre-compiled for MSYS, when RE-compiled from source and
> installed using a different --prefix must operate correctly?

1) *required*, no.
2) *required*, yes.  This should be true regardless if it is for MSYS or 
some other system especially if it is GPL since the GNU Coding Standards 
should have been followed.

>
> Right now, rebase violates both. Personally, I think #1 is a fool's
> game.  Now, #2...is another question. Most well-behavior packages obey
> it, but the upstream rebase package does not.  It can be patched to do
> so, of course -- and THAT patch might actually be accepted upstream
> (most of the other MSYS mods -- e.g. explicitly using dash rather than
> ash -- wouldn't).
>

So, the upstream package needs a fix.

> As far as a proposed patch, I could see either
>
> PATH=${REBASE_PATH:-@...@}
>
> and creating the real rebaseall script from rebaseall.in at configure, or
>
> tp1=${0%/*}
> tp2=${tp1:-.}
> PATH=$(cd $tp2&&  pwd)
>
> to always use rebase.exe from the same directory in which rebaseall
> itself is found. This would facilitate testing an uninstalled rebaseall
> (*) -- except that it would not be able to "find" sed, grep, sort..., so
> we'd need
>
> PATH=$(cd $tp2&&  pwd):@bindir@
>

PATH=$(cd $tp2&&  pwd):@bindir@...

> anyway. (If somebody didn't install *our* sed, grep, and sort -bin
> packages into /, the God help 'em).
>

Ack.  That goes with what is stated in the README.

> (*) You still have to shutdown all other MSYS applications, exit your
> normal shell, launch a new dash shell, then explicitly invoke
> /path/to/my/builddir/rebaseall...so this isn't really all THAT helpful.
>
>>> Oddly, however, nobody in six years on cygwin has ever found it
>>> necessary to use *some other rebase* than the one *shipped with the
>>> rebaseall* script.
>>
>> Ok, I'll take a look.  Maybe because most users would use the setup.exe
>> program to install it and it would end up in the /bin directory is the
>> reason no one has complained.
>
> Sure. Cygwin distributes precompiled binaries in a "install as directed,
> or don't blame us, WJM" fashion.  Traditionally, *msys* packages are
> provided the same way; in the days of msys-dtk-1.0.1 + updates, nobody
> would've expected the coreutils or perl update "overlay" to work
> correctly if it were unpacked in /usr/local/ rather than /.  It MIGHT
> have worked -- but if it didn't the answer would have been "Don't Do That."
>

Ack.

> The core MinGW packages like gcc, binutils, etc, of course, are
> different (I exclude the mingw autotool packages, for reasons that now
> resemble a dead horse).
>

Probably smell like one too. ;p

--
Earnie

Earnie wrote:
>>
>> Wait.  (a) Why wouldn't you, and (b) are the binary *msys* packages
>> uploaded to sourceware expected to work properly if they are NOT
>> installed in the expected directory?
>>
>
> Yes, they should work properly if not installed in the expected
> directory.  I know there are a few that expect to find their files by
> default in /usr/share but those are exceptions easily worked around.
>

I should have instead said:

An expectation of the user is that it works properly.  This is Windows 
and the idiots abound.  I, the user, should be able to install anywhere 
I please, even though I've been warned not to.

Earnie

On 6/1/2010 8:48 AM, Earnie wrote:
> Charles Wilson wrote
>> I know that the msys-autotool packages will have a very hard time if
>> they are not installed into MSYS "/" -- since they will look explicitly
>> for their share/autoconf/*, share/automakeN.NN/*, and share/aclocal/
>> files under / == /usr.
> 
> IIRC there is an environment variable that can be set to control where 
> these are located.

And if somebody is knowledgeable enough to know about that, they can
certainly figure out how to install the msys-autotool packages somewhere
other than /.  But I don't think this list should be responsible for
supporting them, OR for making undue accomodations for that.  Frankly, I
don't have the time.

Use them as provided, installed as directed, or don't come crying here.

>> I know that msys executables are no longer *required* to be installed
>> into the /bin where msys-1.0.dll lives, but...even on linux, if you
>> download an RPM that is supposed to be installed in /usr, and somehow
>> manage to install it in /usr/local -- you'll be very lucky if it works
>> at all.  If you want to install an RPM somewhere else, you're expected
>> to download the SRPM and rebuild.
>>
> 
> Ack, auto-packaging.  But I am very much a believer in build it your 
> self which tends to overcome the obstacles of prepackaged binary 
> releases.  I have though come to trust apt-get enough to use it on my 
> Ubuntu installations.  I never liked RPM syntax enough to learn it.

Yes, your admiration for the gentoo model was obvious from the day
mingwPORT was devised. I hate it; I want to use my computer to get work
done -- not to fuss around incessantly redoing the same crap somebody
else should have already done.

If I want to install MSYS onto three computers, I don't want to waste
time recompiling it all three times.  Nor do I want to recompile it all
even once, and then do all the work the distributor SHOULD have done,
and create my own installation image for the other two computers.

Or, worse, expect MSYS novices on my development team to do any of the
above.

> Yes, this would work up to a point.  That point being one of ``make 
> install prefix=/my/depot/release/directory''. 

No, actually the GCS explicitly requires that if you change the "prefix"
at 'make install' time, that the images installed into this new prefix
are NOT modified from whatever was established by the original prefix at
configure time.

So (assuming rebase actually HAD a configure stage; it doesn't currently
but the above proposal assumes that one would be added), 'make install
prefix=/tmp/for/packaging' would still work.  Assuming the end user
actually unpacked the tarball thus created into the original configured
prefix.

> I loathe DESTDIR and 
> changing prefix at install time is not supposed to disturb the build 
> process.

Err...yes...that statement contradicts what you said above.

> Isn't this more like what other distributed package scripts do?

Sometimes.  .pc files have prefix=@prefix@; they don't do fancy
where-am-i path shenanigans.  .la files have explicitly absolute paths
based on @prefix@ too.  Some /bin/scripts do it one way, some another.

>> Right now, rebase violates both. Personally, I think #1 is a fool's
>> game.  Now, #2...is another question. Most well-behavior packages obey
>> it, but the upstream rebase package does not.  It can be patched to do
>> so, of course -- and THAT patch might actually be accepted upstream
>> (most of the other MSYS mods -- e.g. explicitly using dash rather than
>> ash -- wouldn't).
>>
> 
> So, the upstream package needs a fix.

OK.

> PATH=$(cd $tp2&&  pwd):@bindir@...

Well, all of this pre-supposes that a configure script is added.  So
it's a bigger upstream change than you might think.

I'll look into it.

--
Chuck

On 6/1/2010 8:58 AM, Earnie wrote:
> Earnie wrote:
>>>
>>> Wait.  (a) Why wouldn't you, and (b) are the binary *msys* packages
>>> uploaded to sourceware expected to work properly if they are NOT
>>> installed in the expected directory?
>>>
>>
>> Yes, they should work properly if not installed in the expected
>> directory.  I know there are a few that expect to find their files by
>> default in /usr/share but those are exceptions easily worked around.
>>
> 
> I should have instead said:
> 
> An expectation of the user is that it works properly.  This is Windows 
> and the idiots abound.  I, the user, should be able to install anywhere 
> I please, even though I've been warned not to.

In this case, I adopt the cygwin motto: We're Just Mean.  Don't do that;
if you do, understand the consequences: you get to keep all the pieces
if it breaks.

--
Chuck

Charles Wilson wrote:
> On 6/1/2010 8:58 AM, Earnie wrote:

>> An expectation of the user is that it works properly.  This is Windows
>> and the idiots abound.  I, the user, should be able to install anywhere
>> I please, even though I've been warned not to.
>
> In this case, I adopt the cygwin motto: We're Just Mean.  Don't do that;
> if you do, understand the consequences: you get to keep all the pieces
> if it breaks.
>

Agreed.

Earnie

On Thursday 27 May 2010 16:03:10 Charles Wilson wrote:
> > And fix rebaseall and peflagsall so that it isn't setting the
> > PATH or at least provide a REBASE_PATH to be prepended to PATH.
>
> Look at the code in rebaseall first, and then tell me if you still
> think it'd be just great to use a native rebase with rebaseall. If
> so, then I'll add REBASE_PATH support.

Oh my.  So much ado about nothing, (and The Bard's version was so much 
more entertaining).

Really, this is a moot point.  I would class the entire rebase package 
as a system administrative tool; as such, had it been my choice I 
would have located it in /sbin, (or /usr/sbin, which of course would 
be the same in MSYS anyway).  It wasn't my choice, and the upstream 
developer chose somewhere else.  So what?  For the limited use I 
might make of rebase, that doesn't inconvenience me; nor should it 
inconvenience anyone else.  I really can't get myself worked up over 
this non-issue.  It isn't worth it.

-- 
Regards,
Keith.

On 6/1/2010 4:09 PM, Keith Marshall wrote:
> Oh my.  So much ado about nothing,

Not exactly. Remember this whole discussion came about because of the
perceived need to be able to rebase msys-1.0.dll itself; hence, should
rebase.exe itself be an MSYS or MinGW app.  That's a bit more important
than where, exactly, an msys version ought to be installed.

> (and The Bard's version was so much 
> more entertaining).

On this I agree.

> Really, this is a moot point.  I would class the entire rebase package 
> as a system administrative tool; as such, had it been my choice I 
> would have located it in /sbin, (or /usr/sbin, which of course would 
> be the same in MSYS anyway).  It wasn't my choice, and the upstream 
> developer chose somewhere else.  So what?  For the limited use I 
> might make of rebase, that doesn't inconvenience me; nor should it 
> inconvenience anyone else.  I really can't get myself worked up over 
> this non-issue.  It isn't worth it.

Ack.  (I assume you still want a MinGW version of the exe, so that it
can be used to rebase msys-1.0.dll, regardless of whether there also
exists an msys version nor where that msys version is installed?)

--
Chuck

Charles Wilson wrote:
> Ack.  (I assume you still want a MinGW version of the exe, so that it
> can be used to rebase msys-1.0.dll, regardless of whether there also
> exists an msys version nor where that msys version is installed?)
>

Yes, this is needed, at least until the base address is redone by Cesar 
but even then it would be good to have a "native" version of rebase for 
those who don't want MSYS but still want to be able to rebase.

Earnie

On Wednesday 02 June 2010 04:01:01 Charles Wilson wrote:
> > Oh my.  So much ado about nothing,
>
> Not exactly.

I was referring, in particular, to the side issue of installation 
path; IMO, that *is* much ado about nothing.

> Remember this whole discussion came about because of 
> the perceived need to be able to rebase msys-1.0.dll itself; hence,
> should rebase.exe itself be an MSYS or MinGW app.  That's a bit
> more important than where, exactly, an msys version ought to be
> installed.

Of course it is, but we were becoming side tracked by this (IMO) 
irrelevant side issue.  The only issue we, as package maintainers, 
need to be concerned about, related to installation paths, is that 
any MSYS installation, which is derived from our installation kits, 
remains homogeneous and functional wrt its sysroot.  If anyone wants 
to fragment it, and install its various components in non-standard 
arbitrary locations, then they are on their own; I will not support 
them, and if they break it, they get to keep all the pieces.

> > ...  I really can't get myself worked up over this non-issue.  
> > It isn't worth it. 

So, back on topic...

> Ack.  (I assume you still want a MinGW version of the exe, so that
> it can be used to rebase msys-1.0.dll, regardless of whether there
> also exists an msys version nor where that msys version is
> installed?)

Yes.  I think that makes sense.  We have identified, (and at least two 
of us have experienced), a situation where msys-1.0.dll itself cannot 
be loaded; a Google search on "msys reserve space cygwin heap" turns 
up many other instances of users reporting a similar problem in their 
own pet fora, (which never percolate back to us), so it is an issue 
we need to address.  Providing a native Win32 build of rebase.exe 
could be potentially useful, to help resolve the issue whenever it 
arises; more proactively, we should also consider whether there is a 
more suitable base address which could be chosen for msys-1.0.dll, 
which might reduce the number of occurrences of the problem in the 
first place[*].

[*] One forum thread Google found at http://www.avrfreaks.net suggests that 
some antivirus products may cause this problem; specifically, they 
seem to consider the memory range in which msys-1.0.dll wants to load 
as "protected", and they block the load as an "identified threat"; of 
course this is circumstantial hearsay -- it may have some substance, 
but curiously, having experienced this problem a few weeks ago, and 
having used a native rebase.exe to work around it, I can no longer 
reproduce the problem with a fresh and unmodified installation of 
MSYS-1.0.14 today, (msysCORE, bash and coreutils only), so I'm 
completely confused about what may have caused it.

-- 
Regards,
Keith.

On 6/2/2010 10:47 AM, Earnie wrote:
> 
> Yes, this is needed, at least until the base address is redone by Cesar 
> but even then it would be good to have a "native" version of rebase for 
> those who don't want MSYS but still want to be able to rebase.

I've autoconfifies rebase, adapted it to build (mostly) OOB for MSYS,
MinGW, as well as Cygwin, addressed all of the issues raised in this
thread, and fixed a number of additional bugs discovered along the way
-- which took a heck of a lot more effort than it should have.

I sent this patch upstream to Jason, but he's a bit busy right now with
cygwin-python; he promises to take a look once that python problem is
resolved.

I'm going to hold off on updating the msys-rebase and mingw-rebase
packages until we see what Jason does with my patch, and I have a new
upstream source release to work with.

--
Chuck

Danny Smith wrote:
>
> One possible scheme is to choose a base address based on the first
> letter of the DLL name.
>
> No need to reinvent the wheel here.  The --enable-auto-image-base that Mumut
> Khan contributed to binutils does it even better:
>
> /* Use the output file to create a image base for relocatable DLLs.  */

I tried --enable-auto-image-base on the MSYS DLL and it didn't seem to 
do any harm. The new base address chosen was 0x6F080000. Any objections 
to start using --enable-auto-image-base on the MSYS DLL?

Cesar

On 6/4/2010 9:05 PM, Cesar Strauss wrote:
> Danny Smith wrote:
>>
>> One possible scheme is to choose a base address based on the first
>> letter of the DLL name.
>>
>> No need to reinvent the wheel here.  The --enable-auto-image-base that Mumut
>> Khan contributed to binutils does it even better:
>>
>> /* Use the output file to create a image base for relocatable DLLs.  */

Well, what Danny didn't point out was that --enable-auto-image-base
doesn't concern itself with the memory footprint of the DLL, so even if
(a) there were no hash collisions (e.g. no two DLLs were ever assigned
the SAME image base), and (b) auto-image-base was used to build EVERY
DLL in the entire MSYS and MinGW distributions...

You still could have DLLs "overlapping" in memory:

    0x68EC0000 DLL #1 image base
       |
       |
       |                    DLL #2 image base  0x68F40000
       |                                            |
       |                                            |
    0x69000000 DLL #1 image base + DLL footprint    |
                                                    |
             DLL #2 image base + DLL footprint 0x69040000

Leading to the windows runtime loader relocating one or the other of the
offending DLL (if both are loaded by the same process), leading to ...
not relocating them to the same place in both parent and child.

Even so, (b) is not true for the universe of MSYS and MinGW DLLs -- and,
for that matter, there is no way to guarantee (a)!

And THAT's the reason for 'rebaseall': the idea is to systematically set
the image base address for ALL DLLs in MinGW+MSYS (originally: Cygwin),
taking into account the memory footprint of each one, so that there ARE
no overlaps in that universe.

But it's a brute force tool; it would be nice to TRY and avoid overlaps
at all, to begin with.  And --enable-auto-image-base is a way to TRY to
do that, for all DLLs *other* than the core one
(cygwin1.dll/msys-1.0.dll) -- that's why libtool uses the flag by
default, for all DLLs built using that tool. (But not all msys or mingw
DLLs are built using libtool).

> I tried --enable-auto-image-base on the MSYS DLL and it didn't seem to 
> do any harm. The new base address chosen was 0x6F080000. Any objections 
> to start using --enable-auto-image-base on the MSYS DLL?

But here, we're talking about a *single* DLL.  We're just trying to pick
a memory location; almost any location will do, subject to certain
constraints.  When MSYS was forked, the "appropriate" location was
changed from cygwin's 0x61000000 to 0x71000000, simply so that it
wouldn't clash (otherwise, running cygwin apps from inside an msys shell
would die, immediately, IIRC -- even though doing that, or vice versa,
is (a) not supported, and (b) often dies anyway).

It appears, given what we've discovered on this thread, that 0x71000000
was a poor choice (being outside the "legal" zone).

So, for a new choice, we probably want:
  (a) other than 0x61000000
  (b) *below* 0x61300000 (because many of the MSYS DLLs
      DO, in fact, use --enable-auto-image-base, even if not ALL of
      them do so)
  (c) in the legal range
  (d) does not overlap with any of the excluded ranges, taking into
      account the MSYS image size of 0x00104000, as of 1.0.14 (and
      the cygwin image size of 0x00300000, as of 1.7.5)

So, I'm thinking any choice from 0x60000000 to 0x60E00000 would be good.
 --enable-auto-image-base won't do that (it's not supposed to); in the
*specific* case of the msys DLL we should not use it, but instead choose
the image base explicitly.  Since we already do this (poorly), we're
left simply with changing the existing number:

0x71000000 -> 0x60C00000 ?

That leaves plenty of "growth" between the top of msys-1.0 and the
bottom of cygwin1.dll, even if msys gets to be as big as the current
cygwin, while still not overlapping anything in the normal
--enable-auto-image-base range.

--
Chuck

On Saturday 05 June 2010 14:50:23 Charles Wilson wrote:
> So, I'm thinking any choice from 0x60000000 to 0x60E00000 would be
> good. --enable-auto-image-base won't do that (it's not supposed
> to); in the *specific* case of the msys DLL we should not use it,
> but instead choose the image base explicitly.  Since we already do
> this (poorly), we're left simply with changing the existing number:
>
> 0x71000000 -> 0x60C00000 ?

Hmm.  To facilitate testing of your tentative mingw-get manifests, 
I've set up a completely fresh msysCORE-1.0.14 based install on my 
Win2K box.  I started out minimally, with just the core package plus 
bash-3.1.17-3-msys-1.0.13-bin; this reproduces the "cannot reserve 
space for cygwin's heap" failure on initial msys-1.0.dll load.

Rebasing to 0x60c00000 gets around that, and allows the shell to load.  
It remains functional, on subsequently adding your older *statically* 
linked coreutils-5.97-2-msys-1.0.11-bin package; however, if I then 
upgrade to dynamically linked coreutils-5.97-3-msys-1.0.13-bin, while 
I can still start the shell session, *none* of the coreutils programs 
will execute -- they all die with a similar "cannot reserve space..."
message.  I tried rebasing again, to 0x60b80000 -- I don't know if 
that's a valid decrement -- to no avail; normal service resumed when 
I eventually reduced the base address to 0x60b00000, so perhaps we 
could consider that as a more suitable setting for subsequent 
releases?  Or, maybe we should go even lower?

-- 
Regards,
Keith.

On 7/2/2010 9:24 AM, Keith Marshall wrote:
> On Saturday 05 June 2010 14:50:23 Charles Wilson wrote:
>> So, I'm thinking any choice from 0x60000000 to 0x60E00000 would be
>> good. --enable-auto-image-base won't do that (it's not supposed
>> to); in the *specific* case of the msys DLL we should not use it,
>> but instead choose the image base explicitly.  Since we already do
>> this (poorly), we're left simply with changing the existing number:
>>
>> 0x71000000 -> 0x60C00000 ?
> 
> Hmm.  To facilitate testing of your tentative mingw-get manifests, 
> I've set up a completely fresh msysCORE-1.0.14 based install on my 
> Win2K box.  I started out minimally, with just the core package plus 
> bash-3.1.17-3-msys-1.0.13-bin; this reproduces the "cannot reserve 
> space for cygwin's heap" failure on initial msys-1.0.dll load.
> 
> Rebasing to 0x60c00000 gets around that, and allows the shell to load.  
> It remains functional, on subsequently adding your older *statically* 
> linked coreutils-5.97-2-msys-1.0.11-bin package; however, if I then 
> upgrade to dynamically linked coreutils-5.97-3-msys-1.0.13-bin, while 
> I can still start the shell session, *none* of the coreutils programs 
> will execute -- they all die with a similar "cannot reserve space..."
> message.  I tried rebasing again, to 0x60b80000 -- I don't know if 
> that's a valid decrement -- to no avail; normal service resumed when 
> I eventually reduced the base address to 0x60b00000, so perhaps we 
> could consider that as a more suitable setting for subsequent 
> releases?  Or, maybe we should go even lower?
> 
On 7/2/2010 9:24 AM, Keith Marshall wrote:
> On Saturday 05 June 2010 14:50:23 Charles Wilson wrote:
>>
>> 0x71000000 -> 0x60C00000 ?
>
> Rebasing to 0x60c00000 gets around that, and allows the shell to
> load.
...
> normal service resumed when
> I eventually reduced the base address to 0x60b00000, so perhaps we
> could consider that as a more suitable setting for subsequent
> releases?  Or, maybe we should go even lower?

Well, if anything between 0x60000000 to 0x60E00000 would be ok, then why
not go "halfway down" to 0x60800000 just to be on the safe side?

(FWIW, I have the whole dynamically-linked schmeil installed, and my
msys-1.0.dll image base is 0x71000000, so...I don't actually see any o
these problems.

Today.

(Hmmm...the rebase progress stalled because I was waiting for upstream.
Jason said he needed to handle the cygwin-python-2.6 release rollout
first.  That happened the week before last -- looks like it's time for
me to ping him again.)

--
Chuck