[MinGW-notify] [ mingw-Bugs-1437313 ] MSYS shell introduces extraneous SOH bytes in expressions

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Bugs item #1437313, was opened at 2006-02-23 02:08
Message generated for change (Comment added) made by keithmarshall
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=102435&aid=1437313&group_id=2435

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: MSYS
Group: None
Status: Closed
Resolution: Out of Date
Priority: 1
Private: No
Submitted By: Keith Marshall (keithmarshall)
Assigned to: Earnie Boyd (earnie)
Summary: MSYS shell introduces extraneous SOH bytes in expressions

Initial Comment:
Command substitutions, delimited by either back-quotes
or by $(...), introduce an extraneous SOH byte
following the substituted text, when the substitution
itself is enclosed within a double quoted expression
which extends beyond the end of the command substitution.

This appears to affect only substitutions involving
CRLF delimited text output, as illustrated by:

  $ echo "`printf 'foo\n'` bar" | od -cb
  0000000   f   o   o       b   a   r  \n
          146 157 157 040 142 141 162 012
  0000010

  $ echo "`printf 'foo\r\nbar\r\n'`" | od -cb
  0000000   f   o   o  \r  \n   b   a   r  \n
          146 157 157 015 012 142 141 162 012
  0000011

  $ echo "`printf 'foo\r\nbar\r\n'` baz" | od -cb
  0000000   f   o   o  \r  \n   b   a   r 001       b 
 a   z  \n
          146 157 157 015 012 142 141 162 001 040 142
141 172 012
  0000016

  $ echo "`printf 'foo\r\nbar\n'`"" baz" | od -cb
  0000000   f   o   o  \r  \n   b   a   r       b   a 
 z  \n
          146 157 157 015 012 142 141 162 040 142 141
172 012
  0000015

  $ echo "`printf 'foo\r\nbar\n'` baz" | od -cb
  0000000   f   o   o  \r  \n   b   a   r       b   a 
 z  \n
          146 157 157 015 012 142 141 162 040 142 141
172 012
  0000015

Observe that:

1) The extraneous byte seems always to be SOH.

2) The extraneous byte appears *only* when the output from
   the substituted command is CRLF delimited text,
*and* the
   double-quoted expression extends *beyond* the end of the
   command substitution.

3) It is only the terminal CRLF, on the final line of
output
   from the substituted command, that participates in the
   generation of the extraneous byte; any other CRLF line
   terminators, appearing earlier in the output, remain
   unchanged in the substitution.

4) If the end of the command substitution is coincident
with
   the end of the double-quoted expression, then *both* the
   terminal LF *and* its associated CR are dropped from the
   substituted command output;  OTOH, if the double-quoted
   expression extends beyond the end of the substitution,
   then only the terminal LF is dropped, and its associated
   CR is *replaced* by SOH.

Repeat these examples on a GNU/Linux box produces
identically
the same results, *except* that CRs are preserved
unchanged, in *all*
cases, e.g.

  $ echo "`printf 'foo\r\nbar\r\n'` baz" | od -cb
  0000000   f   o   o  \r  \n   b   a   r  \r       b 
 a   z  \n
          146 157 157 015 012 142 141 162 015 040 142
141 172 012
  0000016

From this, it appears that the problem lies
specifically in the
mishandling of line terminators, when the output from
the substituted
command is CRLF delimited text.  Even more
specifically, it lies in
the inconsistent handling of the terminal CRLF,
depending on whether
the end of the command substitution is coincident with
the end of the
enclosing double-quoted expression, or that expression
extends beyond
the end of the command substitution, i.e.

a) in the case of coincident ends of substitution and
expression,
   the terminal CRLF is discarded, in its entirety.

b) in the case of the double-quoted expression
extending beyond
   the end of the command substitution, only the
terminal LF is
   discarded, and its associated CR is erroneously
replaced by
   a single SOH.

----------------------------------------------------------------------

>Comment By: Keith Marshall (keithmarshall)
Date: 2012-04-12 14:24

Message:
I guess executive2002 has commented on the wrong ticket; the comment
appears to be, in no way, related to this issue.

----------------------------------------------------------------------

Comment By: Gaetano  (executive2002)
Date: 2012-04-12 09:40

Message:
Thank you very much to everyone.

As I reported in my first issue, I'm using Windows 7 Enterprise and not
Linux, so I cannot modify/view the bash.

Another thing: the problem is with floating point numbers contained in an
array, not on strings. The problem of LF, in other words, should not be
applied.

If I "cout" the numbers on stdout, they are "represented" as expected: the
problem is in memory. The compiler rounds in a wrong way the number in
memory.

Ciao 

----------------------------------------------------------------------

Comment By: Perry Werneck (perrywerneck)
Date: 2012-04-12 09:01

Message:
Hi,

Sorry! I thought the installer version was enough to say the MinGW level.
(-;

perry@rv410 ~
$ uname -a
MINGW32_NT-6.1 RV410 1.0.17(0.48/3/2) 2011-04-24 23:39 i686 Msys

perry@rv410 ~
$ sh --version
GNU bash, version 3.1.17(1)-release (i686-pc-msys)
Copyright (C) 2005 Free Software Foundation, Inc.

I just installed prebuilt MinGW & Msys, all other packages are being built
from source using a bash script to download, configure and install one by
one. I can´t use gtk prebuilt packages because the target is a MSAA bridge
for ATK and, to do this, I´ll need to apply a few patches in gtk+.

I´m using the same prefix for every apps just to keep all the development
stuff in the same path. The idea is to, at some point, run make
DESTDIR=PATH in all packages and build a new GTK3 bundle install with nsys
to simplify new developments.

By the way, the earnie comments helped a lot!! I´ll try to rebuild
everithing using his workaround  for ^A 

----------------------------------------------------------------------

Comment By: Earnie Boyd (earnie)
Date: 2012-04-12 06:47

Message:
I agree that this is long ago fixed.

----------------------------------------------------------------------

Comment By: Keith Marshall (keithmarshall)
Date: 2012-04-12 06:35

Message:
Earnie,

I know *what* pkg-config is.  I also understand, from many years of
participation on the autoconf ML, that various source packages may include
pkg-config, as a component of their individual build systems, and that such
distributions may not be compatible, one with another.  Thus, received
wisdom is that each package should maintain its own (private) pkg-config
installation, and hence my concern that the OP has a public installation,
in /mingw/bin.

That aside, I'm not going to waste my time tracking down a pkg-config
distribution, which may be, but equally well may not be compatible with the
version used by the OP, just to confirm whether or not his issue is the
same as that which I reported originally.  AFAIAC, this issue was resolved
long ago; I certainly cannot reproduce the original fault with current
sh.exe, whether run as such, or as bash.exe, on the basis of my original
test case.

----------------------------------------------------------------------

Comment By: Earnie Boyd (earnie)
Date: 2012-04-12 05:10

Message:
Keith,

pkg-config is part of the GLIB distributions and Perry may have built the
package himself or downloaded one of Tor's packages.  See
http://www.mingw.org/wiki/Bootstrapping_GLIB_with_MinGW.

As for gtk3 and gdk-pixbuf I have successfully configure, make and make
install these monsters.  Notes at http://bit.ly/stHdlF if you want to look.

----------------------------------------------------------------------

Comment By: Keith Marshall (keithmarshall)
Date: 2012-04-12 04:08

Message:
Telling me what version of the installer-wrapper you used to install
doesn't convey much useful information about the actual package versions
you've installed, but yes, for a recent install with this as starting
point, your installation should be current.  To be certain, what do

  $ uname -a

and

  $ sh --version

say, (and are these definitely the versions running your failing script).

If your script definitely is running on current versions, it seems highly
improbable that this is the same issue.  I have some concern about your
pkg-config.exe living in /mingw/bin; this isn't a MinGW standard package,
and is normally specific to the particular package build system which
delivers it.  Thus, it should really be sequestered in its own, source
package specific, libexec path; leaving it in /mingw/bin creates potential
for conflict, when two independent source packages try to install their
own, possibly incompatible versions.

In any case, I don't have any pkg-config.exe on my box, and since I can't
be sure that any version I acquire would be identical to yours, I can't
pursue the issue unless you can make a copy available.  You won't be able
to attach it here, but if you mail it to me privately, I'll do so.

----------------------------------------------------------------------

Comment By: Perry Werneck (perrywerneck)
Date: 2012-04-12 02:15

Message:
Hi,

Well, the entire MinGW/Msys was downloaded and installed a few days ago
using mingw-get-inst-20111118.exe. Is it the latest one? 

Changing the script isn´t so easy since the problem is happening in the
configure scripts from gtk3 and gdk-pixbuf.

----------------------------------------------------------------------

Comment By: Keith Marshall (keithmarshall)
Date: 2012-04-12 02:02

Message:
Wow!  A real "blast from the past"!  What museum did you raid, for a
version of MSYS-bash which still exhibits this issue?

The problem, as originally reported, has long since been fixed; the only
residual (potential) issue is, as Earnie has noted, that any CRLF delimited
stream is now reduced unconditionally to LF delimited, so this previously
failing case:

$ echo "`printf 'foo\r\nbar\r\n'` baz" | od -cb
0000000 f o o \r \n b a r 001 b a z \n
146 157 157 015 012 142 141 162 001 040 142 141 172 012
0000016

now results in:

$ echo "`printf 'foo\r\nbar\r\n'` baz" | od -cb
0000000   f   o   o  \n   b   a   r       b   a   z  \n
146 157 157 012 142 141 162 040 142 141 172 012
0000014

so if you really need CRLF, you either need to add an extra CR:

$ echo "`printf 'foo\r\r\nbar\r\n'` baz" | od -cb
0000000   f   o   o  \r  \n   b   a   r       b   a   z  \n
146 157 157 015 012 142 141 162 040 142 141 172 012
0000015

or, (probably better), filter the output:

$ echo "`printf 'foo\r\nbar\r\n'` baz" | unix2dos |od -cb
0000000   f   o   o  \r  \n   b   a   r       b   a   z  \r  \n
146 157 157 015 012 142 141 162 040 142 141 172 015 012
0000016

Note that this notion of preserving CR relates to an extremely contrived
case; in the general case, conversion of CRLF to LF, (which is the normal
line ending in MSYS), seems reasonable.  IMO, this requires no further
action, and could be closed as "fixed", (to my satisfaction, anyway).

----------------------------------------------------------------------

Comment By: Perry Werneck (perrywerneck)
Date: 2012-04-11 18:13

Message:
Sample script showing the same problem:

#!/bin/bash
PKG_CONFIG=/mingw/bin/pkg-config.exe
GDK_PIXBUF_PACKAGES=gobject-2.0
GDK_PIXBUF_EXTRA_LIBS=$STATIC_LIB_DEPS $MATH_LIB $MEDIA_LIB
GDK_PIXBUF_EXTRA_CFLAGS=

FAILED_STRING="`$PKG_CONFIG --libs $GDK_PIXBUF_PACKAGES`
$GDK_PIXBUF_EXTRA_LIBS"
NON_FAILED_STRING=`$PKG_CONFIG --libs $GDK_PIXBUF_PACKAGES`
$GDK_PIXBUF_EXTRA_LIBS

echo FAILED=$FAILED_STRING
echo NON_FAILED=$NON_FAILED_STRING

----------------------------------------------------------------------

Comment By: Earnie Boyd (earnie)
Date: 2006-11-28 09:04

Message:
Logged In: YES 
user_id=15438
Originator: NO

The 3.1.0 version of bash needs to be patched to preserve the \r in the
text.

boyde@OH6000GBOYDE /devshop/msys/pkgupdate/depot
$ echo "`printf 'foo\r\nbar\r\n'`" | od -cb
0000000   f   o   o  \n   b   a   r  \n
        146 157 157 012 142 141 162 012
0000010

boyde@OH6000GBOYDE /devshop/msys/pkgupdate/depot
$ sh --version
GNU bash, version 3.1.0(3)-release (i686-pc-msys)
Copyright (C) 2005 Free Software Foundation, Inc.

----------------------------------------------------------------------

Comment By: Earnie Boyd (earnie)
Date: 2006-02-23 05:40

Message:
Logged In: YES 
user_id=15438

OTTWL I'm thinking this is a result of modifications to bash
to remove (or modify) \r from script files and piped input.
 This should only be done for commands to be executed.  If
someone has time to look at the bash source MSYS uses, it
would be a great thing.

Earnie

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=102435&aid=1437313&group_id=2435

[MinGW-notify] [ mingw-Bugs-1437313 ] MSYS shell introduces extraneous SOH bytes in expressions

A native Windows port of the GNU Compiler Collection (GCC)

[MinGW-notify] [ mingw-Bugs-1437313 ] MSYS shell introduces extraneous SOH bytes in expressions