#908 MSYS shell introduces extraneous SOH bytes in expressions

MSYS
closed
Earnie Boyd
None
out-of-date
Known_bugs
2013-01-28
2006-02-23
Keith Marshall
No

Command substitutions, delimited by either back-quotes
or by $(...), introduce an extraneous SOH byte
following the substituted text, when the substitution
itself is enclosed within a double quoted expression
which extends beyond the end of the command substitution.

This appears to affect only substitutions involving
CRLF delimited text output, as illustrated by:

$ echo "`printf 'foo\n'` bar" | od -cb
0000000 f o o b a r \n
146 157 157 040 142 141 162 012
0000010

$ echo "`printf 'foo\r\nbar\r\n'`" | od -cb
0000000 f o o \r \n b a r \n
146 157 157 015 012 142 141 162 012
0000011

$ echo "`printf 'foo\r\nbar\r\n'` baz" | od -cb
0000000 f o o \r \n b a r 001 b
a z \n
146 157 157 015 012 142 141 162 001 040 142
141 172 012
0000016

$ echo "`printf 'foo\r\nbar\n'`"" baz" | od -cb
0000000 f o o \r \n b a r b a
z \n
146 157 157 015 012 142 141 162 040 142 141
172 012
0000015

$ echo "`printf 'foo\r\nbar\n'` baz" | od -cb
0000000 f o o \r \n b a r b a
z \n
146 157 157 015 012 142 141 162 040 142 141
172 012
0000015

Observe that:

1) The extraneous byte seems always to be SOH.

2) The extraneous byte appears *only* when the output from
the substituted command is CRLF delimited text,
*and* the
double-quoted expression extends *beyond* the end of the
command substitution.

3) It is only the terminal CRLF, on the final line of
output
from the substituted command, that participates in the
generation of the extraneous byte; any other CRLF line
terminators, appearing earlier in the output, remain
unchanged in the substitution.

4) If the end of the command substitution is coincident
with
the end of the double-quoted expression, then *both* the
terminal LF *and* its associated CR are dropped from the
substituted command output; OTOH, if the double-quoted
expression extends beyond the end of the substitution,
then only the terminal LF is dropped, and its associated
CR is *replaced* by SOH.

Repeat these examples on a GNU/Linux box produces
identically
the same results, *except* that CRs are preserved
unchanged, in *all*
cases, e.g.

$ echo "`printf 'foo\r\nbar\r\n'` baz" | od -cb
0000000 f o o \r \n b a r \r b
a z \n
146 157 157 015 012 142 141 162 015 040 142
141 172 012
0000016

From this, it appears that the problem lies
specifically in the
mishandling of line terminators, when the output from
the substituted
command is CRLF delimited text. Even more
specifically, it lies in
the inconsistent handling of the terminal CRLF,
depending on whether
the end of the command substitution is coincident with
the end of the
enclosing double-quoted expression, or that expression
extends beyond
the end of the command substitution, i.e.

a) in the case of coincident ends of substitution and
expression,
the terminal CRLF is discarded, in its entirety.

b) in the case of the double-quoted expression
extending beyond
the end of the command substitution, only the
terminal LF is
discarded, and its associated CR is erroneously
replaced by
a single SOH.

Discussion

1 2 > >> (Page 1 of 2)
  • Earnie Boyd
    Earnie Boyd
    2006-02-23

    • priority: 5 --> 6
     
  • Earnie Boyd
    Earnie Boyd
    2006-02-23

    Logged In: YES
    user_id=15438

    OTTWL I'm thinking this is a result of modifications to bash
    to remove (or modify) \r from script files and piped input.
    This should only be done for commands to be executed. If
    someone has time to look at the bash source MSYS uses, it
    would be a great thing.

    Earnie

     
  • Earnie Boyd
    Earnie Boyd
    2006-11-28

    • priority: 6 --> 9
     
  • Earnie Boyd
    Earnie Boyd
    2006-11-28

    Logged In: YES
    user_id=15438
    Originator: NO

    The 3.1.0 version of bash needs to be patched to preserve the \r in the text.

    boyde@OH6000GBOYDE /devshop/msys/pkgupdate/depot
    $ echo "`printf 'foo\r\nbar\r\n'`" | od -cb
    0000000 f o o \n b a r \n
    146 157 157 012 142 141 162 012
    0000010

    boyde@OH6000GBOYDE /devshop/msys/pkgupdate/depot
    $ sh --version
    GNU bash, version 3.1.0(3)-release (i686-pc-msys)
    Copyright (C) 2005 Free Software Foundation, Inc.

     
  • Perry Werneck
    Perry Werneck
    2012-04-12

    Sample script showing the same problem:

    #!/bin/bash
    PKG_CONFIG=/mingw/bin/pkg-config.exe
    GDK_PIXBUF_PACKAGES=gobject-2.0
    GDK_PIXBUF_EXTRA_LIBS=$STATIC_LIB_DEPS $MATH_LIB $MEDIA_LIB
    GDK_PIXBUF_EXTRA_CFLAGS=

    FAILED_STRING="`$PKG_CONFIG --libs $GDK_PIXBUF_PACKAGES` $GDK_PIXBUF_EXTRA_LIBS"
    NON_FAILED_STRING=`$PKG_CONFIG --libs $GDK_PIXBUF_PACKAGES` $GDK_PIXBUF_EXTRA_LIBS

    echo FAILED=$FAILED_STRING
    echo NON_FAILED=$NON_FAILED_STRING

     
  • Keith Marshall
    Keith Marshall
    2012-04-12

    Wow! A real "blast from the past"! What museum did you raid, for a version of MSYS-bash which still exhibits this issue?

    The problem, as originally reported, has long since been fixed; the only residual (potential) issue is, as Earnie has noted, that any CRLF delimited stream is now reduced unconditionally to LF delimited, so this previously failing case:

    $ echo "`printf 'foo\r\nbar\r\n'` baz" | od -cb
    0000000 f o o \r \n b a r 001 b a z \n
    146 157 157 015 012 142 141 162 001 040 142 141 172 012
    0000016

    now results in:

    $ echo "`printf 'foo\r\nbar\r\n'` baz" | od -cb
    0000000 f o o \n b a r b a z \n
    146 157 157 012 142 141 162 040 142 141 172 012
    0000014

    so if you really need CRLF, you either need to add an extra CR:

    $ echo "`printf 'foo\r\r\nbar\r\n'` baz" | od -cb
    0000000 f o o \r \n b a r b a z \n
    146 157 157 015 012 142 141 162 040 142 141 172 012
    0000015

    or, (probably better), filter the output:

    $ echo "`printf 'foo\r\nbar\r\n'` baz" | unix2dos |od -cb
    0000000 f o o \r \n b a r b a z \r \n
    146 157 157 015 012 142 141 162 040 142 141 172 015 012
    0000016

    Note that this notion of preserving CR relates to an extremely contrived case; in the general case, conversion of CRLF to LF, (which is the normal line ending in MSYS), seems reasonable. IMO, this requires no further action, and could be closed as "fixed", (to my satisfaction, anyway).

     
  • Keith Marshall
    Keith Marshall
    2012-04-12

    • priority: 9 --> 1
    • status: open --> open-out-of-date
     
  • Perry Werneck
    Perry Werneck
    2012-04-12

    Hi,

    Well, the entire MinGW/Msys was downloaded and installed a few days ago using mingw-get-inst-20111118.exe. Is it the latest one?

    Changing the script isn´t so easy since the problem is happening in the configure scripts from gtk3 and gdk-pixbuf.

     
  • Keith Marshall
    Keith Marshall
    2012-04-12

    Telling me what version of the installer-wrapper you used to install doesn't convey much useful information about the actual package versions you've installed, but yes, for a recent install with this as starting point, your installation should be current. To be certain, what do

    $ uname -a

    and

    $ sh --version

    say, (and are these definitely the versions running your failing script).

    If your script definitely is running on current versions, it seems highly improbable that this is the same issue. I have some concern about your pkg-config.exe living in /mingw/bin; this isn't a MinGW standard package, and is normally specific to the particular package build system which delivers it. Thus, it should really be sequestered in its own, source package specific, libexec path; leaving it in /mingw/bin creates potential for conflict, when two independent source packages try to install their own, possibly incompatible versions.

    In any case, I don't have any pkg-config.exe on my box, and since I can't be sure that any version I acquire would be identical to yours, I can't pursue the issue unless you can make a copy available. You won't be able to attach it here, but if you mail it to me privately, I'll do so.

     
  • Earnie Boyd
    Earnie Boyd
    2012-04-12

    Keith,

    pkg-config is part of the GLIB distributions and Perry may have built the package himself or downloaded one of Tor's packages. See http://www.mingw.org/wiki/Bootstrapping_GLIB_with_MinGW.

    As for gtk3 and gdk-pixbuf I have successfully configure, make and make install these monsters. Notes at http://bit.ly/stHdlF if you want to look.

     
1 2 > >> (Page 1 of 2)