GraphicsMagick / Bugs / #659 Can not escape $0 in a batch command

Andrew Dryga - 2022-01-12

Package and version: https://pkgs.alpinelinux.org/flag/community/graphicsmagick/1.3.37-r0

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrew Dryga - 2022-01-12

I've also tried using single quotes in batch script but no luck.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrew Dryga - 2022-01-12

"text 0,193 \"$0 down\"", "text 0,193 \"$"''"0 down\"", "$""0 down" also don't work.

Last edit: Andrew Dryga 2022-01-12

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrew Dryga - 2022-01-12

"text 0,193 '%$0 down'" doesn't fail but generates image with % prefix written on the image. If %% is added just one % is written to an image ("text 0,193 '%%$0 down'").

This trick doesn't work at all if you do just "text 0,193 ' $0 down'".

So any visible symbol before $ is added to an image (which we don't want) and any invisible symbol (like space) is ignored and image header is corrupted.

Last edit: Andrew Dryga 2022-01-12

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Standard escaping options outside of batch work just fine:

$ gm convert -font helvetica -background transparent -gravity North -fill black -pointsize 70 -draw "text 0,193 '\$0 down'" -strip "/tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLL
C9gAbOWB/gm-FsmTmMP34mvLC9gAbOaB.miff" "/tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/gm-FsmTmMP5PMDLC9gAbOcB.png"

$ gm convert -font helvetica -background transparent -gravity North -fill black -pointsize 70 -draw "text 0,193 '$""0 down'" -strip "/tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbL
LC9gAbOWB/gm-FsmTmMP34mvLC9gAbOaB.miff" "/tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/gm-FsmTmMP5PMDLC9gAbOcB.png"

Andrew Dryga - 2022-01-12

Here is the source miff image for the batch script

gm-FsmTmLyrVGTLC9gAbOSB.miff

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Bob Friesenhahn - 2022-01-12

It is good that you have isolated this to being specifically related to 'batch' mode. I believe that this mode has its own shell parsing syntax to emulate the parsing by a real Unix shell.

It would be interesting to see if using the '-escape windows' mode avoids it. Of course, Windows command shell syntax/parsing is flimsy compared with a Unix shell.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrew Dryga - 2022-01-12

@bfriesen I tried "-escape windows" with many of the options, unfortunately I did not see any difference:

$ history | grep "windows" 78 gm batch -escape windows -stop-on-error on /tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/script.gm 80 gm batch -escape windows -stop-on-error on /tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/script.gm 87 gm batch -escape windows -stop-on-error on /tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/script.gm 89 gm batch -escape windows -stop-on-error on /tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/script.gm 91 gm batch -escape windows -stop-on-error on /tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/script.gm 93 gm batch -escape windows -stop-on-error on /tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/script.gm 95 gm batch -escape windows -stop-on-error on /tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/script.gm 97 gm batch -escape windows -stop-on-error on /tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/script.gm 99 gm batch -escape windows -stop-on-error on /tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/script.gm 101 gm batch -escape windows -stop-on-error on /tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/script.gm 103 gm batch -escape windows -stop-on-error on /tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/script.gm 105 gm batch -escape windows -stop-on-error on /tmp/imaginator/photo-transforms/andrew_tmp/batch-FsmTmMP1hbLLC9gAbOWB/script.gm

So almost all of the options above are tried both on unix and windows escaping.

Last edit: Andrew Dryga 2022-01-12

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrew Dryga - 2022-01-13

@bfriesen I'm not good at languages gm is written in but I can try to take a look at that scripting language if you can point me to a place in the source code :)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Bob Friesenhahn - 2022-01-13
  
  On Thu, 13 Jan 2022, Andrew Dryga wrote:
  
  @bfriesen I'm not good at languages gm is written in but I can try to take a look at that scripting language if you can point me to a place in the source code :)
  
  That would be great. If you can find the point of the logic error,
  and see how to fix it, that would save a lot of time.
  
  The problem is almost certainly in magick/command.c and in the
  ParseUnixCommandLine() function:
  
  while (c != EOF) { switch (c) { case '\'': while((c = fgetc(in)) != '\'') { if (p >= limit ) { while ((c = fgetc(in)) != '\n'); return 0; } *p++ = c; } break; case '"': while((c = fgetc(in)) != '"') { if (c == '\\') { int next = fgetc(in); if (next != '\\' && next != '"') *p++ = c; c = next; } if (p >= limit ) { while ((c = fgetc(in)) != '\n'); return 0; } *p++ = c; } break;
  
  Bob
  
  Bob Friesenhahn
  bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
  GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
  Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Bob Friesenhahn - 2022-01-15

As far as I can see, it should not be necessary to escape $0 because batch mode does not pay attention to things which look like shell variables and it has no special support for $ at all. If this $ symbol is breaking the MIFF format then that would clearly be a bug.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Bob Friesenhahn - 2022-01-15

Likewise if image drawing/annotation is broken by a $ symbol, that would clearly be a bug.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrew, I downloaded your input file and copied your original script (no backslash) to a file. I stripped the path part of the file names so that all of the files are in the same directory. I executed the script as you described. There were no errors reported and the image appeared to render perfectly with the $ symbols as intended.

Since it works perfectly for me, I suspect that there is an issue due to font files, or your locale, or perhaps something to do with character encoding (e.g. UTF-8).

I have these settings in my environment:

LANG=en_US.UTF-8
LANGUAGE=en
LC_ADDRESS=en_US.UTF-8
LC_IDENTIFICATION=en_US.UTF-8
LC_MEASUREMENT=en_US.UTF-8
LC_MONETARY=en_US.UTF-8
LC_NAME=en_US.UTF-8
LC_NUMERIC=en_US.UTF-8
LC_PAPER=en_US.UTF-8
LC_TELEPHONE=en_US.UTF-8
LC_TIME=en_US.UTF-8

and if I execute 'gm convert -list type' I do have real font files for some of the fonts you requested, but not all of them:

Path: /usr/local/lib/GraphicsMagick-1.4/config/type-ghostscript.mgk

Name                             Family                  Style   Stretch  Weight
--------------------------------------------------------------------------------
AvantGarde-Book                  AvantGarde              normal  normal    400
AvantGarde-BookOblique           AvantGarde              oblique normal    400
AvantGarde-Demi                  AvantGarde              normal  normal    600
AvantGarde-DemiOblique           AvantGarde              oblique normal    600
Bookman-Demi                     Bookman                 normal  normal    600
Bookman-DemiItalic               Bookman                 italic  normal    600
Bookman-Light                    Bookman                 normal  normal    300
Bookman-LightItalic              Bookman                 italic  normal    300
Courier                          Courier                 normal  normal    400
Courier-Bold                     Courier                 normal  normal    700
Courier-Oblique                  Courier                 oblique normal    400
Courier-BoldOblique              Courier                 oblique normal    700
Helvetica                        Helvetica               normal  normal    400
Helvetica-Bold                   Helvetica               normal  normal    700
Helvetica-Oblique                Helvetica               italic  normal    400
Helvetica-BoldOblique            Helvetica               italic  normal    700
Helvetica-Narrow                 Helvetica Narrow        normal  condensed 400
Helvetica-Narrow-Oblique         Helvetica Narrow        oblique condensed 400
Helvetica-Narrow-Bold            Helvetica Narrow        normal  condensed 700
Helvetica-Narrow-BoldOblique     Helvetica Narrow        oblique condensed 700
NewCenturySchlbk-Roman           NewCenturySchlbk        normal  normal    400
NewCenturySchlbk-Italic          NewCenturySchlbk        italic  normal    400
NewCenturySchlbk-Bold            NewCenturySchlbk        normal  normal    700
NewCenturySchlbk-BoldItalic      NewCenturySchlbk        italic  normal    700
Palatino-Roman                   Palatino                normal  normal    400
Palatino-Italic                  Palatino                italic  normal    400
Palatino-Bold                    Palatino                normal  normal    700
Palatino-BoldItalic              Palatino                italic  normal    700
Times-Roman                      Times                   normal  normal    400
Times-Bold                       Times                   normal  normal    700
Times-Italic                     Times                   italic  normal    400
Times-BoldItalic                 Times                   italic  normal    700
Symbol                           Symbol                  normal  normal    400

I suggest running your script with 'MAGICK_DEBUG=render' in the environment and see if the 'text' commands which are printed are what you expect. Also, check if your locale or character set could be causing issues.

I am seeing render debug output like this:

09:13:52 0:0.076326  0.070u 218960 render.c/DrawImage/4830/Draw:
    text 0,114 '$336 Monthly'
09:13:52 0:0.124416  0.120u 218960 render.c/DrawImage/4830/Draw:
    text 0,193 '$0 down'
09:13:52 0:0.172647  0.170u 218960 render.c/DrawImage/4830/Draw:
    text 0,246 'For 75 months at 2.9% APR. $22,995 OAC'

Bob Friesenhahn - 2022-01-16

assigned_to: Bob Friesenhahn
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Bob Friesenhahn - 2022-01-16

If you have a MIFF file which is reported to have an invalid header (the apparent actual complaint), please gzip it and attach it to this issue so we can see what is wrong with it.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrew Dryga - 2022-01-16

Hello Bob,

We also were digging into this and confirmed that this is hard to reproduce locally (the same version of GM work just fine on my Mac), I even started to blame something with Linux Alpine and their shell and upgrading production to the latest version of it to see if the issue would be resolved.

The env of our containers is:

export LANG='en_US.UTF-8' export PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' export PWD='/opt/imaginator' export SHLVL='1' export TERM='xterm'

Corrupted miff file is attached.

gm-FsrJl2BTf-38txcAh_yB.miff
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Kryzhanovsky - 2022-01-16

Hello everyone. After a series of experiments, the following statements were obtained...

The problem is not reproduced with random pictures from the Internet. I've tried different combinations of '$0' commands with random pictures from the internet, the problem doesn't reproduce. Hence the idea that '$0' is not really the root problem.

The problem does not depend on the operating system and GM version (I tried both GraphicsMagick 1.3.36 20201226 Q16 from the package, and locally built fresh code from the repository). At least, Ubuntu and Alpine behave the same, either indicating that there is some problem with a particular image, or processing the image without error.

But with the original picture attached to the thread (https://sourceforge.net/p/graphicsmagick/bugs/659/#a2fa) everything is not so good. Firstly, even a simple conversion to another format (gm convert src.miff dst.jpg) doesn't work in Ubuntu, I get the error "gm convert: Unexpected end-of-file". Secondly, in Alpine the conversion works for some reason, but when I execute the script (which starts the thread) I get the error "convert: Improper image header".

I hope this information is of some help to you.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Bob Friesenhahn - 2022-01-16

What is the origin of this bad file? How did it come to be?
ImageMagick (as delivered by Ubuntu 20.04) does not like this file either, but it likes all of the other ones.

The MIFF format supports a sequence by concatenating MIFF "files". When it gets to the end of reading the last file in a sequence, it expects that the next character read will be an EOF character. However, it appear that a 'n' is read instead, and in fact the last character of the problematic file is an 'n'.

The big question is how this mystery 'n' got appended to this file?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Sergey Kryzhanovsky - 2022-01-16

Found it. It is about 'bold' in 'helvetica-bold' in the script. That is, I removed the 'bold' from the script and as a result I got a processed image, and not an error message in Alpina. True, the situation did not become clearer from this. :)

Last edit: Sergey Kryzhanovsky 2022-01-16

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Bob Friesenhahn - 2022-01-16

While executing the script on my machine (removing all the full path components) I did not generate/encounter any MIFFs with an extra character appended.

Regardless, I just pushed Mercurial changeset 16645:a45a3d5e77ae, which causes the MIFF reader to return what it has already read if it encounters a non-'i' character rather than an EOF. The first character of a MIFF file (and for a component of a MIFF sequence) should be an 'i' for 'id'.

It is still to be determined how this file got an extra character appended. It could have been due to GraphicsMagick or some other cause.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrew Dryga - 2022-01-17

I'm attaching full flow of transforms. What we are doing is:

["convert", "-resize", "6250x4000>", "-strip", "/tmp/imaginator/photo-transforms/andrew_tmp/fetcher-FsrqkNjbsx78txcAEsiE", "/tmp/imaginator/photo-transforms/andrew_tmp/gm-FsrqkN8a08_8txcAEsmE.jpg"]

Call to external service which takes gm-FsrqkN8a08_8txcAEsmE.jpg and returned an image remove_bg-FsrqkOBQCLL8txcAEsoE.png

["convert", "-trim", "-strip", "/tmp/imaginator/photo-transforms/andrew_tmp/remove_bg-FsrqkOBQCLL8txcAEsoE.png", "/tmp/imaginator/photo-transforms/andrew_tmp/gm-FsrqkOipU1_8txcApn0B.miff"]

["convert", "-background", "none", "-extent", "1092x819-184-307", "-strip", "/tmp/imaginator/photo-transforms/andrew_tmp/gm-FsrqkOipU1_8txcApn0B.miff", "/tmp/imaginator/photo-transforms/andrew_tmp/gm-FsrqkO2EF9v8txcApn2B.miff"]

["batch", "-stop-on-error", "on", "/tmp/imaginator/photo-transforms/andrew_tmp/batch-FsrqkPOx38f8txcApn6B/script.gm"]

gm.zip
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrew Dryga - 2022-01-17

Just for the context: the external service is an ML model which takes an image, finds a car on that image and removes all the background adding the shadow under it.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Bob Friesenhahn - 2022-01-17

This seems like a fascinating and brilliant solution.

Is there still a problem to be solved? Has it been found that GraphicsMagick is somehow writing MIFF files that it is then unable to read (due to an extra appended character)?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andrew Dryga - 2022-01-19

@bfriesen oh sorry for being confusing. I explained it all just so that you see everything we do with an image and see all input/output files so that we can identify when the image gets corrupted. I think the core issue was in our ML service and looks like the issue is not reproducing for a few days already.

Thank you for your help!

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Bob Friesenhahn - 2022-01-19

status: open --> closed-works-for-me
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Can not escape $0 in a batch command

Swiss army knife of image processing

Group

Searches

Help

#659 Can not escape $0 in a batch command

Discussion

Bob