#1517 shell command line parsing error?

[Ryo]

Hello,

I'm not sure this is a bug or not, but I get

gnuplot> ! touch hello.ps; cat hello.ps | sed 's#/a#b#g' > hello.eps
gnuplot> ! touch hello-2.ps; cat hello-2.ps | sed 's#/a#b#g' > hello-2.eps
^
expecting exponent

Gnuplot seems to be interpreting "eps" as a function name?

I was trying to find a documentation of how gnuplot processes the string after the "!" command but couldn't. I naïvely thought that the string (after the "@" substitution) should be passed to the shell unmodified . . .

Sorry if I'm missing something.

Best regards,

Ryo

[Hans-Bernhard Broeker]

Gnuplot seems to be interpreting "eps" as a function name?

No. It's interpreting "2.e" as the beginning of a floating point number, but didn't expect the exponent part to be continued as "2.eps" And the reason it's even parsing that line is that there are '#' characters in it, which have gnuplot try and isolate the comment from the rest of the line. As part of that, the line is tokenized, but 2.eps fails tokenization.

[Ryo]

I see. That seems to me to indicate that the behavior is not ideal and can be improved. As my example shows, the "#" character is a "normal" character for shell if enclosed in quotes.

I may well be wrong but I guess that gnuplot can leave "#" alone and pass the whole string to the shell because "#" is a comment character also for the shell. All gnuplot would have to do is to take care of "\" at the end of the line for line continuation.

Thank you for your response.

Ryo

[Ethan Merritt]

Just a thought -

You can avoid these issues by using

return = system("some complicated system command with # | ! and whatever")

rather than

!some complicated system command with # | ! and whatever

[Ryo]

Hello,

I don't quite know how the gnuplot developers handle bugs and request, but it seems strange to me that the existence of an alternative (the "system" function) makes the error of the "!" command "invalid".

Shouldn't this report remain open, even if its priority is low?

Or, if there is no intention of fixing the problem of the "!" command, I suppose exactly how the "!" command handles the given character string should be documented. In that case, the issue would become a "bug" in the documentation.

Regards,

Ryo

[Ethan Merritt]

I don't mind re-opening the issue, but I'm not entirely convinced there is a true bug here. I can't find any clear statement in the documentation of the expected scope of the "!" character. The docs say:

 There are two ways of spawning a shell command: using `system` command
 or via `!` ($ if using VMS). The former command takes a string as a
 parameter and thus it can be used anywhere among other gnuplot commands,
 while the latter syntax requires to be the only command on the line. Control
 will return immediately to `gnuplot` after this command is executed.  For
 example, in MS-DOS or OS/2,

The thing is, normally a gnuplot "command" only extends to the next token that matches semicolon ";" or comment delimiter "#". Placing the entire line in quotes prevents it from being tokenised, which is why system("rest of line") works. You are expecting that the presence of a "!" at the start of the line will bypass normal interpretation of the line contents. That isn't unreasonable, but it doesn't match the way gnuplot's input line parsing actually works. By the time we find that the first token is "!", the line has already been tokenised (or failed to be tokenised in this particular case).

Possibly we could short-circuit normal processing of the input if the line starts with "!" but then I think we might run into other issues. For one thing that might well prevent wrapping a long command onto subsequent lines by ending each partial line with "\". I'd have to stare at the sequence of events.

Anyhow fine, it's now open and available as a project for someone to formalize what the behaviour is expected to be.

[Ryo]

First of all, thank you for your thoughtful response.

You state that the "!" command is a regular gnuplot command and should obey the same parsing rules as regular commands do. I agree that that is one logical solution to the problem.

But, the current behavior of the "!" command does not agree with your statement. Far from it.

For example,

gnuplot> ! date; echo hello

If the "!" command were a regular gnuplot command, it would end at the semicolon and "echo hello" would be regarded as a separate gnuplot command. But, since gnuplot doesn't have a command called "echo", the above line should raise an "invalid command" error on "echo".

But the fact is that the above line just works: It makes the shell execute the "date" command and then the "echo" command. This means that gnuplot does not parse the above line in the same way as it does a regular gnuplot command.

Another example is

gnuplot> ! "date; echo hello"
sh: date; echo hello: command not found
gnuplot>

It demonstrates that gnuplot does not just pass the argument to the shell: It does some parsing.

Finally, this is a modification to my original bug report:

1) gnuplot> ! echo A/a/aaEEE | sed 's#/a#b#g'
   /AaabEEE
2) gnuplot> ! echo /Aaa/aEEE | sed 's#/a#b#g' > tmp2.eps
3) gnuplot> ! echo /Aaa/aEEE | sed 's#/a#b#g' > tmp-2.eps
                                                ^
            expecting exponent

4) gnuplot> print '#'
   #
5) gnuplot> print '#' > tmp2.eps
            undefined variable: tmp2

6) gnuplot> print '#' > tmp-2.eps
                        ^
            expecting exponent

7) gnuplot>

Please notice the inconsistency between (2) and (5).

From these observations, I think we can conclude that

a) Codes like (3) fails not because gnuplot parses the "!" command in the same way as it does regular gnuplot commands.

b) Gnuplot parses the "!" command differently than it does regular gnuplot commands.

c) There is some bug in the parser of the "!" command.

So, how to fix the bug? I can think of three different solutions.

i) Make the "!" command a regular gnuplot as you envision, in which case even the simple command like

gnuplot> ! date; echo hello

would be an error. You would have to write

gnuplot> ! "date; echo hello"

I think this is a fine solution, except you would lose backward compatibility (existing gnuplot script would cease to work).

ii) Don't parse the argument to the "!" command; just send it to the shell as is, except that gnuplot still has to handle line continuation and potentially macro expansion.

iii) Fix the bug in the current parser.

Personally, I'd like solution 2. The shell command lines in Makefile work like that and work great.

Best regards,

Ryo

[Ethan Merritt]

Clarification: The gnuplot command line input is processed in multiple stages. In one stage the line is split up into "tokens"; in a subsequent stage the tokens are interpreted sequentially by the parser. The "!" bypasses the second stage, so normal gnuplot parsing does not happen. But your problem case is failing in the first step, before parsing even starts. In other words, this isn't a problem with the parser, it's a problem with the initial handling of line input. FWIW the relevant part of the gnuplot code is in scanner.c.

[Ryo]

Thank you for correcting me. Yes, after posting my last message, I realized that I should have said that the problem seems to be in your tokenizer.

You must know this well, but some tokenizing rules of the shell are different from those of gnuplot. For example, image-ver2.eps is a valid shell token, but it consists of five tokens "image", "-", "ver2", ".", and "eps" in gnuplot. So, your tokenizer would have to know the tokenization rules of the shell to handle the "!" command correctly.

For this reason, many applications that call external interpreters avoid tokenizing the strings they send to the interpreters. They just don't want to deal with different grammars. An example is Makefile, as I mentioned before. Another is the "system" function of gnuplot, C language, Ruby, Fortran (as an extension to the language), etc.

Best regards,

Ryo

[Ethan Merritt]

The attached patch fixes your particular problem case.

But I'd like to think about it more and test further before making this change in CVS. In particular I think the syntax and scope of the string send to the shell needs to be better defined.

Should the shell escape via '!' be legal anywhere a new gnuplot command would be legal? That currently fails in a case like

if (something) {
    ! shell command A
} else {
    ! shell command B
}

because anything inside a bracketed clause is not perceived as being at the start of an input line.

What exactly is expected to happen if you have a script that says

gnuplot> ! echo foo # this is a comment, or maybe it isn't

[Ryo]

if (something) {
! shell command A
} else {
! shell command B
}

I see. That looks tough.

Perhaps mine is a stupid idea, but what about treating everything after "!" as a single token, in the same way as you tokenize a single-quote string? That is,

gnuplot>! shell command A # comment \
another shell command? # comment2

would give us two tokens

1) !

2) 'shell command A # comment another shell command? # comment2'

The character string is then given to the shell as is.

As it happens, '#' is also the comment character of the shell; as a result, the shell ignores everything after the first '#' character. This behavior happens to be consistent with the shell, in the sense that (in the bourne shell language)

c='echo hello # comment \
date # comment2'
eval $c

is equivalent to

echo hello # comment date # comment2

and so only the echo command is invoked.

As to your example

gnuplot>! echo foo # this is a comment, or maybe it isn't

I would think that we can give the shell the whole string after "!" and let the shell decide. Again, because '#' happens to be the shell comment character, your example works as most users expect.

Regards,

Ryo

[Ethan Merritt]

I have applied the patch attached above to CVS for 5.0 and 5.1 but do not plan to do anything else at this time.