gnuplot-beta — Developers and beta testers of new gnuplot code

Re: Main branch core dump in fillbetween.dem

[Ethan M. <merritt@u.washington.edu>]

On Wednesday 21 July 2004 04:44 am, Hans-Bernhard Broeker wrote:
> 
> Well, to put it none too subtly: terminal 'dumb' doesn't support PM3D
> function filled_polygon, so graphics.c:1810 is calling NULL pointer.
> Instant crash guaranteed.

I will fix the crash.  But what behavior is preferred in this case?

approximate with rectangles or yerrorbars rather than polygons
(consistent with documentation),

plot one or both of the curves with lines only
(consistent with previous behaviour of filledcurves although
not what the docs say),

or

int_warn(NO_CARET,"plotting style not available for this terminal")


-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Ethan M. <merritt@u.washington.edu>]

[Another long post. Please read it in the hope that all will
become clear.  Or at least become clearer than it is at
present.]

Let me start with a plea to drop the term "late evaluation",
which I think is side-tracking the current discussion.
Whatever it may mean to different people, I don't think it is
a useful way of thinking about my patchsets.

The evaluation of expressions containing string functions is no
different from the evaluation of all other gnuplot functions.
The evaluation is never "late"; it happens when it happens.

Consider the existing case with no strings:

# user function is defined
   f(x) = sin(x)    
# user function is evaluated and result goes in var1
   var1 = f(3.0)
# user function is stored with plot for use in plotting,
# evaluation happens at plot time
   plot 'data' using 1:(f($2))

In the first two cases the function is evaluated at the
time the command line is parsed.
In the third case the function is evaluated once per data
point, but not until the file is read in during plotting.
This is obvious, I hope.

So....
What I did was expand the variety of functions to include
string-valued functions.  But otherwise the evaluation works
the same way

# (1) User function is defined
   f(x) = sprintf("%7.5f",x)
# (2) Pre-defined function is evaluated and result stored
   var1 = sprintf("%2.1f",x)
# (3) Same thing using user function instead
   var2 = f(x)
# (4) User function is stored with plot for use in plotting
   plot 'data' using 1:2:(f($2))

(1) Needs a bit more code to make it work, but is straightforward
(2) This is, I thought, what everyone was asking for but now we
seem to be arguing about
(3) Combination of (1) and (2); evaluation is done in two steps but
still at the time the command line is parsed.
(4) Evaluation of f(x), which is really sprintf("fmt",x) in this case,
does not happen until the data is read in.   In fact the command
I show here as an example is currently broken because of issues
with assumed column contents (see other thread).  But I was 
planning to fix this for the next version of the patchsets.

The point is that in each case, string or no string, the evaluation 
happens at the time it is needed. For 'set <foo>' commands 
this is usually at the time the command line is parsed, but for
'plot/splot/fit' commands it happens elsewhere.  

That inevitably leads to the question "how does gnuplot know that
evaluation is needed?", and that's where the fun (or argument) started.
In all the examples I gave above it is conceptually all the same
whether the variables and functions take on int- or real- values or
whether they take on string values.

But there are a few operations where the two possibilites diverge.
For example, real-valued variables are used (and changed) during
fitting.  I can't imagine that being a meaningful operation for a 
string variable.

Conversely, one thing we do with strings is to print them,
something that doesn't happen to numbers or at least only via
an intermediate string.  Printing a string constant is easy; we do
that already.  But how do you store, or mark, a string-valued
function so the gnuplot knows it must be evaluated and printed?
That is the heart of this discussion.   

I picked on up a suggestion from Hans to mark a string-valued
function by inserting some "magic characters" inside the quotes.
I thought that was a clever idea because it is transparent to the
existing code.  I am bemused by the controversy this seems to
have generated.  

There are two issues:  
how it looks on the command line, and how it is stored internally.

Look, I really don't care how we do it.  This seemed to me a clever
representation.  In addition to being compatible with the 
current code, it means that the internal representation is the
same as the command-line representation (quote + magic chars).
But these are mere conveniences, and are totally separate from
the core functionality.  Feel free to propose other represenations,
and explain why they are better.



And now, finally to Volker's specific questions....

On Wednesday 21 July 2004 12:33 am, Volker Dobler wrote:
> I think 98 sites which do memory allocation themself is more
> dangerous than replacing 98 call sites with one tested and
> correct call to m_quote_capture.

I have no comment at this time.
I will revisit your earlier patch and see what you did there.
Are you willing to help fix things if this mass substitution 
breaks them?

> Regarding the discussion about late evaluation:  I would like
> to _object_ to late evaluation.  Late evaluation will intruduce
> much more problems in user space than it will solve.

Petr said this too.  I am totally not understanding this 
concern.  But then again, I don't even know what you think
"late evaluation" means. 
Could you give an example?

> If I construct a string (regardless if done by the '' + trick or sprintf
> or anyting else) I expect it to be the way I constructed it;
> at least that's the behavior of all programming languages I am
> familiar with.

Yes.  That's fine.  That's what the sprintf() function does -
it constructs a string permanently and stores it in a variable.
That variable always contains the string you constructed.
It does not change.  Ever.

The point which is confusing is that string may in fact contain a
command.  But even this is not new - the hot-key bindings are
strings that contain commands. The command they contain is
executed "when needed", which in the case of key bindings
is in response to an event generated externally.
If you type 'bind' it will print a list of these stored commands.
It's the same with the string variables. 
If you type 'show VAR' it will show you what command is stored
there.  But there are other places in the code that can actually
execute that command when needed.

Users are under no obligation to store commands in string
variables any more than they are required to program new hot-key
bindings.  It's just something you *can* do if it proves useful.
  

-- 
Ethan A Merritt

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Ethan M. <merritt@u.washington.edu>]

On Wednesday 21 July 2004 02:29 am, mi...@ph... wrote:
> > Regarding the discussion about late evaluation:  I would like
> > to _object_ to late evaluation.  Late evaluation will intruduce
> > much more problems in user space than it will solve.

Like what, for example?

So far I don't know of any bad side to this capability.
Sure, most code won't use it - but that is the case for many
features.  And the beauty of this way of doing things is that
it is actually simpler to handle the general case than it is
to add a lot of special cases.

> The late evaluation can be just a flag to "set label". That's where we
> expect this feature to happen most frequently.
> set label 1 sprintf("fitted values a=%g b=%g") at 1,1 lateeval

Yes, that would be possible.  But why only labels?
Why not plot titles, key titles, fit logfile names, and all the
rest?


-- 
Ethan A Merritt

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Ethan M. <merritt@u.washington.edu>]

On Wednesday 21 July 2004 03:37 am, Dave Denholm wrote:
> This may be a stupid question, but does this syntax extend to
> arbitrary expressions, or just variable names ?
>
>   filename = 'sprintf("foobar%d.ps", i+1)'
> or
>   maptofilename(i)=i+1
>   filename = 'sprintf("foobar%d.ps", maptofilename(i))'

Both of those would work. 
This is a new capability added to the existing expression evaluation
code.  It still handles everything it already handled, but now it
knows how to handle at least some operations on strings also.

-- 
Ethan A Merritt

Re: Main branch core dump in fillbetween.dem

[Hans-Bernhard B. <br...@ph...>]

On Wed, 21 Jul 2004, Lars Hecking wrote:

>  I often use the "dumb" terminal with make check as some form of benchmark:
[...]
> 
> #0  0x00000000 in ?? ()
> #1  0x00059864 in finish_filled_curve (points=4, corners=0xffbfe178, 
>     plot=0x1d6960) at graphics.c:1810
> #2  0x0005d1f4 in fill_between (x1=10, yl1=280, yu1=16.733201000000001, 
>     x2=20, yl2=191, yu2=13.820275000000001, plot=0x1d6960) at graphics.c:2089

Well, to put it none too subtly: terminal 'dumb' doesn't support PM3D
function filled_polygon, so graphics.c:1810 is calling NULL pointer.
Instant crash guaranteed.

Ethan?

>  Btw. have we officially ditched the requirement that filenames must be
>  8+3 chars max?

Not officially, but we've let it happen.  And renaming in CVS is just
painful enough to let them stick.

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Main branch core dump in fillbetween.dem

[Lars H. <lhe...@us...>]

 I often use the "dumb" terminal with make check as some form of benchmark:

$ time GNUTERM=dumb make check >/dev/null 2>&1

 This fails with the current cvs head:

...
          @@@@
         @@  @@
        @gnuplot
      @@  ****  @@
     @@***    ***@@
    @**          **@
  @@**            **@@
  @** $$$      $$$ **@
   *  $ $      $ $  *
  *   $$   >    $$   *
  *         >        *
  *         >        *
  *   *      >   *   *
  **  **        **  **
   *    **    **    *
   **    gnuplot   **
     *    ****    *
      ***      ***
         ******


Press Return to continuemake[2]: *** [check-noninteractive] Error 139
make[2]: Leaving directory `/home/lhecking/src/cvs/gnuplot/demo'
make[1]: *** [check-am] Error 2

 A recompile with -g later, I get a core cump and the following backtrace:

#0  0x00000000 in ?? ()
#1  0x00059864 in finish_filled_curve (points=4, corners=0xffbfe178, 
    plot=0x1d6960) at graphics.c:1810
#2  0x0005d1f4 in fill_between (x1=10, yl1=280, yu1=16.733201000000001, 
    x2=20, yl2=191, yu2=13.820275000000001, plot=0x1d6960) at graphics.c:2089
#3  0x0005b494 in plot_betweencurves (plot=0x1d6960) at graphics.c:1998
#4  0x00056ef0 in do_plot (plots=0x1d6960, pcount=3) at graphics.c:1512
#5  0x000ab114 in eval_plots () at plot2d.c:1986
#6  0x000a0b70 in plotrequest () at plot2d.c:248
#7  0x00027820 in plot_command () at command.c:1045
#8  0x00025d04 in command () at command.c:511
#9  0x000255c4 in do_line () at command.c:370
#10 0x00090fd8 in load_file (fp=0x1cab18, name=0x1d0600 "fillbetween.dem", 
    can_do_args=0 '\0') at misc.c:269
#11 0x0002720c in load_command () at command.c:871
#12 0x00025d04 in command () at command.c:511
#13 0x000255c4 in do_line () at command.c:370
#14 0x00090fd8 in load_file (fp=0x1cab08, name=0xffbff1b0 "all.dem", 
    can_do_args=0 '\0') at misc.c:269
#15 0x0009f448 in main (argc=1, argv=0xffbff020) at plot.c:622

 It works fine with the x11 terminal. I can reproduce this on Solaris and
 Linux, so it's not an OS specific issue.

 To recreate it non-interactively:

$ cd demo
$ GNUPLOT_DIRVER_DIR=`pwd`/../src GNUTERM=dumb ../src/gnuplot
...
Terminal type set to 'dumb'
gnuplot> lo "fillbetween.dem"
Segmentation Fault (core dumped)
$ 

 Btw. have we officially ditched the requirement that filenames must be
 8+3 chars max?

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Dave D. <dde...@es...>]

Ethan A Merritt <merritt@u.washington.edu> writes:

>> So far, variables stored values, not expressions to be evaluated at
>> some later time.  If at all possible, I'ld like to keep it that way for
>> strings, if only to minimize the amount of documentation we and the users
>> will need to fully explain all this.
>
> I disagree, and I think the record of requests for enhancement
> support me on this point.  People really want a way of embedding 
> variables into strings, and having the current value of the variable
> substituted in at the time the string is printed.  It doesn't matter
> whether you call that "re-evaluation of a string" or "storing a function
> instead of a string", or "storing an expression to be evaluated later",
> that is the desired capability.
>

There is a demand for some way of making a plot with labels showing
the current value of a variable. I don't think this necessarily
translates into a need for dynamic substitution at plot time.
(But I think that point has already been made)


>> I'm opposed to having sprintf() *inside* the quotes.  It causes new
>> problems like the '' vs. "" saving issue that we don't really need, for no
>> real gain.
>
> On the contrary, it's a huge gain.  It means that a very powerful
> ability is introduced in a uniform way, yet requires minimal or no
> change to the existing code or to the existing storage mechanisms
> for strings.
>
> Anything you do _outside_ the quotes means that every single place
> that tests for a string constant has to be re-written to handle the
> possible substitution of other syntactic entities instead.
>


In any big program, there comes a time for refactoring. It seems to me
that, at the start of a development phase, the fact that a feature can
be inserted with minimal code impact does not *necessarily* mean it is
the best way to do it. (And even at the end of a coding cycle, it may
still be better to leave it out than to distort the syntax or limit
the future possibilities. I'm speaking generally here, not about this
change in particular.



On the subject of requested features / large changes...

is there a case for introducing array variables which can be read from
a file ?  Or is this starting to tread on octave's territory ?

Loading up a file as a 2-d array of strings (which can be converted to
numbers), and being able to plot from an array, may unblock a number
of problems.


dd
-- 
Dave Denholm              <dde...@es...>       http://www.esmertec.com

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Dave D. <dde...@es...>]

Ethan A Merritt <merritt@u.washington.edu> writes:

> On Sunday 18 July 2004 07:03 am, Hans-Bernhard Broeker wrote:
>> > 3) user-defined string-valued functions.
>>
>> Users will almost certainly want to be able to do something like this:
>>
>> 	filename(i)=sprintf("foobar%d.ps", i)
>>
>> 	i=15
>>
> But that example does not require a user-defined function.
> That is the behaviour you would get anyway, courtesy of
> the automagic string evaluation code already written.
>
> filename = 'sprintf("foobar%d.ps",i)'


Sorry, I'm keeping up with this topic...

This may be a stupid question, but does this syntax extend to
arbitrary expressions, or just variable names ?

  filename = 'sprintf("foobar%d.ps", i+1)'

or

  maptofilename(i)=i+1
  filename = 'sprintf("foobar%d.ps", maptofilename(i))'

dd
-- 
Dave Denholm              <dde...@es...>       http://www.esmertec.com

Re: [ gnuplot-Patches-992149 ] String variables revisited

[<mi...@ph...>]

> Regarding the discussion about late evaluation:  I would like
> to _object_ to late evaluation.  Late evaluation will intruduce
> much more problems in user space than it will solve.

The late evaluation can be just a flag to "set label". That's where we
expect this feature to happen most frequently.
set label 1 sprintf("fitted values a=%g b=%g") at 1,1 lateeval

Or, there could be "sprintfLate", but to be authorized only for strings
for "set label", "set title", "(s)plot"... i.e., where the string is used
immediately.
---
PM

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Volker D. <v.d...@we...>]

Ethan Merritt wrote:
> Consider a typical call site, one of the 98 you counted:
> 
> set.c (set_fontpath) line 1801:
> 
> if (isstring(c_token)) {
> int len;
> char *ss = gp_alloc(token_len(c_token), "tmp storage");
> len = (collect? strlen(collect) : 0);
> quote_str(ss,c_token,token_len(c_token));
> collect = gp_realloc(collect, len+1+strlen(ss)+1, "tmp fontpath");
> 
> I am not clever enough to re-write isstring() and quote_str() such
> that the code at this call site works when the length of the string
> changes mid-stream. So trying to be clever in isstring() and
> quote_str() paradoxically makes *more* work, since every call site
> would have to be inspected and re-written for compatibility.
> 
Yes, but this I think this is easy:  I took a look at my old `value
substitution` patch (the python like syntax of "%(foo).2f"):
There are two functions: quote_str and m_quote_capture.
Using quote_str you will have to do memory allocation yourself,
whereas m_quote_capture will do allocation for you if I remember
correctly.  My old patch replaced (hopfully) all the invocations
of quote_str with m_quote_capture and did substitution there. 

> If the program were being re-written from scratch, then yes.
> But in the interest of sanity and not introducing 98 possible
> new bug sites, I would rather go for a solution that leaves the
> input code intact ...
>
I think 98 sites which do memory allocation themself is more
dangerous than replacing 98 call sites with one tested and
correct call to m_quote_capture.

Regarding the discussion about late evaluation:  I would like
to _object_ to late evaluation.  Late evaluation will intruduce
much more problems in user space than it will solve.  If I
construct a string (regardless if done by the '' + trick or sprintf
or anyting else) I expect it to be the way I constructed it;
at least that's the behavior of all programming languages I am
familiar with.  I do not see any problem which could be solved
easier with late evaluation, at least not from user side:  It is
very easy to reconstruct the string if a variable change should
be reflected in the string.

IMHO the substitution should be done right during parsing,
no late evaluation and m_quote_capture is a good place for
it.

Volker
 
--
Volker Dobler
_______________________________________________________
WEB.DE Video-Mail - Sagen Sie mehr mit bewegten Bildern
Informationen unter: http://freemail.web.de/?mc=021199

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Ethan M. <merritt@u.washington.edu>]

On Tuesday 20 July 2004 01:53 am, Hans-Bernhard Broeker wrote:
> > (3) It adds a check for this string-valued function in exactly two
> > places where a function name was not previously accepted by
> > the parser. 
> 
> I think what I'm getting at that this check had better go into isstring()
> instead --- the function that every command parser fragment is supposed to
> be using to check whether the upcoming command line token is a string or
> not.

That turns out not to help.
Consider a typical call site, one of the 98 you counted:

set.c (set_fontpath) line 1801:

        if (isstring(c_token)) {
            int len;
            char *ss = gp_alloc(token_len(c_token), "tmp storage");
            len = (collect? strlen(collect) : 0);
            quote_str(ss,c_token,token_len(c_token));
            collect = gp_realloc(collect, len+1+strlen(ss)+1, "tmp fontpath");
            if (len != 0) {
                strcpy(collect+len+1,ss);
                *(collect+len) = PATHSEP;
            }
            else
                strcpy(collect,ss);
            free(ss);
            ++c_token;
	}

I am not clever enough to re-write isstring() and quote_str() such
that the code at this call site works when the length of the string
changes mid-stream.  So trying to be clever in isstring() and
quote_str() paradoxically makes *more* work, since every call site
would have to be inspected and re-written for compatibility.
 
If the program were being re-written from scratch, then yes.
But in the interest of sanity and not introducing 98 possible
new bug sites, I would rather go for a solution that leaves the
input code intact and focuses instead on identifying a small
number of places where the string is actually used for
something.

case 1: the string is printed. I dealt with this by adding code to
write_multiline().

case 2: the string is used as a file name.   I haven't handled this
yet, but I propose to create a new routine gp_fopen() which 
contains the same check for magic characters at the start of a
string that I inserted into write_multiline().  Yes, this requires
changing call sites from  fd = fopen("...") to fd = gp_fopen("...")
but this would be a piece of cake compared to rethinking and
rewriting 98 sites like the one above .

case 3: Are there any more end-uses for a string?


> Ah, thanks, now this is a whole lot clearer than before.  So this patch is
> what I've been referring to as late evaluation, i.e. the technique of
> storing an expression, rather than its result, to be evaluated at the
> latest possible time.  I'm still not confinced that 'sprintf( is a
> suitable choice of trigger string, but now at least I understand what
> you're doing.

I am not trying to convince anyone that "sprintf" is the best
choice.  The only virtue it has is that we can refer people to
"man sprintf" for help with the format specifiers. I would rather
have either an alternate quote character (something different
from ' or ") or a single magic character following a normal
quote character.

	set title %<expression>%
or
	set title "!<expression>"

Unfortunately the '!' character is a unary operator already,
as are '+' and '^' and many of the other obvious candidates.
So sticking it in front of a general expression is at best a bit
confusing, and at worst ambiguous to parse. I hesitate to
suggest '$', but at least it is not a unary operator.


The other alternative is a pseudo-function such as
	set title %(<expression>)
but that leads us right back to the problem of those 98
call sites that are expecting a quote character.


Please note that this doesn't get away from having to type
"sprintf" if that is in fact the function you want to evaluate.
So even with a single-%-character quoting convention, the
full form would be
	set title %sprintf("<format>",var1,var2)%

but if we allow other string-valued functions, then any one
of them might replace "sprintf" in this example.
Hypothetically it would look like
	set title %myfunc(par1,par2)%


-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Hans-Bernhard B. <br...@ph...>]

On Mon, 19 Jul 2004, Ethan Merritt wrote:

> > In short, all 98 calls of isstring(), and all 19
> > calls of m_quote_capture and all 62 of quote_str()
> 
> No.  You are misunderstanding, or I am totally failing to describe
> things properly.  

A combination of both, it appears.  I had lost overview somewhere along
the line of this, and you didn't see it happen.

> Patchset 1:
> -----------------
> 
> This patchset does 3 things.
[...]

> (3) It adds a check for this string-valued function in exactly two
> places where a function name was not previously accepted by
> the parser. 

I think what I'm getting at that this check had better go into isstring()
instead --- the function that every command parser fragment is supposed to
be using to check whether the upcoming command line token is a string or
not.

> Patchset 2
> ---------------
> 
> Adds a string-function evaluator. I don't know what the right term is
> for this, but basically it just triggers the existing function
> evaluation code. It is directly modelled on do_line(), except that it
> requires the top level function being evaluated to return a string.
> 
> At a single call site in write_multiline(), it checks for a magic
> leading sequence of characters in every string that is printed. If the
> sequence is recognized, it filters the string through the
> string-function evaluator.

Ah, thanks, now this is a whole lot clearer than before.  So this patch is
what I've been referring to as late evaluation, i.e. the technique of
storing an expression, rather than its result, to be evaluated at the
latest possible time.  I'm still not confinced that 'sprintf( is a
suitable choice of trigger string, but now at least I understand what
you're doing.

I.e. we're actually in perfect agreement as to what the necessary
features are --- we just used so completely different terms that we
confused each other completely.  Thanks for staying with me long enough
that we could finally sort this out.

> Possible next steps (hypothetical patchset 3)
> -------------------------------------------------------
> 
> Indeed there are other places that the current syntax does not
> allow a function name including, as you point out, the specification
> of file names.  This could be addressed in several ways.
> 
> (1) We can expand the legal syntax at these places one by
> one as they are needed or requested.

That might be an endless journey.  Better do it in a more brutal fashion:
either by documenting and recommending the ''+ trick to signal a string
expression coming up, or by changing isstring() and friends, as I
suggested earlier.

> (2) We could replace fopen() with an newly-defined gp_fopen() and
> add the string evaluation code only in the new routine.

> No comment on the DOS/Windows issue, but from my unix-centric
> perspective this is not at all what the single-quote convention is 
> expected to do.   Following the conventions of both sh- and csh-
> derived shells,  enclosing a character string inside single quotes
> should mean that absolutely nothing at all is done to it. 

And so far, before your second patchset, nothing is --- that patchset
actually breaks that rule.  Single quoted string are used as-is,
unmodified.  Except that when they're written out to save files, which may
not even stay on the same platform as the gnuplot executable they were
built on, it's unreliable to store them as single-quoted ones.  That's
where conv_text() comes in and generates an ASCII-only, portable
representation for us. The actual problem here was simply a bug in
conv_text().

This save file issue is one which we can get no guidance from Unix shells
from --- they don't save states of internal settings to files meant to be
read back in.

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Ethan A M. <merritt@u.washington.edu>]

On Monday 19 July 2004 02:18 pm, Ethan Merritt wrote:
>
> Please try it out.  It catches those already except for the tic labels and
> key entries, which ought to go through write_multiline() but don't
> currently. I'll add that to the patch, or just add it to cvs separately.

Heh. I was mis-remembering.  Most of the tic labels do already use
write_multiline(). It's only the colorbox tics that bypass it for some
reason. So that and the key entries are the missing pieces I can find.



-- 
Ethan A Merritt
Department of Biochemistry & Biomolecular Structure Center
University of Washington, Seattle

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Ethan M. <merritt@u.washington.edu>]

> > I may be missing some major class of possible uses for strings,
> > but it seems to me that about covers it right there.  What else did
> > you have in mind?
> 
> Every usage of strings anywhere in gnuplot.  Datafile names, output file
> names, 'print' strings, plot elements (axis labels, tick labels, labels,
> title, key title, key entries), 

Please try it out.  It catches those already except for the tic labels and
key entries, which ought to go through write_multiline() but don't currently.
I'll add that to the patch, or just add it to cvs separately.

> In short, all 98 calls of isstring(), and all 19
> calls of m_quote_capture and all 62 of quote_str()

No.  You are misunderstanding, or I am totally failing to describe
things properly.  The great thing about this approach, which you
suggested yourself, is that none of those need to be changed.

> fit 'update' and 'via' files, save/load/call file names, 'cd' names,
> loadpath names, terminal-wide or label-wise font names.

I agree that filenames are not yet handled in the patchsets.
I was not trying for 100% complete coverage in the first go-round.

I will describe the patchsets from a new angle; let's see if 
I can make it more understandable this time:

Patchset 1:
-----------------

This patchset does 3 things.

(1) It adds STRING as a legal "value"  (gp_types.h data structure)
and overloads concatenation onto the + operation applied to STRINGS.

(2) It adds a single string-valued function, sprintf(...).
This works automagically anywhere that the gnuplot parser accepts
a function name. So simple assigment statements, like
	LABEL = sprintf("whatever",var1,var2,var3)
just work, with no new code needed.

(3) It adds a check for this string-valued function in exactly two
places where a function name was not previously accepted by
the parser. These are
	set label <something>
and
	set [xyz...]label <something>

The latter routine handles titles as well, so the coverage is
more complete than you might think at first blush.


Patchset 2
---------------

Adds a string-function evaluator. I don't know what the right term
is for this, but basically it just triggers the existing function evaluation
code. It is directly modelled on do_line(), except that it requires
the top level function being evaluated to return a string.

At a single call site in write_multiline(), it checks for a magic
leading sequence of characters in every string that is printed.
If the sequence is recognized, it filters the string through the
string-function evaluator.

This is a beautifully simple change.  At one shot it implements
variable substitution into all strings printed via write_multiline().
It also has the effect that the *current* value of variables is
substituted in, which is a major bonus in its own right.   This
is what nearly everyone expects gnuplot to do now, but it
doesn't.


Possible next steps (hypothetical patchset 3)
-------------------------------------------------------

Indeed there are other places that the current syntax does not
allow a function name including, as you point out, the specification
of file names.  This could be addressed in several ways.

(1) We can expand the legal syntax at these places one by
one as they are needed or requested.  Here is the code that
went in for labels; other places would be more or less the same:

+#ifdef GP_STRING_VARS
+    /* Allow creation of label text using sprintf() */
+    if (equals(c_token,"sprintf")) {
+       struct value a = {STRING,{NULL}};
+       (void) const_express(&a);
+       this_label->text = a.v.string_val;
+    } else
+#endif
+
+    /* get text from string */
     if (!END_OF_COMMAND && isstring(c_token)) {

(2) We could replace fopen() with an newly-defined gp_fopen() and
add the string evaluation code only in the new routine.

(3) We could instead use the approach of my older "userstrings" patch,
which essentially implements command-line macro definitions. This would
not require changing the  individual parsing fragments, but would open
up a new front in the argument^h^h^h^h err.. discussion.
Since the macro-substitution is done on the input line as a text string,
it is independent of the parsing routines that will later come into play.


> I honestly don't see the need for late evaluation from the user interface
> side of things. I don't see a compelling reason why
> 
> 	set title {some expression involving variable i}
> 	i = 5
> 	plot something
> 	i = 6
> 	plot something else
> 
> has to produce two different title strings, without even offering the
> option of getting the same title on both of them.  

Straw man argument.  There are plenty of options for having the
title come out the same.  What we are talking about is how to get
it to come out differently.

If you want a more obvious, frequently requested, example:

set title 'sprintf("Fit cycle %d finished at `date`, A = %7.4f B = %7.4f", \
			ncyc,A,B)'
n = n+1; fit f(x) 'data' via A,B; plot f(x)
n = n+1; fit f(x) 'data' via A,B; plot f(x)
<wash, rinse, repeat as often as you like>


> 	x = 6
> 	set label 'sprintf("%g", x)' at x, f(x)
> 	x = 1
> 
> show the label at position (6, f(6)), but print the string as '1.0'?
> From where I sit, that makes no sense whatsoever.

Well, so don't do that.  There is no requirement that you have to
select plot-time evaluation of strings if it doesn't make sense.

> So here's a new summary of the matter: late evaluation should be a 
> subject kept separate from that of string variables.

That's why I split things into two patchsets. 
But I hardly think they are independent. 

> If we add late evaluation, it should be added it for both strings and numbers.

Please. Just try the patchsets.   
It does work for both strings and numbers, at least within the realm
it is trying to address.  I'm sure you can come up with things it 
doesn't do at all, but that by itself is not a good argument against
using it for the things it *does* do.




And on the tangential matter of quote styles:

> > [aside:  About the only thing I use the existing single-quote mode for
> > is to allow inclusion of double-quotes in the string without having to
> > escape them with backslashes.  But I don't think that was the original
> > intent. What else is it useful for?]
> 
> E.g. for input of strings in terminals like LaTeX or PostScript enhanced,
> where you really don't want to have to type every \ character twice, and
> for DOS/Windows filenames, where \n processing would produce rather
> surprising results.

No comment on the DOS/Windows issue, but from my unix-centric
perspective this is not at all what the single-quote convention is 
expected to do.   Following the conventions of both sh- and csh-
derived shells,  enclosing a character string inside single quotes
should mean that absolutely nothing at all is done to it.  No fiddling
with back-slashes, no substitution of variables, no execution of
shell escapes.  IMHO gnuplot should do the same thing - save the
single-quoted string as is with no fiddling.   Right now gnuplot
takes the more complicated and error-prone route of trying to
figure out what would have been required to type the string as a
double-quoted string instead, and saving that.  For what gain?
It should just save the string as entered, with a flag that it was 
given in single quotes.

-- 
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center
Mailstop 357742
University of Washington, Seattle, WA 98195

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Hans-Bernhard B. <br...@ph...>]

On Mon, 19 Jul 2004, Ethan Merritt wrote:

> On Monday 19 July 2004 01:35 am, you wrote:
> > On Sun, 18 Jul 2004, Ethan A Merritt wrote:
> > > Anything you do _outside_ the quotes means that every single place
> > > that tests for a string constant has to be re-written to handle the
> > > possible substitution of other syntactic entities instead.
> >
> > And everything you do inside them means you have to handle the
> > substitution at every single point of the code that actually uses a
> > string.  That's not necessarily a much smaller set of places.  It could
> > actually be a good deal larger.
> 
> Have you actually tried the patchset?

Not really.  I've relied on your descriptions of it for now.

> Adding the test+evaluation in one place only, the text-printing
> routine write_multiline(), already catches a large majority of 
> useful cases.  Adding it to the file-open code (not sure exactly
> how many places that is) would catch most of the rest.

Catching most isn't the issue.  Catching all of them is.

> I may be missing some major class of possible uses for strings,
> but it seems to me that about covers it right there.  What else did
> you have in mind?

Every usage of strings anywhere in gnuplot.  Datafile names, output file
names, 'print' strings, plot elements (axis labels, tick labels, labels,
title, key title, key entries), fit 'update' and 'via' files,
save/load/call file names, 'cd' names, loadpath names, terminal-wide or
label-wise font names. In short, all 98 calls of isstring(), and all 19
calls of m_quote_capture and all 62 of quote_str()

> > It could almost certainly all be handled by extending m_quote_capture
> > to proceed evaluating a string-valued expression.
> 
> That is the second stage, yes, but the first stage is teaching all the
> parsing routines to accept something other than a quoted string in
> all the places they currently expect it.

Not really.  The trick I have in mind is to teach m_quote_capture()  and
friends themselves to accept a quoted string followed by whatever else
there is to the string-valued expression it started off.  Just as we
currently parse the function to be plotted by having the parser eat up
as much of the command line as fits the syntax of an expression, a 
string would continue until the next piece of syntax doesn't match the 
syntax of a string expression any longer.  

> > > Assume for the moment we use % for this purpose. Then the
> > > documentation would read:
> > > 	"<expression>"	evaluate immediately, with substitution
> > > 	'<expression>'		evaluate immediately, no substitution
> > > 	%<expression>%	evaluate later, with substitution at that time
> >
> > I honestly don't see why we need the middle variant.  What is the
> > difference between evaluation and substitution that makes you want
> > control over each of them independently?
> 
> ???
> That's what we have *now*.   You want to remove it?

No.  But you had me confused over what your terminology means there.

Now, looking at all this a bit closer, it seems like you're understanding
"substitution" to mean the stuff we already have for the first two cases
(\n, \0123, `backquotes`, ...), and "evaluation" for the new string stuff.  

Well, so far in gnuplot, late evaluation has been limited to the
definition of user-defined functions --- all other expressions are
evaluated to fixed results before the command they appear in is finished
executing.  That method has served us well so far, and before we stray
from that path, we should have a solid reason for doing so.

I honestly don't see the need for late evaluation from the user interface
side of things. I don't see a compelling reason why

	set title {some expression involving variable i}
	i = 5
	plot something
	i = 6
	plot something else

has to produce two different title strings, without even offering the
option of getting the same title on both of them.  Even closer to the
point, why should

	x = 6
	set label 'sprintf("%g", x)' at x, f(x)
	x = 1

show the label at position (6, f(6)), but print the string as '1.0'?
From where I sit, that makes no sense whatsoever.

So here's a new summary of the matter: late evaluation should be a 
subject kept separate from that of string variables.  If we add
late evaluation, it should be added it for both strings and numbers.

> [aside:  About the only thing I use the existing single-quote mode for
> is to allow inclusion of double-quotes in the string without having to
> escape them with backslashes.  But I don't think that was the original
> intent. What else is it useful for?]

E.g. for input of strings in terminals like LaTeX or PostScript enhanced,
where you really don't want to have to type every \ character twice, and
for DOS/Windows filenames, where \n processing would produce rather
surprising results.  Although the latter can be circumvented by using / to
separate directories, which works just as well, but most Windowsers don't
know that.

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Ethan M. <merritt@u.washington.edu>]

On Monday 19 July 2004 01:35 am, you wrote:
> On Sun, 18 Jul 2004, Ethan A Merritt wrote:
> > Anything you do _outside_ the quotes means that every single place
> > that tests for a string constant has to be re-written to handle the
> > possible substitution of other syntactic entities instead.
>
> And everything you do inside them means you have to handle the
> substitution at every single point of the code that actually uses a
> string.  That's not necessarily a much smaller set of places.  It could
> actually be a good deal larger.

Have you actually tried the patchset?
Adding the test+evaluation in one place only, the text-printing
routine write_multiline(), already catches a large majority of 
useful cases.  Adding it to the file-open code (not sure exactly
how many places that is) would catch most of the rest.
The minority of text-printing that does not go through write_multiline(),
with tic labels being the prime example, should be converted to do so.
That would at the same time address several queries about why
some of the current text options don't work for tic labels.

I may be missing some major class of possible uses for strings,
but it seems to me that about covers it right there.  What else did
you have in mind?

> It could almost certainly all be handled by extending m_quote_capture
> to proceed evaluating a string-valued expression.

That is the second stage, yes, but the first stage is teaching all the
parsing routines to accept something other than a quoted string in
all the places they currently expect it.

> > Assume for the moment we use % for this purpose. Then the
> > documentation would read:
> > 	"<expression>"	evaluate immediately, with substitution
> > 	'<expression>'		evaluate immediately, no substitution
> > 	%<expression>%	evaluate later, with substitution at that time
>
> I honestly don't see why we need the middle variant.  What is the
> difference between evaluation and substitution that makes you want
> control over each of them independently?

???
That's what we have *now*.   You want to remove it?

It's the third variant I'm trying to add - evaluation deferred until
plot time.

[aside:  About the only thing I use the existing single-quote mode for is to
allow inclusion of double-quotes in the string without having to escape
them with backslashes.  But I don't think that was the original intent.
What else is it useful for?]



-- 
Ethan A Merritt

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Hans-Bernhard B. <br...@ph...>]

On Sun, 18 Jul 2004, Ethan A Merritt wrote:

> Anything you do _outside_ the quotes means that every single place
> that tests for a string constant has to be re-written to handle the
> possible substitution of other syntactic entities instead.

And everything you do inside them means you have to handle the
substitution at every single point of the code that actually uses a
string.  That's not necessarily a much smaller set of places.  It could
actually be a good deal larger.

It could almost certainly all be handled by extending m_quote_capture
to proceed evaluating a string-valued expression.

> Assume for the moment we use % for this purpose. Then the
> documentation would read:
> 	"<expression>"	evaluate immediately, with substitution
> 	'<expression>'		evaluate immediately, no substitution
> 	%<expression>%	evaluate later, with substitution at that time

I honestly don't see why we need the middle variant.  What is the
difference between evaluation and substitution that makes you want
control over each of them independently?

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: [ gnuplot-Bugs-993127 ] Difference between ' and " not maintained internally

[Hans-Bernhard B. <br...@ph...>]

On Sun, 18 Jul 2004, Ethan A Merritt wrote:

> On Sunday 18 July 2004 02:42 pm, Hans-Bernhard Broeker wrote:

> > Can you demonstrate an actual difference in behaviour from CVS
> > gnuplot caused by this?

> Sure. This is a stupid example, but it's very simple:
[...]

Ah, yes.  But that can just as easily be discribed as a bug in the
re-quoting routine conv_text(): it should backslash-escape back-ticks, but
it doesn't.  That's all there is to it.  And since saving single-quoted
string constants is not really a safe option, I would strongly suggest
leaving the label_struct alone and fixing conv_text() instead.

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: [ gnuplot-Bugs-993127 ] Difference between ' and " not maintained internally

[Ethan A M. <merritt@u.washington.edu>]

On Sunday 18 July 2004 02:42 pm, Hans-Bernhard Broeker wrote:
> > I consider this a bug, because you do not get the same
> > behaviour from a saved file that you got from the original.
>
> I don't really think this is true.  Not as of right now, before
> introduction of the string-variables patchsets, that is.
>
> Can you demonstrate an actual difference in behaviour from CVS
> gnuplot caused by this?

Sure. This is a stupid example, but it's very simple:

set title 'Now is `date`'
plot x
save "temp"
load "temp"

The first plot will have as a title    Now is `date`
After save/reload is has a title     Now is Sun Jul 18 16:18:31 PDT 2004


-- 
Ethan A Merritt
Department of Biochemistry & Biomolecular Structure Center
University of Washington, Seattle

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Ethan A M. <merritt@u.washington.edu>]

> So far, variables stored values, not expressions to be evaluated at
> some later time.  If at all possible, I'ld like to keep it that way for
> strings, if only to minimize the amount of documentation we and the users
> will need to fully explain all this.

I disagree, and I think the record of requests for enhancement
support me on this point.  People really want a way of embedding 
variables into strings, and having the current value of the variable
substituted in at the time the string is printed.  It doesn't matter
whether you call that "re-evaluation of a string" or "storing a function
instead of a string", or "storing an expression to be evaluated later",
that is the desired capability.

> I'm opposed to having sprintf() *inside* the quotes.  It causes new
> problems like the '' vs. "" saving issue that we don't really need, for no
> real gain.

On the contrary, it's a huge gain.  It means that a very powerful
ability is introduced in a uniform way, yet requires minimal or no
change to the existing code or to the existing storage mechanisms
for strings.

Anything you do _outside_ the quotes means that every single place
that tests for a string constant has to be re-written to handle the
possible substitution of other syntactic entities instead.

But let me repeat again, I do not care what the exact quoting style
is.   If it saves confusion, I'm perfectly happy to add a 3rd quote character
besides ' and ", and reserve this new quote character for the case of
re-evaluation at plot time.   That would require changing is_quote(),
which is easy, and a lot of places that explicitly test for ' or ", which is
more annoying but certainly doable.

Assume for the moment we use % for this purpose. Then the
documentation would read:
	"<expression>"	evaluate immediately, with substitution
	'<expression>'		evaluate immediately, no substitution
	%<expression>%	evaluate later, with substitution at that time

-- 
Ethan A Merritt
Department of Biochemistry & Biomolecular Structure Center
University of Washington, Seattle

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Hans-Bernhard B. <br...@ph...>]

On Sun, 18 Jul 2004, Ethan A Merritt wrote:

> On Sunday 18 July 2004 07:03 am, Hans-Bernhard Broeker wrote:
> > >
> > > 1) gprintf("format",mantissa,exponent)
> > >     Is that the form it should take?
> >
> > No.  It should be gprintf("format", number).  The crucial difference
> > is that C's sprintf() supports multiple % formats and uses up exactly
> > one argument per format specifier, whereas gprintf() only ever has one
> > argument, but may use more than one format specifier with it.
> 
> Let me see if I understand this...
> 
> The user would type, for example
> 	set label gprintf("format",var)
> and internally this would be converted into a call to the existing
> function of the form
>     gprintf( (char *)temp, sizeof(temp),
> 	(char *)format,
> 	(double)current_radix,
> 	(double)var);
> followed by copying temp into the appropriate place, in this case
> the label structure.
> 
> From the user's point of view (and the parser's), gprintf always has
> exactly two parameters:  (char *)format and (double)var.

Yes.  It may be useful / necessary to allow for the logarithm base 
(current_radix), too, but we can worry about that later.

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: [ gnuplot-Bugs-993127 ] Difference between ' and " not maintained internally

[Hans-Bernhard B. <br...@ph...>]

On Sun, 18 Jul 2004, Ethan A Merritt wrote:

> is already there, and is broken (IMHO).
> In practice that rule is  'single quoted strings are not 
> processed when typed on the command line, but are
> processed later on whenever they are used, or saved'.

No, it isn't.  The only processing unquoted strings have been receiving so
far as "un-processing" them, i.e. inverting the processing of "" quoted
strings before they get written out or shown, for the benefit of getting
guaranteed displayable output... a '' string could contain liberal <Tab>
or non-ASCII characters, e.g., which it may not be wise to display or
save as they are.

> I consider this a bug, because you do not get the same
> behaviour from a saved file that you got from the original.

I don't really think this is true.  Not as of right now, before
introduction of the string-variables patchsets, that is.

Can you demonstrate an actual difference in behaviour from CVS
gnuplot caused by this?

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Hans-Bernhard B. <br...@ph...>]

On Sun, 18 Jul 2004, Ethan A Merritt wrote:

> But that example does not require a user-defined function.
> That is the behaviour you would get anyway, courtesy of
> the automagic string evaluation code already written.
> 
> filename = 'sprintf("foobar%d.ps",i)'
> set output filename

If that indeed works, and recomputes the sprintf every time the 'set
output' command is reissued, then by comparison with existing gnuplot
machinery for user-defined objects, 'filename' is a function, not an
variable. So far, the gnuplot paradigm always was that variables have a
static value (unless they're the dummy in a plot command, i.e. 'x', 'y',
't', ...).

I.e. so far, variables stored values, not expressions to be evaluated at
some later time.  If at all possible, I'ld like to keep it that way for
strings, if only to minimize the amount of documentation we and the users
will need to fully explain all this.

> They would need to check that the string is
> not really a constant, but instead holds a sprintf() command. 

I'm opposed to having sprintf() *inside* the quotes.  It causes new
problems like the '' vs. "" saving issue that we don't really need, for no
real gain.

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: [ gnuplot-Patches-992149 ] String variables revisited

[Ethan A M. <merritt@u.washington.edu>]

On Sunday 18 July 2004 07:03 am, Hans-Bernhard Broeker wrote:
> >
> > 1) gprintf("format",mantissa,exponent)
> >     Is that the form it should take?
>
> No.  It should be gprintf("format", number).  The crucial difference
> is that C's sprintf() supports multiple % formats and uses up exactly
> one argument per format specifier, whereas gprintf() only ever has one
> argument, but may use more than one format specifier with it.

Let me see if I understand this...

The user would type, for example
	set label gprintf("format",var)
and internally this would be converted into a call to the existing
function of the form
    gprintf( (char *)temp, sizeof(temp),
	(char *)format,
	(double)current_radix,
	(double)var);
followed by copying temp into the appropriate place, in this case
the label structure.

From the user's point of view (and the parser's), gprintf always has
exactly two parameters:  (char *)format and (double)var.

Is that correct?   

-- 
Ethan A Merritt
Department of Biochemistry & Biomolecular Structure Center
University of Washington, Seattle

Re: [ gnuplot-Bugs-993127 ] Difference between ' and " not maintained internally

[Ethan A M. <merritt@u.washington.edu>]

> A more important test case to keep in mind is:
>
> gnuplot> set xlabel 'single\single'
> gnuplot> show xlabel
>
>         xlabel is "single\single", offset at 0.000000, 0.000000
>
> Note how the \ from the single-quoted string got doubled.
>
> Generally speaking I'm not quite sure the syntax you
> suggested for that string variables patch is suitable.  For
> one thing, it breaks the 'single quoted strings are used
> as-is without further processing' scheme.  

This is not something I am suggesting; it is something that
is already there, and is broken (IMHO).
In practice that rule is  'single quoted strings are not 
processed when typed on the command line, but are
processed later on whenever they are used, or saved'.
I consider this a bug, because you do not get the same
behaviour from a saved file that you got from the original.

I did not change or propose anything new for the string
variables patch (yet).  I am just pointing out how it already
works.  It was a happy accident that this bug allowed me
to give an example of delayed evaluation, but I still consider
it a bug.

> In my own
> deliberations abotu this, I never intended for the sprintf()
> to be *inside* the quotes.

Heh.  In fact it was your suggestion that made me realize
this was possible.  You said:

> So far, my vague idea was to require string-valued expressions
> to always start with a string constant, for its benefit.  I.e. we would have
>       
>       plot ''+g(p,q)  # string --> computed filename

I thought this was very clever; all the existing code will continue
to recognize it as a string constant, while any new code can peer
at the start of the string to see a function evaluation is needed.
To make it blindingly obvious what is going on, I used the 7 character
string "sprintf" instead of the 1 character string "+" you used in your
example.  But I already stated that I have no string preference for
what this magic string would be, other than it had better be something
not normally printed as the start of a string constant.
Dave Denholm suggested using tripled-quotes:  """   but I don't
like that because it looks like an attempt to insert a literal quote mark.

Another suggestion from quite a while back was to use the form
%(<expr>) to indicate an expression that was to be evaluated at
plot time.  That could be generalized to:
	"%(<expr>)"
for strings, including "%(sprintf(...))", where in this formulation there
is nothing special about the characters "sprintf"; it just happens to
be the function that is to be evaluated. 


-- 
Ethan A Merritt
Department of Biochemistry & Biomolecular Structure Center
University of Washington, Seattle