gnuplot-beta — Developers and beta testers of new gnuplot code

faq: which terminal for OpenOffice.org

[Petr M. <mi...@ph...>]

I've made some updates to faq.

There is a question "How do I include my graphs in <word processor>?"
I think it would be useful to complete it by "in OpenOffice.org". Can
someone worked this out?

---
Petr Mikulik

Re: [Gnuplot-bugs] bugs/problems in gnuplot 3.8k (cvs)

[<br...@ph...>]

> 1) pdf driver:  that's a great new driver -- I really love that addition!
[...]
> somehow, other pdf tools can inform acroread to set the page setup to
"landscape",
> so printing is ok without twiddling with page setup settings first.

Well, knowing the fact that other programs can manage this doesn't exactly
help us implement
it for ourselves...  We would need to know *how* they do it.

> 2) readline(?), SIGTSTP, and gnuplot_x11 communication  (and libreadline?)

As far as I can see, at least a significant part of that problem lies with
libreadline itself.  gnuplot
should probably catch SIGTSTP and do something with it, though.

> 3) I'm just starting to use gnuplot on M$ Win-XY for my first time.
> obviously it's great too, but by default I'm hardly missing the
> possibility to use shell commands in plot "datafile" using
>
> plot "< some command"
>
> but by default PIPES config is not set for windows builds :-(

That's because it has a side effect which the average Windows user would
quite certainly be quite irritated about: it opens an extra console window
if started from
outside a console.  The average Windowser would perceive that as a bug of
the program, not
as a feature --- I've been there and got quite a collection of mails about
it, when I accidentally put
a similar behaviour in the first distribution of the 3.7.3 W32 build...

I've been contemplating a more Unix-ish Win32 version, i.e. a console
application that still has a
GUI graph window --- I had long thought this was utterly impossible to do,
but it appears I may have
been wrong.  I made some inroads, but haven't quite reached a usable overall
result yet.

OTOH, I don't see what's stopping you from building your own w32 version
with -DPIPES=1
enabled...

> I've read your comments about PIPES in config/makefile.mgw
> but for me (Windows novice) it's unclear if this is a problem
> for _all_ windows versions (e.g. would WinXP be any better?).

The problem doesn't depend on the version of Windows --- it's a fundamental
difference in the
way Unix and Windows work.  X11 programs have a plain stdin and stdout
channel, just like
console ones, whereas Windows GUI apps don't. This was even worse in Win16,
which wgnuplot
was originally made for, and which made that silly home-grown terminal
emulator a necessity.

> right now I still hope (dream?) that it'll be possible to use MSYS
> to allow "real" shell commands in plot "< ..."

What exactly is "MSYS"?  Anyway: if you miss Unix-ish behaviour real bad, I
would suggest
you get yourself cygwin (including X11) installed ;->

Re: Remove path to VERSION file, this is now taken of by TEXINPUTS.

[Petr M. <mi...@ph...>]

> If they're sufficiently Unix-like, I see no particular reason why the
> approach in docs/Makefile.in shouldn't work for makefile.mgw, too. Or you
> can use the "export"  directive of GNU make, i.e.
>
>       export TEXINPUTS=... whatever...

It doesn't work.

Thus I've chosen "copy ../VERSION ." which works OK.

---
Petr Mikulik

howto drive wgnuplot for the records

[kemal a. <as...@cl...>]

I would like to put the following  in the records.
thanks to :
Hans-Bernhard Broeker <br...@ph...>.
here is a working solution to drive gnuplot  Version 3.7 patchlevel 3.
on windows2K built with VC6.0.


// consol.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <iostream>
#include <stdio.h>

using namespace std;


int main(void)
{
        FILE *chkdsk;
        if ( (chkdsk = _popen( "pgnuplot.exe", "wrt")) !=
NULL)
        {
                fputs("plot sin(x)\n", chkdsk);
                fflush(chkdsk);  //Works perfectly now
                system("pause"); //needed otherwise gnuplot closes when
//the console app ends to fast to see anything
        }
		printf("DONE!\n");

        return 0;

}

libgd + symbol fonts [was Re: Universal enhanced text mode!]

[Ethan M. <merritt@u.washington.edu>]

Just an update.

I added this code to CVS back in November, with a caveat that full
support for symbol fonts in the png/jpeg terminal required a patch
for libgd.

I am delighted to report that Tom Boutell has added the patch into
libgd. As of version 2.0.21 (now available for download) both the
Microsoft symbol.ttf and the Adobe Symbol fonts are handled correctly,
and "just work" from gnuplot.  Actually any Adobe custom-encoded
font should work, not just Symbol. Apple fonts should work also,
but I haven't tested them.



iso-8859-xx encodings
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

A related relatively minor problem remains, however, in that
there is currently no mechanism for translating the requested
character encoding from gnuplot through libgd to the font libraries.

libgd's own native font (default if you don't specify one) happens
to be iso-8859-2.   When you request a specific TTF font you=20
get iso-8859-1 so far as I can tell, but I am not sure if this is
guaranteed. =20

There is a mechanism for requesting 8-bit unicode, Shift-JIS, or big5=20
characters, however. So in principal gnuplot could  convert strings
to unicode if it wants characters not in iso-8859-1, but that
sounds like too much work.  I may see if I can further hack
libgd itself to know more about character encodings in some future
release.

--=20
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center Box 357742
University of Washington, Seattle, WA 98195

Re: Remove path to VERSION file, this is now taken of by TEXINPUTS.

[Hans-Bernhard B. <br...@ph...>]

On Fri, 16 Jan 2004, Petr Mikulik wrote:

> 	Hi Lars,
> 
> > Modified files:
> >        gnuplot/docs/: titlepag.tex
> >
> > Log message:
> >        Remove path to VERSION file, this is now taken of by TEXINPUTS.
> 
> well, it seems it is not easily portable by setting TEXINPUTS in a Makefile
> for different systems, like Windows and OS/2. Do you know how to set an
> environmental variable in Makefiles for these systems? 

If they're sufficiently Unix-like, I see no particular reason why the
approach in docs/Makefile.in shouldn't work for makefile.mgw, too. Or you
can use the "export"  directive of GNU make, i.e.

	export TEXINPUTS=... whatever...

There's an older method using the special target ".EXPORT:", but I'm not
sure whether that still exists.  It's not documented any longer, it seems.

> Thus, do you really prefer the above "copy" method, or would rather go back
> to \openin ../VERSION as it was until now?

\openin ../VERSION breaks in other ways, so we can't usefully go back to
that (it fails to work in out-of-source-directory builds on Unix-like
platforms).

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: Remove path to VERSION file, this is now taken of by TEXINPUTS.

[Lars H. <lhe...@us...>]

> The only solution I've found is to add "cp VERSION" command here:
> 
> gnuplot.dvi: gnuplot.tex $(D)titlepag.tex
>     cp gnuplot.tex $(D)gp_tex2.tex
>     cp VERSION $(D)
>     cd $(D) && latex gp_tex2.tex && latex gp_tex2.tex && latex gp_tex2.tex
> 
> Thus, do you really prefer the above "copy" method, or would rather go back
> to \openin ../VERSION as it was until now?

 It breaks when you build outside the source dir, that's why the change
 was made.

Remove path to VERSION file, this is now taken of by TEXINPUTS.

[Petr M. <mi...@ph...>]

	Hi Lars,

> Modified files:
>        gnuplot/docs/: titlepag.tex
>
> Log message:
>        Remove path to VERSION file, this is now taken of by TEXINPUTS.

well, it seems it is not easily portable by setting TEXINPUTS in a Makefile
for different systems, like Windows and OS/2. Do you know how to set an
environmental variable in Makefiles for these systems? I tried it for
Makefile.mgw, but haven't found any way to do it.

The only solution I've found is to add "cp VERSION" command here:

gnuplot.dvi: gnuplot.tex $(D)titlepag.tex
    cp gnuplot.tex $(D)gp_tex2.tex
    cp VERSION $(D)
    cd $(D) && latex gp_tex2.tex && latex gp_tex2.tex && latex gp_tex2.tex

Thus, do you really prefer the above "copy" method, or would rather go back
to \openin ../VERSION as it was until now?

---
Petr Mikulik

with_image_120303.diff

[Arnd B. <arn...@we...>]

Hi,

next I wanted to try the with_image patch,
(I followed the guideline for patching from Ethan,
http://www.bmsc.washington.edu/people/merritt/gnuplot/instructions.html
)

However, patching color.c and gplt_x11.c I get

patching file src/color.c
Hunk #2 FAILED at 525.
1 out of 2 hunks FAILED -- saving rejects to file src/color.c.rej

patching file src/gplt_x11.c
Hunk #2 FAILED at 373.
Hunk #3 succeeded at 483 (offset 2 lines).
Hunk #4 succeeded at 582 (offset 3 lines).
Hunk #5 succeeded at 1441 (offset 1 line).
Hunk #6 succeeded at 2217 (offset 1 line).
Hunk #7 FAILED at 2319.
Hunk #8 succeeded at 3003 (offset 1 line).
Hunk #9 succeeded at 3390 (offset 1 line).
2 out of 9 hunks FAILED -- saving rejects to file src/gplt_x11.c.rej

I had a look at gplt_x11.c but I don't understand
the code (well enough ;-) to fix this...
((If someone has the time at some point for doing this
I promise to test the patch, in which I am very much interested ;-))

Is there any chance that it gets integrated into the CVS version ?

Many thanks,

Arnd

Re: Request for including your software in our database

[Hans-Bernhard B. <br...@ph...>]

[I'm cc'ing this reply to the developers list informational purposes...]

Feel free to go right ahead with this.  

But, for the future, please note that mail like this had better to go the
administrative contact addresses mentioned on the opening screen of the
program, or in the documentation, rather than individual developers...

If you offer the program for download, we may want to link to your
page as an additional mirror site, too, so please send us a linkable
URL once you have completed construction of this database of yours.

On Thu, 15 Jan 2004, Dr G P S Raghava wrote:

> We are maintaining a database of free softwares for general purpose
> (www.imtech.res.in) which are available free of cost for general user
> and academic community. The aim of this project is to help the
> scientific community in distributing these software. We are a academic
> institute are not charging any money from users who wish to download
> these software from our site. we have found your software to be useful
> for general Information Technology professionals, general computer users
> and others also. We wish to include your software in our next database
> release. We already have a copy of software from web. In case , you do
> not agree please initimate us, for removal of your software from our
> database. presently your software will be available from our site i.e.
> gnuplot (software name) at www.imtech.res.in/fsgp/. whenever you are not
> interested to distribute these software from our site , please write to
> us . We will remove your software from our database. Initially we plan
> to provide a link to the original software but later we have realized
> that a number of software are not available from net or their original
> site is not accessible or some sites may close frequently. As well as
> some users do not have internet connections particularly in developing
> countries , so off-line distribution will help them. Thus, we thought we
> should keep one copy at our site and also provide a link to original
> site for downloading these software. We encourage users to download
> software from their original site rather than from our site (We clearly
> mention on home page of the database). This way user will get latest
> version. We also mention clearly that user should read terms and
> conditions and copy right statement laid for users.

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: pasting code into gnuplot

[Arnd B. <arn...@we...>]

Hi,

for me both pasting from X11 and piping (from python)
works great now - Many many thanks   !

The example for accessing gnuplot variables
and waiting for mouse-clicks from python can be obtained as
  http://www.physik.tu-dresden.de/~baecker/python/GnuplotBiDir.py
see
  http://www.physik.tu-dresden.de/~baecker/python/gnuplot.html
for a few notes.

Arnd

Re: pasting code into gnuplot

[Ethan A M. <merritt@u.washington.edu>]

On Thursday 08 January 2004 12:05 pm, Arnd Baecker wrote:
> I would like to do something like this
> #####################################
> from GnuplotBiDir import Gnuplot
> gp=Gnuplot()
> gp("set mouse")       # activate mouse
> gp("plot sin(x)")
> print "Now get coordinates of a mouse-click:"
> gp("pause mouse 'click with mouse' ")
> #raw_input("and press enter here after that")
> mouse_x=gp.getvar("MOUSE_X")
> mouse_y=gp.getvar("MOUSE_Y")
> print mouse_x,mouse_y
> ######################################

> Of course, this cannot work as the calling program does not
> wait for the mouse-click.

I have placed a fix for this in CVS.  'pause mouse' was terminating
on either a mouse click or a <cr> on stdin.  That is OK for
terminal input, but with piped input there may already be multiple
commands backed up in the pipe, in which case it would not wait
for a mouse click.  As of now the behaviour is changed to only
terminate 'pause mouse' on a mouse click or on ctrl-C.
I also added some sanity checking so that you cannot get
stuck waiting for 'pause mouse' if mousing is not active.

Note that requesting "set mouse" before plotting is essential.
The default for piped input is to disable mousing.

-- 
Ethan A Merritt
Department of Biochemistry & Biomolecular Structure Center
University of Washington, Seattle

Re: pasting code into gnuplot

[Daniel J S. <dan...@ie...>]

Ethan A Merritt wrote:

>On Saturday 10 January 2004 01:07 pm, Petr Mikulik wrote:
>  
>
>>  Or would this mean big non-portability to different X11 systems?
>>    
>>
>
>I don't know.  But if we were going to move to a threaded implementation
>of gnuplot, I would want to look at making even larger scale use of it.
>For example, would it make any sense to spawn off a new thread for each
>new plot?  That would be an easier way to achieve multiple active
>plots than Dan's current dream of making all the internal data structures
>into linked lists.
>

There goes my pipe dream.

Dan

Re: pasting code into gnuplot

[Hans-Bernhard B. <br...@ph...>]

On Sat, 10 Jan 2004, Petr Mikulik wrote:

> As I said earlier, I don't know about the techniques you describe in the
> previous letters .. but cannot be something modern used to get rid of these
> problems? 

Modern solutions have the crucial problem of being modern; meaning that
they're not particularly unified or reliably available all across the
spread of platforms we're dealing with.

>   Or would this mean big non-portability to different X11 systems?

I strongly suspect so.

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: pasting code into gnuplot

[Ethan A M. <merritt@u.washington.edu>]

On Saturday 10 January 2004 01:07 pm, Petr Mikulik wrote:
> cannot be something modern used to get rid of these
> problems? E.g. pthreads. There can be a thread waiting for mouse/hotkey
> actions, and it may be triggered by mutexes or semaphores.

Putting all the mouse handling in a separate thread would work (I think).
I don't at the moment see how it would work for hotkeys, though.
A hot-key triggered command should be treated identically to the
same thing typed from the command line, including effects on subsequent
plotting commands.  Is that possible with pthreads?  That is, are
changes to global variables in one thread seen by the others?

> Get rid of stdin to gnuplot_x11 and use named pipe or fifo.

I am not aware of a problem with the input stream to gnuplot_x11.
What did you have in mind?  The problem we have at the moment
is with multiple inputs to gnuplot itself.

>   Or would this mean big non-portability to different X11 systems?

I don't know.  But if we were going to move to a threaded implementation
of gnuplot, I would want to look at making even larger scale use of it.
For example, would it make any sense to spawn off a new thread for each
new plot?  That would be an easier way to achieve multiple active
plots than Dan's current dream of making all the internal data structures
into linked lists.


-- 
Ethan A Merritt
Department of Biochemistry & Biomolecular Structure Center
University of Washington, Seattle

Re: pasting code into gnuplot

[Petr M. <mi...@ph...>]

As I said earlier, I don't know about the techniques you describe in the
previous letters .. but cannot be something modern used to get rid of these
problems? E.g. pthreads. There can be a thread waiting for mouse/hotkey
actions, and it may be triggered by mutexes or semaphores. Get rid of stdin
to gnuplot_x11 and use named pipe or fifo.
  Or would this mean big non-portability to different X11 systems?

---
Petr Mikulik

Re: pasting code into gnuplot

[Hans-Bernhard B. <br...@ph...>]

On Sat, 10 Jan 2004, Daniel J Sebald wrote:

[....]
> You'll have to forgive me.  I've tried following the code to see exactly 
> the path that the commands come in via stdin, where the pipe is read for 
> the mouse, etc. and I'm having a difficult time keeping track of it all. 

The core function is generally term->waitforinput(), which in the case of
the x11 terminal is provided by X11_waitforinput() in term/x11.trm.  
There are different call paths leading there.  Let me provide an overview.
Cscope helps:

C symbol: waitforinput

  File       Function      Line
0 term_api.h <global>       172 int (*waitforinput) __PROTO((void ));
1 command.c  pause_command  961 if (term && term->waitforinput) {
2 command.c  pause_command  964 term->waitforinput();
3 command.c  fgets_ipc     2212 if (term && term->waitforinput) {
4 command.c  fgets_ipc     2219 c = term->waitforinput();
5 readline.c getc_wrapper    84 if (term && term->waitforinput && interactive) {
6 readline.c getc_wrapper    88 c = term->waitforinput();
7 readline.c ansi_getc      775 if (term && term->waitforinput && interactive)
8 readline.c ansi_getc      776 c = term->waitforinput();

The call paths ultimately start at read_line in command.c.  I'll assume
MOUSE is active:

no READLINE (neither gnuplot's own, nor -lreadline):
	GET_STRING ==> fgets
some READLINE, !interactive: 
	fgets_ipc ==> term->waitforinput
GNU libreadline, interactive:
	read_line ==> rlgets ==> readline_ipc ==> rl_callback_read_char
	(getc_wrapper passed to libreadline as input hook)
gnuplot readline.c, interactive:
	read_line ==> rlgets ==> readline_ipc ==> readline
	==> special_getc ==> ansi_getc

So that's how we get to the core function. Now let's look at
X11_waitforinput itself: it uses select() to dispatch between the two
input streams.  The root of all our trouble is that select() operates not
on <stdio.h> objects like stdin, but rather on unbuffered Unix file
handles.  I.e. select doesn't know a thing about <stdio.h> buffering.

> I find it hard to fathom that this is not possible.  

It's quite certainly possible.  The tricky part is to make it possible
in the context of a massively portable program like gnuplot, without 
breaking the whole program for use on other platforms.

> I mean, doesn't this seem like a fundamental task of operating systems?  
> That is, to distribute I/O around the system?  Let's consider an fread()
> or a fgets() or fgetc().  

For starters, fgetc() and friends aren't even functions of the operating
system to begin with.  They're functions of the C runtime library --- but
ISO/ANSI C doesn't standardize asynchronous I/O. 

> Can't I just do an fread() from the stdin (the place where the commands
> are typed in) and whatever is there I store in a "command buffer" and as
> I'm putting the characters in a command buffer I watch for the "new
> line" or "end of gnuplot command" character.

No, you can't.  The reason is that file I/O on Unix (and all other
platforms I've seen) is by default a "blocking" operation:

> If there is nothing in the buffer or there are less characters than
> asked for, doesn't the fread() simple return and EOF or -1 or number of
> bytes read?

If there's buffered input, but less than requested, it'll return what
there is. But if there's currently _nothing_ in the buffer, it won't
return at all until there is.  You only get an EOF condition if the stream
is actually ended (typical Unix setup: if you typed Ctrl-D at the
keyboard), not if there's just no buffered input available at the moment.

> >F) The usual way around this dilemma is to use poll or select to notify
> >   us when either or both of two input streams is presenting new data.
> >   We currently use select in X11_waitforinput().
> >
> 
> Yes, this is one way.  But if "fread()" is inherently forced to return 
> something even if the buffer is empty, isn't that a form of polling the 
> input as well?  

The problem is we *can't* force fread() to do that.  Non-blocking I/O
is not foreseen in the context of <stdio.h> functions.

> As I said before, it seems to me that current Gnuplot 
> really isn't wanting polling for "characters available", rather it is 
> attempting to poll for "gnuplot command in buffer available" and "mouse 
> command in buffer available".

I'm quite sure that reading characters versus lines is unrelated to the
problem at hand.

> >G) Unfortunately, poll/select notification depends on whether the
> >   input stream is buffered or non-buffered.   I do not know if this is
> >   supposed to be the case, but that's what we see in practice.

Indeed select() doesn't know a thing about buffering, so it will not
report a stream as having data available if all of it is buffered.  

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: pasting code into gnuplot

[Daniel J S. <dan...@ie...>]

Ethan Merritt wrote:

>On Friday 09 January 2004 14:50, Daniel J Sebald wrote:
>  
>
>>If you turn off buffering immediately after startup, what if a
>>number of Gnuplot commands are already in the buffer?
>>    
>>
>
>Since piping now works, this is demonstrated not to be a problem.
>Of course if you have a test example that fails, I will glumly
>retract that optimistic statement.
>
>  
>
>>Thus the only way to resolve the race condition, in one form
>>or another from Gnuplot, is for there to be an ability for the "setvbuf"
>>command to return the contents of the buffer right before it was
>>changed.
>>    
>>
>
>Not possible. See "man setvbuf".
>

I searched for info.  As usual, the man doesn't seem to give details of 
the tricky aspects.

>>1)  Fix up the I/O scheme so that gnuplot doesn't wait on keyboard
>>input.  Whenever keyboard input is available then parse it.
>>    
>>
>
>Now you are getting near to the heart of it.  Most of our current
>problem comes from not being able to tell when input is available
>without trying to read it.  And if we try to read it, then we are stuck
>waiting until success.
>
>The central problem is that there are few (actually none that I know of)
>portable methods of reading from two or more input sources at the same
>time.  I know perfectly well how to do it under VMS but, sadly, other
>operating systems are not so capable [grin, duck, and run].
>

You'll have to forgive me.  I've tried following the code to see exactly 
the path that the commands come in via stdin, where the pipe is read for 
the mouse, etc. and I'm having a difficult time keeping track of it all. 
 So I'm going to play devil's advocate here and ask naive questions.

I find it hard to fathom that this is not possible.  I mean, doesn't 
this seem like a fundamental task of operating systems?  That is, to 
distribute I/O around the system?  Let's consider an fread() or a 
fgets() or fgetc().  Can't I just do an fread() from the stdin (the 
place where the commands are typed in) and whatever is there I store in 
a "command buffer" and as I'm putting the characters in a command buffer 
I watch for the "new line" or "end of gnuplot command" character.  When 
the end of command character is found, then shuffle that string over to 
the current command parsing stuff.  If there is nothing in the buffer or 
there are less characters than asked for, doesn't the fread() simple 
return and EOF or -1 or number of bytes read?

Then, do the same with the mouse stream.

It sounds to me, correct me if I'm wrong, that Gnuplot is now attempting 
a graceful method of trying to get a complete command from both the 
mouse and the keyboard whenever either is available.  Is this what you 
mean by asynchronous?  If an operating system can do that, that is 
impressive.  But no, I would not expect that to be a feature of all 
operating systems.  It seems to me then, that for Gnuplot (to be 
portable) is the one that needs to do all the work of reconstructing the 
input streams.

Let me attempt a description of what I'm saying without writing the full 
code.  Say there is a "keyboard_stream" and a "mouse_stream".  We make 
internal buffers inside Gnuplot of length, say 200.

keyboard_command_buff[201];
mouse_command_buff[201];
keyboard_buff_ptr = 0;
mouse_buff_ptr = 0;

while (not_end_of_Gnuplot) {

    int number_chars;

    /* Check if there is keyboard input, if so check if a valid command 
is complete. */
    number_chars = fread(keyboard_stream, 
&keyboard_command_buff[keyboard_buff_ptr], 200-keyboard_buff_ptr);   /* 
That is, read up to whatever is space is left in the buffer */
    if (number_chars > 0) {
       [search through the characters just read and test for end of command]
       if (this is valid end of command)  {
           execute gnuplot command;
           copy any chunk left in buffer to the start of buffer;
           keyboard_buff_ptr = 0;
       } else {
           keyboard_buff_ptr += number_chars;
       }
    }
     if (end of buffer has been reached)
         [issue error to the console]


    /* Check if there is mouse input, if so check if a valid mouse 
string is complete. */
   [Do the same for the mouse stream as is done above for the keyboard.]

}


The input method just keeps going around in the loop and never waits on 
anything.  It may be that only a few characters are read at a time, but 
so what.  It isn't like this needs to be super efficient as though it 
were a time critical communication scenario.  We're waiting on mouse and 
keyboard commands which are rather slow I assume.

Is there some issue with buffered mouse characters piling up in the 
stream while a Gnuplot command is being processed?

(more below)

>The current scheme under linux  depends on select(). This scheme,
>problems and all, is the closest thing to simultaneous/asynchronous
>input that I know of in standard linux/unix.
>That said, I am hardly an expert on linux asynchronous I/O.
>
>Here is the bind we are in, as I understand it for the linux+X11 case:
>

Nice summary.  Thanks.

(more below)

>A) We have terminal input coming in on stdin; we have mouse input
>   coming in via a pipe.
>
>B) These two input sources are asynchrounous.  That is, we don't
>   know which one will be the next to provide new input.
>

The scheme I describe above attempts to handle this, I think.

>C) We have non-mouse (font, window size, etc) information coming
>   back over the pipe also. This could be split off from the mouse pipe,
>   but that does not make things any easier, so let's disregard it.
>

This should be fine I think.

>D) If you do a read/scanf/gets/whatever from stdin then you will
>   miss all mouse input until you get the next chunk of terminal input.
>

Please explain why this is.  I'm to lazy to do a test trial at this 
hour, but won't fread() on stdin just send back and EOF or -1 or 
something if there are no characters in the buffer?

>E) Conversely if you do a read/scanf/gets/whatever on the mouse pipe
>   then you can't even look for new terminal commands until after the
>   next mouse event (which may never arrive).
>
>F) The usual way around this dilemma is to use poll or select to notify
>   us when either or both of two input streams is presenting new data.
>   We currently use select in X11_waitforinput().
>

Yes, this is one way.  But if "fread()" is inherently forced to return 
something even if the buffer is empty, isn't that a form of polling the 
input as well?  As I said before, it seems to me that current Gnuplot 
really isn't wanting polling for "characters available", rather it is 
attempting to poll for "gnuplot command in buffer available" and "mouse 
command in buffer available".

>G) Unfortunately, poll/select notification depends on whether the
>   input stream is buffered or non-buffered.   I do not know if this is
>   supposed to be the case, but that's what we see in practice.
>
>H) More things seem to work when stdin is set to unbuffered,
>   hence the call to setvbuf().  Other things broke, because this caused
>   the current contents of the buffer to be lost.   Note that this behaviour
>   is not documented anywhere I can find, but it makes sense and is
>   easily demonstrated. I am optimistic that this problem is minimized
>   by moving the setvbuf() call to the head of the program.
>
>I) Just to make things a little bit worse, we have commands like
>   'pause <n>' and 'pause mouse'.
>

Hey, these are easy to handle for the scheme above.  Deactivate the 
input test according to some timer, i.e.,

if (time expired >= n) {
  n = 0;
   /* The junk above */
}

Gnuplot will simply stop checking for input in the appropriate stream 
for <n> seconds.  (This may not work... but that would be my first 
attempt at such.)

>J) Just to make things a little bit worse yet, we try to support gnu
>   libreadline,  which has its own quirks about handling input streams.
>

I would hope the this is fairly transparent to the C environment.

>K) And if all that was not complicated enough, people want to cut-and-paste
>   from X11.  This involves cooperation from the X server and from the
>   application windows (xterm, nedit, or whatever) on both the cut and the
>   paste end.  So far as I know, there is no guarantee that just because
>   it works for gnuplot running in an xterm window it will also work inside
>   konsole or kterm or Eterm or <alphabetofyourchoice>term.
>

Again, I would hope this is transparent to the C environment.  The 
cutting and pasting simply puts stuff into the "stdin" buffer, I'm guessing.

>
>Now that I understand the problem with setvbuf(), I am considerably
>more hopeful that we can get everything to work using the current
>scheme.  More hopeful than I was a week ago.
>poll/select is *supposed* to work, after all.
>But items I, J and K may still cause us problems.
>
>  
>
>>3)  Since gnuplot only has a "current plot" replot and not information
>>for all its plots, there needs to be some variable that keeps track of
>>which plot number corresponds with the "current plot" replot info.  That
>>way, the "replot" command can be reissued when the window number and the
>>replot info match.
>>    
>>
>
>That is a separate topic.
>Let's not mix the two.
>

Right.  That is something completely different.  I'm confused enough the 
way it is.

Anyway, tell me what is wrong with the approach I describe.  I guess my 
main point is that if you want Gnuplot to be portable, it's the one that 
will have to do all the work in determining "valid command line string 
available" and "valid mouse string available", not the system.

Dan

Re: pasting code into gnuplot

[Ethan M. <merritt@u.washington.edu>]

On Friday 09 January 2004 14:50, Daniel J Sebald wrote:
> If you turn off buffering immediately after startup, what if a
> number of Gnuplot commands are already in the buffer?

Since piping now works, this is demonstrated not to be a problem.
Of course if you have a test example that fails, I will glumly
retract that optimistic statement.

> Thus the only way to resolve the race condition, in one form
> or another from Gnuplot, is for there to be an ability for the "setvbuf=
"
> command to return the contents of the buffer right before it was
> changed.

Not possible. See "man setvbuf".

> I think the best solution here is to keep the two input streams
> independent.
[snip rest of long description]

What you describe is exactly what we are already doing.
Keeping track of multiple pipes is not a problem. Figuring out
how to read from more than one at a time *is* a problem.

> 1)  Fix up the I/O scheme so that gnuplot doesn't wait on keyboard
> input.  Whenever keyboard input is available then parse it.

Now you are getting near to the heart of it.  Most of our current
problem comes from not being able to tell when input is available
without trying to read it.  And if we try to read it, then we are stuck
waiting until success.

The central problem is that there are few (actually none that I know of)
portable methods of reading from two or more input sources at the same
time.  I know perfectly well how to do it under VMS but, sadly, other
operating systems are not so capable [grin, duck, and run].

The current scheme under linux  depends on select(). This scheme,
problems and all, is the closest thing to simultaneous/asynchronous
input that I know of in standard linux/unix.
That said, I am hardly an expert on linux asynchronous I/O.

Here is the bind we are in, as I understand it for the linux+X11 case:

A) We have terminal input coming in on stdin; we have mouse input
   coming in via a pipe.

B) These two input sources are asynchrounous.  That is, we don't
   know which one will be the next to provide new input.

C) We have non-mouse (font, window size, etc) information coming
   back over the pipe also. This could be split off from the mouse pipe,
   but that does not make things any easier, so let's disregard it.

D) If you do a read/scanf/gets/whatever from stdin then you will
   miss all mouse input until you get the next chunk of terminal input.

E) Conversely if you do a read/scanf/gets/whatever on the mouse pipe
   then you can't even look for new terminal commands until after the
   next mouse event (which may never arrive).

F) The usual way around this dilemma is to use poll or select to notify
   us when either or both of two input streams is presenting new data.
   We currently use select in X11_waitforinput().

G) Unfortunately, poll/select notification depends on whether the
   input stream is buffered or non-buffered.   I do not know if this is
   supposed to be the case, but that's what we see in practice.

H) More things seem to work when stdin is set to unbuffered,
   hence the call to setvbuf().  Other things broke, because this caused
   the current contents of the buffer to be lost.   Note that this behavi=
our
   is not documented anywhere I can find, but it makes sense and is
   easily demonstrated. I am optimistic that this problem is minimized
   by moving the setvbuf() call to the head of the program.

I) Just to make things a little bit worse, we have commands like
   'pause <n>' and 'pause mouse'.

J) Just to make things a little bit worse yet, we try to support gnu
   libreadline,  which has its own quirks about handling input streams.

K) And if all that was not complicated enough, people want to cut-and-pas=
te
   from X11.  This involves cooperation from the X server and from the
   application windows (xterm, nedit, or whatever) on both the cut and th=
e
   paste end.  So far as I know, there is no guarantee that just because
   it works for gnuplot running in an xterm window it will also work insi=
de
   konsole or kterm or Eterm or <alphabetofyourchoice>term.

Now that I understand the problem with setvbuf(), I am considerably
more hopeful that we can get everything to work using the current
scheme.  More hopeful than I was a week ago.
poll/select is *supposed* to work, after all.
But items I, J and K may still cause us problems.

> 3)  Since gnuplot only has a "current plot" replot and not information
> for all its plots, there needs to be some variable that keeps track of
> which plot number corresponds with the "current plot" replot info.  Tha=
t
> way, the "replot" command can be reissued when the window number and th=
e
> replot info match.

That is a separate topic.
Let's not mix the two.

--=20
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center Box 357742
University of Washington, Seattle, WA 98195

Re: pasting code into gnuplot

[Daniel J S. <dan...@ie...>]

Arnd Baecker wrote:

>Hi,
>
>On Wed, 7 Jan 2004, Ethan Merritt wrote:
>
>  
>
>>On Wednesday 07 January 2004 15:54, Ethan Merritt wrote:
>>    
>>
>>>if we are going to
>>>switch to unbuffered mode we should first make sure the
>>>buffer is empty, or else store it elsewhere for later execution.
>>>
>>>I don't yet see a trivial way of doing this, but ...
>>>      
>>>
>>Trivial indeed. I was focused on the "or else", but in fact
>>the first alternative is the easy one.  The buffer is empty
>>when the program starts, so I moved the call to setvbuf()
>>out of X11_init() and put it right at the start of plot.c where
>>the other I/O streams are being initialized.
>>
>>The X11 paste buffer is now working, and piping through awk
>>now works too.  I left the call to setvbuf() protected by
>>#ifdef X11 but I doubt that it needs to be.  Are any non-X11
>>platforms also suffering lost characters in the input stream?
>>    
>>
>
>Well, for me the picture is slightly different:
> a) on a PIV machine: no problems, pasting works, piping works
> b) on my PIII (which I used to do the testing):
>    - pasting does not work the first time
>    - piping seems to work fine now
>
>    I haven't done thorough testing yet, but this seems
>    to improve!
>  
>

 From Hans:

 > I suspect we'll have to avoid stdin buffering and possibly blocking
 > I/O altogether. I.e. don't switch off buffering halfway into the program,
 > but immediately after startup, before the first character is read.

 From Arnd:

>Is the pasting problem related with what Hans just wrote/suggested
>"I.e. don't switch off buffering halfway into the program, but
>immediately after startup"?
>((Speculating: maybe a "fast" machine is fast enough to switch off
>buffering before something is written to the buffer?))
>

I wonder if there is any way of avoiding a race from the level of 
Gnuplot.  If you turn off buffering immediately after startup, what if a 
number of Gnuplot commands are already in the buffer?  (Sounds like a 
plausible situation.)  Then the commands will be lost.

Simply tossing out what is in the buffer of course won't do.

It seems to me that resolving the race condition can only be done at a 
low level.  Thus the only way to resolve the race condition, in one form 
or another from Gnuplot, is for there to be an ability for the "setvbuf" 
command to return the contents of the buffer right before it was 
changed.  Then one would have to reconstruct the input stream by storing 
that somewhere.  You can't store what is currently in the buffer then 
set the I/O stream to non-buffered as two independent actions. 
 Something could enter the I/O buffer in that time between the save and 
the set, for multitasking systems, and again characters will be lost. 
 Only the low level routines can protect the buffer from something new 
being entered.  Right?

I think the best solution here is to keep the two input streams 
independent.  Leave the keyboard stream as it is, no "setvbuf".  Create 
different pipes for the mouse (that can later be configured), I contend. 
 That means that upon creating an X11 plot window a pipe ID of some sort 
is passed to gnuplot_x11.  Create a variable in the plot structure of 
gnuplot_x11 and save that pipe ID in there and whenever there is a mouse 
action for that window, send its primitive mouse information out that 
pipe.  Whatever is on the other end of the pipe interprets the mouse 
commands.  The default handler on the gnuplot side is to reissue the 
"replot" command if appropriate.  Be consistent with who creates and 
deletes the pipes.  Probably gnuplot should create the pipes and 
gnuplot_x11 should be the one to delete them.  (That way, gnuplot_x11 
will never be sending data to a defunct pipe.  Note, it doesn't need to 
be Gnuplot that creates the pipe ID.  Perhaps a higher application could 
create the pipe and give a pipe ID as an argument to the "set term x11" 
command.)  From the gnuplot_x11 side of things it is fairly 
straightforward.  The work lies on the gnuplot side of things:

1)  Fix up the I/O scheme so that gnuplot doesn't wait on keyboard 
input.  Whenever keyboard input is available then parse it.

2)  A system for interpreting primitives coming across the various 
"mouse pipes"... or can I take the liberty of being ultra-silly here and 
coin the phrase "mouse holes", or better still "mouse traps" :) ... 
 Seriously, it would involve a mechanism for keeping track of the pipes, 
a linked-list sort of thing for the default handler.  Or an alternative 
would be one mouse hole (sick of this phrase already?) and the primitive 
info would include a plot number along with it.  [I'd prefer a linked 
list of connections, myself.]

3)  Since gnuplot only has a "current plot" replot and not information 
for all its plots, there needs to be some variable that keeps track of 
which plot number corresponds with the "current plot" replot info.  That 
way, the "replot" command can be reissued when the window number and the 
replot info match.

It sounds like a lot of work, but I don't think it is IMHO.  Linked 
lists are fairly straightforward, especially when they are so simple. 
 Putting effort into a new scheme seems wiser to me than attempting to 
resolve the race condition.  Are there unforeseen problems with losing 
primitives in the pipe somehow?  I don't know.

Any thoughts anyone?  Is keeping the mouse and keyboard separate a 
necessity?

Dan

Re: pasting code into gnuplot

[Arnd B. <arn...@we...>]

On Fri, 9 Jan 2004, Hans-Bernhard Broeker wrote:

> On Thu, 8 Jan 2004, Arnd Baecker wrote:
>
> > After this the mouse coordinates in the plot window
> > are not updated anymore.
>
> That's not exactly a surprise.  When you made a plot to some other
> terminal driver, you essentially decoupled gnuplot_x11 from gnuplot,
> making several kinds of mouse interaction unavailable.  IOW,
> 'set term post ; replot' essentially acts as a (temporary) equivalent
> of 'set mouse off'.

Ok, I wasn't aware of this - so the solution ist pretty easy,
just add a replot at the end, i.e.
  set xrange [0:10]
  print "blurb"
  plot sin(x)
  set out "tst.ps"
  set term post
  rep
  set out
  set term x11
  replot
and the mouse works fine again.

Many thanks,

Arnd

Re: pasting code into gnuplot

[Hans-Bernhard B. <br...@ph...>]

On Thu, 8 Jan 2004, Arnd Baecker wrote:

> After this the mouse coordinates in the plot window
> are not updated anymore. 

That's not exactly a surprise.  When you made a plot to some other
terminal driver, you essentially decoupled gnuplot_x11 from gnuplot,
making several kinds of mouse interaction unavailable.  IOW,
'set term post ; replot' essentially acts as a (temporary) equivalent
of 'set mouse off'.

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: pasting code into gnuplot

[Arnd B. <arn...@we...>]

Hi,

sorry for creating confusion, both machines now behave the same way.
(It seems that my first checkout from CVS still contained the old
version. To check on the other machine I did another checkout
a couple of hours later ... - mea culpa, I should have thought
about the sourceforge delay ;-)

To summarize:
  - pasting works
  - piping works

However, I observe the following (small) glitch:
Again pasting the following
  plot sin(x)
  set out "t1.ps"
  set term post
  rep
  set term x11
  set out
After this the mouse coordinates in the plot window
are not updated anymore. After pressing
 "b" or "7"   (`builtin-toggle-ratio`) or zooming
it works again.


And now for something related ;-):
To make me fully happy I would like to ask if someone
has an idea on the following:

I would like to do something like this
#####################################
from GnuplotBiDir import Gnuplot
gp=Gnuplot()
gp("set mouse")       # activate mouse
gp("plot sin(x)")
print "Now get coordinates of a mouse-click:"
gp("pause mouse 'click with mouse' ")
#raw_input("and press enter here after that")
mouse_x=gp.getvar("MOUSE_X")
mouse_y=gp.getvar("MOUSE_Y")
print mouse_x,mouse_y
######################################
Here gp("...") sends the string to gnuplot via a pipe.
and gp.getvar("variable") fetches the variable given as argument.

Of course, this cannot work as the calling program does not
wait for the mouse-click. Uncommenting the raw_input
allows for the expected behaviour, but one has to press
enter after the mouse-click.
Is there a better way for doing this ?


Again many thanks,

Arnd

Re: pasting code into gnuplot

[Petr M. <mi...@ph...>]

> If so, they've kept themselves remarkably quiet, or haven't even
> noticed the problem yet.
>
> There are only a few noteworthy candidates besides X11, to begin with:
>
> *) OS/2

There are no problems there. There is a bidirectional named pipe
gnuplot/pm.trm <=> gnupmdrv.exe, and mousing commands gnupmdrv => gnuplot
go via shared memory and an event semaphore. Commands from user/gnuplot and
mouse/hotkey/gnupmdrv don't mix because the waiting for mouse commands is
encapsulated in a thread, see command.c.

---
Petr Mikulik

Re: pasting code into gnuplot

[Arnd B. <arn...@we...>]

Hi,

On Wed, 7 Jan 2004, Ethan Merritt wrote:

> On Wednesday 07 January 2004 15:54, Ethan Merritt wrote:
> >
> > if we are going to
> > switch to unbuffered mode we should first make sure the
> > buffer is empty, or else store it elsewhere for later execution.
> >
> > I don't yet see a trivial way of doing this, but ...
>
> Trivial indeed. I was focused on the "or else", but in fact
> the first alternative is the easy one.  The buffer is empty
> when the program starts, so I moved the call to setvbuf()
> out of X11_init() and put it right at the start of plot.c where
> the other I/O streams are being initialized.
>
> The X11 paste buffer is now working, and piping through awk
> now works too.  I left the call to setvbuf() protected by
> #ifdef X11 but I doubt that it needs to be.  Are any non-X11
> platforms also suffering lost characters in the input stream?

Well, for me the picture is slightly different:
 a) on a PIV machine: no problems, pasting works, piping works
 b) on my PIII (which I used to do the testing):
    - pasting does not work the first time
    - piping seems to work fine now

    I haven't done thorough testing yet, but this seems
    to improve!

Is the pasting problem related with what Hans just wrote/suggested
"I.e. don't switch off buffering halfway into the program, but
immediately after startup"?
((Speculating: maybe a "fast" machine is fast enough to switch off
buffering before something is written to the buffer?))

Many thanks,

Arnd