gnuplot-beta — Developers and beta testers of new gnuplot code

Re: gplt_x11.c: undeclared SELECT_TYPE_ARG1

[Lars H. <lhe...@us...>]

Petr Mikulik writes:
> Compiling gplt_x11.c fails under OS/2 -- I have XFree86 3.3.2 -- can someone
> help? I guess this is some kind of new code -- there are no SELECT_TYPE_*
> constants in any X11 header file.
 
 config/config.* need to be updated.

2004-01-07  Lars Hecking  <lhe...@nm...>

        * configure.in, configure.vms, src/fit.c, src/gplt_x11.c, src/stdfn.h,
        term/ggi.trm, term/x11.trm: Use AC_FUNC_SELECT_ARGTYPES instead of
        AC_FUNC_SELECT and related changes.

        * config/config.amg: Updated.

Release plan for gnuplot 4.0

[Petr M. <mi...@ph...>]

I propose to release gnuplot 4.0 now -- within 1 or 2 weeks max.

From the implementation point of view, the following should be done:

* Fix gplt_x11.c to compile on OS/2 again (my mail from few minutes ago)

* SF Bug "885279 Reading '%s' timefmt is not 64 bit safe" to be included
  or not?


Other changes needed (Lars probably knows best what to change):
* the VERSION, README, faq/ et al files need to be upgraded to 4.0.
* Web page should be updated.


Optional: release gnuplot-3.8k right now as 4.0 prerelease, and 4.0 when
all the files with new version 4 are edited.


***

Further, gnuplot deserves new web page. I've made some work for that. Please
have a look at
    physics.muni.cz/~mikulik/gp/gpweb-001.zip

It should replace the current www.gnuplot.info and gnuplot.sf.net.
I will also close my gnuplot section on my web page -- I contributed its
info into the proposed main gnuplot web. Maybe someone also wants to do this
as well.

You are welcome to send patches or replacements etc to me. But avoid writing
"please write about ..." -- do it yourself and send me the solution. Well,
the most economic way for me is if somebody else takes care about the web
... especially, if people don't like my proposed design.

The proposed images for the main site could be replaced by some more
beautiful, like here:
http://ayapin.film.s.dendai.ac.jp/~matuda/Gnuplot/pm3d.html
You can propose something.

Can someone contribute link to an archive of the gnuplot newsgroups?

Have fun too.

---
Petr Mikulik

gplt_x11.c: undeclared SELECT_TYPE_ARG1

[Petr M. <mi...@ph...>]

Compiling gplt_x11.c fails under OS/2 -- I have XFree86 3.3.2 -- can someone
help? I guess this is some kind of new code -- there are no SELECT_TYPE_*
constants in any X11 header file.


gcc -DOS2 -DHAVE_CONFIG_H -ffloat-store -O4 -mpentium -Wall
-Wno-unused -Wno-comment -Zmt -DPM3D -Zmt -DUSE_MOUSE -DHAVE_CONFIG_H
 -ffloat-store -IE:/XFree86/include -c -o gplt_x11.o gplt_x11.c

gplt_x11.c: In function `mainloop':
gplt_x11.c:724: `SELECT_TYPE_ARG1' undeclared (first use in this function)
gplt_x11.c:724: (Each undeclared identifier is reported only once
gplt_x11.c:724: for each function it appears in.)
gplt_x11.c:724: parse error before `nfds'
gplt_x11.c:741: `nfds' undeclared (first use in this function)
gplt_x11.c:784: `SELECT_TYPE_ARG234' undeclared (first use in this function)
gplt_x11.c:784: `SELECT_TYPE_ARG5' undeclared (first use in this function)
gplt_x11.c:784: parse error before `timer'
gplt_x11.c:723: warning: `nf' might be used uninitialized in this function
make: *** [gplt_x11.o] Error 1

using columns for labels

[Giuseppe G. N. A. <Giu...@ct...>]

Hi,

how can one use the content (in general, an alphanumeric one) of a given 
column to produce a label?

Say, something of the kind:

plot 'datafile.dat' u 1:2 t "$3" w p

where $3 would be expected to pick the content of the 3rd column
(presently, it just gives the string $3 invariably).

Thanks.

Giuseppe.

Re: faq: which terminal for OpenOffice.org

[Petr M. <mi...@ph...>]

Except for OO, please contribute TeX and metapost/font very below...


> > There is a question "How do I include my graphs in <word processor>?"
> > I think it would be useful to complete it by "in OpenOffice.org". Can
> > someone worked this out?
>
> OpenOffice on Windows supports copy-paste from gnuplot, and in general,
> OOo can handle several file formats gnuplot generates: EMF, WMF, EPS (with
> the usual caveat that you need a PS printer), DXF and a whole slew of
> pixel formats (PNG, P[BGP]M, GIF, JPG).  The major difference between OOo
> and MS Office is that there don't seem to be import filters for CGM, HPGL
> or CorelDraw images.  But maybe I just didn't look closely enough.
>
> In other words: the text, as it is right now, mostly fits OpenOffice.org.

png and bitmaps are fine; or vectorial formats, I've tried gnuplot + OO on
Linux:
- emf is OK, in colors
- dxf lacks many features
- wmf terminal is not available on gnuplot/Linux

Can someone try gnuplot with wmf and put it onto OO? What are differences to
emf? Which has smaller size and more features => thus what we can recommend?
If the analysis is positive, wmf generation should be switched on for every
platform.

***

I propose these changes to the FAQ -- please update it as necessary so that
I can commit it:

Many word processors can use Encapsulated PostScript for graphs.
This can be generated by the \verb+set terminal postscript eps [color]+
command.
Note that it is a good idea to check and correct the bounding box of the
graphs in the eps files (manually or by the fixbb script from gnuplot
webpage), as you have to correct this box for any eps figure produced by
whichever program.
Some (most?) word processors do not preview the actual image in the eps
file, and you have to add the preview image yourself. You can use the GSView
viewer for this (available for OS/2, Windows and X11), or ??? some Unix ps
tool???. Note that the preview image increases size of the eps file; the
smallest increase you may get by choosing Tiff 6 Packbits.

Into some office applications, like OpenOffice.org, or into applications in
the Windows world, you can insert vectorial images produces by the emf or
wmf terminal type.

With \TeX, it depends on what you use to print your dvi files...

With \TeX processed by pdftex or pdflatex, you can use png and pdf terminal
types.


???? Metapost ??? Metafont ???

Re: faq: which terminal for OpenOffice.org

[Hans-Bernhard B. <br...@ph...>]

On Tue, 10 Feb 2004, Petr Mikulik wrote:

> I've made some updates to faq.
> 
> There is a question "How do I include my graphs in <word processor>?"
> I think it would be useful to complete it by "in OpenOffice.org". Can
> someone worked this out?

OpenOffice on Windows supports copy-paste from gnuplot, and in general,
OOo can handle several file formats gnuplot generates: EMF, WMF, EPS (with
the usual caveat that you need a PS printer), DXF and a whole slew of
pixel formats (PNG, P[BGP]M, GIF, JPG).  The major difference between OOo
and MS Office is that there don't seem to be import filters for CGM, HPGL
or CorelDraw images.  But maybe I just didn't look closely enough.

In other words: the text, as it is right now, mostly fits OpenOffice.org.

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

faq: which terminal for OpenOffice.org

[Petr M. <mi...@ph...>]

I've made some updates to faq.

There is a question "How do I include my graphs in <word processor>?"
I think it would be useful to complete it by "in OpenOffice.org". Can
someone worked this out?

---
Petr Mikulik

Re: [Gnuplot-bugs] bugs/problems in gnuplot 3.8k (cvs)

[<br...@ph...>]

> 1) pdf driver:  that's a great new driver -- I really love that addition!
[...]
> somehow, other pdf tools can inform acroread to set the page setup to
"landscape",
> so printing is ok without twiddling with page setup settings first.

Well, knowing the fact that other programs can manage this doesn't exactly
help us implement
it for ourselves...  We would need to know *how* they do it.

> 2) readline(?), SIGTSTP, and gnuplot_x11 communication  (and libreadline?)

As far as I can see, at least a significant part of that problem lies with
libreadline itself.  gnuplot
should probably catch SIGTSTP and do something with it, though.

> 3) I'm just starting to use gnuplot on M$ Win-XY for my first time.
> obviously it's great too, but by default I'm hardly missing the
> possibility to use shell commands in plot "datafile" using
>
> plot "< some command"
>
> but by default PIPES config is not set for windows builds :-(

That's because it has a side effect which the average Windows user would
quite certainly be quite irritated about: it opens an extra console window
if started from
outside a console.  The average Windowser would perceive that as a bug of
the program, not
as a feature --- I've been there and got quite a collection of mails about
it, when I accidentally put
a similar behaviour in the first distribution of the 3.7.3 W32 build...

I've been contemplating a more Unix-ish Win32 version, i.e. a console
application that still has a
GUI graph window --- I had long thought this was utterly impossible to do,
but it appears I may have
been wrong.  I made some inroads, but haven't quite reached a usable overall
result yet.

OTOH, I don't see what's stopping you from building your own w32 version
with -DPIPES=1
enabled...

> I've read your comments about PIPES in config/makefile.mgw
> but for me (Windows novice) it's unclear if this is a problem
> for _all_ windows versions (e.g. would WinXP be any better?).

The problem doesn't depend on the version of Windows --- it's a fundamental
difference in the
way Unix and Windows work.  X11 programs have a plain stdin and stdout
channel, just like
console ones, whereas Windows GUI apps don't. This was even worse in Win16,
which wgnuplot
was originally made for, and which made that silly home-grown terminal
emulator a necessity.

> right now I still hope (dream?) that it'll be possible to use MSYS
> to allow "real" shell commands in plot "< ..."

What exactly is "MSYS"?  Anyway: if you miss Unix-ish behaviour real bad, I
would suggest
you get yourself cygwin (including X11) installed ;->

Re: Remove path to VERSION file, this is now taken of by TEXINPUTS.

[Petr M. <mi...@ph...>]

> If they're sufficiently Unix-like, I see no particular reason why the
> approach in docs/Makefile.in shouldn't work for makefile.mgw, too. Or you
> can use the "export"  directive of GNU make, i.e.
>
>       export TEXINPUTS=... whatever...

It doesn't work.

Thus I've chosen "copy ../VERSION ." which works OK.

---
Petr Mikulik

howto drive wgnuplot for the records

[kemal a. <as...@cl...>]

I would like to put the following  in the records.
thanks to :
Hans-Bernhard Broeker <br...@ph...>.
here is a working solution to drive gnuplot  Version 3.7 patchlevel 3.
on windows2K built with VC6.0.


// consol.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <iostream>
#include <stdio.h>

using namespace std;


int main(void)
{
        FILE *chkdsk;
        if ( (chkdsk = _popen( "pgnuplot.exe", "wrt")) !=
NULL)
        {
                fputs("plot sin(x)\n", chkdsk);
                fflush(chkdsk);  //Works perfectly now
                system("pause"); //needed otherwise gnuplot closes when
//the console app ends to fast to see anything
        }
		printf("DONE!\n");

        return 0;

}

libgd + symbol fonts [was Re: Universal enhanced text mode!]

[Ethan M. <merritt@u.washington.edu>]

Just an update.

I added this code to CVS back in November, with a caveat that full
support for symbol fonts in the png/jpeg terminal required a patch
for libgd.

I am delighted to report that Tom Boutell has added the patch into
libgd. As of version 2.0.21 (now available for download) both the
Microsoft symbol.ttf and the Adobe Symbol fonts are handled correctly,
and "just work" from gnuplot.  Actually any Adobe custom-encoded
font should work, not just Symbol. Apple fonts should work also,
but I haven't tested them.



iso-8859-xx encodings
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

A related relatively minor problem remains, however, in that
there is currently no mechanism for translating the requested
character encoding from gnuplot through libgd to the font libraries.

libgd's own native font (default if you don't specify one) happens
to be iso-8859-2.   When you request a specific TTF font you=20
get iso-8859-1 so far as I can tell, but I am not sure if this is
guaranteed. =20

There is a mechanism for requesting 8-bit unicode, Shift-JIS, or big5=20
characters, however. So in principal gnuplot could  convert strings
to unicode if it wants characters not in iso-8859-1, but that
sounds like too much work.  I may see if I can further hack
libgd itself to know more about character encodings in some future
release.

--=20
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center Box 357742
University of Washington, Seattle, WA 98195

Re: Remove path to VERSION file, this is now taken of by TEXINPUTS.

[Hans-Bernhard B. <br...@ph...>]

On Fri, 16 Jan 2004, Petr Mikulik wrote:

> 	Hi Lars,
> 
> > Modified files:
> >        gnuplot/docs/: titlepag.tex
> >
> > Log message:
> >        Remove path to VERSION file, this is now taken of by TEXINPUTS.
> 
> well, it seems it is not easily portable by setting TEXINPUTS in a Makefile
> for different systems, like Windows and OS/2. Do you know how to set an
> environmental variable in Makefiles for these systems? 

If they're sufficiently Unix-like, I see no particular reason why the
approach in docs/Makefile.in shouldn't work for makefile.mgw, too. Or you
can use the "export"  directive of GNU make, i.e.

	export TEXINPUTS=... whatever...

There's an older method using the special target ".EXPORT:", but I'm not
sure whether that still exists.  It's not documented any longer, it seems.

> Thus, do you really prefer the above "copy" method, or would rather go back
> to \openin ../VERSION as it was until now?

\openin ../VERSION breaks in other ways, so we can't usefully go back to
that (it fails to work in out-of-source-directory builds on Unix-like
platforms).

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: Remove path to VERSION file, this is now taken of by TEXINPUTS.

[Lars H. <lhe...@us...>]

> The only solution I've found is to add "cp VERSION" command here:
> 
> gnuplot.dvi: gnuplot.tex $(D)titlepag.tex
>     cp gnuplot.tex $(D)gp_tex2.tex
>     cp VERSION $(D)
>     cd $(D) && latex gp_tex2.tex && latex gp_tex2.tex && latex gp_tex2.tex
> 
> Thus, do you really prefer the above "copy" method, or would rather go back
> to \openin ../VERSION as it was until now?

 It breaks when you build outside the source dir, that's why the change
 was made.

Remove path to VERSION file, this is now taken of by TEXINPUTS.

[Petr M. <mi...@ph...>]

	Hi Lars,

> Modified files:
>        gnuplot/docs/: titlepag.tex
>
> Log message:
>        Remove path to VERSION file, this is now taken of by TEXINPUTS.

well, it seems it is not easily portable by setting TEXINPUTS in a Makefile
for different systems, like Windows and OS/2. Do you know how to set an
environmental variable in Makefiles for these systems? I tried it for
Makefile.mgw, but haven't found any way to do it.

The only solution I've found is to add "cp VERSION" command here:

gnuplot.dvi: gnuplot.tex $(D)titlepag.tex
    cp gnuplot.tex $(D)gp_tex2.tex
    cp VERSION $(D)
    cd $(D) && latex gp_tex2.tex && latex gp_tex2.tex && latex gp_tex2.tex

Thus, do you really prefer the above "copy" method, or would rather go back
to \openin ../VERSION as it was until now?

---
Petr Mikulik

with_image_120303.diff

[Arnd B. <arn...@we...>]

Hi,

next I wanted to try the with_image patch,
(I followed the guideline for patching from Ethan,
http://www.bmsc.washington.edu/people/merritt/gnuplot/instructions.html
)

However, patching color.c and gplt_x11.c I get

patching file src/color.c
Hunk #2 FAILED at 525.
1 out of 2 hunks FAILED -- saving rejects to file src/color.c.rej

patching file src/gplt_x11.c
Hunk #2 FAILED at 373.
Hunk #3 succeeded at 483 (offset 2 lines).
Hunk #4 succeeded at 582 (offset 3 lines).
Hunk #5 succeeded at 1441 (offset 1 line).
Hunk #6 succeeded at 2217 (offset 1 line).
Hunk #7 FAILED at 2319.
Hunk #8 succeeded at 3003 (offset 1 line).
Hunk #9 succeeded at 3390 (offset 1 line).
2 out of 9 hunks FAILED -- saving rejects to file src/gplt_x11.c.rej

I had a look at gplt_x11.c but I don't understand
the code (well enough ;-) to fix this...
((If someone has the time at some point for doing this
I promise to test the patch, in which I am very much interested ;-))

Is there any chance that it gets integrated into the CVS version ?

Many thanks,

Arnd

Re: Request for including your software in our database

[Hans-Bernhard B. <br...@ph...>]

[I'm cc'ing this reply to the developers list informational purposes...]

Feel free to go right ahead with this.  

But, for the future, please note that mail like this had better to go the
administrative contact addresses mentioned on the opening screen of the
program, or in the documentation, rather than individual developers...

If you offer the program for download, we may want to link to your
page as an additional mirror site, too, so please send us a linkable
URL once you have completed construction of this database of yours.

On Thu, 15 Jan 2004, Dr G P S Raghava wrote:

> We are maintaining a database of free softwares for general purpose
> (www.imtech.res.in) which are available free of cost for general user
> and academic community. The aim of this project is to help the
> scientific community in distributing these software. We are a academic
> institute are not charging any money from users who wish to download
> these software from our site. we have found your software to be useful
> for general Information Technology professionals, general computer users
> and others also. We wish to include your software in our next database
> release. We already have a copy of software from web. In case , you do
> not agree please initimate us, for removal of your software from our
> database. presently your software will be available from our site i.e.
> gnuplot (software name) at www.imtech.res.in/fsgp/. whenever you are not
> interested to distribute these software from our site , please write to
> us . We will remove your software from our database. Initially we plan
> to provide a link to the original software but later we have realized
> that a number of software are not available from net or their original
> site is not accessible or some sites may close frequently. As well as
> some users do not have internet connections particularly in developing
> countries , so off-line distribution will help them. Thus, we thought we
> should keep one copy at our site and also provide a link to original
> site for downloading these software. We encourage users to download
> software from their original site rather than from our site (We clearly
> mention on home page of the database). This way user will get latest
> version. We also mention clearly that user should read terms and
> conditions and copy right statement laid for users.

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: pasting code into gnuplot

[Arnd B. <arn...@we...>]

Hi,

for me both pasting from X11 and piping (from python)
works great now - Many many thanks   !

The example for accessing gnuplot variables
and waiting for mouse-clicks from python can be obtained as
  http://www.physik.tu-dresden.de/~baecker/python/GnuplotBiDir.py
see
  http://www.physik.tu-dresden.de/~baecker/python/gnuplot.html
for a few notes.

Arnd

Re: pasting code into gnuplot

[Ethan A M. <merritt@u.washington.edu>]

On Thursday 08 January 2004 12:05 pm, Arnd Baecker wrote:
> I would like to do something like this
> #####################################
> from GnuplotBiDir import Gnuplot
> gp=Gnuplot()
> gp("set mouse")       # activate mouse
> gp("plot sin(x)")
> print "Now get coordinates of a mouse-click:"
> gp("pause mouse 'click with mouse' ")
> #raw_input("and press enter here after that")
> mouse_x=gp.getvar("MOUSE_X")
> mouse_y=gp.getvar("MOUSE_Y")
> print mouse_x,mouse_y
> ######################################

> Of course, this cannot work as the calling program does not
> wait for the mouse-click.

I have placed a fix for this in CVS.  'pause mouse' was terminating
on either a mouse click or a <cr> on stdin.  That is OK for
terminal input, but with piped input there may already be multiple
commands backed up in the pipe, in which case it would not wait
for a mouse click.  As of now the behaviour is changed to only
terminate 'pause mouse' on a mouse click or on ctrl-C.
I also added some sanity checking so that you cannot get
stuck waiting for 'pause mouse' if mousing is not active.

Note that requesting "set mouse" before plotting is essential.
The default for piped input is to disable mousing.

-- 
Ethan A Merritt
Department of Biochemistry & Biomolecular Structure Center
University of Washington, Seattle

Re: pasting code into gnuplot

[Daniel J S. <dan...@ie...>]

Ethan A Merritt wrote:

>On Saturday 10 January 2004 01:07 pm, Petr Mikulik wrote:
>  
>
>>  Or would this mean big non-portability to different X11 systems?
>>    
>>
>
>I don't know.  But if we were going to move to a threaded implementation
>of gnuplot, I would want to look at making even larger scale use of it.
>For example, would it make any sense to spawn off a new thread for each
>new plot?  That would be an easier way to achieve multiple active
>plots than Dan's current dream of making all the internal data structures
>into linked lists.
>

There goes my pipe dream.

Dan

Re: pasting code into gnuplot

[Hans-Bernhard B. <br...@ph...>]

On Sat, 10 Jan 2004, Petr Mikulik wrote:

> As I said earlier, I don't know about the techniques you describe in the
> previous letters .. but cannot be something modern used to get rid of these
> problems? 

Modern solutions have the crucial problem of being modern; meaning that
they're not particularly unified or reliably available all across the
spread of platforms we're dealing with.

>   Or would this mean big non-portability to different X11 systems?

I strongly suspect so.

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: pasting code into gnuplot

[Ethan A M. <merritt@u.washington.edu>]

On Saturday 10 January 2004 01:07 pm, Petr Mikulik wrote:
> cannot be something modern used to get rid of these
> problems? E.g. pthreads. There can be a thread waiting for mouse/hotkey
> actions, and it may be triggered by mutexes or semaphores.

Putting all the mouse handling in a separate thread would work (I think).
I don't at the moment see how it would work for hotkeys, though.
A hot-key triggered command should be treated identically to the
same thing typed from the command line, including effects on subsequent
plotting commands.  Is that possible with pthreads?  That is, are
changes to global variables in one thread seen by the others?

> Get rid of stdin to gnuplot_x11 and use named pipe or fifo.

I am not aware of a problem with the input stream to gnuplot_x11.
What did you have in mind?  The problem we have at the moment
is with multiple inputs to gnuplot itself.

>   Or would this mean big non-portability to different X11 systems?

I don't know.  But if we were going to move to a threaded implementation
of gnuplot, I would want to look at making even larger scale use of it.
For example, would it make any sense to spawn off a new thread for each
new plot?  That would be an easier way to achieve multiple active
plots than Dan's current dream of making all the internal data structures
into linked lists.


-- 
Ethan A Merritt
Department of Biochemistry & Biomolecular Structure Center
University of Washington, Seattle

Re: pasting code into gnuplot

[Petr M. <mi...@ph...>]

As I said earlier, I don't know about the techniques you describe in the
previous letters .. but cannot be something modern used to get rid of these
problems? E.g. pthreads. There can be a thread waiting for mouse/hotkey
actions, and it may be triggered by mutexes or semaphores. Get rid of stdin
to gnuplot_x11 and use named pipe or fifo.
  Or would this mean big non-portability to different X11 systems?

---
Petr Mikulik

Re: pasting code into gnuplot

[Hans-Bernhard B. <br...@ph...>]

On Sat, 10 Jan 2004, Daniel J Sebald wrote:

[....]
> You'll have to forgive me.  I've tried following the code to see exactly 
> the path that the commands come in via stdin, where the pipe is read for 
> the mouse, etc. and I'm having a difficult time keeping track of it all. 

The core function is generally term->waitforinput(), which in the case of
the x11 terminal is provided by X11_waitforinput() in term/x11.trm.  
There are different call paths leading there.  Let me provide an overview.
Cscope helps:

C symbol: waitforinput

  File       Function      Line
0 term_api.h <global>       172 int (*waitforinput) __PROTO((void ));
1 command.c  pause_command  961 if (term && term->waitforinput) {
2 command.c  pause_command  964 term->waitforinput();
3 command.c  fgets_ipc     2212 if (term && term->waitforinput) {
4 command.c  fgets_ipc     2219 c = term->waitforinput();
5 readline.c getc_wrapper    84 if (term && term->waitforinput && interactive) {
6 readline.c getc_wrapper    88 c = term->waitforinput();
7 readline.c ansi_getc      775 if (term && term->waitforinput && interactive)
8 readline.c ansi_getc      776 c = term->waitforinput();

The call paths ultimately start at read_line in command.c.  I'll assume
MOUSE is active:

no READLINE (neither gnuplot's own, nor -lreadline):
	GET_STRING ==> fgets
some READLINE, !interactive: 
	fgets_ipc ==> term->waitforinput
GNU libreadline, interactive:
	read_line ==> rlgets ==> readline_ipc ==> rl_callback_read_char
	(getc_wrapper passed to libreadline as input hook)
gnuplot readline.c, interactive:
	read_line ==> rlgets ==> readline_ipc ==> readline
	==> special_getc ==> ansi_getc

So that's how we get to the core function. Now let's look at
X11_waitforinput itself: it uses select() to dispatch between the two
input streams.  The root of all our trouble is that select() operates not
on <stdio.h> objects like stdin, but rather on unbuffered Unix file
handles.  I.e. select doesn't know a thing about <stdio.h> buffering.

> I find it hard to fathom that this is not possible.  

It's quite certainly possible.  The tricky part is to make it possible
in the context of a massively portable program like gnuplot, without 
breaking the whole program for use on other platforms.

> I mean, doesn't this seem like a fundamental task of operating systems?  
> That is, to distribute I/O around the system?  Let's consider an fread()
> or a fgets() or fgetc().  

For starters, fgetc() and friends aren't even functions of the operating
system to begin with.  They're functions of the C runtime library --- but
ISO/ANSI C doesn't standardize asynchronous I/O. 

> Can't I just do an fread() from the stdin (the place where the commands
> are typed in) and whatever is there I store in a "command buffer" and as
> I'm putting the characters in a command buffer I watch for the "new
> line" or "end of gnuplot command" character.

No, you can't.  The reason is that file I/O on Unix (and all other
platforms I've seen) is by default a "blocking" operation:

> If there is nothing in the buffer or there are less characters than
> asked for, doesn't the fread() simple return and EOF or -1 or number of
> bytes read?

If there's buffered input, but less than requested, it'll return what
there is. But if there's currently _nothing_ in the buffer, it won't
return at all until there is.  You only get an EOF condition if the stream
is actually ended (typical Unix setup: if you typed Ctrl-D at the
keyboard), not if there's just no buffered input available at the moment.

> >F) The usual way around this dilemma is to use poll or select to notify
> >   us when either or both of two input streams is presenting new data.
> >   We currently use select in X11_waitforinput().
> >
> 
> Yes, this is one way.  But if "fread()" is inherently forced to return 
> something even if the buffer is empty, isn't that a form of polling the 
> input as well?  

The problem is we *can't* force fread() to do that.  Non-blocking I/O
is not foreseen in the context of <stdio.h> functions.

> As I said before, it seems to me that current Gnuplot 
> really isn't wanting polling for "characters available", rather it is 
> attempting to poll for "gnuplot command in buffer available" and "mouse 
> command in buffer available".

I'm quite sure that reading characters versus lines is unrelated to the
problem at hand.

> >G) Unfortunately, poll/select notification depends on whether the
> >   input stream is buffered or non-buffered.   I do not know if this is
> >   supposed to be the case, but that's what we see in practice.

Indeed select() doesn't know a thing about buffering, so it will not
report a stream as having data available if all of it is buffered.  

-- 
Hans-Bernhard Broeker (br...@ph...)
Even if all the snow were burnt, ashes would remain.

Re: pasting code into gnuplot

[Daniel J S. <dan...@ie...>]

Ethan Merritt wrote:

>On Friday 09 January 2004 14:50, Daniel J Sebald wrote:
>  
>
>>If you turn off buffering immediately after startup, what if a
>>number of Gnuplot commands are already in the buffer?
>>    
>>
>
>Since piping now works, this is demonstrated not to be a problem.
>Of course if you have a test example that fails, I will glumly
>retract that optimistic statement.
>
>  
>
>>Thus the only way to resolve the race condition, in one form
>>or another from Gnuplot, is for there to be an ability for the "setvbuf"
>>command to return the contents of the buffer right before it was
>>changed.
>>    
>>
>
>Not possible. See "man setvbuf".
>

I searched for info.  As usual, the man doesn't seem to give details of 
the tricky aspects.

>>1)  Fix up the I/O scheme so that gnuplot doesn't wait on keyboard
>>input.  Whenever keyboard input is available then parse it.
>>    
>>
>
>Now you are getting near to the heart of it.  Most of our current
>problem comes from not being able to tell when input is available
>without trying to read it.  And if we try to read it, then we are stuck
>waiting until success.
>
>The central problem is that there are few (actually none that I know of)
>portable methods of reading from two or more input sources at the same
>time.  I know perfectly well how to do it under VMS but, sadly, other
>operating systems are not so capable [grin, duck, and run].
>

You'll have to forgive me.  I've tried following the code to see exactly 
the path that the commands come in via stdin, where the pipe is read for 
the mouse, etc. and I'm having a difficult time keeping track of it all. 
 So I'm going to play devil's advocate here and ask naive questions.

I find it hard to fathom that this is not possible.  I mean, doesn't 
this seem like a fundamental task of operating systems?  That is, to 
distribute I/O around the system?  Let's consider an fread() or a 
fgets() or fgetc().  Can't I just do an fread() from the stdin (the 
place where the commands are typed in) and whatever is there I store in 
a "command buffer" and as I'm putting the characters in a command buffer 
I watch for the "new line" or "end of gnuplot command" character.  When 
the end of command character is found, then shuffle that string over to 
the current command parsing stuff.  If there is nothing in the buffer or 
there are less characters than asked for, doesn't the fread() simple 
return and EOF or -1 or number of bytes read?

Then, do the same with the mouse stream.

It sounds to me, correct me if I'm wrong, that Gnuplot is now attempting 
a graceful method of trying to get a complete command from both the 
mouse and the keyboard whenever either is available.  Is this what you 
mean by asynchronous?  If an operating system can do that, that is 
impressive.  But no, I would not expect that to be a feature of all 
operating systems.  It seems to me then, that for Gnuplot (to be 
portable) is the one that needs to do all the work of reconstructing the 
input streams.

Let me attempt a description of what I'm saying without writing the full 
code.  Say there is a "keyboard_stream" and a "mouse_stream".  We make 
internal buffers inside Gnuplot of length, say 200.

keyboard_command_buff[201];
mouse_command_buff[201];
keyboard_buff_ptr = 0;
mouse_buff_ptr = 0;

while (not_end_of_Gnuplot) {

    int number_chars;

    /* Check if there is keyboard input, if so check if a valid command 
is complete. */
    number_chars = fread(keyboard_stream, 
&keyboard_command_buff[keyboard_buff_ptr], 200-keyboard_buff_ptr);   /* 
That is, read up to whatever is space is left in the buffer */
    if (number_chars > 0) {
       [search through the characters just read and test for end of command]
       if (this is valid end of command)  {
           execute gnuplot command;
           copy any chunk left in buffer to the start of buffer;
           keyboard_buff_ptr = 0;
       } else {
           keyboard_buff_ptr += number_chars;
       }
    }
     if (end of buffer has been reached)
         [issue error to the console]


    /* Check if there is mouse input, if so check if a valid mouse 
string is complete. */
   [Do the same for the mouse stream as is done above for the keyboard.]

}


The input method just keeps going around in the loop and never waits on 
anything.  It may be that only a few characters are read at a time, but 
so what.  It isn't like this needs to be super efficient as though it 
were a time critical communication scenario.  We're waiting on mouse and 
keyboard commands which are rather slow I assume.

Is there some issue with buffered mouse characters piling up in the 
stream while a Gnuplot command is being processed?

(more below)

>The current scheme under linux  depends on select(). This scheme,
>problems and all, is the closest thing to simultaneous/asynchronous
>input that I know of in standard linux/unix.
>That said, I am hardly an expert on linux asynchronous I/O.
>
>Here is the bind we are in, as I understand it for the linux+X11 case:
>

Nice summary.  Thanks.

(more below)

>A) We have terminal input coming in on stdin; we have mouse input
>   coming in via a pipe.
>
>B) These two input sources are asynchrounous.  That is, we don't
>   know which one will be the next to provide new input.
>

The scheme I describe above attempts to handle this, I think.

>C) We have non-mouse (font, window size, etc) information coming
>   back over the pipe also. This could be split off from the mouse pipe,
>   but that does not make things any easier, so let's disregard it.
>

This should be fine I think.

>D) If you do a read/scanf/gets/whatever from stdin then you will
>   miss all mouse input until you get the next chunk of terminal input.
>

Please explain why this is.  I'm to lazy to do a test trial at this 
hour, but won't fread() on stdin just send back and EOF or -1 or 
something if there are no characters in the buffer?

>E) Conversely if you do a read/scanf/gets/whatever on the mouse pipe
>   then you can't even look for new terminal commands until after the
>   next mouse event (which may never arrive).
>
>F) The usual way around this dilemma is to use poll or select to notify
>   us when either or both of two input streams is presenting new data.
>   We currently use select in X11_waitforinput().
>

Yes, this is one way.  But if "fread()" is inherently forced to return 
something even if the buffer is empty, isn't that a form of polling the 
input as well?  As I said before, it seems to me that current Gnuplot 
really isn't wanting polling for "characters available", rather it is 
attempting to poll for "gnuplot command in buffer available" and "mouse 
command in buffer available".

>G) Unfortunately, poll/select notification depends on whether the
>   input stream is buffered or non-buffered.   I do not know if this is
>   supposed to be the case, but that's what we see in practice.
>
>H) More things seem to work when stdin is set to unbuffered,
>   hence the call to setvbuf().  Other things broke, because this caused
>   the current contents of the buffer to be lost.   Note that this behaviour
>   is not documented anywhere I can find, but it makes sense and is
>   easily demonstrated. I am optimistic that this problem is minimized
>   by moving the setvbuf() call to the head of the program.
>
>I) Just to make things a little bit worse, we have commands like
>   'pause <n>' and 'pause mouse'.
>

Hey, these are easy to handle for the scheme above.  Deactivate the 
input test according to some timer, i.e.,

if (time expired >= n) {
  n = 0;
   /* The junk above */
}

Gnuplot will simply stop checking for input in the appropriate stream 
for <n> seconds.  (This may not work... but that would be my first 
attempt at such.)

>J) Just to make things a little bit worse yet, we try to support gnu
>   libreadline,  which has its own quirks about handling input streams.
>

I would hope the this is fairly transparent to the C environment.

>K) And if all that was not complicated enough, people want to cut-and-paste
>   from X11.  This involves cooperation from the X server and from the
>   application windows (xterm, nedit, or whatever) on both the cut and the
>   paste end.  So far as I know, there is no guarantee that just because
>   it works for gnuplot running in an xterm window it will also work inside
>   konsole or kterm or Eterm or <alphabetofyourchoice>term.
>

Again, I would hope this is transparent to the C environment.  The 
cutting and pasting simply puts stuff into the "stdin" buffer, I'm guessing.

>
>Now that I understand the problem with setvbuf(), I am considerably
>more hopeful that we can get everything to work using the current
>scheme.  More hopeful than I was a week ago.
>poll/select is *supposed* to work, after all.
>But items I, J and K may still cause us problems.
>
>  
>
>>3)  Since gnuplot only has a "current plot" replot and not information
>>for all its plots, there needs to be some variable that keeps track of
>>which plot number corresponds with the "current plot" replot info.  That
>>way, the "replot" command can be reissued when the window number and the
>>replot info match.
>>    
>>
>
>That is a separate topic.
>Let's not mix the two.
>

Right.  That is something completely different.  I'm confused enough the 
way it is.

Anyway, tell me what is wrong with the approach I describe.  I guess my 
main point is that if you want Gnuplot to be portable, it's the one that 
will have to do all the work in determining "valid command line string 
available" and "valid mouse string available", not the system.

Dan

Re: pasting code into gnuplot

[Ethan M. <merritt@u.washington.edu>]

On Friday 09 January 2004 14:50, Daniel J Sebald wrote:
> If you turn off buffering immediately after startup, what if a
> number of Gnuplot commands are already in the buffer?

Since piping now works, this is demonstrated not to be a problem.
Of course if you have a test example that fails, I will glumly
retract that optimistic statement.

> Thus the only way to resolve the race condition, in one form
> or another from Gnuplot, is for there to be an ability for the "setvbuf=
"
> command to return the contents of the buffer right before it was
> changed.

Not possible. See "man setvbuf".

> I think the best solution here is to keep the two input streams
> independent.
[snip rest of long description]

What you describe is exactly what we are already doing.
Keeping track of multiple pipes is not a problem. Figuring out
how to read from more than one at a time *is* a problem.

> 1)  Fix up the I/O scheme so that gnuplot doesn't wait on keyboard
> input.  Whenever keyboard input is available then parse it.

Now you are getting near to the heart of it.  Most of our current
problem comes from not being able to tell when input is available
without trying to read it.  And if we try to read it, then we are stuck
waiting until success.

The central problem is that there are few (actually none that I know of)
portable methods of reading from two or more input sources at the same
time.  I know perfectly well how to do it under VMS but, sadly, other
operating systems are not so capable [grin, duck, and run].

The current scheme under linux  depends on select(). This scheme,
problems and all, is the closest thing to simultaneous/asynchronous
input that I know of in standard linux/unix.
That said, I am hardly an expert on linux asynchronous I/O.

Here is the bind we are in, as I understand it for the linux+X11 case:

A) We have terminal input coming in on stdin; we have mouse input
   coming in via a pipe.

B) These two input sources are asynchrounous.  That is, we don't
   know which one will be the next to provide new input.

C) We have non-mouse (font, window size, etc) information coming
   back over the pipe also. This could be split off from the mouse pipe,
   but that does not make things any easier, so let's disregard it.

D) If you do a read/scanf/gets/whatever from stdin then you will
   miss all mouse input until you get the next chunk of terminal input.

E) Conversely if you do a read/scanf/gets/whatever on the mouse pipe
   then you can't even look for new terminal commands until after the
   next mouse event (which may never arrive).

F) The usual way around this dilemma is to use poll or select to notify
   us when either or both of two input streams is presenting new data.
   We currently use select in X11_waitforinput().

G) Unfortunately, poll/select notification depends on whether the
   input stream is buffered or non-buffered.   I do not know if this is
   supposed to be the case, but that's what we see in practice.

H) More things seem to work when stdin is set to unbuffered,
   hence the call to setvbuf().  Other things broke, because this caused
   the current contents of the buffer to be lost.   Note that this behavi=
our
   is not documented anywhere I can find, but it makes sense and is
   easily demonstrated. I am optimistic that this problem is minimized
   by moving the setvbuf() call to the head of the program.

I) Just to make things a little bit worse, we have commands like
   'pause <n>' and 'pause mouse'.

J) Just to make things a little bit worse yet, we try to support gnu
   libreadline,  which has its own quirks about handling input streams.

K) And if all that was not complicated enough, people want to cut-and-pas=
te
   from X11.  This involves cooperation from the X server and from the
   application windows (xterm, nedit, or whatever) on both the cut and th=
e
   paste end.  So far as I know, there is no guarantee that just because
   it works for gnuplot running in an xterm window it will also work insi=
de
   konsole or kterm or Eterm or <alphabetofyourchoice>term.

Now that I understand the problem with setvbuf(), I am considerably
more hopeful that we can get everything to work using the current
scheme.  More hopeful than I was a week ago.
poll/select is *supposed* to work, after all.
But items I, J and K may still cause us problems.

> 3)  Since gnuplot only has a "current plot" replot and not information
> for all its plots, there needs to be some variable that keeps track of
> which plot number corresponds with the "current plot" replot info.  Tha=
t
> way, the "replot" command can be reissued when the window number and th=
e
> replot info match.

That is a separate topic.
Let's not mix the two.

--=20
Ethan A Merritt       merritt@u.washington.edu
Biomolecular Structure Center Box 357742
University of Washington, Seattle, WA 98195