gnuplot-beta — Developers and beta testers of new gnuplot code

Re: Variable color demo page (website)

[Ethan A M. <sf...@us...>]

On Friday, October 05, 2012 01:57:44 pm Tait wrote:
> 
> Someone just asked, "how do I map a 4th dimension to line
> color?" I was surprised to discover that the varcolor demo,
> which answers exactly this question, is nowhere to be found on
> http://gnuplot.sourceforge.net/demo_4.6/.
> 
> How does one go about updating/creating a patch for the demo pages?

If you want to contribute a new demo, go right ahead.

Or are you asking how to get an existing demo like varcolor.dem 
included in the on-line collection?  
The thing is, there's not room on one page to index
every demo in the collection.   I thought that these two already
covered the case in question

  http://gnuplot.sourceforge.net/demo_4.6/rgb_variable.html
  http://gnuplot.sourceforge.net/demo_cvs/heatmaps.html

But it is correct that one uses points and the other surfaces,
so perhaps it isn't obvious that the same commands would work
for a plot "with lines".

	Ethan

Variable color demo page (website)

[Tait <gnu...@t4...>]

Someone just asked, "how do I map a 4th dimension to line
color?" I was surprised to discover that the varcolor demo,
which answers exactly this question, is nowhere to be found on
http://gnuplot.sourceforge.net/demo_4.6/.

How does one go about updating/creating a patch for the demo pages?

[PATCH] better citizen gnuplot emacs mode

[<gn...@di...>]

The gnuplot emacs mode was creating bindings in the global comint mode,
overwriting those that were already there. This resulted in breakage when those
keys were used in other non-gnuplot comint buffers. Attached patch applies the
bindings ONLY to the gnuplot comint instance.

dima

Re: SourceForge reorganization may affect gnuplot development site

[Tait <gnu...@t4...>]

>>> I'm not sure the repositories themselves are being replaced, but the
>>> other items are changing (e.g., bug tracker, etc.).  However, CVS seems
>>> to be downplayed a bit and only SVN, mercurial and git are listed as
>>> supported.
>>
>> Probably not at this point, but I suspect that sooner or later CVS
>> will be phased out and it might happen that it won't be supported in
>> future at all. [...]
>
> I would say that either git or mercurial would be a better choice than
> SVN.

(Disclosure: I like git; I find mercurial gets underfoot and is clumsy.
But nothing is worse than CVS.)

SVN has one distinct advantage over both git and mercurial: it is
the lowest-common denominator. Both Mercurial and Git have excellent
utilities that allow the end user to use git/mercurial to interact with
an SVN server. Those who don't want to learn or be forced into a newer
VCS like git or mercurial can use SVN itself. SVN becomes the best of
all worlds. It retains familiarity for CVS users, and git/mercurial fans
can use whichever they prefer and have most or all of the benefits of
their tool of choice.

Someone mentioned Perforce. This is a commercial product whose licenses
are not free. I don't think we -- or SourceForge -- will adopt it. The
feedback I've heard on it has mostly been negative. Bazar would be worth
consideration, but it's not one of the choices listed by SourceForge.

Re: SourceForge reorganization may affect gnuplot development site

[Daniel J S. <dan...@ie...>]

On 10/04/2012 10:15 PM, Allin Cottrell wrote:
> On Thu, 4 Oct 2012, Daniel J Sebald wrote:
>
>> On 10/04/2012 04:35 AM, Mojca Miklavec wrote:
>>> On Thu, Oct 4, 2012 at 10:29 AM, Daniel J Sebald wrote:
>>>>
>>>> I'm not sure the repositories themselves are being replaced, but the
>>>> other items are changing (e.g., bug tracker, etc.).  However, CVS seems
>>>> to be downplayed a bit and only SVN, mercurial and git are listed as
>>>> supported.  So, if when upgrading, gnuplot is forced to choose one of
>>>> those three, then it is essentially the same as replacing.  Not that
>>>> "upgrading" from CVS is bad.  (I've used mercurial pretty much and that
>>>> seems nice.)
>>>
>>> Probably not at this point, but I suspect that sooner or later CVS
>>> will be phased out and it might happen that it won't be supported in
>>> future at all. [...]
>>
>> I would say that either git or mercurial would be a better choice than
>> SVN.
>
> No doubt opinions will differ on this. Svn is closest to CVS, and
> therefore easiest to learn for people who are used to CVS, while
> offering various advantages over the latter. Git is very
> full-featured and is the VC system used by many leading open-source
> projects (notably the Linux kernel and the GNU C library) and
> therefore may be worth learning even though it's not close to CVS.
>
> For me personally, mercurial would be the last choice, simply
> because up to this point no software that I'm interested in resides
> in a mercurial repository.

Another thing to consider is the HTML-based interface for the 
repository.  That can often be helpful for browsing the latest 
developments, patches that might have introduced a bug, etc.  Also, if I 
want to describe something to someone who may not be energized enough to 
clone the repository I'll give a link to the HTML file.

Here's an example of mercurial's interface:

http://hg.savannah.gnu.org/hgweb/octave

which I believe is part of the mercurial software, i.e., just set up an 
address for the repository and it will automatically generate the page 
in a browser.  The one category in the menu list I'd like to point out 
is the "graph", something that git doesn't appear to have (or I'm not 
aware of).  Here's an example of a git repository HTMl interface:

http://sourceware.org/git/?p=glibc.git;a=summary

The graph shows the branch history for the project, e.g., where they 
bifurcate, where they combine.  This is something that can be really 
useful on a bigger project.  (Does gnuplot need such a thing?  Not 
sure.)  It's not the best graph tool I've seen, as it is 
line/page-based, but one gets used to it.

The best graph and diff utilities I've seen are for a commercial product 
called Perforce.  It's graph is graphical based where one can zoom in on 
the graph, scroll a position bar, click on a node and it takes a person 
to that commit.  Perforce's diff utility will display the before and 
after file side-by-side with color coded adds and deletes.  It's really 
nice.  git/mercurial seem a tie in this category tipping toward 
mercurial for its graph printout, which could be better.

Dan

Re: SourceForge reorganization may affect gnuplot development site

[Allin C. <cot...@wf...>]

On Thu, 4 Oct 2012, Daniel J Sebald wrote:

> On 10/04/2012 04:35 AM, Mojca Miklavec wrote:
>> On Thu, Oct 4, 2012 at 10:29 AM, Daniel J Sebald wrote:
>>>
>>> I'm not sure the repositories themselves are being replaced, but the
>>> other items are changing (e.g., bug tracker, etc.).  However, CVS seems
>>> to be downplayed a bit and only SVN, mercurial and git are listed as
>>> supported.  So, if when upgrading, gnuplot is forced to choose one of
>>> those three, then it is essentially the same as replacing.  Not that
>>> "upgrading" from CVS is bad.  (I've used mercurial pretty much and that
>>> seems nice.)
>>
>> Probably not at this point, but I suspect that sooner or later CVS
>> will be phased out and it might happen that it won't be supported in
>> future at all. [...]
>
> I would say that either git or mercurial would be a better choice than
> SVN.

No doubt opinions will differ on this. Svn is closest to CVS, and 
therefore easiest to learn for people who are used to CVS, while 
offering various advantages over the latter. Git is very 
full-featured and is the VC system used by many leading open-source 
projects (notably the Linux kernel and the GNU C library) and 
therefore may be worth learning even though it's not close to CVS.

For me personally, mercurial would be the last choice, simply 
because up to this point no software that I'm interested in resides 
in a mercurial repository.

Allin Cottrell

Re: SourceForge reorganization may affect gnuplot development site

[Daniel J S. <dan...@ie...>]

On 10/04/2012 04:35 AM, Mojca Miklavec wrote:
> On Thu, Oct 4, 2012 at 10:29 AM, Daniel J Sebald wrote:
>>
>> I'm not sure the repositories themselves are being replaced, but the
>> other items are changing (e.g., bug tracker, etc.).  However, CVS seems
>> to be downplayed a bit and only SVN, mercurial and git are listed as
>> supported.  So, if when upgrading, gnuplot is forced to choose one of
>> those three, then it is essentially the same as replacing.  Not that
>> "upgrading" from CVS is bad.  (I've used mercurial pretty much and that
>> seems nice.)
>
> Probably not at this point, but I suspect that sooner or later CVS
> will be phased out and it might happen that it won't be supported in
> future at all. They wanted to completely drop support for CVS already,
> but maybe too many projects complained, so they left it. CVS does
> forget parts of history (when folder is deleted, file
> properties/executable bits etc. are lost) and I find it very difficult
> to navigate through changes.
>
> I have found a nice workaround (which works for me) - converting CVS
> to git and then working with git. And I don't seem to be the only one
> doing that. Once I got used to cherry-picking commits (that works
> amazing) and other sugars, it is difficult to think of going back.
> That feature would be extremely useful in particular for "backporting"
> bug fixes from trunk to branch-X-Y-stable.
>
> In my opinion switching to a different version control system will be
> necessary at one point, so it might be better to do it sooner rather
> than later. (I love git, but any other version control you mention
> above is better than CVS, and the decision/transition has to be made
> by those who are contributing most. It would only make me happy as the
> user if the upstream would use a different VCS.)

I would say that either git or mercurial would be a better choice than 
SVN.  git and mercurial are both "distributed" meaning that once a 
person has a local copy of the repository, the whole commit history is 
present (at least as far back as the project developers have chosen). 
These two make working with changesets amongst programmers much more fluent.

Right now, these advantages probably wouldn't be utilized because things 
are fairly stable.  But there has been a lot of discussion in the past 
regarding bigger projects like reorganizing the core code to make 
managing multiple plots a reality, as well as a number of other things.

I haven't used git much.  It seems nice, but I felt the "dock" or 
"platform" concept was extra work.  I think there are ways to bypass 
that, but mercurial seems fairly efficient.  The things I like about 
mercurial (and git probably has this too) is a built-in diff feature. 
It is so nice to just simply type:

hg diff

anywhere within the project's directory hierarchy and get a really quick 
record of the differences between the repository and the developing code.

hg update --clean

will remove any local changes and revert to the latest tip in the 
repository.

hg commit

will commit changes into the repository.  Then

hg export tip > somefile.patch

will create a diff file that includes the commit message.  (That might 
obviate the need for a ChangeLog file.  Developers would have to come up 
with a standard for creating commit messages.)  The "tip" means just the 
latest commit, but I believe multiple commits can be included in the 
export so several commits can be sent to others.

hg rollback

Will undo the most recent commit.  Just one.  git might do things better 
in the rollback area.

If there were something that could convert the CVS record to a git or 
mercurial record, that would be great.  If the change-log can't be 
incorporated that way, maybe one of the developers could write a C 
similar program that will properly incorporate the change-log 
entries...provided the repository is in some form of ASCII.

Dan

Release announcement - gnuplot version 4.6.1

[Ethan A M. <sf...@us...>]

Release 4.6.1 is now available for download from SourceForge.
As usual the initial release is a source tarball.
Pre-built binaries for Windows and other platforms will come 
later when/if they are prepared by their respective volunteers.

The full release announcement is appended below.

	happy gnuplotting,

		Ethan (gnuplot development team)





%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


		   GNUPLOT VERSION 4.6.1
	    ===================================

This is an incremental release of gnuplot version 4.6 containing various bug
fixes and a couple of new features.

A synopsis of changes since the previous patchlevel (version 4.6.0)
is given below and in the NEWS file.  Detailed information is in ChangeLog.

New features, changes and fixes since gnuplot version 4.6.0
===========================================================

* NEW syntax hints inside Emacs gnuplot-mode
* NEW support tabulation (set table) of pixel values from image plot styles
* NEW support tabulation of variable color column
* CHANGE emf output modified for better compatibility with MS Office programs
* CHANGE canvas terminal loads appropriate font file for UTF-8 encoding
* CHANGE skip execution of empty iteration loops in set and do commands
* CHANGE build scripts modified to accommodate automake 1.12
* CHANGE new policy: objects given in screen coords are not clipped to graph
* CHANGE Draw the z-axis label at a fixed distance to the left of the z-axis
* CHANGE "unset object N" succeeds even if there is currently no object N
* FIX margin space required for rotated axis tic labels
* FIX check for NaN values in binary input
* FIX backslash handling in enhanced text strings
* FIX cairo terminals sometimes lost the line segment before a polygon
* FIX interactive toggle of multiplots in svg
* FIX failure to balance {} if an input file did not end with a newline
* FIX strlen() and substring operators correctly handle UTF-8
* FIX initialization of history when configured --with-readline=bsd
* FIX set term cairolatex pdf mono
* FIX palette-related corruption in some cairolatex output
* FIX preserve number of active call arguments across a nested call command
* FIX wxt terminal mutex protecting execution of the command list
* FIX apply clipping to the interior fill of circles and ellipses
* FIX corruption of weights used for plotting with smooth acsplines
* FIX skip columnheader line when applying "every" filter
* FIX handle out-of-range pm3d values when cb axis is set to log scale
* FIX top/bottom color distinction in hidden3d when not using palette/RGB colors
* FIX allow toggling on/off of more than 10 plots in windows terminal
* FIX color printing from windows terminal
* FIX set term win font ",<size>"
* FIX incorrect return for acos(x) when imag(x) > 0 (bug present since v3.7)
      incorrect return for asin(x) when imag(x) > 0 (bug in 4.4.4, 4.6.0)
      incorrect asinh(x) when real(x) < 0 && imag(x) == 0 (bug in 4.4.4, 4.6.0)
* FIX keep sufficient precision in canvas and svg coords to report time in msec
* FIX the input buffer was not always extended correctly inside a { clause }
* FIX some cairolatex set_color requests were being ignored
* FIX calculated value of kernel density mean and sigma
* FIX emf terminal dashed line support


		 NOTES TO PACKAGERS AND TESTERS 
		===============================

Configuration options for interactive use
-----------------------------------------

The 4.6 source code supports three primary cross-platform output modes
in addition to several platform-specific modes.

1) Cairo/pango/wxWidgets
   These terminals were introduced in version 4.4 and are now the most
   stable and full-featured option.  This set of terminals includes
   - pngcairo, pdfcairo, epscairo, and cairolatex for output to a file
   - wxt for interactive display
   This is the default configuration, but requires prior installation of
   libcairo, libpango, libcairo, libwxgtk, and related support libraries
   To disable these terminals:
       ./configure --disable-wxt --without-cairo

2) Qt
   The new qt terminal supports interactive display with menu-driven
   output to png, svg or pdf.
   Requires libqt version >= 4.5
       ./configure --enable-qt

3) X11 (the "classic" interactive interface)
   This used to be the preferred interactive interface, but the newer
   wxt and qt terminals offer nicer output and a wider range of features.


Options for output to files
---------------------------

Of course the terminals (output modes) present in previous gnuplot versions
are also still available. These include, among many more obscure options:
- png/jpeg/gif output via libgd
- PostScript
- Many flavors of TeX/LaTeX output, including TikZ and ConTeXt (new)
- Bitmapped output to support many older devices (e.g. HP deskjet, epson,
  seiko printers, pbm bitmapped graphics files) is available if needed
  but is no longer configured in by default.  Note that the bitmap code
  copyright is more restrictive than the rest of the gnuplot code.
      ./configure --with-bitmap-terminals

Options for generating interactive plots for web display
--------------------------------------------------------

- Mouseable output for display on the web can be created using either 
  the canvas terminal (HTML5 2D canvas element) or the svg terminal.
  Both allow zooming, toggling plot elements on/off, and user-scriptable
  hot keys.

Online demo plots
-----------------

Demo plots illustrating new and old features are online at

	http://gnuplot.sourceforge.net/demo/


		          OTHER NOTES
		===============================

Installation
------------

You can download a source tarball for gnuplot version 4.6.1 from the 
gnuplot development site on SourceForge.

	http://sourceforge.net/project/showfiles.php?group_id=2055

Installation instructions are available in the source itself; the short
version for linux/unix-like systems is to unpack the tarball and then
build it:
		cd gnuplot-4.6.1 ; ./configure ; make
test it:
		make check
install it:
		make install

Pay careful attention to the output of the ./configure script.
It may indicate that some output drivers have been omitted because the
necessary support libraries were not found.  In general you need to have
previously installed the "*-devel-*" versions of these libraries. 


Known issues
------------

- Mac OSX ships with a terminal input library that appears to be GNU
libreadline, but isn't really.  The program tries to cope with this, but
you may get better results by configuring gnuplot to use either its own
built-in readline routines or the real GNU libreadline.

- The gnuplot build system is not very good at figuring out where to find
or install LaTeX-related files.  This can affect use of the new lua/tikz
and ConTeXt terminals.

- You can configure support for both wxt and qt into the same gnuplot
executable, but only one of these two output modes can be used in any
given gnuplot session.


Support
-------
Please report all bugs and installation problems to the bug tracker
on SourceForge:

	http://sourceforge.net/tracker/?group_id=2055&atid=102055

There is also an gnuplot discussion forum on usenet group

	comp.graphics.apps.gnuplot


Development
-----------
Gnuplot development is quite active. The development branch on SourceForge
contains preliminary implementations of many new features.  The current
development branch is labeled version 4.7, but will be released either as
4.8 or 5.0.  Feedback and contributions of code are very welcome.

Re: [PATCH] New pm3d corners2color function

[sfeam (E. Merritt) <eam...@gm...>]

On Thursday, 04 October 2012, Clement Law wrote:
> Hello,
> 
> I've written a small patch[a] that adds a harmonic mean function[1] to
> pm3d corners2color. I've pasted into this private pastebin because the
> mailing list seems to chomp up my attachment.

Got it, thanks.
It looks straightforward and can go into CVS after testing.

Just a note for the future:  
That pastebin interface is atrocious.
All I can manage to extract from it is a 20K+ line dump of every 
difference in your build tree, regardless of whether it's relevant
to the harmonic mean patch or just CVS tracking cruft.  Because it
was obvious that the work as described should only touch
pm3d/set/save/show, I was able to poke around with an editor to
extract what I needed but it was painful.
    diff -ur cvs/gnuplot/src/ mytree/gnuplot/src
would produce a far more usable patch.

> The correct usages of this harmonic mean is syntactically identical
> to, say, if you wanted to use geometrical mean (geomean):
> 
> set pm3d corners2color harmean
> 
> The reason why I added another 'mean' function is to complete the set.
> Gnuplot at the moment as two of the three 'pythagorean means'[2]. The
> harmonic mean is useful as it can offer a truer sense of the average.
> See this physics example [3].
> 
> One question I have is that the harmonic mean is not defined for
> numbers less than or equal to 0. Should I have harmean4() return 0 in
> these cases?
> 
> I hope it's acceptable, I've never written a patch for any open source
> software before.
> 
> Clement Law
> 
> [a] http://rifers.org/paste/harmean/show/1883
> [1] http://en.wikipedia.org/wiki/Harmonic_mean
> [2] http://en.wikipedia.org/wiki/Pythagorean_means
> [3] http://en.wikipedia.org/wiki/Harmonic_mean#Physics
> 
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
> 

[PATCH] New pm3d corners2color function

[Clement L. <the...@gm...>]

Hello,

I've written a small patch[a] that adds a harmonic mean function[1] to
pm3d corners2color. I've pasted into this private pastebin because the
mailing list seems to chomp up my attachment.

The correct usages of this harmonic mean is syntactically identical
to, say, if you wanted to use geometrical mean (geomean):

set pm3d corners2color harmean

The reason why I added another 'mean' function is to complete the set.
Gnuplot at the moment as two of the three 'pythagorean means'[2]. The
harmonic mean is useful as it can offer a truer sense of the average.
See this physics example [3].

One question I have is that the harmonic mean is not defined for
numbers less than or equal to 0. Should I have harmean4() return 0 in
these cases?

I hope it's acceptable, I've never written a patch for any open source
software before.

Clement Law

[a] http://rifers.org/paste/harmean/show/1883
[1] http://en.wikipedia.org/wiki/Harmonic_mean
[2] http://en.wikipedia.org/wiki/Pythagorean_means
[3] http://en.wikipedia.org/wiki/Harmonic_mean#Physics

Re: SourceForge reorganization may affect gnuplot development site

[Mojca M. <moj...@gm...>]

On Thu, Oct 4, 2012 at 10:29 AM, Daniel J Sebald wrote:
>
> I'm not sure the repositories themselves are being replaced, but the
> other items are changing (e.g., bug tracker, etc.).  However, CVS seems
> to be downplayed a bit and only SVN, mercurial and git are listed as
> supported.  So, if when upgrading, gnuplot is forced to choose one of
> those three, then it is essentially the same as replacing.  Not that
> "upgrading" from CVS is bad.  (I've used mercurial pretty much and that
> seems nice.)

Probably not at this point, but I suspect that sooner or later CVS
will be phased out and it might happen that it won't be supported in
future at all. They wanted to completely drop support for CVS already,
but maybe too many projects complained, so they left it. CVS does
forget parts of history (when folder is deleted, file
properties/executable bits etc. are lost) and I find it very difficult
to navigate through changes.

I have found a nice workaround (which works for me) - converting CVS
to git and then working with git. And I don't seem to be the only one
doing that. Once I got used to cherry-picking commits (that works
amazing) and other sugars, it is difficult to think of going back.
That feature would be extremely useful in particular for "backporting"
bug fixes from trunk to branch-X-Y-stable.

In my opinion switching to a different version control system will be
necessary at one point, so it might be better to do it sooner rather
than later. (I love git, but any other version control you mention
above is better than CVS, and the decision/transition has to be made
by those who are contributing most. It would only make me happy as the
user if the upstream would use a different VCS.)

Mojca

Re: SourceForge reorganization may affect gnuplot development site

[Daniel J S. <dan...@ie...>]

On 10/03/2012 05:03 PM, Ethan A Merritt wrote:
> SourceForge is "upgrading" (in other words replacing) its current project
> source code repositories, mailing lists, bug trackers, etc.
>
> It is utterly unclear to me how much this will affect maintenance
> or use of the gnuplot project pages, but I guess we will find out.
>
> More information:
>      https://sourceforge.net/p/upgrade/
>
> If nothing else, apparently the URL to access the project pages
> will change.

I'm not sure the repositories themselves are being replaced, but the 
other items are changing (e.g., bug tracker, etc.).  However, CVS seems 
to be downplayed a bit and only SVN, mercurial and git are listed as 
supported.  So, if when upgrading, gnuplot is forced to choose one of 
those three, then it is essentially the same as replacing.  Not that 
"upgrading" from CVS is bad.  (I've used mercurial pretty much and that 
seems nice.)

The example project (Allura itself) doesn't seem too much different from 
the existing SourceForge, if perhaps less feature rich.  The color coded 
diffs viewing isn't much to speak of.  One has to use the horizontal 
slide bar quite a bit and then the colored line backgrounds only extend 
as far as the original window, so that isn't much use if the different 
parts are out to the right side of the window.

gnuplot isn't likely to use the discussion forum, I assume.

What this paste bin is, I have no idea.  It looks like hunks of code 
pasted there without username association or description.

The one sort of new thing (or perhaps it is simply something I hadn't 
noticed with the existing SourceForge) is every 5th or 6th page 
navigation a Flash advert comes up.

I'd say wait a while and see if Allure is improved and what the future 
is for CVS support.

Dan

Re: SourceForge reorganization may affect gnuplot development site

[Ethan A M. <sf...@us...>]

On Wednesday, October 03, 2012 03:45:15 pm Allin Cottrell wrote:
> On Wed, 3 Oct 2012, Ethan A Merritt wrote:
> 
> > SourceForge is "upgrading" (in other words replacing) its current project
> > source code repositories, mailing lists, bug trackers, etc.
> >
> > It is utterly unclear to me how much this will affect maintenance
> > or use of the gnuplot project pages, but I guess we will find out.
> 
> Hmm, if I go to that "upgrade" page I see text that appears to make 
> the upgrade optional, per project. Is that misleading? Are we really 
> bound to upgrade regardless?

From their Email to me:

 Early next year, we hope to be 100% migrated off of the classic platform. 
 [...]  We'd like for you to have the opportunity to upgrade on your own 
 schedule, rather than ours. You can read about the upgrade process at 
 https://sourceforge.net/p/upgrade/ and then press the 'Upgrade' button
 next to your project name.

To me that sounds like 'any time you like so long as it is in the next
3 months'.

	Ethan

Re: SourceForge reorganization may affect gnuplot development site

[Allin C. <cot...@wf...>]

On Wed, 3 Oct 2012, Ethan A Merritt wrote:

> SourceForge is "upgrading" (in other words replacing) its current project
> source code repositories, mailing lists, bug trackers, etc.
>
> It is utterly unclear to me how much this will affect maintenance
> or use of the gnuplot project pages, but I guess we will find out.
>
> More information:
>    https://sourceforge.net/p/upgrade/
>
> If nothing else, apparently the URL to access the project pages
> will change.

Hmm, if I go to that "upgrade" page I see text that appears to make 
the upgrade optional, per project. Is that misleading? Are we really 
bound to upgrade regardless?

Allin Cottrell

SourceForge reorganization may affect gnuplot development site

[Ethan A M. <sf...@us...>]

SourceForge is "upgrading" (in other words replacing) its current project 
source code repositories, mailing lists, bug trackers, etc.

It is utterly unclear to me how much this will affect maintenance
or use of the gnuplot project pages, but I guess we will find out.

More information:
    https://sourceforge.net/p/upgrade/

If nothing else, apparently the URL to access the project pages
will change.

       Ethan

Re: [PATCH] x11 terminal respects the requested aspect ratio

[Ethan A M. <sf...@us...>]

On Wednesday, October 03, 2012 01:39:08 am Dima Kogan wrote:
> > On Sat, 29 Sep 2012 17:07:51 -0700
> > Ethan Merritt <merritt@u.washington.edu> wrote:
> >
> > > 6. Removing duplicate messages (such as duplicate consecutive V commands) sounds
> > >    great. We should do it
> >   OK. Low priority because it's relatively rare for normal plots.
> 
> I did a first pass at this. Patch attached. Good news is that as expected, the
> traffic drops dramatically. Inboard timing drops from about 0.9s to about 0.45s.
> The outboard, however, drops from 0.85s to 0.01s! The reason the inboard didn't
> drop as much is that it still has to parse the original huge data file. I have
> some lingering concerns about the patch I'm attaching. 

> I use ftell() to check to see that the V commands are indeed consecutive.
> This might have a non-negligible cost, so I'd check before committing this.

That's clever, but I agree that there is potential cost.
Other terminal drivers do the same job without resorting to ftell().
The trick is that any command that potentially affects the current
active position must either update or invalidate the inboard copy
of x_last and y_last.   For example,  term->put_text() would set
  x_last = y_last = INVALID;           /* #define INVALID -1 */
before leaving.

The down side is that unlike your ftell() version this approach requires
finding all the places that might affect current position.  I've attached a 
first-pass patch that catches most of them, but I probably missed some.
Other terminal drivers can serve as a model.

 
> At this point, my test case is clearly broken since we've been able to optimize
> away all its complexity.

Right.  I modified your original data generation script to produce longer
vectors and some gaps, so the the plot would contain move commands as well
as vector commands.  This gives a more realistic mix of commands:

perl -e 'for(0..2000000) \
 { print "$_ " . sin($_/1000) . "\n"; print "\n" if $_ % 100 == 0; }' \
 > breaks.ascii

With this test data the reduction in size from removing redundant commands
is less than 10%.  I tested using the "uniq" command rather than patching 
the driver source code.  10% max didn't seem very significant to me,
which was why I said it was low priority.

> Is the same optimization valid for P commands? 

I don't think so.  But it does apply to M commands.

> I'm thinking of just generating a bunch of discrete points, and sending 
> them over as P commands. That sounds good, right?

I don't think that the active position after drawing a point symbol is 
guaranteed to be at the center of the point.  So in the sequence
  Move(x,y);  Point(x,y);  Move(x,y); Vector(x1,y1);
the second Move is not redundant.

Of course, we could change the code so that Point(x,y) it _is_ guaranteed
to leave the active position at (x,y).  

> > > 7. General thoughts about speeding up the inboard -> outboard link by making
> > >    some things binary. I like binary. Making this switch will probably make some
> > >    things break at first, but it'll likely be worth it. If we're touching that
> > >    at all, more radical methods may be better. For instance, instead of a V
> > >    command for each point, we could have a V command that predeclares a long
> > >    stream of points. 
> >   We do. That's what the X11_POLYLINE code is for.
> 
> That's not quite what I meant. The polyline code we have chunks up data from the
> V command we receive before sending it to X. I'm talking about doing a similar
> thing, but in the inboard->outboard link: chunk up the V commands before sending
> it across. I'll write and benchmark a patch in a bit. 

I doubt you'll see any gain. Since there is not a flush() command between 
successive writes to the output stream, the system should do this chunking for you
anyhow.

> Are P and V commands the main ones to worry about?

M and V.

	Ethan

Re: [PATCH] x11 terminal respects the requested aspect ratio

[Dima K. <gn...@di...>]

> On Sat, 29 Sep 2012 17:07:51 -0700
> Ethan Merritt <merritt@u.washington.edu> wrote:
>
> > 5. Adding default window size to the inboard x11 driver so that the 'terminal
> >    xlib' produces plots that have decent defaults when sent to the outboard
> >    driver manually. I'll do this.
>   OK.  It's not just xlib, however.  I think (not 100% sure) that the same 
>   problem arises whenever (ipc_back_fd == IPC_BACK_UNUSABLE), e.g. x11 output
>   from a script run non-interactively.

Still on my list; haven't gotten to it yet.


> > 6. Removing duplicate messages (such as duplicate consecutive V commands) sounds
> >    great. We should do it
>   OK. Low priority because it's relatively rare for normal plots.

I did a first pass at this. Patch attached. Good news is that as expected, the
traffic drops dramatically. Inboard timing drops from about 0.9s to about 0.45s.
The outboard, however, drops from 0.85s to 0.01s! The reason the inboard didn't
drop as much is that it still has to parse the original huge data file. I have
some lingering concerns about the patch I'm attaching. I use ftell() to check to
see that the V commands are indeed consecutive. This might have a non-negligible
cost, so I'd check before committing this.

At this point, my test case is clearly broken since we've been able to optimize
away all its complexity. Is the same optimization valid for P commands? What
would be a good real-world test case where this optimization isn't valid? I'm
thinking of just generating a bunch of discrete points, and sending them over as
P commands. That sounds good, right?


> > 7. General thoughts about speeding up the inboard -> outboard link by making
> >    some things binary. I like binary. Making this switch will probably make some
> >    things break at first, but it'll likely be worth it. If we're touching that
> >    at all, more radical methods may be better. For instance, instead of a V
> >    command for each point, we could have a V command that predeclares a long
> >    stream of points. 
>   We do. That's what the X11_POLYLINE code is for.

That's not quite what I meant. The polyline code we have chunks up data from the
V command we receive before sending it to X. I'm talking about doing a similar
thing, but in the inboard->outboard link: chunk up the V commands before sending
it across. I'll write and benchmark a patch in a bit. Are P and V commands the
main ones to worry about?

dima

autoconf/automake complaints

[Mojca M. <moj...@gm...>]

Hello,

I would first like to ask why the file "Makefile.in" is present in
"docs", but nowhere else. I'm forced to run the ./prepare script
because the majority of files (./configure, Makefile.in in other
directories) are missing, but on the other hand I'm using a newer
version of automake and once I run ./prepare I'm unable to switch
branches until I remove/revert the file docs/Makefile.in after the
./prepare script modifies the file. Am I missing something or would it
be possible to remove that file from repository?

Also, since very very recently (just a couple of days) I'm getting the
following complaints from the ./prepare script:

aclocal: warning: autoconf input should be named 'configure.ac', not
'configure.in'
make: `Makefile.am' is up to date.
make: `Makefile.am' is up to date.
make: `Makefile.am' is up to date.
make: `Makefile.am' is up to date.
make: `Makefile.am' is up to date.
aclocal: warning: autoconf input should be named 'configure.ac', not
'configure.in'
automake: warning: autoconf input should be named 'configure.ac', not
'configure.in'
configure.in:10: warning: AM_INIT_AUTOMAKE: two- and three-arguments
forms are deprecated.  For more info, see:
configure.in:10:
http://www.gnu.org/software/automake/manual/automake.html#Modernize-AM_INIT_AUTOMAKE-invocation
automake: warning: autoconf input should be named 'configure.ac', not
'configure.in'
automake: warning: autoconf input should be named 'configure.ac', not
'configure.in'

The gnuplot source code was successfully prepared.
Run configure now, then make && make install to build and install gnuplot.

I believe it's connected with a local update of autotools. I have the
following versions installed:

    automake: 1.12.4
    autoconf: 2.69

but I wouldn't know when I last updated any of them (they get updated
automatically every now and then when I run "port upgrade outdated").
On the other computer I still have automake 1.12.3 and that one
doesn't seem to complain.

May I please kindly request to test the scripts with the latest
automake/autoconf and possibly adapt them to the latest syntax?

Thank you very much,
    Mojca


Citations from the above mentioned website:

Although using some of the following macros was required in past
releases, you should not use any of them in new code. All these macros
will be removed in the next major Automake version; if you are still
using them, running autoupdate should adjust your configure.ac
automatically (see Using autoupdate to Modernize configure.ac). Do it
NOW!

-----------------------

AM_INIT_AUTOMAKE([OPTIONS])
Runs many macros required for proper operation of the generated Makefiles.

Today, AM_INIT_AUTOMAKE is called with a single argument: a
space-separated list of Automake options that should be applied to
every Makefile.am in the tree. The effect is as if each option were
listed in AUTOMAKE_OPTIONS (see Options).

This macro can also be called in another, deprecated form (support for
which will be removed in the next major Automake release (1.13)):
AM_INIT_AUTOMAKE(PACKAGE, VERSION, [NO-DEFINE]). In this form, there
are two required arguments: the package and the version number. This
form is obsolete because the package and version can be obtained from
Autoconf's AC_INIT macro (which itself has an old and a new form).

If your configure.ac has:
          AC_INIT([src/foo.c])
          AM_INIT_AUTOMAKE([mumble], [1.5])

you should modernize it as follows:
          AC_INIT([mumble], [1.5])
          AC_CONFIG_SRCDIR([src/foo.c])
          AM_INIT_AUTOMAKE

Re: [PATCH] x11 terminal respects the requested aspect ratio

[Mojca M. <moj...@gm...>]

On Sun, Sep 30, 2012 at 2:07 AM, Ethan Merritt wrote:
>
>> 6. Removing duplicate messages (such as duplicate consecutive V commands) sounds
>>    great. We should do it
>   OK. Low priority because it's relatively rare for normal plots.

Not necessary, in particular when plotting "using 0:1" with a huge
number of points (or almost any number bigger than screen resolution).
Actually, it is what I do all the time.

A dedicated program for drawing long signals that I'm using goes even
a step further and does the following optimisation:
- keep collecting all the points that would use the same pixel in x-direction
- out of those points remember the minimum, maximum, first and last value of y.

This only works for plots (1) with more or less sequential values of x
(most plots?) and it only makes any difference (2) when the number of
points is significantly bigger than width of the plot. I agree with
Ethan that most plots don't meet the second criteria, but for plots
that do, it makes an enormous difference (unless the bottleneck is
reading the data from input).

When drawing one million points on the screen with 1000 pixels this
results in 2-4 points per 1000 values. That's perfectly consistent
with Ethan's observation of 50-fold speed-up (and again, it is
something that I do on a daily basis).

I'll try to figure out how exactly you do the benchmarking of Dima's
patches. I'm very curious about the effect on Mac OS X.

(I really hated x11 terminal on Mac OS X, but now that I'm using those
long traces and want to zoom in, AquaTerm still lacks mouse support
and Qt has some serious efficiency problems, x11 is currently the only
terminal left that does the job quickly and a lot more efficient than
the rest.)

Mojca

Re: [PATCH] x11 terminal respects the requested aspect ratio

[Ethan M. <merritt@u.washington.edu>]

On Saturday, 29 September 2012, Dima Kogan wrote:
> Hi!
> 
> > On Sat, 29 Sep 2012 11:44:09 -0700
> > "sfeam (Ethan Merritt)" <eam...@gm...> wrote:
> >
> > On Saturday, 29 September 2012, Dima Kogan wrote:
> > > > On Fri, 28 Sep 2012 10:37:06 -0700

> > 1) The output of the ascii version is hugely redundant. Most of the
> > successive V commands are identical because the vectors are shorter
> > than the resolution of the plot coordinates.  If this were a common
> > thing we would do well to add a filtering step in x11.trm so that
> > a new "V" command is only sent if it differs from the previous command.
> > For example running the ascii output through "uniq" reduces the file
> > size by a factor of 7, with a speed increase of 50x(!!) running 
> > through gnuplot_x11.
> 
> Sounds great. Should I do this, or do you want to?

I will look into it.

> > 3) I worry that the binary code might not work on all supported
> > platforms.  Since it's just a few lines of code, I suggest that rather
> > than replacing the existing 'V' command we add a parallel command
> > 'B' for the bainry version.  In x11.trm the choice between using 'V'
> > or 'B' could be a compile-time option.   We can default to the binary
> > version, but anyone having problems with it could revert to the old
> > ascii version with a configuration flag.
> 
> My preference would be to switch entirely so that the amount of code being
> maintained doesn't grow, but you're the boss. :)

I am inclined to let both sit in CVS while we test.  If no problems turn
up with the binary version we can remove the old ascii code before release.

> Keeping track of all the patches, proposals, the following are currently being
> considered:
> 
> 1. Patch for variable-width, space-separated fields to allow 5-digit values 
  Merged yesterday.

> 2. Missing legend label with tall windows. Caused by the 'if (x < 10000 && y <
>    10000)' test in x11.trm. Removing this test entirely seems to work for the
>    most part. ... should be removed entirely.
  Merged yesterday.

> 3. Dead code such as the 'else if (*buffer == X11_GR_FILLED_POLYGON)' block.
  Merged yesterday.

> 4. strtolstrtol() instead of scanf() sounds like an easy win. Can you think of
>    any reason to NOT make that change?
  Only that you are in the process of replacing it altogether with a 
  binary version :-)
  But if it's a win for 'V' it's almost certainly a win everywhere else as well.

> 5. Adding default window size to the inboard x11 driver so that the 'terminal
>    xlib' produces plots that have decent defaults when sent to the outboard
>    driver manually. I'll do this.
  OK.  It's not just xlib, however.  I think (not 100% sure) that the same 
  problem arises whenever (ipc_back_fd == IPC_BACK_UNUSABLE), e.g. x11 output
  from a script run non-interactively.

> 6. Removing duplicate messages (such as duplicate consecutive V commands) sounds
>    great. We should do it
  OK. Low priority because it's relatively rare for normal plots.

> 7. General thoughts about speeding up the inboard -> outboard link by making
>    some things binary. I like binary. Making this switch will probably make some
>    things break at first, but it'll likely be worth it. If we're touching that
>    at all, more radical methods may be better. For instance, instead of a V
>    command for each point, we could have a V command that predeclares a long
>    stream of points. 
  We do. That's what the X11_POLYLINE code is for.

	Ethan

Re: [PATCH] x11 terminal respects the requested aspect ratio

[Dima K. <gn...@di...>]

Hi!

> On Sat, 29 Sep 2012 11:44:09 -0700
> "sfeam (Ethan Merritt)" <eam...@gm...> wrote:
>
> On Saturday, 29 September 2012, Dima Kogan wrote:
> > > On Fri, 28 Sep 2012 10:37:06 -0700
> 
> Hi Dima,
> 
> Just some preliminary thoughts...
> I quickly replicated your benchmarks and see roughly the same results
> (faster CPU but limited memory; 32bit environment).
> 
> However, I have a few general concerns.
> 
> 1) The output of the ascii version is hugely redundant. Most of the
> successive V commands are identical because the vectors are shorter
> than the resolution of the plot coordinates.  If this were a common
> thing we would do well to add a filtering step in x11.trm so that
> a new "V" command is only sent if it differs from the previous command.
> For example running the ascii output through "uniq" reduces the file
> size by a factor of 7, with a speed increase of 50x(!!) running 
> through gnuplot_x11.

Sounds great. Should I do this, or do you want to?


> 2) The binary output is not terminating the "V" records with a '\n'.
> This is fine on linux, but there is a long history of problem reports
> on Windows arising from very long buffers sent through a pipe.  I don't
> know the details well enough to say whether this would trigger similar
> problems, but I do worry.  I think it's worth also trying a variant
> that writes a trailing '\n'. If writing a newline for every V command 
> causes a significant slowdown then perhaps we could generalize the
> existing code in X11_filled_polygon() that breaks long binary buffers
> into smaller chunks.

The binary-V implementation I had wouldn't touch the trailing '\n', since a
particular binary field size is assumed. Furthermore, the binary numbers could
have '\n' bytes in them, making a trailing '\n' even less meaningful. Out of
curiosity, I changed to code to output a '\n' after each binary V record. There
was a slight, but noticeable speed penalty. If we go down the binary-data path,
I'd prefer to not to add the '\n' simply because if anything ever looks at that
byte, then something had already gone wrong.


> 3) I worry that the binary code might not work on all supported
> platforms.  Since it's just a few lines of code, I suggest that rather
> than replacing the existing 'V' command we add a parallel command
> 'B' for the bainry version.  In x11.trm the choice between using 'V'
> or 'B' could be a compile-time option.   We can default to the binary
> version, but anyone having problems with it could revert to the old
> ascii version with a configuration flag.

My preference would be to switch entirely so that the amount of code being
maintained doesn't grow, but you're the boss. :)



> Oh, and I noticed something else about the new aspect-ratio code while
> running the benchmarks.  I use
>    ./gnuplot_x11 -noevents < foo.x11
> to benchmark the outboard driver.  Since there is no feedback in place,
> the inboard driver doesn't know about the aspect ratio and the plot
> is not scaled properly to the plot window.  Now this is not the usual
> path for display x11 output, but I wonder if we can fix that easily.
> Maybe disabling the rescaling code if the feedback pipe is not present?
> Or maybe sending an initial set of scaling commands that are always
> correct, which are later over-ridden as needed in the interactive case
> but remain in effect for the case of -noevents or no feedback pipe?

I'll take a look into this.

Keeping track of all the patches, proposals, the following are currently being
considered:

1. Patch for variable-width, space-separated fields to allow 5-digit values to
   'just work'; sent out to this list a few days ago. There was a concern that
   the extra spaces carry with them a performance cost. The benchmarks I sent
   out indicate that the performance cost is negligible at worst. I think this
   patch should be merged

2. Missing legend label with tall windows. Caused by the 'if (x < 10000 && y <
   10000)' test in x11.trm. Removing this test entirely seems to work for the
   most part. The only issue I've seen is that if the window is too tall, the
   label can overflow past the plot edge. However, the existing test is a very
   poor way to check for that condition anyway; and if an overflow is detected
   then both the legend label AND the legend symbol should be removed. So I
   argue the current test is not useful, and should be removed entirely.

3. Dead code such as the 'else if (*buffer == X11_GR_FILLED_POLYGON)' block. I'd
   like to push for these to be removed. These make it more difficult for new
   people to look at the codebase, and make maintenance more difficult than it
   should be.

4. strtolstrtol() instead of scanf() sounds like an easy win. Can you think of
   any reason to NOT make that change?

5. Adding default window size to the inboard x11 driver so that the 'terminal
   xlib' produces plots that have decent defaults when sent to the outboard
   driver manually. I'll do this.

6. Removing duplicate messages (such as duplicate consecutive V commands) sounds
   great. We should do it

7. General thoughts about speeding up the inboard -> outboard link by making
   some things binary. I like binary. Making this switch will probably make some
   things break at first, but it'll likely be worth it. If we're touching that
   at all, more radical methods may be better. For instance, instead of a V
   command for each point, we could have a V command that predeclares a long
   stream of points. So for instance you'd have "V100\n" to indicate that 100
   binary tuples follow. This would reduce the overhead even more, but would
   require read_input() to be changed to work with long streams of binary data.
   I prefer a method like this much more than what was in the last patch. If
   you'd consider merging something like this, tell me and I'll write it.

dima

Re: [PATCH] x11 terminal respects the requested aspect ratio

[sfeam (E. Merritt) <eam...@gm...>]

On Saturday, 29 September 2012, Dima Kogan wrote:
> Results. Each record is usertime,systemtime in seconds.
> 
> |                     | %04d      | %d        | %d, 16-bit binary V | %d, 32-bit binary V |
> |---------------------+-----------+-----------+---------------------+---------------------|
> | inboard from ASCII  | 1.91,0.21 | 1.86,0.21 | 1.53,0.19           | 1.53,0.21           |
> | inboard from binary | 0.95,0.20 | 0.89,0.20 | 0.56,0.16           | 0.56,0.20           |
> | outboard            | 0.82,0.11 | 0.85,0.11 | 0.19,0.10           | 0.21,0.11           |
> 
> First off, we see that splitting the fields with whitespace speeds up
> the inboard driver a little bit, while slowing down the outboard one a
> little bit. Not completely sure why, but the difference is negligible,
> so I didn't go digging.
> 
> However, sending the data over in binary produces HUGE performance
> gains. The inboard driver is 37% faster (when the original input is
> binary too), while the outboard one is a whopping 76% faster. Not quite
> sure why this is so uneven; maybe the outboard driver parses the data
> more times than it needs to?

I get a 50% speed improvement in the ascii version by replacing scanf() with strtol()
(see attached patch).  
That reduces the speed difference between ascii and binary from ~4x to ~2x.

There is a slight further improvement by replacing the second call to strtol() with atoi();

	Ethan

Re: [PATCH] x11 terminal respects the requested aspect ratio

[sfeam (E. Merritt) <eam...@gm...>]

On Saturday, 29 September 2012, Dima Kogan wrote:
> > On Fri, 28 Sep 2012 10:37:06 -0700

Hi Dima,

Just some preliminary thoughts...
I quickly replicated your benchmarks and see roughly the same results
(faster CPU but limited memory; 32bit environment).

However, I have a few general concerns.

1) The output of the ascii version is hugely redundant. Most of the
successive V commands are identical because the vectors are shorter
than the resolution of the plot coordinates.  If this were a common
thing we would do well to add a filtering step in x11.trm so that
a new "V" command is only sent if it differs from the previous command.
For example running the ascii output through "uniq" reduces the file
size by a factor of 7, with a speed increase of 50x(!!) running 
through gnuplot_x11.

2) The binary output is not terminating the "V" records with a '\n'.
This is fine on linux, but there is a long history of problem reports
on Windows arising from very long buffers sent through a pipe.  I don't
know the details well enough to say whether this would trigger similar
problems, but I do worry.  I think it's worth also trying a variant
that writes a trailing '\n'. If writing a newline for every V command 
causes a significant slowdown then perhaps we could generalize the
existing code in X11_filled_polygon() that breaks long binary buffers
into smaller chunks.

3) I worry that the binary code might not work on all supported
platforms.  Since it's just a few lines of code, I suggest that rather
than replacing the existing 'V' command we add a parallel command
'B' for the bainry version.  In x11.trm the choice between using 'V'
or 'B' could be a compile-time option.   We can default to the binary
version, but anyone having problems with it could revert to the old
ascii version with a configuration flag.

I'll play around with more serious benchmarking if I get time later
this weekend.

Oh, and I noticed something else about the new aspect-ratio code while
running the benchmarks.  I use
   ./gnuplot_x11 -noevents < foo.x11
to benchmark the outboard driver.  Since there is no feedback in place,
the inboard driver doesn't know about the aspect ratio and the plot
is not scaled properly to the plot window.  Now this is not the usual
path for display x11 output, but I wonder if we can fix that easily.
Maybe disabling the rescaling code if the feedback pipe is not present?
Or maybe sending an initial set of scaling commands that are always
correct, which are later over-ridden as needed in the interactive case
but remain in effect for the case of -noevents or no feedback pipe?

	Ethan



> I just ran some benchmarks myself. The results are quite interesting.
> 
> First off, the test machine description:
> 
> gcc (Debian 4.7.0-12) 4.7.0
> 
> CPU:
> vendor_id	: GenuineIntel
> cpu family	: 6
> model		: 15
> model name	: Intel(R) Core(TM)2 CPU         T7400  @ 2.16GHz
> stepping	: 6
> 
> This is a 2-core machine, but everything in gnuplot is single-threaded
> so this doesn't matter.
> 
> I generated 2 large-ish data files. Both contain an identical sinusoid:
> one stores it in ascii, another in binary with packed single-precision
> floats. This isn't what we're testing, but I wanted to get this
> conversion step into the data as a reference. Commands to generate data:
> 
> $ perl -e 'for(0..2000000) { print pack "f*", $_, sin($_/100000);        }' > dat.bin
> $ perl -e 'for(0..2000000) { print "$_ " .        sin($_/100000) . "\n"; }' > dat.ascii
> 
> 
> For each gnuplot build I tested, I timed 3 things:
> 
> 1. inboard x11 from binary (with 'terminal xlib')
> 2. inboard x11 from ascii  (with 'terminal xlib')
> 3. outboard x11            (just gnuplot_x11 executable)
> 
> I tested 4 different gnuplot builds:
> 
> 1. before the split-printf patch (uses %04d format)
> 2. after the split-printf patch (uses space-separated %d format)
> 3. after the split-printf-patch but ALSO sending the V command in
> binary. V commands were the bulk of the xlib-generated data stream, so
> this is our hotspot. 16-bit integers for each argument of V
> 4. Same as previous, but using 32-bit integers
> 
> ASCII input tests were run by making a 'tst.gp' file with
> 
> ================
> set term xlib
> set output "out.xlib"
> plot "dat.ascii" with lines
> ================
> 
> Then generating timings by running multiple times
> $ time ./gnuplot tst.gp
> $ time ./gnuplot_x11 < out.xlib > /dev/null
> 
> 
> Binary input tests were run similarly, but with a 'tst.gp' such as
> 
> ================
> set term xlib
> set output "out.xlib"
> plot "dat.bin" binary format="%float32%float32" with lines
> ================
> 
> 
> Results. Each record is usertime,systemtime in seconds.
> 
> |                     | %04d      | %d        | %d, 16-bit binary V | %d, 32-bit binary V |
> |---------------------+-----------+-----------+---------------------+---------------------|
> | inboard from ASCII  | 1.91,0.21 | 1.86,0.21 | 1.53,0.19           | 1.53,0.21           |
> | inboard from binary | 0.95,0.20 | 0.89,0.20 | 0.56,0.16           | 0.56,0.20           |
> | outboard            | 0.82,0.11 | 0.85,0.11 | 0.19,0.10           | 0.21,0.11           |
> 
> First off, we see that splitting the fields with whitespace speeds up
> the inboard driver a little bit, while slowing down the outboard one a
> little bit. Not completely sure why, but the difference is negligible,
> so I didn't go digging.
> 
> However, sending the data over in binary produces HUGE performance
> gains. The inboard driver is 37% faster (when the original input is
> binary too), while the outboard one is a whopping 76% faster. Not quite
> sure why this is so uneven; maybe the outboard driver parses the data
> more times than it needs to?
> 
> None of this is really surprising, but it really reinforces the earlier
> point that seeking performance gains in our ASCII representation is
> foolish, leading to minimal speedups while making the code less
> manageable. When I started this, I wasn't going to advocate that we
> move to a binary data stream, but the speedup is so significant that we
> really should, I think.
> 
> I'm attaching a patch that changes the V command to work in binary.
> Note that this patch is not at all good-enough to merge yet; I'm
> attaching it so that others could run these tests as well. So, should
> we move the intensive commands to binary? What commands are these,
> other than 'V'?
> 
> dima
> 

Re: [PATCH] x11 terminal respects the requested aspect ratio

[Dima K. <gn...@di...>]

> On Fri, 28 Sep 2012 10:37:06 -0700
> Dima Kogan <gn...@di...> wrote:
>
> > > Hi Ethan.
> > > 
> > > Good you found the issue. Do you know why field widths are
> > > specified at all? Why don't we PRINT2("V%d %d\n") and then
> > > sscanf("%d %d", ...) on the other side? No efficiency is gained
> > > by specifying the widths, it only builds in limitations and makes
> > > the code brittle, as we have just observed.
> > 
> > The basic X11 framework was in place long before my involvement, so
> > I do not know.  My best guess is that the fixed field width was
> > chosen so as to reduce the amount of data sent through the
> > gnuplot->gnuplot_x11 channel.
> > Adding a space in front of every coordinate pair would increase the
> > traffic by as much as 25%.
> > 
> > As I recall, when the polygon encoding was switched from formatted
> > to binary (2004), the advocates of binary encoding/decoding had
> > benchmarks showing data transfer really was a performance
> > bottleneck, and reducing the number of bytes sent over the channel
> > resulted in faster plotting. I do not know if this would still be
> > true on modern machines.
> > 
> > We should benchmark before and after your latest patch.
> 
> The extra space can also save bytes when the values being sent across
> have fewer than 4 digits in them. As I see it, one should use ASCII
> data links if they want robustness and readability, and binary ones
> if they want speed. Here we're sending data in ASCII, while worrying
> about a few extra cycles. If you make up a test that benchmarks
> before and after this patch, I'll write another version of the data
> passing, that uses binary data; if there're any performance gains
> here, that's where they are.


I just ran some benchmarks myself. The results are quite interesting.

First off, the test machine description:

gcc (Debian 4.7.0-12) 4.7.0

CPU:
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 CPU         T7400  @ 2.16GHz
stepping	: 6

This is a 2-core machine, but everything in gnuplot is single-threaded
so this doesn't matter.

I generated 2 large-ish data files. Both contain an identical sinusoid:
one stores it in ascii, another in binary with packed single-precision
floats. This isn't what we're testing, but I wanted to get this
conversion step into the data as a reference. Commands to generate data:

$ perl -e 'for(0..2000000) { print pack "f*", $_, sin($_/100000);        }' > dat.bin
$ perl -e 'for(0..2000000) { print "$_ " .        sin($_/100000) . "\n"; }' > dat.ascii


For each gnuplot build I tested, I timed 3 things:

1. inboard x11 from binary (with 'terminal xlib')
2. inboard x11 from ascii  (with 'terminal xlib')
3. outboard x11            (just gnuplot_x11 executable)

I tested 4 different gnuplot builds:

1. before the split-printf patch (uses %04d format)
2. after the split-printf patch (uses space-separated %d format)
3. after the split-printf-patch but ALSO sending the V command in
binary. V commands were the bulk of the xlib-generated data stream, so
this is our hotspot. 16-bit integers for each argument of V
4. Same as previous, but using 32-bit integers

ASCII input tests were run by making a 'tst.gp' file with

================
set term xlib
set output "out.xlib"
plot "dat.ascii" with lines
================

Then generating timings by running multiple times
$ time ./gnuplot tst.gp
$ time ./gnuplot_x11 < out.xlib > /dev/null


Binary input tests were run similarly, but with a 'tst.gp' such as

================
set term xlib
set output "out.xlib"
plot "dat.bin" binary format="%float32%float32" with lines
================


Results. Each record is usertime,systemtime in seconds.

|                     | %04d      | %d        | %d, 16-bit binary V | %d, 32-bit binary V |
|---------------------+-----------+-----------+---------------------+---------------------|
| inboard from ASCII  | 1.91,0.21 | 1.86,0.21 | 1.53,0.19           | 1.53,0.21           |
| inboard from binary | 0.95,0.20 | 0.89,0.20 | 0.56,0.16           | 0.56,0.20           |
| outboard            | 0.82,0.11 | 0.85,0.11 | 0.19,0.10           | 0.21,0.11           |

First off, we see that splitting the fields with whitespace speeds up
the inboard driver a little bit, while slowing down the outboard one a
little bit. Not completely sure why, but the difference is negligible,
so I didn't go digging.

However, sending the data over in binary produces HUGE performance
gains. The inboard driver is 37% faster (when the original input is
binary too), while the outboard one is a whopping 76% faster. Not quite
sure why this is so uneven; maybe the outboard driver parses the data
more times than it needs to?

None of this is really surprising, but it really reinforces the earlier
point that seeking performance gains in our ASCII representation is
foolish, leading to minimal speedups while making the code less
manageable. When I started this, I wasn't going to advocate that we
move to a binary data stream, but the speedup is so significant that we
really should, I think.

I'm attaching a patch that changes the V command to work in binary.
Note that this patch is not at all good-enough to merge yet; I'm
attaching it so that others could run these tests as well. So, should
we move the intensive commands to binary? What commands are these,
other than 'V'?

dima

Re: [PATCH] x11 terminal respects the requested aspect ratio

[Dima K. <gn...@di...>]

> > Hi Ethan.
> > 
> > Good you found the issue. Do you know why field widths are specified at
> > all? Why don't we PRINT2("V%d %d\n") and then sscanf("%d %d", ...) on
> > the other side? No efficiency is gained by specifying the widths, it
> > only builds in limitations and makes the code brittle, as we have just
> > observed.
> 
> The basic X11 framework was in place long before my involvement, so I do 
> not know.  My best guess is that the fixed field width was chosen so as 
> to reduce the amount of data sent through the gnuplot->gnuplot_x11
> channel.
> Adding a space in front of every coordinate pair would increase the
> traffic by as much as 25%.
> 
> As I recall, when the polygon encoding was switched from formatted to
> binary (2004), the advocates of binary encoding/decoding had benchmarks
> showing data transfer really was a performance bottleneck, and reducing
> the number of bytes sent over the channel resulted in faster plotting.
> I do not know if this would still be true on modern machines.
> 
> We should benchmark before and after your latest patch.

The extra space can also save bytes when the values being sent across
have fewer than 4 digits in them. As I see it, one should use ASCII data
links if they want robustness and readability, and binary ones if they
want speed. Here we're sending data in ASCII, while worrying about a few
extra cycles. If you make up a test that benchmarks before and after
this patch, I'll write another version of the data passing, that uses
binary data; if there're any performance gains here, that's where they
are.

> 
> > I'm attaching a patch that removes hardcoded field sizes from ALL %d
> > and %u fields. Some extra care had to be taken in places where a
> > numerical field is followed by a string (%s), since the '4' was
> > hardcoded in those places too. The bug you described is now gone, and
> > the code is more robust.
> > 
> > I noticed another, very related issue. x11.trm has
> > 
> >     /* badly outrange labels can overflow into text field */
> >     if (x < 10000 && y < 10000) {
> > 	PRINT3("T%d %d %s\n", x, y, str);
> >     }
> > 
> > Here we're hardcoding the 10000 again. This has the effect of not
> > printing the legend label when the window is skinny (as you were making
> > it). This test needs to be adjusted or removed entirely; not sure
> > which. CVS says this test was added by lhecking in 1998. This predates
> > all of sourceforge. Newsgroup logs go back that far, but I didn't see
> > any mention of this issue. Thoughts?
> 
> That does look like an early instance of the same problem.
> But I wonder how that code ever triggered?

Is lhecking still around? Think he still remembers?

> 
> > Finally, in writing the patch, I stumbled on what looks like a bug in
> > what looks like dead code. In gplt_x11.c, look at the
> > 
> >     else if (*buffer == X11_GR_FILLED_POLYGON) {	/* filled
> > [snip]
> > I suspect the whole block is dead, and this bug is thus
> > never hit. If this whole block is truly dead, it should be removed.
> > dima
> 
> I think it must be left over from when the choice between binary/ascii
> polygon encoding was a configuration option.  Since 2009 the ascii 
> option no longer exists.  So yeah, that section is dead code.

Great. Can you remove it then?

Dima