gnuplot-beta

change in handling missing data.

[Dmitri A. S. <das...@gm...>]

I there any way I can get an old (4.0) behavior with new
gnuplot when plotting data files with missing data?

E.g. consider the file (say "test1.dat")

# -- begin
0 1
1 0
NaN NaN
2 1
1 2
# -- end

In gnuplot 4.0
plot "test1.dat" with line
produces plot with two segments.

With 4.1, first one has to do
set datafile missing "NaN"
(that still would not handle "nan")
but after that
plot "test1.dat" with line
would connect the segments along (1 0) (2 1) line.

I am not to argue wich way is better (though I would prefer
gnuplot to handle IEEE special numbers automatically),
I am just wondering if this behavior (to connect the segments)
is indeed the expected one (not a bug) and if it is indeed so,
is there any variable I can set to get old style behavior?

Sincerely,

Dmitri.
--

Re: change in handling missing data.

[Daniel J S. <dan...@ie...>]

Dmitri A. Sergatskov wrote:
> I there any way I can get an old (4.0) behavior with new
> gnuplot when plotting data files with missing data?

At one point I had created a patch for better control of this and gave a demo illustrating how to create plots with breaks in the line and simply skipping points.  I thought it was a good idea for a feature, i.e., a method of telling gnuplot how it should handle various types, but it didn't get much traction.

> 
> E.g. consider the file (say "test1.dat")
> 
> # -- begin
> 0 1
> 1 0
> NaN NaN
> 2 1
> 1 2
> # -- end
> 
> In gnuplot 4.0
> plot "test1.dat" with line
> produces plot with two segments.
> 
> With 4.1, first one has to do
> set datafile missing "NaN"
> (that still would not handle "nan")

There is patch for multiple missing, e.g., set missint "NaN" "nan" "?".

Give that a try if you need such a feature.

> but after that
> plot "test1.dat" with line
> would connect the segments along (1 0) (2 1) line.
> 
> I am not to argue wich way is better (though I would prefer
> gnuplot to handle IEEE special numbers automatically),
> I am just wondering if this behavior (to connect the segments)
> is indeed the expected one (not a bug) and if it is indeed so,
> is there any variable I can set to get old style behavior?

I say it should be configurable.  Anyway, the thread of this discussion in the past is titled:

Subject: Re: MISSING and UNDEFINED
Date: Sun, 18 Jun 2006 18:00:57 -0500
From: Daniel J Sebald <dan...@ie...>

There are some PNGs there showing the output of the demo.

Dan

Re: change in handling missing data.

[Daniel J S. <dan...@ie...>]

Daniel J Sebald wrote:

> From: Daniel J Sebald <danielDOTsebaldATieeeDOTorg>

Oh no!  I sent my email address inside the body.  Shouldn't do that...

Re: change in handling missing data.

[Ethan A M. <merritt@u.washington.edu>]

On Sunday 06 August 2006 02:20 pm, Dmitri A. Sergatskov wrote:
> I there any way I can get an old (4.0) behavior with new
> gnuplot when plotting data files with missing data?

Short answer:  Yes.
	plot 'test1.dat' using ($1):($2) with lines

Longer answer:

The whole "missing" issue drives me to distraction.
No two people expect the same thing from it, so it's no wonder
that many bug reports are filed against it.  3.7 and 4.0 did 
not behave identically, so existing scripts are not an 
infallible guide either.

For what it's worth... 
Your "NaN" strings *are* being read as legal IEEE format numbers.  
Using "set datafile missing ..." is not the issue.
The issue is what to do when a number is not legal (NaN, Inf,
underflow). 
Plot, or not plot? 
Increment the line number, or not?
Error message?

	EAM

 
> E.g. consider the file (say "test1.dat")
> 
> # -- begin
> 0 1
> 1 0
> NaN NaN
> 2 1
> 1 2
> # -- end
> 
> In gnuplot 4.0
> plot "test1.dat" with line
> produces plot with two segments.
> 
> With 4.1, first one has to do
> set datafile missing "NaN"
> (that still would not handle "nan")
> but after that
> plot "test1.dat" with line
> would connect the segments along (1 0) (2 1) line.
> 
> I am not to argue wich way is better (though I would prefer
> gnuplot to handle IEEE special numbers automatically),
> I am just wondering if this behavior (to connect the segments)
> is indeed the expected one (not a bug) and if it is indeed so,
> is there any variable I can set to get old style behavior?
> 
> Sincerely,
> 
> Dmitri.
> --
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
> 

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742

Re: change in handling missing data.

[Dmitri A. S. <das...@gm...>]

On 8/6/06, Ethan A Merritt <merritt@u.washington.edu> wrote:
> On Sunday 06 August 2006 02:20 pm, Dmitri A. Sergatskov wrote:
> > I there any way I can get an old (4.0) behavior with new
> > gnuplot when plotting data files with missing data?
>
> Short answer:  Yes.
>         plot 'test1.dat' using ($1):($2) with lines

I figure this out. This has a couple problems.
The issue came about from using gnuplot from octave. Since
octave interpreter writes the (temporary) data file for gnuplot to
plot, it knows the data format and thus does not need to specify
"using" string.
A simple benchmark (time gnuplot < cmd) shows that (on my computer
at least) plot "file" using ($1):($2) takes about 150% as long
as plot "file".
So I was hoping there is some setting that would allow old
behavior. Apparently it is not the case, so be it, we will adapt.
Perhaps it is time to convert to "binary" datafile format...

>
> Longer answer:
>
> The whole "missing" issue drives me to distraction.
> No two people expect the same thing from it, so it's no wonder
> that many bug reports are filed against it.  3.7 and 4.0 did
> not behave identically, so existing scripts are not an
> infallible guide either.
>
> For what it's worth...
> Your "NaN" strings *are* being read as legal IEEE format numbers.
> Using "set datafile missing ..." is not the issue.
> The issue is what to do when a number is not legal (NaN, Inf,
> underflow).

It is an issue, because previously I did not bother to check
if NaNs in my files written as "NaN" or as "nan".
Also, how do I handle file that has both "NaN" and "Inf"?

> Plot, or not plot?
> Increment the line number, or not?
> Error message?

I understand this problem. I also prefere the old (4.0) behavior.
For one thing, since the plot is the graphical representation of
the data I want to have a visual feedback that I have "funny"
datapoints. Broken line provides such a feedback. Plotting
over it -- hides it.
Again, I am not asking to change anything. At least not at this
moment...

>
>         EAM
>

Sincerely,

Dmitri.
--

Re: change in handling missing data.

[Daniel J S. <dan...@ie...>]

Dmitri A. Sergatskov wrote:

> So I was hoping there is some setting that would allow old
> behavior. Apparently it is not the case, so be it, we will adapt.
> Perhaps it is time to convert to "binary" datafile format...

That would give a speed increase, but of course there isn't the flexibility with DF_MISSING, DF_UNDEFINED as there is with ASCII files, unless we want to define strings some how... but I think that is low priority.  Binary is for speed purposes.

>>The issue is what to do when a number is not legal (NaN, Inf,
>>underflow).
> 
> 
> It is an issue, because previously I did not bother to check
> if NaNs in my files written as "NaN" or as "nan".
> Also, how do I handle file that has both "NaN" and "Inf"?

[snip]

> I understand this problem. I also prefere the old (4.0) behavior.
> For one thing, since the plot is the graphical representation of
> the data I want to have a visual feedback that I have "funny"
> datapoints. Broken line provides such a feedback. Plotting
> over it -- hides it.
> Again, I am not asking to change anything. At least not at this
> moment...

You are confirming the issue, however.  I say give the user the flexibility and not try to interpret what they mean, then it is no more worries for developers.

Dan

Re: change in handling missing data.

[Daniel J S. <dan...@ie...>]

Ethan A Merritt wrote:
> On Sunday 06 August 2006 02:20 pm, Dmitri A. Sergatskov wrote:
> 
>>I there any way I can get an old (4.0) behavior with new
>>gnuplot when plotting data files with missing data?
> 
> 
> Short answer:  Yes.
> 	plot 'test1.dat' using ($1):($2) with lines
> 
> Longer answer:
> 
> The whole "missing" issue drives me to distraction.
> No two people expect the same thing from it, so it's no wonder
> that many bug reports are filed against it.  3.7 and 4.0 did 
> not behave identically, so existing scripts are not an 
> infallible guide either.
> 
> For what it's worth... 
> Your "NaN" strings *are* being read as legal IEEE format numbers.  
> Using "set datafile missing ..." is not the issue.
> The issue is what to do when a number is not legal (NaN, Inf,
> underflow). 
> Plot, or not plot? 
> Increment the line number, or not?
> Error message?

I agree.  No one particular way of dealing with this will satisfy everybody.  Hence I say it should be configurable.  And doing so would really be a nice feature that users like.

The problem with satisfying everyone right now is that there simply isn't enough variation in treating the data.  Being so limited is the problem.  We have expanded things a bit with DF_MISSING, but not enough.

There should be

1) Enough variety to satisfy everyone.
2) And/or configurability.
3) Multiple definitions for the class of data type.

We don't want to get so abstract that things become arcane, so lets go with say three types with some default behavior.  Internally these would be

DF_MISSING:  Simply SKIP, no break between points
DF_UNDEFINED:  BREAK between points.
DF_NAN:  Don't know, also BREAK?

Well, then we also allow some configurability with multiple definitions.  That would be enough flexibility so that the user has more than one way of attacking this problem.  S/he could add more strings to a data class or switch around strings in the definition of these classes, or configure the manner in which each class is treated.

With a demo, the user should get the idea clearly.

Dan

Re: change in handling missing data.

[Ethan A M. <merritt@u.washington.edu>]

On Sunday 06 August 2006 09:07 pm, Dmitri A. Sergatskov wrote:
> > For what it's worth...
> > Your "NaN" strings *are* being read as legal IEEE format numbers.
> > Using "set datafile missing ..." is not the issue.
> > The issue is what to do when a number is not legal (NaN, Inf,
> > underflow).
> 
> It is an issue, because previously I did not bother to check
> if NaNs in my files written as "NaN" or as "nan".

gnuplot reads the number using strtod().
The linux man page for strtod() says:

      An infinity is either ``INF'' or ``INFINITY'', disregarding case.
      A  NAN  is  ``NAN''  (disregarding  case) optionally followed by `(', a
      sequence of characters, followed by ')'.  The character  string  speci-
      fies in an implementation-dependent way the type of NAN.

> Also, how do I handle file that has both "NaN" and "Inf"?

They are the same, for this purpose.
Do not try to describe them via "set datafile missing";
just let them be read in as floating point numbers.

It is in general not safe to read files without the "using"
specifier unless you are absolutely certain what each line contains.
We have tried to maintain backwards compatibility with old behaviour,
but recent extensions and new plot modes require "using" in order
to work at all. For example, in order to read color information from
the data file, or point size, you need extra columns.  But if you
do not tell the program which column is which, it will get confused
and plot the wrong thing altogether. 

> A simple benchmark (time gnuplot < cmd) shows that (on my computer
> at least) plot "file" using ($1):($2) takes about 150% as long
> as plot "file".

That's a bit hard to believe.  

How big are your data files?

Do they have many columns per line, or only two? 
(This used to be an issue but I thought we fixed it).

Are you willing/able to run profiling so that you can tell us
which routine[s] the time is being lost to?

	EAM

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742

Re: change in handling missing data.

[Daniel J S. <dan...@ie...>]

Ethan A Merritt wrote:

>>A simple benchmark (time gnuplot < cmd) shows that (on my computer
>>at least) plot "file" using ($1):($2) takes about 150% as long
>>as plot "file".
> 
> 
> That's a bit hard to believe.  

Actually, it may be correct.  Isn't the difference between "using 1:2" and "using ($1):($2)" that the latter passes the data through a function?  (In this case a very simple function.)  That is, the "action table" code is called.

Try just "using 1:2", Dmitri, and see if that speeds things back to what you expect.

Dan

Re: change in handling missing data.

[Dmitri A. S. <das...@gm...>]

On 8/6/06, Ethan A Merritt <merritt@u.washington.edu> wrote:

> > It is an issue, because previously I did not bother to check
> > if NaNs in my files written as "NaN" or as "nan".
>
> gnuplot reads the number using strtod().
> The linux man page for strtod() says:
>
>       An infinity is either ``INF'' or ``INFINITY'', disregarding case.
>       A  NAN  is  ``NAN''  (disregarding  case) optionally followed by `(', a
>       sequence of characters, followed by ')'.  The character  string  speci-
>       fies in an implementation-dependent way the type of NAN.


Here is the data file ("t3.dat"):
# -- begin
0 2
2 0
NaN NaN
4 2
2 4
# -- end

with gnuplot 4.1 (today's CVS snapshot):
gnuplot> plot "t3.dat"
              ^
         Bad data on line 3

gnuplot> set datafile missing "NaN"
gnuplot> plot "t3.dat"
gnuplot>

(gnuplot 4.0 just plots it)


> That's a bit hard to believe.
>
> How big are your data files?

For the benchamrk I made 2 column data file
x = (-10,10), y = sin(x)
with 1e6 (one million) rows.

cmd is just a one line script
first it is:

plot "t.dat"

[dima@das200 tmp]$ time gnuplot < cmd

real    0m3.814s
user    0m3.488s
sys     0m0.152s

Now changed it to

plot "t.dat" using 1:2

[dima@das200 tmp]$ time gnuplot < cmd

real    0m3.831s
user    0m3.528s
sys     0m0.176s

And finally changed it to

plot "t.dat" using ($1):($2)

[dima@das200 tmp]$ time gnuplot < cmd

real    0m11.086s
user    0m5.828s
sys     0m4.600s

So the wall clock actually increased by factor of 4 mostly due to
"sys" time (originally I just looked at user time -- that is where 150%
came from). I tried both X11 terminal and "dumb" (and output to
/dev/null)  w/o any noticeable difference. These numbers do not
include time of
gnuplot_x11 doing actual drawing.

I did multiple passes, so I am pretty sure the data file is in disk cache.

All this is on FedoraCore 5/ Pentiun4 2.6GHz / 2GB of RAM

>
>         EAM
>

Sincerely,

Dmitri.
--

Re: change in handling missing data.

[Ethan A M. <merritt@u.washington.edu>]

On Sunday 06 August 2006 10:07 pm, Dmitri A. Sergatskov wrote:
> For the benchamrk I made 2 column data file
> x = (-10,10), y = sin(x)
> with 1e6 (one million) rows.
> 
> plot "t.dat" using 1:2
> 
> [dima@das200 tmp]$ time gnuplot < cmd
> 
> real    0m3.831s
> user    0m3.528s
> sys     0m0.176s
> 
> plot "t.dat" using ($1):($2)
> 
> [dima@das200 tmp]$ time gnuplot < cmd
> 
> real    0m11.086s
> user    0m5.828s
> sys     0m4.600s

OK, I can confirm that to some extent.
Benchmarked on a hyperthreaded P4 at 2.6GHz
(I used  set term post; set output '/dev/null')

For  plot "t.dat" using 1:2
2.665u 0.094s 0:02.75 100.0%    0+0k 0+0io 0pf+0w

For plot "t.dat" using ($1):($2) 
4.456u 2.342s 0:06.79 100.0%    0+0k 0+0io 0pf+0w

> I did multiple passes, so I am pretty sure the data file is in disk cache. 
> All this is on FedoraCore 5/ Pentiun4 2.6GHz / 2GB of RAM

So it runs faster on my machine, but yes it spends more time on the second test.
Here are the gprof outputs:

plot "t.dat" using 1:2
======================
Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls   s/call   s/call  name
 27.98      2.37     2.37  1000000     0.00     0.00  store2d_point
 20.19      4.08     1.71  1000000     0.00     0.00  df_tokenise
 15.23      5.37     1.29  1000041     0.00     0.00  PS_vector
 13.70      6.53     1.16  1000001     0.00     0.00  df_readascii
 10.15      7.39     0.86  1000001     0.00     0.00  df_gets
  7.79      8.05     0.66        1     0.66     1.99  plot_lines
  2.01      8.22     0.17        1     0.17     6.44  get_data
  0.83      8.29     0.07  2000000     0.00     0.00  gp_strtod
  0.59      8.34     0.05  2000000     0.00     0.00  check_missing
  0.59      8.39     0.05  1000001     0.00     0.00  df_readline
 
plot "t.dat" using $1:$2
========================
Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls   s/call   s/call  name
 20.97      2.65     2.65  1000000     0.00     0.00  df_tokenise
 19.46      5.11     2.46  1000000     0.00     0.00  store2d_point
 15.51      7.07     1.96  1000001     0.00     0.00  df_readascii
  9.18      8.23     1.16  1000041     0.00     0.00  PS_vector
  8.23      9.27     1.04  1000001     0.00     0.00  df_gets
  5.66      9.99     0.72  2000002     0.00     0.00  evaluate_at
  4.67     10.58     0.59        1     0.59     1.78  plot_lines
  2.85     10.94     0.36        1     0.36    10.70  get_data
  1.78     11.16     0.23  2000000     0.00     0.00  f_dollars
  1.62     11.37     0.21  2000002     0.00     0.00  execute_at
  1.50     11.56     0.19  2000002     0.00     0.00  push
  1.27     11.72     0.16  2000002     0.00     0.00  pop
  1.27     11.88     0.16  2000000     0.00     0.00  check_missing
  1.11     12.02     0.14  4000000     0.00     0.00  real
  1.03     12.15     0.13  1000001     0.00     0.00  df_readline
  0.87     12.26     0.11  2000057     0.00     0.00  Gcomplex
  0.71     12.35     0.09  2000000     0.00     0.00  gp_strtod
  0.55     12.42     0.07  2000002     0.00     0.00  check_stack
  0.36     12.46     0.05                             df_determine_matrix_info
  0.32     12.50     0.04                             more_on_stack
  0.24     12.53     0.03  2000002     0.00     0.00  reset_stack

As Daniel said, the extra time is due to the overhead of
evaluating each coordinate as an expression rather than as a constant.
( evaluate_at f_dollars push pop check_stack reset_stack )
I don't see any horrible time-wasters in the profile, even when I break
it down to individual lines of code.  However, I still don't know which
bit, exactly, is being accounted for as "system time". Perhaps there
is some system call we can manage to do with out, but at present I
can't think what it would be.

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742

Re: change in handling missing data.

[Daniel J S. <dan...@ie...>]

Ethan A Merritt wrote:

> plot "t.dat" using 1:2
> ======================
> Each sample counts as 0.01 seconds.
>   %   cumulative   self              self     total
>  time   seconds   seconds    calls   s/call   s/call  name
>  27.98      2.37     2.37  1000000     0.00     0.00  store2d_point
>  20.19      4.08     1.71  1000000     0.00     0.00  df_tokenise
>  15.23      5.37     1.29  1000041     0.00     0.00  PS_vector
>  13.70      6.53     1.16  1000001     0.00     0.00  df_readascii
>  10.15      7.39     0.86  1000001     0.00     0.00  df_gets
>   7.79      8.05     0.66        1     0.66     1.99  plot_lines
>   2.01      8.22     0.17        1     0.17     6.44  get_data
>   0.83      8.29     0.07  2000000     0.00     0.00  gp_strtod
>   0.59      8.34     0.05  2000000     0.00     0.00  check_missing
>   0.59      8.39     0.05  1000001     0.00     0.00  df_readline
>  
> plot "t.dat" using $1:$2
> ========================
> Each sample counts as 0.01 seconds.
>   %   cumulative   self              self     total
>  time   seconds   seconds    calls   s/call   s/call  name
>  20.97      2.65     2.65  1000000     0.00     0.00  df_tokenise
>  19.46      5.11     2.46  1000000     0.00     0.00  store2d_point
>  15.51      7.07     1.96  1000001     0.00     0.00  df_readascii
>   9.18      8.23     1.16  1000041     0.00     0.00  PS_vector
>   8.23      9.27     1.04  1000001     0.00     0.00  df_gets
>   5.66      9.99     0.72  2000002     0.00     0.00  evaluate_at
>   4.67     10.58     0.59        1     0.59     1.78  plot_lines
>   2.85     10.94     0.36        1     0.36    10.70  get_data
>   1.78     11.16     0.23  2000000     0.00     0.00  f_dollars
>   1.62     11.37     0.21  2000002     0.00     0.00  execute_at
>   1.50     11.56     0.19  2000002     0.00     0.00  push
>   1.27     11.72     0.16  2000002     0.00     0.00  pop
>   1.27     11.88     0.16  2000000     0.00     0.00  check_missing
>   1.11     12.02     0.14  4000000     0.00     0.00  real
>   1.03     12.15     0.13  1000001     0.00     0.00  df_readline
>   0.87     12.26     0.11  2000057     0.00     0.00  Gcomplex
>   0.71     12.35     0.09  2000000     0.00     0.00  gp_strtod
>   0.55     12.42     0.07  2000002     0.00     0.00  check_stack
>   0.36     12.46     0.05                             df_determine_matrix_info
>   0.32     12.50     0.04                             more_on_stack
>   0.24     12.53     0.03  2000002     0.00     0.00  reset_stack
> 
> As Daniel said, the extra time is due to the overhead of
> evaluating each coordinate as an expression rather than as a constant.
> ( evaluate_at f_dollars push pop check_stack reset_stack )

I would have thought that evaluate_at or execute_at would have been where the increase is,  but that doesn't appear to be much in either case.  I would say PS_vector would be the same amount of CPU in both cases.  In the first case it is 15% in the second 9%.  That would mean 0.15 Total_1 = 0.9 Total_2, or Total_2 = 1.67 Total_1.  Is my thinking correct on that?  So a 67% increase in CPU consumption?

> I don't see any horrible time-wasters in the profile, even when I break
> it down to individual lines of code.  However, I still don't know which
> bit, exactly, is being accounted for as "system time". Perhaps there
> is some system call we can manage to do with out, but at present I
> can't think what it would be.

Good question.  evaluate_at, execute_at, push, pop, real, Gcomplex, check_stack, more_on_stack, and reset_stack don't appear to have an system calls.  df_determine_matrix_info has an "fseek" which is something that could be a potential time waster, however gprof doesn't indicate that being called 1 million times.

I'm not sure why df_determine_matrix_info is called in one case over the other, but what is peculiar is that gprof doesn't have anythingthing listed under the number of calls for df_determine_matrix_info or more_on_stack.

Dan

Re: change in handling missing data.

[Daniel J S. <dan...@ie...>]

Dmitri A. Sergatskov wrote:
 
> 
> Here is the data file ("t3.dat"):
> # -- begin
> 0 2
> 2 0
> NaN NaN
> 4 2
> 2 4
> # -- end
> 
> with gnuplot 4.1 (today's CVS snapshot):
> gnuplot> plot "t3.dat"
>               ^
>          Bad data on line 3

Try

  plot "t3.dat" using 1:2

without setting data file missing.  That appears to work the same.  Not sure why.

Dan

Re: change in handling missing data.

[Ethan A M. <merritt@u.washington.edu>]

On Sunday 06 August 2006 10:45 pm, Daniel J Sebald wrote:
> Dmitri A. Sergatskov wrote:
> > 0 2
> > 2 0
> > NaN NaN
> > 4 2
> > 2 4
> > 
> > with gnuplot 4.1 (today's CVS snapshot):
> > gnuplot> plot "t3.dat"
> >               ^
> >          Bad data on line 3


You need to say 
	plot "t3.dat" using ($1):($2)

>   plot "t3.dat" using 1:2
> 
> That appears to work the same.  Not sure why.

This is explained in the help files, with examples.
Really.
Now I agree that it's a peculiar way for it to work, but
it is in fact doing exactly what it is documented to do.

Someone long ago decided that  "using 1:2" should
behave differently than "using ($1):($2)". I don't know
why, but we're stuck with it unless we want to break 
with all previous versions.  

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742

Re: change in handling missing data.

[Daniel J S. <dan...@ie...>]

Ethan A Merritt wrote:

> 
> You need to say 
> 	plot "t3.dat" using ($1):($2)

Oh yeah.  I recall that now from the demo examples.


> Someone long ago decided that  "using 1:2" should
> behave differently than "using ($1):($2)". I don't know
> why, but we're stuck with it unless we want to break 
> with all previous versions.  

This sounds like a familiar discussion.

Dan

Re: change in handling missing data.

[Dmitri A. S. <das...@gm...>]

On 8/7/06, Ethan A Merritt <merritt@u.washington.edu> wrote:
> On Sunday 06 August 2006 10:07 pm, Dmitri A. Sergatskov wrote:
...
>
> For plot "t.dat" using ($1):($2)
> 4.456u 2.342s 0:06.79 100.0%    0+0k 0+0io 0pf+0w
>
> > I did multiple passes, so I am pretty sure the data file is in disk cache.
> > All this is on FedoraCore 5/ Pentiun4 2.6GHz / 2GB of RAM
>
> So it runs faster on my machine, but yes it spends more time on the second test.

I am surprised that the difference is so large. I tried running both
hyperthreaded and non-hyperthreaded and it is pretty much the same.
May be the data format matters?

My file is available at
ftp://coffee.phys.unm.edu/pub/dima/incoming/t-das.dat.bz2

(it is 11 Meg)

You can drop off yours to there as well.

I've also tried GNU time (/usr/bin/time, instead of using
built-in bash time); it gives slightly more info, and the only big
difference I've noticed was:

(plot w/o using):

Involuntary context switches: 452

(plot with using ($1):($2))

Involuntary context switches: 1165

Perhaps evaluation of a token trashes the CPU cache ...

> --
> Ethan A Merritt
>

Sincerely,

Dmitri.
--

Re: change in handling missing data.

[Ethan A M. <merritt@u.washington.edu>]

On Monday 07 August 2006 12:44 am, Dmitri A. Sergatskov wrote:
> On 8/7/06, Ethan A Merritt <merritt@u.washington.edu> wrote:
> > On Sunday 06 August 2006 10:07 pm, Dmitri A. Sergatskov wrote:

> > For plot "t.dat" using ($1):($2)
> > 4.456u 2.342s 0:06.79 100.0%    0+0k 0+0io 0pf+0w
> >
> > > I did multiple passes, so I am pretty sure the data file is in disk cache.
> > > All this is on FedoraCore 5/ Pentiun4 2.6GHz / 2GB of RAM
> >
> > So it runs faster on my machine, but yes it spends more time on the second test.
> 
> I am surprised that the difference is so large. I tried running both
> hyperthreaded and non-hyperthreaded and it is pretty much the same.
> My file is available at
> ftp://coffee.phys.unm.edu/pub/dima/incoming/t-das.dat.bz2

I get the same timings as before if I use your input data file.
Benchmark script
  set term post
  set output '/dev/null'
  plot "t-das.dat" using 1:2 with lines
Timing
  2.728u 0.133s 0:02.86 99.6%     0+0k 0+0io 0pf+0w

Same with
  plot "t-das.dat" using ($1):($2) with lines
Timing
  4.479u 2.295s 0:06.77 99.8%     0+0k 0+0io 0pf+0w

System info
-----------
lascaux [188] uname -a
Linux lascaux 2.6.12-24mdksmp #1 SMP Mon Jul 17 12:46:00 MDT 2006 
         i686 Intel(R) Pentium(R) 4 CPU 2.60GHz unknown GNU/Linux
lascaux [189] head -1 /proc/meminfo
MemTotal:       514176 kB

gcc version 4.0.1 
CFLAGS = -Wall -g  -O

-- 
Ethan A Merritt
Biomolecular Structure Center
University of Washington, Seattle 98195-7742

Re: change in handling missing data.

[<br...@ph...>]

Dmitri A. Sergatskov wrote:
> Here is the data file ("t3.dat"):
> # -- begin
> 0 2
> 2 0
> NaN NaN
> 4 2
> 2 4
> # -- end

I think one important question here has been entirely overlooked. 
Dmitri, you said that this was *octave* writing this data file to be 
used by gnuplot, right?  Well, here's the question: why is octave 
writing numerical garbage into its output, and why should gnuplot be the 
place to worry about that?