gnuplot-info

[Gnuplot-info] plot dstat output cvs file

[Bart V. <Bar...@te...>]

I'm trying to plot an output file from dstat obtained with

   dstat -tv --output dstat_timed.csv

the data-file looks as follows:

-----------------------------------------------------------------------------
"Dstat 0.7.2 CSV output"
"Author:","Dag Wieers 
<da...@wi...>",,,,"URL:","http://dag.wieers.com/home-made/dstat/"
"Host:","maverick",,,,"User:","mc303"
"Cmdline:","dstat -tv --output dstat_timed.csv",,,,"Date:","30 Oct 2013 
11:19:41 CET"

"system","procs",,,"memory 
usage",,,,"paging",,"dsk/total",,"system",,"total cpu usage",,,,,
"time","run","blk","new","used","buff","cach","free","in","out","read","writ","int","csw","usr","sys","idl","wai","hiq","siq"
30-10 
11:19:41,0.000,0.0,1.284,729153536.0,416268288.0,1697247232.0,1353175040.0,0.0,0.0,230338.748,250793.406,1390.199,2597.766,12.913,5.164,74.172,7.655,0.005,0.092
30-10 
11:19:42,0.0,0.0,0.0,729186304.0,416268288.0,1697263616.0,1353125888.0,0.0,0.0,0.0,0.0,955.0,1634.0,3.535,4.040,92.424,0.0,0.0,0.0
30-10 
11:19:43,0.0,0.0,1.0,729186304.0,416268288.0,1697263616.0,1353125888.0,0.0,0.0,0.0,0.0,1104.0,2043.0,5.612,4.082,89.796,0.0,0.0,0.510
30-10 
11:19:44,0.0,0.0,1.0,729194496.0,416268288.0,1697263616.0,1353117696.0,0.0,0.0,0.0,0.0,1071.0,1841.0,6.186,3.093,90.722,0.0,0.0,0.0
30-10 
11:19:45,0.0,0.0,17.0,730214400.0,416268288.0,1697263616.0,1352097792.0,0.0,0.0,0.0,4096.0,1313.0,2408.0,7.143,3.571,89.286,0.0,0.0,0.0
-----------------------------------------------------------------------------

I'm trying to plot with the following script:

---------------------------------------------------
set datafile separator ","

set xlabel "Time"
set xdata time
set timefmt "%d-%m %H:%M:%S"
set format x "%H:%M"

set title "procs"
plot "dstat_timed.csv" u 1:2 t columnheader(2) w lp, \
      "" using 1:3 t columnheader(3) w lp, \
      "" using 1:4 t columnheader(4) w lp
---------------------------------------------------

but unfortunately, this only works if I uncomment the first 6 lines in 
the data-file.

I already tried plotting with

   plot "dstat_timed.csv" u 1:2 every ::6 t columnheader(2) w lp

but this didn't help.

How can I tell gnuplot to ignore the first 6 lines in the data-file?

Thanks for your help,
Bart

Re: [Gnuplot-info] plot dstat output cvs file

[Theo <th...@gm...>]

your data looks a bit distorted (return/newlines), maybe it's because of 
the mail program.

If I correct this and apply all your settings and

plot 'Data.csv' u 1:4 every ::6

with gnuplot 4.6.3 I do not have problems.

Do you get error messages? What is actually plotted?
Maybe try first simpler data to find out what's going wrong.

Theo.

Can you plot e.g. this?

"blabla1"
"blabla2"
"blabla3"
"blabla4"
"blabla5"
"blabla6"
30-10 11:19:40,1.0,2.0,3.0
30-10 11:19:41,1.1,2.1,3.1
30-10 11:19:42,1.2,2.2,3.2
30-10 11:19:43,1.3,2.3,3.3
30-10 11:19:44,1.4,2.4,3.4

Re: [Gnuplot-info] plot dstat output cvs file

[Bart V. <Bar...@te...>]

On 10/30/2013 10:06 PM, Theo wrote:
> your data looks a bit distorted (return/newlines), maybe it's because of
> the mail program.

True.

> If I correct this and apply all your settings and
>
> plot 'Data.csv' u 1:4 every ::6
>
> with gnuplot 4.6.3 I do not have problems.
>
> Do you get error messages? What is actually plotted?
> Maybe try first simpler data to find out what's going wrong.
>
> Theo.
>
> Can you plot e.g. this?
>
> "blabla1"
> "blabla2"
> "blabla3"
> "blabla4"
> "blabla5"
> "blabla6"
> 30-10 11:19:40,1.0,2.0,3.0
> 30-10 11:19:41,1.1,2.1,3.1
> 30-10 11:19:42,1.2,2.2,3.2
> 30-10 11:19:43,1.3,2.3,3.3
> 30-10 11:19:44,1.4,2.4,3.4

First of all, my version of gnuplot is gnuplot 4.6 patchlevel 3.

Secondly, If I plot the following data

"blabla1"
"blabla2"
"blabla3"
"blabla4"
"blabla5"
"blabla6"
"blabla7"
30-10 11:19:40,1.0,2.0,3.0
30-10 11:19:41,1.1,2.1,3.1
30-10 11:19:42,1.2,2.2,3.2
30-10 11:19:43,1.3,2.3,3.3
30-10 11:19:44,1.4,2.4,3.4

using the script

------ cut here -----------------------------------------------
set loadpath '../data'
set datafile separator ","

set xlabel "Time"
set xdata time
set timefmt "%d-%m %H:%M:%S"
set format x "%H:%M"

plot "dstat_test.csv" u 1:2 every ::6 t columnheader(2) w lp, \
      "" using 1:3 every ::6 t columnheader(3) w lp, \
      "" using 1:4 every ::6 t columnheader(4) w lp
------ cut here -----------------------------------------------

then I correctly get three straight lines.  If I however remove the line 
"blabla5" so that the data-file looks more like a dstat data output file:

"blabla1"
"blabla2"
"blabla3"
"blabla4"

"blabla6"
"blabla7"
30-10 11:19:40,1.0,2.0,3.0
30-10 11:19:41,1.1,2.1,3.1
30-10 11:19:42,1.2,2.2,3.2
30-10 11:19:43,1.3,2.3,3.3
30-10 11:19:44,1.4,2.4,3.4

and i use the same script for plotting, then the plot is wrong and I 
only see three dots plottet vertically above each other (at y-values 
1.5, 2.5 and 3.5).

So the empty line is causing me trouble here... but I don't know why :-(

Any help appreciated.

Kind regards,
Bart

Re: [Gnuplot-info] plot dstat output cvs file

[Theo <th...@gm...>]

what happens if you additionally apply the following?

set datafile commentschar '"'

Then it seems to work even with the empty line.
However, you might have to skip the automatic columnheader() and set the 
title manually. Maybe there is still a way to extract the title 
automatically?!

plot "Data.dat" u 1:2 t "Title 2" w lp, "" using 1:3 t "Title 3" w lp, 
"" using 1:4 t "Title 4" w lp

Re: [Gnuplot-info] plot dstat output cvs file

[Theo <th...@gm...>]

Hi Bart,

I read again the description of 'every' carefully (which I also 
recommend to you ;-).
Obviously, empty lines in data rows seperate individual data blocks.
So, if there is an empty line in your data it starts a new block.

Let's use the simplified data:

"blabla1",,,
"blabla2",,,
"blabla3",,,
"blabla4",,,

"blabla6",,,
"date time","A","B","C"
30-10 11:19:40,1.0,2.0,3.0
30-10 11:19:41,1.1,2.1,3.1
30-10 11:19:42,1.2,2.2,3.2
30-10 11:19:43,1.3,2.3,3.3
30-10 11:19:44,1.4,2.4,3.4

The command to plot this:

plot 'Data.dat' u 1:2 every 1:1:2:1 w lp t "Title 2"    # and so on...
    which is equivalent to
plot 'Data.dat' u 1:2 every ::2:1 w lp t "Title 2"

# explanation of every 1:1:2:1
data points increment is 1
block increment is 1
starting point is 2 (points in each block start from 0)
starting block is 1 (blocks start from 0)

However, I have not found a way to automatically take the columnheaders 
from line 2 of block 1.
gnuplot seems to always take line 0 from block 0.

Re: [Gnuplot-info] plot dstat output cvs file

[Bart V. <Bar...@te...>]

On 11/07/2013 08:39 PM, Theo wrote:
> Hi Bart,
>
> I read again the description of 'every' carefully (which I also
> recommend to you ;-).
> Obviously, empty lines in data rows seperate individual data blocks.
> So, if there is an empty line in your data it starts a new block.

True.  I was aware of that.  But indeed you are right, I should've read 
the description of 'every' more carefully :-)

> Let's use the simplified data:
>
> "blabla1",,,
> "blabla2",,,
> "blabla3",,,
> "blabla4",,,
>
> "blabla6",,,
> "date time","A","B","C"
> 30-10 11:19:40,1.0,2.0,3.0
> 30-10 11:19:41,1.1,2.1,3.1
> 30-10 11:19:42,1.2,2.2,3.2
> 30-10 11:19:43,1.3,2.3,3.3
> 30-10 11:19:44,1.4,2.4,3.4
>
> The command to plot this:
>
> plot 'Data.dat' u 1:2 every 1:1:2:1 w lp t "Title 2"    # and so on...
>      which is equivalent to
> plot 'Data.dat' u 1:2 every ::2:1 w lp t "Title 2"
>
> # explanation of every 1:1:2:1
> data points increment is 1
> block increment is 1
> starting point is 2 (points in each block start from 0)
> starting block is 1 (blocks start from 0)
>
> However, I have not found a way to automatically take the columnheaders
> from line 2 of block 1.
> gnuplot seems to always take line 0 from block 0.

Indeed.  And that is exactly still my annoying problem :-)  If anybody 
knows a way how to take the correct columnheaders, feel free to let me know!

Kind regards,
Bart

Re: [Gnuplot-info] plot dstat output cvs file

[Theo <th...@gm...>]

Hi Bart,

there is a solution :-)
Not very elegant but the result is important...
Basically, you are creating a few dummy plots thereby extracting the 
columnheaders into string variables.
Then you create your desired plot using those variables.

Let me know whether it also works for you.

Theo.


I used the reduced data "Bart.dat":


"Dstat 0.7.2 CSV output"
"Author:","Dag Wieers <da...@wi...>",,
"Host:","maverick",,
"Cmdline:","dstat -tv --output dstat_timed.csv",,

"system","procs",,
"time","run","blk","new"
30-10 11:19:41,0.000,0.0,1.284,729153536.0
30-10 11:19:42,0.0,0.0,0.0,729186304.0
30-10 11:19:43,0.0,0.0,1.0,729186304.0
30-10 11:19:44,0.0,0.0,1.0,729194496.0
30-10 11:19:45,0.0,0.0,17.0,730214400.0



set datafile separator ","

# extract the columheaders from block 1 and point 1
# and store them in variables
# this will create some warnings which can be ignored
plot "Bart.dat" u (TitleCol2=stringcolumn(2),1):0 every ::1:1:1:1
plot "Bart.dat" u (TitleCol3=stringcolumn(3),1):0 every ::1:1:1:1
plot "Bart.dat" u (TitleCol4=stringcolumn(4),1):0 every ::1:1:1:1

set xlabel "Time"
set xdata time
set timefmt "%d-%m %H:%M:%S"
set format x "%H:%M"
set xrange [*:*]

plot "Bart.dat" u 1:2 every ::2:1 w lp t TitleCol2, \
              '' u 1:3 every ::2:1 w lp t TitleCol3, \
		 '' u 1:4 every ::2:1 w lp t TitleCol4