Re: [Gnuplot-info] columns with time format as "y"

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

I have had the data separator set to comma for few years as the
data is a CSV file.

The issue is the leading comma.  It appears that the date
format assumes the time information is in the first two fields
of the data.  Ergo the leading comma of the data has caused
error messages to effect of invalid time from gnuplot which is
why the data is pre-processed to remove the leading comma of
the data before processed by gnuplot.

The issue is not if I can plot the data of a CSV file.  I have
been plotting CSV files for a number of years.  I have scripts
that run every 15 minutes creating over 20 plots from 5
different CSV files for number of years.  I like to eliminate
the extra external step that processes the CSV data to remove
the leading comma in the data.  The data that is very large
really needs to be refreshed every 5 minutes, but cannot as the
data has to be passed over twice to plot the data with
gnuplot.  If I did not have to have the external step to
remove the leading comma in the data then this very large data
could be plotted every 5 minutes.  The smaller CSV files also
would plot much faster be eliminating the extra step to remove
the leading comma before gnuplot can process the CSV data. 

John L. Males
Toronto, Ontario
Canada
21 April 2020 16:33 -0400 EDT

================================================================

2020-04-21 20:16:43+0000-UTC Time: 1587500203 PC/System time

21 Apr 20:16:43 ntpdate[34041]: ntpdate 4.2.8p12-a (1)

21 Apr 20:16:58 ntpdate[35558]: step time server 216.232.132.31
offset 0.001401 sec

FreeBSD 11.3-STABLE FreeBSD 11.3-STABLE #0 r349903: Thu Jul 11
16:13:47 UTC 2019
ro...@re...:/usr/obj/usr/src/sys/GENERIC 

(Work in progress alternative to Linux Kernel of its own right,
 Debian, and
 other Linux based Kernel distributions determined.)

Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz
Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz (1396.86-MHz K8-class
CPU) Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz (1396.86-MHz
K8-class CPU) Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz
(1396.86-MHz K8-class CPU) Intel(R) Core(TM) i3-2367M CPU @
1.40GHz (1396.86-MHz K8-class CPU) Intel(R) Core(TM) i3-2367M
CPU @ 1.40GHz (1396.86-MHz K8-class CPU)

dev.cpu.0.temperature: 60.0C
dev.cpu.1.temperature: 60.0C
dev.cpu.2.temperature: 56.0C
dev.cpu.3.temperature: 56.0C
hw.acpi.thermal.tz0.temperature: 65.1C

vmstat -s:

   618156 cpu context switches
    67434 device interrupts
    11717 software interrupts
   408902 traps
 53956889 system calls
       27 kernel threads created
     1727  fork() calls
      335 vfork() calls
        0 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
     3773 vnode pager pageins
    36603 vnode pager pages paged in
        1 vnode pager pageouts
        1 vnode pager pages paged out
        0 page daemon wakeups
    39220 pages examined by the page daemon
        0 clean page reclamation shortfalls
        0 pages reactivated by the page daemon
    75655 copy-on-write faults
      429 copy-on-write optimized faults
   214323 zero fill pages zeroed
        0 zero fill pages prezeroed
        2 intransit blocking page faults
   448906 total VM faults taken
     3540 page faults requiring I/O
        0 pages affected by kernel thread creation
    68956 pages affected by  fork()
    11877 pages affected by vfork()
        0 pages affected by rfork()
   479243 pages freed
        0 pages freed by daemon
   117279 pages freed by exiting processes
    52928 pages active
    34022 pages inactive
        0 pages in the laundry queue
   103525 pages wired down
  3862291 pages free
     4096 bytes per page
   226307 total name lookups
          cache hits (86% pos + 6% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

Boot time : 1587499634

procs     memory        page                    disks
faults        cpu0     cpu1     cpu2     cpu3 r b w     avm
fre  flt  re  pi  po    fr   sr ad0 pa0   in    sy    cs us sy
id us sy id us sy id us sy id 0 0 0 2129420 15449104  769   0
6   0   821   67   0   0  115 92235  1057  3  6 91  4  6 90  4
6 91  3  6 90

memory info:

real memory  = 17179869184 (16384 MB)
avail memory = 16495013888 (15730 MB)

last pid: 44946;  load averages:  0.23,  0.33,  0.24  up
0+00:09:45    20:16:59 40 processes:  1 running, 39 sleeping

Mem: 207M Active, 133M Inact, 405M Wired, 77M Buf, 15G Free
Swap: 48G Total, 48G Free

hw.physmem: 17053859840
hw.usermem: 16629661696
hw.realmem: 17179869184

             total       used       free     shared
buffers     cached Mem:      16210872     625800
15585072          0          0          0 Swap:
50331644          0   50331644

swapinfo:

Device          1K-blocks     Used    Avail Capacity
/dev/ada0s1b     50331644        0 50331644     0%

vmstat:

procs     memory        page                    disks
faults         cpu r b w     avm     fre  flt  re  pi  po
fr   sr ad0 pa0   in    sy    cs us sy id 0 0 0 2129484
15448752  773   0   6   0   824   67   0   0  116 92247  1060
4  6 90

Message replied to:

Date: Mon, 20 Apr 2020 22:17:35 +0200
From: "t.sefzick" <t.s...@gm...>
To: "John L. Males" <jl...@gm...>
Cc: eam...@gm..., gnu...@li...,
t.s...@gm... Subject: Re: [Gnuplot-info] columns with time
format as "y"

> The separator of your data is the comma, so
> 
> set datafile separator comma
> 
> splits your data into
> 
> <empty> date time value1 value2
> 
> Because of the empty first column you need to start
> with column 2:
> 
> set timefmt "%Y-%m-%d,%H:%M:%S"
> plot "datafile.dat" using 2:4
> 
> On Mon, Apr 20, 2020 at 08:00:02PM +0000, John L. Males wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Hello,
> >
> > As follow on to the OP question I have data format of the
> > form:
> >
> > ,2019-04-16,20:30:00,30522,5
> > ,2019-04-16,20:31:00,30523,6
> > ,2019-04-16,20:32:00,30524,7
> > ,2019-04-16,20:33:00,30525,8
> > ,2019-04-16,20:34:00,30526,9
> > ,2019-04-16,20:35:00,30527,10
> > ,2019-04-16,20:36:00,30528,11
> > ,2019-04-16,20:37:00,30529,12
> > ,2019-04-16,20:38:00,30530,13
> > ,2019-04-16,20:39:00,30531,14
> >
> > Yes, the data lines start with "," for reasons I will skip
> > other than to say this is the data by design.
> >
> > I had tried a:
> >
> > set timefmt ",%Y-%m-%d,%H:%M:%S"
> >
> > a few years ago and again a few months ago with no success.
> > Just an error message to effect of invalid time format in
> > data.  I have had to add extra external step to filter the
> > data.  The issue is the data is very large so it adds a
> > couple minutes to filter the data before I can input the
> > date to gnuplot.  I really would like to eliminate the data
> > filter step that eliminates the leading comma of the data
> > as this data is processed every 15 minutes for now as I
> > cannot filter the data every 5 minutes due to the added
> > filtering step that then uses:
> >
> > set timefmt "%Y-%m-%d,%H:%M:%S"
> >
> >
> > John L. Males
> > Toronto, Ontario
> > Canada
> > 20 April 2020 16:00 -0400 EDT
> >
> >
> > ================================================================
> >
> > 2020-04-20 19:45:51+0000-UTC Time: 1587411951 PC/System time
> >
> > 20 Apr 19:45:51 ntpdate[64207]: ntpdate 4.2.8p12-a (1)
> >
> > 20 Apr 19:46:06 ntpdate[65654]: step time server
> > 128.100.100.128 offset 0.003394 sec
> >
> > FreeBSD 11.3-STABLE FreeBSD 11.3-STABLE #0 r349903: Thu Jul
> > 11 16:13:47 UTC 2019
> > ro...@re...:/usr/obj/usr/src/sys/GENERIC
> >
> > (Work in progress alternative to Linux Kernel of its own
> > right, Debian, and
> >  other Linux based Kernel distributions determined.)
> >
> > Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz
> > Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz (1396.86-MHz
> > K8-class CPU) Intel(R) Core(TM) i3-2367M CPU @ 1.40GHz
> > (1396.86-MHz K8-class CPU) Intel(R) Core(TM) i3-2367M CPU @
> > 1.40GHz (1396.86-MHz K8-class CPU) Intel(R) Core(TM)
> > i3-2367M CPU @ 1.40GHz (1396.86-MHz K8-class CPU) Intel(R)
> > Core(TM) i3-2367M CPU @ 1.40GHz (1396.86-MHz K8-class CPU)
> >
> > dev.cpu.0.temperature: 73.0C
> > dev.cpu.1.temperature: 73.0C
> > dev.cpu.2.temperature: 69.0C
> > dev.cpu.3.temperature: 69.0C
> > hw.acpi.thermal.tz0.temperature: 75.1C
> >
> > vmstat -s:
> >
> >  83430749 cpu context switches
> >   1277504 device interrupts
> >    368458 software interrupts
> >  31639133 traps
> > 264899647 system calls
> >        27 kernel threads created
> >      1920  fork() calls
> >       596 vfork() calls
> >         0 rfork() calls
> >         0 swap pager pageins
> >         0 swap pager pages paged in
> >         0 swap pager pageouts
> >         0 swap pager pages paged out
> >      6321 vnode pager pageins
> >     73954 vnode pager pages paged in
> >       107 vnode pager pageouts
> >       107 vnode pager pages paged out
> >         0 page daemon wakeups
> >  12090933 pages examined by the page daemon
> >         0 clean page reclamation shortfalls
> >         0 pages reactivated by the page daemon
> >    113093 copy-on-write faults
> >       463 copy-on-write optimized faults
> >  28402363 zero fill pages zeroed
> >         0 zero fill pages prezeroed
> >        48 intransit blocking page faults
> >  32157102 total VM faults taken
> >      6075 page faults requiring I/O
> >         0 pages affected by kernel thread creation
> >     93134 pages affected by  fork()
> >     20968 pages affected by vfork()
> >         0 pages affected by rfork()
> >  29860181 pages freed
> >         0 pages freed by daemon
> >  14599509 pages freed by exiting processes
> >    344499 pages active
> >   1010164 pages inactive
> >     76679 pages in the laundry queue
> >    262762 pages wired down
> >   2358662 pages free
> >      4096 bytes per page
> >   4907338 total name lookups
> >           cache hits (96% pos + 1% neg) system 0%
> > per-directory deletions 0%, falsehits 0%, toolong 0%
> >
> > Boot time : 1587398332
> >
> > procs     memory        page                    disks
> > faults        cpu0     cpu1     cpu2     cpu3 r b w     avm
> > fre  flt  re  pi  po    fr   sr ad0 pa0   in    sy    cs us
> > sy id us sy id us sy id us sy id 0 0 0 30655696 9434572
> > 2359   0 0   0  2190  887   0   0   94 19431  6120 13  4 83
> > 13  4 83 13 4 83 13  4 83
> >
> > memory info:
> >
> > real memory  = 17179869184 (16384 MB)
> > avail memory = 16495013888 (15730 MB)
> >
> > last pid: 71600;  load averages:  0.53,  0.79,  0.81  up
> > 0+03:47:14    19:46:06 53 processes:  1 running, 52 sleeping
> >
> > Mem: 1346M Active, 3946M Inact, 300M Laundry, 1027M Wired,
> > 442M Buf, 9213M Free Swap: 48G Total, 48G Free
> >
> > hw.physmem: 17053859840
> > hw.usermem: 15977410560
> > hw.realmem: 17179869184
> >
> >              total       used       free     shared
> > buffers     cached Mem:      16210872    2735676
> > 13475196          0          0          0 Swap:
> > 50331644          0   50331644
> >
> > swapinfo:
> >
> > Device          1K-blocks     Used    Avail Capacity
> > /dev/ada0s1b     50331644        0 50331644     0%
> >
> > vmstat:
> >
> > procs     memory        page                    disks
> > faults         cpu r b w     avm     fre  flt  re  pi  po
> > fr   sr ad0 pa0   in    sy    cs us sy id 0 0 0 30655696
> > 9434576 2359   0   0   0  2190  887   0   0   94 19430  6120
> > 13  4 83
> >
> >
> > Message replied to:
> >
> > Date: Mon, 20 Apr 2020 12:00:40 -0700
> > From: Ethan Merritt <eam...@gm...>
> > To: gnu...@li...
> > Subject: Re: [Gnuplot-info] columns with time format as "y"
> >
> >
> > > On Monday, 20 April 2020 11:30:03 PDT co...@tr...
> > > wrote:
> > > >
> > > > Hello All,
> > > >
> > > > I need help with the following problem:
> > > >
> > > > Let's say my input data look like (columns delimited by
> > > > \t):
> > > >
> > > >    2019-04-16 20:30:00	30522	5
> > > >    2019-04-16 20:31:00	30523	6
> > > >    2019-04-16 20:32:00	30524	7
> > > >    2019-04-16 20:33:00	30525	8
> > > >    2019-04-16 20:34:00	30526	9
> > > >    2019-04-16 20:35:00	30527	10
> > > >    2019-04-16 20:36:00	30528	11
> > > >    2019-04-16 20:37:00	30529	12
> > > >    2019-04-16 20:38:00	30530	13
> > > >    2019-04-16 20:39:00	30531	14
> > > >
> > > > Column 1 & 2 (space separated) considered as date/time.
> > > >
> > > > Plotting Col. 3 or 4 against date/time works fine:
> > > >
> > > >    set xdata time
> > > >    set timefmt "%Y-%m-%d\t%H:%M:%S"
> > > >    set format x "%d.%m. %H:%M"
> > > >    plot 'my_file.txt' using 1:3 title "col 3" axis x1y1
> > > > with linespoints pt7 lc 4, '' using 1:4 title "col 4"
> > > > axis x1y1 with linespoints pt7 lc 7
> > > >
> > > > Special feature: Although columns 1 and 2 *together*
> > > > give the timestamp, only "1:..." is given for "using"
> > > > statement. This method is often found in examples.
> > > >
> > > > But now I would like - don't ask me why - to plot the
> > > > timestamp against column 3, i.e. column 3 as "x" and the
> > > > timestamp (columns 1 & 2) as "y".
> > > >
> > > > I was trying:
> > > >
> > > >    set ydata time "%Y-%m-%d\t%H:%M:%S"
> > > >    set timefmt "%Y-%m-%d\t%H:%M:%S"
> > > >    set format y "%d.%m. %H:%M"
> > > >    plot 'my_file.txt' using 3:1 title "col 3" axis x1y1
> > > > with linespoints pt7 lc 4
> > > >
> > > > Result: only date is used for plotting "y", time
> > > > ignored!
> > >
> > > Replace the tab with a space in timefmt:
> > >
> > >     set timefmt "%Y-%m-%d %H:%M:%S"
> > >
> > > The space character in the format will match any
> > > whitespace sequence, including <tab>.
> > >
> > > 	Ethan
> > >
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > gnuplot-info mailing list
> > > gnu...@li...
> > > Membership management via:
> > > https://lists.sourceforge.net/lists/listinfo/gnuplot-info
> > -----BEGIN PGP SIGNATURE-----
> >
> > iF0EARECAB0WIQQxRId2q5JPHFiozTr5X9dS0HpoEAUCXp3/QwAKCRD5X9dS0Hpo
> > EAx0AJ464SqTe0oieSwe8My/Yc6XCKHS7ACg2wPr2cEnr89Hg1SLbwB+LfpUVUE=
> > =czcZ
> > -----END PGP SIGNATURE-----
> >
> >
> > _______________________________________________
> > gnuplot-info mailing list
> > gnu...@li...
> > Membership management via:
> > https://lists.sourceforge.net/lists/listinfo/gnuplot-info
-----BEGIN PGP SIGNATURE-----

iF0EARECAB0WIQQxRId2q5JPHFiozTr5X9dS0HpoEAUCXp9YgAAKCRD5X9dS0Hpo
EIAXAJ9lX26pI9z+B+dAWYLNTDVfNl3XJwCePEYcozQnzh3X9na5UWxnLsRjfc4=
=OaJE
-----END PGP SIGNATURE-----

Re: [Gnuplot-info] columns with time format as "y"

A portable, multi-platform, command-line driven graphing utility

Re: [Gnuplot-info] columns with time format as "y"