#2388 missing precision with timecolumn() together with table

[theozh]

When plotting timedate data into a table precision gets lost.
Let's call this a bug or at least an unnecessary limitation...
I don't see a reason why timedata plotted into a table should be limited to 4 digits, instead of covering second or even millisecond precision.

Code:

### missing precision with timecolumn() together with table
reset session

# create some random test data
myTimeFmt = "%d.%m.%Y %H:%M:%S"
set print $Data
    do for [i=0:20] {
        print sprintf("%s %.3f",strftime(myTimeFmt,time(0)+i*1200),rand(0))
    }
set print

set table $TimeTable
    plot $Data u (timecolumn(1,myTimeFmt)):3 w table
unset table
print $TimeTable

set format x "%H:%M" timedate
plot $Data u (timecolumn(1,myTimeFmt)):3 w lp pt 7 ti "Original data", \
     $TimeTable u 1:2 w lp pt 6 lc "red" ti "Data from table"
### end of code

Result:

 1.60948e+09     0.72
 1.60948e+09     0.539
 1.60948e+09     0.206
 1.60948e+09     0.797
 1.60948e+09     0.231
 1.60948e+09     0.844
 1.60949e+09     0.2
 1.60949e+09     0.008
 1.60949e+09     0.209
 1.60949e+09     0.857
 1.60949e+09     0.518
 1.60949e+09     0.471
 1.60949e+09     0.01
 1.60949e+09     0.909
 1.60949e+09     0.62
 1.6095e+09      0.029
 1.6095e+09      0.039
 1.6095e+09      0.422
 1.6095e+09      0.927
 1.6095e+09      0.786
 1.6095e+09      0.243

[theozh]

the following is also limited in precision, it is basically identical to what timecolumn() does.

plot $Data u (strptime(myTimeFmt,strcol(1)." ".strcol(2))):3 w table

[Ethan Merritt]

From "help with table"

Numerical data is always written with format %g. If you want some other format
use sprintf or gprintf to create a formatted string.

plot file using ("File 1"):1:2:3 with table
plot file using (sprintf("%4.2f",$1)) : (sprintf("%4.2f",$3)) with table

In your test case that would mean replacing

set table $TimeTable
    plot $Data u (timecolumn(1,myTimeFmt)):3 w table
unset table

with

set table $TimeTable
    plot $Data u (sprintf("%.3f",timecolumn(1,myTimeFmt))):3 w table
unset table

[theozh]

Sorry, my fault for not having checked help with table. Besides this, I thought standard precision for %g was 6 digits. So, no bug at all. Happy New Year!

[theozh]

Now, I nevertheless have an issue with this:

How would I use this together with smooth freq, e.g. for creating histograms with a bin size of a few hours or minutes or even seconds?

Variation 1:

set table $TimeTable
    plot $Data u (timecolumn(1,myTimeFmt))):(1) smooth freq
unset table

Variation 2: (using sprintf())

set table $TimeTable
    plot $Data u (sprintf("%.3f",timecolumn(1,myTimeFmt))):(1) smooth freq 
unset table

Variation 3: (using sprintf() and with table)

set table $TimeTable
    plot $Data u (sprintf("%.3f",timecolumn(1,myTimeFmt))):(1) smooth freq w table
unset table

Neither Variation 1, nor Variation 2, nor Variation 3 will do the job.
Variation 1 does not work because of missing precision and
Variation 2 does not work because smooth freq will not work with string values and
Variation 3 does not work because smooth freq does not work together with with table.

Can I maybe set the default precision somewhere to 9 digits?
Well, I could think of some workaround to subtract the initial time...
but have I overlooked something?

[Ethan Merritt]

I think I am failing to understand what plot you are trying to generate. It looks to me that variation 1 and variation 2 both produce the expected result, although variation 2 does require a cast from string to real:

 plot $Data u (real(sprintf("%.3f",timecolumn(1,myTimeFmt)))):(1) smooth freq w table

If the intent is to add 3 decimal places of precision to the generated time data, maybe what you want is:

myTimeFmt3 = "%d.%m.%Y %H:%M:%.3S"
set print $Data
    do for [i=0:20] {
        print sprintf("%s %.3f",strftime(myTimeFmt3,time(0.)+i*1200),rand(0))
    }
set print

[theozh]

Sorry, for not providing a clearer example. Let me try again.
I want a histogram of time data in a table. (I guess your first example above will not work, see Example A below).

Code:

### How to get a time histogram into a table?
reset session

$Data <<EOD
2021-02-10 00:00:01
2021-02-10 00:00:01
2021-02-10 00:00:01
2021-02-10 00:00:02
2021-02-10 00:00:02
2021-02-10 00:00:03
2021-02-10 00:00:04
2021-02-10 00:00:04
2021-02-10 00:00:04
2021-02-10 00:00:04
2021-02-10 00:00:04
EOD


# this will give a graph with bars heights: 3, 2, 1 and 5
plot $Data u (timecolumn(1,"%Y-%m-%d %H:%M:%S")):(1) smooth freq w boxes

# Version A: smooth freq w table
set table $Histogram
    plot $Data u (timecolumn(1,"%Y-%m-%d %H:%M:%S")):(1) smooth freq w table
unset table
print $Histogram

# Version B: smooth freq without table and formatting
set table $Histogram
    plot $Data u (strftime("%Y-%m-%d %H:%M:%S",timecolumn(1,"%Y-%m-%d %H:%M:%S"))):(1) smooth freq w table
unset table
print $Histogram

# Version C: smooth freq without table and date formatting
set table $Histogram
    # will give an error!
    # plot $Data u (strftime("%Y-%m-%d %H:%M:%S",timecolumn(1,"%Y-%m-%d %H:%M:%S"))):(1) smooth freq
unset table

# Version D: smooth freq without table
set table $Histogram
    plot $Data u (timecolumn(1,"%Y-%m-%d %H:%M:%S")):(1) smooth freq
unset table
print $Histogram

# Version E: smooth freq without table with set format x
set table $Histogram
    set format x "%.0f"
    plot $Data u (timecolumn(1,"%Y-%m-%d %H:%M:%S")):(1) smooth freq
unset table
print $Histogram

# Version F: smooth freq without table with set format timedate
set table $Histogram
    set format x "%Y-%m-%d %H:%M:%S" timedate
    plot $Data u (timecolumn(1,"%Y-%m-%d %H:%M:%S")):(1) smooth freq
unset table
print $Histogram

### end of code

Results:

Version A: smooth and with table do not work together

 1.61292e+09     1      
 1.61292e+09     1      
 1.61292e+09     1      
 1.61292e+09     1      
 1.61292e+09     1      
 1.61292e+09     1      
 1.61292e+09     1      
 1.61292e+09     1      
 1.61292e+09     1      
 1.61292e+09     1      
 1.61292e+09     1  

Version B: smooth and with table do not work together

 2021-02-10 00:00:01     1      
 2021-02-10 00:00:01     1      
 2021-02-10 00:00:01     1      
 2021-02-10 00:00:02     1      
 2021-02-10 00:00:02     1      
 2021-02-10 00:00:03     1      
 2021-02-10 00:00:04     1      
 2021-02-10 00:00:04     1      
 2021-02-10 00:00:04     1      
 2021-02-10 00:00:04     1      
 2021-02-10 00:00:04     1      

Version C: smooth does not work together with time format

Error: ... x range is invalid

Version D: all values =1.61292e+09 (because default formatting style is %g, with 6 digits)

# Curve 0 of 1, 5 points
# Curve title: "$Data u (timecolumn(1,"%Y-%m-%d %H:%M:%S")):(1)"
# x y type
 1.61292e+09  3  i
 1.61292e+09  2  i
 1.61292e+09  1  i
 1.61292e+09  5  i

Version E: explicit set format x "%.0f"

# Curve 0 of 1, 5 points
# Curve title: "$Data u (timecolumn(1,"%Y-%m-%d %H:%M:%S")):(1)"
# x y type
1612915201  3  i
1612915202  2  i
1612915203  1  i
1612915204  5  i

Version F: explicit set format x "%Y-%m-%d %H:%M:%S" timedate

# Curve 0 of 1, 5 points
# Curve title: "$Data u (timecolumn(1,"%Y-%m-%d %H:%M:%S")):(1)"
# x y type
"2021-02-10 00:00:01"  3  i
"2021-02-10 00:00:02"  2  i
"2021-02-10 00:00:03"  1  i
"2021-02-10 00:00:04"  5  i

Conclusion:

So, I was missing to simply set the desired format in the table environment beforehand via set format x .... But I cannot not set it within the plot command itself. No bug at all, just a misunderstanding. And the fact that smooth does not work together with with table is most likely an intended behaviour?!

[Ethan Merritt]

I would use

   myfmt = "%Y-%m-%d %H:%M:%S"
   set xtics time format myfmt
   plot $Data u (timecolumn(1,myfmt)):(1) smooth freq w impulse lw 6
   set table $Histogram
   replot
   unset table

The intent of with table is to capture the input processing from the using specifier. All other plot properties are ignored, including smooth, variable, axes, etc.