#1812 Reading from binary data file starts to corrupt the read in data.

[Adrian]

Version 5.0 patchlevel 3 last modified 2016-02-21

I create a small binary file and it is ok, but when the binary file gets to a certain size, it goes all wonky. Here is a C++ binary file generator:

#include <limits> 
#include <iostream> 
#include <assert.h> 

template <typename T1, typename T2> 
void write(std::ostream& os, T1 x, T2 y) 
{ 
        using namespace std; 
        static int counter = 0; 
        assert(0 <= x); 
        assert(0 <= y); 
        assert(x < numeric_limits<double>::max()); 
        assert(y < numeric_limits<double>::max()); 
        double array[2] = { (double)x, (double)y }; 
        os.write((char*)array, sizeof(array)); 
        os.flush(); 
        ++counter; 
} 

int main() 
{ 
        using namespace std; 
        ofstream os; 
        os.open("data.dat"); 
        uint64_t y_value = 0; 
        for (uint64_t x_value = 0; x_value < 5000; ++x_value) 
        { 
                write(os, x_value, y_value); 
                if (x_value % 10 == 0) 
                { 
                        ++y_value; 
                } 
        } 
        return 0; 
} 

It just creates a stepping function, increasing 1 every 10. I then read the file using the command:

plot 'data.dat' binary format='%lf%lf' 0:1 

If I only allow 2052 points, it's fine, but when I get to 2053, I start getting some randomness happening. That last point becomes around (0, 2.45e201). The rest of the data hasn't been corupted at this point, though if I continue, it can get a lot worse.

Text data with a gnuplot line for reading in text data doesn't have this issue.

[Ethan Merritt]

Works fine here:

$ setenv CPPFLAGS "-std=gnu++11"
$ make make_data
$ ./make_data
$ ls -asl data.dat
80 -rw-r--r-- 1 merritt users 80000 Jun 10 23:33 data.dat
gnuplot_5.0.3
gnuplot> set term png
gnuplot> set output 'foo.png'
gnuplot> plot 'data.dat' using 0:1 binary format='%lf%lf' 

Output plot attached

[Adrian]

Ok, well. I got the latest Windows download. Maybe that makes a difference?

This is what I got:

[Adrian]

BTW, that isn't even the same range you plotted. o.O Something is very wrong there.

Here is the plot without the using keyword:

[Ethan Merritt]

I cannot tell from what you have shown so far whether the problem is in the data file creation or the plotting afterwards. Can you attach the data.dat file you created so I can see if gnuplot can read it here?

[Adrian]

Sure thing. Data file attached.

[Hans-Bernhard Broeker]

That file is not in binary format. It's plain text, plottable with

plot 'data.dat' u 1:2

[Ethan Merritt]

Right. That file seems to contain garbage.
Now I'm going to attach the file I created here using the C++ program you posted.
If you can read it in your gnuplot, then I think we have proved that your problem is on the C++ end rather than the gnuplot end.

[Adrian]

Hmmm. That's not good. That was build with VS2015. Good to know.

Thanks.

A

[Hans-Bernhard Broeker]

Hmmm... neither VS2015, nor Cygwin or MinGW64 GCC accepts that source here. The all reject the definition

ofstream os;

Looks like the source misses

#include <fstream>

And once that bug is fixed, VS2015 generates a program that replaces 0x0a by 0x0d 0x0a. I.e. the output stream is in text mode, so it believes it has to generate DOS-style line endings.

[Hans-Bernhard Broeker]

The root cause is that the program fails to set the output file to binary mode.

        os.open("data.dat", ios::binary); 

[Adrian]

Yes, I figured out that last night. It's been a while since I generated a binary file. :/

Thanks.

[Tatsuro MATSUOKA]

For mingw 64,

#include <inttypes.h>

is also required for "uint64_t".

        os.open("data_mgw.dat", ios::binary); 

ios::binary is always required for C++ for windows for binary io like "wb" for C on windows.
(On Cygwin ios::binary is not required because it is posix compatible.)

Ethan.
This is not a bug of gnuplot.
Please close it