gnuplot-info

[Gnuplot-info] read file to datablock

[theozh <th...@gm...>]

Hi, I would like to read a file (as is) into a datablock.
I basically found two ways to do this. See code below.

First question:
My concern is that in Method1 the file has to be read/accessed (row)-times from disk whereas Method2 only once. If I have a file with several thousands or million rows on a very slow network connection this probably would take rather long, correct? I currently cannot test.
Can anybody comment on this?

Second question:
When reading the file via Method2 the datablock contains a space at every beginning of the line.
Can anybody explain why and how to avoid? Bug or feature?

Assuming the file "Test.dat" looks for example like the following:

# Test.dat
# Some test data
# more comments
1 2 3
4 5 6
7 8 9

Here is the code:

### Read file to datablock
reset session
FILE = "Test.dat"
set datafile separator "\n"
set datafile commentschar ""

# Method1
stats FILE u 0 nooutput
set print $Data1
set table $Dummy
do for [row=0:STATS_records-1] {
    plot FILE u (a=stringcolumn(1),a) every ::row::row with table
    print a
}
unset table
set print
print $Data1

# Method2
set table $Data2
    plot FILE u (stringcolumn(1)) with table
unset table
print $Data2

### end of code

Re: [Gnuplot-info] read file to datablock

[theozh <th...@gm...>]

Thanks, in priciple a nice idea.
But probably works only under Linux?!
I guess 'cat' doesn't exists in a simple Win7 system.
I tried with 'type' instead, but no success.


On Saturday, 10 November 2018 07:05:46 theozh wrote:
> Hi, I would like to read a file (as is) into a datablock.
> I basically found two ways to do this. See code below.

I would do something very different.

set print "head"
print "$Data1 << EOD"
set print "tail"
print "EOD"
unset print

load "< cat head Test.dat tail"
print $Data1

		Ethan

Re: [Gnuplot-info] read file to datablock

[Ethan A M. <eam...@gm...>]

On Saturday, 10 November 2018 19:02:12 theozh wrote:
> Thanks, in priciple a nice idea.
> But probably works only under Linux?!
> I guess 'cat' doesn't exists in a simple Win7 system.
> I tried with 'type' instead, but no success.

I can't help with Windows.
A quick bit of googling suggests that the equivalent of
"cat" in the Windows PowerShell is "get-content".  But I
do not know whether that works from inside gnuplot.

	Ethan


> 
> On Saturday, 10 November 2018 07:05:46 theozh wrote:
> > Hi, I would like to read a file (as is) into a datablock.
> > I basically found two ways to do this. See code below.
> 
> I would do something very different.
> 
> set print "head"
> print "$Data1 << EOD"
> set print "tail"
> print "EOD"
> unset print
> 
> load "< cat head Test.dat tail"
> print $Data1
> 
> 		Ethan

Re: [Gnuplot-info] read file to datablock

[theozh <th...@gm...>]

Thank you again!
I am not familiar with Win Powershell... would take a while to figure out.

But do you have an idea why Method2 gets this space at the beginning of the line?

### Read file to datablock
reset session
FILE = "Test.dat"
set datafile separator "\n"
set datafile commentschar ""

# Method2
set table $Data2
    plot FILE u (stringcolumn(1)) with table
unset table
print $Data2


Am 10.11.2018 um 19:30 schrieb Ethan A Merritt:
> On Saturday, 10 November 2018 19:02:12 theozh wrote:
>> Thanks, in priciple a nice idea.
>> But probably works only under Linux?!
>> I guess 'cat' doesn't exists in a simple Win7 system.
>> I tried with 'type' instead, but no success.
> 
> I can't help with Windows.
> A quick bit of googling suggests that the equivalent of
> "cat" in the Windows PowerShell is "get-content".  But I
> do not know whether that works from inside gnuplot.
> 
> 	Ethan
> 

Re: [Gnuplot-info] read file to datablock

[Ethan A M. <eam...@gm...>]

On Saturday, 10 November 2018 20:14:53 theozh wrote:
> Thank you again!
> I am not familiar with Win Powershell... would take a while to figure out.
> 
> But do you have an idea why Method2 gets this space at the beginning of the line?

"with table" places a " " character at the front of every field.
This insures that the fields can be distinguish even if you mess
up in specifying a field separator.

	Ethan


> 
> ### Read file to datablock
> reset session
> FILE = "Test.dat"
> set datafile separator "\n"
> set datafile commentschar ""
> 
> # Method2
> set table $Data2
>     plot FILE u (stringcolumn(1)) with table
> unset table
> print $Data2
> 
> 
> Am 10.11.2018 um 19:30 schrieb Ethan A Merritt:
> > On Saturday, 10 November 2018 19:02:12 theozh wrote:
> >> Thanks, in priciple a nice idea.
> >> But probably works only under Linux?!
> >> I guess 'cat' doesn't exists in a simple Win7 system.
> >> I tried with 'type' instead, but no success.
> > 
> > I can't help with Windows.
> > A quick bit of googling suggests that the equivalent of
> > "cat" in the Windows PowerShell is "get-content".  But I
> > do not know whether that works from inside gnuplot.
> > 
> > 	Ethan
> > 
> 
> 
> 
> _______________________________________________
> gnuplot-info mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-info

Re: [Gnuplot-info] read file to datablock

[theozh <th...@gm...>]

Hi Ethan,
based on your suggestion... the following seems to work under Windows.
In case anybody on Windows might be interested.
Although pretty ugly and cumbersome...
1. writing 2 files to disk
2. reading and concatenating 3 files from disk
3. writing another file to disk
4. and reading this file from disk

I am wondering whether this will work properly with extremely large files on extremely slow network drives?

Thanks,
Theo.

### Read file "as is" to datablock under Windows
reset session
FILE = "Test.dat"

# Method3
set print "head"
print "$Data3 << EOD"
set print "tail"
print "EOD"
set print

system "type head ".FILE." tail > Temp.dat"
load "Temp.dat"
print $Data3
### end of code

Re: [Gnuplot-info] read file to datablock

[Ethan A M. <eam...@gm...>]

On Monday, 12 November 2018 20:41:56 theozh wrote:
> Hi Ethan,
> based on your suggestion... the following seems to work under Windows.
> In case anybody on Windows might be interested.
> Although pretty ugly and cumbersome...
> 1. writing 2 files to disk
> 2. reading and concatenating 3 files from disk
> 3. writing another file to disk
> 4. and reading this file from disk
> 
> I am wondering whether this will work properly with extremely large files on extremely slow network drives?

Copying the original data into new file first and then reading it in a
separate step is strictly worse than directing it through a pipe.

So no, I would not recommend the file-based Method3 you show below.

Hopefully someone more familiar than I am with MSWin can help with
the commands needed to direct the catenation operation through a pipe.

	Ethan



> 
> Thanks,
> Theo.
> 
> ### Read file "as is" to datablock under Windows
> reset session
> FILE = "Test.dat"
> 
> # Method3
> set print "head"
> print "$Data3 << EOD"
> set print "tail"
> print "EOD"
> set print
> 
> system "type head ".FILE." tail > Temp.dat"
> load "Temp.dat"
> print $Data3
> ### end of code
> 
> 
> 
> _______________________________________________
> gnuplot-info mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-info

Re: [Gnuplot-info] read file to datablock

[theozh <th...@gm...>]

Hi Ethan,
I agree, that this was not a good solution.

Ok, here is another attempt, however, still not so nice with creating the temporary second file 'head' on disk.
It seems like to work even without the end-of-data-marker (i.e. your 'tail' file containing 'EOD')

Would it a big thing to implement such a 'file-to-datablock-import' in gnuplot directly? Or not worth it?

Theo.

### Read file "as is" to datablock under Windows
reset session
FILE = "Test.dat"

# Method3
set print "head"
print "$Data3 <<EOD"
set print

load "<type head & type ".FILE
print $Data3
### end of code

Re: [Gnuplot-info] read file to datablock

[theozh <th...@gm...>]

ok, I think here we go :-)
You concatenate the text '$Data <<EOD' and the content of the file using the commands 'echo' and 'type' at which the character '<' has to be escaped by '^'.
So easy if you know how.

The advantages over the earlier mentioned other methods:
- empty lines are preserved
- no additional space character at the beginnig of the line
- no loading of any extra assistant files


### Read file "as is" to datablock under Windows
reset session
FILE = "Test.dat"

load "< echo $Data ^<^<EOD & type ".FILE
print "$Data"
print $Data
### end of code

Re: [Gnuplot-info] read file to datablock

[Ethan A M. <eam...@gm...>]

On Tuesday, 13 November 2018 22:02:42 theozh wrote:
> ok, I think here we go :-)
> You concatenate the text '$Data <<EOD' and the content of the file using the commands 'echo' and 'type' at which the character '<' has to be escaped by '^'.
> So easy if you know how.
> 
> The advantages over the earlier mentioned other methods:
> - empty lines are preserved
> - no additional space character at the beginnig of the line
> - no loading of any extra assistant files
> 
> 
> ### Read file "as is" to datablock under Windows
> reset session
> FILE = "Test.dat"
> 
> load "< echo $Data ^<^<EOD & type ".FILE
> print "$Data"
> print $Data
> ### end of code

Do you understand why the < characters need to be escaped?
It seems worth documenting this technique somewhere and it would
be nice to explain when this is or isn't necessary.
Would it would to put in another set of quotes instead?
   load "< echo '$Data <<EOD' & type ".FILE

Glad you got it working.

	Ethan

Re: [Gnuplot-info] read file to datablock

[theozh <th...@gm...>]

> Do you understand why the < characters need to be escaped?
> It seems worth documenting this technique somewhere and it would
> be nice to explain when this is or isn't necessary.
> Would it would to put in another set of quotes instead?
>    load "< echo '$Data <<EOD' & type ".FILE
> 

For some reason this does not work:
load "< echo '$Data <<EOD' & type ".FILE
I don't know why.

The '<' characters need to be escaped because otherwise they would be interpreted as redirection.
I found this information here:
https://ss64.com/nt/syntax-redirection.html

I haven't yet succeeded to put "EOD" at the end if the very last character of the datafile is not "\n". But apparently there is no need to put EOD at the end because it seems to work without.

Yes, it would be nice if this will be documented somewhere "officially" :-)
And maybe a more advanced Win-user can comment on this.

Re: [Gnuplot-info] read file to datablock

[theozh <th...@gm...>]

just for curiosity...
Do you know if there is also a way under Linux without creating and loading two helper files?
Do you actually need the "EOD" at the end under Linux?

Re: [Gnuplot-info] read file to datablock

[theozh <th...@gm...>]

Thanks again, Ethan!
just for completeness, such that the world can read your valuable comments (see below).
Theo.
> 
>     just for curiosity...
>     Do you know if there is also a way under Linux without creating and loading two helper files?
> 
> 
> I'm sure there are several, but I thought the shell syntax would not be applicable on MSWin. 
> Here is one way
> 
> load '< echo "\$DATA << EOD" & cat file.dat'
>  
> 
>     Do you actually need the "EOD" at the end under Linux?
> 
> 
> You do if it is being read from a file or input stream that may contain other 
> commands after the data block.  The EOD signals the end of the data block and the
> start of regular commands. But in your specific case there is nothing after the data
> and the true end-of-file serves the same purpose.  So the answer is "no" you do not
> need the EOD if the load operation is only filling a datablock.
> 
>               Ethan
>