#58 add partial read support to readflex

Chris Marshall

The current readflex() implementation is pretty much
all or nothing as far as reading chunks of data. This
makes IO processing fragile since hitting the end of file
causes readflex to barf as shown in this pdl2 session:

pdl> $data = sequence(8,15)

pdl> writeflex('data_8x15.dat', $data)

pdl> ?vars
PDL variables in package main::

Name Type Dimension Flow State Mem
$data Double D [8,15] P 0.94KB
$Pi Double D [] P 0.01KB

pdl> $ndata = readflex('data_8x15.dat', [ { Type=>'double', NDims=>2, Dims=>[8,16] } ])
Runtime error: Couldn't read enough data from 'data_8x15.dat' at /cygdrive/f/perl/usr_lib/i686-cygwin-thread-multi-64int/PDL/Core.pm line 196
PDL::Core::barf('Couldn\'t read enough data from \'data_8x15.dat\'') called at /cygdrive/f/perl/usr_lib/i686-cygwin-thread-multi-64int/PDL/IO/FlexRaw.pm line 467
PDL::IO::FlexRaw::readchunk('FileHandle=GLOB(0x3969348)', 'PDL=SCALAR(0x3967d78)', 1024, 'data_8x15.dat') called at /cygdrive/f/perl/usr_lib/i686-cygwin-thread-multi-64int/PDL/IO/FlexRaw.pm line 563

After the error, no data has been read. It would be more
convenient/useful if readflex returned status indicated
whether a read was complete or not and if not, that the
partial data correctly read (like the the first 15 rows of
input data) is available for use.

This feature would help to address the need for record
based IO in readflex as one could just read a large
number of records and then make use of the ones that
successfully read in...


  • Chris Marshall
    Chris Marshall

    It would be useful just to have the failure message give information on how much data was successfully read in as a PDL shape.