Partial line reading (issue + solution)

2010-10-07
2013-05-28
  • Nobody/Anonymous

    Currently we're using LiveGraph to monitor a CSV file that is being created at the same time.
    The CSV file contains a lot of columns (~80).
    Perhaps this is the reason that sometimes the updates fail in case the horizontal axis is set to the values of one of the dataseries.
    This then corrupts the view, because the X-value of all plots jumps to zero and back.

    I solved this by the following  code below (I know it's not beautiful, but it works for me…).
    NOTE: This will cause a hangup in case some of the data lines contain less data than the number of header rows.

    The cause of this issue lies in the fact that 'readline()' returns a partial line in case the reader reaches the end of the file.
    I noticed this by writing each line read to a log file.
    It then showed that lines were read partially in some cases.

    // Add a class variable containing the number of items in the header row:
    private int numHeaderTokens = 0;

    In method "readFromStream":
    Substitute
    if (line.length() > 0) {
        processLine(line);
        linesRead++;
    }

    By
    if (line.length() > 0) {
    try
    {
        processLine(line);
        linesRead++;

      in.mark(1000);
    }
    catch (DataFormatException de)
    {
    in.reset();
    }
    }
    else
    {
    in.mark(1000);
    }

    In method "processSeriesLabelsLine" store the number of items in the header row:
    Insert line
    numHeaderTokens = tok.getTokens().size();
    Before line
    notifyLabelsSet(Collections.unmodifiableList(tok.getTokens()));

    In method "processDataLine" only process the data line if sufficient items are included:
    Insert lines
    int numDataTokens = tok.getTokens().size();
    if (numHeaderTokens >numDataTokens)
    {
    // Line partially read -> throw format exception.
    throw new DataFormatException("insufficient data elements");
    }

    Before line
    notifyDataLineRead(tok.getTokens(), nextDatasetFileIndex++);

     
  • Anonymous - 2011-02-16

    Hello,
    Thank you for this post. Your explanation makes sense.  I have files that get long, but only have 5 columns.
    I applied your changes, and the graph still gets corrupted when the cache is used, but not with zero values, but with spikes.

    Can you explain or provide code as to how you logged the data read to a file?  I'd like to do this to help me debug. Thank you.

     
  • Nobody/Anonymous

    In order to log the data that was read,
    I used the following lines (line numbers after inclusion of new code lines):

    Line 4 (inserted after "import … BufferedReader"):
    import java.io.FileWriter;

    Line 88 (inserted before "…. numHeaderTokens…."):
    private FileWriter loggingFile = null;

    Lines 134-140 (at end of procedure "public DataStreamReader(InputStream is) "):
       try {
       this.loggingFile = new FileWriter("C:\\temp\\parsed.txt");
       }
       catch (IOException e)
       {
        // print "This is not going OK";
       }

    Lines 213/214 (start of 'try' block, before 'processLine(line);'):
           loggingFile.write(line);
           loggingFile.write("\r\n");

    Lines 223-225 (start of 'catch' block, before the 'in.reset();'):
           loggingFile.write("format Exception:\r\n");
           loggingFile.write(line);
           loggingFile.write("\r\n");

    Note that line numbers are only correct after addition of the indicated changes above.
    However, the context information should give sufficient hint.

     
  • Nobody/Anonymous

    Karl, … needless to add, but if you find out the issue, please post the solution as well…

     
  • Nobody/Anonymous

    i'm fairly new to java and i need some help. so once i edited the DataStreamReader class in the jar file, how can i re-compress it into a jar file so I can run it with the added code? How can i do that in the software that i'm using (NetBeans)? Because when i try re-zipping it and then changing the .zip to .jar it does not work. Thanks for the help!

     
  • Nobody/Anonymous

    The real problem here is that LiveGraph is reading incomplete lines. It should not process any data line until it finds an end-of-line marker for that line. Until this is fixed, the solution is to set up the writing application to perform line-buffered output, in which the system automatically flushes data to the output file for each line that is output, and to make sure that the application writes complete lines all at once, perhaps by batching up all the data to be written into a single write() call. If line-buffering can't be set explicitly, you may be able to fake it by following each write() call, which writes a complete line, with a flush() call, which forces the buffered data to be written to the file system.

     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks