Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

SFK Addcr error: out of memory

Meseret
2012-11-11
2013-01-06
  • Meseret
    Meseret
    2012-11-11

    Hello,

    SFK addcr have crashed  with large input files due to out of memory on my windows 2003 server. Any fix and/or thoughts please. I am using the latest realse of SFK.

    Thanks

     
  • stahlworks
    stahlworks
    2012-11-11

    SFK addcr cannot handle large input files, because the whole text is loaded to memory to allow easy overwriting of the input file. Please describe exactly what you mean by "crash". It should just shown an error, but then continue on the next file.
    - do you see some kind of popup instead?
    - what kind of command syntax are you using?
    - do you want to convert the large files as well?

     
  • Meseret
    Meseret
    2012-11-11

    Thanks for your Reply!

    I'm running a cmd as:

    C:\convert\sfk166.exe addcr inputfile.dat

    Where inputfile.dat is a text file of size 3GB

    The error message I got is: "error out of memory"

    It works geat when the input file size is 7MB

    Thanks for your help

     
  • Meseret
    Meseret
    2012-11-11

    What the input file size limit for SFK addcr?

    Any suggestion as a work around to process a large input file?
    I have a  3GB production file that needs to procces using addcr.

    Appreciate your help on this matter!

     
  • stahlworks
    stahlworks
    2012-11-11

    The file size limit is 2 GBytes approximately. Three alternatives:

    1. Use the freeware HxD Hex Editor and replace all 0d by 0d0a
       using data types: hex-values. I just tried it and the tool says
       it would take about halve an hour approx. Not sure if HxD supports
       replacement from a batch file.

    2. with SFK XE (commercial), it can be done with a command like:

    sfk replace -usetmp -showtmp inputfile.dat -binary /0d// /0a/0d0a/ -yes
    

    Explanation: in case that there are unexpected CR (0x0d) characters, they are
    stripped, and then all LF (0x0a) endings are replaced by CRLF (0x0d0a).
    Option -usetmp allows the use of a temporary file, which is required
    to avoid the main memory limit. Should take 5 to 10 minutes on a 3 GB text.

    3. write and compile a simple C program that reads the file line by line,
    writing another output file line by line. type "sfk samp cpp" for details.

     
  • Meseret
    Meseret
    2012-11-11

    Thank you so much, I am working on option 1 & 3 to see if I can get it work.

    What would be the procedure to get the SFK XE (commercial) ? I just want to get ready as an alterantive on this.

     
  • stahlworks
    stahlworks
    2012-11-12

    Thinking about it, another alternative might be

    sfk filter inputfile.dat >outputfile.txt
    

    which however requires inputfile not to contain any binaries, and lines larger than 4000 characters will be split.

    XE is available under http://stahlworks.com/sfkxe.html