SFK Addcr error: out of memory

  • Meseret

    Meseret - 2012-11-11


    SFK addcr have crashed  with large input files due to out of memory on my windows 2003 server. Any fix and/or thoughts please. I am using the latest realse of SFK.


  • stahlworks

    stahlworks - 2012-11-11

    SFK addcr cannot handle large input files, because the whole text is loaded to memory to allow easy overwriting of the input file. Please describe exactly what you mean by "crash". It should just shown an error, but then continue on the next file.
    - do you see some kind of popup instead?
    - what kind of command syntax are you using?
    - do you want to convert the large files as well?

  • Meseret

    Meseret - 2012-11-11

    Thanks for your Reply!

    I'm running a cmd as:

    C:\convert\sfk166.exe addcr inputfile.dat

    Where inputfile.dat is a text file of size 3GB

    The error message I got is: "error out of memory"

    It works geat when the input file size is 7MB

    Thanks for your help

  • Meseret

    Meseret - 2012-11-11

    What the input file size limit for SFK addcr?

    Any suggestion as a work around to process a large input file?
    I have a  3GB production file that needs to procces using addcr.

    Appreciate your help on this matter!

  • stahlworks

    stahlworks - 2012-11-11

    The file size limit is 2 GBytes approximately. Three alternatives:

    1. Use the freeware HxD Hex Editor and replace all 0d by 0d0a
       using data types: hex-values. I just tried it and the tool says
       it would take about halve an hour approx. Not sure if HxD supports
       replacement from a batch file.

    2. with SFK XE (commercial), it can be done with a command like:

    sfk replace -usetmp -showtmp inputfile.dat -binary /0d// /0a/0d0a/ -yes

    Explanation: in case that there are unexpected CR (0x0d) characters, they are
    stripped, and then all LF (0x0a) endings are replaced by CRLF (0x0d0a).
    Option -usetmp allows the use of a temporary file, which is required
    to avoid the main memory limit. Should take 5 to 10 minutes on a 3 GB text.

    3. write and compile a simple C program that reads the file line by line,
    writing another output file line by line. type "sfk samp cpp" for details.

  • Meseret

    Meseret - 2012-11-11

    Thank you so much, I am working on option 1 & 3 to see if I can get it work.

    What would be the procedure to get the SFK XE (commercial) ? I just want to get ready as an alterantive on this.

  • stahlworks

    stahlworks - 2012-11-12

    Thinking about it, another alternative might be

    sfk filter inputfile.dat >outputfile.txt

    which however requires inputfile not to contain any binaries, and lines larger than 4000 characters will be split.

    XE is available under http://stahlworks.com/sfkxe.html


Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.

No, thanks