Large Files, time and memory

2006-03-03
2012-11-13
  • I opened a text file about 150M in size.
    Notepad++ launched, and then proceeded to lock up my system while it churned away.  Memory usage of the application went to 400M, then dropped to 15M is slowly climbed, while fluctuation back to 340M where is finally completed.  the entire process took 6 minutes.

    Will large file processing ever be enhanced, or should I consider a certain file size the "max"?

    Thanks!

     
    • NOT QUITE !

      Having been hit by this bug while synchronizing two computers, I found that the file size is irrelavant.  What is apparently happening is a work buffer is overflowing and changing one byte of the CURRENT file it is handling.  The file size can be 1K or 127M, only the size of the work buffer matters.  This bug has hit files  that are 1k, 25k, 1.6M, and even 6+G is size yet left alone files that are the same size ranges and even hit different files of the same directory download.  So you may think it is only files of 127M size, but it is actually a work buffer overflowing and messing with the current file of ANY size at the time the overflow occures!

      It took almost a month to synchronize over 20G of files on the two computers because of this bug.  I had to copy from one to the other, delete files that were the same, update the old files (old &/ new), place the files into the correct directories (was working with the files while synchronizing), then repeat in the opposite direction until ALL files & directories were identical.  (I like to do things the hard way to learn the pitfalls, then devise a simpler/better way (especially when I haven't found programs that do things MY WAY or close to it).)

       
    • -
      -
      2006-03-04

      Would it be possible to add a warning if a file is over X megs, giving the user the option to load or cancel. I have also had problems when I opened huge log files and everything freezes up.

       
    • Careful !
      there is a big bug that puts rubish in files larger than 127KB:
      http://sourceforge.net/tracker/index.php?func=detail&aid=1354429&group_id=95717&atid=612382

       
    • Also when I try to open a very large file ( .txt ) in my case with more than 100 000 lines. The line number margin work not good. I can see number margin restart to 1.

       
    • Having looked at the source and found it to use edit methods/tasks that I saw in use 25 years ago in 8-bit editers, the delay on large files can be improved by updating the scintillia core.

      One of the simplest tests I do of an editor is the replace command.  N++ uses
      1. find target
      2. delete target (move thru end of buffer)
      3. make room for new (move thru end of buffer)
      4. write new

      A long time ago I modified such a replace to do
      1. sizedif = sizeof(new) - sizeof(target)
      2. find target
      3. if sizedif < 0 delete sizedif
      4. if sizedif > 0 make room sizedif
      5. write new

      Notice that the old way requires two rewrites of memory all the way to the end of the buffer for EACH found target.  While my way ONLY rewrites memory ONCE for target & new of different sizes and NEVER for same sized.

      The old editor that I modified took over five minutes to do a simple replace in a 350k file and only a minute afterwards.  By the way, the N++ test that I ran took over eight hours to do one 5-letter replace 650,000 times in a single 140M file (possibly 2 hours if modified).
       

       
    • Eric
      Eric
      2006-04-08

      Careful !
      there is a big bug that puts rubish in files larger than 127KB:
      http://sourceforge.net/tracker/index.php?func=detail&;aid=1354429&group_id=95717&atid=612382