Improvement/Support of Big File Handling
Notepad++ project is moving to GitHub:
Brought to you by:
donho
Menu
▾
▴
Hi Notepad++ folk,
we had much blaming about notepad++ isn´t able to show and open files with xx mb. My tests show it´s critical from ~60mb up. Don didn´t write much to this issue which could have the reason that the kind of these messages was only blaming without helping anyone.
I was thinking of writing a plugin but then thought trying to fix the biggest issue (in my eyes) of the program would be more worth it. On files with some mb where you often use the hex plugin the whole program is also very slow(in opening and all features).
This post follows two goals.
1. Name the reason of the crashes because with size of ~60mb there shouldn´t be crashes because of memory allocation?!?
2. Finding ideas to have big file support in notepad++ and also speed up opening and handling much. I think in programs like hexworkshop which I like very much because of it´s speed the file won´t be load at all at any time and this should be the only possible solution. I am unsure if this is possible to do with scintilla and the second bigger problem are the plugins. Maybe the api has to change in a big way.
I hope this starts a small dicussion which brings notepad++ ahead
if you wanto to and have time, try to repeat those tests with SciTE editor too, which is the "official" editor made by scintilla developers. Scintilla is the editor components beneath Notepad++. Maybe we could obtain some basic indications on the limits of scintilla itself.
I think other improvement to make is speed up column mode edition. It is problem yet for much more small files. I know I have to search in the forum by same topic and by others workarounds (column editor or crtl+R find), always same answer, but column mode edition is more flexible.
Congratulations and thanks by it great text editor.
Sorry my English.
"we had much blaming about notepad++ isn´t able to show and open files with xx mb. My tests show it´s critical from ~60mb up."
Perhaps the proper answer is "Don't use NP++ for really big files?"
NP++ is wonderful, but I don't believe there is a "one size fits all" editor. I use NP++ as my usual editor, but there are things it *doesn't* do (like have an actual script *language* embedded in which you can extend the editor, ala emacs). So while I have NP++ installed, I have an assortment of other editors as well.
For instance, I don't generally need to *edit* really huge files - just view them. For that, a port of Unix "less" is likely to get the nod.
Folks looking for editors that handle huge files might poke around here:
http://TextEditors.org
It's the most comprehensive listing of text editors for any platform I'm aware of. And it's a wikiwiki site, so feel free to add ones you know about that the site doesn't list. (Yes, NP++ is listed, as part of the Scintilla editors family. http://texteditors.org/cgi-bin/wiki.pl?NotePadPlusPlus\)
Thanks but this is an answer that brings no advantage to anyone.
< Perhaps the proper answer is "Don't use NP++ for really big files?"
< [...]
< NP++ is wonderful, but I don't believe there is a "one size fits all" editor
That might be often right but it´s also an easy excuse. For me notepad++ fits for almost everything. I use Visual Studio on windows and Eclipse on Linux for developing c/c++ but for everything else also viewing source files I use Notepad++. I use hexworkshop for interpret files thats right. But also this has a limitation where I like the plugin for Notepad++ very much (thanks Jens :) ).
I would be willing to work with others on changing the file management. First good improvements were made on v5 but I hope more is possible. I´m not so fast in understanding source code written by others and also I don´t want to waste time in searching if the things I want to do are possible if there are others that know this project much better (Don, Jens, Harry...)
"That might be often right but it´s also an easy excuse."
Indeed, but riddle me this: how often do you need to open a really big file? What kind of files are they? Do you need to *edit* them, or just view them?
Don's answer about requiring the whole file to be loaded for syntax highlighting is one thing I would expect, but that brings up another question. Syntax highlighting is useful for viewing the structure of program code. Do you actually have modules of source code that are that large? I can see it being potentially useful for XML files, which can get huge, but I prefer a specialized tool if I need to deal with stuff like that. It might also be useful for log files if you define a syntax highlighter for the structure of that particular log, but again, a specialized tool is probably a better bet. (For log files I want a viewer with a regular expression search facility, since what I'm interested in will be specific messages related to errors. I don't care about the rest of the file.)
I like things like less for file viewing because they begin displaying the file as soon as enough is buffered from disk to do so. I don't have long wait for things to show at all. And I never need to *edit* files that size, so the fact it's a viewer only isn't an issue. (Less can also call whatever is defined as the standard editor on a system to edit the file being viewed, but it's up to the user to know if doing that makes sense for the file being viewed.)
One option for NP++ might be a "view" mode for big files that explicitly turns *off* syntax highlighting to save time, and doesn't provide editing.
One thing is NPP's own file handling code etc., but isn't part of the "bigfile" limitation bound to the Scintilla control? AFAIK, current scintilla doesn't handle very large files well, as it was written with smaller (source code) files in mind...
> One thing is NPP's own file handling code etc., but isn't part of the "bigfile" limitation bound to the
> Scintilla control?
It´s relatively sure that the crash happens in Scintilla yes.
> AFAIK, current scintilla doesn't handle very large files well, as it was written with smaller (source code)
> files in mind...
That might be but Notepad++ controls scintilla and scintilla is also "only" for the visual output so Notepad++ can control how much and what should be shown by Scintilla.
Maybe the only really good working change would be to open big files with some limitations but I would love to see everything working.
>> AFAIK, current scintilla doesn't handle very large files well, as it was written with smaller (source code)
>> files in mind...
> That might be but Notepad++ controls scintilla and scintilla is also "only" for the visual output so Notepad++ can control how much and what should be shown by Scintilla.
> Maybe the only really good working change would be to open big files with some limitations but I would love to see everything working.
That would require relatively copmlex "paging" code though, especially if you want syntax highlighting to still work. If you "only" want to be able to read through huge log files with no coloring, it's more doable, but there's still complications (search/find/replace, line numbers, caching/paging scheme, et cetera).
Frankly, I'm not sure if it would be worth the code complexity (and bug potential) - yeah, it does suck that you don't have "one editor to rule them all", but imho the alternative to using a different editor would be upgrading the core scintilla component, or use another edit component (which would mean large bodies of NPP code would need total rewrite).
> we had much blaming about notepad++ isn´t able to show and open files with xx mb.
> My tests show it´s critical from ~60mb up.
> Don didn´t write much to this issue which could have the reason
> that the kind of these messages was only blaming without helping anyone.
Well, here's the official answer for your question : we don't really know up to how many MB notepad++ is able to open a file - it depends on the amount of available memory in your system.
The current implementation to open a file is loading whole file into Notepad++ - which makes sure the syntax highlighting won't be interrupted.
Handling a important size of file needs a more sophisticated method to synchronize view and loading of a part of file.
It means also that some existing features will be sacrificed, for example syntax highlighting.
This issue is taken into account in the Notepad++ future evolution - if we have enough time and if there's no technical impact.
Don
Your answer didn´t really go into the direction I want but thank you anyway.
> Well, here's the official answer for your question : we don't really know up to how many MB notepad++ is able > to open a file - it depends on the amount of available memory in your system.
Sure. But it is more a restriction of the OS then of the memory because 4gb in my case is more then enough.
> The current implementation to open a file is loading whole file into Notepad++ - which makes sure the syntax
> highlighting won't be interrupted.
>
> Handling a important size of file needs a more sophisticated method to synchronize view and loading of a part
> of file.
> It means also that some existing features will be sacrificed, for example syntax highlightin
I hoped for a more technical discussion. If I think more about syntax highlighting I could think of problems you direct to. They should be handlable (but I know too less about the way scintilla does this).
Won´t it possible until the point this change will find a way into the code to produce an error message instead a bad crash?
> Won´t it possible until the point this change will find a way
> into the code to produce an error message instead a bad crash?
I agree with you that'll be a better management.
The problem is the crash comes from sciLexer.dll that makes difficult (impossible?) to catch the exception (the same situation from the plugins).
I did try but in vain, but if you have any idea to prevent Notepad++ from the crash, you are more than welcome.
Don
>> Won´t it possible until the point this change will find a way
>> into the code to produce an error message instead a bad crash?
>>
> I agree with you that'll be a better management.
> The problem is the crash comes from sciLexer.dll that makes
> difficult (impossible?) to catch the exception (the same situation
> from the plugins).
>
> I did try but in vain, but if you have any idea to prevent Notepad++
> from the crash, you are more than welcome.
>
I guess the crash is a win32 exception rather than a thrown but unhandled
C++ exception? In that case, a C++ style exception handler won't help you.
But it's likely that you can use a SEH-style exception handler to catch
the error - whether you can recover from the crash is another story, though
(if Scintilla's internal state is messed up, being able to catch the error
might not do much good).
Thing is, SEH exception handlers are compiler-specific, so you'd probably
be doing MSVC vendor lock-in if you use them (dunno if GCC supports it and
whether it's using the same syntax). Look up the MSVC __try and __except
keywords.
Really neat tip Sune!
Just played around with __try and __except, it can capture "access violation" exception.
There won't be the crash anymore while opening a huge file in the future version of Notepad++. Instead Notepad++ will just refuse to open it.
Don
Ok thats a word, thanks.
Will look into it tomorrow. I hoped Scintilla returned any strange result notepad++ didn´t handled yet, but this sounds bad. Doing any size limit isn´t possible because vista can handle much more then XP (tested myselv) and more memory has to be better (don´t tested this :-D)
One way to have n++ work better with large files is for it to check the size of the file before loading it and if it's above some number (like a couple or few MB) then disable highlighting and other features which slow things down.
Given that many other programs work with large files (some load the whole file in memory and some don't), n++ can be made to do so too.
For most memory issues, probably, any error message needs to come from the OS. And most OS's suck bad at these types of error messages. It's some other process's fault. Well yeah the program may not have made sure there was enough memory before trying to use it but the OS can do something other than allow a silent seizure to take place. OS's are still infants.
> For most memory issues, probably, any error message needs to come from the OS. And most OS's suck bad at these
> types of error messages. It's some other process's fault. Well yeah the program may not have made sure there
> was enough memory before trying to use it but the OS can do something other than allow a silent seizure to
> take place. OS's are still infants.
I don´t know if you have a real technical beackground here. It´s not really possible to show how much memory availible is and no one would try this. A program can easily try to allocate memory and if this fails doing any workaround/stoping or exit with a error where you know what happens.
The system error isn´t really an error message it says only that the program crashes on an instruction.
> I don´t know if you have a real technical beackground here. It´s not really possible to show how much memory availible is and no one would try this. A program can easily try to allocate memory and if this fails doing any workaround/stoping or exit with a error where you know what happens.
The system error isn´t really an error message it says only that the program crashes on an instruction.
I do have a technical background and I am a programmer but I haven't had to be involved in these issues so there is a lot I don't know. I have worked much with C++ and memory allocation in the past. Yes one should catch any allocation failures. But I doubt you are right that a program can't check how much memory is available given that there are many 3rd party programs that work at a level where this must be necessary and even free utilities report on various states of memory.
Yes the system errors typically suck and could be much more rebust. It's easy to blame it on the architecture but it is the OS's responsibility to manage memory. This is why is say that OS's are infants.
> But I doubt you are right that a program can't check how much memory is available given that there are many 3rd
> party programs that work at a level where this must be necessary and even free utilities report on various states
> of memory.
Yeah that might be right. But often you need big chunks of memory (for allocate arrays for example) and to find out if those chunks are availible you can´t use any windows api function. But this discussion is nothing that helps Notepad++ I also doubt that I will ever see such a big file support in here :-D
>> But I doubt you are right that a program can't check how much memory
>> is available given that there are many 3rd party programs that work
>> at a level where this must be necessary and even free utilities report
>> on various states of memory.
>>
> Yeah that might be right. But often you need big chunks of memory
> (for allocate arrays for example) and to find out if those chunks
> are availible you can´t use any windows api function. But this
> discussion is nothing that helps Notepad++ I also doubt that I
> will ever see such a big file support in here :-D
>
You can query available memory (there's various stats) to find out
what reasonable "big buffer sizes" are, for instance if you want to
have a dynamically sized cache or the like. Those stats should only
be seen as a rule of thumb, though, and not a guarantee - we live
under a multitasking operating system and have to cooperate with
other running processes. Furthermore, issues like memory fragmentation
can play a role.
The correct thing to do is, of course, to handle memory allocation
failure gracefully. And to avoid allocating memory in ways that lead
to heap fragmentation...
>You can query available memory (there's various stats) to find out
what reasonable "big buffer sizes" are, for instance if you want to
have a dynamically sized cache or the like. Those stats should only
be seen as a rule of thumb, though, and not a guarantee - we live
under a multitasking operating system and have to cooperate with
other running processes. Furthermore, issues like memory fragmentation
can play a role.
>The correct thing to do is, of course, to handle memory allocation
failure gracefully. And to avoid allocating memory in ways that lead
to heap fragmentation...
Ok - I'm the guy that Simon was asking about my technical background again.
Now with your new info, I'll better detail what I'm suggesting. Say a n++ user wants to open a 100 mb file, so he he "opens" it. N++ needs to check the file size of every file before it loads it. N++ sees that the user is trying to open a big file, so N++ checks with Windows to see roughly how much memory N++'s process can get. If there isn't significantly MORE than enough memory to be had, N++ aborts the "open" with a nice message: "file too large for available memory". If there seems to be enough memory, then N++ does a "new" (alloc) for the full amount that it will need for the file and "catches" the fail (if that's what happens) with a nice error message. Of course, if N++ runs out of stack memory or uses a bad pointer, then we're left to the OS to "manage" the crash. Frankly, this is so straightforward, I don't know why so many Windows programs, including Microsoft's, and other OS's seem to have so much trouble with this (and things like choking on a control/binary character). There's two guilty parties: the OS's and the programs, even Microsoft's own programs.
> Now with your new info, I'll better detail what I'm suggesting. Say a n++
> user wants to open a 100 mb file, so he he "opens" it. N++ needs to check
> the file size of every file before it loads it. N++ sees that the user is
> trying to open a big file, so N++ checks with Windows to see roughly how much
> memory N++'s process can get.
>
That's doable without much fuzz.
> If there isn't significantly MORE than enough memory to be had, N++ aborts
> the "open" with a nice message: "file too large for available memory". If
> there seems to be enough memory, then N++ does a "new" (alloc) for the full
> amount that it will need for the file and "catches" the fail (if that's what
> happens) with a nice error message.
>
That's easy as well... BUT... things are more complicated than that :)
I haven't dug into the internal structures used by Scintilla, but I think it's
safe to say it might not be trivial guesstimating just how much memory you need
(ie., to perhaps store things like line-number indices, syntax coloring info,
parsing state, ...) - and if there's a bunch of dynamic allocations going on,
you risk the heap fragmentation issue, which can be a major pain as well.
> Of course, if N++ runs out of stack memory or uses a bad pointer, then we're
> left to the OS to "manage" the crash.
>
That kind of stuff can usually be caught with exception handling, unless you're
in a *really* messed up situation. Can't always be fixed, but often you can at
least exit relatively gracefully, instead of just crashing and discarding file
modifications.
> Frankly, this is so straightforward, I don't know why so many Windows programs,
> including Microsoft's, and other OS's seem to have so much trouble with this
> (and things like choking on a control/binary character). There's two guilty
> parties: the OS's and the programs, even Microsoft's own programs.
>
Well, memory allocation isn't exactly trivial, once you go beyond basic usage :)
And keep in mind that Scintilla wasn't really designed with huge files in mind,
so trying to open a 50+meg logfile should probably be seen as abuse...
>> Of course, if N++ runs out of stack memory or uses a bad pointer, then we're
>> left to the OS to "manage" the crash.
>>
>That kind of stuff can usually be caught with exception handling, unless you're
>in a *really* messed up situation. Can't always be fixed, but often you can at
>least exit relatively gracefully, instead of just crashing and discarding file
>modifications.
I thought of an easy stop on processing the file if a buffer (re)allocation fails
so I looked into the code today and have to agree Don (didn´t thought I get it
really better to work then he ;) the Scintilla system is really complex and don´t
seem to make handling errors really easy or possible. Maybe I try a further step
later.
>That's easy as well... BUT... things are more complicated than that :)
Yeah spent weeks in 2000 trying to fix someone else's way uncessarily complicated memory allocation on a Windows (those Window's API memory functions are long since obsolete, actually they were obsolete for new use then) and couldn't really fix the problems, only hide them. (I was told the original programmer was superb and the code was solid. BS.)
> Well, memory allocation isn't exactly trivial, once you go beyond basic usage :)
And keep in mind that Scintilla wasn't really designed with huge files in mind,
so trying to open a 50+meg logfile should probably be seen as abuse...
I'm sure Scintilla is the real wild card. I have opened files bigger than 50 mb with n++ and had luck. (several versions back, haven't had to since) Usually I only had to check on a couple things and then close the file. Searching worked. It would be nice to know what Scintilla's limitations are (to the extent they can be known) and work around them as necessary.
"once you go beyond basic usage" - the vast majority of programs including a text/data editor like n++ should not go beyond "basic usage" of memory (and the amount memory used in a perfect world/OS should only matter for speed) - but you have to work with what you got
Hi all,
Thought my c++ is nowhere near good enough to help, I would definitely like to add a +1 for this. Notepad++ is such a fantastic editor but not being able to open large (log) files with it is a real pain. True, I do it rarely, but that makes it all the more annoying to have to install another app that virtually never gets used just for that! On *nix less and vi do the job but it would be truely fantastic to be able to open log files and search with notepad++ (you shouldn't really need to do much else), I'm pretty sure I could get it into our company's standard install for both desktop and server if we could open large files with it...
I see some limits on certain things have been introduced with 5, are we heading in this direction?
Cheers
Anton