I've found my self looking for a good open source text editor that is truly 64 bit. I've had great difficulty in this search. A friend of mine suggested I apply notepad++ to my problems. It of course doesn't work either, but it's open source.
I will be fairly routinely generating 6GB+ text files that contain 6M+ rows with about 100 columns. I need a simple way to briefly inspect them and remove the first line, before processing them in MATLAB.
Would it be difficult/possible to compile the notepad++ source code into a 64bit app capable of opening such files.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks Fabio, that's what I wanted to know. And while some of what Sune said is true, what isn't true is his claim that it's rare to need to open such large files. I know I've seen lots of inquiries about this, granted 99% were from lazy network admins that needed to open log files they had neglected. But, this situation is certainly not rare, and it will be getting more common.
And while i never asserted that 64bit was more efficient, it is fairly evident from practical experience that you are not going to open a file bigger than 2GB with a 32bit app. For whatever reason they flip out when the file size exceeds 2GBs. I never really looked for any that supported the /3GB boot ini switch because it wouldn't have helped at all.
@Jocelyn, thanks for chiming in. My group is probably going to run all the metabolic models that we need, in the coming days. Those results are expected to keep us busy for at least six months. So I will keep an eye out for the 64bit version of notepad++. I'm not completely knowledgeable on this matter but it seemed like most of the 64 bit programs still had 32bit memory addressing. Because,. I downloaded at least 2 that were 64 bit, and they would just crash on loading the file, when they reached 2GB memory usage.
For those not familliar.: A 32 bit app generally can only access 2GB of ram unless it is /3GB aware, and can access 3GB of ram. I'm not aware if 64 bit programs have such limits, but they can certainly open files that are terabytes in size with no problem. I might even claim they are more efficient since they can open the files whereas a 32 bit program definitely will not. (of course I'm told that other OS's won't be limited in this way, and they could also end up being the solution as well)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hope you find a program that you can use, rather than waiting for Notepad++ to be able to handle huge files (which I doubt will happen). How much editing do you need to do on the huge file, though?
If you only need to move the first line to another file, you could use LTFViewer to verify the data, and then sed/awk/whatever to extract the first line and copy the rest to another file.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Well It'd still be nice to skim the file before committing the hour it takes for Matlab to import it. And I guess I should have mentioned that I'm not deleting the first line, I need to keep it. Just move it to it's own txt file really.
I don't know about SED, so I'll look into that. If this txt file editor thing doesn't work I can probably get somebody to write up something in perl. I'm guessing that what your saying is that compiling something isn't feasible.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
With such a big file, I would probably use an hex editor to open it, one that does not load the file in memory to open it (i.Hex would be good, but not the last version I tested of Hexplorer). You can quickly scan through your file to check if it look correct, without having to completely load it. And to extract the lines you could use "head" and "tail" (the standard un*x utils) : "head -1 filename" will give you the first line and "tail +2 filename" will give you the file starting at the 2nd line (note the "+" and "-").
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
64bit really has nothing to do with efficient handling of text files - you very rarely need to load insanely large amounts of text all at once, and 32bit editors can do partial loading just as well as a 64bit editors; this would likely be a much smarter scenario for you, since you say you need to "briefly inspect" the file.
Notepad++ isn't geared towards large files, and (at least a few versions ago) would choke on files even in the ~100meg range. The scintilla editing component is designed towards source code size files, not humongous datasets :)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
@ Sune your probably right when speaking of 2GB and smaller files, but through this process it seemed that if the file exceeds the usual 2GB address space that 32bit programs are generally limited to, that they just choke. I tried all the ones I could find too. Several 64bit ones proved to still have 2 GB limits, I didn't find a single 32bit editor that would even partially open these big files I have. Fortunately, today I finally found one call EMeditor that is available in what I'm going to call "true 64bit". It opens them right up, no wait, and runs great. It cost $40 though. It's probably worth it too. Just thought I'd mention it, because In searching I saw alot of people have this problem, but never get any real useful answers.
I'm glad that you've already found a solution. However, a 64-bit version of N++ can't be obtained very easily: someone's already tried it, and the result was that the compiler reported hundreds of errors. And anyway, as Sune says, it wouldn't be a solution. N++ just can't handle files of that size.
This is just to tell you that you must look for another editor (and I see you've already done it).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yeah, I did a (quick) attempt at a 64bit build as well, for the heck of it, and it seemed way too much bother to be worth it (not to mention that scintilla didn't build 64bit clean back then either, dunno what the status is now). Also, just fixing the build errors is no guarantee that it would actually work, you'd have to compile at very high warning level plus do some manual code inspection as well...
And altogether it seemed pretty pointless because of the way scintilla is structured :)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
countdown: I've cleaned up the code quite a lot this summer, and a 64 bit version is in the plans for me in the near-mid future. Not 100% certain it would fix your memory problems, but I'm not 100% on that. We'll know when I'll get there.
joce.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks Fabio, that's what I wanted to know. And while some of what Sune said is true, what isn't true is his claim that it's rare to need to open such large files. I know I've seen lots of inquiries about this, granted 99% were from lazy network admins that needed to open log files they had neglected. But, this situation is certainly not rare, and it will be getting more common.
And while i never asserted that 64bit was more efficient, it is fairly evident from practical experience that you are not going to open a file bigger than 2GB with a 32bit app. For whatever reason they flip out when the file size exceeds 2GBs. I never really looked for any that supported the /3GB boot ini switch because it wouldn't have helped at all.
@Jocelyn, thanks for chiming in. My group is probably going to run all the metabolic models that we need, in the coming days. Those results are expected to keep us busy for at least six months. So I will keep an eye out for the 64bit version of notepad++. I'm not completely knowledgeable on this matter but it seemed like most of the 64 bit programs still had 32bit memory addressing. Because,. I downloaded at least 2 that were 64 bit, and they would just crash on loading the file, when they reached 2GB memory usage.
For those not familliar.: A 32 bit app generally can only access 2GB of ram unless it is /3GB aware, and can access 3GB of ram. I'm not aware if 64 bit programs have such limits, but they can certainly open files that are terabytes in size with no problem. I might even claim they are more efficient since they can open the files whereas a 32 bit program definitely will not. (of course I'm told that other OS's won't be limited in this way, and they could also end up being the solution as well)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
James, it might not be that uncommon to view several-gigabyte logs files - but how often do you need to <b>edit</b> those files? Imho you're better off finding a read-only viewer (or an editor built to handle huge files by partial loading) than trying to shoehorn a general editor into something it wasn't built for :)
Again, it would be no problem for a 32bit program to handle huge files like that, but there's certain assumptions it <i>cannot</i> make... like trying to load the entire file, using "int" datatype for line numbers and file size, etc. I guess I should try hunting down such an editor :)
PS: other (32bit) OSes have pretty much the same limits as Windows, because you're dealing with a 32bit address space. Oh, and if you mark your 32bit executables as large-addresspace-aware, they can use up to around 4GB if you run them under a 64bit Windows :)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I still don't get the obsession with trying to jump through hoops getting a *programmer's* file editor to handle multi-gigabyte files, to be honest... better to focus on core functionality, bugfixes etc.
I still don't get the obsession with trying to jump through hoops getting a <b>programmer's</b> file editor to handle multi-gigabyte files, to be honest... better to focus on core functionality, bugfixes etc.
The reason I would like a 64 bit version of Notepad++ is because I'm seeing weird behaviour when editing files on a 64 bit environment. Had the issue recently where I was trying to modify the applicationHost.config file on a Windows Server 2012 Standard 64-bit server and kept on encountering different content when opening the same file in Windows NotePad (x64) and NotePad++ (x86).
For the time being stopped using NotePad++ on the server but read on ServerFault that it could be due to an architecture issue, but tried the suggestion and still had the same problem.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
+1. Editing files that sail under the File System Redirector flag (http://msdn.microsoft.com/en-us/library/windows/desktop/aa384187%28v=vs.85%29.aspx) is practically impossible while using a 32 bit version of Notepad++.
Last edit: sAnexeh 2015-01-06
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
This just bit me too. To me, all that old discussion about efficiency and file size is irrelevant. I can't believe I didn't run into this before... I probably did and beat my head against the wall for a while before moving on. I've been using Notepad++ for a long time but have to give it up given this problem.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
To be fair, most modernly compiled apps should require little to know work to be recompiled in 64 bits. Additionally, WoW64 won't be around forever. Most well-maintained projects are switching to x64.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've found my self looking for a good open source text editor that is truly 64 bit. I've had great difficulty in this search. A friend of mine suggested I apply notepad++ to my problems. It of course doesn't work either, but it's open source.
I will be fairly routinely generating 6GB+ text files that contain 6M+ rows with about 100 columns. I need a simple way to briefly inspect them and remove the first line, before processing them in MATLAB.
Would it be difficult/possible to compile the notepad++ source code into a 64bit app capable of opening such files.
Thanks Fabio, that's what I wanted to know. And while some of what Sune said is true, what isn't true is his claim that it's rare to need to open such large files. I know I've seen lots of inquiries about this, granted 99% were from lazy network admins that needed to open log files they had neglected. But, this situation is certainly not rare, and it will be getting more common.
And while i never asserted that 64bit was more efficient, it is fairly evident from practical experience that you are not going to open a file bigger than 2GB with a 32bit app. For whatever reason they flip out when the file size exceeds 2GBs. I never really looked for any that supported the /3GB boot ini switch because it wouldn't have helped at all.
@Jocelyn, thanks for chiming in. My group is probably going to run all the metabolic models that we need, in the coming days. Those results are expected to keep us busy for at least six months. So I will keep an eye out for the 64bit version of notepad++. I'm not completely knowledgeable on this matter but it seemed like most of the 64 bit programs still had 32bit memory addressing. Because,. I downloaded at least 2 that were 64 bit, and they would just crash on loading the file, when they reached 2GB memory usage.
For those not familliar.: A 32 bit app generally can only access 2GB of ram unless it is /3GB aware, and can access 3GB of ram. I'm not aware if 64 bit programs have such limits, but they can certainly open files that are terabytes in size with no problem. I might even claim they are more efficient since they can open the files whereas a 32 bit program definitely will not. (of course I'm told that other OS's won't be limited in this way, and they could also end up being the solution as well)
Sorry about the repost above
Large address space aware programs can see 3GB of workspace, on 64bit, and 32 with the right boot switches.
I need to edit the file, once after it's been created.
Thanks for the links, I scribbled down 6 programs that might help, if I decide I don't want to spend money.
Thanks, for letting me know it is pointless to try and compile something.
Actually, 32bit programs *can* use ~4GB of memory on 64bit Windows - see Mark Russinovich's excellent blog: http://blogs.technet.com/markrussinovich/archive/2008/11/17/3155406.aspx .
Hope you find a program that you can use, rather than waiting for Notepad++ to be able to handle huge files (which I doubt will happen). How much editing do you need to do on the huge file, though?
If you only need to move the first line to another file, you could use LTFViewer to verify the data, and then sed/awk/whatever to extract the first line and copy the rest to another file.
If all your doing is removing the first line it sounds like you might have more luck with a program like sed/perl/etc
Well It'd still be nice to skim the file before committing the hour it takes for Matlab to import it. And I guess I should have mentioned that I'm not deleting the first line, I need to keep it. Just move it to it's own txt file really.
I don't know about SED, so I'll look into that. If this txt file editor thing doesn't work I can probably get somebody to write up something in perl. I'm guessing that what your saying is that compiling something isn't feasible.
With such a big file, I would probably use an hex editor to open it, one that does not load the file in memory to open it (i.Hex would be good, but not the last version I tested of Hexplorer). You can quickly scan through your file to check if it look correct, without having to completely load it. And to extract the lines you could use "head" and "tail" (the standard un*x utils) : "head -1 filename" will give you the first line and "tail +2 filename" will give you the file starting at the 2nd line (note the "+" and "-").
64bit really has nothing to do with efficient handling of text files - you very rarely need to load insanely large amounts of text all at once, and 32bit editors can do partial loading just as well as a 64bit editors; this would likely be a much smarter scenario for you, since you say you need to "briefly inspect" the file.
Notepad++ isn't geared towards large files, and (at least a few versions ago) would choke on files even in the ~100meg range. The scintilla editing component is designed towards source code size files, not humongous datasets :)
@ Sune your probably right when speaking of 2GB and smaller files, but through this process it seemed that if the file exceeds the usual 2GB address space that 32bit programs are generally limited to, that they just choke. I tried all the ones I could find too. Several 64bit ones proved to still have 2 GB limits, I didn't find a single 32bit editor that would even partially open these big files I have. Fortunately, today I finally found one call EMeditor that is available in what I'm going to call "true 64bit". It opens them right up, no wait, and runs great. It cost $40 though. It's probably worth it too. Just thought I'd mention it, because In searching I saw alot of people have this problem, but never get any real useful answers.
http://www.emeditor.com/
I'm glad that you've already found a solution. However, a 64-bit version of N++ can't be obtained very easily: someone's already tried it, and the result was that the compiler reported hundreds of errors. And anyway, as Sune says, it wouldn't be a solution. N++ just can't handle files of that size.
This is just to tell you that you must look for another editor (and I see you've already done it).
Yeah, I did a (quick) attempt at a 64bit build as well, for the heck of it, and it seemed way too much bother to be worth it (not to mention that scintilla didn't build 64bit clean back then either, dunno what the status is now). Also, just fixing the build errors is no guarantee that it would actually work, you'd have to compile at very high warning level plus do some manual code inspection as well...
And altogether it seemed pretty pointless because of the way scintilla is structured :)
countdown: I've cleaned up the code quite a lot this summer, and a 64 bit version is in the plans for me in the near-mid future. Not 100% certain it would fix your memory problems, but I'm not 100% on that. We'll know when I'll get there.
joce.
Thanks Fabio, that's what I wanted to know. And while some of what Sune said is true, what isn't true is his claim that it's rare to need to open such large files. I know I've seen lots of inquiries about this, granted 99% were from lazy network admins that needed to open log files they had neglected. But, this situation is certainly not rare, and it will be getting more common.
And while i never asserted that 64bit was more efficient, it is fairly evident from practical experience that you are not going to open a file bigger than 2GB with a 32bit app. For whatever reason they flip out when the file size exceeds 2GBs. I never really looked for any that supported the /3GB boot ini switch because it wouldn't have helped at all.
@Jocelyn, thanks for chiming in. My group is probably going to run all the metabolic models that we need, in the coming days. Those results are expected to keep us busy for at least six months. So I will keep an eye out for the 64bit version of notepad++. I'm not completely knowledgeable on this matter but it seemed like most of the 64 bit programs still had 32bit memory addressing. Because,. I downloaded at least 2 that were 64 bit, and they would just crash on loading the file, when they reached 2GB memory usage.
For those not familliar.: A 32 bit app generally can only access 2GB of ram unless it is /3GB aware, and can access 3GB of ram. I'm not aware if 64 bit programs have such limits, but they can certainly open files that are terabytes in size with no problem. I might even claim they are more efficient since they can open the files whereas a 32 bit program definitely will not. (of course I'm told that other OS's won't be limited in this way, and they could also end up being the solution as well)
James, it might not be that uncommon to view several-gigabyte logs files - but how often do you need to <b>edit</b> those files? Imho you're better off finding a read-only viewer (or an editor built to handle huge files by partial loading) than trying to shoehorn a general editor into something it wasn't built for :)
Again, it would be no problem for a 32bit program to handle huge files like that, but there's certain assumptions it <i>cannot</i> make... like trying to load the entire file, using "int" datatype for line numbers and file size, etc. I guess I should try hunting down such an editor :)
PS: other (32bit) OSes have pretty much the same limits as Windows, because you're dealing with a 32bit address space. Oh, and if you mark your 32bit executables as large-addresspace-aware, they can use up to around 4GB if you run them under a 64bit Windows :)
http://sourceforge.net/apps/ideatorrent/notepad-plus/ideatorrent/idea/24/ shows various solutions. Perhaps you can try adding one, or voting for an existing one (be sure to read how to work around the brokn vote system).
CChris
I still don't get the obsession with trying to jump through hoops getting a *programmer's* file editor to handle multi-gigabyte files, to be honest... better to focus on core functionality, bugfixes etc.
Perhaps tage a look at http://www.donationcoder.com/Forums/bb/index.php?topic=19684 - the most promising tool so far seems to be LTFViewer, or UltraEdit if you need *editing*.
I still don't get the obsession with trying to jump through hoops getting a <b>programmer's</b> file editor to handle multi-gigabyte files, to be honest... better to focus on core functionality, bugfixes etc.
Perhaps tage a look at http://www.donationcoder.com/Forums/bb/index.php?topic=19684 - the most promising tool so far seems to be LTFViewer, or UltraEdit if you need *editing*.
The reason I would like a 64 bit version of Notepad++ is because I'm seeing weird behaviour when editing files on a 64 bit environment. Had the issue recently where I was trying to modify the applicationHost.config file on a Windows Server 2012 Standard 64-bit server and kept on encountering different content when opening the same file in Windows NotePad (x64) and NotePad++ (x86).
For the time being stopped using NotePad++ on the server but read on ServerFault that it could be due to an architecture issue, but tried the suggestion and still had the same problem.
+1. Editing files that sail under the File System Redirector flag (http://msdn.microsoft.com/en-us/library/windows/desktop/aa384187%28v=vs.85%29.aspx) is practically impossible while using a 32 bit version of Notepad++.
Last edit: sAnexeh 2015-01-06
This just bit me too. To me, all that old discussion about efficiency and file size is irrelevant. I can't believe I didn't run into this before... I probably did and beat my head against the wall for a while before moving on. I've been using Notepad++ for a long time but have to give it up given this problem.
To be fair, most modernly compiled apps should require little to know work to be recompiled in 64 bits. Additionally, WoW64 won't be around forever. Most well-maintained projects are switching to x64.
I would like to use N++ as a lightweight code editor in a x64 only (no WOW) environment.