Thread: [nedit-develop] Does NEdit 5.6 support editing huge files (over 4GB) ?
Brought to you by:
tringali
From: ardi <ard...@gm...> - 2015-03-27 17:00:58
|
Hi, I'm subscribing to nedit-develop because I cannot find nedit-discuss anymore (I suppose that list no longer exists). I tried to open a huge file (over 4GB) with a 64bit executable of NEdit 5.5 some months ago, and it failed. Has this been improved in 5.6? If negative, what would be needed to turn NEdit huge-file compatible? Is it a lot of effort, or can I try it myself? Is there any limit imposed by the XmText Motif widget, or does NEdit jus pass text fragments to XmText rather than the whole file? Thanks! ardi |
From: ardi <ard...@gm...> - 2015-03-28 14:34:34
|
Thanks a lot for your comments and advice. BTW, do you mean that NEdit sends the whole text file to the XmText widget? I remember that some old Motif docs recommended to avoid sending large texts to XmText and suggested doing so by "pseudo-paging" between the widget and the main RAM. If you're sure the XmText widget holds the whole text, I'll take a look at the OpenMotif source code, maybe it needs some patching for supporting over 4GB texts. And, where's the fork you mention? How can I locate it? Thanks a lot! ardi On Friday, March 27, 2015, Scott Tringali <sc...@tr...> wrote: > It's open source, you are definitely welcome to try. Someone also forked > the project but I have no idea what's going on over there, maybe they've > done some work on it? > > There might be some unintentional bugs in using file offsets and the > like. But in general, I'm pretty sure the original design of nedit is that > the just data has to fit into virtual memory, as there's no support for > paging in and out to a file. We're also limited to how Motif handles such > things. > > 5.6 was just a final packaging up of what various distros were already > delivering out of our CVS repro. Anything you see as 5.5 was likely very > close to 5.6. > > On 03/27/2015 01:00 PM, ardi wrote: > > I'm subscribing to nedit-develop because I cannot find nedit-discuss >> anymore (I suppose that list no longer exists). >> >> I tried to open a huge file (over 4GB) with a 64bit executable of >> NEdit 5.5 some months ago, and it failed. Has this been improved in >> 5.6? >> >> If negative, what would be needed to turn NEdit huge-file compatible? >> Is it a lot of effort, or can I try it myself? Is there any limit >> imposed by the XmText Motif widget, or does NEdit jus pass text >> fragments to XmText rather than the whole file? >> > |
From: Ivan S. J. <isj...@i1...> - 2015-03-28 21:33:20
|
On Friday 27 March 2015 18:00:51 ardi wrote: > Hi, > > I'm subscribing to nedit-develop because I cannot find nedit-discuss > anymore (I suppose that list no longer exists). > > I tried to open a huge file (over 4GB) with a 64bit executable of > NEdit 5.5 some months ago, and it failed. Has this been improved in > 5.6? I did look into it several years ago. I seem to recall that offsets in the the buffer are 32-bit, so you'd probably get into trouble at ~2GB. Hmmm.... Looking at textBuf.h reveals that plain 'int' is used, so nedit woud start failing at files larger than 2^31-1 bytes. > If negative, what would be needed to turn NEdit huge-file compatible? > Is it a lot of effort, or can I try it myself? I would say it is a medium-sized task, but will require good tools for finding lossy 64->32 bit conversions. If someone can help me with the Motif/Lesstif issues that may crop up then I'm willing to give it a shot over the next few weeks. /isj |
From: Ivan S. J. <isj...@i1...> - 2015-04-02 17:56:49
|
On Saturday 28 March 2015 22:15:34 I wrote: > > If someone can help me with the Motif/Lesstif issues that may crop up then I'm willing to give it a shot over the next few weeks. One challenge is that XtMalloc() is extensively used instead of malloc(). XtMalloc/XtRealloc/XtCalloc take the size as a 'Cardinal' which is 32-bit. XtMalloc/XtRealloc/XtCalloc print a warning and exit on out-of-memory. Possibilities: 1: Change to malloc/realloc/calloc/free pro: easy, well-known, better warnings from tools con: on out-of-memory nedit will core/exit 2: Implement nedit_alloc/nedit_realloc/nedit_calloc/... pro: clean exit on out-of-memory con: more work for me. Any strong opinions? /isj |
From: ardi <ard...@gm...> - 2015-03-29 16:02:44
|
On Saturday, March 28, 2015, Ivan Skytte Jørgensen <isj...@i1...> wrote: > On Friday 27 March 2015 18:00:51 ardi wrote: > > If negative, what would be needed to turn NEdit huge-file compatible? > > Is it a lot of effort, or can I try it myself? > > I would say it is a medium-sized task, but will require good tools for > finding lossy 64->32 bit conversions. One good approach would be to start changing counters to "size_t", compile in 64bit, and enable compiler warnings to the most pedantic ones. I think both gcc and clang would do a good task here, showing warnings in all places where 64bits are being clamped to 32 bits. If someone can help me with the Motif/Lesstif issues that may crop up then > I'm willing to give it a shot over the next few weeks. > If NEdit isn't sending the whole text to the widget, I don't think there will be any Motif issues when loading >4GB files, as Motif won't be seeing them. ardi |
From: Ivan S. J. <isj...@i1...> - 2015-03-29 16:47:26
|
On Sunday 29 March 2015 18:02:37 ardi wrote: > On Saturday, March 28, 2015, Ivan Skytte Jørgensen <isj...@i1...> > wrote: > > > On Friday 27 March 2015 18:00:51 ardi wrote: > > > If negative, what would be needed to turn NEdit huge-file compatible? > > > Is it a lot of effort, or can I try it myself? > > > > I would say it is a medium-sized task, but will require good tools for > > finding lossy 64->32 bit conversions. > > > One good approach would be to start changing counters to "size_t", compile > in 64bit, and enable compiler warnings to the most pedantic ones. The tricky part is to identify which values are sizes and which values are offsets. > both gcc and clang would do a good task here, showing warnings in all > places where 64bits are being clamped to 32 bits. I have Flexelint, which is better than all the compilers I have encountered. I will use that. > If someone can help me with the Motif/Lesstif issues that may crop up then > > I'm willing to give it a shot over the next few weeks. > > > > If NEdit isn't sending the whole text to the widget, I don't think there > will be any Motif issues when loading >4GB files, as Motif won't be seeing > them. I'm more concerned about the motif/lesstif/openmotif compatibility issues. There has been issues with nedit randomly coredumping or not even starting. My X/Window and Motif knowledge is quite rusty by now. /isj |
From: ardi <ard...@gm...> - 2015-03-30 15:19:38
|
On Sunday, March 29, 2015, Ivan Skytte Jørgensen <isj...@i1...> wrote: > On Sunday 29 March 2015 18:02:37 ardi wrote: > > On Saturday, March 28, 2015, Ivan Skytte Jørgensen <isj...@i1... > <javascript:;>> > > wrote: > > > > > On Friday 27 March 2015 18:00:51 ardi wrote: > > > > If negative, what would be needed to turn NEdit huge-file compatible? > > > > Is it a lot of effort, or can I try it myself? > > > > > > I would say it is a medium-sized task, but will require good tools for > > > finding lossy 64->32 bit conversions. > > > > > > One good approach would be to start changing counters to "size_t", > compile > > in 64bit, and enable compiler warnings to the most pedantic ones. > > The tricky part is to identify which values are sizes and which values are > offsets. > Maybe you can safely make all numbers signed (ptrdiff_t). Yes, size_t can hold a number twice bigger than ptrdiff_t, but at some point the code will need to cast from sizes to offsets and vice versa, so you will never be able to work with a file larger than the largest offset. If you cast to ptrdiff_t everything from the very same moment the file is loaded, maybe it would be safe. > > If NEdit isn't sending the whole text to the widget, I don't think there > > will be any Motif issues when loading >4GB files, as Motif won't be > seeing > > them. > > I'm more concerned about the motif/lesstif/openmotif compatibility issues. > There has been issues with nedit randomly coredumping or not even starting. > My X/Window and Motif knowledge is quite rusty by now. > I've been using NEdit 5.5 on OSX for many years without a crash (compiled as a 64 bit executable). Didn't know there were any problems, although I was using an old OpenMotif 2.3.x version. The only bug I used to hit was the copy/cut/paste key shortcuts stopping to respond, but I thought that was an XQuartz issue. ardi |
From: Ivan S. J. <isj...@i1...> - 2018-05-23 15:36:59
|
Just a (very delayed) update. I had a stab at it a few years ago, but I had to give up. 'int' is used almost everywhere, and when seeing a variable/parameter named "pos" it takes a long time to figure out if it is a byte offset from the start of buffer, window position, a logical cursor position, and absolute column, relative column, etc. Also, in the bowels of the macro language byte offsets are stored as 32-bit integers and it seems a major undertaking in enhancing it to support 64-bit integers. My opinion is that bringing 64-bit file support to nedit is infeasible with the current codebase. /isj |