From: David P. <da...@ac...> - 2006-08-31 06:01:23
|
A short, sad story: I spent most of the day writing coding in a single source file. As I have done thousands of times, I save-exit JOE, then run make. The linker returns an error about "main" not being found. That's strange, maybe I accidentally deleted it? Open the file, it's empty. What, did I open a new file by mistake? Running an "ls -l" shows the file as zero bytes. Huh? In the next minutes, I find out that someone filled up the NFS mount, causing JOE to unlink and recreate the file, but without being able to write any data. Source code, say good-bye. The correct fix is to add an option for reliable saves as the default: - open a temporary file in the same directory as the save target (i.e. on the same device) - write to the temp file - close the temp file (check the return status, as NFS can error here) - fsync the temp file - rename the temp file to the target name Those more familiar with the JOE source feel free to volunteer (saver confused me a bit), otherwise I can add this. |
From: Brian C. <B.C...@po...> - 2006-08-31 08:08:03
|
On Thu, Aug 31, 2006 at 01:01:17AM -0500, David Phillips wrote: > The correct fix is to add an option for reliable saves as the default: > > - open a temporary file in the same directory as the save target (i.e. > on the same device) - set owner/group (if possible) and mode bits on this file to be the same as the target [I believe there are races which need to be avoided here but can't remember the details] > - write to the temp file > - close the temp file (check the return status, as NFS can error here) > - fsync the temp file - stat() the temp file and check its size, for paranoia? - rename the original file to the backup name (unless -nobackups) > - rename the temp file to the target name A side-effect of this approach will be that you could end up being able to 'edit' files where you have create/delete permissions on the directory, but not write permissions on the file itself. The file will end up being owned by you, not the original owner. $ echo "abc" >test.txt $ sudo chown 0:0 test.txt $ ls -l test.txt -rw-r--r-- 1 root wheel 4 Aug 31 08:57 test.txt $ echo "def" >test.txt.new $ mv -f test.txt.new test.txt $ ls -l test.txt -rw-r--r-- 1 candlerb candlerb 4 Aug 31 08:57 test.txt This may or may not be desirable. As an aside: remember that joe will still need to keep the original file open and locked, to prevent a second person editing it concurrently. Some care is needed when doing a ^K D, either to keep the 'temp file' handle open and locked, or to open and lock the target again (which may fail, and so mean switching to read-only mode) Just a few thoughts. Regards, Brian. |
From: Koblinger E. <eg...@uh...> - 2006-08-31 10:04:58
|
On Thu, Aug 31, 2006 at 01:01:17AM -0500, David Phillips wrote: > The correct fix is to add an option for reliable saves as the default: > > - open a temporary file in the same directory as the save target (i.e. > on the same device) > - write to the temp file > - close the temp file (check the return status, as NFS can error here) > - fsync the temp file > - rename the temp file to the target name There are drawbacks with this method as well: - you lose the ability to edit a part of a file (e.g. joe foo.txt,0,512) - lose the ability to edit special files (e.g. joe /dev/fd0) - symlinks need special care - hardlinks are handled differently - keeping the perms of the original file needs special care - keeping the owner of the original file becomes sometimes impossible - doesn't work if you have write permission to the file, but not to its directory - double disk space needed (might be a problem for large files) However, I agree that this might be an _option_, keeping the old way of saving around. Furthermore, it would be nice (even with the old method) if joe checked for all errors you mentioned above and informed you if any of these happened. This way even if the file was destroyed on the disk, it would remain in joe's memory buffer and you could quickly make some disk space or save it on another partition. Basically "disk full" is a condition that is not handled correctly by most of the applications. In order for your system to work correctly, you have to make sure you always have a reasonable amount of free space on your disk. BTW you cannot fsync after you've closed the file, so the order should be fsync followed by close. On the other hand, I don't think sync'ing is joe's job. In fact I do think it's not joe's job. It's the job of the OS to decide when to sync, to highly increase performance. If your data and your work is really so extremely critical, then you should create regular backups (and use a source code versioning system anyway), buy a UPS, buy a larger disk and write a cron job that sends notification at 80% of usage, to make sure you'll never run out of disk space. Neither of the standard utilities (cp, cat ...) sync the newly created files, and I guess text editors (vi ...) don't do so either, so I can't see any reason why joe should. I only see a drawback of syncing: save (or exit) time of joe might grow significally. -- Egmont |
From: Dirk S. <sch...@do...> - 2006-08-31 11:35:50
|
On Thursday, 31. August 2006 12:04, Koblinger Egmont wrote: > On Thu, Aug 31, 2006 at 01:01:17AM -0500, David Phillips wrote: > > The correct fix is to add an option for reliable saves as the default: > > > > - open a temporary file in the same directory as the save target (i.e. > > on the same device) > > - write to the temp file > > - close the temp file (check the return status, as NFS can error here) > > - fsync the temp file - rename the original file :-) and then > > - rename the temp file to the target name =2D either remove the original file or rename it to a backup file > There are drawbacks with this method as well: > > - you lose the ability to edit a part of a file (e.g. joe foo.txt,0,512) Yes. This must be handled as special case. > - lose the ability to edit special files (e.g. joe /dev/fd0) Yes. same here > - symlinks need special care > - hardlinks are handled differently Yes, they might get broken or replaced > - keeping the perms of the original file needs special care > - keeping the owner of the original file becomes sometimes impossible > - doesn't work if you have write permission to the file, but not to its > directory > - double disk space needed (might be a problem for large files) joe could just do as described, then compare the stat()-entries and display= =20 warnings... > However, I agree that this might be an _option_, keeping the old way of > saving around. =C2=A1=C2=A1=C2=A1=C2=A1=C2=A1=C2=A1 exactly !!!!!! > Furthermore, it would be nice (even with the old method) if joe checked f= or > all errors you mentioned above and informed you if any of these happened. > This way even if the file was destroyed on the disk, it would remain in > joe's memory buffer and you could quickly make some disk space or save it > on another partition. =C2=A1=C2=A1=C2=A1=C2=A1=C2=A1=C2=A1=C2=A1=C2=A1=C2=A1=C2=A1=C2=A1 exactly = !!!!!!!!!! > Basically "disk full" is a condition that is not handled correctly by most > of the applications. In order for your system to work correctly, you have > to make sure you always have a reasonable amount of free space on your > disk. I must admid that I thought that joe handled "disk full" errors. Hmm... > BTW you cannot fsync after you've closed the file, so the order should be > fsync followed by close. > > On the other hand, I don't think sync'ing is joe's job. In fact I do think > it's not joe's job. It's the job of the OS to decide when to sync, to > highly increase performance.=20 Right. Or maybe it's even the filesystem's job. Sync'ing should not make a= =20 difference in the case of "disk full" - either the filesystem should tell y= ou=20 anyway, even before sync'ing or it should reserve the space before actually= =20 writing the data. I thought most filesystems would do so. Sync'ing should=20 only matter for power outages or a shutdown. > If your data and your work is really so=20 > extremely critical, then you should create regular backups (and use a > source code versioning system anyway),=20 I would really LOVE to see this as an option for joe: Be able to make numbered backups (FILENAME.~1~, FILENAME.~2~, ...) like ema= cs=20 can do! This is extremely useful esp. for software developing. Sometimes you screw = up=20 something without exactly knowing the problem, or you find out some time=20 later.=20 Some years ago I write a shell script around joe that made numbered backup= s=20 of everything I edited with joe and while being at it, corrected the dates,= =20 permissions and ownership (if possible) and took care of filename length=20 problems. Meanwhile I hacked joe itself to use the gnu fileutil 'cp' to copy the=20 original file onto itself with numbered backups forced on before saving -=20 this creates a backup with every metada copied. From the error message (if= =20 there is one) I can tell if making the backup worked. I personally cannot live without this feature. It saved my ass more than on= ce. Obvious disadvantage: It only works if the system has the gnu fileutils. And I was too lazy to make contortions to keep the other backup options, so= my=20 hacked joe can only make numbered backups or no backups at all. > buy a UPS, buy a larger disk and=20 > write a cron job that sends notification at 80% of usage, to make sure > you'll never run out of disk space.=20 Well, according to murphy's law you discover that you should have bought a = new=20 HD 5 minutes too late to get to the shop in time ;-)=20 > Neither of the standard utilities (cp,=20 > cat ...) sync the newly created files, and I guess text editors (vi ...) > don't do so either, so I can't see any reason why joe should. I only see a > drawback of syncing: save (or exit) time of joe might grow significally. Right. It's not joe's job. It's the filesystem's or the OS's job. Cheers Dirk |
From: Thorsten G. <tg...@mi...> - 2006-08-31 12:48:33
|
David Phillips dixit: >- close the temp file (check the return status, as NFS can error here) While I agree with your point (e.g. save to .filename.tmp, then rename filename into .filename.bak, then rename .filename.tmp to filename), I wonder if the "normal" backup file algorithm would be enough (probab- ly not, since you've exited joe already, causing your modifications to be lost, and, if saving twice, probably the backup file too), and, why are you using NFS instead of a local checkout and CVS? bye, //mirabile -- I believe no one can invent an algorithm. One just happens to hit upon it when God enlightens him. Or only God invents algorithms, we merely copy them. If you don't believe in God, just consider God as Nature if you won't deny existence. -- Coywolf Qi Hunt |
From: David P. <da...@ac...> - 2006-08-31 13:41:27
|
On 8/31/06, Koblinger Egmont <eg...@uh...> wrote: > There are drawbacks with this method as well: > [snip] Very good points. This is why I sent a message to the list instead of hacking on the code first. > Basically "disk full" is a condition that is not handled correctly by most > of the applications. In order for your system to work correctly, you have to > make sure you always have a reasonable amount of free space on your disk. Agreed. However, it does happen, and if there's one application that I want to be reliable, it's my text editor. > BTW you cannot fsync after you've closed the file, so the order should be > fsync followed by close. You're correct. For some reason I was thinking there was a version that worked on paths. > On the other hand, I don't think sync'ing is joe's job. In fact I do think > it's not joe's job. I disagree. A reliable application should sync when necessary. You save a file, causing JOE to overwrite the old one. A moment later, the machine crashes, the power goes off, etc. Your old version is already gone, and your new version does not yet exist. > If your data and your work is really so extremely > critical, then you should create regular backups (and use a source code > versioning system anyway), Of course. But how often do you commit? Are you happy to lose your latest changes? On 8/31/06, Dirk Schenkewitz <sch...@do...> wrote: > I would really LOVE to see this as an option for joe: > Be able to make numbered backups (FILENAME.~1~, FILENAME.~2~, ...) > like emacs can do! I agree, especially with the current -backpath limitation of storing all names in a single directory. On 8/31/06, Thorsten Glaser <tg...@mi...> wrote: > why are you using NFS instead of a local checkout and CVS? We use NFS mounted home directories on an EMC. It is far more reliable than a local disk drive, has separate daily backups, etc. However, that doesn't help when an application destroys the current day's data. |
From: Thorsten G. <tg...@mi...> - 2006-08-31 14:00:21
|
David Phillips dixit: >On 8/31/06, Thorsten Glaser <tg...@mi...> wrote: >> why are you using NFS instead of a local checkout and CVS? > >We use NFS mounted home directories on an EMC. It is far more >reliable than a local disk drive, has separate daily backups, etc. >However, that doesn't help when an application destroys the current >day's data. True, but do you have access to a local disc as well? In that case, I'd do my souce work on there and cvs commit when necessary. It's also much faster compiling locally than via NFS. //mirabile -- I believe no one can invent an algorithm. One just happens to hit upon it when God enlightens him. Or only God invents algorithms, we merely copy them. If you don't believe in God, just consider God as Nature if you won't deny existence. -- Coywolf Qi Hunt |
From: Koblinger E. <eg...@uh...> - 2006-08-31 14:23:45
|
On Thu, Aug 31, 2006 at 08:41:23AM -0500, David Phillips wrote: > I disagree. A reliable application should sync when necessary. > [...] > Of course. But how often do you commit? Are you happy to lose your > latest changes? No, I'm not happy. But this happens once a year or so. I'm still much happier with this than if I had to wait seconds after _each_ file save in my joe, which happens approx. a hundred times a day. The OS syncs the file in at most 30 seconds (in Linux, IIRC). So there's a gap of 30 seconcs after you save the file when a sync makes a difference in case of power outage. But how often do you save your file in joe? In every half minute? I don't think so. I think half hour is a better guess. And then comes an outage, and half hour of your work disappears. (If you had an UPS with an appropriate software, joe would save your work in DEADJOE.) So, as we can see, an autosave feature could make anyone much much happier than syncing. Syncing is a minor gain unnecesarily wasting system resources. If you still need it, you're free to mount your whole partition with the "sync" flag. But let's not make joe slower for such a minor gain. -- Egmont |