|
From: Szakacsits S. <sz...@si...> - 2004-02-25 12:18:40
|
Hello Pelle, I CC: linux-ntfs-dev because you made an interesting point. On Wed, 25 Feb 2004, Per Olofsson wrote: > I have a question regarding ntfsclone: Please feel free to ask anytime. I'm very interested what bothers users, what they think, what they find confusing, and so on. > Since ntfsclone uses sparse files, where holes represent unused blocks > (clusters) of the cloned partition, I thought it would be possible to > use this functionality to avoid writing unused blocks when restoring > an image to a hard disc partition. Right. Here is the simplest scenario: backup: ntfsclone -o mybackup.img /dev/hda1 restore: ntfsclone -O /dev/hda1 mybackup.img The restoration will not write the unused blocks back to the disk. Both action should take about the same time and this time is limited only by the maximal speed of the disk and how good the kernel is scheduling the disk IO. Disk seeks are minimalised by ntfsclone. > But as far as I can tell, sparse files are just a filesystem level > optimisation, Right. > and there is no way for programs to tell where the holes in a sparse > file actually are. Well, there is (FIBMAP ioctl) but it's not used because it's very inefficient. Only XFS supports this efficiently. I wrote about these in the ntfsclone manual. Did you check it out? http://linux-ntfs.sourceforge.net/man/ntfsclone.html > It seems like tar and cp just scans the file for large chunks of > zeroes and then assumes that they represent holes. Unfortunately true. It's also mentioned in the ntfsclone manual. > If this method is used to restore an image to a partition, I assume > that if there actually *is* files on the image containing longer > streams of zeroes, these will simply not get written to the target > partition, thus being replaced by whatever garbage was there > before. This is not true. There are only two ways to restore an image to a partition: 1) ntfsclone - it does the job correctly and saves the time writting back unused blocks. 2) dd - it writes _everything_. It will also write the zeros, thus it's slower. What about cp and tar? They are unable to restore an _image_ to a block device (partition), they will give an error when one tries to open the device. And now comes the interesting part. What if you want to copy/create an image with cp or tar with the -S option to a _file_? Can they mess up data? The answer is again no. Both will create a new file and if they detect there is somewhere only zeros then altough they skip that part in the target but the filesystem will set them 0. They will be also sparse files, no messed up data. I hope I could explain the issue, if not please give a specific example when you think the process would be unreliable. > And therefore, this use of ntfsclone is not reliable. Is this correct? It's not correct. I don't know many NTFS imaging softwares but I've heard several times from several sources that e.g. Ghost can trash NTFS. Partimage, Mondo (based on Partimage) is unreliable. I wrote about it here longer: http://www.partimage.org/forums/viewtopic.php?t=5 So the only tools I would trust at present to backup/restore NTFS is dd and ntfsclone. Well, actually I do use it for almost a year _very_ intensively because it's needed for development (backups and automatic testing all the time) and I've never had any problem. Others didn't report problems either. Szaka |
|
From: Per O. <pe...@ds...> - 2004-02-25 15:43:43
|
On Wed, Feb 25, 2004 at 13:09 +0100, Szakacsits Szabolcs wrote: > Right. Here is the simplest scenario: > > backup: ntfsclone -o mybackup.img /dev/hda1 > restore: ntfsclone -O /dev/hda1 mybackup.img > > The restoration will not write the unused blocks back to the disk. Oh, sorry, I overlooked this one for some reason. It indeed writes a sparse file image reliably. > Well, there is (FIBMAP ioctl) but it's not used because it's very > inefficient. Only XFS supports this efficiently. I wrote about these in > the ntfsclone manual. Did you check it out? Yes, I did, but you only talked about efficiency. I was more interested in whether it was possible to _reliably_ detect where holes in a sparse file are located. The method above which uses ntfsclone to restore the image doesn't look at where the holes are, it reads the actual NTFS filesystem to determine which parts to write onto the target device. > What about cp and tar? They are unable to restore an _image_ to a block > device (partition), they will give an error when one tries to open the > device. Okay, I thought that they would. In principle, they should be able to write to a block device, but I assume that they won't because of some built-in sanity checks then? > And now comes the interesting part. What if you want to copy/create an > image with cp or tar with the -S option to a _file_? Can they mess up > data? The answer is again no. Both will create a new file and if they > detect there is somewhere only zeros then altough they skip that part in > the target but the filesystem will set them 0. They will be also sparse > files, no messed up data. > > I hope I could explain the issue, if not please give a specific example > when you think the process would be unreliable. This is what I thought would be unreliable: cp --sparse=always sparse-image-file.img /dev/hda1 But since you're saying that cp won't allow that, this apparently isn't an issue. > > And therefore, this use of ntfsclone is not reliable. Is this correct? > > It's not correct. I don't know many NTFS imaging softwares but I've heard > several times from several sources that e.g. Ghost can trash NTFS. > Partimage, Mondo (based on Partimage) is unreliable. I wrote about it here > longer: Note that I was referring only to this specific potential use of ntfsclone, not ntfsclone in itself. My intent was not to say that ntfsclone is an unreliable program as such. Still, there is one drawback with ntfsclone. It's not possible, AFAIK, to restore images efficiently from a pipe. If you want to put an image on a web server, or more importantly, if you want to multicast an image to many computers, you need to have some kind of temporary storage on the computer(s) you're writing the image to. I have written a utility quite similar to ntfsclone which instead of sparse files uses "sparse images" to store the filesystem. The images are stored in a special format which explicitly marks the segments of the filesystem which are unused, and leaves those out the file. With this, you can do something like: On server: ntfsimage /dev/hda1 >image.simg On client: wget -O- http://server/image.simg | simgrestore /dev/hda1 And no unused blocks need to be transferred. Unfortunately I haven't spent much time on this utility so it's still buggy, although it works on simple test cases. It would be interesting to try to integrate this functionality into ntfsclone. -- Pelle |
|
From: Szakacsits S. <sz...@si...> - 2004-02-25 16:59:21
|
On Wed, 25 Feb 2004, Per Olofsson wrote: > > Well, there is (FIBMAP ioctl) but it's not used because it's very > > inefficient. Only XFS supports this efficiently. I wrote about these in > > the ntfsclone manual. Did you check it out? > > Yes, I did, but you only talked about efficiency. True, maybe I should move those parts to the end of the manual or just delete it completely, if it confuses/scares people. > > What about cp and tar? They are unable to restore an _image_ to a block > > device (partition), they will give an error when one tries to open the > > device. > > Okay, I thought that they would. In principle, they should be able to > write to a block device, but I assume that they won't because of some > built-in sanity checks then? AFAIR somebody wanted to or sent a patch for this to tar developers but I don't know the status. As for cp, I remembered it doesn't work. Now I tried again your example below on a lopback device and it worked. That's bad. One could misuse cp and indeed lose data, as you wrote. Well, I think I did only minimum one million times in my life 'cp floppy.img /dev/floppy'. Dumb me ... > This is what I thought would be unreliable: > > cp --sparse=always sparse-image-file.img /dev/hda1 > > But since you're saying that cp won't allow that, this apparently > isn't an issue. You were right. Unfortunately this is possible and nobody should ever use it. I must definitely mention this in the manual ... thank you! > Note that I was referring only to this specific potential use of > ntfsclone, not ntfsclone in itself. I wanted it to be the simplest to use and make it work easily with standard tools. The end result seems to be a bit confusing and potential misuse. I also don't like the syntax ntfsclone --[output|overwrite] [destination|-] source this would have been better ntfsclone [--overwrite] source [destination] But I can't change since this would be fatal: old version: ntfsclone --overwrite destination source new version: ntfsclone --overwrite source destination There is a synonym for 'overwrite'? I thought about using 'restore' but it;s not always that. It's really used as 'overwrite'. > Still, there is one drawback with ntfsclone. It's not possible, AFAIK, > to restore images efficiently from a pipe. If you want to put an image > on a web server, or more importantly, if you want to multicast an > image to many computers, you need to have some kind of temporary > storage on the computer(s) you're writing the image to. The 'temporary storage' isn't true but the 'efficiently' is true. E.g. ntfsclone -o - dev | gzip -2c | ssh -C host 'gunzip -c | dd of=dev' No temporary storage but it also doesn't have optimal network and disk usage. > I have written a utility quite similar to ntfsclone which instead of > sparse files uses "sparse images" to store the filesystem. The images > are stored in a special format which explicitly marks the segments of > the filesystem which are unused, and leaves those out the file. With > this, you can do something like: > > On server: > ntfsimage /dev/hda1 >image.simg > > On client: > wget -O- http://server/image.simg | simgrestore /dev/hda1 > > And no unused blocks need to be transferred. About one year ago, after lot of discussions with Ian Jackson, we agreed to make three different utilities: - ntfsimage and restoreimage - ntfsclone He wrote the first two, exactly what you wrote but never sent to us. So only ntfsclone left. I needed it badly for development to debug user issues (see --metadata option) and to do things as fast and efficient as possible (no much free time to play with NTFS). Full backup functionality came mostly as a side-effect. You can read about these discussions in the archive, just search for ntfsimage. http://marc.theaimsgroup.com/?l=linux-ntfs-dev&w=2 > Unfortunately I haven't spent much time on this utility so it's still > buggy, although it works on simple test cases. It would be interesting > to try to integrate this functionality into ntfsclone. Yes, there is definitely need for it. I also mentioned this earlier. I'll think about it again how it would be the best to progress :) Suggestions are of course also welcomed :) Szaka |
|
From: Szakacsits S. <sz...@si...> - 2004-02-25 19:47:07
|
On Wed, 25 Feb 2004, Szakacsits Szabolcs wrote: > On Wed, 25 Feb 2004, Per Olofsson wrote: > > > This is what I thought would be unreliable: > > > > cp --sparse=always sparse-image-file.img /dev/hda1 > > > > But since you're saying that cp won't allow that, this apparently > > isn't an issue. > > You were right. Unfortunately this is possible and nobody should ever use > it. I must definitely mention this in the ntfsclone manual ... thank you! I had some time to think about the issue, not just quickly answer email ;) Actually it's a bug in cp. What's cp job? To copy files reliable. Can it do that when used in the above case (block device as the destination)? No. Sure it's a bug. If cp detects sparseness in the source file it will skip on the destination file and leave the skipped data unchanged. Source and destination won't be the same. It wasn't a copy operation. cp should fill those skipped areas with zeros, well actually never skip in these cases. cp should check the file type of the destination and if it's block device then fallback to --sparse=never. % cp --version cp (coreutils) 5.0 Szaka |
|
From: Jim M. <ji...@me...> - 2004-03-07 23:42:18
|
Szakacsits Szabolcs <sz...@si...> wrote: >> > This is what I thought would be unreliable: >> > >> > cp --sparse=always sparse-image-file.img /dev/hda1 ... > Actually it's a bug in cp. What's cp job? To copy files reliable. Can it > do that when used in the above case (block device as the destination)? No. > > Sure it's a bug. If cp detects sparseness in the source file it will skip > on the destination file and leave the skipped data unchanged. Source and > destination won't be the same. It wasn't a copy operation. cp should fill > those skipped areas with zeros, well actually never skip in these cases. > > cp should check the file type of the destination and if it's block device > then fallback to --sparse=never. Thank you for the report. Did you actually run a command like this? cp --sparse=always sparse-image-file.img /dev/hda1 Did it produce any diagnostic? On what OS? I did the following on a linux-2.6.2 system: # dd bs=1 seek=100k of=a < /dev/null # cp --sparse=always a /dev/fd0 cp: writing `/dev/fd0': Invalid argument [Exit 1] The diagnostic was due to a failing ftruncate. Changing cp as you suggest is one way to solve this problem. Another would be to make cp fail when it's asked to do something that doesn't make sense: preserve holes in a non-regular destination file. I'll think it over... |
|
From: Szakacsits S. <sz...@si...> - 2004-03-08 00:24:20
|
On Mon, 8 Mar 2004, Jim Meyering wrote: > Did you actually run a command like this? > > cp --sparse=always sparse-image-file.img /dev/hda1 Yes and no. I thought this is impossible earlier (I remembered I tried it once) but Pelle suggested it can be done and I tested it on a loop device. It indeed worked, I mean the file was incorrectly copied. > Did it produce any diagnostic? On what OS? AFAIR Linux 2.4.latest (when I sent my report). > I did the following on a linux-2.6.2 system: > > # dd bs=1 seek=100k of=a < /dev/null > # cp --sparse=always a /dev/fd0 > cp: writing `/dev/fd0': Invalid argument > [Exit 1] > > The diagnostic was due to a failing ftruncate. Yes, this is what should happen as I remembered. But because I don't have floppy drives anymore, test partitions, etc I tried loop device. It worked so I thought I remembered wrong. On the other hand, I also remember a bunch of 'cp something /dev/fd0' instructions from many documents and if you search the net you can also find them. But I also remember, I never liked cp to a device because I've found it unreliable in general (I blamed bad floppies). This sparse issue could explain it (autodetect). Also, if the length of the SOURCE was the same as the device size then perhaps no ftruncate ... Sorry I don't know cp's history. Szaka |
|
From: Szakacsits S. <sz...@si...> - 2004-03-08 00:47:16
|
On Mon, 8 Mar 2004, Jim Meyering wrote:
>
> Changing cp as you suggest is one way to solve this problem.
> Another would be to make cp fail when it's asked to do
> something that doesn't make sense: preserve holes in a
> non-regular destination file.
I think the later one isn't correct:
1) it's not cp's job to preserve holes. It's an optimization
but it can not be done _always_.
2) people suppose cp works in the non-regular destination
file case. This would get broken with the later change.
BTW, if you're interested in large sparse file handling optimization
(cp, cat, etc) currently having horrible performance (coredumps, sparse
matrices, etc) then you might take a look at the relevant part (HANDLING
LARGE SPARSE FILES) of
http://linux-ntfs.sourceforge.net/man/ntfsclone.html
Szaka
|
|
From: Jim M. <ji...@me...> - 2004-03-08 15:09:50
|
Szakacsits Szabolcs <sz...@si...> wrote: > On Mon, 8 Mar 2004, Jim Meyering wrote: >> Changing cp as you suggest is one way to solve this problem. >> Another would be to make cp fail when it's asked to do >> something that doesn't make sense: preserve holes in a >> non-regular destination file. > > I think the later one isn't correct: > > 1) it's not cp's job to preserve holes. It's an optimization > but it can not be done _always_. > > 2) people suppose cp works in the non-regular destination > file case. This would get broken with the later change. Your comments make me wonder if my proposal was clear enough. Here it is, in more detail: I will probably change cp so that commands like this fail: cp --sparse=always SRC /dev/hda1 That is, if --sparse=always is given and the destination refers to an existing, non-regular file, then cp will fail. Note that cp's default behavior (with --sparse=auto, or with no --sparse=... option) works just fine in that it copies sequences of zeroes from SRC to the destination rather than trying to use lseek to create zero-filled holes. The problem under discussion arises only when --sparse=always is used and when the destination is a non-regular file like /dev/hda1. > BTW, if you're interested in large sparse file handling optimization > (cp, cat, etc) currently having horrible performance (coredumps, sparse > matrices, etc) then you might take a look at the relevant part (HANDLING > LARGE SPARSE FILES) of > > http://linux-ntfs.sourceforge.net/man/ntfsclone.html As you know, copying sparse files efficiently is not even possible on many file systems. If someone submits a clean patch to make things more efficient for some file system types, I'll be happy to consider it. |
|
From: Per O. <pe...@ds...> - 2004-02-25 18:14:32
|
On Wed, Feb 25, 2004 at 17:50 +0100, Szakacsits Szabolcs wrote: > > Still, there is one drawback with ntfsclone. It's not possible, AFAIK, > > to restore images efficiently from a pipe. If you want to put an image > > on a web server, or more importantly, if you want to multicast an > > image to many computers, you need to have some kind of temporary > > storage on the computer(s) you're writing the image to. > > The 'temporary storage' isn't true but the 'efficiently' is true. E.g. > > ntfsclone -o - dev | gzip -2c | ssh -C host 'gunzip -c | dd of=dev' > > No temporary storage but it also doesn't have optimal network and disk > usage. What I meant was that, as you say, you can't get efficiency without temporary storage. Of course, you don't need temporary storage if you use the inefficient approach. > About one year ago, after lot of discussions with Ian Jackson, we agreed > to make three different utilities: > > - ntfsimage and restoreimage > - ntfsclone > > He wrote the first two, exactly what you wrote but never sent to us. I see. The reason I never sent anything to you is that my utilities are just a hack and don't really work as they are now. Are ntfsimage and restoreimage available somewhere? > So only ntfsclone left. I needed it badly for development to debug user > issues (see --metadata option) and to do things as fast and efficient as > possible (no much free time to play with NTFS). Full backup functionality > came mostly as a side-effect. ... > Yes, there is definitely need for it. I also mentioned this earlier. I'll > think about it again how it would be the best to progress :) Suggestions > are of course also welcomed :) ntfsclone and ntfsimage do essentially the same thing, they skip over unused blocks. The output format is merely a detail and could be chosen with a command-line option. I think it would be better if they were one utility, in order to reduce duplicated efforts. -- Pelle |
|
From: Szakacsits S. <sz...@si...> - 2004-02-25 20:18:41
|
On Wed, 25 Feb 2004, Per Olofsson wrote: > I see. The reason I never sent anything to you is that my utilities > are just a hack That's ok :) > and don't really work as they are now. That's not so ok :) > Are ntfsimage and restoreimage available somewhere? No, AFAIK. The only ntfsimage reference google finds is here: http://packages.debian.org/changelogs/pool/main/c/chiark-utils/chiark-utils_4.0.0/changelog > ntfsclone and ntfsimage do essentially the same thing, they skip over > unused blocks. The output format is merely a detail and could be > chosen with a command-line option. I think it would be better if they > were one utility, in order to reduce duplicated efforts. I think that was also what I suggested. ntfsimage and ntfsrestore? Or just ntfsimage? I'm also quite sure when we would be ready Ian will also arrive saying "here is my ntfsimage I promised a year ago!" :) Merging the two using the name ntfsimage would solve my "ntfsclone syntax problem" also :) Szaka |
|
From: Per O. <pe...@ds...> - 2004-02-25 21:44:22
|
On Wed, Feb 25, 2004 at 21:09 +0100, Szakacsits Szabolcs wrote: > > On Wed, 25 Feb 2004, Per Olofsson wrote: > > > I see. The reason I never sent anything to you is that my utilities > > are just a hack > > That's ok :) > > > and don't really work as they are now. > > That's not so ok :) If you're interested, you can get the source from <http://dsv.su.se/~pelle/ntfsimage/>. It worked when I tried it on a filesystem that I had created with mkntfs, but it didn't work on a real Windows partition. I don't know if it actually compiles with the latest libntfs either. The format of the sparse images are documented in comments in the source code. The biggest problem with this code is that it doesn't do enough error checking. > I think that was also what I suggested. ntfsimage and ntfsrestore? Or just > ntfsimage? The point of not calling the restoration utility ntfsrestore is that it isn't really specific to NTFS - it could work with any filesystem. Only the image creator (ntfsimage) is specific to NTFS. > I'm also quite sure when we would be ready Ian will also arrive saying > "here is my ntfsimage I promised a year ago!" :) Surely :) > Merging the two using the name ntfsimage would solve my "ntfsclone syntax > problem" also :) Yes :) I'll have to look more at the ntfsclone source code and see if I can work out a patch, but I can't promise anything. My ntfsimage utility also has the feature of skipping the page file and the bad clusters file, which perhaps also could be added to ntfsclone as a command line option. -- Pelle |
|
From: Szakacsits S. <sz...@si...> - 2004-02-26 23:00:30
|
On Wed, 25 Feb 2004, Per Olofsson wrote: > If you're interested, you can get the source from > <http://dsv.su.se/~pelle/ntfsimage/>. OK but unfortunately I don't have much time for it. I'd prefer if you send working patches for integrating ntfsimage and ntfsclone and I'd be _very_ glad to revise, test, use it. > It worked when I tried it on a filesystem that I had created with > mkntfs, but it didn't work on a real Windows partition. Hmmm, if you go ahead I could take a look. > I don't know if it actually compiles with the latest libntfs either. I could also try this one. > The format of the sparse images are documented in comments in the > source code. The biggest problem with this code is that it doesn't do > enough error checking. Indeed. Error checking is very important, it's a must everywhere. > The point of not calling the restoration utility ntfsrestore is that > it isn't really specific to NTFS - it could work with any > filesystem. Only the image creator (ntfsimage) is specific to NTFS. Hmmm, Ian argued exactly the same way :) The problem, see e.g. how tools are used: gzip - gunzip bzip2 - bunzip2 compress - uncompress zip - unzip tar - tar xdelta - xdelta There is a pattern. ntfsimage - ntfsrestore makes sense. Just using ntfsimage to image/restore also makes sense. {ntfsimage, fooimage, barimage, ...} - simgrestore somehow doesn't really fit. Where will you distribute simgrestore? Everywhere? They will conflict. Separate package? That's a maintaince and user pain. > I'll have to look more at the ntfsclone source code and see if I can > work out a patch, In theory you should mostly look only at clone_ntfs() and integrate it there. > My ntfsimage utility also has the feature of skipping the page file I thought about that also. My problem is, I don't have knowledge about if the page file is allowed or it is possible to be used between reboots. E.g. Linux swap can definitely be used. So IMHO it's better to backup unnecessarily all the time than lose data potentially in some cases rarely. > and the bad clusters file, Good point! ntfsclone incorrectly tries to save them, I added this to the TODO for fixing. > which perhaps also could be added to ntfsclone as a command line > option. Yes, the later needs to be fixed and the pagefile could have an option. But I have one related question. How do you know where is the real page file? There can be several files called pagefile.sys on an NTFS volume. Szaka |
|
From: Per O. <pe...@ds...> - 2004-02-27 20:49:24
|
On Thu, Feb 26, 2004 at 23:50 +0100, Szakacsits Szabolcs wrote:
>
> OK but unfortunately I don't have much time for it. I'd prefer if you send
> working patches for integrating ntfsimage and ntfsclone and I'd be _very_
> glad to revise, test, use it.
Sure. As I said, I can't promise anything, but I might come up with
something.
> There is a pattern. ntfsimage - ntfsrestore makes sense. Just using
> ntfsimage to image/restore also makes sense.
>
> {ntfsimage, fooimage, barimage, ...} - simgrestore somehow doesn't really
> fit. Where will you distribute simgrestore? Everywhere? They will
> conflict. Separate package? That's a maintaince and user pain.
Good point. One advantage with having a separate utility, however, is
that it can be made very small and doesn't have to be linked to
libntfs. This can be useful for boot floppies where space is
constrained, but since we're moving away from using floppies I guess
this isn't as important anymore.
> In theory you should mostly look only at clone_ntfs() and integrate it
> there.
OK.
> I thought about that also. My problem is, I don't have knowledge about if
> the page file is allowed or it is possible to be used between reboots.
> E.g. Linux swap can definitely be used. So IMHO it's better to backup
> unnecessarily all the time than lose data potentially in some cases
> rarely.
Yes, that's a good point too. As I said, I haven't done any successful
real-world testing so I don't if it's necessary to preserve the page
file or not.
> But I have one related question. How do you know where is the real page
> file? There can be several files called pagefile.sys on an NTFS volume.
My program just looks in the root directory after it, so it doesn't
work with non-standard locations. But there can only be one
pagefile.sys in the root directory, right?
--
Pelle
|
|
From: Szakacsits S. <sz...@si...> - 2004-03-01 18:15:14
|
On Fri, 27 Feb 2004, Per Olofsson wrote: > Good point. One advantage with having a separate utility, however, is > that it can be made very small and doesn't have to be linked to > libntfs. This can be useful for boot floppies where space is > constrained, but since we're moving away from using floppies I guess > this isn't as important anymore. That's true. Floppies aren't really used anymore however space will matter for a while, e.g. now a commom limit is around 50 MB (USB stick, mini disk, etc) and people want to utilize it as efficiently as possible. > My program just looks in the root directory after it, so it doesn't > work with non-standard locations. But there can only be one > pagefile.sys in the root directory, right? Yes, only one case-sensitive. But there could be more case-insensitives. Which one is the right? Could its real location be somewhere else? Well, these last two questions are basically the same. Might pagefile.sys has data used between reboots, as sometimes Linux swap? Investigating the issue would take too much time and IMHO there are more important things to do. Of course this doesn't mean if one is interested in the answers is disallowed to research and investigate the issue. Actually I would be quite happy if one would have the time do it. Szaka |
|
From: Per O. <pe...@ds...> - 2004-03-01 19:19:46
|
On Mon, Mar 01, 2004 at 19:02 +0100, Szakacsits Szabolcs wrote: > Might pagefile.sys has data used between reboots, as sometimes Linux swap? The only persistent use of Linux swap I know of is software suspend. But hibernation on Windows is done to hiberfil.sys or something like that, I think. There could be other uses of pagefile.sys of course. > Investigating the issue would take too much time and IMHO there are more > important things to do. I agree. -- Pelle |
|
From: Szakacsits S. <sz...@si...> - 2004-03-24 14:03:29
|
On Mon, 1 Mar 2004, Szakacsits Szabolcs wrote: > Might pagefile.sys has data used between reboots, as sometimes Linux swap? Veteran Microsoft engineer writes to his blog at http://blogs.msdn.com/larryosterman/archive/2004/03/18/92010.aspx Nowadays, NT doesn't always write the entire contents of memory out - it's controlled by a setting in the Startup and Recovery settings dialog on the Advanced tab of the system control panel applet - there are 4 choices - None, a small "minidump", a Kernel memory dump and a full memory dump. Only the full memory dump will write all of RAM, the others limit the amount of memory that's written out. But it still goes to the paging file. Of course I believe when I checked it out, although I suspected this. And since there is this often used thus robust feature then perhaps it's also used in other cases ... This also lead again to a very important question: how do we know if Windows crashed and NTFS is dirty? I thought the VOLUME_IS_DIRTY flag is supposed to mark this. I checked it out and it doesn't. After a crash it isn't set. Apparently it isn't even set when NTFS is mounted. Szaka |
|
From: Anton A. <ai...@ca...> - 2004-03-24 14:48:13
|
On Wed, 2004-03-24 at 14:03, Szakacsits Szabolcs wrote: > On Mon, 1 Mar 2004, Szakacsits Szabolcs wrote: > > > Might pagefile.sys has data used between reboots, as sometimes Linux swap? > > Veteran Microsoft engineer writes to his blog at > > http://blogs.msdn.com/larryosterman/archive/2004/03/18/92010.aspx > > Nowadays, NT doesn't always write the entire contents of memory out - > it's controlled by a setting in the Startup and Recovery settings dialog > on the Advanced tab of the system control panel applet - there are 4 > choices - None, a small "minidump", a Kernel memory dump and a full > memory dump. Only the full memory dump will write all of RAM, the others > limit the amount of memory that's written out. But it still goes to the > paging file. > > Of course I believe when I checked it out, although I suspected this. > And since there is this often used thus robust feature then perhaps > it's also used in other cases ... Possible. Certainly, now we know that we really should always copy the pagefile in ntfsclone, too... > This also lead again to a very important question: how do we know if > Windows crashed and NTFS is dirty? Really, the only way is to scan $LogFile to determine the state. If the log is fully check pointed and no uncommitted transactions are remaining it means the volume is clean. And otherwise it means it is dirty. I have been planning to implement this in ntfsprogs (and in the kernel for that matter) but haven't had time as it is rather a lot of work. > I thought the VOLUME_IS_DIRTY flag is supposed to mark this. I checked it > out and it doesn't. After a crash it isn't set. Apparently it isn't even > set when NTFS is mounted. I think it does but I think it is only set while data is being actively written to disk (I may well be wrong though). Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/ |
|
From: Szakacsits S. <sz...@si...> - 2004-03-25 09:15:30
|
On Wed, 24 Mar 2004, Anton Altaparmakov wrote: > Really, the only way is to scan $LogFile to determine the state. If the > log is fully check pointed and no uncommitted transactions are remaining > it means the volume is clean. And otherwise it means it is dirty. I > have been planning to implement this in ntfsprogs (and in the kernel for > that matter) but haven't had time as it is rather a lot of work. Why is it a lot of work? Unknown? Just a "clean" or "dirty" answer would be enough. No need to redo or undo the log/update records. I think this would be the most important next step to solve. > > I thought the VOLUME_IS_DIRTY flag is supposed to mark this. I checked it > > out and it doesn't. After a crash it isn't set. Apparently it isn't even > > set when NTFS is mounted. > I think it does but I think it is only set while data is being actively > written to disk (I may well be wrong though). So we could check VOLUME_IS_DIRTY and $LogFile then mount only if none is dirty. Szaka |
|
From: Anton A. <ai...@ca...> - 2004-03-25 09:28:27
|
On Thu, 25 Mar 2004, Szakacsits Szabolcs wrote: > On Wed, 24 Mar 2004, Anton Altaparmakov wrote: > > Really, the only way is to scan $LogFile to determine the state. If the > > log is fully check pointed and no uncommitted transactions are remaining > > it means the volume is clean. And otherwise it means it is dirty. I > > have been planning to implement this in ntfsprogs (and in the kernel for > > that matter) but haven't had time as it is rather a lot of work. > > Why is it a lot of work? Unknown? > > Just a "clean" or "dirty" answer would be enough. No need to redo or > undo the log/update records. Oh, of course. I have absolutely no intention of implementing redo/undo of the journal. Or at least not until we have a complete ntfs rw library and driver, then maybe I will look into it but otherwise no way... The difficulty is that there is no single bit saying "journal clean" or "journal dirty". You have to actually parse the entirety of $LogFile reading every single record, parsing each of them, and finding the last written record. Then you have to backtrace what you did till you find the last checkpointed record and then I am not too sure. Probably something along the lines of "if any records requiring undo or redo are present after the last checkpoint record assume journal is dirty otherwise assume it is clean". When we find it is clean we then proceed to empty the journal as we do now so that a Windows boot doesn't gratuitously perform a redo pass on us and corrupt something. This is a lot of code most of which requires existing $LogFile analysis and further reverse engineering. This is not a piece of cake... Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/ |
|
From: Szakacsits S. <sz...@si...> - 2004-04-08 20:19:00
|
On Thu, 25 Mar 2004, Anton Altaparmakov wrote:
> The difficulty is that there is no single bit saying "journal clean" or
> "journal dirty".
On NTFS 3.1 in the 'flags' field of the restart area, the bit 1 (starting
from 0) tells apparently if NTFS is supposed to be clean or not. If the
value is set ('flags' field is 2) then NTFS is "clean", or at least it was
shutdown properly.
Confirmations? Rebuttals?
However all comments and guesses related to dirty/clean NTFS I've seen in
the NTFS doc and in the sources are bogus (at least on NTFS 3.1).
Szaka
|
|
From: Anton A. <ai...@ca...> - 2004-04-09 10:36:18
|
On Thu, 8 Apr 2004, Szakacsits Szabolcs wrote:
> On Thu, 25 Mar 2004, Anton Altaparmakov wrote:
> > The difficulty is that there is no single bit saying "journal clean" or
> > "journal dirty".
>
> On NTFS 3.1 in the 'flags' field of the restart area, the bit 1 (starting
> from 0) tells apparently if NTFS is supposed to be clean or not. If the
> value is set ('flags' field is 2) then NTFS is "clean", or at least it was
> shutdown properly.
>
> Confirmations? Rebuttals?
Er who says that? It certainly isn't the case for LFS 1.1 (the only one I
know about). I believe XP uses LFS 1.1, too (as do NT4 and Win2k).
The easiest way to find out which version of LFS you are looking at is to
run ntfsdump_logfile /dev/partition | more or ntfsdump_logfile -f
ntfs_image_file | more. You need the more as the log record stuff is a
lot of bogus crap at the moment and it outputs tons of rubbish. But I
have been doing lots of work on the restart pages parts and what I say
there should be correct and the output from ntfsdump_logfile regarding the
restart pages should be correct, too.
> However all comments and guesses related to dirty/clean NTFS I've seen in
> the NTFS doc and in the sources are bogus (at least on NTFS 3.1).
Have you looked at the most recent include/ntfs/logfile.h? That contains
a lot of good info about the restart pages and how dirty/clean can be
distinguished. And that is I believe correct for LFS 1.1.
If you believe they are bogus then please let me know which bits and
why...
Best regards,
Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/
|
|
From: Szakacsits S. <sz...@si...> - 2004-04-09 12:42:17
|
On Fri, 9 Apr 2004, Anton Altaparmakov wrote:
> On Thu, 8 Apr 2004, Szakacsits Szabolcs wrote:
> > On Thu, 25 Mar 2004, Anton Altaparmakov wrote:
> > > The difficulty is that there is no single bit saying "journal clean" or
> > > "journal dirty".
> >
> > On NTFS 3.1 in the 'flags' field of the restart area, the bit 1 (starting
> > from 0) tells apparently if NTFS is supposed to be clean or not. If the
> > value is set ('flags' field is 2) then NTFS is "clean", or at least it was
> > shutdown properly.
> >
> > Confirmations? Rebuttals?
>
> Er who says that?
Apparently NTFS 3.1 says so. You never checked?
> It certainly isn't the case for LFS 1.1 (the only one I know about).
> I believe XP uses LFS 1.1, too (as do NT4 and Win2k).
AFAIK, the only LFS user is NTFS. NTFS 3.0 and earlier isn't able to
access/mount NTFS 3.1. So NTFS 3.1 LFS version could be anything.
Moreover this feature is backward compatible. What happens if 3rd party
softwares can't interpret the flag? They waste time trying to recover a
consistent NTFS until they find so.
Microsoft invested a lot to improve NTFS for version 3.1 and the boot
time. Probably it was realised it's more efficient just mark the volume
clean if it was clean at shutdown and make a quick check for it at boot
time than parsing several times (redo, undo, etc) 50-100+ MB journal files
needlessly.
> The easiest way to find out which version of LFS you are looking at is to
> run ntfsdump_logfile /dev/partition | more or ntfsdump_logfile -f
> ntfs_image_file | more. You need the more as the log record stuff is a
> lot of bogus crap at the moment and it outputs tons of rubbish. But I
> have been doing lots of work on the restart pages parts and what I say
> there should be correct and the output from ntfsdump_logfile regarding the
> restart pages should be correct, too.
Yes, people could help out here also:
download/install ntfsprogs
./configure
make install
cd ntfsprogs
make ntfsdump_logfile
cp .libs/ntfsdump_logfile /usr/bin/
repeatedly crash (make an 'ntfsclone' just for your extra safety) and
shutdown Windows properly then run
ntfsdump_logfile partition | head -40 | grep flags
If Windows is XP/W2K3/Longhorn then
flags = 2 apparently means clean shutdown, no recovery needed
flags = 0 means unknows state, recovery needed
> > However all comments and guesses related to dirty/clean NTFS I've seen in
> > the NTFS doc and in the sources are bogus (at least on NTFS 3.1).
>
> Have you looked at the most recent include/ntfs/logfile.h? That contains
> a lot of good info about the restart pages and how dirty/clean can be
> distinguished. And that is I believe correct for LFS 1.1.
Yes, I meant for those with NTFS 3.1 and
http://linux-ntfs.sourceforge.net/ntfs/files/logfile.html
http://linux-ntfs.sourceforge.net/ntfs/concepts/restart.html
> If you believe they are bogus then please let me know which bits and
> why...
All comments because they are always the same on NTFS 3.1 independently if
Windows crashed or not.
Szaka
|
|
From: Anton A. <ai...@ca...> - 2004-04-09 13:22:55
|
On Fri, 9 Apr 2004, Szakacsits Szabolcs wrote:
> On Fri, 9 Apr 2004, Anton Altaparmakov wrote:
> > On Thu, 8 Apr 2004, Szakacsits Szabolcs wrote:
> > > On Thu, 25 Mar 2004, Anton Altaparmakov wrote:
> > > > The difficulty is that there is no single bit saying "journal clean" or
> > > > "journal dirty".
> > >
> > > On NTFS 3.1 in the 'flags' field of the restart area, the bit 1 (starting
> > > from 0) tells apparently if NTFS is supposed to be clean or not. If the
> > > value is set ('flags' field is 2) then NTFS is "clean", or at least it was
> > > shutdown properly.
> > >
> > > Confirmations? Rebuttals?
> >
> > Er who says that?
>
> Apparently NTFS 3.1 says so. You never checked?
What do you mean by NTFS 3.1? Do you mean the NTFS driver that comes with
XP? The one with XPSP1? The one with XPSP2? The one with 2003 or
Longhorn? Note all these drivers are things like version 5.1, etc. NTFS
3.1 is the NTFS specification not the driver. Also the NTFS version is
not tied to the LFS version in any way... They are entirely separate
drivers and NTFS driver registers as a client to the LFS driver. I do not
believe NTFS knows anything at all about LFS versions.
Certainly on my XPSP1 test system, I see differences between crashed NTFS
and cleanly shutdown NTFS in the $LogFile restart pages. And the
differences are as I have described in include/ntfs/logfile.h.
> > It certainly isn't the case for LFS 1.1 (the only one I know about).
> > I believe XP uses LFS 1.1, too (as do NT4 and Win2k).
>
> AFAIK, the only LFS user is NTFS. NTFS 3.0 and earlier isn't able to
> access/mount NTFS 3.1. So NTFS 3.1 LFS version could be anything.
Yes, LFS allows only one client registration per logfile. And yes only
NTFS uses LFS I believe. NTFS 3.1 LFS makes no sense. NTFS and LFS are
quite separate things...
> Moreover this feature is backward compatible. What happens if 3rd party
> softwares can't interpret the flag? They waste time trying to recover a
> consistent NTFS until they find so.
>
> Microsoft invested a lot to improve NTFS for version 3.1 and the boot
> time. Probably it was realised it's more efficient just mark the volume
> clean if it was clean at shutdown and make a quick check for it at boot
> time than parsing several times (redo, undo, etc) 50-100+ MB journal files
> needlessly.
>
> > The easiest way to find out which version of LFS you are looking at is to
> > run ntfsdump_logfile /dev/partition | more or ntfsdump_logfile -f
> > ntfs_image_file | more. You need the more as the log record stuff is a
> > lot of bogus crap at the moment and it outputs tons of rubbish. But I
> > have been doing lots of work on the restart pages parts and what I say
> > there should be correct and the output from ntfsdump_logfile regarding the
> > restart pages should be correct, too.
>
> Yes, people could help out here also:
>
> download/install ntfsprogs
> ./configure
> make install
> cd ntfsprogs
> make ntfsdump_logfile
> cp .libs/ntfsdump_logfile /usr/bin/
>
> repeatedly crash (make an 'ntfsclone' just for your extra safety) and
> shutdown Windows properly then run
>
> ntfsdump_logfile partition | head -40 | grep flags
Also make sure to crash windows at least once while idle and at least once
while it is doing active i/o. (E.g. use explorer to start copying your
Program Files directory to your temp directory or something like that and
when your HD is going like mad, press the reset button.)
> If Windows is XP/W2K3/Longhorn then
>
> flags = 2 apparently means clean shutdown, no recovery needed
> flags = 0 means unknows state, recovery needed
I have never seen the flags be anything other than 0 on XPSP1 and W2K.
> > > However all comments and guesses related to dirty/clean NTFS I've seen in
> > > the NTFS doc and in the sources are bogus (at least on NTFS 3.1).
> >
> > Have you looked at the most recent include/ntfs/logfile.h? That contains
> > a lot of good info about the restart pages and how dirty/clean can be
> > distinguished. And that is I believe correct for LFS 1.1.
>
> Yes, I meant for those with NTFS 3.1 and
>
> http://linux-ntfs.sourceforge.net/ntfs/files/logfile.html
> http://linux-ntfs.sourceforge.net/ntfs/concepts/restart.html
>
> > If you believe they are bogus then please let me know which bits and
> > why...
>
> All comments because they are always the same on NTFS 3.1 independently if
> Windows crashed or not.
Well, they are not the same at all on XPSP1, at least not on mine! They
are only the same if the system was completely idle when it crashed (i.e.
the log file is fully checkpointed and no i/o is outstanding).
Best regards,
Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/
|
|
From: Szakacsits S. <sz...@si...> - 2004-04-09 14:43:44
|
On Fri, 9 Apr 2004, Anton Altaparmakov wrote:
> On Fri, 9 Apr 2004, Szakacsits Szabolcs wrote:
> >
> > Apparently NTFS 3.1 says so. You never checked?
>
> What do you mean by NTFS 3.1? Do you mean the NTFS driver that comes with
> XP? The one with XPSP1? The one with XPSP2? The one with 2003 or
> Longhorn? Note all these drivers are things like version 5.1, etc. NTFS
> 3.1 is the NTFS specification not the driver.
I mean NTFS 3.1. So I mean all drivers that supports NTFS 3.1. But note,
again, this feature doesn't break anything. If a driver knows about this
flag/feature then it can query the state otherwise it does recovery, just
as before.
If you like, forget everything about NTFS versions and distinguishing NTFS
and LFS. Just focus on that bit. It would be backward compatible, there
would have been no need to bump up LFS version because that could break
compatibility.
> Certainly on my XPSP1 test system, I see differences between crashed NTFS
> and cleanly shutdown NTFS in the $LogFile restart pages. And the
> differences are as I have described in include/ntfs/logfile.h.
That's interesting since on my XPSP1 none of the dirty related comments
are right.
Ok, let's take those comments in order:
RESTART_AREA:
"client_free_list At present on dirty volume this is 0xffff."
Always -1.
"client_in_use_list On a clean volume this is 0xffff. On a dirty
volume this is != 0xffff. At present on dirty
volume this is 0."
Always 0.
"unknown; It seems like it = 0 if clean and it != 0
if dirty."
This seems to be a counter related to the total number of checkpoints or
number of event logs or something like that.
LOG_CLIENT_RECORD:
"seq_number; ??? It is 1 when clean and 0 when dirty,
but don't know if this is always the case."
Always 0.
> > > It certainly isn't the case for LFS 1.1 (the only one I know about).
> > > I believe XP uses LFS 1.1, too (as do NT4 and Win2k).
> >
> > AFAIK, the only LFS user is NTFS. NTFS 3.0 and earlier isn't able to
> > access/mount NTFS 3.1. So NTFS 3.1 LFS version could be anything.
>
> Yes, LFS allows only one client registration per logfile. And yes only
> NTFS uses LFS I believe. NTFS 3.1 LFS makes no sense. NTFS and LFS are
> quite separate things...
True from pedantry point of view but I doubt even Microsoft cares. Show me
an example when LFS is used outside of NTFS.
> Also make sure to crash windows at least once while idle and at least once
> while it is doing active i/o. (E.g. use explorer to start copying your
> Program Files directory to your temp directory or something like that and
> when your HD is going like mad, press the reset button.)
That's not ok. Some cases send reset notifications to the OS and they can
do a fast flush. Just unplug the power cable (for your own risk).
Of course you could also sniff the journal events by Captive NTFS, VMware,
etc.
> > If Windows is XP/W2K3/Longhorn then
> >
> > flags = 2 apparently means clean shutdown, no recovery needed
> > flags = 0 means unknows state, recovery needed
>
> I have never seen the flags be anything other than 0 on XPSP1 and W2K.
OK but that's not rebuttal.
Szaka
|
|
From: Anton A. <ai...@ca...> - 2004-04-09 21:33:42
|
On Fri, 9 Apr 2004, Szakacsits Szabolcs wrote: > On Fri, 9 Apr 2004, Anton Altaparmakov wrote: > > On Fri, 9 Apr 2004, Szakacsits Szabolcs wrote: > > > Apparently NTFS 3.1 says so. You never checked? > > > > What do you mean by NTFS 3.1? Do you mean the NTFS driver that comes with > > XP? The one with XPSP1? The one with XPSP2? The one with 2003 or > > Longhorn? Note all these drivers are things like version 5.1, etc. NTFS > > 3.1 is the NTFS specification not the driver. > > I mean NTFS 3.1. So I mean all drivers that supports NTFS 3.1. But note, > again, this feature doesn't break anything. If a driver knows about this > flag/feature then it can query the state otherwise it does recovery, just > as before. > > If you like, forget everything about NTFS versions and distinguishing NTFS > and LFS. Just focus on that bit. It would be backward compatible, there > would have been no need to bump up LFS version because that could break > compatibility. True. Although the LFS drivers are quite happy to recognise any LFS version including strange ones likes 0.-1... > > Certainly on my XPSP1 test system, I see differences between crashed NTFS > > and cleanly shutdown NTFS in the $LogFile restart pages. And the > > differences are as I have described in include/ntfs/logfile.h. > > That's interesting since on my XPSP1 none of the dirty related comments > are right. > > Ok, let's take those comments in order: > > RESTART_AREA: > > "client_free_list At present on dirty volume this is 0xffff." > > Always -1. > > "client_in_use_list On a clean volume this is 0xffff. On a dirty > volume this is != 0xffff. At present on dirty > volume this is 0." > Always 0. > > "unknown; It seems like it = 0 if clean and it != 0 > if dirty." > > This seems to be a counter related to the total number of checkpoints or > number of event logs or something like that. > > LOG_CLIENT_RECORD: > > "seq_number; ??? It is 1 when clean and 0 when dirty, > but don't know if this is always the case." > > Always 0. > > > > > It certainly isn't the case for LFS 1.1 (the only one I know about). > > > > I believe XP uses LFS 1.1, too (as do NT4 and Win2k). > > > > > > AFAIK, the only LFS user is NTFS. NTFS 3.0 and earlier isn't able to > > > access/mount NTFS 3.1. So NTFS 3.1 LFS version could be anything. > > > > Yes, LFS allows only one client registration per logfile. And yes only > > NTFS uses LFS I believe. NTFS 3.1 LFS makes no sense. NTFS and LFS are > > quite separate things... > > True from pedantry point of view but I doubt even Microsoft cares. Show me > an example when LFS is used outside of NTFS. I don't know any. I don't think anything other than NTFS uses LFS. > > Also make sure to crash windows at least once while idle and at least once > > while it is doing active i/o. (E.g. use explorer to start copying your > > Program Files directory to your temp directory or something like that and > > when your HD is going like mad, press the reset button.) > > That's not ok. Some cases send reset notifications to the OS and they can > do a fast flush. Just unplug the power cable (for your own risk). > > Of course you could also sniff the journal events by Captive NTFS, VMware, > etc. > > > > If Windows is XP/W2K3/Longhorn then > > > > > > flags = 2 apparently means clean shutdown, no recovery needed > > > flags = 0 means unknows state, recovery needed > > > > I have never seen the flags be anything other than 0 on XPSP1 and W2K. > > OK but that's not rebuttal. I have to apologise at this point: I must have somehow managed to muddle my logfile copies. I took a new copy just now and it is just like you say and nothing like my old copy. So I must have been working with w2k rather than xp logfile all along. )-: I can confirm the same as you see. This makes everything even more difficult. We will need to parse the NTFS version and then deal with the logfile depending on what we see there. So on NTFS 3.1 with LFS 1.1, we will compare the restart pages (_not_ binary compare but logical compare) and if identical and flags = 2 we can say clean and dirty otherwise. And on NTFS 1.x, 2.x, and 3.0 with LFS 1.1 we need to check the restart pages and if identical and the criteria I describe in ntfs/logfile.h are met (i.e. the logfile is shut, no clients are registered in use) it is clean and assumed dirty otherwise (although it could still be clean in that case but that is too complicated for us to figure out at this point as it involves parsing the whole logfile and checking that all ops are checkpointed). Anyway, it is better to be erring on the safe side IMO. We can always provide a force option for ignoring the fact that we think the journal is dirty. Without the force we just work read-only. The above should cover all NTFS and LFS versions in use outside of MS and perhaps beta-testers of early windows nt 3.x versions but I suspect there aren't any of those around nowadays... Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/ |
