fstransform-devel Mailing List for file-system transformation tool
Status: Beta
Brought to you by:
paperinik
Menu
▾
▴
fstransform-devel — general discussion about fstransform, including features, bugs and clarification
You can subscribe to this list here.
| 2014 |
Jan
(3) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(4) |
Nov
(12) |
Dec
|
|---|
|
From: Massimiliano G. <mas...@gm...> - 2014-11-23 22:06:35
|
On 11/23/14 19:48, Tydyn Rain St. Clair wrote:> Greetings, > > I was using fstransform via an Ubuntu Live CD to convert my installed > Ubuntu system from XFS to EXT4. It had an error, and I accidentally > canceled the operation. Now my entire file system is locked up somewhere > and I have no particular clue of how to restore it back to the hard drive. > fsremap cannot continue the operation because it can't find the log file in > /var/temp/fstransform. I've included said log file. Is there any way to > restore the file system with my Ubuntu installation at this point, or > should I just reformat and reinstall? > > Thanks for any help you could provide. > Hello, the log file you sent was crucial to solve the problem :) Your data can be recovered easily :) A word of warning: using fstransform from a Live CD is VERY dangerous, as Live CDs normally do not preserve *any* data across reboots! In case of power failure, the logs needed to resume fsremap would be *lost*, and data being converted by fsremap becomes basically unrecoverable. A proper solution is to add a "persistence" device to the Live CD, so that any modified file, including logs, will be preserved across reboots. The error that caused fstransform to abort was that Ubuntu took the liberty of removing the directory /media/ubuntu/Ubuntu when fstransform unmounted your device, so fstransform could not mount it again in the same place and continue. There is no fsremap log file because the error happened BEFORE starting fsremap. If fstransform is still running, the solution is to recreate the directory /media/ubuntu/Ubuntu, mount read-only your /dev/sda3 on it and type CONTINUE at fstrasform prompt. Since I guess fstransform is no longer running, resuming is a bit more complicated. > 2014-11-23 15:29:08 fstransform: preparing to transform device '/dev/sda3' to file-system type 'ext4' > 2014-11-23 15:29:08 fstransform: device is mounted at '/media/ubuntu/Ubuntu' with file-system type 'xfs' > ... > 2014-11-23 15:29:08 fstransform: connected loop device '/dev/loop1' to file '/media/ubuntu/Ubuntu/.fstransform.loop.9508' > ... > 2014-11-23 17:09:30 fstransform: mounting again device '/dev/sda3' read-only > 2014-11-23 17:09:30 mount: mount point /media/ubuntu/Ubuntu does not exist All your files are currently stored inside the EXT4-formatted file ".fstransform.loop.9508" inside the XFS-formatted device /dev/sda3. To access them, run as root the commands: mkdir /mnt # this may fail, directory may exists already mkdir /tmp/fstransform.loop mount /dev/sda3 /mnt -o ro losetup /dev/loop2 /mnt/.fstransform.loop.9508 mount /dev/loop2 /tmp/fstransform.loop and your files will be in /tmp/fstransform.loop: ls -l /tmp/fstransform.loop Note that I added "-o ro" to the mount command above. This will let you access your data read-only, which is needed to resume the transformation. You can remove that option if you want to access your data read-write *instead* of converting it. To resume the conversion, run *after* the commands above (make sure to add persistence before running them if you use a Live CD!): umount /tmp/fstransform.loop losetup -d /dev/loop2 fsremap /dev/sda3 /mnt/.fstransform.loop.9508 Regards, Massimiliano Ghilardi |
|
From: Olaf L. <ol...@me...> - 2014-11-19 00:25:50
|
Heya! > Googling a bit, this thread contains some advice - "lowering all the > dirty memory thresholds": > http://oss.sgi.com/archives/xfs/2013-08/msg00733.html > > Pity, I hoped the on-disk inode cache would be enough to support this > specific case. Yeah, I guess it's not really worth the effort to wait for weeks or months. It's more important, that the 2nd backup server starts backuping again. > Regards, > > Massimiliano > > > P.S. was /spare/inode_cache on a fast disk too? > Well, not a blazing-fast disk - a regular SATA HDD (hdparm -t yields 60 MiB/s) connected to the internal controller which is maybe 4 years old. |
|
From: Massimiliano G. <mas...@gm...> - 2014-11-18 23:53:56
|
On 11/18/14 23:18, Olaf Leidinger wrote:> Am Tue, 18 Nov 2014 23:01:26 +0100 > schrieb Massimiliano Ghilardi <mas...@gm...>: > >> On 11/18/14 22:34, Olaf Leidinger wrote: >>> [...] >>> I guess the RAID6 array has some problem and needs to be rebuilt. >>> >>> Best, >>> >>> Olaf >> >> That's a BIG RED flag. >> >> I wouldn't suggest running fstransform on a _degraded_ RAID disk even >> to my enemies... fstransform is a BIG gun and must be used with care. >> Using it on such a setup verges on being suicidal. > > Oh, no, this isn't what I wanted to say. The raid controller reports > perfect health conditions. I just meant, that there might be something > wrong internally (which isn't reported) due to the slow speed. But as > raw-reads (hdparm -t) report a nice speed of 100 MiB/s, I guess it's > XFS fault, not the fault of the underlying block layer. > Hello, this is much better - I mean, for the 4TB data on your 12TB RAID :) But even in the ideal case the conversion time would be at least 2-3 days, plus 1-2 more days to fill with zeroes the 8GB of free space. from your logs I can deduce something more: > 08:22:42 fsmove: progress: 0.3% done, 4.2 terabytes still to preallocate > 08:23:04 fsmove: progress: 0.6% done, 4.1 terabytes still to preallocate, estimated 1 hour and 25 minutes left > [...] > 08:42:00 fsmove: progress: 4.9% done, 4.0 terabytes still to preallocate, estimated 4 hours left > 08:49:40 fsmove: progress: 5.2% done, 3.9 terabytes still to preallocate, estimated 3 hours left Until here, it was reasonably fast: 5% of fsmove in 30 minutes. > 08:50:45 fsmove: progress: 5.5% done, 3.9 terabytes still to preallocate, estimated 1 hour and 40 minutes left > [...] > 09:50:06 fsmove: progress: 7.0% done, 3.9 terabytes still to preallocate, estimated 5 hours left here it slows down: 1 hour to go from 5% to 7% > 10:08:55 fsmove: progress: 7.4% done, 3.9 terabytes still to preallocate, estimated 1 day and 7 hours left > [...] > 10:34:43 fsmove: progress: 8.0% done, 3.8 terabytes still to preallocate, estimated 1 day and 20 hours left more slowdown: 45 minutes to go from 7% to 8% > 11:00:49 fsmove: progress: 8.3% done, 3.8 terabytes still to preallocate, estimated 1 day and 15 hours left > [...] > 15:05:12 fsmove: progress: 11.0% done, 3.7 terabytes still to preallocate, estimated 5 days left and it only gets worse. > (canceled at this point, as the log was spammed with these XFS messages all over, again) In my opinion, the slowness is a combination of the huge number of hard links and this kernel error message: > XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) Googling a bit, this thread contains some advice - "lowering all the dirty memory thresholds": http://oss.sgi.com/archives/xfs/2013-08/msg00733.html Pity, I hoped the on-disk inode cache would be enough to support this specific case. Regards, Massimiliano P.S. was /spare/inode_cache on a fast disk too? |
|
From: Olaf L. <ol...@me...> - 2014-11-18 22:16:27
|
Hey! > The good news is that by using preallocation, the original file > system was not modified at all when you stopped the conversion > (except for the huge sparse file /mirror/.fstransform.loop.682). That's quite good to hear. Actually, fstransform cleaned up nicely and deleted the loop file afterwards. > The bad news is that fstransform with preallocation is MUCH faster > than normal conversion, and if that one was taking N months, the > normal conversion will take about N*2 months. > > [...] > Unluckily, your 4.2 terabytes disk looks quite slower than that > - but maybe it was due to the XFS errors, or maybe due to the huge number > of hard links. I guess it must be the number of hardlinks. The filesystem was quite fast when initially being created. And hdparm -t still yields ~ 100 MiB/s. Thanks for all the help and suggestions. Even I can't really use fstransform in this special case, it's sure a very nice tool to have! Best, Olaf |
|
From: Massimiliano G. <mas...@gm...> - 2014-11-18 22:02:14
|
On 11/18/14 22:34, Olaf Leidinger wrote: >[...] > I guess the RAID6 array has some problem and needs to be rebuilt. > > Best, > > Olaf That's a BIG RED flag. I wouldn't suggest running fstransform on a _degraded_ RAID disk even to my enemies... fstransform is a BIG gun and must be used with care. Using it on such a setup verges on being suicidal. Regards, Massimiliano |
|
From: Massimiliano G. <mas...@gm...> - 2014-11-18 21:53:19
|
On 11/18/14 22:34, Olaf Leidinger wrote: > Hello! > >> what do you mean by "preallocating memory" ? >> The only preallocation coming to my mind is inside fsremap, >> and it's supposed to allocate half available RAM, so it should be >> pretty fast. >> >> Don't tell me you used the _experimental_ "prealloc" support when >> converting to ext4... > > Um, not by choice, it was enabled by default. I just started the transformation via: > > fstransform --opts-fsmove="--inode-cache=/spare/inode_cache" /dev/sda1 ext4 > > Compiled from the github repository. > > > fstransform: starting version 0.9.3.work11, checking environment > fstransform: checking for which... '/usr/bin/which' > fstransform: checking for expr... '/usr/bin/expr' > fstransform: checking for id... '/usr/bin/id' > fstransform: parsing command line arguments > fstransform: options '--inode-cache=/spare/inode_cache' for fsmove specified on command line > fstransform: checking for stat... '/usr/bin/stat' > fstransform: checking for mkfifo... '/usr/bin/mkfifo' > fstransform: checking for blockdev... '/sbin/blockdev > fstransform: checking for fsck... '/sbin/fsck' > fstransform: checking for mkfs... '/sbin/mkfs' > fstransform: checking for mount... '/bin/mount' > fstransform: checking for umount... '/bin/umount' > fstransform: checking for mkdir... '/bin/mkdir' > fstransform: checking for rmdir... '/bin/rmdir' > fstransform: checking for rm... '/bin/rm' > fstransform: checking for dd... '/bin/dd' > fstransform: checking for sync... '/bin/sync' > fstransform: checking for fsmove... '/usr/local/sbin/fsmove' > fstransform: checking for fsremap... '/usr/local/sbin/fsremap' > fstransform: checking for fsck(source file-system)... '/sbin/fsck' > fstransform: checking for fsck(target file-system)... '/sbin/fsck' > fstransform: looking for optional commands > fstransform: checking for fsattr... '/usr/local/sbin/fsattr' > fstransform: checking for sleep... '/bin/sleep' > fstransform: checking for date... '/bin/date' > 08:22:27 fstransform: environment check passed. > 08:22:27 fstransform: saving output of this execution into /var/tmp/fstransform/fstransform.log.682 > 08:22:27 fstransform: preparing to transform device '/dev/sda1' to file-system type 'ext4' > 08:22:27 fstransform: device is mounted at '/mirror' with file-system type 'xfs' > 08:22:27 fstransform: enabling preallocation speedup: target FSTYPE is 'ext4' and command 'fsattr' is available > 08:22:27 fstransform: device raw size = 11999933069824 bytes > 08:22:27 fstransform: creating sparse loop file '/mirror/.fstransform.loop.682' inside device '/dev/sda1'... > 08:22:29 dd: 1+0 records out > 08:22:29 dd: 1 byte (1 B) copied, 0.000278164 s, 3.6 kB/s > 08:22:29 fstransform: connected loop device '/dev/loop0' to file '/mirror/.fstransform.loop.682' > 08:22:29 fstransform: formatting loop device '/dev/loop0' with file-system type 'ext4'... > 08:22:33 fstransform: mounting loop device '/dev/loop0' on '/tmp/fstransform.loop.682' ... > 08:22:34 fstransform: loop device '/dev/loop0' mounted successfully. > 08:22:34 fstransform: preallocating loop file contents. > 08:22:34 fstransform: this may take some time, please be patient... > 08:22:42 fsmove: progress: 0.3% done, 4.2 terabytes still to preallocate > 08:23:04 fsmove: progress: 0.6% done, 4.1 terabytes still to preallocate, estimated 1 hour and 25 minutes left [...] > 14:06:21 fsmove: progress: 10.7% done, 3.7 terabytes still to preallocate, estimated 3 days left > 15:05:12 fsmove: progress: 11.0% done, 3.7 terabytes still to preallocate, estimated 5 days left > > (canceled at this point, as the log was spammed with these XFS messages all over, again) > >> A copy of the log files would help to understand what's happening. >> You can find them in /var/tmp/fstransform/ - before sending them, >> check there are no sensitive information inside, as they often >> contain the names of the files found in the partition being converted. > > I'm sorry, there is no log. Probably, as I canceled the transformation > this afternoon. At the moment, the server is doing something else, > thus I can't restart fstransform to produce logs. But I guess you're right, > it's probably an XFS bug when allocating large file, maybe only in combination > with this hardware. > > How would I disable the preallocation? But I guess without preallocation > progress wouldn't be faster? > > I guess the RAID6 array has some problem and needs to be rebuilt. > > Best, > > Olaf > > Hello Olaf, you're right, preallocation was enabled by default (AAAAAAARGH). To disable it, run "fstransform --prealloc=no ..." I will IMMEDIATELY disable it in github sources. I have good news and bad ones. The good news is that by using preallocation, the original file system was not modified at all when you stopped the conversion (except for the huge sparse file /mirror/.fstransform.loop.682). The bad news is that fstransform with preallocation is MUCH faster than normal conversion, and if that one was taking N months, the normal conversion will take about N*2 months. My experience with fast, directly connected SATA magnetic disks is about 1GB per minute: I've converted a 750 GB disk in about 12 hours. By fast, I mean >100GB/s peak speed (you can check it with "hdparm -t DEVICE") Unluckily, your 4.2 terabytes disk looks quite slower than that - but maybe it was due to the XFS errors, or maybe due to the huge number of hard links. Regards, Massimiliano |
|
From: Olaf L. <ol...@me...> - 2014-11-18 21:35:44
|
Hello! > what do you mean by "preallocating memory" ? > The only preallocation coming to my mind is inside fsremap, > and it's supposed to allocate half available RAM, so it should be > pretty fast. > > Don't tell me you used the _experimental_ "prealloc" support when > converting to ext4... Um, not by choice, it was enabled by default. I just started the transformation via: fstransform --opts-fsmove="--inode-cache=/spare/inode_cache" /dev/sda1 ext4 Compiled from the github repository. fstransform: starting version 0.9.3.work11, checking environment fstransform: checking for which... '/usr/bin/which' fstransform: checking for expr... '/usr/bin/expr' fstransform: checking for id... '/usr/bin/id' fstransform: parsing command line arguments fstransform: options '--inode-cache=/spare/inode_cache' for fsmove specified on command line fstransform: checking for stat... '/usr/bin/stat' fstransform: checking for mkfifo... '/usr/bin/mkfifo' fstransform: checking for blockdev... '/sbin/blockdev fstransform: checking for fsck... '/sbin/fsck' fstransform: checking for mkfs... '/sbin/mkfs' fstransform: checking for mount... '/bin/mount' fstransform: checking for umount... '/bin/umount' fstransform: checking for mkdir... '/bin/mkdir' fstransform: checking for rmdir... '/bin/rmdir' fstransform: checking for rm... '/bin/rm' fstransform: checking for dd... '/bin/dd' fstransform: checking for sync... '/bin/sync' fstransform: checking for fsmove... '/usr/local/sbin/fsmove' fstransform: checking for fsremap... '/usr/local/sbin/fsremap' fstransform: checking for fsck(source file-system)... '/sbin/fsck' fstransform: checking for fsck(target file-system)... '/sbin/fsck' fstransform: looking for optional commands fstransform: checking for fsattr... '/usr/local/sbin/fsattr' fstransform: checking for sleep... '/bin/sleep' fstransform: checking for date... '/bin/date' 08:22:27 fstransform: environment check passed. 08:22:27 fstransform: saving output of this execution into /var/tmp/fstransform/fstransform.log.682 08:22:27 fstransform: preparing to transform device '/dev/sda1' to file-system type 'ext4' 08:22:27 fstransform: device is mounted at '/mirror' with file-system type 'xfs' 08:22:27 fstransform: enabling preallocation speedup: target FSTYPE is 'ext4' and command 'fsattr' is available 08:22:27 fstransform: device raw size = 11999933069824 bytes 08:22:27 fstransform: creating sparse loop file '/mirror/.fstransform.loop.682' inside device '/dev/sda1'... 08:22:29 dd: 1+0 records out 08:22:29 dd: 1 byte (1 B) copied, 0.000278164 s, 3.6 kB/s 08:22:29 fstransform: connected loop device '/dev/loop0' to file '/mirror/.fstransform.loop.682' 08:22:29 fstransform: formatting loop device '/dev/loop0' with file-system type 'ext4'... 08:22:33 fstransform: mounting loop device '/dev/loop0' on '/tmp/fstransform.loop.682' ... 08:22:34 fstransform: loop device '/dev/loop0' mounted successfully. 08:22:34 fstransform: preallocating loop file contents. 08:22:34 fstransform: this may take some time, please be patient... 08:22:42 fsmove: progress: 0.3% done, 4.2 terabytes still to preallocate 08:23:04 fsmove: progress: 0.6% done, 4.1 terabytes still to preallocate, estimated 1 hour and 25 minutes left 08:23:29 fsmove: progress: 0.9% done, 4.1 terabytes still to preallocate, estimated 1 hour and 30 minutes left 08:24:11 fsmove: progress: 1.2% done, 4.1 terabytes still to preallocate, estimated 1 hour and 50 minutes left 08:24:32 fsmove: progress: 1.5% done, 4.1 terabytes still to preallocate, estimated 1 hour and 50 minutes left 08:25:14 fsmove: progress: 1.8% done, 4.1 terabytes still to preallocate, estimated 1 hour and 50 minutes left 08:25:36 fsmove: progress: 2.1% done, 4.1 terabytes still to preallocate, estimated 1 hour and 50 minutes left 08:25:52 fsmove: progress: 2.5% done, 4.1 terabytes still to preallocate, estimated 1 hour and 50 minutes left 08:25:54 fsmove: progress: 2.8% done, 4.1 terabytes still to preallocate, estimated 1 hour and 40 minutes left 08:26:14 fsmove: progress: 3.1% done, 4.0 terabytes still to preallocate, estimated 1 hour and 40 minutes left 08:26:36 fsmove: progress: 3.4% done, 4.0 terabytes still to preallocate, estimated 1 hour and 40 minutes left 08:27:05 fsmove: progress: 3.7% done, 4.0 terabytes still to preallocate, estimated 2 hours left 08:27:42 fsmove: progress: 4.0% done, 4.0 terabytes still to preallocate, estimated 2 hours left 08:28:13 fsmove: progress: 4.3% done, 4.0 terabytes still to preallocate, estimated 2 hours left 08:35:05 fsmove: progress: 4.6% done, 4.0 terabytes still to preallocate, estimated 4 hours left 08:42:00 fsmove: progress: 4.9% done, 4.0 terabytes still to preallocate, estimated 4 hours left 08:49:40 fsmove: progress: 5.2% done, 3.9 terabytes still to preallocate, estimated 3 hours left 08:50:45 fsmove: progress: 5.5% done, 3.9 terabytes still to preallocate, estimated 1 hour and 40 minutes left 09:03:31 fsmove: progress: 5.8% done, 3.9 terabytes still to preallocate, estimated 2 hours left 09:09:05 fsmove: progress: 6.1% done, 3.9 terabytes still to preallocate, estimated 3 hours left 09:24:03 fsmove: progress: 6.4% done, 3.9 terabytes still to preallocate, estimated 4 hours left 09:37:32 fsmove: progress: 6.7% done, 3.9 terabytes still to preallocate, estimated 5 hours left 09:50:06 fsmove: progress: 7.0% done, 3.9 terabytes still to preallocate, estimated 5 hours left 10:08:55 fsmove: progress: 7.4% done, 3.9 terabytes still to preallocate, estimated 1 day and 7 hours left 10:18:35 fsmove: progress: 7.7% done, 3.8 terabytes still to preallocate, estimated 1 day and 20 hours left 10:34:43 fsmove: progress: 8.0% done, 3.8 terabytes still to preallocate, estimated 1 day and 20 hours left 11:00:49 fsmove: progress: 8.3% done, 3.8 terabytes still to preallocate, estimated 1 day and 15 hours left 11:26:28 fsmove: progress: 8.6% done, 3.8 terabytes still to preallocate, estimated 2 days left 11:43:49 fsmove: progress: 8.9% done, 3.8 terabytes still to preallocate, estimated 2 days left 12:01:52 fsmove: progress: 9.2% done, 3.8 terabytes still to preallocate, estimated 3 days left 12:25:29 fsmove: progress: 9.5% done, 3.8 terabytes still to preallocate, estimated 3 days left 12:48:16 fsmove: progress: 9.8% done, 3.8 terabytes still to preallocate, estimated 3 days left 13:12:36 fsmove: progress: 10.1% done, 3.7 terabytes still to preallocate, estimated 4 days left 13:39:16 fsmove: progress: 10.4% done, 3.7 terabytes still to preallocate, estimated 4 days left 14:06:21 fsmove: progress: 10.7% done, 3.7 terabytes still to preallocate, estimated 3 days left 15:05:12 fsmove: progress: 11.0% done, 3.7 terabytes still to preallocate, estimated 5 days left (canceled at this point, as the log was spammed with these XFS messages all over, again) > A copy of the log files would help to understand what's happening. > You can find them in /var/tmp/fstransform/ - before sending them, > check there are no sensitive information inside, as they often > contain the names of the files found in the partition being converted. I'm sorry, there is no log. Probably, as I canceled the transformation this afternoon. At the moment, the server is doing something else, thus I can't restart fstransform to produce logs. But I guess you're right, it's probably an XFS bug when allocating large file, maybe only in combination with this hardware. How would I disable the preallocation? But I guess without preallocation progress wouldn't be faster? I guess the RAID6 array has some problem and needs to be rebuilt. Best, Olaf |
|
From: Massimiliano G. <mas...@gm...> - 2014-11-18 21:17:37
|
On 11/18/14 08:04, Olaf Leidinger wrote: > Heya! > > It seems the filesystem is a bit slow. It would take months to do the > conversion. At the moment it's stuck while preallocationg memory. When > reaching about 30% progress, the estimated time is over a month. > > During preallocating, the kernel log yields messages à la: > > XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) > > After googling, the recommended solution was running defrag on XFS. > I did this, however, no luck so far. The estimations are on a rise. > > Also, I found some bug report for older XFS versions, however, I run > this transformation on Debian Jessie (kernel 3.16), so it shouldn't be > affected. > > Any idea how the preallocation can be speeded up? > > Best, > Olaf > Hello Olaf, what do you mean by "preallocating memory" ? The only preallocation coming to my mind is inside fsremap, and it's supposed to allocate half available RAM, so it should be pretty fast. Don't tell me you used the _experimental_ "prealloc" support when converting to ext4... A copy of the log files would help to understand what's happening. You can find them in /var/tmp/fstransform/ - before sending them, check there are no sensitive information inside, as they often contain the names of the files found in the partition being converted. Conversion to/from XFS is not the smoothest one, but works fairly well as long as XFS file system does not get full. The message > XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) looks like an XFS bug. Regards, Massimiliano |
|
From: Olaf L. <ol...@me...> - 2014-11-18 07:04:02
|
Heya! It seems the filesystem is a bit slow. It would take months to do the conversion. At the moment it's stuck while preallocationg memory. When reaching about 30% progress, the estimated time is over a month. During preallocating, the kernel log yields messages à la: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) After googling, the recommended solution was running defrag on XFS. I did this, however, no luck so far. The estimations are on a rise. Also, I found some bug report for older XFS versions, however, I run this transformation on Debian Jessie (kernel 3.16), so it shouldn't be affected. Any idea how the preallocation can be speeded up? Best, Olaf |
|
From: Massimiliano G. <mas...@gm...> - 2014-11-03 18:25:54
|
On 11/03/14 01:45, Olaf Leidinger wrote: > Heya! > > So this will reduce the amount of memory needed for the conversion? > > This would be marvellous! One backup directory set contains approximately > 6 million hard links (according to your find command). Thus, and to speed up > the conversion, I ordered the backup server to delete old backups and it's > still on it. At the moment, the file system is up to 50% free space (~5 TiB) > again (coming from completely full) . Yet, I doubt that the deleting+converting > is finished before my vacation, so I'll postpone it until after the vacation. > > Thanks for adding this new feature! > > Best, > Olaf > Hello Olaf, yes, the disk-based inode cache greatly reduces the memory needed by fsmove during conversion of file systems with a lot of hard links. With it, the consumed memory does not depend at all on the number of hard links. 6 millions hard links could be managed also by the original code, but "6 millions hard links in a single directory" makes me think that summing other similar directories, the original code may easily exceed the available RAM (it needed some hundred bytes per hard link). With the new code, it shouldn't be a problem at all, provided you specify with --inode-cache=DIR a file system with a LOT of free inodes (again, remember the command "df -i") Regards, Max |
|
From: Olaf L. <ol...@me...> - 2014-11-03 00:45:46
|
Heya! So this will reduce the amount of memory needed for the conversion? This would be marvellous! One backup directory set contains approximately 6 million hard links (according to your find command). Thus, and to speed up the conversion, I ordered the backup server to delete old backups and it's still on it. At the moment, the file system is up to 50% free space (~5 TiB) again (coming from completely full) . Yet, I doubt that the deleting+converting is finished before my vacation, so I'll postpone it until after the vacation. Thanks for adding this new feature! Best, Olaf -- Please consider sending me encrypted E-Mails. If you wonder how to do that, just ask. |
|
From: Massimiliano G. <mas...@gm...> - 2014-11-02 23:08:17
|
On 10/29/14 23:58, Olaf Leidinger wrote: > On 10/29/14 23:36, Massimiliano Ghilardi wrote: >> Yes, it preserves hard links. In order to do that, it keeps an >> in-memory cache of hard links found so far. If the cache gets larger >> than available RAM, it starts swapping to disk and you're in trouble: >> it will take ages to finish, or, if it runs out of swap space, it >> will crash. > > Actually, I can't really guess how many hard links are there on this > partition. The server has 8 GiB of RAM + 8 GiB of swap... I'll try and > report back how it worked. > > Best, > Olaf Hello Olaf, I added support for disk-based inode cache in the latest version of fstranform. While not released yet, initial tests are successful. You can download it from https://github.com/cosmos72/fstransform either with the 'download ZIP' button on the right, or by cloning the GIT repository. To actually enable the disk-based inode cache, you must add the option --inode-cache="DIR" to fsmove. If you use the main fstransform script, the option becomes --opts-fsmove="--inode-cache=DIR" where DIR is be the path of a non-existing directory (fsmove will create it) - it can also be inside the filesystem to be transformed. Such directory will be filled with one soft-link per group of hard-links pointing to the same file, so it must be inside a filesystem with plenty of available inodes ("df -i" is your friend). Best regards, Max |
|
From: Massimiliano G. <mas...@gm...> - 2014-10-29 23:14:36
|
On 10/29/14 23:58, Olaf Leidinger wrote: > Heya! > >> On the good side, it's quite well tested, it works with file systems >> full up to 95% (90% on XFS), and can even resume a job after a crash >> or power failure. > > 90% when converting to or when converting from XFS? 90% when converting _to_ XFS. It's an estimation, not an absolute number... on a very large disk it may be slightly higher. > Actually, I can't really guess how many hard links are there on this > partition. The server has 8 GiB of RAM + 8 GiB of swap... I'll try and > report back how it worked. > > Best, > Olaf you can count the number of hard links with this command: find /path/to/filesystem -xdev -type f \! -links 1 | wc -l it's not exact, as each file with N links will be counted N times, but you will get a rough idea. Regards, Max |
|
From: Olaf L. <ol...@me...> - 2014-10-29 22:59:29
|
Heya! > On the good side, it's quite well tested, it works with file systems > full up to 95% (90% on XFS), and can even resume a job after a crash > or power failure. 90% when converting to or when converting from XFS? > On the bad side, it is quite slow (1GB/minute on a single hard disk) > and actually the main problem is that if it fails catastrophically, > you have no backup. Not having a backup is not such a big deal in our case, this is a backup server. The original data is on our file server. We'd only loose the time evolution of the files from the file server. > Yes, it preserves hard links. In order to do that, it keeps an > in-memory cache of hard links found so far. If the cache gets larger > than available RAM, it starts swapping to disk and you're in trouble: > it will take ages to finish, or, if it runs out of swap space, it > will crash. Actually, I can't really guess how many hard links are there on this partition. The server has 8 GiB of RAM + 8 GiB of swap... I'll try and report back how it worked. Best, Olaf |
|
From: Massimiliano G. <mas...@gm...> - 2014-10-29 22:36:56
|
On 10/29/14 16:51, Olaf Leidinger wrote: > Dear List, > > today, I learnt about fstransform and it seems like a wonderful tool! > I'm wondering why I didn't find it earlier. It seems to good, to be > true! > > One thing I didn't find in the docs, however, is, if hard links are > preserved during the conversion. > > We have a huge block device, about 11 TiB, provided by a 3ware > controller via RAID6. On top of it, there is a XFS file system, which > is used to store incremental backups of user data -- mostly quite small > files. This incremental backup makes heavy use of hard links. If the > linking was broken, the file system would quickly fill up. > > Best, > > Olaf > > ------------------------------------------------------------------------------ > _______________________________________________ > Fstransform-devel mailing list > Fst...@li... > https://lists.sourceforge.net/lists/listinfo/fstransform-devel > Hello Olaf, I wrote fstransform exactly because it's not obvious that it's possible to implement it at all. On the good side, it's quite well tested, it works with file systems full up to 95% (90% on XFS), and can even resume a job after a crash or power failure. On the bad side, it is quite slow (1GB/minute on a single hard disk) and actually the main problem is that if it fails catastrophically, you have no backup. Yes, it preserves hard links. In order to do that, it keeps an in-memory cache of hard links found so far. If the cache gets larger than available RAM, it starts swapping to disk and you're in trouble: it will take ages to finish, or, if it runs out of swap space, it will crash. So it can handle a reasonable number of hard links - let's say one million hard links are ok, while 100 millions are asking for trouble. Regards, Max |
|
From: Olaf L. <ol...@me...> - 2014-10-29 16:28:59
|
Dear List, today, I learnt about fstransform and it seems like a wonderful tool! I'm wondering why I didn't find it earlier. It seems to good, to be true! One thing I didn't find in the docs, however, is, if hard links are preserved during the conversion. We have a huge block device, about 11 TiB, provided by a 3ware controller via RAID6. On top of it, there is a XFS file system, which is used to store incremental backups of user data -- mostly quite small files. This incremental backup makes heavy use of hard links. If the linking was broken, the file system would quickly fill up. Best, Olaf |
|
From: Massimiliano G. <mas...@gm...> - 2014-01-17 02:28:53
|
On 01/16/14 18:50, Thomas Eschenbacher wrote: > Hello Massimiliano, > > during the past few days I successfully converted a 500GB > USB HD from jfs to ext3 (as an intermediate step to btrfs). > My PC cashed in between, but your tools were able to resume > the job without any difficulties. > > => Thank you very very much for writing such a great tool !!! > > You did a really great job :-) > > have a nice week, > Thomas > Hello Thomas, thanks :) when I wrote fstransform, I added A LOT of checks and safety features exactly because it was performing risky operations on others' data. Can you believe I left my PC running thousands of automatic tests for days on randomly generated file systems - both real and simulated - to be confident that fstransform was sufficiently stable and bug-free? The ability to resume an interrupted/crashed job was the final safeguard that gave me the feeling it was good enough to be published. Good to know you agree :) Massimiliano P.S. last time I tried btrfs (admittedly 2 years ago), it did not like at all the operations performed by fstransform - I was not able to have fstransform running reliably on it, neither as source nor as destination file system. I hope btrfs matured in the meantime, but still I recommend you to be extra careful if you want to use fstransform with it. |
|
From: Massimiliano G. <mas...@gm...> - 2014-01-17 02:11:00
|
Hello Bill, this particular problem (umount: device is busy) happened to me several times. Usually it's not a serious issue, because fstransform will not exit when an error occurs: it will instead pause and wait until it gets user input. The user can either fix the problem and tell CONTINUE to fstransform, or he can kill it with CTRL+C, ENTER, or anything else. You're right about the fact that fstransform could perform more checks - as trying to detect as early as possible if some programs (including itself) is using the file system. You were very unlucky that in your case it was impossible to fix the problem while fstransform waited, as ITS EXISTENCE was the problem... Best regards, Massimiliano Ghilardi P.S. I manually approved your message, as normally only members can write to fst...@li... - I'd recommended you to subscribe using the form at https://lists.sourceforge.net/lists/listinfo/fstransform-devel On 01/16/14 08:28, Bill Barry wrote: > Hello list, > I used fstransform to transform a 2TB disk from jfs to ext4. I made > a mistake when doing this and later was able to recover. I just thought you > would be interested in what kind of mistakes are possible. > > The mistake I made was to mount the disk I wanted to transform and execute > fstransform from the root directory of that mounted partition. Everything > went well for a very long time until I got this message > > ------------------------------------------------------------- [...more lines omitted ...] > 2014-01-14 13:10:00 fstransform: unmounting device '/dev/sdd1' before disk > check > 2014-01-14 13:10:00 umount: /mnt/2tb: device is busy. > 2014-01-14 13:10:00 umount: (In some cases useful info about processes that > use > 2014-01-14 13:10:00 umount: the device is found by lsof(8) or fuser(1)) > > 2014-01-14 13:10:00 ERROR! fstransform: command '/bin/umount /dev/sdd1' > failed (exit status 1) > this is potentially a problem. > you can either quit now by pressing ENTER or > CTRL+C, > > or, if you know what went wrong, you can fix it > yourself, > then manually run the command '/bin/umount > /dev/sdd1' > (or something equivalent) > and finally resume this script by typing CONTINUE > and pressing ENTER: > 2014-01-14 13:57:48 fstransform: exiting. > ------------------------------------------------------------- > > As you can see the umount failed because I was executing the script from > the directory /mnt/2tb/ which was the root of /dev/sdd1. > > Since the loop file had been completed. I was able to manually run fsremap > and finish the process. > > I guess it would be useful to check at the beginning of the process to see > if someone has made a crazy mistake like this. > > Thanks, > Bill Barry > > > > ------------------------------------------------------------------------------ > CenturyLink Cloud: The Leader in Enterprise Cloud Services. > Learn Why More Businesses Are Choosing CenturyLink Cloud For > Critical Workloads, Development Environments & Everything In Between. > Get a Quote or Start a Free Trial Today. > http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Fstransform-devel mailing list > Fst...@li... > https://lists.sourceforge.net/lists/listinfo/fstransform-devel > |
|
From: Bill B. <bi...@bi...> - 2014-01-16 07:28:57
|
Hello list,
I used fstransform to transform a 2TB disk from jfs to ext4. I made
a mistake when doing this and later was able to recover. I just thought you
would be interested in what kind of mistakes are possible.
The mistake I made was to mount the disk I wanted to transform and execute
fstransform from the root directory of that mounted partition. Everything
went well for a very long time until I got this message
-------------------------------------------------------------
2014-01-14 12:57:41 fsmove: progress: 95.3% done, 58.2 gigabytes still to
move, estimated 40 minutes left
2014-01-14 13:06:48 fsmove: job completed.
2014-01-14 13:06:48 fstransform: unmounting and running '/sbin/fsck' (disk
check) on loop file '/mnt/2tb/.fstransform.loop.27259'
2014-01-14 13:06:55 fsck: fsck from util-linux 2.20.1
2014-01-14 13:10:00 fsck: /dev/loop0: 1709026/122101760 files (0.9%
non-contiguous), 316980982/488378000 blocks
2014-01-14 13:10:00 fstransform: disconnected loop device '/dev/loop0' from
file '/mnt/2tb/.fstransform.loop.27259'
2014-01-14 13:10:00 fstransform: unmounting device '/dev/sdd1' before disk
check
2014-01-14 13:10:00 umount: /mnt/2tb: device is busy.
2014-01-14 13:10:00 umount: (In some cases useful info about processes that
use
2014-01-14 13:10:00 umount: the device is found by lsof(8) or fuser(1))
2014-01-14 13:10:00 ERROR! fstransform: command '/bin/umount /dev/sdd1'
failed (exit status 1)
this is potentially a problem.
you can either quit now by pressing ENTER or
CTRL+C,
or, if you know what went wrong, you can fix it
yourself,
then manually run the command '/bin/umount
/dev/sdd1'
(or something equivalent)
and finally resume this script by typing CONTINUE
and pressing ENTER:
2014-01-14 13:57:48 fstransform: exiting.
-------------------------------------------------------------
As you can see the umount failed because I was executing the script from
the directory /mnt/2tb/ which was the root of /dev/sdd1.
Since the loop file had been completed. I was able to manually run fsremap
and finish the process.
I guess it would be useful to check at the beginning of the process to see
if someone has made a crazy mistake like this.
Thanks,
Bill Barry
|