Thread: [Fstransform-devel] preservation of hard links
Status: Beta
Brought to you by:
paperinik
Menu
▾
▴
fstransform-devel
|
From: Olaf L. <ol...@me...> - 2014-10-29 16:28:59
|
Dear List, today, I learnt about fstransform and it seems like a wonderful tool! I'm wondering why I didn't find it earlier. It seems to good, to be true! One thing I didn't find in the docs, however, is, if hard links are preserved during the conversion. We have a huge block device, about 11 TiB, provided by a 3ware controller via RAID6. On top of it, there is a XFS file system, which is used to store incremental backups of user data -- mostly quite small files. This incremental backup makes heavy use of hard links. If the linking was broken, the file system would quickly fill up. Best, Olaf |
|
From: Massimiliano G. <mas...@gm...> - 2014-10-29 22:36:56
|
On 10/29/14 16:51, Olaf Leidinger wrote: > Dear List, > > today, I learnt about fstransform and it seems like a wonderful tool! > I'm wondering why I didn't find it earlier. It seems to good, to be > true! > > One thing I didn't find in the docs, however, is, if hard links are > preserved during the conversion. > > We have a huge block device, about 11 TiB, provided by a 3ware > controller via RAID6. On top of it, there is a XFS file system, which > is used to store incremental backups of user data -- mostly quite small > files. This incremental backup makes heavy use of hard links. If the > linking was broken, the file system would quickly fill up. > > Best, > > Olaf > > ------------------------------------------------------------------------------ > _______________________________________________ > Fstransform-devel mailing list > Fst...@li... > https://lists.sourceforge.net/lists/listinfo/fstransform-devel > Hello Olaf, I wrote fstransform exactly because it's not obvious that it's possible to implement it at all. On the good side, it's quite well tested, it works with file systems full up to 95% (90% on XFS), and can even resume a job after a crash or power failure. On the bad side, it is quite slow (1GB/minute on a single hard disk) and actually the main problem is that if it fails catastrophically, you have no backup. Yes, it preserves hard links. In order to do that, it keeps an in-memory cache of hard links found so far. If the cache gets larger than available RAM, it starts swapping to disk and you're in trouble: it will take ages to finish, or, if it runs out of swap space, it will crash. So it can handle a reasonable number of hard links - let's say one million hard links are ok, while 100 millions are asking for trouble. Regards, Max |
|
From: Olaf L. <ol...@me...> - 2014-10-29 22:59:29
|
Heya! > On the good side, it's quite well tested, it works with file systems > full up to 95% (90% on XFS), and can even resume a job after a crash > or power failure. 90% when converting to or when converting from XFS? > On the bad side, it is quite slow (1GB/minute on a single hard disk) > and actually the main problem is that if it fails catastrophically, > you have no backup. Not having a backup is not such a big deal in our case, this is a backup server. The original data is on our file server. We'd only loose the time evolution of the files from the file server. > Yes, it preserves hard links. In order to do that, it keeps an > in-memory cache of hard links found so far. If the cache gets larger > than available RAM, it starts swapping to disk and you're in trouble: > it will take ages to finish, or, if it runs out of swap space, it > will crash. Actually, I can't really guess how many hard links are there on this partition. The server has 8 GiB of RAM + 8 GiB of swap... I'll try and report back how it worked. Best, Olaf |
|
From: Massimiliano G. <mas...@gm...> - 2014-10-29 23:14:36
|
On 10/29/14 23:58, Olaf Leidinger wrote: > Heya! > >> On the good side, it's quite well tested, it works with file systems >> full up to 95% (90% on XFS), and can even resume a job after a crash >> or power failure. > > 90% when converting to or when converting from XFS? 90% when converting _to_ XFS. It's an estimation, not an absolute number... on a very large disk it may be slightly higher. > Actually, I can't really guess how many hard links are there on this > partition. The server has 8 GiB of RAM + 8 GiB of swap... I'll try and > report back how it worked. > > Best, > Olaf you can count the number of hard links with this command: find /path/to/filesystem -xdev -type f \! -links 1 | wc -l it's not exact, as each file with N links will be counted N times, but you will get a rough idea. Regards, Max |
|
From: Massimiliano G. <mas...@gm...> - 2014-11-02 23:08:17
|
On 10/29/14 23:58, Olaf Leidinger wrote: > On 10/29/14 23:36, Massimiliano Ghilardi wrote: >> Yes, it preserves hard links. In order to do that, it keeps an >> in-memory cache of hard links found so far. If the cache gets larger >> than available RAM, it starts swapping to disk and you're in trouble: >> it will take ages to finish, or, if it runs out of swap space, it >> will crash. > > Actually, I can't really guess how many hard links are there on this > partition. The server has 8 GiB of RAM + 8 GiB of swap... I'll try and > report back how it worked. > > Best, > Olaf Hello Olaf, I added support for disk-based inode cache in the latest version of fstranform. While not released yet, initial tests are successful. You can download it from https://github.com/cosmos72/fstransform either with the 'download ZIP' button on the right, or by cloning the GIT repository. To actually enable the disk-based inode cache, you must add the option --inode-cache="DIR" to fsmove. If you use the main fstransform script, the option becomes --opts-fsmove="--inode-cache=DIR" where DIR is be the path of a non-existing directory (fsmove will create it) - it can also be inside the filesystem to be transformed. Such directory will be filled with one soft-link per group of hard-links pointing to the same file, so it must be inside a filesystem with plenty of available inodes ("df -i" is your friend). Best regards, Max |
|
From: Olaf L. <ol...@me...> - 2014-11-03 00:45:46
|
Heya! So this will reduce the amount of memory needed for the conversion? This would be marvellous! One backup directory set contains approximately 6 million hard links (according to your find command). Thus, and to speed up the conversion, I ordered the backup server to delete old backups and it's still on it. At the moment, the file system is up to 50% free space (~5 TiB) again (coming from completely full) . Yet, I doubt that the deleting+converting is finished before my vacation, so I'll postpone it until after the vacation. Thanks for adding this new feature! Best, Olaf -- Please consider sending me encrypted E-Mails. If you wonder how to do that, just ask. |
|
From: Massimiliano G. <mas...@gm...> - 2014-11-03 18:25:54
|
On 11/03/14 01:45, Olaf Leidinger wrote: > Heya! > > So this will reduce the amount of memory needed for the conversion? > > This would be marvellous! One backup directory set contains approximately > 6 million hard links (according to your find command). Thus, and to speed up > the conversion, I ordered the backup server to delete old backups and it's > still on it. At the moment, the file system is up to 50% free space (~5 TiB) > again (coming from completely full) . Yet, I doubt that the deleting+converting > is finished before my vacation, so I'll postpone it until after the vacation. > > Thanks for adding this new feature! > > Best, > Olaf > Hello Olaf, yes, the disk-based inode cache greatly reduces the memory needed by fsmove during conversion of file systems with a lot of hard links. With it, the consumed memory does not depend at all on the number of hard links. 6 millions hard links could be managed also by the original code, but "6 millions hard links in a single directory" makes me think that summing other similar directories, the original code may easily exceed the available RAM (it needed some hundred bytes per hard link). With the new code, it shouldn't be a problem at all, provided you specify with --inode-cache=DIR a file system with a LOT of free inodes (again, remember the command "df -i") Regards, Max |
|
From: Olaf L. <ol...@me...> - 2014-11-18 07:04:02
|
Heya! It seems the filesystem is a bit slow. It would take months to do the conversion. At the moment it's stuck while preallocationg memory. When reaching about 30% progress, the estimated time is over a month. During preallocating, the kernel log yields messages à la: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) After googling, the recommended solution was running defrag on XFS. I did this, however, no luck so far. The estimations are on a rise. Also, I found some bug report for older XFS versions, however, I run this transformation on Debian Jessie (kernel 3.16), so it shouldn't be affected. Any idea how the preallocation can be speeded up? Best, Olaf |
|
From: Massimiliano G. <mas...@gm...> - 2014-11-18 21:17:37
|
On 11/18/14 08:04, Olaf Leidinger wrote: > Heya! > > It seems the filesystem is a bit slow. It would take months to do the > conversion. At the moment it's stuck while preallocationg memory. When > reaching about 30% progress, the estimated time is over a month. > > During preallocating, the kernel log yields messages à la: > > XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) > > After googling, the recommended solution was running defrag on XFS. > I did this, however, no luck so far. The estimations are on a rise. > > Also, I found some bug report for older XFS versions, however, I run > this transformation on Debian Jessie (kernel 3.16), so it shouldn't be > affected. > > Any idea how the preallocation can be speeded up? > > Best, > Olaf > Hello Olaf, what do you mean by "preallocating memory" ? The only preallocation coming to my mind is inside fsremap, and it's supposed to allocate half available RAM, so it should be pretty fast. Don't tell me you used the _experimental_ "prealloc" support when converting to ext4... A copy of the log files would help to understand what's happening. You can find them in /var/tmp/fstransform/ - before sending them, check there are no sensitive information inside, as they often contain the names of the files found in the partition being converted. Conversion to/from XFS is not the smoothest one, but works fairly well as long as XFS file system does not get full. The message > XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) looks like an XFS bug. Regards, Massimiliano |
|
From: Olaf L. <ol...@me...> - 2014-11-18 21:35:44
|
Hello! > what do you mean by "preallocating memory" ? > The only preallocation coming to my mind is inside fsremap, > and it's supposed to allocate half available RAM, so it should be > pretty fast. > > Don't tell me you used the _experimental_ "prealloc" support when > converting to ext4... Um, not by choice, it was enabled by default. I just started the transformation via: fstransform --opts-fsmove="--inode-cache=/spare/inode_cache" /dev/sda1 ext4 Compiled from the github repository. fstransform: starting version 0.9.3.work11, checking environment fstransform: checking for which... '/usr/bin/which' fstransform: checking for expr... '/usr/bin/expr' fstransform: checking for id... '/usr/bin/id' fstransform: parsing command line arguments fstransform: options '--inode-cache=/spare/inode_cache' for fsmove specified on command line fstransform: checking for stat... '/usr/bin/stat' fstransform: checking for mkfifo... '/usr/bin/mkfifo' fstransform: checking for blockdev... '/sbin/blockdev fstransform: checking for fsck... '/sbin/fsck' fstransform: checking for mkfs... '/sbin/mkfs' fstransform: checking for mount... '/bin/mount' fstransform: checking for umount... '/bin/umount' fstransform: checking for mkdir... '/bin/mkdir' fstransform: checking for rmdir... '/bin/rmdir' fstransform: checking for rm... '/bin/rm' fstransform: checking for dd... '/bin/dd' fstransform: checking for sync... '/bin/sync' fstransform: checking for fsmove... '/usr/local/sbin/fsmove' fstransform: checking for fsremap... '/usr/local/sbin/fsremap' fstransform: checking for fsck(source file-system)... '/sbin/fsck' fstransform: checking for fsck(target file-system)... '/sbin/fsck' fstransform: looking for optional commands fstransform: checking for fsattr... '/usr/local/sbin/fsattr' fstransform: checking for sleep... '/bin/sleep' fstransform: checking for date... '/bin/date' 08:22:27 fstransform: environment check passed. 08:22:27 fstransform: saving output of this execution into /var/tmp/fstransform/fstransform.log.682 08:22:27 fstransform: preparing to transform device '/dev/sda1' to file-system type 'ext4' 08:22:27 fstransform: device is mounted at '/mirror' with file-system type 'xfs' 08:22:27 fstransform: enabling preallocation speedup: target FSTYPE is 'ext4' and command 'fsattr' is available 08:22:27 fstransform: device raw size = 11999933069824 bytes 08:22:27 fstransform: creating sparse loop file '/mirror/.fstransform.loop.682' inside device '/dev/sda1'... 08:22:29 dd: 1+0 records out 08:22:29 dd: 1 byte (1 B) copied, 0.000278164 s, 3.6 kB/s 08:22:29 fstransform: connected loop device '/dev/loop0' to file '/mirror/.fstransform.loop.682' 08:22:29 fstransform: formatting loop device '/dev/loop0' with file-system type 'ext4'... 08:22:33 fstransform: mounting loop device '/dev/loop0' on '/tmp/fstransform.loop.682' ... 08:22:34 fstransform: loop device '/dev/loop0' mounted successfully. 08:22:34 fstransform: preallocating loop file contents. 08:22:34 fstransform: this may take some time, please be patient... 08:22:42 fsmove: progress: 0.3% done, 4.2 terabytes still to preallocate 08:23:04 fsmove: progress: 0.6% done, 4.1 terabytes still to preallocate, estimated 1 hour and 25 minutes left 08:23:29 fsmove: progress: 0.9% done, 4.1 terabytes still to preallocate, estimated 1 hour and 30 minutes left 08:24:11 fsmove: progress: 1.2% done, 4.1 terabytes still to preallocate, estimated 1 hour and 50 minutes left 08:24:32 fsmove: progress: 1.5% done, 4.1 terabytes still to preallocate, estimated 1 hour and 50 minutes left 08:25:14 fsmove: progress: 1.8% done, 4.1 terabytes still to preallocate, estimated 1 hour and 50 minutes left 08:25:36 fsmove: progress: 2.1% done, 4.1 terabytes still to preallocate, estimated 1 hour and 50 minutes left 08:25:52 fsmove: progress: 2.5% done, 4.1 terabytes still to preallocate, estimated 1 hour and 50 minutes left 08:25:54 fsmove: progress: 2.8% done, 4.1 terabytes still to preallocate, estimated 1 hour and 40 minutes left 08:26:14 fsmove: progress: 3.1% done, 4.0 terabytes still to preallocate, estimated 1 hour and 40 minutes left 08:26:36 fsmove: progress: 3.4% done, 4.0 terabytes still to preallocate, estimated 1 hour and 40 minutes left 08:27:05 fsmove: progress: 3.7% done, 4.0 terabytes still to preallocate, estimated 2 hours left 08:27:42 fsmove: progress: 4.0% done, 4.0 terabytes still to preallocate, estimated 2 hours left 08:28:13 fsmove: progress: 4.3% done, 4.0 terabytes still to preallocate, estimated 2 hours left 08:35:05 fsmove: progress: 4.6% done, 4.0 terabytes still to preallocate, estimated 4 hours left 08:42:00 fsmove: progress: 4.9% done, 4.0 terabytes still to preallocate, estimated 4 hours left 08:49:40 fsmove: progress: 5.2% done, 3.9 terabytes still to preallocate, estimated 3 hours left 08:50:45 fsmove: progress: 5.5% done, 3.9 terabytes still to preallocate, estimated 1 hour and 40 minutes left 09:03:31 fsmove: progress: 5.8% done, 3.9 terabytes still to preallocate, estimated 2 hours left 09:09:05 fsmove: progress: 6.1% done, 3.9 terabytes still to preallocate, estimated 3 hours left 09:24:03 fsmove: progress: 6.4% done, 3.9 terabytes still to preallocate, estimated 4 hours left 09:37:32 fsmove: progress: 6.7% done, 3.9 terabytes still to preallocate, estimated 5 hours left 09:50:06 fsmove: progress: 7.0% done, 3.9 terabytes still to preallocate, estimated 5 hours left 10:08:55 fsmove: progress: 7.4% done, 3.9 terabytes still to preallocate, estimated 1 day and 7 hours left 10:18:35 fsmove: progress: 7.7% done, 3.8 terabytes still to preallocate, estimated 1 day and 20 hours left 10:34:43 fsmove: progress: 8.0% done, 3.8 terabytes still to preallocate, estimated 1 day and 20 hours left 11:00:49 fsmove: progress: 8.3% done, 3.8 terabytes still to preallocate, estimated 1 day and 15 hours left 11:26:28 fsmove: progress: 8.6% done, 3.8 terabytes still to preallocate, estimated 2 days left 11:43:49 fsmove: progress: 8.9% done, 3.8 terabytes still to preallocate, estimated 2 days left 12:01:52 fsmove: progress: 9.2% done, 3.8 terabytes still to preallocate, estimated 3 days left 12:25:29 fsmove: progress: 9.5% done, 3.8 terabytes still to preallocate, estimated 3 days left 12:48:16 fsmove: progress: 9.8% done, 3.8 terabytes still to preallocate, estimated 3 days left 13:12:36 fsmove: progress: 10.1% done, 3.7 terabytes still to preallocate, estimated 4 days left 13:39:16 fsmove: progress: 10.4% done, 3.7 terabytes still to preallocate, estimated 4 days left 14:06:21 fsmove: progress: 10.7% done, 3.7 terabytes still to preallocate, estimated 3 days left 15:05:12 fsmove: progress: 11.0% done, 3.7 terabytes still to preallocate, estimated 5 days left (canceled at this point, as the log was spammed with these XFS messages all over, again) > A copy of the log files would help to understand what's happening. > You can find them in /var/tmp/fstransform/ - before sending them, > check there are no sensitive information inside, as they often > contain the names of the files found in the partition being converted. I'm sorry, there is no log. Probably, as I canceled the transformation this afternoon. At the moment, the server is doing something else, thus I can't restart fstransform to produce logs. But I guess you're right, it's probably an XFS bug when allocating large file, maybe only in combination with this hardware. How would I disable the preallocation? But I guess without preallocation progress wouldn't be faster? I guess the RAID6 array has some problem and needs to be rebuilt. Best, Olaf |
|
From: Massimiliano G. <mas...@gm...> - 2014-11-18 22:02:14
|
On 11/18/14 22:34, Olaf Leidinger wrote: >[...] > I guess the RAID6 array has some problem and needs to be rebuilt. > > Best, > > Olaf That's a BIG RED flag. I wouldn't suggest running fstransform on a _degraded_ RAID disk even to my enemies... fstransform is a BIG gun and must be used with care. Using it on such a setup verges on being suicidal. Regards, Massimiliano |
|
From: Massimiliano G. <mas...@gm...> - 2014-11-18 21:53:19
|
On 11/18/14 22:34, Olaf Leidinger wrote: > Hello! > >> what do you mean by "preallocating memory" ? >> The only preallocation coming to my mind is inside fsremap, >> and it's supposed to allocate half available RAM, so it should be >> pretty fast. >> >> Don't tell me you used the _experimental_ "prealloc" support when >> converting to ext4... > > Um, not by choice, it was enabled by default. I just started the transformation via: > > fstransform --opts-fsmove="--inode-cache=/spare/inode_cache" /dev/sda1 ext4 > > Compiled from the github repository. > > > fstransform: starting version 0.9.3.work11, checking environment > fstransform: checking for which... '/usr/bin/which' > fstransform: checking for expr... '/usr/bin/expr' > fstransform: checking for id... '/usr/bin/id' > fstransform: parsing command line arguments > fstransform: options '--inode-cache=/spare/inode_cache' for fsmove specified on command line > fstransform: checking for stat... '/usr/bin/stat' > fstransform: checking for mkfifo... '/usr/bin/mkfifo' > fstransform: checking for blockdev... '/sbin/blockdev > fstransform: checking for fsck... '/sbin/fsck' > fstransform: checking for mkfs... '/sbin/mkfs' > fstransform: checking for mount... '/bin/mount' > fstransform: checking for umount... '/bin/umount' > fstransform: checking for mkdir... '/bin/mkdir' > fstransform: checking for rmdir... '/bin/rmdir' > fstransform: checking for rm... '/bin/rm' > fstransform: checking for dd... '/bin/dd' > fstransform: checking for sync... '/bin/sync' > fstransform: checking for fsmove... '/usr/local/sbin/fsmove' > fstransform: checking for fsremap... '/usr/local/sbin/fsremap' > fstransform: checking for fsck(source file-system)... '/sbin/fsck' > fstransform: checking for fsck(target file-system)... '/sbin/fsck' > fstransform: looking for optional commands > fstransform: checking for fsattr... '/usr/local/sbin/fsattr' > fstransform: checking for sleep... '/bin/sleep' > fstransform: checking for date... '/bin/date' > 08:22:27 fstransform: environment check passed. > 08:22:27 fstransform: saving output of this execution into /var/tmp/fstransform/fstransform.log.682 > 08:22:27 fstransform: preparing to transform device '/dev/sda1' to file-system type 'ext4' > 08:22:27 fstransform: device is mounted at '/mirror' with file-system type 'xfs' > 08:22:27 fstransform: enabling preallocation speedup: target FSTYPE is 'ext4' and command 'fsattr' is available > 08:22:27 fstransform: device raw size = 11999933069824 bytes > 08:22:27 fstransform: creating sparse loop file '/mirror/.fstransform.loop.682' inside device '/dev/sda1'... > 08:22:29 dd: 1+0 records out > 08:22:29 dd: 1 byte (1 B) copied, 0.000278164 s, 3.6 kB/s > 08:22:29 fstransform: connected loop device '/dev/loop0' to file '/mirror/.fstransform.loop.682' > 08:22:29 fstransform: formatting loop device '/dev/loop0' with file-system type 'ext4'... > 08:22:33 fstransform: mounting loop device '/dev/loop0' on '/tmp/fstransform.loop.682' ... > 08:22:34 fstransform: loop device '/dev/loop0' mounted successfully. > 08:22:34 fstransform: preallocating loop file contents. > 08:22:34 fstransform: this may take some time, please be patient... > 08:22:42 fsmove: progress: 0.3% done, 4.2 terabytes still to preallocate > 08:23:04 fsmove: progress: 0.6% done, 4.1 terabytes still to preallocate, estimated 1 hour and 25 minutes left [...] > 14:06:21 fsmove: progress: 10.7% done, 3.7 terabytes still to preallocate, estimated 3 days left > 15:05:12 fsmove: progress: 11.0% done, 3.7 terabytes still to preallocate, estimated 5 days left > > (canceled at this point, as the log was spammed with these XFS messages all over, again) > >> A copy of the log files would help to understand what's happening. >> You can find them in /var/tmp/fstransform/ - before sending them, >> check there are no sensitive information inside, as they often >> contain the names of the files found in the partition being converted. > > I'm sorry, there is no log. Probably, as I canceled the transformation > this afternoon. At the moment, the server is doing something else, > thus I can't restart fstransform to produce logs. But I guess you're right, > it's probably an XFS bug when allocating large file, maybe only in combination > with this hardware. > > How would I disable the preallocation? But I guess without preallocation > progress wouldn't be faster? > > I guess the RAID6 array has some problem and needs to be rebuilt. > > Best, > > Olaf > > Hello Olaf, you're right, preallocation was enabled by default (AAAAAAARGH). To disable it, run "fstransform --prealloc=no ..." I will IMMEDIATELY disable it in github sources. I have good news and bad ones. The good news is that by using preallocation, the original file system was not modified at all when you stopped the conversion (except for the huge sparse file /mirror/.fstransform.loop.682). The bad news is that fstransform with preallocation is MUCH faster than normal conversion, and if that one was taking N months, the normal conversion will take about N*2 months. My experience with fast, directly connected SATA magnetic disks is about 1GB per minute: I've converted a 750 GB disk in about 12 hours. By fast, I mean >100GB/s peak speed (you can check it with "hdparm -t DEVICE") Unluckily, your 4.2 terabytes disk looks quite slower than that - but maybe it was due to the XFS errors, or maybe due to the huge number of hard links. Regards, Massimiliano |
|
From: Olaf L. <ol...@me...> - 2014-11-18 22:16:27
|
Hey! > The good news is that by using preallocation, the original file > system was not modified at all when you stopped the conversion > (except for the huge sparse file /mirror/.fstransform.loop.682). That's quite good to hear. Actually, fstransform cleaned up nicely and deleted the loop file afterwards. > The bad news is that fstransform with preallocation is MUCH faster > than normal conversion, and if that one was taking N months, the > normal conversion will take about N*2 months. > > [...] > Unluckily, your 4.2 terabytes disk looks quite slower than that > - but maybe it was due to the XFS errors, or maybe due to the huge number > of hard links. I guess it must be the number of hardlinks. The filesystem was quite fast when initially being created. And hdparm -t still yields ~ 100 MiB/s. Thanks for all the help and suggestions. Even I can't really use fstransform in this special case, it's sure a very nice tool to have! Best, Olaf |
|
From: Massimiliano G. <mas...@gm...> - 2014-11-18 23:53:56
|
On 11/18/14 23:18, Olaf Leidinger wrote:> Am Tue, 18 Nov 2014 23:01:26 +0100 > schrieb Massimiliano Ghilardi <mas...@gm...>: > >> On 11/18/14 22:34, Olaf Leidinger wrote: >>> [...] >>> I guess the RAID6 array has some problem and needs to be rebuilt. >>> >>> Best, >>> >>> Olaf >> >> That's a BIG RED flag. >> >> I wouldn't suggest running fstransform on a _degraded_ RAID disk even >> to my enemies... fstransform is a BIG gun and must be used with care. >> Using it on such a setup verges on being suicidal. > > Oh, no, this isn't what I wanted to say. The raid controller reports > perfect health conditions. I just meant, that there might be something > wrong internally (which isn't reported) due to the slow speed. But as > raw-reads (hdparm -t) report a nice speed of 100 MiB/s, I guess it's > XFS fault, not the fault of the underlying block layer. > Hello, this is much better - I mean, for the 4TB data on your 12TB RAID :) But even in the ideal case the conversion time would be at least 2-3 days, plus 1-2 more days to fill with zeroes the 8GB of free space. from your logs I can deduce something more: > 08:22:42 fsmove: progress: 0.3% done, 4.2 terabytes still to preallocate > 08:23:04 fsmove: progress: 0.6% done, 4.1 terabytes still to preallocate, estimated 1 hour and 25 minutes left > [...] > 08:42:00 fsmove: progress: 4.9% done, 4.0 terabytes still to preallocate, estimated 4 hours left > 08:49:40 fsmove: progress: 5.2% done, 3.9 terabytes still to preallocate, estimated 3 hours left Until here, it was reasonably fast: 5% of fsmove in 30 minutes. > 08:50:45 fsmove: progress: 5.5% done, 3.9 terabytes still to preallocate, estimated 1 hour and 40 minutes left > [...] > 09:50:06 fsmove: progress: 7.0% done, 3.9 terabytes still to preallocate, estimated 5 hours left here it slows down: 1 hour to go from 5% to 7% > 10:08:55 fsmove: progress: 7.4% done, 3.9 terabytes still to preallocate, estimated 1 day and 7 hours left > [...] > 10:34:43 fsmove: progress: 8.0% done, 3.8 terabytes still to preallocate, estimated 1 day and 20 hours left more slowdown: 45 minutes to go from 7% to 8% > 11:00:49 fsmove: progress: 8.3% done, 3.8 terabytes still to preallocate, estimated 1 day and 15 hours left > [...] > 15:05:12 fsmove: progress: 11.0% done, 3.7 terabytes still to preallocate, estimated 5 days left and it only gets worse. > (canceled at this point, as the log was spammed with these XFS messages all over, again) In my opinion, the slowness is a combination of the huge number of hard links and this kernel error message: > XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) Googling a bit, this thread contains some advice - "lowering all the dirty memory thresholds": http://oss.sgi.com/archives/xfs/2013-08/msg00733.html Pity, I hoped the on-disk inode cache would be enough to support this specific case. Regards, Massimiliano P.S. was /spare/inode_cache on a fast disk too? |
|
From: Olaf L. <ol...@me...> - 2014-11-19 00:25:50
|
Heya! > Googling a bit, this thread contains some advice - "lowering all the > dirty memory thresholds": > http://oss.sgi.com/archives/xfs/2013-08/msg00733.html > > Pity, I hoped the on-disk inode cache would be enough to support this > specific case. Yeah, I guess it's not really worth the effort to wait for weeks or months. It's more important, that the 2nd backup server starts backuping again. > Regards, > > Massimiliano > > > P.S. was /spare/inode_cache on a fast disk too? > Well, not a blazing-fast disk - a regular SATA HDD (hdparm -t yields 60 MiB/s) connected to the internal controller which is maybe 4 years old. |
Thanks for helping keep SourceForge clean.
X