A brief history. This is a 4-disk mergerfs / snapraid setup. Disk "d2" was dying, so I added a new disk as "d5", migrated data to it, removed the old disk, replaced "d2" with the (new) "d5". In my first attempt to sync after doing this, I got a crash. I haven't loaded it under gdb or done any further debugging.
-Nick
$ snapraid -v -v sync
Self test...
Loading state from /var/lib/snapraid/00/snapraid.content...
WARNING! Renaming disk 'd5' to 'd2'
Internal inconsistency when allocating file 'tv/complete/MST3K/MST3K - Complete 35 DVD Collection/MST3K DVD 31/MST3K - S10E01 - Soultaker.avi' at position '0/2806' for existing extent '0:1' in disk 'd2'
Stacktrace of snapraid v12.1-1-gbcb0ab2, gcc 7.5.0, 64-bit, PATH_MAX=4096 [bt] 01: snapraid(fs_allocate+0x144) [0x41b004]
addr2line: 'snapraid': No such file
exit:1 [bt] 02: snapraid() [0x42219a]
addr2line: 'snapraid': No such file
exit:1 [bt] 03: snapraid(state_read+0x211) [0x423871]
addr2line: 'snapraid': No such file
exit:1 [bt] 04: snapraid(main+0x10f9) [0x409559]
addr2line: 'snapraid': No such file
exit:1 [bt] 05: /lib64/libc.so.6(__libc_start_main+0xef) [0x7f983499429d]
??:0 [bt] 06: snapraid(_start+0x2a) [0x409dda]
addr2line: 'snapraid': No such file
exit:1
Please report this error to the SnapRAID Forum: https://sourceforge.net/p/snapraid/discussion/1677233/
Aborted (core dumped)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
This is commit b424a503a0179af5a8f19ad548e5782626458411 (12.2 tag + "Add checksums for v12.2")
nick@imrahil:~/build/snapraid> gcc --version
gcc (SUSE Linux) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Anything else that would be helpful, let me know.
Thanks!
-Nick
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It's a bit strange, because Snapraid recognizes that your new d5 was the old d2, and it tries to rename it. This means that the filesystem UUIDs of the two disk are the same.
How did you initialize and migrate data to the new d5 ?
A possible workaround could be to force a UUID change, like with:
tune2fs-Urandom/dev/sdX1
Otherwise, you can also temporarily remove d5 from the snapraid array configuration, run a sync, add it back, and sync again.
Anyway, I'll investigate this condition, but it may take some time.
Ciao,
Andrea
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It's a bit strange, because Snapraid recognizes that your new d5 was the old d2, and it tries to rename it. This means that the filesystem UUIDs of the two disk are the same.
Yeah, I thought that this should work. Obviously.
How did you initialize and migrate data to the new d5 ?
Added the disk normally, with an XFS if it makes a difference.
Created a d5 entry, and added it to mergerfs via xattr.
Checked things out, made sure it all looked reasonable.
/disks/av/01# rsync -av --progress ./snapraid /disks/av/04
Note: One of the snapraid content files is stored in each of
/disks/av/<nn>/snapraid</nn>
Fixup fstab, and other files, rebooted (I couldn't umount one of
the filesystems, so this was required).
# snapraid -v -v sync
A possible workaround could be to force a UUID change, like with:
tune2fs -U random /dev/sdX1
Otherwise, you can also temporarily remove d5 from the snapraid array configuration, run a sync, add it back, and sync again.
Anyway, I'll investigate this condition, but it may take some time.
I will try one of these in an hour or two and report back.
In the meantime, I did some poking. In fs_allocate(), parity_extent
points at a completely different file than does extent and
file_extent. I haven't dug into the code enough to figure out how or
why this has occurred, but I'd be glad to help from this end.
Thanks much for the help!
-Nick
Last edit: Nick LeRoy 2022-10-26
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Do you mean that you intentionally created a new filesystem in the new disk with the same UUID of the old one?
SnapRAID uses the filesystem UUID to identify if a disk is replaced. This could have invalidated some assumptions made by SnapRAID. At least, it's not a condition I ever tested.
Ciao,
Andrea
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Do you mean that you intentionally created a new filesystem in the new disk with the same UUID of the old one?
No, new (random) UUID.
SnapRAID uses the filesystem UUID to identify if a disk is replaced. This could have invalidated some assumptions made by SnapRAID. At least, it's not a condition I ever tested.
I have observed this in the past. I've done this "replace the disk"
thing once before with SnapRAID.
Great tool, by the way, thank you!
-Nick
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
A brief history. This is a 4-disk mergerfs / snapraid setup. Disk "d2" was dying, so I added a new disk as "d5", migrated data to it, removed the old disk, replaced "d2" with the (new) "d5". In my first attempt to sync after doing this, I got a crash. I haven't loaded it under gdb or done any further debugging.
-Nick
$ snapraid -v -v sync
Self test...
Loading state from /var/lib/snapraid/00/snapraid.content...
WARNING! Renaming disk 'd5' to 'd2'
Internal inconsistency when allocating file 'tv/complete/MST3K/MST3K - Complete 35 DVD Collection/MST3K DVD 31/MST3K - S10E01 - Soultaker.avi' at position '0/2806' for existing extent '0:1' in disk 'd2'
Stacktrace of snapraid v12.1-1-gbcb0ab2, gcc 7.5.0, 64-bit, PATH_MAX=4096
[bt] 01: snapraid(fs_allocate+0x144) [0x41b004]
addr2line: 'snapraid': No such file
exit:1
[bt] 02: snapraid() [0x42219a]
addr2line: 'snapraid': No such file
exit:1
[bt] 03: snapraid(state_read+0x211) [0x423871]
addr2line: 'snapraid': No such file
exit:1
[bt] 04: snapraid(main+0x10f9) [0x409559]
addr2line: 'snapraid': No such file
exit:1
[bt] 05: /lib64/libc.so.6(__libc_start_main+0xef) [0x7f983499429d]
??:0
[bt] 06: snapraid(_start+0x2a) [0x409dda]
addr2line: 'snapraid': No such file
exit:1
Please report this error to the SnapRAID Forum:
https://sourceforge.net/p/snapraid/discussion/1677233/
Aborted (core dumped)
Before anybody asks:
This is commit b424a503a0179af5a8f19ad548e5782626458411 (12.2 tag + "Add checksums for v12.2")
nick@imrahil:~/build/snapraid> gcc --version
gcc (SUSE Linux) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Anything else that would be helpful, let me know.
Thanks!
-Nick
Last edit: Nick LeRoy 2022-10-25
Looking again at the dump, I realized what it was, so I reran it with the full path. Improved backtrace:
root@imrahil[521]:/var/log/snapraid# /usr/local/bin/snapraid -v -v sync
Self test...
Loading state from /var/lib/snapraid/00/snapraid.content...
WARNING! Renaming disk 'd5' to 'd2'
Internal inconsistency when allocating file 'tv/complete/MST3K/MST3K - Complete 35 DVD Collection/MST3K DVD 31/MST3K - S10E01 - Soultaker.avi' at position '0/2806' for existing extent '0:1' in disk 'd2'
Stacktrace of snapraid v12.1-1-gbcb0ab2, gcc 7.5.0, 64-bit, PATH_MAX=4096
[bt] 01: /usr/local/bin/snapraid(fs_allocate+0x144) [0x41b004]
/local/home/nick/build/snapraid/cmdline/elem.c:1270
[bt] 02: /usr/local/bin/snapraid() [0x42219a]
/local/home/nick/build/snapraid/cmdline/state.c:2038
[bt] 03: /usr/local/bin/snapraid(state_read+0x211) [0x423871]
/local/home/nick/build/snapraid/cmdline/state.c:3817
[bt] 04: /usr/local/bin/snapraid(main+0x10f9) [0x409559]
/local/home/nick/build/snapraid/cmdline/snapraid.c:1342
[bt] 05: /lib64/libc.so.6(__libc_start_main+0xef) [0x7f547286c29d]
??:0
[bt] 06: /usr/local/bin/snapraid(_start+0x2a) [0x409dda]
/home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:122
Hi Nick,
It's a bit strange, because Snapraid recognizes that your new d5 was the old d2, and it tries to rename it. This means that the filesystem UUIDs of the two disk are the same.
How did you initialize and migrate data to the new d5 ?
A possible workaround could be to force a UUID change, like with:
Otherwise, you can also temporarily remove d5 from the snapraid array configuration, run a sync, add it back, and sync again.
Anyway, I'll investigate this condition, but it may take some time.
Ciao,
Andrea
On Wed, Oct 26, 2022 at 1:51 AM Andrea Mazzoleni
amadvance@users.sourceforge.net wrote:
Yeah, I thought that this should work. Obviously.
Added the disk normally, with an XFS if it makes a difference.
Created a d5 entry, and added it to mergerfs via xattr.
/disks/av/01# rsync -av --progress --exclude=snapraid
--remove-source-files . /disks/av/04
Checked things out, made sure it all looked reasonable.
/disks/av/01# rsync -av --progress ./snapraid /disks/av/04
Note: One of the snapraid content files is stored in each of
/disks/av/<nn>/snapraid</nn>
Fixup fstab, and other files, rebooted (I couldn't umount one of
the filesystems, so this was required).
# snapraid -v -v syncI will try one of these in an hour or two and report back.
In the meantime, I did some poking. In fs_allocate(), parity_extent
points at a completely different file than does extent and
file_extent. I haven't dug into the code enough to figure out how or
why this has occurred, but I'd be glad to help from this end.
Thanks much for the help!
-Nick
Last edit: Nick LeRoy 2022-10-26
Do you mean that you intentionally created a new filesystem in the new disk with the same UUID of the old one?
SnapRAID uses the filesystem UUID to identify if a disk is replaced. This could have invalidated some assumptions made by SnapRAID. At least, it's not a condition I ever tested.
Ciao,
Andrea
On Wed, Oct 26, 2022 at 1:18 PM Andrea Mazzoleni
amadvance@users.sourceforge.net wrote:
No, new (random) UUID.
I have observed this in the past. I've done this "replace the disk"
thing once before with SnapRAID.
Great tool, by the way, thank you!
-Nick