Dump/Restore ext2/3/4 filesystem backup / Bugs / #157 dump fails to restore incremental if directory was removed

Frank Bicknell - 2012-06-21

seems important as this is a restore failure

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Frank Bicknell - 2012-06-21

priority: 5 --> 7
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Frank Bicknell - 2012-06-21

Tried with an ext2 filesystem; same result.

However, I did answer "n" twice:

----------------------------------
rename ./testing/foo to ./testing/RSTTMP0155649
Calculate node updates.
Find unreferenced names.
deleteino: out of range 0
abort? [yn] n
deleteino: 0 not found
abort? [yn] n
^Crestore interrupted, continue? [yn] n
----------------------------------

And after the second 'n', it launched into CPU 100% land. I left it
there until I was pretty sure it wasn't coming out and as you see above
interrupted it.

A subsequent umount and fsck found the filesystem ok. Just that funnly
"leftover" where foo was before.

So I've decided to try this tactic on my own system test I was running
earlier and see if it does the same for a live system.

More to come.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Frank Bicknell - 2012-06-22

I went back and looked up Christoph's e-mail here in sourceforge and it finally dawned on me that he was trying to say that back-revving to 4b42 was what produced a successful result.

So I tracked down this version in binary form and he's right: it works! So not sure whether it was 43 or 44 that introduced the problem, but from the email thread it was broken in 43.

My test result (anti climactic now)

------------------------------------------
...
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

bick-ubtu3:~$ sudo mount /dev/mapper/loop0p1 /mnt/t
bick-ubtu3:~$ cd /mnt/t
bick-ubtu3:/mnt/t$ sudo restore -rf /tmp/level0.dump
restore: ./lost+found: File exists
bick-ubtu3:/mnt/t$ sudo restore -rvf /tmp/level1.dump
Verify tape and initialize maps
Input is from a local file/pipe
Input block size is 32
Dump date: Fri Jun 22 09:06:36 2012
Dumped from: Fri Jun 22 09:05:59 2012
Level 1 dump of /mnt/t on bick-ubtu3:/dev/mapper/loop0p1
Label: none
Begin incremental restore
Initialize symbol table.
Extract directories from tape
Mark entries to be removed.
rename ./testing/foo to ./testing/RSTTMP0131074
Calculate node updates.
Find unreferenced names.
Remove old nodes (directories).
Remove node ./testing/RSTTMP0131074
Extract new leaves.
Check pointing the restore
Add links
Set directory mode, owner, and times.
Check the symbol table.
Check pointing the restore
bick-ubtu3:/mnt/t$ ls -lR
.:
total 1736
drwx------ 2 root root 16384 Jun 22 09:04 lost+found
-rw------- 1 root root 1756656 Jun 22 09:07 restoresymtable
drwxr-xr-x 2 root root 4096 Jun 22 09:06 testing
ls: cannot open directory ./lost+found: Permission denied

./testing:
total 0
bick-ubtu3:/mnt/t$
------------------------------------------

Perfect!

So I guess just consider this a long-winded submission of a bug report for 4b43 as tested in 4b44...

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Frank Bicknell - 2012-06-22

I was able to successfully restore dumps (including incremental) created with 0.4b44 using b42. So clearly the problem is in restore, not dump.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

sangamesh - 2012-07-13

I had the same problem and decided to dig in a little bit. I found the problem to be that the deleted directory's entry structure is getting added twice to the remove list. In "findunreflinks" function in restore.c the deleted directory's entry structure is added again to the remove list, this makes the removelist a circular linked list with the entrys next poniting to itself and then it goes into an infinite loop.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

John Cairns - 2014-01-20

I experienced this bug as well. I was able to work around by using restore in interactive mode (-i).

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Andy Valencia - 2014-04-11

Just bit by this in Debian Wheezy, fully up-to-date.
So, the maintainer realizes that the headline is basically "dump project no longer supports incremental backups"? I had to go dig up a version 42 and build it for myself to get us back online here. Sobering.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ernest Beinrohr - 2014-06-12

same here. just happened. versio 44 cannot restore, downgrade to 42 allowed me to restore.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ben Harris - 2014-08-01

My testing indicates that this bug was introduced in revision 1.39 of restore/restore.c. If I build 0.4b44 with restore.c rolled back to 1.38, the bug doesn't manifest. If I roll restore.c forward to 1.39, the bug appears.

I've attached a tarball of a minimal test case to save others else the trouble of creating one. It contains level0.dump and level1.dump, and running the following as root should demonstrate the problem:

mkdir restoretest cd restoretest restore -rf ../level0.dump restore -rf ../level1.dump

testcase.tar.gz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ben Harris - 2014-08-01

More analysis, building on sangamesh's work above:

The bug is that in findunreflinks(), the code for removing unreferenced directories unconditionally calls deleteino() and then adds the node to removelist even though this might already have been done. This causes a panic in deleteino() because e_ino is zero, and causes an infinite loop when anything tries to traverse removelist. The fix will be to skip directories that are already scheduled for deletion.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Ben Harris - 2014-08-01

And here's a possible patch to fix the problem. It assumes that if an entry has e_ino == 0, that means it's already been passed to deletino() and added to removelist. I think this is true. It's certainly the case that all additions to removelist have e_ino == 0, so this should stop the hangs and the panics from removino(). At worst, it will leave spurious directories around after a restore.

restore-rmdir.patch

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mike Frysinger - 2016-05-09

thanks, pushed in commit fe2d1f1ee25e363900592260f7fb0223e3a64464

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mike Frysinger - 2016-05-09

status: open --> closed-fixed

Group: -->
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

dump fails to restore incremental if directory was removed

Group

Searches

Help

#157 dump fails to restore incremental if directory was removed

Discussion