Menu
▾
▴
dump-users
|
From: Brent B. <br...@ke...> - 2005-11-25 18:52:10
|
I'm trying to solve a problem that I've been having for a *long* time:
When I do multivolume dumps, doing 'restore -C' on them almost always
reveals corruption on files at and just after the point where the
archive spans to the next tape. I've learned from the man page for the
st driver that this is to be expected if write buffering is on (which it
is by default):
MT_ST_BUFFER_WRITES (Default: true)
Buffer all write operations in fixed block mode. If this option
is false and the drive uses a fixed block size, then all write
operations must be for a multiple of the block size. This
option must be set false to write reliable multi-volume
archives.
MT_ST_ASYNC_WRITES (Default: true)
When this options is true write operations return immediately
without waiting for the data to be transferred to the drive if
the data fits into the driver's buffer. The write threshold
determines how full the buffer must be before a new SCSI write
command is issued. Any errors reported by the drive will be
held until the next operation. This option must be set false to
write reliable multi-volume archives.
So...that would mean that buffered writes and async writes should be
turned off if you expect to span tapes with dump (or tar, or anything
else probably), right? So I do 'mt stoptions 4', and I still get:
BOT ONLINE IM_REP_EN
from 'mt status'. That's odd. The 'IM_REP_ON' at the end means it's
still doing things asyncronously. Again, from the 'st' driver man page:
GMT_IM_REP_EN(x): Immediate report mode. This bit is set if
there are no guarantees that the data has been physically
written to the tape when the write call returns. It is set zero
only when the driver does not buffer data and the drive is set
not to buffer data.
So apparently it ignored my 'mt stoptions 4' call, because IM_REP_EN is
still on, thus write buffers are still on, thus I'll probably get a
corrupted backup again if I try to span multiple tapes. Now I try
another mt command, 'mt drvbuffer 0' to just shut off all buffers
completely. Now 'mt status' gives me:
BOT ONLINE
Good! No more IM_REP_EN now. I'm safe, right? We'll see. So I get
ready to do a dump, and here's what that looks like:
village:~# mt erase
village:~# mt rewind
village:~# mt datcompression 2
Compression on.
Compression capable.
Decompression capable.
village:~# mt drvbuffer 0
village:~# mt stoptions 4
village:~# mt status
drive type = Generic SCSI-2 tape
drive status = 637534720
sense key error = 0
residue count = 0
file number = 0
block number = 0
Tape block size 512 bytes. Density code 0x26 (unknown).
Soft error count since last status=0
General status bits on (41000000):
BOT ONLINE
So the tape table of contents is erased, it's at the beginning, hardware
compression is on, all my nasty write-buffering is off, and 'mt status'
is giving me only "BOT ONLINE" with no "IM_REP_EN" to prove it's okay.
So I can dump? Here we go:
village:~# dump -0au /srv
DUMP: Date of this level 0 dump: Fri Nov 25 12:23:24 2005
DUMP: Dumping /dev/sdb6 (/srv) to /dev/nst0
DUMP: Label: /srv
DUMP: Writing 10 Kilobyte records
DUMP: mapping (Pass I) [regular files]
DUMP: mapping (Pass II) [directories]
DUMP: estimated 58195461 blocks.
DUMP: Volume 1 started with block 1 at: Fri Nov 25 12:24:28 2005
DUMP: 0.00% done at 3 kB/s, finished in 4931:10
DUMP: 0.00% done at 3 kB/s, finished in 4947:55
DUMP: Interrupt received.
DUMP: Do you want to abort dump?: ("yes" or "no") DUMP: Interrupt
received.
DUMP: Interrupt received.
DUMP: Do you want to abort dump?: ("yes" or "no") DUMP: Do you want
to abort dump?: ("yes" or "no") yes
DUMP: The ENTIRE dump is aborted.
village:~#
And that's what it does... Everything looks fine through Phase II, then
the tape drive starts, and it seems to be streaming just fine with no
shoeshining of the tape at all...except...the hard drive is showing
hardly any activity at all. What is it streaming? The tape motor is
running, the tape drive activity light is on...but the hard drive is
sending...nothing. Finally, as you can see above, dump ends up spending
so much time waiting to start Phase III that it prints its percent done
progress messages before it even gets that far. (And the progress
messages say it's doing 3kB/s, and that it will finish in about five
thousand hours.) All the while the tape drive motor is running, and
it's happily streaming *something*, though with nothing coming from the
hard drive, I can't imagine what. If I do 'mt tell' after aborting it,
I get:
village:~# mt tell
At block 3980.
So I'm now almost two megabytes into the tape media, I never even got to
Phase III, thus it didn't even start archiving data yet. Very, very
weird.
Sorry about the long detailed message -- usually if you post a brief
message, nobody can help you, so I thought I'd post the whole thing.
What I'm hoping someone here can do is just tell me what *they* do for
successful multivolume dumps, which can be verified by 'restore -C' as
good. I've seen lots of posts in the dump mailing lists from people
with this problem, and elsewhere, but no solutions, and it doesn't seem
so much a dump problem as a general tape drive problem, since it's the
SCSI tape driver that seems to create this scenario, and it would
probably happen just the same with tar or anything that tries to span
data across tape volumes on Linux.
Can anyone help with this? If you're successfully writing (and
verifying!) multivolume tape archives, how are you doing it? The
problem only shows up on a 'restore -C' -- during the write, it looks
okay -- so if you don't actually verify, you don't really know that you
aren't having this problem, too.
--
+ Brent A. Busby, UNIX Systems Admin + "It's like being +
+ James Franck / Enrico Fermi Institute + blindsided by a +
+ The University of Chicago + flying dwarf..." +
|
|
From: Kenneth P. <sh...@se...> - 2005-11-25 20:30:49
|
--On Friday, November 25, 2005 12:52 PM -0600 Brent Busby <br...@ke...> wrote: > I've seen lots of posts in the dump mailing lists from people with this > problem, and elsewhere, but no solutions, and it doesn't seem so much a > dump problem as a general tape drive problem, since it's the SCSI tape > driver that seems to create this scenario, and it would probably happen > just the same with tar or anything that tries to span data across tape > volumes on Linux. While I have no solution, I'd suggest checking with the linux-scsi mailing list, as someone there with a good understanding of the st driver may know what's going on. <http://vger.kernel.org/majordomo-info.html> <http://vger.kernel.org/vger-lists.html> Let us know what you find. It sounds like an interesting problem. |
|
From: Stelian P. <st...@po...> - 2005-11-28 20:19:36
|
Le vendredi 25 novembre 2005 =E0 12:30 -0800, Kenneth Porter a =E9crit : > --On Friday, November 25, 2005 12:52 PM -0600 Brent Busby=20 > <br...@ke...> wrote: >=20 > > I've seen lots of posts in the dump mailing lists from people with th= is > > problem, and elsewhere, but no solutions, and it doesn't seem so much= a > > dump problem as a general tape drive problem, since it's the SCSI tap= e > > driver that seems to create this scenario, and it would probably happ= en > > just the same with tar or anything that tries to span data across tap= e > > volumes on Linux. >=20 > While I have no solution, I'd suggest checking with the linux-scsi mail= ing=20 > list, as someone there with a good understanding of the st driver may k= now=20 > what's going on. >=20 > <http://vger.kernel.org/majordomo-info.html> > <http://vger.kernel.org/vger-lists.html> >=20 > Let us know what you find. It sounds like an interesting problem. I think he is already reading the linux-scsi mailing list, but in case he isn't it may be useful to also CC: the scsi tape driver maintainer, Kai Makisara, in your mails. Good luck. Stelian. --=20 Stelian Pop <st...@po...> |
|
From: Brent B. <br...@ke...> - 2006-01-07 06:55:18
|
Since some of you said to write back if I found a solution... After much searching, I did finally find the way to make multivolume archives reliable instead of corrupting at the end-of-tape points. It was back in the archives of this list, but was only mentioned there once and never again, so... mt stclearoptions async-writes buffer-writes I did this, and a 'restore -C' verified good across a 4-volume set of tapes. It makes me wonder if this problem is actually the reason that Amanda can't span tapes -- perhaps the developers never did get around this issue with the st tape driver, and gave up. (Officially, the word on Amanda is that you can't -- is this the only thing stopping them?) -- + Brent A. Busby, UNIX Systems Admin + "It's like being + + James Franck / Enrico Fermi Institute + blindsided by a + + The University of Chicago + flying dwarf..." + |
|
From: Eric J. <eje...@sw...> - 2006-01-07 16:05:00
|
Hi Brent, Thanks very much for posting this. One request for clarification: Do those options need to be cleared before *writing* the dump, or only when trying restore the dump? I suspect the former, but wanted to check. Also (and please forgive my ignorance of this topic), is there a straightforward way to see what options are set for the drive already? It wasn't obvious in the 'mt' man page. Finally, I assume that these options would need to be set/cleared after each reboot, but not in between? Thanks very much, Eric |
|
From: <br...@ke...> - 2006-01-08 03:43:27
|
On Sat, Jan 07, 2006 at 11:04:47AM -0500, Eric Jensen wrote:
> Thanks very much for posting this. One request for clarification:
>
> Do those options need to be cleared before *writing* the dump, or only
> when trying restore the dump? I suspect the former, but wanted to
> check.
Before writing. I did the successful verify that way too, with
buffering and async write turned off (the same as during the dump), but
I'd imagine that on reads, the caching probably doesn't do any harm at
either setting and doesn't really matter.
> Also (and please forgive my ignorance of this topic), is there a
> straightforward way to see what options are set for the drive already?
> It wasn't obvious in the 'mt' man page.
I don't know a way to do that without changing anything, but I did
notice that whenever *any* tape driver option is changed with
stclearoptions type commands, the driver seems to very helpfully send
the new current state of *all* of the options to syslog. There's
probably a way to achieve that without changing an option, but I don't
know offhand. Also, the defaults for all of the tape driver settings
are in the man page for the 'st' driver, so if you haven't changed
anything, that should tell you what everything should be starting at.
> Finally, I assume that these
> options would need to be set/cleared after each reboot, but not in
> between?
Yes, the settings seem to be persistent for as long as the system is
booted up. The mt-st package includes a program called stinit which
gets run at boot time by an init script. Stinit looks at
/etc/stinit.def for any sysadmin-supplied settings for any tape drives
you'd like to have autoconfigured at boot time, and uses them to set up
all your tape drives for you ('man stinit').
--
+ Brent A. Busby, UNIX Systems Admin + "It's like being +
+ James Franck / Enrico Fermi Institute + blindsided by a +
+ The University of Chicago + flying dwarf..." +
|
|
From: Stelian P. <st...@po...> - 2006-01-07 23:16:53
|
Le samedi 07 janvier 2006 =E0 00:54 -0600, Brent Busby a =E9crit : > Since some of you said to write back if I found a solution... Thanks for keeping us informed... > After much searching, I did finally find the way to make multivolume=20 > archives reliable instead of corrupting at the end-of-tape points. It=20 > was back in the archives of this list, but was only mentioned there onc= e=20 > and never again, so... Indeed, this recalls me something. > mt stclearoptions async-writes buffer-writes >=20 > I did this, and a 'restore -C' verified good across a 4-volume set of=20 > tapes. It makes me wonder if this problem is actually the reason that=20 > Amanda can't span tapes -- perhaps the developers never did get around=20 > this issue with the st tape driver, and gave up. (Officially, the word= =20 > on Amanda is that you can't -- is this the only thing stopping them?) No, the fact that Amanda does not support multiple tapes has nothing to do with tape driver issues. It's just that the single tape assumption is deeply buried into the Amanda's algorithms and I think they don't have enough active developers to tackle this. Stelian. --=20 Stelian Pop <st...@po...> |
|
From: Brent B. <br...@ke...> - 2006-01-11 23:02:06
|
Can anyone imagine what might have happened here: I do 'restore -rv -b64 -f /dev/st0' to restore a level 0 dump from DAT tape. All the directories seem to get restored, and the stdout prints 'Making node' for each of them as they are encountered on the tape -- but alarmingly, there are no regular files! I can also do 'restore -i' and see all the files in the catalog, but when extracting, I never get any data, just directories. I tried to reassure myself that it was just extracting the directories before it gets to the data (and I never did play out the whole tape to test that -- it would take hours, and I'm still checking things), but the thing that really scares me is that there are long pauses -- sometimes *very* long pauses, about as long as the actual extraction of regular files might take, in between the creation of some of these dir nodes. That doesn't sound good. So, does restore always create all dir nodes on extraction before moving on to regular files? If so, is there any reason there would be pauses of several minutes in between the dirs? If not, then is there some way you can imagine that my settings could be correct enough to let it read the dump label, the catalog, and the path names, but not correct enough to see the data? It's very weird. And by the way, this _is_ mission critical, so I do appreciate any help you can give here.... -- + Brent A. Busby, UNIX Systems Admin + "It's like being + + James Franck / Enrico Fermi Institute + blindsided by a + + The University of Chicago + flying dwarf..." + |
|
From: Brent B. <br...@ke...> - 2006-01-12 04:07:10
|
Looks like my data did finally extract. Sorry for raising the alarm -- I was a little alarmed myself... <g> -- + Brent A. Busby, UNIX Systems Admin + "It's like being + + James Franck / Enrico Fermi Institute + blindsided by a + + The University of Chicago + flying dwarf..." + |
Thanks for helping keep SourceForge clean.
X