Re: [Dar-support] Multiple slices on LTO6 tape

[Petr S. <sez...@se...>]

Hi all,




I will try to calm down the enthusiasm or adoration of LTFS for tape backup.

I have read about 30 google links, discussions and have browsed the first 10
pages of google results (many links are just presentation of commercial 
products.) I have also consulted a colleague who has been using LTO6,7, and 
now LTO8 for years for storing TBs of astronomical images from meteor 
cameras . And I can also say that part of my scepticism is based on a 15+ 
years of experience with LTO-1 (even first European samples from IBM)  and 
30+years experience with  magnetic tapes QIC 60 MB on PC AT and 150MB, 250MB
on Sun Sparc 1....), 2 and 4GB on DAT.




In general, the operation of tape interface is basically the same for tens 
of years (even before QIC the half-inch tapes with vacuum loading- so sexy 
in pictures of mainframe halls - were working similarly on unix machines).

The ancient name of the tape device was also STREAMER. So the idea is based 
on continuous stream of data which are written forward and backward after 
shifting the heads up and down. The so called helican scan drives as DAT 
were just slowly moving writting short tilted tracks as a VHS cassette. The 
data were written in fixed blocks and files were ended by tapemarks. Last 
one is called EOD (there are two of them) . The classical  ways of storing 
data were two - tar and dd . Some ideas from that time was also based on 
cpio or pax format - but is was more after the SunOS was turned into Solaris
- AT&T based. For backup of system the programs "dump"   and "restore" 
(ufsdump ufsrestore on Solaris) were most common.

For driving DAT tapes a new kernel on SunOS had to be compiled with 
extensions (larger capacity, compression etc.)

Also the mtx program for driving autoloaders and tape robots is still valid 
after long time.

I was running also various automating backup SW as Amanda (on DAT robot with
6 tapes) , afbackup (on LTO-1 with 7 tapes robot ). So far I did not dare to
try Bacula or BareOS , despite I have read many times the whole doc. 





The problem occured when the vendors wanted to sell the LTO drives to home 
users with MS windows - as the professional IT marked was saturated by LTO -
extremely expensive (for home market various Exabyte drives DLT and AIT 
units much cheaper were intended before). And as Windows users like to click
and need to see trees of files, the idea was born to simulate disk on tape  
 (in fact old SunOS based on BSD Unix was able to boot from tape similarly 
to disk boot) defining separate index . But this required the support for 
multiple partitions on HW level (also augmented by the RF chip which is in 
every LTO medium). And this was not available until LTO5. 





The idea of LTFS is very simple but it seems that despite proclamation of 
openness , the devil is in details. Every drive manufacturer has proprietary
extensions in tape interface, which is somehow used by other proprietary 
vendors of tape backup systems.  So in fact LTFS is compatible accross all 
devices and systems, but only on a basic level.




E.g. many vendors claim that they support spanning of backups accross 
multiple tapes, which would be fantastics. 


And in fact there is a specification in the standard  




https://www.snia.org/sites/default/files/technical-work/ltfs/release/SNIA-
LTFS-Format-2.4.0-TechPosition.pdf





(also see older version on http://docs.oracle.com/cd/E19957-01/LTFSSpec/
LTFSSpec.pdf) 





Chapter or Annex F.1 speaks about spanning accross tapes, but it is not 
clear whether it is implemented in all LTFS SW packages. But here 




https://www.oracle.com/webfolder/s/delivery_production/docs/FY14h1/doc4/
281882-LTFS%20For%20Dummies.pdf  





is nothing about it. 


I have a feeling that the full LTFS advantage is used only by expensive 
proprietary solutions (e.g. the yotta is expensive as well - order of 
hundreds EUR) 





I would appreciate if anybody will try to save a file larger then tape size 
on LTFS in his/her environment. But from brief description of several linux 
ltfs implementations it seems that the system will respond with error.




In addition what I know (I did not dare to destroy my tapes with ltfs) - 
unlike tar dd cpio and other stuff - where you can write on tape what you 
want and  can overwrite it from every tape mark by a new content, 


the ltfs must be initialized - and as it is done in HW (from LTO5 higher 
only) , you cannot easily use such tape and overwrite it. It must be erased 
in a special way.  Some people have reported than the tape was not usable 
anymore after such attempt (I can imagine the implementation of this 
unformatting methods may not be perfect in non-commercial solutions.)







I will also react to the statements given below:




As LTFS does not support special files as devices, system files, files with 
special access rights, sparse files (although in spec is something about 
sparse files) symlinks etc , it is not good replacement of traditional 
solutions.

Of course I can create a cpio, tar or dar archive and then save it on LTFS 
tape as a one file ...

BUT as said above I probably will ot be able to span the large file over 
more tapes..

So with additional complexity added I am at the same problem with which I 
was opening this thread. 


How to span big backup over multiple tapes...





In fact I have in one directory or in one disk partition typically several -
up to ten TB of astronomical data which must be stored in consistent way - 
there are many subdirectories and would be rather difficult to select when 
to tear the tree 





So taking it back and forth I think the dar with dar_split or dar_xform (for
fixed size slices) is a great tool.

The only problem I have (and the reason why I did not try Bacula) is that I 
want to be able to read the tape on low level without installing complex 
system - of course I need dd or tar as well - better 'star' from  Schilly 
tools.

So the static binary of dar (and dar_split) can satisfy my needs. But I 
often need to read an abandoned tape after years (no label or catalog is 
usually available after several years ;-). So the dar catalogue (or 
extracted index ) is perfect for this purpose - but the slices should be 
somehow readable individually, which seems to be a design problem of dar.







Anyway I am looking forward to hear here about using LTFS with dar or in 
fact any experience with LTO or other tapes and DAR. 





Best regards, Petr

  

---------- Původní e-mail ----------
Od: me...@rm...
Komu: dar...@li...
Datum: 9. 6. 2022 11:51:01
Předmět: [Dar-support] Multiple slices on LTO6 tape 
"Dear Petr, Dear Denis, 

> BTW - LTFS has a lot of problems - so it is not good replacement for dar 
> on tapes (it seems not to support multiple tapes either, moreover the 
> symlink and special files cannot be written - so you cannot backup the 
> system partition with it ... 

Petr, you are right, LTFS cannot user Special file names, etc. or 
rights. But that is all managed by DAR. 
All LTFS has to do is: 
1. save content -> dar slices and maybe the dar catalogue files 
2. Restore everythere -> Can be read on Linux, Windows, Mac 
3. Be reliable -> ECC is used on LTO-Tapes [1] 


IMHO LTFS is the ideal combination for using dar on tapes. You do not 
have to deal with dd / buffers / running out space. Just use the tape 
like an external drive. 

Then I finished my test, I'll create a Repository on github, containing 
a Windows-CMD or Powershell-Script and an Linux-Bash-Script to backup 
and restore files on LTFS using dar. 

Beside of the gui, DAR is able to do all the stuff other tools in the 
LTFS-field do, e.g. YoYotta [2]: 
"During the archive YoYotta calculates both MD5 and xxHash checksums and 
then automatically uses these to read back and verify every file. 
YoYotta remembers the contents of every tape created so later on you can 
search to find the location of shots." 

Is anyone interested in this, or already using LTFS and DAR? 

Does anyone feel like testing the scripts or even just commenting? 

[1] https://www.lto.org/2019/12/how-does-lto-tape-do-it/ 
[2] https://yoyotta.com/yoyotta/ltfs.html 

"