redoubt-checkins Mailing List for redoubt: secure offsite backups

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Update of /cvsroot/redoubt/docs
In directory usw-pr-cvs1:/tmp/cvs-serv17668

Modified Files:
	index.html 
Added Files:
	client.html 
Log Message:
beginnings of client specification

--- NEW FILE: client.html ---
<html>

<head>

<title>Redoubt: client specification</title>

</head>

<body bgcolor="#ffffff">

<h1>Redoubt: client specification version 0.1</h1>
<p>
<STRONG>SUBJECT TO CHANGE WITHOUT NOTICE</STRONG>

<h2>Backup Process</h2>
<p>
The backup process is as follows:

<h3>File Listing</h3>
<p>
Find all files that should be backed up, taking into account exclude
and include expressions (stored as part of the client configuration),
but not taking into account previous backups or the current state of
the server.  This list of files is stored temporarily on disk (due to
its potentially large size) including file metadata.
<p>
[TODO: specification for the listing file]

<h3>File Selection</h3>
<p>
During the selection stage, the client determines the exact work to be
done during the backup.  This includes:
<ul>
<li>list of files to be deleted
<li>list of files for full archival (metadata and data)
<li>list of files for metadata-only update
</ul>
<p>
Take the list of files to be backed up and compare with the client
index (the known state of the backup archive, specification below).
Files that have been removed since the last backup are marked for
deletion, files that changed will be marked for full archival or
metadata-only update (subject to the configuration of the client,
allowing less important files to be backed up less often).

<h3>File Deletion</h3>
<p>
Next, files to be deleted are removed from the archive (in order to
reduce usage on the server before sending new data).
<p>
Each deletion request is sent to the storage agent.

<h3>File Packaging</h3>
<p>
For each file, the packaging agent creates a cpio-formatted archive,
compresses the archive (unless additional compression would be
pointless such as for files that are compressed or contain random
data) and encrypts it.  It then checks the client index to see if that
file already has been assigned a name.  If not, a random name is
assigned to the file and stored in the client index.  Optionally, the agent
can create files with random keys that are stored on the server too
and used to encrypt files.
<p>
[TODO: handling of files that are hard links]
<p>
Each encrypted file is sent to the storage agent.

<h3>Client Index Updates</h3>
<p>
As the storage agent confirms each transaction (deletion or package),
the state of each file is updated in the client index file.  Deleted
files are marked as deleted and are only removed from the client index
after a configurable amount of time has passed (and the storage agent
confirms that it has deleted all versions of the file).
<p>
The client index is not saved on the server due to possibly very large
size.

<hr>

<h2>Restoration Process</h2>
<p>
If the client index exists, the client can:
<ul>
<li>retrieve an individual file from the server as it was last backed up
<li>ask the storage agent for a list of available versions of the file
and then retrieve one of those versions
</ul>
If the client index is lost, the client has several options to offer:
<ul>
<li>start from scratch (delete everything & full backup)
<li>download minimum to restore index (virtually everything, the
latest copy of each non-deleted file [TODO: first block of each file?])
<li>do a complete restore
</ul>

<hr>

<h3>Appendix: client index specification</h3>
<p>
The client index is stored as a linked list of variable length
entries.  Each entry contains the entry length, the original file name
and its length, the metadata information and its length, and the
random file name and its length.
<p>
By using variable length entries, it is possible to store long
original file names without wasting disk space and also to expand the
metadata information or support alternative client operating systems.
<p>
The structure of each entry is shown here:
<p>
<table border=1 cellpadding=5>
<tr><td>entry length<td>unsigned 16-bit integer, little-endian
<tr><td>filename length<td>unsigned 16-bit integer, little-endian
<tr><td>filename string<td>char array, NUL terminated
<tr><td>metadata length<td>unsigned 16-bit integer, little-endian
<tr><td>metadata C struct<td>on Unix systems, struct stat
<tr><td>random name length<td>unsigned 8-bit integer
<tr><td>random name string<td>char array, NUL terminated
</table>
<p>
Length field values count the terminating NUL characters in the
length calculation.
<p>
[TODO: allow length of zero for random name length so format can be
reused for temporary listing during discovery stage?]
<p>
[TODO: sorting of list or use of hash or tree for faster lookups and
comparison against temporary listing?]

<h3>Appendix: cpio specification</h3>
<p>
This format matches the standard "newc" or "crc" cpio formats except
that the trailer has explicitly been removed to save space.
<pre>
[cpio archive] = [cpio file] | [cpio archive] + [cpio file]

[cpio file] = [cpio header] + [cpio file name] + [cpio file data]

[cpio header] = All the fields in the header are ISO 646 strings of
		hexadecimal numbers (case does not matter, but
		lowercase is used), left padded with '0' character,
		not NUL terminated.

                field name	bytes	notes
                [c_magic]	6	"070701" for "new" portable format;
					"070702" for CRC format
		[c_ino]		8
		[c_mode]	8	definition below
		[c_uid]		8
		[c_gid]		8
		[c_nlink]	8
		[c_mtime]	8
		[c_filesize]	8	must be 0 for FIFOs and directories
		[c_maj]		8
		[c_min]		8
		[c_rmaj]	8	only valid for devices, 0 otherwise
		[c_rmin]	8	only valid for devices, 0 otherwise
		[c_namesize]	8	count includes terminating NUL
		[c_chksum]	8	0 for "new" portable format;
					for CRC format the sum of all the
					bytes in the file

[cpio file name] = [file name] + '\0' + [pad 4]

[cpio file data] = [file data] + [pad 4]

[pad 4] = 0-3 bytes to end field on 4-byte boundary

[c_mode] = c_mode bits defined as follows (in hexadecimal and
           traditional octal)

/* st_mode bits */
#define C_ISSOCK	0000c000	00000140000
#define C_ISLNK		0000a000	00000120000
#define C_ISCTG         00009000	00000110000
#define C_ISREG         00008000	00000100000
#define C_ISBLK         00006000	00000060000
#define C_ISDIR         00004000	00000040000
#define C_ISCHR         00002000	00000020000
#define C_ISFIFO        00001000	00000010000
#define C_ISUID         00000800	00000004000
#define C_ISGID         00000400	00000002000
#define C_ISVTX         00000200	00000001000
#define C_IRUSR         00000100	00000000400
#define C_IWUSR         00000080	00000000200
#define C_IXUSR         00000040	00000000100
#define C_IRGRP         00000020	00000000040
#define C_IWGRP         00000010	00000000020
#define C_IXGRP         00000008	00000000010
#define C_IROTH         00000004	00000000004
#define C_IWOTH         00000002	00000000002
#define C_IXOTH         00000001	00000000001
</pre>

</body>

</html>

Index: index.html
===================================================================
RCS file: /cvsroot/redoubt/docs/index.html,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** index.html	2002/01/20 20:23:36	1.3
--- index.html	2002/01/21 00:18:03	1.4
***************
*** 118,121 ****
--- 118,122 ----

  <ul>
+ <li><a href=http://redoubt.sourceforge.net/client.html>Client Specification</a>
  <li><a href=http://redoubt.sourceforge.net/protocol.html>Client/Server Protocol</a>
  </ul>

redoubt-checkins Mailing List for redoubt: secure offsite backups

redoubt-checkins — used to report CVS checkins