Quillen S3 Backup Code
Status: Alpha
Brought to you by:
grourk
File | Date | Author | Commit |
---|---|---|---|
src | 2007-08-16 | grourk | [r19] Overhaul of the chunk database system. Still s... |
README | 2007-07-01 | grourk | [r15] Significant changes to make Quillen more robust... |
quillen.config | 2007-06-04 | grourk | [r12] Adding sample config file. |
QUILLEN - By Greg Nelson ------------------------ Quillen is designed to backup your important documents or other data to your Amazon S3 account. Information about Amazon S3 can be found at http://www.amazon.com/s3. You must obtain an S3 account from Amazon prior to using Quillen. The way Quillen works is it will split your files into chunks and store those individual chunks on S3. Chunks are variable size and are extracted based on the file contents. This chunking system is very efficient because it minimizes the amount of data that has to be transferred between successive versions of a file. Say, for example, you have a file foo.txt that is 1MB. When you back up this file, Quillen splits it into chunks and then stores a "shadow file" that tells Quillen how to re-assemble the file later (during a restore operation). The chunks are identified by a hash of their contents. Therefore, chunks can be shared between files that have similar content. Now, say you append a single byte to the beginning of foo.txt. If Quillen used fixed-size chunks at fixed offsets, then every chunk in the file would have changed! If each chunk is 8K, then all 128 existing chunks would have to be deleted from S3 and 128 chunks would have to be re-uploaded to S3, each only differing by 1 byte. (Actually, 129 chunks would be uploaded and the last chunk would contain only one byte). Quillen, on the other hand, will only need to change one single chunk. Quillen's chunking system is very robust to all kinds of changes made to a file. It therefore does a good job of minimizing data transfer. And, since chunks can be shared amongst files, it does a good job of minimizing data storage. Quillen is designed to be a simple, command line tool that can be crontabbed and forgotten about. INSTALLATION: ------------- 1. Simply drop the quillen.jar file wherever you want and put your AWS acess key in a file quillen.config. Put your access key ID on the first line and your secret access key on the second line. 2. Create the quillen buckets (this will create two buckets on S3 for quillen) with: java -jar quillen.jar -command createBuckets USAGE: ------ Quillen puts your data into "vaults". You can backup files and directories to a vault, restore from a vault, and list files in a vault. To BACKUP a file foo.txt and put it in a vault bar: java -jar quillen.jar -command backup -vault bar -path foo.txt To RESTORE foo.txt from vault bar and save it to baz.txt: java -jar quillen.jar -command restore -vault bar -saveTo baz.txt To BACKUP a directory dir and put it in a vault bar: java -jar quillen.jar -command backup -vault bar -path dir To RESTORE a single file dir/foo.txt in vault bar and save it to dir2: java -jar quillen.jar -command restoreFile -vault bar -file dir/foo.txt -saveTo dir2 To LIST FILES in vault bar: java -jar quillen.jar -command listFiles -vault bar To DELETE vault bar: java -jar quillen.jar -command delete -vault bar To DELETE A SINGLE FILE foo.txt from vault bar: java -jar quillen.jar -command deleteFile -vault bar -file foo.txt To USE A DIFFERENT CONFIG FILE other than the default quillen.config: java -jar quillen.jar -command backup -vault bar -path foo.txt -config YOURCONFIGFILE UNINSTALLATION: --------------- To delete from S3 the two buckets quillen created in step 2 of INSTALLATION: java -jar quillen.jar -command -deleteBuckets This will completely remove any trace of quillen from your S3. QUESTIONS, COMMENTS, CONCERNS: ------------------------------ Please direct them to me at grourk+quillen@gmail.com. Thank you!