Synopsis

Synopsis

The project is aimed at helping techies who use the versioning feature of Amazon's S3 service -- it does not deal with "normal" S3 buckets, as for those there are a variety of tools out there already (file browsers, sync tools etc.). However, in the case of versioning-enabled buckets, most (if not all) file browsers for S3 will only list the most recent version of the bucket, and do not provide any support for any previous versions. As such, imagine that you have 100 files that are updated daily in the bucket, after a month (~30 days), each file will have 30 versions -- and as such, the total number of objects stored in the S3 bucket will be 30 x 100 = 3,000 objects. Out of those, using any of the today's browser, you will only be able to see 30 objects (the latest versions). Even more, deleting those 30 latest versions will show in any S3 browser the bucket as empty, however, when you try to delete the bucket, Amazon will rightly report the bucket as non-empty and you will not be allowed to delete the bucket -- which still has 29 x 100 = 2,900 objects in it, sitting there, taking space and costing you money. (Please note that at the time I'm writing this, even Amazon's own web based S3 browser shows the bucket as empty in this case!)

Even more confusingly, let's say you have a file abc.txt which has 100 versions and you delete the latest one (what actually happens behind the scenes is S3 creates a new empty version which is called a "delete marker"). At this point, the file "disappears" from your S3 browser and you are convinced it's deleted. A few months later you create a new file called abc.txt again, totally different from the original one. At this point this new file inherits all the previous versions of the previous abc.txt file even though they are not related -- which can cause confusion!

This project is aimed at assisting with all these problems, and make it easier to purge a whole bucket, delete all the versions of a certain file and so on. The aim is to have this tool available as both a command-line as well as provide a GUI (Swing?) interface to it.

The project is Java-based, using maven as a build tool. I've used JDK 1.6 throughout but it might be the case that future versions will require migrating it to Java7, we'll see. It relies on Amazon's SDK for Java, and makes use of Apache Commons CLI, Apache Commons Lang and Apache Commons Collections. There are no databases involved, no Spring (at the moment) or any other frameworks. We use JUnit for unit tests.

In terms of IDE, I personally use Eclipse -- you will have to install the plugins for subversion and maven -- but since this is a maven-ized project, any other IDE should work, including simply compiling from the command line!