#78 Application and script abstract classes

[Anonymous]

Originally created by: jcflack

Application and script abstract classes

The goal of this work is to widen the population that can easily write reliable new Amanda applications and scripts to support specialized backup needs. The topmost section of the Amanda wiki Development Tasks list presents several useful applications that could be added to Amanda, and that list has been unchanged for seven years.

Arguably, the slow pace of new application and script development relates to the level at which the current Application API and Script API are defined. Those documents spell out the arguments and file descriptors passed to an application or script process, and the concrete syntax of messages to be exchanged.

Specification at that level has the advantage of being language-agnostic: an Amanda application or script could in theory be written in any language, as long as it consumes and produces the needed arguments and messages properly. Its disadvantage is a level of detail far removed from the practical problem a would-be application or script writer wants to solve. It tends to narrow the potential population of Amanda application or script authors: rather than including any Amanda user with a problem to solve and an idea how to do so, it realistically demands some expertise in IPC protocols and enough Amanda internals knowledge to fill in the actual behavior behind some of the syntax. (The many amanda-hackers messages exchanged during this work illustrate some of the traps for the unwary.)

While language-agnostic in theory, applications and scripts for Amanda are likely to be written in perl, given its heavy use in Amanda proper. Specifically, they are likely to be object-oriented perl code that extends one of the base classes Amanda::Application or Amanda::Script. They do inherit a small amount of common functionality from their base classes, but not nearly as much as they could. Inheriting classes are still on their own, for example, to produce and consume the API-defined IPC messages, ensuring valid syntax.

Less-thin abstract base classes could encapsulate much more of that common work, and present a traditional, object-oriented API where new application or script code may override just a few key methods and rely on default behavior wherever customization is not needed. That can make applications and scripts much faster to develop, and easier to review for correctness. At the same time, it simplifies future evolution of the IPC messages, something that would be increasingly impractical if a growing set of scripts and applications all contain duplicated code with those details baked in, but is simple with methods inherited from one central place.

This work, therefore, introduces the new abstract classes Amanda::Application::Abstract and Amanda::Script::Abstract. To stay compatible with existing applications and scripts, this work makes no changes to the existing classes Amanda::Application and Amanda::Script. Those classes are simply inherited by these new base classes, which are implemented in pure perl and provide a richer object-oriented API to applications and scripts that choose to inherit from them.

This layering also means that if it is ever desired to write an Amanda application or script in another scripting language than perl, these two pure-perl classes are essentially what could be translated to provide an "Amanda application/script API binding" for that language.

Included sample applications / scripts

Included here are four new applications to show the simplicity of development: amooraw (merely amraw redone in OO style), amgrowingfile (useful for a single large file known to only monotonically grow; can do incremental levels), amgrowingzip (like amgrowingfile but for a ZIP archive), and amopaquetree (backup of a directory/file tree where restoration of individual files won't be needed, but with fine-grained incremental backup down to only changed regions within files, rather than entire files that may contain small changes).

The amopaquetree application is also one that might itself be used as a base class for a specialized application applying the same opaque-tree, fine-grained-increments approach to a specific database management system, for example.

Four new scripts are also provided: am389bak for taking a consistent snapshot of a 389 Directory Server instance before estimate or backup, amsvnmakehotcopy to do the same for a Subversion repository, amlvmsnapshot to allow backing up from a snapshot of a filesystem using LVM, and amlibvirtfsfreeze to freeze and thaw filesystems in a libvirt-supported guest VM, so the host filesystem can be snapshotted for backup at a moment when the guest image files are consistent.

These four scripts, especially, should be considered experimental or demo quality at this stage. They do little error checking of external commands they execute (though they work fine when nothing goes wrong), and, to be useful in most real environments, they are likely to need short C-language setuid wrappers to be written for those few external commands, and such wrappers are not included in this commit. A configurable, secure, general-purpose permission-granting wrapper would greatly simplify development of scripts like these, but a design for that remains future work.

Future work

Elevated permissions

As mentioned above, a design for a general-purpose and securely configurable way to grant elevated permission to specific actions executed from selected applications or scripts (as opposed to having to write some C analog of runtar for each needed case) could be a challenging design problem, but one that would greatly simplify application/script development for real environments.

I/O involving child processes

The example applications and scripts presented here do less than they ought in the way of capturing standard and error output from child processes they execute, interpreting and responding appropriately, or passing modified messages upstream to Amanda. Fully robust implementations would include that, ideally without adding such complexity it obscures the outlines of the code.

Doing such work at the low level of, say, perl's open3 is too longwinded to be ideal. Experienced Amanda developers may prefer to work with the features of Amanda::MainLoop, already familiar from other parts of the guts of Amanda. For other potential application or script developers, who may not have deep Amanda hacking experience but will know perl, the familiarity and clear, intuitive syntax of perl's IPC::Run might be more appealing.

IPC::Run is a CPAN module that need not be present on every system with perl, and might not be a suitable dependency for Amanda core. Because of its functionality and appealing syntax, though, it might be something that specialized applications or scripts might rely on if the author prefers. Such applications or scripts would need to avoid breaking make check, and return a suitable error to amcheck, on systems where the module is not present. It would be simple to provide some stub support in Amanda::Application::Abstract and Amanda::Script::Abstract to simplify that.

[Anonymous]

Originally posted by: jcflack

AppScriptWithAbstractClasses.pdf
Have just pushed one more new app, amzfs-holdsend, giving a third choice (in addition to amzfs-snapshot and amzfs-sendrecv) for how to approach ZFS backups.

Also attached here is a PDF file of the new generated docs for ease of review.

[Anonymous]

Originally posted by: chassell

I'll need to catch up and read more of this but let me say it merges beautifully and seems to disrupt nothing at all except to add more files / functions and a little bit of documentation changes.

I'm not sure it will be welcomed as a direct benefit but it can be a very clear "addition" of commands.