Read Me

AutoDeploy

A framework in support of Visible Ops step #3, to assist system administrators with creating automated, repeatable deployments of productions systems.

OVERVIEW

In support of the ITPI in general (http://www.itpi.org/home/aboutus.php) and Visible Ops step #3 in specific (http://www.itpi.org/home/visibleops.php), I have put together this framework.

In Visible Ops, step #3 is defined as:

3. Establish Repeatable Build Library
The highest return on investment is implementing effective release management processes. This step creates repeatable builds for the most critical assets and services, to make it “cheaper to rebuild than to repair.”

This is something I agreed highly with, and put together this project with these goals:

1) Provide a framework to deploy unix based systems in an automated fashion.
2) Framework should provide automation by calling out to locally created scripts of commands.
3) Deployment of applications managed through this framework should be a hands off process.
4) Local OS customizations should be able to be managed as well.

This framework has the following dependancies:

1) Must create scripts for installation of the various managed applications.
2) Need a system in place to deploy out OS templates (Jumpstart, Ghost, VMware Templates, etc).
3) Unix based system. This framework has not been build with Windows in mind at all.
4) A central repository accessible via http to store configs and software.
5) It requires the perl library XML::Simple. This can be installed by "perl -MCPAN -e 'install XML::Simple'".

Contents of the framework consist of:

1) deploy.pl - Wrapper script that loads all the configurations and runs the install scripts for the enabled applications one by one.
2) deployconfig.xml - Config file stored on the repository server, deploy.pl will grab this config. Location defined in localconfig.xml. Can be renamed to anything.
3) localconfig.xml - Config file to bootstrap deploy.pl, giving it the location of the archive server and central config. Defines what application to install. Can be renamed to anything, is the only command line argument to deploy.pl.
4) localstate.xml - State file generated by deploy.pl consisting of currently installed applications and versions.

That is it. I have tried to keep this framework simple and small as possible so that any and all work is spent on creating and maintaining a series of scripts to install all locally support applications. These install scripts work your build library. This deployment framework is a way to manage it.

CONCEPTS

When you deploy a new server, there should be no manual work done, no thinking required. You do all your thinking one time, up front, in the creation of the scripts. If this is done correctly, then deployment of a server becomes a function that any level of employee can handle, and you are guaranteed to always be at a known state upon deployment. Furthermore, deployment will always take a known amount of time.

This opens up new opportunities for many administrations. For instance, it is accepted that a large uptime means that a reboot is "scary" as there is a high possibility that something will go wrong. This leads to patches not being rolled out, and the systems slowly over time deviates away from a known state. But what if instead, you redeployed each server once a month? Or once each scheduled down time? If deployment is automated, this would allow you deploy out patches and return to a known state with minimal effort. Staying as close as possible to a known state will help with troubleshooting and reproducing problems when they occur. At this point reboots are never an issue, you now redeploy servers possibly as often as some administrators reboot.

The idea here is that we have moved far beyond where server should be hand crafted. Within Visual Ops they talk of making machines into fuses and not works of art. To stay with that metaphor, this framework is the assembly line for creating those fuses.

Think of this situation -

You have a VMware ESX infrastructure which makes it very easy to deploy new servers. You template out a base OS image. You create a series of installation scripts for each application that matters in your environment. And you create a local configuration script for each server function that you have.

When you want to deploy a new server, you deploy a server template with ESX. Log into the VM, and copy in deploy.pl and a local config file. You run deploy.pl which goes through and installs each needed application in turn, then finally runs a local server configuration script that does final prep for the server for its new role. Then reboot. Server is deployed.

Then you have the possibility of using this same framework to auto-deploy application upgrades. A new version of httpd is released, you tweak your original installation script for it to use the new sources, increment the version number, and update the deployconfig.xml configuration to reflect the new version. Then you have deploy.pl croned out to run every night at 2am, it sees the locally installed version is no longer the same as on the repository and updates the local installation. If you have a large number of servers using the same applications, you can have them all updated through this single action.

Beyond production systems, this framework also makes it easy to deploy out identical development systems as well. Just have a slightly different local setup script for the dev systems.

USAGE

The easiest way to understand the usage of this framework is to examine an example installation.

First off, you build your software repository. This consists of a http accessible server with your deployconfig.xml, local build scripts, and source tarballs that you need. For example you may have a server like so:

---
bash-3.00# ls -R htdocs
htdocs:
config  files

htdocs/config:
deployconfig.xml  httpd_v1.sh       openssl_v1.sh     wwwserverlocalsetup_v3.sh

htdocs/files:
httpd-2.2.3.tar.gz     openssl-0.9.8d.tar.gz
---

Within the deployconfig.xml you would define your server setup:

---
<serverconfig>
        <repbase>http://ip.add.re.ss/</repbase>
        <repfiles>files/</repfiles>
        <appconfigbase>config/</appconfigbase>
        <repconfig>deployconfig.xml</repconfig>
        <builddir>/builds/source</builddir>
</serverconfig>
---

<repfiles> and <appconfigbase> are appended to <repbase> to create the URL paths that the various files live under. <repconfig> defines the name of the config file this is set in, and <builddir> gives a path for the various builds to happen under for possible compilation purposes.

Further in the config, you would have a line for each application managed by the framework. For example:

---
<application>
        <name>httpd</name>
        <installlocation>/apps/httpd</installlocation>
        <version>2.26</version>
        <internalversion>1</internalversion>
        <archive>httpd-2.2.3.tar.gz</archive>
        <basedir>httpd-2.2.3</basedir>
        <installscript>httpd_v1.sh</installscript>
</application>

<application>
        <name>openssl</name>
        <installlocation>/apps/ssl</installlocation>
        <version>0.9.8d</version>
        <internalversion>1</internalversion>
        <archive>openssl-0.9.8d.tar.gz</archive>
        <basedir>openssl-0.9.8d</basedir>
        <installscript>openssl_v1.sh</installscript>
        <dep>httpd</dep>
</application>

<application>
	<name>wwwserverlocalsetup</name>
	<internalversion>3</internalversion>
	<installscript>wwwserverlocalsetup_v3.sh</installscript>
	<dep>httpd</dep>
</application>
---

Each item defined in a <application> block is handed off as a env var to the various scripts. You can create as many or as few definitions in each block as needed, with a few exceptions. <name> and <internalversion> must be set, as these are used to track what applications are installed through the localstate.xml file. <installscript> must be defined as well, otherwise it is not possible to have anything happen locally.

On the local server managed by this framework, you would have a localconfig.xml file, with various definitions, starting with a block to define the server settings.

---
<serverconfig>
        <repbase>http://ip.add.re.ss/</repbase>
        <repfiles>files/</repfiles>
        <appconfigbase>config/</appconfigbase>
        <repconfig>deployconfig.xml</repconfig>
        <builddir>/builds/source</builddir>
</serverconfig>
---

This should mirror the settings in the deployconfig.xml, as this simple bootstraps the framework. Beyond the bootstrapping information you have the settings that are unique to this server:

---
<localconfig>
        <installapps>
                < wwwserverlocalsetup>1</wwwserverlocalsetup>
                <openssl>1</openssl>
                <httpd>1</httpd>
                <otherapp>0</opterapp>
        </installapps>
        <localstate>/config/localstate.xml</localstate>
</localconfig>
---

This block tells deploy.pl what applications belong on this server.

With the infrastructure in place, deployment of the server then becomes as simple as:

---
~# ./deploy.pl localconfig.xml
---

Then the frame work takes care of the rest.

INSTALLSCRIPTS

Install scripts can be written in any language you want, or can even be executables. They are simple downloaded from the repository and executed.

The only requirement the install scripts have is, they must return a non-zero exit status if they fail. This is used by deploy.pl to report back success or failure after execution of the script. Every item defined within a <application> block will be passed to the script as an environment variable, and can be used in any manner by the install script.