ubumirror Files

UBUMIRROR - an 'intelligent' APT repository mirror
---------


Written by Jeff MacLoue <jeff@macloue.com>, released under terms of GPL.

==============
0. Quick Start
==============

!!! Please don't run this script as root, you may damage your system, your
    files or your karma if you do.

To get a mirror of 32-bit Ubuntu Lucid Lynx which can be used for installation
and regular updates, put something like that in your cron (note the shell
expansion):

/usr/local/bin/ubumirror.pl -C lucid{,-updates,-security}/{main,restricted}

This will make a partial mirror of http://archive.ubuntu.com/ubuntu/ under
/var/www/htdocs/ubuntu/ so http://your-server/ubuntu/ can serve everything to
the network installer. /var/spool/ubumirror/ directory will be used for
temporary files in the process.

More advanced example is:

ubumirror.pl -u http://ua.archive.ubuntu.com/ubuntu/ -C {lucid,natty}{,-updates,-security}/{main,restricted}

Note the trailing slash in mirror URL, it is mandatory.

This will mirror both Lucid Lynx and Natty Narwhal from Ukraine's local Ubuntu
Archive.


============
1. The Task
============

The good thing about APT repository is a package pool. A single copy of a
package file is stored at every time - this reduces the distribution site size
and makes "continuous release" process easier.

The bad thing about APT repository is, well, a package pool. Mirroring an
archive with several Ubuntu versions and all the universe/multiverse stuff can
be a real pain if you are low (or, okay, greedy) on disk space or slow
Internet connection.

What's the point of doing an rsync of the entire archive if you have only
Lucid Lynx in your office? What's the point to mirror all the multiverse swamp
if you only need one or two packages there? So the task is rather simple: get
only files you absolutely need to have.

This sounds like an easy task - retrieve package index files for your
distribution and repositories required, parse them for references to package
pool, retrieve the packages. That's roughly all the ubumirror.pl script does,
plus removing the package files not referenced in the indices.

Of course there is little to no point to do this for one or two computers, it
is perfectly possible to live a good life without local repository mirror at
all. But, well, in a remote office with cheap ADSL and ten or fifteen Ubuntu
workstations distribution updates and new workstation installations are
complicated. A laptop with partial archive mirror is a real life-saver there.


===================
1a. Why ubumirror?
===================

There is a ready-made (and maybe even official) solution, apt-mirror
(http://apt-mirror.sourceforge.net). I tried it but it's not what I need - it
tends to invoke multiple wget instances by default (which, er, is rather nice
but I don't like it) and doesn't mirror .udeb files used for system
installation. So ubumirror.pl tries to be simplier and smarter at the same
time.

To sum up, ubumirror is simple and targeted specifically at Ubuntu while
apt-mirror is more advanced solution targeted at Debian.


=============
2. The Means
=============

The script uses wget for all the network operations. There is little use for
rsync as you need individual files and not the complete directories, and there
is no point in doing massive network operations in Perl.

Getopt::Std, File::Find, URI and IO::Zlib modules are used in operation. They
may or may not come with your distribution (they do with the Slackware 13.37 I
use, they are available in CentOS 5 as separate packages as well). 

cp from GNU coreutils is used for recursive file copy.

It is in theory possible to port all this to a non-UNIX platform, patches are
welcome.


========================
3. Command-Line Options
========================

ubumirror.pl [-OPTIONS [-MORE_OPTIONS]] [--] REPOSITORY ...

--help and --version standard options are recognized.

The following single-character options are accepted:

  -a <arch>  Architecture (default i386)
  -u <url>   Base repository URL (default http://archive.ubuntu.com/ubuntu/)
  -s <path>  Spool directory to store work files (default /var/spool/ubumirror)
  -d <path>  Directory to store repository mirror (default /var/www/htdocs/ubuntu)
  -w <cmd>   How to invoke WGET (default /usr/bin/wget)
  -C         Don't do pool cleanup
  -v         Verbose output
  -D         Debug output
  --         Stop processing for options

You need to specify at least one repository to proceed.

The repositories are specified as <distribution>/<repo>, e.g., lucid/main. As
mentioned above, it's best to use shell expansion capabilities to keep the
command line shorter.


=================
4. The Operation
=================

ubumirror starts with changing to the spool directory. Then it invokes wget to
retrieve package index and Release/Release.gpg files which APT uses.

For package indexes, Packages.gz is used as it is considerably smaller than
uncompressed Packages and still readable with standard IO::Zlib Perl module.

Then the retrieved Packages.gz are parsed to get all the Filename: and Size:
line pairs. A hash %Pool is built from this information.

Then the /var/www/htdocs/ubuntu/pool/ directory contents is compared with
%Pool - if the package isn't there or has different size it is marked for
retrieval.

Then wget is invoked again to retrieve them.

And finally after retrieving all the files the indices and other files from
spool directory are copied over /var/www/htdocs/ubuntu/dists/ - to get a
consistent mirror. Unlike apt-mirror, ubumirror is not very concerned with
keeping the mirror consistent at every time for sake of simplicity.

Optionally, if -C is specified on the command line, the /pool/ directory is
scanned again and everything not referenced in the package files is deleted to
save space. Please use with caution.


============================
5. BUGS, TODOs, Suggestions
============================

Probably over 9000. This is an initial public release, quality should be
considered alpha.

Bug reports and patches can be submitted at SourceForge or sent to me at
jeff@macloue.com.