rtklib-bouncer_r — Bouncer_r - Mirror Management for Reposotories

[david <dfa...@sb...>]

Pedro Fernandes Macedo wrote:

> david wrote:
>
>> The final piece of the puzzle will be moving bouncer_r from a data 
>> pull model to a data push model. I was browsing through various 
>> bit_torrent clients for hints or a messaging passing class that can 
>> be pulled out and used by bouncer.
>
>
>
> Have you looked at the way that Nagios does its checking for free disk 
> space/free mem/cpu usage/etc on the hosts it monitors? That's a bit 
> like what I was thinking on doing... To make possible to the bouncer 
> server to ask the mirrors for a checkup , it could be possible to make 
> a little daemon listen on a port , waiting from a request from 
> bouncer_server. Then it would run the following check (this check 
> could also be set to run just after the rsync run, if the mirror admin 
> wants it like that)
>
> The local bouncer app would download a file from the bouncer_server 
> with all the data necessary to check the completeness . Then the 
> mirror checks its completeness using this file and sends an answer to 
> the bouncer server (maybe by doing a request to 
> http://bouncer/somepage.php?lots_of_flags&complete=1).
>
> In the case of a remote request , the reply could be sent back using 
> the same connection that was initially used to request the check, but 
> I preffer doing the request to the url always.
>
> And this would change the need to download a few GB of data to just a 
> few MB (but  then , we would have to convince the mirror admins to run 
> this daemon)
>
I hacked together a a quick mirror completeness flag at 
http://rtklib.org/bouncer_r/python/metadata/.  It is basically a very 
small timestamp.xml that is cheap to send back and forth and a larger 
filelist.xml to grab in case the timestamp indicates a change.  But I 
would rather use full blown repomd as found at 
http://linux.duke.edu/projects/metadata/ so we can use repoview at 
http://linux.duke.edu/projects/mini/repoview/ to create the .html pages 
on bouncer that do the redirection. 

> -- 
> Pedro Macedo
>
I just got back from the store with a two liter bottle of diet coke in 
perpetration of a look through the nagios code;

I am writing bounce_r with a data pull fall back so if a mirror doesn't 
have to buy into running a daemon if they don't want to.  I think a 
listener daemon is the best answer long term!

A basic transaction would consist of something like this.  daemon 
messages identified with (d) http/ftp with (h) and rsync (r)

repomaster->bouncer (d)
   I'm fedora core,version one, arch i386
   I've just updated
bouncer->repomaster (h)
   check status of repomd.xml for repo(core,1,i386)
   if repomd changed
        run bouncer.update.repo(core,1,i386)
         mark all mirrors of (core,1,i386) out of date
         for mirror in mirrors(core,1,i386)
             bouncer->mirror(d)
                  repo(core,1,i386) has changed
                  mirror->repomaster(r)
                     update repo(core,1,i386)
               mirror->bouncer (d)
                     I've completed update of repo(core,1,i386)
               bouncer->mirror (h)
                   check status of repomd.xml
               bouncer changes mirror status to up to date

The loop through the mirror list should be multi threaded, maybe with 
some sort of ranking and tiering  system.

I was thinking the daemon should be easily called from the command line 
via a bash script.  No need for it to interact directly with the 
database.  On the bouncer_r server the listener could fire off something 
like
    sentry -command update --product core --version 1 --arch i386
and
    sentry -command updated --mirror kernel.org --product core 
--version 1 --arch i386

Buyin from the mirrors is optional because we can fall back to 
polling.   But if can get the system working.  I think users will demand 
it.  On the other hand if we keep track of how may bytes we grab from 
different mirrors vs their available bandwidth(why I would eventfully 
incorporating a dns server), the up2date mirrors can serve the new stuff 
while the out of date mirrors can server stuff like .ISO of other repos 
that have not changed.

As a proof of concept we can use trigger_daemon.php+flags as the trigger 
which could pass the exact same flags as the daemon would.  Hmm can php 
call a bash script?

I've created a list called bou...@li... at 
http://sourceforge.net/mail/?group_id=140672 so we can keep archive of 
our conversation and allow other to follow our thoughts and add 
suggestions without being cc in email.

thanks
-dtf