Re: [Apt-proxy-users] cache cleaning: deleting obsolete files

[Micha <ear...@gm...>]

Chris Halls <ha...@de...>:

| On Thursday 23 March 2006 03:55, Micha wrote:
| > | Your configuration as shown should work... but only if a-p has been
| > | running for a whole day. Try setting cleanup_freq to something small =
so
| > | the cleanup cycle is started quickly, e.g. 30m.
| >
| > Hey, wait ! Looks like i misinterpreted it.
| > I thought it would check the date...but it checks uptime ?
| > That would explain a lot. I shutdown the box every day.
| > I just thought the cleanup didn't work correctly, somehow.
|=20
| It's a bug. See this report:
|=20
| http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D343922

ok, let's have some closer look.


I restarted ap with the following config:

port =3D 9999
min_refresh_delay =3D 1h
complete_clientless_downloads =3D 1
debug =3D all:9
telnet_port =3D 2222
telnet_user =3D apt-proxy
telnet_password =3D apt-proxy
timeout =3D 30
cache_dir =3D /var/cache/apt-proxy
passive_ftp =3D on
cleanup_freq =3D 1m
max_age =3D 1m
max_versions =3D 1
dynamic_backends =3D off

ps aux (process list) shows,

<aptproxy  6538  0.0  2.5  13868  6432 ?        S    00:56   0:00 /usr/bin/=
python2.3 /usr/bin/twistd --pidfile=3D/var/run/apt-proxy//apt-proxy.pid --r=
undir=3D/var/run/apt-proxy/ --python=3D/usr/sbin/apt-proxy --logfile=3D/var=
/log/apt-proxy.log --no_save>

and the log says, first of all, for stopping:

2006/03/24 01:00 CET [-] Received SIGTERM, shutting down.
2006/03/24 01:00 CET [-] (Port 9999 Closed)
2006/03/24 01:00 CET [-] Stopping factory <apt_proxy.apt_proxy.Factory inst=
ance at 0xb74b2c4c>
2006/03/24 01:00 CET [-] Traceback (most recent call last):
	  File "/usr/lib/python2.3/site-packages/twisted/scripts/twistd.py", line =
197, in runApp
	    app.runReactorWithLogging(config, oldstdout, oldstderr)
	  File "/usr/lib/python2.3/site-packages/twisted/application/app.py", line=
 131, in runReactorWithLogging
	    reactor.run()
	  File "/usr/lib/python2.3/site-packages/twisted/internet/posixbase.py", l=
ine 218, in run
	    self.mainLoop()
	  File "/usr/lib/python2.3/site-packages/twisted/internet/posixbase.py", l=
ine 226, in mainLoop
	    self.runUntilCurrent()
	--- <exception caught here> ---
	  File "/usr/lib/python2.3/site-packages/twisted/internet/base.py", line 5=
41, in runUntilCurrent
	    call.func(*call.args, **call.kw)
	  File "/usr/lib/python2.3/site-packages/twisted/internet/tcp.py", line 81=
4, in connectionLost
	    self.factory.doStop()
	  File "/usr/lib/python2.3/site-packages/twisted/internet/protocol.py", li=
ne 64, in doStop
	    self.stopFactory()
	  File "/usr/share/python-support/apt-proxy/apt_proxy/apt_proxy.py", line =
1922, in stopFactory
	    packages.cleanup(self)
	  File "/usr/share/python-support/apt-proxy/apt_proxy/packages.py", line 2=
98, in cleanup
	    backend.get_packages_db().cleanup()
	exceptions.AttributeError: 'str' object has no attribute 'get_packages_db'
=09
2006/03/24 01:00 CET [-] Received SIGTERM, shutting down.
2006/03/24 01:00 CET [-] Main loop terminated.
2006/03/24 01:00 CET [-] Server Shut Down.


and for the restart (shortened):

2006/03/24 01:02 CET [-] twistd 2.2.0 (/usr/bin/python2.3 2.3.5) starting up
2006/03/24 01:02 CET [-] reactor class: twisted.internet.selectreactor.Sele=
ctReactor
2006/03/24 01:02 CET [-] Loading /usr/sbin/apt-proxy...
2006/03/24 01:02 CET [-] /usr/lib/python2.3/site-packages/twisted/manhole/t=
elnet.py:8: exceptions.DeprecationWarning: As of Twisted 2.1, twisted.proto=
cols.telnet is deprecated.  See twisted.conch.telnet for the current, suppo=
rted API.
2006/03/24 01:02 CET [-] WARNING: apt-proxy has not been tested under this =
version of twisted (2.2.0).
2006/03/24 01:02 CET [-] WARNING: although it should work without problem.
2006/03/24 01:02 CET [-] [config] config value min_refresh_delay=3D3600
2006/03/24 01:02 CET [-] [config] config value complete_clientless_download=
s=3DTrue
2006/03/24 01:02 CET [-] [config] config value telnet_port=3D2222
2006/03/24 01:02 CET [-] [config] config value telnet_user=3Dapt-proxy
2006/03/24 01:02 CET [-] [config] config value cleanup_freq=3D60
2006/03/24 01:02 CET [-] [config] config value max_versions=3D1
2006/03/24 01:02 CET [-] [config] config value max_age=3D60
2006/03/24 01:02 CET [-] [config] config value passive_ftp=3DTrue
2006/03/24 01:02 CET [-] [config] config value dynamic_backends=3DFalse

-> [backends stuff] ->

2006/03/24 01:02 CET [-] Loaded.
2006/03/24 01:02 CET [-] apt_proxy.apt_proxy.Factory starting on 9999
2006/03/24 01:02 CET [-] Starting factory <apt_proxy.apt_proxy.Factory inst=
ance at 0xb77e840c>
2006/03/24 01:02 CET [-] [debug] Created new BackendServer: ftp://ftp.infor=
matik.hu-berlin.de/pub/Java/Linux/debian
2006/03/24 01:02 CET [-] [debug] Created new BackendServer: ftp://ftp.tux.o=
rg/java/debian
2006/03/24 01:02 CET [-] [debug] Created new BackendServer: http://www.line=
x.org/sources/linex/debian
2006/03/24 01:02 CET [-] [debug] Created new BackendServer: http://download=
.videolan.org/pub/videolan/debian
2006/03/24 01:02 CET [-] [debug] Created new BackendServer: ftp://ftp.nerim=
.net/debian-marillat
2006/03/24 01:02 CET [-] [debug] Created new BackendServer: http://kibi.dyn=
dns.org/packages
2006/03/24 01:02 CET [-] [debug] Created new BackendServer: http://secure-t=
esting.debian.net/debian-secure-unstable
2006/03/24 01:02 CET [-] [debug] Created new BackendServer: http://secure-t=
esting.debian.net/debian-secure-testing
2006/03/24 01:02 CET [-] [debug] Created new BackendServer: http://ftp2.de.=
debian.org/debian
2006/03/24 01:02 CET [-] set uid/gid 107/65534

and a little later,

2006/03/24 01:03 CET [-] [debug] Doing periodic cleaning up
2006/03/24 01:03 CET [-] [debug] Verifying database: /var/cache/apt-proxy/.=
apt-proxy/db/access.db
2006/03/24 01:03 CET [-] [debug] Opening database /var/cache/apt-proxy/.apt=
-proxy/db/access.db
2006/03/24 01:03 CET [-] [debug] Verifying database: /var/cache/apt-proxy/.=
apt-proxy/db/update.db
2006/03/24 01:03 CET [-] [debug] Opening database /var/cache/apt-proxy/.apt=
-proxy/db/update.db
2006/03/24 01:03 CET [-] [debug] Periodic cleaning done
2006/03/24 01:04 CET [-] [debug] Doing periodic cleaning up
2006/03/24 01:04 CET [-] [debug] Periodic cleaning done
2006/03/24 01:05 CET [-] [debug] Doing periodic cleaning up
2006/03/24 01:05 CET [-] [debug] Periodic cleaning done
2006/03/24 01:06 CET [-] [debug] Doing periodic cleaning up
2006/03/24 01:06 CET [-] [debug] Periodic cleaning done

oops, that was fast...i missed to lookup the cache volume, before.
It should have been something about 200M,  though.
That's what i donwloaded the last days.
However, now i have
/dev/hda13            3.7G  328K  3.7G   1% /var/cache/apt-proxy

find . -name *.deb reveals: not a single file left.
So far for fiddling with testbeds...but it's ok, for now.
I should learn how to do it right, before piling up 4 GB again.

Q: What is preceeding, max_age or max_versions ?
I was thinking max_versions=3D1 means at least one version will be
kept anyway. Do i have to set max_age=3Doff to make it work this=20
way ?

-- I set it to off now :)


| The cleanup does a lot of sleeping in between each disk access=20

ok. Maybe that's why i couldn't see it, i only used top.


| > There's no commandline to trigger cleanup, right?
|=20
| No, but it sounds like it might be a good idea to add this.

Indeed. Always fiddling with the config isn't that cool...
It would also be handy to have a 'simulate' option (show what would be done)
which could save some people's cache (see above) ;o)

| That is the HTTP socket. You can't do telnet commands to this port, only=
=20
| requests for files.

Yes, but i wondered why then no error mesg from any side...looks
like i have to learn something about tje telnet protocol yet.
Anyway, i mainly wanted to test if i can call on the ports at all.
I wonder now if that port 9998 just wasn't available from the AP side?
I never tried a dummy port...! Indeed:

/var r: telnet localhost 1234567
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

/var r: telnet localhost 2222
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

| Have a look at the debug log. It should say if it is listening. My versio=
n=20
| says:
|=20
| 2006/03/01 10:31 GMT [-] twisted.manhole.telnet.ShellFactory starting on =
2222
| 2006/03/01 10:31 GMT [-] Starting factory <twisted.manhole.telnet.ShellFa=
ctory=20
| instance at 0xb77611ec>

I see no related AP log message at all, nor any error log in the syslogs
- only the connect attempt. And i _can_ do telnet, local and from other mac=
hines.
It's just not working.=20
=20
| > At this occasion i noticed that AP keeps a direct stream even if it can=
't
| > use the cache anymore. Well done.
|=20
| Erm, that is just a lucky accident :)

 :) Be the force with us.


ps. I'm free to do some tuning now...so i just I recreated the partition=20
with dir_index, let's see what it does...
 mke2fs /dev/hda13 -c -T largefile -m 0 -O dir_index -O sparse_user

-T largefile (1 inode per 1MB) created:

Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         128
Inode blocks per group:   4


-T news (1 inode / 4k block) would do:

Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         32576
Inode blocks per group:   1018

I can get a list of the installed packages' sizes with
aptitude search ~i -F "%D %p"
and it appears there are very few packages with MB sizes (>=3D10000kB),=20
like some kernel sources, and most notably the 144 MB openoffice=20
clipart .deb file. This leaves1650 packages with plain average 540kB.
(However, there are 250 additional virtual packges with <N/A> size)
Looks like largefile is ok, right ?

   =B0
 /\/