Menu
▾
▴
apt-proxy-users
|
From: Micha <ear...@gm...> - 2006-03-19 02:37:09
|
Hello, i'm no more on the list (too less problems with AP :) -> please 'reply to all'. The cache here is about 3.6 GB, and unfortunately my i can't give the cache more than 4GB total (seperate partition) without rebuilding the whole partition table... I switched from debian testing to unstable, yesterday, and there's still stable + testing stuff in the cache. Plus repositores for obsolete backends that i don't use anymore. However, i can find files from 2005 in the cache, and more than 1 version anyway. For example, openoffice.org_2.0.0-5_i386.deb openoffice.org_2.0.2-1_i386.deb mozilla-browser_1.7.12-1.1_i386.deb mozilla-browser_1.7.12-1_i386.deb mozilla-browser_1.7.8-1_i386.deb mozilla-firefox_1.0.4-2_i386.deb mozilla-firefox_1.0.6-5_i386.deb mozilla-firefox_1.0.7-1_i386.deb The latter, found in debian/pool/main/m/mozilla-firefox, is a good example for a related issue, that is, changing package names and/or classification. Debian unstable has firefox version 1.5, which i can find now in debian/pool/main/f/firefox: firefox_1.5.dfsg+1.5.0.1-4_i386.deb mozilla-firefox_1.5.dfsg+1.5.0.1-4_all.deb The question is, can i have AP 'clean up' all files that are older than a specified time regardless of the number of versions ? Can i modify the config temporarily to delete anything older than (today) 1 day, and that would leave me with 'sid' only ? These are the related config settings: min_refresh_delay = 1h cleanup_freq = 1d max_age = 1d max_versions = 1 dynamic_backends = off So, how can i have AP clear the old files ? |
|
From: Chris H. <ha...@de...> - 2006-03-22 23:47:46
|
On Sunday 19 March 2006 02:36, Micha wrote:
> The latter, found in debian/pool/main/m/mozilla-firefox,
> is a good example for a related issue, that is, changing
> package names and/or classification. Debian unstable has firefox
> version 1.5, which i can find now in debian/pool/main/f/firefox:
> firefox_1.5.dfsg+1.5.0.1-4_i386.deb
> mozilla-firefox_1.5.dfsg+1.5.0.1-4_all.deb
This particular problem is very hard to solve, and I don't think it is worth
spending time trying to do this.
> The question is, can i have AP 'clean up' all files that are older
> than a specified time regardless of the number of versions ?
> Can i modify the config temporarily to delete anything older than (today) 1
> day, and that would leave me with 'sid' only ?
>
> These are the related config settings:
>
> min_refresh_delay = 1h
> cleanup_freq = 1d
> max_age = 1d
> max_versions = 1
> dynamic_backends = off
>
> So, how can i have AP clear the old files ?
Your configuration as shown should work... but only if a-p has been running
for a whole day. Try setting cleanup_freq to something small so the cleanup
cycle is started quickly, e.g. 30m.
Alternatively, use a find command to remove the files from the cache:
find /var/cache/apt-proxy/* -amin +1440 -exec rm {} \;
(1440 is 60*24 minutes = 1 day)
Chris
|
|
From: Micha <ear...@gm...> - 2006-03-23 03:55:24
|
Chris Halls <ha...@de...>:
| This particular problem is very hard to solve
=20
| Your configuration as shown should work... but only if a-p has been runni=
ng=20
| for a whole day. Try setting cleanup_freq to something small so the clean=
up=20
| cycle is started quickly, e.g. 30m.
Hey, wait ! Looks like i misinterpreted it.
I thought it would check the date...but it checks uptime ?
That would explain a lot. I shutdown the box every day.
I just thought the cleanup didn't work correctly, somehow.
Hmmm...i still would prefer to run the cleanup only on demand,
i do compiling/installing more often, and the ap background process could=20
slow down things when i like to have 100% cpu myself...
There's no commandline to trigger cleanup, right?
Perhaps simply restart then with cleanup_freq =3D 1m...However:
Tried it right now, but i can't see any twisted process activity,=20
nor anything in the ap log ?=20
How about telnet, connecting to the running server ?=20
Can i trigger something that way ?
Oh, I just see,=20
"Warning: As of Twisted 2.1, twisted.protocols.telnet is deprecated. =20
See twisted.conch.telnet for the current, supported API."
(Plus general warning for twisted 2.2.0 but normal operation are ok, so...)
I never used the python thing, and have no clue what i could do with it=20
anyway, is there any information available ?
I just enabled the config options for port 998 and restarted ap and tried a=
=20
simple telnet connect but i can't get any connection there, it seems to work
however on the normal http server port 9999. (without password request...?)
/ r: telnet localhost 9998
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
/etc r: telnet localhost 9999
Trying 127.0.0.1...
Connected to woody.
Escape character is '^]'.
I also dug holes into xinetd / libwrap / ident so they should be ok with=20
telnet queries. libwrap (hosts.allow) is open for "apt-proxy:" and there's=
=20
anyway no port assigned enywhere (in /etc/services) ....=20
so why not 9998 ? Is there a mistake ?
I just installed identtestd, that works ok too.
=20
| Alternatively, use a find command to remove the files from the cache:
|=20
| find /var/cache/apt-proxy/* -amin +1440 -exec rm {} \;
ok that's what i wanted to hear, anyway :)
Though it's coming too late. Cache ran out of space, and i decided
to start from scratch, this time, since i just yesterday moved all machines
to unstable, (many laptops you know) and there's much turnover, and i=20
think most packages will be cached again very soon.
Pity, but not critical (by now).
At this occasion i noticed that AP keeps a direct stream even if it can't=20
use the cache anymore. Well done.
=B0
/\/
|
|
From: Chris H. <ha...@de...> - 2006-03-23 14:28:45
|
On Thursday 23 March 2006 03:55, Micha wrote: > | Your configuration as shown should work... but only if a-p has been > | running for a whole day. Try setting cleanup_freq to something small so > | the cleanup cycle is started quickly, e.g. 30m. > > Hey, wait ! Looks like i misinterpreted it. > I thought it would check the date...but it checks uptime ? > That would explain a lot. I shutdown the box every day. > I just thought the cleanup didn't work correctly, somehow. It's a bug. See this report: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=343922 > Hmmm...i still would prefer to run the cleanup only on demand, > i do compiling/installing more often, and the ap background process could > slow down things when i like to have 100% cpu myself... The cleanup does a lot of sleeping in between each disk access so that it does not disturb any busy processes. It should use less than 0.5% CPU, depending on your processor speed. > There's no commandline to trigger cleanup, right? No, but it sounds like it might be a good idea to add this. > Perhaps simply restart then with cleanup_freq = 1m...However: > Tried it right now, but i can't see any twisted process activity, > nor anything in the ap log ? That's strange. I see activity on my version, although I am working on a different version of the code. > How about telnet, connecting to the running server ? It should be possible, but I can't get this working properly either - maybe it is because of the deprecated error that you saw. It is probably best for you to use find to remove the files for now, until bug #343922 is fixed. > I just enabled the config options for port 998 and restarted ap and tried a > simple telnet connect but i can't get any connection there, It will only enable the telnet server if you specify a port, username and password. Maybe something was not spelled correctly. > it seems to > work however on the normal http server port 9999. (without password > request...?) That is the HTTP socket. You can't do telnet commands to this port, only requests for files. > I also dug holes into xinetd / libwrap / ident so they should be ok with > telnet queries. libwrap (hosts.allow) is open for "apt-proxy:" and there's > anyway no port assigned enywhere (in /etc/services) .... > so why not 9998 ? Is there a mistake ? Have a look at the debug log. It should say if it is listening. My version says: 2006/03/01 10:31 GMT [-] twisted.manhole.telnet.ShellFactory starting on 2222 2006/03/01 10:31 GMT [-] Starting factory <twisted.manhole.telnet.ShellFactory instance at 0xb77611ec> (my telnet socket is port 2222) > At this occasion i noticed that AP keeps a direct stream even if it can't > use the cache anymore. Well done. Erm, that is just a lucky accident :) Chris |
|
From: Micha <ear...@gm...> - 2006-03-24 03:38:11
|
Chris Halls <ha...@de...>: | On Thursday 23 March 2006 03:55, Micha wrote: | > | Your configuration as shown should work... but only if a-p has been | > | running for a whole day. Try setting cleanup_freq to something small = so | > | the cleanup cycle is started quickly, e.g. 30m. | > | > Hey, wait ! Looks like i misinterpreted it. | > I thought it would check the date...but it checks uptime ? | > That would explain a lot. I shutdown the box every day. | > I just thought the cleanup didn't work correctly, somehow. |=20 | It's a bug. See this report: |=20 | http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D343922 ok, let's have some closer look. I restarted ap with the following config: port =3D 9999 min_refresh_delay =3D 1h complete_clientless_downloads =3D 1 debug =3D all:9 telnet_port =3D 2222 telnet_user =3D apt-proxy telnet_password =3D apt-proxy timeout =3D 30 cache_dir =3D /var/cache/apt-proxy passive_ftp =3D on cleanup_freq =3D 1m max_age =3D 1m max_versions =3D 1 dynamic_backends =3D off ps aux (process list) shows, <aptproxy 6538 0.0 2.5 13868 6432 ? S 00:56 0:00 /usr/bin/= python2.3 /usr/bin/twistd --pidfile=3D/var/run/apt-proxy//apt-proxy.pid --r= undir=3D/var/run/apt-proxy/ --python=3D/usr/sbin/apt-proxy --logfile=3D/var= /log/apt-proxy.log --no_save> and the log says, first of all, for stopping: 2006/03/24 01:00 CET [-] Received SIGTERM, shutting down. 2006/03/24 01:00 CET [-] (Port 9999 Closed) 2006/03/24 01:00 CET [-] Stopping factory <apt_proxy.apt_proxy.Factory inst= ance at 0xb74b2c4c> 2006/03/24 01:00 CET [-] Traceback (most recent call last): File "/usr/lib/python2.3/site-packages/twisted/scripts/twistd.py", line = 197, in runApp app.runReactorWithLogging(config, oldstdout, oldstderr) File "/usr/lib/python2.3/site-packages/twisted/application/app.py", line= 131, in runReactorWithLogging reactor.run() File "/usr/lib/python2.3/site-packages/twisted/internet/posixbase.py", l= ine 218, in run self.mainLoop() File "/usr/lib/python2.3/site-packages/twisted/internet/posixbase.py", l= ine 226, in mainLoop self.runUntilCurrent() --- <exception caught here> --- File "/usr/lib/python2.3/site-packages/twisted/internet/base.py", line 5= 41, in runUntilCurrent call.func(*call.args, **call.kw) File "/usr/lib/python2.3/site-packages/twisted/internet/tcp.py", line 81= 4, in connectionLost self.factory.doStop() File "/usr/lib/python2.3/site-packages/twisted/internet/protocol.py", li= ne 64, in doStop self.stopFactory() File "/usr/share/python-support/apt-proxy/apt_proxy/apt_proxy.py", line = 1922, in stopFactory packages.cleanup(self) File "/usr/share/python-support/apt-proxy/apt_proxy/packages.py", line 2= 98, in cleanup backend.get_packages_db().cleanup() exceptions.AttributeError: 'str' object has no attribute 'get_packages_db' =09 2006/03/24 01:00 CET [-] Received SIGTERM, shutting down. 2006/03/24 01:00 CET [-] Main loop terminated. 2006/03/24 01:00 CET [-] Server Shut Down. and for the restart (shortened): 2006/03/24 01:02 CET [-] twistd 2.2.0 (/usr/bin/python2.3 2.3.5) starting up 2006/03/24 01:02 CET [-] reactor class: twisted.internet.selectreactor.Sele= ctReactor 2006/03/24 01:02 CET [-] Loading /usr/sbin/apt-proxy... 2006/03/24 01:02 CET [-] /usr/lib/python2.3/site-packages/twisted/manhole/t= elnet.py:8: exceptions.DeprecationWarning: As of Twisted 2.1, twisted.proto= cols.telnet is deprecated. See twisted.conch.telnet for the current, suppo= rted API. 2006/03/24 01:02 CET [-] WARNING: apt-proxy has not been tested under this = version of twisted (2.2.0). 2006/03/24 01:02 CET [-] WARNING: although it should work without problem. 2006/03/24 01:02 CET [-] [config] config value min_refresh_delay=3D3600 2006/03/24 01:02 CET [-] [config] config value complete_clientless_download= s=3DTrue 2006/03/24 01:02 CET [-] [config] config value telnet_port=3D2222 2006/03/24 01:02 CET [-] [config] config value telnet_user=3Dapt-proxy 2006/03/24 01:02 CET [-] [config] config value cleanup_freq=3D60 2006/03/24 01:02 CET [-] [config] config value max_versions=3D1 2006/03/24 01:02 CET [-] [config] config value max_age=3D60 2006/03/24 01:02 CET [-] [config] config value passive_ftp=3DTrue 2006/03/24 01:02 CET [-] [config] config value dynamic_backends=3DFalse -> [backends stuff] -> 2006/03/24 01:02 CET [-] Loaded. 2006/03/24 01:02 CET [-] apt_proxy.apt_proxy.Factory starting on 9999 2006/03/24 01:02 CET [-] Starting factory <apt_proxy.apt_proxy.Factory inst= ance at 0xb77e840c> 2006/03/24 01:02 CET [-] [debug] Created new BackendServer: ftp://ftp.infor= matik.hu-berlin.de/pub/Java/Linux/debian 2006/03/24 01:02 CET [-] [debug] Created new BackendServer: ftp://ftp.tux.o= rg/java/debian 2006/03/24 01:02 CET [-] [debug] Created new BackendServer: http://www.line= x.org/sources/linex/debian 2006/03/24 01:02 CET [-] [debug] Created new BackendServer: http://download= .videolan.org/pub/videolan/debian 2006/03/24 01:02 CET [-] [debug] Created new BackendServer: ftp://ftp.nerim= .net/debian-marillat 2006/03/24 01:02 CET [-] [debug] Created new BackendServer: http://kibi.dyn= dns.org/packages 2006/03/24 01:02 CET [-] [debug] Created new BackendServer: http://secure-t= esting.debian.net/debian-secure-unstable 2006/03/24 01:02 CET [-] [debug] Created new BackendServer: http://secure-t= esting.debian.net/debian-secure-testing 2006/03/24 01:02 CET [-] [debug] Created new BackendServer: http://ftp2.de.= debian.org/debian 2006/03/24 01:02 CET [-] set uid/gid 107/65534 and a little later, 2006/03/24 01:03 CET [-] [debug] Doing periodic cleaning up 2006/03/24 01:03 CET [-] [debug] Verifying database: /var/cache/apt-proxy/.= apt-proxy/db/access.db 2006/03/24 01:03 CET [-] [debug] Opening database /var/cache/apt-proxy/.apt= -proxy/db/access.db 2006/03/24 01:03 CET [-] [debug] Verifying database: /var/cache/apt-proxy/.= apt-proxy/db/update.db 2006/03/24 01:03 CET [-] [debug] Opening database /var/cache/apt-proxy/.apt= -proxy/db/update.db 2006/03/24 01:03 CET [-] [debug] Periodic cleaning done 2006/03/24 01:04 CET [-] [debug] Doing periodic cleaning up 2006/03/24 01:04 CET [-] [debug] Periodic cleaning done 2006/03/24 01:05 CET [-] [debug] Doing periodic cleaning up 2006/03/24 01:05 CET [-] [debug] Periodic cleaning done 2006/03/24 01:06 CET [-] [debug] Doing periodic cleaning up 2006/03/24 01:06 CET [-] [debug] Periodic cleaning done oops, that was fast...i missed to lookup the cache volume, before. It should have been something about 200M, though. That's what i donwloaded the last days. However, now i have /dev/hda13 3.7G 328K 3.7G 1% /var/cache/apt-proxy find . -name *.deb reveals: not a single file left. So far for fiddling with testbeds...but it's ok, for now. I should learn how to do it right, before piling up 4 GB again. Q: What is preceeding, max_age or max_versions ? I was thinking max_versions=3D1 means at least one version will be kept anyway. Do i have to set max_age=3Doff to make it work this=20 way ? -- I set it to off now :) | The cleanup does a lot of sleeping in between each disk access=20 ok. Maybe that's why i couldn't see it, i only used top. | > There's no commandline to trigger cleanup, right? |=20 | No, but it sounds like it might be a good idea to add this. Indeed. Always fiddling with the config isn't that cool... It would also be handy to have a 'simulate' option (show what would be done) which could save some people's cache (see above) ;o) | That is the HTTP socket. You can't do telnet commands to this port, only= =20 | requests for files. Yes, but i wondered why then no error mesg from any side...looks like i have to learn something about tje telnet protocol yet. Anyway, i mainly wanted to test if i can call on the ports at all. I wonder now if that port 9998 just wasn't available from the AP side? I never tried a dummy port...! Indeed: /var r: telnet localhost 1234567 Trying 127.0.0.1... telnet: Unable to connect to remote host: Connection refused /var r: telnet localhost 2222 Trying 127.0.0.1... telnet: Unable to connect to remote host: Connection refused | Have a look at the debug log. It should say if it is listening. My versio= n=20 | says: |=20 | 2006/03/01 10:31 GMT [-] twisted.manhole.telnet.ShellFactory starting on = 2222 | 2006/03/01 10:31 GMT [-] Starting factory <twisted.manhole.telnet.ShellFa= ctory=20 | instance at 0xb77611ec> I see no related AP log message at all, nor any error log in the syslogs - only the connect attempt. And i _can_ do telnet, local and from other mac= hines. It's just not working.=20 =20 | > At this occasion i noticed that AP keeps a direct stream even if it can= 't | > use the cache anymore. Well done. |=20 | Erm, that is just a lucky accident :) :) Be the force with us. ps. I'm free to do some tuning now...so i just I recreated the partition=20 with dir_index, let's see what it does... mke2fs /dev/hda13 -c -T largefile -m 0 -O dir_index -O sparse_user -T largefile (1 inode per 1MB) created: Block size: 4096 Fragment size: 4096 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 128 Inode blocks per group: 4 -T news (1 inode / 4k block) would do: Block size: 4096 Fragment size: 4096 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 32576 Inode blocks per group: 1018 I can get a list of the installed packages' sizes with aptitude search ~i -F "%D %p" and it appears there are very few packages with MB sizes (>=3D10000kB),=20 like some kernel sources, and most notably the 144 MB openoffice=20 clipart .deb file. This leaves1650 packages with plain average 540kB. (However, there are 250 additional virtual packges with <N/A> size) Looks like largefile is ok, right ? =B0 /\/ |
Thanks for helping keep SourceForge clean.
X