You can subscribe to this list here.
2000 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(2) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2001 |
Jan
|
Feb
(5) |
Mar
|
Apr
(2) |
May
(3) |
Jun
|
Jul
|
Aug
(2) |
Sep
|
Oct
|
Nov
|
Dec
|
2002 |
Jan
(2) |
Feb
(5) |
Mar
(14) |
Apr
(1) |
May
(7) |
Jun
(2) |
Jul
(7) |
Aug
(13) |
Sep
(21) |
Oct
(3) |
Nov
|
Dec
|
2003 |
Jan
(6) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(5) |
Oct
(5) |
Nov
|
Dec
(3) |
2004 |
Jan
(11) |
Feb
(2) |
Mar
(4) |
Apr
|
May
(9) |
Jun
|
Jul
(1) |
Aug
(12) |
Sep
(6) |
Oct
(7) |
Nov
(10) |
Dec
(3) |
2005 |
Jan
(22) |
Feb
(20) |
Mar
(5) |
Apr
(10) |
May
(15) |
Jun
(14) |
Jul
(9) |
Aug
(3) |
Sep
(7) |
Oct
(1) |
Nov
(3) |
Dec
(12) |
2006 |
Jan
(1) |
Feb
(1) |
Mar
(5) |
Apr
|
May
|
Jun
(2) |
Jul
|
Aug
|
Sep
(2) |
Oct
|
Nov
(2) |
Dec
(2) |
2007 |
Jan
|
Feb
(3) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
From: Thorsten G. <tgu...@gm...> - 2002-08-13 20:36:17
|
Hi *, In my apt-proxy.conf I specified several servers: ,---- | ... | add_backend /main/ \ | $APT_PROXY_CACHE/main/ \ | ftp.de.debian.org::debian/ \ | ftp2.de.debian.org::debian/ \ | ftp.us.debian.org::debian/ \ | ftp.uk.debian.org::debian/ | ... `---- quoting man apt-proxy.conf: ,---- | ... would turn into a back-end request of first | `somelocalmirror::helixcode/debian/woody/main/binary-i386/x11/foo_1-1.deb' | and if that failed, ^^^^^^^^^^^^^^^^^^^^^^ | `spidermonkey.helixcode.com::http/distributions/debian/woody/main/binary-i386/x11/foo_1-1.deb' `---- So I expected that I wouldn't notice it if ftp.de.debian.org is offline while ftp2.de.debian.org is online. Well - I noticed it ;-) Comments? regards Thorsten -- There's a door Where does it go? It stays where it is, I think. (Terry Pratchett, Eric) |
From: Chris H. <chr...@gm...> - 2002-08-09 11:31:41
|
Thanks Craig for answering this; just a quick comment: On Fri, Aug 09, 2002 at 10:08:42AM +1200, Nigel Walters wrote: > I thought rsync would be helpful mainly for subsequent=20 > OpenOffice downloads. Unfortunately, rsync doesn't deal efficiently with compressed files such as =2Edebs, and you get worse performance overall with rsync - an rsynced .deb= is typically reported as 0.99-1.1 times the original size, i.e. you get almost no saving and often it is actually worse due to the protocol overhead, and the load it puts on the rsync server. It is only worth using rsync for updating the large uncompressed Packages files found on the debian archvies; everything else is unnecessary, and puts unwanted load on the server. I changed the default apt-proxy.conf to reflect this in my last upload. The default was rsync becuase that was the only protocol apt-proxy origianally supported, and I did not want to change the default immeditately until the new http/ftp support had been proved to be reliable first. You can see the rsync stats by setting KEEP_STATS or DEBUG in apt-proxy.conf, if you would like to test it out yourself. Chris |
From: Craig F. <cr...@wi...> - 2002-08-08 22:51:03
|
I put those very lines (for hhtp) into my own apt-proxy.conf and sources.list and everything worked just fine, both for an 'apt-get update' and an 'apt-get install -d openoffice.org'. Here are the exact excerpts from those files: /etc/apt-proxy/apt-proxy.conf: add_backend /openoffice/ \ $APT_PROXY_CACHE/openoffice/ \ http://www.phy.olemiss.edu/openoffice/ /etc/apt/sources.list: deb http://localhost:9999/openoffice/ testing main contrib deb http://localhost:9999/openoffice/ unstable main contrib With a "500 Bad Configuration" error, it could be that another line (unrelated to your openoffice backend) in your apt-proxy.conf is messed up. Do you have any other backends setup under apt-proxy that do work? Go through your entire config file to make sure there isn't another problem. If you can't find any problems in the config file, feel free to email me your whole apt-proxy.conf file of the list, and I will look it over myself. -- Craig Foster cr...@wi... Nigel Walters <li...@ih...> said: > On Thu, 08 Aug 2002 14:40, you wrote: > > I think you are confusing the rsync and http syntax. As you have it setup, > > apt-proxy is looking for an rsync server at www.phy.olemiss.edu. I have no > > idea about whether that host is running an rsync server, but your line in > > sources.list accessed it by http. Even if rsync is setup on that server, > > the Packages file would likely be small and probably not worth the extra > > server overhead caused by rsync. Just use http: > > > > Yes rsync is set up on this server but I went back and retested using http > with the lines suggested and still it did not work. (Error was 500 Bad > Configuration). I thought rsync would be helpful mainly for subsequent > OpenOffice downloads. > > > > add_backend /openoffice/ \ > > $APT_PROXY_CACHE/openoffice/ \ > > http://www.phy.olemiss.edu/openoffice/ > > > > Then add this to your sources.list files on various Debian hosts: > > > > deb http://<apt-proxy-host>:9999/openoffice/ testing main contrib > > > > (replacing <apt-proxy-host> with the real hostname of course). > > So anyone who knows a working pair of lines for sources.list and > apt-proxy.conf to retrieve the OpenOffice debs from www.phy.olemiss.edu. I > would be very pleased to see them. > > Thanks > Nigel > > > > > ------------------------------------------------------- > This sf.net email is sponsored by:ThinkGeek > Welcome to geek heaven. > http://thinkgeek.com/sf > _______________________________________________ > Apt-proxy-users mailing list > Apt...@li... > https://lists.sourceforge.net/lists/listinfo/apt-proxy-users > |
From: Nigel W. <li...@ih...> - 2002-08-08 22:11:33
|
On Thu, 08 Aug 2002 14:40, you wrote: > I think you are confusing the rsync and http syntax. As you have it setup, > apt-proxy is looking for an rsync server at www.phy.olemiss.edu. I have no > idea about whether that host is running an rsync server, but your line in > sources.list accessed it by http. Even if rsync is setup on that server, > the Packages file would likely be small and probably not worth the extra > server overhead caused by rsync. Just use http: > Yes rsync is set up on this server but I went back and retested using http with the lines suggested and still it did not work. (Error was 500 Bad Configuration). I thought rsync would be helpful mainly for subsequent OpenOffice downloads. > add_backend /openoffice/ \ > $APT_PROXY_CACHE/openoffice/ \ > http://www.phy.olemiss.edu/openoffice/ > > Then add this to your sources.list files on various Debian hosts: > > deb http://<apt-proxy-host>:9999/openoffice/ testing main contrib > > (replacing <apt-proxy-host> with the real hostname of course). So anyone who knows a working pair of lines for sources.list and apt-proxy.conf to retrieve the OpenOffice debs from www.phy.olemiss.edu. I would be very pleased to see them. Thanks Nigel |
From: Manuel E. S. <ra...@bi...> - 2002-08-08 11:39:01
|
On Tue, Aug 06, 2002 at 05:27:57AM -0000, Craig Foster wrote: > Okay, I've had a couple more evenings to sit down and play with apt-proxy- > v2. I figured out a couple of things (such as the existence > of /etc/init.d/apt-proxy-v2 (-; ), and I have also come across a few > oddities. Some of these I think I mentioned in my last email to the list, > but I'll elaborate more here: > > 1) apt-proxy-v2 does not seem to serve files from its own cache to clients. > This happens for Packages.gz and Release files as well as packages. Those files are mutable (meaning that a file with the same name may change over time, other versioned files like .deb's don't change) and receive different treatment. > Lines 15-20: > 06/08/2002 00:14 [AptProxy,0,127.0.0.1] [debug:9]CHECKING_CACHED > 06/08/2002 00:14 [-] [verify:9]Process Status: -1 > 06/08/2002 00:14 [-] [verify:9]unknown file: not verified > 06/08/2002 00:14 [-] > 06/08/2002 00:14 [-] [verify:9]verication failed > 06/08/2002 00:14 [-] [debug:9]NOT_CACHED Integrity verification seams to have failed, which makes the cached file irelevant. But the file is considered unknonw by the file verifier so the validation should not fail. There is something wrong with the FileVerifier class. > 2) apt-proxy-v2 gives 403 Forbidden responses for certain files. For > instance, trying to do an "apt-get install gs" always yields: > > Err http://localhost woody/main gs 6.53-3 > 403 Forbidden > Failed to fetch http://localhost:8000/debian/pool/main/g/gs/gs_6.53- > 3_i386.deb 403 Forbidden > > This behavior is the same for both sid and woody on multiple backend > servers. The file in question does exist and is accessible via HTTP on the > backend servers. The apt-proxy-v2 log shows the following: > > 06/08/2002 00:39 [AptProxy,2,127.0.0.1] [debug:9]Connection: keep-alive > 06/08/2002 00:39 [AptProxy,2,127.0.0.1] [debug:9]User-Agent: Debian > APT-HTTP/1.3 > 06/08/2002 00:39 [AptProxy,2,127.0.0.1] [debug:9] > 06/08/2002 00:39 [AptProxy,2,127.0.0.1] [debug:9]/../ in simplified uri It is refusing to serve the file for security reasons, because after trying to simplify all '..' ocurrences some where left, and that is a security problem. simplify_path and a complicated uri are provably to blame. > 4) twistd eats a lot of CPU cycles for a rather long time after downloads > are complete and the client has disconnected. What is it doing? It is generating Packages.gz from Packages or viceversa depending on the backend involved. It shouldn't happen when downloading .deb files. > 5) I see mention of memory leaks in the TODO file, so I assume everyone > knows about this, but: Memory usage by twistd gradually grows over time as > apt-proxy-v2 serves more and more requests. My system finally OOM'ed and the > vm killed twistd after a couple of "apt-get install -d kde"'s on my firewall > machine (which does not have any of X installed). I haven't been very careful with memory leaks, mainly because I don't really know how to control them in python, this confirms my fears :( > 6) Does apt-proxy-v2 really NEED to be a daemon that runs constantly (as > opposed to a daemon controlled by inet.d). I understand that there is > maintenance to be done on the cache, but it would seem that all of that could > be handled at the end of requests and/or by cron scripts. Likewise, I > understand that there may a slight performance degradation if the process > needs to be started by inet.d, but for small sites, having the large daemon > in memory all the time is overkill. Perhaps there could be an option to > either run apt-proxy-v2 as a daemon or from inet.d (as is done in the samba > and apache packages). Running apt-proxy-v2 from inet.d would also help > lessen the effects of the apparent memory leak in apt-proxy-v2 since a new > daemon would be started with each request and ended afterwards. This is > similar in principle to the way apache kills off child processes after they > have served so many pages in order to keep memory leaks under control. What > am I missing here that necessitates having twistd running all the time? The inetd afair, was a limitation of twisted, apt-proxy-v2 is based on it, and twisted did'nt support inetd based daemons when development started. apt-proxy-v2 is not so big when you start it, the big problem is the memory leaks. The daemon does a very light weight cache waking to make sure that there are no files there which it is not keeping track of, and it would have loking issues to do outsite of the main daemon. There are other looking issues which get quite simplified by running a permanent daemon. That said, it would be nice to have a cut-down inetd mode, but it will probably not happen soon. And from your previous email: apt-proxy-v2 should happyly work with an apt-proxy-v1 cache directory, and I believe that apt-proxy-v1 will do the same with an apt-proxy-v2 cache directory, but Chris should confirm the second. What will fail in strange ways is runnig both at the same time on the same cache directory, because they don't do locking with one another and will certainly step on each other's toes. Hope that helps ranty PS: I am not willing to work on apt-proxy-v2 right now because working remotely was not taking me anywhere, I will be back with my development machine on mid augost. You are very welcomed to try to debug apt-proxy-v2, just keep in mind that I will be more helpful when I come back with my development machine/permanent internet access. -- --- Manuel Estrada Sainz <ra...@de...> <ra...@bi...> <ra...@us...> ------------------------ <man...@hi...> ------------------- Let us have the serenity to accept the things we cannot change, courage to change the things we can, and wisdom to know the difference. |
From: Chris H. <chr...@gm...> - 2002-08-08 07:21:21
|
Hi Craig, Thank you very much for testing out v2. Your feedback is really helpful. I've not had any time yet to look at these issues, and Manuel is still on holiday, so nothing has happened yet. On Thu, Aug 01, 2002 at 06:29:50AM -0000, Craig Foster wrote: > I installed apt-proxy-v2 and started it up with a simple 'apt-proxy-v2&'.= =20 > Here are my first impressions (and a few questions). >=20 > 1) Overall performance seems really good. It is much faster than v1. Well, that was the main goal so at least we're on target there :) > 2) After an 'apt-get update' apt-proxy-v2 eats up all spare CPU cycles f= or a=20 > good while (on my K6-233 test box) after the download is completed. Memo= ry=20 > usage also increases during this time. What is it doing? I've seen similar behaviour and I suspect it may be in the database code. It builds databases of files in the cache and access times and I suspect there may be a problem. > 3) apt-proxy-v2 seems to cache both Package/Release files and packages = just=20 > fine, but it never serves up the cached files! Instead it downloads them= =20 > from the backend with every request. The debugging messages say: >=20 > [debug:9]CHECKING_CACHED > [debug:9]NOT_CACHED >=20 > even when the file has been cached. This is great for building a partial= =20 > mirror, but kinda defeats the purpose of a proxy, eh? ;-) Heh, well, I'd suspected something like that was happening but you've confimed it now. > 4) Over time, apt-proxy-v2 continues to grow in memory with each success= ive=20 > download. I've generally killed the process when it reached around 40MB = in=20 > memory. Oops, something has a nice memory leak I guess. > 5) What is the proper way to start apt-proxy-v2? I've just been doing= =20 > a 'apt-proxy-v2&' from the command prompt for testing. Do I need any=20 > parameters? Is it designed to run under inetd or as a standalone server? Currently as a standalone server, although the twisted guys added inetd support for us recently. The package installs an init.d script, so you can start/stop with /etc/init.d/apt-proxy-v2 start|stop. For testing in the foreground, there is a script called aptest in the source package - just run that from the command line. You can run that script directly within a working directory of a CVS checkout. > 6) Is it safe to use the same cache directory for v1 and v2? It sure wo= uld=20 > be nice not to have 2 separate partial mirrors being built while I test t= his=20 > thing out! I'm not sure - I wanted to check with Manuel before doing it myself! > Overall, that's not bad for an alpha package! I'll be following its prog= ress=20 > closely. Let me know how many of these issues are matters of my usage an= d=20 > configuration. Thanks again for your feedback. I don't think your usage or configuration caused any of the problems. Chris |
From: Chris H. <chr...@gm...> - 2002-08-08 07:04:36
|
On Thu, Aug 08, 2002 at 12:22:22PM +1200, Nigel Walters wrote: > Hi, >=20 > My sources.list file contains the line >=20 > deb http://www.phy.olemiss.edu/openoffice/ testing main contrib >=20 > This works fine. I have tried to convert this to use apt-proxy. The attem= pt I=20 > think should work (but doesn't !) is >=20 > add_backend /openoffice/ \=20 > $APT_PROXY_CACHE/openoffice/ \ > www.phy.olemiss.edu::openoffice/ This syntax is for rsync backends, but you need http: add_backend /openoffice/ \=20 $APT_PROXY_CACHE/openoffice/ \ http://www.phy.olemiss.edu/openoffice/ Chris |
From: Nigel W. <li...@ih...> - 2002-08-08 00:24:57
|
Hi, My sources.list file contains the line deb http://www.phy.olemiss.edu/openoffice/ testing main contrib This works fine. I have tried to convert this to use apt-proxy. The attempt I think should work (but doesn't !) is add_backend /openoffice/ \ $APT_PROXY_CACHE/openoffice/ \ www.phy.olemiss.edu::openoffice/ I would love to know the correct line, but I would also like to understand what I am doing wrong ... particularly how to select the "correct URL prefix" even from reading working files and the man pages I don't get this parameter. Thanks Nigel |
From: Craig F. <cr...@wi...> - 2002-08-06 05:28:08
|
Okay, I've had a couple more evenings to sit down and play with apt-proxy- v2. I figured out a couple of things (such as the existence of /etc/init.d/apt-proxy-v2 (-; ), and I have also come across a few oddities. Some of these I think I mentioned in my last email to the list, but I'll elaborate more here: 1) apt-proxy-v2 does not seem to serve files from its own cache to clients. This happens for Packages.gz and Release files as well as packages. The attached file (getpackages) contains the log messages for the second of two "apt-get update"'s. The second "apt-get update" was executed immedately after the twistd process had gone idle from the first one, and min_refresh_delay is set to 1d in /etc/apt-proxy/apt-proxy-v2.conf. Therefore, apt-proxy-v2 should have served up Packages.gz and Release from its cache, but instead it downloaded them again from the backend and sent them to the client (again). Lines which may be relevant or important include: Line 10 (note the two very different dates): 06/08/2002 00:14 [AptProxy,0,127.0.0.1] [debug:9]If-Modified-Since: Fri, 19 Jul 2002 19:03:33 GMT Lines 15-20: 06/08/2002 00:14 [AptProxy,0,127.0.0.1] [debug:9]CHECKING_CACHED 06/08/2002 00:14 [-] [verify:9]Process Status: -1 06/08/2002 00:14 [-] [verify:9]unknown file: not verified 06/08/2002 00:14 [-] 06/08/2002 00:14 [-] [verify:9]verication failed 06/08/2002 00:14 [-] [debug:9]NOT_CACHED These messages were for the retrieval of Packages.gz. Similar messages exist for Release as well. Any idea of where to start looking to debug this one? 2) apt-proxy-v2 gives 403 Forbidden responses for certain files. For instance, trying to do an "apt-get install gs" always yields: Err http://localhost woody/main gs 6.53-3 403 Forbidden Failed to fetch http://localhost:8000/debian/pool/main/g/gs/gs_6.53- 3_i386.deb 403 Forbidden This behavior is the same for both sid and woody on multiple backend servers. The file in question does exist and is accessible via HTTP on the backend servers. The apt-proxy-v2 log shows the following: 06/08/2002 00:39 [AptProxy,2,127.0.0.1] [debug:9]Connection: keep-alive 06/08/2002 00:39 [AptProxy,2,127.0.0.1] [debug:9]User-Agent: Debian APT-HTTP/1.3 06/08/2002 00:39 [AptProxy,2,127.0.0.1] [debug:9] 06/08/2002 00:39 [AptProxy,2,127.0.0.1] [debug:9]/../ in simplified uri 06/08/2002 00:39 [AptProxy,2,127.0.0.1] (harmless warning): discarding zero-length data for request <GET /debian/pool/main/g/gs/gs_6.53-3_i386.deb HTTP/1.1> 06/08/2002 00:39 [AptProxy,2,127.0.0.1] [debug:9]Client connection closed Note that the "simplified uri" and "harmless warning" lines do not occur in the logs for successful transfers. 3) apt-roxy-vs fails to start upon installation. I didn't save the actual error message, but I seem to remember that there was a permissions error relating to the log file. I also remember that the aptproxy user was created after that error. Perhaps this order is reversed? At any rate, doing an "/etc/init.d/apt-proxy-v2 start" after the install starts it up normally. I will try to take the time to do a complete uninstall and reinstall sometime soon to get the details of all of this. 4) twistd eats a lot of CPU cycles for a rather long time after downloads are complete and the client has disconnected. What is it doing? 5) I see mention of memory leaks in the TODO file, so I assume everyone knows about this, but: Memory usage by twistd gradually grows over time as apt-proxy-v2 serves more and more requests. My system finally OOM'ed and the vm killed twistd after a couple of "apt-get install -d kde"'s on my firewall machine (which does not have any of X installed). 6) Does apt-proxy-v2 really NEED to be a daemon that runs constantly (as opposed to a daemon controlled by inet.d). I understand that there is maintenance to be done on the cache, but it would seem that all of that could be handled at the end of requests and/or by cron scripts. Likewise, I understand that there may a slight performance degradation if the process needs to be started by inet.d, but for small sites, having the large daemon in memory all the time is overkill. Perhaps there could be an option to either run apt-proxy-v2 as a daemon or from inet.d (as is done in the samba and apache packages). Running apt-proxy-v2 from inet.d would also help lessen the effects of the apparent memory leak in apt-proxy-v2 since a new daemon would be started with each request and ended afterwards. This is similar in principle to the way apache kills off child processes after they have served so many pages in order to keep memory leaks under control. What am I missing here that necessitates having twistd running all the time? -- Craig Foster cr...@wi... |
From: Craig F. <cr...@wi...> - 2002-08-01 06:29:55
|
I installed apt-proxy-v2 and started it up with a simple 'apt-proxy-v2&'. Here are my first impressions (and a few questions). 1) Overall performance seems really good. It is much faster than v1. 2) After an 'apt-get update' apt-proxy-v2 eats up all spare CPU cycles for a good while (on my K6-233 test box) after the download is completed. Memory usage also increases during this time. What is it doing? 3) apt-proxy-v2 seems to cache both Package/Release files and packages just fine, but it never serves up the cached files! Instead it downloads them from the backend with every request. The debugging messages say: [debug:9]CHECKING_CACHED [debug:9]NOT_CACHED even when the file has been cached. This is great for building a partial mirror, but kinda defeats the purpose of a proxy, eh? ;-) 4) Over time, apt-proxy-v2 continues to grow in memory with each successive download. I've generally killed the process when it reached around 40MB in memory. 5) What is the proper way to start apt-proxy-v2? I've just been doing a 'apt-proxy-v2&' from the command prompt for testing. Do I need any parameters? Is it designed to run under inetd or as a standalone server? 6) Is it safe to use the same cache directory for v1 and v2? It sure would be nice not to have 2 separate partial mirrors being built while I test this thing out! Overall, that's not bad for an alpha package! I'll be following its progress closely. Let me know how many of these issues are matters of my usage and configuration. -- Craig Foster cr...@wi... |
From: Chris H. <chr...@gm...> - 2002-07-23 12:46:41
|
Hi folks, apt-proxy-v2 is at alpha stage. The main implementation has been done by Manuel, and I have been testing it a small production environment for a couple of weeks. I've added the apt-proxy-v2 package to the 'unstable' repository on sourceforce (deb http://apt-proxy.sourceforge.net/apt-proxy unstable/). This version of apt-proxy is still under development and is not fully stable, but is already much quicker than apt-proxy v1, except for Packages file updates using rsync. It will install parallel to apt-proxy: names that clash with v1 have a -v2 suffix, and will be renamed once apt-proxy-v2 reaches version 2.0 and replaces apt-proxy v1. Known issues: - Rsync backend support is not yet available - Multiple backend support is not yet implemented - apt-proxy-import appears to be buggy still - Occasionaly it is necessary to restart the apt-get operation. You can improve stablility somewhat by setting in /etc/apt/apt.conf: Aquire::http::Pipeline-Depth "0"=20 This is a temporary workaround and is only necessary until the problem has been found. This development version listens on port 8000 instead of 9999, so you can easily switch between the two by changing the port in /etc/apt/sources.list. That's it, except a big THANK YOU! to Manuel, who is taking a well-earned break at the moment. Chris |
From: Chris H. <chr...@gm...> - 2002-07-16 11:15:27
|
Hi Ashley, On Mon, Jul 15, 2002 at 04:58:06PM +0100, Ashley Collins wrote: > deb http://mookie:9999/grustibus/ ./ The './' is not recognised at the moment - you should use '/' instead. > > But I had no luck. So, I tried making it: > > deb http://mookie:9999/grustibus/ unstable main That would not work because this type of deb line makes apt prepend dists/binary-$ARCH to the URL requested. > Can anyone help me? I use grustibus as an example, but I really want to use > this to cache the Xfree86 4.2.0 previews and KDE3 packages. Here are the settings I am using for XF4.2 and KDE3: apt-proxy.conf: add_backend /kde3/ $APT_PROXY_CACHE/kde/ http://kde3.geniussystems.net/debian/ add_backend /xfree/ $APT_PROXY_CACHE/xfree/ http://www.A-Eskwadraat.nl/~bas/x4.2/ sources.list: deb http://aptproxy:9999/kde3 / deb http://aptproxy:9999/xfree sid/$(ARCH)/ HTH, Chris |
From: Ashley C. <ash...@mo...> - 2002-07-15 15:58:41
|
Hello, I have a question about how to turn lines from sources.list into apt-prox= y=20 backends. An example of this is the grustibus package provided by adding this to=20 /etc/apt/sources.list =09deb http://debian.sente.pl/ ./ I've tried to add this to my apt-proxy.conf as: #--- Grustibus repository add_backend /grustibus/ \ =20 $APT_PROXY_CACHE/debian/grustibus/ \ =20 http://grustibus.sourceforge.net/debian I then tried changing the sources.list entry to this (mookie is the name = of=20 apt-proxy cache machine): =09deb http://mookie:9999/grustibus/ ./ But I had no luck. So, I tried making it: =09deb http://mookie:9999/grustibus/ unstable main But that did not work either. I both cases I get errors in apt-proxy.log=20 saying that Release, Packages or Packages.gz not existing on any server. Can anyone help me? I use grustibus as an example, but I really want to u= se=20 this to cache the Xfree86 4.2.0 previews and KDE3 packages. Many thanks. Ashley --=20 "If things seem a bit weird today, you've probably got CAPS on in vi..." |
From: Manuel E. S. <ran...@ra...> - 2002-07-15 11:23:01
|
Hello, This time is for real, I will have no internet access at all so there is no way I can do any apt-proxy related work within the next couple of weeks. And anyway, I have been messing eveything up since I work remotely, this is a pain :( In case someone want's to try to fix something, you will find a couple of patches at http://ranty.ddts.net/~ranty/apt-proxy/cvs/apt-proxy/ and my development files. Take care ranty -- --- Manuel Estrada Sainz <ra...@de...> <ra...@bi...> <ra...@us...> ------------------------ <man...@hi...> ------------------- God grant us the serenity to accept the things we cannot change, courage to change the things we can, and wisdom to know the difference. |
From: 'Chris H. <chr...@gm...> - 2002-07-04 06:48:02
|
Hi Roland, Thank you very much for doing that detailed analysis on the rsync protocol - I was not aware that compression could be selectively turned off on a per-server basis. When I went to look at the code, I discovered that that it was actually written to rsync the Packages.gz the first time as you suggested, but that due to a typo the uncompressed file would be retrieved every time instead. I attach a patch that fixes this. This change will be in the next upload I make. Thanks again, Chris |
From: Chris H. <chr...@gm...> - 2002-07-01 08:10:24
|
On Mon, Jul 01, 2002 at 03:52:21PM +1000, Roland Gerlach wrote: > The first fetch of the main/Packages.gz file appears to have downloaded the > 7M Packages file to generate the Packages.gz file. I appreciate that > apt-proxy rsync's the Packages file to transfer only differences but could > apt-proxy detect that the Packages file is missing and download the > Packages.gz file (only 2M) and gunzip it to create the initial Packages > file? When downloading the uncompressed packages file, apt-proxy uses this command: $RSYNC_CMD -z $TMP_URL $TMP_FILE The -z tells rsync to compress the on-the-wire transfer, so you should not have seen 7MB be transferred. Did you measure how much traffic was generated? Chris |
From: Roland G. <rol...@me...> - 2002-07-01 05:52:37
|
With the imminent release of woody, I've turned my apt sources.list file to sid. The first fetch of the main/Packages.gz file appears to have downloaded the 7M Packages file to generate the Packages.gz file. I appreciate that apt-proxy rsync's the Packages file to transfer only differences but could apt-proxy detect that the Packages file is missing and download the Packages.gz file (only 2M) and gunzip it to create the initial Packages file? Just trying to conserve bandwidth. Cheers, Roland. -- Tell me and I'll forget; show me and I may remember; involve me and I'll understand - Chinese Proverb. |
From: Chris H. <chr...@gm...> - 2002-06-04 18:37:15
|
Hi Thomas, On Tue, Jun 04, 2002 at 10:18:56AM -0500, Thomas Cook wrote: > I am trying to set up a partial Debian archive mirror for my network using > apt-proxy. Unfortunately, I don=B9t have 40+ Gb to use, so I would like = to > only mirror i386 testing (contrib., main, non-free, non-us, no source). I'm not sure if you understand what apt-proxy does. It is not a mirroring tool that creates a complete mirror of all packages - see debmirror for that. But if you are concerned about disk space, apt-proxy more suitable. apt-proxy will only mirror packages that are requested by clients (machines running apt), so if you only tell your clients to use apt-proxy for i386 testing packages, apt-proxy will only mirror those files in i386 testing that have been requested. > After reading the man pages and the README file, my impression is that th= is > is done in the add_backend section of apt-proxy.conf, but I'm a little fu= zzy > on the details. Yes, that's right. > In looking through the Debian archive by hand, I can see that under the > stable 'potato' directories, there are real .deb packages, but under test= ing > and unstable, there is only a package list. Do I aim apt-proxy at the li= st > files? Do I need to pick each package out of the stable dir? apt-proxy determines which files to download by looking at the requests received from apt clients. You just need to configure apt-proxy with the toplevel directory of the archive hierarchy using add_backend as given in the example apt-proxy.conf. As you say, it means apt-proxy could potentially mirror packages from all the distributions - potato,woody,sid etc., but it will only populate the cache with files that have been requested. Provided your clients are not configured to request packages from stable or sid from apt-proxy, those files will not be mirrored. Chris |
From: Thomas C. <sys...@bl...> - 2002-06-04 15:18:59
|
I'm not sure if this list is still up, but I though I would give it a shot. I am trying to set up a partial Debian archive mirror for my network using apt-proxy. Unfortunately, I don=B9t have 40+ Gb to use, so I would like to only mirror i386 testing (contrib., main, non-free, non-us, no source). After reading the man pages and the README file, my impression is that this is done in the add_backend section of apt-proxy.conf, but I'm a little fuzz= y on the details. In looking through the Debian archive by hand, I can see that under the stable 'potato' directories, there are real .deb packages, but under testin= g and unstable, there is only a package list. Do I aim apt-proxy at the list files? Do I need to pick each package out of the stable dir? I would really appreciate someone pointing me in the right direction, thanks, -Tom |
From: Christian W. M. <wo...@dk...> - 2002-05-28 16:26:05
|
Hi, > Steven and Christian, please let me know if the problems you reported = are > solved. It seems to work now, thanks for the update. Christian |
From: Chris H. <chr...@gm...> - 2002-05-28 07:47:10
|
Hi all, I've investigated the bugs reported on this list over the past few days and have put up a new version in the sourceforge repository (unstable section): http://apt-proxy.sourceforge.net/apt-proxy/unstable/apt-proxy_1.3.1.2pre1_a= ll.deb Steven and Christian, please let me know if the problems you reported are solved. Changelog: apt-proxy (1.3.1.2pre1) unstable; urgency=3Dlow * Depend on binutils for ar (closes: #143573) * Accept '~' characters in URLs * Clean up URLs used internally * apt-proxy-import: copy files instead of linking if using dpkg-name from Potato * Copy links to .debs as files, not links on rysnc backends. This made it impossible to download some potato .debs on ftp.de.debian.org. * Refuse to run apt-proxy as the root user. This is not a good idea security-wise. * Check for write permission in backend directories Thanks, Chris --=20 Chris Halls | Frankfurt, Germany |
From: Christian W. M. <wo...@dk...> - 2002-05-27 19:43:06
|
Hi, I just tried apt-proxy on a debian potato setup. It seems like it has = problems with symlinks in the debian archive or maybe I missed = something? It seems to download only the symlinks and not the files they = are refering to: An error full execution: ----------------------------------------------------------------- don:/etc/bind# apt-get upgrade Reading Package Lists... Done Building Dependency Tree... Done The following packages have been kept back mc mc-common=20 36 packages upgraded, 0 newly installed, 0 to remove and 2 not upgraded. Need to get 16.0MB of archives. After unpacking 10.3MB will be used. Do you want to continue? [Y/n]=20 Err http://debian.dkik.dk potato/main e2fsprogs 1.18-3.0potato1 404 File does not exist on any server 0% [Waiting for file]^C don:/etc/bind#=20 ----------------------------------------------------------------- sources.list from don: ----------------------------------------------------------------- deb http://debian.dkik.dk:9999/main potato main contrib non-free deb http://debian.dkik.dk:9999/non-US potato/non-US main contrib = non-free ----------------------------------------------------------------- From the log of apt-proxy: ----------------------------------------------------------------- [26464 21:32:31]: apt-proxy 1.3.1 [26464 21:32:31]: GET = /main/dists/potato/main/binary-i386/base/e2fsprogs_1.18-3.0potato1.deb = HTTP/1.1 [26464 21:32:31]: Host: debian.dkik.dk:9999 [26464 21:32:31]: Connection: keep-alive [26464 21:32:31]: Connection keep-alive [26464 21:32:31]: User-Agent: Debian APT-HTTP/1.2 26464 21:32:31]:=20 [26464 21:32:31]: Finished processing request [26464 21:32:31] Mon May 27 21:32:31 CEST 2002 Request = /main/dists/potato/main/binary-i386/base/e2fsprogs_1.18-3.0potato1.deb [26464 21:32:31]: Looking for file = `/main/dists/potato/main/binary-i386/base/e2fsprogs_1.18-3.0potato1.deb' [26464 21:32:31]: verify_file = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/e2fsprogs_1= .18-3.0potato1.deb [26464 21:32:31]: -- Created lock = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/e2fsprogs_1= .18-3.0potato1.deb.lock [26464 21:32:31]: Fetching = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/e2fsprogs_1= .18-3.0potato1.deb from ftp.de.debian.org::debian/ = ftp2.de.debian.org::debian/ ftp.se.debian.org::debian/ = ftp.us.debian.org::debian/ ftp.uk.debian.org::debian/ [26464 21:32:31]: Trying ftp.de.debian.org::debian/... [26464 21:32:31]: = download_url:ftp.de.debian.org::debian/dists/potato/main/binary-i386/base= /e2fsprogs_1.18-3.0potato1.deb = dest:/mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base = file:e2fsprogs_1.18-3.0potato1.deb.partial = [realname:/mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/e= 2fsprogs_1.18-3.0potato1.deb] IsDir:[] [26464 21:32:31]: Found best basis for = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/e2fsprogs_1= .18-3.0potato1.deb: [26464 21:32:31]: rsync --timeout=3D30 -v --perms --links --times = --partial = ftp.de.debian.org::debian/dists/potato/main/binary-i386/base/e2fsprogs_1.= 18-3.0potato1.deb = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/e2fsprogs_1= .18-3.0potato1.deb.partial [26464 21:32:32] wrote 153 bytes read 545 bytes 465.33 bytes/sec total = size is 71 speedup is 0.10 cp: = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/e2fsprogs_1= .18-3.0potato1.deb: No such file or directory [26464 21:32:32]: -- Removed lock = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/e2fsprogs_1= .18-3.0potato1.deb.lock [26464 21:32:32] Bad URL = /main/dists/potato/main/binary-i386/base/e2fsprogs_1.18-3.0potato1.deb = File does not exist on any server [26464 21:32:32]: write_header 404 File does not exist on any server, = keep-alive:keep-alive content:text/html size:250 date: enc: request: [26464 21:32:32]: GET = /main/dists/potato/main/binary-i386/base/gzip_1.2.4-33.1.deb HTTP/1.1 [26464 21:32:32]: Host: debian.dkik.dk:9999 [26464 21:32:32]: Connection: keep-alive [26464 21:32:32]: Connection keep-alive [26464 21:32:32]: User-Agent: Debian APT-HTTP/1.2 26464 21:32:32]:=20 [26464 21:32:32]: Finished processing request [26464 21:32:32] Mon May 27 21:32:32 CEST 2002 Request = /main/dists/potato/main/binary-i386/base/gzip_1.2.4-33.1.deb [26464 21:32:32]: Looking for file = `/main/dists/potato/main/binary-i386/base/gzip_1.2.4-33.1.deb' [26464 21:32:32]: verify_file = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/gzip_1.2.4-= 33.1.deb [26464 21:32:32]: -- Created lock = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/gzip_1.2.4-= 33.1.deb.lock [26464 21:32:32]: Fetching = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/gzip_1.2.4-= 33.1.deb from ftp.de.debian.org::debian/ ftp2.de.debian.org::debian/ = ftp.se.debian.org::debian/ ftp.us.debian.org::debian/ = ftp.uk.debian.org::debian/ [26464 21:32:32]: Trying ftp.de.debian.org::debian/... [26464 21:32:32]: = download_url:ftp.de.debian.org::debian/dists/potato/main/binary-i386/base= /gzip_1.2.4-33.1.deb = dest:/mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base = file:gzip_1.2.4-33.1.deb.partial = [realname:/mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/g= zip_1.2.4-33.1.deb] IsDir:[] [26464 21:32:32]: Found best basis for = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/gzip_1.2.4-= 33.1.deb: [26464 21:32:32]: rsync --timeout=3D30 -v --perms --links --times = --partial = ftp.de.debian.org::debian/dists/potato/main/binary-i386/base/gzip_1.2.4-3= 3.1.deb = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/gzip_1.2.4-= 33.1.deb.partial [26464 21:32:34] wrote 143 bytes read 520 bytes 265.20 bytes/sec total = size is 56 speedup is 0.08 cp: = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/gzip_1.2.4-= 33.1.deb: No such file or directory [26464 21:32:34]: -- Removed lock = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/gzip_1.2.4-= 33.1.deb.lock [26464 21:32:34] Bad URL = /main/dists/potato/main/binary-i386/base/gzip_1.2.4-33.1.deb File does = not exist on any server [26464 21:32:34]: write_header 404 File does not exist on any server, = keep-alive:keep-alive content:text/html size:240 date: enc: request: ----------------------------------------------------------------- An ls of the archive ----------------------------------------------------------------- [root@compaq log]$ ls -l = /mnt/data/apt-proxy/debian/dists/potato/main/binary-i386/base/=20 total 16 lrwxrwxrwx 1 aptproxy nogroup 71 May 27 21:04 = e2fsprogs_1.18-3.0potato1.deb.partial -> = ../../../../../pool/main/e/e2fsprogs/e2fsprogs_1.18-3.0potato1_i386.deb lrwxrwxrwx 1 aptproxy nogroup 56 May 27 21:04 = gzip_1.2.4-33.1.deb.partial -> = ../../../../../pool/main/g/gzip/gzip_1.2.4-33.1_i386.deb lrwxrwxrwx 1 aptproxy nogroup 56 May 27 21:04 = libc6_2.1.3-20.deb.partial -> = ../../../../../pool/main/g/glibc/libc6_2.1.3-20_i386.deb lrwxrwxrwx 1 aptproxy nogroup 70 May 27 21:04 = libncurses5_5.0-6.0potato2.deb.partial -> = ../../../../../pool/main/n/ncurses/libncurses5_5.0-6.0potato2_i386.deb lrwxrwxrwx 1 aptproxy nogroup 65 May 27 21:04 = libstdc++2.10_2.95.2-13.1.deb.partial -> = ../../../../../pool/main/g/gcc/libstdc++2.10_2.95.2-13.1_i386.deb lrwxrwxrwx 1 aptproxy nogroup 70 May 27 21:04 = ncurses-bin_5.0-6.0potato2.deb.partial -> = ../../../../../pool/main/n/ncurses/ncurses-bin_5.0-6.0potato2_i386.deb [root@compaq log]$=20 -------------------------------------------------------------- But these are all dangling symlinks... From apt-proxy.conf: -------------------------------------------------------------- APT_PROXY_CACHE=3D/mnt/data/apt-proxy add_backend /main/ \ $APT_PROXY_CACHE/debian/ \ ftp.de.debian.org::debian/ \ ftp2.de.debian.org::debian/ \ ftp.se.debian.org::debian/ \ ftp.us.debian.org::debian/ \ ftp.uk.debian.org::debian/ add_backend /non-US/ \ $APT_PROXY_CACHE/non-US/ \ ftp.de.debian.org::debian-non-US/ \ ftp2.de.debian.org::debian-non-US/ \ ftp.se.debian.org::debian-non-US/ \ ftp.uk.debian.org::debian/non-US/ add_backend /security/ \ $APT_PROXY_CACHE/security/ \ security.debian.org::debian-security/ \ non-us.debian.org::debian-security/ -------------------------------------------------------------- Christian |
From: Chris H. <chr...@gm...> - 2002-05-26 20:53:55
|
Hi Steven, On Sat, May 25, 2002 at 09:58:53PM -0400, Steven G. Johnson wrote: > Hi, I am using apt-proxy 1.3.1 under Debian/stable (potato) on x86, and I > ran into what appears to be a serious bug in apt-proxy-import. (It looks > like this utility hasn't been tested under potato; see below.) You're right - I don't use Potato. I've had a report from Thorsten Gunkel about this, but I have not had the chance to try to reproduce it myself yet. > First of all, it appears that apt-proxy-import deletes all of the .deb > files in the directory you point it to. Is this the intended > behavior? (It doesn't seem to be documented in the man page.) I don't > think that such destructive behavior should be the default. No, the indended behaviour is to rename the .debs using dpkg-name, and to avoid touching the original files, use the -k option to create symlinks in a seperate directory (dpkg-name.links), and then to delete them afterwards. > It wouldn't be so bad if it just moved the files, rather than deleting > them (although I still think it should only copy and not move), but it > appears there is a bug that causes this to fail. Yes, it's not indended and is a bug. > root# apt-proxy-import --verbose tst > tst: dpkg-name: can't find `-k' > moved `ash_0.3.5-11_i386.deb' to `/var/cache/apt-proxy/dpkg-name.links/ash_0.3.5-11_i386.deb' > skipped ash_0.3.5-11_i386.deb > > Note the message about "-k"...the apt-proxy-import script is passing the > "-k" option to dpkg-name, but this option is apparently unsupported under > potato. Hmm, I guess it won't work properly in that case. > Also, it claims to be moving the .deb to > /var/cache/apt-proxy/dpkg-name.links/, but the dpkg-name.links > subdirectory is nonexistent. That is very odd: In apt-proxy-import, mkdir is run like this just before dpkg-name is run: mkdir "$APT_PROXY_CACHE/dpkg-name.links/" It must be because dpkg-name on Potato interprets the argument to the -k option in a strange way: $DPKG_NAME -k -s "$APT_PROXY_CACHE/dpkg-name.links" "$directory"/*.deb \ | line_feedback I guess we should change the behaviour of apt-proxy-import for Potato. I was hoping Woody would have been released by now and I could avoid having to install a Potato system to track this down :) Chris -- Chris Halls | Frankfurt, Germany |
From: Steven G. J. <st...@ab...> - 2002-05-26 01:55:49
|
Hi, I am using apt-proxy 1.3.1 under Debian/stable (potato) on x86, and I ran into what appears to be a serious bug in apt-proxy-import. (It looks like this utility hasn't been tested under potato; see below.) First of all, it appears that apt-proxy-import deletes all of the .deb files in the directory you point it to. Is this the intended behavior? (It doesn't seem to be documented in the man page.) I don't think that such destructive behavior should be the default. It wouldn't be so bad if it just moved the files, rather than deleting them (although I still think it should only copy and not move), but it appears there is a bug that causes this to fail. In particular, I imported a directory full of .deb files, and it seems that more than half of them were skipped (according to the apt-proxy-import output), and I can no longer find the skipped .deb files anywhere (neither in their original directory nor under /var/cache/apt-proxy). As a simple test case, I created a directory tst/ containing ash_0.3.5-11_i386.deb, and the output was: root# apt-proxy-import --verbose tst tst: dpkg-name: can't find `-k' moved `ash_0.3.5-11_i386.deb' to `/var/cache/apt-proxy/dpkg-name.links/ash_0.3.5-11_i386.deb' skipped ash_0.3.5-11_i386.deb Note the message about "-k"...the apt-proxy-import script is passing the "-k" option to dpkg-name, but this option is apparently unsupported under potato. Also, it claims to be moving the .deb to /var/cache/apt-proxy/dpkg-name.links/, but the dpkg-name.links subdirectory is nonexistent. Thanks for your help. Cordially, Steven G. Johnson |
From: Chris H. <chr...@gm...> - 2002-05-02 08:02:04
|
On Thu, May 02, 2002 at 10:49:00AM +1000, Roland Gerlach wrote: > Periodically I burn a CD of the .deb files in apt-proxy directory on the > work system and copy the files into the apt-proxy directory at home. > Apt-proxy seams to cope with this and when I upgrade the home systems, it > performs the rsync which sends and receives only a couple of hundred bytes > to confirm that the file is up-to-date. Still much better than performing > an rsync with a previous version. >=20 > Unfortunately, when I upgrade another system at home, apt-proxy performs = the > same rsync over and over again. I'd really like to avoid the additional > traffic - any suggestions? Hmm, that behaviour sounds odd. I can't tell you why it's happening like that. Maybe you can add DEBUG=3Dtrue to /etc/apt-proxy/apt-proxy.conf and send me a copy of /var/log/apt-proxy.log for such a session? Normally, apt-proxy will not try to re-rsync a .deb once it has it in its cache. This is because once a .deb is put in the archive, it is never changed - changes are uploaded with a new filename. Packages and Sources files do get re-updated - if the inode change time is older than BACKEND_FREQ minutes. But that only happens when you run apt-get update, not apt-get install.=20 > Today I discovered apt-proxy-import and will give it a try tonight to see= if > it avoids these extra rsyncs. I don't think it will, because it just copies the file into the correct directory, and renames it if necessary. That is the same as your copy from CD. HTH, Chris --=20 Chris Halls | Frankfurt, Germany Yahoo:hagga12000 ICQ:36940860 MSN:chr...@ni... |