apt-proxy-users

[Apt-proxy-users] Performance problems on Debian/mips

[Manuel E. S. <ra...@de...>]

 Hello,

 First of all thanks a lot for apt-proxy, its a great program and an
 even better idea.

 I what to make clear that I don't what to start a flamewar and I'm not
 trying to force anything on anyone.

 apt-proxy runs quite slowly on the SGI Indy that I want to use as
 proxy, and I realized that it is written in shell script using the
 "Little powerful tools" philosophy, which has a poor performance. I
 never thought that a proxy could be written in shell script, quite
 impressive.

 Now to the point, what would be the drawbacks of using perl, python or
 even C instead of shell script?  At some point I may try to rewrite
 apt-proxy using one of those and I would like to know what you think
 about it.

 Take care

 	ranty

 
-- 
--- Manuel Estrada Sainz <ra...@de...>
			 <ra...@at...>
------------------------ <ra...@so...> ---------------------------------
God grant us the serenity to accept the things we cannot change, courage to
change the things we can, and wisdom to know the difference.

Re: [Apt-proxy-users] Performance problems on Debian/mips

[Chris H. <chr...@ni...>]

Hi Manual,

On Thu, Feb 28, 2002 at 08:10:24PM +0100, Manuel Estrada Sainz wrote:
>  First of all thanks a lot for apt-proxy, its a great program and an
>  even better idea.

:-)

>  I what to make clear that I don't what to start a flamewar and I'm not
>  trying to force anything on anyone.
> 
>  apt-proxy runs quite slowly on the SGI Indy that I want to use as
>  proxy, and I realized that it is written in shell script using the
>  "Little powerful tools" philosophy, which has a poor performance. I
>  never thought that a proxy could be written in shell script, quite
>  impressive.
> 
>  Now to the point, what would be the drawbacks of using perl, python or
>  even C instead of shell script?  At some point I may try to rewrite
>  apt-proxy using one of those and I would like to know what you think
>  about it.

As you say, the idea itself is great, but apt-proxy is not all that
wonderful. apt-proxy itself was almost unusable during 2001, because 
changes to the Debian archives (the pool structure) caused it to stop
working as intended.  During that period I think quite a few people tried
it, found it was very difficult to set up and use, and some had a go at
writing their own.  I know of at least 3 other similar projects.  I tried
debproxy over on Sourceforge, but found the code to be too buggy to be fun
to use.  When apt-proxy came up for adoption in Debian I took the chance to
incorporate the fixes I had already made into the Debian distribution, with
a view to making sure that Woody had a working apt-proxy.  

I do not really see apt-proxy in its present form as having a shelf life
beyond the Woody release, because I expect (and hope) that another
implementation will have caught up with apt-proxy, in terms of stability,
ease of installation, and features.  There isn't anything ready in all areas
yet, so apt-proxy 1.x lives on.  In the meantime I'm just having fun getting
experience as a package maintainer, upstream developer and doing something
useful at the same time :)

It would probably be a good idea for me to put this info up on the home
page, together with links to all the other projects.  I would have thought
it would be better for people to work together on one of these than have so
many little projects.  At the moment, I think aptcached looks the most
promising.  The code looks nice and clean but I haven't actually installed
it to give it a test run yet.

You can have a look here:
http://talk.trekweb.com/~jasonb/software.shtml

Chris
-- 
Chris Halls | Frankfurt, Germany <><
Yahoo:hagga12000 ICQ:36940860 MSN:chr...@ni...

Re: [Apt-proxy-users] Performance problems on Debian/mips

[<je...@au...>]

On Thu, Feb 28, 2002 at 08:10:24PM +0100, Manuel Estrada Sainz wrote:
>  Now to the point, what would be the drawbacks of using perl, python or
>  even C instead of shell script?  At some point I may try to rewrite
>  apt-proxy using one of those and I would like to know what you think
>  about it.

It's rumoured that apt-proxy was written in shell because someone made a
bet with Rusty that it couldn't be done.  So he did it.

IMO the best way to write a proxy for Debian would be to write an
apt-handler that knows it's going through a proxy.  And also a server
implementation.

I'm not sure what other projects are doing but I don't really have a
need for a proxy because I have more bandwidth than hard drive space and
I don't like to let my downloads full below 1.0 x the average user
download (my ISP gives a limit of 10 x the age user download).

-- 
Jeremy Lunn
Melbourne, Australia
http://www.jabber.org/ - the next generation of Instant Messaging.

Re: [Apt-proxy-users] Performance problems on Debian/mips

[Chris H. <chr...@ni...>]

On Fri, Mar 01, 2002 at 09:39:53AM +1100, Jeremy Lunn wrote:
> It's rumoured that apt-proxy was written in shell because someone made a
> bet with Rusty that it couldn't be done.  So he did it.

*grin*

Ah, now we have a reason why apt-proxy is a shell script that actually makes
sense :-D

> IMO the best way to write a proxy for Debian would be to write an
> apt-handler that knows it's going through a proxy.  And also a server
> implementation.

I see some disadvantages:

 - It only works with apt.  I don't know about other people, but I often use
   apt-proxy during a new install.  debootstrap uses busybox wget.

 - Every box has to have client sofware installed and kept up to date

What advantage would it bring?

> I'm not sure what other projects are doing but I don't really have a
> need for a proxy because I have more bandwidth than hard drive space and
> I don't like to let my downloads full below 1.0 x the average user
> download (my ISP gives a limit of 10 x the age user download).

Lucky you :)

Chris

Re: [Apt-proxy-users] Performance problems on Debian/mips

[Manuel E. S. <ra...@de...>]

On Thu, Feb 28, 2002 at 09:14:32PM +0100, Chris Halls wrote:
> Hi Manual,
> 
> On Thu, Feb 28, 2002 at 08:10:24PM +0100, Manuel Estrada Sainz wrote:
> >  Now to the point, what would be the drawbacks of using perl, python or
> >  even C instead of shell script?  At some point I may try to rewrite
> >  apt-proxy using one of those and I would like to know what you think
> >  about it.
> 
> I do not really see apt-proxy in its present form as having a shelf life
> beyond the Woody release, because I expect (and hope) that another
> implementation will have caught up with apt-proxy, in terms of stability,
> ease of installation, and features.  There isn't anything ready in all areas
> yet, so apt-proxy 1.x lives on.

 How about 'word by word' rewrite of apt-proxy in perl so we don't have
 to worry about logic errors and once done we will have plenty of space
 for improvements and performance enhancements.

 By 'word by word' rewrite I mean keeping all funcion names and as much
 of the structure as posible.

> It would probably be a good idea for me to put this info up on the home
> page, together with links to all the other projects.

 Please do.

> I would have thought it would be better for people to work together on
> one of these than have so many little projects.

 And I believe that apt-proxy should be the one, it is the most mature,
 and it goes right to the point.

> At the moment, I think aptcached looks the most promising.  The code
> looks nice and clean but I haven't actually installed it to give it a
> test run yet.

 Well, having apt-proxy around makes me think that aptcached looks a bit
 'short minded'. And anyway, if we do the rewrite proposed above we
 could take the nice parts of aptcached and use the in apt-proxy.

 Reading the goal's of aptcached I think that we could convince Jason
 with a 'perl improved'/'http capable' version of apt-proxy. Specially
 if there is room for some of his code I guess :)

 If I do the rewrite and it works, would you adopt it as upstream?

 And to start contributing on this, attached goes some scripts I wrote
 to import .deb into a pool structure without Package lists.

 Take care

 	ranty
-- 
--- Manuel Estrada Sainz <ra...@de...>
			 <ra...@at...>
------------------------ <ra...@so...> ---------------------------------
God grant us the serenity to accept the things we cannot change, courage to
change the things we can, and wisdom to know the difference.

Re: [Apt-proxy-users] Performance problems on Debian/mips

[Chris H. <chr...@ni...>]

Hi Manuel, sorry I took a while to get back to you.

On Sat, Mar 02, 2002 at 12:58:09PM +0100, Manuel Estrada Sainz wrote:
> On Thu, Feb 28, 2002 at 09:14:32PM +0100, Chris Halls wrote:
>=20
>  How about 'word by word' rewrite of apt-proxy in perl so we don't have
>  to worry about logic errors and once done we will have plenty of space
>  for improvements and performance enhancements.
>=20
>  By 'word by word' rewrite I mean keeping all funcion names and as much
>  of the structure as posible.

I don't know how much would be left after doing that :-)

There are quite a few of the functions that are very rsync specific and
could do with cleaning up, although I guess it would still at least provide
a framework.  If doing the rewrite myself, I guess I would consider starting
it like that.

> > At the moment, I think aptcached looks the most promising.  The code
> > looks nice and clean but I haven't actually installed it to give it a
> > test run yet.
>=20
>  Well, having apt-proxy around makes me think that aptcached looks a bit
>  'short minded'. And anyway, if we do the rewrite proposed above we
>  could take the nice parts of aptcached and use the in apt-proxy.
>=20
>  Reading the goal's of aptcached I think that we could convince Jason
>  with a 'perl improved'/'http capable' version of apt-proxy. Specially
>  if there is room for some of his code I guess :)
>=20
>  If I do the rewrite and it works, would you adopt it as upstream?

I got hold of Jason today on IRC and we discussed the whole thing.  He's
joined this list so we can discuss here.

It turns out that there are some issues with aptcached that could cause
problems:

 - aptcached depends on POE.  It was so Jason could avoid having to do
   events loops, but POE is still oficially alpha, is not in Potato and has
   a large runtime overhead,  Jason reckoned around 2MB.  I'm not against
   using libraries to help simplify coding and make it more maintainable,
   but I think that POE is too big a price to pay.  Jason understands that
   and is open to dropping POE, as long as someone else does the rewrite :-)

 - It doesn't mirror the directory structure.  That certainly simplifies the
   code but has scalability problems and you don't end up with a partial
   mirror.

 - HTTP 1.1 keep-alives aren't working (yet).

So it looks like a fair amount must be rewritten anyway.  Jason said he's
open to help with whatever we decide to do, which I'm very grateful for.

If you have the time to work on a rewrite that's great by me - I'd rather
have several people involved than just me.  It helps keep bugs shallow and
for the program to grow faster.  The only thing I would ask for is to make
sure others can work with the code too.  In particular, documenting what
code is supposed to do and helping to make sure it is production-ready.

>  And to start contributing on this, attached goes some scripts I wrote
>  to import .deb into a pool structure without Package lists.

Thanks for the scripts.  It's a nice idea, to build the path using dpkg
--info and one I had not thought of.  It only works for mirrors that use the
current Debian mirror structure, and would have to be changed if the mirror
structure were to be changed by the FTP maintainers in the future.

The time taken to write e.g. the apt-proxy-import script was probably less
than half the total time it took to get it into the package, because I had
to write documentation and do stuff like handling typical errors and provide
--help text. =20

It would still be a fair amount of work to get your scripts finished.  I'm
not sure whether you are providing them for comment or as a contribution to
the package.  If you want to help out, I would need you to help get things
nearer to a completed state.  Otherwise, I would be comitting myself to a
lot of extra work which I probably wouldn't have the time to do.

Does that sound OK?  I've set up an #apt-proxy channel on openprojects IRC
so maybe we can all talk there sometime.

Thanks,
Chris (who's glad things are picking up)
--=20
Chris Halls | Frankfurt, Germany
Yahoo:hagga12000 ICQ:36940860 MSN:chr...@ni...

Re: [Apt-proxy-users] Performance problems on Debian/mips

[Manuel E. S. <ra...@de...>]

On Tue, Mar 05, 2002 at 07:30:44PM +0100, Chris Halls wrote:
> Hi Manuel, sorry I took a while to get back to you.
> 
[snip]
> 
> I got hold of Jason today on IRC and we discussed the whole thing.  He's
> joined this list so we can discuss here.
> 
> It turns out that there are some issues with aptcached that could cause
> problems:
> 
>  - aptcached depends on POE.  It was so Jason could avoid having to do
>    events loops, but POE is still oficially alpha, is not in Potato and has
>    a large runtime overhead,  Jason reckoned around 2MB.  I'm not against
>    using libraries to help simplify coding and make it more maintainable,
>    but I think that POE is too big a price to pay.  Jason understands that
>    and is open to dropping POE, as long as someone else does the rewrite :-)
> 
>  - It doesn't mirror the directory structure.  That certainly simplifies the
>    code but has scalability problems and you don't end up with a partial
>    mirror.
> 
>  - HTTP 1.1 keep-alives aren't working (yet).
> 
> So it looks like a fair amount must be rewritten anyway.  Jason said he's
> open to help with whatever we decide to do, which I'm very grateful for.
> 
> If you have the time to work on a rewrite that's great by me - I'd rather
> have several people involved than just me.  It helps keep bugs shallow and
> for the program to grow faster.  The only thing I would ask for is to make
> sure others can work with the code too.  In particular, documenting what
> code is supposed to do and helping to make sure it is production-ready.

 I don't have a whole lot of time right now, but I will be rewriting
 apt-proxy as time permits, I will make the ongoing work publically
 available. Maybe it could be hosted in apt-proxy's CVS.

> >  And to start contributing on this, attached goes some scripts I wrote
> >  to import .deb into a pool structure without Package lists.
> 
> Thanks for the scripts.  It's a nice idea, to build the path using dpkg
> --info and one I had not thought of.  It only works for mirrors that use the
> current Debian mirror structure, and would have to be changed if the mirror
> structure were to be changed by the FTP maintainers in the future.
> 
> The time taken to write e.g. the apt-proxy-import script was probably less
> than half the total time it took to get it into the package, because I had
> to write documentation and do stuff like handling typical errors and provide
> --help text.  
> 
> It would still be a fair amount of work to get your scripts finished.  I'm
> not sure whether you are providing them for comment or as a contribution to
> the package.  If you want to help out, I would need you to help get things
> nearer to a completed state.  Otherwise, I would be comitting myself to a
> lot of extra work which I probably wouldn't have the time to do.

 It was meant as a contribution, but I accept the criticism, how about
 integrating it into "apt-proxy-import --mirror=pool", so if the file is
 not in the current listings it will use pool mirror structure. I would
 also update the manpage. Hmm, some places at "apt-proxy-import" read
 "echo" where is should be "echo -e" so it interprets escaped secuences,
 I will fix that also.

> Does that sound OK?  I've set up an #apt-proxy channel on openprojects IRC
> so maybe we can all talk there sometime.

 My nick in openprojects IRC is 'ranty', feel free to invite me over to
 #apt-proxy whenever you see me around.

 more TODO stuff :)
 ---------------
 
 About apt-proxy, I would like it to generate package lists with the
 package it already has, maybe even with a different release name.

 deb http://apt-proxy.x.x:9999/main/ unstable #same as it is now
 deb http://apt-proxy.x.x:9999/main/ mirror   #Packages would be generated
					      # by apt-proxy

 By making mirror a different release we could use 
 	APT::Default-Release "mirror"
 to try to stay with those packages and not download new ones while
 being able to install new stuff.

 apt-move those a preaty good job generating packages eficiently, we
 could take a look.

 I know this is not to happen soon, I just write it here for the record.

 Take care 

 	ranty


-- 
--- Manuel Estrada Sainz <ra...@de...>
			 <ra...@at...>
------------------------ <ra...@so...> ---------------------------------
God grant us the serenity to accept the things we cannot change, courage to
change the things we can, and wisdom to know the difference.

Re: [Apt-proxy-users] Performance problems on Debian/mips

[Jason B. <ja...@ed...>]

On Tuesday 05 March 2002 01:30 pm, you wrote:
<snip>
> I got hold of Jason today on IRC and we discussed the whole thing.  He's
> joined this list so we can discuss here.
>
> It turns out that there are some issues with aptcached that could cause
> problems:
>
>  - aptcached depends on POE.  It was so Jason could avoid having to do
>    events loops, but POE is still oficially alpha, is not in Potato and has
>    a large runtime overhead,  Jason reckoned around 2MB.  I'm not against
>    using libraries to help simplify coding and make it more maintainable,
>    but I think that POE is too big a price to pay.  Jason understands that
>    and is open to dropping POE, as long as someone else does the rewrite
> :-)

I talked to Chris again last Thursday or Friday on IRC and we started looking 
for POE alternatives, which I'm very much open to.  The usage of POE brings 
up memory requirements to somewhere around 6MB, which is quite large.

I've read through the docs of NetServer::Generic and it looks very promising. 
It supports multiple modes of operation, including select and fork, and a 
client mode which could replace my client side POE stuff that actually talks 
to the HTTP mirrors.

>  - It doesn't mirror the directory structure.  That certainly simplifies
> the code but has scalability problems and you don't end up with a partial
> mirror.

I can see the benefits to both a partial mirror and (of course) scalability.

What's the best way to go about maintaining a directory stucture?  Should the 
daemon handle it or could a script be fashioned to deal with reading the 
package lists and figuring out where things should go, freeing the daemon 
from having to deal with rather large package lists.

>  - HTTP 1.1 keep-alives aren't working (yet).
>
> So it looks like a fair amount must be rewritten anyway.  Jason said he's
> open to help with whatever we decide to do, which I'm very grateful for.

Yes.

> If you have the time to work on a rewrite that's great by me - I'd rather
> have several people involved than just me.  It helps keep bugs shallow and
> for the program to grow faster.  The only thing I would ask for is to make
> sure others can work with the code too.  In particular, documenting what
> code is supposed to do and helping to make sure it is production-ready.
>
<snip>
> Thanks,
> Chris (who's glad things are picking up)

Well, at least they were last week. :)