Enlightenment

On Mon, 14 Aug 2006 12:08:06 +0900 Carsten Haitzler (The Rasterman)
<raster@...> wrote:

> It has come to my attention that.... yet again we are killing systems.
>=20
> yes - we are becoming a burden on yet more cvs servers. we are
> monsters! :(
>=20
> anyway - we have been living on caosity's cvs for a while now - but
> we are killing it (sorry kainx!)
>=20
> so its time to finally bite the bullet and dredge up the issue of us
> needing servers again.
>=20
> here is what i think we need:
>=20
> 1. devel "cvs" server + future web server (for downloads too of
> official tarballs etc.)
> 2. an anonymous "cvs" server and possibly second download mirror.
>=20
> so 2 systems really.
>=20
> i hear that svn is significantly less load for anonymous access - even
> developer - who has experience with this server-side? can you confirm
> or deny? i would consider a possible move to svn if we can keep our
> history from cvs.
>=20
> so - let the flames begin.

I have only mild server side svn experience, not enough to warrant
comments.  All I can say is that I have a preference for svn, and as
far as I can tell, all those issues that people seem to have with it
have been addressed.  I know I have seen a page comparing cvs and svn,
and every single svn issue they raised was fixed in later versions.

Didn't we say at the outset that extra mirrors are a goal, and that we
should build an easy to setup mirror package of some sort?  How's that
going?

Is it time to seek donations for our own server?

You have mentioned recently that there are bounties for coding E, can
the same source of those funds donate some cvs server resources?

On Monday, 14 August 2006, at 12:08:06 (+0900),
Carsten Haitzler wrote:

> anyway - we have been living on caosity's cvs for a while now - but
> we are killing it (sorry kainx!)
>
> so its time to finally bite the bullet and dredge up the issue of us
> needing servers again.

What we really need is anoncvs mirrors.  We need systems that we can
point "anoncvs.enlightenment.org" to.  A nice round-robin DNS should
solve the load problem quite nicely.

> here is what i think we need:
> 
> 1. devel "cvs" server + future web server (for downloads too of official
> tarballs etc.)
> 2. an anonymous "cvs" server and possibly second download mirror.
> 
> so 2 systems really.

The developer CVS server is doing acceptably, is it not?

We are working on obtaining another server for the E project to have
as its very own.  Hopefully that will pan out.

> i hear that svn is significantly less load for anonymous access -
> even developer - who has experience with this server-side? can you
> confirm or deny?  i would consider a possible move to svn if we can
> keep our history from cvs.

It's not true.  SVN requires a lot more overhead (including Apache
with SVN and DAV modules), uses a BDB backend (you remember your love
of BDB, right?), and requires DOUBLE the amount of disk space for a
checkout.  Yes, I said double.  Furthermore, branching and tagging
don't really exist for SVN (it uses "copies" which, while they may be
zero overhead on the server, are murder on the checkout).  And last I
checked, you could not keep your history.

CVS is the devil we know.  There's really nothing we need it to do
that it doesn't do.  I see no compelling reason to move.

Michael

-- 
Michael Jennings (a.k.a. KainX)  http://www.kainx.org/  <mej@...>
n + 1, Inc., http://www.nplus1.net/       Author, Eterm (www.eterm.org)
-----------------------------------------------------------------------
 "Love?!  What does *love* have to do with *marriage*?!"
           -- Peter Jurasik (Ambassador Londo Mollari), "Babylon Five"

On Mon, 14 Aug 2006 13:59:58 +1000 David Seikel <onefang@...> babbl=
ed:

> On Mon, 14 Aug 2006 12:08:06 +0900 Carsten Haitzler (The Rasterman)
> <raster@...> wrote:
>=20
> > It has come to my attention that.... yet again we are killing systems=
.
> >=20
> > yes - we are becoming a burden on yet more cvs servers. we are
> > monsters! :(
> >=20
> > anyway - we have been living on caosity's cvs for a while now - but
> > we are killing it (sorry kainx!)
> >=20
> > so its time to finally bite the bullet and dredge up the issue of us
> > needing servers again.
> >=20
> > here is what i think we need:
> >=20
> > 1. devel "cvs" server + future web server (for downloads too of
> > official tarballs etc.)
> > 2. an anonymous "cvs" server and possibly second download mirror.
> >=20
> > so 2 systems really.
> >=20
> > i hear that svn is significantly less load for anonymous access - eve=
n
> > developer - who has experience with this server-side? can you confirm
> > or deny? i would consider a possible move to svn if we can keep our
> > history from cvs.
> >=20
> > so - let the flames begin.
>=20
> I have only mild server side svn experience, not enough to warrant
> comments.  All I can say is that I have a preference for svn, and as
> far as I can tell, all those issues that people seem to have with it
> have been addressed.  I know I have seen a page comparing cvs and svn,
> and every single svn issue they raised was fixed in later versions.
>=20
> Didn't we say at the outset that extra mirrors are a goal, and that we
> should build an easy to setup mirror package of some sort?  How's that
> going?

yes - but a primary anon cvs that we personally manage woudl be best - we=
 dont
have that atm. i think its time we stop burdeni0ng other peoples systems
though. maybe svn is also a good move?

> Is it time to seek donations for our own server?

well we started with that and had a stop-gap measure from caos as it was =
the
same color host location. kind of an intermediate step - but i am thinkin=
g it
is time to pull out this card... but ... let the flames begin.

> You have mentioned recently that there are bounties for coding E, can
> the same source of those funds donate some cvs server resources?

possibly. :) but for now maybe not. we might manage a donated server or 2=
...

--=20
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
=E8=A3=B8=E5=A5=BD=E5=A4=9A
Tokyo, Japan (=E6=9D=B1=E4=BA=AC =E6=97=A5=E6=9C=AC)

On Mon, 14 Aug 2006 00:03:07 -0400 Michael Jennings <mej@...> babbl=
ed:

> On Monday, 14 August 2006, at 12:08:06 (+0900),
> Carsten Haitzler wrote:
>=20
> > anyway - we have been living on caosity's cvs for a while now - but
> > we are killing it (sorry kainx!)
> >
> > so its time to finally bite the bullet and dredge up the issue of us
> > needing servers again.
>=20
> What we really need is anoncvs mirrors.  We need systems that we can
> point "anoncvs.enlightenment.org" to.  A nice round-robin DNS should
> solve the load problem quite nicely.

yeah - though that might be harder than we think. the other problem here =
is
consistency - we want a user using the 1 server once they started as if a=
non
servers don't update at the same time - they will have problems with
consistency. having a faster/lighter/easier access method (svn? who has
something to say? i prefer the "devil you know" but since this problem ke=
eps
coming up... time to bit the bullet?)

> > here is what i think we need:
> >=20
> > 1. devel "cvs" server + future web server (for downloads too of offic=
ial
> > tarballs etc.)
> > 2. an anonymous "cvs" server and possibly second download mirror.
> >=20
> > so 2 systems really.
>=20
> The developer CVS server is doing acceptably, is it not?

indeed it is :)

> We are working on obtaining another server for the E project to have
> as its very own.  Hopefully that will pan out.

this was kind of part of what i was getting to - to make sure that we don=
t
tread on your feet any more - move to our own box - and yes - anoncvs mir=
rors
will be good - but maybe using svn in the end will make this easier.  i a=
m
broaching the topic to start just this discussion. i am not set on a path=
 at
all - but if we are causing pain - we need to address it.

> > i hear that svn is significantly less load for anonymous access -
> > even developer - who has experience with this server-side? can you
> > confirm or deny?  i would consider a possible move to svn if we can
> > keep our history from cvs.
>=20
> It's not true.  SVN requires a lot more overhead (including Apache
> with SVN and DAV modules), uses a BDB backend (you remember your love
> of BDB, right?), and requires DOUBLE the amount of disk space for a
> checkout.  Yes, I said double.  Furthermore, branching and tagging

checkouts from svn are just ridiculous. agreed. but its the server side i=
 am
asking about. as i said - i HEARD it is easier on the server - i am after
details from those having been there, done that. yeah - bdb - oh yay. let=
s
break format all the time.. ;)

> don't really exist for SVN (it uses "copies" which, while they may be
> zero overhead on the server, are murder on the checkout).  And last I
> checked, you could not keep your history.

i hear there are migration tools - but i will wait to hear what tools the=
y are
and how well they work before ever moving.

> CVS is the devil we know.  There's really nothing we need it to do
> that it doesn't do.  I see no compelling reason to move.

agreed - unless svn is less overhead. that is the question. an svn anon s=
erver
serving 1000 users - and then an cvs anon server serving the same src and=
 the
same people - which one consumes less disk io/cpu/memory? thats what i wa=
nt to
know - and if it is less - is it significantly less? or just like 5% (not=
 worth
worrying about)

> Michael
>=20
> --=20
> Michael Jennings (a.k.a. KainX)  http://www.kainx.org/  <mej@...>
> n + 1, Inc., http://www.nplus1.net/       Author, Eterm (www.eterm.org)
> -----------------------------------------------------------------------
>  "Love?!  What does *love* have to do with *marriage*?!"
>            -- Peter Jurasik (Ambassador Londo Mollari), "Babylon Five"
>=20
> -----------------------------------------------------------------------=
--
> Using Tomcat but need to do more? Need to support web services, securit=
y?
> Get stuff done quickly with pre-integrated technology to make your job =
easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geron=
imo
> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=
=3D121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
>=20


--=20
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
=E8=A3=B8=E5=A5=BD=E5=A4=9A
Tokyo, Japan (=E6=9D=B1=E4=BA=AC =E6=97=A5=E6=9C=AC)

On Mon, Aug 14, 2006 at 12:03:07AM -0400, Michael Jennings wrote:
> On Monday, 14 August 2006, at 12:08:06 (+0900),
> Carsten Haitzler wrote:

> > i hear that svn is significantly less load for anonymous access -
> > even developer - who has experience with this server-side? can you
> > confirm or deny?  i would consider a possible move to svn if we can
> > keep our history from cvs.
> 
> It's not true.  SVN requires a lot more overhead (including Apache
> with SVN and DAV modules), uses a BDB backend (you remember your love
> of BDB, right?), and requires DOUBLE the amount of disk space for a
> checkout.  Yes, I said double.  Furthermore, branching and tagging
> don't really exist for SVN (it uses "copies" which, while they may be
> zero overhead on the server, are murder on the checkout).  And last I
> checked, you could not keep your history.

There are tools to move from cvs to svn. E.g. http://cvs2svn.tigris.org/
I've never used one, so I can't vouch for them, but it seems like it
would be a fairly straightforward operation.

I'm still curious if anyone has any stats on actual CPU load for
checkouts from CVS vs. SVN. "More apps involved" doesn't necessarily
equate to "more load". Neither does more disk space used. (But without
stats, I can't do anythign but speculate -- and I don't have the time to
generate such stats right now...)

> 
> CVS is the devil we know.  There's really nothing we need it to do
> that it doesn't do.  I see no compelling reason to move.

The main thing that SVN has that won me over is the ability to do diffs
without hitting the server. I generally check diffs a LOT before
checking in. With CVS this is a slow pita. I also find svn log more
useful than cvs log. But that's probably just familiarity.

Thats my 2c :)

rephorm

On Mon, 14 Aug 2006 13:42:19 +0900 Carsten Haitzler (The Rasterman)
<raster@...> wrote:

> On Mon, 14 Aug 2006 00:03:07 -0400 Michael Jennings <mej@...>
> babbled:
>=20
> > On Monday, 14 August 2006, at 12:08:06 (+0900),
> > Carsten Haitzler wrote:
> >
> > It's not true.  SVN requires a lot more overhead (including Apache
> > with SVN and DAV modules), uses a BDB backend (you remember your
> > love of BDB, right?), and requires DOUBLE the amount of disk space
> > for a checkout.  Yes, I said double.  Furthermore, branching and
> > tagging
>=20
> checkouts from svn are just ridiculous. agreed. but its the server
> side i am asking about. as i said - i HEARD it is easier on the
> server - i am after details from those having been there, done that.
> yeah - bdb - oh yay. lets break format all the time.. ;)

I think the backend issue is one of those "they fixed that since"
issues I was talking about.  These days you get a choice of backends.

Yes, your local working copy takes up double the disk space, because it
keeps a pristine copy of what was checked out.  While this takes up
more space for the source code on a developers box, it has it's
advantages.  How many developers are that tight for space that they
can't spare some for one more copy of the source code?  It really
becomes a case of which particular trade off do you want?

On Mon, 14 Aug 2006 15:00:46 +1000 David Seikel <onefang@...> babbl=
ed:

> On Mon, 14 Aug 2006 13:42:19 +0900 Carsten Haitzler (The Rasterman)
> <raster@...> wrote:
>=20
> > On Mon, 14 Aug 2006 00:03:07 -0400 Michael Jennings <mej@...>
> > babbled:
> >=20
> > > On Monday, 14 August 2006, at 12:08:06 (+0900),
> > > Carsten Haitzler wrote:
> > >
> > > It's not true.  SVN requires a lot more overhead (including Apache
> > > with SVN and DAV modules), uses a BDB backend (you remember your
> > > love of BDB, right?), and requires DOUBLE the amount of disk space
> > > for a checkout.  Yes, I said double.  Furthermore, branching and
> > > tagging
> >=20
> > checkouts from svn are just ridiculous. agreed. but its the server
> > side i am asking about. as i said - i HEARD it is easier on the
> > server - i am after details from those having been there, done that.
> > yeah - bdb - oh yay. lets break format all the time.. ;)
>=20
> I think the backend issue is one of those "they fixed that since"
> issues I was talking about.  These days you get a choice of backends.
>=20
> Yes, your local working copy takes up double the disk space, because it
> keeps a pristine copy of what was checked out.  While this takes up
> more space for the source code on a developers box, it has it's
> advantages.  How many developers are that tight for space that they
> can't spare some for one more copy of the source code?  It really
> becomes a case of which particular trade off do you want?

its not the space - its the extra rsync times having to scan 2x as many
files. :)

but - if snc server-side is a big improvement for anon access - i will be
willing to forgo such performance in the name of better anon

--=20
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
=E8=A3=B8=E5=A5=BD=E5=A4=9A
Tokyo, Japan (=E6=9D=B1=E4=BA=AC =E6=97=A5=E6=9C=AC)

On Mon, 14 Aug 2006 14:11:00 +0900 Carsten Haitzler (The Rasterman)
<raster@...> wrote:

> On Mon, 14 Aug 2006 15:00:46 +1000 David Seikel <onefang@...>
> babbled:
>=20
> > On Mon, 14 Aug 2006 13:42:19 +0900 Carsten Haitzler (The Rasterman)
> > <raster@...> wrote:
> >=20
> > > On Mon, 14 Aug 2006 00:03:07 -0400 Michael Jennings
> > > <mej@...> babbled:
> > >=20
> > > > On Monday, 14 August 2006, at 12:08:06 (+0900),
> > > > Carsten Haitzler wrote:
> > > >
> > > > It's not true.  SVN requires a lot more overhead (including
> > > > Apache with SVN and DAV modules), uses a BDB backend (you
> > > > remember your love of BDB, right?), and requires DOUBLE the
> > > > amount of disk space for a checkout.  Yes, I said double.
> > > > Furthermore, branching and tagging
> > >=20
> > > checkouts from svn are just ridiculous. agreed. but its the server
> > > side i am asking about. as i said - i HEARD it is easier on the
> > > server - i am after details from those having been there, done
> > > that. yeah - bdb - oh yay. lets break format all the time.. ;)
> >=20
> > I think the backend issue is one of those "they fixed that since"
> > issues I was talking about.  These days you get a choice of
> > backends.
> >=20
> > Yes, your local working copy takes up double the disk space,
> > because it keeps a pristine copy of what was checked out.  While
> > this takes up more space for the source code on a developers box,
> > it has it's advantages.  How many developers are that tight for
> > space that they can't spare some for one more copy of the source
> > code?  It really becomes a case of which particular trade off do
> > you want?
>=20
> its not the space - its the extra rsync times having to scan 2x as
> many files. :)

That's not a server side issue.  It's only the developers working copy
that has this issue.

On Mon, 14 Aug 2006 15:18:40 +1000 David Seikel <onefang@...> babbl=
ed:

> On Mon, 14 Aug 2006 14:11:00 +0900 Carsten Haitzler (The Rasterman)
> <raster@...> wrote:
>=20
> > On Mon, 14 Aug 2006 15:00:46 +1000 David Seikel <onefang@...>
> > babbled:
> >=20
> > > On Mon, 14 Aug 2006 13:42:19 +0900 Carsten Haitzler (The Rasterman)
> > > <raster@...> wrote:
> > >=20
> > > > On Mon, 14 Aug 2006 00:03:07 -0400 Michael Jennings
> > > > <mej@...> babbled:
> > > >=20
> > > > > On Monday, 14 August 2006, at 12:08:06 (+0900),
> > > > > Carsten Haitzler wrote:
> > > > >
> > > > > It's not true.  SVN requires a lot more overhead (including
> > > > > Apache with SVN and DAV modules), uses a BDB backend (you
> > > > > remember your love of BDB, right?), and requires DOUBLE the
> > > > > amount of disk space for a checkout.  Yes, I said double.
> > > > > Furthermore, branching and tagging
> > > >=20
> > > > checkouts from svn are just ridiculous. agreed. but its the serve=
r
> > > > side i am asking about. as i said - i HEARD it is easier on the
> > > > server - i am after details from those having been there, done
> > > > that. yeah - bdb - oh yay. lets break format all the time.. ;)
> > >=20
> > > I think the backend issue is one of those "they fixed that since"
> > > issues I was talking about.  These days you get a choice of
> > > backends.
> > >=20
> > > Yes, your local working copy takes up double the disk space,
> > > because it keeps a pristine copy of what was checked out.  While
> > > this takes up more space for the source code on a developers box,
> > > it has it's advantages.  How many developers are that tight for
> > > space that they can't spare some for one more copy of the source
> > > code?  It really becomes a case of which particular trade off do
> > > you want?
> >=20
> > its not the space - its the extra rsync times having to scan 2x as
> > many files. :)
>=20
> That's not a server side issue.  It's only the developers working copy
> that has this issue.

i know! :)

--=20
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
=E8=A3=B8=E5=A5=BD=E5=A4=9A
Tokyo, Japan (=E6=9D=B1=E4=BA=AC =E6=97=A5=E6=9C=AC)

LS0tLS1CRUdJTiBQR1AgU0lHTkVEIE1FU1NBR0UtLS0tLQ0KSGFzaDogU0hBMQ0KDQpBbSBNb24s
IDE0IEF1ZyAyMDA2IDAwOjAzOjA3IC0wNDAwDQpzY2hyaWViIE1pY2hhZWwgSmVubmluZ3MgPG1l
akBrYWlueC5vcmc+Og0KDQo+IE9uIE1vbmRheSwgMTQgQXVndXN0IDIwMDYsIGF0IDEyOjA4OjA2
ICgrMDkwMCksDQo+IENhcnN0ZW4gSGFpdHpsZXIgd3JvdGU6DQo+IA0KPiA+IGFueXdheSAtIHdl
IGhhdmUgYmVlbiBsaXZpbmcgb24gY2Fvc2l0eSdzIGN2cyBmb3IgYSB3aGlsZSBub3cgLSBidXQN
Cj4gPiB3ZSBhcmUga2lsbGluZyBpdCAoc29ycnkga2FpbnghKQ0KPiA+DQo+ID4gc28gaXRzIHRp
bWUgdG8gZmluYWxseSBiaXRlIHRoZSBidWxsZXQgYW5kIGRyZWRnZSB1cCB0aGUgaXNzdWUgb2Yg
dXMNCj4gPiBuZWVkaW5nIHNlcnZlcnMgYWdhaW4uDQo+IA0KPiBXaGF0IHdlIHJlYWxseSBuZWVk
IGlzIGFub25jdnMgbWlycm9ycy4gIFdlIG5lZWQgc3lzdGVtcyB0aGF0IHdlIGNhbg0KPiBwb2lu
dCAiYW5vbmN2cy5lbmxpZ2h0ZW5tZW50Lm9yZyIgdG8uICBBIG5pY2Ugcm91bmQtcm9iaW4gRE5T
IHNob3VsZA0KPiBzb2x2ZSB0aGUgbG9hZCBwcm9ibGVtIHF1aXRlIG5pY2VseS4NCj4gDQo+ID4g
aGVyZSBpcyB3aGF0IGkgdGhpbmsgd2UgbmVlZDoNCj4gPiANCj4gPiAxLiBkZXZlbCAiY3ZzIiBz
ZXJ2ZXIgKyBmdXR1cmUgd2ViIHNlcnZlciAoZm9yIGRvd25sb2FkcyB0b28gb2Ygb2ZmaWNpYWwN
Cj4gPiB0YXJiYWxscyBldGMuKQ0KPiA+IDIuIGFuIGFub255bW91cyAiY3ZzIiBzZXJ2ZXIgYW5k
IHBvc3NpYmx5IHNlY29uZCBkb3dubG9hZCBtaXJyb3IuDQo+ID4gDQo+ID4gc28gMiBzeXN0ZW1z
IHJlYWxseS4NCj4gDQo+IFRoZSBkZXZlbG9wZXIgQ1ZTIHNlcnZlciBpcyBkb2luZyBhY2NlcHRh
Ymx5LCBpcyBpdCBub3Q/DQo+IA0KPiBXZSBhcmUgd29ya2luZyBvbiBvYnRhaW5pbmcgYW5vdGhl
ciBzZXJ2ZXIgZm9yIHRoZSBFIHByb2plY3QgdG8gaGF2ZQ0KPiBhcyBpdHMgdmVyeSBvd24uICBI
b3BlZnVsbHkgdGhhdCB3aWxsIHBhbiBvdXQuDQo+IA0KPiA+IGkgaGVhciB0aGF0IHN2biBpcyBz
aWduaWZpY2FudGx5IGxlc3MgbG9hZCBmb3IgYW5vbnltb3VzIGFjY2VzcyAtDQo+ID4gZXZlbiBk
ZXZlbG9wZXIgLSB3aG8gaGFzIGV4cGVyaWVuY2Ugd2l0aCB0aGlzIHNlcnZlci1zaWRlPyBjYW4g
eW91DQo+ID4gY29uZmlybSBvciBkZW55PyAgaSB3b3VsZCBjb25zaWRlciBhIHBvc3NpYmxlIG1v
dmUgdG8gc3ZuIGlmIHdlIGNhbg0KPiA+IGtlZXAgb3VyIGhpc3RvcnkgZnJvbSBjdnMuDQo+IA0K
PiBJdCdzIG5vdCB0cnVlLiAgU1ZOIHJlcXVpcmVzIGEgbG90IG1vcmUgb3ZlcmhlYWQgKGluY2x1
ZGluZyBBcGFjaGUNCj4gd2l0aCBTVk4gYW5kIERBViBtb2R1bGVzKSwgdXNlcyBhIEJEQiBiYWNr
ZW5kICh5b3UgcmVtZW1iZXIgeW91ciBsb3ZlDQo+IG9mIEJEQiwgcmlnaHQ/KSwgYW5kIHJlcXVp
cmVzIERPVUJMRSB0aGUgYW1vdW50IG9mIGRpc2sgc3BhY2UgZm9yIGENCj4gY2hlY2tvdXQuICBZ
ZXMsIEkgc2FpZCBkb3VibGUuICBGdXJ0aGVybW9yZSwgYnJhbmNoaW5nIGFuZCB0YWdnaW5nDQo+
IGRvbid0IHJlYWxseSBleGlzdCBmb3IgU1ZOIChpdCB1c2VzICJjb3BpZXMiIHdoaWNoLCB3aGls
ZSB0aGV5IG1heSBiZQ0KPiB6ZXJvIG92ZXJoZWFkIG9uIHRoZSBzZXJ2ZXIsIGFyZSBtdXJkZXIg
b24gdGhlIGNoZWNrb3V0KS4gIEFuZCBsYXN0IEkNCj4gY2hlY2tlZCwgeW91IGNvdWxkIG5vdCBr
ZWVwIHlvdXIgaGlzdG9yeS4NClRoYXQncyBub3QgdHJ1ZSB0aGF0IFNWTiBuZWVkcyBsb3QgbW9y
ZSBvdmVyaGVhZC4gQXQgbGVhc3QgaWYgeW91IGNvbXBpbGUgc3VidmVyc2lvbiBmcm9tIHNvdXJj
ZSB5b3UgY2FuIGRpc2FibGUgdW5uZWVkZWQgZmVhdHVyZXMgZS5nLiBhcGFjaGUgYW5kIGphdmEt
YmluZGluZ3Mgc3VwcG9ydC4NCk9uIGdlbnRvbyB5b3UgY2FuIGVuYWJsZS9kaXNhYmxlIGZvbGxv
d2luZyBmZWF0dXJlcyB3aGVuIGVtZXJnaW5nIHN1YnZlcnNpb246DQphcGFjaGUyLCBiYXNoLWNv
bXBsZXRpb24sDQpiZXJrZGIsIHN1YnZlcnNpb24gaGFzIGFsc28gYSBmbGF0IGZpbGUgZGIgc3Vw
cG9ydChmc2ZzIHdoaWNoIGlzIHN0YW5kYXJkIHdoZW4gY3JlYXRpbmcgYSBzdWJ2ZXJzaW9uIHJl
cG8pDQpqYXZhIChhZmFpayBvbmx5IGJpbmRpbmdzIHN1cHBvcnQpDQpubHMNCnBlcmwgKGFmYWlr
IG9ubHkgYmluZGluZ3Mgc3VwcG9ydCkNCnB5dGhvbiAoYWZhaWsgb25seSBiaW5kaW5ncyBzdXBw
b3J0KQ0KemxpYg0KZW1hY3MNCm5vd2ViZGF2IChubyB3ZWJkYXYgc3VwcG9ydCB2aWEgdGhlIG5l
b24gbGliKQ0KcnVieSAoYWZhaWsgb25seSBiaW5kaW5ncyBzdXBwb3J0KQ0KDQpzbyB5b3UgY2Fu
IHN0cmlwIGRvd24gdGhlIHJlcXVpcmVtZW50cyB0byB0aGUgbWluaW11bSBpZiB5b3Ugd2FudC4N
CkFwYWNoZSArIFdlYmRhdiBpcyBvbmx5IG5lZWRlZCBpZiB5b3Ugd2FudCB0aGUgYWNjZXNzIHRv
IHRoZSByZXBvcyB2aWEgd2ViZGF2Lg0KDQp0aGVyZSBhcmUgbm93IGF0IGxlYXN0IG9uZSBjdnMy
c3ZuIGNvbnZlcnNpb24gdG9vbCB3aXRoIHdoaWNoIHlvdSBjYW4ga2VlcCB5b3VyIGhpc3Rvcnkg
b2YgY3ZzLg0KDQpTdGVwaGFuDQoNCj4gDQo+IENWUyBpcyB0aGUgZGV2aWwgd2Uga25vdy4gIFRo
ZXJlJ3MgcmVhbGx5IG5vdGhpbmcgd2UgbmVlZCBpdCB0byBkbw0KPiB0aGF0IGl0IGRvZXNuJ3Qg
ZG8uICBJIHNlZSBubyBjb21wZWxsaW5nIHJlYXNvbiB0byBtb3ZlLg0KPiANCj4gTWljaGFlbA0K
PiANCg0KDQotIC0tIA0KRWluIFJpbmcsIHNpZSB6dSBrbmVjaHRlbiwgc2llIGFsbGUgenUgZmlu
ZGVuLA0KSW5zIER1bmtlbCB6dSB0cmVpYmVuIHVuZCBld2lnIHp1IGJpbmRlbg0KSW0gTGFuZGUg
TW9yZG9yLCB3byBkaWUgU2NoYXR0ZW4gZHJvaG4uDQotLS0tLUJFR0lOIFBHUCBTSUdOQVRVUkUt
LS0tLQ0KVmVyc2lvbjogR251UEcgdjEuNC41IChHTlUvTGludXgpDQoNCmlRQ1ZBd1VCUk9CT1di
R1J4Q25IYmF2VEFRTGpUd1FBcjEwT3Z1ZTJIN0J1T0llQWpnakNQV2ZmT2xPOE41RGMNCnhkQmJ1
MmpQNm1OaC8wNDlsT2Y0V1duZkk0cmZZakIrVW1zWVh3T0greHYwTktLalVLdGRTdy9hdmpNQXJX
QUINCkZ2TG5vazEyV0RSVmZDaTYvRFA4aG4ya1cwR2dmYTVUSXhmeklPK2IrR21MSFlWeGNzWGNM
MmxFVk5YMXNmS2sNCnNSTlpJeVJDYmprPQ0KPXpid0YNCi0tLS0tRU5EIFBHUCBTSUdOQVRVUkUt
LS0tLQ0K

On Mon, 14 Aug 2006 12:20:09 +0200 Stephan Wezel <s.wezel@...> babbled:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Am Mon, 14 Aug 2006 00:03:07 -0400
> schrieb Michael Jennings <mej@...>:
> 
> > On Monday, 14 August 2006, at 12:08:06 (+0900),
> > Carsten Haitzler wrote:
> > 
> > > anyway - we have been living on caosity's cvs for a while now - but
> > > we are killing it (sorry kainx!)
> > >
> > > so its time to finally bite the bullet and dredge up the issue of us
> > > needing servers again.
> > 
> > What we really need is anoncvs mirrors.  We need systems that we can
> > point "anoncvs.enlightenment.org" to.  A nice round-robin DNS should
> > solve the load problem quite nicely.
> > 
> > > here is what i think we need:
> > > 
> > > 1. devel "cvs" server + future web server (for downloads too of official
> > > tarballs etc.)
> > > 2. an anonymous "cvs" server and possibly second download mirror.
> > > 
> > > so 2 systems really.
> > 
> > The developer CVS server is doing acceptably, is it not?
> > 
> > We are working on obtaining another server for the E project to have
> > as its very own.  Hopefully that will pan out.
> > 
> > > i hear that svn is significantly less load for anonymous access -
> > > even developer - who has experience with this server-side? can you
> > > confirm or deny?  i would consider a possible move to svn if we can
> > > keep our history from cvs.
> > 
> > It's not true.  SVN requires a lot more overhead (including Apache
> > with SVN and DAV modules), uses a BDB backend (you remember your love
> > of BDB, right?), and requires DOUBLE the amount of disk space for a
> > checkout.  Yes, I said double.  Furthermore, branching and tagging
> > don't really exist for SVN (it uses "copies" which, while they may be
> > zero overhead on the server, are murder on the checkout).  And last I
> > checked, you could not keep your history.
> That's not true that SVN needs lot more overhead. At least if you compile
> subversion from source you can disable unneeded features e.g. apache and
> java-bindings support. On gentoo you can enable/disable following features
> when emerging subversion: apache2, bash-completion, berkdb, subversion has
> also a flat file db support(fsfs which is standard when creating a subversion
> repo) java (afaik only bindings support) nls
> perl (afaik only bindings support)
> python (afaik only bindings support)
> zlib
> emacs
> nowebdav (no webdav support via the neon lib)
> ruby (afaik only bindings support)
> 
> so you can strip down the requirements to the minimum if you want.
> Apache + Webdav is only needed if you want the access to the repos via webdav.
> 
> there are now at least one cvs2svn conversion tool with which you can keep
> your history of cvs.

ok- but what about runtime performance? doe svn use "50%" of the system load
(ram/cpu/whatever) compare to cvs? 20% 150%? 200%? what? a rough ballpark thing
is what i'd like to know from people with experience of going form cvs to svn
or svn to cvs - will it make our "anon cvs serves" handle more users with the
same src being shipped given the same resources? if more - about how much more?

> Stephan
> 
> > 
> > CVS is the devil we know.  There's really nothing we need it to do
> > that it doesn't do.  I see no compelling reason to move.
> > 
> > Michael
> > 
> 
> 
> - -- 
> Ein Ring, sie zu knechten, sie alle zu finden,
> Ins Dunkel zu treiben und ewig zu binden
> Im Lande Mordor, wo die Schatten drohn.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (GNU/Linux)
> 
> iQCVAwUBROBOWbGRxCnHbavTAQLjTwQAr10Ovue2H7BuOIeAjgjCPWffOlO8N5Dc
> xdBbu2jP6mNh/049lOf4WWnfI4rfYjB+UmsYXwOH+xv0NKKjUKtdSw/avjMArWAB
> FvLnok12WDRVfCi6/DP8hn2kW0Ggfa5TIxfzIO+b+GmLHYVxcsXcL2lEVNX1sfKk
> sRNZIyRCbjk=
> =zbwF
> -----END PGP SIGNATURE-----
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

Carsten Haitzler (The Rasterman) wrote:
> It has come to my attention that.... yet again we are killing systems.
>
> yes - we are becoming a burden on yet more cvs servers. we are monsters! :(
>
> anyway - we have been living on caosity's cvs for a while now - but we are
> killing it (sorry kainx!)
>
> so its time to finally bite the bullet and dredge up the issue of us needing
> servers again.
>
> here is what i think we need:
>
> 1. devel "cvs" server + future web server (for downloads too of official
> tarballs etc.)
> 2. an anonymous "cvs" server and possibly second download mirror.
>
> so 2 systems really.
>
> i hear that svn is significantly less load for anonymous access - even
> developer - who has experience with this server-side? can you confirm or deny?
> i would consider a possible move to svn if we can keep our history from cvs.
>
> so - let the flames begin.
>
>   
Now I know this is probably not going to sound right but how about a git 
repository?.

I'll give you some information about in case you're not up to speed with 
it and/or what it can do. If you are then sorry for wasting your time.

It is a distributed revision control system but it has a whole bunch of 
very cool features. Here are a few of them, stolen from git's homepage:
- supports rapid and convenient branching and merging
- repositories can be easily accessed via the efficient Git protocol 
(optionally wrapped in ssh)
- they can also be accessed simply using HTTP - you can publish your 
repository anywhere without _any_ special webserver configuration 
required. (that should make it very easy to mirror).
- very fast and scales well even when working with large projects and 
long histories
- commonly an order of magnitude faster than most other revision control 
systems, and several orders of magnitude faster on some operations

You can find the said webpage on http://git.or.cz/
There's a section called Git for CVS users you might want to check out.

I've actually worked with if for a couple of small projects of mine. 
Anyway here are a few extra things to note.

Git in itself is rather complicated to use. It is designed to be quite 
low-level. However, there are programs like cogito, that work on top of 
the low-level git commands. Now I've used subversion for a couple of 
years, and I've used CVS for a very short time and if you've used either 
one of them, cogito is a snap.

There are conversion tools available, but again I haven't tested them 
just yet. But if you're interested I can give it a spin and tell you 
guys how everything goes.

Here's a page to check out CVS importing: 
http://www.kernel.org/pub/software/scm/git/docs/cvs-migration.html

There's a tool called gitweb, which is a web interface for Git 
repositories. For an example, you might want to try the kernel's git 
page: http://www.kernel.org/git/

I for one, think it would be pretty cool if enlightenment was to be 
developed using cogito. It allows for very quick branching/merging, it's 
very fast and quite easy to pick up as well. I haven't used it for 
anything massive though but you could always just set up a read-only 
repository and ask people to download from it. All you have to do is 
convert your CVS tree into a git tree and copy that on one of your 
webservers.

That's my (rather long) 2 cents anyway.

Cheers,
Eugen.

The reason I wanted to go the CVS way was so that we could avoid the
importing hell you generally go through when switching systems (both
on the server side and the client side).
On that side I found DCVS: http://www.elego-software-solutions.com/dcvs/

On the other hand, I've been staring at that git website and git
sounds a lot like SVN, so I'm all for it.

--Sthitha

On 8/14/06, Eugen Minciu <minciue@...> wrote:
> > I dont know much about Git cogito, but if there's a distributed system
> > that can directly work with CVS (as in sync to cvs each time rather
> > than importing), we could set up anoncvs as distributed and leave
> > dev-cvs as it is.
>
> I don't think you can do that with git. And I don't think it's better then actually using git itself. If you haven't used git/cogito you could take a few minutes and check out the links in my last email.
>
> I'll try to look for some articles/info about distributed development vs centralized development later. Git/cogito is used by the Linux kernel (and lately the Gnome project as well, I think). So it's proven to be reliable and fast (it was initially created by Linus himself as a replacement for the commerical BitKeeper).
>
> Cheers.
>


On 8/14/06, Sthithaprajna Garapaty <iamsthitha@...> wrote:
> > Hey,
> > I think that's a great idea for anoncvs.
> > Instead of having one central server  that might get choked anyway,
> > maybe we should try a distributed system so everyone can pitch in and
> > give some bandwidth/cpu.
> > I dont know much about Git cogito, but if there's a distributed system
> > that can directly work with CVS (as in sync to cvs each time rather
> > than importing), we could set up anoncvs as distributed and leave
> > dev-cvs as it is.
> >
> > On the SVN debate, I dunno anything about performance, but I would
> > love dev-cvs to go svn :) (And there are tools to get us there without
> > losing history)
> >
> > --Sthitha
> >

Eugen Minciu [2006-08-14 15:29]:
> Carsten Haitzler (The Rasterman) wrote:
> > It has come to my attention that.... yet again we are killing systems.
> >
> > yes - we are becoming a burden on yet more cvs servers. we are monsters=
! :(
> >
> > anyway - we have been living on caosity's cvs for a while now - but we =
are
> > killing it (sorry kainx!)
> >
> > so its time to finally bite the bullet and dredge up the issue of us ne=
eding
> > servers again.
> >
> > here is what i think we need:
> >
> > 1. devel "cvs" server + future web server (for downloads too of official
> > tarballs etc.)
> > 2. an anonymous "cvs" server and possibly second download mirror.
> >
> > so 2 systems really.
> >
> > i hear that svn is significantly less load for anonymous access - even
> > developer - who has experience with this server-side? can you confirm o=
r deny?
> > i would consider a possible move to svn if we can keep our history from=
 cvs.
> >
> > so - let the flames begin.
> >
> >  =20
> Now I know this is probably not going to sound right but how about a git=
=20
> repository?.

git kicks Subversion's ass. At first, the decentralized nature of git
might feel weird, but you'll get used to it, and love it (I've been
using git since the day when Bitkeeper ended it's free-as-in-beer
license thing, I think it was 2005-07-01).

Disk space usage on the developer's box is considerably worse than CVS
of course. The read-only git daemon (the one you'd use for
anongit.e.org) is light on ressources I believe. Developer access would
be done via ssh.

cogito isn't needed for a sane git experience any more (it was, in the
early days).

Regards,
Tilman

--=20
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

> It's not true.  SVN requires a lot more overhead (including Apache
> with SVN and DAV modules),

They aren't required, they just make things easier. You can use a
standalone svn daemon, or AFAIK have a client side client and server
side client who talk over SSH (similar to rsync's two-client mode?).
I've only ever used Apache + mod_svn myself though.

Even if you do go the apache + mod_svn route, since when was apache
known for being bloated and slow? (Sure it's slower than things like
lighttpd, but even with all the bloat turned on the apache overhead is
still tiny compared to the processor time spent getting actual work
done)


> uses a BDB backend (you remember your love of BDB, right?)

Since a few years ago, it's been able to use a filesystem based
database, which I've never had problems with (BDB kept me away from
SVN for a while too, since it kept giving me headaches...)


> and requires DOUBLE the amount of disk space for a
> checkout.  Yes, I said double.

Having a local copy of the unmodified source makes things like taking
diffs or reverting changes *much* faster, and zero load on the server.

Disk space shouldn't be much of an issue with today's drives; Raster's
point about running rsync on his local checkout will indeed use twice
the bandwidth if it's done the naive way, but I'm sure there are some
optimisations that could be done. (I'm interested in knowing why
someone would want to rsync their local copy anyway, rather than
checking into the server on one box and checking out elsewhere)


> Furthermore, branching and tagging
> don't really exist for SVN (it uses "copies" which, while they may be
> zero overhead on the server, are murder on the checkout).

I've found tagging much cleaner, and you don't need to check anything
out at all:

svn cp http://server/project/trunk http://server/project/tags/0.4.2

(if you want to retroactively tag an old version of the trunk, just
add "-r <revision number>" before the trunk URL). Practically zero
overhead on the server (all it stores in the copy is a reference to the
trunk filename and revision), and zero overhead on the client.

Branching I've only experimented with, but don't use regularly enough
to say whether it's better or worse than CVS's.


> And last I checked, you could not keep your history.

The cvs2svn script seems to have worked fine for other large, active
projects (mplayer, gaim, and inkscape are the ones in my src folder)


Unfortunately for the topic at hand, the only thing I can't say for
certain is "SVN is better at dealing with server-killing loads caused
by vast numbers of anon checkouts". Using my otherwise unused 200MHz
test server, browsing the HTTP interface is as fast as browsing plain
text files, but I don't know if it'll stay that fast with thousands of
users at once...

    -- Shish

Strangely enough, another large, old open source project I am involved
in is having exactly the same conversation.  One of them came up with
this -

http://wiki.metalforge.net/doku.php/user:rednaxela:scmtable

It may be helpful.

On Tue, 15 Aug 2006 09:01:06 +1000
David Seikel <onefang@...> wrote:

> Strangely enough, another large, old open source project I am involved
> in is having exactly the same conversation.  One of them came up with
> this -
> 
> http://wiki.metalforge.net/doku.php/user:rednaxela:scmtable
> 
> It may be helpful.
> 

Git isn't there. Now I hope I understand all of the points in that table, and I also hope I understand git well enough, but here's a listing for git (based on my knowledge).

- Protocol: dedicated / ssh / http-based/ email (at least partially).
- Learning curve from CVS: medium (imho)
- Network based access: No, it uses smoke signals. Doh!
- Multiplatform: Yes.
- Access control lists: Yes, with SSH
- Read-only access to everyone: Yes
- Supported by Sourceforge & co.: Yes, with SSH (though I don't think it's that relevant here).
- Receive email notification upon a commit: Almost, you have to do a chmod +x on a script to enable it to be run. ( see http://www.kernel.org/pub/software/scm/git/docs/hooks.txt ) 
- Readily available/easily installable software: Yes
- Tracking of when merges are done: Yes
- Good branch handling: Yes
- Efficient use of resources: From what I can tell it's impressive, the git developers say it's around an order of magnitude faster, sometimes more. It was designed with a primary focus on efficiency so I tend to think it's pretty good.
- Global revisioning: Yes
- support for symbolic tagging within one repository: Yes
- Atomic checkins: Yes (I think)
- Maximum ability to do SCCS operations without access to repository: Yes (and it's a great thing to have imho)
- good binary file handling: From what I've read yes, I believe GZip compression is used on binary data.
- Ability to do local branches: Yes
- Rename support: Yes.
- Web view of repository: Yes

So this is _my_ knowledge of how git works. If it's wrong sorry for any confusions this may cause. Anyway, I'll stop zealoting for git from now on, I just think it's really cool. 

But even if you guys won't go with it, I would still suggest using Subversion which I believe to be better then CVS.

Cheers,
Eugen.

On Mon, 14 Aug 2006 23:51:19 +0100 Shish <shish@...> babb=
led:

> > It's not true.  SVN requires a lot more overhead (including Apache
> > with SVN and DAV modules),
>=20
> They aren't required, they just make things easier. You can use a
> standalone svn daemon, or AFAIK have a client side client and server
> side client who talk over SSH (similar to rsync's two-client mode?).
> I've only ever used Apache + mod_svn myself though.
>=20
> Even if you do go the apache + mod_svn route, since when was apache
> known for being bloated and slow? (Sure it's slower than things like
> lighttpd, but even with all the bloat turned on the apache overhead is
> still tiny compared to the processor time spent getting actual work
> done)
>=20
>=20
> > uses a BDB backend (you remember your love of BDB, right?)
>=20
> Since a few years ago, it's been able to use a filesystem based
> database, which I've never had problems with (BDB kept me away from
> SVN for a while too, since it kept giving me headaches...)
>=20
>=20
> > and requires DOUBLE the amount of disk space for a
> > checkout.  Yes, I said double.
>=20
> Having a local copy of the unmodified source makes things like taking
> diffs or reverting changes *much* faster, and zero load on the server.

load on the sever here isnt important - u devs cause very little/no load =
- its
anon users that kill things. :) but yes - it is a good point... but. doub=
le the
files.

> Disk space shouldn't be much of an issue with today's drives; Raster's
> point about running rsync on his local checkout will indeed use twice
> the bandwidth if it's done the naive way, but I'm sure there are some
> optimisations that could be done. (I'm interested in knowing why
> someone would want to rsync their local copy anyway, rather than
> checking into the server on one box and checking out elsewhere)

ok- double the bandwidth and double the scan time. rsync has to scan a lo=
t
more files and in fact the scan of my files takes more time than the sync=
. yes
- i disabled checksumming because that just makes it intolerable.

why rsync? do you have more than computer? how do you copy your email, co=
de,
files, porn, music etc. to a new system? if it s laptop and doesnt life o=
n
your network all the time for example? rsync.

i rsync msot of my dot-files and homedir between desktop, laptop, work et=
c.
etc. etc. - this way all my files, work, state, notes, wallpapers, config=
,
email, music, etc. etc. - follow me form one system to another - seamless=
ly
(well with a single rsync) and i am not tethered to a lan and nfs - i can=
 just
continue where i left on on my laptop on a plane, train, in the airport, =
on a
park bench, at work, and then back home to my desktop, over to the couch =
in the
living room - whereever i go - onto whatever machine i may use, my "digit=
al
life" follows me. i can just keep doing what i was doing. ESPECIALLY impo=
rtant
is that my source code follows me - thus my cvs checkouts. this acts also=
 as a
backup mechanism as i also rsync to a central sever 2 or more times a day=
 - and
a copy of my data exists on several systems - if i lose a disk - not a bi=
g
problem. i have several backups - several of them even "offsite" :)

if you don't have at least a desktop and a laptop and have never tried wo=
rking
offline - you wont understand - but rsync is the ultimate tool for this k=
ind of
semi connected/disconnected kind of independent working ability.

>=20
> > Furthermore, branching and tagging
> > don't really exist for SVN (it uses "copies" which, while they may be
> > zero overhead on the server, are murder on the checkout).
>=20
> I've found tagging much cleaner, and you don't need to check anything
> out at all:
>=20
> svn cp http://server/project/trunk http://server/project/tags/0.4.2
>=20
> (if you want to retroactively tag an old version of the trunk, just
> add "-r <revision number>" before the trunk URL). Practically zero
> overhead on the server (all it stores in the copy is a reference to the
> trunk filename and revision), and zero overhead on the client.
>=20
> Branching I've only experimented with, but don't use regularly enough
> to say whether it's better or worse than CVS's.
>=20
>=20
> > And last I checked, you could not keep your history.
>=20
> The cvs2svn script seems to have worked fine for other large, active
> projects (mplayer, gaim, and inkscape are the ones in my src folder)
>=20
>=20
> Unfortunately for the topic at hand, the only thing I can't say for
> certain is "SVN is better at dealing with server-killing loads caused
> by vast numbers of anon checkouts". Using my otherwise unused 200MHz
> test server, browsing the HTTP interface is as fast as browsing plain
> text files, but I don't know if it'll stay that fast with thousands of
> users at once...
>=20
>     -- Shish
>=20
> -----------------------------------------------------------------------=
--
> Using Tomcat but need to do more? Need to support web services, securit=
y?
> Get stuff done quickly with pre-integrated technology to make your job =
easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geron=
imo
> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=
=3D121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
>=20


--=20
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
=E8=A3=B8=E5=A5=BD=E5=A4=9A
Tokyo, Japan (=E6=9D=B1=E4=BA=AC =E6=97=A5=E6=9C=AC)

On Tue, 15 Aug 2006 09:01:06 +1000 David Seikel <onefang@...> babbl=
ed:

> Strangely enough, another large, old open source project I am involved
> in is having exactly the same conversation.  One of them came up with
> this -
>=20
> http://wiki.metalforge.net/doku.php/user:rednaxela:scmtable
>=20
> It may be helpful.

mercurial looks god there - but svn is the easiest learning curve :)


--=20
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
=E8=A3=B8=E5=A5=BD=E5=A4=9A
Tokyo, Japan (=E6=9D=B1=E4=BA=AC =E6=97=A5=E6=9C=AC)

Carsten Haitzler (The Rasterman) wrote:
> On Tue, 15 Aug 2006 09:01:06 +1000 David Seikel <onefang@...> babbled:
> 
>> Strangely enough, another large, old open source project I am involved
>> in is having exactly the same conversation.  One of them came up with
>> this -
>>
>> http://wiki.metalforge.net/doku.php/user:rednaxela:scmtable
>>
>> It may be helpful.
> 
> mercurial looks god there - but svn is the easiest learning curve :)

I may be little off topic. But as a novice in sys admin. I found lot
more easier to config svn then cvs. I try to test BDB filesystem and
also FSFS and found no trouble dumping from BDB to FSFS, managing
concurrent branches against trunk and taging branches and trunk to a
given TAG so that you and up with several build possible out of
checkout. Cause of the little overhead I didn't try all those with CVS
so my comment may not help. For as little I have read, I can't even know
where all this going. Is there some people wanting to migrate to svn.

For such I may only say : why broking something that working? If you do
found something problematic, find the solution, and solution don't mean
replacement but neither discard it.

My 2 cent,
LMA1980

On 8/14/06, The Rasterman Carsten Haitzler <raster@...> wrote:
> On Tue, 15 Aug 2006 09:01:06 +1000 David Seikel <onefang@...> babbled:
>
> > Strangely enough, another large, old open source project I am involved
> > in is having exactly the same conversation.  One of them came up with
> > this -
> >
> > http://wiki.metalforge.net/doku.php/user:rednaxela:scmtable
> >
> > It may be helpful.
>
> mercurial looks god there - but svn is the easiest learning curve :)
>

I feel compelled to mention monotone as I use it for a few projects. I
personally like it quite a bit but it's still being fairly heavily
changed (although they do a real good job of making the upgrade path
easy).

http://venge.net/monotone

It's also fully distributed, which may or may not be a good thing
depending on what you want. (Each user has their own copy of the
entire history, it is not possible to just have a checkout.)

-- 
Justin Patrin

On Monday, 14 August 2006, at 13:42:19 (+0900),
Carsten Haitzler wrote:

> yeah - though that might be harder than we think. the other problem
> here is consistency - we want a user using the 1 server once they
> started as if anon servers don't update at the same time - they will
> have problems with consistency.

Not necessarily.  A nice lengthy TTL on the records (say, 12 hours?)
will make sure they get the same mirror, at least throughout any given
day.  The problems we'd face really wouldn't be any different from any
other set of mirrors.

> having a faster/lighter/easier access method (svn? who has something
> to say? i prefer the "devil you know" but since this problem keeps
> coming up... time to bit the bullet?)

What it's telling us is that any single server is going to have its
ass handed to it by E anoncvs users.  I don't think replacing one with
another is the solution, SCM system notwithstanding.

> this was kind of part of what i was getting to - to make sure that
> we dont tread on your feet any more - move to our own box - and yes
> - anoncvs mirrors will be good - but maybe using svn in the end will
> make this easier.  i am broaching the topic to start just this
> discussion. i am not set on a path at all - but if we are causing
> pain - we need to address it.

Honestly, it hasn't affected us really at all.  We're more concerned
about the users....

> checkouts from svn are just ridiculous. agreed. but its the server
> side i am asking about. as i said - i HEARD it is easier on the
> server - i am after details from those having been there, done that.

I have not run an anon SVN server, so I can't speak to that point.  I
can only say that it makes no sense that a standalone, dedicated
server like cvs would have less overhead than Apache, which we all
know is a web server, a shared filesystem server, an embedded language
engine, a proxy server, an authentication and authorization system,
...need I go on?  :-)

> yeah - bdb - oh yay. lets break format all the time.. ;)

If you think BDB was bad, Subversion is just as bad.  Upgrading from
one version of svn-server to another can often involve a change of
repository format, in which case the ONLY migration path is the
following:

1.  Use the old version of the server to create a "dump" of the
    repository.  You must remember to do this BEFORE upgrading, or
    pray that you have a static copy around.  I speak from experience
    when I say this gets nasty.  To say nothing of what happens when
    the dump fails....
2.  Upgrade SVN.
3.  Load the repository from the dump file, creating each revision
    from scratch based on the file data.  This takes an amount of time
    directly proportional to the number of revisions and the
    complexity of each changeset.  Again, you just have to hope it
    doesn't fail.

But that's a bit off-topic....  I don't have actual data for you, but
I can't imagine how CVS would be harder on the server than SVN....

> agreed - unless svn is less overhead. that is the question. an svn
> anon server serving 1000 users - and then an cvs anon server serving
> the same src and the same people - which one consumes less disk
> io/cpu/memory? thats what i want to know - and if it is less - is it
> significantly less? or just like 5% (not worth worrying about)

I was speaking to Inc the other night, and he (more or less) offered
an SVN mirror if we were interested.  I said we would be interested in
any mirror that could sync off the CVS server and help share some of
the burden, SVN included.  If he's still willing to follow through on
that, it might be a good opportunity to compare the two.



On Sunday, 13 August 2006, at 23:49:06 (-0500),
brian.mattern@... wrote:

> There are tools to move from cvs to
> svn. E.g. http://cvs2svn.tigris.org/ I've never used one, so I can't
> vouch for them, but it seems like it would be a fairly
> straightforward operation.

"Seems" and "is" are entirely different.  You'd think doing a
repository dump would be a "fairly straightforward operation" too, but
I've had that choke on me too.

> The main thing that SVN has that won me over is the ability to do
> diffs without hitting the server. I generally check diffs a LOT
> before checking in. With CVS this is a slow pita. I also find svn
> log more useful than cvs log. But that's probably just familiarity.

It's quite easy to check out a parallel tree or do an export and
generate local diffs from that.  With CVS, you have the option of
equipping yourself to do local diffs or using the server.  With SVN,
you have no choice at all.



On Monday, 14 August 2006, at 12:20:09 (+0200),
Stephan Wezel wrote:

> That's not true that SVN needs lot more overhead. At least if you
> compile subversion from source you can disable unneeded features

This is an interesting point.  Has anyone ever tried building a
version of CVS which could do anoncvs but nothing else?  No commits?
That might have an interesting impact on the overhead question.



On Monday, 14 August 2006, at 15:29:28 (+0300),
Eugen Minciu wrote:

> Now I know this is probably not going to sound right but how about a
> git repository?.

git is quite compelling, actually.  Subversion's chief "advantage" is
also its Achilles heel:  Just like CVS, it lacks fundamental
capabilities of distributed development.  git is basically a GPL
implementation of the key design concepts of BitKeeper.  I think it
has a lot more to offer over CVS than SVN.



On Monday, 14 August 2006, at 10:30:51 (-0500),
Sthithaprajna Garapaty wrote:

> The reason I wanted to go the CVS way was so that we could avoid the
> importing hell you generally go through when switching systems (both
> on the server side and the client side).

The only reason to switch that we're looking at right now is to solve
the load issue.  I think raster's made it pretty clear that the focus
of this discussion is how to best utilize server resources, not what
everyone's favorite SCM system is.  I'm definitely going to look at
DCVS, but I don't think it's applicable to the topic at hand.



On Monday, 14 August 2006, at 23:51:19 (+0100),
Shish wrote:

> I've only ever used Apache + mod_svn myself though.

Exactly.  Have you met anyone who uses SVN without Apache?  Neither
have I.

> Even if you do go the apache + mod_svn route, since when was apache
> known for being bloated and slow?

Since about 1.2, I think.

> I've found tagging much cleaner, and you don't need to check anything
> out at all:
> 
> svn cp http://server/project/trunk http://server/project/tags/0.4.2

FUD.  You can tag CVS without having a checkout.  (Try cvs -H rtag.)
And SVN doesn't even HAVE tagging.  It has copying, which contrary to
popular (SVN developer) belief, is NOT the same thing.  A (non-branch)
tag is a symbolic name assigned to a particular state of the
repository (i.e., a changeset).  It is not a copy.

But again, this is all moot.  I'm sure there are numerous people who
would love to argue with me till doomsday about how great and
wonderful Subversion is; many others have already tried.  It does
absolutely NOTHING to address the question at hand:  Is anonymous SVN
easier on the server than anonymous cvs?

> Unfortunately for the topic at hand, the only thing I can't say for
> certain is "SVN is better at dealing with server-killing loads caused
> by vast numbers of anon checkouts".

While this comment would certainly be relevant, your subsequent
comments lead me to believe you don't have the evidence to back it
up.  We're talking about significant numbers of checkouts (i.e.,
apples), and you're talking about web browsing (oranges).

Michael

-- 
Michael Jennings (a.k.a. KainX)  http://www.kainx.org/  <mej@...>
n + 1, Inc., http://www.nplus1.net/       Author, Eterm (www.eterm.org)
-----------------------------------------------------------------------
 "The breakup was mutual, but it was more mutual on my part."
                                                        -- Beth O'Hara

On 8/14/06, Michael Jennings <e-devel@...> wrote:
> On Monday, 14 August 2006, at 13:42:19 (+0900),
> Carsten Haitzler wrote:

<snip>

> > checkouts from svn are just ridiculous. agreed. but its the server
> > side i am asking about. as i said - i HEARD it is easier on the
> > server - i am after details from those having been there, done that.
>
> I have not run an anon SVN server, so I can't speak to that point.  I
> can only say that it makes no sense that a standalone, dedicated
> server like cvs would have less overhead than Apache, which we all
> know is a web server, a shared filesystem server, an embedded language
> engine, a proxy server, an authentication and authorization system,
> ...need I go on?  :-)
>
> > yeah - bdb - oh yay. lets break format all the time.. ;)
>
> If you think BDB was bad, Subversion is just as bad.  Upgrading from
> one version of svn-server to another can often involve a change of
> repository format, in which case the ONLY migration path is the
> following:

as others have pointed out, you do not have to use BDB, and i would go
so far as to say FSFS is the only practical way to use subversion.
However, FSFS has the distinct disadvantage of being very slow on
initial checkout (svn folks themselves say about 2x as slow as BDB),
since it only stores deltas, and must essentially read the entire
server repository to reply. As much as I like svn, i think this fact
alone i think eliminates SVN as a choice for anonymous access.

<snip>

> On Monday, 14 August 2006, at 23:51:19 (+0100),
> Shish wrote:
>
> > I've only ever used Apache + mod_svn myself though.
>
> Exactly.  Have you met anyone who uses SVN without Apache?  Neither
> have I.

i have "used" a repository that uses svnserve, but only administered
any using webdav. From the user's perspective, it is indistinguishable
except that the repo url begins with "svn://" instead of "http://"

> > Even if you do go the apache + mod_svn route, since when was apache
> > known for being bloated and slow?
>
> Since about 1.2, I think.
>
> > I've found tagging much cleaner, and you don't need to check anything
> > out at all:
> >
> > svn cp http://server/project/trunk http://server/project/tags/0.4.2
>
> FUD.  You can tag CVS without having a checkout.  (Try cvs -H rtag.)
> And SVN doesn't even HAVE tagging.  It has copying, which contrary to
> popular (SVN developer) belief, is NOT the same thing.  A (non-branch)
> tag is a symbolic name assigned to a particular state of the
> repository (i.e., a changeset).  It is not a copy.

speaking of FUD...

how is a zero-copy copy different from a tag? yes, you have to name it
correctly to avoid confusion, and you have to be careful not to modify
it (although you can set also props to prevent that, and the "tags"
directory is a good hint), but for all intents and purposes, it is
equivalent to a tag. svn log will even give you a full history of
files in the branch/tag beyond the branch/tag point.
<http://svnbook.red-bean.com/nightly/en/svn.branchmerge.tags.html>

also, you do not need to do a full check out after creating a
tag/branch/copy. you can use svn switch. to use the above example:

svn cp http://server/project/trunk http://server/project/tags/0.4.2
cd /path/to/your/checkout/of/trunk
svn switch http://server/project/tags/0.4.2

svn switch only contacts the server to get any changes since your most
recent update.
<http://svnbook.red-bean.com/nightly/en/svn.branchmerge.switchwc.html>

> But again, this is all moot.  I'm sure there are numerous people who
> would love to argue with me till doomsday about how great and
> wonderful Subversion is; many others have already tried.  It does
> absolutely NOTHING to address the question at hand:  Is anonymous SVN
> easier on the server than anonymous cvs?

i will stop trying to convince you then ;) . and my answer to the
question at hand is: most probably no. i would guess that git would be
a better solution, but i have never used it.

>
> > Unfortunately for the topic at hand, the only thing I can't say for
> > certain is "SVN is better at dealing with server-killing loads caused
> > by vast numbers of anon checkouts".
>
> While this comment would certainly be relevant, your subsequent
> comments lead me to believe you don't have the evidence to back it
> up.  We're talking about significant numbers of checkouts (i.e.,
> apples), and you're talking about web browsing (oranges).

on the other hand, i would say there is sufficient (albeit
theoretical) evidence that SVN would not be a good performer for
high-load anonymous access.

d#

>
> Michael
>

-- 
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

On 8/14/06, The Rasterman Carsten Haitzler <raster@...>
>
> 1. devel "cvs" server + future web server (for downloads too of official
> tarballs etc.)
> 2. an anonymous "cvs" server and possibly second download mirror.
>
> so 2 systems really.
>
> i hear that svn is significantly less load for anonymous access - even
> developer - who has experience with this server-side? can you confirm or deny?
> i would consider a possible move to svn if we can keep our history from cvs.
>
> so - let the flames begin.
>

What about git? You can easily move cvs history over to git.

-- 
things i hate about my linux pc:

1. it takes more than a second to boot up
2. keeps asking about filenames and directories
3. does not remember what i was working on yesterday
4. does not remember all the changes i have ever made
5.cannot figure out necessary settings by itself

I will try to benchmark cvs, subversion and git today. If this doesn't take me too long I might throw in Mercurial as well.

I'll set up a (insert one of the above SCMs) server on my desktop, with the repository on a tmpfs, with the enlightenment repository, and I'll grab it 100 times with my laptop on tmpfs as well, or maybe even more. 

That way I think I have a pretty good way of assessing exactly how well the server handles that kind of a load. I'll use tmpfs so that, my hdds aren't a bottleneck. The network works quite well, I hit 9MB/s on a regular basis so that should not be a problem either. 

There are basically 3 things I will be checking, and I'll average the results:
1) How much does it take to checkout the repository?
2) What is the average load on the machine during this time, in terms of CPU and Memory
3) How much network bandwidth is being used in the process.

I'll post the exact details when I'm done but I think it's better that we see some hard numbers and not just try to convince everyone to use our own favorite. (Though in the process I hope I'll be able to convince you to use git).

I'll post the exact way I did this so that people may replicate what I've done if anyone desires so.

Cheers,
Eugen.

On Tue, 15 Aug 2006 00:41:34 -0400 Michael Jennings <e-devel@...>
wrote:

> Exactly.  Have you met anyone who uses SVN without Apache?  Neither
> have I.

I'm using SVN without Apache.  Your anti-SVN message is starting to
sound like it's coming from someone that hasn't actually run a SVN
server recently.  It's like the "E is bloated and slow" I still get
very now and then, "sure E14 was, recent versions are anything but".

> FUD.  You can tag CVS without having a checkout.  (Try cvs -H rtag.)
> And SVN doesn't even HAVE tagging.  It has copying, which contrary to
> popular (SVN developer) belief, is NOT the same thing.  A (non-branch)
> tag is a symbolic name assigned to a particular state of the
> repository (i.e., a changeset).  It is not a copy.

Tagging and branching, when was the last time either occurred to E?
=20
> But again, this is all moot.  I'm sure there are numerous people who
> would love to argue with me till doomsday about how great and
> wonderful Subversion is; many others have already tried.  It does
> absolutely NOTHING to address the question at hand:  Is anonymous SVN
> easier on the server than anonymous cvs?

I'm beginning to think that no one has any recent relevant experience
to answer that question.

And what do you think about opening some mirrors ?

I know that raster enjoys knowing the numbers of people using the cvs, 
etc...
Maybe, mirrors could give statistics and works togethers...

An other possibility is to make the official source's tree under cvs, 
and make a git mirror, an other under svn etc...


I could host one. Just said me if you are interessted.


Stepane 'rookmoot' Bauland

On Tue, Aug 15, 2006 at 11:41:09AM +0300, Eugen Minciu wrote:
> I will try to benchmark cvs, subversion and git today. If this doesn't take me too long I might throw in Mercurial as well.

Thanks! Be sure to post the results on the web somewhere, as I'm sure
other large projects may be interested in this sort of data.

rephorm

I was just about to say the same thing; the only thing I lack is a
suitably huge CVS repository to start with -- is there any way for me
to make a local mirror of E's whole repository? Google fails me the
specific procedure...


> tmpfs,

In real life disks will be involved, it'd probably be good to take
them into account. (I'd recommend doing both, to see what effect the
disks have)


> I'll grab it 100 times with my laptop

I was thinking a more appropriate benchmark would be to see how many
parallel checkouts can be done at once before performance starts
dropping

    -- Shish

On Tue, 15 Aug 2006 16:55:50 +0100 Shish <shish@...>
wrote:

> I was just about to say the same thing; the only thing I lack is a
> suitably huge CVS repository to start with -- is there any way for me
> to make a local mirror of E's whole repository? Google fails me the
> specific procedure...
>=20
>=20
> > tmpfs,
>=20
> In real life disks will be involved, it'd probably be good to take
> them into account. (I'd recommend doing both, to see what effect the
> disks have)
>=20
>=20
> > I'll grab it 100 times with my laptop
>=20
> I was thinking a more appropriate benchmark would be to see how many
> parallel checkouts can be done at once before performance starts
> dropping

Don't forget the most crucial of benchmarks - how long will it take
raster to rsync his local working copy?

On Tue, 15 Aug 2006 16:55:50 +0100
Shish <shish@...> wrote:

> In real life disks will be involved, it'd probably be good to take
> them into account. (I'd recommend doing both, to see what effect the
> > I'll grab it 100 times with my laptop
 ... 
> I was thinking a more appropriate benchmark would be to see how many
> parallel checkouts can be done at once before performance starts
> dropping
> 
>     -- Shish
> 

The problem comes from the fact that I can't really limit the connection speed in a process-based way. 

Basically, when I try to run x threads simultaneously the hdd will pull the speed of the entire process down and you have basically no way of telling which SCM is faster. 

They would probably all finish at once, or close to that, because they're limited by the speed of the device. 

Now I'd love to be able to run multiple threads simultaneously with tmpfs but this is a big repository and I haven't got the RAM to duplicate it that much.

This benchmark is, well, a benchmark, so a big ball of salt should be carried around.

But I tend to believe that at least in terms of CPU, the load on the server is equivalent if you have one client churning away at 4MB/s or a lot of smaller ones doing the same thing.

It's really the memory that is affected but I understood this wasn't really the issue.

The script is almost done, all I have to setup now is the git repository and I'm done.

I have a problem because I didn't get all the cvs (just e17) and I don't have the history information anymore (you can't keep it unless you use a mechanism like rsync). 

I hope this isn't an issue but I will post the script along with the results and I encourage anyone with enough time on their hands to run the test on their own machines/network.

Cheers, 
Eugen.

> I have a problem because I didn't get all the cvs (just e17) and I don't
> have the history information anymore (you can't keep it unless you use
> a mechanism like rsync). 

It was mentioned earlier that the large amount of history information
was SVN's bottleneck when you check out a fresh copy; I imagine it
would be a burden for CVS too -- hence why I was asking for some way of
downloading the entire repository, history included~

    -- Shish

On Tuesday, 15 August 2006, at 19:01:54 (+1000),
David Seikel wrote:

> Your anti-SVN message is starting to sound like it's coming from
> someone that hasn't actually run a SVN server recently.

I admin 2 and am a developer on a third.  But by all means, please
feel free to continue assuming I must not know what I'm talking about
because my opinion differs with yours.  It's much easier to write me
off than to actually consider what I have to say.

Like the guy who tried to explain to me what tagging is, why a copy
really is a tag, and how to use "svn switch" as though I were a
cluebie.  Very cute.

> Tagging and branching, when was the last time either occurred to E?

July 17th or so...at least that's the most recent a cursory search
found.  Why do you ask?




On Tuesday, 15 August 2006, at 16:55:50 (+0100),
Shish wrote:

> I was just about to say the same thing; the only thing I lack is a
> suitably huge CVS repository to start with -- is there any way for
> me to make a local mirror of E's whole repository?

Rsync access is provided to the complete anoncvs repository.

rsync anoncvs.enlightenment.org::anoncvs/e/

Includes revision history and everything.

Michael

-- 
Michael Jennings (a.k.a. KainX)  http://www.kainx.org/  <mej@...>
n + 1, Inc., http://www.nplus1.net/       Author, Eterm (www.eterm.org)
-----------------------------------------------------------------------
 "That's why C++ is so disappointing:  it does nothing to address some
  of the most fundamental problems in C, and its most important
  addition (classes) builds on the deficient C type model."
                       -- Peter Van Der Linden, "Expert C Programming"

On Tue, 15 Aug 2006 13:57:42 -0400 Michael Jennings <e-devel@...>
wrote:

> But by all means, please feel free to continue assuming I must not
> know what I'm talking about because my opinion differs with yours.
> It's much easier to write me off than to actually consider what I
> have to say.

I'm not assuming anything, I am responding to what you are actually
saying.  It's not a matter of opinion, you have stated things as fact
that are not true.  Others have corrected what you have been saying.

On Wednesday, 16 August 2006, at 04:09:54 (+1000),
David Seikel wrote:

> I'm not assuming anything, I am responding to what you are actually
> saying.

That was the first time you'd actually responded to *me*, and the only
other response you made indirectly to me was about the backend.  I'll
refer you to previous e-mails as to why the filesystem backend has
significant performance issues (re: deltas) which make it
counter-productive toward the performance goal.

> It's not a matter of opinion, you have stated things as fact that
> are not true.

Name one thing.

It's true that SVN is almost always used with DAV and Apache.  Sure,
there are exceptions here and there, but they are the minority.

The BDB backend is THE choice for performance, particularly with
anonymous access.  I haven't heard anyone contradict that.

SVN does not support tagging and branching.  It supports copying,
which creates a copy-on-write replica of part of the repository in
another location in the repository.  While this can perform the same
functions as tagging and branching, it is NOT the same thing.  That's
not an opinion; that is fact.  Tagging is creating a label on a
particular tree at a particular point, not creating a copy of the tree
at that point.  Branching is forking a tree at a particular point.
Copying is closer to branching than tagging, but it's still not the
same.  The fundamental distinction is that tagging and branching
happen *to* a tree; copying is necessarily *outside* the tree.  I'm
sorry if you can't see that, but I (and many other developers) can and
have run into the problems it creates.

> Others have corrected what you have been saying.

Whatever you say.

We're talking about performance here, and I don't see further
discussion of Subversion's capabilities (or lacks thereof) are
furthering that goal.

Michael

-- 
Michael Jennings (a.k.a. KainX)  http://www.kainx.org/  <mej@...>
n + 1, Inc., http://www.nplus1.net/       Author, Eterm (www.eterm.org)
-----------------------------------------------------------------------
 "I now cry streams of blood because I had to take my stand.  I crush
  my eyes beneath my heel as my heart pulses in my hand."
                                                         -- "Forsaken"

Hi,

I've spent a lot of time on this, but the results are ... shall we say ... interesting. Before proceeding I have to remind you, again that I only used e17 without the version info, so you will probably want to try this out for yourself.

The script I wrote checked the average time for a checkout. I found that checkouts from the git server take a pheonmenally long time and I was too tired to wait for it (somewhere along the lines of 3-5 minutes, not sure really). 

That makes me think I may have not set it up. If there's anything I should have done, or more info you would like, include it here.

I've also found that git offers another very interesting thing. Rsync as a protocol for its client (similar to the way it uses http, there's no need for any extra modules or configuration, you just copy the tree somewhere and it works.

I also kept an eye on the CPU and memory load on the server. So here's what I found.

CVS: 
- Average checkout time: 41.843s
- CPU used: Constantly around 70% (something like 60-80%
- MEM used: 2-3%

SVN (svnserve):
- Average checkout time: 27.921s
- CPU used: 50-90%
- Mem used: 2%

SVN (http):
- Average checkout time: 61.239s
- CPU used: 70-90%
- Mem used: 10-12% *

Git/HTTP:
- Average checkout time: 98.962s
- CPU used: < 4%
- Mem used: 10-12 *

* Somehow I think Apache caches a lot of the stuff out because the memory gets allocated and remains there after the fun is over.

So this looks very weird. And at the end of the day it doesn't really prove too much. SVN and Git (particularly Git) are really gentle on the server's resources at the expense of higher download times.

But this is once again, a single connection, in a pedal-to-the-metal attempt to get the repository as quickly as possible. I have not optimized anything in here, but feel free to do it yourselves.

The script is written in Ruby, just make sure to change the constants for the commands at the top of the file. I eventually ran in 20 times because 100 seemed like way too much.

Hope you guys can use the data, or at least the script.

Cheers,
Eugen.

On Tuesday, 15 August 2006, at 22:12:47 (+0300),
Eugen Minciu wrote:

> CVS: 
> - Average checkout time: 41.843s
> - CPU used: Constantly around 70% (something like 60-80%
> - MEM used: 2-3%
> 
> SVN (svnserve):
> - Average checkout time: 27.921s
> - CPU used: 50-90%
> - Mem used: 2%
> 
> SVN (http):
> - Average checkout time: 61.239s
> - CPU used: 70-90%
> - Mem used: 10-12% *
> 
> Git/HTTP:
> - Average checkout time: 98.962s
> - CPU used: < 4%
> - Mem used: 10-12 *

Finally!  Some useful data.

> So this looks very weird. And at the end of the day it doesn't
> really prove too much. SVN and Git (particularly Git) are really
> gentle on the server's resources at the expense of higher download
> times.

I agree regarding git, but not SVN.  60-80% averages out to around
70%, and so does 50-90%.  So it's not really any better on resources
than CVS, and SVN+HTTP is significantly worse (which is as I predicted
and expected).

Was this with the BDB or FS backend?

Can you try multiple invocations of the script from the same host?
Like say 20 or 30 simultaneous checkouts?

I'd also be interested in comparisons of incremental checkouts.  As I
understand it, Git trees are complete repositories, not just
checkouts.  So a checkout-to-checkout comparison is unfair as Git is
downloading a crapload more data (history and such).

Thanks!
Michael

-- 
Michael Jennings (a.k.a. KainX)  http://www.kainx.org/  <mej@...>
n + 1, Inc., http://www.nplus1.net/       Author, Eterm (www.eterm.org)
-----------------------------------------------------------------------
 "Nerds make the best lovers.  That's why I'm in Speed School." 
                                                       -- Angela Smith

On 8/15/06, Michael Jennings <e-devel@...> wrote:
>
> I'd also be interested in comparisons of incremental checkouts.  As I
> understand it, Git trees are complete repositories, not just
> checkouts.  So a checkout-to-checkout comparison is unfair as Git is
> downloading a crapload more data (history and such).
>
> Thanks!
> Michael

Do we have any current statistics about the load on a CVS mirror?
Might be easier to find more mirrors if we can give them some idea
what sort of bandwidth, disk use and CPU load they can expect.

Thanks,
Nathan

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Eugen Minciu wrote:
<snip>
> Now I know this is probably not going to sound right but how about a git 
> repository?.
<snip>

You may be interested in reading our[0] thread[1] about our search for
an SCM, with us deciding, in the end, on git. Mainly git is compared
against Perforce, but it should be applicable here.

A little information about our repository[2] for comparison: 60,000 code
commits with a 95MB repository averaging 50 commits per day

- -sandalle

[0] http://www.sourcemage.org/
[1] http://article.gmane.org/gmane.linux.sourcemage.discuss/2423
  Shows use cases, timing, and space requirements.
[2] Source: http://lwn.net/Articles/145233/ dated August 2, 2005

- --
Eric Sandall                     |  Source Mage GNU/Linux Developer
eric@...                  |  http://www.sourcemage.org/
http://eric.sandall.us/          |  SysAdmin @ Shock Physics @ WSU
http://counter.li.org/  #196285  |  http://www.shock.wsu.edu/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFE4j9aHXt9dKjv3WERAg86AJ9f7qcgxx36SENy3/qsbsxiOjnMZwCfcapl
ZrudiTytfhXz9uLZU97dO4c=
=9Q20
-----END PGP SIGNATURE-----

well, thats what this is KainX. I have pretty much secured a donated box. I
have also been working with raster to find the right box for us to buy. the
place that I was going to have host the mirror said that its time for us to
go get our own box, so I let raster know that. I'll update you more on the
specs of the donated server and the price of the other as soon as my
contacts get back to me....  I've been working to get boxes
donated... I've contacted a few places but no other luck besides the one
box.... so far.....
-- 
--Inc
 http://www.inc-omplete.org
 http://www.inc-corporate.org

Well I went outside for a while and it did me good. I have a new ideea for testing out the scm servers. Yay ;)

Instead of trying to actually save the items on my hard disk I'll try to redirect them to /dev/null. I allready saw how I can do this with cvs and I hope I'll find a way to do it with both svn and git.

So I'll be modifying the script, I'll have it create around 500 threads so that each thread will receive about 10K/s. I guess that should be a pretty acurate description of how much load a large system takes upon itself.

Cheers,
Eugen.

It seems to me that our SCM system feature requirements are extremely
limited. We hardly ever tag or branch, let alone do merging between
branches or anything resembling changeset management.

I think CVS amply provides the features we need. It's simple and robust.
It's far from perfect, but as Michael says - It's the devil we know.

I don't believe changing SCM will make any significant difference to the 
problems that we appear to have, except possibly on a very short term. 
Even in the unlikely case that some SCM system is twice(four/ten/... 
times) as fast as CVS the problem we have will resurface if/when the 
user base grows accordingly.

Unless there happens to be an SCM system that is just incredibly more 
efficient than CVS I also think that we would make a change for the 
wrong reason. Changing SCM system should IMO be done only if we 
want/need particular features not available in CVS.

In the unfortunate case that it is concluded that we want to switch away 
from CVS I hope it will be to git. Not because I know it from personal 
experience, but simply because it is used by linux and xorg, which are 
two projects I respect.

/Kim

On Tuesday, 15 August 2006, at 15:04:06 (-0700),
Inc wrote:

> well, thats what this is KainX. I have pretty much secured a donated
> box. I have also been working with raster to find the right box for
> us to buy. the place that I was going to have host the mirror said
> that its time for us to go get our own box, so I let raster know
> that. I'll update you more on the specs of the donated server and
> the price of the other as soon as my contacts get back to me....
> I've been working to get boxes donated... I've contacted a few
> places but no other luck besides the one box.... so far.....

It seems to me that our best bet would be to get the donated box
online and start dividing the load between the two anoncvs boxes.  We
may not need a 2nd box.



On Wednesday, 16 August 2006, at 00:08:18 (+0200),
Kim Woelders wrote:

> It seems to me that our SCM system feature requirements are
> extremely limited. We hardly ever tag or branch, let alone do
> merging between branches or anything resembling changeset
> management.

I know you tag quite a bit, and I personally maintain at least 2
branches at any one time.

> Unless there happens to be an SCM system that is just incredibly
> more efficient than CVS I also think that we would make a change for
> the wrong reason. Changing SCM system should IMO be done only if we
> want/need particular features not available in CVS.

Hear hear!

Michael

-- 
Michael Jennings (a.k.a. KainX)  http://www.kainx.org/  <mej@...>
n + 1, Inc., http://www.nplus1.net/       Author, Eterm (www.eterm.org)
-----------------------------------------------------------------------
 "I want to stand with you on a mountain.  I want to bathe with you
  in the sea.  I want to lay like this forever, until the sky falls
  down on me."                -- Savage Garden, "Truly, Madly, Deeply"

Kim Woelders wrote:
> 
> 
> 
> It seems to me that our SCM system feature requirements are extremely
> limited. We hardly ever tag or branch, let alone do merging between
> branches or anything resembling changeset management.
> 
> I think CVS amply provides the features we need. It's simple and robust.
> It's far from perfect, but as Michael says - It's the devil we know.
> 
> I don't believe changing SCM will make any significant difference to the
> problems that we appear to have, except possibly on a very short term.
> Even in the unlikely case that some SCM system is twice(four/ten/...
> times) as fast as CVS the problem we have will resurface if/when the
> user base grows accordingly.
> 
> Unless there happens to be an SCM system that is just incredibly more
> efficient than CVS I also think that we would make a change for the
> wrong reason. Changing SCM system should IMO be done only if we
> want/need particular features not available in CVS.
> 
> In the unfortunate case that it is concluded that we want to switch away
> from CVS I hope it will be to git. Not because I know it from personal
> experience, but simply because it is used by linux and xorg, which are
> two projects I respect.
> 
> /Kim

Well said Kim.

I agree completely.  I'd like to see a multi-connection test of cvs vs
svn, but I also won't promise to set it up myself. :)  It was nice of
raster to open this debate, but we have running servers with cvs (and
webvc) and should really have a compelling reason to move.  As you can
see, I'm not much of a fan of change for change's sake. ;)

-- 
Kevin

On Wed, 16 Aug 2006 01:07:12 +0300 Eugen Minciu <minciue@...> babbled:

> Well I went outside for a while and it did me good. I have a new ideea for
> testing out the scm servers. Yay ;)
> 
> Instead of trying to actually save the items on my hard disk I'll try to
> redirect them to /dev/null. I allready saw how I can do this with cvs and I
> hope I'll find a way to do it with both svn and git.
> 
> So I'll be modifying the script, I'll have it create around 500 threads so
> that each thread will receive about 10K/s. I guess that should be a pretty
> acurate description of how much load a large system takes upon itself.

actually - i think we need to know how this works WITH data on disk - why? some
scm's may invoke much more disk IO than others and thus bottleneck at the disk
earlier than others. we need to know. unless you mean save on the client side -
then thats fine. but its server-side we really care about here - remember :)

word seems to have it that git is "da shitsnizzle" when it comes to performance
- but i am going to want to see the numbers. how many clients can connect and
checkout and/or update and how long does it take vs. the load on the server etc.

> Cheers,
> Eugen.
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

On Mon, 14 Aug 2006 19:07:13 +0200 Tilman Sauerbeck <tilman@...>
babbled:

> Eugen Minciu [2006-08-14 15:29]:
> > Carsten Haitzler (The Rasterman) wrote:
> > > It has come to my attention that.... yet again we are killing systems.
> > >
> > > yes - we are becoming a burden on yet more cvs servers. we are monsters! :
> > > (
> > >
> > > anyway - we have been living on caosity's cvs for a while now - but we are
> > > killing it (sorry kainx!)
> > >
> > > so its time to finally bite the bullet and dredge up the issue of us
> > > needing servers again.
> > >
> > > here is what i think we need:
> > >
> > > 1. devel "cvs" server + future web server (for downloads too of official
> > > tarballs etc.)
> > > 2. an anonymous "cvs" server and possibly second download mirror.
> > >
> > > so 2 systems really.
> > >
> > > i hear that svn is significantly less load for anonymous access - even
> > > developer - who has experience with this server-side? can you confirm or
> > > deny? i would consider a possible move to svn if we can keep our history
> > > from cvs.
> > >
> > > so - let the flames begin.
> > >
> > >   
> > Now I know this is probably not going to sound right but how about a git 
> > repository?.
> 
> git kicks Subversion's ass. At first, the decentralized nature of git
> might feel weird, but you'll get used to it, and love it (I've been
> using git since the day when Bitkeeper ended it's free-as-in-beer
> license thing, I think it was 2005-07-01).
> 
> Disk space usage on the developer's box is considerably worse than CVS
> of course. The read-only git daemon (the one you'd use for
> anongit.e.org) is light on ressources I believe. Developer access would
> be done via ssh.
> 
> cogito isn't needed for a sane git experience any more (it was, in the
> early days).

i think i should check git. i know svn enough to know its a viable move from
cvs - but i know very little of git - but from the comments - it seems good
(these days) - though much more radical than  move to svn :)

> Regards,
> Tilman
> 
> -- 
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing on usenet and in e-mail?
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

On Tue, 15 Aug 2006 14:58:42 -0400 Michael Jennings <e-devel@...> babbled:

> On Wednesday, 16 August 2006, at 04:09:54 (+1000),
> David Seikel wrote:
> 
> > I'm not assuming anything, I am responding to what you are actually
> > saying.
> 
> That was the first time you'd actually responded to *me*, and the only
> other response you made indirectly to me was about the backend.  I'll
> refer you to previous e-mails as to why the filesystem backend has
> significant performance issues (re: deltas) which make it
> counter-productive toward the performance goal.
> 
> > It's not a matter of opinion, you have stated things as fact that
> > are not true.
> 
> Name one thing.
> 
> It's true that SVN is almost always used with DAV and Apache.  Sure,
> there are exceptions here and there, but they are the minority.
> 
> The BDB backend is THE choice for performance, particularly with
> anonymous access.  I haven't heard anyone contradict that.
> 
> SVN does not support tagging and branching.  It supports copying,
> which creates a copy-on-write replica of part of the repository in
> another location in the repository.  While this can perform the same
> functions as tagging and branching, it is NOT the same thing.  That's
> not an opinion; that is fact.  Tagging is creating a label on a
> particular tree at a particular point, not creating a copy of the tree
> at that point.  Branching is forking a tree at a particular point.
> Copying is closer to branching than tagging, but it's still not the
> same.  The fundamental distinction is that tagging and branching
> happen *to* a tree; copying is necessarily *outside* the tree.  I'm
> sorry if you can't see that, but I (and many other developers) can and
> have run into the problems it creates.

i'm  on the  "much of a muchness" as to the copy vs. tag/branch thing here -
others may like to fine-grain use each and all - but i personally am quite
happy with a copy (on write) :)

NB - i didn't mean to make this be a "lets argue for the most powerful scms on
the planet" and if it does X or not - i know i am definitely not after "more
features" - i want to see if we can make anon "cvs" scale better on 1 box
before having to move to anoncvs mirror farms.

i just "heard" that svn is lighter on a system for anonymous access and i
wanted to find out if it is true - and if so - by how much (if it is by 10% -
forget it - but if you can server 3 or 5 or 10 times as many users given the
same  data and resources before you run out... then i see at least one viable
alternative to be CONSIDERED) :)

so basically - what i am saying is... lets keep this on topic - your original
comments were definitely read by me and noted. but now we have eugen going to
do some actual "benchmarks" and that is going to "excite" me :)

> > Others have corrected what you have been saying.
> 
> Whatever you say.
> 
> We're talking about performance here, and I don't see further
> discussion of Subversion's capabilities (or lacks thereof) are
> furthering that goal.

EXAAAAACTLY! :) on topic :)

> Michael
> 
> -- 
> Michael Jennings (a.k.a. KainX)  http://www.kainx.org/  <mej@...>
> n + 1, Inc., http://www.nplus1.net/       Author, Eterm (www.eterm.org)
> -----------------------------------------------------------------------
>  "I now cry streams of blood because I had to take my stand.  I crush
>   my eyes beneath my heel as my heart pulses in my hand."
>                                                          -- "Forsaken"
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

On Wed, 16 Aug 2006 02:10:02 +1000 David Seikel <onefang@...> babbled:

> On Tue, 15 Aug 2006 16:55:50 +0100 Shish <shish@...>
> wrote:
> 
> > I was just about to say the same thing; the only thing I lack is a
> > suitably huge CVS repository to start with -- is there any way for me
> > to make a local mirror of E's whole repository? Google fails me the
> > specific procedure...
> > 
> > 
> > > tmpfs,
> > 
> > In real life disks will be involved, it'd probably be good to take
> > them into account. (I'd recommend doing both, to see what effect the
> > disks have)
> > 
> > 
> > > I'll grab it 100 times with my laptop
> > 
> > I was thinking a more appropriate benchmark would be to see how many
> > parallel checkouts can be done at once before performance starts
> > dropping
> 
> Don't forget the most crucial of benchmarks - how long will it take
> raster to rsync his local working copy?

hahahahahahahahha! :) a lot longer i imagine - but as i said - if this improves
anon access for everyone... it's good.


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

On Tue, 15 Aug 2006 16:55:50 +0100 Shish <shish@...> babbled:

> I was just about to say the same thing; the only thing I lack is a
> suitably huge CVS repository to start with -- is there any way for me
> to make a local mirror of E's whole repository? Google fails me the
> specific procedure...
> 
> 
> > tmpfs,
> 
> In real life disks will be involved, it'd probably be good to take
> them into account. (I'd recommend doing both, to see what effect the
> disks have)
> 
> 
> > I'll grab it 100 times with my laptop
> 
> I was thinking a more appropriate benchmark would be to see how many
> parallel checkouts can be done at once before performance starts
> dropping

exactly- and then how does it drop. and what is the load on the server?

>     -- Shish
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

On Tue, 15 Aug 2006 18:34:57 -0400 Kevin Brosius <cobra@...> babbled:

> Kim Woelders wrote:
> > 
> > 
> > 
> > It seems to me that our SCM system feature requirements are extremely
> > limited. We hardly ever tag or branch, let alone do merging between
> > branches or anything resembling changeset management.
> > 
> > I think CVS amply provides the features we need. It's simple and robust.
> > It's far from perfect, but as Michael says - It's the devil we know.
> > 
> > I don't believe changing SCM will make any significant difference to the
> > problems that we appear to have, except possibly on a very short term.
> > Even in the unlikely case that some SCM system is twice(four/ten/...
> > times) as fast as CVS the problem we have will resurface if/when the
> > user base grows accordingly.
> > 
> > Unless there happens to be an SCM system that is just incredibly more
> > efficient than CVS I also think that we would make a change for the
> > wrong reason. Changing SCM system should IMO be done only if we
> > want/need particular features not available in CVS.
> > 
> > In the unfortunate case that it is concluded that we want to switch away
> > from CVS I hope it will be to git. Not because I know it from personal
> > experience, but simply because it is used by linux and xorg, which are
> > two projects I respect.
> > 
> > /Kim
> 
> Well said Kim.
> 
> I agree completely.  I'd like to see a multi-connection test of cvs vs
> svn, but I also won't promise to set it up myself. :)  It was nice of
> raster to open this debate, but we have running servers with cvs (and
> webvc) and should really have a compelling reason to move.  As you can
> see, I'm not much of a fan of change for change's sake. ;)

the compelling reason is that (apparently) anoncvs is straining under its load
(again) and this happened so quickly that i am putting up a debate for
alternate solutions - willing to discuss really anything :)

-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

On Wed, 16 Aug 2006 00:08:18 +0200 Kim Woelders <kim@...> babbled:

> 
> It seems to me that our SCM system feature requirements are extremely
> limited. We hardly ever tag or branch, let alone do merging between
> branches or anything resembling changeset management.
> 
> I think CVS amply provides the features we need. It's simple and robust.
> It's far from perfect, but as Michael says - It's the devil we know.
> 
> I don't believe changing SCM will make any significant difference to the 
> problems that we appear to have, except possibly on a very short term. 
> Even in the unlikely case that some SCM system is twice(four/ten/... 
> times) as fast as CVS the problem we have will resurface if/when the 
> user base grows accordingly.
> 
> Unless there happens to be an SCM system that is just incredibly more 
> efficient than CVS I also think that we would make a change for the 
> wrong reason. Changing SCM system should IMO be done only if we 
> want/need particular features not available in CVS.
> 
> In the unfortunate case that it is concluded that we want to switch away 
> from CVS I hope it will be to git. Not because I know it from personal 
> experience, but simply because it is used by linux and xorg, which are 
> two projects I respect.

i agree with everything you said - we need very few features - we need history
management so we can see who did what to what and when - that is enough to be
able to roll back. i am looking at git, but i would only move if it really is
an order of magnitude faster (and i dont mean by download times but by how many
users can a single server handle before it becomes useless).

i had been hoping to get benchmarks - and - we have some! yay! i do agree they
need to test what happens if you get 10 or 20 clients at once using it.

svn is also an option - but so far it deosn't seem to really be far ahead
enough of cvs to warrant much further looking :(

of course we could consider much more exotic solutions...

insead of anon cvs we could provide server-side regular "make dist" tarballs
and go back to a download-only distriubtion mehcanism.

anonymous users really have little use for cvs beyond limiting download sizes
of updates. what alternative options do we have?

raw tarballs of cvs module checkouts generated by servers - make dist tarballs
of particular sub-projects - making a cvs checkout file tree available over
http, ftp or rsync?

we can look at this instead...

> /Kim
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

On Tue, 15 Aug 2006 19:11:57 +0300 Eugen Minciu <minciue@...> babbled:

> On Tue, 15 Aug 2006 16:55:50 +0100
> Shish <shish@...> wrote:
> 
> > In real life disks will be involved, it'd probably be good to take
> > them into account. (I'd recommend doing both, to see what effect the
> > > I'll grab it 100 times with my laptop
>  ... 
> > I was thinking a more appropriate benchmark would be to see how many
> > parallel checkouts can be done at once before performance starts
> > dropping
> > 
> >     -- Shish
> > 
> 
> The problem comes from the fact that I can't really limit the connection
> speed in a process-based way. 
> 
> Basically, when I try to run x threads simultaneously the hdd will pull the
> speed of the entire process down and you have basically no way of telling
> which SCM is faster. 

in a real-life scm this will happen too. some scm's will hit disk harder than
others - this is a crucial measurement :)

> They would probably all finish at once, or close to that, because they're
> limited by the speed of the device. 
> 
> Now I'd love to be able to run multiple threads simultaneously with tmpfs but
> this is a big repository and I haven't got the RAM to duplicate it that much.
> 
> This benchmark is, well, a benchmark, so a big ball of salt should be carried
> around.
> 
> But I tend to believe that at least in terms of CPU, the load on the server
> is equivalent if you have one client churning away at 4MB/s or a lot of
> smaller ones doing the same thing.

nb - when i said load - i meant the whole kit & kaboodle. ie cpu, memory
footprint, context switch overhead, IO overhead, etc.

an scm that accesses only 100mb of disk data will only need 100mb of ram cache
to remain "fast". an scm that accesses 1000mb of "unique" disk data will eat up
much more cache space and thus degrade performance faster as repositories get
bigger. if it needs more malloc()ed ram to run - then that is less ram for
cache - etc. etc.

we need to know the overall performance as it would be on a long-running live
system (ie data is already loaded and cached after some initial
checkouts/updates) and it is humming along and now may get N clients wanting to
checkout and/or update. yes your network will be a limiting factor if you can
push 100mbit of traffic without a problem.

now if you can - try "slowing down" the server. if you can use speedstepping or
cpu throttling to artificially make it a slower system - use hdparm if you can
to slow the disk down. boot with a mem=X option to limit ram (and try it with
different levels). then "Relatively" the network will seem to get faster. this
of course is if it can handle pushing data at 100mbit without problems to many
clients and the network is our limiter... :)

> It's really the memory that is affected but I understood this wasn't really
> the issue.
> 
> The script is almost done, all I have to setup now is the git repository and
> I'm done.
> 
> I have a problem because I didn't get all the cvs (just e17) and I don't have
> the history information anymore (you can't keep it unless you use a mechanism
> like rsync). 
> 
> I hope this isn't an issue but I will post the script along with the results
> and I encourage anyone with enough time on their hands to run the test on
> their own machines/network.
> 
> Cheers, 
> Eugen.
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

"Carsten Haitzler (The Rasterman)" wrote:
> 
> 
> On Tue, 15 Aug 2006 18:34:57 -0400 Kevin Brosius <cobra@...> babbled:
> 
> > Kim Woelders wrote:
> > >
> > >
> > >
> > > It seems to me that our SCM system feature requirements are extremely
> > > limited. We hardly ever tag or branch, let alone do merging between
> > > branches or anything resembling changeset management.
> > >
> > > I think CVS amply provides the features we need. It's simple and robust.
> > > It's far from perfect, but as Michael says - It's the devil we know.
> > >
> > > I don't believe changing SCM will make any significant difference to the
> > > problems that we appear to have, except possibly on a very short term.
> > > Even in the unlikely case that some SCM system is twice(four/ten/...
> > > times) as fast as CVS the problem we have will resurface if/when the
> > > user base grows accordingly.
> > >
> > > Unless there happens to be an SCM system that is just incredibly more
> > > efficient than CVS I also think that we would make a change for the
> > > wrong reason. Changing SCM system should IMO be done only if we
> > > want/need particular features not available in CVS.
> > >
> > > In the unfortunate case that it is concluded that we want to switch away
> > > from CVS I hope it will be to git. Not because I know it from personal
> > > experience, but simply because it is used by linux and xorg, which are
> > > two projects I respect.
> > >
> > > /Kim
> >
> > Well said Kim.
> >
> > I agree completely.  I'd like to see a multi-connection test of cvs vs
> > svn, but I also won't promise to set it up myself. :)  It was nice of
> > raster to open this debate, but we have running servers with cvs (and
> > webvc) and should really have a compelling reason to move.  As you can
> > see, I'm not much of a fan of change for change's sake. ;)
> 
> the compelling reason is that (apparently) anoncvs is straining under its load
> (again) and this happened so quickly that i am putting up a debate for
> alternate solutions - willing to discuss really anything :)

Hmmm. I'm not sure I see the logic that way.  We moved off sf to get the
ability to do rsync backup of cvs (one of the reasons, along with some
intermittent performance of dev and anon cvs.)  We now have solid dev
cvs performance, rsync access, and better anon access (IMO.)

Are you against requesting offers of open anon mirrors?  With the rsync
access, that seems easy, but of you want a anon cvs farm that has full
status monitoring, then you're stepping the complexity up a bit.

I guess I don't buy the 'straining under the load and happening quickly'
as a compelling reason. ;)  It's a different situation.

-- 
Kevin

On Wed, 16 Aug 2006 08:07:16 +0900
Carsten Haitzler (The Rasterman) <raster@...> wrote:

> actually - i think we need to know how this works WITH data on disk - why? some
> scm's may invoke much more disk IO than others and thus bottleneck at the disk
> earlier than others. we need to know. unless you mean save on the client side -
> then thats fine. but its server-side we really care about here - remember :)
Well ... it was client side, but it didn't work as I couldn't find a way to send the files to the bit bucket with either svn or git.

> word seems to have it that git is "da shitsnizzle" when it comes to performance
> - but i am going to want to see the numbers. how many clients can connect and
> checkout and/or update and how long does it take vs. the load on the server etc.

Well I oficially declare my numbers crap. I hold my laptop responsible for this mess and I don't really trust a single number here. And I'll explain why in a minute.

Here are the numbers, but as I said, I really don't think they should be considered usefull. Consider this to be FUD.

CVS: 
- CPU Load: Max
- Mem Load: 45%
- Checkout Time: 296.374s

SVN:
- CPU Load: 20% ( It was 20% in the last test as well I believe, all processes seem to share the same CPU usage.)
- Mem Load: 40%
- Checkout Time: 658.576

SVN/HTTP:
- CPU Load: 70-90%
- Mem Load: 60%
- Checkout Time: 874.618

GIT (git protocol):
- CPU Load: Max
- Mem Load: 10-15%
- Checkout Time: 2345.243

GIT (http):
- CPU Load: < 5%
- Mem Load: 10-12%
- Checkout Time: A lot, from what I could see, it was cloning at a very low rate.

I used actual disk access on both the client and the server, Apache2-mpm-worker, SVN with FSFS and that's about it. A spec of machines would be missing the point. The point is they aren't any good for these kinds of "benchmarks".

It looks really suspicious to me that times seem to increase linearly among the tested SCMs, even though this wasn't the case in the first test. 

Also, I often found network traffic dropping to 0 because the laptop's HDD couldn't take the heat anymore.

Also, the version of the script I tested with did not wait after one SCM finished and moved directly to the next one. I don't think that was a good idea.

Now, Git with http was excellent in terms of CPU & Memory usage but I'm not really sure why it wouldn't go past 1.5MB/s (ever). It would have probably taken me an hour or so, and that's on a LAN which is pretty much unacceptable.

But here's the script. Feel free to change the number of threads at the top of the script (it's currently at 20) but please make note that you need about 180MBx number of threads available.

Change the lines near the top to specify how the checkout commands should be run in your environment. If you want to add another test but lack the Ruby know-how just tell me what you need and I'll patch it up for you.

Finally make a new directory, copy the script there and execute 'ruby scm_benchmark.rb'. And that's it.

So ultimately I urge someone with more patience and two solid machines to give my little test a spin. Please don't make any decision based on my data. Pretty please! And Carsten, if this doesn't turn you on, I'm sorry ;)

Cheers,
Eugen.

Now I wonder what that 'Attatch' button does? ...
Sorry bout that.

Eugen.

You can always ask some help on git@....
Personally I have been following the development of git from a technical
point of view out of interest, but at the moment I do not have a linux box
unfortunately so I have not been able to play with it. It also has a tool
(gitk) that visualizes merges etc.)

What I can say is that from a usability point of view it is very fast (that
was Linus' number one design rule when he wrote git)
For download performance it is probably important to pack the repository
before checking out from a client. Pack files are supposed to help a lot
with regards to network performance if I understand correct.

Some links :

http://git.or.cz/gitwiki/GitBenchmarks
http://www.kernel.org/pub/software/scm/git/docs/cvs-migration.html

Which projects use git? (X.org e.g.)
http://git.or.cz/gitwiki/GitProjects
http://freedesktop.org/wiki/UsingGit

the web interface :
http://www.kernel.org/git/

some history of why Linus started it.
http://git.or.cz/gitwiki/GitHistory

Hope this is of help to some.
D=E8nis


On 8/16/06, Eugen Minciu <minciue@...> wrote:
>
> On Wed, 16 Aug 2006 08:07:16 +0900
> Carsten Haitzler (The Rasterman) <raster@...> wrote:
>
> > actually - i think we need to know how this works WITH data on disk -
> why? some
> > scm's may invoke much more disk IO than others and thus bottleneck at
> the disk
> > earlier than others. we need to know. unless you mean save on the clien=
t
> side -
> > then thats fine. but its server-side we really care about here -
> remember :)
> Well ... it was client side, but it didn't work as I couldn't find a way
> to send the files to the bit bucket with either svn or git.
>
> > word seems to have it that git is "da shitsnizzle" when it comes to
> performance
> > - but i am going to want to see the numbers. how many clients can
> connect and
> > checkout and/or update and how long does it take vs. the load on the
> server etc.
>
> Well I oficially declare my numbers crap. I hold my laptop responsible fo=
r
> this mess and I don't really trust a single number here. And I'll explain
> why in a minute.
>
> Here are the numbers, but as I said, I really don't think they should be
> considered usefull. Consider this to be FUD.
>
> CVS:
> - CPU Load: Max
> - Mem Load: 45%
> - Checkout Time: 296.374s
>
> SVN:
> - CPU Load: 20% ( It was 20% in the last test as well I believe, all
> processes seem to share the same CPU usage.)
> - Mem Load: 40%
> - Checkout Time: 658.576
>
> SVN/HTTP:
> - CPU Load: 70-90%
> - Mem Load: 60%
> - Checkout Time: 874.618
>
> GIT (git protocol):
> - CPU Load: Max
> - Mem Load: 10-15%
> - Checkout Time: 2345.243
>
> GIT (http):
> - CPU Load: < 5%
> - Mem Load: 10-12%
> - Checkout Time: A lot, from what I could see, it was cloning at a very
> low rate.
>
> I used actual disk access on both the client and the server,
> Apache2-mpm-worker, SVN with FSFS and that's about it. A spec of machines
> would be missing the point. The point is they aren't any good for these
> kinds of "benchmarks".
>
> It looks really suspicious to me that times seem to increase linearly
> among the tested SCMs, even though this wasn't the case in the first test=
.
>
> Also, I often found network traffic dropping to 0 because the laptop's HD=
D
> couldn't take the heat anymore.
>
> Also, the version of the script I tested with did not wait after one SCM
> finished and moved directly to the next one. I don't think that was a goo=
d
> idea.
>
> Now, Git with http was excellent in terms of CPU & Memory usage but I'm
> not really sure why it wouldn't go past 1.5MB/s (ever). It would have
> probably taken me an hour or so, and that's on a LAN which is pretty much
> unacceptable.
>
> But here's the script. Feel free to change the number of threads at the
> top of the script (it's currently at 20) but please make note that you ne=
ed
> about 180MBx number of threads available.
>
> Change the lines near the top to specify how the checkout commands should
> be run in your environment. If you want to add another test but lack the
> Ruby know-how just tell me what you need and I'll patch it up for you.
>
> Finally make a new directory, copy the script there and execute 'ruby
> scm_benchmark.rb'. And that's it.
>
> So ultimately I urge someone with more patience and two solid machines to
> give my little test a spin. Please don't make any decision based on my da=
ta.
> Pretty please! And Carsten, if this doesn't turn you on, I'm sorry ;)
>
> Cheers,
> Eugen.
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronim=
o
> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=
=3D121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
>

On Wed, 16 Aug 2006, Carsten Haitzler (The Rasterman) wrote:
>
> i think i should check git. i know svn enough to know its a viable move from
> cvs - but i know very little of git - but from the comments - it seems good
> (these days) - though much more radical than  move to svn :)

Then try xcb :) (http://freedesktop.org/wiki/UsingGit)

Vincent

> now if you can - try "slowing down" the server.

I'm running a box I found in the trash, with a "designed for windows
95" sticker on the side; I would hope that's slow enough :P

On the down side, cvs2svn is taking ages -- it's currently translating
cvs commit 3000 / 25000 after about 3 hours... (the only *problem*
encountered with conversion is that two people had used the name "a"
for different branches / tags)

    -- Shish

Dènis Riedijk escreveu:
> What I can say is that from a usability point of view it is very fast 
> (that was Linus' number one design rule when he wrote git)
> For download performance it is probably important to pack the 
> repository before checking out from a client. Pack files are supposed 
> to help a lot with regards to network performance if I understand 
> correct.
>
> Some links :
[snip]
Adding to that list, here's the mail from cairo when they converted to 
git, lot's of usefull info: 
http://lists.freedesktop.org/archives/cairo/2006-February/006335.html
I'm pretty sure some of that is outdated, since I remind posts changing 
some things and scripts created to handle things easier and then it was 
before some other things got fixed in git. I couldn't find if these 
where put somewhere, their wiki doesn't have anything about it.

Solerman

Carsten Haitzler (The Rasterman) wrote:
>
> i think i should check git. i know svn enough to know its a viable move from
> cvs - but i know very little of git - but from the comments - it seems good
> (these days) - though much more radical than  move to svn :)
>
>   
Of course one reason for using SVN rather than any other non-cvs system
is that it is almost exactly the same to a user. No radical updates - it
"just works" ;)

A

David Seikel wrote:
> On Tue, 15 Aug 2006 00:41:34 -0400 Michael Jennings <e-devel@...>
> wrote:
>
>   
>> FUD.  You can tag CVS without having a checkout.  (Try cvs -H rtag.)
>> And SVN doesn't even HAVE tagging.  It has copying, which contrary to
>> popular (SVN developer) belief, is NOT the same thing.  A (non-branch)
>> tag is a symbolic name assigned to a particular state of the
>> repository (i.e., a changeset).  It is not a copy.
>>     
>
> Tagging and branching, when was the last time either occurred to E?
>  
>   
Tagging may be a "symbolic name" in CVS but the svn "tagged copy" idea 
is pretty much the same - it is a snapshot of the repository at a point 
in time given a name.
Remember here that SVN uses light copy - it is not a copy of the actual 
repos at all - list a "link" very like cvs tags, when you think about it.

A

Hello again,

I was thinking that you might want to try something if you get a second machine up.

>From the man page of git-cvsimport:
"Imports a CVS repository into git. It will either create a new repository, or incrementally import into an existing one."

So you might want to try using a "git mirror" of what you guys actually have on CVS. Now personally, I think moving to git alltogether is much better then this "follow the cvs server" sort of thing. There are just a couple of things that are really cool about git (and any distributed scm) versus cvs or svn. 

For those of you who know git well enough, sorry for writing this stuff, but I imagine there's an equal number of devs who don't so I just wanted to give people an idea on what it's about.

1) Local commits. You clone a remote repository (you get the history and everything) and you can then commit to your own local copy of the repository. 

Not only that but you can do just about everything you would do with cvs locally without even needing access to the server. Then when you're done you can merge your superfanstasticincredible feature with the main repository.

2) Local tags and branches. Well, it's part of what I said above but it should get some special notice. You get to branch from what you just cloned and so on ...

3) It's easy to setup. Basically you don't need to set it up at all. You just copy the .git/ directory from your repository to a server in something like http://e.org/e.git and you're all done. Then anyone can mirror it very easily.

4) The repository is well compacted with git-repack. For example, the part of the repository I had was 135MB in size (e17 without the history). With git-repack it all shrunk down to 52MB. Thanks for pointing that out, I wasn't aware of it.

5) The server seems to handle the load quite well, even though downloads may be a tad slower. Ultimately, the compression might turn out to make retrieving faster, and it is probably 

Now I know those so called benchmarks of mine don't indicate these but I have strong doubts about the conditions in which they are run. I believe git to be much faster then cvs if properly set up. And I've seen first hand that it doesn't beat the server to death on 20 simultaneous checkouts.

6) It can be mirrored and offered for retrieval in a variety of methods. There's git:// rsync:// and http:// and it can probably be setup with other webservers like lighttpd (though my attempts to use lighttpd with a git repo misteriously failed).

7) A larger emphasys on branching may actually be good. You might eventually want to create a more stable branch, or you might want to try to rewrite some library in the EFL, in a new cooler way or whatever.

Ultimately that's my best argument for it. With git you can do stuff like that in a more natural way and I generally found that it offers an almost unlimited degree of freedom. 

I tend to believe it's more efficient but I can't really back my claim up with consistent numbers. I have provided a way to test it but my hardware isn't cut out for that. I'm still waiting for someone else's benchmark...

But finally, the best way to test would be to use that second machine as a git mirror and see how well it behaves. You might even go as far as denying anoncvs for a while and telling people to use git for anonymous access instead. 

Well, anyway, you guys know better then I do. If you decide to keep on using CVS, I could make some sort of script to do CVS updates and make dist stuff on an event (like upon email notification or whatever), but it would have to be in Ruby. I can do that if you want to use SVN or Git as well. Your call. 

Cheers,
Eugen.

On Wed, 16 Aug 2006 14:51:39 +0100 Andrew Williams <andy@...>
babbled:

> Carsten Haitzler (The Rasterman) wrote:
> >
> > i think i should check git. i know svn enough to know its a viable move from
> > cvs - but i know very little of git - but from the comments - it seems good
> > (these days) - though much more radical than  move to svn :)
> >
> >   
> Of course one reason for using SVN rather than any other non-cvs system
> is that it is almost exactly the same to a user. No radical updates - it
> "just works" ;)

true - no radical learning curve and "back to work". of course - as i said - we
could have other options, like nuke anoncvs and replace it with server-side
generated tarballs.

> A
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

On Tuesday, 15 August 2006, at 14:48:36 (-0500),
Nathan Ingersoll wrote:

> Do we have any current statistics about the load on a CVS mirror?
> Might be easier to find more mirrors if we can give them some idea
> what sort of bandwidth, disk use and CPU load they can expect.

The repository currently requires just over 800MB.  The anoncvs server
has transmitted 206GB in the past 19 days, so about 11GB/day or so.
CPU load depends entirely on the box, but anoncvs is a P4 1.7 GHz with
15 minute load averages fairly consistently under 2.  And it's doing
several things, not just anoncvs.  So YMMV.



On Wednesday, 16 August 2006, at 08:43:22 (+0900),
Carsten Haitzler wrote:

> the compelling reason is that (apparently) anoncvs is straining
> under its load (again) and this happened so quickly that i am
> putting up a debate for alternate solutions - willing to discuss
> really anything :)

Not exactly.  The machine is doing fine; the only problem is that I
had to limit simultaneous CVS pserver connections to 1 per IP and 24
total to keep the box usable.  The result is that some people will
have their connections refused during peak times, but the machine
spends an awful lot of time doing nothing too, so unless everyone hits
it all at once, it's fine.

Would it help if I figured out the peak times for anoncvs and made
those public so that folks could schedule updates for other times?



On Tuesday, 15 August 2006, at 20:20:40 (-0400),
Kevin Brosius wrote:

> Hmmm. I'm not sure I see the logic that way.  We moved off sf to get
> the ability to do rsync backup of cvs (one of the reasons, along
> with some intermittent performance of dev and anon cvs.)  We now
> have solid dev cvs performance, rsync access, and better anon access
> (IMO.)

I agree.  I don't think it's quite time to throw in the towel.  I
think we just need one or two additional anoncvs boxes to share the
load.

> Are you against requesting offers of open anon mirrors?  With the
> rsync access, that seems easy, but of you want a anon cvs farm that
> has full status monitoring, then you're stepping the complexity up a
> bit.

As it stands now, anyone can run an anoncvs mirror simply by rsyncing
periodically.  If we had enough volunteers, we could create a mirror
list....

Michael

-- 
Michael Jennings (a.k.a. KainX)  http://www.kainx.org/  <mej@...>
n + 1, Inc., http://www.nplus1.net/       Author, Eterm (www.eterm.org)
-----------------------------------------------------------------------
 "May the forces of evil become confused on the way to your house."
                                                      -- George Carlin

* Eugen Minciu (minciue@...) wrote:
> I was thinking that you might want to try something if you get a
> second machine up.
> 
> >From the man page of git-cvsimport:
> "Imports a CVS repository into git. It will either create a new
> repository, or incrementally import into an existing one."
> 
> So you might want to try using a "git mirror" of what you guys
> actually have on CVS. Now personally, I think moving to git
> alltogether is much better then this "follow the cvs server" sort of
> thing. There are just a couple of things that are really cool about
> git (and any distributed scm) versus cvs or svn. 
> 
> For those of you who know git well enough, sorry for writing this
> stuff, but I imagine there's an equal number of devs who don't so I
> just wanted to give people an idea on what it's about.

Git is interesting and all, and may offer some development advantages,
but I keep harking back to 2 statements made earlier in this thread: 1)
that we're interested in performance, and 2) the developer base is
pretty small vs the userbase.

In terms of (1), git may run slightly faster in some cases, but you're
saying it's actually a little worse for checkouts (especially fresh
ones, which I would not be surprised to find most users doing as they
build or use Gentoo or whathaveyou).

More importantly, this isn't a thread about developers or even would-be
developers complaining it's too hard to do what they want to contribute
to the source tree due to the SCM choice. This is all about users
checking out code that's in development.

Personally I don't get why we're not creating nightly diffs and a
tarball once a week and letting people use that (it's not exactly
difficult to script). I highly doubt most users care about checkins in
the last few hours, and I suggested doing so I think way back in January
on IRC..

term

(I had a few minutes to read e-devel. It happens.)

On Thu, 17 Aug 2006 02:19:28 -0400
Lyle Kempler <term-e1@...> wrote:

> Git is interesting and all, and may offer some development advantages,
> but I keep harking back to 2 statements made earlier in this thread: 1)
> that we're interested in performance, and 2) the developer base is
> pretty small vs the userbase.
> 
> In terms of (1), git may run slightly faster in some cases, but you're
> saying it's actually a little worse for checkouts (especially fresh
> ones, which I would not be surprised to find most users doing as they
> build or use Gentoo or whathaveyou).

As I said, I'm not convinced that GIT is actually slower, there may be a bottleneck on the client's machine and packing data does seem to improve the process. For a single checkout Git/git takes 1:51 vs "so long I got bored and killed it" and Git/http takes 1:03 vs 1:38. 

But as I said, I'm quite convinced one of my machine's HDDs is actually the bottleneck, which is why I'm asking for someone to give us some other data.
 
> More importantly, this isn't a thread about developers or even would-be
> developers complaining it's too hard to do what they want to contribute
> to the source tree due to the SCM choice. This is all about users
> checking out code that's in development.

Oh, I never said it was. From what I can see you guys are quite happy working with CVS. But it's never a bad idea to look at the alternatives. 

Even if you're not finding it too hard to do what you want now, you should check to see if other systems will actually let you do more. 

I'm just saying, in theory git can probably do more. The development model can be almost the same as with CVS or it can evolve into something completely diffrent. 

Whether that makes sense for Enlightenment, or whether you really care is ultimately your decision. One should choose the tool which is best suited for his work.

Cheers,
Eugen.

On Wed, 16 Aug 2006 23:43:47 -0400
Michael Jennings <e-devel@...> wrote:

> CPU load depends entirely on the box, but anoncvs is a P4 1.7 GHz with
> 15 minute load averages fairly consistently under 2.  And it's doing

> Not exactly.  The machine is doing fine; the only problem is that I
> had to limit simultaneous CVS pserver connections to 1 per IP and 24
> total to keep the box usable.  The result is that some people will
> have their connections refused during peak times, but the machine
> spends an awful lot of time doing nothing too, so unless everyone hits
> it all at once, it's fine.

Any statistics on the load when 24 anon checkouts occur?

Cheers,
Eugen.

On 8/17/06, Lyle Kempler <term-e1@...> wrote:
> Personally I don't get why we're not creating nightly diffs and a
> tarball once a week and letting people use that (it's not exactly
> difficult to script). I highly doubt most users care about checkins in
> the last few hours, and I suggested doing so I think way back in January
> on IRC..
>

Actually they are. You'd be surprised how many people build E (and
related stuff) multiple times a day from anoncvs. Again, I agree with
the majority that is suggesting the mirrors idea. CVS has its quirks
here and there, but its been doing everything we wanted for a long
time. I think that having a 2 or 3 anoncvs mirrors should solve our
problems.

-- 
Hisham Mardam Bey
MSc (Computer Science)
http://hisham.cc/
+9613609386
Codito Ergo Sum (I Code Therefore I Am)

Hisham Mardam Bey wrote:
> On 8/17/06, Lyle Kempler <term-e1@...> wrote:
>   
>> Personally I don't get why we're not creating nightly diffs and a
>> tarball once a week and letting people use that (it's not exactly
>> difficult to script). I highly doubt most users care about checkins in
>> the last few hours, and I suggested doing so I think way back in January
>> on IRC..
>>
>>     
>
> Actually they are. You'd be surprised how many people build E (and
> related stuff) multiple times a day from anoncvs. Again, I agree with
> the majority that is suggesting the mirrors idea. CVS has its quirks
> here and there, but its been doing everything we wanted for a long
> time. I think that having a 2 or 3 anoncvs mirrors should solve our
> problems.

I'm agreeing Hisham, the way to got mirrors is better i think... If 
raster (etc...) wants to change cvs for svn, git or anything else isn't 
the problems. Having more than one server is always a good idea.

See ya CodeWaWa :)

On Thu, 17 Aug 2006 10:15:13 +0200 St=C3=A9phane Bauland <joligardon@...=
l.com>
babbled:

> Hisham Mardam Bey wrote:
> > On 8/17/06, Lyle Kempler <term-e1@...> wrote:
> >  =20
> >> Personally I don't get why we're not creating nightly diffs and a
> >> tarball once a week and letting people use that (it's not exactly
> >> difficult to script). I highly doubt most users care about checkins =
in
> >> the last few hours, and I suggested doing so I think way back in Jan=
uary
> >> on IRC..
> >>
> >>    =20
> >
> > Actually they are. You'd be surprised how many people build E (and
> > related stuff) multiple times a day from anoncvs. Again, I agree with
> > the majority that is suggesting the mirrors idea. CVS has its quirks
> > here and there, but its been doing everything we wanted for a long
> > time. I think that having a 2 or 3 anoncvs mirrors should solve our
> > problems.
>=20
> I'm agreeing Hisham, the way to got mirrors is better i think... If=20
> raster (etc...) wants to change cvs for svn, git or anything else isn't=
=20
> the problems. Having more than one server is always a good idea.
>=20
> See ya CodeWaWa :)

i am wanting to open a discussion to solutions. due to peak loads kainx h=
as had
to limit connection #'s - fair enough. this is bad. several anoncvs serve=
rs
will help. i think maybe we need to explore other options.

i am loathe to change scm's unless we have to. if we can avoid it - great=
. i do
NOT deny other scm's have benefits - can do things cvs can't etc. etc. - =
but
frankly - cvs does everything us developers need. we use only a subset of=
 cvs
too. we don't need the complex features in other scm's - nice though they=
 may
be. the point of this is to try and make anonymous "up to date" source co=
de
access scale better.

it seems that every scm has its positives and negatives. svn and cvs seem
evenly matched performance-wise, give or take. svn increases the local ch=
eckout
cost. git makes this even larger, but server-side is very light. of these=
 3
main contenders none stand out as a major leap forward for anon access.

so we are down to
1. anon cvs mirrors
2. another system (tarball snaps, etc. etc).

personally i would have no problem in a server-side auto-build of tarball=
s.
what do people think? should we perhaps have the anoncvs server do daily =
(or
maybe several times per day) builds of packages? not rpm or deb but "make
dist"; ie something like the following run maybe every 4 or 8 hours? once=
 a day?

# how much archive to keep in Kb
MAX=3D10000000
# date & time snap
DAT=3D`date '+%F_%T'`
# make the snap dir
mkdir -p /pub/snaps/$DAT
# latest link to automated snarfers can get the latest always
ln -sf /pub/snaps/$DAT /pub/snaps/LATEST
# get cvs
cvs co e17
cd ./e17
# for everything in the cvs tree with a configure.in
for I in `find . -name configure.in -print`; do
  DIR=3D`dirname $I`
  PRJ=3D`basename $DIR`
  pushd $DIR
  # just because it has a configure.in does not mean we want a snap - add=
 a
  # magic file to tell us we want to do the snap
  if [ -f ./.DO-SNAP ]; then
    ./autogen.sh
    make dist
    mv $PRJ-*.tar.gz /pub/snaps/$DAT/
  fi
  popd
done
# clean up the checkout
rm -rf ./e17
# clean up old snaps if our archive got to big
SIZ=3D`du -s /pub/snaps | awk '{printf("%s", $1);}'`
while [ $SIZ -gt $MAX ]; do
  DEL=3D`/bin/ls /pub/snaps | head -1`
  rm -rf /pub/snaps/$DEL
  SIZ=3D`du -s /pub/snaps | awk '{printf("%s", $1);}'`
done

We can now make "/pub/snaps" available over http/ftp etc. etc. - hell, ev=
en
generate .torrent seed files. maybe stop advertising anoncvs - and encour=
age
peoples "easy_e17.sh" or "get_e.sh" or emerges etc. to use this repositor=
y?
they won't need "autofoo" anymore as the configure etc. will already be
generated. this also catches broken packages - as they simply will refuse=
 to
build a dist tarball and be skipped. well badly broken - if they don't pa=
ss
make distcheck - well... we don't catch that (we can add it though)=20

what do you say? would people be willing to switch to such snaps?

--=20
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
=E8=A3=B8=E5=A5=BD=E5=A4=9A
Tokyo, Japan (=E6=9D=B1=E4=BA=AC =E6=97=A5=E6=9C=AC)

On 8/17/06, The Rasterman Carsten Haitzler <raster@...> wrote:
>
>
>
> personally i would have no problem in a server-side auto-build of
> tarballs.
> what do people think? should we perhaps have the anoncvs server do daily
> (or
> maybe several times per day) builds of packages? not rpm or deb but "make
> dist"; ie something like the following run maybe every 4 or 8 hours? once
> a day?


I think that while this option might be useful for load, it takes a lot more
bandwidth, and is bad for users, as they would have to download hundreds of
megabytes each time a 1 Kb patch is in.

I vote for the cvs mirrors as at least it's easier on the user's AND the
server's bandwidth.


-- 
Chady 'Leviathan' Kassouf
http://chady.net/

On 8/17/06, Chady Kassouf <chady.kassouf@...> wrote:
>
>
> On 8/17/06, The Rasterman Carsten Haitzler <raster@...> wrote:
> >
> >
> > personally i would have no problem in a server-side auto-build of
> tarballs.
> > what do people think? should we perhaps have the anoncvs server do daily
> (or
> > maybe several times per day) builds of packages? not rpm or deb but "make
> > dist"; ie something like the following run maybe every 4 or 8 hours? once
> a day?
>
>
> I think that while this option might be useful for load, it takes a lot more
> bandwidth, and is bad for users, as they would have to download hundreds of
> megabytes each time a 1 Kb patch is in.
>
> I vote for the cvs mirrors as at least it's easier on the user's AND the
> server's bandwidth.
>

I totally agree and second that vote.

-- 
Hisham Mardam Bey
MSc (Computer Science)
http://hisham.cc/
+9613609386
Codito Ergo Sum (I Code Therefore I Am)

On Thu, 17 Aug 2006 16:50:22 +0200 "Chady Kassouf"
<chady.kassouf@...> wrote:

> On 8/17/06, The Rasterman Carsten Haitzler <raster@...>
> wrote:
> >
> >
> >
> > personally i would have no problem in a server-side auto-build of
> > tarballs.
> > what do people think? should we perhaps have the anoncvs server do
> > daily (or
> > maybe several times per day) builds of packages? not rpm or deb but
> > "make dist"; ie something like the following run maybe every 4 or 8
> > hours? once a day?
>=20
>=20
> I think that while this option might be useful for load, it takes a
> lot more bandwidth, and is bad for users, as they would have to
> download hundreds of megabytes each time a 1 Kb patch is in.
>=20
> I vote for the cvs mirrors as at least it's easier on the user's AND
> the server's bandwidth.

We have two types of users, those that daily snapshots are good for,
and the cvs junkies.

Let cvs junkies rsync against the unpacked tarballs?

Snapshot it once a day, but build hourly patches against that days
snapshot?

Just throwing out some random ideas.

* Chady Kassouf (chady.kassouf@...) wrote:
> On 8/17/06, The Rasterman Carsten Haitzler <raster@...>
> wrote:
> >personally i would have no problem in a server-side auto-build of
> >tarballs.  what do people think? should we perhaps have the anoncvs
> >server do daily (or maybe several times per day) builds of packages?
> >not rpm or deb but "make dist"; ie something like the following run
> >maybe every 4 or 8 hours? once a day?
> I think that while this option might be useful for load, it takes a
> lot more bandwidth, and is bad for users, as they would have to
> download hundreds of megabytes each time a 1 Kb patch is in.
> 
> I vote for the cvs mirrors as at least it's easier on the user's AND
> the server's bandwidth.

Eventually I'll corner KainX and he'll tell me what setting I don't have
in mutt to read raster's emails other than through quotes.. anyway..

Were I suggesting merely a new tarball every day, then I'd agree. My
argument was for a weekly tarball and patches otherwise (and it doesn't
just have to be patches since the last weekly tarball). A lot of CPU
time and some bandwidth is consumed determining what changed since the
last time you sync'd up. If you _knew_ where that last sync point was
(e.g., yesterday's diff), then you'd be saving everyone a lot of
resources by just getting the pregenerated, precompressed version,
instead of making CVS do the diff, compressing on the fly (I'd like to
believe most people are using -z3 :)). As I read over and over that
people are using scripts to retrieve everything (be it ebuilds or
hand-rolled), this could easily be switched to.

* Hisham Mardam Bey (hisham.mardambey@...) wrote:
> On 8/17/06, Lyle Kempler <term-e1@...> wrote:
> > Personally I don't get why we're not creating nightly diffs and a
> > tarball once a week and letting people use that (it's not exactly
> > difficult to script). I highly doubt most users care about checkins
> > in the last few hours, and I suggested doing so I think way back in
> > January on IRC..
> Actually they are. You'd be surprised how many people build E (and
> related stuff) multiple times a day from anoncvs. Again, I agree with
> the majority that is suggesting the mirrors idea. CVS has its quirks
> here and there, but its been doing everything we wanted for a long
> time. I think that having a 2 or 3 anoncvs mirrors should solve our
> problems.

I am actually surprised that so many people are "a CVS junkie" :) ..
that's good feedback to have. I never advocated dumping the anoncvs
servers. I was saying that we could reduce demand (and therefore improve
performance for the remainder) by providing tarballs. And, as raster
said, there's nothing keeping us from producing patches every 4 hours,
etc..

term

On Thu, 17 Aug 2006 16:50:22 +0200 "Chady Kassouf" <chady.kassouf@...>
babbled:

> On 8/17/06, The Rasterman Carsten Haitzler <raster@...> wrote:
> >
> >
> >
> > personally i would have no problem in a server-side auto-build of
> > tarballs.
> > what do people think? should we perhaps have the anoncvs server do daily
> > (or
> > maybe several times per day) builds of packages? not rpm or deb but "make
> > dist"; ie something like the following run maybe every 4 or 8 hours? once
> > a day?
> 
> 
> I think that while this option might be useful for load, it takes a lot more
> bandwidth, and is bad for users, as they would have to download hundreds of
> megabytes each time a 1 Kb patch is in.
> 
> I vote for the cvs mirrors as at least it's easier on the user's AND the
> server's bandwidth.

actually - they can just download the tarball that changed (and that they
want) :)

frankly though - i think this will then discourage users to update as often as
they need to download more - and this will also be good. remember it also
removes all the "what version of autofoo will work" thing. they no longer need
to know nor care (and a "user" should never need to know or care or have any
autofoo even installed)

-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

On Fri, 18 Aug 2006 03:37:06 +1000 David Seikel <onefang@...> babbled:

> On Thu, 17 Aug 2006 16:50:22 +0200 "Chady Kassouf"
> <chady.kassouf@...> wrote:
> 
> > On 8/17/06, The Rasterman Carsten Haitzler <raster@...>
> > wrote:
> > >
> > >
> > >
> > > personally i would have no problem in a server-side auto-build of
> > > tarballs.
> > > what do people think? should we perhaps have the anoncvs server do
> > > daily (or
> > > maybe several times per day) builds of packages? not rpm or deb but
> > > "make dist"; ie something like the following run maybe every 4 or 8
> > > hours? once a day?
> > 
> > 
> > I think that while this option might be useful for load, it takes a
> > lot more bandwidth, and is bad for users, as they would have to
> > download hundreds of megabytes each time a 1 Kb patch is in.
> > 
> > I vote for the cvs mirrors as at least it's easier on the user's AND
> > the server's bandwidth.
> 
> We have two types of users, those that daily snapshots are good for,
> and the cvs junkies.
> 
> Let cvs junkies rsync against the unpacked tarballs?
> 
> Snapshot it once a day, but build hourly patches against that days
> snapshot?
> 
> Just throwing out some random ideas.

i like the idea of rsync'ed tarball trees (ie do the tarball snap - then do an
unpack of them somewhere and offer that unpacked tree as a public rsync server).


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

On Thu, 17 Aug 2006 16:45:19 -0400 Lyle Kempler <term-e1@...>
babbled:

> * Chady Kassouf (chady.kassouf@...) wrote:
> > On 8/17/06, The Rasterman Carsten Haitzler <raster@...>
> > wrote:
> > >personally i would have no problem in a server-side auto-build of
> > >tarballs.  what do people think? should we perhaps have the anoncvs
> > >server do daily (or maybe several times per day) builds of packages?
> > >not rpm or deb but "make dist"; ie something like the following run
> > >maybe every 4 or 8 hours? once a day?
> > I think that while this option might be useful for load, it takes a
> > lot more bandwidth, and is bad for users, as they would have to
> > download hundreds of megabytes each time a 1 Kb patch is in.
> > 
> > I vote for the cvs mirrors as at least it's easier on the user's AND
> > the server's bandwidth.
> 
> Eventually I'll corner KainX and he'll tell me what setting I don't have
> in mutt to read raster's emails other than through quotes.. anyway..
> 
> Were I suggesting merely a new tarball every day, then I'd agree. My
> argument was for a weekly tarball and patches otherwise (and it doesn't
> just have to be patches since the last weekly tarball). A lot of CPU
> time and some bandwidth is consumed determining what changed since the
> last time you sync'd up. If you _knew_ where that last sync point was
> (e.g., yesterday's diff), then you'd be saving everyone a lot of
> resources by just getting the pregenerated, precompressed version,
> instead of making CVS do the diff, compressing on the fly (I'd like to
> believe most people are using -z3 :)). As I read over and over that
> people are using scripts to retrieve everything (be it ebuilds or
> hand-rolled), this could easily be switched to.
> 
> * Hisham Mardam Bey (hisham.mardambey@...) wrote:
> > On 8/17/06, Lyle Kempler <term-e1@...> wrote:
> > > Personally I don't get why we're not creating nightly diffs and a
> > > tarball once a week and letting people use that (it's not exactly
> > > difficult to script). I highly doubt most users care about checkins
> > > in the last few hours, and I suggested doing so I think way back in
> > > January on IRC..
> > Actually they are. You'd be surprised how many people build E (and
> > related stuff) multiple times a day from anoncvs. Again, I agree with
> > the majority that is suggesting the mirrors idea. CVS has its quirks
> > here and there, but its been doing everything we wanted for a long
> > time. I think that having a 2 or 3 anoncvs mirrors should solve our
> > problems.
> 
> I am actually surprised that so many people are "a CVS junkie" :) ..
> that's good feedback to have. I never advocated dumping the anoncvs
> servers. I was saying that we could reduce demand (and therefore improve
> performance for the remainder) by providing tarballs. And, as raster
> said, there's nothing keeping us from producing patches every 4 hours,
> etc..

agreed. i am not advocating REMOVING anoncvs as such - just removing it as a
"first priority anonymous source access" mechanism. putting anoncvs access info
into fineprint in some obscure page linked off a download page. moving the
"primary" access to something more server-friendly that matches people's needs
better. people dont NEED cvs. developers do. they simply need a way to "get the
latest code". also for those speaking of the bandwidth issues - a LOT of people
DONT keep their cvs trees after build they re-checkout a new tree.

so what we need to do is try alleviate the need for anoncvs mirrors and if
there are mirrors - make them "last longer" bye making an alternate way of
accessing updates available that gives people what they really need in a way
that is light on our resources, and thus can scale better.

imho making dist tarballs is also a good testing mechanism. that is our "end
product" anyway - a cvs tree where you ned to run autogen.sh (autofoo) is NOT
our "target product". it is an intermediate state developers can easily work
with. end users will work with the result of make dist and then do the usual
tar -xf file.tar.gz; cd file; ./configure && make && make install

if we move and encourage the primary mechanism to be this with regular enough
updates that people aren't waiting impatiently for that update/fix - then we
may solve a lot of our problems?

-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

On Friday, 18 August 2006, at 08:21:17 (+0900),
Carsten Haitzler wrote:

> agreed. i am not advocating REMOVING anoncvs as such - just removing
> it as a "first priority anonymous source access" mechanism. putting
> anoncvs access info into fineprint in some obscure page linked off a
> download page. moving the "primary" access to something more
> server-friendly that matches people's needs better.

Unfortunately, the only thing this will *actually* accomplish is to
alter the #1 question in #E from "Where does raster get all those cute
inflatable sheep??" to "How do I get the latest E from CVS?"  :-(

> people dont NEED cvs. developers do. they simply need a way to "get
> the latest code". also for those speaking of the bandwidth issues -
> a LOT of people DONT keep their cvs trees after build they
> re-checkout a new tree.

Maybe someone should write a cvs pserver patch which disallows
checkouts and only permits updates.  That would alleviate that problem
quite quickly.  :-)

Michael

-- 
Michael Jennings (a.k.a. KainX)  http://www.kainx.org/  <mej@...>
n + 1, Inc., http://www.nplus1.net/       Author, Eterm (www.eterm.org)
-----------------------------------------------------------------------
 "Men are not subtle.  We are obvious.  Women know what men want.  Men
  know what men want.  What do we want?  We want women!  That's it.
  It's the only thing we know for sure."             -- Jerry Seinfeld

Michael Jennings wrote:
> On Friday, 18 August 2006, at 08:21:17 (+0900),
> Carsten Haitzler wrote:
>
>   
>> agreed. i am not advocating REMOVING anoncvs as such - just removing
>> it as a "first priority anonymous source access" mechanism. putting
>> anoncvs access info into fineprint in some obscure page linked off a
>> download page. moving the "primary" access to something more
>> server-friendly that matches people's needs better.
>>     
>
> Unfortunately, the only thing this will *actually* accomplish is to
> alter the #1 question in #E from "Where does raster get all those cute
> inflatable sheep??" to "How do I get the latest E from CVS?"  :-(
>   
^^^ cute inflatable sheep? i wanna know where he gets them really :)
>   
<snip>

On Thursday, 17 August 2006, at 11:15:50 (+0300),
Eugen Minciu wrote:

> Any statistics on the load when 24 anon checkouts occur?

Not specifically, but over the past 24 hours, the max 1 minute load
average was 17.55, and the max 15 minute load average was 8.56.

Looks like 1800-2200 UTC is our peak time frame.

Michael

-- 
Michael Jennings (a.k.a. KainX)  http://www.kainx.org/  <mej@...>
n + 1, Inc., http://www.nplus1.net/       Author, Eterm (www.eterm.org)
-----------------------------------------------------------------------
 "The first lesson reading teaches is how to be alone."
                                                   -- Jonathan Franzen

Hi everyone.

I've been doing some thinking today. And I've been doing some testing as well. And there are a few things I realized.

The first thing I realized is a reason why the pseudo-benchmark I created was giving out evil data. In git's case this is because git does a lot of extra operations on the client's disk (unpacking and such) which take up a lot of time. During that time the server wouldn't be under any load, which shows why the load on the server wasn't anywhere near constant.

And then I realized why git wasn't really doing so well. And at the same time this is the reason why cvs isn't doing so well and, frankly no scm possibly could.

The problem is that you have a truckload of binary data in the repository. There are many reasons why this shouldn't be so.

1) Binary data is way better off distributed in the form of archives, that can be mirrored by anyone (I'm thinking at least SF). That way people can get that data a lot faster and your server is happy too.

2) You don't change the binary data that much. And even when you do so, you could pacakge your data into archives like imlib2-data.tar.bz2 so that you repackage less.

3) Changes in binary data don't generally affect dependencies. They're not like API changes or whatever. Most of the time people will just need to grab one updated archive and that's it.

4) You could then use pkg-config to ensure the right version of the data is actually installed from your configure scripts.

5) Let's do some simple math. 

You have 100MB worth of files. These account to 60MB binary and 40MB text. When you try to compress this, as git does, you get around 50MB binary and 8MB test. So that accounts for almost 60MB. 

That means for every 60 people that would simultaneously download through CVS you can have 100 download through git (let's just ignore the other factors and focus on bandwith a little).

Now suppose you have 40MB of text. With git you can then down to about 20% of the original size (maybe less, who knows). That means you could (in theory) actually have 5 times more downloads with git then with CVS.

Now I'm not saying to not keep that data in a repo. You obviously have to. I'm just saying there's no need for people to have anonymous access to that repo, it could be for developers only.

So, my suggestions are:
1) Move the data into its own repository
2) Convert the two repositories to git
3) Make that data repository devel-only.
4) Split the data into small packages (one for each data/ dir in the tree, I guess)
5) Make the source require the data through pkg-config
6) Have the data released as tarballs once it's changed (you can have that happen automatically with git, I'm assuming you can with the others as well)

And that's it. But for all this babbling, is this really worth it? 
Like I said, I found client-side disk I/O to make the benchmarks mostly useless. But they still provide me with a good overview on server-side CPU & Memory usage

So I opted for a new approach. I would have two terminals on my client. In one I'd do something like 'sleep 5 ; svn checkout ...'. In the second I'd do 'time read'. I would press enter once when network traffic actually began and once again when it stopped and that showed me how much everything took.

So here's the timings. The repos have no history attatched.

Repo with data:
CVS: 			0:46
SVN(svnserve): 		1:16
SVN(HTTP): 		1:58
GIT(git): 		1:23
GIT(HTTP): 		1:53

Same repo without data:
CVS: 			0:12
SVN(svnserve):		0:28
SVN(http):		0:37
GIT(http):		0:13

And what about Git with its built in protocol? Just six seconds. How's that for taking some load off :) Of course you have to add/substract 1s for my timings on the keyboard but you get the overall idea.

This is a very complicated way of doing things. But data should probably be separated from code. And it should probably be distributed in small archives. And people shouldn't have to use an SCM to get it.

So ... Wadda ya say. Is this too complicated/ not worth it / stupid / braindamaged / interesting ?

My brain farts more things like that on a regular basis. If the above makes sense, let me know and I'll give you a couple of other ideas as well :d

Eugen.

P.S: I knew Linus wouldn't lie ;)

Chady Kassouf wrote:
> On 8/17/06, The Rasterman Carsten Haitzler <raster@...> wrote:
>>
>>
>>
>> personally i would have no problem in a server-side auto-build of
>> tarballs.
>> what do people think? should we perhaps have the anoncvs server do daily
>> (or
>> maybe several times per day) builds of packages? not rpm or deb but "make
>> dist"; ie something like the following run maybe every 4 or 8 hours? once
>> a day?
> 
> 
> I think that while this option might be useful for load, it takes a lot
> more
> bandwidth, and is bad for users, as they would have to download hundreds of
> megabytes each time a 1 Kb patch is in.
> 
> I vote for the cvs mirrors as at least it's easier on the user's AND the
> server's bandwidth.
> 
forget the tarballs.  As a user I vote also for the cvs mirrors.

On Fri, 18 Aug 2006 06:04:37 +0300 Eugen Minciu <minciue@...> babbl=
ed:

> Hi everyone.
>=20
> I've been doing some thinking today. And I've been doing some testing a=
s
> well. And there are a few things I realized.
>=20
> The first thing I realized is a reason why the pseudo-benchmark I creat=
ed was
> giving out evil data. In git's case this is because git does a lot of e=
xtra
> operations on the client's disk (unpacking and such) which take up a lo=
t of
> time. During that time the server wouldn't be under any load, which sho=
ws why
> the load on the server wasn't anywhere near constant.
>=20
> And then I realized why git wasn't really doing so well. And at the sam=
e time
> this is the reason why cvs isn't doing so well and, frankly no scm poss=
ibly
> could.
>=20
> The problem is that you have a truckload of binary data in the reposito=
ry.
> There are many reasons why this shouldn't be so.
>=20
> 1) Binary data is way better off distributed in the form of archives, t=
hat
> can be mirrored by anyone (I'm thinking at least SF). That way people c=
an get
> that data a lot faster and your server is happy too.
>=20
> 2) You don't change the binary data that much. And even when you do so,=
 you
> could pacakge your data into archives like imlib2-data.tar.bz2 so that =
you
> repackage less.
>=20
> 3) Changes in binary data don't generally affect dependencies. They're =
not
> like API changes or whatever. Most of the time people will just need to=
 grab
> one updated archive and that's it.
>=20
> 4) You could then use pkg-config to ensure the right version of the dat=
a is
> actually installed from your configure scripts.
>=20
> 5) Let's do some simple math.=20
>=20
> You have 100MB worth of files. These account to 60MB binary and 40MB te=
xt.
> When you try to compress this, as git does, you get around 50MB binary =
and
> 8MB test. So that accounts for almost 60MB.=20
>=20
> That means for every 60 people that would simultaneously download throu=
gh CVS
> you can have 100 download through git (let's just ignore the other fact=
ors
> and focus on bandwith a little).
>=20
> Now suppose you have 40MB of text. With git you can then down to about =
20% of
> the original size (maybe less, who knows). That means you could (in the=
ory)
> actually have 5 times more downloads with git then with CVS.
>=20
> Now I'm not saying to not keep that data in a repo. You obviously have =
to.
> I'm just saying there's no need for people to have anonymous access to =
that
> repo, it could be for developers only.
>=20
> So, my suggestions are:
> 1) Move the data into its own repository

not going to happen. the data is an internal part of the projects - it ge=
ts
modifed 8new icons, images etc.) and is part of the build process. so not=
 going
to happen. the code is useless without the data - there is no point split=
ting
it and doing so is a tonne of work that makes building more painful for
developers and users.

> 2) Convert the two repositories to git
> 3) Make that data repository devel-only.
> 4) Split the data into small packages (one for each data/ dir in the tr=
ee, I
> guess)
> 5) Make the source require the data through pkg-config
> 6) Have the data released as tarballs once it's changed (you can have t=
hat
> happen automatically with git, I'm assuming you can with the others as =
well)

at this point - why bother with git at all. just ake tarball snaps. much =
less
effort.

> And that's it. But for all this babbling, is this really worth it?=20
> Like I said, I found client-side disk I/O to make the benchmarks mostly
> useless. But they still provide me with a good overview on server-side =
CPU &
> Memory usage
>=20
> So I opted for a new approach. I would have two terminals on my client.=
 In
> one I'd do something like 'sleep 5 ; svn checkout ...'. In the second I=
'd do
> 'time read'. I would press enter once when network traffic actually beg=
an and
> once again when it stopped and that showed me how much everything took.
>=20
> So here's the timings. The repos have no history attatched.
>=20
> Repo with data:
> CVS: 			0:46
> SVN(svnserve): 		1:16
> SVN(HTTP): 		1:58
> GIT(git): 		1:23
> GIT(HTTP): 		1:53
>=20
> Same repo without data:
> CVS: 			0:12
> SVN(svnserve):		0:28
> SVN(http):		0:37
> GIT(http):		0:13
>=20
> And what about Git with its built in protocol? Just six seconds. How's =
that
> for taking some load off :) Of course you have to add/substract 1s for =
my
> timings on the keyboard but you get the overall idea.
>=20
> This is a very complicated way of doing things. But data should probabl=
y be
> separated from code. And it should probably be distributed in small arc=
hives.
> And people shouldn't have to use an SCM to get it.
>=20
> So ... Wadda ya say. Is this too complicated/ not worth it / stupid /
> braindamaged / interesting ?
>=20
> My brain farts more things like that on a regular basis. If the above m=
akes
> sense, let me know and I'll give you a couple of other ideas as well :d
>=20
> Eugen.
>=20
> P.S: I knew Linus wouldn't lie ;)

though git seems nice - i am beginning to think its not going to solve a =
lot. we
need to really just provide alternate mechanisms to get the code and moe
anoncvs mirros i think.

> -----------------------------------------------------------------------=
--
> Using Tomcat but need to do more? Need to support web services, securit=
y?
> Get stuff done quickly with pre-integrated technology to make your job =
easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geron=
imo
> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=
=3D121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
>=20


--=20
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
=E8=A3=B8=E5=A5=BD=E5=A4=9A
Tokyo, Japan (=E6=9D=B1=E4=BA=AC =E6=97=A5=E6=9C=AC)

On Fri, 18 Aug 2006 14:24:45 +0900
Carsten Haitzler (The Rasterman) <raster@...> wrote:

> > 1) Move the data into its own repository

> 
> not going to happen. the data is an internal part of the projects - it gets
> modifed 8new icons, images etc.) and is part of the build process. so not going
> to happen. the code is useless without the data - there is no point splitting
> it and doing so is a tonne of work that makes building more painful for
> developers and users.
> 
I can agree to the fact that it is hard. But it's only hard because it would take some amount of time to perform. The process of doing so isn't that complex.

Yes, the code does require the data so ...
> > 5) Make the source require the data through pkg-config

Secondly, I looked around to see just how much data was modified in e17 in the last month (with git you could have e16 as a separate repo, as I doubt it generates that much traffic).

So here it is:
apps/e: 8 modifications
proto/etk: 8 modifications
libs/ewl: 6 modifications
proto/estickies: 6 modifications
apps/entrance: 4 modifications
proto/emphasis: 4 modifications
apps/elicit: 2 modifications
apps/e_utils: 1 modification
libs/edje: 1 modification

That's it. Now how many modification did the source code itself receive during this period? 

> at this point - why bother with git at all. just ake tarball snaps. much less
> effort.

Things would get up to 7 times faster then they are now, it's a lot easier to make multiple repositories, you don't have to mess with .cvsignore, and because it sends compressed data over the network you use up less bandwidth (if that's of any concern). It's a lot easier to mirror too.
 
> 
> though git seems nice - i am beginning to think its not going to solve a lot. we
> need to really just provide alternate mechanisms to get the code and moe
> anoncvs mirros i think.
> 

At this point I doubt you guys agree with my view. So, my opinion is probably wrong and this is just a case of me being hard-headed. It would take about a week to get everything into place and there are probably many easier ways to get the same results.

Eugen.

On Fri, 18 Aug 2006 12:43:06 +0300 Eugen Minciu <minciue@...> babbled:

> On Fri, 18 Aug 2006 14:24:45 +0900
> Carsten Haitzler (The Rasterman) <raster@...> wrote:
> 
> > > 1) Move the data into its own repository
> 
> > 
> > not going to happen. the data is an internal part of the projects - it gets
> > modifed 8new icons, images etc.) and is part of the build process. so not
> > going to happen. the code is useless without the data - there is no point
> > splitting it and doing so is a tonne of work that makes building more
> > painful for developers and users.
> > 
> I can agree to the fact that it is hard. But it's only hard because it would
> take some amount of time to perform. The process of doing so isn't that
> complex.
> 
> Yes, the code does require the data so ...
> > > 5) Make the source require the data through pkg-config
> 
> Secondly, I looked around to see just how much data was modified in e17 in
> the last month (with git you could have e16 as a separate repo, as I doubt it
> generates that much traffic).
> 
> So here it is:
> apps/e: 8 modifications
> proto/etk: 8 modifications
> libs/ewl: 6 modifications
> proto/estickies: 6 modifications
> apps/entrance: 4 modifications
> proto/emphasis: 4 modifications
> apps/elicit: 2 modifications
> apps/e_utils: 1 modification
> libs/edje: 1 modification

and in the month before that more - not to mention data additions. and there
will be a whole bunch more soon.

> That's it. Now how many modification did the source code itself receive
> during this period? 
> 
> > at this point - why bother with git at all. just ake tarball snaps. much
> > less effort.
> 
> Things would get up to 7 times faster then they are now, it's a lot easier to
> make multiple repositories, you don't have to mess with .cvsignore, and
> because it sends compressed data over the network you use up less bandwidth
> (if that's of any concern). It's a lot easier to mirror too. 

would they get 7 times faster? with all of the history in the git repo too?

> > 
> > though git seems nice - i am beginning to think its not going to solve a
> > lot. we need to really just provide alternate mechanisms to get the code
> > and moe anoncvs mirros i think.
> > 
> 
> At this point I doubt you guys agree with my view. So, my opinion is probably
> wrong and this is just a case of me being hard-headed. It would take about a
> week to get everything into place and there are probably many easier ways to
> get the same results.
> 
> Eugen.
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

OK. I'm done supporting git now ;) Time for other ideas.
>From what has allready been presented I find anoncvs mirrors as a better approach to tarballs, as many of us actually don't have the bandwith to download a dist tarball every week.

CVS has hooks from what I can see.Yay. So after every commit we could set something up on the server to:

Idea 1:
1) export the directory main directory (like apps/e)
2) make an archive of the export.

Idea 2:
The zip utility can update only a part of the archive that has changed. You could run something like zip -u e17 path/to/e17. I made a zip of the whole e17 module, changed a single file then ran update. It ran and updated everything in 11.5 seconds on my laptop. On your machine it could probably happen in less then 5. 

Again, you could combine this with the above and generate archives like apps_entrance.zip or whatever, which would update in less then a second.

Thus, you could store just one version of the file at all times. People don't get confused with versions, they're always in sync with the repo and they can easily check if they're up to date with the help of md5 hashes. I could even write you some script which checks to see if a user is up to date and downloads the modified archives.

Cheers,
Eugen.

On 8/18/06, Eugen Minciu <minciue@...> wrote:
> OK. I'm done supporting git now ;) Time for other ideas.
> >From what has allready been presented I find anoncvs mirrors as a better approach to tarballs, as many of us actually don't have the bandwith to download a dist tarball every week.
>
> CVS has hooks from what I can see.Yay. So after every commit we could set something up on the server to:
>
> Idea 1:
> 1) export the directory main directory (like apps/e)
> 2) make an archive of the export.
>
> Idea 2:
> The zip utility can update only a part of the archive that has changed. You could run something like zip -u e17 path/to/e17. I made a zip of the whole e17 module, changed a single file then ran update. It ran and updated everything in 11.5 seconds on my laptop. On your machine it could probably happen in less then 5.
>
> Again, you could combine this with the above and generate archives like apps_entrance.zip or whatever, which would update in less then a second.
>
> Thus, you could store just one version of the file at all times. People don't get confused with versions, they're always in sync with the repo and they can easily check if they're up to date with the help of md5 hashes. I could even write you some script which checks to see if a user is up to date and downloads the modified archives.
>

Except that this would cause the file to be redownloaded every time a
change happens. This will eat up far more bandwidth than using an anon
repo (anoncvs, assuming users update) or with a base tarball with diff
tarballs.

(Just a random note, both Portage (Gentoo) and BitBake (OpenEmbedded)
keep the checked out CVS tree around and simply update it for a new
build. They also keep around the source tarballs that they download,
although some people delete them for more space (although they of
course shouldn't).)

-- 
Justin Patrin

On Thu, 17 Aug 2006 22:55:02 -0400
Michael Jennings <e-devel@...> wrote:

> On Thursday, 17 August 2006, at 11:15:50 (+0300),
> Eugen Minciu wrote:
> 
> > Any statistics on the load when 24 anon checkouts occur?
> 
> Not specifically, but over the past 24 hours, the max 1 minute load
> average was 17.55, and the max 15 minute load average was 8.56.
> 
> Looks like 1800-2200 UTC is our peak time frame.
> 

Is there anyway we can tell if the main problem is new checkouts or updates?  I ask because it seems the main problem is new checkouts and cpu seems less of a problem than bandwidth.  At least it seems that way as I have read this thread.

Here is my thinking on the matter.  I update my repository about 3 times a week.  If this kind of use is OK?  Then my proposal could be something along these lines.  

    1.  Weekly cvs tarbals.
    2.  anon cvs with checkout disabled. cvs only has update ability ( so users can update the tarball only). Tarballs could be bittorent pushed if wanted.
    3.  Have a "recomended" update script downloadable from one of the sites get-e maybe.

or if you wanted to go really crazy.  Require login for anon and sign up for a login on get-e maybe.  Do some minor hacking of cvs,git,svn or something that provides the following.

    1.  use sqlite or berklydb to track users a little.
    2.  limits the number of checkouts.  (say one per 3 months)
    3.  limits the number of updates.  (say 3 per week :-))

The above has the disadvantage of not scaling well across mutiple cvs mirrors.  However if the web site knows about the cvs mirrors it can send new users to a mirror say round robin or all mirrors can have a full user list updated by the web server say rsync style or even scp on a timer.

If your are really interested in one of these options I can see about making the changes to cvs etc.  There maybe many other reasons this is not practical just thought I would share.

As a casual user I really don't want to lose cvs update but I understand if that is what is needed (not using gentoo btw).  I watch this project very closely though I have not been envolved much ( 2 patches weee ).  Anyway I hope this is a help not hinderance to this discussion.

Frederick

On Friday, 18 August 2006, at 12:43:06 (+0300),
Eugen Minciu wrote:

> At this point I doubt you guys agree with my view. So, my opinion is
> probably wrong and this is just a case of me being hard-headed.

It's not a question of right or wrong.  It's a question of return on
investment.  Making these changes would be very timeconsuming, not to
mention the alterations it would require to automated build scripts
and the like.  And I don't think we'd gain much from it; CVS handles
binary files just fine.  They only affect disk space in the repo, not
bandwidth.

Michael

-- 
Michael Jennings (a.k.a. KainX)  http://www.kainx.org/  <mej@...>
n + 1, Inc., http://www.nplus1.net/       Author, Eterm (www.eterm.org)
-----------------------------------------------------------------------
 "I've been looking for a Savior in these dirty streets, 
  Looking for a Savior beneath these dirty sheets." 
                                               -- Tori Amos, "Crucify"

On Fri, 18 Aug 2006 10:39:42 -0500 Frederick Reeve <cylix@...> babbled:

> On Thu, 17 Aug 2006 22:55:02 -0400
> Michael Jennings <e-devel@...> wrote:
> 
> > On Thursday, 17 August 2006, at 11:15:50 (+0300),
> > Eugen Minciu wrote:
> > 
> > > Any statistics on the load when 24 anon checkouts occur?
> > 
> > Not specifically, but over the past 24 hours, the max 1 minute load
> > average was 17.55, and the max 15 minute load average was 8.56.
> > 
> > Looks like 1800-2200 UTC is our peak time frame.
> > 
> 
> Is there anyway we can tell if the main problem is new checkouts or updates?
> I ask because it seems the main problem is new checkouts and cpu seems less
> of a problem than bandwidth.  At least it seems that way as I have read this
> thread.
> 
> Here is my thinking on the matter.  I update my repository about 3 times a
> week.  If this kind of use is OK?  Then my proposal could be something along
> these lines.  
> 
>     1.  Weekly cvs tarbals.
>     2.  anon cvs with checkout disabled. cvs only has update ability ( so
> users can update the tarball only). Tarballs could be bittorent pushed if
> wanted.
>     3.  Have a "recomended" update script downloadable from one of the sites
> get-e maybe.
> 
> or if you wanted to go really crazy.  Require login for anon and sign up for
> a login on get-e maybe.  Do some minor hacking of cvs,git,svn or something
> that provides the following.
> 
>     1.  use sqlite or berklydb to track users a little.
>     2.  limits the number of checkouts.  (say one per 3 months)
>     3.  limits the number of updates.  (say 3 per week :-))

requiring a login would also allow us to track users more easily - especially
"problem" users - as you say, but we would need infrastructure to have people
create logins by themselves (a web page most likely) - not keen on doing that
myself.

> The above has the disadvantage of not scaling well across mutiple cvs
> mirrors.  However if the web site knows about the cvs mirrors it can send new
> users to a mirror say round robin or all mirrors can have a full user list
> updated by the web server say rsync style or even scp on a timer.
> 
> If your are really interested in one of these options I can see about making
> the changes to cvs etc.  There maybe many other reasons this is not practical
> just thought I would share.
> 
> As a casual user I really don't want to lose cvs update but I understand if
> that is what is needed (not using gentoo btw).  I watch this project very
> closely though I have not been envolved much ( 2 patches weee ).  Anyway I
> hope this is a help not hinderance to this discussion.
> 
> Frederick
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)

* Carsten Haitzler (raster@...) wrote:
> On Thu, 17 Aug 2006 16:45:19 -0400 Lyle Kempler
> <term-e1@...> babbled:
> > * Chady Kassouf (chady.kassouf@...) wrote:
> > > On 8/17/06, The Rasterman Carsten Haitzler <raster@...>
> > > wrote:
> > > >personally i would have no problem in a server-side auto-build of
> > > >tarballs.  what do people think? should we perhaps have the
> > > >anoncvs server do daily (or maybe several times per day) builds
> > > >of packages?  not rpm or deb but "make dist"; ie something like
> > > >the following run maybe every 4 or 8 hours? once a day?
> > > I think that while this option might be useful for load, it takes
> > > a lot more bandwidth, and is bad for users, as they would have to
> > > download hundreds of megabytes each time a 1 Kb patch is in.
> > > 
> > > I vote for the cvs mirrors as at least it's easier on the user's
> > > AND the server's bandwidth.
> > Eventually I'll corner KainX and he'll tell me what setting I don't
> > have in mutt to read raster's emails other than through quotes..
> > anyway..
> > 
> > Were I suggesting merely a new tarball every day, then I'd agree. My
> > argument was for a weekly tarball and patches otherwise (and it
> > doesn't just have to be patches since the last weekly tarball). A
> > lot of CPU time and some bandwidth is consumed determining what
> > changed since the last time you sync'd up. If you _knew_ where that
> > last sync point was (e.g., yesterday's diff), then you'd be saving
> > everyone a lot of resources by just getting the pregenerated,
> > precompressed version, instead of making CVS do the diff,
> > compressing on the fly (I'd like to believe most people are using
> > -z3 :)). As I read over and over that people are using scripts to
> > retrieve everything (be it ebuilds or hand-rolled), this could
> > easily be switched to.
> > 
> > * Hisham Mardam Bey (hisham.mardambey@...) wrote:
> > > On 8/17/06, Lyle Kempler <term-e1@...> wrote:
> > > > Personally I don't get why we're not creating nightly diffs and
> > > > a tarball once a week and letting people use that (it's not
> > > > exactly difficult to script). I highly doubt most users care
> > > > about checkins in the last few hours, and I suggested doing so I
> > > > think way back in January on IRC..
> > > Actually they are. You'd be surprised how many people build E (and
> > > related stuff) multiple times a day from anoncvs. Again, I agree
> > > with the majority that is suggesting the mirrors idea. CVS has its
> > > quirks here and there, but its been doing everything we wanted for
> > > a long time. I think that having a 2 or 3 anoncvs mirrors should
> > > solve our problems.
> > I am actually surprised that so many people are "a CVS junkie" :) ..
> > that's good feedback to have. I never advocated dumping the anoncvs
> > servers. I was saying that we could reduce demand (and therefore
> > improve performance for the remainder) by providing tarballs. And,
> > as raster said, there's nothing keeping us from producing patches
> > every 4 hours, etc..
> agreed. i am not advocating REMOVING anoncvs as such - just removing
> it as a "first priority anonymous source access" mechanism. putting
> anoncvs access info into fineprint in some obscure page linked off a
> download page. moving the "primary" access to something more
> server-friendly that matches people's needs better. people dont NEED
> cvs. developers do. they simply need a way to "get the latest code".
> also for those speaking of the bandwidth issues - a LOT of people DONT
> keep their cvs trees after build they re-checkout a new tree.
> 
> so what we need to do is try alleviate the need for anoncvs mirrors
> and if there are mirrors - make them "last longer" bye making an
> alternate way of accessing updates available that gives people what
> they really need in a way that is light on our resources, and thus can
> scale better.

Looking at get-e.org, the download page points to the E17 User's Guide,
which starts with "Installing from CVS". If we instead changed the
download page to "download this script and run it, you'll need wget and
tar and bzip2 and ..", then new users could convert over without even
having to concern themselves about CVS. If the script had the ability to
download only a portion of the tree -- say, "minimal" vs "gadgets" vs
"full", etc -- that might also increase everyone's milage. I think a lot
of this is people just get everything because they're not familar enough
to know what they need or what might be cool (of course this all goes
back to packaging, no official recent releases.. etc). The script could
even check dependencies (which may lower the number of questions we
get). IIRC, someone may have even proposed this a long time ago..

> imho making dist tarballs is also a good testing mechanism. that is
> our "end product" anyway - a cvs tree where you ned to run autogen.sh
> (autofoo) is NOT our "target product". it is an intermediate state
> developers can easily work with. end users will work with the result
> of make dist and then do the usual tar -xf file.tar.gz; cd file;
> ./configure && make && make install

Absolutely. A more wild possibility is that because most of our
configure scripts accept the same options, the script could even collect
htem and pass them in (so --prefix could be handed down from the
download or perhaps another script, and users can just walk off until
the build is done or have it run non-interactively from a cronjob,
etc..). IMO (and it's a somewhat uninformed one as of late), most users
want access to "product" and the obstacles to installation send them
looking to anoncvs.

> if we move and encourage the primary mechanism to be this with regular
> enough updates that people aren't waiting impatiently for that
> update/fix - then we may solve a lot of our problems?

I'd like to believe that, though instant access to changes is alluring,
(just look at news sites).. :)

term

Lyle Kempler wrote:

> Looking at get-e.org, the download page points to the E17 User's Guide,
> which starts with "Installing from CVS". If we instead changed the
> download page to "download this script and run it, you'll need wget and
> tar and bzip2 and ..", then new users could convert over without even
> having to concern themselves about CVS. If the script had the ability to
> download only a portion of the tree -- say, "minimal" vs "gadgets" vs
> "full", etc -- that might also increase everyone's milage. I think a lot
> of this is people just get everything because they're not familar enough
> to know what they need or what might be cool (of course this all goes
> back to packaging, no official recent releases.. etc). The script could
> even check dependencies (which may lower the number of questions we
> get). IIRC, someone may have even proposed this a long time ago..
> 

so these scripts would build all EFL & some set of applications (who
decides what applications?).  Of course we are assuming that these all
build without issue, as in "alice in wonderland".  I have no trust in
scripts for the entire build process when we are talking about pre-alpha
software.

> IMO (and it's a somewhat uninformed one as of late), most users
> want access to "product" and the obstacles to installation send them
> looking to anoncvs.

You have taken a poll of what all users want?  Some of us (users) are
trying to contribute by staying close to the development by building,
from scratch, on the various Linux O/S flavors to quickly surface
issues.  The hope is that it aids the cause.  Admittedly, perhaps we are
not your eventual std end users.

Eventually, when e17 is released I will be working with rpm's on this
Fedora Core system.  For now I build from scratch.

> 
> I'd like to believe that, though instant access to changes is alluring,
> (just look at news sites).. :)
> 
For me, I consider it more than alluring.

* Laurence Vanek (lvanek@...) wrote:
> Lyle Kempler wrote:
> > Looking at get-e.org, the download page points to the E17 User's
> > Guide, which starts with "Installing from CVS". If we instead
> > changed the download page to "download this script and run it,
> > you'll need wget and tar and bzip2 and ..", then new users could
> > convert over without even having to concern themselves about CVS. If
> > the script had the ability to download only a portion of the tree --
> > say, "minimal" vs "gadgets" vs "full", etc -- that might also
> > increase everyone's milage. I think a lot of this is people just get
> > everything because they're not familar enough to know what they need
> > or what might be cool (of course this all goes back to packaging, no
> > official recent releases.. etc). The script could even check
> > dependencies (which may lower the number of questions we get). IIRC,
> > someone may have even proposed this a long time ago..
> so these scripts would build all EFL & some set of applications (who
> decides what applications?).  Of course we are assuming that these all
> build without issue, as in "alice in wonderland".  I have no trust in
> scripts for the entire build process when we are talking about
> pre-alpha software.

Why is it when we suggest convienence solutions, people respond
defensively? This is not an attack on your ability to keep using
anoncvs, building things by hand, or whatever your specific desired
working methodology is. Enlightenment is about providing you with
_options_. The script _could_ feasibly (easily) simply have a
--download-only feature. It _could_ feasibly give up as soon as an error
is found, hell, if we had warningless builds, it _could_ feasibly tell
GCC to stop on a warning. Or you could simply not use those settings.
You don't have to trust anything you don't want to.

But, I suspect (and I'll say it again, it's a guess) that a lot of users
are willing to try it, and with some practice, we might be able to get a
lot of milage out of it. If that's not what you're after, then keep
using anoncvs or rsync or the freedesktop tarballs, and build it
yourself.

> > IMO (and it's a somewhat uninformed one as of late), most users want
> > access to "product" and the obstacles to installation send them
> > looking to anoncvs.
> You have taken a poll of what all users want?

I expected "it's a somewhat uninformed one as of late" to implicitly
translate to a response of "no, I haven't taken a poll of what all users
want". Read what I said; you shouldn't have to ask that.

> Some of us (users) are trying to contribute by staying close to the
> development by building, from scratch, on the various Linux O/S
> flavors to quickly surface issues.  The hope is that it aids the
> cause.  Admittedly, perhaps we are not your eventual std end users.

You say it for me: "some of us". I don't think anyone, the whole context
of the conversation, has ever wanted to prevent you from contributing,
however you achieve that. My suggesting tarballs, or any of the other
suggestions made by others, are an attempt to improve performance and
possibly make things easier for those who are interested. No more.

> Eventually, when e17 is released I will be working with rpm's on this
> Fedora Core system.  For now I build from scratch.

And that's fine -- finding bugs is a huge advantage of open source
software, and we all welcome the help. Don't take these suggestions as
an affront to that.

> > I'd like to believe that, though instant access to changes is
> > alluring, (just look at news sites).. :)
> For me, I consider it more than alluring.

Relax; we're not raising the price. :)

term

On Sat, 19 Aug 2006 15:36:55 -0400 Lyle Kempler <term-e1@...>
babbled:

definitely. this is not about removing things - but maybe shuffling visibility
of things like cvs from first port of call, to last port of call, and encourage
users to use alternate mechanisms to get the code that is lighter on our
resources. in addition if these mechanisms align with our final distribution
mechanism (tarball + configure + make + make install) we get to heavily test it
now - as opposed to later.

> * Laurence Vanek (lvanek@...) wrote:
> > Lyle Kempler wrote:
> > > Looking at get-e.org, the download page points to the E17 User's
> > > Guide, which starts with "Installing from CVS". If we instead
> > > changed the download page to "download this script and run it,
> > > you'll need wget and tar and bzip2 and ..", then new users could
> > > convert over without even having to concern themselves about CVS. If
> > > the script had the ability to download only a portion of the tree --
> > > say, "minimal" vs "gadgets" vs "full", etc -- that might also
> > > increase everyone's milage. I think a lot of this is people just get
> > > everything because they're not familar enough to know what they need
> > > or what might be cool (of course this all goes back to packaging, no
> > > official recent releases.. etc). The script could even check
> > > dependencies (which may lower the number of questions we get). IIRC,
> > > someone may have even proposed this a long time ago..
> > so these scripts would build all EFL & some set of applications (who
> > decides what applications?).  Of course we are assuming that these all
> > build without issue, as in "alice in wonderland".  I have no trust in
> > scripts for the entire build process when we are talking about
> > pre-alpha software.
> 
> Why is it when we suggest convienence solutions, people respond
> defensively? This is not an attack on your ability to keep using
> anoncvs, building things by hand, or whatever your specific desired
> working methodology is. Enlightenment is about providing you with
> _options_. The script _could_ feasibly (easily) simply have a
> --download-only feature. It _could_ feasibly give up as soon as an error
> is found, hell, if we had warningless builds, it _could_ feasibly tell
> GCC to stop on a warning. Or you could simply not use those settings.
> You don't have to trust anything you don't want to.
> 
> But, I suspect (and I'll say it again, it's a guess) that a lot of users
> are willing to try it, and with some practice, we might be able to get a
> lot of milage out of it. If that's not what you're after, then keep
> using anoncvs or rsync or the freedesktop tarballs, and build it
> yourself.
> 
> > > IMO (and it's a somewhat uninformed one as of late), most users want
> > > access to "product" and the obstacles to installation send them
> > > looking to anoncvs.
> > You have taken a poll of what all users want?
> 
> I expected "it's a somewhat uninformed one as of late" to implicitly
> translate to a response of "no, I haven't taken a poll of what all users
> want". Read what I said; you shouldn't have to ask that.
> 
> > Some of us (users) are trying to contribute by staying close to the
> > development by building, from scratch, on the various Linux O/S
> > flavors to quickly surface issues.  The hope is that it aids the
> > cause.  Admittedly, perhaps we are not your eventual std end users.
> 
> You say it for me: "some of us". I don't think anyone, the whole context
> of the conversation, has ever wanted to prevent you from contributing,
> however you achieve that. My suggesting tarballs, or any of the other
> suggestions made by others, are an attempt to improve performance and
> possibly make things easier for those who are interested. No more.
> 
> > Eventually, when e17 is released I will be working with rpm's on this
> > Fedora Core system.  For now I build from scratch.
> 
> And that's fine -- finding bugs is a huge advantage of open source
> software, and we all welcome the help. Don't take these suggestions as
> an affront to that.
> 
> > > I'd like to believe that, though instant access to changes is
> > > alluring, (just look at news sites).. :)
> > For me, I consider it more than alluring.
> 
> Relax; we're not raising the price. :)
> 
> term
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> enlightenment-devel mailing list
> enlightenment-devel@...
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster@...
裸好多
Tokyo, Japan (東京 日本)