Re: [E-devel] [PATCH] ecore main loop "fd_handlers_to_call" list optimization

[Carsten H. (T. R. <ra...@ra...>]

On Thu, 2 Dec 2010 16:20:36 -0500 Mike Blumenkrantz <mi...@ze...> said:

> On Thu, 2 Dec 2010 13:18:02 +0100
> Cedric BAIL <ced...@fr...> wrote:
> 
> > On Thu, Dec 2, 2010 at 1:08 PM, Iván Briano (Sachiel)
> > <sac...@gm...> wrote:
> > > 2010/12/2 Mike McCormack <mj....@sa...>:
> > >> On 12/02/2010 08:20 PM, Lucas De Marchi wrote:
> > >>> On Thu, Dec 2, 2010 at 8:34 AM, Cedric BAIL<ced...@fr...>  wrote:
> > >>>> On Thu, Dec 2, 2010 at 7:22 AM, Mike
> > >>>> McCormack<mj....@sa...> wrote:
> > >>>>> Rather than using malloc'ed list entries in the mail loop, use a
> > >>>>> single linked in-place list.
> > >>>>>
> > >>>>> This avoid lots of mallocs and frees as the main loop iterates.
> > >>>>
> > >>>> I like the idea, but I fear that doing change to the core main loop
> > >>>> structure could break things. Maybe we should post pone this change to
> > >>>> 1.1. From my point of view we already did to much change to ecore main
> > >>>> loop during the beta cycle.
> > >>>
> > >>> I agree.
> > >>>
> > >>> How long are we for 1.0 release?
> > >>
> > >> Come on guys.
> > >>
> > >> The only reason that you're seeing this is that I don't commit directly
> > >> to SVN.
> > >>
> > >> This is a tweak on fixes that have been put in by others in the last few
> > >> days. If you object to this, please read the SVN commits list and object
> > >> to the other patches in the same area.
> > >
> > > True, all the refactoring of fd handlers was a big enough thing to put
> > > off until 1.1,
> > > and it was done directly in SVN. So what's the reason to not commit
> > > this one now?
> > 
> > I know I should have objected at that time, but I didn't. Right now I
> > think we already have to much change in the ecore main loop to feel
> > safe with all possible scenario (glib integration, multiple ecore main
> > loop imbrication and all) that I would really prefer to postpone more
> > change in this aread. But that's just my opinion. If you really want
> > to do it, wait at least for beta3 to be released tomorrow.
> > 
> > Thanks,
> This change absolutely must go through before 1.0 if we want to have a
> respectable main loop, and the following data will prove it.  This data can
> easily be replicated using the client/server_bench.c files in ecore/examples.
> 
> ecore revision 54616:
> =======================
> 25000 fd test:
> Connection lost! #10400!
> ^C
> Time elapsed for 25000 connections: 371.606825 seconds
> # specimen	experiment time	starting time	ending time
> 1	645014268	3993714	649007982
> ./bench_client  4.38s user 0.70s system 1% cpu 6:11.76 total
> (Note: at this point I killed it because after 10400 fds, ecore stopped
> and no longer processed any connections)
> 
> 10000 fd test:
> Connection lost! #8933!
> ^C
> Time elapsed for 10000 connections: 443.156038 seconds
> # specimen	experiment time	starting time	ending time
> 1	746600745	3902720	750503465
> ./bench_client  0.55s user 0.28s system 0% cpu 7:23.28 total
> (Note: at this point I killed it because after 8933 fds, ecore stopped
> and no longer processed any connections)
> 
> 5000 fd test:
> Time elapsed for 5000 connections: 45.284662 seconds
> # specimen	experiment time	starting time	ending time
> 1	259416526	3992216	263408742
> ./bench_client  0.14s user 0.16s system 0% cpu 45.331 total
> 
> 2000 fd test:
> Time elapsed for 2000 connections: 3.086964 seconds
> # specimen	experiment time	starting time	ending time
> 1	101628352	3916119	105544471
> ./bench_client  0.05s user 0.07s system 3% cpu 3.107 total
> 
> 1000 fd test:
> Time elapsed for 1000 connections: 0.048472 seconds
> # specimen	experiment time	starting time	ending time
> 1	47202900	3945319	51148219
> ./bench_client  0.02s user 0.03s system 96% cpu 0.060 total
> 
> A note about the above results: The numbers for the first two tests were the
> "best" that I could achieve.  I ran both tests multiple times to attempt the
> highest number of connections possible.  When ecore showed no further
> connections after a full minute, I ended the test.  The reason why the 25k fd
> test achieves more connections is because it sends more attempts in the same
> amount of time, allowing for more to be successfully picked up (like throwing
> a bunch of marbles at something far away, only a few will hit).
> 
> Here are the exact same tests once again.
> ecore revision HEAD:
> =======================
> 25000 fd test with printf disabled:
> Time elapsed for 25000 connections: 1.141623 seconds
> # specimen	experiment time	starting time	ending time
> 1	1136243319	3866130	1140109449
> ./bench_client  0.43s user 0.91s system 98% cpu 1.356 total
> 
> 25000 fd test:
> Time elapsed for 25000 connections: 3.439958 seconds
> # specimen	experiment time	starting time	ending time
> 1	-892811351	3977067	-888834284
> ./bench_client  0.44s user 0.94s system 37% cpu 3.651 total
> 
> 10000 fd test:
> Time elapsed for 10000 connections: 0.479854 seconds
> # specimen	experiment time	starting time	ending time
> 1	470182903	3876967	474059870
> ./bench_client  0.17s user 0.38s system 97% cpu 0.567 total
> 
> 5000 fd test:
> Time elapsed for 5000 connections: 0.247735 seconds
> # specimen	experiment time	starting time	ending time
> 1	237503799	3889656	241393455
> ./bench_client  0.10s user 0.18s system 95% cpu 0.295 total
> 
> 2000 fd test:
> Time elapsed for 2000 connections: 0.096664 seconds
> # specimen	experiment time	starting time	ending time
> 1	94169842	3970562	98140404
> ./bench_client  0.04s user 0.07s system 95% cpu 0.119 total
> 
> 1000 fd test:
> Time elapsed for 1000 connections: 0.047393 seconds
> # specimen	experiment time	starting time	ending time
> 1	46859592	3859111	50718703
> ./bench_client  0.02s user 0.04s system 97% cpu 0.060 total
> 
> 
> I think the data here speaks for itself, and this is only showing current svn
> HEAD.
> 
> 25000 fd test with Mike's newest patch:
> Time elapsed for 25000 connections: 1.131604 seconds
> # specimen	experiment time	starting time	ending time
> 1	1127458798	3833513	1131292311
> ./bench_client  0.43s user 0.91s system 99% cpu 1.345 total
> 
> Again, a clear, though smaller, improvement.  Regardless of whether this
> specific patch makes it into 1.0, our policy here should definitely not be
> "how fast can we revert the work on ecore?", it should be "how fast and
> efficiently can we test/fix this issue to make sure it's working as
> intended?" because it's definitely a bug.
> -- 
> Mike Blumenkrantz
> Zentific: Our boolean values are huge.

i think we're confusing 2 issues. 1 is moving from O(n2) to O(n) and having a
mainloop that handles lots of fd's at all vs a a micro-optimisation patch on
top of it that makes very little difference at all. yes - it does malloc less.
yes it will fragment less. yes it is a bit faster. the code is shorter. not
disputed.

the question is - is that patch on top "part of the original O(n2) -> O(n) work
or is it separate. i'm on the fence. it could have gone in as part of mikez's
work and no one would have peeped. at the same time the code it implements for
a list is very different to everything else in ecore that uses lists. i don't
see a technical problem with the code in the patch - more of a clarity issue
that you see something wanting to be a list and not actually using normal list
infra. chances are this bit of ecore isnt going to see much work over time - so
is that actually an issue? all good questions. i'm tempted to say "just let it
in as it'd have been fine were it part of the original O(n2) -> O(n) fix".

-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    ra...@ra...