Re: [Freevo-devel] Re: [Freevo-cvslog] Kaa r1479 - trunk/beacon/src

[Jason T. <ta...@ur...>]

On Sun, 2006-04-23 at 17:09 +0200, Dirk Meyer wrote:
> Jason Tackaberry wrote:
> > But the main benefit here is with a list of extensions that we know are
> > special, beacon can do O(1) lookups:
> >
> >    ext = os.path.splitext(self._beacon_data['name'])[1]
> >    special_exts = kaa.metadata.get_special_exts_for_type(ext)
> >    listdir_file_map = self._beacon_parent._beacon_listdir(cache=True)[1]
> >    for ext in special_exts:
> >       if basename+ext in listdir_file_map:
> >           mtime += listdir_file_map[basename+ext][3][stat.ST_MTIME]
> >
> > Maybe you want to avoid splitext() or whatever, but you get the idea.
> > This is just rough code.  But the above logic reduces the complexity to
> > O(n) and when n=2500, that's a difference of 6.25 million iterations.
> 
> Sounds like a good speedup. I will keep that in mind. For now, beacon
> works like it is now and speedups are not very high an my todo list. I
> first need to finish rom drive support.

Ok, so this logic is checked in now, and it's muuuuccch better (which we
fully expected).  beacon now no longer churns at 100% for a few seconds
when starting the dirname=/usr/bin search.  Now it completes in a
fraction of a second.

Tonight I'll turn my attention to the normalization stuff.
Normalization accounts for about 40% of the time it takes to do the
dirname query.  However, 40% of that 40% is from cPickle, and there's
nothing that can be done about that unless we decide to write our own
simpler serialization stuff (that supports only the primitives types
plus list,tuple,dict).

Jason.