Re: [PyIndexer] App-side join?

[Marcus C. <ma...@wr...>]

On Mon, 17 Dec 2001 at 13:49:05 +0000, Chris Withers wrote:

> Marcus Collins wrote:
> > 
> > The app-side time to join a list of document ids returned from MySQL
> > into a MySQL list should not be too bad. (The 'obvious' way of doing it
> > takes 50ms for 10000 ids on my box, once the ids have been returned from
> > MySQL. The main time expense, I think, would be inside MySQLdb.)
> 
> I'm guessing by this you mean doing each boolean term as a seperate SELECT and
> using python to combine the returned results?
> What was the method you used?

Oh hang, sorry... I meant fetching all the results from a previous query
and using string.join() to generate an SQL list (the list of doc ids to
be checked in the next query). Those are both cheap.

Results to follow, but essentially MySQL has no problem handling IN tests 
with 10K members on a 1M row table -- worst-case, where all results are
returned, is under two seconds, of which 10% is spent by Python fetching 
the results and string.joining the list. With one in ten results being
returned, you're looking at a query taking around 0.5s. This excludes
the initial lookups and other joins, but it is positive.

Cheers

-- Marcus