[Javtools-poolmngr] Re : Chunking

[Isabelle M. <met...@pa...>]

Hi everyone,

I don't think chunking is a good idea, it doesn't scale well (as Amit =
pointed out), plus the databases already do it. For example, if you send =
a query for 10000 rows to Oracle, it sends it back in chunks of =
something like 50 rows. The model underlying resultsets is that a cursor =
is opened, and then there are repeated fetches until there is no more =
data.=20

I'm also not clear on how you see the technical implementation of =
chunking. Let's say the query is "select * from authors where first_name =
like 'ISA%'.
How are you going to tell the database to return only N rows? I don't =
think you can. So the only thing you can do is sit there waiting for =
data, collecting it until there are N items, and then pass them on to =
the user. What have you gained in doing that? IMO : nothing.

What might be useful, though, is the possibility to cache a resultset =
(i.e. save it so that the query doesn't have to be executed next time =
around). But :

(1) a piggish user might use up all available memory for one gigantic =
query, so you need to have policies in place for max space per =
connection
(2) you need a policy for invalidating stale data (that has been sitting =
there for a long time without being requested)
(3) you might to collect usage patterns for each cached resultset, and =
adapt the cache to that pattern. For ex. if there are 10000 rows in the =
cache, but in 100 requests nobody ever went beyond row 1000, you might =
as well throw out the last 9000. Think searches on search engines : how =
many times do you look at the 11th page?
(4) you need a notion of "cache-ability". For example, there would be no =
point in saving "select * from authors where first_name =3D 'Ronald" and =
last_name =3D 'Reagan'. (too specific =3D=3D> likelihood of being =
requested again too small)


Isabelle