[Exist-development] optimizing positional predicates for fast paging

[Michael S. <so...@if...>]

I wonder whether eXist is currently optimizing xquery evaluations that use
positional predicates, or similar.  It doesn't seem like it, based on some
poking around in the code, and I think this could be a very fruitful area.
What I mean is an expression like:

collection("/")/ (//elem) [position() = (1 to 10)]

- when there are a lot of //elem, evaluation can be short-cut by ignoring
all but the first 10: ie the first "page" of results.  Currently it
*appears* to me as if the entire (//elem) node sequence is being
constructed, and then filtered.

In web applications, a great many queries are limited in exactly this way:
[position() = ($n to $n + $page-size)], so it seems like an area that merits
attention.  And the query-writer doesn't have enough expressive power to
push the positional predicate into the innermost expression, especially if
they are passing paging parameters via a wrapping API layer like REST.  

There is also a problem when coding FLWOR expressions (which you have to do
in order to sort the results), where the only option for paging is to wrap
the entire FLWOR in a subsequence function call (or postfix a predicate like
above).  Some kind of internal expression rewriting that pushes the paging
predicate into the FLWOR evaluation could yield a huge performance
improvement in such cases.  Even when results are sorted, one might see a
performance gain by using a value index scan.

I started to look into implementing this myself, but I wanted to see whether
any work had been done already, or if anyone had any other thoughts about
whether this is worthwhile?

-Mike