[Semediawiki-user] Intelligent caching (Was: Page cache and inline queries)

[Markus <ma...@ai...>]

On Tuesday 21 November 2006 17:22, Yaron Koren wrote:
> We had this problem too in our implementation at discoursedb.org; our
> solution was to have the "Refresh" tab show up for all users, not just
> administrators, on every page. That way anyone who adds or changes content
> can refresh the data themselves if they don't see their changes appearing.
> The danger with this approach is that it could led to site slowness, but =
we
> haven't observed that yet on our site.
>
> If you want to do it, you just have to comment out the "if" check (
> "if($wgUser->isAllowed('delete')){" ) in the file "SMW_RefreshTab.php".

[Remark: this is mainly restricted to admins since we did not want to show =
too=20
many options to users who might not even be aware of their use.]

>
> -Yaron


We are aware of the problem, but it is not easy to find a suitable solution=
=2E=20
The issue involves various research and engineering challenges (see comment=
s=20
below), and I am happy to cooperate with anyone who wants to work on a=20
solution. We are already supervising a Master's thesis that is concerned wi=
th=20
caching of queries in semantic storage systems, but this is just a first=20
step. We can also act as external supervisors for any student's projects th=
at=20
look into this direction -- feel free to contact us!

Now back to the actual problem:

We could of course change the queries in such a way that they are re-run vi=
a=20
AJAX for everyone who has JavaScript (while the others see the current stat=
ic=20
version). The problem with this is that executing a query will be a=20
significant performance bottleneck that one rather wants to avoid. Executin=
g=20
a query on a page is certainly more time consuming than refreshing a normal=
=20
page without a query, i.e. if you have a wiki that is small enough to switc=
h=20
off the caching for all pages, then a non-cached query handling might be=20
possible. Otherwise you should be happy if your servers manage to handle th=
e=20
additional load of even the cached queries (note that SMW includes many=20
switches to further restrict queries).

Currently, our rationale is that the traditional way of creating up-to-date=
=20
listings is to manually edit them. Inline queries only require someone to=20
press "refresh" once in a while, or to wait until the page-cache is=20
automatically cleared next time (this will happen frequently enough on larg=
e=20
wikis anyway). So things already become easier and more acurate.

=46or the future, it will be necessary to think about site-wide query cachi=
ng=20
and subquery caching: in addition to storing the query output in one page,=
=20
one might consider the above JavaScript solution, but with an additional=20
cache being used for query answering. This could increase performance of th=
e=20
otherwise problematic Special:Ask as well. The cache would then be refreshe=
d=20
in different ways, but it clearly is not trivial to do this (i.e. to find o=
ut=20
which queries are affected by a particular change in the wiki). One questio=
n=20
also is how many answers of each query can reasonably be cached in the=20
available storage space without slowing down cache retrieval too much. And=
=20
one has to look at the problem of using existing cache entries to answer=20
related new queries (for a simple example: if you ask for the same query as=
=20
before, but with less attributes printed, then one can obviously use the=20
cached results of the first query -- provided that you have a fast way of=20
figuring this out). An external cache could be updated by a robot that uses=
=20
seperate resources.

We are moving towards an architecture that enables transparent use of=20
different query backends and caches, so the technical foundation for the=20
above will be available ... any volunteers? (background in database=20
optimisation and/or caching methods and/or RDF/OWL/SPARQL could obviously b=
e=20
useful)

Cheers,

Markus

>
> On 11/21/06, Fernando Correia <fer...@gm...> wrote:
> > How do you handle the page cache for pages that use inline queries?
> >
> > If a page has an inline query, for instance, to show all pages from a
> > category, and I edit another page to insert it on the category, the page
> > with the query will not reflect this change.
> >
> > Only if I purge the page cache manually it will be updated.
> >
> > This behavior seems to make the inline queries not so useful, because
> > there is a high chance they will be showing incorrect data as the wiki
> > evolves.
> >
> > I'd like to know if anyone has developed strategies to handle this.
> >
> > Since I'm new to MediaWiki, I don't know what strategy would be better.
> >
> > Thank you.
> >
> >
> > -----------------------------------------------------------------------=
=2D-
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to share
> > your
> > opinions on IT & business topics through brief surveys - and earn cash
> > http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=
=3DDEVDEV
> >
> > _______________________________________________
> > Semediawiki-user mailing list
> > Sem...@li...
> > https://lists.sourceforge.net/lists/listinfo/semediawiki-user

=2D-=20
Markus Kr=F6tzsch
Institute AIFB, University of Karlsruhe, D-76128 Karlsruhe
ma...@ai...        phone +49 (0)721 608 7362
www.aifb.uni-karlsruhe.de/WBS/     fax +49 (0)721 693  717