Semantic MediaWiki

Hey James,

Great that you are looking into this. Are you aware of Nicschayns Summer of
Code project [0]? He'll be doing stuff in this direction. Better coordinate
to avoid both doing the same work in parallel :)

I think your approach is good in general, as an easy-to-implement tweak
that can have quite a big impact on performance even though not as much as
more complex caching strategies might.

> public function getHash() { return md5( $this->m_querystring .
$this->m_limit . serialize( $this->m_extraprintouts ) ); }

Since stuff here is not sorted, you can have the same query with different
hashes. Might be easy to sort in some way and not have this.

> The cache object is generated using the $wgMemc class

I'm in favor of using wfGetCache( CACHE_ANYTHING ).

To wrap up: I think this is ok, although the integration could be better.
One concern I have is that if this is enabled, and you change something
that you want to see reflected in your query right away, there is no way to
invalidate the cache. If that gets addressed, such a change can go in SMW
as far as I'm concerned. Improving the integration can be done later,
although this might not be much later, depending on Nicschayns progress.

[0] https://www.mediawiki.org/wiki/User:Nischayn22/Gsoc

Cheers

--
Jeroen De Dauw
http://www.bn2vs.com
Don't panic. Don't be evil.
--

Hi,

Right now I am still working on the ground level i.e. the smwStore as many
things can be optimized there; for e.g. on each page edit all the smw data
is rewritten to the database even if nothing was changed.

Soon after I complete working on the smwStore I will look into Caching
Queries and SpecialPages.
However, your implementation will be helpful.

I did a similar thing with wgMemc for a SpecialPage of SMW.


		global $wgMemc;
		$key = $this->getName(); // we use the Name as the cache key

$key = $key.$limit.$offset; // modified key to contain limit and offset
information
$res = $wgMemc->get($key);

if($res == '' || $nocache == true){//Check to see if cache is old or empty
$res = $this->getResults( $options );
$wgMemc->set( $key, $res, 60 * 60 ); //update the cache with expiration
time 1 hour ( can be changed )
}
else
$wgOut->addHTML( "<p>This is a cached result</p>" );



On Wed, May 30, 2012 at 3:25 AM, Jeroen De Dauw <jeroendedauw@...:

> Hey James,
>
> Great that you are looking into this. Are you aware of Nicschayns Summer
> of Code project [0]? He'll be doing stuff in this direction. Better
> coordinate to avoid both doing the same work in parallel :)
>
> I think your approach is good in general, as an easy-to-implement tweak
> that can have quite a big impact on performance even though not as much as
> more complex caching strategies might.
>
>
> > public function getHash() { return md5( $this->m_querystring .
> $this->m_limit . serialize( $this->m_extraprintouts ) ); }
>
> Since stuff here is not sorted, you can have the same query with different
> hashes. Might be easy to sort in some way and not have this.
>
>
> > The cache object is generated using the $wgMemc class
>
> I'm in favor of using wfGetCache( CACHE_ANYTHING ).
>

Why? wgMemc works whether you use Memc or not. And memc gives the best
performance.


> To wrap up: I think this is ok, although the integration could be better.
> One concern I have is that if this is enabled, and you change something
> that you want to see reflected in your query right away, there is no way to
> invalidate the cache. If that gets addressed, such a change can go in SMW
> as far as I'm concerned. Improving the integration can be done later,
> although this might not be much later, depending on Nicschayns progress.
>
> [0] https://www.mediawiki.org/wiki/User:Nischayn22/Gsoc
>
> Cheers
>
> --
> Jeroen De Dauw
> http://www.bn2vs.com
> Don't panic. Don't be evil.
> --
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Semediawiki-devel mailing list
> Semediawiki-devel@...
> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
>
> --
Cheers,

Nischay Nahata
http://nischayn22.in

Hi,

just wanted to put in my 2c.

On 30 May 2012 16:21, Nischay Nahata <nischayn22@...> wrote:
> Right now I am still working on the ground level i.e. the smwStore as many
> things can be optimized there; for e.g. on each page edit all the smw data
> is rewritten to the database even if nothing was changed.

I am really looking forward to this. I had to implement something like
this today to figure out if a cache purge was necessary after some
property changed. Would be great, if you could build in a hook that is
called on change. (James: Working on caching in SemanticGlossary right
now. Will probably contact you later today.)


>> > The cache object is generated using the $wgMemc class
>>
>> I'm in favor of using wfGetCache( CACHE_ANYTHING ).
>
> Why? wgMemc works whether you use Memc or not. And memc gives the best
> performance.

Globals are evil. However, instead of a fixed wfGetCache(
CACHE_ANYTHING ), why not use wfGetMainCache()?
Or better yet, something like $cache = ($smwgCacheType !== null)?
wfGetCache( $smwgCacheType ):wfGetMainCache();
This way people could specify their preferred cache type in
$smwgCacheType and if they did not, you'd fall back on the main cache.

I would probably also replace the expiry time of 60 * 60 by some
configurable value. IMHO if you can ensure that you invalidate the
cache when necessary, there should not be a reason to have an expiry
time at all. Or do I miss something here?

Cheers,
Stephan

Hi,

Sorry that I wasn't able to respond earlier but we made some effort to
make the integration bit more sound and add a functionality that
enables SMWQueryCache to invalidate query cache objects based on
individual objects that are contained within each query. For a more
exhaustive explanation see [1].

## General changes
We leave the class SMWQuery unharmed and instead having an extended
class SMWQueryExtended the implements the getHash() functionality.

All functionality has been transferred into its own class SMWQueryCache.

SMWAskPage and SMWQueryProcessor now use:
$querycache = new SMWQueryCache;
$res = $querycache->getQueryCacheResult( $query, $params )

We introduce a new method (createAssociatedQueryCache) with which
queries are able to track individual objects that are contained within
each query. Should one of those objects change/or become obsolete
linked queries are invalidated immediately. The so called associated
cache objects are kept as a separate cache entity (see more [1])
allowing to avoid additional database select/write within the
SMWQueryCache class (which was the purpose after all).

For cache invalidation we rely on three hooks
$wgHooks['SMWStore::updateDataBefore'][],
$wgHooks['smwDeleteSemanticData'][] whereby the only new hook we need
is $wgHooks['smwChangeTitle'] because when a page gets moved neither
of the above hooks have been activated.

In general their are only three minor changes to the SMW core that
would be necessary to allow to use all functionality provided by the
class SMWQueryCache.

* SMWAskPage
* SMWQueryProcessor
* $wgHooks['smwChangeTitle']

## Respond to earlier emails

>> Are you aware of Nicschayns Summer of Code project [0]?
I was ware of it but since our users started to use Special:Ask
queries more often during the Special:Search session we have seen an
increased need to bundle those identical queries.

>> Better coordinate to avoid both doing the same work in parallel :)
The general idea of this email was to push some ideas forward before
everything is set in stone.

>> Since stuff here is not sorted, you can have the same query with different hashes
Actually we can't sort since the result is stored the way printout
needs the nested objects meaning when the same query is executed but
"hypothetical" columns are switched getQueryResult would return with a
different order in $res therefore a different sorting of columns have
to have a different hash key in order for the PrinterResult to match
$res and the printout statements.


> I'm in favor of using wfGetCache( CACHE_ANYTHING ).
Done

>>  integration could be better
Everything is transferred into the SMWQuery class

>> you change something that you want to see reflected in your query right away
Disabling query cache completely by setting set $smwgQueryCache =
false or for a specific query to set $params['cache'] to false which
then works on a per query basis.
See also [2]

>> no way to invalidate the cache
Using associated cache objects, see above explanation and [3]

>> Globals are evil
We moved away from wgMemc and instead use wfGetCache( ... )

>> replace the expiry time ... configurable value
Using $smwgQueryCacheExpiry and see also [4]

[1] https://github.com/mwjames/smw-querycache

[2] https://github.com/mwjames/smw-querycache#integration-with-srf

[3] https://github.com/mwjames/smw-querycache#query-invalidation

[4] https://github.com/mwjames/smw-querycache#query-cache-parameters-localsettingsphp

Cheers

mwjames

Just so you know: you could submit a change on Gerrit in case some of your
implementation could be merged with SMW
permanently and used by everyone else. The code can be reviewed there very
well as compared to github.

I am still working on the SMW store interface and will look into caching
queries later. If your implementation could be used
by SMW we can avoid doing the same tasks separately.

On Thu, Jun 7, 2012 at 2:42 AM, James HK <jamesin.hongkong.1@...:

> Hi,
>
> Sorry that I wasn't able to respond earlier but we made some effort to
> make the integration bit more sound and add a functionality that
> enables SMWQueryCache to invalidate query cache objects based on
> individual objects that are contained within each query. For a more
> exhaustive explanation see [1].
>
> ## General changes
> We leave the class SMWQuery unharmed and instead having an extended
> class SMWQueryExtended the implements the getHash() functionality.
>
> All functionality has been transferred into its own class SMWQueryCache.
>
> SMWAskPage and SMWQueryProcessor now use:
> $querycache = new SMWQueryCache;
> $res = $querycache->getQueryCacheResult( $query, $params )
>
> We introduce a new method (createAssociatedQueryCache) with which
> queries are able to track individual objects that are contained within
> each query. Should one of those objects change/or become obsolete
> linked queries are invalidated immediately. The so called associated
> cache objects are kept as a separate cache entity (see more [1])
> allowing to avoid additional database select/write within the
> SMWQueryCache class (which was the purpose after all).
>
> For cache invalidation we rely on three hooks
> $wgHooks['SMWStore::updateDataBefore'][],
> $wgHooks['smwDeleteSemanticData'][] whereby the only new hook we need
> is $wgHooks['smwChangeTitle'] because when a page gets moved neither
> of the above hooks have been activated.
>
> In general their are only three minor changes to the SMW core that
> would be necessary to allow to use all functionality provided by the
> class SMWQueryCache.
>
> * SMWAskPage
> * SMWQueryProcessor
> * $wgHooks['smwChangeTitle']
>
> ## Respond to earlier emails
>
> >> Are you aware of Nicschayns Summer of Code project [0]?
> I was ware of it but since our users started to use Special:Ask
> queries more often during the Special:Search session we have seen an
> increased need to bundle those identical queries.
>
> >> Better coordinate to avoid both doing the same work in parallel :)
> The general idea of this email was to push some ideas forward before
> everything is set in stone.
>
> >> Since stuff here is not sorted, you can have the same query with
> different hashes
> Actually we can't sort since the result is stored the way printout
> needs the nested objects meaning when the same query is executed but
> "hypothetical" columns are switched getQueryResult would return with a
> different order in $res therefore a different sorting of columns have
> to have a different hash key in order for the PrinterResult to match
> $res and the printout statements.
>
>
> > I'm in favor of using wfGetCache( CACHE_ANYTHING ).
> Done
>
> >>  integration could be better
> Everything is transferred into the SMWQuery class
>
> >> you change something that you want to see reflected in your query right
> away
> Disabling query cache completely by setting set $smwgQueryCache =
> false or for a specific query to set $params['cache'] to false which
> then works on a per query basis.
> See also [2]
>
> >> no way to invalidate the cache
> Using associated cache objects, see above explanation and [3]
>
> >> Globals are evil
> We moved away from wgMemc and instead use wfGetCache( ... )
>
> >> replace the expiry time ... configurable value
> Using $smwgQueryCacheExpiry and see also [4]
>
> [1] https://github.com/mwjames/smw-querycache
>
> [2] https://github.com/mwjames/smw-querycache#integration-with-srf
>
> [3] https://github.com/mwjames/smw-querycache#query-invalidation
>
> [4]
> https://github.com/mwjames/smw-querycache#query-cache-parameters-localsettingsphp
>
> Cheers
>
> mwjames
>



-- 
Cheers,

Nischay Nahata
http://nischayn22.in