From: Yaron K. <ya...@gm...> - 2008-07-03 17:51:25
|
Hi everyone, I'm pleased to announce the release of the site Referata, at referata.com: a hosting site for SMW-based semantic wikis. This is not the first site to offer hosting of wikis using Semantic MediaWiki (that's Wikia, as of a few months ago), but it is the first to also offer the usage of Semantic Forms, Semantic Drilldown, Semantic Calendar, Semantic Google Maps and some of the other related extensions you've probably heard about; Widgets, Header Tabs, etc. As such, I consider it the first site that lets people create true collaborative databases, where many people can work together on a set of well-structured data. Basic usage of Referata is free: once you register, you can create as many wikis as you want, at a subdomain of referata.com (e.g., mywiki.referata.com). The site has two additional levels of usage: "premium", which costs $20 a month, and "enterprise", which costs $250 a month. The most important additional feature of premium membership is that you can set the wiki to be private (readable only by users), and the most important additional feature of enterprise membership is that you can have a "virtual domain", meaning that you can have the wiki show up at non-Referata URL that you own. There's a special subdomain I created just for user testing: scratchpad.referata.com, where anyone can go to create semantic properties, queries, templates, forms etc. If you do use the site, please feel free to offer any feedback or bug reports, either to this email address or to in...@re.... Thank you, Yaron Koren |
From: <zeh...@mo...> - 2008-07-04 11:29:13
|
I'm using MediaWiki: 1.11.1 (r30349) Semantic MediaWiki (version 1.2c-SVN) I recently felt that my test wiki was rather slow and I thought I investigate. I basically just added the line wfDebug("\nsmwfProcessInlineQueryParserFunction:".print_r($params,true)); after the lines $params = func_get_args(); array_shift( $params ); // we already know the $parser ... in the function smwfProcessInlineQueryParserFunction in the file SMW_GlobalFunctions.php. In the MW debugging log file I see that whenever I call a page which does not contain any SMW related data (i.e. Special:SpecialPages or just edit a new empty page) the function smwfProcessInlineQueryParserFunction is called several times with paramters containing a "typical type of query" which are used on other pages of the wiki (although not sure where exactly). There seems two types of queries which turn up but not always 100% identical. Reloading i.e. the Special:SpecialPages shows that smwfProcessInlineQueryParserFunction is called a different number of times (e.g. 9 or 13 times) each time the page is loaded with either the same/similar query or a different one (kind of random). Calling a page with a SMW query the parameters of smwfProcessInlineQueryParserFunction show the expected parameters for that query. I tried to disable all caching in LocalSettings but nothing changed. Where do these queries come from which are automatically filled into the parameters of smwfProcessInlineQueryParserFunction? Should smwfProcessInlineQueryParserFunction be called at all when loading a page without any SMW stuff? Thanks, Gu |
From: Markus K. <ma...@se...> - 2008-07-07 07:49:59
|
On Freitag, 4. Juli 2008, zeh...@mo... wrote: > I'm using > MediaWiki: 1.11.1 (r30349) > Semantic MediaWiki (version 1.2c-SVN) > > I recently felt that my test wiki was rather slow and I thought I > investigate. I basically just added the line > wfDebug("\nsmwfProcessInlineQueryParserFunction:".print_r($params,true)); > after the lines > $params = func_get_args(); > array_shift( $params ); // we already know the $parser ... > in the function smwfProcessInlineQueryParserFunction in the file > SMW_GlobalFunctions.php. > > In the MW debugging log file I see that whenever I call a page which > does not contain any SMW related data (i.e. Special:SpecialPages or just > edit a new empty page) the function smwfProcessInlineQueryParserFunction is > called several times with paramters containing a "typical type of query" > which are used on other pages of the wiki (although not sure where > exactly). There seems two types of queries which turn up but not always > 100% identical. > > Reloading i.e. the Special:SpecialPages shows that > smwfProcessInlineQueryParserFunction is called a different number of times > (e.g. 9 or 13 times) each time the page is loaded with either the > same/similar query or a different one (kind of random). > Calling a page with a SMW query the parameters of > smwfProcessInlineQueryParserFunction show the expected parameters for that > query. > > I tried to disable all caching in LocalSettings but nothing changed. > > Where do these queries come from which are automatically filled into the > parameters of smwfProcessInlineQueryParserFunction? > > Should smwfProcessInlineQueryParserFunction be called at all when loading a > page without any SMW stuff? Yes that can happen. SMW and MediaWiki use "jobs" to execute certain update tasks in small pieces. For example, if you change the type of a property, all pages using that property will be updated with such a job. MediaWiki creates similar jobs when templates have changed. Jobs are run during normal wiki operation when another operation is executed, so it is normal that one call to you wiki will lead to several other pages being processed in the background. This explains the queries. You can view the number of queued jobs on Special:Statistics, and you can use the MediaWiki script runJobs.php to execute them directly. Please let us know if there is an actual bug in the job generation, e.g. if you find a very large amount of jobs being constantly queued in the wiki. If the jobs are normal but the queries are still too slow, try to upgrade to the new SMWSQLStore2 (or wait until the SMW 1.2 release where you find proper instructions on that). It is significantly faster on many queries. Upgrading an existing SMW SVN installation that already uses the new store will require a refreshData with option -vftp and another normal one with option -v (and possibly -s on large wikis). Regards, Markus > > Thanks, > Gu > > > > ------------------------------------------------------------------------- > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! > Studies have shown that voting for your favorite open source project, > along with a healthy diet, reduces your potential for chronic lameness > and boredom. Vote Now at http://www.sourceforge.net/community/cca08 > _______________________________________________ > Semediawiki-user mailing list > Sem...@li... > https://lists.sourceforge.net/lists/listinfo/semediawiki-user -- Markus Krötzsch Semantic MediaWiki http://semantic-mediawiki.org http://korrekt.org ma...@se... |
From: Markus K. <ma...@se...> - 2008-07-07 07:58:17
|
Hi Yaron, congratulations for making this step! I am confident that this hosting option can be of much use to many people who look for some additional technical assistance and customisation for their wikis. I also hope that this new occupation will not stop but rather further the great work you already did on Semantic Forms and your other extensions. Best regards, Markus On Donnerstag, 3. Juli 2008, Yaron Koren wrote: > Hi everyone, > > I'm pleased to announce the release of the site Referata, at referata.com: > a hosting site for SMW-based semantic wikis. This is not the first site to > offer hosting of wikis using Semantic MediaWiki (that's Wikia, as of a few > months ago), but it is the first to also offer the usage of Semantic Forms, > Semantic Drilldown, Semantic Calendar, Semantic Google Maps and some of the > other related extensions you've probably heard about; Widgets, Header Tabs, > etc. As such, I consider it the first site that lets people create true > collaborative databases, where many people can work together on a set of > well-structured data. > > Basic usage of Referata is free: once you register, you can create as many > wikis as you want, at a subdomain of referata.com (e.g., > mywiki.referata.com). The site has two additional levels of usage: > "premium", which costs $20 a month, and "enterprise", which costs $250 a > month. The most important additional feature of premium membership is that > you can set the wiki to be private (readable only by users), and the most > important additional feature of enterprise membership is that you can have > a "virtual domain", meaning that you can have the wiki show up at > non-Referata URL that you own. > > There's a special subdomain I created just for user testing: > scratchpad.referata.com, where anyone can go to create semantic properties, > queries, templates, forms etc. > > If you do use the site, please feel free to offer any feedback or bug > reports, either to this email address or to in...@re.... > > Thank you, > Yaron Koren -- Markus Krötzsch Semantic MediaWiki http://semantic-mediawiki.org http://korrekt.org ma...@se... |
From: <zeh...@mo...> - 2008-07-07 09:46:17
|
Oh I see, thanks. Wasn't aware of MW 'jobs' at all. My job queue has a length of 26,393 according to Special:Statistics. However the table mbw_jobs has 27222 entries where 27125 have job_cmd='refreshLinks' and 97 have job_cmd='htmlCacheUpdate'. I have recently increased the size of the test wiki to about 28000 pages (which all include the same template) and I believe I can vaguely remember to have run the maintenance script refreshLinks.php at one point in order to try to sort something out. I can't remember the details but I didn't know it would generate such a long-lasting effect. I did run refreshData.php of course but I don't think that's connected with these jobs (I usually run refreshData.php -p once and than refreshData.php in 1000 page id chunks (using the -e option)). I'm waiting for a good moment to switch to the new storage version but as I have made a number of modifications it will be a bit more work. Thanks again for the info, it is rather comforting to have a logical explanation for such 'strange' problems. Cheers, Gu Quoting Markus Krötzsch <ma...@se...>: > On Freitag, 4. Juli 2008, zeh...@mo... wrote: > > I'm using > > MediaWiki: 1.11.1 (r30349) > > Semantic MediaWiki (version 1.2c-SVN) > > > > I recently felt that my test wiki was rather slow and I thought I > > investigate. I basically just added the line > > wfDebug("\nsmwfProcessInlineQueryParserFunction:".print_r($params,true)); > > after the lines > > $params = func_get_args(); > > array_shift( $params ); // we already know the $parser ... > > in the function smwfProcessInlineQueryParserFunction in the file > > SMW_GlobalFunctions.php. > > > > In the MW debugging log file I see that whenever I call a page which > > does not contain any SMW related data (i.e. Special:SpecialPages or just > > edit a new empty page) the function smwfProcessInlineQueryParserFunction > is > > called several times with paramters containing a "typical type of query" > > which are used on other pages of the wiki (although not sure where > > exactly). There seems two types of queries which turn up but not always > > 100% identical. > > > > Reloading i.e. the Special:SpecialPages shows that > > smwfProcessInlineQueryParserFunction is called a different number of times > > (e.g. 9 or 13 times) each time the page is loaded with either the > > same/similar query or a different one (kind of random). > > Calling a page with a SMW query the parameters of > > smwfProcessInlineQueryParserFunction show the expected parameters for that > > query. > > > > I tried to disable all caching in LocalSettings but nothing changed. > > > > Where do these queries come from which are automatically filled into the > > parameters of smwfProcessInlineQueryParserFunction? > > > > Should smwfProcessInlineQueryParserFunction be called at all when loading > a > > page without any SMW stuff? > > Yes that can happen. > > SMW and MediaWiki use "jobs" to execute certain update tasks in small pieces. > > For example, if you change the type of a property, all pages using that > property will be updated with such a job. MediaWiki creates similar jobs when > > templates have changed. Jobs are run during normal wiki operation when > another operation is executed, so it is normal that one call to you wiki will > > lead to several other pages being processed in the background. This explains > > the queries. > > You can view the number of queued jobs on Special:Statistics, and you can use > > the MediaWiki script runJobs.php to execute them directly. Please let us know > > if there is an actual bug in the job generation, e.g. if you find a very > large amount of jobs being constantly queued in the wiki. > > If the jobs are normal but the queries are still too slow, try to upgrade to > > the new SMWSQLStore2 (or wait until the SMW 1.2 release where you find proper > > instructions on that). It is significantly faster on many queries. Upgrading > > an existing SMW SVN installation that already uses the new store will require > > a refreshData with option -vftp and another normal one with option -v (and > possibly -s on large wikis). > > Regards, > > Markus > > > > > Thanks, > > Gu > > > > > > > > ------------------------------------------------------------------------- > > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! > > Studies have shown that voting for your favorite open source project, > > along with a healthy diet, reduces your potential for chronic lameness > > and boredom. Vote Now at http://www.sourceforge.net/community/cca08 > > _______________________________________________ > > Semediawiki-user mailing list > > Sem...@li... > > https://lists.sourceforge.net/lists/listinfo/semediawiki-user > > > > -- > Markus Krötzsch > Semantic MediaWiki http://semantic-mediawiki.org > http://korrekt.org ma...@se... > |
From: Sergey C. <sem...@an...> - 2008-07-07 18:37:25
|
Gu, you might want to kill refreshLinks and htmlCacheUpdate jobs and run refreshLinks.php instead, then just clean cache by touching LocalSettings.php or something else - this might be usual work for you if you have that many pages with one template. At least that's how we do updates on Ardorado.com when popular templates get updated, especially if that's happening multiple times in a row. BTW, can you share which wiki you're doing this on? I'm quite interested in large SMW instances. Thank you, Sergey On Mon, Jul 7, 2008 at 5:45 AM, <zeh...@mo...> wrote: > Oh I see, thanks. Wasn't aware of MW 'jobs' at all. > > My job queue has a length of 26,393 according to Special:Statistics. > > However the table mbw_jobs has 27222 entries where 27125 have > job_cmd='refreshLinks' and 97 have job_cmd='htmlCacheUpdate'. > > I have recently increased the size of the test wiki to about 28000 pages > (which > all include the same template) and I believe I can vaguely remember to have > run > the maintenance script refreshLinks.php at one point in order to try to > sort > something out. I can't remember the details but I didn't know it would > generate > such a long-lasting effect. > > I did run refreshData.php of course but I don't think that's connected with > these jobs (I usually run refreshData.php -p once and than refreshData.php > in > 1000 page id chunks (using the -e option)). > > I'm waiting for a good moment to switch to the new storage version but as I > have > made a number of modifications it will be a bit more work. > > Thanks again for the info, it is rather comforting to have a logical > explanation > for such 'strange' problems. > > Cheers, > Gu > > Quoting Markus Krötzsch <ma...@se...>: > > > On Freitag, 4. Juli 2008, zeh...@mo... wrote: > > > I'm using > > > MediaWiki: 1.11.1 (r30349) > > > Semantic MediaWiki (version 1.2c-SVN) > > > > > > I recently felt that my test wiki was rather slow and I thought I > > > investigate. I basically just added the line > > > > wfDebug("\nsmwfProcessInlineQueryParserFunction:".print_r($params,true)); > > > after the lines > > > $params = func_get_args(); > > > array_shift( $params ); // we already know the $parser ... > > > in the function smwfProcessInlineQueryParserFunction in the file > > > SMW_GlobalFunctions.php. > > > > > > In the MW debugging log file I see that whenever I call a page which > > > does not contain any SMW related data (i.e. Special:SpecialPages or > just > > > edit a new empty page) the function > smwfProcessInlineQueryParserFunction > > is > > > called several times with paramters containing a "typical type of > query" > > > which are used on other pages of the wiki (although not sure where > > > exactly). There seems two types of queries which turn up but not always > > > 100% identical. > > > > > > Reloading i.e. the Special:SpecialPages shows that > > > smwfProcessInlineQueryParserFunction is called a different number of > times > > > (e.g. 9 or 13 times) each time the page is loaded with either the > > > same/similar query or a different one (kind of random). > > > Calling a page with a SMW query the parameters of > > > smwfProcessInlineQueryParserFunction show the expected parameters for > that > > > query. > > > > > > I tried to disable all caching in LocalSettings but nothing changed. > > > > > > Where do these queries come from which are automatically filled into > the > > > parameters of smwfProcessInlineQueryParserFunction? > > > > > > Should smwfProcessInlineQueryParserFunction be called at all when > loading > > a > > > page without any SMW stuff? > > > > Yes that can happen. > > > > SMW and MediaWiki use "jobs" to execute certain update tasks in small > pieces. > > > > For example, if you change the type of a property, all pages using that > > property will be updated with such a job. MediaWiki creates similar jobs > when > > > > templates have changed. Jobs are run during normal wiki operation when > > another operation is executed, so it is normal that one call to you wiki > will > > > > lead to several other pages being processed in the background. This > explains > > > > the queries. > > > > You can view the number of queued jobs on Special:Statistics, and you can > use > > > > the MediaWiki script runJobs.php to execute them directly. Please let us > know > > > > if there is an actual bug in the job generation, e.g. if you find a very > > large amount of jobs being constantly queued in the wiki. > > > > If the jobs are normal but the queries are still too slow, try to upgrade > to > > > > the new SMWSQLStore2 (or wait until the SMW 1.2 release where you find > proper > > > > instructions on that). It is significantly faster on many queries. > Upgrading > > > > an existing SMW SVN installation that already uses the new store will > require > > > > a refreshData with option -vftp and another normal one with option -v > (and > > possibly -s on large wikis). > > > > Regards, > > > > Markus > > > > > > > > Thanks, > > > Gu > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! > > > Studies have shown that voting for your favorite open source project, > > > along with a healthy diet, reduces your potential for chronic lameness > > > and boredom. Vote Now at http://www.sourceforge.net/community/cca08 > > > _______________________________________________ > > > Semediawiki-user mailing list > > > Sem...@li... > > > https://lists.sourceforge.net/lists/listinfo/semediawiki-user > > > > > > > > -- > > Markus Krötzsch > > Semantic MediaWiki http://semantic-mediawiki.org > > http://korrekt.org ma...@se... > > > |
From: <zeh...@mo...> - 2008-07-08 09:59:23
|
Thanks for the tip, I realised that running refreshLinks.php didn't create the jobs but is rather a quicker alternative. I hope eventually templates will be changed very infrequently but right now I'm still experimenting so it will happen more often and it's good to know how to keep the database up-to-date (besides the SMW_refreshData). So far I'm more trying to find ways which allow to query the semantic data and herefore I have been using a wiki with only a small number of pages. But now I want to make sure that the response times are also acceptable with a larger number of pages. So as a second step I have loaded about 6000 main pages each with 4 to 6 subpages (about 28000 pages). That's maybe about half of the total number of content pages expected (I'm not sure if you consider this already 'large'). As query pages can contain several {{#ask}} queries returning unique lists of all values of some common properties (including n-ary parts) it becomes slower just to display those with an increasing number of data pages and therefore properties in the wiki. But considering the slow hardware I use now the speed is still ok (and should even improve when I get the time and courage to update to use the new storage engine). Cheers, Gu Quoting Sergey Chernyshev <sem...@an...>: > Gu, you might want to kill refreshLinks and htmlCacheUpdate jobs and run > refreshLinks.php instead, then just clean cache by touching > LocalSettings.php or something else - this might be usual work for you if > you have that many pages with one template. At least that's how we do > updates on Ardorado.com when popular templates get updated, especially if > that's happening multiple times in a row. > > BTW, can you share which wiki you're doing this on? I'm quite interested in > large SMW instances. > > Thank you, > > Sergey > > > On Mon, Jul 7, 2008 at 5:45 AM, <zeh...@mo...> wrote: > > > Oh I see, thanks. Wasn't aware of MW 'jobs' at all. > > > > My job queue has a length of 26,393 according to Special:Statistics. > > > > However the table mbw_jobs has 27222 entries where 27125 have > > job_cmd='refreshLinks' and 97 have job_cmd='htmlCacheUpdate'. > > > > I have recently increased the size of the test wiki to about 28000 pages > > (which > > all include the same template) and I believe I can vaguely remember to > have > > run > > the maintenance script refreshLinks.php at one point in order to try to > > sort > > something out. I can't remember the details but I didn't know it would > > generate > > such a long-lasting effect. > > > > I did run refreshData.php of course but I don't think that's connected > with > > these jobs (I usually run refreshData.php -p once and than refreshData.php > > in > > 1000 page id chunks (using the -e option)). > > > > I'm waiting for a good moment to switch to the new storage version but as > I > > have > > made a number of modifications it will be a bit more work. > > > > Thanks again for the info, it is rather comforting to have a logical > > explanation > > for such 'strange' problems. > > > > Cheers, > > Gu > > > > Quoting Markus Krötzsch <ma...@se...>: > > > > > On Freitag, 4. Juli 2008, zeh...@mo... wrote: > > > > I'm using > > > > MediaWiki: 1.11.1 (r30349) > > > > Semantic MediaWiki (version 1.2c-SVN) > > > > > > > > I recently felt that my test wiki was rather slow and I thought I > > > > investigate. I basically just added the line > > > > > > wfDebug("\nsmwfProcessInlineQueryParserFunction:".print_r($params,true)); > > > > after the lines > > > > $params = func_get_args(); > > > > array_shift( $params ); // we already know the $parser ... > > > > in the function smwfProcessInlineQueryParserFunction in the file > > > > SMW_GlobalFunctions.php. > > > > > > > > In the MW debugging log file I see that whenever I call a page which > > > > does not contain any SMW related data (i.e. Special:SpecialPages or > > just > > > > edit a new empty page) the function > > smwfProcessInlineQueryParserFunction > > > is > > > > called several times with paramters containing a "typical type of > > query" > > > > which are used on other pages of the wiki (although not sure where > > > > exactly). There seems two types of queries which turn up but not > always > > > > 100% identical. > > > > > > > > Reloading i.e. the Special:SpecialPages shows that > > > > smwfProcessInlineQueryParserFunction is called a different number of > > times > > > > (e.g. 9 or 13 times) each time the page is loaded with either the > > > > same/similar query or a different one (kind of random). > > > > Calling a page with a SMW query the parameters of > > > > smwfProcessInlineQueryParserFunction show the expected parameters for > > that > > > > query. > > > > > > > > I tried to disable all caching in LocalSettings but nothing changed. > > > > > > > > Where do these queries come from which are automatically filled into > > the > > > > parameters of smwfProcessInlineQueryParserFunction? > > > > > > > > Should smwfProcessInlineQueryParserFunction be called at all when > > loading > > > a > > > > page without any SMW stuff? > > > > > > Yes that can happen. > > > > > > SMW and MediaWiki use "jobs" to execute certain update tasks in small > > pieces. > > > > > > For example, if you change the type of a property, all pages using that > > > property will be updated with such a job. MediaWiki creates similar jobs > > when > > > > > > templates have changed. Jobs are run during normal wiki operation when > > > another operation is executed, so it is normal that one call to you wiki > > will > > > > > > lead to several other pages being processed in the background. This > > explains > > > > > > the queries. > > > > > > You can view the number of queued jobs on Special:Statistics, and you > can > > use > > > > > > the MediaWiki script runJobs.php to execute them directly. Please let us > > know > > > > > > if there is an actual bug in the job generation, e.g. if you find a very > > > large amount of jobs being constantly queued in the wiki. > > > > > > If the jobs are normal but the queries are still too slow, try to > upgrade > > to > > > > > > the new SMWSQLStore2 (or wait until the SMW 1.2 release where you find > > proper > > > > > > instructions on that). It is significantly faster on many queries. > > Upgrading > > > > > > an existing SMW SVN installation that already uses the new store will > > require > > > > > > a refreshData with option -vftp and another normal one with option -v > > (and > > > possibly -s on large wikis). > > > > > > Regards, > > > > > > Markus > > > > > > > > > > > Thanks, > > > > Gu > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW! > > > > Studies have shown that voting for your favorite open source project, > > > > along with a healthy diet, reduces your potential for chronic lameness > > > > and boredom. Vote Now at http://www.sourceforge.net/community/cca08 > > > > _______________________________________________ > > > > Semediawiki-user mailing list > > > > Sem...@li... > > > > https://lists.sourceforge.net/lists/listinfo/semediawiki-user > > > > > > > > > > > > -- > > > Markus Krötzsch > > > Semantic MediaWiki http://semantic-mediawiki.org > > > http://korrekt.org ma...@se... > > > > > > |