John,

You may want to check SMW trunk.  The built-in csv result format now uses php://temp instead of php://memory so you should now be able to export large csv files without using the SRF-Ploticus workaround.

And since the built-in now works for large result sets, I also cleaned up and removed the csv imageformat in Ploticus to tighten the code and eliminate the duplicate functionality.

See https://bugzilla.wikimedia.org/17760 for more info.

If you don't want to live on the edge and use code from trunk, you can simply change:

    $csv = fopen('php://memory', 'r+'); 
to 
   $csv = fopen('php://temp', 'r+');

in includes\SMW_QP_CSV.php.

Best,
Joel

On Feb 19, 2009, at 1:52 PM, johnmajor wrote:


A few general responses:

0) SMW is often pitched as an extension Wikipedia should adopt.  Wikipedia
is going on 3million english pages, and I can easily imagine queries
returning 10s of thousands of rows (return all species and geographic ranges
of butterflies).
SMW *is* incredibly powerful, but if it is seriously not meant to scale past
a few thousand pages, the general utility is going to be small.

1)Thanks for the good suggestions as to how to extract these large datasets
outside of SMW.  I already make heavy use of mwclient, and have a shortcut
solution for crudely embedding results of large queries (ie: using the
embedded SQL extension in abusive ways).  However, it seems a key feature of
SMW is the ability to query the semantic content stored in the wiki, and
present it in wiki pages.  Using bots, and processing data dumps, will allow
me to answer the question  "give me the average age of my 200,000 users",
but does not allow me to embed the result(in a clean fashion) in a page-
truth be told, nor does my inline SQL query.
Help SMW! you're my only hope.

2) I completely agree. It is immensely helpful to read the message boards,
documentation, and source code.  However, this exact topic was not covered,
or at least I did not find a clear discussion. Hence my post, which I think
the discussion around will be useful to future hunters of information about
scalability issues surrounding SMW.



--
View this message in context: http://www.nabble.com/Odd-behaviour-of-SMW-query-results-in-CSV-format....-truncated-results%2C-memory-exceptions%2C-and-%27Are-there-streaming-output-types-%27-tp22088074p22107249.html
Sent from the Semantic Mediawiki - User mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Semediawiki-user mailing list
Semediawiki-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/semediawiki-user

--
Joel Natividad
joel@natividads.com
Think Different!