Re: [Semediawiki-user] SMW using all available memory for large query

[Brian <Bri...@co...>]

Regarding this issue, I eventually dumped Category:Papers using mysql,
stripped out all of the Paper templates and converted them to bibtex (all in
Python, using MySQLdb). The entire process takes less than three seconds for
almost 20,000 papers and produces an 18MB file. This is clearly the way to
do export if you need it to be fast.

Marty, I do plan to release the paper system. I am still working on it on a
day-to-day basis and it is running on a private wiki. I've already got
enough beta users, but I will post to this list as soon as I GPL the source.

Cheers,

On Tue, Jan 13, 2009 at 2:44 PM, moostro <wei...@gm...> wrote:

>
> Dear Brian,
>
> I wonder whether you might provide a link to the SMW+SF bibliography system
> that you have designed.
>
> I've put together something much smaller, and I never finished this work.
> My needs are also smaller (on the order of a few hundred sources, or maybe
> a
> thousand eventually).  Perhaps I could use your work on my SMW+SF
> installation, and save some time, if you don't mind sharing.
>
> best,
>
> Marty
>
>
> Brian-3 wrote:
> >
> > I have managed to turn SMW+SF into a bibliography system. I have styled
> > the
> > properties based on the allowed bibtex fields. Then, I imported our
> > bibliography, and now I am working on export. I managed to get #ask to
> > output bibtex using a template, and it works quite well for smallish
> > queries. However, we have 15,000 pages in Category:Papers, and when I try
> > to
> > export the whole thing SMW because a memory hog. I gave it up to 1.5 GB
> of
> > memoryand after 10 minutes it failed running out of memory.
> >
> > I will provide the key bits of code if they are helpful:
> >
> >    - Template:Paper http://pastebin.ca/1292085
> >       - For adding a new paper
> >       - Template:Bibtex http://pastebin.ca/1292087
> >       - Used to create bibtex output from #ask
> >    - The #ask query: http://pastebin.ca/index.php
> >       - http://pastebin.ca/1292088
> >
> > My hope was that this bib system would allow my users to grow our
> > bibliography to > 100,000 items, but I have to come up with some method
> of
> > exporting the whole thing.
> >
> > Thoughts?
> >
> > Cheers,
> > Brian
> >
> > --
> > (Not sent from my iPhone)
> >
> >
> ------------------------------------------------------------------------------
> >
> > _______________________________________________
> > Semediawiki-user mailing list
> > Sem...@li...
> > https://lists.sourceforge.net/lists/listinfo/semediawiki-user
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/SMW-using-all-available-memory-for-large-query-tp21140220p21445173.html
> Sent from the Semantic Mediawiki - User mailing list archive at Nabble.com.
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by:
> SourcForge Community
> SourceForge wants to tell your story.
> http://p.sf.net/sfu/sf-spreadtheword
> _______________________________________________
> Semediawiki-user mailing list
> Sem...@li...
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>