Re: [VuFind-General] Harvester memory issue?

[Demian K. <dem...@vi...>]

Thanks for noticing this -- the harvester was using recursion, so it was a bit of a memory hog.  I have rewritten it to run through a flat loop, and it should now be significantly more efficient.  You can download the fixed version here:

https://vufind.svn.sourceforge.net/svnroot/vufind/trunk/harvest/harvest_oai.php

Please let me know if you still have trouble!

- Demian

> -----Original Message-----
> From: Lingley, Dean R [mailto:dli...@pu...]
> Sent: Monday, March 14, 2011 6:43 PM
> To: Demian Katz
> Cc: 'vuf...@li...'; Riehle, Matthew T
> Subject: Harvester memory issue?
> 
> Hi Demian,
> Trying to harvest hathitrust and when it gets to about 180,000 recs the
> process dies due to lack of memory, just curious if you've tried to
> harvest a collection this large or ran into a similar issue?  Memory
> usage seems to increase with each 500 recs harvested.
> 
> Thanks,
> Dean