Re: [VuFind-Tech] question about harvesting hathi trust records using the OAI service

[Erik M. <mit...@wf...>]

Thanks!

Erik



Erik Mitchell, Ph.D.
Assistant Director for Technology Services
Z. Smith Reynolds Library
Wake Forest University
mit...@wf...
336.655.5290

On Nov 9, 2010, at 9:47 AM, Demian Katz <dem...@vi...> wrote:

 In the interest of modularity, harvest and index are separate actions…  so
when you harvest with OAI, you download a bunch of XML files into a
directory.  You then need to run an import script to load those files from
the directory into your Solr index.  The harvest directory contains a
batch-import-marc.sh script which you can use to send all of the MARCXML
files in a directory to SolrMarc for indexing.  This will probably serve
your purposes, though it is currently extremely slow because it loads a
separate instance of SolrMarc for each individual record.  It would probably
be more efficient to introduce an intermediate layer to combine the separate
XML files into one big blob prior to indexing for improved performance, but
I haven’t had a chance to work on that problem yet.



Let me know if you need more help!



- Demian



*From:* Mitchell, Erik [mailto:mit...@wf...]
*Sent:* Tuesday, November 09, 2010 9:02 AM
*To:* vuf...@li...
*Subject:* [VuFind-Tech] question about harvesting hathi trust records using
the OAI service



Hi all,

I may be missing something completely obvious here but I have been doing
some harvesting of hathi trust records using the new OAI tool and am having
issues understanding how the indexing process works.  I am beginning to get
the idea that I need to run the import process separately from the harvest
process?  Do I also need to create Marc records from the MARCXML that I am
getting from hathi?

many thanks :)

Erik

-- 
Erik Mitchell, Ph.D.
Assistant Director for Technology Services
Z. Smith Reynolds Library
Wake Forest University
http://erikmitchell.info