Hello all,

 

I just finished up some automation scripts to get records from Archivists’ Toolkit into VuFind as marc records.  The scripts worked fine in tests, but yesterday when I went to do the initial import of over 6,100 records it was impossibly slow.  The solr injest was taking 22 minutes per record and it only got through 10 of them before the server went down and stopped the process. 

 

After transformation, each marc record is stored on the VuFind server in an individual file.  A separate script feeds each file to import-marc.sh as it is ready and it looks like that is where it is choking.  Any ideas on how to speed up this initial import?  After this initial batch we shouldn’t be injesting more than a handful of records/updates per day.

 

Matt Shoemaker

Director of Digital Collections and Systems

The Historical Society of Pennsylvania

1300 Locust Street

Philadelphia, PA 19107

Tel: 215/732-6200 ext. 201

Fax: 215/732-2680

mshoemaker@hsp.org