I just finished up some automation scripts to get records from Archivists’ Toolkit into VuFind as marc records. The scripts worked fine in tests, but yesterday when I went to do the initial import of over 6,100 records it was impossibly slow. The solr injest was taking 22 minutes per record and it only got through 10 of them before the server went down and stopped the process.
After transformation, each marc record is stored on the VuFind server in an individual file. A separate script feeds each file to import-marc.sh as it is ready and it looks like that is where it is choking. Any ideas on how to speed up this initial import? After this initial batch we shouldn’t be injesting more than a handful of records/updates per day.
Director of Digital Collections and Systems
The Historical Society of Pennsylvania
1300 Locust Street
Philadelphia, PA 19107
Tel: 215/732-6200 ext. 201