From: Tod O. <to...@uc...> - 2012-12-11 22:08:02
|
Returning to capturing output from the harvest scripts, I'd like some input on a minor point. Currently stdout gets informative messages like so: Now Importing /data/magma/vufind2/local/harvest/auth/auth_full_marc_utf-8_00_121206230000.mrc ... /usr/local/bin/java -Xms512m -Xmx512m -Dsolr.core.name=authority -Dsolr.indexer.properties=/data/magma/vufind2/import/marc_auth.properties,/data/magma/vufind2/import/marc_auth.properties -jar /data/magma/vufind2/import/SolrMarc.jar /data/magma/vufind2/local/import/import_auth.properties /data/magma/vufind2/local/harvest/auth/auth_full_marc_utf-8_00_121206230000.mrc and all of the solrmarc messages (record number, stack traces on failure, etc.) go to stderr. I kind of think that the options are: (a) everything goes to a log file, (b) stdout can go to the terminal and stderr should go to the log file, or (c) maybe that chatty "Now importing..." goes to stdout/terminal and all else goes to stderr/the log. Are there any strong feelings about which is the right way? Personally, I'm kind of inclined towards (c), but maybe sites who are in production have a different view. -Tod On Oct 31, 2012, at 12:32 PM, Demian Katz <demian.katz@VILLANOVA.EDU> wrote: > I don't have a problem with changing the batch MARC import scripts to capture stderr; I believe that when they were originally written, all SolrMarc output was written to stdout -- it began using stderr more appropriately in relatively recent updates. > > The only other refactoring you might need to do is to allow a way of specifying a full directory path -- right now, the scripts assume that all files live under VuFind's harvest directory, but in a situation not linked to the OAI harvester, the files might be somewhere else. You might also want to add a switch to disable the "move to processed directory" functionality and/or a switch to control logging (i.e. optionally disable by sending to null). > > - Demian > >> -----Original Message----- >> From: Tod Olson [mailto:to...@uc...] >> Sent: Wednesday, October 31, 2012 1:14 PM >> To: Demian Katz >> Cc: Tod Olson; vuf...@li... Tech Mailinglist >> Subject: Re: VF2.0 import scripts taking more that one file >> >> Yes, looking at the harvest/ scripts for marc records, I see see that stdout >> is directed to an output file, but stderr is not written to disk. Since stderr >> has all of the error info, I'm inclined to capture it. I can also see where >> people would not want the error logs taking up disc space, since there's a >> message for every record. But sending all output to a file is a little more >> cron-friendly. >> >> I may be willing to refactor a couple of those batch scripts (no commitment >> yet), but I'd like a little input on what sort of requirements other sites >> would have. >> >> -Tod >> >> On Oct 31, 2012, at 11:22 AM, Tod Olson <to...@uc...> >> wrote: >> >>> Aha, I'd dismissed harvest as exclusively the province of OAI. Thanks for >> correcting that. >>> >>> I'll pop a patch into JIRA when I can. >>> >>> -Tod >>> >>> On Oct 30, 2012, at 10:52 PM, Demian Katz <dem...@vi...> >>> wrote: >>> >>>> There are batch import scripts in the harvest directory -- you might be >> able to use those. If not, perhaps some refactoring can make all the existing >> tools more flexible. Also, if you add -p support to the auth script, please >> submit a patch and I'll be happy to merge that into master. >>>> >>>> thanks, >>>> Demian >>>> ________________________________________ >>>> From: Tod Olson [to...@uc...] >>>> Sent: Tuesday, October 30, 2012 8:06 PM >>>> To: vuf...@li... Tech Mailinglist >>>> Subject: [VuFind-Tech] VF2.0 import scripts taking more that one file >>>> >>>> I find that it would be useful for my site if the import-marc.sh and >> import-marc-auth.sh. I could easily hack those two shell scripts to take some >> arbitrary number of files as arguments and loop over them, and submit a patch. >> Would that be of use to other sites? >>>> >>>> Otherwise, I'll just write wrappers around them for local use. >>>> >>>> The one interface change that I'd want to implement: it would be easier if >> I changed import-marc-auth.sh to take a profile file with a -p argument like >> import-marc.sh. >>>> >>>> -Tod >>>> >>>> Tod Olson <to...@uc...> >>>> Systems Librarian >>>> University of Chicago Library >>>> >>>> >>>> >>>> >>>> --------------------------------------------------------------------------- >> --- >>>> Everyone hates slow websites. So do we. >>>> Make your web apps faster with AppDynamics >>>> Download AppDynamics Lite for free today: >>>> http://p.sf.net/sfu/appdyn_sfd2d_oct >>>> _______________________________________________ >>>> Vufind-tech mailing list >>>> Vuf...@li... >>>> https://lists.sourceforge.net/lists/listinfo/vufind-tech >>> > |