Menu
▾
▴
Re: [VuFind-General] Remove records
|
From: Robert H. <rh...@vi...> - 2009-07-23 16:28:28
|
You can already easily do both of the following, although as Alan Rykhus points out in nearly every case where you might want to use the 'ignore' capability, the 'delete' will accomplish the job better. To do the delete you could add the following to your index spec: delete = 998a, (map.delete_map), DeleteRecordIfFieldEmpty map.delete_map.d = null map.delete_map.displayRawIfMissing = original value The advantage of implementing it this way (as opposed to the way suggested by Andrew below) is that if there are several different values that indicate that a given record should be deleted, it is very easy to implement. I guess one disadvantage is that the field 'delete' would be added to the solr index for that record, and the schema would have to be modified to accept that additional field. To implement the 'ignore' capability, in the new SolrMarc release there is a utility function called 'filterrecords' that can be used as unix stream filter to only pass records that meet a certain criteria. Which I guess is the opposite of what Andrew proposes. cat records.mrc | filterrecords '998a/k' | indexfile which would only process records that contain 'k' in a 998a subfield. But since the filtersrecords program is merely a shell script that sets the marc.include_if_present property, it would be fairly easy to create a different script that uses the marc.include_if_missing property and set the value to 998a/s which would cause the filter program to only pass through records that didn't have a 998a subfield, or if it has a 998a subfield only those records that don't have a 's' value in that field. Again as Alan Rykhus points out, care must be used when using any 'ignore' so that if you are updating an existing index any records in the input that are ignored, that already exist in the index will remain unchanged. This can be useful to speed up reindexing in the case where a new index rule is added, or an existing index rule is changed, so that you can reindex only those records that you know will be produce different results based on the new indexing spec. -Bob Haschart Andrew Nagy wrote: > A nice addition to the marc map properties file in solrmarc would be > the following: > > delete = 998a=d > > What this would do is delete any records that come through where the > 998a field has the value of 'd'. Additionally it would beneficial to > have an ignore for suppressed records such as: > > ignore = 998a=s > > When a record has value of 's' in the 998a field - the record would be > ignored. > > I will take a look at coding this in - unless someone has the urge to > build it quickly. > > Andrew > > On Wed, Jul 22, 2009 at 5:38 PM, Robert Haschart <rh...@vi... > <mailto:rh...@vi...>> wrote: > > Well then I have good news. The ability to specify that a record > should be deleted based on the results of indexing a particular > field is already in SolrMarc. > > For our catalog certain "locations" for items are used to indicate > that a particular copy of an item should not be visible. In our > location_map.properties these locations are explicitly mapped to > "null". Based on the following rule, if all copies of a given > item should not be visible, than the item itself is deleted from > the index (not just not indexed, actually deleted, so that if > daily update changed the last visible copy of an item to a hidden > location, the index entry is deleted.) > > location_facet = 999k:999l, location_map.properties, > DeleteRecordIfFieldEmpty > > Additionally if a custom indexing function is needed for indexing > but again the end result should be that the item is deleted if > custom function returns an empty set you can do the following: > > shadowed_location_facet = customDeleteRecordIfFieldEmpty, > getShadowedLocation(shadowed_location_map.properties, > deleteHIDDEN, noExtraIds) > > > Lastly there is also already a capability of providing the > SolrMarc indexer a list of ids in a text file (with a .del > suffix) and all of the ids listed will be deleted. > > -Bob Haschart > > > Andrew Nagy wrote: > >> Cool! I know Eric may not be necessarily working with Marc >> records - but we do hope to add delete capabilities to SolrMarc. >> This way you can add to the marc mapping what field you use to >> denote a deleted record and let SolrMarc handle the rest. >> >> Andrew >> >> On Thu, Jul 16, 2009 at 11:45 AM, Eric Lease Morgan >> <em...@nd... <mailto:em...@nd...>> wrote: >> >> >> On Jul 16, 2009, at 11:05 AM, >> vuf...@li... >> <mailto:vuf...@li...> >> wrote: >> >> >> How do I remove records in batch that I loaded in error? >> > >> > Grant - I have been building some record "maintenance" >> features in >> > the admin >> > module with VuFind but it's not fully fleshed out and ready >> for use. >> > >> > Until then - you can delete records from the Solr index >> directly. >> > Here is >> > the documentation on this. Let me know if you have any >> questions on >> > how to >> > do this. >> > >> > >> http://wiki.apache.org/solr/UpdateXmlMessages#head-f5553a8214869b11708ad5476e05e7b05a024c23 >> >> >> >> I have also thought about this problem, and through my >> investigations >> I saw a file named util/deletes.php which may do the job for >> you, but >> since I broke my PHP implementation I no longer have the PHP >> command- >> line utility. I hope to write soon something a delete >> application in >> Perl. It should be less than a dozen lines long. >> >> -- >> Eric Lease Morgan >> Head, Digital Access and Information Architecture Department >> Hesburgh Libraries, University of Notre Dame >> >> (574) 631-8604 >> >> >> >> >> >> ------------------------------------------------------------------------------ >> Enter the BlackBerry Developer Challenge >> This is your chance to win up to $100,000 in prizes! For a >> limited time, >> vendors submitting new applications to BlackBerry App >> World(TM) will have >> the opportunity to enter the BlackBerry Developer Challenge. >> See full prize >> details at: http://p.sf.net/sfu/Challenge >> _______________________________________________ >> VuFind-General mailing list >> VuF...@li... >> <mailto:VuF...@li...> >> https://lists.sourceforge.net/lists/listinfo/vufind-general >> >> >>------------------------------------------------------------------------ >> ------------------------------------------------------------------------------ >> Enter the BlackBerry Developer Challenge This is your chance to >> win up to $100,000 in prizes! For a limited time, vendors >> submitting new applications to BlackBerry App World(TM) will have >> the opportunity to enter the BlackBerry Developer Challenge. See >> full prize details at: http://p.sf.net/sfu/Challenge >> >>------------------------------------------------------------------------ >> >>_______________________________________________ >>VuFind-General mailing list >>VuF...@li... <mailto:VuF...@li...> >>https://lists.sourceforge.net/lists/listinfo/vufind-general >> >> > > |