From: Markus F. <in...@fl...> - 2010-05-13 11:58:32
|
I am struggeling with configuring automatic deduplication in Vufind. I tried to follow: http://wiki.apache.org/solr/Deduplication Actually nothing happens. All records are being imported without any deduplication. What am I missing? Thanks Markus --- In /usr/local/vufind/solr/biblio/conf/solrconfig.xml I added <requestHandler name="/update" class="solr.XmlUpdateRequestHandler" > <lst name="defaults"> <str name="update.processor">dedupe</str> </lst> </requestHandler> <updateRequestProcessorChain name="dedupe"> <processor class="org.apache.solr.update.processor.SignatureUpdateProcessorFactory"> <bool name="enabled">true</bool> <bool name="overwriteDupes">true</bool> <str name="signatureField">dedupeHash</str> <str name="fields">reference,issn</str> <str name="signatureClass">org.apache.solr.update.processor.Lookup3Signature</str> </processor> <processor class="solr.LogUpdateProcessorFactory" /> <processor class="solr.RunUpdateProcessorFactory" /> </updateRequestProcessorChain> <!-- The field "reference" corresponds to MARC 773g, is mapped in marc.poperties and has also an additional entry in schema.xml --> --- In /usr/local/vufind/solr/biblio/conf/schema.xml I added the field <field name="dedupeHash" type="string" stored="true" indexed="true" multiValued="false" /> |
From: Demian K. <dem...@vi...> - 2010-05-13 12:45:26
|
I've never worked with this feature myself, and I'm not sure if any other VuFinders have either -- you might actually get a better response from the solr-user list. One first thought, though -- as a sanity check, have you tried comparing the signature values between two known duplicate records? That might help point you in the direction of the problem (i.e. either the signature isn't generating the way you think it is, or something else in the chain isn't acting correctly on duplicate signatures). - Demian > -----Original Message----- > From: Markus Fischer [mailto:in...@fl...] > Sent: Thursday, May 13, 2010 7:58 AM > To: vuf...@li... > Subject: [VuFind-Tech] SOLR built in deduplication > > I am struggeling with configuring automatic deduplication in Vufind. I > tried to follow: > > http://wiki.apache.org/solr/Deduplication > > Actually nothing happens. All records are being imported without any > deduplication. > > What am I missing? > > Thanks > Markus > > --- > In /usr/local/vufind/solr/biblio/conf/solrconfig.xml I added > > <requestHandler name="/update" class="solr.XmlUpdateRequestHandler" > > <lst name="defaults"> > <str name="update.processor">dedupe</str> > </lst> > </requestHandler> > > <updateRequestProcessorChain name="dedupe"> > <processor > class="org.apache.solr.update.processor.SignatureUpdateProcessorFactory > "> > <bool name="enabled">true</bool> > <bool name="overwriteDupes">true</bool> > <str name="signatureField">dedupeHash</str> > <str name="fields">reference,issn</str> > <str > name="signatureClass">org.apache.solr.update.processor.Lookup3Signature > </str> > </processor> > <processor class="solr.LogUpdateProcessorFactory" /> > <processor class="solr.RunUpdateProcessorFactory" /> > </updateRequestProcessorChain> > > <!-- The field "reference" corresponds to MARC 773g, is mapped in > marc.poperties and has also an additional entry in schema.xml --> > > --- > In /usr/local/vufind/solr/biblio/conf/schema.xml I added the field > > <field name="dedupeHash" type="string" stored="true" indexed="true" > multiValued="false" /> > > ----------------------------------------------------------------------- > ------- > > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech |