VuFind

> But anyway, as a result of all of this, our codebases nowhere near match
> anymore and it's unlikely we'll ever completely rejoin the trunk.

I'd love to see reunification eventually, not just so the trunk can steal your good stuff but also so that you can benefit from the improvements in the mainline code...  but I understand that we've probably gone well beyond the point where that is practical, and if you're comfortable with your local version, there's no reason to shove the mainline down your throat!  Of course, each of your innovations that we migrate into the trunk gets us a little closer to being on the same page.

> That's not a huge source of concern for us, except that we would still
> like to see others benefit from the good bits of what we've done where
> possible, so maybe a sensible way forward would be for me to check out
> your authority branch and attempt to merge my browse code into it,
> keeping track of any changes to existing code I make along the way.
> Does that sound workable?

That definitely sounds workable.  If you want to create a branch of my branch in order to achieve this, by all means go ahead -- I assume you already have commit access, but please let me know if you need anything.  A few things are going to change by the time the branch gets merged into the trunk (mostly to do with the import process, since the next SolrMarc release will allow me to clean up some of the hacky BeanShell stuff I've done), but I think the essentials should remain pretty much the same.

- Demian

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Ya'aqov Ziso <yaaqovz@gm...> - 2010-08-12 17:52 Attachments: Message as HTML      *Mark, extra baggage overhead and expenses are involved in current authority processing. Once the enhancement/simplification Demian, Ralph and I are working on will be completed for VuFind, you at NAL may be tempted otherwise -- :) Cheers, Ya'aqov On Wed, Aug 11, 2010 at 8:00 PM, Mark Triggs <mtriggs@...> wrote: > Ah! Sorry, my explanation reflects my tech-centric view of the world: I > just process the data without worrying where it comes from :) > > I think you'll find us at the NLA particularly uninteresting, then, > because we don't do any sort of automated bulk updates of our authority > data. No vendor-supplied updates or anything like that: any changes > made are done by our cataloguing staff on a case-by-case basis. Our > staff might pull in new headings from the Library of Congress, but > that's an ad-hoc sort of procedure. > > So, from the point of view of my browse, I'm just focusing on tracking > updates to our own authority data, nothing more broad than that. > > Cheers, > > Mark > > > Ya'aqov Ziso <yaaqovz@...> writes: > >> Mark, yes, we are getting clearer. Authority data is not >> static. Similarly to the bib indexes being dynamically changing due to >> adds, edits, and deletes -- authority indexes are being dynamically >> updated as well by or via the Library of Congress. >> >> Libraries' access to these authority data/indexes is achieved usually >> via a vendor. My initial question to you has been: how often are you >> updating authority data (whereby authority data/files are available to >> you probably via vendor) ? >> >> [if your access to those files is different, not via a vendor, how >> so?] >> >> The project Demian and I are engaged in involves access to THAT >> authority data, attempting to make it available for faceted displays >> of authority data available globally (via WorldCat) AND separately, >> access to the authority data available locally (via vendor+processing >> such as what you described). > > -- > Mark Triggs > <mtriggs@...> > -- if you're not part of the problem, you're not part of the solution | 856 217 3456 | yaaQovz@... *

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Mark Triggs <mtriggs@nl...> - 2010-08-13 00:29 Demian Katz <demian.katz@...> writes: >> But anyway, as a result of all of this, our codebases nowhere near match >> anymore and it's unlikely we'll ever completely rejoin the trunk. > > I'd love to see reunification eventually, not just so the trunk can > steal your good stuff but also so that you can benefit from the > improvements in the mainline code... but I understand that we've > probably gone well beyond the point where that is practical, and if > you're comfortable with your local version, there's no reason to shove > the mainline down your throat! Of course, each of your innovations > that we migrate into the trunk gets us a little closer to being on the > same page. Yep, agreed all round, and I'm always happy to do the legwork required to merge things from our code back into the main trunk. Most of our stuff isn't *too* bizarre, so cleaning up and merging shouldn't be too much effort in general. > That definitely sounds workable. If you want to create a branch of my > branch in order to achieve this, by all means go ahead -- I assume you > already have commit access, but please let me know if you need > anything. A few things are going to change by the time the branch > gets merged into the trunk (mostly to do with the import process, > since the next SolrMarc release will allow me to clean up some of the > hacky BeanShell stuff I've done), but I think the essentials should > remain pretty much the same. Excellent, sounds good. I've pushed my preliminary work into: https://vufind.svn.sourceforge.net/svnroot/vufind/branches/authority-with-alphabetic-browse and will keep fiddling about with it today. So far I've got the basic browse indexing working, and have my Solr browse request handler hooked into the biblio Solr core and giving results that look about right for topic browse. Next I'll look at adding some sort of basic web interface and add a couple more example browses (title and author, I guess). One thing I've hit that I wasn't completely sure about was what to do with my Java sources and the .jar files they produce. I see a couple of .jar files in the SVN repository that don't have corresponding source files anywhere, but I felt funny about including .jar files without also providing their source code. At the moment I've just checked in the whole lot--source files, an ant build script, and the .jar files it produces. Mark -- Mark Triggs Systems Administrator Business Systems Support, The National Library of Australia <mtriggs@...>

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Mark Triggs <mtriggs@nl...> - 2010-08-13 07:51 Just to follow up to myself, I've just checked in my work for the afternoon. I've got a basic browse interface now (linked from the footer), and browses for authors and topics. To test it out: * check out the URL below and configure things as normal * Index your auth data with the ./import-marc-auth.sh script * Index your bib data with the ./import-marc.sh script * Create the SQLite bits using: cd alphabetic-browse ant jars ./index.sh * Fire it all up and hope for the best ;) Cheers, Mark Mark Triggs <mtriggs@...> writes: > Excellent, sounds good. I've pushed my preliminary work into: > > https://vufind.svn.sourceforge.net/svnroot/vufind/branches/authority-with-alphabetic-browse > > and will keep fiddling about with it today. -- Mark Triggs Systems Administrator Business Systems Support, The National Library of Australia <mtriggs@...>

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Demian Katz <demian.katz@vi...> - 2010-08-13 12:51 > Excellent, sounds good. I've pushed my preliminary work into: > > https://vufind.svn.sourceforge.net/svnroot/vufind/branches/authority- > with-alphabetic-browse > > and will keep fiddling about with it today. Great, thanks! Looks like you've made good progress already. This reminds me of an unrelated subject -- are you still using the nla and/or nla.dead branches? Seems a pretty good chance that nla.dead is unnecessary, and even the main nla one hasn't changed since 2008. You're welcome to keep them if you need them, but if they're obsolete, perhaps we should delete them to clean up the branches folder a bit. > One thing I've hit that I wasn't completely sure about was what to do > with my Java sources and the .jar files they produce. I see a couple of > .jar files in the SVN repository that don't have corresponding source > files anywhere, but I felt funny about including .jar files without also > providing their source code. At the moment I've just checked in the > whole lot--source files, an ant build script, and the .jar files it > produces. Most of the .jar files included with VuFind come from other projects (i.e. SolrMarc) and thus have their source code represented elsewhere. I think there might also be one or two that actually have the source embedded inside the .jar. If there's some example you're thinking of that isn't covered by either of these cases, please let me know and I'll investigate further. In any case, I don't have a problem with including some Java code with VuFind if that's the most logical place to keep it, though I wonder in this case if it would actually make more sense to keep the Java somewhere in your alphabetic browse repository and then just move the jar files over to VuFind. I assume your new code is branched from your old code, and keeping it all in one place would seem to make it easier to maintain. Putting the code in VuFind's repository invites people to change it, and if alternate versions exist in some other repository, it seems likely to lead to confusion. Perhaps I'm misunderstanding the nature of the code you're talking about, though! - Demian

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Mark Triggs <mtriggs@nl...> - 2010-08-14 01:36 Demian Katz <demian.katz@...> writes: >> Excellent, sounds good. I've pushed my preliminary work into: >> >> https://vufind.svn.sourceforge.net/svnroot/vufind/branches/authority- >> with-alphabetic-browse >> >> and will keep fiddling about with it today. > > Great, thanks! Looks like you've made good progress already. > > This reminds me of an unrelated subject -- are you still using the nla > and/or nla.dead branches? Seems a pretty good chance that nla.dead is > unnecessary, and even the main nla one hasn't changed since 2008. > You're welcome to keep them if you need them, but if they're obsolete, > perhaps we should delete them to clean up the branches folder a bit. Oops, I'd forgotten those were there. The ".dead" one was certainly not important (any system I use for more than about three minutes has numerous .dead files ;), and the "nla" one was a years-old old attempt at merging our search code back in the trunk. I've deleted both now. > Most of the .jar files included with VuFind come from other projects > (i.e. SolrMarc) and thus have their source code represented elsewhere. > I think there might also be one or two that actually have the source > embedded inside the .jar. If there's some example you're thinking of > that isn't covered by either of these cases, please let me know and > I'll investigate further. Ah, that makes sense, thanks. > I assume your new code is branched from your old code, and keeping it > all in one place would seem to make it easier to maintain. Putting > the code in VuFind's repository invites people to change it, and if > alternate versions exist in some other repository, it seems likely to > lead to confusion. Perhaps I'm misunderstanding the nature of the > code you're talking about, though! Not at all--I think that's a good approach. I've merged the changes I've made for VuFind back into: https://code.nla.gov.au/redmine/projects/nlabrowse and have just included the .jar files I built from that source tree. The build process for VuFind should be pretty simple--just a checkout of that code, and "ant jars -Dsolr.war=/usr/local/vufind/solr/jetty/webapps/solr.war", as described in the README. I've also added in title browse this morning, so I'm hoping that things are now at a point where it'd be worth someone else trying to get this working to see which of my assumptions were wrong :) Here's the steps I've followed: $ svn co https://vufind.svn.sourceforge.net/svnroot/vufind/branches/authority-with-alphabetic-browse/ [the usual installation/configuration steps for a fresh install] $ VUFIND_HOME=$PWD ./import-marc.sh ~/bibdump.mrc $ VUFIND_HOME=$PWD ./import-marc-auth.sh ~/authdump.mrc $ ./index-alphabetic-browse.sh $ ./vufind.sh start and then follow the new link from the footer to try it out. I haven't made any attempt at Windows support just yet, because I don't have much in the way of batch-file-fu, nor access to a Windows machine to test it on (life's good ;). I think Windows support would just require an index-alphabetic-browse.sh and the Windows port of GNU sort we discussed before, though, if anyone wants to have a go... Enjoy your weekend, Mark -- Mark Triggs <mtriggs@...>

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Demian Katz <demian.katz@vi...> - 2010-08-16 16:03 > Oops, I'd forgotten those were there. The ".dead" one was certainly not > important (any system I use for more than about three minutes has > numerous .dead files ;), and the "nla" one was a years-old old attempt > at merging our search code back in the trunk. I've deleted both now. Thanks! > I've also added in title browse this morning, so I'm hoping that things > are now at a point where it'd be worth someone else trying to get this > working to see which of my assumptions were wrong :) Great -- I'll try to start playing with this either this week or next, depending on how my pre-vacation schedule works out. One thought in the meantime: one feature of the authority index I put together is that it can hold data from a variety of different sources. For example, you might index local records exported from your ILS plus the full Name Authority File and Subject Headings from the Library of Congress. The "source" facet is designed to allow control over this -- you can filter down to just the part you want. I assume that your tool doesn't currently account for this. At a glance, it looks like you aren't specifying a secondary .properties file for import-marc-auth.sh, which will result in the default "Unknown" value for source, and I'm guessing that when the browse headings are created, they're based on the whole index rather than a particular facet. It may not be worth worrying too much about this until more use cases actually exist... but being able to create browse headings based on a particular source facet value will probably be a useful feature in the future. - Demian

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Mark Triggs <mtriggs@nl...> - 2010-08-17 00:48 Hi Demian, Demian Katz <demian.katz@...> writes: >> I've also added in title browse this morning, so I'm hoping that things >> are now at a point where it'd be worth someone else trying to get this >> working to see which of my assumptions were wrong :) > > Great -- I'll try to start playing with this either this week or next, > depending on how my pre-vacation schedule works out. Excellent, thanks. > One thought in the meantime: one feature of the authority index I put > together is that it can hold data from a variety of different sources. > For example, you might index local records exported from your ILS plus > the full Name Authority File and Subject Headings from the Library of > Congress. The "source" facet is designed to allow control over this > -- you can filter down to just the part you want. I assume that your > tool doesn't currently account for this. At a glance, it looks like > you aren't specifying a secondary .properties file for > import-marc-auth.sh, which will result in the default "Unknown" value > for source, and I'm guessing that when the browse headings are > created, they're based on the whole index rather than a particular > facet. It may not be worth worrying too much about this until more > use cases actually exist... but being able to create browse headings > based on a particular source facet value will probably be a useful > feature in the future. You're right that it currently just uses the entire index, but changing it to limit to a subset of the authority records probably isn't a huge deal. The code that pulls out the headings: https://code.nla.gov.au/redmine/projects/nlabrowse/repository/revisions/master/entry/browse-indexing/PrintBrowseHeadings.java#L135 already has some notion of a predicate to determine whether a given heading should be extracted or not, so this mechanism could potentially be used to limit to certain sources. Besides that, the browse handler would just need to be changed to apply a filter to the TermQueries it does when populating a browse. I'll continue to mull it over, and maybe we can discuss this further if/when some use cases pop up? I don't think it would require any fundamental changes to the way the browse currently works, and it's always nice to add features based on feedback from users :) Thanks (and enjoy your vacation!), Mark -- Mark Triggs Systems Administrator Business Systems Support, The National Library of Australia <mtriggs@...>

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Ya'aqov Ziso <yaaqovz@gm...> - 2010-08-17 02:20 Attachments: Message as HTML      ....Besides that, the browse handler would just need to be changed to apply a filter to the TermQueries it does when populating a browse. I'll continue to mull it over, and maybe we can discuss this further if/when some use cases pop up? I don't think it would require any fundamental changes to the way the browse currently works, .... =========== Mark, please indulge my questions/comments: 1. Let's assume someone's browsing the name heading 'Twiggs, Mark, 1980-' If there's an authority record for that name heading, its record will be found in the global Name Authority File (NAF) index; if NAL (local) has a work needing that heading, the SAME authority record in NAF will be called, and that SAME record will have tagging indicating it's also been processed locally for NAL. Does this scheme work out with your suggestion? 2. I'm not sure how you want to test queries to your local authority index if what you have is a locally produced (by your local catalogers) index. I believe you need a copy of global NAF and a method to tag local NAL's headings in NAF, distinguishing local from global. Please correct me if necessary. Ya'aqov

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Mark Triggs <mtriggs@nl...> - 2010-08-17 03:02 Hi Ya'aqov, Ya'aqov Ziso <yaaqovz@...> writes: > ....Besides that, the browse handler would just need to be changed to > apply a filter to the TermQueries it does when populating a > browse. I'll continue to mull it over, and maybe we can discuss this > further if/when some use cases pop up? I don't think it would require > any fundamental changes to the way the browse currently works, .... > > =========== > > Mark, please indulge my questions/comments: > > 1. Let's assume someone's browsing the name heading 'Twiggs, Mark, 1980-' > > If there's an authority record for that name heading, its record will > be found in the global Name Authority File (NAF) index; if NAL (local) > has a work needing that heading, the SAME authority record in NAF will > be called, and that SAME record will have tagging indicating it's also > been processed locally for NAL. Does this scheme work out with your > suggestion? I think so, but it's quite possible my expectations of a browse are different to other people's, since my only experience with browse is that I've tried to implement one :) For a name heading browse at the National Library of Australia, we want a browse to include: * Headings from our bibliographic data; plus * Headings from our (local) authority which are linked from bibliographic data That is, our browse only shows headings that are needed by some work in our collection, and which are therefore already in our local authority data. We don't need to index the global name authority file to do this. My understanding from Demian's comment was that some people will want to import authority records from multiple sources: from their local ILS and from the Library of Congress, at least. With multiple sources of authority data, I can imagine two types of browse: * A browse where the user explicitly indicates which source of authority data they want to browse up-front * A browse which tries to combine all sources of authority data in some way In my previous message I was assuming the former, but maybe we're really talking about the latter here? i.e. a browse which includes all sources of authority data by default, but which allows the user to narrow their browse to only include authority data from one particular source? > 2. I'm not sure how you want to test queries to your local authority > index if what you have is a locally produced (by your local > catalogers) index. I believe you need a copy of global NAF and a > method to tag local NAL's headings in NAF, distinguishing local > from global. Please correct me if necessary. That sounds right to me--I assumed that's what the "source" index in Demian's work is for. I was imagining that you load in the authority records from the Library of Congress with a source of "global" (or "loc", or whatever) and then load your local authority file with a source like "local". Then, to add the ability to narrow to my browse code, I'd need to provide the ability to only show authority data from the source specified by the user. Does that make sense? Please excuse my ignorance :) Mark -- Mark Triggs Systems Administrator Business Systems Support, The National Library of Australia <mtriggs@...>

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Demian Katz <demian.katz@vi...> - 2010-08-17 12:52 > That sounds right to me--I assumed that's what the "source" index in > Demian's work is for. I was imagining that you load in the authority > records from the Library of Congress with a source of "global" (or > "loc", or whatever) and then load your local authority file with a > source like "local". Then, to add the ability to narrow to my browse > code, I'd need to provide the ability to only show authority data from > the source specified by the user. That's exactly right... the most obvious use case is only browsing the local headings, but the source facet also allows an "expanded browse" that pulls in headings not available in the library -- useful as a sanity check when somebody does a valid search for something that isn't held, and also useful when combined with other modules of VuFind like the WorldCat interface. There's one extra wrinkle that makes this a little more complicated, which I think is part of what Ya'aqov was getting at -- if we just load all the local headings with one "source" value and all the remote headings with another "source" value, we end up with duplicate headings in the full index, since some values are going to be in both places. One approach we have discussed involves using the XC Metadata Services Toolkit to process the remote records and add a flag to those which are held locally. If the local records are a subset of the remote records, this would solve the problem completely -- you wouldn't need to load the local records at all. If the overlap is more complicated, the problem is harder to solve, but having a flag could still assist in deduplication. Anyway, it's still something to think about further -- the use cases remain hypothetical for the moment. I see that you don't seem to be planning on attending the VuFind conference next month... but if you're interested in participating in any discussions on this (or another) subject, we might be able to patch you in remotely... though the time zone difference is likely to be a problem. - Demian

[VuFind-General] Alphabetical subject browse - implementing libraries? From: Ya'aqov Ziso <yaaqovz@gm...> - 2010-08-17 17:47 Attachments: Message as HTML      Indeed and exactly. The proposed framework is to check the existence of a name or subject first in their largest pool available (that is WorldCat's identities/NAF or LCSH/Terminologies). With both these global files available (via API) for matching and processing local bibs there will be NO NEED to create authority records also locally, that would be unnecessary duplication. All necessary will be to create one VuFind authority index (source) based on NAF and or LCSH (available via API). in the records of this authority index all needed will be to tag those that are local (evidently a subset of the global). That seems possible via the XC Metadata Services Kit under construction and as Demian indicates enable us to 'provide the ability to only show authority data from the source specified by the user'. The local sets will ALWAYS be a subset of the global set. NAF and Terminologies thesauri files are created collectively by members of libraries all over the world, no one library holding them all. The comment that some people will want to import authority records from multiple sources may be very useful to a situation where NAL (Australia) have authority records not available in NAF (primarily US). It seems that NAL's authority records (in fact anyone's local authority records not available at large) are in MARC21 authority format can be ADDED to the VuFind source index to be created. Demian, does this seem correct to you? if yes there will be a MARC21 minimal format to be adhered to an vetted with NAL and others interested. Demian, the suggestion to patch Mark remotely to a VuFind conference discussion on these issues sounds great. please count me in. Mark, it's worth loosing sleep over such discussion!. Ya'aqov

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Demian Katz <demian.katz@vi...> - 2010-08-17 18:37 Attachments: Message as HTML      See my comments below in blue. The local sets will ALWAYS be a subset of the global set. NAF and Terminologies thesauri files are created collectively by members of libraries all over the world, no one library holding them all. I don’t know that they will ALWAYS be a subset – in some cases, isn’t it possible that the local file may contain locally-created headings that do not exist in the global file? In this case, there is overlap between the sets, but neither one is an exact subset of the other. It seems that NAL's authority records (in fact anyone's local authority records not available at large) are in MARC21 authority format can be ADDED to the VuFind source index to be created. Demian, does this seem correct to you? if yes there will be a MARC21 minimal format to be adhered to an vetted with NAL and others interested. Yes, it’s very easy to add new authority records to VuFind’s index. As long as the same basic types of information (heading, see also, use for) are present somewhere in the MARC record, you can create custom maps to load them to the correct fields in the Solr index. - Demian

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Ya'aqov Ziso <yaaqovz@gm...> - 2010-08-17 18:57 Attachments: Message as HTML      Thank you Demian. The premise of the proposed model is that global files NAF/LCSH are available via API, as they provide the most conclusive information. If we create one VuFind source that includes the local records AND the global files (all adhering to MARC format), the global files will be the larger, more conclusive files. Per Mark's NAL situation -- in the created VuFind index (including global+local) the local will be a subset, tagged as local, right? Ya'aqov On Tue, Aug 17, 2010 at 1:37 PM, Demian Katz <demian.katz@...: > See my comments below in blue. > > > > The local sets will ALWAYS be a subset of the global set. NAF and > Terminologies thesauri files are created collectively by members of > libraries all over the world, no one library holding them all. > > I don’t know that they will ALWAYS be a subset – in some cases, isn’t it > possible that the local file may contain locally-created headings that do > not exist in the global file? In this case, there is overlap between the > sets, but neither one is an exact subset of the other. > > > It seems that NAL's authority records (in fact anyone's local authority > records not available at large) are in MARC21 authority format can be ADDED > to the VuFind source index to be created. Demian, does this seem correct to > you? if yes there will be a MARC21 minimal format to be adhered to an vetted > with NAL and others interested. > > Yes, it’s very easy to add new authority records to VuFind’s index. As > long as the same basic types of information (heading, see also, use for) are > present somewhere in the MARC record, you can create custom maps to load > them to the correct fields in the Solr index. > > > > - Demian >

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Demian Katz <demian.katz@vi...> - 2010-08-17 19:00 Attachments: Message as HTML      Right now, if Mark loaded his own records and used my harvesting tool to get the full NAF, he would end up with one set of records tagged as NAF and one set of records tagged as local. Some records would exist in both sets. In the hypothetical future scenario where we use XC MST to append a “local” flag to selected NAF records, we would have a little more control over how things worked out, but we would still potentially have three distinct sets of records: “in NAF, not in local set”, “in NAF AND in local set”, “in local set, not in NAF.” - Demian From: Ya'aqov Ziso [mailto:yaaqovz@...] Sent: Tuesday, August 17, 2010 2:57 PM To: Demian Katz; Mark Triggs; vufind-general@...; Ralph LeVan Subject: Re: Alphabetical subject browse - implementing libraries? Thank you Demian. The premise of the proposed model is that global files NAF/LCSH are available via API, as they provide the most conclusive information. If we create one VuFind source that includes the local records AND the global files (all adhering to MARC format), the global files will be the larger, more conclusive files. Per Mark's NAL situation -- in the created VuFind index (including global+local) the local will be a subset, tagged as local, right? Ya'aqov On Tue, Aug 17, 2010 at 1:37 PM, Demian Katz <demian.katz@...>> wrote: See my comments below in blue. The local sets will ALWAYS be a subset of the global set. NAF and Terminologies thesauri files are created collectively by members of libraries all over the world, no one library holding them all. I don’t know that they will ALWAYS be a subset – in some cases, isn’t it possible that the local file may contain locally-created headings that do not exist in the global file? In this case, there is overlap between the sets, but neither one is an exact subset of the other. It seems that NAL's authority records (in fact anyone's local authority records not available at large) are in MARC21 authority format can be ADDED to the VuFind source index to be created. Demian, does this seem correct to you? if yes there will be a MARC21 minimal format to be adhered to an vetted with NAL and others interested. Yes, it’s very easy to add new authority records to VuFind’s index. As long as the same basic types of information (heading, see also, use for) are present somewhere in the MARC record, you can create custom maps to load them to the correct fields in the Solr index. - Demian

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Ya'aqov Ziso <yaaqovz@gm...> - 2010-08-17 19:34 Attachments: Message as HTML      *This is exciting Demian. Thinking ahead, when LCSH is added: can LCSH be added into the SAME source (having different tagging for its fields) and retaining the same three sets? Also, per set of *“in local set, not in NAF.” -- what's the check? i.e., how does one verify 'Twiggs, Mark, 1980-' in NAF is not the same as 'Twiggs, Mark' in local? *Ya'aqov * *On Tue, Aug 17, 2010 at 2:00 PM, Demian Katz <demian.katz@...> * wrote: > > *Right now, if Mark loaded his own records and used my harvesting tool to > get the full NAF, he would end up with one set of records tagged as NAF and > one set of records tagged as local. Some records would exist in both sets. > * > > * * > > *In the hypothetical future scenario where we use XC MST to append a > “local” flag to selected NAF records, we would have a little more control > over how things worked out, but we would still potentially have three > distinct sets of records: “in NAF, not in local set”, “in NAF AND in local > set”, “c.”* > > * * > > *- Demian* > > * * > > *From: Ya'aqov Ziso [mailto:yaaqovz@...] > Sent: Tuesday, August 17, 2010 2:57 PM > To: Demian Katz; Mark Triggs; vufind-general@...; Ralph > LeVan > Subject: Re: Alphabetical subject browse - implementing libraries?* > > * * > > *Thank you Demian. The premise of the proposed model is that global files > NAF/LCSH are available via API, as they provide the most conclusive > information. If we create one VuFind source that includes the local records > AND the global files (all adhering to MARC format), the global files will be > the larger, more conclusive files. > > Per Mark's NAL situation -- in the created VuFind index (including > global+local) the local will be a subset, tagged as local, right?* * > > Ya'aqov* > > *On Tue, Aug 17, 2010 at 1:37 PM, Demian Katz <demian.katz@...> > wrote:* > > *See my comments below in blue.* > > * * > > *The local sets will ALWAYS be a subset of the global set. NAF and > Terminologies thesauri files are created collectively by members of > libraries all over the world, no one library holding them all.* > > *I don’t know that they will ALWAYS be a subset – in some cases, isn’t it > possible that the local file may contain locally-created headings that do > not exist in the global file? In this case, there is overlap between the > sets, but neither one is an exact subset of the other.* > > * > It seems that NAL's authority records (in fact anyone's local authority > records not available at large) are in MARC21 authority format can be ADDED > to the VuFind source index to be created. Demian, does this seem correct to > you? if yes there will be a MARC21 minimal format to be adhered to an vetted > with NAL and others interested. * > > *Yes, it’s very easy to add new authority records to VuFind’s index. As > long as the same basic types of information (heading, see also, use for) are > present somewhere in the MARC record, you can create custom maps to load > them to the correct fields in the Solr index.* > > * * > > *- Demian* > > * * > * *

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Demian Katz <demian.katz@vi...> - 2010-08-17 19:43 Attachments: Message as HTML      Since name and subject authority records function much the same way, I don’t see a problem with sharing the Solr index and setting up a similar processing workflow for all of the data. As for verifying whether the same heading in multiple places actually represents the same thing… that’s a tricky problem. Presumably when we come up with rules for matching bibliographic records to authority records in the MST, similar rules could be applied to match one set of authority records against another set and identify potential problem areas. It won’t be simple, though! - Demian From: Ya'aqov Ziso [mailto:yaaqovz@...] Sent: Tuesday, August 17, 2010 3:35 PM To: Demian Katz Cc: Mark Triggs; vufind-general@...; Ralph LeVan Subject: Re: Alphabetical subject browse - implementing libraries? This is exciting Demian. Thinking ahead, when LCSH is added: can LCSH be added into the SAME source (having different tagging for its fields) and retaining the same three sets? Also, per set of “in local set, not in NAF.” -- what's the check? i.e., how does one verify 'Twiggs, Mark, 1980-' in NAF is not the same as 'Twiggs, Mark' in local? Ya'aqov On Tue, Aug 17, 2010 at 2:00 PM, Demian Katz <demian.katz@...>> wrote: Right now, if Mark loaded his own records and used my harvesting tool to get the full NAF, he would end up with one set of records tagged as NAF and one set of records tagged as local. Some records would exist in both sets. In the hypothetical future scenario where we use XC MST to append a “local” flag to selected NAF records, we would have a little more control over how things worked out, but we would still potentially have three distinct sets of records: “in NAF, not in local set”, “in NAF AND in local set”, “c.” - Demian From: Ya'aqov Ziso [mailto:yaaqovz@...>] Sent: Tuesday, August 17, 2010 2:57 PM To: Demian Katz; Mark Triggs; vufind-general@...>; Ralph LeVan Subject: Re: Alphabetical subject browse - implementing libraries? Thank you Demian. The premise of the proposed model is that global files NAF/LCSH are available via API, as they provide the most conclusive information. If we create one VuFind source that includes the local records AND the global files (all adhering to MARC format), the global files will be the larger, more conclusive files. Per Mark's NAL situation -- in the created VuFind index (including global+local) the local will be a subset, tagged as local, right? Ya'aqov On Tue, Aug 17, 2010 at 1:37 PM, Demian Katz <demian.katz@...>> wrote: See my comments below in blue. The local sets will ALWAYS be a subset of the global set. NAF and Terminologies thesauri files are created collectively by members of libraries all over the world, no one library holding them all. I don’t know that they will ALWAYS be a subset – in some cases, isn’t it possible that the local file may contain locally-created headings that do not exist in the global file? In this case, there is overlap between the sets, but neither one is an exact subset of the other. It seems that NAL's authority records (in fact anyone's local authority records not available at large) are in MARC21 authority format can be ADDED to the VuFind source index to be created. Demian, does this seem correct to you? if yes there will be a MARC21 minimal format to be adhered to an vetted with NAL and others interested. Yes, it’s very easy to add new authority records to VuFind’s index. As long as the same basic types of information (heading, see also, use for) are present somewhere in the MARC record, you can create custom maps to load them to the correct fields in the Solr index. - Demian

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Mark Triggs <mtriggs@nl...> - 2010-08-18 07:01 Demian Katz <demian.katz@...> writes: >> That sounds right to me--I assumed that's what the "source" index in >> Demian's work is for. I was imagining that you load in the authority >> records from the Library of Congress with a source of "global" (or >> "loc", or whatever) and then load your local authority file with a >> source like "local". Then, to add the ability to narrow to my browse >> code, I'd need to provide the ability to only show authority data from >> the source specified by the user. > > That's exactly right... the most obvious use case is only browsing > the local headings, but the source facet also allows an "expanded > browse" that pulls in headings not available in the library -- useful > as a sanity check when somebody does a valid search for something that > isn't held, and also useful when combined with other modules of VuFind > like the WorldCat interface. Right, that makes sense--I've been thinking about it backwards. So we're really talking about having a browse over headings pulled from bibliographic data that allows you to "mix in" headings (and their associated see also/use instead/notes) taken from one or more sources of authority data. I'm still not too worried about extending what I've got to handle this kind of thing. I've just tried: * Duplicating my current subject browse SQLite database * Renaming "headings" to "headings_local" * Duplicating that table and calling it "headings_loc" The resulting database is 900MB and has around 500,000 headings in it (250k per table). Merging those tables together in the most naive way I can think of (pulling 20 ordered headings from each table, then merging & sorting them back in Java land and taking the top 20) still only takes around 5-10ms. I haven't tested the performance of the required queries against the Lucene authority index, but I'm confident they'll be comparable. So, I think that taking authority data from multiple sources and building SQLite indexes that are fast to query in a reasonably ad-hoc fashion (returning headings from only some or all sources) will be feasible. However... > There's one extra wrinkle that makes this a little more complicated, > which I think is part of what Ya'aqov was getting at -- if we just > load all the local headings with one "source" value and all the remote > headings with another "source" value, we end up with duplicate > headings in the full index, since some values are going to be in both > places. One approach we have discussed involves using the XC Metadata > Services Toolkit to process the remote records and add a flag to those > which are held locally. If the local records are a subset of the > remote records, this would solve the problem completely -- you > wouldn't need to load the local records at all. If the overlap is > more complicated, the problem is harder to solve, but having a flag > could still assist in deduplication. Anyway, it's still something to > think about further -- the use cases remain hypothetical for the > moment. Ah, I understand. I'll show my ignorance here, but would it be sufficient for a browse (and I realise this is more broad than just browse) to just treat the heading from the 1xx of an authority record as the key for de-duping? In my blissfully ignorant little world, I'm imagining two records like: Source: LOC Heading: Cooking See also: Home economics Food Instead of: Cookery Source: local Heading: Cooking See also: Gastronomy Instead of: Cookery getting merged into one browse entry with indicators to show which bits came from which source, like: Cooking See also: Home economics [LOC] Food [LOC] Gastronomy [local] Use instead of: Cookery [local] or something like that. I understand this is going to look strange if two authority records are actually not referring to the same thing, but I guess I don't have enough of a feel of how often that would happen. It also leads to the possibility of circular "See also" and "Use instead of" references, where one source of authority data says "B: See also: A" and the other says "A: See also: B". I consider this a feature :) > I see that you don't seem to be planning on attending the VuFind > conference next month... but if you're interested in participating in > any discussions on this (or another) subject, we might be able to > patch you in remotely... though the time zone difference is likely to > be a problem. Yep, it would be lovely to meet everyone but it's a long trip, and the timezones are indeed troublesome :) But that's OK: I'm not sure I have any great insight into the particulars of authority data anyway, and the community around the mailing lists makes it easy keep abreast of what's going on (even if most of you *are* sleeping while I'm typing. Except for Greg :). I still lurk on #vufind on Freenode as _mst too, should our hours ever happen to line up... Cheers, Mark -- Mark Triggs Systems Administrator Business Systems Support, The National Library of Australia <mtriggs@...>

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Demian Katz <demian.katz@vi...> - 2010-08-18 14:32 > or something like that. I understand this is going to look strange if > two authority records are actually not referring to the same thing, but > I guess I don't have enough of a feel of how often that would happen. I think it is likely to be a much bigger problem in name authorities than in subject authorities, where there is a bigger chance of a lot of different "Smith, John" entries showing up representing completely different individuals. Still, it's probably the best compromise you can achieve if you're just using software to merge data without having human intervention to puzzle out the actual relationships. > It also leads to the possibility of circular "See also" and "Use instead > of" references, where one source of authority data says "B: See also: A" > and the other says "A: See also: B". I consider this a feature :) Circular "see also" is fine... but circular "use instead of" seems potentially more troublesome! In any case, as I said to Greg, perhaps our best bet is to try to normalize this stuff before it gets into VuFind. I think the issue is too complex to be really smart about it on the fly! - Demian

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Ya'aqov Ziso <yaaqovz@gm...> - 2010-08-18 18:13 Attachments: Message as HTML      ,,, So we're really talking about having a browse over headings pulled from bibliographic data that allows you to "mix in" headings (and their associated see also/use instead/notes) taken from one or more sources of authority data ... >> Mark, just to make sure: the authority file (NAF, LCSH) is not derived from Voyager bibs (as prototyped at NAL). It is constructed separately, in view of data in the bibs. An authority record's main, or usefor, entries are established as a way to match UseFor onto bibs and flip them (the bibs) into the main of the authority record. (The bibs cannot be self referential in order to correct themselves, nor do they have sufficient data for that). For many a reason the 1xx alone could not be the only key for deduping; it needs to be combined with some other elements in the authority record. As Demian points out, mechanisms for authority processing of names are distinguished from those for subjects. Examples such as the 'Cookery' and|or for similar names (Smith, John) will all be part of the processing to be handled in the XC Metadata Kit. That processing will ascertain that bibs are match/flipped to the correct name or subject heading, then imported back to Voyager, life's good. Ya'aqov

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Demian Katz <demian.katz@vi...> - 2010-09-13 17:12 Mark, Thanks again for all of your work on this. As of r3000, I have merged your branch into the trunk and made a few adjustments (minor markup improvements, cosmetic changes, addition of support in both the classic and default themes, etc.). I have also opened a JIRA ticket as a reminder that we need to add Windows support and write documentation (which I'll try to take care of in the near future): http://vufind.org/jira/browse/VUFIND-321 I have noticed one significant problem that I can't easily solve, but perhaps you'll have some thoughts on it... Title browsing isn't taking stopwords into account, so all titles starting with "The" are showing up under "T." In the Solr index, we have a title_sort field which is used for sorting purposes and has any leading stopwords stripped off (accomplished via application of the MARC skip indicator). Is there any way that the heading browse could be made smart enough to make use of this information to allow proper sorting of titles? If nothing else, might it be possible to compare the length of the title_sort field with the title field in order to change "The Title is Long" to "Title is Long, The" automatically? - Demian > -----Original Message----- > From: Mark Triggs [mailto:mtriggs@...] > Sent: Friday, August 13, 2010 9:36 PM > To: Demian Katz > Cc: vufind-general@... > Subject: Re: [VuFind-General] Alphabetical subject browse - > implementing libraries? > > Demian Katz <demian.katz@...> writes: > > >> Excellent, sounds good. I've pushed my preliminary work into: > >> > >> > https://vufind.svn.sourceforge.net/svnroot/vufind/branches/authority- > >> with-alphabetic-browse > >> > >> and will keep fiddling about with it today. > > > > Great, thanks! Looks like you've made good progress already. > > > > This reminds me of an unrelated subject -- are you still using the > nla > > and/or nla.dead branches? Seems a pretty good chance that nla.dead > is > > unnecessary, and even the main nla one hasn't changed since 2008. > > You're welcome to keep them if you need them, but if they're > obsolete, > > perhaps we should delete them to clean up the branches folder a bit. > > Oops, I'd forgotten those were there. The ".dead" one was certainly > not > important (any system I use for more than about three minutes has > numerous .dead files ;), and the "nla" one was a years-old old attempt > at merging our search code back in the trunk. I've deleted both now. > > > > Most of the .jar files included with VuFind come from other projects > > (i.e. SolrMarc) and thus have their source code represented > elsewhere. > > I think there might also be one or two that actually have the source > > embedded inside the .jar. If there's some example you're thinking of > > that isn't covered by either of these cases, please let me know and > > I'll investigate further. > > Ah, that makes sense, thanks. > > > > I assume your new code is branched from your old code, and keeping it > > all in one place would seem to make it easier to maintain. Putting > > the code in VuFind's repository invites people to change it, and if > > alternate versions exist in some other repository, it seems likely to > > lead to confusion. Perhaps I'm misunderstanding the nature of the > > code you're talking about, though! > > Not at all--I think that's a good approach. I've merged the changes > I've made for VuFind back into: > > https://code.nla.gov.au/redmine/projects/nlabrowse > > and have just included the .jar files I built from that source tree. > > The build process for VuFind should be pretty simple--just a checkout > of > that code, and "ant jars > -Dsolr.war=/usr/local/vufind/solr/jetty/webapps/solr.war", as described > in the README. > > I've also added in title browse this morning, so I'm hoping that things > are now at a point where it'd be worth someone else trying to get this > working to see which of my assumptions were wrong :) Here's the steps > I've followed: > > $ svn co > https://vufind.svn.sourceforge.net/svnroot/vufind/branches/authority- > with-alphabetic-browse/ > > [the usual installation/configuration steps for a fresh install] > > $ VUFIND_HOME=$PWD ./import-marc.sh ~/bibdump.mrc > > $ VUFIND_HOME=$PWD ./import-marc-auth.sh ~/authdump.mrc > > $ ./index-alphabetic-browse.sh > > $ ./vufind.sh start > > and then follow the new link from the footer to try it out. > > I haven't made any attempt at Windows support just yet, because I don't > have much in the way of batch-file-fu, nor access to a Windows machine > to test it on (life's good ;). I think Windows support would just > require an index-alphabetic-browse.sh and the Windows port of GNU sort > we discussed before, though, if anyone wants to have a go... > > Enjoy your weekend, > > Mark > > -- > Mark Triggs > <mtriggs@...>

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Mark Triggs <mtriggs@nl...> - 2010-09-13 21:08 Thanks for doing the work to merge it all in--that's great to see. Ah, title browse: I think I'd blocked out this little quirk of our implementation. Being too lazy/evil to modify our MARC imported to handle the skip indicator, I just used a handful of string replacements to drop words like "The" off my title sort keys :P (I know, I know, I'm a terrible person). However, given that you've got this information in your indexes already, it shouldn't be too difficult to make use of it. By default the browse lists are just built by enumerating all the values in a single Lucene field, but the browse indexing code has hooks that let you provide your own class for extracting the browse list in some other way. I should be able to walk through the index a document at a time, using 'title_sort' for the sort key and 'title' for the display text (using the stored values of those fields), so I'll have a go at this when I get to work this morning and hopefully send you a patch shortly. Thanks, Mark Demian Katz <demian.katz@...> writes: > Mark, > > Thanks again for all of your work on this. As of r3000, I have merged > your branch into the trunk and made a few adjustments (minor markup > improvements, cosmetic changes, addition of support in both the > classic and default themes, etc.). I have also opened a JIRA ticket > as a reminder that we need to add Windows support and write > documentation (which I'll try to take care of in the near future): > > http://vufind.org/jira/browse/VUFIND-321 > > I have noticed one significant problem that I can't easily solve, but > perhaps you'll have some thoughts on it... Title browsing isn't > taking stopwords into account, so all titles starting with "The" are > showing up under "T." In the Solr index, we have a title_sort field > which is used for sorting purposes and has any leading stopwords > stripped off (accomplished via application of the MARC skip > indicator). Is there any way that the heading browse could be made > smart enough to make use of this information to allow proper sorting > of titles? If nothing else, might it be possible to compare the > length of the title_sort field with the title field in order to change > "The Title is Long" to "Title is Long, The" automatically? -- Mark Triggs <mtriggs@...>

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Mark Triggs <mtriggs@nl...> - 2010-09-14 01:37 Hi again, I've just finished making some changes to support having a sort key come from a different Lucene field, and I've committed my change here: https://vufind.svn.sourceforge.net/svnroot/vufind/branches/alphabetic-browse-fixes/ It's a very small patch since most of the work happens in the nla-browse-handler project: just a new jar file and a tiny modification to the shell script to use a custom class for building title browses. Let me know if I've messed anything up :) Thanks, Mark Mark Triggs <mtriggs@...> writes: > I should be able to walk through the index a document at a time, using > 'title_sort' for the sort key and 'title' for the display text (using > the stored values of those fields), so I'll have a go at this when I > get to work this morning and hopefully send you a patch shortly. -- Mark Triggs Systems Administrator Business Systems Support, The National Library of Australia <mtriggs@...>

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Demian Katz <demian.katz@vi...> - 2010-09-14 14:27 Thanks for doing this so quickly! Just one question -- how is the BIBLEECH environment variable being used? Is the Java looking for this value explicitly? Would it be better to make this a new parameter to the build_browse function which then gets passed to Java via a command-line property setting? I'm starting to think about the Windows batch file port of the current shell script, and I think life would be a little easier if I could avoid having to manipulate the environment (it might or might not be a problem... but it seems like a potential source of platform-specific eccentricities that we might as well stay away from if we can). thanks, Demian > -----Original Message----- > From: Mark Triggs [mailto:mtriggs@...] > Sent: Monday, September 13, 2010 9:37 PM > To: vufind-general@... > Subject: Re: [VuFind-General] Alphabetical subject browse - > implementing libraries? > > Hi again, > > I've just finished making some changes to support having a sort key > come > from a different Lucene field, and I've committed my change here: > > > https://vufind.svn.sourceforge.net/svnroot/vufind/branches/alphabetic- > browse-fixes/ > > It's a very small patch since most of the work happens in the > nla-browse-handler project: just a new jar file and a tiny modification > to the shell script to use a custom class for building title browses. > > Let me know if I've messed anything up :) > > Thanks, > > Mark > > > Mark Triggs <mtriggs@...> writes: > > > I should be able to walk through the index a document at a time, > using > > 'title_sort' for the sort key and 'title' for the display text (using > > the stored values of those fields), so I'll have a go at this when I > > get to work this morning and hopefully send you a patch shortly. > > -- > Mark Triggs > Systems Administrator > Business Systems Support, The National Library of Australia > <mtriggs@...> > > ----------------------------------------------------------------------- > ------- > Start uncovering the many advantages of virtual appliances > and start using them to simplify application deployment and > accelerate your shift to cloud computing. > http://p.sf.net/sfu/novell-sfdev2dev > _______________________________________________ > VuFind-General mailing list > VuFind-General@... > https://lists.sourceforge.net/lists/listinfo/vufind-general

Re: [VuFind-General] Alphabetical subject browse - implementing libraries? From: Mark Triggs <mtriggs@nl...> - 2010-09-14 21:00 Ah, good point. I've modified the code to look for the "bibleech" Java System property if the "BIBLEECH" environment isn't set and have modified the browse indexing script to use that instead. Changes again pushed to: https://vufind.svn.sourceforge.net/svnroot/vufind/branches/alphabetic-browse-fixes The "BIBLEECH" class get used here: https://code.nla.gov.au/redmine/projects/nlabrowse/repository/revisions/master/entry/browse-indexing/PrintBrowseHeadings.java#L118 and is responsible for doing the extraction of sort keys and headings from the Lucene indexes. The default "Leech" implementation just enumerates the value of a single field, whereas the VuFindTitleLeech takes the sort key values from the title sort field, and the heading values from title field. Leech was a silly name really :) Demian Katz <demian.katz@...> writes: > Thanks for doing this so quickly! Just one question -- how is the > BIBLEECH environment variable being used? Is the Java looking for > this value explicitly? Would it be better to make this a new > parameter to the build_browse function which then gets passed to Java > via a command-line property setting? I'm starting to think about the > Windows batch file port of the current shell script, and I think life > would be a little easier if I could avoid having to manipulate the > environment (it might or might not be a problem... but it seems like a > potential source of platform-specific eccentricities that we might as > well stay away from if we can). -- Mark Triggs <mtriggs@...>

2 messages have been excluded from this view by a project administrator.

Next Messages