From: Edwards, R. <rya...@cs...> - 2010-01-22 01:05:32
|
I found that when I uncommented the cover images, reviews, and excerpts info in the VuFind config.ini file, our Syndetics solutions content now appears. Thanks, Ryan Ryan Edwards Systems and Web Development Librarian CSU Channel Islands (805) 437-3198 rya...@cs... <mailto:rya...@cs...> |
From: Reginald A. (ext) <r....@vm...> - 2010-01-25 12:54:33
|
Hi, Can anyone tell me what type of database VuFind is compatible with and if an ILS component is mandatory? Any suggestions or suggested reading are welcome, Reginald Amade Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief |
From: Demian K. <dem...@vi...> - 2010-01-25 14:00:52
|
VuFind was written with MySQL in mind as the back-end database, and most installations use MySQL. However, since the database access goes through PEAR's DB_DataObject framework, it shouldn't be too difficult to adapt the code to work with a different database. You would have to change the install process to set up the appropriate database structure in the other database system, and you would have to change the connection string in config.ini to use the appropriate format, but it might not take much more than that. As for the ILS component, it is definitely not mandatory -- several existing VuFind installations are running without an ILS. You have two options here: either edit the templates to disable functionality that won't work without an ILS (i.e. real-time availability status, some of the "My Account" pages), or write an ILS driver (as specified at http://vufind.org/wiki/building_an_ils_driver) to simulate the ILS functionality you need. I'll be happy to provide more details if you still have questions. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, January 25, 2010 7:32 AM To: vuf...@li... Subject: [VuFind-General] vufind requirements Hi, Can anyone tell me what type of database VuFind is compatible with and if an ILS component is mandatory? Any suggestions or suggested reading are welcome, Reginald Amade Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief |
From: Reginald A. (ext) <r....@vm...> - 2010-01-25 14:14:02
|
Hi Demian, I'm having trouble understanding how metadata harvested by the Service Provider using OAI-PMH can/must be transformed into a format that is VuFind compliant. My understanding is that the Data Provider will return metadata in oai-dc format. If this is correct then additional conversion/transformation needs to be done before VuFind can work with the data? Does my understanding make any sense? Thnx Reginald ________________________________ Van: Demian Katz [mailto:dem...@vi...] Verzonden: maandag 25 januari 2010 14:59 Aan: Reginald Amade (ext); vuf...@li... Onderwerp: RE: vufind requirements VuFind was written with MySQL in mind as the back-end database, and most installations use MySQL. However, since the database access goes through PEAR's DB_DataObject framework, it shouldn't be too difficult to adapt the code to work with a different database. You would have to change the install process to set up the appropriate database structure in the other database system, and you would have to change the connection string in config.ini to use the appropriate format, but it might not take much more than that. As for the ILS component, it is definitely not mandatory -- several existing VuFind installations are running without an ILS. You have two options here: either edit the templates to disable functionality that won't work without an ILS (i.e. real-time availability status, some of the "My Account" pages), or write an ILS driver (as specified at http://vufind.org/wiki/building_an_ils_driver) to simulate the ILS functionality you need. I'll be happy to provide more details if you still have questions. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, January 25, 2010 7:32 AM To: vuf...@li... Subject: [VuFind-General] vufind requirements Hi, Can anyone tell me what type of database VuFind is compatible with and if an ILS component is mandatory? Any suggestions or suggested reading are welcome, Reginald Amade Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief |
From: Demian K. <dem...@vi...> - 2010-01-25 15:31:07
|
You are correct -- right now, VuFind is designed to most easily ingest data in MARC format. While VuFind has an OAI module, it is currently incomplete (as recently discussed on this list). I am currently in the process of changing the way VuFind deals with records to make it more easily compatible with other formats. If you are planning on using the stable RC2 release, your best bet is to harvest the metadata, convert it to MARC format (which may well be possible with existing tools if you chain them together in the right order), and then index it in the standard VuFind fashion. If you would like to work with newer experimental code and are willing to do some coding yourself, it might make sense to build on my current work in progress and use the OAI-DC data more directly. For this, you would have to write an indexer to parse the OAI-DC records and send them to the Solr index (probably the hardest part) and possibly also a record driver so VuFind could appropriately display the records (if it's default index-based display was not good enough). Again, let me know if you need more details. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, January 25, 2010 9:14 AM To: Demian Katz; vuf...@li... Subject: RE: vufind requirements Hi Demian, I'm having trouble understanding how metadata harvested by the Service Provider using OAI-PMH can/must be transformed into a format that is VuFind compliant. My understanding is that the Data Provider will return metadata in oai-dc format. If this is correct then additional conversion/transformation needs to be done before VuFind can work with the data? Does my understanding make any sense? Thnx Reginald ________________________________ Van: Demian Katz [mailto:dem...@vi...] Verzonden: maandag 25 januari 2010 14:59 Aan: Reginald Amade (ext); vuf...@li... Onderwerp: RE: vufind requirements VuFind was written with MySQL in mind as the back-end database, and most installations use MySQL. However, since the database access goes through PEAR's DB_DataObject framework, it shouldn't be too difficult to adapt the code to work with a different database. You would have to change the install process to set up the appropriate database structure in the other database system, and you would have to change the connection string in config.ini to use the appropriate format, but it might not take much more than that. As for the ILS component, it is definitely not mandatory -- several existing VuFind installations are running without an ILS. You have two options here: either edit the templates to disable functionality that won't work without an ILS (i.e. real-time availability status, some of the "My Account" pages), or write an ILS driver (as specified at http://vufind.org/wiki/building_an_ils_driver) to simulate the ILS functionality you need. I'll be happy to provide more details if you still have questions. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, January 25, 2010 7:32 AM To: vuf...@li... Subject: [VuFind-General] vufind requirements Hi, Can anyone tell me what type of database VuFind is compatible with and if an ILS component is mandatory? Any suggestions or suggested reading are welcome, Reginald Amade Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief |
From: Reginald A. (ext) <r....@vm...> - 2010-02-01 13:14:29
|
Thanks Demian, For your clear answer. I understand that VuFind 1. most easily ingests data in MARC format: this suggests a conversion step OAI-DC to MARC 2. has an OAI module that is incomplete and requires additional programming to facilitate data ingestion I have some additional questions though... What do not understand is how the Solr component fits in? What is the flow once data provider records (OAI-DC) are received up to saving them in the (VuFind) repository? Where can I find an ERD? Is the schema fixed? We would like to have manual management functionality for VuFind, or does that come out-of-the-box? Thanks for your reply Reginald ________________________________ Van: Demian Katz [mailto:dem...@vi...] Verzonden: maandag 25 januari 2010 15:31 Aan: Reginald Amade (ext); vuf...@li... Onderwerp: RE: vufind requirements You are correct -- right now, VuFind is designed to most easily ingest data in MARC format. While VuFind has an OAI module, it is currently incomplete (as recently discussed on this list). I am currently in the process of changing the way VuFind deals with records to make it more easily compatible with other formats. If you are planning on using the stable RC2 release, your best bet is to harvest the metadata, convert it to MARC format (which may well be possible with existing tools if you chain them together in the right order), and then index it in the standard VuFind fashion. If you would like to work with newer experimental code and are willing to do some coding yourself, it might make sense to build on my current work in progress and use the OAI-DC data more directly. For this, you would have to write an indexer to parse the OAI-DC records and send them to the Solr index (probably the hardest part) and possibly also a record driver so VuFind could appropriately display the records (if it's default index-based display was not good enough). Again, let me know if you need more details. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, January 25, 2010 9:14 AM To: Demian Katz; vuf...@li... Subject: RE: vufind requirements Hi Demian, I'm having trouble understanding how metadata harvested by the Service Provider using OAI-PMH can/must be transformed into a format that is VuFind compliant. My understanding is that the Data Provider will return metadata in oai-dc format. If this is correct then additional conversion/transformation needs to be done before VuFind can work with the data? Does my understanding make any sense? Thnx Reginald ________________________________ Van: Demian Katz [mailto:dem...@vi...] Verzonden: maandag 25 januari 2010 14:59 Aan: Reginald Amade (ext); vuf...@li... Onderwerp: RE: vufind requirements VuFind was written with MySQL in mind as the back-end database, and most installations use MySQL. However, since the database access goes through PEAR's DB_DataObject framework, it shouldn't be too difficult to adapt the code to work with a different database. You would have to change the install process to set up the appropriate database structure in the other database system, and you would have to change the connection string in config.ini to use the appropriate format, but it might not take much more than that. As for the ILS component, it is definitely not mandatory -- several existing VuFind installations are running without an ILS. You have two options here: either edit the templates to disable functionality that won't work without an ILS (i.e. real-time availability status, some of the "My Account" pages), or write an ILS driver (as specified at http://vufind.org/wiki/building_an_ils_driver) to simulate the ILS functionality you need. I'll be happy to provide more details if you still have questions. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, January 25, 2010 7:32 AM To: vuf...@li... Subject: [VuFind-General] vufind requirements Hi, Can anyone tell me what type of database VuFind is compatible with and if an ILS component is mandatory? Any suggestions or suggested reading are welcome, Reginald Amade Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief |
From: Andrew N. <as...@gm...> - 2010-02-01 21:47:54
|
The OAI module within VuFind has fallen by the wayside over time - but was originally designed to be an OAI server. This meant that your VuFind install would allow others to harvest the records from your VuFind instance via OAI. There is no code in this module to process OAI harvesting into VuFind. Andrew On Mon, Feb 1, 2010 at 7:46 AM, Reginald Amade (ext) <r....@vm...> wrote: > Thanks Demian, > > > > For your clear answer. > > I understand that VuFind > > 1. most easily ingests data in MARC format: this suggests a conversion > step OAI-DC to MARC > 2. has an OAI module that is incomplete and requires additional > programming to facilitate data ingestion > > > > I have some additional questions though… > > What do not understand is how the Solr component fits in? > > What is the flow once data provider records (OAI-DC) are received up to > saving them in the (VuFind) repository? > > Where can I find an ERD? Is the schema fixed? > > We would like to have manual management functionality for VuFind, or does > that come out-of-the-box? > > > > Thanks for your reply > > > > Reginald > > > > > > > > > ------------------------------ > > *Van:* Demian Katz [mailto:dem...@vi...] > *Verzonden:* maandag 25 januari 2010 15:31 > > *Aan:* Reginald Amade (ext); vuf...@li... > *Onderwerp:* RE: vufind requirements > > > > You are correct -- right now, VuFind is designed to most easily ingest data > in MARC format. While VuFind has an OAI module, it is currently incomplete > (as recently discussed on this list). I am currently in the process of > changing the way VuFind deals with records to make it more easily compatible > with other formats. If you are planning on using the stable RC2 release, > your best bet is to harvest the metadata, convert it to MARC format (which > may well be possible with existing tools if you chain them together in the > right order), and then index it in the standard VuFind fashion. If you > would like to work with newer experimental code and are willing to do some > coding yourself, it might make sense to build on my current work in progress > and use the OAI-DC data more directly. For this, you would have to write an > indexer to parse the OAI-DC records and send them to the Solr index > (probably the hardest part) and possibly also a record driver so VuFind > could appropriately display the records (if it's default index-based display > was not good enough). Again, let me know if you need more details. > > > > - Demian > > > > *From:* Reginald Amade (ext) [mailto:r....@vm...] > *Sent:* Monday, January 25, 2010 9:14 AM > *To:* Demian Katz; vuf...@li... > *Subject:* RE: vufind requirements > > > > Hi Demian, > > > > I’m having trouble understanding how metadata harvested by the Service > Provider using OAI-PMH can/must be transformed into a format that is VuFind > compliant. My understanding is that the Data Provider will return metadata > in oai-dc format. If this is correct then additional > conversion/transformation needs to be done before VuFind can work with the > data? > > > > Does my understanding make any sense? > > > > Thnx > > Reginald > > > ------------------------------ > > *Van:* Demian Katz [mailto:dem...@vi...] > *Verzonden:* maandag 25 januari 2010 14:59 > *Aan:* Reginald Amade (ext); vuf...@li... > *Onderwerp:* RE: vufind requirements > > > > VuFind was written with MySQL in mind as the back-end database, and most > installations use MySQL. However, since the database access goes through > PEAR's DB_DataObject framework, it shouldn't be too difficult to adapt the > code to work with a different database. You would have to change the > install process to set up the appropriate database structure in the other > database system, and you would have to change the connection string in > config.ini to use the appropriate format, but it might not take much more > than that. > > > > As for the ILS component, it is definitely not mandatory -- several > existing VuFind installations are running without an ILS. You have two > options here: either edit the templates to disable functionality that won't > work without an ILS (i.e. real-time availability status, some of the "My > Account" pages), or write an ILS driver (as specified at > http://vufind.org/wiki/building_an_ils_driver) to simulate the ILS > functionality you need. > > > > I'll be happy to provide more details if you still have questions. > > > > - Demian > > > > *From:* Reginald Amade (ext) [mailto:r....@vm...] > *Sent:* Monday, January 25, 2010 7:32 AM > *To:* vuf...@li... > *Subject:* [VuFind-General] vufind requirements > > > > Hi, > > > > Can anyone tell me what type of database VuFind is compatible with and if > an ILS component is mandatory? > > Any suggestions or suggested reading are welcome, > > > > Reginald Amade > > > > > > Disclaimer: www.vmm.be/disclaimer > > > > Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief > > Disclaimer: www.vmm.be/disclaimer > > > > Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief > > Disclaimer: www.vmm.be/disclaimer > > Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief > > > > ------------------------------------------------------------------------------ > The Planet: dedicated and managed hosting, cloud storage, colocation > Stay online with enterprise data centers and the best network in the > business > Choose flexible plans and management services without long-term contracts > Personal 24x7 support from experience hosting pros just a phone call away. > http://p.sf.net/sfu/theplanet-com > _______________________________________________ > VuFind-General mailing list > VuF...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-general > > |
From: Demian K. <dem...@vi...> - 2010-02-02 15:16:44
|
As Andrew said, the existing OAI module is probably not suitable for your needs, but I'll try to answer your questions on the assumption that you're willing to build something new to meet your needs. First of all, there is no formal ERD at the moment -- there is some documentation in the wiki that gives a higher-level view of VuFind's index, but it is largely out of date. Also, since the schema is not fixed and can be easily customized, you may find it easier to look at the configuration directly to get a sense of how things work. There are three important pieces that tell most of the story: 1.) solr/biblio/conf/schema.xml defines all of the fields in VuFind's index as well as the data types that may be used to define fields. 2.) import/marc.properties defines how MARC fields are mapped to the various VuFind indexes. Obviously, if you're not working with MARC, you won't have to worry about these mappings directly, but they are a useful reference if you are trying to determine the meaning of a particular field. 3.) web/conf/searchspecs.yaml defines which indexes are searched and how they are used in relevance ranking for each of the search types available to the user (all fields, title, subject, etc.) Obviously, there's a bit of a learning curve here, and not every feature of these files is going to be obvious at first glance... but the general idea shouldn't be too hard to figure out. Assuming that you're using the new experimental "record driver" code, the other important piece is the record driver, which defines how data from the index is presented to the user. The default record driver relies on the index fields for display and may need to be changed if you drastically change the indexing... but you also have the option of building your own driver and customizing things however you want. As for the workflow, it's fairly straightforward in theory (though I'm sure there are some complicated details): 1.) Retrieve the record(s). 2.) Assign unique but reproducible IDs to records so you can apply updates in the future and avoid duplication or ID collisions. 3.) Map the record to the fields of the index and post it to Solr. As far as how the Solr component fits in, it's really the heart of VuFind, performing two critical tasks: storing all of your metadata, and providing all of the indexing and search functionality necessary to retrieve it. VuFind adds a convenient layer on top of Solr to help you define useful searches and present the data, but Solr does all the heavy lifting in the background. Finally, regarding management, what functionality did you have in mind? VuFind's administration module offers some very basic options (basic delete/view records), and you can do some more functions through Solr itself. However, it's helpful to have a good understanding of Solr's role as an index engine. If you expect it to work like a relational database (which I did at first), you may be shocked at its limitations... however, its strengths are elsewhere. It is completely optimized toward doing fast lookups and giving information (like faceting) about indexed data. It's much better than a database for searching through vast amounts of data quickly, but it simply isn't designed to do some things that a database can. Most significant: once a record is indexed, you can't change it. You can REPLACE it with a new record with the same ID, but you can't update it through Solr -- there is no way of adding data to a field or performing a global replace. Your data management needs to happen somewhere else; Solr just needs to be fed the latest fully-formed versions of records. Obviously, understanding and working around this limitation is important when figuring out your workflows. Apologies for the great length of this reply, but I hope it contains some of the information you need. If you still need more, just let me know and I'll be happy to elaborate or clarify. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, February 01, 2010 7:47 AM To: Demian Katz; vuf...@li... Subject: RE: vufind requirements Thanks Demian, For your clear answer. I understand that VuFind 1. most easily ingests data in MARC format: this suggests a conversion step OAI-DC to MARC 2. has an OAI module that is incomplete and requires additional programming to facilitate data ingestion I have some additional questions though... What do not understand is how the Solr component fits in? What is the flow once data provider records (OAI-DC) are received up to saving them in the (VuFind) repository? Where can I find an ERD? Is the schema fixed? We would like to have manual management functionality for VuFind, or does that come out-of-the-box? Thanks for your reply Reginald ________________________________ Van: Demian Katz [mailto:dem...@vi...] Verzonden: maandag 25 januari 2010 15:31 Aan: Reginald Amade (ext); vuf...@li... Onderwerp: RE: vufind requirements You are correct -- right now, VuFind is designed to most easily ingest data in MARC format. While VuFind has an OAI module, it is currently incomplete (as recently discussed on this list). I am currently in the process of changing the way VuFind deals with records to make it more easily compatible with other formats. If you are planning on using the stable RC2 release, your best bet is to harvest the metadata, convert it to MARC format (which may well be possible with existing tools if you chain them together in the right order), and then index it in the standard VuFind fashion. If you would like to work with newer experimental code and are willing to do some coding yourself, it might make sense to build on my current work in progress and use the OAI-DC data more directly. For this, you would have to write an indexer to parse the OAI-DC records and send them to the Solr index (probably the hardest part) and possibly also a record driver so VuFind could appropriately display the records (if it's default index-based display was not good enough). Again, let me know if you need more details. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, January 25, 2010 9:14 AM To: Demian Katz; vuf...@li... Subject: RE: vufind requirements Hi Demian, I'm having trouble understanding how metadata harvested by the Service Provider using OAI-PMH can/must be transformed into a format that is VuFind compliant. My understanding is that the Data Provider will return metadata in oai-dc format. If this is correct then additional conversion/transformation needs to be done before VuFind can work with the data? Does my understanding make any sense? Thnx Reginald ________________________________ Van: Demian Katz [mailto:dem...@vi...] Verzonden: maandag 25 januari 2010 14:59 Aan: Reginald Amade (ext); vuf...@li... Onderwerp: RE: vufind requirements VuFind was written with MySQL in mind as the back-end database, and most installations use MySQL. However, since the database access goes through PEAR's DB_DataObject framework, it shouldn't be too difficult to adapt the code to work with a different database. You would have to change the install process to set up the appropriate database structure in the other database system, and you would have to change the connection string in config.ini to use the appropriate format, but it might not take much more than that. As for the ILS component, it is definitely not mandatory -- several existing VuFind installations are running without an ILS. You have two options here: either edit the templates to disable functionality that won't work without an ILS (i.e. real-time availability status, some of the "My Account" pages), or write an ILS driver (as specified at http://vufind.org/wiki/building_an_ils_driver) to simulate the ILS functionality you need. I'll be happy to provide more details if you still have questions. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, January 25, 2010 7:32 AM To: vuf...@li... Subject: [VuFind-General] vufind requirements Hi, Can anyone tell me what type of database VuFind is compatible with and if an ILS component is mandatory? Any suggestions or suggested reading are welcome, Reginald Amade Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief |
From: Reginald A. (ext) <r....@vm...> - 2010-02-03 15:09:31
|
You're absolutely right. I am willing to build something that meets our needs. As Analyst/Designer I need to be sure I understand what challenges we are facing. Let me try to explain our situation. We are an organization responsible for environmental issues. One of the projects is to set up an environmental portal where public users can easily access stored media. A common model is used where partners can participate by making their library available. The procedures used so far made synchronisation a tedious and time consuming task with low data quality. We have a business case that aims to improve both user experience, maintenance and data quality. Our technologies/products of choice are VuFind and the OAI-PMH protocol. As such our approach is 1. Setting up the harvesting both from a service provider perspective as well as the data provider perspective. a. Given the fact that we are dealing with 25+ data providers and 8 identified types, it is anticipated that we need a solid design in order to provide an easy scalable and extensible basis for future connection of new data providers b. Automatic harvesting of offered meta data 2. Setting up a VuFind instance that ingests the harvested meta data Obviously I hope to get some more insight on what I need to do on the harvesting side in order to enable VuFind to publish the data. So far your feedback is telling me that 1. VuFind can easily ingest metadata in MARCXML format using the latest solrMARC tool 2. The previous point suggests that the records are when a. Using an ILS, exported from it b. Not using an ILS, converted directly from oai-dc (eg to MARCXML) I am wondering what the benefits/drawbacks are of (not)using an ILS? Like Voyager. Does VuFind/Solr need an ILS? We are not using one! In fact VuFind acts as a presentation layer, while Solr is performing the hard work under the hood. VuFind does not physically store the Solr created index in its MySQL database. The MySQL database is used to realize VuFind's functionality? The concern of management. We think that if a partner has a "small" volume of anticipated changes/additions we could give them some management functionality (in)directly. Ideally this is on the Solr index, but maybe generating MARCXML and feeding that to the Solr index is the way to go? I hope that you can answer my questions or point to resources! Thanks Reginald ________________________________ Van: Demian Katz [mailto:dem...@vi...] Verzonden: dinsdag 2 februari 2010 16:17 Aan: Reginald Amade (ext); vuf...@li... Onderwerp: RE: vufind requirements As Andrew said, the existing OAI module is probably not suitable for your needs, but I'll try to answer your questions on the assumption that you're willing to build something new to meet your needs. First of all, there is no formal ERD at the moment -- there is some documentation in the wiki that gives a higher-level view of VuFind's index, but it is largely out of date. Also, since the schema is not fixed and can be easily customized, you may find it easier to look at the configuration directly to get a sense of how things work. There are three important pieces that tell most of the story: 1.) solr/biblio/conf/schema.xml defines all of the fields in VuFind's index as well as the data types that may be used to define fields. 2.) import/marc.properties defines how MARC fields are mapped to the various VuFind indexes. Obviously, if you're not working with MARC, you won't have to worry about these mappings directly, but they are a useful reference if you are trying to determine the meaning of a particular field. 3.) web/conf/searchspecs.yaml defines which indexes are searched and how they are used in relevance ranking for each of the search types available to the user (all fields, title, subject, etc.) Obviously, there's a bit of a learning curve here, and not every feature of these files is going to be obvious at first glance... but the general idea shouldn't be too hard to figure out. Assuming that you're using the new experimental "record driver" code, the other important piece is the record driver, which defines how data from the index is presented to the user. The default record driver relies on the index fields for display and may need to be changed if you drastically change the indexing... but you also have the option of building your own driver and customizing things however you want. As for the workflow, it's fairly straightforward in theory (though I'm sure there are some complicated details): 1.) Retrieve the record(s). 2.) Assign unique but reproducible IDs to records so you can apply updates in the future and avoid duplication or ID collisions. 3.) Map the record to the fields of the index and post it to Solr. As far as how the Solr component fits in, it's really the heart of VuFind, performing two critical tasks: storing all of your metadata, and providing all of the indexing and search functionality necessary to retrieve it. VuFind adds a convenient layer on top of Solr to help you define useful searches and present the data, but Solr does all the heavy lifting in the background. Finally, regarding management, what functionality did you have in mind? VuFind's administration module offers some very basic options (basic delete/view records), and you can do some more functions through Solr itself. However, it's helpful to have a good understanding of Solr's role as an index engine. If you expect it to work like a relational database (which I did at first), you may be shocked at its limitations... however, its strengths are elsewhere. It is completely optimized toward doing fast lookups and giving information (like faceting) about indexed data. It's much better than a database for searching through vast amounts of data quickly, but it simply isn't designed to do some things that a database can. Most significant: once a record is indexed, you can't change it. You can REPLACE it with a new record with the same ID, but you can't update it through Solr -- there is no way of adding data to a field or performing a global replace. Your data management needs to happen somewhere else; Solr just needs to be fed the latest fully-formed versions of records. Obviously, understanding and working around this limitation is important when figuring out your workflows. Apologies for the great length of this reply, but I hope it contains some of the information you need. If you still need more, just let me know and I'll be happy to elaborate or clarify. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, February 01, 2010 7:47 AM To: Demian Katz; vuf...@li... Subject: RE: vufind requirements Thanks Demian, For your clear answer. I understand that VuFind 1. most easily ingests data in MARC format: this suggests a conversion step OAI-DC to MARC 2. has an OAI module that is incomplete and requires additional programming to facilitate data ingestion I have some additional questions though... What do not understand is how the Solr component fits in? What is the flow once data provider records (OAI-DC) are received up to saving them in the (VuFind) repository? Where can I find an ERD? Is the schema fixed? We would like to have manual management functionality for VuFind, or does that come out-of-the-box? Thanks for your reply Reginald ________________________________ Van: Demian Katz [mailto:dem...@vi...] Verzonden: maandag 25 januari 2010 15:31 Aan: Reginald Amade (ext); vuf...@li... Onderwerp: RE: vufind requirements You are correct -- right now, VuFind is designed to most easily ingest data in MARC format. While VuFind has an OAI module, it is currently incomplete (as recently discussed on this list). I am currently in the process of changing the way VuFind deals with records to make it more easily compatible with other formats. If you are planning on using the stable RC2 release, your best bet is to harvest the metadata, convert it to MARC format (which may well be possible with existing tools if you chain them together in the right order), and then index it in the standard VuFind fashion. If you would like to work with newer experimental code and are willing to do some coding yourself, it might make sense to build on my current work in progress and use the OAI-DC data more directly. For this, you would have to write an indexer to parse the OAI-DC records and send them to the Solr index (probably the hardest part) and possibly also a record driver so VuFind could appropriately display the records (if it's default index-based display was not good enough). Again, let me know if you need more details. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, January 25, 2010 9:14 AM To: Demian Katz; vuf...@li... Subject: RE: vufind requirements Hi Demian, I'm having trouble understanding how metadata harvested by the Service Provider using OAI-PMH can/must be transformed into a format that is VuFind compliant. My understanding is that the Data Provider will return metadata in oai-dc format. If this is correct then additional conversion/transformation needs to be done before VuFind can work with the data? Does my understanding make any sense? Thnx Reginald ________________________________ Van: Demian Katz [mailto:dem...@vi...] Verzonden: maandag 25 januari 2010 14:59 Aan: Reginald Amade (ext); vuf...@li... Onderwerp: RE: vufind requirements VuFind was written with MySQL in mind as the back-end database, and most installations use MySQL. However, since the database access goes through PEAR's DB_DataObject framework, it shouldn't be too difficult to adapt the code to work with a different database. You would have to change the install process to set up the appropriate database structure in the other database system, and you would have to change the connection string in config.ini to use the appropriate format, but it might not take much more than that. As for the ILS component, it is definitely not mandatory -- several existing VuFind installations are running without an ILS. You have two options here: either edit the templates to disable functionality that won't work without an ILS (i.e. real-time availability status, some of the "My Account" pages), or write an ILS driver (as specified at http://vufind.org/wiki/building_an_ils_driver) to simulate the ILS functionality you need. I'll be happy to provide more details if you still have questions. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, January 25, 2010 7:32 AM To: vuf...@li... Subject: [VuFind-General] vufind requirements Hi, Can anyone tell me what type of database VuFind is compatible with and if an ILS component is mandatory? Any suggestions or suggested reading are welcome, Reginald Amade Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief |
From: Till K. <kin...@gm...> - 2010-02-03 15:52:09
|
Reginald Amade (ext) schrieb: > 1. VuFind can easily ingest metadata in MARCXML format using the > latest solrMARC tool Yes, true. > I am wondering what the benefits/drawbacks are of (not)using an ILS? > Like Voyager. You only need an ILS if you want to use any of the functionality it offers (beyond the catalogue), like library user management, item acquisition, cataloging, loan management... As I understand your use case you need none of that functionality. > Does VuFind/Solr need an ILS? We are not using one! You don't need an ILS. We and several others are running VuFind without an ILS. To start, just kick out all the references to an ILS out of the VuFind user interface (mainly item status display). > In fact VuFind acts as a presentation layer, while Solr is performing > the hard work under the hood. Correct. > VuFind does not physically store the Solr created index in its MySQL > database. No. Solr puts the data into its index on index. > The MySQL database is used to realize VuFind’s functionality? Yes, VuFind uses MySQL to track session data (by default, you may use any other session handler available in PHP, too), user tags and comments, user item lists etc. > The concern of management. Management of what? Data management? VuFind doesn't do any data management. It just searches an Solr index cleverly and displays the results nicely :-). And it brings along one way to build that Solr index out of MARC data through solrmarc. But it has no tools to manage or aggregate data. That's something an ILS does, but most ILSes are restricted to very "narrow" data management use cases in libraries (usually cataloging by hand and some way of batch import). So a traditional ILS isn't the solution for your needs either, I think. If I understand you correctly, you need a solution to manage data retrieved from different libraries (in your case through OAI). Neither VuFind nor Solr are good at that. We use a very heavy commercial software for that, but our use case is a large library federation that manages all its metadata cooperatively in that central system. Maybe the Extensible Catalog Project (XC, http://www.extensiblecatalog.org/) has something suitable for you? Till |
From: Demian K. <dem...@vi...> - 2010-02-03 15:37:51
|
To answer a few of your questions: 1.) Does VuFind/Solr need an ILS? No -- the main reason for using an ILS with VuFind is to get live availability status of items (i.e. is the book on the shelf?) and to manage patron-oriented features (what are my fines? what do I have checked out?). If you are dealing with a repository of digital documents, none of this stuff really applies to you, so you should be able to simply disable the ILS-oriented functionality. I eventually plan to add a special "No ILS" setting to VuFind to make it really easy to disable this stuff. For now, you'll have to edit a few templates to remove irrelevant functionality, but this shouldn't be too much work. 2.) VuFind does not physically store the Solr created index in its MySQL database. The MySQL database is used to realize VuFind's functionality? This is correct -- Solr stores all of the record information. The MySQL database is used for tracking user accounts and user-entered metadata (tags, comments, favorites lists). The vast majority of lookups hit Solr directly, but occasionally a MySQL call will be issued to pull in extra data associated with a given Solr record ID. 3.) Management issues. I would advise against using Solr as your main point for record management. It's only an index, and it's generally best to leave it at that for several reasons: a) as previously mentioned, managing Solr records is a lot less convenient than managing something like a relational database b) you may need to completely rebuild your index from time to time (due to Solr/VuFind upgrades or reconfiguration), so it's best if you have some easy way of getting ALL of the data you want indexed without having to re-extract it from Solr itself. c) the Solr index files can get huge, so for backup purposes, you may want a more compact master format from which to reconstruct your site in a catastrophe. That's not to say that you can't back up Solr -- it's actually quite easy to manage -- but maintaining several days' worth of Solr backups could get expensive quickly. You might be better off with, for example, weekly Solr backups and daily record database backups. d) Solr performs best if you write to it relatively infrequently. If you make constant changes without reoptimizing the index, performance will degrade, and constantly reoptimizing is a CPU-intensive process. Due to all of those factors, your best bet may be to maintain a database-driven system of all of your records complete with management capabilities. This system wouldn't need much in the way of search capabilities (Solr has that covered), but it could be used for managing all the accounts you are harvesting from, doing the actual harvesting, and exporting either a full dump of all records, or a dump of all records changed/added/deleted since a certain date. This way, you can do a full index rebuild with relative ease when you have to, and the rest of the time you can run a nightly process to get the latest changes, index them in Solr, and then reoptimize the index at a low-volume time. In a sense, this harvester/record database would fit into your VuFind setup in the position where most users have an ILS. 4.) VuFind can easily ingest metadata in MARCXML format using the latest solrMARC tool. VuFind is actually designed to read in binary MARC21 data... but since you can easily convert MARCXML to MARC21, your statement is essentially true. There's a good chance you can get away with simply using MARC as part of your workflow, though there are a couple of reasons that you might want to use the new record driver code and develop your own non-MARC importer and VuFind plug-in: a) MARC has some fairly restrictive length limits, so if you want to do full-text indexing of long articles, you may run into problems. On the other hand, if you're only doing basic top-level metadata, MARC should be fine. b) Writing your own record driver may make it easier to customize the record displays if you have a lot of metadata fields that don't readily map to the MARC standard. Again, if this is essentially standard bibliographic information, you may not have any problems. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Wednesday, February 03, 2010 10:09 AM To: Demian Katz Cc: vuf...@li... Subject: RE: vufind requirements You're absolutely right. I am willing to build something that meets our needs. As Analyst/Designer I need to be sure I understand what challenges we are facing. Let me try to explain our situation. We are an organization responsible for environmental issues. One of the projects is to set up an environmental portal where public users can easily access stored media. A common model is used where partners can participate by making their library available. The procedures used so far made synchronisation a tedious and time consuming task with low data quality. We have a business case that aims to improve both user experience, maintenance and data quality. Our technologies/products of choice are VuFind and the OAI-PMH protocol. As such our approach is 1. Setting up the harvesting both from a service provider perspective as well as the data provider perspective. * Given the fact that we are dealing with 25+ data providers and 8 identified types, it is anticipated that we need a solid design in order to provide an easy scalable and extensible basis for future connection of new data providers * Automatic harvesting of offered meta data 1. Setting up a VuFind instance that ingests the harvested meta data Obviously I hope to get some more insight on what I need to do on the harvesting side in order to enable VuFind to publish the data. So far your feedback is telling me that 1. VuFind can easily ingest metadata in MARCXML format using the latest solrMARC tool 2. The previous point suggests that the records are when a. Using an ILS, exported from it b. Not using an ILS, converted directly from oai-dc (eg to MARCXML) I am wondering what the benefits/drawbacks are of (not)using an ILS? Like Voyager. Does VuFind/Solr need an ILS? We are not using one! In fact VuFind acts as a presentation layer, while Solr is performing the hard work under the hood. VuFind does not physically store the Solr created index in its MySQL database. The MySQL database is used to realize VuFind's functionality? The concern of management. We think that if a partner has a "small" volume of anticipated changes/additions we could give them some management functionality (in)directly. Ideally this is on the Solr index, but maybe generating MARCXML and feeding that to the Solr index is the way to go? I hope that you can answer my questions or point to resources! Thanks Reginald ________________________________ Van: Demian Katz [mailto:dem...@vi...] Verzonden: dinsdag 2 februari 2010 16:17 Aan: Reginald Amade (ext); vuf...@li... Onderwerp: RE: vufind requirements As Andrew said, the existing OAI module is probably not suitable for your needs, but I'll try to answer your questions on the assumption that you're willing to build something new to meet your needs. First of all, there is no formal ERD at the moment -- there is some documentation in the wiki that gives a higher-level view of VuFind's index, but it is largely out of date. Also, since the schema is not fixed and can be easily customized, you may find it easier to look at the configuration directly to get a sense of how things work. There are three important pieces that tell most of the story: 1.) solr/biblio/conf/schema.xml defines all of the fields in VuFind's index as well as the data types that may be used to define fields. 2.) import/marc.properties defines how MARC fields are mapped to the various VuFind indexes. Obviously, if you're not working with MARC, you won't have to worry about these mappings directly, but they are a useful reference if you are trying to determine the meaning of a particular field. 3.) web/conf/searchspecs.yaml defines which indexes are searched and how they are used in relevance ranking for each of the search types available to the user (all fields, title, subject, etc.) Obviously, there's a bit of a learning curve here, and not every feature of these files is going to be obvious at first glance... but the general idea shouldn't be too hard to figure out. Assuming that you're using the new experimental "record driver" code, the other important piece is the record driver, which defines how data from the index is presented to the user. The default record driver relies on the index fields for display and may need to be changed if you drastically change the indexing... but you also have the option of building your own driver and customizing things however you want. As for the workflow, it's fairly straightforward in theory (though I'm sure there are some complicated details): 1.) Retrieve the record(s). 2.) Assign unique but reproducible IDs to records so you can apply updates in the future and avoid duplication or ID collisions. 3.) Map the record to the fields of the index and post it to Solr. As far as how the Solr component fits in, it's really the heart of VuFind, performing two critical tasks: storing all of your metadata, and providing all of the indexing and search functionality necessary to retrieve it. VuFind adds a convenient layer on top of Solr to help you define useful searches and present the data, but Solr does all the heavy lifting in the background. Finally, regarding management, what functionality did you have in mind? VuFind's administration module offers some very basic options (basic delete/view records), and you can do some more functions through Solr itself. However, it's helpful to have a good understanding of Solr's role as an index engine. If you expect it to work like a relational database (which I did at first), you may be shocked at its limitations... however, its strengths are elsewhere. It is completely optimized toward doing fast lookups and giving information (like faceting) about indexed data. It's much better than a database for searching through vast amounts of data quickly, but it simply isn't designed to do some things that a database can. Most significant: once a record is indexed, you can't change it. You can REPLACE it with a new record with the same ID, but you can't update it through Solr -- there is no way of adding data to a field or performing a global replace. Your data management needs to happen somewhere else; Solr just needs to be fed the latest fully-formed versions of records. Obviously, understanding and working around this limitation is important when figuring out your workflows. Apologies for the great length of this reply, but I hope it contains some of the information you need. If you still need more, just let me know and I'll be happy to elaborate or clarify. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, February 01, 2010 7:47 AM To: Demian Katz; vuf...@li... Subject: RE: vufind requirements Thanks Demian, For your clear answer. I understand that VuFind 1. most easily ingests data in MARC format: this suggests a conversion step OAI-DC to MARC 2. has an OAI module that is incomplete and requires additional programming to facilitate data ingestion I have some additional questions though... What do not understand is how the Solr component fits in? What is the flow once data provider records (OAI-DC) are received up to saving them in the (VuFind) repository? Where can I find an ERD? Is the schema fixed? We would like to have manual management functionality for VuFind, or does that come out-of-the-box? Thanks for your reply Reginald ________________________________ Van: Demian Katz [mailto:dem...@vi...] Verzonden: maandag 25 januari 2010 15:31 Aan: Reginald Amade (ext); vuf...@li... Onderwerp: RE: vufind requirements You are correct -- right now, VuFind is designed to most easily ingest data in MARC format. While VuFind has an OAI module, it is currently incomplete (as recently discussed on this list). I am currently in the process of changing the way VuFind deals with records to make it more easily compatible with other formats. If you are planning on using the stable RC2 release, your best bet is to harvest the metadata, convert it to MARC format (which may well be possible with existing tools if you chain them together in the right order), and then index it in the standard VuFind fashion. If you would like to work with newer experimental code and are willing to do some coding yourself, it might make sense to build on my current work in progress and use the OAI-DC data more directly. For this, you would have to write an indexer to parse the OAI-DC records and send them to the Solr index (probably the hardest part) and possibly also a record driver so VuFind could appropriately display the records (if it's default index-based display was not good enough). Again, let me know if you need more details. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, January 25, 2010 9:14 AM To: Demian Katz; vuf...@li... Subject: RE: vufind requirements Hi Demian, I'm having trouble understanding how metadata harvested by the Service Provider using OAI-PMH can/must be transformed into a format that is VuFind compliant. My understanding is that the Data Provider will return metadata in oai-dc format. If this is correct then additional conversion/transformation needs to be done before VuFind can work with the data? Does my understanding make any sense? Thnx Reginald ________________________________ Van: Demian Katz [mailto:dem...@vi...] Verzonden: maandag 25 januari 2010 14:59 Aan: Reginald Amade (ext); vuf...@li... Onderwerp: RE: vufind requirements VuFind was written with MySQL in mind as the back-end database, and most installations use MySQL. However, since the database access goes through PEAR's DB_DataObject framework, it shouldn't be too difficult to adapt the code to work with a different database. You would have to change the install process to set up the appropriate database structure in the other database system, and you would have to change the connection string in config.ini to use the appropriate format, but it might not take much more than that. As for the ILS component, it is definitely not mandatory -- several existing VuFind installations are running without an ILS. You have two options here: either edit the templates to disable functionality that won't work without an ILS (i.e. real-time availability status, some of the "My Account" pages), or write an ILS driver (as specified at http://vufind.org/wiki/building_an_ils_driver) to simulate the ILS functionality you need. I'll be happy to provide more details if you still have questions. - Demian From: Reginald Amade (ext) [mailto:r....@vm...] Sent: Monday, January 25, 2010 7:32 AM To: vuf...@li... Subject: [VuFind-General] vufind requirements Hi, Can anyone tell me what type of database VuFind is compatible with and if an ILS component is mandatory? Any suggestions or suggested reading are welcome, Reginald Amade Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief Disclaimer: www.vmm.be/disclaimer Kent u onze nieuwsbrief al? www.vmm.be/nieuwsbrief |