From: Demian K. <dem...@vi...> - 2010-04-26 14:50:39
|
The "stored feed" solution sounds, as you say, pretty complex -- I have a feeling you might run into scalability problems there, though perhaps I'm not fully understanding your proposal. One other thought on the "date indexed" field -- There has been some discussion of using some of the XC tools with VuFind (metadata services toolkit, etc.). The XC approach to index-building is to harvest records via OAI-PMH. One of the XC tools is a program which grabs records from the ILS and presents them through an OAI-PMH interface. It seems like it may make sense to add an OAI-PMH harvester to VuFind so that it can take advantage of this tool (not to mention all the other OAI-PMH sources out in the wild). Obviously, there are a number of challenges in switching to this workflow, and it's yet to be seen exactly how it fits into the VuFind picture. However, assuming other details are sorted out, the OAI-PMH datestamp provides a clear way of tracking when things changed. It would survive a full reindex since a reindex in the OAI-PMH context would simply be a re-harvest. Obviously, this doesn't completely solve the problem -- what if your OAI-PMH source blows up and has to be rebuilt? But it might be a step in the right direction, and it's worth thinking about... but obviously it's not a short-term solution! - Demian From: Brad Dewar [mailto:bd...@st...] Sent: Monday, April 26, 2010 8:43 AM To: vuf...@li... Subject: Re: [VuFind-Tech] VUFIND-167 (RSS Feature is not really RSS) Here are two different approaches I've been thinking about lately. I don't think either is THE approach, but maybe they are steps in the right direction: A "date indexed" field in Solr. Set up the feed as an RSS representation of the search results - exactly the way it is now, but sort desc by date indexed. Newest items are always at the top of your RSS reader (even though they may not be particularly _relevant_ results). You run into problems if (when) you re-index everything though.... It's my impression (guess, really) that the RSS feeds aren't used very much. Given that, it wouldn't be too problematic to store each unique request. Every time someone requests a feed, store some representation of it on disk. Edit each of those static files onCommit (merge new items into the top of the file). Feed readers get fed from the static files rather than by doing a direct search. That sounds like it could get pretty complex, though. You'd also have a problem knowing when to delete unused feed files (if ever). Doesn't have to be files - could use MySQL. If someone has an idea for persisting 'date-indexed' through a re-index, that would likely be much easier (it's so much closer to what's there now). Brad From: Greg Pendlebury [mailto:gre...@gm...] Sent: April-23-10 9:20 PM To: Tuan Nguyen Cc: vuf...@li... Subject: Re: [VuFind-Tech] VUFIND-167 (RSS Feature is not really RSS) I have no idea on the technical correctness of the idea, but it's pretty far from the expected behaviour for RSS consumers. People would expect individual entries so that their readers and aggregators will tell them what exactly changed, not just that something changed and they are going to have to sift through the results to find it. Syndication requires timeliness and I can't see a way around that without VuFind becoming authoritative about times, or find an external source. I suspect the easiest way out is to add some sort of syndication/new items interface to the ILMS Driver, and disable the feature in VuFind unless we find that interface. On 24 April 2010 06:45, Tuan Nguyen <tu...@yo...<mailto:tu...@yo...>> wrote: I'm don't know much about RSS either. The only problem I could see with this is if a record is deleted and another is added, then the number of results will not change and the feed reader won't pick it up. On Apr 23, 2010, at 1:36 PM, Demian Katz wrote: I've just been doing some reading about RSS, which in spite of its ubiquity, I have never read about at the spec-level before. It's a whole lot simpler (and more simplistic) than I had imagined. Two things really surprised me: 1.) There is no defined request format. I had assumed there was a standard protocol for requesting RSS feeds, but (unless I'm missing something big), there really isn't. It's up to the developer if they want to parameterize their feeds and exactly what they represent in the list. 2.) No date-related information is required in the feed. Anything dealing with when a resource was created or changed is an optional field. I had assumed that some kind of date information was required so that feed aggregators could correctly interpolate feeds and detect changes. These revelations led me to a new thought on how to solve this issue. Our current solution is to literally render our search results as an RSS feed. This is problematic because (due to a lack of "last changed" data) we can't order the feed in a way that puts the newest results at the top of the list. Without some kind of change-based sorting, it's nearly useless as a real RSS feed because it doesn't highlight changes. However, what if we take things up a level? Rather than representing the search results as individual items within the RSS feed, what if we represent the search ITSELF as the item? i.e.: <item> <title>Author Search: charles dickens</title> <link> http://vufind.org/demo/Search/Results?lookfor=charles+dickens&type=Author</link<http://vufind.org/demo/Search/Results?lookfor=charles+dickens&type=Author%3c/link>> <description>Search with 187 results.</description> </item> I suspect that the most common scenario that will interest a user in returning to a search is discovering that new items matching the terms have arrived. When this happens, the description will change. Would that be enough for an aggregator to treat this as a new item? If so, perhaps this is a simple starting point for more functional RSS -- it's not as glamorous as showing the individual new search results, but it serves as a notification that something has changed, and it can prompt the user to repeat the search. As I've said, I don't have a lot of experience with RSS, so maybe this is a silly idea or simply won't work. However, if anyone with a little more knowledge of the topic would care to comment, I would appreciate it! This is the closest thing I've come up with to a simple solution to this complex problem, and it would be nice if we can make it work. Trying to solve it from other angles seems to run up against the wall of insufficient input data, and I really don't know how to get around that in a non-institution-specific way. thanks, Demian ------------------------------------------------------------------------------ _______________________________________________ Vufind-tech mailing list Vuf...@li...<mailto:Vuf...@li...> https://lists.sourceforge.net/lists/listinfo/vufind-tech ------------------------------------------------------------------------------ _______________________________________________ Vufind-tech mailing list Vuf...@li...<mailto:Vuf...@li...> https://lists.sourceforge.net/lists/listinfo/vufind-tech |