Menu
▾
▴
#22 harvesting and periodic re-indexing of content
This issue is mostly related to full text search but it could be applicable to
all 9s metadata. Some contributors have asked about how we would pick
up changes to .txt files living on their servers. Two approaches come to
mind:
1) Periodically crawl all the nines:text links and re-index if the last-
modified value in the http headers is newer than the last-indexed date.
2) Set a periodic time (weekly?) where all content is re-indexed.
The second approach is more blunt but it has the advantage of setting
expectations. If #2 is resource intensive we could do a combination of
the two where we issue a crawler on a set schedule.
A third approach would be for the content author to notify the Collex
system of updates, but this seems cumbersome and error prone.
Logged In: YES
user_id=1526673
Originator: NO
I believe it makes much more sense for us to think of this in terms of all 9s metadata rather than just the full text indexing. And shouldn't this be part of the admin interface? If it were, this "third approach" may be just the thing. I'm not sure how "cumbersome and error prone" it would actually be. It seems to me, in fact, to be more in line with what scholars expect from a peer-reviewed resource -- that they'd notify the system if post-peer review changes in their material warrant it (and they'd need to notify the editors, as well.) Let's talk about this with Jerry before anybody tackles it.