Re: [VuFind-Tech] Limit for NoILS->getNewItems()

[Demian K. <dem...@vi...>]

Mario,

This code was written a very long time ago, and there may be better approaches now, but my recollection is that the issue here is that the ILS driver allows you to retrieve record IDs in a paged fashion, but there is no guarantee that the Solr index will have all of these records present (there is likely to be a gap since may VuFind instances load records on a timed cycle, so the newest records may not have reached the index yet). This means we have two possible problems:

1.) If we take only the number of IDs needed to fill a single page of results, we may end up with incomplete or even totally empty pages of results, which will cause confusion for the user.

2.) If we take ALL of the IDs and try to figure out which specific ones are available in Solr, we are likely to run into the maxBooleanClauses limitation of Solr (set to a maximum of 1024 by default in VuFind).

So the compromise we came up with was to fetch a number of IDs equal to 10 pages of results and hope that at least a significant number of these are available. This gives us fully functional pagination without overflowing the Boolean clause limit... but of course it is only showing a subset of the real results.

You could certainly bump up the numbers a bit without breaking anything, but that will likely slow things down. Trying a more precise approach is probably possible, but at the cost of a lot of extra complicated code.

More recently, the ability was added to drive the new items screen using VuFind's internal change tracking functionality, which completely removes the ILS driver from the picture and makes things both simpler and more consistent. It doesn't offer the same filtering options as the ILS-driven approach, but these filters could be implemented using Solr fairly easily if actually needed. I would recommend that this may be the best approach for you, if indexing last change dates is feasible with your data. See:

https://github.com/vufind-org/vufind/blob/master/config/vufind/searches.ini#L431

and

https://vufind.org/wiki/indexing:tracking_record_changes

for more details.

Please let me know if you have any questions, or if you still think we should make changes to the existing functionality. I'm happy to discuss this further!

- Demian

-----Original Message-----
From: Mario Trojan [mailto:mar...@un...] 
Sent: Monday, May 15, 2017 4:18 AM
To: vuf...@li...
Subject: [VuFind-Tech] Limit for NoILS->getNewItems()

Hello everybody,

during some tests with the ILS Drivers (especially Vufind\ILS\Driver\NoILS->getNewItems), i realized that the limit parameter is not used as documented. The documentation says:

@param int $limit The size of each page of results to retrieve

The function is called in
Vufind\Controller\Plugin\NewItems->getBibIDsFromCatalog
there you can see that it is used as: $perPage * $resultPages

(which is used in SearchController->newitemresultsAction())

So if you have 20 max results per Page and want to display 10 pages, the limit parameter will be set to the value 200. Does this really make sense?

In my understanding, the purpose is to:
1) Get the count of all items who meet the conditions (to know how many pages are needed)
2) Get only the IDs of all items for a certain page to display

(a current problem we have is that our users want to know how many new items really exist, and they always see 200 because it is the maximum value passed to the function)

So, wouldn't it be good to change this behaviour in vufind itself? What do you think?

Regards,
Mario


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsdm.link%2Fslashdot&data=02%7C01%7Cdemian.katz%40villanova.edu%7C06c1d0825e314997739708d49b6af5ba%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636304331074117918&sdata=jD52DmSO6UD7CxxodB0gMVg7TsI39YdzN2xHhAgsxs4%3D&reserved=0
_______________________________________________
Vufind-tech mailing list
Vuf...@li...
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fvufind-tech&data=02%7C01%7Cdemian.katz%40villanova.edu%7C06c1d0825e314997739708d49b6af5ba%7C765a8de5cf9444f09cafae5bf8cfa366%7C0%7C0%7C636304331074117918&sdata=qaZS33UuzRejrKVjDjXyxF4fgeX%2BBuO6PsHRMJyk6WU%3D&reserved=0