Heritrix: Internet Archive Web Crawler

Tracker: Feature Requests

7 add dummy items at heads of BDB queues for performance - ID: 1262665

Last Update: Comment added ( karl-ia )

Details:

Per Mark Hayes of Sleepycat, our current method of
finding the 'head' of one of our virtual queues, a
range search, will hold a lock on the preceding
database record (the last item in the previous queue).

Inserting a dummy placeholder item at the head
position, and then moving forward from there, would
avoid this potential for lock contention.

Submitted:

Gordon Mohr ( gojomo ) - 2005-08-18 02:08

Priority:

Status:

Closed

Resolution:

None

Assigned:

Gordon Mohr

Category:

None

Group:

1.6.0

Visibility:

Public

Comments ( 2 )

Date: 2007-03-14 01:43 Sender: karl-ia This issue is now discussed in the new JIRA tracker at http://webteam.archive.org/jira/browse/HER-962 -- please add further comments at that location.
Date: 2005-11-07 23:50 Sender: gojomo Logged In: YES user_id=144912 Added. The 'cap' is a zero-length entry inserted at the 'origin' key of each queue, at the time the queue is first instantiated. getNextNearestItem() does a getSearchKey() to this cap, then proceeds to next, to get the real 'head' of a queue. Commit comment: Implementation of [ 1262665 ] add dummy items at heads of BDB queues for performance * BdbWorkQueue.java in constructor, add cap * BdbFrontier.java pass frontier into constructor to allow insertion of cap * BdbMultipleWorkQueues.java addCap() inserts the zero-length placeholder; getNextNearestItem() seeks to cap then gets next entry. Closing as finished (internal change with no easily testable expression).

Date: 2007-03-14 01:43
Sender: karl-ia

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-962 -- please add further
comments at that location.

Date: 2005-11-07 23:50
Sender: gojomo

Logged In: YES
user_id=144912

Added. The 'cap' is a zero-length entry inserted at the
'origin' key of each queue, at the time the queue is first
instantiated. getNextNearestItem() does a getSearchKey() to
this cap, then proceeds to next, to get the real 'head' of a
queue. Commit comment:

Implementation of [ 1262665 ] add dummy items at heads of
BDB queues for performance
* BdbWorkQueue.java
in constructor, add cap
* BdbFrontier.java
pass frontier into constructor to allow insertion of cap
* BdbMultipleWorkQueues.java
addCap() inserts the zero-length placeholder;
getNextNearestItem() seeks to cap then gets next entry.

Closing as finished (internal change with no easily testable
expression).

Attached File

Changes ( 5 )

Field	Old Value	Date	By
close_date	-	2005-11-07 23:50	gojomo
status_id	Open	2005-11-07 23:50	gojomo
assigned_to	nobody	2005-10-04 22:36	gojomo
artifact_group_id	None	2005-09-23 20:53	gojomo
priority	6	2005-09-23 18:37	gojomo

Heritrix: Internet Archive Web Crawler

Tracker: Feature Requests

Comments ( 2 )

Attached File

No Files Currently Attached

Changes ( 5 )

About SourceForge

Find Software

Develop Software

Community

Help