Heritrix: Internet Archive Web Crawler

Tracker: Feature Requests

7 'total' bytes/fetches quota options in QuotaEnforcer - ID: 1393254

Last Update: Comment added ( karl-ia )

Details:

QuotaEnforcer only offers maximums for the 'success'
bytes/fetches -- not all fetches. On the list, Bjarne
of NB.DK points out they've hit sites where hundreds of
MB of non-success content have been fetched before a
10MB success quota is reached.

The QuotaEnforcer should also offer a max for the
'total' byte/fetch counts (which are also already being
tallied).

Operators might prefer this to a 'success' quota, or
want to use both together.

Submitted:

Gordon Mohr ( gojomo ) - 2005-12-30 00:03

Priority:

Status:

Closed

Resolution:

None

Assigned:

Gordon Mohr

Category:

None

Group:

1.8.0

Visibility:

Public

Comments ( 3 )

Date: 2007-03-14 01:45 Sender: karl-ia This issue is now discussed in the new JIRA tracker at http://webteam.archive.org/jira/browse/HER-993 -- please add further comments at that location.
Date: 2006-01-23 18:47 Sender: gojomo Logged In: YES user_id=144912 Added. Commit comment: Implementation of [ 1393254 ] 'total' bytes/fetches quota options in QuotaEnforcer * CrawlSubStats.java tally 'total' (successes+other responses) numbers properly * QuotaEnforcer.java add settings for all-response totals and KBs, for server/host/group tallies
Date: 2005-12-30 00:05 Sender: gojomo Logged In: YES user_id=144912 Additionally, the text/help descriptions of the current settings should be more clear that they only apply to 'successes'.

Attached File

Changes ( 3 )

Field	Old Value	Date	By
status_id	Open	2006-03-17 21:12	gojomo
artifact_group_id	None	2006-03-17 21:12	gojomo
close_date	-	2006-03-17 21:12	gojomo

Heritrix: Internet Archive Web Crawler

Tracker: Feature Requests

Comments ( 3 )

Attached File

No Files Currently Attached

Changes ( 3 )

About SourceForge

Find Software

Develop Software

Community

Help