Heritrix: Internet Archive Web Crawler

Tracker: Feature Requests

5 Rotation of crawl logs - ID: 1222764

Last Update: Comment added ( karl-ia )

Details:

Add facility that allows rotation off of crawl logs.
Wanted by AUS crawl because logs are huge becoming
unmanageable.

Submitted:

Michael Stack ( stack-sf ) - 2005-06-17 16:27

Priority:

Status:

Closed

Resolution:

None

Assigned:

Karl Thiessen

Category:

None

Group:

1.6.0

Visibility:

Public

Comments ( 2 )

Date: 2007-03-14 01:42 Sender: karl-ia This issue is now discussed in the new JIRA tracker at http://webteam.archive.org/jira/browse/HER-948 -- please add further comments at that location.
Date: 2005-06-17 16:48 Sender: stack-sf Logged In: YES user_id=924942 Added. Below is commit message. Assigning Karl for testing of new feature. See user manual note for how its supposed to work. Implement '[ 1222764 ] Rotation of crawl logs'. * src/articles/user_manual.xml Note on new rotate logs feature. * src/java/org/archive/crawler/admin/CrawlJob.java Line lengths. Added rotateLogs and (unimplemented) checkpoint jmx operations. * src/java/org/archive/crawler/admin/CrawlJobHandler.java (rotateLogs): Added. * src/java/org/archive/crawler/checkpoint/CheckpointContext.java Minor refactoring. Not yet complete (Added getting of linked list of checkpoints, finished support for checkpoint prefix). * src/java/org/archive/crawler/framework/AbstractTracker.java javadoc. line lengths. * src/java/org/archive/crawler/framework/CrawlController.java Formatting. Added notes on checkpoint implemetation. (rotateLogFiles): Added null param and suffix string overrides. Refactored orginal method. * src/java/org/archive/io/GenerationFileHandler.java Formatting. * src/webapps/admin/logs.jsp Show 'rotate logs' link if a current job and its paused. * src/webapps/admin/console/action.jsp Add rotateLogs action.

Date: 2007-03-14 01:42
Sender: karl-ia

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-948 -- please add further
comments at that location.

Date: 2005-06-17 16:48
Sender: stack-sf

Logged In: YES
user_id=924942

Added. Below is commit message.

Assigning Karl for testing of new feature.

See user manual note for how its supposed to work.

Implement '[ 1222764 ] Rotation of crawl logs'.
* src/articles/user_manual.xml
Note on new rotate logs feature.
* src/java/org/archive/crawler/admin/CrawlJob.java
Line lengths.
Added rotateLogs and (unimplemented) checkpoint jmx
operations.
* src/java/org/archive/crawler/admin/CrawlJobHandler.java
(rotateLogs): Added.
* src/java/org/archive/crawler/checkpoint/CheckpointContext.java
Minor refactoring. Not yet complete (Added getting of
linked list of
checkpoints, finished support for checkpoint prefix).
* src/java/org/archive/crawler/framework/AbstractTracker.java
javadoc. line lengths.
* src/java/org/archive/crawler/framework/CrawlController.java
Formatting. Added notes on checkpoint implemetation.
(rotateLogFiles): Added null param and suffix string
overrides. Refactored
orginal method.
* src/java/org/archive/io/GenerationFileHandler.java
Formatting.
* src/webapps/admin/logs.jsp
Show 'rotate logs' link if a current job and its paused.
* src/webapps/admin/console/action.jsp
Add rotateLogs action.

Attached File

Changes ( 4 )

Field	Old Value	Date	By
artifact_group_id	None	2005-09-23 21:08	gojomo
status_id	Open	2005-06-17 16:48	stack-sf
assigned_to	nobody	2005-06-17 16:48	stack-sf
close_date	-	2005-06-17 16:48	stack-sf

Heritrix: Internet Archive Web Crawler

Tracker: Feature Requests

Comments ( 2 )

Attached File

No Files Currently Attached

Changes ( 4 )

About SourceForge

Find Software

Develop Software

Community

Help