Heritrix: Internet Archive Web Crawler

Tracker: Feature Requests

5 [jmx] progress statistics as notification - ID: 1374947

Last Update: Comment added ( karl-ia )

Details:

Added progress statistics as notifications.

They are emitted at whatever the rate set for logging
in progress stats log (Its set in order file).

The emission is a little perverse. It is an instance
of AttributeChangeNotification:
http://www.cise.ufl.edu/depot/www/JDK-1.5.0/api/javax/management/AttributeC
hangeNotification.html.

source, seqnumber and timestamp should be obvious.
Message is the progress statistics as a string.
attributename is 'progressStatistics'. Attribute type
is 'java.util.HashMap'. old value is old statistics
HashMap, new value is new statistics as HashMap.

I toyed with sending each of the elements of the
hashmap as its own attributechangenotification (if has
changed since last progressstats time) but seems like
we'd be giving out 12 or 14 messages every 20 seconds
or so for each instance. The implementation would be
cleaner but might be too many messages.

Also, to make these notifications show up in CrawlJob,
had to rework StatisticsTracker so that its progress
statistics logging could be noticed in crawl controller
and subsequently in CrawlJob. Not pretty.

We might want to add registration of parties interested
in StatisticsTracker -- but then we'd have a 3rd event
publisher (Currently we have Crawl Status events
managed by CrawlController and Crawl URI changes
managed by frontier). Might want to amalgamate
eventing in Heritrix -- especially as the extant
eventing mechanism could do with some rework.

Submitted:

Michael Stack ( stack-sf ) - 2005-12-07 01:33

Priority:

Status:

Closed

Resolution:

None

Assigned:

Michael Stack

Category:

API

Group:

1.8.0

Visibility:

Public

Comments ( 3 )

Date: 2007-03-14 01:45 Sender: karl-ia This issue is now discussed in the new JIRA tracker at http://webteam.archive.org/jira/browse/HER-986 -- please add further comments at that location.
Date: 2005-12-07 22:27 Sender: stack-sf Logged In: YES user_id=924942 Changed how progress stats are sent as notifications. See commit below. More on '[ 1374947 ] [jmx] progress statistics as notification' Chatting more with Danny and looking at what the progress statistics looks like in MC4J and in JConsole, instead send progress stats hash as OpenType -- instead of of as Map -- in userdata field of a plain Notification rather than as data for an AttributeChangeNotification. Doing it this way, can see the notification both as a string in the message field and the OpenType can be interrogated by JConsole in the userdata field. * src/java/org/archive/crawler/admin/CrawlJob.java Removed keeping around cached statistics now we no longer send old and new but just the new. Convert the statistics Map to a CompositeData. Added keeping around CompositeType reused everytime we send a progress stats notification. Also, was sending progress stats notifications from wrong bean -- from the CrawlController override instead of out of CrawlJob. Fixed. Removed package name from Notification names. Redundant. * src/java/org/archive/crawler/admin/StatisticsTracker.java * src/java/org/archive/crawler/framework/StatisticsTracking.java Go back to plain Map. * src/java/org/archive/util/JmxUtils.java (createCompositeType): Added. * src/java/org/archive/util/JmxUtilsTest.java Added test of new createCompositeType method.
Date: 2005-12-07 01:55 Sender: stack-sf Logged In: YES user_id=924942 Implemented. Closing. Refactoring of eventing is covered by RFE '[ 1350723 ] Allow any processor emit crawl status change events'

Attached File

Changes ( 2 )

Field	Old Value	Date	By
status_id	Open	2005-12-07 01:55	stack-sf
close_date	-	2005-12-07 01:55	stack-sf

Heritrix: Internet Archive Web Crawler

Tracker: Feature Requests

Comments ( 3 )

Attached File

No Files Currently Attached

Changes ( 2 )

About SourceForge

Find Software

Develop Software

Community

Help