Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-673 -- please add further
comments at that location.

Logged In: YES
user_id=144912

Closing as implemented; further fixes/enhancements to basic
ExtractorXML can be reported as new issues.

Logged In: YES
user_id=144912

Patterned after Igor's ExtractorCSS, I've created a
super-simple ExtractorXML. Commit comment:

Work towards [ 806831 ] XMLExtractor (XML/RSS)
* ExtractorXML.java
initial commit; based on ExtractorCSS;
extract HTTP URIs in XML attributes and simple elements
(with only the URI as non-whitespace content)

This seems enough to parse RSS, so this should probably be
added to the extractors in the default profiles.

Logged In: YES
user_id=924942

Here are rss parsers:

https://rome.dev.java.net/
http://jakarta.apache.org/commons/sandbox/feedparser/

Logged In: YES
user_id=924942

Interesting article on why xml on the web has failed. Aside
from that has interesting discussion of xml charset handling
-- or lack thereof -- over http:

http://www.xml.com/pub/a/2004/07/21/dive.html

Logged In: YES
user_id=924942

Changed summary again. Most rss feeds have a 'http' scheme.

Advantage to having a XMLExtractor -- the XMLExtractor would
do the rss files too; probably no advantage to an
exclusively rss extractor -- would be being able to exploit
the encoding stipulation at the head of the xml file. This
said, rare would be the case where the XML file would be in
an encoding that was other than single-byte or single-byte
amenable (UTF-8).

Another reason we might consider an XMLExtractor would be if
a purposed extractor was more performant than the UE (Would
it take much to make the processors standalone callable from
the command-line? Would make comparison the easier).

Logged In: YES
user_id=144912

UniversalExtractor might be considered 'rude' by some sites
-- in particular, discovering URIs that are specifically
crawler traps (in that they mark the offending crawler as
being too aggressive).

So there could be a good reason to split RSS/XML off
separately, to allow finer settings about aggressiveness.
For example, for RSS/XML, we probably want only URIs
delimited by \' or \" -- not delimited by spaces.

Logged In: YES
user_id=715474

I looked into RSS files and with out parsing rss documents
we ca not separate them from any other xml files. This might
be expensive just so that we can separate them from any
other xml files in order to parse possible relative URLs. In
general, RSS specs are not clear on how relative URLs should
be resolved. And as far as I learned about RSS files is that
relative urls are to be avoided for now. So we might just go
ahead and parse absolute URLs from all xml files with
universal extractor. An operator can just simple add filter
*.xml to this extractor and that is it.

Logged In: YES
user_id=892643

The universal extractor (ExtractorUniversal) does an ok job
of getting absolute URLs from XML files. It is unlikely that
a dedicated XML extractor could do much better. A
specialised RSS extractor might though be able to get
relative URLs that the universal extractor can not.

Logged In: YES
user_id=924942

Updated summary to talk of a FetchRSS and a (SAX)
ExtractorXML processor.