Heritrix: Internet Archive Web Crawler

This issue is now discussed in the new JIRA tracker at
http://webteam.archive.org/jira/browse/HER-989 -- please add further
comments at that location.

Logged In: YES
user_id=924942

Actually close.

Logged In: YES
user_id=924942

Added a user-agent. Here's from an apache access log.

127.0.0.1 - - [18/Aug/2006:17:12:31 -0700] "GET /test.arc.gz
HTTP/1.1" 206 14858 "-" "org.archive.io.arc.ARCReader" "-"

Thanks for testing Karl.

Closing. Commit below:

Finish '[ 1387423 ] [arcreader] Fetch records and iterate
remote ARCs'
* src/java/org/archive/io/WriterPoolMember.java
* src/java/org/archive/io/arc/ARCReader.java
Javadoc edit.
* src/java/org/archive/io/arc/ARCReaderFactory.java
Add a user agent.
* src/java/org/archive/io/arc/ARCReaderFactoryTest.java
Minor change to commented-out test (Test is commented
out because
requires net but is useful testing ARCReading over http).

Logged In: YES
user_id=144912

add user-agent if possible then close

Logged In: YES
user_id=1269624

This works just fine, but doesn't supply a User-Agent to the
HTTP server it's getting the remote ARC from. Something
like "ARCreader 1.8.x (archive.org)" might be appropriate here.

Logged In: YES
user_id=924942

Finished up. Now possible to pass an URL to ARC and have it
read iterate over records across network stream.

Assigning Karl.

TO TEST:

+ Verify ARCReader works at it used to (Don't have to spend
too much time on this 'cos lots of unit tests that verify
ARCReader operation -- but maybe I broke the cmdline
interface some?).
+ Test using ARCReader to read remote ARCs specified with an
HTTP URL.


Finish up on '[ 1387423 ] [arcreader] Fetch records and
iterate remote ARCs'/
* src/java/org/archive/io/RepositionableInputStream.java
Add constructor so can set buffer size. Added javadoc of
experience
using this instance. Fix our bypassing underlying
BufferedInputStream
buffer by calling its read rather than read against
passed stream
directly. Add overrides to ensure position always gets
updated on read.
(position): Cleaned up math.
* src/java/org/archive/io/RepositionableInputStreamTest.java
More thorough, awkward testing of RepositionableInputStream.
* src/java/org/archive/io/arc/ARCReader.java
(alignedOnFirstRecord): Added. Flag is false when
backing stream did
not start at zeroth position (e.g. the ARCReader was
made against an
offset into a stream).
* src/java/org/archive/io/arc/ARCReaderFactory.java
Moved the gets around so gets that take File are grouped
together. Same
for those that take URL. Added argument to the
compressed arc reader
constructor -- an alignedOnFirstRecord argument. I
removed the
get override for URL that allowed specifying an offset.
Wait till someone
needs it. Meantime, its clear which get pulls ARCs local
-- the one
that doesn't take an offset. The one that does take an
offset tries to
read the ARCs remotely (Even if offset is zero -- note
its now possible
to iterate over complete ARC contents over a network
stream).
* src/java/org/archive/io/arc/ARCReaderFactoryTest.java
Changed commented out test used testing reading ARCs
over net.