I am new to Aperture, and will use Aperture mainly for crawling web pages. How does Aperture tell if a web page has changed? Does it compare content? Can it use ETags?
It uses the java HttpURLConnection.setIfModifiedSince method to make the underlying HTTP request contain the the If-Modified-Since header, with the timestamp of last recorded modification. Then it relies on the server returning the HTTP 304 response in case a file has not been modified.
In this way we can do this with a single HTTP request, no need to issue a separate HTTP HEAD, and a HTTP GET afterwards.
The relevant code is in HttpAccessor class.
Log in to post a comment.
Sign up for the SourceForge newsletter:
You seem to have CSS turned off.
Please don't fill out this field.