Simple Web Spider / News: Recent posts

Changes:
- FIX: Reduce logging of exception stack trace, if only standard HTTP warnings are logged like socket timeout
- ADD: allow to use configure file for most configurations
- ADD: Host throttler to try to reduce requesting same domain
- UPDATE: Upgrade db4o 7.11 to 7.12
- UPDATE: Upgrade HTTPClient 3 to 4
- UPDATE: Separate databases between queue and hashes.
- UPDATE: configure to use java logger as default, if available. And setup logging to file.
- UPDATE: bootstrapping URLs will be used at first on every boot and not imported to database base anymore.... read more

This version has stabilized the memory usage by using new database engine db4o. Handling url database with a size greater than one giga byte is no problem anymore.

Important parsing improvements and fixes for Simple Web Spider. A tool to spider the web indefinitely... Can be used with proxies like YaCy (p2p crawler and search portal) and for generating web access for obfuscating web access, e.g. if your government do Telecommunications data retention.

Memory and speed improvements

Replace HtmlCleaner and XPath with own parser Fixes 152909
add Skipping binary content Fixes 152910
add Add PMD to eclipse, too
fix Do not log explicit ignored protocols Fixes 152940
fix Do not log explicit ignored mime types Fixes 152941

Minor distribution changes
Major build management improvements

New release 0.0.7 is available...