DeDuplicator (Heritrix add-on) download

The DeDuplicator is an add-on module (plug-in) for the web crawler Heritrix. It offers a means to reduce the amount of duplicate data collected in a series of snapshot crawls.

Project Activity

See All Activity >

License

GNU Library or Lesser General Public License version 2.0 (LGPLv2)

Follow DeDuplicator (Heritrix add-on)

DeDuplicator (Heritrix add-on) Web Site

Other Useful Business Software

Train ML Models With SQL You Already Know

BigQuery automates data prep, analysis, and predictions with built-in AI assistance.

Build and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.

Start Free

Rate This Project

User Reviews

Be the first to post a review of DeDuplicator (Heritrix add-on)!

Additional Project Details

Languages

English

Intended Audience

Advanced End Users, Developers, System Administrators

User Interface

Plugins

Programming Language

Java

Related Categories

Java Internet Software, Java Web Scrapers

Registered

2006-11-06

Similar Business Software

SurveyJS

SurveyJS is an embeddable, self-hosted, white-label form builder for teams building custom forms, surveys, questionnaires, and other data collection tools inside web applications. It runs entirely on the client and is fully compatible with all modern JavaScript frameworks, including React,...

See Software
Oxylabs

Oxylabs is a market leader in web intelligence with enterprise-grade, ethical, and compliant solutions. Its proxy infrastructure spans one of the largest global networks, offering residential, ISP, mobile, datacenter, & dedicated datacenter proxies, along with Web Unblocker – an AI-driven...

See Software
PathSolutions TotalView

PathSolutions TotalView network monitoring and troubleshooting software bridges the gap between NETWORK MONITORING and TROUBLESHOOTING RESOLUTION telling you WHEN, WHERE and WHY network errors occur. PathSolutions TotalView continuously monitors and tracks the performance of every device and...

See Software
Grafana Cloud

Grafana Labs delivers the leading AI-powered observability platform, built around Grafana—the world’s most widely adopted open source technology for dashboards and visualization. Recognized as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms (3x) and furthest in...

See Software
JDisc Discovery

JDisc Discovery is a comprehensive network inventory and IT asset management solution designed to help organizations gain clear, up-to-date visibility into their IT environment. It automatically scans and maps devices across the network, including servers, workstations, virtual machines, and...

See Software
Files.com

Files.com is a cloud-native Managed File Transfer (MFT) platform that unifies file transfers, sharing, and automation across any cloud, protocol, or partner. It connects 50+ storage systems — including Amazon S3, Azure, Google Drive, SharePoint, Dropbox, and Box — presenting them as a single...

See Software

Report inappropriate content