DeDuplicator (Heritrix add-on) download

The DeDuplicator is an add-on module (plug-in) for the web crawler Heritrix. It offers a means to reduce the amount of duplicate data collected in a series of snapshot crawls.

Project Activity

See All Activity >

License

GNU Library or Lesser General Public License version 2.0 (LGPLv2)

Follow DeDuplicator (Heritrix add-on)

DeDuplicator (Heritrix add-on) Web Site

Other Useful Business Software

Forever Free Full-Stack Observability | Grafana Cloud

Our generous forever free tier includes the full platform, including the AI Assistant, for 3 users with 10k metrics, 50GB logs, and 50GB traces.

Built on open standards like Prometheus and OpenTelemetry, Grafana Cloud includes Kubernetes Monitoring, Application Observability, Incident Response, plus the AI-powered Grafana Assistant. Get started with our generous free tier today.

Create free account

Rate This Project

User Reviews

Be the first to post a review of DeDuplicator (Heritrix add-on)!

Additional Project Details

Languages

English

Intended Audience

Advanced End Users, Developers, System Administrators

User Interface

Plugins

Programming Language

Java

Related Categories

Java Internet Software, Java Web Scrapers

Registered

2006-11-06

Similar Business Software

Oxylabs

Oxylabs is a market leader in web intelligence with enterprise-grade, ethical, and compliant solutions. Its proxy infrastructure spans one of the largest global networks, offering residential, ISP, mobile, datacenter, & dedicated datacenter proxies, along with Web Unblocker – an AI-driven...

See Software
Grafana Cloud

Grafana Labs delivers the leading AI-powered observability platform, built around Grafana—the world’s most widely adopted open source technology for dashboards and visualization. Recognized as a Leader in the 2025 Gartner® Magic Quadrant™ for Observability Platforms, Grafana Labs supports more...

See Software
PathSolutions TotalView

PathSolutions TotalView network monitoring and troubleshooting software bridges the gap between NETWORK MONITORING and TROUBLESHOOTING RESOLUTION telling you WHEN, WHERE and WHY network errors occur. PathSolutions TotalView continuously monitors and tracks the performance of every device and...

See Software
Files.com

Files.com is a cloud-native Managed File Transfer (MFT) platform that unifies file transfers, sharing, and automation across any cloud, protocol, or partner. It connects 50+ storage systems — including Amazon S3, Azure, Google Drive, SharePoint, Dropbox, and Box — presenting them as a single...

See Software
Google Cloud BigQuery

BigQuery is a serverless, multicloud data warehouse that simplifies the process of working with all types of data so you can focus on getting valuable business insights quickly. At the core of Google’s data cloud, BigQuery allows you to simplify data integration, cost effectively and securely...

See Software
JDisc Discovery

JDisc Discovery is a comprehensive network inventory and IT asset management solution designed to help organizations gain clear, up-to-date visibility into their IT environment. It automatically scans and maps devices across the network, including servers, workstations, virtual machines, and...

See Software