Search Results for "deduplication"
Sort By:
Showing 56 open source projects for "deduplication"
View related business solutions-
Host LLMs in Production With On-Demand GPUsDeploy your model, get an endpoint, pay only for compute time. No GPU provisioning or infrastructure management required.
-
Demo Series - Small Business Backup By VeeamWatch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.
-
1
ipwb
A distributed and persistent archive replay system using IPFS
InterPlanetary Wayback (ipwb) facilitates permanence and collaboration in web archives by disseminating the contents of WARC files into the IPFS network. IPFS is a peer-to-peer content-addressable file system that inherently allows deduplication and facilitates opt-in replication. ipwb splits the header and payload of WARC response records before disseminating into IPFS to leverage the deduplication, builds a CDXJ index with references to the IPFS hashes returned, and combines the header and payload from IPFS at the time of replay. An important aspect of archival replay systems is rewriting various resource references for proper memento reconstruction so that they are dereferenced properly from the archive from around the same datetime as of the root memento and not from the live site (in which case the resource might have changed or gone missing). ... -
2
Kopia
Cross-platform backup tool for Windows, macOS and Linux
...Kopia has both a command-line interface for automation and scripting as well as a graphical UI for interactive use, making it suitable for advanced users and those who prefer visual tools. Its architecture supports end-to-end encryption, optional compression, and deduplication, so multiple backups can share data efficiently, and repositories can be stored securely even in untrusted locations. -
3
Zingg
Scalable master data management and identity resolution
...It uses machine learning to learn how records should be compared, reducing the need for brittle hand-written matching rules. The project is designed for data engineering and analytics teams working on customer 360, supplier 360, deduplication, fuzzy matching, data quality, and golden record workflows. Zingg runs on Apache Spark and can scale to large data lake, warehouse, and cloud platform environments. It supports configuration-driven pipelines where users define input data, match fields, training data, models, and output destinations. Its main value is helping organizations unify fragmented records into reliable entity clusters while keeping the process trainable, explainable, and repeatable. -
4
Note67
A private, local meeting notes assistant
...Users can record meetings directly from their microphone, view live transcriptions, filter by speaker, and export structured summaries, making it useful for professionals who need searchable, organized records of discussions. It also features thoughtful signal processing such as voice activity detection and echo deduplication to improve transcription accuracy, and provides standard note-taking features. -
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
5
FEAPDER
Powerful Python crawler framework for scalable web scraping tasks
...It also integrates monitoring and alerting capabilities to help developers track crawler performance and detect issues during execution. feapder includes browser rendering support for handling dynamic web pages and provides mechanisms for large-scale data deduplication during crawling. -
6
wagmi
React Hooks for Ethereum
wagmi is a collection of React Hooks containing everything you need to start working with Ethereum. wagmi makes it easy to "Connect Wallet," display ENS and balance information, sign messages, interact with contracts, and much more, all with caching, request deduplication, and persistence. We create a wagmi Client and pass it to the WagmiConfig React Context. The client is set up to use the ethers Default Provider and automatically connect to previously connected wallets. Next, we use the useConnect hook to connect an injected wallet (e.g. MetaMask) to the app. Finally, we show the connected account's address with useAccount and allow them to disconnect with useDisconnect. ... -
7
DOLMA
Data and tools for generating and inspecting OLMo pre-training data
DOLMA (Data Optimization and Learning for Model Alignment) is a framework designed to manage large-scale datasets for training and fine-tuning language models efficiently. -
8
Bareos
Bareos is a cross-network Open Source backup solution
Bareos (Backup Archiving Recovery Open Sourced) is an enterprise-grade open-source backup solution forked from Bacula. It offers robust backup, restore, and archiving features for Linux, Windows, and macOS systems. Bareos supports encrypted and deduplicated backups across networks, making it ideal for managing large infrastructure backups. -
9
AutoCVE
Agent-driven automated CVE discovery platform for source code auditing
AutoCVE is an agent-driven platform for automating authorized source-code auditing and CVE discovery. It covers the workflow from project selection and repository import to vulnerability verification and submission-ready report generation. An orchestrator coordinates specialized agents for reconnaissance, scanning, triage, deep analysis, verification, and final consolidation. Users can choose enhanced scanning, intelligent auditing, or comprehensive auditing based on speed and depth... -
Build Securely on Azure with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
10
ReClip
Download videos from almost any website
...Users can paste multiple URLs at once, select output formats such as MP4 or MP3, and choose quality settings before downloading. The system also includes features like automatic URL deduplication and batch processing to improve usability. -
11
ip2region
Ip2region is an offline IP-to-Region localization library and IP data
...It ships with raw IP datasets and optimized XDB database files for city-level region lookup. Returned records can include country, province, city, ISP, and ISO country-code information. The XDB format supports large datasets, deduplication, compression, and customizable region fields. Query implementations are available for many languages, including Go, Java, Python, PHP, Rust, C, C++, JavaScript, C#, Lua, Erlang, and others. Several cache strategies allow applications to trade memory use for faster query performance, with fully cached lookups operating at microsecond-level speeds. ... -
12
OpenMeter
Metering and Billing for AI, API and DevOps
OpenMeter is an open-source metering and billing platform designed to collect, aggregate, and analyze usage events from APIs, cloud infrastructure, and software services in real time, enabling flexible usage-based billing for SaaS, AI, and DevOps offerings. It supports high-scale event ingestion and deduplication to accurately record how customers consume billable resources such as API calls, compute time, or storage, and then correlates that usage with payment systems and billing plans to automate invoicing and revenue recognition. The system includes metering, storage, cataloging of products and pricing rules, and tools to enforce limits or quotas, supporting both self-service customer portals and internal dashboards. ... -
13
ClusterFuzz
Scalable fuzzing infrastructure
ClusterFuzz is a scalable fuzzing infrastructure that finds security and stability issues in software. Google uses ClusterFuzz to fuzz all Google products and as the fuzzing backend for OSS-Fuzz. ClusterFuzz provides many features which help seamlessly integrate fuzzing into a software project's development process. Can run on any size cluster (e.g. OSS-Fuzz instance runs on 100,000 VMs). Fully automatic bug filing, triage and closing for various issue trackers (e.g. Monorail, Jira).... -
14
OpenArchiver
An open-source platform for legally compliant email archiving
...It’s designed for scenarios where reliable, tamper-proof archiving and full-text search across both emails and attachments are essential for legal discovery, compliance, or long-term records retention. The platform combines a modern web UI with powerful backend services, including fast indexing, deduplication, encryption at rest, and asynchronous ingestion workflows, making it suitable for both small teams and enterprise deployments. Beyond simply capturing email, it emphasizes security and auditability with features like secure storage formats, file integrity verification, and detailed audit trails of user interactions. -
15
SWR
React Hooks library for remote data fetching
The name “SWR” is derived from stale-while-revalidate, a HTTP cache invalidation strategy popularized by HTTP RFC 5861. SWR is a strategy to first return the data from cache (stale), then send the fetch request (revalidate), and finally come with the up-to-date data. With SWR, components will get a stream of data updates constantly and automatically. And the UI will be always fast and reactive. With just one single line of code, you can simplify the logic of data fetching in your project,... -
16
NeMo Curator
Scalable data pre processing and curation toolkit for LLMs
NeMo Curator is a Python library specifically designed for fast and scalable dataset preparation and curation for large language model (LLM) use-cases such as foundation model pretraining, domain-adaptive pretraining (DAPT), supervised fine-tuning (SFT) and paramter-efficient fine-tuning (PEFT). It greatly accelerates data curation by leveraging GPUs with Dask and RAPIDS, resulting in significant time savings. The library provides a customizable and modular interface, simplifying pipeline... -
17
Nuke
Image loading system
Nuke ILS provides an efficient way to download and display images in your app. It's easy to learn and use thanks to a clear and concise API. Its architecture enables many powerful features while offering virtually unlimited possibilities for customization. Despite the number of features, the framework is lean and compiles in just under 3 seconds¹. Nuke has an automated test suite 2x the size of the codebase itself, ensuring excellent reliability. Every feature is carefully designed and... -
18
Pachyderm
Data-Centric Pipelines and Data Versioning
...Automatic immutable data lineage and data versioning of all data types. Autoscaling and parallel processing built on Kubernetes for resource orchestration. Uses standard object stores for data storage with automatic deduplication. Runs across all major cloud providers and on-premises installations. Automatic and intelligent versioning of even the largest data sets of unstructured and structured data. Git-like structure enables effective team collaboration. Full versioning for metadata including all analysis, parameters, artifacts, models, and intermediate results. ... -
19
syzkaller
syzkaller is an unsupervised coverage-guided kernel fuzzer
syzkaller is Google’s coverage-guided, feedback-driven kernel fuzzer designed to uncover reliability and security bugs in operating system kernels at scale. It automatically generates, mutates, and minimizes system call programs, then drives them through a specialized executor (syz-executor) to exercise deep kernel paths. The system integrates tightly with sanitizers such as KASAN, KMSAN, KCSAN, and UBSAN to surface memory safety, concurrency, and undefined behavior issues with actionable... -
20
BmuS
BmuS - Powerful linux backup program deduplication, encryption & more
...Visit the Quick Start Guide and FAQ on how to install Docker and Bmus on a Mac or Windows. https://www.youtube.com/watch?v=ksfYJlpqfCw BmuS features encryption, deduplication, Cloud Backups and much more. One of the key features that has received special attention (or is it called “Love”?) is the dashboard ( https://www.back-me-up-scotty.com/dashboards/bmus_dashboard.html ), which is probably the most unique feature of BmuS, apart from the fact that only a few backup tools can back up files AND MySQL-DB. -
21
Blazefox
Blazefox: Fast, safe file deduplication and management tool for Linux
Blazefox is a powerful and efficient file deduplication and management tool designed for both everyday users and system administrators. It scans directories recursively, identifies duplicate files using advanced content hashing, and provides safe options to remove, move, or rename duplicates. Blazefox supports flexible filtering with regex/glob patterns, interactive and dry-run modes, and advanced conflict resolution strategies, ensuring no data is lost accidentally. -
22
Kemono Downloader
Kemono Downloader - A cross-platform Python app built with PyQt6
Welcome to Kemono Downloader, a versatile Python-based desktop application built with PyQt6, designed to download content from Kemono.su. This tool enables users to archive individual posts or entire creator profiles from services like Patreon, Fanbox, and more, supporting a wide range of file types with customizable settings and advanced features. -
23
zpaqfranz
Zpaq compatible archiver for Win, Linux, Free/OpenBSD, Solaris & MacOS
High-Performance Archiver & Backup for Mission-Critical Data Bulletproof archival for multi-TB datasets, NAS, and VM backups Built for speed: multi-core CPUs and modern SSD/NVMe drives to achieve 1 GB/s+ throughput, up to 3x faster than 7-Zip Far more efficient and versatile than Time Machine or ZFS snapshots Key Features: ✓ Deduplicated, incremental & versioned backups ✓ Anti-ransomware integrity checks & deep validation ✓ Multi-hash support: SHA-2, SHA-3, BLAKE3, XXH3 &... -
24
ByteSync
All-in-One Sync, Backup & Deduplication Tool
ByteSync is a powerful open-source tool for remote file comparison and synchronization, designed to transfer only what truly matters: the differences. Whether you're syncing large files, backing up remote folders, or handling extensive sets of small files, ByteSync delivers speed, security, and efficiency—without unnecessary data transfers. Compatible with Windows, macOS, and Linux, ByteSync leverages cloud-based orchestration with end-to-end encryption, eliminating the need for VPNs or... -
25
baresnap
C11 bare-metal backup suite. FastCDC/AES-NI turbo silicon pipelines.
BareSnap is an open-source backup tool built with a "Spartan" philosophy that prioritizes raw hardware performance ("real metal"). It strictly rejects modern corporate abstractions like Docker, Kubernetes, or heavy external dependencies. KEY FEATURES: * High Performance: Uses a linear entropy pipeline (FastCDC + XDelta3 + LZ4/ZSTD) capable of processing gigabytes of local data in 0.3 seconds. * Adaptive Cryptography: Includes a self-contained module that dynamically ...
