Search Results for "deduplication" - Page 2
Sort By:
Showing 54 open source projects for "deduplication"
View related business solutions-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
1
text-dedup
All-in-one text de-duplication
...It supports Jaccard similarity thresholding, parallel execution, and flexible deduplication strategies, making it ideal for cleaning web-scraped data, language model training datasets, or document archives. -
2
orogene
Makes `node_modules/` happen. Fast. No fuss
Orogene is a next-generation package manager designed for Node.js environments, focusing on speed, efficiency, and seamless integration with tools that utilize node_modules/, such as bundlers and CLI applications. It employs a central store for dependencies, deduplicates packages, and leverages copy-on-write techniques on supported filesystems to minimize disk usage and accelerate loading times. Orogene aims to provide a robust and user-friendly experience, ensuring that developers can... -
3
XSIBackup-DC
VMWare ESXi Virtual Machine Backup, runs in free ESXi
©XSIBackup-DC is a C binary that runs in the ©ESXi shell and behaves as a backup and replica service. It follows the same principles as our previous ©XSIBackup editions, that is: a utility that runs in the hypervisor command line and accepts arguments and values stating which VMs to backup and where to copy them. It accepts local and remote SSH paths, thus you can backup and replicate VMs to local datastores or to remote repositories in any additional ©ESXi or Linux server. Download... -
4
graphql-react
A GraphQL client for React using modern context and hooks APIs
graphql-react is a lightweight GraphQL client for React that emphasizes efficiency and simplicity. It provides built-in caching, request deduplication, and declarative data fetching without the complexity of larger GraphQL clients. -
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
5
XSIBackup-Classic
Local and over IP backups for ©VMWare ©ESXi 5.1 to 6.7
...Consists in a cron programmable service running in the hypervisor. Hot backups, fully functional, standalone, ideal for dedicated servers. Full version supports quiescing, differential backup, deduplication and much more... Download link: https://33hops.com/xsibackup-vmware-esxi-backup.html Free VMWare backup solution for the ESXi hypervisor. Linux style command line tool. Crontab with configurable disk space autoprovision, set the room you want to reserve for your backups and rotate them. Full backups may be done with vmkfstools or by means of Rsync and its Delta algorithm (read: https://33hops.com/blog_xsibackup-rsync-considerations.html), to a local datastore or over TCP/IP tunneled through SSH, so any transfer over IP is secure. -
6
wadmerge
Merges WAD files for Doom engine games.
...Takes any number of input wads, and produces a sorted single output .WAD file. Runs on Linux and Windows, does not duplicate wad entries and allows space saving with an optional 'deduplication' method, where different entries which have the same data, only have the data stored once. -
7
This is a simple BASH script which will backup source directories to a destination target, such as a mounted NFS share. It will create a full directory structure matching the source, but only use disk space for new/changed files. This is done by linking existing files. Essentially, it backs up like an incremental backup, and consumes space like an incremental backup, but restores like a full backup.
-
8
Open Source Data Quality and Profiling
World's first open source data quality & data preparation project
This project is dedicated to open source data quality and data preparation solutions. Data Quality includes profiling, filtering, governance, similarity check, data enrichment alteration, real time alerting, basket analysis, bubble chart Warehouse validation, single customer view etc. defined by Strategy. This tool is developing high performance integrated data management platform which will seamlessly do Data Integration, Data Profiling, Data Quality, Data Preparation, Dummy Data... -
9
BackupPC
BackupPC is a high-performance, enterprise-grade system
BackupPC is a high-performance, enterprise-grade system for backing up Linux, Windows, and macOS systems to a central server using rsync, tar, SMB, or FTP. It uses hard-linking and compression to efficiently store multiple backups of the same files and offers a web interface for administration and restore. BackupPC is suitable for environments where centralized, disk-based backups are needed. -
Demo Series - Small Business Backup By VeeamWatch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.
-
10
UrBackup
Open Source Client/Server Network Backup System
Efficient Client-Server Backup system for Linux, macOS and Windows. A client for Windows, FreeBSD and Linux lets you backup open files and complete partition images (Windows). Backups are stored to disks in a efficient way (deduplication) on either Windows or Linux servers. -
11
KonDB
Organizer for your Kontakt files
- Easily Organize, Sort Through, and Load your Kontakt NKI/NKM Files in Windows - This project was built to satisfy a need to have a separate app to manage a large Kontakt files collection. - This app does not delete/move/rename/etc. your files, but it will make management of your files a lot easier. Getting Started: Drag-and-drop files/folders into this app from your computer at any time to add to your index. If you drag in the same file(s) twice, it won't be re-added. Drag-and-drop... -
12
tbackup2
Backup tool
Backup files using rsync and deduplication with hardlinks. Written in ruby. -
13
PRIMAL_RAD
DICOM tag modification and routing tool
PRIMAL can be used to route, modify and store DICOM Radiology image files. Typically this tool is put between a image acquisition station (modality) and a PACS to route images and modify the DICOM tags in ways that are not easy to do on the acquisition station but needs to be done prior to the study arriving at the PACS. PRIMAL is designed to turn a Fedora or Redhat Linux server into a DICOM routing appliance. Most information can be viewed through the web interface. Commercial... -
14
Offnet
Program that saves complete web pages retaining multiple timestamps
Offnet is an open source tool for mirroring web pages. It lets you manage several snapshots of web pages in order to retain the newest as well as some elder versions of a web page. It has also file based deduplication features to store content efficiently. The application comes with an integrated web server to navigate directly through your web browser. Project goals: - Web page downloads for less experienced users, including easy setup - Project based page maintanance - Not too plain functions that include also multiple snapshots per project - Iterative, understandable and storage efficient data structure to enable more manual control over stored pages (meta files editable with Easy Folder Morpher) - Retain archived files and query links as original, altering links only during query Current status: - Alpha stadium, archivation quality below Heritrix -
15
RSMQ
Redis Simple Message Queue
A lightweight Redis-backed message queue for handling simple distributed messaging and job processing. -
16
GraphQL multipart request specification
A spec for GraphQL multipart form requests (file uploads).
An interoperable multipart form field structure for GraphQL requests, used by various file upload client/server implementations. An “operations object” is an Apollo GraphQL POST request (or array of requests if batching). An “operations path” is an object-path string to locate a file within an operations object. -
17
DataCleaner
Data quality analysis, profiling, cleansing, duplicate detection +more
DataCleaner is a data quality analysis application and a solution platform for DQ solutions. It's core is a strong data profiling engine, which is extensible and thereby adds data cleansing, transformations, enrichment, deduplication, matching and merging. Website: http://datacleaner.github.io -
18
ZipSnapNG
Portable multi-platform backup tool with advanced features
ZipSnapNG is a portable multi-platform backup tool which, despite its simplicity (200Kb and no installation required), provides advanced features such as incremental backup, data deduplication, encryption and configurable retention. It may run through the command line or within a graphical interface. -
19
cyphertite
Secure, encrypted, zero-knowledge online data backup.
Introducing Cyphertite 2.0: An all new look and feel in Windows! Cyphertite is a tar-like secure remote archiver. It deduplicates, compresses, and encrypts data prior to transmission, providing total privacy while reducing unnecessary wire traffic. It seamlessly supports IPv6 and IPv4 on a variety of platforms. -
20
Lessfs is an userspace (fuse) inline data de-duplicating filesystem for Linux that includes support for lzo or QuickLZ compression and encryption.
-
21
lsdup
List files with duplicate contents
lsdup, short for "list duplicates", is a command line tool which can assist in finding and removing files with identical contents. Unlike many file deduplication tools, this has a "comparison mode" which lets you tell lsdup to "only read the files in this directory, do not consider any for duplicates... but, the files in that other directory may have duplicates, so delete (or move) any of those." lsdup also has normal deduplication, where, if given a single directory, it will sort the originals from the duplicates. ... -
22
ZPAQ
Archiver and compression library
zpaq is a journaling archive for incremental backups with deduplication. Journaling means that the archive is append-only and you can revert to an earlier state to recover old versions of files. Compression is based on the open ZPAQ standard, a self-describing format that preserves compatibility as improved algorithms are developed. The default compression level is faster than zip with better compression. -
23
-
24
-
25
MarcXimiL is a flexible multi-platform bibliographic similarity analysis framework. Features: deduplication, information monitoring, visual analysis, plagiarism detection. Supported: MARCXML, OAI-PMH2 harvesting, and importation of text MARC.