Search Results for "duplicate"
Sort By:
Showing 264 open source projects for "duplicate"
View related business solutions-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
1
Czkawka
Multi functional app to find duplicates, empty folders, similar images
Czkawka (Polish for “hiccup”) is a lightning‑fast, multi‑purpose file cleaning tool written in Rust. It helps users declutter storage by finding duplicate files, similar images or audio, empty folders, and unusually large files through CPU‑efficient multithreading. Available with both GUI (GTK‑based) and CLI versions for flexible usage. -
2
FDUPES
FDUPES is a program for identifying or deleting duplicate files
FDUPES is a lightweight command-line utility that helps users find and optionally delete duplicate files within specified directories by comparing file contents, which can be extremely useful for cleaning up storage clutter or organizing large collections of files. It works by scanning directories and subdirectories, identifying sets of files with identical content through size and hash comparisons, and then listing them together so users can examine duplicates. -
3
Mihomo
A simple Python Pydantic model for Honkai
Mihomo is a Python client library leveraging Pydantic to model parsed Honkai: Star Rail user data from the Mihomo public API. It provides structured types, type hints, and convenience methods to fetch and transform player profiles, daily stats, and character details efficiently. -
4
Beets
Open-source music library management system
Beets catalogs your music collection with a variety of tools for manipulating and accessing music. -
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
5
Calibre-Web Automated
Calibre-Web but Automated and with tons of New Features
Calibre-Web-Automated (CWA) is an all-in-one, self-hosted solution for managing an ebook library that combines the modern, lightweight web UI style of Calibre-Web with the deeper tooling and conversion capabilities associated with Calibre. The goal is to reduce the common “two-service” setup where users run Calibre-Web for browsing and Calibre separately for conversions, metadata fixes, and automation, by packaging those workflows together in a single system. CWA keeps the familiar strengths... -
6
claude-real-video
Let Claude (or any LLM) actually watch a video
...It accepts public video URLs or local files and extracts the visual and audio evidence an AI model needs. Instead of sampling at a fixed interval, it detects scene changes and removes near-duplicate frames to reduce unnecessary token usage. It can transcribe audio, generate a clean output folder, and create a local viewer with the video, keyframe grid, and transcript. The tool can also be installed as a Claude Code skill so an agent can process videos more directly. It is useful for reviewing screen recordings, summarizing visual content, analyzing tutorials, and turning video into LLM-readable context. -
7
DouK-Downloader
TikTok releases/likes/compilations/live streams/videos/atlases/music
...The software goes beyond simple downloading by offering comprehensive data collection features, including comments, user statistics, and trending data such as hot boards and search results. It supports automated workflows like incremental downloads, duplicate detection, and persistent storage of collected data in formats like CSV, XLSX, and SQLite, making it useful for both content archiving and analytics. The project also includes a Web UI interface and API support, allowing remote access, automation, and integration into larger systems or private servers. -
8
JSON Lint
JSON Lint for PHP
...Its lint method returns detailed parsing errors without decoding the input, while parse produces PHP values and throws exceptions when the document is invalid. Error messages include the relevant line, nearby text, a position marker, and the tokens expected by the parser. Optional flags can reject duplicate object keys, preserve them under generated names, or collect repeated values into arrays. The parser can also return associative arrays and accept JavaScript-style inline or block comments. UTF-8 validation detects malformed byte sequences and reports the first invalid position. The package supports a wide range of PHP releases and installs through Composer with an included jsonlint executable. -
9
Meta Package Manager
Wraps all package managers with a unifying CLI
Meta Package Manager wraps all package managers with a unifying CLI, and provides the MPM CLI, a wrapper around all package managers. MPM is like yt-dlp, but for package managers instead of videos. MPM solves XKCD #1654 - Universal Install Script. List installed packages. List duplicate installed packages. Search for packages. Install a package, remove a package, and list outdated packages. Sync local package infos. Upgrade all outdated packages. Backup list of installed packages to TOML file. Restore/install list of packages from TOML files. Pin-point commands to a subset of package managers (include/exclude selectors). ... -
Host LLMs in Production With On-Demand GPUsDeploy your model, get an endpoint, pay only for compute time. No GPU provisioning or infrastructure management required.
-
10
ydata-profiling
Create HTML profiling reports from pandas DataFrame objects
ydata-profiling primary goal is to provide a one-line Exploratory Data Analysis (EDA) experience in a consistent and fast solution. Like pandas df.describe() function, that is so handy, ydata-profiling delivers an extended analysis of a DataFrame while allowing the data analysis to be exported in different formats such as html and json. -
11
Promxy
An aggregating proxy to enable HA prometheus
...Similarly prometheus has no real built-in query federation, which means that you end up with N sources in grafana which is (1) confusing to grafana users and (2) has no support for aggregation across the sources. Promxy enables an HA prometheus setup by "merging" the data from the duplicate hosts (so if there is a gap in one, promxy will fill with the other). In addition Promxy provides a single datasource for all promql queries. -
12
diskover-community
Open source file indexing & storage analytics powered by Elasticsearch
...By indexing file metadata from sources such as local file systems, network shares like NFS and SMB, and cloud storage, the tool provides a centralized way to analyze heterogeneous storage environments. Diskover also helps identify outdated or unused files, duplicate data, and inefficient storage usage that can waste resources or increase operational costs. A Python-based indexing engine performs the scanning and indexing tasks. -
13
SortPhotos
SortPhotos is a Python script that organizes photos and videos
...SortPhotos includes options for copying versus moving files, recursive searches, silent or test modes, and customizable start times for when a “day” begins. It also prevents duplicate files by comparing content, with an option to keep duplicates if needed. With support for automation through launch agents or cron jobs, SortPhotos is well-suited for photographers, archivists, and anyone looking to streamline large personal or professional media collections. -
14
Percona Toolkit
A collection of advanced open source command-line tools
...Utilities such as pt-query-digest analyze slow logs and packet captures to surface hotspots and regressions, while pt-online-schema-change applies ALTERs with minimal blocking by copying and swapping tables. Consistency tools like pt-table-checksum and pt-table-sync detect and reconcile replication drift across replicas or shards. Other helpers find duplicate indexes, unused keys, skewed data distributions, and configuration issues that quietly erode performance. Each tool favors guardrails: dry runs, throttling, and resumable operations reduce the risk of harming production workloads. Operators and DBAs rely on the toolkit to bring observability and predictability to busy database fleets without hand-rolled scripts. -
15
supabase-py
Python Client for Supabase. Query Postgres from Flask, Django
Python Client for Supabase. Query Postgres from Flask, Django, FastAPI. Python user authentication, security policies, edge functions, file storage, and realtime data streaming. Good first issue. -
16
FramePack
Lets make video diffusion practical
FramePack explores compact representations for sequences of image frames, targeting tasks where many near-duplicate frames carry redundant information. The idea is to “pack” frames by detecting shared structure and storing differences efficiently, which can accelerate training or inference on video-like data. By reducing I/O and memory bandwidth, datasets become lighter to load while models still see the essential temporal variation. The repository demonstrates both packing and unpacking steps, making it straightforward to integrate into preprocessing pipelines. ... -
17
Grabber
Very customizable imageboard/booru downloader with powerful filenaming
Very customizable imageboard/booru browser and mass downloader, with powerful file naming features. Thanks to its powerful naming features, you just have to set your filename and save the directory using all the tokens available, and the program will generate a filename using the image's information. With this, you can store and manage your pictures in advanced directory structures, and save images with custom filenames. Imageboard/booru downloader which can download thousands of images from... -
18
janitor
Simple tools for data cleaning in R
janitor provides simple, convenient tools for data cleaning, formatting, and exploration in R. It is especially useful for cleaning messy data frames, removing duplicates, formatting column names, and producing frequency tables in a tidy workflow. -
19
Laravel Gamify
Gamify your Laravel app with Reputation Points & Achievements Badges
Use qcod/laravel-gamify to quickly add reputation point & badges in your Laravel app. If you are installing on Laravel 5.4 or lower you will need to manually register the Service Provider by adding it in the config/app.php providers array. -
20
Proxifly Free Proxy List
Free HTTP, SOCKS4, & SOCKS5 Proxy List * Updated every 5 minutes
Proxifly Free Proxy List by Proxifly is a public repository that provides automatically refreshed lists of working proxies. It collects HTTP, HTTPS, SOCKS4, and SOCKS5 proxies from around the web and validates them on a frequent schedule. The repository separates proxy data by protocol and provides the lists in multiple file formats. Its main value is freshness, since the list is updated every few minutes and removes duplicates. It can be useful for testing network tools, validating proxy... -
21
DefectDojo
DefectDojo is a DevSecOps and vulnerability management tool
DefectDojo is a security orchestration and vulnerability management platform. DefectDojo allows you to manage your application security program, maintain product and application information, triage vulnerabilities and push findings to systems like JIRA and Slack. DefectDojo enriches and refines vulnerability data using a number of heuristic algorithms that improve with the more you use the platform. DefectDojo integrates with 85+ security tools. DefectDojo has bi-directional integration with... -
22
Mediary Scout
Agent-driven media library for your cloud drives
Mediary Scout is an agent-driven media library that acquires movies, television shows, and anime for supported cloud drives. An LLM agent searches connected indexers, selects a result according to quality and subtitle preferences, transfers it, and verifies what arrived. The library tracks season coverage, missing episodes, airing status, and scheduled gap-filling tasks. It supports 115, Quark, and GuangYaPan as separate workspaces without requiring local media downloads. Users can run an... -
23
Nano ID
A secure, URL-friendly, unique string ID generator for JavaScript
Nano ID is a library for generating random IDs. Likewise UUID, there is a probability of duplicate IDs. However, this probability is extremely small. Meanwhile, a lot of projects generate IDs in small numbers. For those projects, the ID length could be reduced without risk. This calculator aims to help you realize the extent to which the ID length can be reduced. Instead of using the unsafe Math.random(), Nano ID uses the crypto module in Node.js and the Web Crypto API in browsers. ... -
24
trackerslist
Updated list of public BitTorrent trackers
trackerslist is a repository that provides continuously updated lists of public BitTorrent trackers, designed to improve torrent download performance and peer discovery. The project is maintained by an automated system that regularly checks tracker availability, removes duplicates, and ranks them based on reliability and latency. It offers multiple formats of tracker lists, including HTTP, HTTPS, WebSocket, and IP-based versions, making it compatible with a wide range of BitTorrent clients.... -
25
parcel/css
A CSS parser, transformer, and minifier written in Rust
...Many other CSS parsers treat property values as an untyped series of tokens. This means that each transformer that wants to do something with these values must interpret them itself, leading to duplicate work and inconsistencies. @parcel/css parses all values using the grammar from the CSS specification and exposes a specific value type for each property. Built on the cssparser and selectors crates created by Mozilla and used by Firefox and Servo. These provide a solid general-purpose CSS-parsing foundation on top of which @parcel/css implements support for all specific CSS rules and properties. ...
