Search Results for "duplicates"
Sort By:
Showing 211 open source projects for "duplicates"
View related business solutions-
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
1
Czkawka
Multi functional app to find duplicates, empty folders, similar images
Czkawka (Polish for “hiccup”) is a lightning‑fast, multi‑purpose file cleaning tool written in Rust. It helps users declutter storage by finding duplicate files, similar images or audio, empty folders, and unusually large files through CPU‑efficient multithreading. Available with both GUI (GTK‑based) and CLI versions for flexible usage. -
2
FDUPES
FDUPES is a program for identifying or deleting duplicate files
...It works by scanning directories and subdirectories, identifying sets of files with identical content through size and hash comparisons, and then listing them together so users can examine duplicates. Once duplicates are identified, the tool offers interactive deletion options where users can choose which copy to keep and which to remove, or apply flags to automate deletion while preserving a single instance. Because it operates directly on file content rather than just filenames, fdupes can accurately detect true copies and guide cleaning operations in data cleanup or migration tasks. ... -
3
JobFunnel
Scrape job websites into a single spreadsheet with no duplicates.
Scrape job websites into a single spreadsheet with no duplicates. Automated tool for scraping job postings into a .csv file. You can search for jobs with YAML configuration files or by passing command arguments. By performing regular scraping and reviewing, you can cut through the noise of even the busiest job markets. Run funnel with your settings YAML to populate your master CSV file with jobs from available providers. -
4
AIOStreams
One addon to rule them all
AIOStreams is a “super-add-on” for Stremio that consolidates results from many add-ons and debrid services into a single, customizable feed. Instead of juggling multiple add-ons, you point AIOStreams at them; it then queries, merges, de-duplicates, and re-ranks everything according to your rules. The project includes a powerful filtering and sorting engine—think conditions on resolution, codec, provider, cached status, seed count, and more—so power users can shape results precisely. It also provides a template-based formatter to control how streams are labeled in the UI, complete with live preview for quick iteration. ... -
Go from Code to Production URL in SecondsSkip the Kubernetes configs. Cloud Run handles HTTPS, scaling, and infrastructure automatically. Two million requests free per month.
-
5
Proxifly Free Proxy List
Free HTTP, SOCKS4, & SOCKS5 Proxy List * Updated every 5 minutes
...The repository separates proxy data by protocol and provides the lists in multiple file formats. Its main value is freshness, since the list is updated every few minutes and removes duplicates. It can be useful for testing network tools, validating proxy support, research, or controlled development environments. Because free proxies can be unreliable or unsafe, users should avoid sending sensitive data through them and should follow applicable laws and service rules. -
6
Kuroba Experimental
Free and open source image board browser
...Ability to attach multiple media files to reply, attach media files that were shared by external apps (even by some keyboards), attach remote media files by URL, etc. New image downloader. Allows downloading images while the app is in the background, retrying failed to download images, resolving duplicates, etc. -
7
janitor
Simple tools for data cleaning in R
janitor provides simple, convenient tools for data cleaning, formatting, and exploration in R. It is especially useful for cleaning messy data frames, removing duplicates, formatting column names, and producing frequency tables in a tidy workflow. -
8
trackerslist
Updated list of public BitTorrent trackers
trackerslist is a repository that provides continuously updated lists of public BitTorrent trackers, designed to improve torrent download performance and peer discovery. The project is maintained by an automated system that regularly checks tracker availability, removes duplicates, and ranks them based on reliability and latency. It offers multiple formats of tracker lists, including HTTP, HTTPS, WebSocket, and IP-based versions, making it compatible with a wide range of BitTorrent clients. The lists can be integrated into torrent files or magnet links to enhance connectivity and download speeds. Additionally, the project includes specialized lists for alternative networks such as I2P and Yggdrasil, expanding its usability in privacy-focused environments. ... -
9
Aves
Aves is a gallery and metadata explorer app, built for Android
Aves is a feature-rich gallery app focused on organizing, browsing, and viewing photos and videos with precision. It indexes media on device storage and surfaces powerful ways to explore content, such as by folders, albums, dates, tags, and embedded metadata. The viewer prioritizes smooth gestures (pinch-to-zoom, pan, fling) and a clean, immersive interface that keeps controls out of the way until needed. Aves emphasizes metadata awareness, reading details like EXIF and other tags to enable... -
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
10
PostgREST
REST API for any Postgres database
...The structural constraints and permissions in the database determine the API endpoints and operations. Using PostgREST is an alternative to manual CRUD programming. Custom API servers suffer problems. Writing business logic often duplicates, ignores or hobbles database structure. Object-relational mapping is a leaky abstraction leading to slow imperative code. The PostgREST philosophy establishes a single declarative source of truth: the data itself. It’s easier to ask PostgreSQL to join data for you and let its query planner figure out the details than to loop through rows yourself. ... -
11
PhotoPrism
AI-Powered Photos App for the Decentralized Web 🌈💎✨
PhotoPrism® is an AI-Powered Photos App for the Decentralized Web. It makes use of the latest technologies to tag and find pictures automatically without getting in your way. You can run it at home, on a private server, or in the cloud. Our mission is to provide the most user- and privacy-friendly solution to keep your pictures organized and accessible. That's why PhotoPrism was built from the ground up to run wherever you need it, without compromising freedom, privacy, or functionality. -
12
SortPhotos
SortPhotos is a Python script that organizes photos and videos
...SortPhotos includes options for copying versus moving files, recursive searches, silent or test modes, and customizable start times for when a “day” begins. It also prevents duplicate files by comparing content, with an option to keep duplicates if needed. With support for automation through launch agents or cron jobs, SortPhotos is well-suited for photographers, archivists, and anyone looking to streamline large personal or professional media collections. -
13
websocket for Go
Minimal and idiomatic WebSocket library for Go
...RFC 7692 permessage-deflate compression. Compile to Wasm. Transparent message buffer reuse with wsjson and wspb subpackages. Gorilla writes directly to a net.Conn and so duplicates features of net/http.Client. Gorilla requires registering a pong callback before sending a Ping. Compare godoc of nhooyr.io/websocket with gorilla/websocket side by side. Will enable easy HTTP/2 support in the future. -
14
fastdup
An unsupervised and free tool for image and video dataset analysis
fastdup is a powerful free tool designed to rapidly extract valuable insights from your image & video datasets. Assisting you to increase your dataset images & labels quality and reduce your data operations costs at an unparalleled scale. -
15
MusicBrainz Server
Server for the MusicBrainz project (website, API, database tools)
MusicBrainz Server is the web application powering the MusicBrainz open music metadata database. It handles artist, album, track, and recording entities and allows users to submit edits, vote, merge duplicates, and browse relationships like releases or linked ISRCs. The server supports full-text search, advanced filtering, web APIs, data dumps, and integration with other services like Cover Art Archive and Discogs. Its architecture ensures referential integrity and versioned change tracking so edits can be audited or reverted, and moderation workflows manage conflicting contributions. ... -
16
ComfyUI Essentials
Essential nodes that are weirdly missing from ComfyUI core
ComfyUI_essentials is a ComfyUI custom node collection that adds practical nodes the author considers missing from the ComfyUI core. The project focuses on useful workflow building blocks rather than generic duplicates, with nodes for image handling, mask processing, sampling, segmentation, conditioning, text, and miscellaneous operations. Its image tools include functions for batching, cropping, flipping, resizing, compositing, background removal, color matching, LUT application, sharpening, tiling, and latent previewing. Its mask tools include blur, smoothing, fixing, flipping, color-based masks, segmentation masks, bounding boxes, transition masks, and batch utilities. ... -
17
EPLB
Expert Parallelism Load Balancer
...In EP, different “experts” are mapped to different GPUs or nodes, so load imbalance becomes a performance bottleneck if certain experts are invoked much more often. EPLB solves this by duplicating heavily used experts (redundancy) and then placing those duplicates across GPUs to even out computational load. It uses policies like hierarchical load balancing (grouped experts placed at node and then GPU level) and global load balancing depending on configuration. The logic is implemented in eplb.py and supports predicting placements given estimated expert usage weights. EPLB aims to reduce hot-spotting and ensure more uniform usage of compute resources in large MoE deployments. -
18
gitignore.io
Create useful .gitignore files for your project
...You can access it from a clean web UI, a simple REST API, or the command line, making it easy to script into new-project scaffolds and automation. The generator accepts multiple technologies in one request, normalizes duplicates, and orders rules sensibly so the result is readable and effective. Templates are versioned and updated over time as tools evolve, helping teams avoid accidentally committing build artifacts, credentials, caches, and other noisy files. The repository includes documentation, example invocations, and contribution guidelines so users can add or refine templates. -
19
Skills Janitor
Audit, track usage, and compare your Claude Code skills
The Skills Janitor project is a lightweight plugin designed to manage, audit, and optimize AI agent skill ecosystems, particularly for environments like Claude Code and OpenAI Codex. It functions as a “maintenance layer” for AI skills by automatically scanning installed skill directories, identifying duplicates, and analyzing their structure and usage. One of its core purposes is to help developers maintain a clean and efficient skill environment, especially as the number of installed skills grows over time. The system provides a set of command-based tools that allow users to perform health checks, generate reports, and automatically fix issues such as broken or redundant skills. ... -
20
Node Modules Inspector
Interactive UI for local node modules inspection
This is a tool (CLI + interactive UI) for inspecting the node_modules directory of a JavaScript/TypeScript project, created by Anthony Fu. It supports projects using npm, pnpm or bun. The idea is to help developers visualise the dependency graph, see which dependencies are installed, their sizes, types (ESM vs CJS), origins (catalog vs registry), and filter or build a static report of a project’s dependency tree. The project includes a web UI version that you can try at node-modules.dev, and... -
21
Pandas Profiling
Create HTML profiling reports from pandas DataFrame objects
...File sizes, creation dates, dimensions, indication of truncated images and existance of EXIF metadata. Mostly global details about the dataset (number of records, number of variables, overall missigness and duplicates, memory footprint). Comprehensive and automatic list of potential data quality issues (high correlation, skewness, uniformity, zeros, missing values, constant values, between others). -
22
CleanVision
Automatically find issues in image datasets
CleanVision automatically detects potential issues in image datasets like images that are: blurry, under/over-exposed, (near) duplicates, etc. This data-centric AI package is a quick first step for any computer vision project to find problems in the dataset, which you want to address before applying machine learning. CleanVision is super simple -- run the same couple lines of Python code to audit any image dataset! The quality of machine learning models hinges on the quality of the data used to train them, but it is hard to manually identify all of the low-quality data in a big dataset. ... -
23
anti-copy-paster
A plugin for IntelliJ IDEA for just-in-time code duplicates extraction
The plugin monitors the copying and pasting that takes place inside the IDE. As soon as a code fragment is pasted, the plugin checks if it introduces code duplication, and if it does, the plugin calculates a set of code metrics for it, and these metrics are compared against the currently selected metrics thresholds. If the chosen thresholds are surpassed, the plugin suggests the developer to perform the Extract Method refactoring and applies the refactoring if necessary. -
24
WizTree
WizTree quickly analyzes disk space, showing large files and folders.
WizTree is an ultra-fast and intuitive disk space analyzer that helps you instantly see what’s taking up space on your hard drives. Using advanced technology, it reads the Master File Table (MFT) directly on NTFS drives, making it dramatically faster than traditional scanners. The program displays your files and folders in a clear, visual tree-map view, showing their relative sizes so you can quickly locate and delete large or unnecessary items. -
25
RemoveDuplicate
A software that can helps you remove duplicate files
RemoveDuplicate is a Windows application designed to help users find and remove duplicate files within folders. It uses MD5 hashing to identify identical files and provides a user-friendly interface to manage and delete duplicates while keeping one copy of each file.
