Search Results for "distributed storage"
Sort By:
Showing 29 open source projects for "distributed storage"
View related business solutions-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
Anna’s Archive
Comprehensive search engine for books, papers, comics, magazines
...The project includes all the infrastructure required to run a full instance locally or in production, combining web servers, databases, and search indexing systems into a scalable architecture. It relies heavily on technologies such as Elasticsearch for search functionality and MariaDB for structured data storage, enabling fast and efficient querying across massive datasets. The system is designed with redundancy and replication in mind, allowing distributed deployments and mirrored environments to handle high traffic and large data volumes. It also includes tooling for importing datasets, managing metadata, and maintaining structured archives using custom formats. -
2
diskover-community
Open source file indexing & storage analytics powered by Elasticsearch
Diskover Community Edition is an open source file system indexing and storage analytics platform designed to help organizations understand and manage large volumes of file data. It crawls file systems and indexes metadata using Elasticsearch, enabling fast search, analysis, and organization of files stored across different storage systems. It allows administrators and users to explore file structures, monitor storage usage, and gain insights into how data is distributed across infrastructure. ... -
3
huey
A little task queue for python
huey is a lightweight task queue for Python applications. It gives developers a clean API for running background jobs outside the main request or execution flow. The project supports several storage backends, including Redis, Valkey, Redict, SQLite, the file system, and in-memory storage. It can execute tasks with processes, threads, or greenlets, which makes it adaptable to different workloads. Huey also supports scheduled tasks, recurring tasks, retries, task priorities, result storage, expiration, locking, rate limits, timeouts, pipelines, groups, and chords. ... -
4
Databend
Cloud-native open source data warehouse for analytics and AI queries
...Built in Rust, the system focuses on high performance, scalability, and efficient data processing for analytical queries. It is designed with a separation of compute and storage, allowing compute nodes to scale independently while storing data in object storage systems. This architecture enables cost-efficient storage and elastic scaling for workloads that involve large datasets and complex queries. Databend provides a unified engine capable of handling analytics, vector search, and full-text search within a single platform. ... -
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
5
Smallpond
A lightweight data processing framework built on DuckDB and 3FS
smallpond is a lightweight distributed data processing framework built by DeepSeek, designed to scale DuckDB workloads over clusters using their 3FS (Fire-Flyer File System) backend. The idea is to preserve DuckDB’s fast analytics engine but lift it from single-node to multi-node settings, giving you the ability to operate on large datasets (e.g. petabyte scale) without moving to a heavyweight system like Spark. -
6
Flan Scan
A pretty sweet vulnerability scanner
Flan Scan is a lightweight open-source network vulnerability scanner designed to make it easy to detect exposed services, open ports, and associated vulnerabilities across IP ranges or network segments as part of security audit and compliance workflows. It is essentially a thin wrapper around the widely-used Nmap scanner, augmenting it with scripts and tooling that transform raw Nmap output into vulnerability-focused reports that map detected services to known CVEs, making results more... -
7
ddia
Designing Data-Intensive Application
ddia is a Chinese translation and online reading project for “Designing Data-Intensive Applications.” It covers both the first edition and ongoing second-edition translation work. The repository organizes the book content into chapters about data systems, storage, encoding, replication, partitioning, transactions, distributed systems, consistency, and derived data. It is built for readers who want deep systems design material in Chinese with a structured web reading experience. The project also includes site assets, metadata, layouts, internationalization files, and build configuration for generating the online version. ... -
8
learn_python3_spider
Python crawler tutorial series, learning Python crawlers from 0 to 1
learn_python3_spider is a tutorial and example-code collection for learning web scraping with Python 3 from beginner to advanced topics. It introduces request capture, browser and mobile traffic inspection, and core scraping libraries such as Requests, BeautifulSoup, Selenium, Appium, and Scrapy. Lessons also cover JSON parsing, regular expressions, concurrency, proxy usage, login handling, and CAPTCHA-related techniques. Data can be stored with CSV, MySQL, or MongoDB examples. More advanced... -
9
migrid
A grid middleware with minimal user and resource requirements
[This project moved to Github and is no longer maintained here] Minimum intrusion Grid (MiG) is an attempt to design a new platform for Grid computing which is driven by a stand-alone approach to Grid, rather than integration with existing systems. The goal of the MiG project is to provide Grid infrastructure where the requirements on users and resources alike is as small as possible (minimum intrusion). MiG strives for minimum intrusion but will seek to provide a feature rich and... -
Build Securely on AWS with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
10
Datahosting ipfs kubo-cluster
Managed IPFS Kubo pinning with IPFS Cluster replication
...Retention options include short-term and long-term storage, making it suitable for websites, datasets, backups, and distributed applications. For higher availability, IPFS Cluster replication is available with 1, 2, or 3 replicas, enabling redundancy across nodes and improved resilience against node failure. Bandwidth usage is tracked separately, with included allowances and predictable overage pricing.https://github.com/branislav1989/ipfs-kubo-private-public-ipfs-cluste -
11
CostPal
CostPal is your personal finance manager
...It's purpose is to act as a scratchpad, not a standalone application for managing personal finance - for that you have CostPal. Currently only Dropbox is supported as a cloud storage used for data exchange. -
12
BerryNet
Deep learning gateway on Raspberry Pi and other edge devices
This project turns edge devices such as Raspberry Pi into an intelligent gateway with deep learning running on it. No internet connection is required, everything is done locally on the edge device itself. Further, multiple edge devices can create a distributed AIoT network. At DT42, we believe that bringing deep learning to edge devices is the trend towards the future. It not only saves costs of data transmission and storage but also makes devices able to respond according to the events shown in the images or videos without connecting to the cloud. One of the applications of this intelligent gateway is to use the camera to monitor the place you care about. ... -
13
PerfKit Benchmarker
PerfKit Benchmarker (PKB) contains a set of benchmarks
...It simplifies the process of running complex benchmarks by providing unified command-line workflows that handle resource provisioning, execution, and result collection. The framework includes a comprehensive set of predefined benchmarks covering areas such as compute, storage, networking, and distributed systems workloads. It is widely used by researchers, engineers, and organizations to evaluate cloud architectures and make informed infrastructure decisions. -
14
ECDICT
Free English to Chinese Dictionary Database
ECDICT is a comprehensive English–Chinese dictionary dataset packaged for developers who need an offline, queryable lexicon for applications, NLP, or educational tools. It aggregates headwords, phonetics, parts of speech, translations, and example information into formats that are easy to integrate. The project provides multiple distribution forms—commonly SQLite/CSV/StarDict-style files—so you can choose the right storage and query approach for your app. Because it’s offline and local, it... -
15
haipproxy
Distributed proxy IP pool for web crawlers using Scrapy and Redis
HAipproxy is a distributed proxy IP pool system designed to collect, manage, and provide large numbers of proxy addresses for web crawling tasks. It automatically crawls proxy resources from the internet and aggregates them into a centralized pool that can be accessed by distributed spiders and scraping systems. It is built using Python and relies on Scrapy for high-performance crawling while Redis is used for data storage, communication, and task coordination between components. ... -
16
Hierarchical cluster engine HCE
Hierarchical Cluster Engine project
The main idea of this project – to implement the solution that can be used to: construct custom network mesh or distributed network cluster structure with several relations types between nodes, formalize the data flow processing goes from upper node level central source point to down nodes and backward, formalize the management requests handling from multiple source points, support native reducing of multiple nodes results (aggregation, duplicates elimination, sorting and so on), internally support powerful full-text search engine and data storage, provide transactions-less and transactional requests processing, support flexible run-time changes of cluster infrastructure, have many languages bindings for client-side integration APIs in one product build on C++ language... ... -
17
RedMQ
Simple, flexible messaging with Redis.
RedMQ is a Redis-based messaging system that is specifically designed to support flexible messaging schemes, exposing an API that allows messages to be filtered by a number of message attributes. At its core is a Python TCP server built on Asynchronous Messaging Protocol and Twisted's AMP module. This means RedMQ can support clients in any language, though only two clients - PHP and Python - have been implemented to date. The API is designed with flexibility in mind, providing native support... -
18
Kadna is a small curses program written in Python that creates a secure and synchronizable database. This is intended for distributed password and small file (e.g. cryptographic keys) storage. This project in not longer maintained.
-
19
This is attempt to make an open source bookmarking system that supports tagging, distributed data storage, genetic "splicing" of strains of bookmarking tags and much more!
-
20
-
21
Universal information crawler is a fast precise and reliable Internet crawler. Uicrawler is a program/automated script which browses the World Wide Web in a methodical, automated manner and creates the index of documents that it accesses.
-
22
iROS is a meta-operating system for technology-rich "interactive rooms". The core components (Event Heap, DataHeap, iCrafter) provide communication, data storage, and service management for an iRoom.
-
23
-
24
rpmdiff is a utility to create a binary patch from one version of an rpm to another. This binary patch can then be distributed to end-users of a distribution who have installed the first version of the rpm and they will be upgraded to the second version.
-
25
Sfooi - distributed serverless p2p/file storage operating over protocol similiar (but not identicall) to airhook, using DHT as routing medium.
