A lightweight data processing framework built on DuckDB and 3FS
smallpond is a lightweight distributed data processing framework built by DeepSeek, designed to scale DuckDB workloads over clusters using their 3FS (Fire-Flyer File System) backend. The idea is to preserve DuckDB’s fast analytics engine but lift it from single-node to multi-node settings, giving you the ability to operate on large datasets (e.g. petabyte scale) without moving to a heavyweight system like Spark. Users write Python-like code (via DataFrame APIs or SQL strings) to express...
Entangled is a distributed hash table (DHT) and peer-to-peer tuple space, based on Kademlia. It is written in Python, and makes use of the Twisted framework. This can be used as a base for creating peer-to-peer (P2P) network applications.