Search Results for "big data" - Page 3
Sort By:
Showing 224 open source projects for "big data"
View related business solutions-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
1
Apache Spark
A unified analytics engine for large-scale data processing
...With Spark Streaming (microbatches) and Structured Streaming, it delivers low-latency event processing suitable for real-time analytics. The built-in MLlib library provides scalable machine learning algorithms, while GraphX enables graph computations integrated with data pipelines. Spark supports multiple languages—Scala, Java, Python, R—and connects with many storage systems like HDFS, S3, Cassandra, and streaming platforms like Kafka, making it a versatile choice for big data workloads in analytics, ETL, and data science. -
2
Bacalhau
Community-driven, simple, yet powerful framework
Bacalhau is a decentralized compute platform for running jobs on data stored across distributed networks, like IPFS or Filecoin, without moving the data to centralized cloud environments. It allows developers to run containerized workloads close to where the data lives, reducing latency, cost, and privacy risks. Bacalhau supports various runtime environments and is designed to make decentralized data processing as accessible as traditional cloud computing. It’s especially useful for... -
3
StartOS
Linux server OS optimized for self-hosting
StartOS is a sovereign, self-hosted operating system built by Start9 Labs to empower individuals with digital independence. Designed to run on personal servers, it provides a privacy-first interface for installing, managing, and running decentralized applications without needing technical expertise. StartOS includes services like Bitcoin nodes, messaging platforms, file hosting, and password managers, all running locally and without third-party control. With a user-friendly UI and strong... -
4
PHP7
PHP7 / Laravel Multi-format Streaming Parser
When it comes to parsing XML/CSV/JSON/... documents, there are 2 approaches to consider. DOM loading loads all the documents, making it easy to navigate and parse, and as such provides maximum flexibility for developers. Streaming implies iterating through the document, acts like a cursor, and stops at each element in its way, thus avoiding memory overkill. Thus, when it comes to big files, callbacks will be executed meanwhile file is downloading and will be much more efficient as far as... -
Demo Series - Small Business Backup By VeeamWatch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.
-
5
Coding Interview University
A complete computer science study plan to become a software engineer
Coding Interview University is a multi-month self-study plan that grew from a short personal to-do list into a comprehensive roadmap for preparing technical interviews at large software companies. The author used this plan to transition into a Software Development Engineer role at Amazon and emphasizes that most learners will not need to study as many hours to succeed. The repository focuses on practical readiness over memorization, guiding you to learn core computer science topics to about... -
6
HASH
The best way to use and work with blocks
...You can read more about our big-picture vision at hash.dev -
7
Kinto
A generic JSON document store with sharing and synchronisation options
...Kinto is used at Mozilla and released under the Apache v2 license. It’s hard for frontend developers to respect users' privacy when building applications that work offline, store data remotely and synchronize across devices. Existing solutions either rely on big corporations that crave user data or require a non-trivial amount of time and expertise to set up a new server for every new project. We want to help developers focus on the front, and we don’t want the challenge of storing user data to get in their way. ... -
8
manticoresearch
Easy to use open source fast database for search
...Modern MPP architecture and smart query parallelization capabilities allow to fully utilize all your CPU cores to lower response time as much as possible, when needed. Powerful and fast full-text searching which works fine for small and big datasets. Columnar storage support via the Manticore Columnar Library for bigger datasets (much bigger than can fit in RAM). SQL-first: Manticore's native syntax is SQL. It speaks SQL over HTTP and uses the MySQL protocol (you can use your preferred MySQL client). JSON over HTTP: to provide a more programmatic way to manage your data and schemas, Manticore provides a HTTP JSON protocol. ... -
9
Volcano
A Cloud Native Batch System (Project under CNCF)
...It provides a suite of mechanisms that are commonly required by many classes of batch & elastic workload including machine learning/deep learning, bioinformatics/genomics, and other "big data" applications. These types of applications typically run on generalized domain frameworks like TensorFlow, Spark, Ray, PyTorch, MPI, etc, which Volcano integrates with. Volcano builds upon a decade and a half of experience running a wide variety of high-performance workloads at scale using several systems and platforms, combined with best-of-breed ideas and practices from the open-source community. ... -
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
10
CleanVision
Automatically find issues in image datasets
CleanVision automatically detects potential issues in image datasets like images that are: blurry, under/over-exposed, (near) duplicates, etc. This data-centric AI package is a quick first step for any computer vision project to find problems in the dataset, which you want to address before applying machine learning. CleanVision is super simple -- run the same couple lines of Python code to audit any image dataset! The quality of machine learning models hinges on the quality of the data used to train them, but it is hard to manually identify all of the low-quality data in a big dataset. ... -
11
Ptah.sh
Self-hosted alternative to Heroku
Ptah.sh is a Fair Source self-hosting deployment platform - an alternative to Heroku/Vercel and other Big Corp software. We believe that indie, startups, and small to medium businesses must not suffer from unpredicted billing or bare-metal/VPS configurations. Designed for indie developers and SMBs, Ptah.sh offers the simplest hosting experience. -
12
MobX
A Simple, scalable state management
MobX is a battle tested library that makes state management simple and scalable by transparently applying functional reactive programming (TFRP). Write minimalistic, boilerplate free code that captures your intent. Trying to update a record field? Use the good old JavaScript assignment. Updating data in an asynchronous process? No special tools are required, the reactivity system will detect all your changes and propagate them out to where they are being used. All changes to and uses of... -
13
Sail
A drop-in Apache Spark replacement written in Rust
Sail is an open-source distributed computation framework designed to unify batch processing, stream processing, and AI workloads into a single, high-performance engine. It is built entirely in Rust, eliminating JVM overhead and enabling predictable performance, fast startup times, and improved memory safety compared to traditional big data frameworks. Sail is compatible with the Spark Connect protocol, which means existing Spark SQL and DataFrame workloads can run without code changes, making adoption seamless for teams already using Spark-based pipelines. The framework is designed to operate across a variety of environments, including local machines, Kubernetes clusters, and cloud deployments, allowing flexible scaling based on workload requirements. ... -
14
Etaoin
Pure Clojure Webdriver protocol implementation
A pure Clojure implementation of the Webdriver protocol, named after Etaoin Shrdlu, a typing machine that came to life after a mysterious note was produced on it. Use the Etaoin library to automate a browser, test your frontend behavior, simulate human actions or whatever you want. Currently supports Chrome, Firefox, Safari, and Edge. Can either connect to a remote WebDriver process or have Etaoin launch one for you. Run your unit tests directly from Emacs by pressing C-t t as usual. Can... -
15
AlphaTree
DNN && GAN && NLP && BIG DATA
AlphaTree is an educational repository that provides a visual roadmap of deep learning models and related artificial intelligence technologies. The project focuses on explaining the historical development and relationships between major neural network architectures used in modern machine learning. It presents diagrams and documentation describing the evolution of models such as LeNet, AlexNet, VGG, ResNet, DenseNet, and Inception networks. The repository organizes these architectures into a... -
16
CubeFS
cloud-native file store
CubeFS is a new generation cloud-native storage that supports access protocols such as S3, HDFS, and POSIX. It is widely applicable in various scenarios such as big data, AI/LLMs, container platforms, separation of storage and computing for databases and middleware, data sharing and protection, etc. Compatible with various access protocols such as S3, POSIX, HDFS, etc., and the access between protocols can be interoperable. Support replicas and erasure coding engines, users can choose flexibly according to business scenarios. ... -
17
NeuroMatch Academy (NMA)
NMA Computational Neuroscience course
NMA Computational Neuroscience course. We have curated a curriculum that spans most areas of computational neuroscience (a hard task in an increasingly big field!). We will expose you to both theoretical modeling and more data-driven analyses. The Neuro Video Series is a series of 12 videos that covers basic neuroscience concepts and neuroscience methods. These videos are completely optional and do not need to be watched in a fixed order so you can pick and choose which videos will help you brush up on your knowledge. ... -
18
Angel
A Flexible and Powerful Parameter Server for large-scale ML
Angel is a high-performance distributed machine learning and graph computing platform based on the philosophy of Parameter Server. It is tuned for performance with big data from Tencent and has a wide range of applicability and stability, demonstrating an increasing advantage in handling higher-dimension models. Angel is jointly developed by Tencent and Peking University, taking account of both high availability in industry and innovation in academia. With a model-centered core design concept, Angel partitions the parameters of complex models into multiple parameter-server nodes and implements a variety of machine learning algorithms and graph algorithms using efficient model-updating interfaces and functions, as well as a flexible consistency model for synchronization. ... -
19
Koordinator
A QoS-based scheduling system brings optimal layout and status to work
Koordinator is a modern scheduling system that colocates microservices, AI, and big data workloads on Kubernetes. It achieves high utilization by combining elastic resource quota, efficient pod-packing, over-commitment, and resource sharing with container resource isolation. Koordinator is high-performance, scalable, yet most importantly, proven in mass production environments. It allows you to build container orchestration systems that support enterprise production environments. ... -
20
huihut interview
A summary of C/C++ technical interview basics
...It’s organized to be approachable whether you’re a student preparing for your first internship or an experienced engineer brushing up on fundamentals before a big interview round. -
21
Counter
Web Analytics made simple
...The backend is designed to be resource-light while still providing the essential metrics teams care about, such as page views, referrers, and basic geography. Rather than storing everything in a monolithic database, it combines a fast in-memory store for hot data with archival to a traditional SQL store for long-term retention. The client snippet is tiny and cookieless, which helps reduce layout shift and keeps page performance high. Because it’s a small Go service with a straightforward architecture, it can be self-hosted easily and integrated into existing stacks. The philosophy is to deliver “just enough” analytics for dashboards and reporting without the overhead or tracking footprint of big-box platforms. -
22
zpdf
Zero-copy PDF text extraction library written in Zig
zpdf is a high-performance PDF text extraction library written in Zig that focuses on speed, low overhead, and modern parsing techniques. It leans heavily on memory-mapped file reading and zero-copy patterns where possible, so it can scan large PDFs without repeatedly copying data around in memory. The library supports streaming extraction using efficient arena allocation, making it well suited for workloads that need to process big documents quickly or in batches. It implements multiple PDF decompression filters and handles common font encoding pathways, which are essential for turning raw PDF content streams into readable text. ... -
23
Nano Events
Simple and tiny (107 bytes) event emitter library for JavaScript
Nano Events is a minimalistic, high-performance event emitter library for JavaScript. Its goal is to provide the simplest possible API to add pub/sub capabilities (emitters and listeners) to any JS object or application, while keeping overhead and bundle size extremely small. Rather than offering many complex features, nanoevents focuses on the core primitives: creating an emitter, subscribing to named events, emitting events with arbitrary data, and unsubscribing. Because of its minimal API... -
24
Guide to Technical Interviews
Guided collection and roadmap for preparing technical interviews
This repository is a guided collection and roadmap for preparing technical interviews, covering the gamut from algorithmic challenges and data structures to system design and behavioral preparation. It consolidates resources like interview question lists, practice platforms, mock interview sites, and recommended books or blogs. For individuals targeting big-tech or rigorous interview processes, this acts as a structured study guide rather than a random list of links. The README breaks down preparation into categories — coding problems, system design, mock interview sites — so you can identify gap areas and allocate study time accordingly. ... -
25
Foundatio
Pluggable foundation blocks for building distributed apps
Pluggable foundation blocks for building loosely coupled distributed apps. Includes implementations in Redis, Azure, AWS, RabbitMQ and in memory (for development). When building several big cloud applications we found a lack of great solutions (that's not to say there aren't solutions out there) for many key pieces to building scalable distributed applications while keeping the development experience simple. Wanted to build against abstract interfaces so that we could easily change...
