Search Results for "compression"
Sort By:
Showing 48 open source projects for "compression"
View related business solutions-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
rust-rocksdb
Rust wrapper for rocksdb
Binding is statically linked with a specific version of RocksDB. If you want to build it yourself, make sure you've also cloned the RocksDB and compression submodules. By default, support for the Snappy, LZ4, Zstd, Zlib, and Bzip2 compression is enabled through crate features. If support for all of these compression algorithms is not needed, default features can be disabled and specific compression algorithms can be enabled. The underlying RocksDB does allow column families to be created and dropped from multiple threads concurrently. ... -
2
WAL-G
Archival and Restoration for databases in the Cloud
...To configure the compression method used for backups. Possible options are lz4, lzma, zstd, brotli. The default method is lz4. LZ4 is the fastest method, but the compression ratio is bad. LZMA is way much slower. However, it compresses backups about 6 times better than LZ4. Brotli and zstd are a good trade-off between speed and compression ratio, which is about 3 times better than LZ4. -
3
Reliable PostgreSQL Backup & Restore
Reliable PostgreSQL backup and restore
pgBackRest aims to be a reliable, easy-to-use backup and restore solution that can seamlessly scale up to the largest databases and workloads by utilizing algorithms that are optimized for database-specific requirements. Compression is usually the bottleneck during backup operations but, even with now-ubiquitous multi-core servers, most database backup solutions are still single-process. pgBackRest solves the compression bottleneck with parallel processing. Utilizing multiple cores for compression makes it possible to achieve 1TB/hr raw throughput even on a 1Gb/s link. ... -
4
pg_probackup
Backup and recovery manager for PostgreSQL
pg_probackup is an advanced backup and recovery tool for PostgreSQL developed by PostgresPro. It supports full, incremental, and differential backups with compression and encryption, and provides fine-grained control over backup and restore processes. It’s designed for enterprise-grade backup strategies in PostgreSQL environments. -
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
5
HDF5
Official HDF5® Library Repository
HDF5 (Hierarchical Data Format v5) is a widely-used data management library and file format for storing large and complex scientific data sets efficiently. -
6
ClickHouse Go
Golang driver for ClickHouse
...Bulk write support (for database/sql use begin->prepare->(in loop exec)->commit). AsyncInsert, named and numeric placeholders support. LZ4/ZSTD compression support. External data. -
7
OrioleDB
Building a modern cloud-native storage engine
OrioleDB is a next-generation storage engine for PostgreSQL, aiming to enhance scalability, performance, and modern data management. Designed as a table access method (TAM) extension, it introduces features like MVCC improvements, compression, and efficient concurrency handling. OrioleDB seeks to future-proof PostgreSQL by enabling cloud-native, high-throughput workloads without altering the core PostgreSQL behavior. -
8
TimescaleDB
An open-source time-series SQL database optimized for fast ingest
...Horizontally scale to petabytes. Don’t worry about cardinality. Simplify your stack, ask more complex questions, and build more powerful applications. Don’t break the bank. 94-97% compression rates from best-in-class algorithms. Downsample + retention policies. Decouple compute and storage. You’re in trusted hands. Fully-managed TimescaleDB on AWS, Azure, or GCP in just one click. Top-rated support and ops available 24/7 worldwide. A modern, cloud-native relational database platform for time-series data based on TimescaleDB and PostgreSQL. -
9
DB Dumper
Dump the contents of a database
DB Dumper is a simple PHP library for creating database dumps of MySQL, PostgreSQL, and SQLite databases. It provides a fluent, expressive API that lets developers configure backup paths, credentials, and additional options. DB Dumper is ideal for integrating automated backups into Laravel or other PHP-based applications. -
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
10
GreptimeDB
An open-source, cloud-native, unified time series database for metrics
GreptimeDB treats all time series as contextual events with timestamps, and thus unifies the processing of metrics, logs, and events. It supports analyzing metrics, logs, and events with SQL, PromQL, and streaming with continuous aggregation. GreptimeDB is a time-series database optimized for storing and querying large amounts of time-series data, commonly used in monitoring and IoT applications. -
11
IOWOW
A C utility library and persistent key/value storage engine
IOWOW is a lightweight, high-performance key-value store and data storage library written in C, designed for use in embedded systems and high-speed applications that require efficient file-based storage. -
12
ClickHouse
A fast open-source OLAP database management system
ClickHouse® is a fast, open-source column-oriented database management system that can generate analytical data reports through SQL queries in real time. According to several independent benchmarks, it far exceeds other comparable column-oriented database management systems, working even up to 1000 times faster. It is able to process hundreds of millions to more than a billion rows and tens of gigabytes of data per single server per second. Apart from its blazing speed, ClickHouse is highly... -
13
ArcticDB
ArcticDB is a high performance, serverless DataFrame database
...Giving you the keys to protect your most valuable asset. Supports large concurrent writes to many tables ensuring datasets can be onboarded fast and in the most convenient format. Scale-out architecture and built-in compression ensure that networks and storage are utilized efficiently. Support for DataFrames with thousands of columns enables large trading universes. -
14
TDengine
Open-source time-series database with high-performance and scalability
...TDengine can be widely applied to IoT, Industrial Internet, Connected Vehicles, DevOps, Energy , Finance and many other use-cases. TDengine’s innovative design and purpose-built storage engine outperforms other time-series databases for data ingestion, querying and data compression while significantly reducing storage and computing costs. TDengine’s native distributed architecture powers out-of-the-box scalability and high availability. Nodes can be added through simple configuration to achieve greater data processing power. In addition, this feature is open source. TDengine uses SQL as the query language, thereby reducing learning and migration costs, while adding SQL extensions to handle time-series data better, and supporting convenient and flexible schemaless data ingestion. -
15
Carmine
Redis client and message queue for Clojure
...Flexible, high-performance binary-safe serialization using Nippy. Simple, high-performance message queue (v2+, Redis 2.6+). Simple, high-performance distributed lock (v2+, Redis 2.6+). Pluggable compression and encryption support (v2+). Includes Tundra, an API for replicating data to an additional datastore (v2+, Redis 2.6+). You'll usually want to define a single connection pool, and one connection spec for each of your Redis servers. Note that executing multiple Redis commands in a single wcar request uses efficient Redis pipelining under the hood, and returns a pipeline reply (vector) for easy destructuring, etc. -
16
LinDB
LinDB is a scalable, high performance, high availability database
LinDB is a scalable, high-performance, high-availability distributed time series database. A single server could easily support more than one million write TPS; With fundamental techniques like efficient compression storage and parallel computing, LinDB delivers highly optimized query performance. The multi-channel replication protocol supports any amount of nodes, and ensures the system's availability. Schema-free multi-dimensional data model with Metric, Tags, and Fields; The LinQL is flexible yet handy for real-time data analytics. Horizontal scalable is made simple by adding more new broker and storage nodes without too much thinking and manual operations. ... -
17
IoTDB
Apache IoTDB
Apache IoTDB (Database for Internet of Things) is an IoT native database with high performance for data management and analysis, deployable on the edge and the cloud. Due to its light-weight architecture, high performance and rich feature set together with its deep integration with Apache Hadoop, Spark and Flink, Apache IoTDB can meet the requirements of massive data storage, high-speed data ingestion and complex data analysis in the IoT industrial fields. In the scene of factories, there... -
18
Barman for PostgreSQL
Backup and Recovery Manager for PostgreSQL
...It allows to perform remote backups of multiple servers in business critical environments and help DBAs during the recovery phase. Barman's most wanted features include backup catalogs, retention policies, remote recovery, archiving and compression of WAL files and backups. Barman is written and maintained by PostgreSQL professionals 2ndQuadrant. -
19
SymmetricDS
Database Replication, Filtered Synchronization, and Transformation
SymmetricDS is database replication and file synchronization software. It is a platform-independent, web-enabled, and database-agnostic synchronization tool. With initial data load, bidirectional change data capture, conflict resolution, and data transformation, it is a comprehensive data replication solution. -
20
LocustDB
Massively parallel, high performance analytics database
An experimental analytics database aiming to set a new standard for query performance and storage efficiency on commodity hardware. See How to Analyze Billions of Records per Second on a Single Desktop PC and How to Read 100s of Millions of Records per Second from a Single Disk for an overview of current capabilities. Download the latest binary release, which can be run from the command line on most x64 Linux systems, including Windows Subsystem for Linux. When loading .csv or .csv.gz files... -
21
Hydra Columnar
Postgres-native columnar storage extension
Hydra Columnar is an open-source columnar storage extension for PostgreSQL designed to deliver analytics performance on par with modern data warehouses. It integrates seamlessly with the PostgreSQL ecosystem, allowing users to benefit from columnar compression, vectorized execution, and late materialization without leaving their existing database setup. Hydra enables hybrid row-column storage, making it ideal for OLAP workloads on Postgres. -
22
Arctic TimeSeries and Tick store
High performance datastore for time series and tick data
...Serializes a number of data types eg. Pandas DataFrames, Numpy arrays, Python objects via pickling etc. so you don't have to handle different datatypes manually. Uses LZ4 compression by default on the client side to get big savings on network / disk. Allows you to version different stages of an object and snapshot the state (In some ways similar to git), and allows you to freely experiment and then just revert back the snapshot. [VersionStore only] Does the chunking (breaking a Dataframe to smaller part for you. ... -
23
CursusDB
CursusDB is an open-source distributed in-memory database
CursusDB is a time-series database built for high-performance analytics and data processing, optimized for handling large volumes of sequential data efficiently. -
24
Cetus
Cetus is a high performance middleware that provides routing
Cetus is a high-performance middleware that provides transparent routing between your application and any backend MySQL Servers. Cetus is the middleware of relational database MySQL developed by C language, which mainly provides a comprehensive database access proxy function. The connection mode of Cetus is basically compatible with MySQL, and the application program can access the database through Cetus without modification, which realizes the horizontal expansion and high availability of... -
25
Sqlite Index Blaster
Create huge Sqlite indexes at breakneck speeds
SQLite Blaster is an advanced SQLite extension that enhances database performance by enabling multi-threading, data compression, and memory optimizations. It is designed for applications that require fast local storage with improved query efficiency.
