Vortex

Vortex is a high-performance toolkit designed for working with compressed Apache Arrow arrays, providing functionality for in-memory, on-disk, and over-the-wire data handling. It aims to be an advanced successor to Apache Parquet, offering dramatically faster random access reads and scans, while maintaining similar compression ratios. Vortex's modular design allows for extensibility, enabling developers to implement custom encodings for efficient data management, particularly for large-scale columnar datasets.

Features

Zero-copy integration with Arrow: Enables efficient conversion between Vortex and Apache Arrow arrays
Extensible encoding: Supports a range of highly data-parallel encodings, including FastLanes and FSST
Cascading compression: Allows data to be recursively compressed with nested encodings for optimized storage
Pluggable compression strategies: Default strategy uses BtrBlocks but can be replaced with custom approaches
Random access performance: Delivers 100-200x faster random access reads compared to traditional formats like Parquet
Statistics and compute: Includes compute kernels and lazily computed statistics for enhanced query performance

Project Samples

Project Activity

See All Activity >

License

Apache License V2.0

Follow Vortex

Vortex Web Site

Other Useful Business Software

Stop Storing Third-Party Tokens in Your Database

Auth0 Token Vault handles secure token storage, exchange, and refresh for external providers so you don't have to build it yourself.

Rolling your own OAuth token storage can be a security liability. Token Vault securely stores access and refresh tokens from federated providers and handles exchange and renewal automatically. Connected accounts, refresh exchange, and privileged worker flows included.

Try Auth0 for Free

Rate This Project