Search Results for "performance"

Vortex is a high-performance toolkit designed for working with compressed Apache Arrow arrays, providing functionality for in-memory, on-disk, and over-the-wire data handling. It aims to be an advanced successor to Apache Parquet, offering dramatically faster random access reads and scans, while maintaining similar compression ratios. Vortex's modular design allows for extensibility, enabling developers to implement custom encodings for efficient data management, particularly for large-scale columnar datasets.

High-performance xml pull reader/writer. The reader is almost zero-copy (use of Cow whenever possible) It is easy on memory allocation (the API provides a way to reuse buffers) It supports various encoding (with encoding feature), namespaces resolution, and special characters. Syntax is inspired by xml-rs. When using the serialize feature, quick-xml can be used with serde's Serialize/Deserialize traits.

Cube is the semantic layer for building data applications. It helps data engineers and application developers access data from modern data stores, organize it into consistent definitions, and deliver it to every application. Cube was designed to work with all SQL-enabled data sources, including cloud data warehouses like Snowflake or Google BigQuery, query engines like Presto or Amazon Athena, and application databases like Postgres. Cube has a built-in relational caching engine to provide...

...It’s designed to make working with CSV files at scale easy and efficient, offering over 40 powerful subcommands for tasks like querying, sampling, splitting, deduplicating, and more. qsv is ideal for data engineers, analysts, and developers who need high-performance CSV manipulation on the command line.

log4rs is a highly configurable logging framework modeled after Java's Logback and log4j libraries. If you are using the file rotation in your configuration there is a known substantial performance issue so listen up! By default the gzip feature is enabled and when rolling files it will zip log archives automatically. This is a problem when the log archives are large as the zip happens in the main thread and will halt the process while the zip is completed. Be advised that the gzip feature will be removed from default features as of 1.0.

Low Output Latency streaming HTML rewriter/parser with CSS-selector based API. It is designed to modify HTML on the fly with minimal buffering. It can quickly handle very large documents, and operate in environments with limited memory resources. The crate serves as a back-end for the HTML rewriting functionality of Cloudflare Workers, but can be used as a standalone library with a convenient API for a wide variety of HTML rewriting/analysis tasks. The parser switches back to the tag scanner...