Search Results for "high performance computing"

...The project ports many capabilities of the Hugging Face Transformers ecosystem into the Rust programming language. It allows developers to run state-of-the-art NLP models like BERT, GPT-2, and DistilBERT directly within Rust applications while maintaining high performance and memory efficiency. The library integrates with Rust machine learning infrastructure using crates such as tch-rs and ONNX Runtime for model execution. It also includes tokenization utilities, model architectures, and task-specific pipelines that simplify the development of NLP applications. Because Rust is known for its safety and performance, this project enables developers to deploy modern NLP models in production systems written in Rust.

Lance is a columnar data format that is easy and fast to version, query and train on. It’s designed to be used with images, videos, 3D point clouds, audio and of course tabular data. It supports any POSIX file systems, and cloud storage like AWS S3 and Google Cloud Storage.

...You can co-locate a managed dataset with your application or machine learning model, and accelerate it with Arrow in-memory, SQLite/DuckDB, or with attached PostgreSQL for fast, high-concurrency, low-latency queries. Accelerated engines give you flexibility and control over query cost and performance.

hora is an open-source high-performance vector similarity search library designed for large-scale machine learning and information retrieval systems. The project focuses on approximate nearest neighbor search, a fundamental technique used in modern AI applications such as recommendation systems, image search, and semantic search engines. Hora implements multiple efficient indexing algorithms that allow systems to rapidly search through high-dimensional vectors produced by machine learning models. ...

Weld is a programming language and runtime designed to improve the performance of data-intensive applications by optimizing computations across multiple libraries. Instead of optimizing individual functions independently, Weld introduces an intermediate representation that allows different frameworks to share optimization opportunities. This approach reduces data movement between libraries and enables the system to generate highly optimized machine code for parallel execution. Weld is...