Search Results for "key value"

R-KV is an open-source research project that focuses on improving the efficiency of large language model inference through key-value cache compression techniques. Modern transformer models rely heavily on KV caches during autoregressive decoding, which store intermediate attention states to accelerate generation. However, these caches can consume large amounts of memory, especially in reasoning-oriented models with long context windows. R-KV introduces a method for compressing the KV cache during decoding, allowing models to maintain reasoning performance while reducing memory consumption and computational overhead. ...

...UCCL is designed to work with heterogeneous hardware environments, allowing GPUs from different vendors and network interfaces to communicate efficiently without vendor lock-in. The system also supports specialized workloads such as reinforcement learning weight transfers, key-value cache sharing, and expert parallelism for mixture-of-experts models. Its architecture emphasizes flexibility and extensibility so that developers can implement custom communication protocols tailored to specific machine learning workloads.

KVCache-Factory is an open-source research framework designed to explore and implement unified key-value cache compression techniques for autoregressive transformer models. In large language models, the key-value cache stores intermediate attention states that enable efficient token generation during inference, but these caches can consume large amounts of GPU memory when handling long contexts. KVCache-Factory provides a platform for implementing and evaluating multiple compression strategies that reduce memory usage while preserving model performance. ...

...Traditional retrieval-augmented generation systems rely on real-time retrieval of documents from databases or vector stores during inference. CAG proposes a different approach by preloading relevant knowledge into the model’s context window and precomputing the model’s key-value cache before queries are processed. This strategy allows the model to generate responses using the cached context directly, eliminating the need for repeated retrieval operations during runtime. As a result, the approach can significantly reduce latency and simplify system architecture compared with traditional RAG pipelines. ...

...HelixDB is built from scratch in Rust and uses LMDB as its storage engine, enabling high performance and low-latency query execution. HelixDB also supports additional data formats such as key-value, document, and relational data, making it flexible for a wide range of backend architectures. A central feature of the project is its custom query language, HelixQL, which is fully type-safe and compiled to ensure reliability and correctness in production environments. HelixDB includes built-in capabilities for embeddings, vector search, keyword search, and graph traversal, which are particularly useful for retrieval-augmented generation and agent-based systems.

...This engine enables efficient movement of tensors and model data across heterogeneous environments such as GPU memory, system memory, and distributed storage systems. Mooncake also introduces distributed key-value cache storage that allows inference systems to reuse previously computed attention states, significantly improving throughput in large-scale deployments. The system supports advanced networking technologies such as RDMA and NVMe over Fabric, enabling high-speed communication across clusters.

...Its value comes from presenting progress in a chronological and thematic way, which makes trends, breakthroughs, and shifts in research focus easier to see.

Perico is a dialogue management library. It allows you to specify a dialogue agent: a computer program that can interact with a user (or other agents) using (natural language) messages.