Search Results for "decoding"
Sort By:
Showing 52 open source projects for "decoding"
View related business solutions-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
1
R-KV
Redundancy-aware KV Cache Compression for Reasoning Models
R-KV is an open-source research project that focuses on improving the efficiency of large language model inference through key-value cache compression techniques. Modern transformer models rely heavily on KV caches during autoregressive decoding, which store intermediate attention states to accelerate generation. However, these caches can consume large amounts of memory, especially in reasoning-oriented models with long context windows. R-KV introduces a method for compressing the KV cache during decoding, allowing models to maintain reasoning performance while reducing memory consumption and computational overhead. ... -
2
DFlash
Block Diffusion for Ultra-Fast Speculative Decoding
DFlash is an open-source framework for ultra-fast speculative decoding using a lightweight block diffusion model to draft text in parallel with a target large language model, dramatically improving inference speed without sacrificing generation quality. It acts as a “drafter” that proposes likely continuations which the main model then verifies, enabling significant throughput gains compared to traditional autoregressive decoding methods that generate token by token. -
3
FlashMLA
FlashMLA: Efficient Multi-head Latent Attention Kernels
FlashMLA is a high-performance decoding kernel library designed especially for Multi-Head Latent Attention (MLA) workloads, targeting NVIDIA Hopper GPU architectures. It provides optimized kernels for MLA decoding, including support for variable-length sequences, helping reduce latency and increase throughput in model inference systems using that attention style. The library supports both BF16 and FP16 data types, and includes a paged KV cache implementation with a block size of 64 to efficiently manage memory during decoding. -
4
vLLM
A high-throughput and memory-efficient inference and serving engine
vLLM is a fast and easy-to-use library for LLM inference and serving. High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more. -
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
5
Lucebox
Fast LLM speculative inference server for consumer hardware
Lucebox is a local LLM inference server built for fast generation on consumer hardware. It focuses on custom kernels, speculative prefill, speculative decoding, and model-specific optimizations rather than a generic one-size-fits-all runtime. The project includes a native C++ HTTP server with an OpenAI-compatible API, making it usable with tools that already speak the Chat Completions format. It supports CUDA and ROCm workflows, with Docker images for NVIDIA and AMD GPU setups. The repository also includes harnesses for testing compatibility with clients such as Claude Code, Codex, OpenCode, Hermes, Pi, OpenClaw, and Open WebUI. ... -
6
MiniCPM4.1
Achieving 3+ generation speedup on reasoning tasks
MiniCPM4.1 is an enhanced iteration of the MiniCPM4 architecture, introducing improvements in reasoning capabilities, inference speed, and hybrid operation modes that allow dynamic switching between deep reasoning and standard generation. It builds upon the same efficiency-focused philosophy but further optimizes decoding performance, achieving substantial speed gains in reasoning-intensive tasks while maintaining high-quality outputs. One of its key innovations is the hybrid reasoning mode, which allows developers to control whether the model engages in deeper reasoning processes or faster responses depending on the use case. The model also supports both dense and sparse attention mechanisms, enabling more efficient computation depending on the selected inference framework. ... -
7
Omnilingual ASR
Omnilingual ASR Open-Source Multilingual SpeechRecognition
...It focuses on leveraging self-supervised audio pretraining and scalable fine-tuning so low-resource languages can benefit from high-resource data. The project provides data preparation pipelines, training scripts, decoding utilities, and evaluation tools so researchers can reproduce results and extend to new language sets. It emphasizes modularity: acoustic modeling, language modeling, tokenization, and decoding are separable pieces you can swap or ablate. The repo is aimed at pushing practical multilingual ASR—robust to accents, code-switching, and domain shifts—rather than language-by-language systems. ... -
8
NVIDIA Model Optimizer
A unified library of SOTA model optimization techniques
Model Optimizer is a unified library that provides state-of-the-art techniques for compressing and optimizing deep learning models to improve inference efficiency and deployment performance. It brings together multiple optimization strategies such as quantization, pruning, distillation, and speculative decoding into a single cohesive framework. The library is designed to reduce model size and computational requirements while maintaining accuracy, making it particularly valuable for deploying large models in production environments. It supports a wide range of model types, including large language models, diffusion models, and vision-language models, and integrates with deployment frameworks such as TensorRT and vLLM. ... -
9
Tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models
...It also offers extension mechanisms so that custom encodings can be registered. Internally, it includes the core tokenizer logic (often implemented in Rust or efficient lower-level code), APIs for encoding, decoding, and counting tokens, and binding layers to Python (and sometimes other languages) for easy use. -
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
10
FISSURE
The RF and reverse engineering framework for everyone
FISSURE is an open-source radio frequency analysis and signal intelligence framework built to support software-defined radio research, wireless security experimentation, and protocol reverse engineering. The project brings together tools for capturing, inspecting, decoding, replaying, and analyzing RF signals across a wide range of wireless technologies. It is designed as a practical environment for researchers and operators who need to move from raw spectrum observation to structured investigation without stitching together too many separate utilities by hand. The platform supports workflows related to signal discovery, demodulation, packet inspection, fuzzing, and attack simulation, making it useful for both defensive research and controlled lab testing. ... -
11
mllm
Fast Multimodal LLM on Mobile Devices
...Implemented primarily in C and C++, it is designed to operate with minimal external dependencies while taking advantage of hardware-specific acceleration technologies such as ARM NEON and x86 AVX2 instructions. The system supports multiple optimization techniques including quantization, pruning, and speculative decoding to improve performance while reducing computational overhead. It also provides tools to convert models from popular formats like PyTorch checkpoints into optimized runtime formats that can be executed on supported hardware platforms. -
12
FastDeploy
High-performance Inference and Deployment Toolkit for LLMs and VLMs
...The platform enables developers to deploy trained models quickly using optimized inference pipelines that support GPUs, specialized AI accelerators, and other hardware architectures. FastDeploy includes advanced acceleration technologies such as speculative decoding, multi-token prediction, and efficient KV cache management to improve throughput and latency during inference. It also offers compatibility with OpenAI-style APIs and vLLM-like interfaces, allowing developers to integrate deployed models easily into existing applications and services. -
13
Dolphin
Document Image Parsing via Heterogeneous Anchor Prompting”
Dolphin — maintained by ByteDance — is a project aimed at providing a high-performance, robust, and extensible media or multimedia framework / player infrastructure (or possibly a streaming media solution), intended to meet modern demands for efficiency, flexibility, and integration in media-heavy applications. It seeks to combine performant media playback or handling (audio/video decoding, streaming, buffering) with a modular, developer-friendly API that allows easy embedding into larger applications or services. Because multimedia delivery requirements vary widely (adaptive streaming, live feeds, cross-platform compatibility, custom UI, performance constraints), Dolphin aims to offer a foundation that developers can build upon or adapt to their needs. ... -
14
LTX-Video
Official repository for LTX-Video
LTX-Video is a sophisticated multimedia processing framework from Lightricks designed to handle high-quality video editing, compositing, and transformation tasks with performance and scalability. It provides runtime components that efficiently decode, encode, and manipulate video streams, frame buffers, and audio tracks while exposing a rich API for building customized editing features like transitions, effects, color grading, and keyframe automation. The toolkit is built with both real-time... -
15
stable-diffusion.cpp
Diffusion model(SD,Flux,Wan,Qwen Image,Z-Image,...) inference
stable-diffusion.cpp is a lightweight, high-performance implementation of Stable Diffusion and related generative models written entirely in portable C/C++, designed to run on virtually any device without heavy dependencies. It enables text-to-image and image-to-image generation, supports a growing set of models like SD1.x, SD2.x, SDXL, SD-Turbo, Qwen Image, and more, and is continually updated with support for cutting-edge model variants including video and image editing models. The project... -
16
SGLang
SGLang is a fast serving framework for large language models
SGLang is a fast serving framework for large language models and vision language models. It makes your interaction with models faster and more controllable by co-designing the backend runtime and frontend language. -
17
LLM From Scratch
Build and train a GPT-style language model
...Inspired by Andrej Karpathy’s nanoGPT, the project emphasizes learning through direct implementation and experimentation rather than black-box usage. The workshop documentation explains concepts such as self-attention, embeddings, gradient clipping, optimizer scheduling, and decoding strategies in a practical and approachable way. -
18
TorchAudio
Data manipulation and transformation for audio signal processing
The aim of torchaudio is to apply PyTorch to the audio domain. By supporting PyTorch, torchaudio follows the same philosophy of providing strong GPU acceleration, having a focus on trainable features through the autograd system, and having consistent style (tensor names and dimension names). Therefore, it is primarily a machine learning library and not a general signal processing library. The benefits of PyTorch can be seen in torchaudio through having all the computations be through PyTorch... -
19
Colibrì
Run GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine
...The runtime is implemented in pure C, requires no Python or BLAS during inference, and can operate without a GPU. Compressed attention caches, expert caching, optional hot tiers, and speculative decoding reduce memory pressure and improve repeated use. A planning tool calculates safe disk, RAM, and VRAM placement before loading the model, while a diagnostic command checks system readiness. Colibri includes terminal chat, an OpenAI-compatible text API, and a browser client, but disk-bound generation can be slow on cold caches. -
20
GLM-4.5
GLM-4.5: Open-source LLM for intelligent agents by Z.ai
GLM-4.5 is a cutting-edge open-source large language model designed by Z.ai for intelligent agent applications. The flagship GLM-4.5 model has 355 billion total parameters with 32 billion active parameters, while the compact GLM-4.5-Air version offers 106 billion total parameters and 12 billion active parameters. Both models unify reasoning, coding, and intelligent agent capabilities, providing two modes: a thinking mode for complex reasoning and tool usage, and a non-thinking mode for... -
21
TensorRT LLM
TensorRT LLM provides users with an easy-to-use Python API
...The library focuses on maximizing throughput and minimizing latency through advanced techniques such as quantization, custom attention kernels, and optimized memory management strategies. It includes support for cutting-edge inference methods like speculative decoding and inflight batching, enabling real-time and large-scale AI applications. TensorRT-LLM integrates seamlessly with NVIDIA’s broader inference ecosystem, including Triton Inference Server and distributed deployment frameworks, making it suitable for production environments. -
22
Autonomous Agents
Autonomous Agents (LLMs) research papers. Updated Daily
Autonomous-Agents is a research-focused repository that collects implementations, experiments, and academic resources related to autonomous multi-agent systems and intelligent robotics. The project explores how multiple agents can cooperate and interact with complex environments through machine learning, imitation learning, and multimodal sensing. It includes frameworks that integrate visual perception, tactile sensing, and spatial reasoning to guide the actions of robotic agents during... -
23
NeMo Curator
Scalable data pre processing and curation toolkit for LLMs
NeMo Curator is a Python library specifically designed for fast and scalable dataset preparation and curation for large language model (LLM) use-cases such as foundation model pretraining, domain-adaptive pretraining (DAPT), supervised fine-tuning (SFT) and paramter-efficient fine-tuning (PEFT). It greatly accelerates data curation by leveraging GPUs with Dask and RAPIDS, resulting in significant time savings. The library provides a customizable and modular interface, simplifying pipeline... -
24
Kaldi
kaldi-asr/kaldi is the official location of the Kaldi project
...Kaldi is designed for researchers who need a highly customizable environment to experiment with new algorithms, as well as for practitioners who want robust, production-ready ASR pipelines. It includes extensive tools for data preparation, feature extraction, acoustic and language modeling, decoding, and evaluation. With its modular design, Kaldi allows users to adapt the system to a wide range of languages and domains. As one of the most influential projects in speech recognition, it has become a foundation for much of the modern work in ASR. -
25
MiniCPM4
Ultra-Efficient LLMs on End Device
MiniCPM4 is part of the MiniCPM family of ultra-efficient large language models designed specifically for high performance on edge devices and resource-constrained environments. Unlike traditional large-scale models that require extensive computational resources, MiniCPM4 focuses on delivering competitive reasoning and language capabilities while maintaining significantly lower latency and higher efficiency. It achieves this through optimized architectures, scalable training strategies, and...