Search Results for "mixture"
Sort By:
Showing 55 open source projects for "mixture"
View related business solutions-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
Veeam Data Platform v13.1Try Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
1
MoBA
MoBA: Mixture of Block Attention for Long-Context LLMs
MoBA, short for Mixture of Block Attention, is an open-source research implementation of a novel attention mechanism designed to improve the efficiency of large language models processing extremely long contexts. The architecture adapts ideas from Mixture-of-Experts networks and applies them directly to the attention mechanism of transformer models. Instead of forcing each token to attend to every other token in the sequence, MoBA divides the context into blocks and dynamically routes queries to only the most relevant segments of information. ... -
2
Wan2.2
Wan2.2: Open and Advanced Large-Scale Video Generative Model
Wan2.2 is a major upgrade to the Wan series of open and advanced large-scale video generative models, incorporating cutting-edge innovations to boost video generation quality and efficiency. It introduces a Mixture-of-Experts (MoE) architecture that splits the denoising process across specialized expert models, increasing total model capacity without raising computational costs. Wan2.2 integrates meticulously curated cinematic aesthetic data, enabling precise control over lighting, composition, color tone, and more, for high-quality, customizable video styles. ... -
3
OpenMythos
A theoretical reconstruction of the Claude Mythos architecture
...It divides computation into three main stages, including a pre-processing phase, a looped recurrent reasoning block, and a final output refinement stage, creating a structured pipeline for inference. The architecture incorporates advanced techniques such as mixture-of-experts routing, adaptive computation time, and multiple attention mechanisms to dynamically allocate compute where needed. It is highly configurable through a centralized configuration system, allowing experimentation with different architectural parameters such as loop depth, attention type. -
4
DeepSeek R1
Open-source, high-performance AI model with advanced reasoning
DeepSeek-R1 is an open-source large language model developed by DeepSeek, designed to excel in complex reasoning tasks across domains such as mathematics, coding, and language. DeepSeek R1 offers unrestricted access for both commercial and academic use. The model employs a Mixture of Experts (MoE) architecture, comprising 671 billion total parameters with 37 billion active parameters per token, and supports a context length of up to 128,000 tokens. DeepSeek-R1's training regimen uniquely integrates large-scale reinforcement learning (RL) without relying on supervised fine-tuning, enabling the model to develop advanced reasoning capabilities. ... -
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
5
LingBot-Video
Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence
LingBot-Video is a large-scale mixture-of-experts video generation model focused on embodied intelligence. It is designed to connect video synthesis with physical-world understanding instead of generating only visually appealing clips. The project includes dense and MoE model variants for text-to-image, text-to-video, and text-image-to-video workflows. Its training combines large-scale web video data with more than 70,000 hours of embodied data. -
6
Tile Kernels
A kernel library written in tilelang
Tile Kernels is a DeepSeek kernel library written with TileLang for high-performance AI and machine-learning workloads. It contains specialized kernels for areas such as mixture-of-experts routing, quantization, batched transpose operations, Engram gating, and Manifold HyperConnection components. The project includes both optimized kernel implementations and PyTorch reference versions for comparison and validation. It is aimed at developers and researchers who work close to model internals and need efficient low-level building blocks. ... -
7
HunyuanImage-3.0
A Powerful Native Multimodal Model for Image Generation
...It unifies multimodal understanding and generation in a single autoregressive framework, combining text and image modalities seamlessly rather than relying on separate image-only diffusion components. It uses a Mixture-of-Experts (MoE) architecture with many expert subnetworks to scale efficiently, deploying only a subset of experts per token, which allows large parameter counts without linear inference cost explosion. The model is intended to be competitive with closed-source image generation systems, aiming for high fidelity, prompt adherence, fine detail, and even “world knowledge” reasoning (i.e. leveraging context, semantics, or common sense in generation). ... -
8
LLMs-Zero-to-Hero
From nobody to big model (LLM) hero
...Rather than relying entirely on existing frameworks, the project encourages readers to implement important components themselves in order to gain a deeper understanding of how modern language models work internally. It includes explanations of dense transformer architectures, mixture-of-experts models, training pipelines, and techniques used in contemporary LLM development. -
9
Qwen3-Coder
Qwen3-Coder is the code version of Qwen3
Qwen3-Coder is the latest and most powerful agentic code model developed by the Qwen team at Alibaba Cloud. Its flagship version, Qwen3-Coder-480B-A35B-Instruct, features a massive 480 billion-parameter Mixture-of-Experts architecture with 35 billion active parameters, delivering top-tier performance on coding and agentic tasks. This model sets new state-of-the-art benchmarks among open models for agentic coding, browser-use, and tool-use, matching performance comparable to leading models like Claude Sonnet. Qwen3-Coder supports an exceptionally long context window of 256,000 tokens, extendable to 1 million tokens using Yarn, enabling repository-scale code understanding and generation. ... -
Host LLMs in Production With On-Demand GPUsDeploy your model, get an endpoint, pay only for compute time. No GPU provisioning or infrastructure management required.
-
10
Heretic
Fully automatic censorship removal for language models
...Designed for researchers and advanced users, Heretic makes it possible to study and experiment with uncensored model responses in a reproducible, automated way. The project can decensor many popular dense and some mixture-of-experts (MoE) models, supporting workflows that would otherwise require manual tuning. Beyond simple decensoring, Heretic includes research-oriented options for analyzing model internals and interpretability data. -
11
Xtuner
A Next-Generation Training Engine Built for Ultra-Large MoE Models
Xtuner is a large-scale training engine designed for efficient training and fine-tuning of modern large language models, particularly mixture-of-experts architectures. The framework focuses on enabling scalable training for extremely large models while maintaining efficiency across distributed computing environments. Unlike traditional 3D parallel training strategies, XTuner introduces optimized parallelism techniques that simplify scaling and reduce system complexity when training massive models. ... -
12
Ling-V2
Ling-V2 is a MoE LLM provided and open-sourced by InclusionAI
Ling-V2 is an open-source family of Mixture-of-Experts (MoE) large language models developed by the InclusionAI research organization with the goal of combining state-of-the-art performance, efficiency, and openness for next-generation AI applications. It introduces highly sparse architectures where only a fraction of the model’s parameters are activated per input token, enabling models like Ling-mini-2.0 to achieve reasoning and instruction-following capabilities on par with much larger dense models while remaining significantly more computationally efficient. ... -
13
Ring
Ring is a reasoning MoE LLM provided and open-sourced by InclusionAI
Ring is a reasoning Mixture-of-Experts (MoE) large language model (LLM) developed by inclusionAI. It is built from or derived from Ling. Its design emphasizes reasoning, efficiency, and modular expert activation. In its “flash” variant (Ring-flash-2.0), it optimizes inference by activating only a subset of experts. It applies reinforcement learning/reasoning optimization techniques. -
14
Wan2.1
Wan2.1: Open and Advanced Large-Scale Video Generative Model
...The model supports text-to-video and image-to-video generation tasks with flexible resolution options suitable for various GPU hardware configurations. Wan2.1’s architecture balances generation quality and inference cost, paving the way for later improvements seen in Wan2.2 such as Mixture-of-Experts and enhanced aesthetics. It was trained on large-scale video and image datasets, providing generalization across diverse scenes and motion patterns. -
15
DeepSeek-V3
Powerful AI language model (MoE) optimized for efficiency/performance
DeepSeek-V3 is a robust Mixture-of-Experts (MoE) language model developed by DeepSeek, featuring a total of 671 billion parameters, with 37 billion activated per token. It employs Multi-head Latent Attention (MLA) and the DeepSeekMoE architecture to enhance computational efficiency. The model introduces an auxiliary-loss-free load balancing strategy and a multi-token prediction training objective to boost performance. -
16
MiniMax-M1
Open-weight, large-scale hybrid-attention reasoning model
...It is built on the MiniMax-Text-01 foundation and keeps the same massive parameter budget, but reworks the attention and training setup for better reasoning and test-time compute scaling. Architecturally, it combines Mixture-of-Experts layers with lightning attention, enabling the model to support a native context length of 1 million tokens while using far fewer FLOPs than comparable reasoning models for very long generations. The team emphasizes efficient scaling of test-time compute: at 100K-token generation lengths, M1 reportedly uses only about 25 percent of the FLOPs of some competing models, making extended “think step” traces more feasible. ... -
17
TerraGov Marine Corps
TGMC: TerraGov Marine Corps, a SS13 mod
...The codebase serves as both a live game server foundation and a development platform for contributors who want to expand gameplay features, design new mechanics, or refine existing systems. With its mixture of roleplay, tactical combat, and science fiction setting, TGMC provides a distinctive twist on the SS13 lineage. -
18
MiniCPM5-2B
SOTA on-device LLMs, small yet powerful
...It also provides fine-tuning, quantization, deployment, and agent-skill resources for adapting models to different environments. Earlier MiniCPM releases explore sparse attention, mixture-of-experts designs, long-context modeling, and efficient edge inference. -
19
AI Infra
Understanding of AI Infra: Quantitative Analysis and System Design
...Companion tools help readers reproduce resource calculations and inspect the assumptions behind system-design decisions. Additional material covers accelerators, networking, KV caches, inference serving, training systems, mixture-of-experts models, and performance engineering. The source is written in Markdown and automatically built into a XeLaTeX PDF and a searchable website. Research notes, references, calculation scripts, and historical revisions are preserved alongside the main manuscript. -
20
PRML
PRML algorithms implemented in Python
...Rather than just summarizing concepts, the repository includes working code that demonstrates linear regression and classification, kernel methods, neural networks, graphical models, mixture models with EM algorithms, approximate inference, and sequential data methods — all following the book’s structure and notation. Many of these algorithms are paired with Jupyter notebooks that let users interact with the code, visualize results, and experiment with parameters in a way that deeply strengthens theoretical understanding. -
21
Kaldi
kaldi-asr/kaldi is the official location of the Kaldi project
Kaldi is an open source toolkit for speech recognition research. It provides a powerful framework for building state-of-the-art automatic speech recognition (ASR) systems, with support for deep neural networks, Gaussian mixture models, hidden Markov models, and other advanced techniques. The toolkit is widely used in both academia and industry due to its flexibility, extensibility, and strong community support. Kaldi is designed for researchers who need a highly customizable environment to experiment with new algorithms, as well as for practitioners who want robust, production-ready ASR pipelines. ... -
22
Oh My Hermes
All in one plugin for Hermes Agent
Oh My Hermes is an operating layer for Hermes Agent that adds structured routing, coding workflows, long-term memory, and evidence-aware execution. It keeps Hermes as the conversational front end while deciding which workflow, model category, and specialist capability should handle each task. Work can be split into parallel lanes with isolated worktrees and typed completion states. The plugin includes a reviewed memory system that stores approved records with provenance and aging rules. Its... -
23
Ling
Ling is a MoE LLM provided and open-sourced by InclusionAI
Ling is a Mixture-of-Experts (MoE) large language model (LLM) provided and open-sourced by inclusionAI. The project offers different sizes (Ling-lite, Ling-plus) and emphasizes flexibility and efficiency: being able to scale, adapt expert activation, and perform across a range of natural language/reasoning tasks. Example scripts, inference pipelines, and documentation. -
24
GLM-4.5
GLM-4.5: Open-source LLM for intelligent agents by Z.ai
GLM-4.5 is a cutting-edge open-source large language model designed by Z.ai for intelligent agent applications. The flagship GLM-4.5 model has 355 billion total parameters with 32 billion active parameters, while the compact GLM-4.5-Air version offers 106 billion total parameters and 12 billion active parameters. Both models unify reasoning, coding, and intelligent agent capabilities, providing two modes: a thinking mode for complex reasoning and tool usage, and a non-thinking mode for... -
25
DeepEP
DeepEP: an efficient expert-parallel communication library
DeepEP is a communication library designed specifically to support Mixture-of-Experts (MoE) and expert parallelism (EP) deployments. Its core role is to implement high-throughput, low-latency all-to-all GPU communication kernels, which handle the dispatching of tokens to different experts (or shards) and then combining expert outputs back into the main data flow. Because MoE architectures require routing inputs to different experts, communication overhead can become a bottleneck — DeepEP addresses that by providing optimized GPU kernels and efficient dispatch/combining logic. ...
