DownloadHy4 preview is Tencent’s open-weight flagship Mixture-of-Experts language model designed for advanced reasoning, software engineering, productivity, scientific research, and long-horizon tasks. It contains 770B backbone parameters while activating 49B per token across 78 layers, with 256 routed experts and one shared expert in each MoE layer. Its architecture uses Gated DeepSeek Sparse Attention with IndexCache for cross-layer sparse-index reuse and identity Hyper-Connections to improve information flow. A native 10B-parameter Multi-Token Prediction layer enables speculative decoding for faster inference. Hy4-preview supports a native 1M-token context window, allowing it to process large codebases, numerous files, and complex extended workflows. Tencent specifically optimized the model for software engineering, office and financial analysis, game development, and scientific research.
Features
- 770B total parameters with 49B activated per token
- 256 routed experts plus one shared expert per MoE layer
- Native 1M-token context window
- Gated DeepSeek Sparse Attention with IndexCache
- Identity Hyper-Connections for enhanced information flow
- Native Multi-Token Prediction for speculative decoding
- Deep reasoning and direct no-thinking response modes
- Optimized for coding, productivity, research, and game development
Project Samples
