Search Results for "transformer"
Sort By:
Showing 46 open source projects for "transformer"
View related business solutions-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
1
Megatron
Ongoing research training transformer models at scale
Megatron is a large, powerful transformer developed by the Applied Deep Learning Research team at NVIDIA. This repository is for ongoing research on training large transformer language models at scale. We developed efficient, model-parallel (tensor, sequence, and pipeline), and multi-node pre-training of transformer based models such as GPT, BERT, and T5 using mixed precision. -
2
Heretic
Fully automatic censorship removal for language models
Heretic is an open-source Python tool that automatically removes the built-in censorship or “safety alignment” from transformer-based language models so they respond to a broader range of prompts with fewer refusals. It works by applying directional ablation techniques and a parameter optimization strategy to adjust internal model behaviors without expensive post-training or altering the core capabilities. Designed for researchers and advanced users, Heretic makes it possible to study and experiment with uncensored model responses in a reproducible, automated way. ... -
3
MoBA
MoBA: Mixture of Block Attention for Long-Context LLMs
...The approach allows language models to scale to significantly longer input contexts without the quadratic computational cost normally associated with transformer attention mechanisms. -
4
SageAttention
NeurIPS2025 Spotlight] Quantized Attention
SageAttention is an open-source optimization library designed to accelerate the attention mechanism used in transformer-based neural networks. Since attention operations are often the most computationally expensive component of modern AI models, SageAttention introduces quantization techniques that significantly reduce computational overhead while preserving model accuracy. The system achieves this by using low-precision numerical formats such as INT4, FP8, or INT8 to represent key matrices within the attention computation. ... -
Veeam Data Platform v13.1 - Get Your Free TrialTry Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
5
Intel LLM Library for PyTorch
Accelerate local LLM inference and finetuning
Intel LLM Library for PyTorch is an open-source acceleration library developed to optimize large language model inference and fine-tuning on Intel hardware platforms. Built as an extension of the PyTorch ecosystem, the library enables developers to run modern transformer models efficiently on Intel CPUs, GPUs, and specialized AI accelerators. The framework provides hardware-aware optimizations and low-precision computation techniques that significantly improve the performance of large language models while reducing memory consumption. IPEX-LLM supports a wide range of popular models, including architectures such as LLaMA, Mistral, Qwen, and other transformer-based systems. ... -
6
Train LLM From Scratch
A straightforward method for training your LLM
Train LLM From Scratch is an educational PyTorch project that shows how to build and train a transformer-based language model from the ground up. It is based on the architecture described in Attention Is All You Need and is designed to make the training pipeline understandable rather than hidden behind a large framework. The repository walks through the process from downloading data to generating text with a trained model. It supports training smaller or larger models, including million- and billion-parameter configurations depending on available hardware. ... -
7
TigerBot
TigerBot: A multi-language multi-task LLM
...The project focuses on building high-performance models capable of handling both English and Chinese tasks while maintaining strong reasoning and conversational abilities. TigerBot models are based on modern transformer architectures and are trained on large datasets that cover multiple domains and languages. The project provides both base models and chat-optimized variants that can be used for dialogue systems, question answering, and general language understanding tasks. In addition to model weights, the repository includes training scripts, inference tools, and configuration files that allow researchers and developers to reproduce experiments or fine-tune the models for specific applications. -
8
LLMs-from-scratch
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
...The repository favors clear Python and NumPy or PyTorch implementations that can be run and modified without heavyweight frameworks obscuring the logic. Chapters and notebooks progress from tiny toy models to more capable transformer stacks, including sampling strategies and evaluation hooks. The focus is on readability, correctness, and experimentation, making it ideal for students and practitioners transitioning from theory to working systems. By the end, you have a grounded sense of how data pipelines, optimization, and inference interact to produce fluent text. -
9
LLMs-Zero-to-Hero
From nobody to big model (LLM) hero
...Rather than relying entirely on existing frameworks, the project encourages readers to implement important components themselves in order to gain a deeper understanding of how modern language models work internally. It includes explanations of dense transformer architectures, mixture-of-experts models, training pipelines, and techniques used in contemporary LLM development. -
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
10
ML Retreat
Machine Learning Journal for Intermediate to Advanced Topics
...Rather than functioning as a traditional tutorial series, the repository is organized as a learning journey that progressively explores increasingly advanced subjects. Topics include large language models, graph neural networks, mechanistic interpretability, transformer architectures, and emerging research areas such as quantum machine learning. The repository includes references to influential research papers, lectures, and educational content from well-known machine learning educators. -
11
ModernBERT
Bringing BERT into modernity via both architecture changes and scaling
ModernBERT is an open-source research project that modernizes the classic BERT encoder architecture by incorporating recent advances in transformer design, training techniques, and efficiency improvements. The goal of the project is to bring BERT-style models up to date with the capabilities of modern large language models while preserving the strengths of bidirectional encoder architectures used for tasks such as classification, retrieval, and semantic search. ModernBERT introduces architectural improvements that enhance both training efficiency and inference performance, making the model more suitable for modern large-scale machine learning pipelines. ... -
12
LightLLM
LightLLM is a Python-based LLM (Large Language Model) inference
...LightLLM is designed to handle large-scale model workloads in production environments, supporting efficient batching and GPU utilization for fast inference across multiple requests. Its architecture allows models to be deployed with minimal overhead while maintaining compatibility with popular transformer-based model families such as LLaMA and GPT-style architectures. -
13
MiniOneRec
Minimal reproduction of OneRec
...The framework provides an end-to-end pipeline for building generative recommender systems, including semantic identifier construction, supervised fine-tuning, and reinforcement learning-based optimization. Semantic IDs are created using techniques such as quantized variational autoencoders to convert item features into token sequences that can be modeled by transformer architectures. Developers can train and evaluate recommendation models using different backbone language models while benefiting from the generative framework’s parameter efficiency and scalability. -
14
R-KV
Redundancy-aware KV Cache Compression for Reasoning Models
R-KV is an open-source research project that focuses on improving the efficiency of large language model inference through key-value cache compression techniques. Modern transformer models rely heavily on KV caches during autoregressive decoding, which store intermediate attention states to accelerate generation. However, these caches can consume large amounts of memory, especially in reasoning-oriented models with long context windows. R-KV introduces a method for compressing the KV cache during decoding, allowing models to maintain reasoning performance while reducing memory consumption and computational overhead. ... -
15
Happy-LLM
Large Language Model Principles and Practice Tutorial from Scratch
...The project guides learners through the entire conceptual and practical pipeline of modern LLM development, starting with foundational natural language processing concepts and gradually progressing to advanced architectures and training techniques. It explains the Transformer architecture, pre-training paradigms, and model scaling strategies while also providing hands-on coding examples so readers can implement and experiment with their own models. The tutorial emphasizes practical understanding by walking users through building and training small language models, including tokenizer construction, pre-training workflows, and fine-tuning methods. -
16
llm.c
LLM training in simple, raw C/CUDA
llm.c is a minimalist, systems-level implementation of a small transformer-based language model in C that prioritizes clarity and educational value. By stripping away heavy frameworks, it exposes the core math and memory flows of embeddings, attention, and feed-forward layers. The code illustrates how to wire forward passes, losses, and simple training or inference loops with direct control over arrays and buffers. -
17
AirLLM
AirLLM 70B inference with single 4GB GPU
...This layer-wise inference approach allows models with tens of billions of parameters to run on devices with only a few gigabytes of VRAM. AirLLM preprocesses model weights so that each transformer layer can be loaded independently during computation, reducing the memory footprint while still performing full inference. As a result, developers can experiment with models that previously required specialized high-end GPUs. -
18
MING
A large-scale model of medical consultation in Chinese
...This interactive capability makes it suitable for conversational health applications, patient triage scenarios, and educational demonstrations. The model is built on transformer-based architectures using frameworks such as PyTorch and integrates with Hugging Face tooling for training and inference workflows. -
19
KVCache-Factory
Unified KV Cache Compression Methods for Auto-Regressive Models
KVCache-Factory is an open-source research framework designed to explore and implement unified key-value cache compression techniques for autoregressive transformer models. In large language models, the key-value cache stores intermediate attention states that enable efficient token generation during inference, but these caches can consume large amounts of GPU memory when handling long contexts. KVCache-Factory provides a platform for implementing and evaluating multiple compression strategies that reduce memory usage while preserving model performance. ... -
20
MaxText
A simple, performant and scalable Jax LLM
MaxText is a high-performance, highly scalable open-source framework designed to train and fine-tune large language models using the JAX ecosystem. The project acts as both a reference implementation and a practical training library that demonstrates best practices for building and scaling transformer-based language models on modern accelerator hardware. It is optimized to run efficiently on Google Cloud TPUs and GPUs, enabling researchers and engineers to train models ranging from small experiments to extremely large distributed workloads. The framework focuses on simplicity while still supporting advanced techniques such as model sharding, distributed computation, and high-throughput training pipelines. ... -
21
llms-from-scratch-cn
Build a large language model from 0 only with Python foundation
...Rather than focusing on using pre-trained models through APIs, the project emphasizes understanding the internal mechanisms of modern language models, including tokenization, attention mechanisms, transformer architecture, and training workflows. Through a collection of notebooks, code examples, and translated learning materials, users can explore how to implement components such as multi-head attention, data loaders, and training pipelines using Python and PyTorch. -
22
LLM Foundry
LLM training code for MosaicML foundation models
Introducing MPT-7B, the first entry in our MosaicML Foundation Series. MPT-7B is a transformer trained from scratch on 1T tokens of text and code. It is open source, available for commercial use, and matches the quality of LLaMA-7B. MPT-7B was trained on the MosaicML platform in 9.5 days with zero human intervention at a cost of ~$200k. Large language models (LLMs) are changing the world, but for those outside well-resourced industry labs, it can be extremely difficult to train and deploy these models. ... -
23
MiniMind
Train a 26M-parameter GPT from scratch in just 2h
minimind is a framework that enables users to train a 26-million-parameter GPT (Generative Pre-trained Transformer) model from scratch in approximately two hours. It provides a streamlined process for data preparation, model training, and evaluation, making it accessible for individuals and organizations to develop their own language models without extensive computational resources. -
24
LLM-Pruner
On the Structural Pruning of Large Language Models
...Large language models often require enormous computational resources, making them expensive to deploy and inefficient for many practical applications. LLM-Pruner addresses this issue by identifying and removing non-essential components within transformer architectures, such as redundant attention heads or feed-forward structures. The framework relies on gradient-based analysis to determine which parameters contribute least to model performance, enabling targeted structural pruning rather than simple weight removal. After pruning, the framework applies lightweight fine-tuning methods such as LoRA to recover performance using relatively small datasets and short training times. -
25
How to Train Your GPT
Build a modern LLM from scratch. Every line commented
How to Train Your GPT is an interactive textbook that teaches users how to build, train, and run a modern language model from scratch. It is written for learners with minimal machine-learning background, using simple explanations, commented code, and practical examples. The project covers the same broad family of architecture behind systems such as GPT-style models, LLaMA-style models, Claude-style systems, and Mistral-style models. It includes chapters and topic explainers on tokenizers,...
