Search Results for "optimizer"

...It distills the GPT architecture into a few hundred lines of Python code, making it far easier to understand than large, production-scale implementations. The repo is organized with a training pipeline (dataset preprocessing, model definition, optimizer, training loop) and inference script so you can train a small GPT on text datasets like Shakespeare or custom corpora. It emphasizes readability and clarity: the training loop is cleanly written, and the code avoids heavy abstractions, letting students follow the architecture step by step. While simple, it can still train non-trivial models on modern GPUs and generate coherent text. ...

Catalyst is a PyTorch framework for accelerated Deep Learning research and development. It allows you to write compact but full-featured Deep Learning pipelines with just a few lines of code. With Catalyst you get a full set of features including a training loop with metrics, model checkpointing and more, all without the boilerplate. Catalyst is focused on reproducibility, rapid experimentation, and codebase reuse so you can break the cycle of writing another regular train loop and make...

...The repository provides code for training and evaluating learned optimizers that can generalize across different problem types, such as quadratic functions and image classification tasks (MNIST and CIFAR-10). Using TensorFlow, it defines a meta-optimizer model that learns by observing and adapting to the optimization trajectories of other models. The project allows users to compare performance between traditional optimizers and the learned optimizer (L2L) on various benchmarks, demonstrating how optimization strategies can be learned through experience. The design supports both single-variable and high-dimensional problems, and includes tools for evaluating how well a learned optimizer performs on unseen tasks.