DownloadFairseq(-py) is a sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling and other text generation tasks. We provide reference implementations of various sequence modeling papers. Recent work by Microsoft and Google has shown that data parallel training can be made significantly more efficient by sharding the model parameters and optimizer state across data parallel workers. These ideas are encapsulated in the new FullyShardedDataParallel (FSDP) wrapper provided by fairscale. Fairseq can be extended through user-supplied plug-ins. Models define the neural network architecture and encapsulate all of the learnable parameters. Criterions compute the loss function given the model outputs and targets. Tasks store dictionaries and provide helpers for loading/iterating over Datasets, initializing the Model/Criterion and calculating the loss.
Features
- Multi-GPU training on one machine or across multiple machines (data and model parallel)
- Fast generation on both CPU and GPU with multiple search algorithms implemented
- Gradient accumulation enables training with large mini-batches even on a single GPU
- Mixed precision training (trains faster with less GPU memory on NVIDIA tensor cores)
- Easily register new models, criterions, tasks, optimizers and learning rate schedulers
- Flexible configuration based on Hydra allowing a combination of code, command-line and file based configuration
Project Samples
