Fairseq(-py) is a sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling and other text generation tasks. We provide reference implementations of various sequence modeling papers. Recent work by Microsoft and Google has shown that data parallel training can be made significantly more efficient by sharding the model parameters and optimizer state across data parallel workers. These ideas are encapsulated in the new FullyShardedDataParallel (FSDP) wrapper provided by fairscale. Fairseq can be extended through user-supplied plug-ins. Models define the neural network architecture and encapsulate all of the learnable parameters. Criterions compute the loss function given the model outputs and targets. Tasks store dictionaries and provide helpers for loading/iterating over Datasets, initializing the Model/Criterion and calculating the loss.

Features

  • Multi-GPU training on one machine or across multiple machines (data and model parallel)
  • Fast generation on both CPU and GPU with multiple search algorithms implemented
  • Gradient accumulation enables training with large mini-batches even on a single GPU
  • Mixed precision training (trains faster with less GPU memory on NVIDIA tensor cores)
  • Easily register new models, criterions, tasks, optimizers and learning rate schedulers
  • Flexible configuration based on Hydra allowing a combination of code, command-line and file based configuration

Project Samples

Project Activity

See All Activity >

License

MIT License

Follow Fairseq

Fairseq Web Site

Other Useful Business Software
Cut Cloud Costs with Google Compute Engine Icon
Cut Cloud Costs with Google Compute Engine

Save up to 91% with Spot VMs and get automatic sustained-use discounts. One free VM per month, plus $300 in credits.

Save on compute costs with Compute Engine. Reduce your batch jobs and workload bill 60-91% with Spot VMs. Compute Engine's committed use offers customers up to 70% savings through sustained use discounts. Plus, you get one free e2-micro VM monthly and $300 credit to start.
Try Compute Engine
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of Fairseq!

Additional Project Details

Programming Language

Python

Related Categories

Python Artificial Intelligence Software, Python Neural Network Libraries

Registered

2021-05-14