Search Results for "generative"

Megatron is a large, powerful transformer developed by the Applied Deep Learning Research team at NVIDIA. This repository is for ongoing research on training large transformer language models at scale. We developed efficient, model-parallel (tensor, sequence, and pipeline), and multi-node pre-training of transformer based models such as GPT, BERT, and T5 using mixed precision. Megatron is also used in NeMo Megatron, a framework to help enterprises overcome the challenges of building and...

Implementation of Make-A-Video, new SOTA text to video generator from Meta AI, in Pytorch. They combine pseudo-3d convolutions (axial convolutions) and temporal attention and show much better temporal fusion. The pseudo-3d convolutions isn't a new concept. It has been explored before in other contexts, say for protein contact prediction as "dimensional hybrid residual networks". The gist of the paper comes down to, take a SOTA text-to-image model (here they use DALL-E2, but the same learning...

This repository records EleutherAI's library for training large-scale language models on GPUs. Our current framework is based on NVIDIA's Megatron Language Model and has been augmented with techniques from DeepSpeed as well as some novel optimizations. We aim to make this repo a centralized and accessible place to gather techniques for training large-scale autoregressive language models, and accelerate research into large-scale training. For those looking for a TPU-centric codebase, we...

...It is built around several guiding principles: moving from outline to detail, from older foundational papers to state-of-the-art work, and from generic to more specialized areas while keeping a focus on impactful contributions. The roadmap organizes papers into categories such as fundamentals, convolutional networks, sequence models, unsupervised learning, generative models, optimization, and application areas like computer vision or NLP. For each section, it suggests an order that lets readers gradually build intuition and then dive deeper into more advanced or recent topics. It is particularly useful for students and engineers who want to systematically improve their understanding rather than randomly picking papers.