Search Results for "node"

...It supports advanced parallelism strategies including tensor, pipeline, data, expert, and context parallelism, enabling training across massive multi-GPU and multi-node clusters. The framework includes mixed-precision training options such as FP16, BF16, FP8, and FP4 to maximize performance and memory efficiency on modern hardware. Megatron-LM is widely used in research and industry for pretraining GPT-, BERT-, T5-, and multimodal-style models, with tooling for checkpoint conversion and interoperability with Hugging Face. ...

evolution-strategies-starter is an archived OpenAI research project that provides a distributed implementation of the algorithm described in the paper “Evolution Strategies as a Scalable Alternative to Reinforcement Learning” by Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever. The repository demonstrates how to scale Evolution Strategies (ES) for reinforcement learning tasks using a master-worker architecture, where the master node broadcasts parameters to multiple workers, and the workers return performance results after evaluation. This approach allows for efficient parallelization and robustness against worker termination, making it ideal for distributed execution on Amazon EC2 spot instances. The codebase supports building custom AMIs with Packer, integrates with MuJoCo for simulation-based experiments, and includes scripts for launching and managing large-scale runs. ...