Megatron-LM
Developed by NVIDIA
Megatron-LM and Megatron Core are NVIDIA-developed GPU-optimized libraries for training transformer models at scale. Megatron Core offers GPU-accelerated building blocks, supporting various parallelism strategies (TP, PP, DP, EP, CP) and mixed-precision training (FP16, BF16, FP8, FP4), ideal for custom training frameworks. Megatron-LM is a reference example combining Megatron Core with pre-configured scripts for research and rapid experimentation. It efficiently trains multi-billion parameter models with high Model FLOP Utilization and supports advanced architectures like MoE.
- GPU-optimized for large-scale Transformer model training
- Supports Tensor, Pipeline, Data, Expert, and Context parallelism strategies
- Provides mixed precision training support (FP16, BF16, FP8, FP4)
- Efficiently trains up to 462B parameter models with up to 47% MFU
- Hugging Face compatibility, MoE models, and emerging optimizers support
desktop