Run distributed training with the SageMaker distributed data parallelism library

The SageMaker distributed data parallelism (SMDDP) library extends SageMaker training capabilities on deep learning models with near-linear scaling efficiency by providing implementations of collective communication operations optimized for AWS infrastructure.

When training large machine learning (ML) models, such as large language models (LLM) and diffusion models, on a huge training dataset, ML practitioners use clusters of accelerators and distributed training techniques to reduce the time to train or resolve memory constraints for models that cannot fit in each GPU memory. ML practitioners often start with multiple accelerators on a single instance and then scale to clusters of instances as their workload requirements increase. As the cluster size increases, so does the communication overhead between multiple nodes, which leads to drop in overall computational performance.

To address such overhead and memory problems, the SMDDP library offers the following.

To learn more about the details of the SMDDP library offerings, proceed to Introduction to the SageMaker distributed data parallelism library.

For more information about training with the model-parallel strategy offered by SageMaker, see also (Archived) SageMaker model parallelism library v1.x.