AI and machine learning training often distribute workloads across multiple GPUs, which necessitates constant data transfer, including gradients and weights. Inefficient data transfer can lead to underutilized resources and inflated training costs. Optimizing this transfer is key to improving performance, and this piece zeroes in on data-distributed training, where identical model copies reside on each GPU. Each GPU processes a portion of the input data, calculates local gradients, and shares these gradients for a unified update across all models. The article compares two Amazon EC2 instance types: the g6e.48xlarge with NVIDIA L40S GPUs and the p4d.24xlarge with NVIDIA A100 GPUs. The key difference lies in their connection methods; the g6e instance uses PCI Express, while the p4d employs NVIDIA NVLink, which provides faster communication. This distinction is significant, especially for workloads with high communication demands. The authors illustrate their findings by running a toy model, specifically a Vision Transformer (ViT) with around 306 million parameters, on both instances to evaluate the impact of GPU-to-GPU communication. They set up a training experiment using PyTorch and include a synthetic dataset to mimic real-world scenarios. The configuration uses the NVIDIA Collective Communications Library (NCCL) for efficient communication. They also utilize the NVIDIA Nsight Systems profiler to analyze the training process, capturing various performance metrics. This structured approach allows them to identify how different configurations and instance types affect the overall training throughput, providing insights into the optimization of data transfer in distributed training setups.