A distributed training method with intergenerational accumulation and cross-node random drop for mechanical fault diagnosis

With increasing complexity of deep neural networks and continuous expansion of training datasets, the computational cost of model training grows exponentially. To reduce training time, distributed training systems leveraging multiple computing devices have been developed for computational acceleration. However, compared with the rapidly increasing computing power, the communication bandwidth between devices increases slowly and becomes a bottleneck restricting the efficiency of distributed training. In this paper, an efficient distributed training method called gradient transfer compression (GTC) is proposed to reduce communication overhead and improve training efficiency. The methodology involves three key techniques: (1) Intergenerational accumulation, where gradients generated over multiple iterations are stored and accumulated, reducing the frequency of communication between computing devices; (2) Cross-node random drop, which synchronizes gradients with a specified ratio to decrease network traffic while ensuring model convergence; and (3) Mixed precision training, which reduces the bandwidth required for gradient communication. The effectiveness of GTC is demonstrated through experiments on two rolling bearing datasets. Compared with the conventional PyTorch distributed training method, the proposed method reduces the GPU memory usage by 97.10 % and 14.02 %, increases the training efficiency by 24.74 % and 8.03 % respectively in two cases, while maintaining the diagnostic performance of the model.