Energy Consumption Analysis of Parallelization Strategies with Adaptive Precision in Deep Learning Models
Charles Cetta
Co-Presenters: Individual Presentation
College: The Dorothy and George Hennings College of Science, Mathematics and Technology
Major: Computer Science
Faculty Research Mentor: Yulia Kumar
Abstract:
This research extends our previous work on distributed training methodologies by investigating the energy efficiency implications of different parallelism strategies and precision levels in deep neural network training. As AI models continue to grow in size and complexity, their energy consumption has become a critical concern for both environmental sustainability and operational costs. Our study proposes a novel framework that combines adaptive precision training with different parallelism strategies to optimize energy efficiency in distributed deep learning systems.The research will systematically evaluate the energy consumption patterns of popular architectures such as ResNet and BERT under various training configurations. We will implement an adaptive precision framework that dynamically adjusts numerical precision based on real-time energy consumption metrics, investigating its interaction with both data and model parallelism approaches. The study will utilize two distinct HPC environments to ensure the generalizability of our findings across different hardware configurations.Through this research, we aim to develop comprehensive guidelines for energy-efficient distributed training that can be applied across various deep learning applications. Expected outcomes include quantitative analyses of energy consumption patterns under different parallelism and precision configurations, a practical framework for adaptive precision training, and recommendations for optimal training strategies based on model architecture and hardware environment. This work will contribute to the growing field of green AI by providing practical strategies for reducing the environmental impact of large-scale machine learning operations.