HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models

The increasing size of language models necessitates a thorough analysis acrossmultiple dimensions to assess trade-offs among crucial hardware metrics such aslatency, energy consumption, GPU memory usage, and performance. Identifyingoptimal model configurations under specific hardware constraints is becomingessential but remains challenging due to the computational load of exhaustivetraining and evaluation on multiple devices. To address this, we introduce HW-GPT-Bench, a hardware-aware benchmark that utilizes surrogate predictions toapproximate various hardware metrics across 13 devices of architectures in the GPT-2 family, with architectures containing up to 1.55B parameters. Our surrogates,via calibrated predictions and reliable uncertainty estimates, faithfully model theheteroscedastic noise inherent in the energy and latency measurements. To estimateperplexity, we employ weight-sharing techniques from Neural Architecture Search(NAS), inheriting pretrained weights from the largest GPT-2 model. Finally, wedemonstrate the utility of HW-GPT-Bench by simulating optimization trajectoriesof various multi-objective optimization algorithms in just a few seconds.