"During prompt processing, the H100s were running at 92% compute utilization. Tensor cores fully saturated. Exactly what you want to see on a $30K GPU."
"The next phase, token generation, ran for 3 to 9 seconds. During that stretch the same GPUs dropped to 30% utilization. The compute cores sat idle while the memory bus worked flat out reading the attention cache."
A global retailer faced high GPU usage during peak holiday traffic due to a 70B model in their product search pipeline. Despite scaling from 24 to 48 H100 GPUs, latency issues persisted. Profiling the serving layer showed that while prompt processing utilized GPUs at 92%, the token generation phase saw a drop to 30% utilization, indicating inefficiencies. This analysis prompted a reevaluation of GPU infrastructure needs for upcoming sales events.
Read at InfoWorld
Unable to calculate read time
Collection
[
|
...
]