#gpu-utilization

[ follow ]
#ai-infrastructure #cost-efficiency #ai-investment #environmental-impact #human-competency #token-pricing
fromInfoWorld
1 week ago

How I doubled my GPU efficiency without buying a single new card

During prompt processing, the H100s were running at 92% compute utilization. Tensor cores fully saturated. Exactly what you want to see on a $30K GPU.
Business intelligence
#ai-infrastructure
more#ai-infrastructure
Artificial intelligence
fromInfoWorld
5 months ago

Maximizing speed: How continuous batching unlocks unprecedented LLM throughput

Continuous batching processes one token at a time across active requests with micro-steps and on-the-fly swaps to maintain full GPU utilization and dramatically increase throughput.
Artificial intelligence
fromTheregister
11 months ago

Wanted: Metric for gauging if GPUs are being used optimally

Efficient usage of costly GPU accelerators in AI processing is crucial, but the industry struggles with effective measurement methods.
Many AI teams overestimate their GPU utilization, limiting performance and increasing costs.
[ Load more ]