Same model, opposite stress test
It's tempting to treat training and inference as the same workload at different scales. They run the same transformer architecture, after all. But the way each one stresses a processor is almost inverted. Training is dominated by large, predictable batches, gradient computation, and throughput measured over hours or days. Inference is dominated by unpredictable request arrival, tight latency budgets, and throughput measured in milliseconds per token.
Batching works differently when a person is waiting
Training gets to choose its batch size. Engineers tune it to keep compute units saturated, because there's no user on the other end of a training step waiting for a response. Inference, especially interactive inference, doesn't get that choice. Requests arrive when they arrive, at whatever size a user's prompt happens to be, and batching them together for efficiency has to be balanced against how long the first user in that batch is willing to wait for the last one to finish.
Autoregressive generation changes the arithmetic
Generating a response token by token means the model reloads its weights, or at minimum touches its key-value cache, for every single token it produces. That repeated, sequential memory access pattern is largely absent from training, where the same weights are reused across a full batch before moving on. It's a large part of why inference tends to be memory-bound in ways training often isn't, and why architecture tuned for training throughput doesn't automatically translate into good inference latency.
Why the distinction should drive infrastructure decisions
Treating inference as an afterthought to training-optimized hardware is one of the more expensive assumptions an infrastructure team can make. The workloads diverge enough that the accelerator best suited to training a model is not automatically the accelerator best suited to serving it in production. That divergence is the entire premise behind building AI inference silicon as its own category, rather than reusing training hardware and hoping the efficiency gap doesn't matter at scale. At the volumes production inference now runs, it does.
Read more about how this shapes BERNIONE's approach in Why Purpose-Built Beats General-Purpose for AI Inference.