Archive · Article

Inference Engineering From First Token to Production Metric

First of five in a written series on serving systems. One request, eight stages, and the two numbers that get reported as one: GPU latency is not request latency. A service that budgets admission by characters admits work it cannot hold.