Inference Costs

Inference is the application of a fully trained model to produce an answer - as opposed to training, in which the model is created in the first place. Inference costs are accordingly the running costs that each individual request causes in operation. While training costs are a one-off, inference costs recur with every use and therefore usually dominate the total bill over the lifetime of an LLM application.

Two Basic Models

• API / Managed: Billing is per token. No infrastructure of your own is needed, but there are unit costs for every request - the more traffic, the higher the running bill.
• Self-hosting: The model runs on your own or rented GPU/compute infrastructure. Costs arise from hardware runtime, not per token. At high, steady volume, the cost per request can end up lower than with the API.

Levers When Self-Hosting

Those who run models themselves reduce inference costs primarily through high GPU utilization instead of expensive idle time. Concrete levers:

• Batching / continuous batching: Multiple requests are bundled and sent through the GPU together instead of processing each one individually.
• Quantization: Smaller number formats (INT8, INT4) reduce memory footprint and compute per request.
• KV cache reuse: Already computed key-value pairs are reused instead of recalculated.
• Utilization: High, steady GPU utilization lowers the cost per request - idle time is the most expensive mode of operation.
• Spot instances: Cheaper, interruptible cloud capacity for workloads that can tolerate delays.

API or Self-Hosting - a TCO Question

The decision between a managed API and in-house operation is a question of total cost of ownership and depends on three factors: request volume, the required latency, and the operational effort for maintenance, scaling, and on-call. Only above a certain, steady volume does self-hosting pay off - at fluctuating or low load, the API usually remains cheaper because no idle hardware has to be paid for. Inference costs are a central building block of LLM cost optimization; in addition, model routing distributes requests across models of differing cost. How we reduce inference costs in the bigger picture is shown on our service page AI Cost Optimization.

Further Resources

• Glossary: LLM Cost Optimization, Token Costs, Model Routing
• Service: AI Cost Optimization