Token Costs

Token costs are the central billing factor when using language models. A token is the smallest processing unit of text a model works with - roughly part of a word, sometimes a short whole word, a punctuation mark, or a syllable. Before a model processes text, it splits it into such tokens; every one of them counts toward the bill.

Language model providers charge per token, usually split into two kinds: input tokens (the entire prompt including context that the model reads) and output tokens (the answer the model generates). With many models, output tokens are considerably more expensive than input tokens, because generation is more demanding than reading.

Token Overhead: Why It Rarely Stays at the User Input

A common misconception is that only the visible text of the user's question gets paid for. In reality, AI agents and long contexts produce a multiple of the tokens compared to the raw input - and every single one is billed. This token overhead includes:

• System prompts: fixed instructions passed to the model again with every request.
• Tool definitions: descriptions of the tools an agent may call, often several hundred tokens per tool.
• Intermediate steps: agents work iteratively - each reasoning and call step reads the history so far again.
• Conversation history: earlier messages travel along into every new call so the context is preserved.

How much of this is possible is limited by the context window - the maximum volume of tokens a model can process per request. The fuller this window is used, the higher the input token costs turn out.

Levers to Reduce Token Costs

• Keep context compact: pass only the documents and history parts truly needed instead of repeating the entire history; summarize long histories rather than feeding them in full.
• Shorter system prompts: phrase instructions concisely - whatever runs along with every request pays off many times over.
• Limit output: cap the answer length via a parameter like max_tokens so expensive output tokens don't accrue unnecessarily.
• Reuse recurring parts via prompt caching: stable blocks such as the system prompt or knowledge base are cached and billed more cheaply on the next call.

Where It Fits in Cost Optimization

Token costs are the building block through which nearly every other cost figure is defined. They are a core topic of LLM cost optimization, which looks at prompt design, model choice, and caching together. The actual operating cost per request is captured by the term inference costs - where token costs come through directly. If you want to reduce them in a structured way, our service page AI Cost Optimization lays out the approach.

Further Resources

• Glossary: LLM Cost Optimization, Prompt Caching, Context Window, Inference Costs
• Service: AI Cost Optimization