Tokens and context windows

The previous lesson said a model maps a token sequence to a distribution over next tokens. This one is about that word token — because it is simultaneously the unit you are billed in, the unit your latency is measured in, and the unit your prompt has to fit into. Three of the four operational concerns in an AI feature are denominated in it.

What a token is

A token is a chunk of text from a fixed vocabulary, learned by byte-pair encoding: start from bytes, repeatedly merge the most frequent adjacent pair, stop at a target vocabulary size. Common words end up as one token, rarer ones as several, and the leading space is usually part of the token — " the" and "the" are different entries.

Some rules of thumb for English prose, and where each one breaks:

That last row is a real production failure: a request that fits comfortably in testing blows the window the first time a user pastes a stack trace with an embedded base64 payload.

Tokenisation also explains a whole family of "the model is stupid" complaints. Asking how many rs are in strawberry is asking a system that never saw individual letters — it saw two or three subword chunks. Asking it to reverse a string, count characters, or do exact arithmetic on long numbers runs into the same wall. These are not reasoning failures, they are representation failures, and the fix is to do the work in C# and hand the model the result.

The context window is a budget

The context window is the maximum number of tokens the model can attend to in one call, and it covers everything: the system prompt, the whole conversation history you are resending, any retrieved documents, tool definitions, tool results, and the tokens the model is about to generate.

That last clause is the one that catches people. Input and output share the window. Fill it to the brim with input and there is no room left to answer.

So budget it explicitly, in this order:

1. Reserve the output first. Set ChatOptions.MaxOutputTokens to what a good answer actually needs, and subtract it from the window before anything else.
2. Fixed costs next. The system prompt and tool schemas are paid on every single call. A 900-token system prompt on a chatty endpoint is a standing charge, not a one-off.
3. Retrieved context. Bounded by however many chunks you decided to include.
4. Conversation history last — because it is the only part you can safely trim.

Counting tokens in .NET

Do not estimate by dividing a character count by four. Microsoft.ML.Tokenizers gives you the real BPE vocabularies in-process, with no network call.

using Microsoft.ML.Tokenizers;
 
// Requires the matching data package: O200kBase for the gpt-4o family,
// Cl100kBase for the gpt-4 and gpt-3.5 family.
Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("gpt-4o");
 
const string prompt = "Summarise the following changelog for a release note.";
 
int count = tokenizer.CountTokens(prompt);
IReadOnlyList<int> ids = tokenizer.EncodeToIds(prompt);
 
Console.WriteLine($"{count} tokens, {prompt.Length} characters");
Console.WriteLine(string.Join(' ', ids));

CountTokens is the cheap call and the one to use in a budget check; EncodeToIds is what you want when a count surprises you and you need to see where the splits actually landed.

Wrapping that in a budget gives you something you can assert on:

using Microsoft.Extensions.AI;
using Microsoft.ML.Tokenizers;
 
/// <summary>Fits a conversation into a window, newest turns first.</summary>
public sealed class PromptBudget(Tokenizer tokenizer, int contextWindow, int reservedForOutput)
{
    // Every message carries a few tokens of role framing on top of its text. The
    // exact number is provider-specific; four is a deliberate over-estimate.
    private const int PerMessageOverhead = 4;
 
    public int Count(ChatMessage message) =>
        tokenizer.CountTokens(message.Text ?? string.Empty) + PerMessageOverhead;
 
    public IReadOnlyList<ChatMessage> Fit(ChatMessage system, IReadOnlyList<ChatMessage> history)
    {
        int remaining = contextWindow - reservedForOutput - Count(system);
 
        // Walk backwards: the most recent turns are the ones worth keeping.
        var kept = new List<ChatMessage>();
        for (int i = history.Count - 1; i >= 0; i--)
        {
            int cost = Count(history[i]);
            if (cost > remaining)
            {
                break;
            }
 
            remaining -= cost;
            kept.Add(history[i]);
        }
 
        kept.Reverse();
        kept.Insert(0, system);
        return kept;
    }
}

Two things are worth stealing from this. It reserves the output before spending anything, so a long conversation degrades by dropping old turns rather than by returning a truncated answer. And it is a plain class over ChatMessage with no provider dependency, so it is unit-testable without a model — which is most of what makes AI code testable at all.

Cost and latency

Billing is per token, quoted per million, and input and output are priced separately — output typically several times higher than input. Two consequences follow, and neither is visible in a demo.

A conversation costs quadratically. There is no server-side memory, so turn n resends turns 1 through n−1. Ten turns of 500 tokens each is not 5,000 billed input tokens, it is roughly 27,500. The mitigations are the ones above — trim history, keep the system prompt short — plus provider-side prompt caching, which discounts a repeated prefix. That is the reason to put stable content (system prompt, tool schemas) first and volatile content last: a cache hit needs an identical prefix.

Latency splits in two. Time to first token is dominated by input length; total time is dominated by output length, because output is generated serially. A 1,000-token answer takes roughly ten times as long to finish as a 100-token one, no matter how short the question was. So:

• Cap MaxOutputTokens and ask for terse output. It is a latency control as much as a cost control.
• Stream when a human is waiting. It does not reduce total time; it moves the perceived wait to time-to-first-token.

using Microsoft.Extensions.AI;
 
var options = new ChatOptions { MaxOutputTokens = 300, Temperature = 0.2f };
 
await foreach (ChatResponseUpdate update in
    client.GetStreamingResponseAsync(messages, options, cancellationToken))
{
    Console.Write(update.Text);
}

Next: the step that turns the model's distribution into the tokens you are billed for, and the settings on it that decide whether an answer is reproducible enough to build on.