Temperature, sampling and determinism

The model's entire output is a probability distribution. Picking one token out of it is a separate step, run by ordinary code that you configure and the model never sees. Every knob on ChatOptions that sounds like a personality setting — Temperature, TopP, Seed — acts in that step, which is why none of them can make the model know anything it did not already know.

This lesson is about that step: what the knobs do, which values are a policy rather than a preference, and what "deterministic" can and cannot mean when the thing you are calling is somebody else's fleet.

From a distribution to one token

A forward pass ends with one raw score — a logit — per entry in the vocabulary. Softmax turns those scores into probabilities that sum to

1. Temperature divides the scores before the exponential, so it decides how much the gaps between them matter:

/// <summary>Turns raw model scores into probabilities, warped by temperature.</summary>
static double[] Softmax(ReadOnlySpan<double> logits, double temperature)
{
    // Temperature 0 is not a value this formula accepts — it is a division by
    // zero. Providers special-case it to "take the highest score", which is why
    // 0 means greedy rather than "very low randomness".
    ArgumentOutOfRangeException.ThrowIfNegativeOrZero(temperature);
 
    double max = double.NegativeInfinity;
    foreach (double logit in logits)
    {
        max = Math.Max(max, logit / temperature);
    }
 
    var weights = new double[logits.Length];
    double sum = 0;
    for (int i = 0; i < logits.Length; i++)
    {
        // Subtracting the maximum before exponentiating is the standard guard
        // against overflow; it cancels out in the division below.
        weights[i] = Math.Exp(logits[i] / temperature - max);
        sum += weights[i];
    }
 
    for (int i = 0; i < weights.Length; i++)
    {
        weights[i] /= sum;
    }
 
    return weights;
}

Four candidate tokens scoring 3, 2, 1 and 0 come out of that like this:

Nothing about the model changed between those columns. The same forward pass, the same scores, three different odds of an unusual word — and the unusual word is where both the interesting phrasing and the fabricated API signature come from.

The knobs, and what each one does

ChatOptions is the provider-independent surface for all of it. Every property is nullable, and null means "whatever the provider's default is" — which is not necessarily the same default next quarter.

Two of those interact badly. Temperature and TopP both narrow or widen the same distribution, so tuning both leaves you unable to say which one produced a change. Pick one, pin the other to its neutral value — TopP = 1f when you are tuning temperature.

Top-p is worth a concrete look, because it is the one whose behaviour changes with the distribution rather than with your setting. At TopP = 0.9f and the T = 1.0 column above, the running total reaches 0.644, then 0.881, then 0.968 — so the nucleus is quick, brown and lazy, renormalised between them, and purple is not merely unlikely but unreachable. On a flatter distribution the same 0.9 might keep two hundred tokens. That is the point of it: it cuts the tail, not a fixed count.

Settings are a policy, not a preference

The useful question is never "what temperature feels right". It is who consumes this output — a parser or a person:

using Microsoft.Extensions.AI;
 
/// <summary>The two policies most features actually need, named once.</summary>
public static class SamplingPolicies
{
    /// <summary>Anything a machine will parse: extraction, classification, routing.</summary>
    public static ChatOptions Deterministic(int maxOutputTokens) => new()
    {
        Temperature = 0f,       // greedy: always take the highest-scoring token
        TopP = 1f,              // neutral, because Temperature is the knob in use
        Seed = 42,              // a request, not a guarantee
        MaxOutputTokens = maxOutputTokens,
    };
 
    /// <summary>Anything a human will read and choose between.</summary>
    public static ChatOptions Drafting(int maxOutputTokens) => new()
    {
        Temperature = 0.8f,
        FrequencyPenalty = 0.3f,   // variety across a long answer, not within a word
        MaxOutputTokens = maxOutputTokens,
    };
}

Read that as a rule with two branches. Temperature 0 for anything a machine will consume, because variety in a field name is not creativity, it is a parse error. A higher temperature only where a human is picking between alternatives, and where seeing the same suggestion twice would be the failure.

The common mistake is reaching for a middle value — 0.3, say — as a compromise. It is not one. It is a small chance of an unusual token on every token of a long answer, which is exactly enough randomness to make a bug irreproducible and not nearly enough to make prose interesting.

What determinism is available

Temperature 0 removes the sampler's randomness. It does not make the call reproducible, and the difference matters the first time a test fails on a Tuesday.

Three things move underneath you:

• Floating-point reductions are not associative. The order in which sums are accumulated on the GPU depends on how your request was batched with other people's, and a different order can flip two near-tied logits. Greedy decoding then picks a different token, and the rest of the answer diverges from there.
• The fleet is heterogeneous. Providers route between hardware generations, and quantisation or kernel differences change the arithmetic.
• Model aliases move. gpt-4o is a pointer. Pin the dated version string in anything you need to hold still, and treat a provider's model update as a dependency upgrade — because that is what it is.

Seed is worth setting anyway. It is a best-effort request that a provider may honour, ignore, or honour only within one revision of one deployment; where it is honoured it removes one source of variation, and where it is not you have lost nothing. Never build a test on it holding.

What you can have is a record good enough to reason from:

using Microsoft.Extensions.AI;
 
/// <summary>Everything needed to argue about an output after the fact.</summary>
public sealed record CallRecord(
    string? ModelId,          // response.ModelId — what actually served the call
    float? Temperature,
    float? TopP,
    long? Seed,
    ChatFinishReason? FinishReason,
    long? InputTokens,
    long? OutputTokens,
    string Text);
 
public static class CallRecordFactory
{
    public static CallRecord From(ChatResponse response, ChatOptions options) => new(
        response.ModelId,
        options.Temperature,
        options.TopP,
        options.Seed,
        response.FinishReason,
        response.Usage?.InputTokenCount,
        response.Usage?.OutputTokenCount,
        response.Text);
}

Store that beside any output you keep. Without the model id and the options, a bug report about a bad answer is unreproducible in principle — you cannot even tell whether the model that produced it still exists.

Testing against a sampled dependency

Non-determinism is not a reason to skip tests. It is a reason to assert different things, at two levels.

Unit tests should not call a model at all. IChatClient is an interface; substitute it and your budgeting, parsing and validation logic becomes ordinary deterministic code with ordinary tests:

using Microsoft.Extensions.AI;
 
/// <summary>Returns canned responses in order, and records what it was asked.</summary>
public sealed class StubChatClient(params string[] responses) : IChatClient
{
    private int _call;
 
    public List<ChatOptions?> Requests { get; } = [];
 
    public Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        Requests.Add(options);
        return Task.FromResult(new ChatResponse(new ChatMessage(ChatRole.Assistant, responses[_call++])));
    }
 
    public IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default) => throw new NotSupportedException();
 
    public object? GetService(Type serviceType, object? serviceKey = null) => null;
 
    public void Dispose() { }
}

That stub is also how you test the thing this lesson is about: assert that your extraction path sent Temperature = 0f. A sampling policy that is only a comment is a sampling policy that will be edited away.

Tests that do call a model assert properties, not strings. Over a fixed set of recorded inputs, check the things that must hold whatever the sampler did: the response deserialises; every cited id appears in the supplied context; a required field is non-empty; the answer stays inside its length budget; a question with no supporting context produces a refusal rather than a guess. Score the set, store the pass rate, and treat a drop as a regression — a single run tells you almost nothing about a distribution.

Next: the other call in the API — the one that returns numbers instead of prose, has no sampler at all, and costs roughly a hundredth as much.