Capabilities and limits

Everything so far has been mechanism. This lesson is the engineering: what a model is reliable at, the specific ways it fails, and what you build around it before it is allowed near a user. If you take one thing from this course into a code review, take this lesson.

What models do well

There is a single heuristic that predicts reliability better than any benchmark:

The closer the answer already is to the input, the more reliable the call.

Transformation of text you supplied is the strong case. Recall of facts you did not supply is the weak one — that is the lossy-compression property from lesson 1, and no amount of prompting repairs it.

The design move that follows is the same every time: shorten the distance between input and answer. Retrieve the document and let the model summarise it rather than asking what the document says. Pass the price from your database rather than asking for it. Do the arithmetic in C#. Almost every serious AI feature is a conventional system that gathers facts, with a model doing the last step of expressing them.

Failure modes

Fabrication. The model produces a fluent, confident, specific, wrong answer — an API that does not exist, a section number that was never written, a citation to a plausible paper. There is no internal signal distinguishing this from a correct answer, because there is no distinction inside the mechanism: both are high-probability continuations. Fluency is not evidence, and self-reported confidence is just more generated text.

Staleness. Weights are frozen at training time. Anything newer — a release from last month, your codebase, today's price — is unknown unless you put it in the prompt.

Prompt injection. Every token in the context is treated alike, so text you retrieved or a user pasted can carry instructions. This is not a filtering problem you can solve with a blocklist; it is the direct consequence of instructions and data sharing one channel. The mitigations are architectural: give the model the least privilege that works, never let its output authorise an action on its own, and treat everything it emits as untrusted input — never concatenate it into SQL, a shell command, a file path or raw HTML.

Silent truncation. Hit MaxOutputTokens mid-sentence and you get a well-formed HTTP 200 containing a partial answer. Check response.FinishReason rather than assuming completion.

Non-determinism. Two identical calls can return different text, and temperature 0 narrows that without closing it — Temperature, sampling and determinism is the whole of that story, and the reason the controls below assert on properties rather than on strings.

Containing them in production

Ask for a shape, then validate it. Free text is unverifiable; a schema is not. Structured output is the containment that does the most work per line of code. Microsoft.Extensions.AI has a generic overload that requests structured output and deserialises the reply.

using System.ComponentModel;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Logging;
 
public sealed class ChangelogSummary
{
    [Description("One sentence, under 100 characters.")]
    public required string Headline { get; init; }
 
    [Description("Three to five user-visible changes.")]
    public required IReadOnlyList<string> Highlights { get; init; }
 
    public required bool ContainsBreakingChange { get; init; }
}
 
public sealed class ChangelogSummariser(IChatClient client, ILogger<ChangelogSummariser> logger)
{
    private static readonly ChatOptions Options = new() { Temperature = 0f, MaxOutputTokens = 500 };
 
    public async Task<ChangelogSummary?> SummariseAsync(string changelog, CancellationToken ct)
    {
        ChatResponse<ChangelogSummary> response =
            await client.GetResponseAsync<ChangelogSummary>(changelog, Options, cancellationToken: ct);
 
        // An unparseable response is a failed call, not a partial success.
        if (!response.TryGetResult(out ChangelogSummary? summary))
        {
            logger.LogWarning("Model output rejected. Raw: {Raw}", response.Text);
            return null;
        }
 
        // The schema constrains the shape. It does not constrain the content —
        // domain rules are still yours to enforce.
        return summary.Highlights.Count is >= 3 and <= 5 ? summary : null;
    }
}

The [Description] attributes are not comments: they are serialised into the schema the model is given, so they are prompt text with a compiler checking where they hang.

Bound the output space wherever you can. The routing example in Embeddings vs completions is the pattern — when the answer must be one of n known things, make the model's job a selection rather than a generation, so an invalid answer is unrepresentable.

Ground answers in text you supplied, and require citations back into it, so a claim without a source is a rejectable output rather than a judgement call. That is the whole subject of the RAG course later in this track.

Put a ceiling on the call. The model has no idea what your budget is, so the guard belongs in the pipeline:

using Microsoft.Extensions.AI;
using Microsoft.ML.Tokenizers;
 
/// <summary>Rejects a request that would exceed a per-call input ceiling.</summary>
public sealed class BudgetGuardChatClient(
    IChatClient innerClient,
    Tokenizer tokenizer,
    int maxInputTokens) : DelegatingChatClient(innerClient)
{
    public override Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        int total = messages.Sum(message => tokenizer.CountTokens(message.Text ?? string.Empty));
        if (total > maxInputTokens)
        {
            throw new InvalidOperationException(
                $"Prompt is {total} tokens; the ceiling is {maxInputTokens}.");
        }
 
        return base.GetResponseAsync(messages, options, cancellationToken);
    }
}

Registered alongside the rest of the pipeline, with timeouts and retries handled where they belong — at the transport:

builder.Services
    .AddChatClient(provider)
    .UseDistributedCache()   // identical prompt + options → no second call
    .UseLogging()
    .UseOpenTelemetry()
    .Use(inner => new BudgetGuardChatClient(inner, tokenizer, maxInputTokens: 8_000));
 
// Model endpoints are slow and occasionally unavailable. Give them a real
// resilience policy rather than the default HttpClient timeout.
builder.Services.AddHttpClient("ai").AddStandardResilienceHandler();

Test properties, not strings. Since output varies, assert the things that must hold: the response deserialises; every cited id exists in the supplied context; Highlights has three to five entries; no answer exceeds the length budget; a question with no supporting context produces a refusal. Run those over a fixed set of recorded inputs and treat a drop in pass rate as a regression — that is what the Microsoft.Extensions.AI.Evaluation packages automate, and it is a course of its own.

Instrument in tokens and money. UseOpenTelemetry() gives you token counts and durations per call; put an alert on cost per request and on the rejection rate from your validators. A rising rejection rate is usually the first visible sign that a provider silently updated a model.

Keep a human where being wrong is expensive. Not as a disclaimer in the footer, but as a step in the workflow: a draft to approve, a suggestion to accept, an action that needs a click. Everything above reduces the rate of bad output. Nothing reduces it to zero, and designing as if it did is the single most common way an AI feature goes wrong in production.

That is the course. You now know what the thing does, what it costs, how it breaks, and roughly what has to sit around it. The rest of this track is depth on each of those — starting with prompts as code rather than folklore.