How language models work

A language model is an unusual dependency to take. It has no schema, its error behaviour is to answer confidently rather than to throw, and it will give you two different strings for the same input. None of that is a defect to be patched around. All of it falls out of one mechanism, and once you have the mechanism the rest of this course is ordinary engineering.

Prediction, not retrieval

Give a model a sequence of tokens and it returns one thing: a probability distribution over every token in its vocabulary for the position that comes next. That is the whole of it. For a ~200,000-token vocabulary, one forward pass produces ~200,000 numbers summing to 1.

Generating a sentence is that step run in a loop. Pick a token from the distribution, append it to the sequence, run the model again on the longer sequence, pick again. This is what autoregressive means, and it has three consequences you will feel immediately:

• Output is serial. The tenth token cannot be computed before the ninth, which is why streaming exists and why output tokens dominate latency.
• The model re-reads everything, every token. Cost scales with the length of the conversation, not just with the length of your question.
• Nothing is looked up. There is no index, no row, no document. There is a function from a token sequence to a distribution.

The picking step is not part of the model. It is sampling code that sits after it, and it is where Temperature, TopP and Seed act. At temperature 0 you take the most probable token every time; raise it and you flatten the distribution and let less likely tokens through. This matters more than it sounds: the model is deterministic, the sampler is not. When you ask for reproducibility later, it is the sampler you are negotiating with, not the weights.

What training produces

Training produces weights — some tens of billions of floating-point numbers — and nothing else. It does not produce a copy of the corpus, an index into it, or any record of what was read.

Two stages get you there:

1. Pretraining. Predict the next token across a very large body of text. This is where the model acquires syntax, idiom, code structure and a lossy, smeared-out impression of a great many facts.
2. Post-training. Instruction tuning and preference optimisation, which teach the model to treat a prompt as a request to be satisfied rather than a passage to be continued. A pretrained-only model asked "How do I add a hosted service?" is quite likely to reply with three more questions, because that is what such text usually looks like.

Facts survive this process the way a compressed image survives compression: recognisably, unevenly, and with no way to tell reconstruction from recall. That gives you four properties to design around.

The fix for all four is the same and it is not a better model: put the text in the prompt. That is the entire premise of retrieval-augmented generation, which is a later course in this track. Here it is enough to know why it has to exist.

Consequences for your code

The API shape follows directly from the mechanism. You send a list of messages, you get tokens back, and everything the model knows about your problem is in that list. There is no session, no server-side memory, no handle to a previous call. A "conversation" is your application resending the history each turn.

In .NET the abstraction over that is IChatClient, from Microsoft.Extensions.AI. It is the HttpClient of this space: one interface, many providers, and middleware in between.

using Microsoft.Extensions.AI;
 
public sealed class SummariseCommand(IChatClient client)
{
    public async Task<string> RunAsync(string changelog, CancellationToken ct)
    {
        ChatResponse response = await client.GetResponseAsync(
            [
                new ChatMessage(ChatRole.System, "You are a release-notes editor. Be terse."),
                new ChatMessage(ChatRole.User, changelog),
            ],
            cancellationToken: ct);
 
        // Usage is nullable: not every provider reports it, and cached or
        // filtered responses may report it partially.
        Console.WriteLine($"in={response.Usage?.InputTokenCount} out={response.Usage?.OutputTokenCount}");
 
        return response.Text;
    }
}

Note what the class depends on. Not a provider SDK, not an HTTP client, not an API key — just IChatClient. That is deliberate, and it is what makes the next lesson's token accounting and the last lesson's failure handling something you can add without touching this file.

Registering a client

Provider choice is one line, at the composition root, and everything after it is middleware.

using System.ClientModel;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using OpenAI;
 
var builder = Host.CreateApplicationBuilder(args);
 
// The only provider-specific line in the application.
IChatClient provider = new OpenAIClient(new ApiKeyCredential(builder.Configuration["Ai:Key"]!))
    .GetChatClient("gpt-4o-mini")
    .AsIChatClient();
 
builder.Services
    .AddChatClient(provider)
    .UseLogging()          // prompts and responses at Trace — never at Information
    .UseOpenTelemetry();   // token counts and durations as real metrics
 
var host = builder.Build();

AddChatClient returns a builder, and each Use… call wraps the client in a decorator. Caching, function invocation and telemetry all arrive the same way. It is the same pipeline idea as ASP.NET Core middleware, applied to a model call — which means the interesting production concerns are composable rather than scattered.

Finally, the property that surprises people. Run the same request twice:

var options = new ChatOptions { Temperature = 0f, MaxOutputTokens = 200 };
 
string first = (await client.GetResponseAsync("Summarise this changelog.", options)).Text;
string second = (await client.GetResponseAsync("Summarise this changelog.", options)).Text;
 
// Very likely similar. Not guaranteed equal — not even at temperature 0.
Console.WriteLine(first == second);

Temperature 0 removes the sampler's randomness but not the rest: floating-point reduction order varies with server-side batching, and providers reroute traffic between hardware and model revisions without telling you. Treat exact-match assertions on model output as tests that will fail on a Tuesday for no reason. The last lesson in this course is about what to assert instead.

Next: what a token actually is, and why the context window is a budget you have to manage rather than a limit you occasionally hit.