Embeddings vs completions

Two lessons in, "calling a model" has meant one thing: send messages, get prose back. There is a second call, it does something completely different, and it is about a hundred times cheaper. Most production AI systems are mostly the second call, with the first one used sparingly at the end.

Two calls, two jobs

A completion runs the autoregressive loop from lesson 1: many forward passes, one per output token, returning text. An embedding call runs a single forward pass and returns a fixed-length vector of floats — 1,536 of them for text-embedding-3-small — positioning the input in a space where geometric closeness approximates similarity of meaning.

The abstraction is the same shape as IChatClient, and it registers the same way:

using System.ClientModel;
using Microsoft.Extensions.AI;
using OpenAI;
 
var openAi = new OpenAIClient(new ApiKeyCredential(builder.Configuration["Ai:Key"]!));
 
builder.Services.AddEmbeddingGenerator(
    openAi.GetEmbeddingClient("text-embedding-3-small").AsIEmbeddingGenerator());

The core method takes a batch, which is the API telling you something: embedding one string per HTTP round trip is the wrong shape. Send them together.

using System.Numerics.Tensors;
using Microsoft.Extensions.AI;
 
public sealed class SemanticSearch(IEmbeddingGenerator<string, Embedding<float>> generator)
{
    public async Task<IReadOnlyList<(string Document, float Score)>> RankAsync(
        string query,
        IReadOnlyList<string> documents,
        CancellationToken ct = default)
    {
        // One call for the query and every document — batching is the whole point.
        GeneratedEmbeddings<Embedding<float>> vectors =
            await generator.GenerateAsync([query, .. documents], cancellationToken: ct);
 
        ReadOnlyMemory<float> queryVector = vectors[0].Vector;
 
        return documents
            .Select((document, i) => (
                Document: document,
                Score: TensorPrimitives.CosineSimilarity(queryVector.Span, vectors[i + 1].Vector.Span)))
            .OrderByDescending(result => result.Score)
            .ToList();
    }
}

Cosine similarity is the angle between two vectors, in [-1, 1]. It ignores magnitude, which is what you want: a two-line note and a two-page document about the same subject should score alike. TensorPrimitives is hardware-accelerated, so brute-forcing a few thousand vectors in memory is genuinely fine — you need a vector database for scale and persistence, not to make the maths fast.

Choosing between them

The question to ask is not "which is better" but "is my output a set of things, or a piece of text?"

Reach for an embedding when the answer is a selection from data you already have:

• Search and retrieval — the R in RAG, the next course but one
• Deduplication, near-duplicate detection, clustering
• Classification and routing by nearest labelled example or centroid
• Recommendation and "more like this"
• Detecting drift, by watching similarity to a reference set over time

Reach for a completion when the answer is text that does not exist yet:

• Summarising, rewriting, translating, extracting into a schema
• Answering in prose from context you supplied
• Judgements that need nuance rather than proximity

A worked example of the difference. To route a support ticket to one of eight queues you could send the ticket and the eight queue descriptions to a completion model and ask it to pick — about 700 input tokens, a second of latency, a small chance of it inventing a ninth queue. Or you could embed the eight descriptions once at startup, embed the ticket, and take the nearest. The second version is faster, cheaper by orders of magnitude, cannot return an invalid queue because the answer is an index into your array, and gives you a similarity score you can threshold to mean "no confident match, send it to a human".

That last property is the one to internalise: an embedding pipeline's output space is bounded by your data. A completion's is not.

Where embeddings are the wrong tool: they capture topical similarity, not truth, polarity or intent. "The deployment succeeded" and "the deployment failed" sit very close together. If negation, ordering or specific numbers matter, similarity will quietly mislead you and you need a completion — or a plain WHERE clause.

Cost and latency

The gap is not marginal. At the time of writing, a small embedding model is priced around two cents per million tokens, against dollars per million for a mid-range chat model's output — two to three orders of magnitude, on an input-only bill, since an embedding produces no output tokens.

Three engineering consequences:

1. Cache aggressively. For a fixed model version an embedding is a pure function of its input, so cache it keyed on a hash of the text plus the model name. Most corpora are re-embedded far more often than they change.
2. Batch, and bound the batch. One request with 200 inputs beats 200 requests. Providers cap both the batch size and the total tokens per request, so chunk your input and use the same PromptBudget habit from Tokens and context windows.
3. Changing embedding model is a migration, not a config change. Vectors from two different models are not comparable — not even different sizes of the same family. Switching means re-embedding the entire corpus and swapping the index atomically. Store the model name and dimension next to every vector, or you will not be able to tell which are stale.

Latency behaves differently too. An embedding call is a single forward pass, so its time depends on input length and nothing else — no serial output phase, no reason to stream. It is fast enough to sit in a request path that a completion call has no business being in.

using System.Runtime.InteropServices;
using System.Security.Cryptography;
using System.Text;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Caching.Distributed;
 
public sealed class CachedEmbeddings(
    IEmbeddingGenerator<string, Embedding<float>> generator,
    IDistributedCache cache)
{
    private const string ModelVersion = "text-embedding-3-small";
 
    public async Task<ReadOnlyMemory<float>> GetAsync(string text, CancellationToken ct = default)
    {
        // The model name is part of the key: vectors from two models never mix.
        string key = $"emb:{ModelVersion}:{Convert.ToHexString(
            SHA256.HashData(Encoding.UTF8.GetBytes(text)))}";
 
        byte[]? cached = await cache.GetAsync(key, ct);
        if (cached is not null)
        {
            return MemoryMarshal.Cast<byte, float>(cached).ToArray();
        }
 
        GeneratedEmbeddings<Embedding<float>> result =
            await generator.GenerateAsync([text], cancellationToken: ct);
 
        ReadOnlyMemory<float> vector = result[0].Vector;
        await cache.SetAsync(key, MemoryMarshal.AsBytes(vector.Span).ToArray(), ct);
        return vector;
    }
}

Next: which model and which library actually make those two calls for you — the first decision on a .NET project, and the one worth keeping reversible.