LLM Inference Serving: Essential Notes

An application sends tokens to a model API and receives tokens back. Below that interface, the serving system has to retain attention state, place it in scarce memory, decide which requests run together, and route later requests toward reusable work.

This guide builds one mental model for that layer. The goal is not to implement GPU kernels. It is to understand enough serving mechanics to design better agent transcripts, interpret latency and cache metrics, and reason about the behaviour of model providers.

1. The Whole Request Path

A request passes through several decisions before output appears:

application constructs messages and tools
        ↓
provider tokenizes the request
        ↓
router chooses an inference worker
        ↓
worker looks for reusable prefix blocks
        ↓
prefill processes the uncached prompt suffix
        ↓
request joins the decode scheduler
        ↓
model produces one or more new tokens per step
        ↓
finished request releases or retains its cache blocks

The stages interact. A router may choose a busy worker because it already holds most of the request's prefix. An idle worker could begin immediately, but would have to repeat the prefill. Keeping a cache entry improves reuse, but consumes memory that could admit another request.

The system is therefore optimizing several quantities at once:

• time to first token (TTFT): delay before generation begins;
• inter-token latency (ITL): delay between generated tokens;
• throughput: tokens or requests completed per unit time;
• memory capacity: how many active and reusable sequences fit;
• cache reuse: how much prompt computation can be skipped;
• fairness: whether large or cache-friendly requests starve other work.

No single scheduling policy maximizes all of them.

2. Prefill and Decode

Autoregressive generation has two phases.

Prefill

During prefill, the model processes the input prompt. The tokens are already known, so the hardware can process many of them in parallel.

[system][tools][history][new user message]
<--------------- prefill ---------------->

Prefill creates the attention state required for generation. Longer uncached prompts mean more prefill work and usually a longer TTFT.

Decode

During decode, the model generates the next token, appends it, then generates the next one:

prompt → token 1 → token 2 → token 3 → ...

Each step depends on the preceding token, so the steps are sequential for one sequence. A serving engine gets useful parallelism by decoding many independent sequences together.

Why the distinction matters

Prefill and decode have different workloads:

• prefill handles many known tokens and is relatively compute-heavy;
• decode handles a small number of new tokens per sequence and repeatedly reads the accumulated attention state;
• prompt caching primarily removes repeated prefill work;
• continuous batching primarily keeps decode capacity occupied across requests.

A provider may even place prefill and decode on separate worker pools. That is an infrastructure choice, not a change to the model's logical operation.

3. What the KV Cache Contains

Self-attention starts from a representation for each token and produces three vectors at every attention layer:

• a query, describing what the current token is looking for;
• a key, describing what a token can be matched against;
• a value, containing information that can be retrieved from that token.

For a newly generated token, the model creates a query and compares it with the keys of preceding tokens. The resulting scores determine a weighted mixture of their values.

The keys and values for earlier tokens do not change during ordinary autoregressive generation. Recomputing them at every decode step would repeat the same work. The serving engine therefore retains them in the KV cache.

earlier tokens:  K₁,V₁  K₂,V₂  K₃,V₃ ... Kₙ,Vₙ
new token:       Qₙ₊₁
                  ↓ compare with earlier keys
                attention-weighted mixture of earlier values

Queries are transient. The cache stores the earlier keys and values needed by future queries.

A useful size formula

For a conventional transformer, an approximate KV-cache size for one sequence is:

2 × layers × KV heads × head dimension × cached tokens × bytes per element
↑
keys and values

For an illustrative model with 32 layers, 8 KV heads, head dimension 128 and two-byte cache elements:

10,000 cached tokens ≈ 1.22 GiB
100,000 cached tokens ≈ 12.2 GiB

The exact footprint depends on the model architecture, precision, sliding or local attention, and serving implementation. Multi-query and grouped-query attention reduce the number of KV heads specifically to reduce this repeated state.

The important scaling law is simple: KV memory grows with cached sequence length, number of layers, KV width, and number of concurrent sequences.

Three caches that should not be confused

An application artefact cache changes what must enter the prompt. A prompt cache avoids reprocessing an unchanged prefix. They solve different problems.

4. Prefix Reuse

After a generation finishes, a later request may begin with the same tokens:

request 1: [system][tools][conversation........]
request 2: [system][tools][conversation........][new]
           <--------- reusable prefix --------><--->

If the corresponding KV state remains available, the serving system can reuse the matching prefix and prefill only the new suffix.

The match is over tokens, not meaning. These prompts may mean the same thing but produce different caches:

"Use the read tool, then answer."
"Answer after using the read tool."

A difference in the middle ends reuse at that point. All later tokens are now a different continuation.

Block-based prefix caching

Serving engines commonly divide the token sequence and its KV state into blocks. A block can be identified from:

• the tokens inside the block;
• the identity of the preceding prefix;
• model and configuration details that affect the generated state.

This forms a chain:

block A hash
    ↓
hash(block B tokens + A prefix)
    ↓
hash(block C tokens + A,B prefix)

Two requests with the same initial blocks can refer to the same cached state. The first differing block starts a new chain.

Application consequences

Early prompt components should be deterministic:

• keep system prompts stable;
• serialize tools in a consistent order;
• avoid timestamps or random identifiers near the beginning;
• treat tool-schema changes as cache-breaking changes;
• append new conversation state instead of rewriting old messages;
• make compaction an intentional cache reset;
• record prompt and tool-loadout versions with cache metrics.

The smallest prompt is not always the cheapest prompt. Deleting a small result from the middle can force a large surviving suffix to be processed again.

5. Affinity and Prefix-Aware Routing

Once several workers serve the same model, the router must choose where each request goes.

Affinity hint

An affinity hint asks the router to prefer the worker associated with an earlier request:

request(session-42)       → router → worker 7
next request(session-42)  → router → worker 7, if practical

This resembles a sticky session. The session or cache key is a routing hint, not proof that worker 7 still holds a valid prefix. The entry may have expired, been evicted, or belonged to a different token sequence on another branch.

The router may also ignore affinity because the preferred worker is unhealthy or overloaded.

Prefix-aware routing

A more informed router estimates which exact prefix blocks each worker holds:

incoming blocks: A B C D E

worker 1: A B C D      80% overlap, busy
worker 2: A B          40% overlap, idle
worker 3: none          0% overlap, lightly loaded

Workers can publish events when blocks are created or evicted. A global index then maps block hashes to likely locations. Systems without precise events may predict cache contents from earlier routing decisions and expire those predictions after a time limit.

Affinity says, "try the same worker." Prefix-aware routing says, "estimate the reusable computation on every worker."

6. The Scheduling Tradeoff

Cache locality is only one part of the routing decision.

Suppose worker 1 holds the whole prefix but has a long queue. Worker 2 is idle but holds nothing:

worker 1 completion estimate
    = queue delay + short cached prefill + decode

worker 2 completion estimate
    = no queue delay + full prefill + decode

For a short prompt, worker 2 may be faster. For a very long prompt, waiting for worker 1 may win. A scheduler can approximate this by scoring both reusable prefix work and current decode load.

The tradeoff also appears in latency metrics:

• preferring cache overlap often improves TTFT;
• concentrating work on cache-rich workers can worsen queueing and ITL;
• spreading work evenly improves balance but may repeat expensive prefills;
• strict affinity simplifies routing but can create hot workers.

This is why an affinity key cannot sensibly guarantee worker placement.

7. Eviction

KV memory is finite. When there is not enough space for new work, the runtime must free, move, or recompute some state.

There are two related decisions:

1. Active-request pressure: which requests are admitted, paused or preempted when their live state does not fit?
2. Reusable-cache pressure: which completed or inactive prefix blocks are retained for possible future hits?

Possible eviction signals include:

• least recently used blocks;
• expired retention time;
• low predicted chance of reuse;
• large memory footprint;
• request priority;
• pressure from active sequences;
• whether another copy exists in a slower memory tier.

Eviction is not necessarily an error. Keeping every old prefix would eventually prevent new requests from running.

What an API client can know

An application can often identify:

• its own prompt or tool-schema change;
• a model or provider switch;
• compaction or branch navigation;
• an idle gap exceeding a documented retention period.

It usually cannot prove that a provider evicted a cache block or routed the request to another GPU. Report such cases as unexplained provider-side misses, not diagnosed evictions.

8. Memory Tiers

The fastest memory is also the scarcest. Serving systems can extend the useful cache capacity by moving state through slower tiers:

accelerator memory   fastest access, highest pressure
        ↓
host memory          larger, transfer required
        ↓
local storage        larger again, much slower
        ↓
remote storage       shareable across nodes, network involved

Moving cached state is useful only when restoration is cheaper than recomputation and does not delay more valuable work.

Two common patterns are:

• offloading: move colder KV state out of accelerator memory and restore it before reuse;
• distributed cache: make blocks discoverable or transferable across workers so requests are less tightly tied to one device.

A cache hit is therefore not a single performance class. A hot accelerator hit, a host-memory restoration and a remote transfer can all avoid recomputation but have different latency.

For an application engineer, the main lesson is to avoid interpreting cached_tokens as a complete latency prediction.

9. Continuous Batching

Traditional static batching waits for a group of requests, runs the group, then finishes the batch. That is awkward for language generation because sequences have different prompt and output lengths.

Continuous batching, also called in-flight or iteration-level batching, reconsiders the active batch at generation boundaries:

step 1: A B C
step 2: A B C D    D is admitted
step 3: A   C D    B has finished
step 4: A E C D    E uses the free capacity

Each active sequence contributes its next decode token. Finished requests leave and waiting requests enter without waiting for the longest sequence in an old batch to finish.

Why it helps

• the device spends less time underutilized;
• short requests are not permanently coupled to long ones;
• the server can sustain more concurrent traffic;
• scheduling can respond to changing KV-memory capacity.

What it does not guarantee

Continuous batching primarily improves fleet throughput. A single request can still slow down under contention because it shares steps and memory bandwidth with other requests.

This explains why two stable application metrics should be kept separate:

• TTFT, dominated by routing, queueing and prefill;
• ITL or output tokens per second, dominated by decode scheduling and load.

10. Paged Attention

The KV cache grows one token at a time and different sequences stop at different lengths. Requiring one large contiguous memory region per sequence creates two forms of waste:

• reservation waste: capacity is held for tokens that may never be generated;
• fragmentation: free memory exists but not in a convenient contiguous region.

PagedAttention applies an operating-system idea. The logical KV sequence is divided into fixed-size blocks that can live in non-contiguous physical memory:

logical sequence: [block 0][block 1][block 2][block 3]
                       │       │       │       │
physical memory:    slot 7  slot 19  slot 4  slot 12

A block table maps logical positions to physical slots. The attention kernel uses that mapping when reading earlier keys and values.

This enables:

• incremental allocation as sequences grow;
• less fragmentation and over-reservation;
• block-level freeing and eviction;
• sharing identical prefix blocks between requests;
• copy-on-write behaviour when two sequences share a prefix and then diverge.

PagedAttention is not the only possible memory-management design. Its enduring lesson is that efficient serving requires dynamic KV allocation rather than pretending every sequence has one fixed, contiguous maximum-sized buffer.

11. One Worked Trace

Consider an agent session with a stable 60,000-token prefix and a short new user message.

Healthy reuse

1. The application sends the same system prompt, tools and history plus the new suffix.
2. The router finds that worker 4 holds most prefix blocks and that its queue is acceptable.
3. Worker 4 reuses those blocks and prefills only the suffix.
4. The scheduler admits the request into an active decode batch.
5. As other requests finish, new requests join subsequent steps.
6. The completed prefix blocks remain available until pressure or retention policy removes them.

Accidental reset

1. An extension inserts a timestamp into the system prompt.
2. The first token mismatch appears near the start.
3. Existing conversation blocks no longer represent a matching prefix.
4. The router gains little from cache-aware placement.
5. The worker repeats almost the entire prefill.
6. TTFT and uncached-input cost jump even though the visible conversation barely changed.

This is why prefix stability is an application-level correctness and cost property, even though the KV tensors live below the API.

12. What to Measure Above the API

Useful request-level fields include:

Do not label every miss as provider failure. First separate:

known application reset
likely retention expiry
model or provider boundary
unexplained miss

The provider owns routing, eviction and physical placement. The application owns deterministic payload construction and honest observability.

13. The Application-Engineer Stopping Point

An application or agent engineer should be able to explain this chain:

append message
  → match exact prefix blocks
  → route using cache overlap and worker load
  → restore or allocate KV memory
  → prefill uncached tokens
  → join a changing decode batch
  → retain, move or evict blocks

That is enough to:

• design cache-stable prompts and tool protocols;
• reason about compaction economics;
• interpret cache, TTFT and decode metrics;
• ask sensible questions of an inference provider;
• participate in application-level system design.

Implementation knowledge such as CUDA kernels, exact tensor strides, interconnect topology, distributed block transfer protocols and scheduler tuning is optional unless the role owns inference infrastructure.

14. Compact Reference

Sources

• Earendil Engineering, Prompt Caching In Agents, 2026.
• Woosuk Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023.
• vLLM, Automatic Prefix Caching implementation.
• NVIDIA Dynamo, KV-aware routing and router guide.
• NVIDIA TensorRT-LLM, scheduler and in-flight batching.
• Hugging Face Transformers, cache strategies.