From Prompt to Token: The Life of an LLM Request

You send a chatbot a 14-word question and get a paragraph back. In between, the server turned your message into 43 tokens, read all of them in a single pass of the model, then ran the whole model again for every further token of the answer, 92 times. We followed one request through a real server and timed each stage: the first token arrived after 79 milliseconds, the last after 6.8 seconds, and nearly all of that time went to writing the answer, not to reading the question.

The path in one picture

An inference server is two programs and a GPU. The API server speaks HTTP: it receives your request, prepares it, and streams the answer back. The engine core decides which requests the GPU works on at each moment and runs the model. In vLLM they are separate processes, and the server's own log tags every line with which one wrote it:

(APIServer pid=860067) INFO ... "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=860196) INFO ... GPU KV cache size: 497,744 tokens

The request we followed: Qwen3-8B, an 8-billion-parameter open model, served by vLLM 0.31.0 on an NVIDIA GB10, with a short system message and one question. Each number below is the median of five runs, with nothing else on the GPU unless we say so.

Messages become one string

A model doesn't receive a list of messages. It receives one sequence of text, and the chat template, a small program shipped with each model, builds it from your messages. Here is what Qwen3-8B's template produced for our request, line breaks included:

<|im_start|>system
You are a concise assistant for infrastructure engineers.<|im_end|>
<|im_start|>user
In one paragraph, why does an LLM server stream its answer token by token?<|im_end|>
<|im_start|>assistant
<think>

</think>

The markers <|im_start|> and <|im_end|> tell the model where each turn starts and ends. The text stops right after assistant, so the most likely continuation is the assistant's answer. The empty <think> block is how this model is told to skip its reasoning step: we sent enable_thinking: false, and the template wrote an empty block for it.

Each model has its own template, and the model only behaves as trained when it gets the exact layout it was trained on. A server applying the wrong template still answers, just worse, which makes the mistake hard to spot.

The string becomes tokens

The model works on tokens: pieces of text, each with a number in a fixed vocabulary. Qwen3-8B's vocabulary has 151,936 entries. Our 14-word question, alone, became 17 tokens:

In | ␣one | ␣paragraph | , | ␣why | ␣does | ␣an | ␣L | LM | ␣server | ␣stream | ␣its | ␣answer | ␣token | ␣by | ␣token | ?

(␣ marks a leading space, which is part of the token.) Common words are one token each, punctuation gets its own, and "LLM" is rare enough to be split into two. With the system message (9 tokens) and the template's markers (17), the whole prompt is 43 tokens. That count is what the model reads and what a hosted API bills you for: the markers you never typed are paid like the rest.

You can see both steps on any vLLM server: its /tokenize endpoint applies the chat template and returns the tokens without running the model.

$ curl -s localhost:8001/tokenize -H 'Content-Type: application/json' -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [{"role": "user", "content": "Why stream tokens?"}],
    "chat_template_kwargs": {"enable_thinking": false},
    "return_token_strs": true}'

{"count":16,"max_model_len":32768,
 "tokens":[151644,872,198,10234,4269,11211,30,151645,198,151644,77091,198,151667,271,151668,271],
 "token_strs":["<|im_start|>","user","Ċ","Why","Ġstream","Ġtokens","?","<|im_end|>","Ċ","<|im_start|>","assistant","Ċ","<think>","ĊĊ","</think>","ĊĊ"]}

Three words of question, 16 tokens of prompt. (Ġ is the tokenizer's own mark for a leading space, Ċ for a line break.)

The scheduler decides when it runs

The API server hands the 43 tokens to the engine core, where the request joins a waiting queue. The engine core runs a loop (source): pick the work for the next step, run one forward pass of the model on the GPU, collect the results, repeat. A forward pass runs tokens through every layer of the model once.

The scheduler picks that work. It has a budget of tokens per step (--max-num-batched-tokens) and a limit on how many requests share one step. Each step, it hands that budget out to the requests in progress and the ones waiting. The default budget depends on the GPU's memory (source): 2,048 tokens below 70 GiB, 8,192 above, 16,384 from 160 GiB. Our GB10 got 2,048 despite its 121 GiB, because its memory is shared with the CPU and the query vLLM uses to read the GPU's memory size fails on it, which sends vLLM to the smallest default.

Alone on the GPU, our request waited 0.02 milliseconds: the scheduler took it at the very next step. When requests arrive faster than the engine can take them, this wait grows, and it can become most of the time to the first token.

One detail of vLLM's scheduler says a lot about how engines work: it has no "prefill phase" and no "decode phase". Its own comment explains that each request only has a number of tokens already computed and a number of tokens it has, and each step gives tokens to requests so that the first catches up with the second. A new request is behind by its whole prompt; a request that is answering is behind by one token. The same step can serve both.

Prefill: the prompt in one pass

The first step our request got computed all 43 prompt tokens in one forward pass. This is prefill. Each layer handles all the tokens at once, and along the way it stores two vectors per token and per layer in the KV cache: a key and a value, what later tokens look up instead of recomputing the prompt (how big that cache gets). The end of the pass gives the model's prediction for the token after the prompt, which becomes the first token of the answer.

Prefill took 77 ms, and the first token reached our client at 82 ms. The other 5 ms are templating, tokenizing, and the trip back through the engine core, the API server, and HTTP.

77 ms to read 43 tokens is slow compared with what the GPU can do: on the same machine, a 3,500-token prompt takes about 690 ms, 5,100 tokens per second. A forward pass has a floor: it must read the model's weights from memory, about 15 GB of them (all but the table that turns token numbers into vectors), whether it processes 1 token or 43, and this GPU reads memory at about 241 GB/s (measured here). 15 GB at 241 GB/s is 63 ms. For a prompt this short, reading the weights is most of the cost, and it is about the cost of one answer token. Long prompts behave very differently: their cost is in the arithmetic, not in the reading (prefill vs. decode traces both).

Decode: one pass per token

Then the answer comes, one token per step. This is decode: each step runs one new token through the model (the last token produced), reads the KV cache of all the tokens before it, adds its own key and value to the cache, and predicts the next token.

The pass ends with one score for each of the 151,936 entries in the vocabulary. The sampler turns the scores into probabilities and draws one token. Most of the time, there is no real choice: the first five tokens of our answer each had a probability above 99.9%. The sixth was a real draw:

"its" had a probability of 82% and "answers" 18%. These are the probabilities vLLM returns when you ask for logprobs; it computes them from the raw scores, before any sampling setting is applied.

The draw itself used the request's settings. Our request asked for a temperature of 0.7, which sharpens the distribution: 89% against 11% here. It set nothing else, so vLLM filled in the rest from the model's own generation_config.json (loaded by default): keep only the 20 most likely tokens (top_k), then the smallest set holding 95% of the probability (top_p). Whatever is drawn is appended to the sequence, and the next step predicts the token after it.

Each decode step took 73 ms, about the same as the 43-token prefill, for the same reason: every step reads the same 15 GB of weights to produce one token per request. The answer was 93 tokens long; prefill produced the first, so decode ran 92 times: 92 × 73 ms = 6.7 seconds, 99% of the request.

Back to text

Each new token is checked first in the engine core. The request ends when the model produces its end-of-turn token (<|im_end|> for Qwen3) or when it reaches max_tokens, and the scheduler frees its place for the next request. Ours stopped on its own, at 93 tokens. Running out of max_tokens instead cuts the answer mid-sentence, which is why a truncated answer is a configuration problem, not a model problem. (Stop strings you pass in the request are matched later, on the text, in the API server.)

Then every token goes back to the API server, which does two things with it.

• Detokenize. The token numbers become text again. This is done incrementally, because a token isn't always a complete piece of text: our answer started with "An", " L", "LM", and only " L" plus "LM" make " LLM". For languages and emoji whose characters span several tokens, the server waits until the bytes form a complete character before sending it.
• Stream. Each new piece of text goes out as a server-sent event, one small message pushed on the open HTTP connection. Our client received a chunk every 73 ms, the same rhythm as the decode steps.

Streaming exists because of how decode works: the answer is produced one token at a time anyway, so the server can show you the first words 82 ms in instead of making you wait 6.8 seconds for the full paragraph.

Sharing the GPU

A production server is rarely this quiet. We ran the same request five more times while 64 other requests were generating long answers. The server's log, every ten seconds, showed the load:

Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 819.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%

The first token took about three times longer: our request waited for the step in progress to finish, then its prefill shared a step with 64 decoding requests. Each of our tokens took 22% longer, because every step now produced 65 tokens instead of 1. Meanwhile the GPU produced about 820 tokens per second in total, against 14 for our request alone.

This is the trade every serving engine makes: sharing each pass between many requests costs each of them some latency and multiplies the total output. How the engine admits new requests into a running batch is continuous batching; how it keeps 65 KV caches in memory without waste is paged attention.

Reading the numbers

The stages map directly onto the numbers serving dashboards show:

• Time to first token (TTFT) = queue wait + prefill + the trip back. It grows with load (the queue) and with prompt length (prefill).
• Time between tokens (inter-token latency, ITL) = the length of one decode step, which grows with the number of requests in the batch.
• Total time ≈ TTFT + (number of answer tokens − 1) × time between tokens. For our request: 82 ms + 92 × 73 ms = 6.8 s.

So when a request feels slow, the first question is which of the two it is. A slow first token points at the queue or at prefill: too much load, or very long prompts. Slow streaming points at decode: a large batch, a long context, or a model too big for how fast the GPU reads its memory.

Prefill and decode cost the same here because the prompt was tiny. With a prompt of a few thousand tokens they behave like two different workloads, down to the kernels the GPU runs: that is the next piece.