A 27B Model as Your Local Agent: What It Can and Can't Do

An agent re-sends its whole history on every turn: the system prompt, the task, every command it ran and every file it read. By turn 20 of a coding task, that is 20,000 tokens, and a model that read them all again each time would spend most of the session reading. Hosted APIs hide this behind prompt caching and a bill. On a model you run yourself, the question is concrete: does the box remember the history, how many agents can share it, and can the model call tools reliably? We replayed real coding-agent sessions against a 27-billion-parameter model on one desk-side GPU and measured each of these.

The setup

One NVIDIA GB10 (the chip of the DGX Spark: 128 GB of memory shared by CPU and GPU, 273 GB/s), running Qwen3.8-27B with the configuration published in Mia's AI Lab's Qwen3.8-27B-SGLang-DGX-Spark scripts, which start from SGLang's cookbook:

• weights and activations in NVFP4, the output layer in BF16 (checkpoint, about 24 GB);
• FP8 KV cache, 262k-token context, room for 1.38 million cached tokens;
• DFlash2 speculative decoding, 8 draft tokens per step;
• the qwen3_coder tool-call parser and the qwen3 reasoning parser, so tool calls and thinking come back as structured fields of the OpenAI API.

The traffic is not synthetic. nebius/SWE-agent-trajectories holds recorded sessions of SWE-agent fixing real GitHub issues: every prompt, command and tool output. We took 12 resolved sessions of 20 to 40 steps and replayed each step as one request with its exact history, asking for up to 128 tokens of answer with thinking off. The next turn continues with the dataset's own action, so every run sees the same conversation. That measures what the serving side controls: how long the model takes to read its context and start answering, and how fast it then writes.

One agent: the cache does the work

From the second turn on, SGLang's prefix cache already holds nearly everything the agent sent before (on this hybrid model, up to the last point where SGLang saved the recurrent state), and only the newest tokens are read. Over the session, 89.5% of all prompt tokens came from the cache. The first answer token arrived in 0.55 s at the median and 0.83 s at the 90th percentile, and stayed under a second while the history grew tenfold. The two slow turns are the first one (2,144 tokens, nothing cached) and turn 14, where a large tool output arrived: 4,405 new tokens, 2.3 s.

Without the cache, the last turn alone would have meant reading 20,579 tokens at the 1,400-1,800 tokens/s this model reads prompts on this GPU: about 13 s before the first token, against 0.36 s measured. The cache is what makes a local agent feel interactive; prefix caching explains how it works.

Several agents: more work, longer waits

The box does more in total as agents are added, 72 turns per minute with 8 against 25 with one, because continuous batching shares each read of the weights between them. Each agent pays for it: its tokens arrive at half the speed, and its first token waits longer, up to 7 s when its own new tokens and other agents' are read in the same steps. The cache kept up: 93-94% of prompt tokens were cached with 4 and 8 agents, since the 1.38 million tokens of KV cache hold all their histories.

Eight is also the ceiling. Qwen3.8 is a hybrid model: its Gated DeltaNet layers keep a fixed-size recurrent state per request instead of a growing cache, and SGLang sizes a pool of those states at startup. Its log says so: max_running_requests is capped to 8 by the mamba state cache (max_mamba_cache_size=40, 5 state slots per request). A ninth agent waits in the queue.

Writing speed: it depends on what it writes

One request at a time, 1,024 tokens of code came at 48-49 tokens/s, and 900 tokens of prose at 20 tokens/s. The difference is speculative decoding: the draft model guesses code, with its repeated identifiers and syntax, far more often than prose, so each step accepts more tokens. Agent answers in the replay (short commands and edits) landed in between, 35 tokens/s.

At 20-50 tokens/s, a 300-token edit takes 6 to 15 s to write. That is the main difference you will feel from a hosted API, and it adds up over a 30-turn session.

Tool calls

We ran the Berkeley Function Calling Leaderboard's (BFCL v3) single-turn categories through the server's OpenAI tool-calling API, scoring the parsed call against the expected function and arguments with a simplified version of BFCL's AST check:

Not one call, in 1,140 requests, came back as malformed JSON: the parser and the model agree on the format. Choosing the right function and arguments worked about 93% of the time. The weak spot is knowing when not to call: one time in five, given tools that don't fit the question, the model called one anyway. Thinking helped there (on the same 100 questions, 8 fixed and 2 broken, not significant at this size) and not elsewhere, at twice the tokens per answer.

Long context

At 120,000 tokens of context, with 8 numbered codes hidden in it and one asked for, the model found the right one in 20 of 20 trials. Reading that much is the cost: a 128k-token prompt took 125 s to read from scratch, about 1,000 tokens/s. An agent that pastes a whole repository into one turn pays that once; with the prefix cache, later turns do not.

What it can't do

• Write fast. 20-50 tokens/s alone, 19 per agent with 8. Fine for an agent that mostly reads and runs commands; slow for one that writes long files.
• Serve a team. Eight concurrent requests is the hard limit of this configuration, and waits grow well before it.
• Read a large new context quickly. About 1,500 tokens/s: a 30k-token paste is 20 s.

Gotchas

Keep the history stable. The cache matches prompts from the start. An agent framework that rewrites earlier messages (summarizing old turns, reordering tools, putting a timestamp in the system prompt) breaks the match and pays the full read on that turn.

Check the concurrency the server actually allows. The setup asks for 10 concurrent requests; SGLang capped it at 8: the DFlash2 launch uses SGLang's extra_buffer state-cache strategy, which needs 5 state slots per request, and the pool was sized at 4 per request (40 slots for 10). The log line above is the only place this shows.

Tell the model when no tool fits. Irrelevance was the worst category. A "respond without calling a tool if none applies" line in the system prompt, or thinking on, is cheap insurance.

Verdict

As the engine behind a single developer's coding agent, or a few, a 27B model on a GB10 is good enough: interactive turns thanks to the cache, reliable tool calls, recall over long histories. It does not match a hosted API on speed, and it is a one-person machine. The configuration choices that make it work are the ones measured here: prefix caching on, an FP8 KV cache for room, speculative decoding for speed, and a stable prompt prefix on the agent's side.