If you're shopping for a local LLM, one number appears everywhere:

tokens per second.

40 tok/s.

80 tok/s.

150 tok/s.

500 tok/s prefill.

It looks simple. Faster is better.

For ordinary chat, that number can be useful.

For AI agents, coding agents, long-context workflows, and multi-step tool use, it can be deeply misleading.

A model producing 100 tokens per second can feel dramatically slower than a model producing 25 tokens per second.

The reason is that an agent doesn't simply generate text.

It repeatedly reads context, processes new information, generates a response, calls tools, receives their output, builds another prompt, processes that prompt, generates again, and repeats the cycle.

The user doesn't experience "tokens per second."

They experience:

How long did it take the agent to finish the job?

That distinction changes almost everything about how local AI inference should be benchmarked.

The number everyone watches

Suppose you have two models.

Model A

• 100 tok/s generation
• 500 tok/s prompt processing

Model B

• 30 tok/s generation
• 3,000 tok/s prompt processing

At first glance, Model A looks much faster.

100 is more than three times 30.

You might assume Model A is the obvious choice.

Now give both models a 100,000-token coding context and ask them to modify a large project.

Model A spends several minutes processing the prompt before producing anything.

Model B processes the same context much faster, then generates at 30 tok/s.

Suddenly the "slower" model can finish the interaction first.

This isn't hypothetical.

A recent r/LocalLLaMA discussion specifically focused on the fact that agentic workflows expose prefill performance problems that ordinary chat often hides. Users reported that a model could generate at impressive speeds while taking a surprisingly long time to process large agent prompts.

Another user running Qwen 3.6 35B-A3B with a coding agent reported about 9 tokens/s generation with a 36K-token context, but a total response time of roughly 77 seconds once prompt processing was included.

That is the first important lesson:

Generation speed is only one part of inference latency.

What actually happens when an AI agent works?

A normal chatbot interaction looks roughly like this:

User
  ↓
Prompt
  ↓
Prefill
  ↓
Decode
  ↓
Answer

An agent looks more like this:

User
  ↓
Large system prompt
  ↓
Repository / files / history
  ↓
Prefill
  ↓
Model reasoning
  ↓
Tool call
  ↓
Tool output
  ↓
New context
  ↓
Prefill again
  ↓
More reasoning
  ↓
Another tool call
  ↓
More context
  ↓
Prefill again
  ↓
...
  ↓
Final answer

The model may perform dozens or hundreds of inference calls during a single task.

A recent real-world Reddit experiment using Qwen 3.8-27B with an agent harness reported 966 model calls, 972 model-facing tool calls, 130.2 million input tokens and 812,500 output tokens over an eight-hour run. The median root request had a 136.6K-token context, with the 95th percentile around 205.9K.

This is what makes agentic workloads fundamentally different.

A benchmark that says:

104 tok/s

doesn't tell you how long the actual task took.

Prefill and decode are two different problems

To understand why, we need to separate inference into two phases.

Prefill

The model receives the existing prompt and processes it.

That might include:

• System instructions
• Conversation history
• Source code
• Tool definitions
• Tool results
• Retrieved documents
• Previous reasoning
• User instructions

The model processes those tokens to establish the internal state needed to generate the next token.

The result is a cache of attention information, commonly called the KV cache.

Decode

The model then generates tokens one at a time.

During decoding, each new token uses the existing context and adds another entry to the cache.

This is the phase most local-LLM benchmarks emphasize.

For example:

Qwen: 80 tok/s

That is generally a decode measurement.

It tells you how quickly the model can generate output after the relevant context has already been processed.

It doesn't tell you how long it takes to get there.

Why prefill can dominate long-context workloads

Imagine a model that can process:

1,000 prompt tokens per second

and generate:

50 output tokens per second.

Give it a 1,000-token prompt and ask for 500 output tokens.

Approximate time:

Prefill: 1 second

Decode: 10 seconds

Total: ~11 seconds

Generation dominates.

Now give it a 100,000-token prompt.

Prefill: ~100 seconds

Decode: 10 seconds

Total: ~110 seconds

The exact numbers depend heavily on the model, hardware, attention implementation, batching, cache state and runtime. But the principle remains.

The workload has changed from:

mostly generation

to:

mostly reading.

That is exactly what modern coding agents increasingly do.

A Reddit benchmark on an M1 Max found that generation speed remained relatively stable across different tasks while prompt processing time grew substantially as the input became larger.

The 100K-token problem

Now consider an agent working on a large software repository.

The agent may need:

• The system prompt
• Agent instructions
• Tool schemas
• Conversation history
• Files it previously opened
• Relevant source code
• Compiler errors
• Test output
• Git diffs
• Terminal output
• Search results
• Previous tool calls

After several iterations, the context can become enormous.

Modern models advertise context windows of 128K, 256K, 512K or even 1M tokens.

But a larger context window doesn't mean the model can process that context instantly.

It only means the model is capable of accepting it.

Those are very different things.

The brutal example: 128K context

One of the most interesting recent Reddit experiments involved speculative prefill for Qwen 3.6-27B.

On an RTX 3090, the author reported approximately:

24.8 seconds TTFT at 128K

versus roughly:

257 seconds with vanilla llama.cpp

for the tested configuration, around a 10× difference.

The same experiment reported that the model could decode at roughly 74 tok/s, showing the problem clearly: the model could generate quickly once it got there, but getting through the long prompt was the expensive part.

This is why simply saying:

"This GPU runs the model at 74 tok/s"

can be incredibly misleading.

If you have to wait minutes before those 74 tokens per second begin, the user doesn't experience a 74 tok/s system.

They experience:

"Why hasn't the AI answered yet?"

TTFT matters

This leads us to an important metric:

Time to First Token, or TTFT

TTFT measures how long it takes from submitting the request until the first generated token arrives.

It can include:

• Queueing
• Prompt processing
• Chunked prefill
• Scheduling
• Other infrastructure overhead

Inference frameworks such as vLLM explicitly treat TTFT as a first-class performance metric.

For an interactive application, TTFT is often more meaningful than raw generation speed.

Imagine two systems.

System A

100 tok/s decode

30-second TTFT

System B

40 tok/s decode

2-second TTFT

Suppose both produce a 400-token response.

System A:

30 + 4 = 34 seconds

System B:

2 + 10 = 12 seconds

The system advertised as "2.5× faster" is actually almost 3× slower for this interaction.

That is the benchmark trap.

But TTFT still isn't enough

Here's where it gets more interesting.

For an agent, we shouldn't stop at TTFT.

We also care about:

Time per output token

How quickly tokens arrive during generation.

Inter-token latency

How much time passes between individual output tokens.

End-to-end latency

How long the complete request takes.

Tool latency

How long the external tool takes.

Agent iteration latency

How long one complete reasoning → action → observation cycle takes.

Task completion time

How long it takes to actually finish the job.

That final number is arguably the most important one.

The benchmark that actually matters

Imagine asking an AI coding agent:

"Find why the checkout page crashes when a user applies a discount code, fix it, and run the tests."

The agent might do this:

Request
↓
Read repository
↓
Search files
↓
Read checkout code
↓
Reason
↓
Modify file
↓
Run tests
↓
Receive 8,000 tokens of errors
↓
Reason again
↓
Read additional files
↓
Modify code
↓
Run tests again
↓
Receive results
↓
Reason again
↓
Fix bug
↓
Run tests
↓
Done

The model may have generated only 5,000 tokens.

But it could have processed hundreds of thousands of input tokens.

The final output-token count tells you almost nothing about the actual workload.

Agent inference has a hidden input/output asymmetry

Traditional chat often looks like:

Input: 1,000 tokens

Output: 500 tokens

That's manageable.

Agentic workloads can look like:

Input: 100,000 tokens

Output: 1,000 tokens

Then the next turn:

Input: 110,000 tokens

Output: 800 tokens

Then:

Input: 125,000 tokens

Output: 2,000 tokens

The model is spending vastly more computation reading than speaking.

This is why the traditional obsession with output tokens is increasingly disconnected from agent workloads.

Research on LLM serving has recognized this prompt-heavy nature of agent workloads for years. The Preble scheduling research describes agent interactions as repeated LLM calls sharing previous context, with some agent workloads having prompts dramatically longer than their outputs.

The KV cache changes the equation

At this point we need to talk about the KV cache.

During prefill, the model computes attention states for the prompt.

Those states can be stored.

When generating subsequent tokens, the model doesn't need to recompute everything from scratch.

That's the basic idea behind the KV cache.

It is one of the fundamental optimizations that makes autoregressive generation practical.

But the cache consumes memory.

And its size generally grows with the amount of context being retained.

That creates a tradeoff:

More context → more memory → potentially more useful information

but also:

More context → more work and memory pressure.

Recent research reviewing KV-cache optimization describes context length as a first-order constraint for memory capacity, bandwidth and inference throughput as context windows grow from thousands toward millions of tokens.

Prefix caching changes everything

There's an important optimization that can make long-context agents much faster.

Suppose every agent request begins with the same 30,000 tokens:

System prompt
+
Tool definitions
+
Repository instructions
+
Project metadata

Then the user asks something.

After that, the agent calls a tool.

Then it asks the model again.

If the first 30,000 tokens haven't changed, recomputing them is wasteful.

A runtime can reuse the cached prefix.

That's prefix caching.

Instead of:

Process 30K tokens
+
Process 10K new tokens

the runtime can effectively reuse the existing prefix and process only the new portion, subject to the exact caching implementation and cache boundary.

vLLM's documentation describes this explicitly: when the KV cache contains matching blocks from the beginning of a prompt, those blocks can be reused rather than recomputed.

This is one reason two apparently identical models can feel radically different inside an agent.

The model isn't the only thing that matters.

The inference engine matters.

And this creates a nasty benchmarking problem

Imagine benchmark A runs a repeated prompt.

The first request processes everything.

The next 20 requests reuse the same prefix.

The benchmark looks fantastic.

Now benchmark B uses slightly different contexts every time.

Its cache can't reuse as much.

Suddenly benchmark B looks dramatically slower.

Which one represents real agent usage?

It depends.

That's why an emerging class of agent benchmarks is explicitly concerned about prefix-cache poisoning.

One open-source agentic inference benchmark describes the problem directly: naive benchmarks can become artificially fast because repeated prompts hit the prefix cache, so the benchmark ends up measuring cache reuse rather than genuine inference. It deliberately modifies recordings to invalidate the prefix cache while preserving semantic content.

This is a huge deal.

A benchmark can lie without technically reporting incorrect numbers.

The numbers are real.

The workload is just unrealistic.

Real agents don't have perfectly stable prompts

Consider a coding agent.

It starts with:

system prompt
repository instructions
file A
file B
file C

Then it edits File B.

Now the context may effectively become:

system prompt
repository instructions
file A
modified file B
file C

Everything after the modification point may no longer match the previous serialized prefix.

That can reduce cache reuse.

A benchmark that repeatedly feeds an identical prompt won't capture this.

This is why cache-aware benchmarking is becoming important.

The agent's context isn't just text

Another common mistake is to imagine that an agent simply appends messages forever.

Real agent systems have much more structure.

There can be:

• Tool calls
• Tool results
• Function schemas
• Structured outputs
• File contents
• Search results
• Images
• Code blocks
• Hidden reasoning
• Intermediate state
• Compaction summaries
• Retrieved context
• Dynamic system instructions

Every one of these can affect the actual serialized prompt.

That means two prompts that look semantically almost identical to a human may not be identical to the inference engine.

At the token level, small changes can affect caching.

And the cache operates on what the model actually receives, not on what a human considers "basically the same."

Why reasoning models make this worse

Modern reasoning models add another dimension.

They can generate huge amounts of internal reasoning before arriving at the final answer.

For an agent, this can become expensive in two ways.

First:

More generated tokens.

Second:

More context for future turns.

A Reddit user running Qwen 3.8 27B locally reported that the model could consume more than 100K tokens during its planning phase, eventually forcing context compaction in OpenCode.

This creates an interesting feedback loop:

More reasoning
↓
More tokens
↓
Larger context
↓
More prefill work
↓
Longer agent turns
↓
More time available for further reasoning
↓
Even larger context

A model can be extremely capable but become frustratingly slow because it thinks too much.

That's why raw decode speed alone isn't enough.

Reasoning efficiency may matter more than token speed

Suppose Model A solves a problem in:

10,000 reasoning tokens

at 100 tok/s.

Model B solves it in:

2,000 reasoning tokens

at 30 tok/s.

Model A:

100 seconds

Model B:

67 seconds

The slower model wins.

Now include prompt processing, tool calls and context growth, and the difference could become even larger.

This is why a model's tokens per second and its tokens required to solve the task are separate variables.

A useful metric is:

Time to successful task completion

not:

Maximum tokens per second

Prefill and decode are competing for the same hardware

There's another problem when multiple agents run simultaneously.

Suppose you have five agents.

Four are currently generating tokens.

One performs a web search and suddenly receives 20,000 tokens of results.

The fifth agent needs to prefill those 20,000 tokens.

That prefill can compete with the decode workload.

A recent r/LocalLLaMA user reported exactly this behavior with multiple llama.cpp agents: decode performance was good, but when one agent needed to process a few thousand tokens from a web search, the other agents experienced significant slowdowns.

This is not simply a model-speed problem.

It's a scheduler problem.

Why serving systems are separating prefill and decode

Modern inference frameworks increasingly treat prefill and decode as different workloads.

vLLM's experimental disaggregated-prefill architecture explicitly separates them so operators can tune TTFT independently from inter-token latency. The documentation also notes that disaggregation does not inherently increase total throughput. Its purpose is better control over latency and scheduling.

Why?

Because prefill and decode have different hardware characteristics.

Prefill is generally more compute-heavy.

Decode is often much more memory-bandwidth-sensitive.

Trying to optimize both with exactly the same scheduling strategy is inefficient.

Chunked prefill

One solution is chunked prefill.

Instead of processing a giant prompt in one enormous operation, the system divides the prompt into chunks.

For example:

100K-token prompt

↓
16K
16K
16K
16K
16K
16K
4K

The scheduler can interleave these chunks with decode work from other requests.

vLLM's current documentation describes this as a way to balance compute-heavy prefill against memory-bound decode. Smaller chunks generally help protect inter-token latency, while larger chunks can improve TTFT and throughput.

This is a fascinating tradeoff.

You can improve one metric while hurting another.

There is no single magic setting.

Why "maximum tokens per second" can hurt interactive performance

Imagine two workloads arriving at the same server.

Request A

100K-token prompt.

Request B

Short 1K-token prompt.

If the scheduler gives Request A unrestricted priority, Request B may sit behind a huge prefill.

The model could be doing thousands of tokens per second internally while the user waiting for Request B experiences terrible latency.

That's why production inference systems care about:

• TTFT
• ITL
• queueing delay
• throughput
• fairness
• tail latency
• scheduling

rather than only raw generation speed.

Tail latency is particularly important

Average performance can hide ugly behavior.

Suppose:

• Average TTFT: 2 seconds
• 95th percentile TTFT: 20 seconds
• 99th percentile TTFT: 60 seconds

An average benchmark might say:

"2 seconds! Great."

Users might say:

"Sometimes this thing takes a minute to respond."

For an agent system with many sequential calls, tail latency gets even more important.

If a task requires 30 model calls, one horrible delay can hold up the entire workflow.

This is why inference frameworks explicitly expose metrics such as TTFT, TPOT, ITL and end-to-end latency.

The math of an agentic request

A simplified agent task can be thought of as:

Ttask=∑i=1N(Tqueue,i+Tprefill,i+Tdecode,i+Ttool,i)T_{task} = \sum_{i=1}^{N} (T_{queue,i} + T_{prefill,i} + T_{decode,i} + T_{tool,i})

where:

• N = number of model/tool iterations
• Tqueue = time waiting for compute
• Tprefill = time processing the input context
• Tdecode = time generating the output
• Ttool = time spent waiting for external tools

And:

Tdecode≈OiRdecodeT_{decode} \approx \frac{O_i}{R_{decode}}

where:

• OiO_i = output tokens
• RdecodeR_{decode} = decode tokens/sec

Meanwhile:

Tprefill≈f(Pi,hardware,model,runtime,cache)T_{prefill} \approx f(P_i, hardware, model, runtime, cache)

where:

• PiP_i = effective uncached prompt tokens
• hardware matters
• model architecture matters
• runtime implementation matters
• cache reuse matters

This gives us a much better mental model.

A faster decoder only reduces one component.

If prefill or tool latency dominates, increasing decode speed may barely change the total task time.

The "effective context" is more important than the advertised context

A model might support:

1,000,000 tokens

but your workload might have:

200,000 tokens already cached

and only:

10,000 new tokens

to process.

The relevant quantity for that particular request isn't necessarily 210,000 freshly computed tokens.

It's the uncached portion plus the runtime's cache-management overhead.

Conversely, if your context is constantly changing, a huge context window can become expensive.

This is why context length and context utilization need to be benchmarked together.

Long context creates another problem: context quality

There is also a distinction between:

Can the model accept this many tokens?

and:

Does the model effectively use this many tokens?

A million-token context window doesn't automatically mean the model can retrieve every relevant detail equally well.

Long-context research has repeatedly examined degradation, retrieval difficulty and attention allocation as contexts grow.

For agent systems, this creates another tradeoff:

Keeping everything can preserve information.

Keeping everything can also:

• Increase prefill cost
• Increase memory requirements
• Increase scheduling pressure
• Increase retrieval noise
• Increase the chance of irrelevant context influencing the model
• Increase compaction pressure later

Sometimes a smaller, cleaner context is faster and better.

Context compaction is not just a memory trick

When an agent reaches its context limit, it can summarize older information.

That sounds straightforward.

But compaction itself requires another model operation.

You might have:

100K context
↓
Compaction
↓
15K summary
↓
Continue

You saved memory.

But you also paid computation to create the summary.

And you potentially lost information.

One Reddit user working with a 100K local agent reported moving to a dedicated context-management system because native compaction was struggling to preserve important details.

So context management is itself part of inference performance.

The hidden cost of tool outputs

Here's a common mistake in agent design.

An agent calls:

git diff

and gets 12,000 tokens.

Then:

npm test

and gets 18,000 tokens.

Then:

grep

and gets another 8,000.

The model didn't generate those tokens.

But it has to process them.

This means tool verbosity can become an inference bottleneck.

A poorly designed tool can effectively turn:

one operation

into:

tens of thousands of additional model input tokens.

This is why good agent design increasingly treats tool output as a scarce resource.

Don't give the model everything.

Give it what it needs.

The same applies to source code

Imagine an agent opens a 10,000-line file because it needs one function.

The model now has to process that material.

A smarter agent could:

1. Search for the relevant symbol.
2. Read 50 lines around it.
3. Follow dependencies.
4. Expand context only when necessary.

This isn't just an accuracy optimization.

It's an inference-latency optimization.

Good context selection can reduce both:

cost

and

time.

Why agent architecture can matter more than model size

Consider two systems.

Agent A

Uses a 70B model.

Loads 150K tokens every turn.

Poor prefix reuse.

Huge tool outputs.

Agent B

Uses a 30B model.

Loads 30K relevant tokens.

Strong prefix caching.

Efficient tool outputs.

Good context compaction.

Agent B may feel much faster.

This is one reason the "just buy a bigger GPU" approach doesn't always solve agent latency.

Sometimes the problem is upstream.

The agent is feeding the model too much information.

Hardware still matters, a lot

None of this means GPU performance is irrelevant.

Quite the opposite.

Hardware determines:

• Model loading speed
• Prefill throughput
• Decode throughput
• Memory bandwidth
• KV-cache capacity
• PCIe transfer performance
• CPU/GPU synchronization
• Concurrent request capacity

But different workloads stress different hardware properties.

For decode-heavy workloads, memory bandwidth can dominate.

For long-context prefill, compute and attention efficiency become much more important.

For offloaded models, PCIe and system-memory bandwidth can become bottlenecks.

For multiple agents, scheduling and memory capacity become increasingly important.

So the "best GPU for an LLM" question is incomplete.

The correct question is:

Best GPU for which workload?

This explains some strange local-LLM benchmarks

You may see:

Model X: 150 tok/s

and:

Model Y: 45 tok/s

Yet users prefer Model Y.

It might be because Model Y:

• Starts responding faster
• Has lower TTFT
• Uses fewer reasoning tokens
• Has better cache reuse
• Has better prompt processing
• Finishes tasks in fewer iterations
• Produces fewer unnecessary tool calls

The raw tok/s number doesn't capture any of those.

A better benchmark for AI agents

If we actually want to compare agent performance, we should report something closer to this:

MetricWhy it mattersDecode tok/sOutput generation speedPrefill tok/sInput processing speedTTFTTime until the agent starts respondingITLSmoothness of generationE2E latencyTime for one requestCache hit rateHow much context was reusedEffective uncached tokensActual prompt workInput tokens/taskTotal context processedOutput tokens/taskGeneration workloadReasoning tokens/taskThinking overheadTool calls/taskAgent complexityTool latencyExternal waitingContext peakMemory pressureTask success rateWhether the model actually solved itTime to successful completionThe metric users actually care about

And one metric should sit at the top:

Time to Successful Task Completion

Because that's what an agent is for.

A concrete comparison

Imagine two coding agents solving the same bug.

Agent A

Model: 70B

Decode: 100 tok/s

Prefill: 800 tok/s

Average context: 120K

Task iterations: 12

Total input tokens: 1.4M

Total output tokens: 18K

Task time: 18 minutes

Agent B

Model: 32B

Decode: 35 tok/s

Prefill: 2,000 tok/s

Average context: 35K

Task iterations: 8

Total input tokens: 280K

Total output tokens: 7K

Task time: 6 minutes

Agent A wins the benchmark.

Agent B wins the job.

That's the problem with reducing agent performance to a single number.

The future may be dominated by prefill optimization

This isn't just a Reddit observation.

Research and inference engineering are increasingly focused on long-context prefill.

Recent work has explored sparse attention specifically to reduce long-context prefill computation, while other approaches are targeting universal prefill acceleration across different model architectures.

A May 2026 Reddit project demonstrated a reported roughly 10× TTFT improvement on a 128K prompt using speculative prefill.

The direction is clear:

As context windows grow, simply making decode faster isn't enough.

The industry has to make reading the context faster too.

The interesting hardware question

This could eventually change how we buy AI hardware.

Today, people often compare GPUs by:

• VRAM
• TFLOPS
• model size
• tokens/sec

For agents, we may increasingly care about:

Memory bandwidth + compute throughput + cache capacity + interconnect + prefill efficiency + scheduling behavior.

A GPU with slightly lower decode performance but dramatically better long-context prefill could be the better agent machine.

And for multi-GPU systems, interconnect bandwidth becomes even more important.

The same applies to CPU inference.

A massive RAM machine can technically run enormous models and contexts, but if the memory subsystem can't feed the model quickly enough, the system may be technically capable and practically miserable.

Why Apple Silicon is an interesting special case

Apple Silicon creates another interesting benchmark situation because CPU, GPU and unified memory share a memory architecture.

That makes huge models and contexts possible on machines that would otherwise have insufficient dedicated GPU VRAM.

But memory capacity isn't the same as memory performance.

Recent local-inference experiments on Apple Silicon have highlighted the gap between generation speed and prompt-processing speed, particularly as context grows.

So a Mac can have:

64 GB unified memory

and successfully load a model that wouldn't fit inside a conventional 16 GB GPU.

That doesn't mean it will process a 200K-token agent context quickly.

Capacity answers:

Can I load it?

Bandwidth and compute answer:

Can I use it quickly?

Those are different questions.

Why benchmarking should separate cold and warm runs

This is another detail that gets overlooked.

A realistic benchmark should report at least two cases.

Cold context

Nothing is cached.

This measures the cost of building the state from scratch.

Warm context

A large portion of the prefix is already cached.

This measures the benefit of reuse.

Both matter.

A chatbot continuing the same conversation may have a high cache-hit rate.

A coding agent whose files keep changing may have much lower reuse.

A retrieval system might have almost no shared prefix.

A benchmark that reports only one number can hide all of this.

The future benchmark should look more like a workload trace

Instead of:

"Run 128K tokens through the model and report tok/s."

A realistic agent benchmark should record:

Turn 1
Context: 12K
Cache hit: 0%
Output: 800
Tool calls: 2

Turn 2
Context: 19K
Cache hit: 70%
Output: 1,200
Tool calls: 3

Turn 3
Context: 31K
Cache hit: 62%
Output: 900
Tool calls: 4

Turn 4
Context: 48K
Cache hit: 45%
Output: 2,100
Tool calls: 2

Then calculate:

Total wall-clock time

Total tokens processed

Total output tokens

Total tool time

Successful task completion

That tells us something useful.

The uncomfortable conclusion

There is nothing wrong with measuring tokens per second.

It is a useful metric.

The problem is treating it as the metric.

For a simple chatbot, output speed can tell you a lot.

For an AI agent, it can tell you surprisingly little.

An agent is a system.

Its performance depends on:

model + prompt + context + cache + inference engine + scheduler + hardware + tools + agent architecture + reasoning behavior.

Change any one of those and the user's experience can change dramatically.

A model that generates 150 tokens per second can still feel slow.

A model that generates 30 tokens per second can feel fast.

A smaller model can finish a task faster than a larger one.

A slower GPU can beat a faster GPU for a particular workload.

A lower tok/s configuration can produce a better result.

And a benchmark can report a fantastic number while completely missing the thing that matters.

What should you actually look for when choosing a local model for agents?

Don't start with:

"How many tokens per second does it get?"

Ask these instead.

1. How fast is prefill at my actual context length?

Not 2K.

Not 4K.

If your agent normally reaches 80K, benchmark 80K.

2. What is TTFT?

Especially at the context sizes you actually use.

3. How good is prefix caching?

A huge difference for repeated agent prompts.

4. How much context does the agent actually process?

The model's maximum context isn't the same as your workload.

5. How many reasoning tokens does it use?

A faster model that thinks for 10× longer may not be faster.

6. How many tool calls does it need?

Fewer unnecessary iterations can beat raw model speed.

7. How does performance change as context grows?

Benchmark:

8K → 32K → 64K → 128K

Don't assume the scaling is linear.

8. How does it behave with multiple agents?

A single-agent benchmark doesn't tell you what happens when four agents share the GPU.

9. What happens when the KV cache fills?

Memory pressure can completely change performance.

10. How long does it take to actually finish the task?

This is the final metric.

The new mental model for AI inference

For years, we treated LLM speed roughly like this:

How quickly can it generate text?

The agent era requires a different question:

How quickly can the entire system turn a problem into a correct result?

That means we should stop thinking about inference as:

tokens → tokens

and start thinking about it as:

context → reasoning → actions → new context → reasoning → actions → result

The bottleneck can move every few seconds.

At one moment it's prefill.

Then decode.

Then KV-cache memory.

Then a compiler.

Then a web request.

Then another giant prefill.

Then context compaction.

Then decode again.

The fastest component isn't necessarily the bottleneck.

The metric that actually matters

If you're building or buying an AI agent system, here's the number I'd put on the dashboard:

Minutes to successful task completion

Not maximum tok/s.

Not theoretical FLOPS.

Not context-window size.

Not even TTFT by itself.

Give the system a real task.

Let it use its tools.

Let the context grow naturally.

Let the cache behave naturally.

Let it make mistakes.

Then measure how long it takes to produce a correct result.

That is what the user is buying.

And as AI agents move from simple chat into software development, research, automation and long-running workflows, this distinction is going to matter more and more.

The fastest LLM isn't necessarily the one that generates the most tokens.

It's the one that wastes the least time getting to the answer.

And that may be the biggest mistake we're making when we benchmark AI right now.