We Tested Our Concurrency Finding on a Second Model — It Held

Private LLM Infrastructure Research Series · Before building a public cost calculator, we needed to know if one result was a property of the GPU or a property of one model.

September 23, 2026

We Tested Our Concurrency Finding on a Second Model — It Held

The finding we needed to check

Our previous experiment (E006) found something that didn't match the obvious intuition. On a single cloud GPU (NVIDIA L4) already running Llama 3.1 8B at 96% utilization with one request at a time, raising client-side concurrency to 8 simultaneous requests cut the cost per token by 4.8× — from $4.11 to $0.86 per million tokens. GPU power draw barely moved.

A number like that is easy to over-trust. It came from one model, on one GPU, on one workload. Before we build it into anything a reader might rely on — including a self-hosting cost calculator we're building for this site — we wanted to know whether it was a property of this GPU serving concurrent decode workloads, or a property of Llama specifically.

So we ran E007: same hardware, same workload, same methodology as E006, with exactly one variable changed — the model. We substituted Qwen3-8B, one of the Chinese open-weight models we covered in a previous newsletter issue, and reran all four concurrency levels (1, 2, 4, 8) with two repetitions each.

Why concurrency reduces cost per token

Generating a token isn't primarily a compute problem — it's a memory-bandwidth problem. For every token, the GPU has to read the model's full set of weights out of VRAM before it can do the (comparatively small) amount of math involved. That read has to happen regardless of how many requests are using it. Run one request at a time, and the GPU spends much of its time waiting on that memory read, with its compute cores mostly idle. Run 8 requests at once, and that same memory read now serves all 8 simultaneously — you pay the cost of reading the weights once and get roughly 8× the useful output from it.

Diagram showing that at concurrency=1 the GPU is mostly idle waiting on a memory read, while at concurrency=8 the same memory read serves 8 requests at once, yielding 8x the work for the same cost — At concurrency 1, the GPU idles waiting on a memory read. At concurrency 8, that same read serves 8 requests at once — roughly 8× the work for the same cost.

This is why GPU power draw stays essentially flat across concurrency levels in both experiments — the GPU isn't doing more work in a power-draw sense, it's doing less wasted waiting per unit of useful output.

The result: same shape, two architectures

Concurrency	Llama $/1M tokens	Qwen3 $/1M tokens	Difference
1	$4.106	$4.337	Qwen3 +5.6%
2	$2.239	$2.357	Qwen3 +5.3%
4	$1.449	$1.412	Qwen3 −2.6%
8	$1.073	$1.077	+0.4% (statistical noise)

Line chart showing cost per million tokens falling from about $4.10-4.34 at concurrency 1 to about $1.07 at concurrency 8, for both Llama 3.1 8B and Qwen3-8B, with the two lines nearly overlapping throughout — Cost per million tokens falls from ~$4.10–4.34 at concurrency 1 to ~$1.07 at concurrency 8 for both models — the two lines nearly overlap throughout.

Both models showed a monotonic, substantial cost reduction with concurrency, with no sign of plateauing by concurrency=8: Llama dropped 3.83× (concurrency 1→8), Qwen3 dropped 4.03×. GPU power stayed flat for both models across every concurrency level tested (Llama 71.4–71.7W, Qwen3 70.5–71.4W). That's the core result: the mechanism generalizes across two different model architectures on the same GPU. It is not something specific to Llama.

The gap that closes

At concurrency=1, Qwen3 cost about 5–6% more per token than Llama. By concurrency=8, that gap was 0.4% — smaller than the run-to-run noise in this experiment (Llama's own two repetitions at concurrency=8 varied by 4.1% from each other). We can't say with this data whether Qwen3 is "slightly more expensive" as a general property, because two data points per condition don't give a formal statistical test enough power to distinguish a real 0.4% difference from measurement noise. What we can say: the models converge, and by the highest concurrency we tested, cost is indistinguishable between them.

Line chart showing aggregate throughput in tokens per second rising from about 45-48 tok/s at concurrency 1 to about 183 tok/s at concurrency 8 for both models, with Qwen3 starting slightly behind Llama and closing the gap by concurrency 8 — Aggregate throughput rises from ~45–48 tok/s at concurrency 1 to ~183 tok/s at concurrency 8 for both models — Qwen3 starts behind Llama and closes the gap by concurrency 8.

The reason the cost gap closes: Qwen3's throughput scaled slightly better with concurrency (4.03× vs. Llama's 3.83× improvement from concurrency 1 to 8), closing the disadvantage it started with. Worth noting for anyone comparing models casually — a benchmark run only at concurrency=1 would have concluded Qwen3 is the more expensive model to serve. Run at the concurrency you'd actually deploy at, and that conclusion doesn't hold.

One architectural note: Qwen3 was consistently faster per individual request at every concurrency level (11–17% lower mean latency than Llama), despite having more transformer layers (37 vs. Llama's 33). Both models were confirmed at 100% GPU offload, so this reflects a genuine per-architecture decode-speed difference, not a placement artifact. The two models are similar in parameter count (both labeled "8B") but not identical in architecture — Qwen3's VRAM footprint ran consistently 550–750MB higher than Llama's.

What we're not claiming

Not a claim about which model is "better." That's a capability and use-case question, and cost-per-token at a fixed workload doesn't answer it.
Not a statistically confirmed difference in cost at any concurrency level. With two repetitions per condition, we don't have the power to formally test small differences — we can only compare the size of a gap to the size of known measurement noise, which is what the table above does.
Not a claim that this holds for other model sizes, other GPUs, or other workloads. Two 8B models on one L4, one fixed prompt, one price snapshot. That's the scope.

What this means if you're sizing hardware

To make this concrete: take a hypothetical organization of 1,000 employees running a private AI assistant. Assume 700 active users on a given day, 15 messages each, roughly 800 input and 400 output tokens per message — figures in line with typical reported chatbot token ranges. That works out to about 4.2 million output tokens a day, or roughly 130 tokens/second on average, with peak-hour demand plausibly reaching 4–5× that. (This sizing example is an illustrative estimate built from typical usage ranges, not a measured workload.)

One GPU of the class we tested, run at the concurrency level we measured, sustained 180–230 tokens/second. That puts peak demand for this hypothetical org in the range of 3 GPUs of this class — a rough, order-of-magnitude estimate, not a specification.

The real limitation of this estimate: everything we've measured used one fixed prompt at one context length (4,096 tokens). Real traffic has variable-length messages, and KV cache — not model weights — is what actually scales with both concurrent users and context length together. That's the part production sizing guides spend most of their effort on, and it's a gap in our own data so far, not something E006 or E007 measured. Two references worth reading if you're doing this sizing for real:

vLLM's GPU sizing guide, covering prefill vs. decode and KV cache math: centron.de/en/tutorial/vllm-gpu-sizing-guide-for-llm-inference
NVIDIA/VMware's LLM inference sizing and performance guidance: blogs.vmware.com/cloud-foundation

Limitations

One GPU, one deployment, one price snapshot. Every number above is scoped to this L4, this GCP pricing, these two specific 8B quantized checkpoints — not a general claim about L4s, Ollama, or private-LLM serving economics broadly.
Tokenizer, output length, and layer count were deliberately left uncontrolled. They're part of what "different model" means, not confounds we tried to eliminate. Qwen3 ran with think:false throughout — a thinking-enabled comparison would be a different, larger experiment.
GPU-only energy. No usable CPU power telemetry exists on this cloud deployment, consistent with every experiment in this series since E005.
Concurrency beyond 8 is untested. Whether cost keeps falling past 8, or where it plateaus, is genuinely unknown from this data.
Single GPU only. We're currently limited to one GPU on Google Cloud, so this series can't yet test whether cost/token holds, improves, or degrades once inference is split across multiple GPUs — a real gap for anyone sizing beyond what a single card can serve.

What's next

Two extensions are on deck for the next experiment, both testable within a single-GPU budget: pushing concurrency past 8 to find where the cost curve actually plateaus (if it does), and varying context length at fixed concurrency to measure how KV cache growth affects both memory and cost — something this series hasn't tested at all, since context length has been held fixed at 4,096 tokens since the first experiment.

Full results

Llama 3.1 8B

Concurrency	GPU power	GPU util	GPU temp	VRAM	Agg tok/s	Mean latency	p95 latency	Mean TTFT	GPU J/tok	$/1K tok	$/1M tok
1	71.37 W	95.23%	74.68 °C	9,403 MB	48.01	12.82 s	13.84 s	0.091 s	1.487 J	$0.00411	$4.106
2	71.70 W	94.38%	74.51 °C	9,471 MB	88.03	13.92 s	15.70 s	0.116 s	0.815 J	$0.00224	$2.239
4	71.67 W	91.68%	74.49 °C	9,471 MB	136.05	17.89 s	20.01 s	0.132 s	0.527 J	$0.00145	$1.449
8	71.51 W	86.93%	73.79 °C	9,458 MB	183.78	26.50 s	29.70 s	0.176 s	0.389 J	$0.00107	$1.073

Qwen3-8B (`think:false`)

Concurrency	GPU power	GPU util	GPU temp	VRAM	Agg tok/s	Mean latency	p95 latency	Mean TTFT	GPU J/tok	$/1K tok	$/1M tok
1	70.52 W	93.32%	73.35 °C	9,974 MB	45.45	11.47 s	12.35 s	0.082 s	1.552 J	$0.00434	$4.337
2	71.37 W	93.24%	74.03 °C	10,190 MB	83.64	12.52 s	13.65 s	0.117 s	0.853 J	$0.00236	$2.357
4	71.42 W	89.28%	74.09 °C	10,198 MB	139.64	14.77 s	16.00 s	0.133 s	0.511 J	$0.00142	$1.412
8	71.33 W	85.51%	73.94 °C	10,201 MB	183.04	22.57 s	25.02 s	0.167 s	0.390 J	$0.00108	$1.077

All 16 runs completed with status=OK, zero failed requests. Rep-to-rep spread on $/1K tokens was ≤1% for every condition except Llama concurrency=8 (4.10%) and Qwen3 concurrency=4 (0.92%) / concurrency=8 (1.67%).

Provenance

Part of the Private LLM Infrastructure Research Series. Extends E006's concurrency methodology with a controlled model substitution — hardware, workload, and measurement pipeline held identical; only the model changed.

models:   llama3.1:8b (Q4_K_M) · sha256:667b0c1932bc6ffc593ed1d03f895bf2dc8dc6df21db3042284a6f4416b06a29
          qwen3:8b (Q4_K_M) · sha256:a3de86cd1c132c822487ededd47a324c50491393e6565cd14bafa40d0b8e686f
hardware: NVIDIA L4 24GB, GCE g2-standard-4 (4 vCPU), on-demand
prompt:   sha256 3a7ea35e…4a26, identical file used since the first experiment in this series
pricing:  GCP Cloud Billing Catalog API, us-central1 — $0.7096/hr (compute+GPU+disk)
window:   2026-09-22 → 2026-09-23

This is a descriptive report of one deployment's measurements — it answers what happened on this hardware, under this workload, not a general claim about model economics. Figures cited from third-party sizing guides were current as of the dates linked and should be re-verified before relying on them.