01Decode throughput
Per-request decode rate and aggregate throughput at each concurrency level. Output capped at 384 tokens, temperature 0, thinking disabled.
single-stream decode
22.80 tok/s
c1 · mean of 3 runs
best aggregate throughput
31.53 tok/s
c4 · all batches
decode budget
384 tok
per request, temp 0
| concurrency | per-req decode | min – max | ttft mean | aggregate |
|---|---|---|---|---|
| c13 reqs · 3 batch(es) | 22.80 tok/s | 22.56 – 22.99 | 390.1 ms | 22.34 tok/s |
| c24 reqs · 2 batch(es) | 21.22 tok/s | 17.98 – 24.47 | 8,322.2 ms | 24.52 tok/s |
| c48 reqs · 2 batch(es) | 16.95 tok/s | 13.84 – 23.37 | 10,800.1 ms | 31.53 tok/s |
02Prefill & TTFT by prompt size
Time to first token as the prompt grows. Cold = first request for that size; warm = repeated identical prompt (prefix-cached prefill).
| prompt size | ttft cold | ttft warm | prefill rate |
|---|---|---|---|
| ~256actual 185 tokens | 3,611.0 ms | 809.1 ms | 229 tok/s |
| ~1,000actual 712 tokens | 1,073.5 ms | 1,087.2 ms | 655 tok/s |
| ~4,000actual 2,751 tokens | 2,525.5 ms | 2,227.6 ms | 1,235 tok/s |
| ~16,000actual 10,985 tokens | 7,647.3 ms | 690.3 ms | 15,913 tok/s |
| ~64,000actual 43,902 tokens | 25,071.6 ms | 1,065.5 ms | 41,203 tok/s |
| ~128,000actual 87,787 tokens | 29,706.1 ms | 1,685.6 ms | 52,081 tok/s |
03Prefix cache effect
Identical ~16000-token prompt sent repeatedly. The ratio shows how much the serving stack saves on cache hits.
ttft cold
12,332.2 ms
first request, ~10,985 tokens
ttft repeated
727.7 ms
identical prompt, prefix cache hit
cache speedup
16.95×
cold / repeated TTFT