01Decode throughput
Per-request decode rate and aggregate throughput at each concurrency level. Output capped at 384 tokens, temperature 0, thinking disabled.
single-stream decode
29.26 tok/s
c1 · mean of 3 runs
best aggregate throughput
41.67 tok/s
c2 · all batches
decode budget
384 tok
per request, temp 0
| concurrency | per-req decode | min – max | ttft mean | aggregate |
|---|---|---|---|---|
| c13 reqs · 3 batch(es) | 29.26 tok/s | 29.07 – 29.42 | 372.5 ms | 28.53 tok/s |
| c24 reqs · 2 batch(es) | 21.84 tok/s | 20.82 – 23.09 | 488.1 ms | 41.67 tok/s |
| c48 reqs · 2 batch(es) | 21.11 tok/s | 19.62 – 21.98 | 10,006.0 ms | 41.06 tok/s |
02Prefill & TTFT by prompt size
Time to first token as the prompt grows. Cold = first request for that size; warm = repeated identical prompt (prefix-cached prefill).
| prompt size | ttft cold | ttft warm | prefill rate |
|---|---|---|---|
| ~256actual 186 tokens | 533.9 ms | 565.3 ms | 329 tok/s |
| ~1,000actual 721 tokens | 1,187.4 ms | 418.4 ms | 1,723 tok/s |
| ~4,000actual 2,798 tokens | 3,314.1 ms | 525.4 ms | 5,325 tok/s |
| ~16,000actual 11,187 tokens | 12,153.4 ms | 12,021.9 ms | 931 tok/s |
| ~64,000actual 44,710 tokens | 58,854.9 ms | 46,684.8 ms | 958 tok/s |
| ~128,000actual 89,399 tokens | 110,602.3 ms | 64,033.9 ms | 1,396 tok/s |
03Prefix cache effect
Identical ~16000-token prompt sent repeatedly. The ratio shows how much the serving stack saves on cache hits.
ttft cold
11,987.2 ms
first request, ~11,187 tokens
ttft repeated
12,077.8 ms
identical prompt, prefix cache hit
cache speedup
0.99×
cold / repeated TTFT