Union Alpha · G-Bench

01Recipe notes

Quick mode: T2/T3 ran 1 seed

T1 & T4 as configured. Not a full-suite run.

No decode-rate claim: the endpoint buffers output, so decode is unmeasurable and e2e tok/s is reported instead.

Concurrency ladder disabled by choice: the endpoint rate-limits under parallelism (HTTP 429).

T4 graded score rests on 1 gradeable answer of 2 — the judge failed on the other (recorded as a judge error).

T2 negative signal: 40% decoy capture (4/10 t3 runs).

T3: s2_chain completed the flow but returned a wrong final answer.

02Composite score

Weighted gbench-total-v1 breakdown across the four tests.

gbench-total-v1

91.30/100

weighted composite of tests 1–4

test	normalized	weight	contribution
performance	0.7500	0.25	18.75 pts
needle	0.9600	0.30	28.80 pts
tools	0.9500	0.25	23.75 pts
hallucination	1.0000	0.20	20.00 pts

03Performance & capability

TTFT, prefill, decode and end-to-end throughput across sampling passes (Test 1). Decode rate is reported only where it is measurable; e2e tok/s is total tokens / total wall time (prefill + decode) and is the honest figure for an endpoint that buffers its stream.

rtt floor

3,735.3 ms

round-trip to /v1/models

working ctx

196,608 tok

degraded: advertised 262144, works at 196608

pass · context	ttft p50	decode p50	e2e tok/s	itl p95	prefill rate
greedy · 1,024 tok1 runs · fields:	66,094.1 ms	—buffered — not measurable	3.7 tok/s	—	59 tok/s
greedy · 131,072 tok1 runs · fields:	21,959.6 ms	—buffered — not measurable	9.3 tok/s	—	25,208 tok/s
canonical · 1,024 tok1 runs · fields:	21,107.0 ms	—buffered — not measurable	10.6 tok/s	—	unavailable
canonical · 131,072 tok1 runs · fields:	13,029.0 ms	—buffered — not measurable	14.3 tok/s	—	24,939 tok/s
creative · 1,024 tok1 runs · fields:	16,479.2 ms	—buffered — not measurable	13.1 tok/s	—	208 tok/s
creative · 131,072 tok1 runs · fields:	6,466.4 ms	—buffered — not measurable	17.6 tok/s	—	13,659 tok/s

Decode throughput (supplemental probe)

Streaming single-stream decode, 384-token output, temperature 0, thinking tokens included in count.

Decode rate is not measurable for this endpoint: it buffers each answer and flushes it in few stream chunks, so there is no token-level generation window to time. Any tok/s figure derived from chunk arrival would be an artifact (earlier versions of this harness reported thousands of tok/s this way). Honest end-to-end throughput (prefill + decode, total tokens / total wall time): 3.7 tok/s.

04Needle retrieval at depth

Recall by context depth and tier (Test 2). Inline and multi-turn delivery.

mean recall

0.97

over 30 runs (2 seeds × 2 modes × 3 tiers × 3 ctx × 5 depths)

empty answers

1/30

runs returned no text

refusals

0/30

explicit refusal detected

decoy capture

4/30

t3 decoy needles leaked into answers

mode / tier	mean recall
inline / t1_single4,096 tokens · 5 runs	1.00
inline / t1_single117,964 tokens · 5 runs	1.00
inline / t2_double4,096 tokens · 5 runs	1.00
inline / t2_double117,964 tokens · 5 runs	0.80
inline / t3_decoys4,096 tokens · 5 runs	1.00
inline / t3_decoys117,964 tokens · 5 runs	1.00

05Tool calling

Sandboxed scenario completion, timing, and call hygiene (Test 3).

total malformed

across 2 runs

total spurious

extra calls beyond plan

scenario	completion	ttft-tool p50	malformed	spurious	rounds p50
s1_lookup	1.00	1,384.6 ms	0	0	—
s2_chain	1.00	970.7 ms	0	0	—

06Hallucination vs. canonical corpus

Graded semantic answers over a large low-contamination corpus (Test 4).

graded score

1/2

correct · tier t256k · ~205,540 tok delivered

partially correct

0 refused · 0 hallucinated

cold control

6/10

no corpus in context

contamination

graded vs. cold control gap

Delivery tier t256k — statutes in context: lei_10406.

Judge: Qwen3.8-Flash-Next (local vLLM, thinking off) · template t4-semantic-judge-v1.