Ornith-1.5-35B-A3B

01Verdict summary

Advertised context 262,144 — fully usable. Verified to the 327,680-token rejection point (131k / 196k / 262k probes all ok).

MoE throughput profile: ~174 tok/s decode at short context dropping to ~64 tok/s at 131k — attention cost over the long context dominates even for a 3B-active MoE. 2.3× long-context decode penalty.

Concurrency: 3 parallel streams sustain 524 tok/s aggregate at 1k ctx (3.3× solo); at 131k ctx it collapses to solo (1 stream, 66 tok/s) — the long-context KV working set prevents effective batching.

Prefill curve is non-monotonic: 2.9k tok/s @ 256 → 9.8k tok/s @ 16k → 6.2k tok/s @ 131k. Small prompts don't saturate; huge prompts become memory-bandwidth-bound. TTFT @ 131k ≈ 27s.

Sampling sensitivity is high: early stops at temp ≥ 0.7 well below the 256-token cap (canonical@4096: [111, 24, 7] tokens). Greedy is most stable. Serve with temp ≤ 0.4 and/or a strong repetition penalty for long-form output. Decode rates from short runs remain valid (steady-state ITL).

02GBench total score

total

76.55 / 100

performance + needle + tools + hallucination

best dimension

0.95

tool calling

weakest dimension

0.25

hallucination (t256k tier)

test	normalized	weight	contribution
performance	0.8620	0.25	21.55 pts
needle	0.8750	0.30	26.25 pts
tools	0.9500	0.25	23.75 pts
hallucination tier t256k	0.2500	0.20	5.00 pts

03Decode throughput

Baseline decode by context size, canonical sampling pass (temperature, top_k, top_p).

decode @ 1k ctx

174.7 tok/s

canonical · p50/p95 175.2

decode @ 131k ctx

64.1 tok/s

2.3× long-context penalty

RTT floor

50.1 ms p50

10 probes

ctx	ttft raw	ttft corr	prefill tok/s	decode tok/s	itl p50/p95	e2e
256	1,555.9 ms	1,501.5 ms	2,877	168.3/171.4	5.88/7.52 ms	1,636 ms
1024	1,113.8 ms	1,059.3 ms	6,021	174.7/175.2	5.65/7.82 ms	1,711 ms
4096	2,025.5 ms	1,971.0 ms	8,481	174.1/185.0	5.83/8.26 ms	2,133 ms
16384	3,133.0 ms	3,078.6 ms	9,803	137.2/148.7	7.56/9.27 ms	3,847 ms
131072	27,018.2 ms	26,963.8 ms	6,253	64.1/65.6	15.68/19.51 ms	28,828 ms

Greedy pass (seed, temperature — reduced matrix) and creative pass (min_p, repeat_penalty, temperature, top_p — reduced matrix) agree: 154 / 67 tok/s greedy and 136 / 60 tok/s creative at 1k / 131k.

04Sustained concurrency

point	solo decode	qos floor	max sustained	aggregate tok/s
ctx 1024	159.6 tok/s	95.8	3	524.4
ctx 50% (131k)	65.5 tok/s	39.3	1	66.3

Context ceiling: advertised 262,144 = actual working 262,144. Probes — 131,072 ok · 196,608 ok · 262,144 ok · 327,680 rejected (http_400).

05Needle retrieval

180 runs · inline 90 / multi_turn 90 · depths 0.15–1.0 · ctx up to 235,929 · exact-match judge. Decoy capture 0.0% · refusal rate 0.0%. Perfect 1.00 across t1_single and t2_double at every depth and context, both modes — the only losses are t3_decoys at high temperature, with a clear middle dip.

tier · mode	4,096 ctx	117,964 ctx	235,929 ctx	middle dip
t1_single · inline all depths	1.00	1.00	1.00	no
t1_single · multi_turn all depths	1.00	1.00	1.00	no
t2_double · inline all depths	1.00	1.00	1.00	no
t2_double · multi_turn all depths	1.00	1.00	1.00	no
t3_decoys · inline best depth	1.00 @0.15	0.50 @0.15	0.50 @0.15	yes
t3_decoys · multi_turn best depth	1.00 @0.6	0.75 @0.6	0.25 @1.0	yes

06Tool calling

Native wire format · 2 seeds per scenario · all 6 runs completed, 0 malformed, 0 wrong branches. One spurious call per seed on average was always recovered.

scenario	completed	rounds p50	ttft-tool p50	spurious
s1_lookup	1.00	3.00	57,620 ms	2
s2_chain	1.00	4.00	517 ms	2
s3_branch	1.00	5.00	469 ms	2

07Hallucination

Delivery tier t256k — corpus del_3048 + lei_13709 in context, ~206,877 tokens delivered. Graded run scored 0/2 correct: both questions refused, none hallucinated. Cold control (no corpus) also 0/10 — not contaminated. The score reflects over-refusal at 256k delivery, not fabrication.