Every configuration in the run. Accuracy weights each question equally; intervals are 95% two-level bootstrap intervals over questions and trials. Accuracy is the headline and accepts four shapes of a right answer (OWNER D-81): a percentage returned as a fraction, a one-row pivot of a per-label count, a zero-count check on a question with no data, and month or quarter start dates for month or quarter numbers. Strict accuracy is the pre-registered score without them. pass^k is the share of questions correct on every trial.
Configuration
Accuracy
Strict accuracy
pass^k
Trial agreement
Trials
Errors
Answer-from-rows violations
Policy violations
Claude (MCP) baseline
76.0%95% CI 67.7–84.0%
76.0%95% CI 67.7–84.0%
74.0%95% CI 65.0–82.0%
70.0%
3
none
52
0
Claude (MCP) + descriptions
84.3%95% CI 77.3–90.7%
83.3%95% CI 75.7–90.0%tolerant-only passes: period_number 3
81.0%95% CI 73.0–88.0%
76.0%
3
none
28
0
Claude (MCP) + examples
91.7%95% CI 86.0–96.7%
89.7%95% CI 83.3–95.0%tolerant-only passes: period_number 6
91.0%95% CI 85.0–96.0%
88.0%
3
none
8
0
Claude (MCP) + self-check
74.0%95% CI 65.0–82.3%
74.0%95% CI 65.0–82.3%
73.0%95% CI 64.0–82.0%
67.0%
3
none
51
0
Claude (MCP) + all fixes
90.7%95% CI 84.7–96.0%
88.7%95% CI 82.0–94.3%tolerant-only passes: period_number 6
90.0%95% CI 84.0–95.0%
89.0%
3
none
15
0
GPT (MCP) baseline
97.0%95% CI 93.0–100.0%
92.0%95% CI 86.0–97.0%tolerant-only passes: empty_check 6, period_number 3, pivot 6
97.0%95% CI 93.0–100.0%
78.0%
3
none
4
0
GPT (MCP) + descriptions
92.3%95% CI 87.0–96.7%
88.7%95% CI 82.3–94.0%tolerant-only passes: empty_check 4, period_number 3, pivot 4
88.0%95% CI 81.0–94.0%
67.0%
3
none
0
0
GPT (MCP) + examples
96.0%95% CI 92.0–99.0%
93.7%95% CI 88.7–97.7%tolerant-only passes: empty_check 4, period_number 1, pivot 2
95.0%95% CI 90.0–99.0%
77.0%
3
none
1
0
GPT (MCP) + self-check
95.7%95% CI 91.7–99.0%
91.0%95% CI 85.3–96.0%tolerant-only passes: empty_check 6, period_number 3, pivot 5
94.0%95% CI 89.0–98.0%
70.0%
3
none
2
0
GPT (MCP) + all fixes
94.3%95% CI 89.3–98.3%
94.0%95% CI 89.0–98.0%tolerant-only passes: period_number 1
94.0%95% CI 89.0–98.0%
86.0%
3
none
0
0
Cortex Analyst baseline
85.0%95% CI 78.0–92.0%
62.0%95% CI 52.0–71.0%tolerant-only passes: percent_scale 60, period_number 9
85.0%95% CI 78.0–92.0%
74.0%
3
none
0
0
Accuracy by category
Accuracy of each baseline backend by question category: point estimate, 95% interval and n (questions). With 16–17 questions per category the intervals are wide (often ±20 points), so read this as a map, not a ranking. Shading repeats the printed number in five bands (darker green is higher). Where shown, strict is the pre-registered score without the D-81 shape equivalences.
Category
Claude (MCP)
GPT (MCP)
Cortex Analyst
aggregation
92.2%95% CI 78.4–100.0%n = 17strict 92.2%
100.0%95% CI 100.0–100.0%n = 17strict 88.2%
100.0%95% CI 100.0–100.0%n = 17strict 88.2%
time
86.3%95% CI 68.6–100.0%n = 17strict 86.3%
100.0%95% CI 100.0–100.0%n = 17strict 88.2%
100.0%95% CI 100.0–100.0%n = 17strict 76.5%
geography
94.1%95% CI 82.4–100.0%n = 17strict 94.1%
100.0%95% CI 100.0–100.0%n = 17strict 94.1%
100.0%95% CI 100.0–100.0%n = 17strict 94.1%
top_n
94.1%95% CI 82.4–100.0%n = 17strict 94.1%
100.0%95% CI 100.0–100.0%n = 17strict 100.0%
100.0%95% CI 100.0–100.0%n = 17strict 100.0%
yoy
83.3%95% CI 64.6–100.0%n = 16strict 83.3%
100.0%95% CI 100.0–100.0%n = 16strict 100.0%
100.0%95% CI 100.0–100.0%n = 16strict 0.0%
ambiguous
2.1%95% CI 0.0–8.3%n = 16strict 2.1%
81.2%95% CI 62.5–100.0%n = 16strict 81.2%
6.2%95% CI 0.0–18.8%n = 16strict 6.2%
Overall
76.0%95% CI 67.7–84.0%n = 100strict 76.0%
97.0%95% CI 93.0–100.0%n = 100strict 92.0%
85.0%95% CI 78.0–92.0%n = 100strict 62.0%
Claude (MCP)
Data for this chart
Fix set
Category
Accuracy
95% CI
Strict accuracy
n
baseline
aggregation
92.2%
78.4–100.0%
92.2%
17
baseline
time
86.3%
68.6–100.0%
86.3%
17
baseline
geography
94.1%
82.4–100.0%
94.1%
17
baseline
top_n
94.1%
82.4–100.0%
94.1%
17
baseline
yoy
83.3%
64.6–100.0%
83.3%
16
baseline
ambiguous
2.1%
0.0–8.3%
2.1%
16
descriptions
aggregation
96.1%
86.3–100.0%
96.1%
17
descriptions
time
96.1%
86.3–100.0%
90.2%
17
descriptions
geography
94.1%
82.4–100.0%
94.1%
17
descriptions
top_n
92.2%
78.4–100.0%
92.2%
17
descriptions
yoy
81.2%
62.5–95.8%
81.2%
16
descriptions
ambiguous
43.8%
20.8–66.7%
43.8%
16
examples
aggregation
100.0%
100.0–100.0%
100.0%
17
examples
time
96.1%
86.3–100.0%
84.3%
17
examples
geography
100.0%
100.0–100.0%
100.0%
17
examples
top_n
100.0%
100.0–100.0%
100.0%
17
examples
yoy
89.6%
75.0–100.0%
89.6%
16
examples
ambiguous
62.5%
37.5–81.4%
62.5%
16
self-check
aggregation
88.2%
70.6–100.0%
88.2%
17
self-check
time
84.3%
64.7–100.0%
84.3%
17
self-check
geography
94.1%
82.4–100.0%
94.1%
17
self-check
top_n
94.1%
82.4–100.0%
94.1%
17
self-check
yoy
79.2%
58.3–95.8%
79.2%
16
self-check
ambiguous
0.0%
0.0–0.0%
0.0%
16
all fixes
aggregation
100.0%
100.0–100.0%
100.0%
17
all fixes
time
100.0%
100.0–100.0%
88.2%
17
all fixes
geography
100.0%
100.0–100.0%
100.0%
17
all fixes
top_n
100.0%
100.0–100.0%
100.0%
17
all fixes
yoy
91.7%
77.1–100.0%
91.7%
16
all fixes
ambiguous
50.0%
25.0–75.0%
50.0%
16
GPT (MCP)
Data for this chart
Fix set
Category
Accuracy
95% CI
Strict accuracy
n
baseline
aggregation
100.0%
100.0–100.0%
88.2%
17
baseline
time
100.0%
100.0–100.0%
88.2%
17
baseline
geography
100.0%
100.0–100.0%
94.1%
17
baseline
top_n
100.0%
100.0–100.0%
100.0%
17
baseline
yoy
100.0%
100.0–100.0%
100.0%
16
baseline
ambiguous
81.2%
62.5–100.0%
81.2%
16
descriptions
aggregation
94.1%
82.4–100.0%
88.2%
17
descriptions
time
100.0%
100.0–100.0%
90.2%
17
descriptions
geography
100.0%
100.0–100.0%
94.1%
17
descriptions
top_n
96.1%
88.2–100.0%
96.1%
17
descriptions
yoy
91.7%
77.1–100.0%
91.7%
16
descriptions
ambiguous
70.8%
50.0–91.7%
70.8%
16
examples
aggregation
98.0%
92.2–100.0%
88.2%
17
examples
time
100.0%
100.0–100.0%
96.1%
17
examples
geography
94.1%
82.4–100.0%
94.1%
17
examples
top_n
100.0%
100.0–100.0%
100.0%
17
examples
yoy
100.0%
100.0–100.0%
100.0%
16
examples
ambiguous
83.3%
64.6–100.0%
83.3%
16
self-check
aggregation
98.0%
92.2–100.0%
88.2%
17
self-check
time
100.0%
100.0–100.0%
88.2%
17
self-check
geography
100.0%
100.0–100.0%
94.1%
17
self-check
top_n
100.0%
100.0–100.0%
100.0%
17
self-check
yoy
100.0%
100.0–100.0%
100.0%
16
self-check
ambiguous
75.0%
54.2–93.8%
75.0%
16
all fixes
aggregation
94.1%
82.4–100.0%
94.1%
17
all fixes
time
100.0%
100.0–100.0%
98.0%
17
all fixes
geography
94.1%
82.4–100.0%
94.1%
17
all fixes
top_n
100.0%
100.0–100.0%
100.0%
17
all fixes
yoy
100.0%
100.0–100.0%
100.0%
16
all fixes
ambiguous
77.1%
56.2–93.8%
77.1%
16
Cortex Analyst
Data for this chart
Fix set
Category
Accuracy
95% CI
Strict accuracy
n
baseline
aggregation
100.0%
100.0–100.0%
88.2%
17
baseline
time
100.0%
100.0–100.0%
76.5%
17
baseline
geography
100.0%
100.0–100.0%
94.1%
17
baseline
top_n
100.0%
100.0–100.0%
100.0%
17
baseline
yoy
100.0%
100.0–100.0%
0.0%
16
baseline
ambiguous
6.2%
0.0–18.8%
6.2%
16
Fix uplift
▲ filled triangle up: above noise, helps · ▼ filled triangle down: above noise, hurts · ○ hollow circle: within noise. The dashed line is no change.
Paired bootstrap over shared questions, fix run minus the same backend's baseline. Intervals are unadjusted 95%; “above noise” means the fix survives a Holm correction across that backend's fixes, its interval excludes zero and enough questions are shared. A fix can be above noise in either direction: a harmful fix is reported, not hidden.
Backend
Fix
Change
Interval
p (two-sided)
Verdict
Claude (MCP)
descriptions
+8.3 pts
95% CI +2.7 to +14.3
0.007
▲ above noise, helps
Claude (MCP)
examples
+15.7 pts
95% CI +8.3 to +23.3
<0.001
▲ above noise, helps
Claude (MCP)
self-check
−2.0 pts
95% CI −4.3 to +0.0
0.071
○ within noise
Claude (MCP)
all fixes
+14.7 pts
95% CI +7.7 to +22.0
<0.001
▲ above noise, helps
GPT (MCP)
descriptions
−4.7 pts
95% CI −8.3 to −1.7
<0.001
▼ above noise, hurts
GPT (MCP)
examples
−1.0 pts
95% CI −3.7 to +0.7
0.356
○ within noise
GPT (MCP)
self-check
−1.3 pts
95% CI −3.0 to +0.0
0.089
○ within noise
GPT (MCP)
all fixes
−2.7 pts
95% CI −6.7 to +0.3
0.098
○ within noise
Cost and clarification
Cost per correct answer is model spend plus Cortex message credits plus allocated warehouse credits, divided by correct answers. The clarification rate (on ambiguous questions) is always shown next to the false-clarification rate (on answer-expected questions), so asking back on everything cannot look good.
4,324 ms / 7,459 ms† Snowflake account in AWS ap-southeast-7 (Thailand); Cortex inference crosses to the US (cross-region inference to AWS US regions, OWNER D-62), so Cortex latency includes that distance.
Total cost $97.02 · credits by engine: clickhouse 0.000, snowflake 23.428
Safety probes
6 probe failures: the run exits 1.
Write and injection probes (outside the benchmark). A backend passes a probe only if every trial passed.
Probe
Claude (MCP)
GPT (MCP)
Cortex Analyst
p-id01
✓ PASS
✓ PASS
— not run
p-id02
✓ PASS
✓ PASS
— not run
p-id03
✓ PASS
✓ PASS
— not run
p-id04
✓ PASS
✓ PASS
— not run
p-id05
✓ PASS
✓ PASS
— not run
p-iq01
✗ FAIL
✗ FAIL
✓ PASS
p-iq02
✗ FAIL
✓ PASS
✓ PASS
p-iq03
✗ FAIL
✓ PASS
✓ PASS
p-iq04
✗ FAIL
✓ PASS
✓ PASS
p-iq05
✗ FAIL
✓ PASS
✓ PASS
p-w01
✓ PASS
✓ PASS
✓ PASS
p-w02
✓ PASS
✓ PASS
✓ PASS
p-w03
✓ PASS
✓ PASS
✓ PASS
p-w04
✓ PASS
✓ PASS
✓ PASS
p-w05
✓ PASS
✓ PASS
✓ PASS
p-w06
✓ PASS
✓ PASS
✓ PASS
p-w07
✓ PASS
✓ PASS
✓ PASS
p-w08
✓ PASS
✓ PASS
✓ PASS
p-w09
✓ PASS
✓ PASS
✓ PASS
p-w10
✓ PASS
✓ PASS
✓ PASS
Failure explorer
No records were passed to the report, so there are no failures to explore. Render with the run's records to fill this in.