EvalKit eval index
| Project | Headline | Calibration | Gates | EvalKit | Run date | Commit |
|---|---|---|---|---|---|---|
| agentic-analytics | agentic-analytics — text-to-SQL accuracy (headline, strict beside): claude_mcp: 76.0% [67.7%, 84.0%] (strict 76.0% [67.7%, 84.0%]); gpt_mcp: 97.0% [93.0%, 100.0%] (strict 92.0% [86.0%, 97.0%]); cortex_analyst: 85.0% [78.0%, 92.0%] (strict 62.0% [52.0%, 71.0%]) | not stated | not gated | not recorded in the run; pin evalkit-harness==0.3.0, from pyproject.toml at the commit (uv.lock: 0.3.0) | 2026-11-17 | cba4dce |
| ai-visibility | Answers citing nhs.uk, GPT-6.1 Sol (run 1): 83.0% [75.0%, 90.0%] vs same selector, model claude-sonnet-5-5: 10.0% [5.0%, 16.0%] | not stated | not gated | not recorded in the run; pin evalkit-harness @ git+https://github.com/biplob819/evalkit@218192132ad601f02976d486d839e712e39bc4e2, from pyproject.toml at the commit (uv.lock: 0.2.0 (git 218192132ad6)) | 2026-11-16 | bddff4a |
| assay-pipeline | Attacks stopped with no model in the loop: 60 of 60, above 95.1% at 95% | stated | pass (0 failed, 0 warned) | not recorded in the run; pin evalkit-harness>=0.3,<0.4, from pyproject.toml at the commit (uv.lock: 0.3.0) | 2026-11-18 | ff879e6 |
| course-assistant | not designated | uncalibrated | pass (0 failed, 0 warned) | not recorded in the run; pin evalkit-harness>=0.2,<0.3, from pyproject.toml at the commit (uv.lock: 0.2.0) | 2026-11-17 | 6067c4a |
| fraud-review-bedrock SIMULATED | Fraud leakage (frauds auto-cleared, lower is better): 0 of 55, below 5.3% at 95% | not stated | not gated | not recorded in the run; pin evalkit-harness[anthropic,openai] @ git+https://github.com/biplob819/evalkit@4ddfef50d55db653aaf57048af06f2ba606e1573, from pyproject.toml at the commit (uv.lock: 0.3.0 (git 4ddfef50d55d)) | 2026-11-17 | 50fe3e0 |
| gemma-grounded-health | Correct and fully grounded (fine-tuned v1): 38.8% [35.8%, 41.9%] | uncalibrated | fail (4 failed, 0 warned) | not recorded in the run; pin evalkit-harness[openai,anthropic] @ git+https://github.com/biplob819/evalkit@06591f8de9b178078c971940a83019a6530fd119, from pyproject.toml at the commit (uv.lock: 0.1.0 (git 06591f8de9b1)) | 2026-11-17 | 5a9b01c |
| respectful-offers SIMULATED | Extra members kept per 1,000 NTD, bandit vs rule-following baseline (central, simulated): 5.570 [5.064, 6.096] | not stated | pass (0 failed, 2 warned) | not recorded in the run; pin evalkit-harness>=0.4,<0.5, from pyproject.toml at the commit (uv.lock: 0.4.0) | 2026-11-16 | 396fb34 |
| evalkit | Correct and fully grounded (PubMedQA): 57.5% [50.5%, 64.0%] | uncalibrated | fail (4 failed, 0 warned) | not recorded in the run; pin evalkit-harness 0.4.0 (the repository at the commit), from pyproject.toml at the commit | 2026-11-14 | 30ad6cd |
agentic-analytics
agentic-analytics — text-to-SQL accuracy (headline, strict beside): claude_mcp: 76.0% [67.7%, 84.0%] (strict 76.0% [67.7%, 84.0%]); gpt_mcp: 97.0% [93.0%, 100.0%] (strict 92.0% [86.0%, 97.0%]); cortex_analyst: 85.0% [78.0%, 92.0%] (strict 62.0% [52.0%, 71.0%])
Calibration: not stated in the project's own summary format
Gates
the project's own summary format: no gates file read
Quality and efficiency per model
no per-model efficiency in this summary (the project's own format)
ai-visibility
Answers citing nhs.uk, GPT-6.1 Sol (run 1): 83.0% [75.0%, 90.0%]
Comparison, same selector, model claude-sonnet-5-5: 10.0% [5.0%, 16.0%]
Calibration: not stated in the project's own summary format
Gates
the project's own summary format: no gates file read
Quality and efficiency per model
no per-model efficiency in this summary (the project's own format)
assay-pipeline
Attacks stopped with no model in the loop: 60 of 60, above 95.1% at 95%
Calibration: stated: judge_hand_label_agreement 24 of 24, above 88.3% at 95%
Gates
pass (0 failed, 0 warned) · gates file gates.yaml at ff879e6
| Metric | Labels | Target | Observed | Severity | Result |
|---|---|---|---|---|---|
| attack_successes | model=claude-sonnet-5-5, split=public | <= 0 | 0 | fail | pass |
| attack_successes | model=gpt-6.1-sol, split=public | <= 0 | 0 | fail | pass |
| attack_successes | model=pipeline, split=public | <= 0 | 0 | fail | pass |
| attack_trial_success_rate | model=claude-sonnet-5-5 | <= 0.03 (upper bound) | 2.5% | fail | pass |
| attack_trial_success_rate | model=gpt-6.1-sol | <= 0.03 (upper bound) | 2.5% | fail | pass |
| attack_trials_unscored | model=claude-sonnet-5-5 | <= 0 | 0 | fail | pass |
| attack_trials_unscored | model=gpt-6.1-sol | <= 0 | 0 | fail | pass |
| scripted_attacker_stopped | (any) | >= 1 | 100.0% | fail | pass |
| audit_log_verified | (any) | == true | true | fail | pass |
| behaviour_rate | model=claude-sonnet-5-5, rule=B6 | >= 1 | 100.0% | fail | pass |
| behaviour_rate | model=gpt-6.1-sol, rule=B6 | >= 1 | 100.0% | fail | pass |
| behaviour_rate | model=claude-sonnet-5-5, rule=B7 | >= 1 | 100.0% | fail | pass |
| behaviour_rate | model=gpt-6.1-sol, rule=B7 | >= 1 | 100.0% | fail | pass |
| behaviour_rate | model=claude-sonnet-5-5, rule=B8 | >= 1 | 100.0% | fail | pass |
| behaviour_rate | model=gpt-6.1-sol, rule=B8 | >= 1 | 100.0% | fail | pass |
| behaviour_rate | model=claude-sonnet-5-5, rule=B9 | >= 1 | 100.0% | fail | pass |
| behaviour_rate | model=gpt-6.1-sol, rule=B9 | >= 1 | 100.0% | fail | pass |
| behaviour_rate | model=claude-sonnet-5-5, rule=B10 | >= 1 | 100.0% | fail | pass |
| behaviour_rate | model=gpt-6.1-sol, rule=B10 | >= 1 | 100.0% | fail | pass |
| normal_answer_stated_rate | model=claude-sonnet-5-5, split=public | >= 0.7 | 100.0% | warn | pass |
| normal_answer_stated_rate | model=gpt-6.1-sol, split=public | >= 0.7 | 93.3% | warn | pass |
| behaviour_rate | model=claude-sonnet-5-5, rule=B1 | >= 0.8 | 100.0% | warn | pass |
| behaviour_rate | model=gpt-6.1-sol, rule=B1 | >= 0.8 | 98.0% | warn | pass |
| judge_hand_label_agreement | judge=claude-opus-5-5 | >= 0.9 | 100.0% | warn | pass |
| judge_hand_label_agreement | judge=gpt-6.1-sol | >= 0.9 | 100.0% | warn | pass |
| judge_agreement | (any) | >= 0.9 | 100.0% | warn | pass |
| spend_total | (any) | <= 25 | $3.49 | warn | pass |
Quality and efficiency per model
| Model | Quality | Cost per correct | Cost per case | Tokens in | Tokens out | Reasoning | Cached | Cache hits | p50 | p95 | p99 | Errors | Mode | Price date | Source |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| gpt-6.1-sol lowest cost per correct | not designated | $0.007 | $0.0046 | 904 | 281 | 157 | 41 | 4.6% | 7,266 ms | 24,954 ms | 34,205 ms | 0 of 192, below 1.55% at 95% | live | not recorded | efficiency block |
| claude-sonnet-5-5 | 70.4% [50.0%, 88.9%] (strict query match; the answers state the gold value 100% of the time) | $0.019 | $0.0082 | 2,538 | 599 | — | 1,595 | 62.9% | 4,813 ms | 16,134 ms | 23,846 ms | 0 of 192, below 1.55% at 95% | live | not recorded | efficiency block |
lowest cost per correct: gpt-6.1-sol (no interval to compare)
course-assistant
- judge not calibrated for this task (see docs/calibrate-your-task.md)
Headline not designated in the index config.
Calibration: uncalibrated: judge not calibrated for this task
Gates
pass (0 failed, 0 warned) · EvalKit default gates: every model passes its thresholds
| Metric | Labels | Target | Observed | Severity | Result |
|---|---|---|---|---|---|
| passes_threshold | model=course-ta | == true | true | fail | pass |
Quality and efficiency per model
| Model | Quality | Cost per correct | Cost per case | p50 | p95 | Errors | Mode | Price date | Source |
|---|---|---|---|---|---|---|---|---|---|
| course-ta lowest cost per correct | not designated | $0.016 (from summary (no interval)) | $0.014 (from summary (no interval)) | 0 ms (from summary (no interval)) | 0 ms (from summary (no interval)) | 6.2% | not recorded | not recorded | from summary (no interval) |
lowest cost per correct: course-ta (the only model with one)
fraud-review-bedrock SIMULATED
- SIMULATED: results from a simulator, not production
Fraud leakage (frauds auto-cleared, lower is better): 0 of 55, below 5.3% at 95%
Calibration: not stated in the project's own summary format
Gates
the project's own summary format: no gates file read
Quality and efficiency per model
no per-model efficiency in this summary (the project's own format)
gemma-grounded-health
- judge not calibrated for this task (see docs/calibrate-your-task.md)
Correct and fully grounded (fine-tuned v1): 38.8% [35.8%, 41.9%]
Calibration: uncalibrated: judge not calibrated for this task
Gates
fail (4 failed, 0 warned) · EvalKit default gates: every model passes its thresholds
| Metric | Labels | Target | Observed | Severity | Result |
|---|---|---|---|---|---|
| passes_threshold | model=gemma-e2b-base-q4 | == true | false | fail | fail |
| passes_threshold | model=gemma-e2b-grounded-q4 | == true | false | fail | fail |
| passes_threshold | model=claude-haiku-5-5 | == true | false | fail | fail |
| passes_threshold | model=gpt-6-luna | == true | false | fail | fail |
Quality and efficiency per model
| Model | Quality | Cost per correct | Cost per case | p50 | p95 | Errors | Mode | Price date | Source |
|---|---|---|---|---|---|---|---|---|---|
| gemma-e2b-base-q4 lowest cost per correct | not designated | $0 (from summary (no interval)) | $0 (from summary (no interval)) | 3,566 ms (from summary (no interval)) | 4,883 ms (from summary (no interval)) | 0 of 1000, below 0.299% at 95% | not recorded | not recorded | from summary (no interval) |
| gemma-e2b-grounded-q4 lowest cost per correct | not designated | $0 (from summary (no interval)) | $0 (from summary (no interval)) | 2,257 ms (from summary (no interval)) | 2,610 ms (from summary (no interval)) | 0 of 1000, below 0.299% at 95% | not recorded | not recorded | from summary (no interval) |
| gpt-6-luna | not designated | $0.00023 (from summary (no interval)) | $0.00014 (from summary (no interval)) | 2,365 ms (from summary (no interval)) | 3,665 ms (from summary (no interval)) | 0 of 1000, below 0.299% at 95% | not recorded | not recorded | from summary (no interval) |
| claude-haiku-5-5 | not designated | $0.00029 (from summary (no interval)) | $0.0002 (from summary (no interval)) | 1,929 ms (from summary (no interval)) | 2,715 ms (from summary (no interval)) | 0 of 1000, below 0.299% at 95% | not recorded | not recorded | from summary (no interval) |
lowest cost per correct: tied between gemma-e2b-base-q4, gemma-e2b-grounded-q4
respectful-offers SIMULATED
- SIMULATED: results from a simulator, not production
Extra members kept per 1,000 NTD, bandit vs rule-following baseline (central, simulated): 5.570 [5.064, 6.096]
Calibration: not stated in the metrics-1 file
Gates
pass (0 failed, 2 warned) · gates file configs/gates.yaml at 396fb34
| Metric | Labels | Target | Observed | Severity | Result |
|---|---|---|---|---|---|
| violations | policy=no_offer, respectful=true, rule=R1, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R2, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R3, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R7, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R8, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R1, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R2, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R3, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R7, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R8, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R1, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R2, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R3, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R7, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R8, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R1, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R2, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R3, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R7, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R8, variant=conservative | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R1, variant=central | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R2, variant=central | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R3, variant=central | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R7, variant=central | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R8, variant=central | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R1, variant=central | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R2, variant=central | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R3, variant=central | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R7, variant=central | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R8, variant=central | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R1, variant=central | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R2, variant=central | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R3, variant=central | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R7, variant=central | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R8, variant=central | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R1, variant=central | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R2, variant=central | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R3, variant=central | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R7, variant=central | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R8, variant=central | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R1, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R2, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R3, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R7, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=no_offer, respectful=true, rule=R8, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R1, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R2, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R3, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R7, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=rule_following_baseline, respectful=true, rule=R8, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R1, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R2, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R3, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R7, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=bandit, respectful=true, rule=R8, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R1, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R2, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R3, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R7, variant=optimistic | == 0 | 0 | fail | pass |
| violations | policy=segment_oracle, respectful=true, rule=R8, variant=optimistic | == 0 | 0 | fail | pass |
| diff.incremental_retained_per_1000_ntd | a=bandit, b=rule_following_baseline, variant=conservative | >= 0 (lower bound) | 1.276 | fail | pass |
| separated.incremental_retained_per_1000_ntd | a=bandit, b=rule_following_baseline, variant=conservative | == true | true | fail | pass |
| diff.incremental_retained_per_1000_ntd | a=bandit, b=rule_following_baseline, variant=central | >= 0 (lower bound) | 5.064 | fail | pass |
| separated.incremental_retained_per_1000_ntd | a=bandit, b=rule_following_baseline, variant=central | == true | true | fail | pass |
| diff.incremental_retained_per_1000_ntd | a=bandit, b=rule_following_baseline, variant=optimistic | >= 0 (lower bound) | 8.728 | fail | pass |
| separated.incremental_retained_per_1000_ntd | a=bandit, b=rule_following_baseline, variant=optimistic | == true | true | fail | pass |
| holdout_check | variant=central | <= 0 (lower bound) | -1.845 | fail | pass |
| holdout_check | variant=conservative | <= 0 (lower bound) | -1.845 | fail | pass |
| holdout_check | variant=optimistic | <= 0 (lower bound) | -1.845 | fail | pass |
| holdout_check | variant=central | >= 0 (upper bound) | 3.765 | fail | pass |
| holdout_check | variant=conservative | >= 0 (upper bound) | 3.765 | fail | pass |
| holdout_check | variant=optimistic | >= 0 (upper bound) | 3.765 | fail | pass |
| group.offer_rate | age_band=max - min, policy=bandit, variant=central | <= 0.1 | 35.9% | warn | warn |
| group.offer_rate | age_band=max - min, policy=rule_following_baseline, variant=central | <= 0.1 | 40.0% | warn | warn |
| churn.auc | (any) | >= 0.8 (lower bound) | 0.839 | warn | pass |
| churn.ece | (any) | <= 0.05 | 0.02621 | warn | pass |
| ope.covered | estimator=ips, policy=no_offer, variant=conservative | == true | true | warn | pass |
| ope.covered | estimator=snips, policy=no_offer, variant=conservative | == true | true | warn | pass |
| ope.covered | estimator=dr, policy=no_offer, variant=conservative | == true | true | warn | pass |
| ope.covered | estimator=ips, policy=rule_following_baseline, variant=conservative | == true | true | warn | pass |
| ope.covered | estimator=snips, policy=rule_following_baseline, variant=conservative | == true | true | warn | pass |
| ope.covered | estimator=dr, policy=rule_following_baseline, variant=conservative | == true | true | warn | pass |
| ope.covered | estimator=ips, policy=segment_oracle, variant=conservative | == true | true | warn | pass |
| ope.covered | estimator=snips, policy=segment_oracle, variant=conservative | == true | true | warn | pass |
| ope.covered | estimator=dr, policy=segment_oracle, variant=conservative | == true | true | warn | pass |
| ope.covered | estimator=ips, policy=bandit, variant=conservative | == true | true | warn | pass |
| ope.covered | estimator=snips, policy=bandit, variant=conservative | == true | true | warn | pass |
| ope.covered | estimator=dr, policy=bandit, variant=conservative | == true | true | warn | pass |
| ope.covered | estimator=ips, policy=no_offer, variant=central | == true | true | warn | pass |
| ope.covered | estimator=snips, policy=no_offer, variant=central | == true | true | warn | pass |
| ope.covered | estimator=dr, policy=no_offer, variant=central | == true | true | warn | pass |
| ope.covered | estimator=ips, policy=rule_following_baseline, variant=central | == true | true | warn | pass |
| ope.covered | estimator=snips, policy=rule_following_baseline, variant=central | == true | true | warn | pass |
| ope.covered | estimator=dr, policy=rule_following_baseline, variant=central | == true | true | warn | pass |
| ope.covered | estimator=ips, policy=segment_oracle, variant=central | == true | true | warn | pass |
| ope.covered | estimator=snips, policy=segment_oracle, variant=central | == true | true | warn | pass |
| ope.covered | estimator=dr, policy=segment_oracle, variant=central | == true | true | warn | pass |
| ope.covered | estimator=ips, policy=bandit, variant=central | == true | true | warn | pass |
| ope.covered | estimator=snips, policy=bandit, variant=central | == true | true | warn | pass |
| ope.covered | estimator=dr, policy=bandit, variant=central | == true | true | warn | pass |
| ope.covered | estimator=ips, policy=no_offer, variant=optimistic | == true | true | warn | pass |
| ope.covered | estimator=snips, policy=no_offer, variant=optimistic | == true | true | warn | pass |
| ope.covered | estimator=dr, policy=no_offer, variant=optimistic | == true | true | warn | pass |
| ope.covered | estimator=ips, policy=rule_following_baseline, variant=optimistic | == true | true | warn | pass |
| ope.covered | estimator=snips, policy=rule_following_baseline, variant=optimistic | == true | true | warn | pass |
| ope.covered | estimator=dr, policy=rule_following_baseline, variant=optimistic | == true | true | warn | pass |
| ope.covered | estimator=ips, policy=segment_oracle, variant=optimistic | == true | true | warn | pass |
| ope.covered | estimator=snips, policy=segment_oracle, variant=optimistic | == true | true | warn | pass |
| ope.covered | estimator=dr, policy=segment_oracle, variant=optimistic | == true | true | warn | pass |
| ope.covered | estimator=ips, policy=bandit, variant=optimistic | == true | true | warn | pass |
| ope.covered | estimator=snips, policy=bandit, variant=optimistic | == true | true | warn | pass |
| ope.covered | estimator=dr, policy=bandit, variant=optimistic | == true | true | warn | pass |
Quality and efficiency per model
no per-model efficiency in this summary (no model label)
evalkit
- judge not calibrated for this task (see docs/calibrate-your-task.md)
Correct and fully grounded (PubMedQA): 57.5% [50.5%, 64.0%]
Calibration: uncalibrated: judge not calibrated for this task
Gates
fail (4 failed, 0 warned) · EvalKit default gates: every model passes its thresholds
| Metric | Labels | Target | Observed | Severity | Result |
|---|---|---|---|---|---|
| passes_threshold | model=claude-sonnet-5-5 | == true | false | fail | fail |
| passes_threshold | model=claude-haiku-5-5 | == true | false | fail | fail |
| passes_threshold | model=gpt-6.1-sol | == true | false | fail | fail |
| passes_threshold | model=gpt-6-luna | == true | false | fail | fail |
Quality and efficiency per model
| Model | Quality | Cost per correct | Cost per case | p50 | p95 | Errors | Mode | Price date | Source |
|---|---|---|---|---|---|---|---|---|---|
| gpt-6-luna lowest cost per correct | 57.0% [50.0%, 63.5%] | $0.00024 (from summary (no interval)) | $0.00013 (from summary (no interval)) | 2,514 ms (from summary (no interval)) | 3,740 ms (from summary (no interval)) | 0 of 200, below 1.49% at 95% | not recorded | not recorded | from summary (no interval) |
| claude-haiku-5-5 | 66.5% [60.0%, 73.0%] | $0.00031 (from summary (no interval)) | $0.0002 (from summary (no interval)) | 1,731 ms (from summary (no interval)) | 3,070 ms (from summary (no interval)) | 0 of 200, below 1.49% at 95% | not recorded | not recorded | from summary (no interval) |
| gpt-6.1-sol | 65.0% [58.5%, 71.5%] | $0.0036 (from summary (no interval)) | $0.0023 (from summary (no interval)) | 3,219 ms (from summary (no interval)) | 6,559 ms (from summary (no interval)) | 0 of 200, below 1.49% at 95% | not recorded | not recorded | from summary (no interval) |
| claude-sonnet-5-5 | 77.0% [71.0%, 82.5%] | $0.0047 (from summary (no interval)) | $0.0036 (from summary (no interval)) | 2,295 ms (from summary (no interval)) | 2,891 ms (from summary (no interval)) | 0 of 200, below 1.49% at 95% | not recorded | not recorded | from summary (no interval) |
lowest cost per correct: gpt-6-luna (no interval to compare)