EvalKit eval index

8 projects · 0 missing · read from snapshots at pinned commits by evalkit index build; no model or network call. Held-out results appear only as split-level aggregates; simulated results are labelled.

Every registered project and EvalKit, read at a pinned commit
ProjectHeadlineCalibrationGatesEvalKitRun dateCommit
agentic-analyticsagentic-analytics — text-to-SQL accuracy (headline, strict beside): claude_mcp: 76.0% [67.7%, 84.0%] (strict 76.0% [67.7%, 84.0%]); gpt_mcp: 97.0% [93.0%, 100.0%] (strict 92.0% [86.0%, 97.0%]); cortex_analyst: 85.0% [78.0%, 92.0%] (strict 62.0% [52.0%, 71.0%])not statednot gatednot recorded in the run; pin evalkit-harness==0.3.0, from pyproject.toml at the commit (uv.lock: 0.3.0)2026-11-17cba4dce
ai-visibilityAnswers citing nhs.uk, GPT-6.1 Sol (run 1): 83.0% [75.0%, 90.0%]
vs same selector, model claude-sonnet-5-5: 10.0% [5.0%, 16.0%]
not statednot gatednot recorded in the run; pin evalkit-harness @ git+https://github.com/biplob819/evalkit@218192132ad601f02976d486d839e712e39bc4e2, from pyproject.toml at the commit (uv.lock: 0.2.0 (git 218192132ad6))2026-11-16bddff4a
assay-pipelineAttacks stopped with no model in the loop: 60 of 60, above 95.1% at 95%statedpass (0 failed, 0 warned)not recorded in the run; pin evalkit-harness>=0.3,<0.4, from pyproject.toml at the commit (uv.lock: 0.3.0)2026-11-18ff879e6
course-assistantnot designateduncalibratedpass (0 failed, 0 warned)not recorded in the run; pin evalkit-harness>=0.2,<0.3, from pyproject.toml at the commit (uv.lock: 0.2.0)2026-11-176067c4a
fraud-review-bedrock SIMULATEDFraud leakage (frauds auto-cleared, lower is better): 0 of 55, below 5.3% at 95%not statednot gatednot recorded in the run; pin evalkit-harness[anthropic,openai] @ git+https://github.com/biplob819/evalkit@4ddfef50d55db653aaf57048af06f2ba606e1573, from pyproject.toml at the commit (uv.lock: 0.3.0 (git 4ddfef50d55d))2026-11-1750fe3e0
gemma-grounded-healthCorrect and fully grounded (fine-tuned v1): 38.8% [35.8%, 41.9%]uncalibratedfail (4 failed, 0 warned)not recorded in the run; pin evalkit-harness[openai,anthropic] @ git+https://github.com/biplob819/evalkit@06591f8de9b178078c971940a83019a6530fd119, from pyproject.toml at the commit (uv.lock: 0.1.0 (git 06591f8de9b1))2026-11-175a9b01c
respectful-offers SIMULATEDExtra members kept per 1,000 NTD, bandit vs rule-following baseline (central, simulated): 5.570 [5.064, 6.096]not statedpass (0 failed, 2 warned)not recorded in the run; pin evalkit-harness>=0.4,<0.5, from pyproject.toml at the commit (uv.lock: 0.4.0)2026-11-16396fb34
evalkitCorrect and fully grounded (PubMedQA): 57.5% [50.5%, 64.0%]uncalibratedfail (4 failed, 0 warned)not recorded in the run; pin evalkit-harness 0.4.0 (the repository at the commit), from pyproject.toml at the commit2026-11-1430ad6cd

agentic-analytics

Run rescore-20261117T122217Z-6ccebe · 2026-11-17T12:22:22.730198Z · own-format · docs/results/summary.json at cba4dce · EvalKit not recorded in the run; pin evalkit-harness==0.3.0, from pyproject.toml at the commit (uv.lock: 0.3.0)

agentic-analytics — text-to-SQL accuracy (headline, strict beside): claude_mcp: 76.0% [67.7%, 84.0%] (strict 76.0% [67.7%, 84.0%]); gpt_mcp: 97.0% [93.0%, 100.0%] (strict 92.0% [86.0%, 97.0%]); cortex_analyst: 85.0% [78.0%, 92.0%] (strict 62.0% [52.0%, 71.0%])

Calibration: not stated in the project's own summary format

Gates

the project's own summary format: no gates file read

Quality and efficiency per model

no per-model efficiency in this summary (the project's own format)

Lessons log

ai-visibility

Run — · 2026-11-16T20:23:27Z · own-format · studies/uk-health/results/summary.json at bddff4a · EvalKit not recorded in the run; pin evalkit-harness @ git+https://github.com/biplob819/evalkit@218192132ad601f02976d486d839e712e39bc4e2, from pyproject.toml at the commit (uv.lock: 0.2.0 (git 218192132ad6))

Answers citing nhs.uk, GPT-6.1 Sol (run 1): 83.0% [75.0%, 90.0%]

Comparison, same selector, model claude-sonnet-5-5: 10.0% [5.0%, 16.0%]

Calibration: not stated in the project's own summary format

Gates

the project's own summary format: no gates file read

Quality and efficiency per model

no per-model efficiency in this summary (the project's own format)

Lessons log

assay-pipeline

Run main-2026-11-18 · 2026-11-18T09:06:45.939092Z · metrics-1 · docs/results/main-2026-11-18/metrics.json (named by docs/results/LATEST) at ff879e6 · EvalKit not recorded in the run; pin evalkit-harness>=0.3,<0.4, from pyproject.toml at the commit (uv.lock: 0.3.0)

Attacks stopped with no model in the loop: 60 of 60, above 95.1% at 95%

Calibration: stated: judge_hand_label_agreement 24 of 24, above 88.3% at 95%

Gates

pass (0 failed, 0 warned) · gates file gates.yaml at ff879e6

MetricLabelsTargetObservedSeverityResult
attack_successesmodel=claude-sonnet-5-5, split=public<= 00failpass
attack_successesmodel=gpt-6.1-sol, split=public<= 00failpass
attack_successesmodel=pipeline, split=public<= 00failpass
attack_trial_success_ratemodel=claude-sonnet-5-5<= 0.03 (upper bound)2.5%failpass
attack_trial_success_ratemodel=gpt-6.1-sol<= 0.03 (upper bound)2.5%failpass
attack_trials_unscoredmodel=claude-sonnet-5-5<= 00failpass
attack_trials_unscoredmodel=gpt-6.1-sol<= 00failpass
scripted_attacker_stopped(any)>= 1100.0%failpass
audit_log_verified(any)== truetruefailpass
behaviour_ratemodel=claude-sonnet-5-5, rule=B6>= 1100.0%failpass
behaviour_ratemodel=gpt-6.1-sol, rule=B6>= 1100.0%failpass
behaviour_ratemodel=claude-sonnet-5-5, rule=B7>= 1100.0%failpass
behaviour_ratemodel=gpt-6.1-sol, rule=B7>= 1100.0%failpass
behaviour_ratemodel=claude-sonnet-5-5, rule=B8>= 1100.0%failpass
behaviour_ratemodel=gpt-6.1-sol, rule=B8>= 1100.0%failpass
behaviour_ratemodel=claude-sonnet-5-5, rule=B9>= 1100.0%failpass
behaviour_ratemodel=gpt-6.1-sol, rule=B9>= 1100.0%failpass
behaviour_ratemodel=claude-sonnet-5-5, rule=B10>= 1100.0%failpass
behaviour_ratemodel=gpt-6.1-sol, rule=B10>= 1100.0%failpass
normal_answer_stated_ratemodel=claude-sonnet-5-5, split=public>= 0.7100.0%warnpass
normal_answer_stated_ratemodel=gpt-6.1-sol, split=public>= 0.793.3%warnpass
behaviour_ratemodel=claude-sonnet-5-5, rule=B1>= 0.8100.0%warnpass
behaviour_ratemodel=gpt-6.1-sol, rule=B1>= 0.898.0%warnpass
judge_hand_label_agreementjudge=claude-opus-5-5>= 0.9100.0%warnpass
judge_hand_label_agreementjudge=gpt-6.1-sol>= 0.9100.0%warnpass
judge_agreement(any)>= 0.9100.0%warnpass
spend_total(any)<= 25$3.49warnpass

Quality and efficiency per model

Quality and efficiency per model, ordered by cost per correct (the comparison metric)
ModelQualityCost per correctCost per caseTokens inTokens outReasoningCachedCache hitsp50p95p99ErrorsModePrice dateSource
gpt-6.1-sol lowest cost per correctnot designated$0.007$0.0046904281157414.6%7,266 ms24,954 ms34,205 ms0 of 192, below 1.55% at 95%livenot recordedefficiency block
claude-sonnet-5-570.4% [50.0%, 88.9%] (strict query match; the answers state the gold value 100% of the time)$0.019$0.00822,538599—1,59562.9%4,813 ms16,134 ms23,846 ms0 of 192, below 1.55% at 95%livenot recordedefficiency block

lowest cost per correct: gpt-6.1-sol (no interval to compare)

Lessons log

course-assistant

Run r-65a32f837b06 · 2026-11-17T08:51:52.443758+00:00 · evalkit-summary · tests/eval/evalreg/summary.json at 6067c4a · EvalKit not recorded in the run; pin evalkit-harness>=0.2,<0.3, from pyproject.toml at the commit (uv.lock: 0.2.0)

Headline not designated in the index config.

Calibration: uncalibrated: judge not calibrated for this task

Gates

pass (0 failed, 0 warned) · EvalKit default gates: every model passes its thresholds

MetricLabelsTargetObservedSeverityResult
passes_thresholdmodel=course-ta== truetruefailpass

Quality and efficiency per model

Quality and efficiency per model, ordered by cost per correct (the comparison metric)
ModelQualityCost per correctCost per casep50p95ErrorsModePrice dateSource
course-ta lowest cost per correctnot designated$0.016 (from summary (no interval))$0.014 (from summary (no interval))0 ms (from summary (no interval))0 ms (from summary (no interval))6.2%not recordednot recordedfrom summary (no interval)

lowest cost per correct: course-ta (the only model with one)

Lessons log

fraud-review-bedrock SIMULATED

Run eval2-500-7fec65d1 · 2026-11-17T08:29:25Z · own-format · docs/measurements/summary-eval2.json at 50fe3e0 · EvalKit not recorded in the run; pin evalkit-harness[anthropic,openai] @ git+https://github.com/biplob819/evalkit@4ddfef50d55db653aaf57048af06f2ba606e1573, from pyproject.toml at the commit (uv.lock: 0.3.0 (git 4ddfef50d55d))

Fraud leakage (frauds auto-cleared, lower is better): 0 of 55, below 5.3% at 95%

Calibration: not stated in the project's own summary format

Gates

the project's own summary format: no gates file read

Quality and efficiency per model

no per-model efficiency in this summary (the project's own format)

Lessons log

gemma-grounded-health

Run r-f0f3b432a413 · 2026-11-17T05:50:15.439325+00:00 · evalkit-summary · docs/results/main/summary.json at 5a9b01c · EvalKit not recorded in the run; pin evalkit-harness[openai,anthropic] @ git+https://github.com/biplob819/evalkit@06591f8de9b178078c971940a83019a6530fd119, from pyproject.toml at the commit (uv.lock: 0.1.0 (git 06591f8de9b1))

Correct and fully grounded (fine-tuned v1): 38.8% [35.8%, 41.9%]

Calibration: uncalibrated: judge not calibrated for this task

Gates

fail (4 failed, 0 warned) · EvalKit default gates: every model passes its thresholds

MetricLabelsTargetObservedSeverityResult
passes_thresholdmodel=gemma-e2b-base-q4== truefalsefailfail
passes_thresholdmodel=gemma-e2b-grounded-q4== truefalsefailfail
passes_thresholdmodel=claude-haiku-5-5== truefalsefailfail
passes_thresholdmodel=gpt-6-luna== truefalsefailfail

Quality and efficiency per model

Quality and efficiency per model, ordered by cost per correct (the comparison metric)
ModelQualityCost per correctCost per casep50p95ErrorsModePrice dateSource
gemma-e2b-base-q4 lowest cost per correctnot designated$0 (from summary (no interval))$0 (from summary (no interval))3,566 ms (from summary (no interval))4,883 ms (from summary (no interval))0 of 1000, below 0.299% at 95%not recordednot recordedfrom summary (no interval)
gemma-e2b-grounded-q4 lowest cost per correctnot designated$0 (from summary (no interval))$0 (from summary (no interval))2,257 ms (from summary (no interval))2,610 ms (from summary (no interval))0 of 1000, below 0.299% at 95%not recordednot recordedfrom summary (no interval)
gpt-6-lunanot designated$0.00023 (from summary (no interval))$0.00014 (from summary (no interval))2,365 ms (from summary (no interval))3,665 ms (from summary (no interval))0 of 1000, below 0.299% at 95%not recordednot recordedfrom summary (no interval)
claude-haiku-5-5not designated$0.00029 (from summary (no interval))$0.0002 (from summary (no interval))1,929 ms (from summary (no interval))2,715 ms (from summary (no interval))0 of 1000, below 0.299% at 95%not recordednot recordedfrom summary (no interval)

lowest cost per correct: tied between gemma-e2b-base-q4, gemma-e2b-grounded-q4

Lessons log

respectful-offers SIMULATED

Run real-v2 · 2026-11-16T20:12:07.430556Z · metrics-1 · results/kkbox/metrics.json at 396fb34 · EvalKit not recorded in the run; pin evalkit-harness>=0.4,<0.5, from pyproject.toml at the commit (uv.lock: 0.4.0)

Extra members kept per 1,000 NTD, bandit vs rule-following baseline (central, simulated): 5.570 [5.064, 6.096]

Calibration: not stated in the metrics-1 file

Gates

pass (0 failed, 2 warned) · gates file configs/gates.yaml at 396fb34

MetricLabelsTargetObservedSeverityResult
violationspolicy=no_offer, respectful=true, rule=R1, variant=conservative== 00failpass
violationspolicy=no_offer, respectful=true, rule=R2, variant=conservative== 00failpass
violationspolicy=no_offer, respectful=true, rule=R3, variant=conservative== 00failpass
violationspolicy=no_offer, respectful=true, rule=R7, variant=conservative== 00failpass
violationspolicy=no_offer, respectful=true, rule=R8, variant=conservative== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R1, variant=conservative== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R2, variant=conservative== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R3, variant=conservative== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R7, variant=conservative== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R8, variant=conservative== 00failpass
violationspolicy=bandit, respectful=true, rule=R1, variant=conservative== 00failpass
violationspolicy=bandit, respectful=true, rule=R2, variant=conservative== 00failpass
violationspolicy=bandit, respectful=true, rule=R3, variant=conservative== 00failpass
violationspolicy=bandit, respectful=true, rule=R7, variant=conservative== 00failpass
violationspolicy=bandit, respectful=true, rule=R8, variant=conservative== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R1, variant=conservative== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R2, variant=conservative== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R3, variant=conservative== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R7, variant=conservative== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R8, variant=conservative== 00failpass
violationspolicy=no_offer, respectful=true, rule=R1, variant=central== 00failpass
violationspolicy=no_offer, respectful=true, rule=R2, variant=central== 00failpass
violationspolicy=no_offer, respectful=true, rule=R3, variant=central== 00failpass
violationspolicy=no_offer, respectful=true, rule=R7, variant=central== 00failpass
violationspolicy=no_offer, respectful=true, rule=R8, variant=central== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R1, variant=central== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R2, variant=central== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R3, variant=central== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R7, variant=central== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R8, variant=central== 00failpass
violationspolicy=bandit, respectful=true, rule=R1, variant=central== 00failpass
violationspolicy=bandit, respectful=true, rule=R2, variant=central== 00failpass
violationspolicy=bandit, respectful=true, rule=R3, variant=central== 00failpass
violationspolicy=bandit, respectful=true, rule=R7, variant=central== 00failpass
violationspolicy=bandit, respectful=true, rule=R8, variant=central== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R1, variant=central== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R2, variant=central== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R3, variant=central== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R7, variant=central== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R8, variant=central== 00failpass
violationspolicy=no_offer, respectful=true, rule=R1, variant=optimistic== 00failpass
violationspolicy=no_offer, respectful=true, rule=R2, variant=optimistic== 00failpass
violationspolicy=no_offer, respectful=true, rule=R3, variant=optimistic== 00failpass
violationspolicy=no_offer, respectful=true, rule=R7, variant=optimistic== 00failpass
violationspolicy=no_offer, respectful=true, rule=R8, variant=optimistic== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R1, variant=optimistic== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R2, variant=optimistic== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R3, variant=optimistic== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R7, variant=optimistic== 00failpass
violationspolicy=rule_following_baseline, respectful=true, rule=R8, variant=optimistic== 00failpass
violationspolicy=bandit, respectful=true, rule=R1, variant=optimistic== 00failpass
violationspolicy=bandit, respectful=true, rule=R2, variant=optimistic== 00failpass
violationspolicy=bandit, respectful=true, rule=R3, variant=optimistic== 00failpass
violationspolicy=bandit, respectful=true, rule=R7, variant=optimistic== 00failpass
violationspolicy=bandit, respectful=true, rule=R8, variant=optimistic== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R1, variant=optimistic== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R2, variant=optimistic== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R3, variant=optimistic== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R7, variant=optimistic== 00failpass
violationspolicy=segment_oracle, respectful=true, rule=R8, variant=optimistic== 00failpass
diff.incremental_retained_per_1000_ntda=bandit, b=rule_following_baseline, variant=conservative>= 0 (lower bound)1.276failpass
separated.incremental_retained_per_1000_ntda=bandit, b=rule_following_baseline, variant=conservative== truetruefailpass
diff.incremental_retained_per_1000_ntda=bandit, b=rule_following_baseline, variant=central>= 0 (lower bound)5.064failpass
separated.incremental_retained_per_1000_ntda=bandit, b=rule_following_baseline, variant=central== truetruefailpass
diff.incremental_retained_per_1000_ntda=bandit, b=rule_following_baseline, variant=optimistic>= 0 (lower bound)8.728failpass
separated.incremental_retained_per_1000_ntda=bandit, b=rule_following_baseline, variant=optimistic== truetruefailpass
holdout_checkvariant=central<= 0 (lower bound)-1.845failpass
holdout_checkvariant=conservative<= 0 (lower bound)-1.845failpass
holdout_checkvariant=optimistic<= 0 (lower bound)-1.845failpass
holdout_checkvariant=central>= 0 (upper bound)3.765failpass
holdout_checkvariant=conservative>= 0 (upper bound)3.765failpass
holdout_checkvariant=optimistic>= 0 (upper bound)3.765failpass
group.offer_rateage_band=max - min, policy=bandit, variant=central<= 0.135.9%warnwarn
group.offer_rateage_band=max - min, policy=rule_following_baseline, variant=central<= 0.140.0%warnwarn
churn.auc(any)>= 0.8 (lower bound)0.839warnpass
churn.ece(any)<= 0.050.02621warnpass
ope.coveredestimator=ips, policy=no_offer, variant=conservative== truetruewarnpass
ope.coveredestimator=snips, policy=no_offer, variant=conservative== truetruewarnpass
ope.coveredestimator=dr, policy=no_offer, variant=conservative== truetruewarnpass
ope.coveredestimator=ips, policy=rule_following_baseline, variant=conservative== truetruewarnpass
ope.coveredestimator=snips, policy=rule_following_baseline, variant=conservative== truetruewarnpass
ope.coveredestimator=dr, policy=rule_following_baseline, variant=conservative== truetruewarnpass
ope.coveredestimator=ips, policy=segment_oracle, variant=conservative== truetruewarnpass
ope.coveredestimator=snips, policy=segment_oracle, variant=conservative== truetruewarnpass
ope.coveredestimator=dr, policy=segment_oracle, variant=conservative== truetruewarnpass
ope.coveredestimator=ips, policy=bandit, variant=conservative== truetruewarnpass
ope.coveredestimator=snips, policy=bandit, variant=conservative== truetruewarnpass
ope.coveredestimator=dr, policy=bandit, variant=conservative== truetruewarnpass
ope.coveredestimator=ips, policy=no_offer, variant=central== truetruewarnpass
ope.coveredestimator=snips, policy=no_offer, variant=central== truetruewarnpass
ope.coveredestimator=dr, policy=no_offer, variant=central== truetruewarnpass
ope.coveredestimator=ips, policy=rule_following_baseline, variant=central== truetruewarnpass
ope.coveredestimator=snips, policy=rule_following_baseline, variant=central== truetruewarnpass
ope.coveredestimator=dr, policy=rule_following_baseline, variant=central== truetruewarnpass
ope.coveredestimator=ips, policy=segment_oracle, variant=central== truetruewarnpass
ope.coveredestimator=snips, policy=segment_oracle, variant=central== truetruewarnpass
ope.coveredestimator=dr, policy=segment_oracle, variant=central== truetruewarnpass
ope.coveredestimator=ips, policy=bandit, variant=central== truetruewarnpass
ope.coveredestimator=snips, policy=bandit, variant=central== truetruewarnpass
ope.coveredestimator=dr, policy=bandit, variant=central== truetruewarnpass
ope.coveredestimator=ips, policy=no_offer, variant=optimistic== truetruewarnpass
ope.coveredestimator=snips, policy=no_offer, variant=optimistic== truetruewarnpass
ope.coveredestimator=dr, policy=no_offer, variant=optimistic== truetruewarnpass
ope.coveredestimator=ips, policy=rule_following_baseline, variant=optimistic== truetruewarnpass
ope.coveredestimator=snips, policy=rule_following_baseline, variant=optimistic== truetruewarnpass
ope.coveredestimator=dr, policy=rule_following_baseline, variant=optimistic== truetruewarnpass
ope.coveredestimator=ips, policy=segment_oracle, variant=optimistic== truetruewarnpass
ope.coveredestimator=snips, policy=segment_oracle, variant=optimistic== truetruewarnpass
ope.coveredestimator=dr, policy=segment_oracle, variant=optimistic== truetruewarnpass
ope.coveredestimator=ips, policy=bandit, variant=optimistic== truetruewarnpass
ope.coveredestimator=snips, policy=bandit, variant=optimistic== truetruewarnpass
ope.coveredestimator=dr, policy=bandit, variant=optimistic== truetruewarnpass

Quality and efficiency per model

no per-model efficiency in this summary (no model label)

Lessons log

evalkit

Run r-e8dac378d2d3 · 2026-11-14T18:22:09.290564+00:00 · evalkit-summary · docs/results/summary.json at 30ad6cd · EvalKit not recorded in the run; pin evalkit-harness 0.4.0 (the repository at the commit), from pyproject.toml at the commit

Correct and fully grounded (PubMedQA): 57.5% [50.5%, 64.0%]

Calibration: uncalibrated: judge not calibrated for this task

Gates

fail (4 failed, 0 warned) · EvalKit default gates: every model passes its thresholds

MetricLabelsTargetObservedSeverityResult
passes_thresholdmodel=claude-sonnet-5-5== truefalsefailfail
passes_thresholdmodel=claude-haiku-5-5== truefalsefailfail
passes_thresholdmodel=gpt-6.1-sol== truefalsefailfail
passes_thresholdmodel=gpt-6-luna== truefalsefailfail

Quality and efficiency per model

Quality and efficiency per model, ordered by cost per correct (the comparison metric)
ModelQualityCost per correctCost per casep50p95ErrorsModePrice dateSource
gpt-6-luna lowest cost per correct57.0% [50.0%, 63.5%]$0.00024 (from summary (no interval))$0.00013 (from summary (no interval))2,514 ms (from summary (no interval))3,740 ms (from summary (no interval))0 of 200, below 1.49% at 95%not recordednot recordedfrom summary (no interval)
claude-haiku-5-566.5% [60.0%, 73.0%]$0.00031 (from summary (no interval))$0.0002 (from summary (no interval))1,731 ms (from summary (no interval))3,070 ms (from summary (no interval))0 of 200, below 1.49% at 95%not recordednot recordedfrom summary (no interval)
gpt-6.1-sol65.0% [58.5%, 71.5%]$0.0036 (from summary (no interval))$0.0023 (from summary (no interval))3,219 ms (from summary (no interval))6,559 ms (from summary (no interval))0 of 200, below 1.49% at 95%not recordednot recordedfrom summary (no interval)
claude-sonnet-5-577.0% [71.0%, 82.5%]$0.0047 (from summary (no interval))$0.0036 (from summary (no interval))2,295 ms (from summary (no interval))2,891 ms (from summary (no interval))0 of 200, below 1.49% at 95%not recordednot recordedfrom summary (no interval)

lowest cost per correct: gpt-6-luna (no interval to compare)

Lessons log