What AI search cites for UK health questions
citewatch asked 2 AI search models 100 pre-registered questions in 2 runs, and recorded every source they cited and how recently each page was updated.
What this study found
- nhs.uk appears in 83% (83/100; 95% CI 75%–90%) of GPT-6.1 Sol's answers but 10% (10/100; 95% CI 5.0%–16%) of Claude Sonnet 5.5's (run 1; separated after Holm correction, p = 0.002).
- The two models' cited sources overlap little: Jaccard similarity 10% (n=100; 95% CI 7.4%–13%) per question, averaged over questions (run 1).
- Share of citation weight (each answer counted equally, split over the URLs it cites) held by each model's own top 10 publishers (run 1): Claude Sonnet 5.5 24% (n=100; 95% CI 23%–31%); GPT-6.1 Sol 78% (n=100; 95% CI 75%–85%). The bootstrap re-picks the top 10 in each resample, so these intervals lean upward.
- Time-sensitive answers whose newest dated source is over 365 days old (no fresh source at all), both runs pooled: Claude Sonnet 5.5 12% (14/114; 95% CI 5.8%–20%); GPT-6.1 Sol 43% (49/114; 95% CI 31%–55%).
Scope
This study measures sourcing: which websites 2 AI search models cite when asked UK health questions, and how recently those pages were updated. It does not judge whether any answer or page is medically accurate. 100 pre-registered questions across 5 intents, asked twice (runs 1, 2) of Claude Sonnet 5.5, GPT-6.1 Sol with web search, from London, GB: findings are directional for this setup, not a verdict on AI search in general. Every percentage carries its count and 95% interval.
How far to trust these numbers
| Check | Result | What it measures |
|---|---|---|
| Date coverage | 89% (745/841; 95% CI 86%–91%) | cited URLs with a resolved date or a logged reason; the study's 90% target (below target) |
| Date check | 67% (18/27; 95% CI 48%–81%) | parser within 7 days of the checked date; agent-checked, owner-approved (OWNER D-63) |
| Support: Claude Sonnet 5.5 cited, judged by claude-opus-5-5 | 67% (16/24; 95% CI 47%–82%) | cited page states the claim (AI judge) |
| Support: Claude Sonnet 5.5 cited, judged by gpt-6.1-sol | 67% (16/24; 95% CI 47%–82%) | cited page states the claim (AI judge) |
| Support: GPT-6.1 Sol cited, judged by claude-opus-5-5 | 57% (13/23; 95% CI 37%–74%) | cited page states the claim (AI judge) |
| Support: GPT-6.1 Sol cited, judged by gpt-6.1-sol | 43% (10/23; 95% CI 26%–63%) | cited page states the claim (AI judge) |
| Support: Claude Sonnet 5.5 cited, owner's check | 100% (24/24; 95% CI 86%–100%) | owner-labelled (OWNER D-62) |
| Support: GPT-6.1 Sol cited, owner's check | 100% (25/25; 95% CI 87%–100%) | owner-labelled (OWNER D-62) |
Errors: Claude Sonnet 5.5: 200 answers, 0 refusals, 0 transport errors, 48 over the 150-word limit asked for (kept and scored); GPT-6.1 Sol: 200 answers, 0 refusals, 0 transport errors, 0 over the 150-word limit asked for (kept and scored). Date check: 17 dated pages showed the checker no date and 7 did not load; none is scored.
Where the two models differ
Paired differences in presence share for each run's most-cited publishers, with a two-sided bootstrap p-value and Holm correction (alpha 0.05, pre-registered).
| Run | Publisher | Models compared | Difference in presence share | p | p (Holm) | Separated |
|---|---|---|---|---|---|---|
| run 1 | nhs.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | -73 pts (95% CI -82 to -64; n=100) | 0.0002 | 0.0022 | yes |
| run 1 | gov.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | -21 pts (95% CI -29 to -13; n=100) | 0.0002 | 0.0022 | yes |
| run 1 | nhsinform.scot | Claude Sonnet 5.5 − GPT-6.1 Sol | +11 pts (95% CI +4 to +18; n=100) | 0.0040 | 0.0280 | yes |
| run 1 | icb.nhs.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | +10 pts (95% CI +5 to +16; n=100) | 0.0002 | 0.0022 | yes |
| run 1 | wales.nhs.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | +7 pts (95% CI +2 to +12; n=100) | 0.0016 | 0.0128 | yes |
| run 1 | nhsbsa.nhs.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | -6 pts (95% CI -11 to -2; n=100) | 0.0058 | 0.0348 | yes |
| run 1 | nice.org.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | -4 pts (95% CI -9 to +0; n=100) | 0.1242 | 0.6209 | no |
| run 1 | nih.gov | Claude Sonnet 5.5 − GPT-6.1 Sol | -2 pts (95% CI -7 to +3; n=100) | 0.5349 | 1.0000 | no |
| run 1 | uk.com | Claude Sonnet 5.5 − GPT-6.1 Sol | -2 pts (95% CI -6 to +2; n=100) | 0.4382 | 1.0000 | no |
| run 1 | england.nhs.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | -2 pts (95% CI -9 to +5; n=100) | 0.6649 | 1.0000 | no |
| run 1 | bhf.org.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | +0 pts (95% CI -6 to +6; n=100) | 1.0000 | 1.0000 | no |
| run 2 | nhs.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | -71 pts (95% CI -80 to -62; n=100) | 0.0002 | 0.0020 | yes |
| run 2 | gov.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | -21 pts (95% CI -30 to -13; n=100) | 0.0002 | 0.0020 | yes |
| run 2 | icb.nhs.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | +9 pts (95% CI +3 to +15; n=100) | 0.0034 | 0.0238 | yes |
| run 2 | nhsinform.scot | Claude Sonnet 5.5 − GPT-6.1 Sol | +8 pts (95% CI +1 to +16; n=100) | 0.0494 | 0.2964 | no |
| run 2 | wales.nhs.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | +8 pts (95% CI +3 to +14; n=100) | 0.0014 | 0.0112 | yes |
| run 2 | nice.org.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | -5 pts (95% CI -10 to +0; n=100) | 0.0650 | 0.3250 | no |
| run 2 | nih.gov | Claude Sonnet 5.5 − GPT-6.1 Sol | -5 pts (95% CI -10 to +0; n=100) | 0.0660 | 0.3250 | no |
| run 2 | nhsbsa.nhs.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | -3 pts (95% CI -8 to +1; n=100) | 0.2354 | 0.7061 | no |
| run 2 | bhf.org.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | +1 pts (95% CI -5 to +7; n=100) | 0.8689 | 1.0000 | no |
| run 2 | england.nhs.uk | Claude Sonnet 5.5 − GPT-6.1 Sol | -1 pts (95% CI -8 to +6; n=100) | 0.8851 | 1.0000 | no |
How concentrated the citations are
Table: concentration
| Model | Run | Top-10 share | HHI |
|---|---|---|---|
| Claude Sonnet 5.5 | 1 | 24% (n=100; 95% CI 23%–31%) | 0.0103 |
| GPT-6.1 Sol | 1 | 78% (n=100; 95% CI 75%–85%) | 0.2543 |
| Claude Sonnet 5.5 | 2 | 27% (n=100; 95% CI 24%–34%) | 0.0117 |
| GPT-6.1 Sol | 2 | 79% (n=100; 95% CI 75%–86%) | 0.2648 |
Citations per answer
| Model | Run | Intent | Unique URLs per answer | Raw citations per answer | Answers citing nothing |
|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 1 | All intents | 3.82 (95% CI 3.62–4.02; n=100) | 6.41 | 0 |
| Claude Sonnet 5.5 | 1 | Fitness | 4.15 (95% CI 3.73–4.58; n=20) | 6.30 | 0 |
| Claude Sonnet 5.5 | 1 | NHS services | 3.45 (95% CI 3.00–3.95; n=20) | 6.40 | 0 |
| Claude Sonnet 5.5 | 1 | Nutrition | 3.80 (95% CI 3.35–4.24; n=20) | 6.55 | 0 |
| Claude Sonnet 5.5 | 1 | Symptoms | 3.90 (95% CI 3.42–4.43; n=20) | 6.45 | 0 |
| Claude Sonnet 5.5 | 1 | Treatment | 3.80 (95% CI 3.40–4.18; n=20) | 6.35 | 0 |
| GPT-6.1 Sol | 1 | All intents | 3.08 (95% CI 2.85–3.32; n=100) | 4.70 | 0 |
| GPT-6.1 Sol | 1 | Fitness | 3.45 (95% CI 3.00–3.88; n=20) | 5.05 | 0 |
| GPT-6.1 Sol | 1 | NHS services | 2.95 (95% CI 2.40–3.59; n=20) | 4.35 | 0 |
| GPT-6.1 Sol | 1 | Nutrition | 2.95 (95% CI 2.53–3.39; n=20) | 4.75 | 0 |
| GPT-6.1 Sol | 1 | Symptoms | 2.90 (95% CI 2.38–3.47; n=20) | 4.85 | 0 |
| GPT-6.1 Sol | 1 | Treatment | 3.15 (95% CI 2.56–3.76; n=20) | 4.50 | 0 |
| Claude Sonnet 5.5 | 2 | All intents | 3.74 (95% CI 3.56–3.94; n=100) | 6.30 | 0 |
| Claude Sonnet 5.5 | 2 | Fitness | 3.70 (95% CI 3.33–4.06; n=20) | 6.55 | 0 |
| Claude Sonnet 5.5 | 2 | NHS services | 3.60 (95% CI 3.15–4.05; n=20) | 6.45 | 0 |
| Claude Sonnet 5.5 | 2 | Nutrition | 3.60 (95% CI 3.21–4.00; n=20) | 5.75 | 0 |
| Claude Sonnet 5.5 | 2 | Symptoms | 3.90 (95% CI 3.35–4.47; n=20) | 6.20 | 0 |
| Claude Sonnet 5.5 | 2 | Treatment | 3.90 (95% CI 3.53–4.26; n=20) | 6.55 | 0 |
| GPT-6.1 Sol | 2 | All intents | 2.98 (95% CI 2.78–3.19; n=100) | 4.71 | 0 |
| GPT-6.1 Sol | 2 | Fitness | 3.30 (95% CI 2.87–3.75; n=20) | 5.05 | 0 |
| GPT-6.1 Sol | 2 | NHS services | 2.80 (95% CI 2.35–3.25; n=20) | 4.35 | 0 |
| GPT-6.1 Sol | 2 | Nutrition | 2.95 (95% CI 2.43–3.52; n=20) | 4.65 | 0 |
| GPT-6.1 Sol | 2 | Symptoms | 2.90 (95% CI 2.45–3.39; n=20) | 4.80 | 0 |
| GPT-6.1 Sol | 2 | Treatment | 2.95 (95% CI 2.55–3.38; n=20) | 4.70 | 0 |
Stale answers
Table: stale answers (primary and secondary)
| Model | Intent | Definition | Rate | Undated |
|---|---|---|---|---|
| Claude Sonnet 5.5 | All intents | primary | 12% (14/114; 95% CI 5.8%–20%) | 0 |
| Claude Sonnet 5.5 | All intents | secondary | 67% (76/114; 95% CI 56%–77%) | 0 |
| Claude Sonnet 5.5 | Fitness | primary | 15% (3/20; 95% CI 0%–39%) · directional | 0 |
| Claude Sonnet 5.5 | Fitness | secondary | 60% (12/20; 95% CI 31%–88%) · directional | 0 |
| Claude Sonnet 5.5 | NHS services | primary | 3.8% (1/26; 95% CI 0%–12%) · directional | 0 |
| Claude Sonnet 5.5 | NHS services | secondary | 65% (17/26; 95% CI 41%–88%) · directional | 0 |
| Claude Sonnet 5.5 | Nutrition | primary | 0% (0/20; 95% CI 0%–0%) · directional | 0 |
| Claude Sonnet 5.5 | Nutrition | secondary | 70% (14/20; 95% CI 42%–94%) · directional | 0 |
| Claude Sonnet 5.5 | Symptoms | primary | 19% (5/26; 95% CI 5.9%–33%) · directional | 0 |
| Claude Sonnet 5.5 | Symptoms | secondary | 62% (16/26; 95% CI 38%–83%) · directional | 0 |
| Claude Sonnet 5.5 | Treatment | primary | 23% (5/22; 95% CI 0%–50%) · directional | 0 |
| Claude Sonnet 5.5 | Treatment | secondary | 77% (17/22; 95% CI 55%–95%) · directional | 0 |
| GPT-6.1 Sol | All intents | primary | 43% (49/114; 95% CI 31%–55%) | 0 |
| GPT-6.1 Sol | All intents | secondary | 78% (89/114; 95% CI 68%–88%) | 0 |
| GPT-6.1 Sol | Fitness | primary | 15% (3/20; 95% CI 0%–39%) · directional | 0 |
| GPT-6.1 Sol | Fitness | secondary | 65% (13/20; 95% CI 39%–90%) · directional | 0 |
| GPT-6.1 Sol | NHS services | primary | 73% (19/26; 95% CI 47%–95%) · directional | 0 |
| GPT-6.1 Sol | NHS services | secondary | 88% (23/26; 95% CI 70%–100%) · directional | 0 |
| GPT-6.1 Sol | Nutrition | primary | 30% (6/20; 95% CI 0%–62%) · directional | 0 |
| GPT-6.1 Sol | Nutrition | secondary | 70% (14/20; 95% CI 38%–100%) · directional | 0 |
| GPT-6.1 Sol | Symptoms | primary | 46% (12/26; 95% CI 20%–72%) · directional | 0 |
| GPT-6.1 Sol | Symptoms | secondary | 85% (22/26; 95% CI 62%–100%) · directional | 0 |
| GPT-6.1 Sol | Treatment | primary | 41% (9/22; 95% CI 12%–71%) · directional | 0 |
| GPT-6.1 Sol | Treatment | secondary | 77% (17/22; 95% CI 50%–100%) · directional | 0 |
How old the cited pages are
Lag is days from the page's effective date (last reviewed, else modified, else published) to the answer that cited it. Top 30 publishers by dated pages (both models); per-model rows are in summary.json.
| Publisher | Dated citations | Median lag (days) | IQR (days) | Modified after the answer |
|---|---|---|---|---|
| nhs.uk | 144 | 823 | 249–1229 | 0 |
| gov.uk | 28 | 129 | 129–754 | 0 |
| bhf.org.uk | 19 | 69 | 55–368 | 0 |
| england.nhs.uk | 19 | 1217 | 342–1329 | 0 |
| nhsinform.scot | 16 | 97 | 47–112 | 0 |
| icb.nhs.uk | 11 | 397 | 178–627 | 0 |
| wales.nhs.uk | 8 | 458 | 146–671 | 0 |
| healthwatch.co.uk | 6 | 435 | 428–451 | 0 |
| patient.info | 6 | 699 | 383–1120 | 0 |
| uhsussex.nhs.uk | 5 | 909 | 424–935 | 0 |
| clevelandclinic.org | 4 | 124 | 91–263 | 0 |
| curatehealth.co.uk | 4 | 122 | 116–122 | 0 |
| kingstonandrichmond.nhs.uk | 4 | 455 | 212–503 | 0 |
| mayoclinic.org | 4 | 618 | 599–1126 | 0 |
| nhsbsa.nhs.uk | 4 | 1473 | 594–2197 | 0 |
| nice.org.uk | 4 | 949 | 264–1159 | 0 |
| pharmaceutical-journal.com | 4 | 357 | 357–958 | 0 |
| scot.nhs.uk | 4 | 547 | 368–3527 | 0 |
| tommys.org | 4 | 832 | 87–832 | 0 |
| accessdoctor.co.uk | 3 | 24 | 24–29 | 0 |
| cheshireandmerseyside.nhs.uk | 3 | 743 | 565–935 | 0 |
| gloshospitals.nhs.uk | 3 | 603 | 82–603 | 0 |
| gov.wales | 3 | 341 | 263–374 | 0 |
| healthystart.nhs.uk | 3 | 53 | 53–192 | 0 |
| londonambulance.nhs.uk | 3 | 96 | 72–97 | 0 |
| myhealthlondon.nhs.uk | 3 | 25 | 25–25 | 0 |
| nhs.wales | 3 | 347 | 32–773 | 0 |
| nhsggc.scot | 3 | 61 | 52–128 | 0 |
| nutrition.org.uk | 3 | 370 | 20–719 | 0 |
| ombudsman.org.uk | 3 | 49 | 49–49 | 0 |
NHS pages past their review date
| Model | Cited NHS pages past their review-due date |
|---|---|
| Both models | 13% (88/661; 95% CI 10%–17%) |
| Claude Sonnet 5.5 | 1.1% (3/282; 95% CI 0%–2.9%) |
| GPT-6.1 Sol | 22% (85/379; 95% CI 16%–29%) |
Do the two models cite the same sources?
Table: every overlap measure
| Kind | Intent | Run | Model | Value |
|---|---|---|---|---|
| cross_model | All intents | 1 | — | 10% (n=100; 95% CI 7.4%–13%) |
| pooled_top20 | All intents | 1 | — | 11% (n=83; 95% CI 8.2%–22%) |
| cross_model | All intents | 2 | — | 10% (n=100; 95% CI 6.6%–13%) |
| pooled_top20 | All intents | 2 | — | 18% (n=94; 95% CI 10%–24%) |
| self | All intents | both | Claude Sonnet 5.5 | 54% (n=100; 95% CI 48%–60%) |
| self | All intents | both | GPT-6.1 Sol | 67% (n=100; 95% CI 62%–73%) |
| normalised | All intents | both | — | 16% (n=200; 95% CI 12%–21%) |
| cross_model | Fitness | 1 | — | 8.7% (n=20; 95% CI 2.4%–17%) |
| pooled_top20 | Fitness | 1 | — | 9.2% (n=87; 95% CI 2.7%–14%) |
| cross_model | Fitness | 2 | — | 12% (n=20; 95% CI 3.3%–22%) |
| pooled_top20 | Fitness | 2 | — | 9.5% (n=74; 95% CI 4.7%–16%) |
| self | Fitness | both | Claude Sonnet 5.5 | 49% (n=20; 95% CI 36%–64%) |
| self | Fitness | both | GPT-6.1 Sol | 59% (n=20; 95% CI 46%–73%) |
| normalised | Fitness | both | — | 19% (n=40; 95% CI 7.2%–32%) |
| cross_model | NHS services | 1 | — | 11% (n=20; 95% CI 4.5%–19%) |
| pooled_top20 | NHS services | 1 | — | 13% (n=55; 95% CI 5.1%–19%) |
| cross_model | NHS services | 2 | — | 10% (n=20; 95% CI 4.2%–17%) |
| pooled_top20 | NHS services | 2 | — | 12% (n=59; 95% CI 6.5%–19%) |
| self | NHS services | both | Claude Sonnet 5.5 | 54% (n=20; 95% CI 41%–68%) |
| self | NHS services | both | GPT-6.1 Sol | 81% (n=20; 95% CI 68%–92%) |
| normalised | NHS services | both | — | 16% (n=40; 95% CI 7.4%–26%) |
| cross_model | Nutrition | 1 | — | 12% (n=20; 95% CI 6.2%–18%) |
| pooled_top20 | Nutrition | 1 | — | 13% (n=70; 95% CI 7.9%–23%) |
| cross_model | Nutrition | 2 | — | 6.1% (n=20; 95% CI 2.1%–11%) |
| pooled_top20 | Nutrition | 2 | — | 10% (n=70; 95% CI 4.5%–16%) |
| self | Nutrition | both | Claude Sonnet 5.5 | 62% (n=20; 95% CI 49%–75%) |
| self | Nutrition | both | GPT-6.1 Sol | 68% (n=20; 95% CI 54%–81%) |
| normalised | Nutrition | both | — | 14% (n=40; 95% CI 6.7%–23%) |
| cross_model | Symptoms | 1 | — | 10% (n=20; 95% CI 3.9%–16%) |
| pooled_top20 | Symptoms | 1 | — | 10% (n=71; 95% CI 5.9%–19%) |
| cross_model | Symptoms | 2 | — | 11% (n=20; 95% CI 2.7%–23%) |
| pooled_top20 | Symptoms | 2 | — | 6.6% (n=76; 95% CI 2.6%–13%) |
| self | Symptoms | both | Claude Sonnet 5.5 | 52% (n=20; 95% CI 39%–65%) |
| self | Symptoms | both | GPT-6.1 Sol | 72% (n=20; 95% CI 60%–84%) |
| normalised | Symptoms | both | — | 17% (n=40; 95% CI 6.5%–31%) |
| cross_model | Treatment | 1 | — | 9.2% (n=20; 95% CI 3.7%–16%) |
| pooled_top20 | Treatment | 1 | — | 12% (n=68; 95% CI 5.4%–18%) |
| cross_model | Treatment | 2 | — | 10% (n=20; 95% CI 3.4%–17%) |
| pooled_top20 | Treatment | 2 | — | 15% (n=68; 95% CI 6.7%–22%) |
| self | Treatment | both | Claude Sonnet 5.5 | 51% (n=20; 95% CI 40%–62%) |
| self | Treatment | both | GPT-6.1 Sol | 57% (n=20; 95% CI 45%–70%) |
| normalised | Treatment | both | — | 17% (n=40; 95% CI 7.6%–28%) |
Run-to-run stability
| Model | Publisher | Metric | Value |
|---|---|---|---|
| Claude Sonnet 5.5 | all | self_jaccard | 54% (n=100; 95% CI 48%–60%) |
| Claude Sonnet 5.5 | nhsinform.scot | share_change | -3 pts (95% CI -8 to +2; n=100) |
| Claude Sonnet 5.5 | england.nhs.uk | share_change | -3 pts (95% CI -7 to +0; n=100) |
| Claude Sonnet 5.5 | icb.nhs.uk | share_change | +0 pts (95% CI -5 to +5; n=100) |
| Claude Sonnet 5.5 | nhs.uk | share_change | +3 pts (95% CI -2 to +8; n=100) |
| Claude Sonnet 5.5 | bhf.org.uk | share_change | +0 pts (95% CI -4 to +4; n=100) |
| Claude Sonnet 5.5 | wales.nhs.uk | share_change | +1 pts (95% CI -2 to +4; n=100) |
| Claude Sonnet 5.5 | scot.nhs.uk | share_change | +0 pts (95% CI -3 to +3; n=100) |
| Claude Sonnet 5.5 | uhsussex.nhs.uk | share_change | -2 pts (95% CI -5 to +0; n=100) |
| Claude Sonnet 5.5 | clevelandclinic.org | share_change | -1 pts (95% CI -3 to +0; n=100) |
| Claude Sonnet 5.5 | curatehealth.co.uk | share_change | -2 pts (95% CI -5 to +0; n=100) |
| Claude Sonnet 5.5 | healthwatch.co.uk | share_change | +0 pts (95% CI -4 to +4; n=100) |
| Claude Sonnet 5.5 | nhs.wales | share_change | -1 pts (95% CI -4 to +2; n=100) |
| Claude Sonnet 5.5 | patient.info | share_change | +1 pts (95% CI -2 to +5; n=100) |
| Claude Sonnet 5.5 | pharmaceutical-journal.com | share_change | -3 pts (95% CI -7 to +0; n=100) |
| GPT-6.1 Sol | all | self_jaccard | 67% (n=100; 95% CI 62%–73%) |
| GPT-6.1 Sol | nhs.uk | share_change | +1 pts (95% CI -3 to +5; n=100) |
| GPT-6.1 Sol | gov.uk | share_change | +2 pts (95% CI -5 to +9; n=100) |
| GPT-6.1 Sol | england.nhs.uk | share_change | -4 pts (95% CI -10 to +2; n=100) |
| GPT-6.1 Sol | bhf.org.uk | share_change | -1 pts (95% CI -4 to +2; n=100) |
| GPT-6.1 Sol | nhsbsa.nhs.uk | share_change | +0 pts (95% CI -3 to +3; n=100) |
| GPT-6.1 Sol | nhsinform.scot | share_change | +0 pts (95% CI -5 to +5; n=100) |
| GPT-6.1 Sol | nice.org.uk | share_change | +2 pts (95% CI -3 to +7; n=100) |
| GPT-6.1 Sol | nih.gov | share_change | +2 pts (95% CI +0 to +5; n=100) |
| GPT-6.1 Sol | uk.com | share_change | -2 pts (95% CI -5 to +0; n=100) |
| GPT-6.1 Sol | myhealthlondon.nhs.uk | share_change | -2 pts (95% CI -5 to +0; n=100) |
Date coverage
89% (745/841; 95% CI 86%–91%)
Cited URLs with a resolved date or a logged reason for having none (the study's 90% target).
Date check
Held-out pages never used to tune the parser; agent-checked, owner-approved (OWNER D-63).
| Date source | Within 7 days | Exact |
|---|---|---|
| overall | 67% (18/27; 95% CI 48%–81%) | 67% (18/27; 95% CI 48%–81%) |
| http_header | 0% (0/2; 95% CI 0%–66%) · directional | 0% (0/2; 95% CI 0%–66%) · directional |
| meta_tag | 50% (1/2; 95% CI 9.5%–91%) · directional | 50% (1/2; 95% CI 9.5%–91%) · directional |
| page_age | 50% (1/2; 95% CI 9.5%–91%) · directional | 50% (1/2; 95% CI 9.5%–91%) · directional |
| schema_org | 67% (4/6; 95% CI 30%–90%) · directional | 67% (4/6; 95% CI 30%–90%) · directional |
| sitemap | 0% (0/3; 95% CI 0%–56%) · directional | 0% (0/3; 95% CI 0%–56%) · directional |
| visible_text | 100% (12/12; 95% CI 76%–100%) | 100% (12/12; 95% CI 76%–100%) |
| undated | 50% (2/4; 95% CI 15%–85%) · directional | 50% (2/4; 95% CI 15%–85%) · directional |
Support check
A sample of citations: does the cited page state the claim it was cited for? Two AI judges from different families and the owner's own check. Kappa is not reported while the owner's labels are all one value.
| Cited model | Judge | Supported | Agreement with owner | Kappa |
|---|---|---|---|---|
| Claude Sonnet 5.5 | claude-opus-5-5 | 67% (16/24; 95% CI 47%–82%) | 67% (16/24) | not reported |
| Claude Sonnet 5.5 | gpt-6.1-sol | 67% (16/24; 95% CI 47%–82%) | 67% (16/24) | not reported |
| Claude Sonnet 5.5 | owner | 100% (24/24; 95% CI 86%–100%) | — | not reported |
| GPT-6.1 Sol | claude-opus-5-5 | 57% (13/23; 95% CI 37%–74%) | 57% (13/23) | not reported |
| GPT-6.1 Sol | gpt-6.1-sol | 43% (10/23; 95% CI 26%–63%) | 43% (10/23) | not reported |
| GPT-6.1 Sol | owner | 100% (25/25; 95% CI 87%–100%) | — | not reported |
How each check was made
| Check | Method |
|---|---|
| date_accuracy | agent-checked, owner-approved (OWNER D-63) |
| support_hand | owner-labelled (OWNER D-62) |
Errors and exclusions
Table: every error and exclusion count
| Group | Count | Value |
|---|---|---|
| claude-sonnet-5-5 | answers | 200 |
| claude-sonnet-5-5 | over_length | 48 |
| claude-sonnet-5-5 | refusal | 0 |
| claude-sonnet-5-5 | search_error | 0 |
| claude-sonnet-5-5 | transport | 0 |
| claude-sonnet-5-5 | zero_citation | 0 |
| gpt-6.1-sol | answers | 200 |
| gpt-6.1-sol | over_length | 0 |
| gpt-6.1-sol | refusal | 0 |
| gpt-6.1-sol | search_error | 0 |
| gpt-6.1-sol | transport | 0 |
| gpt-6.1-sol | zero_citation | 0 |
| hand_dates | checker_could_not_load | 7 |
| hand_dates | labels | 55 |
| hand_dates | stale_labels | 0 |
| hand_dates | unjudgeable | 17 |
| nhs_no_review_date | blocked | 4 |
| nhs_no_review_date | fetch_error | 4 |
| nhs_no_review_date | not_html | 43 |
| nhs_no_review_date | ok | 151 |
| nhs_no_review_date | robots_disallowed | 33 |
| overlap_excluded | cross_model:all:run1 | 0 |
| overlap_excluded | cross_model:all:run2 | 0 |
| overlap_excluded | cross_model:fitness:run1 | 0 |
| overlap_excluded | cross_model:fitness:run2 | 0 |
| overlap_excluded | cross_model:nhs_services:run1 | 0 |
| overlap_excluded | cross_model:nhs_services:run2 | 0 |
| overlap_excluded | cross_model:nutrition:run1 | 0 |
| overlap_excluded | cross_model:nutrition:run2 | 0 |
| overlap_excluded | cross_model:symptoms:run1 | 0 |
| overlap_excluded | cross_model:symptoms:run2 | 0 |
| overlap_excluded | cross_model:treatment:run1 | 0 |
| overlap_excluded | cross_model:treatment:run2 | 0 |
| overlap_excluded | self:claude-sonnet-5-5:all | 0 |
| overlap_excluded | self:claude-sonnet-5-5:fitness | 0 |
| overlap_excluded | self:claude-sonnet-5-5:nhs_services | 0 |
| overlap_excluded | self:claude-sonnet-5-5:nutrition | 0 |
| overlap_excluded | self:claude-sonnet-5-5:symptoms | 0 |
| overlap_excluded | self:claude-sonnet-5-5:treatment | 0 |
| overlap_excluded | self:gpt-6.1-sol:all | 0 |
| overlap_excluded | self:gpt-6.1-sol:fitness | 0 |
| overlap_excluded | self:gpt-6.1-sol:nhs_services | 0 |
| overlap_excluded | self:gpt-6.1-sol:nutrition | 0 |
| overlap_excluded | self:gpt-6.1-sol:symptoms | 0 |
| overlap_excluded | self:gpt-6.1-sol:treatment | 0 |
| undated_pages | blocked | 9 |
| undated_pages | fetch_error | 5 |
| undated_pages | not_html | 37 |
| undated_pages | ok | 96 |
| undated_pages | robots_disallowed | 74 |
Provenance
| Item | Value |
|---|---|
| Runs | 1, 2 |
| Generated | 2026-11-16T20:23:27+00:00 |
| Spend (USD) | $15.03 |
| Config hash | b1ef32199d482e7d |
| Questions sha256 | aaa3a348ab5844e2ad1fac47e0b68a4b53fb08e1354a0f303cc3d60186bba071 |
| Pre-registration sha256 | 9e227483cc387386c0a2b7ca36161c24f77cd59eaee2eb173978c4a183f60cbb |
| Results format | 1.0 |