citewatch study · UK health · runs 1, 2

Who AI search cites for UK health questions, and how fresh it is

Ask GPT-6.1 Sol a UK health question with web search on (run 1) and nhs.uk is among its sources in 83% (83/100; 95% CI 75%–90%) of answers. Ask Claude Sonnet 5.5 the same question and it is 10% (10/100; 95% CI 5.0%–16%). Same questions, same day, same place: the two models lean on different parts of the web.

This study measures sourcing: which websites 2 AI search models cite when asked UK health questions, and how recently those pages were updated. It does not judge whether any answer or page is medically accurate. 100 pre-registered questions across 5 intents, asked twice (runs 1, 2) of Claude Sonnet 5.5, GPT-6.1 Sol with web search: findings are directional for this setup, not a verdict on AI search in general. Every percentage carries its count and 95% interval.

The full numbers are on the dashboard; what this could become as a data product is in the one-page signal spec. Code, data and every decision are in the open-source repository.

What was measured

100 questions across 5 intents (fitness, NHS services, nutrition, symptoms, treatment), written and frozen before any answer was seen, were each asked of Claude Sonnet 5.5 and GPT-6.1 Sol with web search, in 2 runs: 400 answers in all. citewatch recorded every source each answer cited, then visited each cited page, politely and only where robots.txt allowed, to find when it was last reviewed, modified or published. Every share and rate below carries its count and a 95% interval.

1. The models lean on different sources

Claude Sonnet 5.5 spreads its citations across the wider NHS family and others; its most cited publishers in run 1 were nhsinform.scot (17% (17/100; 95% CI 10%–25%)), england.nhs.uk (13% (13/100; 95% CI 7.0%–20%)), icb.nhs.uk (10% (10/100; 95% CI 5.0%–16%)).

Claude Sonnet 5.5GPT-6.1 Sol
0%25%50%75%100%nhs.uknhs.uk · Claude Sonnet 5.5: 10% (10/100; 95% CI 5.0%–16%)nhs.uk · Claude Sonnet 5.5: 10% (10/100; 95% CI 5.0%–16%)nhs.uk · GPT-6.1 Sol: 83% (83/100; 95% CI 75%–90%)nhs.uk · GPT-6.1 Sol: 83% (83/100; 95% CI 75%–90%)england.nhs.ukengland.nhs.uk · Claude Sonnet 5.5: 13% (13/100; 95% CI 7.0%–20%)england.nhs.uk · Claude Sonnet 5.5: 13% (13/100; 95% CI 7.0%–20%)england.nhs.uk · GPT-6.1 Sol: 15% (15/100; 95% CI 8.0%–22%)england.nhs.uk · GPT-6.1 Sol: 15% (15/100; 95% CI 8.0%–22%)gov.ukgov.uk · Claude Sonnet 5.5: 2.0% (2/100; 95% CI 0%–5.0%)gov.uk · Claude Sonnet 5.5: 2.0% (2/100; 95% CI 0%–5.0%)gov.uk · GPT-6.1 Sol: 23% (23/100; 95% CI 15%–31%)gov.uk · GPT-6.1 Sol: 23% (23/100; 95% CI 15%–31%)nhsinform.scotnhsinform.scot · Claude Sonnet 5.5: 17% (17/100; 95% CI 10%–25%)nhsinform.scot · Claude Sonnet 5.5: 17% (17/100; 95% CI 10%–25%)nhsinform.scot · GPT-6.1 Sol: 6.0% (6/100; 95% CI 2.0%–11%)nhsinform.scot · GPT-6.1 Sol: 6.0% (6/100; 95% CI 2.0%–11%)bhf.org.ukbhf.org.uk · Claude Sonnet 5.5: 7.0% (7/100; 95% CI 3.0%–12%)bhf.org.uk · Claude Sonnet 5.5: 7.0% (7/100; 95% CI 3.0%–12%)bhf.org.uk · GPT-6.1 Sol: 7.0% (7/100; 95% CI 2.0%–12%)bhf.org.uk · GPT-6.1 Sol: 7.0% (7/100; 95% CI 2.0%–12%)icb.nhs.ukicb.nhs.uk · Claude Sonnet 5.5: 10% (10/100; 95% CI 5.0%–16%)icb.nhs.uk · Claude Sonnet 5.5: 10% (10/100; 95% CI 5.0%–16%)icb.nhs.uk · GPT-6.1 Sol: 0% (0/100; 95% CI 0%–0%)icb.nhs.uk · GPT-6.1 Sol: 0% (0/100; 95% CI 0%–0%)nhsbsa.nhs.uknhsbsa.nhs.uk · Claude Sonnet 5.5: 1.0% (1/100; 95% CI 0%–3.0%)nhsbsa.nhs.uk · Claude Sonnet 5.5: 1.0% (1/100; 95% CI 0%–3.0%)nhsbsa.nhs.uk · GPT-6.1 Sol: 7.0% (7/100; 95% CI 2.0%–12%)nhsbsa.nhs.uk · GPT-6.1 Sol: 7.0% (7/100; 95% CI 2.0%–12%)nice.org.uknice.org.uk · Claude Sonnet 5.5: 2.0% (2/100; 95% CI 0%–5.0%)nice.org.uk · Claude Sonnet 5.5: 2.0% (2/100; 95% CI 0%–5.0%)nice.org.uk · GPT-6.1 Sol: 6.0% (6/100; 95% CI 2.0%–11%)nice.org.uk · GPT-6.1 Sol: 6.0% (6/100; 95% CI 2.0%–11%)wales.nhs.ukwales.nhs.uk · Claude Sonnet 5.5: 7.0% (7/100; 95% CI 2.0%–12%)wales.nhs.uk · Claude Sonnet 5.5: 7.0% (7/100; 95% CI 2.0%–12%)wales.nhs.uk · GPT-6.1 Sol: 0% (0/100; 95% CI 0%–0%)wales.nhs.uk · GPT-6.1 Sol: 0% (0/100; 95% CI 0%–0%)nih.govnih.gov · Claude Sonnet 5.5: 2.0% (2/100; 95% CI 0%–5.0%)nih.gov · Claude Sonnet 5.5: 2.0% (2/100; 95% CI 0%–5.0%)nih.gov · GPT-6.1 Sol: 4.0% (4/100; 95% CI 1.0%–8.0%)nih.gov · GPT-6.1 Sol: 4.0% (4/100; 95% CI 1.0%–8.0%)uk.comuk.com · Claude Sonnet 5.5: 2.0% (2/100; 95% CI 0%–5.0%)uk.com · Claude Sonnet 5.5: 2.0% (2/100; 95% CI 0%–5.0%)uk.com · GPT-6.1 Sol: 4.0% (4/100; 95% CI 1.0%–8.0%)uk.com · GPT-6.1 Sol: 4.0% (4/100; 95% CI 1.0%–8.0%)clevelandclinic.orgclevelandclinic.org · Claude Sonnet 5.5: 4.0% (4/100; 95% CI 1.0%–8.0%)clevelandclinic.org · Claude Sonnet 5.5: 4.0% (4/100; 95% CI 1.0%–8.0%)clevelandclinic.org · GPT-6.1 Sol: 1.0% (1/100; 95% CI 0%–3.0%)clevelandclinic.org · GPT-6.1 Sol: 1.0% (1/100; 95% CI 0%–3.0%)scot.nhs.ukscot.nhs.uk · Claude Sonnet 5.5: 5.0% (5/100; 95% CI 1.0%–10%)scot.nhs.uk · Claude Sonnet 5.5: 5.0% (5/100; 95% CI 1.0%–10%)scot.nhs.uk · GPT-6.1 Sol: 0% (0/100; 95% CI 0%–0%)scot.nhs.uk · GPT-6.1 Sol: 0% (0/100; 95% CI 0%–0%)uhsussex.nhs.ukuhsussex.nhs.uk · Claude Sonnet 5.5: 5.0% (5/100; 95% CI 1.0%–10%)uhsussex.nhs.uk · Claude Sonnet 5.5: 5.0% (5/100; 95% CI 1.0%–10%)uhsussex.nhs.uk · GPT-6.1 Sol: 0% (0/100; 95% CI 0%–0%)uhsussex.nhs.uk · GPT-6.1 Sol: 0% (0/100; 95% CI 0%–0%)blog.gov.ukblog.gov.uk · Claude Sonnet 5.5: 2.0% (2/100; 95% CI 0%–5.0%)blog.gov.uk · Claude Sonnet 5.5: 2.0% (2/100; 95% CI 0%–5.0%)blog.gov.uk · GPT-6.1 Sol: 2.0% (2/100; 95% CI 0%–5.0%)blog.gov.uk · GPT-6.1 Sol: 2.0% (2/100; 95% CI 0%–5.0%)
Run 1: share of answers citing each publisher at least once (presence share), top 15 publishers, 95% intervals.

The gap for nhs.uk holds in every run and survives a Holm correction across each run's most cited publishers. 10 of 21 paired comparisons separate the models.

2. They rarely cite the same sources

Comparing the publishers each model cites for the same question, the average overlap (Jaccard similarity) is 10% (n=100; 95% CI 7.4%–13%) in run 1. It ranges from 8.7% (n=20; 95% CI 2.4%–17%) for fitness to 12% (n=20; 95% CI 6.2%–18%) for nutrition questions.

Run 1Run 2
0%25%50%75%100%All intentsAll intents · Run 1: 10% (n=100; 95% CI 7.4%–13%)All intents · Run 1: 10% (n=100; 95% CI 7.4%–13%)All intents · Run 2: 10% (n=100; 95% CI 6.6%–13%)All intents · Run 2: 10% (n=100; 95% CI 6.6%–13%)FitnessFitness · Run 1: 8.7% (n=20; 95% CI 2.4%–17%)Fitness · Run 1: 8.7% (n=20; 95% CI 2.4%–17%)Fitness · Run 2: 12% (n=20; 95% CI 3.3%–22%)Fitness · Run 2: 12% (n=20; 95% CI 3.3%–22%)NHS servicesNHS services · Run 1: 11% (n=20; 95% CI 4.5%–19%)NHS services · Run 1: 11% (n=20; 95% CI 4.5%–19%)NHS services · Run 2: 10% (n=20; 95% CI 4.2%–17%)NHS services · Run 2: 10% (n=20; 95% CI 4.2%–17%)NutritionNutrition · Run 1: 12% (n=20; 95% CI 6.2%–18%)Nutrition · Run 1: 12% (n=20; 95% CI 6.2%–18%)Nutrition · Run 2: 6.1% (n=20; 95% CI 2.1%–11%)Nutrition · Run 2: 6.1% (n=20; 95% CI 2.1%–11%)SymptomsSymptoms · Run 1: 10% (n=20; 95% CI 3.9%–16%)Symptoms · Run 1: 10% (n=20; 95% CI 3.9%–16%)Symptoms · Run 2: 11% (n=20; 95% CI 2.7%–23%)Symptoms · Run 2: 11% (n=20; 95% CI 2.7%–23%)TreatmentTreatment · Run 1: 9.2% (n=20; 95% CI 3.7%–16%)Treatment · Run 1: 9.2% (n=20; 95% CI 3.7%–16%)Treatment · Run 2: 10% (n=20; 95% CI 3.4%–17%)Treatment · Run 2: 10% (n=20; 95% CI 3.4%–17%)
Cross-model overlap: Jaccard similarity of the two models' cited publishers per question, averaged (1 = identical sources, 0 = none in common).

3. One model concentrates, the other spreads

The top 10 publishers hold 78% (n=100; 95% CI 75%–85%) of GPT-6.1 Sol's citation weight and 24% (n=100; 95% CI 23%–31%) of Claude Sonnet 5.5's (each answer counts equally; the interval leans upward because the top 10 is re-picked in every resample).

4. Time-sensitive answers often rest on old pages

For questions where the right answer can change (services, eligibility, waiting times, prices, guidance), an answer is counted as stale when even its newest dated source is over a year old (its date as last reviewed, else modified, else published). That was true of 43% (49/114; 95% CI 31%–55%) of GPT-6.1 Sol's time-sensitive answers and 12% (14/114; 95% CI 5.8%–20%) of Claude Sonnet 5.5's, both runs pooled. On the looser definition, any dated source over a year old, the figures are 78% (89/114; 95% CI 68%–88%) and 67% (76/114; 95% CI 56%–77%). Citations of NHS pages already past their own review-due date: GPT-6.1 Sol 22% (85/379; 95% CI 16%–29%), Claude Sonnet 5.5 1.1% (3/282; 95% CI 0%–2.9%).

Claude Sonnet 5.5GPT-6.1 Sol
0%25%50%75%100%All intentsAll intents · Claude Sonnet 5.5: 12% (14/114; 95% CI 5.8%–20%)All intents · Claude Sonnet 5.5: 12% (14/114; 95% CI 5.8%–20%)All intents · GPT-6.1 Sol: 43% (49/114; 95% CI 31%–55%)All intents · GPT-6.1 Sol: 43% (49/114; 95% CI 31%–55%)FitnessFitness · Claude Sonnet 5.5: 15% (3/20; 95% CI 0%–39%)Fitness · Claude Sonnet 5.5: 15% (3/20; 95% CI 0%–39%)Fitness · GPT-6.1 Sol: 15% (3/20; 95% CI 0%–39%)Fitness · GPT-6.1 Sol: 15% (3/20; 95% CI 0%–39%)NHS servicesNHS services · Claude Sonnet 5.5: 3.8% (1/26; 95% CI 0%–12%)NHS services · Claude Sonnet 5.5: 3.8% (1/26; 95% CI 0%–12%)NHS services · GPT-6.1 Sol: 73% (19/26; 95% CI 47%–95%)NHS services · GPT-6.1 Sol: 73% (19/26; 95% CI 47%–95%)NutritionNutrition · Claude Sonnet 5.5: 0% (0/20; 95% CI 0%–0%)Nutrition · Claude Sonnet 5.5: 0% (0/20; 95% CI 0%–0%)Nutrition · GPT-6.1 Sol: 30% (6/20; 95% CI 0%–62%)Nutrition · GPT-6.1 Sol: 30% (6/20; 95% CI 0%–62%)SymptomsSymptoms · Claude Sonnet 5.5: 19% (5/26; 95% CI 5.9%–33%)Symptoms · Claude Sonnet 5.5: 19% (5/26; 95% CI 5.9%–33%)Symptoms · GPT-6.1 Sol: 46% (12/26; 95% CI 20%–72%)Symptoms · GPT-6.1 Sol: 46% (12/26; 95% CI 20%–72%)TreatmentTreatment · Claude Sonnet 5.5: 23% (5/22; 95% CI 0%–50%)Treatment · Claude Sonnet 5.5: 23% (5/22; 95% CI 0%–50%)Treatment · GPT-6.1 Sol: 41% (9/22; 95% CI 12%–71%)Treatment · GPT-6.1 Sol: 41% (9/22; 95% CI 12%–71%)
Primary definition: share of time-sensitive answers whose newest dated citation is over 365 days old. Per-intent cells rest on few answers and are directional.

Per-intent cells rest on 10 to 13 questions each and are directional; the largest gap is for NHS services questions (GPT-6.1 Sol 73% (19/26; 95% CI 47%–95%) · directional; Claude Sonnet 5.5 3.8% (1/26; 95% CI 0%–12%) · directional).

How far to trust these numbers

What publishers can take from it

citewatch builds a card for each of the 361 cited publishers: how often each model cites it, by topic, who is cited instead, and readability problems found on its cited pages. Across all cards the most common were robots.txt files citewatch could not read (54 publishers), pages with no date citewatch could find (51 publishers), pages dated more than a year before a time-sensitive answer (48 publishers). The advice is about being readable and current: a visible, marked-up date and an available robots.txt. It is not, and will not be, advice on gaming AI answers.

Method and reproducing it

The questions, settings, seeds, metrics and the alpha of 0.05 with Holm correction were pre-registered before the first answer. Intervals come from a question-cluster bootstrap; counts from Wilson intervals. The two study runs cost $15.03 in API calls. Anyone can rerun it with their own keys: install citewatch from the repository, then run citewatch ask, extract, fetch and score.