Who AI search cites for UK health questions, and how fresh it is
Ask GPT-6.1 Sol a UK health question with web search on (run 1) and nhs.uk is among its sources in 83% (83/100; 95% CI 75%–90%) of answers. Ask Claude Sonnet 5.5 the same question and it is 10% (10/100; 95% CI 5.0%–16%). Same questions, same day, same place: the two models lean on different parts of the web.
This study measures sourcing: which websites 2 AI search models cite when asked UK health questions, and how recently those pages were updated. It does not judge whether any answer or page is medically accurate. 100 pre-registered questions across 5 intents, asked twice (runs 1, 2) of Claude Sonnet 5.5, GPT-6.1 Sol with web search: findings are directional for this setup, not a verdict on AI search in general. Every percentage carries its count and 95% interval.
The full numbers are on the dashboard; what this could become as a
data product is in the one-page signal spec. Code, data and every
decision are in the open-source repository.
What was measured
100 questions across 5 intents (fitness, NHS services, nutrition, symptoms, treatment),
written and frozen before any answer was seen, were each asked of Claude Sonnet 5.5 and GPT-6.1 Sol with web
search, in 2 runs: 400 answers in all. citewatch recorded every source each
answer cited, then visited each cited page, politely and only where robots.txt allowed, to find
when it was last reviewed, modified or published. Every share and rate below carries its count
and a 95% interval.
1. The models lean on different sources
Claude Sonnet 5.5 spreads its citations across the wider NHS family and
others; its most cited publishers in run 1 were nhsinform.scot (17% (17/100; 95% CI 10%–25%)), england.nhs.uk (13% (13/100; 95% CI 7.0%–20%)), icb.nhs.uk (10% (10/100; 95% CI 5.0%–16%)).
Claude Sonnet 5.5GPT-6.1 Sol
Run 1: share of answers citing each publisher at least once (presence share), top 15 publishers, 95% intervals.
The gap for nhs.uk
holds in every run and survives a Holm
correction across each run's most cited publishers. 10 of
21 paired comparisons separate the models.
2. They rarely cite the same sources
Comparing the publishers each model cites for the same question, the average overlap
(Jaccard similarity) is 10% (n=100; 95% CI 7.4%–13%) in run 1. It ranges from
8.7% (n=20; 95% CI 2.4%–17%) for
fitness to
12% (n=20; 95% CI 6.2%–18%) for
nutrition questions.
Run 1Run 2
Cross-model overlap: Jaccard similarity of the two models' cited publishers per question, averaged (1 = identical sources, 0 = none in common).
3. One model concentrates, the other spreads
The top 10 publishers hold 78% (n=100; 95% CI 75%–85%) of GPT-6.1 Sol's citation weight and
24% (n=100; 95% CI 23%–31%) of Claude Sonnet 5.5's (each answer counts equally; the interval leans upward
because the top 10 is re-picked in every resample).
4. Time-sensitive answers often rest on old pages
For questions where the right answer can change (services, eligibility, waiting times,
prices, guidance), an answer is counted as stale when even its newest dated source is over a year
old (its date as last reviewed, else modified, else published). That was true of 43% (49/114; 95% CI 31%–55%) of GPT-6.1 Sol's time-sensitive answers and
12% (14/114; 95% CI 5.8%–20%) of Claude Sonnet 5.5's, both runs pooled. On the looser definition, any
dated source over a year old, the figures are 78% (89/114; 95% CI 68%–88%) and
67% (76/114; 95% CI 56%–77%). Citations of NHS pages already past their own review-due date:
GPT-6.1 Sol 22% (85/379; 95% CI 16%–29%), Claude Sonnet 5.5 1.1% (3/282; 95% CI 0%–2.9%).
Claude Sonnet 5.5GPT-6.1 Sol
Primary definition: share of time-sensitive answers whose newest dated citation is over 365 days old. Per-intent cells rest on few answers and are directional.
Per-intent cells rest on 10 to 13 questions each and are directional; the largest gap is for
NHS services questions
(GPT-6.1 Sol 73% (19/26; 95% CI 47%–95%) · directional; Claude Sonnet 5.5
3.8% (1/26; 95% CI 0%–12%) · directional).
How far to trust these numbers
Dates. citewatch found a date, or a logged reason for having none, for
89% (745/841; 95% CI 86%–91%) of cited URLs, just under the study's 90% target. On 55
held-out pages (agent-checked, owner-approved), the parser was within 7 days on
67% (18/27; 95% CI 48%–81%) of the pages that could be judged. Dates read from
visible page text were within 7 days on 100% (12/12; 95% CI 76%–100%); dates from
sitemaps, HTTP headers or the search tool's page age, which readers never see, on
14% (1/7; 95% CI 2.6%–51%) · directional.
Support. On a sample of citations, two AI judges from different families
found the claim stated on the cited page in 67% (16/24; 95% CI 47%–82%) (Claude Sonnet 5.5 cited, judged by Claude Opus 5.5); 67% (16/24; 95% CI 47%–82%) (Claude Sonnet 5.5 cited, judged by GPT-6.1 Sol); 57% (13/23; 95% CI 37%–74%) (GPT-6.1 Sol cited, judged by Claude Opus 5.5); 43% (10/23; 95% CI 26%–63%) (GPT-6.1 Sol cited, judged by GPT-6.1 Sol).
The owner's own check, made at a quick pace of one to two seconds per citation, found
100% (24/24; 95% CI 86%–100%) (Claude Sonnet 5.5 cited); 100% (25/25; 95% CI 87%–100%) (GPT-6.1 Sol cited) supported
(two citations the judges could not assess are in the owner's count only). The judges and the
owner disagree, and this check cannot settle the true rate.
Scope. One country, two models, 100 questions, two runs. This is a
measurement of sourcing, not of medical accuracy, and not advice for anyone choosing a health
source.
What publishers can take from it
citewatch builds a card for each of the 361 cited publishers: how often each model
cites it, by topic, who is cited instead, and readability problems found on its cited pages. Across
all cards the most common were robots.txt files citewatch could not read (54 publishers), pages with no date citewatch could find (51 publishers), pages dated more than a year before a time-sensitive answer (48 publishers). The advice is about being readable and current: a visible, marked-up
date and an available robots.txt. It is not, and will not be, advice on gaming AI answers.
Method and reproducing it
The questions, settings, seeds, metrics and the alpha of 0.05 with Holm correction were
pre-registered before the first answer. Intervals come from a question-cluster bootstrap; counts
from Wilson intervals. The two study runs cost $15.03 in API calls. Anyone can rerun
it with their own keys: install citewatch from the repository, then run
citewatch ask, extract, fetch and score.