German Statutory Health Insurance Websites: What Do You Review When You Cannot Review Everything?

A sentence about a benefit appears on a health insurer's website. The sentence is correct. It was already there four years ago, but back then it included the condition that is now missing.

You do not find sentences like that at a glance. You find them only by going through the entire estate, and no estate is small enough for that. I collected the public pages of 84 websites, portals, and subsidiary web properties run by German statutory health insurers and had them reviewed by machine: 56,198 pages, an average of roughly 670 per property. Hardly any editorial team can review all of them professionally once a year.

Until the end of 2024, I was responsible for member communications at a health insurer. I know the editor's side of the work: the estate that has grown over time, the legacy content from three relaunches, the page nobody touches because it is unclear who has subject-matter responsibility for it.

The Order of the Review

The reflex is “more people.” The arithmetic does not work. Reviewing 670 pages professionally once a year requires capacity that does not exist alongside the day job, and the same task returns the following year. What does work is something else: a justified order.

Prioritization happens anyway. The only question is whether it follows a plan or whatever happens to catch the eye. Pressure on that order is now growing from a direction that did not exist for the public four years ago: ChatGPT has been publicly available since November 2022. A literature review notes growing interest in using AI systems such as ChatGPT to communicate health information.² These systems summarize public web content and detach it from the context of the source page. A benefit sentence without its condition used to be imprecise. Today it becomes the template for an answer produced elsewhere that the editorial team can no longer check.

Researchers from institutions including the University of Zurich, Stanford University, and Northeastern University measured this in April 2025 on 1,508 real search queries about pregnancy and baby care.³ Two numbers stand out: the AI Overview and the featured snippet on the same Google results page contradicted each other in one third of the cases where both appeared. Advice to seek medical assessment appeared in eleven percent of the AI Overviews. Whether a missing condition resurfaces somewhere later is not something the source page controls.

How I Approached It

I am not aware of a study of this design for German statutory health insurance. What I counted were web properties, not insurers: the 84 units are websites, portals, and subsidiary properties, and one insurer can run several of them. For a sense of how much of the sector that covers, the National Association of Statutory Health Insurance Funds lists 93 insurers (as of 24 September 2026).⁴

The review looked in five directions at once: transparency, legal framing, medical communication, contradictions within an insurer's own text, and signals pointing to AI authorship.

What the study excludes is part of the same claim. Accessibility and UX writing are outside the review framework. Anyone expecting a study of every quality dimension will not find one here.

The review runs in three stages. First comes a rule-based pre-screen for defined signal patterns. Then an AI-assisted triage passes conspicuous pages onward and moves the rest into a shallower review stage: lower priority, not clearance. Finally, an in-depth AI review produces structured review records, each with a verbatim passage, an assessment of what is at stake, and an explicit counterargument.

Three safeguards separate this method from simply “running a language model over the site once.”

Evidence requirement. Every record must include a verbatim passage from the page under review. A deterministic comparison then checks whether the quoted passage can actually be found in the captured page text. Of 35,998 review records, this succeeded for 31,347. Another 4,631 quotations could not be located, and 20 records had no usable status. Those stay flagged as needing review rather than going out as substantiated findings.

Two clarifications matter because this number is easy to misread. It counts review records, not pages and not errors: one page can carry several, and none of them is an established defect. The comparison also proves that the passage is present, not that its claim is true.

Second assessment. Two independently configured models assessed the same material in 182 paired cases. They agreed in 75.8 percent of cases. That sounds like a lot, but it is not: once you subtract how often two assessors would agree by chance alone, what remains is moderate agreement (Cohen's kappa 0.532; 95% confidence interval 0.415–0.649). Forty-four cases went on to downstream professional clarification. That is not a flaw in the method but a design principle: two language models are not independent reviewers, and shared training data can produce shared mistakes. Agreement is therefore a triage signal, not proof of truth.

Currency safeguard. The most awkward failure mode occurs when the page under review and the reviewing model share the same outdated knowledge. The model then confirms a superseded statement as correct. Curated currency checks cover topics prone to change.

What comes out at the end is a prioritized worklist. Not findings. AI handles the persistent first pass; the assessment stays with people.

What Recurs Across Many Properties

Across the properties, 290 review patterns recur, 42 of them at the highest priority level. The table shows seven of those 42. The patterns overlap, and none of them is an established finding.

The most important point is not any single number but the spread. The need for review is structural, not confined to a few conspicuous properties.

The first pattern is the most widespread, appearing in 67 of the 84 properties. Consistency of medical statements, by contrast, affects only 24 properties. The spread shows where a shared review route across many properties pays off. It does not show how much work sits in a single property; that takes a look at the property itself.

One qualification should not be skipped: these are review candidates, not established violations. A number in this table means that a person with domain expertise should take a look, not that something is wrong. The study deliberately names no insurers and produces no ranking. A ranking would give the notes a finality that the method does not claim.

The study does not measure error rates. It collects review candidates, and the questions behind them are always the same. Does a statement carry its conditions? Does it still hold? What supports it? Is it medically coherent, and what legal framework does it sit in?

Governance Remains Invisible from the Outside

An additional module checked whether basic governance signals are discoverable along a defined public search path. In other words, can an outside observer tell how an insurer maintains its health content?

Of 84 properties, eleven name a clear editorial responsibility for content. Six show a visible route for reporting a content error. Four say how often they review their content. Three say anything about how they handle AI-assisted text production.

These numbers measure public discoverability only. A missing signal does not mean that internal processes are missing, only that they are not visible to members, researchers, and regulators. For insurers expected to promote digital health literacy, that is itself an issue.

Something similar has been observed before. In summer 2022, Viviane Scherenberg and Melanie Preuß searched the websites of 97 insurers for offers on digital health literacy; such offers were directly findable via the site search for 16 insurers.¹ Different scope, different measurement, but they published their finding back in 2023.

What Editorial Teams Can Do with This

The method shifts the question from “can we review everything?” to “where do we direct limited review capacity first?” Four questions are enough as a framework. Every health and benefit page should be able to answer them:

1. Who has subject-matter responsibility for this statement?
2. Which reference governs it?
3. When was it last reviewed?
4. What happens when the bylaw, the evidence, or the legal framework changes?

Anyone who can answer these four questions for their top pages has a basis for working through the review need in a targeted way. Anyone who cannot at least knows where to start.

In practice, this becomes a prioritized review worklist instead of a full sweep. Fixed review cycles for topics prone to change. Clear subject-matter ownership for health and benefit information. Update triggers for changes to guidelines, law, or benefit status. And visible governance signals, so outsiders can see that the content is maintained.

None of this requires an external audit or new technology. Most of it is organization, not tooling. The full method and the bounded work sample are on the AußenBlick GKV project page.

And the Benefit for Members?

It is indirect, but it is the point. Where insurers work through the prioritized patterns, the stock of unreviewed legacy content shrinks, benefit information carries its conditions, and sources and currency become traceable. That works twice over: for members who read the page directly and for everyone who receives the same information through an AI system, in compressed form and without the context of the source page.

I did not measure that benefit. The study measures review need, not effect. But the direction is plausible: the more current and consistent the source text, the more reliable the outputs that machines derive from it are likely to be.

Sources & References

---

Method, data cutoff, and limits are documented in the preprint. The full text is available at arXiv:2608.03500 and as a Zenodo preprint. A separate, edited snapshot package is published as a dataset. The raw crawl, full page texts, production code, prompts, and thresholds are not public. Data as of May 2026.

Which review step would you add from your own practice, and which of the four governance questions is hardest to answer in your organization?