Evaluation lab

Change detection validity

Five ways to decide that someone's thinking has changed, run on the same 3-year cohort of 5,000 people. The trade-off is the point: catching more decline costs more wrong referrals.

Test-retest (correlation)

0.73

95% CI 0.71–0.74 · 4,481 stable pairs

Practice effect allowed for

0.3 SD

SEdiff 0.6

CUSUM (cumulative sum) in-control run length (h = 4, k = 0.5)

329

checks before a false alarm · 2,000 runs

Usual-range coverage (stable people)

93.2%

target about 90% · per group on Subgroups

Methods compared

Method	Sensitivity (95% CI)	Specificity (95% CI)	False referrals per 5,000 · 3 years	Median months to detection
Fixed cut-off (single test)	33.1% (29.2–37.3)	71.4% (70.0–72.7)	1,283	16.8
5% drop, as in quality indicator LNG004	73.4% (69.5–77.0)	38.2% (36.8–39.7)	2,768	9.3
Reliable change, no confirm check	69.2% (65.1–73.0)	74.1% (72.8–75.4)	1,159	14.2
Reliable change + confirm check (ours)	59.5% (55.3–63.7)	93.3% (92.5–94.0)	300	17.9
Ours + CUSUM drift arm	73.0% (69.0–76.7)	82.7% (81.6–83.8)	775	15.6

Our rule waits for a second low check before referring. That gives up some sensitivity against single-check reliable change, and in return sends about a quarter as many stable people to their doctor. The CUSUM arm catches slow drift sooner, at the cost of more wrong referrals. It stays a clinician-side signal, not a trigger.

Harms of our method

False reassurance
People whose decline was not flagged within 3 years
210 / 519
Anxiety
Stable people asked to check again at least once
2,662 / 4,481
Wrong referral
Stable people sent to the family doctor
300 / 4,481
Delay
Median months from onset of decline to a referral
17.9 months

Repeat testing has noise, so many stable people are asked to check again at some point in 3 years. That is why the check-again message is calm, explains why, and books the repeat for them.

What each part adds (ablations)

Variant	Sensitivity	Specificity	False referrals
Full method	59.5%	93.3%	300
Without readiness gates	57.2%	91.0%	405
Without the confirm check	69.2%	74.1%	1,159
Without practice adjustment	40.1%	98.8%	54
Baseline from one check	51.8%	82.0%	808
CUSUM drift only	73.0%	83.2%	752

How the cohort is built

Data dictionary

People and follow-up: 5,000 people, 3 years, 4 checks a year
Share with true decline: 10% · 0.4–0.8 SD/year
Measurement noise per check: 0.42 SD (15% higher with help or 0–9 years of school)
Practice gain after the baseline: 0.3 SD
Bad days: 8% of checks, -0.9 SD. Readiness questions catch 70%
Seed: 20261113

Method

Sensitivity (95% CI)

Specificity (95% CI)

False referrals per 5,000 · 3 years

Median months to detection

Fixed cut-off (single test)

33.1% (29.2–37.3)

71.4% (70.0–72.7)

1,283

16.8

5% drop, as in quality indicator LNG004

73.4% (69.5–77.0)

38.2% (36.8–39.7)

2,768

9.3

Reliable change, no confirm check

69.2% (65.1–73.0)

74.1% (72.8–75.4)

1,159

14.2

Reliable change + confirm check (ours)

59.5% (55.3–63.7)

93.3% (92.5–94.0)

300

17.9

Ours + CUSUM drift arm

73.0% (69.0–76.7)

82.7% (81.6–83.8)

775

15.6

Variant

Sensitivity

Specificity

False referrals

Full method

59.5%

93.3%

300

Without readiness gates

57.2%

91.0%

405

Without the confirm check

69.2%

74.1%

1,159

Without practice adjustment

40.1%

98.8%

Baseline from one check

51.8%

82.0%

808

CUSUM drift only

73.0%

83.2%

752