# Backtest of the Trends readings: plan Written on 2026-11-06, before the first run. It fixes the question, the sample, the rules and the scoring. Nothing here may change once results are seen. Anything done differently is listed under Deviations at the end, with the reason. This measures past readings. It makes no claim about any week after the data ends. ## What was known before this was written - The site's own figure: a growing reading was followed by another one 12 weeks later 13 percent of the time, across all topics and weeks since 2016. - The latest reading of each topic, as printed on /trends. - The topic list, the reading rule and its three numbers (12 weeks, 25 percent, a floor of 24), which were set after the first counts of the list were seen. - The list itself was fixed on 2026-11-05. It was chosen with knowledge of what mattered by then. So no part of this history is blind to the choice of topics. "Held out" below means held out from anything this test fits. It does not mean unseen by the person who wrote the list. ## Data `lib/website/trends/counts.json`, the 17 topics, weeks 2016-01-04 to 2026-10-26, as retrieved on 2026-11-06. Pinned to that last week, so later weeks do not change the result. Raw counts, as Trends reads them. The all-documents row is not a case. ## Terms `counts[i]` is week i, with week 0 = 2016-01-04. N is the number of weeks. A case is one topic at one origin t, 0 < t <= N, where weeks before t are known and week t is the first unknown week. - B = sum of weeks t-12 to t-1, the window just read. - A = sum of weeks t-24 to t-13, the window before it. - C = sum of weeks t to t+11, the next window. It is never an input. `read(last, previous)` is the rule in `directionOf`, unchanged: - last + previous < 24 is Too few to read. - previous = 0 is Grow. - a change of +25 percent or more is Grow, -25 percent or less is Fade, and between is Hold. The reading at t is `read(B, A)`. The next reading is `read(C, B)`, which is the reading the site would have printed at t+12. ## Sample A case is eligible when all of these hold: 1. t >= 52, so every rule has the year of inputs it needs. 2. t + 12 <= N, so the outcome exists. 3. No week from t-52 to t+11 falls in the stored lapse, 2025-10-06 to 2025-11-24. A case is scored when it is eligible and the reading at t is not Too few to read. A reading of Too few makes no call. Those cases are counted and reported. Periods, split by time, with the split fixed here at 2022-01-03: - Development: the last week of C is before 2022-01-03. - Held out: week t is on or after 2022-01-03. - A case whose C straddles the split is in neither. The only thing fitted is rule T below, on development cases only. Spacing. Weekly origins overlap heavily, so they are not separate results. - Primary sample: held-out origins 12 weeks apart, starting at the first held-out origin. - Also reported: the other 11 possible starting weeks, as a range, and all weekly origins. ## Question 1, primary: the next reading Which rule most often names the reading 12 weeks later? A hit is an exact match of the next reading. If the next reading is Too few to read, every rule misses, and the case stays in the denominator. - R, the reading repeats: the next reading is the reading at t. - H, repeat the last window: assume C = B, which reads Hold. So H always says Hold. - Y, same window last year: the reading that window had one year before, `read` of weeks t-40 to t-29 against t-52 to t-41. If that is Too few, it says Hold. Secondary rules, reported but not part of the verdict: - V, the reading reversed: Grow says Fade, Fade says Grow, Hold says Hold. - T, the reading as history says: for each reading, the most common next reading in the development cases, spaced weekly. Ties go to Hold. Score: hits divided by scored cases. Also reported: hits by reading, so that a rule cannot hide behind the commonest class, and the full table of reading against next reading, next to the plain rate of each reading over the same cases. ## Question 2, primary: the count in the next window Which rule gets closest to C? - R, the reading carried forward: B x 1.25 after Grow, B after Hold, B x 0.75 after Fade. This is the smallest move each reading states. - N1, repeat the last window: B. - N2, mean of the last two windows: (A + B) / 2. - N3, same 12 weeks last year: sum of weeks t-40 to t-29. - N4, the last 52 weeks' pace: sum of weeks t-52 to t-1, x 12 / 52. Score: the sum of |prediction - C| over the sum of C, on the scored cases. Also reported: the number of topics where R beats N1 on the same score, because the large topics weigh most in the pooled figure. ## Verdict rule Intervals: resample the origin dates with replacement, keeping every topic at a date together, 10,000 times, seed 20261106, generator mulberry32. Take the 2.5 and 97.5 percentiles of the difference between two rules. Topics are not treated as independent, because one regulatory agenda can move many of them on one day. For each primary comparison of R against a baseline, in the primary sample: - "Better": R scores better and the interval excludes zero. - "Worse": R scores worse and the interval excludes zero. - "No measured difference": otherwise. Each is stated as measured, whichever it is. ## Secondary descriptions Reported for Grow and for Fade, on held-out scored cases, no verdict: - Went on: after Grow, C > B. After Fade, C < B. - Move kept: after Grow, C >= 1.25 x A. After Fade, C <= 0.75 x A. ## Reconciling the 13 percent Using the site's own loop, every weekly origin t from 24 to N-12, all topics, lapse included: 1. Reproduce the site's cases, repeats and percent exactly. 2. Say what followed those cases: Grow, Hold, Fade, Too few. 3. Give the plain rate of Grow over all readable cases in the same loop. 4. Count the separate runs of consecutive weekly Grow readings in those cases, per topic, to show how many separate episodes the case count stands for. 5. Give the same rate for: eligible cases, development, held out, and the primary sample. 6. Shuffle each topic's weeks independently, 1,000 times, seed 20261106, and give the mean and the 2.5 and 97.5 percentiles of the same rate. This shows what the rate looks like with the order of the weeks removed and each topic's total kept. ## Sensitivity, no verdict - The primary comparisons again with rule 3 of the sample dropped, so the lapse is included. - The development period, scored the same way. ## Reproduction `node scripts/backtest-readings.mjs` writes every number to `lib/website/trends/backtest.json` and every weekly case to `lib/website/trends/backtest-cases.csv`. Adding `--check` recomputes and fails on any difference. The arithmetic gets an independent reimplementation before anything is published. ## Deviations Each entry was added after the first run. The text above this section is unchanged. The hash recorded before that run is of this file with this section reading "None so far." The script rebuilds that version and fails if the hash no longer matches. 1. Correction. Rule Y and rule N3 are named "same window last year", but the weeks written above are wrong. C is weeks t to t+11, so C one year earlier is weeks t-52 to t-41, not t-40 to t-29. The weeks written above sit 12 weeks later in the calendar than C. Reviewing the first run's output found this. The script now uses t-52 to t-41 for N3, and for Y it reads those weeks against t-64 to t-53. Because Y now needs 64 weeks of inputs, rule 1 of the sample is t >= 64 and rule 3 reads from t-64. No held-out case is affected by those two changes. The development period starts 12 weeks later, so T is refitted on slightly fewer cases. The rules as first written stay in the results as Yw and N3w, with their intervals, so both can be seen. 2. Additions, descriptive and with no verdict, made after seeing the first run: - The rate of a Grow followed by a Grow at each of the 12 starting weeks, because the primary sample's rate sat far from the weekly rate. - In the shuffle test, what followed a Grow and a Fade besides Grow after Grow, because the fitted rule T showed the reading working in reverse. - The same intervals at each of the 12 starting weeks, because the first start turned out to be the kindest to the reading. The verdict still comes from the first start alone. 3. The independent check found these in the script. None changed a published score or verdict: - The `scored` column of the cases file ignored rule 3. It now follows the plan. - Sensitivity intervals carried verdict words, which the plan does not give them. Removed. - Rule T sent some ties away from Hold. No tie occurs. Fixed to match the plan. - Some differences and comparisons used rounded values. They now use exact ones.