What this console is

A private dashboard at data-headlines.pierce.tools that turns published-headline performance into editorial decisions.

Every article McClatchy publishes lands in the Tracker (the content team lead's master sheet). Snowflake enriches that with platform-level page-view data—Apple News, SmartNews, Yahoo, Newsbreak, search, social, subscriber, newsletter. Tarrow adds platform-syndication detail. The console reads from that joined surface and asks one question per tile: which combinations of pub, topic, formula, vertical, and writer beat their baseline by enough to act on?

It's the analytics counterpart to the Keywords Decisioning Console: keywords decides what to write next; headlines learns from what was written.

Who uses each surface

Concepts 101 · zero-knowledge primer

The console reuses a handful of recurring terms. If anything below is unfamiliar, start here.

Page-views (PVs / total_pvs)

The total times an article was loaded across every distribution surface in the period (Apple News + SmartNews + Yahoo + Newsbreak + search + social + direct + newsletter + AMP + subscriber). Sourced from STORY_TRAFFIC_MAIN in Snowflake.

Lift

A multiplier comparing one group's median PVs to a baseline. +50% means the group's median article hit 1.5× the baseline. Negative = below baseline. The baseline depends on context: usually the publication's overall median, sometimes the vertical median, sometimes the untagged-formula baseline.

Median (not mean)

The console reports medians everywhere because page-view distributions are heavy-tailed—one viral hit can shift a mean by 5×. Medians are stable; viral outliers don't distort the signal.

Headline formula

A regex-classified template the headline matches. Priority-ordered lead-position patterns (number_lead, heres, quote_lead, name_lead) then structural cues (did_you_miss, what_to_know, explainer, definitional, how_to, question, dash_directive, superlative_list, comparative, negation_lead, curiosity_gap) and a length-based structural fallback (short_punchy, descriptive_standard, descriptive_long) so every headline buckets. See the full taxonomy in the glossary.

Vertical

One of three named TH-channel verticals: Mind/Body, Everyday Living, Experiences. Articles published outside these channels (search/discover work) aren't bucketed.

Content type

Either News & Regional (T1 newspapers—Miami Herald, KC Star, Charlotte Observer, etc.) or L&E (US Weekly, Woman's World, and Life & Style). The active lens at the top of Findings restricts every tile to one or the other. (InTouch and Closer are retired L&E brands the console recognizes only for News-vs-Entertainment tag coloring—they aren't in the active Findings cohort.)

Verdict

Each tile assigns each row a verdict: go (real lift, strong evidence), test (directional, worth piloting), or skip (below baseline or thinly evidenced). Color-coded green / amber / coral on the tile rail and pills.

P-value

The probability the observed lift could have arisen by chance from the baseline distribution. p<0.05 = real signal; p<0.10 = directional; otherwise no detectable difference. Computed via Mann-Whitney U-test (non-parametric—no assumption of normal distributions).

The lens filter (News / L&E / All)

Three pills at the top of Findings. Active pill governs the cohort (a cohort = a group of publications sharing a content type) + primary metrics across every tile below.

Findings: the 7 tiles

The home page (/findings/) renders seven analytical tiles. Each tile is a self-contained question + answer; click anywhere on a tile to open its drawer with the underlying data.

1. Per-publication: top-performing topics—which topics each pub hits hardest.
2. Per-publication: top-performing formulas—which headline patterns earn lift at each pub.
3. Per-vertical formula × site—three TH-channel rails (M/B, EL, Exp) with cluster batting average.
4. Trends Over Time—per-pub PV trajectory by quarter.
5. Formula × Topic × Publication—three-way interaction including weather as a topic dim.
6. Bottom-Performer Patterns—confirmed anti-patterns to avoid.
7. Team Performance—per-author percentile vs publication median.

Above the tiles sit the lens pills (News / L&E / All) and a filter-chip row that narrows every tile at once to one content lane, publication type, or headline formula. Several tiles (and the home page) also carry a recent top-performers panel you can rank by traffic or by reader-engagement.

Tile · Per-publication topics

For each publication, the topics that exceed the pub's median PVs by the largest margin. Lift = (topic median − pub median) ÷ pub median. n = articles in the topic at that pub during the lookback window.

Use: assignment prioritization. If Politics at Miami Herald is +34% with n=120 and p<0.05, that's a durable signal—the next politics piece probably outperforms a same-pub baseline article.

Caveat: topic mix shifts seasonally. Lift numbers in the per-pub formula + vertical-cube tiles default to recency-weighted (last 12 months, decayed by ~0.95^weeks-since-publish ≈ 3-month half-life) so "what's working now" beats "what worked over the year."

Tile · Per-publication formulas

Same shape as Per-publication topics, but classifying by headline formula instead of topic. The baseline is the pub's overall median PVs across all headlines; lift = (formula median − pub median) ÷ pub median. Every headline buckets into one of the formulas in the taxonomy (the length-based structural fallback catches anything without a signature pattern), so there's no untagged baseline.

News editorial truth: direct-declarative + quote-lead likely outperform wordplay on local news in Google News/Discover. The data confirms or revises that prior per publication.

L&E editorial truth: curiosity-gap likely beats spoiled-answer. Confirmed on UsW with +42% vs −35% (n=19 / n=12, p<0.05).

Tile · Per-vertical formula × site cube

Three rails—Mind/Body, Everyday Living, Experiences—each showing the formula × site combinations with the largest lift vs vertical median.

The cluster batting average in the meta line answers: of all CSA-touched articles (those with a cluster_id), what fraction sit in a cluster with a positive cluster_vs_co_median? It's article-level, so larger clusters count more (more articles = more votes). Reported as "1 in N"—lower N = better.

Use: formula assignment at the vertical level, or evaluating whether a creator's cluster strategy is paying off.

Tile · Trends Over Time

Per-publication median PVs by quarter. Lift trend = (latest quarter median − first quarter median) ÷ first quarter median.

Tile · Sweet-spot finder (pub × topic × formula)

A topic might look flat at a pub overall—but a specific formula can unlock it. This tile finds those three-way sweet spots so the per-tile-1 and per-tile-2 single-axis views don't bury them.

What it shows

Every (publication, topic, formula) combination with n ≥ 3 articles, scored by lift vs that publication's median PVs. Sorted by lift descending. Verdict overlays the same go / test / skip rule as everywhere else. The cube is the most fine-grained tile in the console — at small corpus sizes it surfaces only the densest 1-3 cells; as the article volume grows it fills out.

How to use it

• Look for combinations marked go with high n. Those are the assignable sweet spots—pub + topic + formula triples that beat the baseline reliably.
• If a topic shows flat or negative on the per-pub topics tile, scan here before writing it off—the right formula may rescue it.
• Use the highlighted rows to directly compare editorial-principle pairs (e.g. curiosity-gap vs spoiled-answer at UsW).

Weather signal sidecar

The right pane shows the 30-day weather lift on T1 newspapers (median PVs of weather-classified articles vs the T1 baseline) plus the count. Weather is included in the main table as a topic dimension; the sidecar separates it so seasonality is visible at a glance.

Tile · Bottom-Performer Patterns

Anti-patterns: combinations whose median PVs sit ≤0.5× their pub median, with n ≥ 10.

Currently three confirmed patterns: local-team sports without national hook (T1s), spoiled-answer headlines (L&E), punny/wordplay on local news (T1s).

Use: what to avoid. Full guidance with examples lives in the bottom-performer drawer of each per-publication tile.

Tile · Team Performance

Per-author percentile vs publication median. Author shown as initials (first + last name leading letters; collisions disambiguated with a digit suffix). Sorted by pub-percentile descending.

A writer at 70th pctile means their median article ranks at the 70th percentile of articles published at their primary pub—read it like a class rank: their typical article beats 70% of what their pub published.

Two extra percentile readings sit beside the headline one so writers are compared like-for-like: vertical-peer percentile (rank against only other writers in the same content lane—fairer than ranking a recipes writer against a politics writer) and tier-peer percentile (rank against writers at comparable-size publications). The small ± figure on each is the wiggle room from sample size (a bootstrap confidence range).

Recent top-performers panel (+ engagement rank)

A side panel that rides several tiles (and the home page) listing the standout published headlines from the last 7 days. New since the 5/04 docs cut: you choose what to rank them by.

An axis toggle above the list—four buttons: PVs, Engaged %, Time on page, Cluster cont.—lets you choose the ranking metric, and your choice is remembered on your next visit:

An article missing the chosen measure drops off that list (e.g. a piece with no engagement coverage won't appear when you rank by Engaged %). When you rank by PVs, each row also carries a small decile badge (like D8) showing which tenth of its content lane the article lands in by lift (how far it beat the publication median).

Reading the numbers (lift · confidence range · verdict dot · decile)

Four small affordances appear throughout Findings. None require statistics background—here's how to read each at a glance. Full definitions are in the glossary.

• Lift — how far above or below normal a group performed vs a typical article in the same publication or lane. +50% = half again as much (1.5×); −20% = a fifth less; 0% = exactly normal.
• Confidence range — the bracket after a lift number, like [+22, +47], is the likely range of the true effect: about 95% confident the real lift sits between those two numbers. Narrower = more reliable; wider = more uncertainty. It only shows when there are enough articles to compute it (n ≥ 10).
• Verdict dot — the colored dot in each tile's header is that tile's overall call at a glance: green = trusted and actionable (go), amber = worth testing (test), coral/red = skip, or too uncertain to act on yet.
• Decile badge — on top-performer rows ranked by traffic, a badge like D8 says which tenth of its content lane the article sits in by lift (how far it beat the publication median); D10 is the very top tenth (the badge is highlighted when an article reaches it).

Each of these has a (?) help bubble on the live page with the same plain-language definition. The docs and the tooltips are kept in lock-step.

Anatomy: tile + drawer

Every tile has the same skeleton:

• Rail. Vertical accent strip on the left edge. Color encodes verdict—green = go, amber = test, coral = skip.
• Title row (t1). Tile name + (?) help bubble + a one-line meta summary (n records · key signal). Always visible.
• Chevron (▸ / ▾). Right edge of the title row. Click anywhere on the title row to expand or collapse the drawer.
• Drawer (t2). Hidden until expanded. Two-pane: main column with description + data table; side column with recent top-performers (or per-tile signal—formula hit rates, vertical totals + cluster batting average, weather signal).

Default state: all 7 tiles land collapsed. Click to open the drawer for the tile you want to dig into. Multiple drawers can stay open at once.

Grader page + today's grades

Two grader surfaces feed off the same daily run:

• Today's published grades on the home page—4 KPIs (headlines graded · avg score · range · top issue) reflecting the most recent grader run.
• Grader page—full per-headline scorecard for every headline in the lookback window, sorted worst-first. Each headline card expands to show the full rule-by-rule breakdown plus per-vertical tips. The page also carries a 30-day history strip, a per-outlet pass-rate matrix (every rule × every publication), and a criterion-correlations panel (which rules tend to be missed together).

The grader runs daily at 10:17 CDT (15:17 UTC) against the Tracker. It scores each headline against the rule set (a mix of fixed computer checks and a Groq LLM for the four judgment-call rules—active voice, lead burial, curiosity, accuracy), writes the results to docs/grader/index.html + docs/data/grader-signals.json + docs/grader/history.json, and commits. The score is the share of rule-weight a headline earned (see Grader tiers + the score).

Manual trigger (operators only): gh workflow run headline-grader.yml. The in-page Run-now button was removed 2026-05-04—a hardcoded passcode + localStorage PAT isn't a defensible auth surface for a public-readable site.

Grader tiers + the score

Every rule belongs to a tier, and tiers carry different weight. A headline's score is the share of applicable rule-weight it earned—not a simple count of passes.

The score, in one sentence

Add up the weights of the rules a headline passed, divide by the weights of all rules that applied to it, multiply by 100. Rules that don't apply (the platform-title rules, or a keyword rule when no keyword was supplied) are left out of the denominator entirely—they neither help nor hurt. So two headlines can be scored against different rule sets and still compare fairly.

The four tiers + their weights

The Grader page shows these four as a KPI strip with each tier's average pass rate and its weight (e.g. "27% wt"). Within a headline card, a mini stacked bar shows where that headline's points came from by tier (blue = Structure, purple = Formula, green = Quality), with a dark tail for points it didn't earn.

Format-aware grader rules

Some article formats lock the H1 to a pattern that universal rules would penalize. The grader detects the article's Format column and overrides the universal criteria where they conflict with the format spec.

Synthesis rule

Format spec wins where it conflicts with universal rules (since the format spec is the locked editorial standard). Universal rules apply where the format is silent.

Format → required H1 + exempt criteria

Source of truth: csa-content-standards/docs/<format>.md. Each format spec carries the canonical pattern; the grader's _FORMAT_RULES table is a synthesis of those.

What happens if the Format column is blank

The universal rule set applies (the same 19 criteria as everywhere else). No format-specific override fires. The card's meta line shows the author + vertical + platform + brand but no format badge.

The headline experiment (framing test + expert signal)

A live test, run inside the Variant tool, that separates two very different kinds of editorial knowledge: a lever we've proven and want to enforce, and a hypothesis we want to test before trusting. This section is the high-level "what + why"; the operator how-to lives in the Variant tool documentation.

The two halves, and why they're treated differently

The June 2026 headline analysis turned up one finding strong enough to act on now and one question still open. Putting them on the same tool forced a clean split:

• Proven lever — the expert/study signal. When a story is built on a credentialed expert or a named study, naming that in the headline reliably lifts performance (the analysis measured roughly +71% page-views and +50% engagement time). Because it's a measured result, not a guess, it's enforced: it became a scored grader rule (see Expert/study signal), alongside length and the punctuation ban.
• Open hypothesis — the trend-stage framing. Whether describing where a trend sits in its lifecycle changes how readers engage is unknown. So it's tested, not scored: a headline can be written in a test framing without that framing helping or hurting its grade. Scoring a framing would amount to claiming we already know the answer—and we don't yet.

The organizing rule is simple: codify what's proven, validate what's not. A framing only graduates into a scored rule once engagement data picks a winner.

The three framing styles under test

Each is a way of framing the same trend in the headline—same article, different angle on where the trend stands:

Writers begin running the test on 2026-06-10. The framing styles are tuned weekly as data arrives, so they're deliberately kept somewhere editable without a redeploy—not baked into the governed rule library. The test runs alongside normal generation: a writer opts a headline into a test style, or generates the usual production headline; it's one labeled lane, not a separate tool.

The per-headline "receipt"

For the eventual "which style won?" to be honest, every generated headline has to be traceable to the exact setup that produced it—the thresholds, the rule set in force, how the expert signal was delivered, and which framing was under test. So each generation records an immutable config snapshot: a content-addressed "receipt." Two headlines with the same receipt provably ran under the same configuration; a different receipt means something changed. When engagement numbers land weeks later, each headline's outcome attributes to its precise receipt rather than to a guess about what was live at the time.

Bridge to keywords console

The console emits an outbound feed at /data/headline-outcomes.json with every published article's outcome (PV, lift, formula, topic, content type, vertical, publication date). The Keywords Decisioning Console reads that feed for its Decision Log lens—keywords with a published headline get the headline grade + PV outcome paired back automatically.

Inbound bridge (filter Findings tiles by a keyword brief) is stubbed in the t2-side panel—pending the data-keywords brief-id metadata to land in the Tracker.

The (?) help bubbles

Click any (?) icon to open a popover with a tight definition of that term, metric, or tile. Click anywhere outside or press Esc to close. Enter / Space with the icon focused also toggles.

Content lives in docs/js/help-blurbs.js as a single registry—every term documented in one file. To reference a blurb from any page: <span class="help" data-help="some-id"></span>. The icon + popover are auto-hydrated.

What auto-updates vs what's manual

Troubleshooting

Tiles show stale data

Check the live badge in the topbar. If it says missing for a feed, that feed's cron hasn't fired today. If all 3 feeds are bound but data looks old:

• Hard-refresh (Cmd+Shift+R)—Cloudflare's cache can lag a deploy by a few minutes.
• Every JS/CSS/JSON URL is now stamped with ?v=YYYYMMDDHHMM based on the cron run, so each new deploy auto-busts both the browser cache and Cloudflare's CDN cache.
• If the badge says 0 graded, no headlines were published in the last 24h.

Grader page modal won't close

If Cancel doesn't dismiss, click the dimmed area outside the modal box. Resolved as of 2026-05-02 commit 001356d.

Vertical filter doesn't show all options

Three verticals only: Mind/Body, Everyday Living, Experiences. Articles classified as "Discover" in source data are search/discover work mode, not a vertical—they bucket as — in vertical-keyed UI.

Writer initials collide

Two writers with same initials get a digit suffix: RB and RB2. The mapping isn't published; reach out if you need to dereference.

FAQ

Why are PV numbers different from what I see in Amplitude?

This console reads from MCC_PRESENTATION.CONTENT_SCALING_AGENT.TRACKER_ENRICHED, which composes per-platform PVs from STORY_TRAFFIC_MAIN + Tarrow XLSX. Amplitude only covers O&O; MSN traffic for example is Tarrow-only. Numbers will differ.

Why isn't InTouch / Closer / Life & Style on the lens filter?

The filter pills show the top-3 buckets (All / News / L&E), not per-publication pills. The active L&E cohort is US Weekly, Woman's World, and Life & Style; an L&E lens covers all three. InTouch and Closer are not in the active cohort—they're retired brands the console recognizes only for News-vs-Entertainment tag coloring, so they're graded for historical continuity but don't appear in the live Findings cohort.

Why is the (?) icon called the help bubble?

The (?) icon opens a popover with a "what is this?" blurb. The pattern is shared across the keywords + headlines consoles.

Architecture

Static HTML site deployed to Cloudflare Pages. No build step, no React, no bundler—pages are emitted by Python scripts and served directly. JS interactions are vanilla (no framework).

data/                          # source—Snowflake enrichment + Tarrow XLSX
  snowflake_enrichment.json    # ← snowflake_enrich.py (weekly cron)
  Top Stories 2026 Syndication.xlsx  # ← Tarrow (weekly)

tiles_new.py                   # weekly cron → docs/data/tiles.json + docs/index.html
generate_grader.py             # daily cron → docs/grader/index.html + docs/data/grader-signals.json

docs/                          # served by Cloudflare Pages
  index.html                   # home (Findings)
  data/                        # JSON feeds for tile rendering
  js/                          # vanilla JS interactivity
  css/styles.css + legacy-pages.css

Data sources + refresh cadence

File map

Snowflake TRACKER_ENRICHED schema

The vetted boundary table read by snowflake_enrich.py. Path: MCC_PRESENTATION.CONTENT_SCALING_AGENT.TRACKER_ENRICHED.

Columns this console relies on:

Tracker sheet integration

The editorial Tracker is a Google Sheet maintained by the content team lead, treated as the source-of-truth. It flows into the Snowflake model MCC_PRESENTATION.CONTENT_SCALING_AGENT.TRACKER_ENRICHED (rebuilt ~2×/day), and the grader reads that—it switched off the direct Sheets read on 2026-06-05 (the sheet stopped receiving new dated rows, frozen at 6/02, while the Snowflake model stayed current).

Required columns (used by this console): Headline, Author, Brand Type, Publication Date, Published URL/Link, Primary Keywords, Syndication platform, Format, Content Type, Rotation #.

Header aliases are documented in scripts/shared/constants.py—when the content team lead renames a column, the alias mapping keeps the pipeline stable until the next ingest.

Tarrow XLSX intake

Weekly per-platform syndication file from Tarrow (former vendor name retained as data-product label). Provides Apple News / SmartNews / Yahoo / MSN per-platform titles + PVs. MSN data is platform-side only—not in Snowflake—so Tarrow is the only path to MSN signal.

The data team's 2026-04-18 offer to load this XLSX into Snowflake as a proper data model is in flight as substrate work.

Cross-syndication distortion screen

Designed end-to-end; merge-ready when the Marfeel→Snowflake feed lands. Solves a structural data-quality issue that distorts every topic / cluster / format / author rollup the console renders.

Why it exists

On 2026-05-06, exec/leadership flagged that distribution hand-picking already-strong stories for cross-syndication "juices" apparent topic performance—his canonical example was a Field-Level-Media-syndicated piece spread across ~24 syndication targets. When the console reads total_pvs to render leaderboards + cluster aggregates, those juiced articles inflate every rollup they appear in. Operators reading the console can't tell native topic strength from syndication fan-out. The distortion is selection-on-success: syndication is downstream of strong early performance on the origin publication, not a topic-strength signal.

What changes

New columns on TRACKER_ENRICHED (fed by an upcoming MCC_PRESENTATION.CONTENT_SCALING_AGENT.MARFEEL_ARTICLE_BY_MEDIUM table built by the data engineer):

Performance-signal columns switch from total-PVs basis to origin-PVs basis at the same migration:

• is_hit—1 if origin_pvs ≥ pub_median_pvs (was total_pvs ≥)
• article_vs_co_median—uses origin_pvs instead of total_pvs
• cluster_total_pvs + cluster_hits—recomputed on origin_pvs
• cluster_vs_co_median—uses new cluster_total_pvs_trimmed with the heaviest-juiced article per cluster excluded (winsorized)
• article_pv_share_of_domain_month—origin-PVs basis
• author_hit_count + author_hit_rate—origin-PVs basis

Reach + revenue columns stay on total-PVs basis because every PV is real revenue regardless of medium: total_pvs, article_programmatic_revenue_live, author_avg_pvs, author_pv_stddev.

UI surfacing

Two new operator-facing surfaces ride the new columns:

1. Per-article juice chip on every card: 🔁 N-site, X% syndicated when syndication_juice != 'none' (orange for light, red for heavy; hover reveals the medium list).
2. Leaderboard "Hide heavy-juiced" toggle: localStorage-persisted, default off. When on, filters syndication_juice = 'heavy' rows from leaderboard rankings. Article still openable through deeplink.

What this is NOT

• Not a hide-juiced policy. Heavily-syndicated articles still appear on every card view + every leaderboard by default. The toggle is opt-in.
• Not a content-quality judgment. The chip is descriptive, not evaluative—a heavily-juiced article isn't bad content; it just shouldn't be read as topic-strength signal.
• Not a Marfeel feed dependency for correctness logic. The CTE is fail-open (defaults safely if no match): when no Marfeel row matches an article, origin_pvs falls back to total_pvs and syndication_juice defaults to none. Pre-feed-land state behaves identically to pre-screen state.
• Not a replacement for cross_site_* columns (McClatchy-internal newspaper syndication). Marfeel covers external / platform syndication targets; both apply to most articles; both useful.

Implementation

Master design + SQL diffs in ops-hub/docs/cross-syndication-screen.md. UI spec + snowflake_enrich.py diffs in data-headlines/dev-docs/cross-syndication-screen-ui.md. Governance entry in csa-content-standards data-universe-labeling.

Cron schedules

Both workflows can be triggered manually via gh workflow run or the GitHub Actions UI.

Deployment + Cloudflare Pages

Deployed to data-headlines.pierce.tools from the main branch. Auto-deploys on push (~30s). Cloudflare Pages also exposes a default *.pages.dev URL for the project; that endpoint resolves but is operational-only—always use the custom domain for stakeholder-facing links.

Access is gated by Cloudflare Access—visiting any page bounces through a one-time-PIN login (whitelisted emails only) before serving content. SPA-fallback is enabled, which serves /index.html for unknown paths—keep this in mind when adding new sub-pages (use absolute paths if the page sits at a depth where relative resolution would 404).

Anonymization governance

Per ops-hub/ANONYMIZATION.md: no proper colleague names in any committed file. Pierce is the only permitted name. Authors in tiles.json appear as initials; grader-signals.json drops the author field entirely; meeting transcripts are extracted to anonymized summaries and originals deleted.

Enforcement runs as Step 4.5 of /sync-repos—fail-closed on denylist match, transcript-filename match, verbatim-content patterns, or snapshot-directory contamination.

Quality gates

Pre-push gates from ops-hub:

• Anonymization—Step 4.5 of /sync-repos.
• Conciseness—Step 4.6: project descriptions ≤400 chars, nextActions ≤25 words, CONTEXT.md ≤150 lines.

PV / total_pvs

Page-views: total times an article was loaded across every distribution surface (Apple News, SmartNews, Yahoo, Newsbreak, search, social, direct, newsletter, AMP, subscriber). Sourced from STORY_TRAFFIC_MAIN in Snowflake and aggregated into total_pvs on TRACKER_ENRICHED.

Reach metric—fan-out included. total_pvs is the right number for reach + revenue rollups (every PV is real money regardless of medium). It is not the right number for topic-strength / cluster-strength / author-strength judgments—those use origin_pvs post-cross-syndication-screen to remove selection-on-success distortion. See Cross-syndication distortion screen for full context.

Lift

A multiplier comparing one group's median PVs to a baseline. lift = (group_median − baseline_median) ÷ baseline_median. Positive = above baseline, negative = below. The baseline depends on context: usually publication median, sometimes vertical median, sometimes untagged-formula median.

Median (vs mean)

Page-view distributions are heavy-tailed—a single viral hit can shift a mean by 5×. The console uses medians everywhere because they're robust to outliers. When a tile reports n=20 articles with median PVs of 1,200, that's the 10th-ranked article's PV count.

P-value (Mann-Whitney U)

The probability the observed lift could have arisen by chance from the baseline distribution under the null hypothesis (no difference). Computed via Mann-Whitney U-test—non-parametric, rank-based, no normality assumption.

• p < 0.05—conventional significance; treat as real signal
• p < 0.10—directional; worth a controlled pilot
• p ≥ 0.10—no detectable difference; data is too thin

Verdict (go / test / skip)

Verdict color encodes on the tile rail (left edge), pills, and KPI accents. The Verdict column pill in tile tables shows the verdict label first followed by the confidence reading: go · p<0.05, test · p<0.10, test · no diff, skip · p<0.05.

Confidence (high / mod / low)

Overlay on verdict, conveying statistical confidence:

• conf-high (cyan #38bdf8)—p<0.05, n large
• conf-mod (violet #c084fc)—directional p<0.10
• conf-low (slate #94a3b8)—anecdote / no detectable difference

Confidence range on lift (the bracket)

The bracket after a lift number—like [+22, +47]—is the likely range of the true effect: we're about 95% confident the real lift sits between those two numbers. A narrower bracket means a more reliable estimate; a wider one means more uncertainty.

It's a 95% normal-approximation interval around the lift, computed in tiles_new.py:_lift_ci_normal. It only appears when the group has at least 10 articles (and the baseline median is non-zero); below that the data is too thin to bracket, so the cell shows the lift alone. Use the bracket to tell a steady, dependable lift (tight bracket on many articles) from a fluky one (wide bracket on few).

Decile badge (D1–D10)

On top-performer rows ranked by traffic, a small badge like D8 says which tenth of its content lane (vertical) the article lands in by lift (how far it beat the publication median). D10 is the top tenth; D1 is the bottom tenth. The badge is highlighted green when an article reaches the top deciles, amber in the middle, and muted lower down—a one-glance read of "how far up its lane did this land."

Computed per vertical in tiles_new.py (lift_decile_within_vertical): the article's traffic rank within its lane, bucketed into ten. It's a within-lane ranking, so a recipes article is placed against other recipes articles, not against politics. It shows only on the PV ranking axis of the recent top-performers panel.

Engagement metrics (engaged % · time on page · cluster continuation)

Beyond raw clicks, three reader-behavior measures say whether a headline actually delivered. They're now the primary read on the top-performers panel; page-view trend stays as the simple traffic-only reference.

• Engaged % (engaged_session_rate)—the share of visits where readers engaged rather than bounced.
• Time on page (median_time_on_page_seconds)—how long readers stayed, as a median (one marathon reader doesn't distort it).
• Cluster continuation (csa_cluster_continuation_rate)—the share of readers who clicked on to another linked article in the same group. 30%+ is strong, 15–30% middling, under 15% weak.

These are Amplitude/Marfeel-derived and land on each article via the Wing-B engagement substrate plus the recurring Marfeel ETL—live and longitudinally reliable from 2026-05-18 forward. They're NULL when an article has no engagement coverage (it predates the window, or wasn't tracked), so the panel skips an article on any axis it's missing rather than show a misleading zero. The "beat the publication's normal level" standard used by the hit-rate-by-formula sidecar uses these measures where they exist and falls back to plain traffic where they don't.

Cluster batting average

Article-level: of all CSA-touched articles (those with a cluster_id), the share whose cluster's median PV beat the publication's company median (cluster_vs_co_median > 0). Reported as "1 in N"—lower N = better.

Larger clusters get proportionally more votes (each article in a winning cluster counts), so this reads as "how often does a CSA-touched article sit in a cluster that's outperforming."

Current batting average sits in the 1-in-2.5 to 1-in-3 range. The exec/leadership Q3 target is 1 in 3.3 (≈30%).

Methodology shift coming. Post-cross-syndication-screen, cluster_vs_co_median uses cluster_total_pvs_trimmed (heaviest-juiced article per cluster excluded) computed against origin_pvs per article. Expect cluster batting averages to shift downward modestly as previously-juiced clusters revert to native strength. See Cross-syndication distortion screen.

Curiosity gap

Editorial principle: lead with the question, not the answer. "The Secret to Her 20-lb Weight Loss" (gap kept open → click) outperforms "She Lost 20-lbs Eating Kale" (answer spoiled → no click). Active in the L&E lens.

Confirmed at US Weekly with n=19 vs n=12 control, +42% vs −35% lift, p<0.05.

Formula taxonomy (full list)

Canonical list lives in formula_taxonomy.py at repo root — single source of truth shared by the Findings classifier (tiles_new.py:classify_formula) and the published-headline grader (generate_grader.py:_formula). The variant tool's Cloudflare Worker (data-headlines-variant, separate codebase) is synced manually against this list.

Priority-ordered: penalized-filler phrases first (so they bucket correctly even when a credit-worthy pattern also matches), then lead-position patterns, then structural cues, then stylistic markers, then a length-based fallback. First match wins. Every non-empty headline buckets — untagged is reserved for the empty-string degenerate case only.

Role column: credit = legitimate editorial formula, earns "Formula present" credit in the grader, classifier buckets here, variant Worker generates candidates here. penalized = recognized vague-filler pattern, classifier buckets it for cohort analysis but the grader does NOT award credit (the per-criterion penalty fires separately). fallback = length-based classifier-only bucket; doesn't represent an editorial formula.

Penalized-filler phrases (priority 1)

These bucket FIRST so a headline like "X: Everything You Need to Know" doesn't accidentally credit on a stylistic marker further down the list.

Lead-position patterns (priority 2)

Numeric + listicle (priority 3)

list_with_quantifier comes before number_lead so "5 Things to Know" buckets as the more specific listicle pattern rather than the bare number-lead bucket.

Structural cues (priority 4)

Stylistic markers (priority 5)

Fire often, less differentiating. Last in priority so they only catch headlines that don't trigger any more specific editorial pattern above.

Length-based fallback (classifier only)

Headlines that don't trigger any pattern above bucket by word count so per-pub × formula groups still populate. These are NOT editorial formulas — the grader returns "No recognized formula" (no credit) for headlines that fall through to this layer.

Topic taxonomy

Regex-based classifier in tiles_new.py:classify_topic(). Priority-ordered: specific patterns fire first; if no rule matches and Snowflake has an IAB tag, that's used; otherwise a token-cue fallback buckets to wellness_general / lifestyle_general / news_general. untagged is reserved for headlines with zero topical cue words (rare in practice).

Traditional news topics

weather_severe, weather_heatwave, weather_general, politics, crime, sports_local, sports_national, royals, finance.

TH wellness/lifestyle topics

Tuned to the Trend Hunter content corpus that dominates today's dataset:

• wellness_menopause — menopause, perimenopause, hot flashes, HRT, estrogen
• wellness_sleep — sleep, insomnia, circadian, melatonin, naps
• wellness_retreat — retreats, spas, wellness boot camps, sanctuaries
• wellness_stress — stress, anxiety, breathwork, vagus nerve, cortisol, cold plunge, meditation
• wellness_beauty — skincare, beef tallow, wrinkles, botox, red light therapy
• wellness_fitness — run clubs, workouts, pilates, sarcopenia, gym
• wellness_nutrition — protein, electrolytes, hydration, gut health, loaded water, collagen
• celebrity_health — weight loss, Ozempic, surgery, cancer
• home_hazards — mold, microplastics, PFAS, forever chemicals, laundry detergent, tap water
• home_garden — garden, flowers, curb appeal, landscaping, lawn
• real_estate — home inspection, home buying, mortgage, housing market
• recipes — recipes, baking, meal prep, dinner/breakfast
• entertainment — movies, TV shows, Netflix, streaming
• celebrity_general — Kardashian, Jenner, Hollywood stars (when not under a wellness bucket)
• lifestyle_trend — viral, TikTok, trend, Gen Z, "explainer"
• travel_food — travel, destinations, food tours, restaurants

Fallback buckets

wellness_general / lifestyle_general / news_general — fire when a headline contains a general cue word (wellness/health/luxury/trend/report/etc.) but no specific topic pattern matched. These keep per-pub × topic groups populated for analysis even when a headline doesn't fit a named bucket.

Content type (News vs L&E)

Verticals (the 3)

• Mind/Body (color: emerald --vert-mb)—health, fitness, wellness
• Everyday Living (color: amber-orange --vert-el)—home, décor, lifestyle, parenting
• Experiences (color: pink --vert-ex)—travel, recipes, finance, retirement

These are the three named TH-channel verticals. Articles published outside these channels (search/discover work) are bucketed as — in vertical-keyed UI; they aren't a fourth vertical.

Writer initials

Author identifier in committed JSON. Computed from full author name: first letter of first word + first letter of last word. "Jane Doe" → JD. "Alex Roe-Smith" → AR. Collisions get a digit suffix (JD, JD2, …).

Live Tracker sheet uses canonical full AUTHOR field; initials are a render-layer transformation only.

Grader 19 criteria (+ tier weights)

Defined once in generate_grader.py:CRITERIA. A headline's score = (weight of rules it passed) ÷ (weight of rules that applied to it) × 100. Rules with weight 0 are informational (shown, never scored); rules that return "N/A" for a given headline drop out of its denominator. The four tiers carry 27 / 27 / 36 / 9% of the total weight (see Grader tiers + the score).

Weight 0 = informational (shown but not scored)—number and apple_heres, because whether they help depends on which platform the title goes to, and the grader sees only the website headline. The two platform-title rules (an_title_chars, sn_title_chars) would grade the custom Apple News / SmartNews titles, but those live in the Tarrow file that isn't joined to the grader yet, so they read "?" (N/A) for almost every headline and stay out of the score in practice. All 19 grade the H1 from the Tracker, never the platform-specific syndication titles.

Expert/study signal (proven lever)

When an article is built on an expert or a study—a credentialed person, researcher, or doctor, or named research—putting that signal in the headline reliably lifts performance. The June 2026 headline analysis measured this as the single highest-impact finding (roughly +71% page-views and +50% engagement time), which is why it's an enforced, scored grader rule rather than just a tip.

How the rule behaves: it fails only when the source clearly has expert/study material but the headline leaves it out; it passes when the headline surfaces it. It shows "?" (not scored) when the story has no expert/study material to surface, when the brand is US Weekly, Woman's World, or Life & Style (where the rule doesn't apply), or when the tool can't confirm the expert from the article body.

Anti-fabrication is structural: the signal must come from the article body—the writer has done the research. The tool never invents an expert or study the body doesn't support. To pass: name the expert or study in the headline when the story actually has one.

Headline letter grades

Score → letter grade mapping for the visual grade pill:

Grade trends per author are tracked in docs/grader/history.json (30-day rolling window).

Anonymization in practice

Per ops-hub/ANONYMIZATION.md, no proper colleague names appear in any committed file (Pierce is the sole exception).

Where it shows up

• docs/data/tiles.json · team_perf.writers[].initials—first letter of first name + first letter of last name (collisions get a digit suffix). No author field at all.
• docs/data/grader-signals.json · per-headline objects have vertical but no author.
• docs/grader/index.html · the _author_initials() helper renders authors as initials in every hcard. The full name is only used in-process for the AUTHOR_VERTICAL lookup, never serialized.

Mapping back to actual names

The live Tracker sheet has the canonical AUTHOR field. Operators with sheet access can dereference initials by sorting on AUTHOR + matching to the displayed pub-percentile. Initials map is not published.

Enforcement

Pre-push gate: /sync-repos Step 4.5 invokes /anonymization which scans every committed file against the 55-name denylist + transcript-filename patterns + verbatim-content patterns. Fail-closed.

Variant tool · what it is

A live editor-in-the-loop tool at /variant/ that takes one headline + its primary keywords (and optionally the article body) and returns 5–10 publication-ready alternative phrasings, plus one structurally-novel ⚡ wildcard for the long-term cohort study.

The tool answers a different question than Findings or Grader. Findings tells you what shape worked across thousands of past articles. Grader scores one headline. Variant generates options for an article you're about to publish—every option verified against the same production grader, every option containing every primary keyword you supplied, every option diverse enough from the seed that selecting one is a real editorial decision.

Who uses it. Content team writers + the content team lead. Each user gets a cap of 10 generations/day, 200/month, with the team sharing a ~700-runs/month budget ceiling, and the tool runs weekday-only (Mon–Fri, Eastern; closed weekends). Pierce is on an unlimited exception list because he funds the underlying API.

Why it exists. Variants offered for a single article were running lexically too similar—sibling bridges in a cluster looked interchangeable. The variant tool is the editorial-side fix: forced facet diversity across an option set when the article body provides article-supported latitude, plus a long-term study aimed at promoting the most-picked off-pattern wildcard structures into the standard format library.

For the full operator reference—form fields, the style test, the expert lever, reading the result cards, rate limits, troubleshooting—see the Variant tool documentation.

The headline experiment in the Variant tool

The headline experiment (trend-stage framing test + the proven expert/study lever) runs inside the Variant tool. Here's where it sits in this tool's architecture; the writer-facing controls and the A/B report are in the Variant tool documentation.

Three rule homes, three change cadences

The experiment is built on a clean split between what's settled, what's under test, and the record that ties a result back to its setup:

The decision rule: a rule belongs in the settled/scored layer once an engagement winner has been picked and it's being enforced; it stays in the volatile database layer while it's still a hypothesis being injected and validated-but-not-scored. Scored ⇒ settled; under-test ⇒ database.

How it grades

• Expert signal (proven) grades like any scored rule, using the same "score it when the story supplies an expert/study, otherwise N/A" gate the keyword rule already uses.
• Framing (hypothesis) runs through a conformance check that confirms a test headline actually follows its assigned style, but never enters the score—so an unproven framing can't tank a production headline's grade or block generation.

Both the expert directive and the assigned framing ride the existing generation call—no extra LLM round-trips, no added cost. All the schema is additive and every new read fails open: a missing row falls back to bundled defaults exactly like the calibration read, so the experiment can never take the production tool down.

Quickstart · 60 seconds

1. Paste the seed headline you want alternatives for. The tool produces rephrasings of the SAME article—not pitches for a different one.
2. List the primary keywords (load-bearing SEO terms, comma-separated). Every variant the tool offers will contain ALL of them. Strict editorial rule.
3. Optional but recommended: paste the full article body. The tool switches to "maximize difference" mode—pairwise diversity gates kick in, and the grader judges accuracy against the article instead of the headline alone. Without article body, the grader stays conservative.
4. Click Generate variants. Wait 5–15 seconds. Each card shows score + similarity + structural breakdown.
5. Keep the variant(s) you'd ship—the button toggles to Kept ✓; multi-pick is supported and kept state survives regeneration of the same seed (persists server-side for performance analysis). Or Copy to clipboard, or ↻ Regenerate for a fresh batch with the same inputs.

The form fields

Default vs article-mode

The tool runs in one of two regimes depending on whether the article body field is filled. The mode pill in the results header tells you which is active.

Article-mode is what content team should reach for when she has the article in hand. Without it, the tool is honest about its narrower latitude and shows the conservative regime in the mode pill.

The wildcard slot

Every /generate returns exactly one variant tagged ⚡ wildcard. The wildcard is generated under an off-pattern prompt—its job is structural novelty, not faithful rephrasing.

What's different about a wildcard:

• Bypasses the rule-following LLM grader (active voice / lead burial / curiosity). Those criteria are calibrated for ordinary variants and would fight the wildcard's role.
• Still passes every hard structural rule at 100% (length, named-entity lead, formula, primary keywords, banned patterns)—wildcards are publishable shape, not garbage.
• Single accuracy safety-net check via a one-criterion LLM call—catches wildcards that introduce fabricated claims (e.g. "experts say" when no experts are in the source).
• Has its own cosine ceiling (0.95 vs seed) regardless of mode—the cohort study values structural fingerprint novelty, not semantic distance (meaning-difference), so a wildcard at 0.95 cosine with a structurally novel POS (part-of-speech) sequence is exactly what the study wants.

Verify wildcards editorially before publishing—they bypass the LLM judgment criteria by design. The chip + tooltip on each wildcard card discloses this.

The wildcard cohort feeds the format-promotion ledger—recurring wildcard structures that the editor consistently picks become candidate new headline standard formats.

Score floor · what's guaranteed

The grader is a port of the same criteria registry in generate_grader.py that runs daily on published headlines (kept in lock-step by a parity test). The variant tool applies it differently to each cohort.

Hard structural rules · 100% pass on every variant (rule-following + wildcard)

• Length · distinct numbers, not one band. The hard floor is 75 chars—a variant-tool runtime override (VARIANT_CHAR_RANGE.min in index.js) that permits tighter phrasings; it sits below the daily grader's publish floor. The scoring target band is 90–104 (target_min/target_max, matching the daily grader since 2026-06-05). Above target there's an 11-char editorial over-tolerance (VARIANT_CHAR_RANGE.over_tol, added 2026-06-16): options 105–115 chars soft-pass carrying an amber ⚑ editorial-judgment flag instead of dropping, so writers get a fuller set and decide whether to trim; only <75 or >115 drop. This tolerance is variant-tool-only—the daily grader carries no over_tol and still hard-fails any headline over target. So a 75–89-char variant clears the floor and is publishable but scores below one in 90–104; a 105–115-char variant passes flagged. (Rationale: keeps a compressed or slightly-long phrasing from being dropped outright when a fully-compliant set is hard to reach.)
• Named-entity lead · no "The / A / An / It / This / There / He / She / They / Here" lead
• Formula present · matches at least one credit-worthy editorial formula (about two dozen, e.g. Number lead · "Here's" · Possessive · Colon-structured · Em-dash · How / Why / Inside leads · Action-verb · List-with-quantifier · Comparative · and more—see the full formula taxonomy)
• Primary keywords · every primary keyword the operator supplied appears in the variant
• Banned patterns · no question-mark ends · no "What to Know" / "Did you miss" · no all-caps non-acronyms

LLM-judged criteria · ≥85 composite score (rule-following only)

Four binary criteria graded by an LLM: active_voice, no_lead_burial, curiosity, accurate. Composite score weights each—failing one weight-1 criterion = 94 (passes), failing one weight-2 criterion = 88 (passes), failing two = ≤82 (fails). The 85 floor tolerates a single borderline LLM judgment-call miss but rejects combinations or multiple substantive misses.

Wildcards · one-criterion accuracy safety net

Wildcards skip the 4-criterion LLM grader entirely. Instead a single LLM call judges accuracy alone (does the wildcard introduce a falsifiable claim the source contradicts or doesn't cover?). Default PASS. Catches fabrications without fighting the structural-novelty objective.

Active gates · cosine + grader + diversity

If the pool falls below the requested N after all gates, the pipeline runs up to 2 regeneration passes with explicit failure-mode feedback (e.g. "char_count under 80 in 4 variants—extend with article specifics"). After 2 passes, returns whatever's in the pool with a recovery-advice payload.

Quotas + caps

Four guardrails. A weekday-only window (Mon–Fri ET), per-user daily, per-user monthly, and the team-wide budget. Atomic D1 counters—concurrent calls cannot over-shoot. Visible to the user as pills above the form.

UI pills: "today X/Y" · "month X/Y" · "team ~X/Y runs left"—pills turn amber as you approach a cap, red when reached. The team pill converts dollars to estimated runs at the current per-call cost so users see a single shared ceiling in the units they care about.

Unlimited-users exception

The UNLIMITED_USER_EMAILS env var (comma-separated emails) bypasses every cap—the per-user daily and monthly limits and the weekend gate—for listed users. Calls still record into the counter tables and accrue cost for transparency in the usage log—they just never get blocked. Pierce is on this list because he funds the underlying API.

To tune the [vars] block: edit workers/variant/wrangler.toml [vars] (RATE_DAILY · RATE_MONTHLY · BUDGET_MONTHLY_USD · BUDGET_ALERT_FRACTION · EST_COST_PER_CALL_USD) and wrangler deploy.

UNLIMITED_USER_EMAILS lives as a Worker secret (not [vars]) because it lists colleague email addresses—committing those would violate anonymization. Update via echo '$LIST' | wrangler secret put UNLIMITED_USER_EMAILS; see DEPLOY.md §4 for rotation procedure.

If 0 variants returned

The tool tries up to 2 regeneration passes before giving up. If the pool is still empty (or short of N), the response carries a recovery_advice array—specific actionable guidance based on the dominant failure mode that pass.

The UI surfaces it as an amber-bordered "what to try" panel below the (empty) results, with a ↻ Regenerate button next to it. Examples of what the panel says depending on the failure pattern:

• "Most candidates were rejected as too similar to your seed (cosine gate). Try with article body to unlock more varied phrasings."
• "Most candidates failed the rule-based grader—your seed may be too short to extend into the 90–104 char target or contain banned patterns. Try lengthening the seed."
• "You're requiring all 4 primary keywords in every variant—that's a strict bar. Move the least-essential keyword to 'secondary keywords' to widen the pool."
• "Pairwise diversity rejected several candidates—your topic may have only a couple of viable facets. Try generating fewer (e.g., n=5) or remove an 'also avoid' entry."

Form state persists in sessionStorage—the operator can adjust inputs and regenerate without re-pasting everything.

My usage log

Every authenticated page load fetches the user's own usage from D1 (server-side, never localStorage) and populates the pills + the collapsible "My usage log" panel below them.

Personal section (CF-Access-authenticated email is the filter—users see only their own activity):

• Per-day breakdown · last 30 days (calls / variants returned / cost in $)
• Recent generations · last 30 (timestamp, headline preview, returned_n / requested_n, mode, request_id for forensics)

Team-wide visibility in the quota pills above the form: today's team total + this month's spend vs budget are surfaced as part of the standard quota line so concurrent operators see capacity without a separate panel. Per-user monthly breakdown is available via GET /metrics/quota-by-user if needed for ops review; not currently rendered in the UI.

Refreshes on every page load. The cap-enforcement counters are atomic in D1; the read endpoint is purely visibility—concurrent /generate calls cannot over-shoot the limits regardless of what this panel reports.

Architecture · Worker + D1 + Workers AI

Cloudflare Worker at workers/variant/ backed by D1 (shared with workers/history) + Cloudflare Workers AI (embeddings) + Anthropic API (generation + grading). Static page at docs/variant/.

Model dispatch is by env-var prefix: any model name starting with @cf/ routes to Workers AI via env.AI.run (free, account-bound); anything else (e.g. claude-sonnet-4-6) routes to Anthropic via fetch. Workers AI Llama 3.3 70B is a tested fallback for cost-down—quality is meaningfully worse on the rule-following generator (more clichés, weaker prompt adherence) but acceptable for the LLM grader role if needed.

Pipeline · stages in order

One POST /generate call traverses these stages. Up to 2 regeneration passes if the pool falls short.

1. Validate body—headline ≥20 chars, primary keywords required, article ≤50,000 chars, format/persona/platform/site allowlisted, prompt-injection regex screen on the article body.
2. Idempotency check—if client_request_id matches a recent (≤5min) generation for the same email, return its generation_id without re-running the pipeline.
3. Rate-limit gate—read D1 counters for daily/monthly/budget. Skip caps if email is in UNLIMITED_USER_EMAILS.
4. Anonymization scrub—load denylist from the DENYLIST_JSON Cloudflare Secret (cached per-isolate). Replace any matched name with […] in headline, article, primary/secondary keywords, also-avoid entries.
5. Generate · rule-following + wildcard in parallel via callRuleFollowing + callWildcard. Retry once on transient (429/5xx) Anthropic + Workers AI failures.
6. Embed · single Workers AI batch call · seed + each also-avoid + each candidate variant. On second-attempt failure, degrade to cosine=0 (rule-based grader still gates).
7. Cosine gate · per-cohort ceilings (rule-following: 0.97 default / 0.93 article; wildcard: 0.95). Track every dropped candidate in allOfferedCandidates with dropped_at_gate='cosine'.
8. Rule-based grader pre-check · char_count, formula, named-entity lead, primary keywords, banned patterns. Drop on any non-LLM rule failure with dropped_at_gate='grader_rule'.
9. LLM grader · batched call for rule-following candidates only · 4 criteria. Wildcards skip this stage. On grader failure, degrade to rule-based-only score (LLM criteria null) instead of 502.
10. Compose graded set · rule-following with merged LLM results (drop sub-85) + wildcards with rule-based-only score + accuracy safety-net call.
11. Set-level diversity · formula cap (≤2 per formula). Article-mode adds pairwise cosine ≤0.85 + pairwise Jaccard ≤0.45 between rule-following variants.
12. Regen loop · if pool < requestedN and passes < 2, re-prompt the LLM with explicit failure-mode feedback (which gate dropped the most variants this pass).
13. Persist · D1 batch—generation row + every offered candidate (rule + wildcard, kept + dropped, with dropped_at_gate annotation) + back-compat write to legacy variant_offered_wildcards. Atomic.
14. Record cost · atomic D1 counter increment (always—failed pipelines still count for DoS protection). 80%-budget Slack alert fires once per crossing.
15. Return · variants + pipeline stats + quota snapshot + recovery_advice if pool short.

Model assignments

All three LLM roles are env-var-driven. Edit workers/variant/wrangler.toml [vars] block + wrangler deploy to swap any role. Dispatch by prefix: @cf/... → Workers AI (free), anything else → Anthropic.

Cost model. ~$0.06–$0.10 per /generate call on the current Sonnet+Sonnet+Haiku stack. Recent observed range $0.057–$0.099 (varies with article-body length + regen passes).

D1 schema · 18 tables + 1 view

Single D1 database (data-headlines-history) shared with workers/history. Migrations 0001–0014 applied; current schema_version is 14. Migration 0004 added the kept toggle (unkept_at + seed_fingerprint + v_active_kept view); 0006/0007/0008 added partial unique indexes; 0009 lowercased legacy user_email values for case-consistent matching; 0010 added embed_degraded sentinel column on offered_candidates + offered_wildcards and widened the offered_candidates UNIQUE to (generation_id, cohort, variant_text); 0011 widened the variant_selections UNIQUE to match (cohort included) and added partial UNIQUE(is_active=1) on rules_versions + prompts; 0012 added the variant_jobs async-job queue (no version stamp—table-only, so MAX(version) jumps 12→13 across the 0012/0013 pair); 0013 tightened the article-mode pairwise gates; 0014 added the headline-experiment data layer (variant_rule_sets + variant_config_specs, plus rule_set_id/spec_hash/headline_type columns on generations + offered_candidates).

Portability. Every *_at epoch_ms field has an *_iso mirror column populated at write time. Indexes cover every filter/group-by surface (calibration, prompt, model, rules SHA, cluster_key, total_pvs, publish_date, etc.). Foreign keys declared; PRAGMA foreign_keys = ON set per-request in the worker.

Versioning · calibrations / prompts / models / rules

Every variant_generations row carries the full reproducibility tuple—what numeric calibration was active, which exact prompt template was sent, which model handled each role, which csa-content-standards SHA the rule corpus came from, which denylist version was loaded.

Why this matters. The variant tool will be tuned over time. Score floors will move, prompts will be loosened or tightened, models will be swapped. A/B comparisons across versions become trivial SQL: WHERE calibration_id = X vs Y. Rolling improvements are measurable rather than asserted.

How a new calibration ships

1. Edit constants in workers/variant/src/index.js (or wrangler.toml env vars for the LLM models).
2. Run wrangler deploy.
3. On next request, ensureMetadata() reads the active variant_calibrations row. If the new constants don't match, INSERT a new calibration row + deprecate the prior. Worker stamps the new ID onto subsequent generations.
4. Compare A/B over a few days: SELECT calibration_id, AVG(returned_n), AVG(est_cost_usd), AVG(rejected_grader) FROM variant_generations WHERE generated_at > ? GROUP BY calibration_id.

How a new prompt ships

Edit the prompt template in workers/variant/src/llm.js. Worker computes SHA-256 of the system prompt at request time. INSERT-OR-IGNORE into variant_prompts (idempotent on hash). Subsequent generations reference the new prompt_id; A/B comparison via WHERE prompt_rule_id = X.

Cohort study + format-promotion ledger

The wildcard cohort is the long-term feedstock for surfacing new headline standard formats—recurring wildcard structures that the operator consistently picks (and that perform once published) become candidates for promotion to the rule library.

The pipeline

1. Collect · every offered candidate persists with structural fingerprint (cluster_key = formula × leading_pos × first 3 POS tokens). Both kept + dropped wildcards land in variant_offered_candidates.
2. Cluster · scripts/analyze_variant_cohort.py groups wildcards by cluster_key. Per cluster: count offered, count picked, pick rate, performance lift if outcomes joined. Aborts when /export/* is truncated at 5000 rows so reports are never silently biased.
3. Surface candidates · clusters with significantly higher pick rate or PV lift than the rule-following baseline land in variant_format_candidates with status='surfaced'.
4. Review · operator (Pierce + content team lead) reviews surfaced candidates. Status moves to under_review → promoted / rejected.
5. Promote · promoted clusters are codified into csa-content-standards as new headline format guidelines. The promotion event records the csa-content-standards SHA at promotion time, so a reviewer can diff exactly what landed in the standards docs because of this candidate.
6. Audit · variant_format_events append-only log captures every status transition (who promoted what when, against which standards SHA).

What's shipped vs planned

• Shipped: migrations 0003-0014 schema (variant_offered_candidates · variant_format_candidates · variant_format_events · v_active_kept · UNIQUE indexes · lowercased user_email · embed_degraded sentinel · cohort-aware UNIQUE on offered_candidates + variant_selections · partial UNIQUE(is_active=1) on rules_versions + prompts · the variant_jobs async queue · the headline-experiment data layer variant_rule_sets + variant_config_specs with per-call/per-headline attribution columns). Worker writes every offered candidate's fingerprint, including diversity-dropped candidates. scripts/analyze_variant_cohort.py computes cluster pick-rates + bootstrap CIs + edit-distance tail; aborts cleanly on /export/* truncation rather than silently bias; non-zero-baseline guards on candidate-format surfacing; isinstance NaN guards on CI bounds; html.escape on operator-supplied report content.
• Planned: weekly cron to refresh format-candidate rankings; a UI surface for reviewing surfaced candidates; promotion-to-standards integration.

Anonymization · DENYLIST_JSON

The variant tool reads the colleague-name denylist from a Cloudflare Secret (DENYLIST_JSON) at request time—never bundled into the build artifact, never committed to git. Per-isolate cache so we re-parse the secret only on cold start.

Format:

{"version":"2026-05-07","names":["<colleague-1>","<colleague-2>","..."]}

(Legacy: a bare JSON array of strings is also accepted.)

Where it scrubs:

• Pre-call · headline, article body, primary + secondary keywords, also-avoid entries—all scrubbed before they reach the LLM. Names replaced with […] placeholders.
• Post-call · LLM outputs scrubbed before they reach the user (defense in depth—the LLM shouldn't introduce new colleague names but if it echoes one that survived input scrub via partial match, this catches it).
• Hard reject · any candidate variant containing a denylist name post-scrub is dropped from the pool entirely (treated as a denylist failure).

The denylist version stamps onto every variant_generations row (denylist_version column) for reproducibility—if a name is added or removed from the denylist, the version stamp lets you scope queries to before/after the change.

Pierce is the only permitted name across all repos; the denylist excludes him. To set / update:

cd workers/variant && wrangler secret put DENYLIST_JSON < /tmp/denylist.json

Deployment · secrets + migrations

Detail in workers/variant/DEPLOY.md. Summary here.

One-time setup

1. Apply migrations to remote D1 in order, 0001 → 0014 (0001_variant_tables.sql … 0011_selections_cohort_unique_and_active_partials.sql → 0012_async_jobs.sql → 0013_tighten_article_pairwise_gates.sql → 0014_headline_experimentation.sql). Or use npm run migrate:all from workers/variant/. Then run the one-shot seed 0014_seed_phase1_rule_set.sql ONCE after migrating (it's a deliberate operator step, NOT in the migrate chain; idempotent via WHERE NOT EXISTS). Confirm the schema_version: 14 stamp via /healthz.
2. Set worker secrets: WRITE_TOKEN (32-hex shared bearer for /generate /select /outcome), ANTHROPIC_API_KEY (McClatchy-billed), DENYLIST_JSON (anonymization), UNLIMITED_USER_EMAILS (comma-separated emails that bypass quota; secret because it lists colleague addresses), SLACK_WEBHOOK_URL (optional, for 80%-budget alerts).
3. Bundle the rule corpus + denylist before first deploy: python3 scripts/sync_rules.py. (DENYLIST_JSON is runtime-loaded; only RULES.js is bundled.)
4. Add /variant/ to the existing CF Access app on data-headlines.pierce.tools (already covered by the data-headlines.pierce.tools Access policy by default).

Per-deploy

cd workers/variant
python3 scripts/sync_rules.py       # refresh RULES.js from csa-content-standards
wrangler deploy                     # ship Worker
curl -s https://data-headlines-variant.pierce-williams.workers.dev/healthz | jq

The /healthz response surfaces: schema_version, bindings (ai/db/anthropic_key—the legacy KV binding was retired in migration 0003), active models, rules_loaded, denylist configuration + version, active calibration row. /healthz/deep additionally pings Workers AI for a live latency check. All other GET endpoints require X-Auth-Token.

API endpoints

Troubleshooting

Variant-specific glossary

Cohort

Either rule_following (n−1 publishable variants per call, full grader floor) or wildcard (1 off-pattern variant per call, hard rules + accuracy safety-net only). Stored on variant_offered_candidates.cohort.

Cluster key

Compact structural fingerprint computed from formula × leading_pos × first 3 POS tokens (e.g. 'Possessive | noun | NN-NN-VB'). Used as the cohort study's primary GROUP BY for surfacing recurring wildcard structures.

Calibration

A bundled snapshot of every numeric pipeline knob (score_floor · cosine gates · pairwise gates · char range · regen passes). Stored as a row in variant_calibrations; the active row's id stamps onto every generation.

Prompt hash

SHA-256 hex of a system prompt template at the moment of dispatch. Stored in variant_prompts with the full template text. Lets you A/B prompt revisions without losing history.

Article-mode

Pipeline regime when the operator pastes the article body. Activates pairwise diversity gates between rule-following variants (cosine ≤ 0.85, Jaccard ≤ 0.45) and switches the LLM grader's accuracy criterion to "judge against the article" (more permissive). Mode pill in the results header tells the operator which regime is active.

Recovery advice

Specific actionable guidance appended to /generate responses when the pool is short of N. Surfaced in the UI as an amber-bordered panel with concrete advice ("move keyword X to secondary," "try without an article body," etc.) and a regenerate button.

Idempotency replay

If client_request_id matches a prior generation by the same email within 5 minutes, the worker returns the prior generation_id without re-running the pipeline. Prevents double-billing on transient client retries.

Unlimited-users exception

Comma-separated emails in the UNLIMITED_USER_EMAILS env var. Listed users bypass all daily/monthly/team-budget caps. Calls still record into the counter tables for transparency. Pierce is on this list (he funds the underlying API).