Monthly Copilot round-up — QA system playbook¶

Read this before building any month's "What's New in Microsoft 365 Copilot" issue. Companion to whats-new-copilot-pack-playbook.md (the PPTX/carousel pack) and copilot-monthly-blog-screenshots-playbook.md (screenshot capture + placement). This doc covers the QA system — the deterministic checks that replaced eyeballing.

Created 21 Aug 2026, after the August issue (59 sections · 60 images · ~15,200 words).

TL;DR — what to run¶

cd C:\ssClawy\aguidetocloud-revamp
$env:PYTHONIOENCODING = 'utf-8'

python scripts\monthly-blog-qa.py lint                      # every issue, offline, ~1s
python scripts\monthly-blog-qa.py lint --post september     # just this month's
python scripts\monthly-blog-qa.py images manifest --post september
python scripts\monthly-blog-qa.py crosscheck --post september   # prose numbers vs observation
python scripts\monthly-blog-qa.py links --post september    # network; before draft: false
python scripts\monthly-blog-qa.py audit --post september --write-receipt
python scripts\monthly-blog-qa.py verify-receipt --all      # every published issue
python scripts\monthly-blog-qa.py annotations --all         # every image classified, from 2026-09
python scripts\monthly-blog-qa.test.py                      # the guard's own tests

npm run qa:monthly-copilot, :test and :annotations are the registered aliases. An unregistered script is not a mechanism (Rule #14b) — that is why they exist.

Why this exists¶

The August 2026 issue was reviewed by hand, twice, with an independent GPT model. Three defect classes still got through to the final pass — all of them the quiet kind that reads perfectly well:

Defect found by hand	Why no existing guard saw it
A roadmap ID cited with no matching entry in Microsoft's feed	`check-blog-html.mjs` checks HTML structure, not citation truth
A heading whose `For:` date contradicted its own screenshot	Nothing compares prose to pixels
Images referenced but never actually eyeballed	Rule #8 was prose-only; prose rules don't execute

Worse, the human side was unreliable in a measurable way: my own false-positive rate on "missing roadmap ID" claims during that session was 8 out of 8. Every single flagged gap turned out to be wrong on inspection. That is the argument for determinism — not that people are careless, but that this particular check is one humans are bad at and machines are perfect at.

The invariants, and what each one cost to learn¶

lint is offline and fast because it is the only part wired into the real git pre-push hook. Anything slow or network-bound in a push gate teaches people to reach for --no-verify, which kills the mechanism entirely.

Check	Severity	Provenance
Zero sections parsed	error, never a pass	Headings are `##` on standalone issues and `###` on grouped ones. A parser assuming one form reports zero sections for the other and looks like a clean run
Duplicate / gapped section numbers	error	April 2026 numbers its connector table 26–33, colliding with heading 29 and skipping 34
Missing `For:` line	error	The audience/timing convention was adopted partway through the series
Missing source line / source line with no URL	error	March §27 honestly states no roadmap ID exists — the only sourceless line in the corpus
Malformed roadmap ID	error	Defensive; effectively unreachable, since extraction only matches `\d{6}`
Roadmap ID absent from the feed	error, excusable	Added 21 Aug 2026 — see below
Link label ID ≠ its own href ID	error	Added 21 Aug 2026 — a link reading "Roadmap 558938" pointing at 555894 is invisible to every human reviewer
Image missing on disk	error	—
Image with empty `alt`	error	28 broken alts once shipped to production; Hugo's minifier masked them

The roadmap-ID-exists check, and why it is excusable¶

The obvious design is a hard error. I probed before building it (Rule #6) and the data said otherwise:

220 roadmap IDs cited across the eight 2026 issues
3 are absent from the live feed — 531759 (twice) and 660076
531759 is cited independently in both the May and July issues for the same named feature, which is what rules out a typo

So Microsoft genuinely withdraws roadmap items, and a naive hard error would have blocked pushes on two already-published posts on day one. Equally, the absence is not systematic decay: survival by post shows no age gradient (January 100%, May 95.6%, July 93.3%) and the feed still retains 872 Launched items back to CY2024. So the exception list stays small and honest.

The resolution: a hard error unless documented in qa/monthly-copilot/exceptions.json with a code, the section, and why. A typo blocks. A genuine withdrawal is recorded once, in writing, and stops nagging.

⚠️ Never add an exception to silence a problem in the issue you are currently writing. Fix the post. That file is for history that is already live.

The false positive worth remembering¶

The first draft of the label-vs-href check reported 4 failures. All four were wrong. The real source lines look like this:

📖 [Roadmap 558938](…searchterms=558938) · [558934](…searchterms=558934) · [559480](…searchterms=559480)

Only the first label spells out "Roadmap"; the rest are bare numbers. Comparing label-IDs to URL-IDs per line therefore saw {558938} vs {558934,558938,559480} and cried wolf. The fix was to compare per link — which is the actual semantic of the defect — and to ignore links whose href isn't a roadmap URL.

That shape is now regression-locked in monthly-blog-qa.test.py. If someone "simplifies" the check back to per-line, a test fails by name.

The general lesson: a new check's first run is not evidence it works. It is a hypothesis. Inspect what it flags before believing either the check or the content.

Where the gates live¶

Gate	Runs	Fails
`scripts/pre-push-hook.ps1`	on `git push`, ~1.1 s	closed on defects, open when Python is absent
`.github/workflows/monthly-copilot-qa.yml`	on push/PR touching an issue, the tool, exceptions, content images, or the roadmap feed	closed, unconditionally — this is the backstop for the hook's fail-open
`scripts/monthly-blog-qa.test.py`	whenever the tool's own logic is in the push, and in CI	closed
`scripts/monthly-blog-qa.mutate.py`	CI only (~13 s, too slow for the hook)	closed — a MISS means a guard is decorative
`annotations --all`	in the push hook beside the receipt, and again in CI (~0.2 s)	closed, from the 2026-09 issue onward

🔴 CI is a backstop, not a publication gate. deploy.yml hands off to Cloudflare Pages on push — by the time the workflow runs, the post is already public. The pre-push hook is therefore the only genuinely pre-publication gate, which is why the receipt check lives there and CI merely repeats it.

Two structural details that were bugs before they were features:

The self-test trigger sits ABOVE the hook's if (-not $blogChanged) { exit 0 } early exit. It was originally below it, which meant a push changing only monthly-blog-qa.py carried no blog markdown and so skipped the guard's own tests — the one thing that push should have checked.
The hook sets PYTHONIOENCODING=utf-8 and [Console]::OutputEncoding. Findings name sections as §33; without it the blocked-push message — the one thing the author actually reads — arrives as \uFFFD33.
The receipt gate also sits above that early exit, and matches three path shapes the old filter never saw: the issue markdown, static/images/blog/, and qa/monthly-copilot/. Swapping an image changes no markdown at all, so a content-only filter is blind to precisely the failure the receipt exists to catch.
The annotation gate rides the same trigger, immediately after the receipt check — deliberately beside it rather than folded into it, because a receipt is a hashed artefact and a PASS written before the gate existed would satisfy it forever. See The annotation gate below.

CI covers the whole corpus, not just changed files, because an exceptions.json edit or a roadmap-feed refresh can invalidate a post nobody touched.

verify-receipt is deliberately feed-independent. A bot commits static/data/roadmap/latest.json daily, and a bot must never be able to redden main. (lint does read the feed and is wired into CI — that exposure is real and pre-existing, listed under Open items.)

The receipt gate — evidence, not assertion¶

Wiring audit/verify-receipt into automation looked like a half-hour job. A Rule

16 Gate A review killed the design, and both objections reproduced under¶

execution. Both are worth remembering because both looked fine.

1. The receipt hash was not portable. core.autocrlf=true, so one commit is CRLF in the Windows working copy and LF in a Linux CI checkout. Measured on the August issue: 1,264 CRLF sequences, byte hash 51a93af3… locally against 3b62ecf8… on Ubuntu. Wiring the check into CI as proposed would have failed August as a stale receipt on day one — a red build for a post with nothing wrong with it.

Staleness caused by a substantive edit is the feature. Staleness caused by checkout line endings is a bug. Post hashes are now LF-canonical (sha256_textfile, schema 2); image hashes stay byte-exact, because normalising a binary would defeat the whole point.

2. The receipt certified its own JSON, not the QA. It read the arrays recorded at audit time and never re-derived anything. Appending one byte to a reviewed image left verify-receipt green while audit on the identical state reported FAIL — because swapping an image touches no markdown, so the post hash matched and the stale empty arrays still passed.

verify-receipt now re-derives the image manifest from the post and requires every embedded image to exist, hash as recorded, and be backed by a written observation — not a bare hash, and not an **Observed:** heading with nothing under it.

The threat model is accident, not adversary. Sush is the sole author and will not forge his own QA. The realistic failure is swapping a screenshot and forgetting to re-observe it. That reframing is what decides the design: bare-hash acceptance is low risk, but a missing image re-hash is high risk because it fails silently. Re-hashing all 60 August images costs 42 ms, so there was never a performance argument.

Grandfathering. The seven pre-August issues carry 182 unobserved images and 34 unresolved sections between them. Retrofitting that evidence would be invented, not observed, so they sit in legacy-baseline.json instead. The cutoff lives in code (ENFORCEMENT_START), never in the JSON — so exempting a gated post can never be a one-line edit to a data file, and a slug at or after the cutoff fails validation.

Then Gate B found the gate was still mostly theatre¶

The commit above passed its own 63 tests, ran green on Ubuntu, and blocked a real push. It was still verifying far less than it claimed. A Rule #16 Gate B review of that diff found six more defects; all six reproduced, and four let a receipt PASS while actively lying:

What the receipt claimed	What was actually checked
Every section resolved	Nothing. Deleting `sections`, marking one `"unresolved"`, or zeroing `post.sections` all passed
This image was reviewed	Only that some section reviewed that hash — §5's prose certified §40's image
These are the images	A dict keyed by `src`, so a duplicate record with a wrong section and wrong hash was overwritten by a correct one and vanished
Baseline may only shrink	A comment. A backdated `december-2025` slug sorts before the cutoff and would be accepted

Plus: single-post mode computed the baseline errors then discarded them, so the one command a human runs by hand was the one that lied; and malformed receipts failed by AttributeError instead of the named diagnostic the gate exists to print.

The lesson worth carrying forward: a gate that passes its own tests, runs in CI, and blocks real pushes can still be verifying almost nothing. "It went green" is evidence about the test, not about the guard. Both rounds of holes were found by an independent reviewer looking at the diff — neither by the tests, and neither by me.

Mutation testing is the only thing that answers "is this guard real?" Break each guard, require the suite to go red. It stands at 30/30 and has already caught a guard of mine with zero real coverage — the fixture was failing for an unrelated reason, so the test passed for the wrong reason and would have passed with the guard deleted. It now lives in the repo and runs in CI, because a mechanism that depends on someone remembering to run it is already dead (Rule #14b):

python scripts/monthly-blog-qa.mutate.py    # 55 guards, ~20s, no network

A MISS means the suite cannot see that guard disappear — the guard is decorative. A SKIP means an anchor drifted; repair the anchor, never delete the mutation. Every new guard gets a mutation in the same commit, or it is unproven.

Round 2: the one where audit and its own verifier disagreed¶

A third review, of the round-1 diff, found four more. The one worth remembering is not the one that let a lie through — it is the false positive:

audit matched an observation by hash alone. verify-receipt required (section, hash). Put one image in two sections, observe it under only one, and audit printed state: PASS, then wrote a receipt that its own verifier rejected at push time — blocking an honest post with a message the author had just watched pass.

Any two components that decide the same question must derive it the same way. The fix was not to relax the verifier but to make audit strict, so the failure surfaces where it is actionable — at the desk, not at the gate. Rule #15 in miniature: the first instinct was to caption it ("verify is stricter, that's fine"); it was a defect.

The other three: only the literal "unresolved" was rejected, so a missing, null or invented disposition claimed to be evidence; nested image fields were shaped but not typed, so an unhashable value raised a traceback and a JSON true masqueraded as section 1 (Python hashes True == 1); and diagnostics consumed candidates greedily, so messages changed with receipt order.

Two existing mutations had also gone stale against those very edits and had been silently not running. The harness reported them as SKIP rather than as confident false passes — which is precisely why it prints a count and why SKIP must never be ignored.

Now at 86 tests, 19/19 mutations.

Round 3: the extractor that was narrower than the post¶

A fourth review found seven more. Six were variations on two root causes (Rule #11 — fix the class, not the instance), and one class is the most dangerous shape this tool can have:

An extractor narrower than the post reports clean on every surface. lint passed, audit passed, the receipt was written and verify-receipt was happy — because the entry had never been seen. Markdown ![alt](src) screenshots were invisible to all four. Numbered table rows were parsed by lint and then discarded by audit and verify, which printed sections: 0 and PASSED a post that had entries neither had looked at.

There is no failure mode quieter than this. A missing guard at least fails loudly when someone tries it; a narrow extractor produces a green tick over an unexamined page. When you add a form to the post, ask what reads it — and check every reader, not the one you are looking at. The fix was one extract_images() serving lint, audit, the receipt and the ledger, and both audit and verify covering every section the parser finds rather than headings only.

A table row carries no 📖 source line, so derive() finds no roadmap ID in it. It therefore resolves only through a written disposition. That is fail-closed on purpose: the tool refuses to guess, and the author must write down why.

The second class: the receipt was trusted where it could be re-derived. The author-controlled dispositions file was copied straight into evidence — an invented word became a PASS receipt that the verifier rejected at push time (round 2's defect, reappearing through a different door), and an author could hand-assert roadmap_id on a section citing no ID at all. A disposition may now explain a section; it may never relabel one.

The recommendation that was rejected, and why. The reviewer advised having verify-receipt re-derive roadmap status from the feed. That breaks invariant #1: the feed is committed daily by a bot, so a feed-dependent verifier hands a bot the power to redden main. Both findings were closed offline instead — using the post text ("does this section cite any ID?") and the receipt's own recorded roadmap_status ("does it admit NOT-IN-FEED?"). A reviewer's severity is worth taking seriously; a reviewer's proposed fix is not automatically the right one. Size the blast radius before you accept either: a corpus scan showed 0 Markdown images and 0 August table rows, which made two "High" findings latent rather than live — still worth fixing, because they are exactly the silent traps September would spring.

And the lesson that cost the most: asserting a precondition proves nothing. The first version of the new table-row test asserted only that the parser saw the row — and the parser was never the broken half. It passed with the guard deleted. Mutation testing caught it, along with two more stale anchors. The positive end-to-end test is what proves coverage: drive the real command, assert the real exit code.

Now at 110 tests, 30/30 mutations.

Round 4 — what a URL serves, what a reader sees, what counts as evidence¶

Another independent review of the diff returned BLOCK with five High findings. All five were reproduced against the live tool before anything changed, and the frozen August post was probed first (0 table rows, 0 code fences, 0 HTML comments, all 60 images inside sections) to prove each fix could not disturb it.

Gap	Why it survived four rounds
Image casing	`Path.resolve()` on Windows rewrites the path to the on-disk casing, so the check compared a name with itself. `/images/blog/shot.webp` against a file named `Shot.webp` passes every local check and 404s on the case-sensitive host that serves the site. Fix: build the candidate from the requested components, never `resolve()` it; guarantee containment structurally by refusing `..`.
Invisible text parsed as content	A heading inside an HTML comment fabricated a section no reader sees. Comments, fenced blocks and inline code are now blanked with newlines preserved, so every offset and section span still lines up with the real file.
Reference-style images	`![alt][id]` was invisible to every check. `IMG_RE` requires `](` and `REF_IMG_RE` requires `][`, so the two can never double-count.
Orphan images	An image outside every numbered section can never be named by an observation, so it vanished from lint, audit, the receipt and the manifest at once — while all four reported a complete count.
Partial receipt validation	Per-record validation ran only for `roadmap_id`, so relabelling a corroborated section as a manual disposition skipped every check that would have contradicted it. It now runs for every disposition. Also: `str.strip()` does not remove zero-width characters, so a reason of `U+200B` alone read as written evidence — and audit accepted what verify rejected.

The table-row rule, decided by measurement instead of cleverness. Which numbered table rows are feature entries had now been wrong twice. Promoting every numeric first column turned an ordinary | 1 | Business | pricing table into phantom sections. Keying on the post's Quick Jump anchors was worse: March numbers its connectors 30–34 while its Quick Jump lists only three anchors, so five real entries with genuine roadmap IDs silently vanished from the gate. Only a full-corpus lint caught it — the count moved from 0 errors / 19 excused warnings to 1 error / 18 warnings. 🔴 Treat the warning count itself as a signal, not just the exit code.

The rule that survived was derived from all eight published issues rather than guessed: the decision is per TABLE, not per row. A table qualifies when it cites a roadmap URL, or fills a gap in the heading numbering, or starts where the headings stopped. That covers all three real shapes — February 38–45 after the last heading, March 30–34 and April 26–33 between headings — and rejects a pricing table that restarts at 1. A row numbered like an existing heading is reported as a duplicate, never dropped: April really does number two different features 29, and silently discarding it was the same fail-open shape the gate exists to prevent. Restoring that one warning is how the corpus returned to its exact baseline.

Two defects were caught by the harness, not by me. The mutation suite reported a MISS on the traversal guard: my new test used ../outside.webp, which the starts-with-/ check rejects first, so the .. guard itself was never exercised — the dangerous form is site-absolute and traversing (/i/../../outside.webp). And the September dress rehearsal crashed on TypeError: Object of type WindowsPath is not JSON serializable: I had put a Path on an image row that gets serialised into the receipt, which would have broken the first September audit. Neither is visible to a green test suite. Run the rehearsal; believe the MISS.

Now at 131 tests, 41/41 mutations, plus 23 reproduction cases that all fail closed. lint 291 ms · verify-receipt --all 195 ms — both well inside the hook budget. Then CI went red on a guard that was working. The casing fix passed everything on Windows and failed on Ubuntu. Nothing was wrong with the tool: on a case-sensitive filesystem the OS refuses the mis-cased lookup first, so the file is simply missing and the error message differs from the casing-specific one Windows produces. The push was blocked on both platforms — the test had over-specified the message.

🔴 A guard can be genuinely unobservable on a platform. On Linux, deleting the casing check changes nothing any test could detect, so the matching mutation would report a MISS there forever. That is a fact about the filesystem, not a hole in the suite. Both the test and the mutation now probe the filesystem at runtime — never key off os.name, because macOS is case-insensitive too — and the mutation prints n/a where the guard cannot be observed, so a skip is never quietly counted as a pass: 40/40 mutations caught (1 n/a on this platform).

The fix was proved before pushing by forcing the probe to report a case-sensitive filesystem locally and re-running both suites. A Windows-only green run cannot prove the Linux path — simulate the other platform, or the only proof left is CI itself.

The rehearsal that was only pretending to pass¶

The September dress rehearsal — the one artefact that caught the WindowsPath regression — was still a scratch script, and by Rule #14b that means dead. Promoting it into the permanent suite immediately exposed something worse: it had never actually passed. It only printed each stage's return code and then exited 0 regardless, so nobody noticed that all four of its images were being rejected and three of its four sections were unresolved. Its fixture used - observed: where the ledger requires a bold **Observed:** block, and omitted the 📖 marker that makes a line a source line at all. A script that prints results is not a test; only an assertion is a test.

Rebuilt as six real assertions it now drives a realistic unpublished month through lint → audit → receipt → verify → manifest in one pass, and it hunts the false positive that every other test ignores: a gate that blocks an honest post teaches the author to reach for --no-verify, and then it protects nothing. Making it honest also forced the fixture to become realistic — *For:* lines on every section, and a §4 that cites a Message-center article rather than nothing, which is how a genuinely unlisted change actually appears.

A matching mutation now re-introduces the original bug (a Path on an image row) and is caught. 138 tests, 42/42 mutations; lint 288 ms, self-tests 1.1 s.

What four rounds of review actually demonstrated¶

Every round found real defects. None were found by the test suite, by CI, or by me — 63, then 75, then 86 green tests each sat on top of live holes. A green suite is evidence about the tests, not about the guards. The two mechanisms that repeatedly found truth were an independent reviewer reading the diff (Rule #16 Gate B) and mutation testing. Budget for both; a passing build is not a third opinion.

The annotation gate — proving a screenshot was actually marked up¶

Added September 2026, when the issue reached 129 images and Sush asked for the rule to be enforced rather than remembered: "each sessions are standalone and it will forget everything when I launch a new session." That is Rule #14b — a mechanism that depends on anyone remembering is already dead.

Run it:

python scripts\monthly-blog-qa.py annotations --all
python scripts\monthly-blog-qa.py annotations --post september
python scripts\monthly-blog-qa.py annotations --post september --pixels   # advisory report only

Every image in an issue must be classified in qa/monthly-copilot/<slug>.annotations.json, in one of four ways:

Disposition	Means	Evidenced by
`annotated`	our annotator drew the callouts	`source_sha256`, `output_sha256`, `spec`, `scale`, `callouts`, `boxes`
`annotated_at_capture`	the callouts were already in the captured UI	a written `reason`
`created_asset`	a diagram we authored	a written `reason`
`microsoft_artwork`	official Microsoft image, used as-is	a written `reason`

🔴 Why this gate does not look at pixels¶

The obvious design is to prove "this image is annotated" by counting red pixels. It was built, reviewed, and killed. Two independent Gate A reviewers rejected it, and then measurement settled it:

On lab-s12-search-in-rail-annotated.webp — a correctly annotated 337×301 image — the proposed rule (within ±46 per channel of the house red, largest connected component ≥60 px) finds 93 matching pixels whose largest component is 40. It rejects correct work. Widen the tolerance by nine units to ±55 and it returns 171 and passes. A threshold whose answer swings 4× on a 9-unit nudge is not a measurement, it is a coin toss with a number attached.

Two further findings, both verified:

Red proves nothing. 21 of the 29 authored diagrams in the September issue carry a red component above the threshold. Product UI supplies red of its own, so an unrelated red Delete button would happily certify a black annotation beside it.
Requiring the word "red" in alt text is not accessibility. WCAG 1.1.1 asks for equivalent information. "A callout points to Share response" is better alt text than "a red box is shown" — and the colour-word rule would have rejected the better one.

The cost of getting this wrong is asymmetric, which is the whole argument: a gate that blocks correct work teaches --no-verify, and that one switch does not disable one check — it disables the receipt, SEO, internal-content and mobile-layout gates in the same hook. A missed detection costs one image; a false positive costs every gate in the file.

So the gate is entirely deterministic: set membership, hashes, and alt-text regexes. --pixels still prints a red-coverage report, clearly labelled advisory, because it is genuinely useful to a human eye — it just never decides anything.

🔴 A note for the future session that will want to re-add a pixel gate. The advisory report on the September corpus looks encouraging, and that is the trap:
[pixels] annotated            n=92   red px min 6308 median 24340 max 71623
[pixels] annotated_at_capture n=4    red px min 163  median 1230  max 1529
[pixels] created_asset        n=29   red px min 0    median 2345  max 4276
[pixels] microsoft_artwork    n=4    red px min 0    median 0     max 0
annotated (min 6308) and created_asset (max 4276) do not overlap on this one month. That is not a threshold, for three reasons: the counts are raw and not normalised for image area, so they partly measure how big a picture is; n is one issue; and the measured failure was never the whole-image count, it was the largest connected component on a small image, where a correct annotation scored 40 against a required 60. Separation observed once is not a rule, and the asymmetric cost above does not change even if a threshold could be tuned. Report it, do not gate on it.

What it actually checks¶

Completeness, both ways. An image in the post with no record fails; a record the post no longer uses fails. Two images sharing a file name in different folders are refused, not resolved — the sidecar is keyed by name, and there is no correct way to guess which one a record meant.
The bytes on disk are the bytes the record describes. output_sha256 must equal the live file hash. This is the one hard, content-bound proof available, and it is free because audit has already hashed every image. Recolour or swap a screenshot and its claim to be annotated stops being true until someone re-records it — exactly as a changed hash reopens a Rule #8 observation.
An annotation that changed nothing is not an annotation. source_sha256 equal to output_sha256 means the annotator ran and drew nothing.
Alt text may not contradict the annotation. Calling our callouts black fails; our callouts are #CF2626. Calling a product element black passes — two real September alts legitimately do.
An annotated image's alt must say what the callout points at. No colour word is required.

The gate's own first bug, kept as a permanent test¶

Its first run produced 96 false alarms — "needs a written reason" against every one of the 92 annotated records. The data was right and the gate was wrong: a generated annotation does not have prose, it has machine provenance, which is stronger. A reason is what an exemption needs, because "we chose not to annotate this" is a judgement call that only a sentence can carry. Validation is now per-kind, and a test asserts both halves.

This is the same false-positive class Gate A had warned about, caught before shipping rather than after — which is the entire argument for running the reviewers.

The gate's own second bug: two word lists that disagreed¶

Gate B (post-build review, 15 Sep 2026) found the alt-text rule rejected 9 of 10 realistic annotated alt texts — "a red outline around…", "annotated screenshot of…", "a red ellipse surrounds…", "framed in red". It would have blocked correct work on first use.

The decisive tell was that the two matchers contradicted each other: the black matcher treated outlined / framed / ellipse as annotation vocabulary and the alt matcher did not, so "framed in black" was a violation while "framed in red" was not even recognised as an annotation. They had been written independently.

Fix: one vocabulary, declared once (_ANNOT_NOUN / _ANNOT_VERB), used by both. Two lists describing the same concept will always drift.

The black matcher is deliberately narrower than it looks like it should be¶

It matches only phrases that can only describe a mark somebody drew — circled | boxed | ringed | called out | annotated + in black, black callout/annotation/leader line, and in a black box/ring/circle. The obvious additions were tested and rejected because the product supplies them:

Rejected addition	Why it would block correct work
`outlined in black`	how Excel describes a selected cell
`highlighted in black`	a Word highlight
`black arrow`	a real toolbar glyph
`black outline`	a theme preview
`a black box`	ordinary AI-writing idiom — "the model is not a black box"

The cost is asymmetric: a missed stale caption costs one image; a false positive costs the whole hook, because --no-verify disables the receipt, SEO, internal-content and mobile-layout gates in the same run. Detection gap accepted and recorded, not silently traded away.

Known and accepted in the other direction: the alt matcher still admits generic English ("Outlook marks the email as read", "two text boxes appear"). Permissive on purpose.

It immediately earned its keep. Tightening it surfaced a real defect the old rule had masked: lab-s89-task-verified.webp carries 3 callouts and 2 boxes and its alt described none of them. It passed only because the alt said "a red exclamation mark" — a product glyph matching the old generic marks? pattern.

Two traps worth carrying forward¶

Naive substring matching. A check for red matches inside covered, numbered, credit, blurred, hovered, required, predicted. Both matchers use \b boundaries, and a permanent test feeds them exactly those words.
Fixtures that pass vacuously. The suite's published() helper writes no date: — so a fixture built on it is grandfathered, and every annotation assertion would have passed while testing nothing. The annots() driver writes an explicit date for this reason.

Adoption cutoff, asserted in both directions¶

ANNOTATION_POLICY_FROM = (2026, 9). Earlier issues are grandfathered and say so in their output line. Both directions are tested: silently skipping a missing sidecar is how an entirely unclassified future issue escapes, and blocking all eight historical issues is how the gate gets switched off in week one.

🔴 The grandfather branch is the gate's own off switch, so the month must never be guessed. Reading date: alone is not enough — Hugo also accepts date: 2026-9-21 (unpadded), date : …, TOML date = …, or no date at all in favour of lastmod. Every one of those returned None, and None was read as older than the policy: a brand-new, entirely unclassified issue reported grandfathered (0000-00) and exited 0. The month is now resolved from the front matter or the filename, and a post that yields neither fails closed with UNKNOWN MONTH rather than grandfathering itself. All four spellings are permanent tests, and a mutation proves the fail-closed branch is visible to the suite.

Why it is in the hook and not in the receipt¶

Wiring it into audit's receipt state broke two pre-existing receipts() tests, and that was the correct signal rather than a nuisance: a receipt is a hashed artefact recorded at audit time, so a PASS written before this gate existed would satisfy it forever. Enforcement lives in the push hook, which runs annotations --all fresh on every push, so there is no stale evidence to trust. audit still reports the status, because that is the screen an author actually reads.

Proven to fire. Corrupting one record's output_sha256 and driving pre-push-hook.ps1 with git's real stdin protocol exits 1 and names the image and section. A gate never seen to fail is not a gate. Forcing that failure also exposed a crash in the test suite's own failure path — it passed a re.Match where a string was expected, so the drift it detected surfaced as a TypeError instead of a finding.

One house red, declared once¶

annotate_screenshot.py drew (207, 38, 38) / #CF2626; annotate_lib.py drew (206, 38, 38) / #CE2626. One unit on the red channel is invisible to a person and permanent in the file, so nothing ever reported it. Both now import HOUSE_RED from scripts/house_style.py, and a test asserts that neither file re-declares a red of its own — the declaration count is what holds, where correcting both numbers would only have fixed today.

HOUSE_INK was deliberately not unified: annotate_screenshot.py documents its navy as measured pixel-exact from the reference harness post, and silently moving a measurement to make two numbers agree is the wrong fix. Flagged for a human call.

Rule #8 image QA — the part that is still human¶

The tool cannot do this. Sub-agents cannot either: they are text-only and will return a confident, wrong verdict on image content. Only a vision-capable model looking at the actual pixels can do it, and it must write down what it sees while looking, not afterwards.

Workflow:

python scripts\monthly-blog-qa.py images convert --post september   # webp -> png, view can't open webp
python scripts\monthly-blog-qa.py images manifest --post september  # what's referenced / observed

Then, for every image, view the converted PNG and append to qa/monthly-copilot/<slug>.images.md:

## §12 — Section title

`<full 64-char sha256>`

**Observed:** what is literally on screen — UI, app, labels, figures. No
expectation-based language.

**Verdict:** ✅ MATCH / ⚠️ PARTIAL / ❌ MISMATCH — one-line reason.

🔴 Three traps, all of which have bitten:

The parser only accepts a full 64-char SHA-256. The 16-char cache filename and the 12-char manifest display are both useless here. Source of truth is the images manifest output.
view cannot open .webp. Always view the converted PNG.
Image identity is the content hash, not the filename. Replacing an image under the same name correctly loses its reviewed status — that is the point.

audit then cross-references observations against referenced images and refuses to issue a clean receipt while any image is unobserved.

Building next month's issue — the order that works¶

Draft the post (draft: true). Keep the *For:* line and the 📖 source line on every section; both are enforced. Also run python scripts\mc-watchlist.py --month <YYYY-MM> for the Admin watch-list block near the end — see the section directly below.
python scripts\monthly-blog-qa.py lint --post <month> early and often. It is ~1 s; there is no reason to defer it.
Capture screenshots per copilot-monthly-blog-screenshots-playbook.md.
images convert → vision-QA every image → write the observations file.
Read the prose back against the observations. Any sentence that counts something in a screenshot ("five slides are visible", "three panels") must be checked against what the observation file actually recorded. This is the defect class that survived every other gate in August — see below. Run crosscheck --post <month> first: it narrows the read-back to a short list (9 candidates in August). It does not replace step 5 — a clean run is not proof of agreement, only that the shapes it can see agree.
audit --post <month> --write-receipt, then verify-receipt --all. The audit writes the claim; verify re-derives it from disk and is what the push gate runs.
links --post <month> — before flipping the draft, not before every push.
pwsh scripts\hugo-safe.ps1 — never bare hugo.
Push. The hook re-runs lint and the receipt gate. If you re-cropped a screenshot after the audit, this is where it stops you: re-observe the new image and re-run step 6. Explicit pathspecs only — this repo routinely carries ~100 untracked files from concurrent sessions, so git add . / -A / commit -a are banned.
Flip draft: false only when Sush says so.

The Admin watch-list block — trial from September 2026¶

The roadmap only reports things arriving. It structurally cannot report a retirement, a migration or an endpoint move, so every issue up to and including August had a blind spot: the changes that actually bite admins.

python scripts\mc-watchlist.py --month 2026-09 finds candidates from the public Message Center archive (MIT, rebuilt hourly). It prints two tiers: act now (a hard deadline, or a Retirement / Deferred feature tag) and review (the triage pool — including reversals detected by wording rather than by tag). On the August trial it returned 2 and 18.

Publish 2–3, never more than 4. If more qualify, the monthly block is the wrong container. If none do, omit the block entirely — it is not a standing feature.

Placement: after "On the horizon", before "How this issue was put together".

The rules this block must not break¶

Two independent reviewers were run before it was built. One voted to kill it; the other to ship it small. They converged on the same artefact, and these are the terms that made it defensible:

Rule	Why
Never present it as complete	The archive reflects one E5 tenant. Message Center carries a per-tenant "Status for your org" and a per-org relevance score, so the same post genuinely differs between tenants
Conditional, never a directive	"What I'd check: if you keep an allowlist…" — never "you must update your allowlist"
Only `ActionRequiredByDateTime` is a deadline	`EndDateTime`/`StartDateTime` are lifecycle fields. Labelling one "act by" would be a factual error in admin guidance
Write the words yourself	The script emits IDs, metadata and keyword signals — never prose. Generating copy from `Body` risks Microsoft's copyrighted wording, generic AI voice, and overconfident advice at once
One disclaimer, once	Same anti-scrolling-fatigue rule that stripped 53 repeated caption notes from the August issue
Don't commit the archive	2.6 MB, rebuilt hourly. The repo already learned this with the roadmap feed: a local copy of a live feed is a maintenance liability
Never bind a receipt to the live feed	It rebuilds hourly, so every historical receipt would stale instantly and teach `--no-verify`

Why it is not automated yet¶

Deliberate. Sush chose a one-month manual trial to see whether 2–3 items an issue genuinely earn their place before a permanent moving part is added to a gate that already blocks pushes. This is a known exception to "automate it or don't build it" — it survives only because the step above is inside the mandatory pre-flight list. If the September issue ships without the block being considered, that is the trial failing, and the answer is to wire discovery into audit, not to try harder to remember.

The script is intentionally not in the pre-push hook: lint is offline by design, and anything network-bound in a push gate teaches people to bypass it.

The two defect classes that only humans caught in August¶

Both survived lint, the hook, CI and the image manifest, because both live in the gap between the prose and the pixels — which no offline check can see.

1. Prose asserts something countable that the screenshot contradicts. §13 said "Five slides are visible in the thumbnail rail". The rail held four. The observation file had said "four" the whole time; nothing compared the two. Rule #8 produces the evidence but does not close the loop — someone has to read the artifact back against the sentences. crosscheck now narrows this to a short list — see below.

2. A screenshot leaks something that must not ship. The same §13 image carried a complete, working SharePoint sharing URL — tenant host, token, and the ?e= share parameter — with a red annotation box pointing straight at it. Redact in the source image, then look at the result: the first redaction pass still left the ?e= parameter and part of the tenant name legible. A redaction you have not re-viewed is not a redaction. Redact only what leaks — the :f: folder marker stayed, because the body text teaches it.

`crosscheck`, and why it is advisory on purpose¶

crosscheck compares numbers in image-describing prose against numbers in the recorded observation, and prints what disagrees. It exists because of defect class 1 above. It is not a gate, and that is the finding, not a shortcut.

Two independent reviewers were asked to attack and defend the idea before it was built. They split — one said don't build it, one said build it but make it block in lint — so the design was settled by measurement instead of by argument.

Measured against the live August issue, in which the only known defect was already fixed, so every hit is a false positive:

Formulation	False positives	Catches the real §13 defect?
Naive next-token `(number, noun)`	46	only by reading the `Verdict` prose, which quotes the old wrong number
Scoped to the `Observed:` block	21	no
+ strip markdown emphasis	21	yes
+ word boundaries	17	yes
+ ignore `Red box N` labels	13	yes
+ image-cue sentence filter	10	yes
+ hyphen guard (`seventy-nine`)	9	yes

Six tokenisation bugs, each invisible until measured, and each one an English-number parser being rebuilt badly. Two of them flagged correct prose as defective. The remaining 9 are irreducible: one screenshot may legitimately carry "438 available agents" and "230 active agents".

So it runs as an advisory command, never in the hook:

Nine lines to eyeball once a month is a good deal for a human.
Nine false alarms in a push gate teaches --no-verify, which would take the self-tests, lint and the SEO/OG guards down with it (Rule #14b).

A clean run is not proof of agreement. It cannot see a claim whose number follows its noun — §50's "up to 20 agents per site" against an observation reading "Unique agents … 125" is invisible to it. It narrows attention; it does not replace the read-back. Broadening the grammar to catch that shape also catches more false positives — recall and noise are coupled, so the narrow version is deliberate.

The regression test uses the verbatim historical wording from both sides of the real mistake (observation recovered from git show 1c624bdb^), not a fixture written afterwards. That matters: the first implementation passed a synthetic test and still failed the real one, because the stemmer unified slides → slid but left slide alone, so singular never met plural.

Link rot, and why `links` is separate¶

Web search invents learn.microsoft.com URLs. Of four candidates it proposed for the August billing citation, three were 404. Never cite a Learn URL you have not fetched. Note also that the real path was /microsoft-365/copilot/…, not the far more plausible-looking /microsoft-365-copilot/ai-billing/….

links is network-bound, so it is deliberately outside the push gate — a flaky connection must never block a push. Run it before publishing. August baseline: 56 distinct links, all 200 (34 www.microsoft.com, 12 learn.microsoft.com, 5 support.microsoft.com, 5 techcommunity.microsoft.com). support.microsoft.com GUID paths churn far more than Learn paths, so expect those to rot first.

Deliberate non-goals¶

No roadmap-coverage quota. Microsoft publishes no Copilot Cowork features to the M365 roadmap at all, so "N% of sections must cite a roadmap ID" would be a lie dressed as rigour. What audit enforces instead is disposition coverage: every section must be explained, even when the explanation is "no roadmap row exists".
No second roadmap data dump. static/data/roadmap/latest.json is already bot-committed daily. An earlier pass hand-rolled a parallel 2 MB copy with no refresh path — a duplicate of a live feed is a maintenance liability.
No hardcoded month lists. Posts are discovered by glob. The predecessor hardcoded January–August 2026, so September would have been silently skipped by the very tool meant to check it.
No external link checking in lint. It is network-bound and belongs in links, run before publishing — not in a push gate.

Files¶

Path	What
`scripts/monthly-blog-qa.py`	The tool. `lint`, `roadmap`, `audit`, `images`, `inspect`, `links`, `crosscheck`, `verify-receipt`
`scripts/mc-watchlist.py`	Message Center candidate finder for the Admin watch-list block. Read-only, network-bound, deliberately outside the push gate. Emits metadata and signals — never prose
`scripts/monthly-blog-qa.test.py`	185 self-tests — every real defect shape, plus the false positives above
`scripts/monthly-blog-qa.mutate.py`	Mutation harness — breaks each of the 55 guards and requires the suite to notice. Runs in CI. A SKIP means an anchor drifted; repair it, never delete it
`qa/monthly-copilot/<slug>.dispositions.json`	Optional. Author's written explanation for a section the tool cannot resolve. Validated, not trusted — it may explain, never relabel
`qa/monthly-copilot/exceptions.json`	Documented historical anomalies, each with a written reason
`qa/monthly-copilot/legacy-baseline.json`	Pre-August issues grandfathered out of the receipt gate. Must be a subset of `LEGACY_SLUGS` hardcoded in the tool — the shrink-only rule is enforced, not just documented
`qa/monthly-copilot/<slug>.images.md`	Rule #8 observations, one per image
`qa/monthly-copilot/<slug>.json`	Audit receipt (schema 2)
`.github/workflows/monthly-copilot-qa.yml`	CI backstop
`scripts/pre-push-hook.ps1`	The hook git actually invokes (`pre-push-check.ps1` is manual and never runs)

Open items¶

🔴 lint reads the roadmap feed and is wired into CI. A bot commits that feed daily, and the workflow triggers on it, so a roadmap entry disappearing upstream can turn main red without anyone touching the repo. Excused via exceptions.json today, but the structural fix is to split human-content checks from feed-drift checks. verify-receipt was kept feed-independent for exactly this reason.
Nothing blocks draft: false itself. The receipt gate fires on push, which is the last point before Cloudflare publishes — but flipping the frontmatter and pushing in one go is still a single action. Branch protection plus a PR would make CI a real gate; today main accepts direct pushes.
An observation is now bound to (section, hash). ✅ Closed by de979915, and cmd_audit was brought into agreement in round 2 — it had been matching by hash alone, so it passed posts its own verifier rejected. It was previously bound to the hash alone, so prose written under §5 satisfied the gate for an image embedded in §40 — a review of an entirely different feature reading as proof.
The *For:*-date-vs-screenshot contradiction is not machine-checkable and remains part of the vision pass.
crosscheck cannot see a claim whose number follows its noun. Widening the grammar raises false positives, so this is a deliberate ceiling, not a bug to fix.