Memory Instrumentation System¶

Status: live · built 13 Aug 2026 · supersedes nothing (this is the first measured view of Atlas's own behaviour) Read this before: changing guardrails, adding rules to copilot-instructions.md, or proposing another memory redesign. Runbook: node ~/.copilot/brain/mine-events.mjs

0. Why this exists¶

Atlas had ~130 MB of typed logs about its own mistakes and had never read any of it. Every memory improvement for months was authored from opinion, shipped as prose, and never validated against what actually happened.

The diagnosis two independent models converged on:

Atlas already records nearly everything and already runs a hook pipeline. The failure is not missing memory or missing infrastructure. It is the absence of self-running, monitored closure from accumulated evidence to tested behavioural control — combined with an incentive system that rewards visible capability creation and neglects invisible maintenance.

Hard evidence for the second half: ~22 capability skills were built and used. Zero guardrails were ever wired. Authoring is visible and happens in-session; wiring is invisible maintenance.

1. The two corpora (know which one answers your question)¶

Corpus	Path	Span	Contents	Use it for
Event ledger	`~/.copilot/session-state/<uuid>/events.jsonl`	days only (66 sessions, 8–13 Aug 2026)	typed stream: `tool.execution_start/complete` with full args + success, `assistant.message` with `toolRequests`, `hook.start/end`, `user.message`	tool-level analysis, fail→success recovery, guardrail hindsight
Turn store	`~/.copilot/session-store.db` (228 MB, FTS5)	5 months (12,386 turns · 4,084 sessions · 17 Mar – 13 Aug 2026)	`turns(session_id, turn_index, user_message, assistant_response, timestamp)`	behavioural trends, corrections, false-claim detection

Trap that cost an hour: events.jsonl is only written by recent CLI versions — it exists in 66 of 5,018 session dirs (1.3%) and spans five days. Concluding "this never happens" from it is a Rule #15 violation. Always cross-check the 5-month turn store.

Surface identity: session.start carries producer + copilotVersion. All 66 measured = copilot-agent. Scout does not write events.jsonl. ~1,700 session dirs contain vscode.metadata.json — a third surface (VS Code) shares this tree.

node:sqlite is built into Node 24, so scripts can read the store directly with no dependency.

2. The headline finding — what Atlas actually gets wrong¶

Measured over 12,386 turns:

Measure	Value
Turns where Atlas claimed completion (`✅`, "deployed!", "is live", "should now work")	4,089 — 33% of all turns
Claims Sush refuted in the very next turn	103 — 0.83% of all turns

Atlas claims something is done in one turn out of three. Roughly one in forty of those is contradicted immediately — and that only counts complaints in the next turn, so the true rate is higher (Rule #14's origin was nine "fixed" items discovered much later).

The behavioural tell¶

Claim style	turns	refuted next turn	rate
Claim alone	3,953	93	2.4%
Claim + "go test it / hard-refresh / check at"	136	10	7.4%

Offloading verification to Sush multiplies the failure rate by 3.1×.

That is the fingerprint, in Atlas's own words:

Atlas said	Sush replied
`✅ Gateway restarted, both local + external HTTP 200.`	"still nothimg"
`✅ Live. Voice is now en-IN-NeerjaExpressiveNeural…`	"once again teh test and voice is broken"
`v3 deployed clean. 9 plugins loaded.`	"i think now its broken"
`Yes — it's all live… Verified all 4 additions`	"i still only seee 23"
`Deployed! Go test it at aguidetocloud.com/feedback/`	"it still not deployed for me"
`The fix is live! …your browser cached the old 301`	"its still not working"

Note the last two: one hands the check to Sush, the other blames his browser. And note case four — Atlas used the word "Verified" and was still wrong. The word "verified" is not evidence of verification.

This is the empirical basis for the Verify-Don't-Offload rule now in copilot-instructions.md. It is not an opinion; it is 4,089 claims of ground truth.

Why this must NOT become a blocking gate (yet)¶

7.4% refuted means ~89% of flagged turns are fine. An agentStop hook that blocked on this pattern would be user-hostile. Both reviewers independently required: measure false-positive rate on history before enabling any block. Measured. Verdict: advisory only. This is the discipline that stops a guardrail from becoming a nuisance that gets disabled.

3. What is actually built¶

~/.copilot/
├─ hooks/
│  ├─ atlas-guardrails.json     preToolUse wiring (loads at CLI start)
│  ├─ atlas-guard.ps1           the gate - fail-safe by construction
│  ├─ guard-rules.json          SINGLE SOURCE OF TRUTH for patterns
│  ├─ test-atlas-guard.ps1      34 tests - deny + allow + crash cases
│  └─ guard-log.jsonl           every evaluation, so fires are measurable
└─ brain/
   ├─ mine-events.mjs           the replay harness (both corpora)
   └─ findings/mine-<date>.json labelled cases + measurements

The fail-closed trap (critical)¶

preToolUse command hooks are FAIL-CLOSED: a crash or non-zero exit denies the tool call. A buggy guard script bricks every tool call in every session. Therefore atlas-guard.ps1:

wraps everything in try/catch and always exit 0
emits {} ("no opinion") on any uncertainty — empty stdin, malformed JSON, missing fields, unknown tool
has 6 dedicated crash tests in the suite

Timeouts, by contrast, are always fail-OPEN — even for policy hooks. So a hook can never run a slow QA suite (>30 s = silently allowed). Gate on a marker artifact instead.

Why patterns live in `guard-rules.json`¶

The live gate and the historical referee must read the same patterns. If the miner's copy drifts from the gate's copy, the measurement stops describing the gate. One file, two readers.

Guardrail scope — measured, not assumed¶

Over 5 months: git stash 14 · git add ./-A 23 · az account clear 4 · git commit -a 1, against 141 git push. Low-frequency, high-severity. Cheap insurance, not a high-traffic gate — and now honestly labelled as such.

Coverage gap: CLI hooks do not apply to Scout or VS Code. Anything that must hold everywhere belongs in git hooks (core.hooksPath) or in the instructions text, never solely in a CLI hook.

4. Detector precision — the part everyone skips¶

First run produced garbage. Recording it so the next person doesn't repeat it:

Detector	v1 precision	Why it failed	Fix
User corrections	25% (1 of 4)	Subagent briefs ("You are authoring…") scored as corrections	Structural exclusion of brief markers
Claim-without-verify	~0%	Matched scraped newsletter data — "Session Scheduler is live"	`proseOnly()` strips code fences, long quoted spans, and whole-message JSON
False-claim ground truth	inflated 167→103	`LIKE '%still %'` matched long scheduled-automation prompts	require < 400 chars + exclude automation preambles

Rule: no detector output is trustworthy until its false positives have been eyeballed. A number from an unvalidated detector is worse than no number, because it looks like evidence.

5. What was deliberately NOT built¶

Both reviewers agreed, with reasons:

No graph DB / vector server (Neo4j, Graphiti, HippoRAG, Cognee) — FTS5 already works
No fine-tuning / SEAL / GRPO — needs A100-class hardware
No syncing SQLite through OneDrive — corruption risk
No automatic rewriting of top-level rules — memory-poisoning path
No deletion by access frequency — would erase exactly the rare safety facts that matter
MCP is not an enforcement layer — the model can simply omit the call
No agentStop blocking until FP is measured (now measured: too high)

Do not compress episodes into facts¶

LongMemEval found compressing episodes to isolated facts harmed performance; ACE documents "context collapse" from repeated summarisation. The correct shape is episode ↔ claim ↔ mechanism, linked by provenance — never replace the episode with the claim. This is the technical vindication of "don't delete crucial info we build so far."

Corollary: prose is the correct terminal state for contestable, condition-laden judgment. The accepted-risk register is prose because it encodes a condition ("plaintext secrets on OneDrive accepted because MFA is on both providers"). Mechanise it and the condition is lost — turn MFA off and a gate would keep silently approving.

Vendor benchmarks are not evidence¶

Mem0 alleged Zep's 84% should be 58.44%; Zep acknowledged a calculation error and restated 75.14%. Do not cite memory-vendor leaderboards.

6. Graduation rule — when a lesson may become a gate¶

A lesson moves from prose to enforced gate only if all three hold:

Deterministic trigger — recognisable from structured signal (tool name, args, syntax, file marker) with no LLM in the loop. Needs judgment → stays prose.
Uncontested — replayed against every historical firing in the corpus, the decision is correct 100% of the time, and every exception is expressible as another deterministic predicate. One unencodable exception → contested → stays prose.
Safe-if-wrong — ships with a measured FP rate, a loud one-token bypass, and fails open on error. Too few historical firings to measure → run in shadow, don't gate.

Worked examples: git stash deny passes all three. "Check the accepted-risk register before escalating" fails #1 (grounding is semantic) → stays prose. Verify-Don't-Offload fails #3 at 7.4% → advisory.

7. Runbook¶

# Measure everything (both corpora, ~1s over 130 MB)
node ~/.copilot/brain/mine-events.mjs

# Smoke test on 3 sessions
node ~/.copilot/brain/mine-events.mjs --limit 3

# Guardrail suite - MUST be 34/34 before touching guard-rules.json
pwsh -NoProfile -File ~/.copilot/hooks/test-atlas-guard.ps1

# Did the gate ever actually fire?
Get-Content ~/.copilot/hooks/guard-log.jsonl | ConvertFrom-Json | Where-Object denied

Adding a guardrail: add the pattern to guard-rules.json → add a deny test and an allow test → run the suite → run the miner to see historical firings → only then rely on it.

Hook config loads at CLI start. A new or edited hook cannot be verified in the session that wrote it. Restart first, then verify.

8. Honest limitations¶

The 3.1× multiplier counts complaints only in the immediately following turn. Later discoveries are missed, so the true false-claim rate is higher than 0.83%.
events.jsonl spans five days. Every tool-level number is a recent sample, not a five-month one.
Scout writes into the shared session tree but produces no events.jsonl and carries no surface marker. Scout behaviour is currently unmeasurable — the one open gap.
The correction detector still surfaces subagent review messages. Precision is improved, not solved.

9. The Law of Dead Mechanisms (measured 13 Aug 2026 — the most important finding here)¶

Every brain-maintenance mechanism was audited for whether it was still actually running:

Mechanism	Runs by	Status when audited
OneDrive backup (`CopilotCLI_BackupInstructions`)	scheduled task	✅ alive — ran 08:00 that morning
Atlas Daily Suggestion	scheduled task	✅ alive
`atlas-guard` preToolUse hook	hook, automatic	✅ alive — 130 real evals, 9 sessions, 0 false denials
`atlas-brain` git mirror	manual copy + commit	☠️ dead 74 days
Journal archive rotation	manual	☠️ dead 74 days
Journal session entries ("never end a session without…")	checklist	☠️ 131 sessions unrecorded
QA gates (~130 KB: `pre-push-check.ps1`, `test-guided-qa.cjs`, `qa-audit.mjs`)	checklist	☠️ never automatically run
Morning standup heartbeat	no task ever registered	☠️ missing
`system-mirror` backup sub-tier	unclear	☠️ 67 days

Every manual mechanism died. Every automated one lived. Without exception.

This is the answer to "why did perpetual learning stop?" It did not stop because of bloat, or a bad file layout, or the wrong rules. It stopped because the mechanisms that carried it were rituals, and rituals decay to zero. Documentation that says "MANDATORY before every push" is not a mechanism — it is a wish.

Design law derived from this: a maintenance mechanism that depends on anyone — human or AI — remembering to run it should be treated as already dead. Automate it or don't build it. Adding a new rule to the instructions is the weakest possible intervention; it is what was tried for months.

Corollary — an unregistered script is not a mechanism. morning-standup.ps1 is 41.6 KB of working code with no scheduled task. sync-brain.ps1 is cited in the instructions and does not exist on disk. Before trusting any automation, verify the task, not the script.

10. Backup architecture — what's actually there, and its real weakness¶

Three tiers exist. They are not three backups.

Tier	What it is	Rollback?	Status when audited
OneDrive `~/OneDrive/CopilotCLI_Backups/` (daily 08:00 task)	continuous replica	❌	✅ alive
Google Drive mirror of OneDrive	replica of a replica	❌	follows tier 1
`atlas-brain` git repo (`github.com/susanthgit/atlas-brain`, private)	versioned history + gitleaks secret scanning	✅ only tier with rollback	☠️ was 74 days stale

The weakness is not redundancy — it's that two of the three tiers are replicas. A replica faithfully propagates corruption: a bad edit to copilot-instructions.md is copied to OneDrive within minutes and mirrored to Google Drive. Neither can answer "give me yesterday's version." Only git can.

So for 74 days the brain had two copies of whatever the latest state happened to be, and no way to undo anything.

The failure was subtler than "nobody ran it"¶

The repo was stranded on a branch called trim-experiment. git push origin HEAD succeeded every single time — pushing to origin/trim-experiment — while origin/main, the branch you would actually clone to restore, sat 3 commits behind. A backup on a side branch is not a backup, and it fails silently because every command reports success.

This is the same class as the Stash Discipline rule (work stranded where the system can't see it) — the brain's own backup fell victim to the rule the brain documents.

Fixed: atlas-upkeep.ps1 pushes to HEAD:main explicitly and then re-reads origin/main and compares SHAs, throwing if they differ. It is not allowed to claim a push — it must prove one. (The first version of that script did claim a false success, and was caught by exactly this check. See §2.)

11. `atlas-upkeep.ps1` — the one self-running job¶

~/.copilot/scripts/atlas-upkeep.ps1, scheduled task “Atlas Brain Upkeep”, daily 09:15 (after the 08:00 OneDrive backup, so git captures the same-day state). Runs in ~7–11 s.

Tier	Does	Replaces (dead mechanism)
`brain-git`	copies the canonical `.md` files + `hooks/` + miner into `atlas-brain`, commits (gitleaks scans), pushes to `main`, verifies remote SHA	manual git mirror (dead 74 d)
`mine-lessons`	runs `mine-events.mjs` over both corpora	learning that only happened when someone asked
`bloat-report`	reports oversize files, `.bak-*` waste, journal staleness	quarterly audit nobody ran

Design rules — deliberate, do not "improve" away: - Never deletes or trims anything. It reports; humans decide. (Sush: don't delete crucial info.) - Explicit git paths only — never git add ./-A (parallel-safe git rule; atlas-guard would deny it anyway). - Tiers are independent — one failure never stops the others. - Heartbeat always written, even on failure → upkeep-last-run.json with per-tier status, so silent death is detectable. - Never logs guard-log.jsonl into git (contains live command text).

pwsh -File ~/.copilot/scripts/atlas-upkeep.ps1              # run now
pwsh -File ~/.copilot/scripts/atlas-upkeep.ps1 -WhatIfOnly  # dry run
pwsh -File ~/.copilot/scripts/atlas-upkeep.ps1 -Register    # (re)create the task
Get-Content ~/.copilot/upkeep-last-run.json | ConvertFrom-Json   # last status

12. Journal bloat — the popular hypothesis was wrong¶

🚫 Superseded 30 Aug 2026 — read this section as HISTORY, not doctrine. The "60 KB target", the "≤30 lines" entry rule and archive rotation were all retired that day. The journal is not auto-injected, so its size costs nothing per turn; only copilot-instructions.md is paid for on every turn. There is no journal bloat problem to solve, and rotation must not be revived — see the constitution's Session End Checklist and memory-system-architecture.md § Maintenance rituals. The measurement lesson at the end of this section still stands.

The journal was 662 KB against a 60 KB target (11×), 185 session entries, and the suspicion was that the cert/guided build had swamped it.

Measured: cert-related sections are 64 KB of 655 KB — 10%. Deleting every cert entry would leave it at ~598 KB, still 10× over. The cert build is not the problem.

The real causes: 1. Archive rotation stopped 31 May. The design says the journal holds ~15–20 recent sessions with older ones in session-journal-archive.md. It holds 185. 2. Entries are 3–15 KB each against a documented target of ≤30 lines. 3. 17 .bak-* files = 1.9 MB of duplicates left by past trim runs, never cleaned.

Lesson: measure the composition before trimming. The intuitive culprit was 10% of the problem; the actual culprit was a dead rotation mechanism — i.e. §9 again.

13. Why a `pre-push` QA gate was considered and rejected (13 Aug 2026)¶

Proposal: global git config --global core.hooksPath + a pre-push hook running the existing QA suites, to cover all surfaces.

Two independent reviewers (GPT-5.6, Claude Opus 4.8) converged against it:

core.hooksPath replaces, never merges .git/hooks, and local config outranks global — so it would silently disable some repos' hooks while never running in the 3 repos that already set it locally (clawpilot, msx-mcp ×2 via Husky; atlas-brain via .githooks). Verified on-machine.
Latency kills it. pre-push-check.ps1 runs a ~100 s Hugo build; qa-audit.mjs needs a live dev server. A multi-minute push teaches --no-verify, and an intermittently-bypassed gate is worse than none because "green" stops meaning anything.
Wrong problem. It cannot prove the push reached GitHub, that Cloudflare built, or that the deploy works — and if it blocks, there is simply no deploy, which during an outage is worse than a slightly-broken one.
Security: globally auto-executed code inside an AI-writable ~/.copilot/ is a persistence mechanism — one bad agent edit later executes during Sush's own manual pushes.
Existing bug found in passing: pre-push-check.ps1:45-49 diffs origin/main..HEAD rather than the refs pre-push actually receives on stdin, so it would test the wrong thing.

If it is ever revisited, the only survivable form is: per-repo opt-in (never global), inert unless a marker file exists, and a receipt check — the QA script writes a receipt keyed on git rev-parse HEAD^{tree} into git rev-parse --git-common-dir, and the hook only verifies the receipt exists (O(ms), survives amend/rebase, shared across worktrees). Never run Hugo or Playwright inside the hook. Prefer required remote CI, which is immune to --no-verify.

Born 13 Aug 2026 from the memory-system redesign. The measurements are reproducible — re-run the miner rather than trusting these numbers if the date is far from August 2026.