v0.2 · unit level

Memory consistency v0.2: mnemon's storage layer, 9/10 passed

Tested 2026-10-20 · spec CONSISTENCY v0.2 · under test @mnemon-dev/[email protected] · macOS arm64 · no embedding initially

Findings
1. Nine of ten checks pass. The one failure is contradictory facts do not supersede: old and new coexist, both active.
2. It is not flaky: two independent runs produced identical pass/fail results.
3. The more interesting finding: default hybrid retrieval collapses on a shared-prefix corpus — the target never reaches the top 10, while the same data ranks first under search.
4. An embedding control group was added: detection escalates to CONFLICT (0.92), the action is still added. Root cause narrowed to “annotates conflicts, does not resolve them”.

Why a different test design

The v0.1 pilot tested “does it remember across sessions”. That was falsified: the no-plugin baseline passed too, because the host itself persists AGENTS.md and a session cache. So v0.2 tests three deterministic things instead: is it right, does it leak across projects, is the write auditable.

What was tested

@mnemon-dev/[email protected] — CLI with a darwin-arm64 native package, local SQLite, no provider key required;
Layer: storage. The dsh-mnemon plugin orchestration layer above the CLI is out of scope;
Environment: Node v24.20.0 (DSH side) / v22.22.2 (WB side, independent implementation); no embedding at first (embedding_available=false).

Method

The adapter was implemented by adam-wb from the spec alone (never read the reference adapter), covering all six checks; each check uses its own temp directory, removed afterwards;
Retrieval is pinned to search --limit 10; hybrid recall ranking is recorded as an observation only, never as a verdict;
Three outcomes: pass / fail / skip. Skip is reserved for “the check never ran”, rendered separately;
Two independent runs, compared row by row (manifest.runs.stable=true).

Results

Check	Result	Evidence
roundtrip	✅	write returns `action=added`; read returns identical content
persistence-across-process	✅	process A writes → 94,208 B on disk → process B reads it back
audit-trail	✅	`receipt.insight_id_hash == sha256(id)`; the receipt contains no raw text
store-isolation	✅	zero foreign content returned in the reverse direction
contradiction	❌	old and new coexist, both active; `diff_suggestion=UPDATE` but `action=added`
scale-limit	✅	200 writes in 8.5 s; `search` ranks both ends first; 0.95x latency ratio

Plus four regression checks for memory-hub, all green: 9/10 in total.

How the three criteria were judged

No cross-project leakage · pass. The criterion is “is the other store's text returned”, not “is the result count zero”. With no keyword hit, recall falls back to recency (matched_via=time); counting results would misread that fallback as leakage.
Auditable writes · pass, with a caveat. The audit artifact is receipt (hashed, safe to hand across a boundary). The local log echoes raw text — that is a local debugging view, not a boundary leak, but syncing that directory would leak it.
Compaction consistency · fail. Spec v0.1 contradicted itself here (“just record the behaviour” vs “both versions active = fail”). v0.2 rules fail, from the user's point of view: if the old conclusion is still active, the user gets the opposite answer — regardless of whether the cause is a missing model or a deliberate design.

Two findings

1 · Default retrieval path collapses. With 200 entries sharing one prefix, search ranks the target first, while the default recall (hybrid) never puts it in the top 10 — the top slots are all matched_via=entity, score=0.85 near-duplicates. For users: the more memories you have, and the more alike they are, the more likely the default retrieval fails.

2 · Contradiction detection only advises, never acts. diff_suggestion=UPDATE shows it recognised the relation (similarity 0.75), yet the store action stayed added. If the product is “suggest, let the agent decide”, that is a design choice; if it claims auto-correction, it is a defect. We report the fact, not the verdict.

Supplementary group: embedding control (added 2026-10-20)

operator: adam-wb · run 2026-10-20-memory-mnemon-embed · ollama 0.40.6 + nomic-embed-text, 100% coverage · independently re-run by adam-dsh.

Observation	No embedding	With embedding
`diff_suggestion`	`UPDATE` (0.75)	`CONFLICT` (0.92)
`action`	`added`	`added`
`superseded_id`	—	`null`
Retrieval state	both active	both active
Ranking	—	old fact #1 (0.62) > new fact (0.31)

The failure is not environmental. Detection fired (CONFLICT); the action still did not. Per §2.1 the fail stands, and the root cause narrows to “annotates, does not resolve”.
The old conclusion ranks above the new one — worse than a tie, because it looks like a ranked, trustworthy answer.
CONFLICT sits right at the threshold. Replication: adding one word (Pages) drops similarity to 0.787 and turns the verdict into ADD; removing the shared prefix gives 0.873, still CONFLICT. Same meaning, tiny wording change, different classification — so the number is not a stable semantic label.
Shared-prefix corpus: 2 of 3 samples succeeded with embedding. Recorded as insufficient coverage, not as a result (≥10 samples needed).

old: deployment goes through Cloudflare      → recall 0.62 entity
new: no longer Cloudflare, switch to Vercel  → recall 0.311 entity
second write: action=added  diff_suggestion=CONFLICT  superseded_id=null  similarity 0.873

Coverage gaps (not passes)

~~Embedding group uncovered~~ → covered (see above), conclusion unchanged; new gap: the shared-prefix corpus needs ≥10 samples;
The plugin orchestration layer is untested (providers / strategies above the CLI);
No no-plugin baseline for unit checks: these are deterministic, so do not compare them with the v0.1 behavioural results.

Reproduce

cd bench/memory/unit
npm install
node run.mjs          # table for humans / CI
node run.mjs --json   # machine-readable

Evidence: runs/2026-10-20-memory-mnemon/manifest.json (sha256 for adapter / runner / spec, plus two-run stability).

Appeals and corrections

Authors who believe the contradiction verdict is unfair may follow the appeals process: provide evidence → get it listed separately → the correction is recorded. This section never rewrites history, only appends.

2026-10-20: initial draft (review).

2026-10-20: reviewed and signed by JOJO; published.

2026-10-20: embedding control group added (operator: adam-wb, independently re-run by adam-dsh); conclusion unchanged.