Memory plugins, tested: all four passed — and so did the no-plugin baseline
Tested 2026-10-20 · DSH 0.2.1-rc.2 · model deepseek-official · Apple M5 16GB
Findings
1. All four candidates passed 36/36 runs on the v0.1 tasks. But the no-plugin baseline also passed 3/3 once we switched to a random codeword.
2. The codeword was persisted in workspace
3. Cross-session recall is not a differentiator for memory plugins in DSH. Their value has to come from compaction, structured retrieval and governance.
1. All four candidates passed 36/36 runs on the v0.1 tasks. But the no-plugin baseline also passed 3/3 once we switched to a random codeword.
2. The codeword was persisted in workspace
AGENTS.md and the host's session_projcache, and the new session could read it — a DSH capability, not a plugin.3. Cross-session recall is not a differentiator for memory plugins in DSH. Their value has to come from compaction, structured retrieval and governance.
Setup
- DSH 0.2.1-rc.2, headless profile, deepseek-official;
- Fresh
HOMEand workspace per run; hostsession_projcachecleared before new sessions; - Candidates: dsh-memory, dsh-mnemon, openviking-memory, context-lens; control: no plugin;
- Three runs per v0.1 task; one run per codeword baseline task.
Results
| Candidate | M1 cross-session | M3 contradiction | M4 isolation | Notes |
|---|---|---|---|---|
| dsh-memory | 3/3 | 3/3 | 3/3 | median 201s / 98s / 13s |
| dsh-mnemon | 3/3 | 3/3 | 3/3 | median 53s / 9s / 13s |
| openviking-memory | 3/3 | 3/3 | 3/3 | median 316s / 51s / 31s |
| context-lens | 3/3 | 3/3 | 3/3 | median 107s / 14s / 12s |
| baseline (no plugin) | 3/3 | 3/3 | 3/3 | recalled a random codeword |
| dsh-mnemosyne | install rejected (requires dsh-tools ^0.1.5-rc.2) | not testable | ||
Evidence
After replacing generic preferences with an unguessable codeword ZQ-4821, the baseline still answered correctly in a new session. The codeword was found in two places: session A's workspace AGENTS.md, and the host's storages/session_projcache/sessions/*.json summaries.
Separate project roots per session and clearing session_projcache did not stop it — the host has other persistence paths that the current harness cannot disable.
What this means
- For users: you already get basic cross-session continuity without a memory plugin;
- For plugin authors: “remember my preferences” is hard to justify; solve what the host cannot — post-compaction consistency, structured retrieval, multi-project governance, auditability;
- For rankings: publishing a ranking now would mislead, so this issue publishes the methodology finding only.
Next (v0.2)
- Move to scenarios the host cannot cover: forced compaction, cross-project name conflicts, long-horizon fact updates;
- Add unit-level tests against each plugin's store/recall interface;
- Only tasks where the baseline fails and plugins pass count toward a ranking.
Reproduce
cd bench/memory
DEEPSEEK_API_KEY=... ./run.sh --execute --candidate baseline --task M1 --runs 1Revision log
2026-10-20: initial. Pilot finding: v0.1 tasks do not discriminate; rankings paused.