How we test

One sentence: same machine, same model, same inputs, real tasks. Prove the task can fail before comparing who does better.

Scope

• We test: task success, output quality, time and cost, failure modes, fitness boundaries.
• We do not test: installation compatibility (covered by radar / dshbase), security audits, UI polish, author background.

Fairness

• Fixed DSH version, model, machine, inputs and time budget;
• Three runs per task, median reported; fresh HOME and workspace each run;
• Install per official docs; if it does not install, we record that and do not hand-patch;
• Our own plugins are listed separately as “author-tested” and excluded from rankings.

The baseline gate

Every task must first fail a no-plugin baseline. If the baseline passes, the task measures the host or the model, not the plugin — the task is invalid and gets redesigned.

Scoring

• Success 50%, quality 30%, efficiency 20%; weights are fixed before running;
• Quality uses a 1–5 rubric with two reviewers; disagreements above 1 point are re-reviewed;
• No single overall ranking — task-level verdicts only.

Evidence

Every run keeps commands, logs, outputs, screenshots and JSON. Reports include reproducible commands and fixed inputs.

Notice and corrections

Authors are notified by email or GitHub issue with a 72-hour right of reply. Corrections are published with a revision log; nothing is deleted.

Cadence

One category per month: memory and context first, then ComfyUI and Blender.