sheldon/loop-loop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add few-shot calibration examples to evaluator prompt

Browse Source 
Three examples showing bad rubber-stamp, good rejection, and good
pass patterns. Based on Anthropic's harness design recommendation
to calibrate evaluators with few-shot score breakdowns, and
informed by real failures observed in a production loop run.
This commit is contained in:
Sheldon Finlay
2026-03-28 11:15:52 -04:00
parent 80b0f0f4c1
commit 1d059e218b
1 changed files with 30 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
prompts/evaluator/_base.md
View File

@@ -27,6 +27,36 @@ Evaluate story **`{{CURRENT_STORY_ID}}`**.
7. Run quality checks yourself (typecheck, tests, lint)
8. **Actually run the code.** Use whatever tools are available. Code that looks correct but doesn't run is not complete.
## Calibration Examples
<example type="bad-evaluation">
"The generator added rate limiting decorators to all four endpoints. The code looks clean and follows the existing pattern. Tests were not run but the implementation appears correct. PASS."
Why this is wrong: "appears correct" is not verification. The evaluator didn't run tests, didn't check if the limiter instance is actually wired to the app, and didn't read the modified files in full. This is a rubber stamp.
</example>
<example type="good-rejection">
"Checked acceptance criteria for US-001. Criterion 3 says 'both files import get_s3_client from app.core.cdn'. Verified admin_audio.py:8 — correct. Checked admin_parallax_themes.py — file still defines its own get_s3_client() at line 36 and does not import from cdn. Also: admin_parallax_themes.py:96 calls os.path.splitext() but `import os` was removed during the credential cleanup — this will crash at runtime.
REJECT: admin_parallax_themes.py still has local get_s3_client (criterion 3 not met) and missing `import os` will cause NameError on sprite upload."
Why this is good: Verified each criterion against actual code with file paths and line numbers. Caught a regression the generator introduced. Specific and actionable.
</example>
<example type="good-pass">
"Checked all 4 acceptance criteria for US-004:
1. db.query(DailySpin) block is uncommented — verified at shop.py:323-332. ✓
2. Returns success=False with 'Already spun today' message — verified at shop.py:330. ✓
3. TODO comment removed — grep for 'Re-enable for production' returns zero matches. ✓
4. First spin still works — logic only blocks when existing_spin is found. ✓
Ran git diff: only shop.py modified, changes scoped to the daily spin endpoint. No imports removed, no regressions in surrounding code.
PASS."
Why this is good: Each criterion checked against specific lines. Verified no collateral damage. Concise but thorough.
</example>
## Verdict
Write your verdict to `{{LOOP_DIR}}/.verdict` AND include it in your response.