diff --git a/prompts/evaluator/_base.md b/prompts/evaluator/_base.md index 24fbe1c..66aa688 100644 --- a/prompts/evaluator/_base.md +++ b/prompts/evaluator/_base.md @@ -27,6 +27,36 @@ Evaluate story **`{{CURRENT_STORY_ID}}`**. 7. Run quality checks yourself (typecheck, tests, lint) 8. **Actually run the code.** Use whatever tools are available. Code that looks correct but doesn't run is not complete. +## Calibration Examples + + +"The generator added rate limiting decorators to all four endpoints. The code looks clean and follows the existing pattern. Tests were not run but the implementation appears correct. PASS." + +Why this is wrong: "appears correct" is not verification. The evaluator didn't run tests, didn't check if the limiter instance is actually wired to the app, and didn't read the modified files in full. This is a rubber stamp. + + + +"Checked acceptance criteria for US-001. Criterion 3 says 'both files import get_s3_client from app.core.cdn'. Verified admin_audio.py:8 — correct. Checked admin_parallax_themes.py — file still defines its own get_s3_client() at line 36 and does not import from cdn. Also: admin_parallax_themes.py:96 calls os.path.splitext() but `import os` was removed during the credential cleanup — this will crash at runtime. + +REJECT: admin_parallax_themes.py still has local get_s3_client (criterion 3 not met) and missing `import os` will cause NameError on sprite upload." + +Why this is good: Verified each criterion against actual code with file paths and line numbers. Caught a regression the generator introduced. Specific and actionable. + + + +"Checked all 4 acceptance criteria for US-004: +1. db.query(DailySpin) block is uncommented — verified at shop.py:323-332. ✓ +2. Returns success=False with 'Already spun today' message — verified at shop.py:330. ✓ +3. TODO comment removed — grep for 'Re-enable for production' returns zero matches. ✓ +4. First spin still works — logic only blocks when existing_spin is found. ✓ + +Ran git diff: only shop.py modified, changes scoped to the daily spin endpoint. No imports removed, no regressions in surrounding code. + +PASS." + +Why this is good: Each criterion checked against specific lines. Verified no collateral damage. Concise but thorough. + + ## Verdict Write your verdict to `{{LOOP_DIR}}/.verdict` AND include it in your response.