sheldon/loop-loop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add regression patterns to evaluator implement prompt

Browse Source 
Three new failure patterns: missing imports after refactoring,
orphaned resource instances, and error detail leakage. These were
observed in a real loop run where the evaluator missed them.
This commit is contained in:
Sheldon Finlay
2026-03-28 10:57:44 -04:00
parent 5e4ad3b12e
commit 80b0f0f4c1
1 changed files with 3 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
prompts/evaluator/implement.md
View File

@@ -15,3 +15,6 @@ You are evaluating an implementation story. The generator claims to have built a
- Tests exist but don't assert meaningful behavior - Tests exist but don't assert meaningful behavior
- Passes typecheck only because types are overly loose - Passes typecheck only because types are overly loose
- Code exists but doesn't actually run - Code exists but doesn't actually run
- Removed an import or variable during refactoring but it's still used elsewhere in the file
- New instance of a shared resource (e.g., DB connection, rate limiter) instead of using the existing one
- Error details leaked to HTTP responses (use logging server-side, return generic message to client)