⚙️ FEATURE: Execution-based verification loop (write → execute → verify → re-plan)

[FernandoCelmer]

Problem

After each tool-use cycle, pycodeloop does not automatically verify outcomes. The agent writes code, marks the task done, and stops — even if tests are broken or lint fails. The human must manually run verification commands to discover regressions.

This is the core harness mechanism identified in "Code as Agent Harness" (arXiv 2605.18747): agent loops must close the write→execute→verify→re-plan cycle automatically.

Context

The harness pattern:

1. Captures a lint/test baseline before the agent runs
2. After each agent attempt, runs lint + test commands
3. If they fail, calls Canonicalise() to build a structured follow-up prompt with labelled sections: # Previous attempt failed, regression block, lint failures, test failures
4. Retries up to max_attempts (default 3)

Expected behavior

CodeLoop gains a verify_loop() method:

result = await codeloop.verify_loop(
    prompt="Refactor the auth module",
    lint_cmd="ruff check .",
    test_cmd="pytest tests/",
    max_attempts=3,
)

Internally:

1. Capture baseline (lint_ok, test_ok) before first attempt
2. Run agent
3. Run lint + test after each turn
4. If verification fails, inject typed failure block into next prompt
5. If baseline was green and result is red → flag regression explicitly

Suggested implementation

• Add VerifyLoop class in pycodeloop/core/verify.py
• Add canonicalise(lint_output, test_output, baseline) -> str pure function
• Add verify_loop(prompt, lint_cmd, test_cmd, max_attempts=3) to CodeLoop
• Wire into JSONL trace: record verify_attempt, verify_result, regression_detected events

References

• "Code as Agent Harness" — arXiv 2605.18747, Section 3.2 (Feedback-driven control)