From 7e0fde1b2396e91d11601fc1369007e7163ef518 Mon Sep 17 00:00:00 2001 From: Danil Pismenny Date: Thu, 30 Jul 2026 20:25:36 +0500 Subject: [PATCH 1/6] Add real Codex human-gate E2E suite --- CHANGELOG.md | 4 + Makefile | 7 +- README.md | 21 +++ README.ru.md | 21 +++ memory-bank/engineering/testing-policy.md | 7 +- memory-bank/features/FT-016/README.md | 19 +++ memory-bank/features/FT-016/brief.md | 82 ++++++++++++ memory-bank/features/FT-016/design.md | 44 +++++++ .../features/FT-016/implementation-plan.md | 46 +++++++ memory-bank/features/README.md | 5 + memory-bank/ops/development.md | 7 + test/e2e/human-gate.sh | 124 ++++++++++++++++++ test/start_issue.bats | 14 ++ 13 files changed, 398 insertions(+), 3 deletions(-) create mode 100644 memory-bank/features/FT-016/README.md create mode 100644 memory-bank/features/FT-016/brief.md create mode 100644 memory-bank/features/FT-016/design.md create mode 100644 memory-bank/features/FT-016/implementation-plan.md create mode 100755 test/e2e/human-gate.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index d798fbd..1e97552 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ This project follows Semantic Versioning. ## [Unreleased] +### Added + +- Added an opt-in local E2E smoke suite for the real Codex human-gate workflow. + ## [1.13.3] - 2026-07-30 ### Fixed diff --git a/Makefile b/Makefile index 65c3aea..ca87388 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: build install uninstall test print-version bump-patch bump-minor bump-major release-patch release-minor release-major +.PHONY: build install uninstall test e2e-human-gate print-version bump-patch bump-minor bump-major release-patch release-minor release-major PREFIX ?= $(HOME)/.local BINDIR ?= $(PREFIX)/bin @@ -27,11 +27,14 @@ uninstall: test: bash -n scripts/start-issue - shellcheck install.sh scripts/start-issue scripts/build-start-issue scripts/bump-version scripts/prepare-release scripts/lib/start_issue/*.sh + shellcheck install.sh scripts/start-issue scripts/build-start-issue scripts/bump-version scripts/prepare-release scripts/lib/start_issue/*.sh test/e2e/*.sh python3 scripts/check_memory_bank_index.py --max-depth 4 git diff --check bats test +e2e-human-gate: + @bash test/e2e/human-gate.sh + print-version: @awk -F'"' '/^VERSION="/ { print $$2; exit }' scripts/start-issue diff --git a/README.md b/README.md index 2389346..7130612 100644 --- a/README.md +++ b/README.md @@ -269,6 +269,27 @@ State files: /.start-issue/runs//thread-id ``` +### Local real-Codex E2E smoke test + +The normal Bats suite uses a fake Codex CLI. To exercise the real local Codex +CLI, run this opt-in test from a `start-issue` checkout: + +```bash +START_ISSUE_E2E=1 make e2e-human-gate +``` + +The script uses the private `dapi/start-issue-e2e-fixture` repository and its +control issue, requires authenticated `gh`, rejects the fake Codex binary, and +creates an isolated temporary clone and worktree parent. It deletes those after +success; set `START_ISSUE_E2E_KEEP=1` to retain them. To test interactive resume, run: + +```bash +START_ISSUE_E2E=1 \ +test/e2e/human-gate.sh --scenario human-gate +``` + +Exit the resumed Codex session to let the script verify the artifacts. + Configuration precedence: 1. Agent: CLI `--agent` / `--no-agent`, then project config, user config, `START_ISSUE_AGENT`, then built-in default `claude` diff --git a/README.ru.md b/README.ru.md index 0b56dc5..de507f0 100644 --- a/README.ru.md +++ b/README.ru.md @@ -109,6 +109,27 @@ State files: /.start-issue/runs//thread-id ``` +### Локальный E2E smoke test с реальным Codex + +Обычный Bats-набор использует fake Codex CLI. Для проверки с реальным локальным +Codex из checkout `start-issue` выполните opt-in команду: + +```bash +START_ISSUE_E2E=1 make e2e-human-gate +``` + +Скрипт использует приватный репозиторий `dapi/start-issue-e2e-fixture` и его +control issue, требует авторизованный `gh`, не допускает fake Codex и создаёт +отдельный временный clone и worktree parent. После успеха они удаляются; чтобы +сохранить их, задайте `START_ISSUE_E2E_KEEP=1`. Для проверки interactive resume: + +```bash +START_ISSUE_E2E=1 \ +test/e2e/human-gate.sh --scenario human-gate +``` + +Выйдите из возобновлённой Codex-сессии, после чего скрипт проверит артефакты. + ## Использование ```bash diff --git a/memory-bank/engineering/testing-policy.md b/memory-bank/engineering/testing-policy.md index b338e0f..0da6599 100644 --- a/memory-bank/engineering/testing-policy.md +++ b/memory-bank/engineering/testing-policy.md @@ -34,7 +34,7 @@ make test `make test` runs: 1. `bash -n scripts/start-issue` -2. `shellcheck install.sh scripts/start-issue scripts/build-start-issue scripts/bump-version scripts/prepare-release scripts/lib/start_issue/*.sh` +2. `shellcheck install.sh scripts/start-issue scripts/build-start-issue scripts/bump-version scripts/prepare-release scripts/lib/start_issue/*.sh test/e2e/*.sh` 3. `python3 scripts/check_memory_bank_index.py --max-depth 4` 4. `git diff --check` 5. `bats test` @@ -44,6 +44,7 @@ make test - Shell syntax: `bash -n` - Static analysis: `shellcheck` - Behavior/regression tests: Bats under `test/` +- Opt-in real-agent E2E smoke tests: scripts under `test/e2e/`, run manually and never in CI - Memory-bank navigation: `scripts/check_memory_bank_index.py` - Whitespace/conflict-marker check: `git diff --check` @@ -88,6 +89,10 @@ Manual-only verification is acceptable only for: - visual/manual review of long help text or docs when no stable assertion is useful. +Real-agent E2E scripts must require an explicit opt-in environment variable, +avoid fake agent binaries, preserve diagnostic artifacts, and stay outside +`make test` and CI. + For each manual-only gap, record the reason and the manual procedure in the feature plan or final handoff. diff --git a/memory-bank/features/FT-016/README.md b/memory-bank/features/FT-016/README.md new file mode 100644 index 0000000..4fe4d83 --- /dev/null +++ b/memory-bank/features/FT-016/README.md @@ -0,0 +1,19 @@ +--- +title: "FT-016: Real Codex human-gate E2E suite" +doc_kind: feature +doc_function: index +purpose: "Navigation for the opt-in real-Codex human-gate E2E suite feature." +derived_from: + - ../../dna/governance.md + - brief.md + - design.md + - implementation-plan.md +status: active +audience: humans_and_agents +--- + +# FT-016: Real Codex human-gate E2E suite + +- [brief.md](brief.md) — scope and verification contract. +- [design.md](design.md) — selected local E2E design. +- [implementation-plan.md](implementation-plan.md) — execution and test plan. diff --git a/memory-bank/features/FT-016/brief.md b/memory-bank/features/FT-016/brief.md new file mode 100644 index 0000000..0e4101d --- /dev/null +++ b/memory-bank/features/FT-016/brief.md @@ -0,0 +1,82 @@ +--- +title: "FT-016: Real Codex human-gate E2E suite" +doc_kind: feature +doc_function: canonical +purpose: "Canonical problem and verification contract for an opt-in real-Codex human-gate smoke suite." +derived_from: + - ../../flows/feature-flow.md + - ../../engineering/testing-policy.md +status: active +delivery_status: in_progress +audience: humans_and_agents +must_not_define: + - implementation_sequence + - solution_space +--- + +# FT-016: Real Codex human-gate E2E suite + +## What + +### Problem + +The deterministic Bats suite uses a fake Codex executable and cannot validate compatibility with a locally installed real Codex CLI. + +### Outcome + +| Metric ID | Metric | Target | Measurement method | +| --- | --- | --- | --- | +| `MET-01` | Operator can run real-Codex smoke validation | One documented opt-in command per terminal state | Script output and saved state artifacts | + +### Scope + +- `REQ-01` Provide an explicit opt-in local suite for a real Codex `STATUS: DONE` run. +- `REQ-02` Provide a manually completable `STATUS: HUMAN_GATE` resume scenario. +- `REQ-03` Verify state artifacts and reject accidental fake-Codex execution. + +### Non-Scope + +- `NS-01` Do not run real Codex sessions in CI or `make test`. +- `NS-02` Do not alter the human-gate runtime behavior. + +### Constraints / Assumptions + +- `ASM-01` The operator has authenticated `gh` and a current `codex`; the private fixture repository owns the control issue. +- `CON-01` A real agent session can have side effects, so explicit opt-in and an isolated worktree parent are required. + +## Design Requirement Decision + +| Decision | Reason | Downstream owner | +| --- | --- | --- | +| `Design required: yes` | The suite introduces an operator-facing environment and safety contract. | `design.md` | + +## Verify + +### Exit Criteria + +- `EC-01` The `done` scenario validates a real `thread.started` event, state files, and `STATUS: DONE`. +- `EC-02` The `human-gate` scenario validates the reported resume command and `STATUS: HUMAN_GATE` after the operator exits resume. + +### Traceability matrix + +| Requirement ID | Acceptance refs | Checks | Evidence IDs | +| --- | --- | --- | --- | +| `REQ-01`, `REQ-03` | `SC-01` | `CHK-01` | `EVID-01` | +| `REQ-02`, `REQ-03` | `SC-02` | `CHK-02` | `EVID-02` | + +### Acceptance Scenarios + +- `SC-01` With explicit authorization and a real Codex CLI, the `done` scenario finishes successfully and reports preserved artifacts. +- `SC-02` With the same prerequisites, the `human-gate` scenario opens resume and validates its state after the operator exits. + +### Checks + +| Check ID | Covers | How to check | Expected result | Evidence path | +| --- | --- | --- | --- | --- | +| `CHK-01` | `SC-01` | `START_ISSUE_E2E=1 make e2e-human-gate` | `PASS` plus a state path | temporary fixture clone | +| `CHK-02` | `SC-02` | `START_ISSUE_E2E=1 test/e2e/human-gate.sh --scenario human-gate` | `PASS` after resume exits | temporary fixture clone | + +### Evidence + +- `EVID-01` Preserved `e2e.log`, events, last message, and thread-id from `CHK-01`. +- `EVID-02` The same artifacts plus reported resume command from `CHK-02`. diff --git a/memory-bank/features/FT-016/design.md b/memory-bank/features/FT-016/design.md new file mode 100644 index 0000000..e760b38 --- /dev/null +++ b/memory-bank/features/FT-016/design.md @@ -0,0 +1,44 @@ +--- +title: "FT-016: Design" +doc_kind: feature +doc_function: canonical +purpose: "Selected design for the opt-in real-Codex human-gate E2E suite." +derived_from: + - brief.md +status: active +audience: humans_and_agents +must_not_define: + - ft_016_scope + - ft_016_acceptance_criteria + - implementation_sequence +--- + +# FT-016: Design + +## C4 Applicability + +| C4 ID | Decision | Trigger / reason | Artifact | +| --- | --- | --- | --- | +| `C4-00` | not required | A local shell test script adds no runtime boundary. | none | + +## Selected Solution + +- `SOL-01` Use one opt-in Bash runner with `done` and `human-gate` scenarios. +- `SOL-02` Run the source executable from an independently selected target repository and create a unique temporary worktree parent. + +## Accepted Local Decisions + +- `SD-01` Preserve logs and run state on failure or explicit `START_ISSUE_E2E_KEEP=1`; clean disposable resources after success. +- `SD-02` Keep the suite outside CI and `make test`; the operator explicitly authorizes live execution with `START_ISSUE_E2E=1`. + +## Contracts + +| Contract ID | Input / Output | Semantics / Constraints | +| --- | --- | --- | +| `CTR-01` | `START_ISSUE_E2E`, issue, optional project dir | Authorization, target issue, and target repo must be explicit or safely defaulted. | +| `CTR-02` | Codex events and last message | Require `thread.started`, all three state files, and the expected terminal status. | + +## Failure Modes + +- `FM-01` Fake Codex is found on `PATH`; fail before issue work starts. +- `FM-02` Codex, `gh`, issue access, or terminal status is unavailable; fail with the preserved log path. diff --git a/memory-bank/features/FT-016/implementation-plan.md b/memory-bank/features/FT-016/implementation-plan.md new file mode 100644 index 0000000..007548b --- /dev/null +++ b/memory-bank/features/FT-016/implementation-plan.md @@ -0,0 +1,46 @@ +--- +title: "FT-016: Implementation Plan" +doc_kind: feature +doc_function: derived +purpose: "Execution plan for the real-Codex human-gate E2E suite." +derived_from: + - brief.md + - design.md +status: active +audience: humans_and_agents +must_not_define: + - ft_016_scope + - ft_016_selected_design + - ft_016_acceptance_criteria +--- + +# FT-016: Implementation Plan + +## Current State / Reference Points + +| Path | Role | +| --- | --- | +| `scripts/lib/start_issue/agent.sh` | Runtime state and terminal-status contract. | +| `test/helpers/fake-bin/codex` | Deterministic coverage that the new suite must avoid. | +| `Makefile` | Local test command entrypoints. | + +## Test Strategy + +| Surface | Coverage | Local command | CI | +| --- | --- | --- | --- | +| Runner syntax and integration | `bash -n`, shellcheck, documented target | `make test` | existing test job | +| Real Codex done/resume | Manual opt-in `SC-01`, `SC-02` | `CHK-01`, `CHK-02` | excluded by `NS-01` | + +## Work Order + +| Step ID | Implements | Goal | Verifies | +| --- | --- | --- | --- | +| `STEP-01` | `REQ-01` - `REQ-03`, `SOL-01` - `SOL-02` | Add the guarded E2E runner and Make target. | Script help and static checks. | +| `STEP-02` | `REQ-01` - `REQ-03`, `SD-01` - `SD-02` | Document prerequisites, commands, and retained artifacts. | README review and `make test`. | +| `STEP-03` | `SC-01`, `SC-02` | Offer manual acceptance commands. | `CHK-01`, `CHK-02`. | + +## Stop Conditions / Fallback + +| Stop ID | Trigger | Safe fallback | +| --- | --- | --- | +| `STOP-01` | Missing credentials, unavailable issue, or a live Codex failure | Keep deterministic Bats coverage as the release gate and inspect the preserved E2E log. | diff --git a/memory-bank/features/README.md b/memory-bank/features/README.md index e6bcd6a..0ec58d1 100644 --- a/memory-bank/features/README.md +++ b/memory-bank/features/README.md @@ -29,3 +29,8 @@ audience: humans_and_agents - Базовый формат: `FT-XXX/` - Вместо `XXX` используй идентификатор, принятый в проекте: issue id, ticket id или другой стабильный ключ - Один package = одна delivery-единица + +## Feature packages + +- [FT-016: Real Codex human-gate E2E suite](FT-016/README.md) + Opt-in local smoke suite that validates the real Codex human-gate path. diff --git a/memory-bank/ops/development.md b/memory-bank/ops/development.md index 3086fc7..048b9a3 100644 --- a/memory-bank/ops/development.md +++ b/memory-bank/ops/development.md @@ -39,6 +39,13 @@ Install locally from source: make install ``` +Run the opt-in real-Codex human-gate E2E smoke test with a usable issue in the +current repository: + +```bash +START_ISSUE_E2E=1 make e2e-human-gate +``` + ## Daily Commands ```bash diff --git a/test/e2e/human-gate.sh b/test/e2e/human-gate.sh new file mode 100755 index 0000000..4b2a7b5 --- /dev/null +++ b/test/e2e/human-gate.sh @@ -0,0 +1,124 @@ +#!/usr/bin/env bash + +# Opt-in smoke test for a real local Codex human-gate session. +set -euo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +fixture_repo="dapi/start-issue-e2e-fixture" +fixture_issue="1" +scenario="done" + +usage() { + cat <<'EOF' +Usage: START_ISSUE_E2E=1 test/e2e/human-gate.sh [--scenario done|human-gate] + +Runs start-issue against a real Codex CLI using the private fixture repository +dapi/start-issue-e2e-fixture and its control issue #1. It deletes the temporary +clone after a successful run; set START_ISSUE_E2E_KEEP=1 to retain it. The +HUMAN_GATE scenario opens Codex resume interactively; exit it to continue. +EOF +} + +fail() { + printf 'E2E human-gate: %s\n' "$*" >&2 + exit 1 +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --scenario) + scenario="${2:-}" + shift 2 + ;; + --help|-h) + usage + exit 0 + ;; + *) + fail "unknown argument: $1" + ;; + esac +done + +[[ "$scenario" == "done" || "$scenario" == "human-gate" ]] || fail "scenario must be done or human-gate" +[[ "${START_ISSUE_E2E:-}" == "1" ]] || fail "set START_ISSUE_E2E=1 to authorize a real Codex session" +start_issue_bin="${START_ISSUE_E2E_BINARY:-$repo_root/scripts/start-issue}" + +[[ -x "$start_issue_bin" ]] || fail "start-issue executable not found: $start_issue_bin" + +codex_path="$(command -v codex || true)" +[[ -n "$codex_path" ]] || fail "codex is not on PATH" +[[ "$codex_path" != "$repo_root/test/helpers/fake-bin/codex" ]] || fail "PATH resolves codex to the test fake" +gh auth status >/dev/null || fail "gh is not authenticated" + +if codex exec --help 2>&1 | grep -q -- '--ask-for-approval'; then + fail "installed codex exec still advertises --ask-for-approval; use a current Codex CLI" +fi + +fixture_root="$(mktemp -d "${TMPDIR:-/tmp}/start-issue-human-gate.XXXXXX")" +fixture_dir="$fixture_root/fixture" +worktree_parent="$fixture_root/worktrees" +log_path="$fixture_root/e2e.log" +expected_status="DONE" +if [[ "$scenario" == "human-gate" ]]; then + expected_status="HUMAN_GATE" +fi + +prompt=$(cat <&1 | tee "$log_path" +command_status=${PIPESTATUS[0]} +set -e + +[[ $command_status -eq 0 ]] || fail "start-issue exited $command_status; inspect $log_path" + +state_dir="$(awk '/^[[:space:]]*State dir: / { sub(/^[[:space:]]*State dir: /, ""); print; exit }' "$log_path")" +[[ -n "$state_dir" ]] || fail "state directory was not reported; inspect $log_path" +last_message_path="$state_dir/last-message.txt" +worktree_path="$(dirname "$(dirname "$(dirname "$state_dir")")")" + +[[ -f "$state_dir/events.jsonl" ]] || fail "events.jsonl is missing: $state_dir/events.jsonl" +[[ -f "$last_message_path" ]] || fail "last-message.txt is missing: $last_message_path" +[[ -f "$state_dir/thread-id" ]] || fail "thread-id is missing: $state_dir/thread-id" +jq -e 'select(.type == "thread.started") | .thread_id | strings' "$state_dir/events.jsonl" >/dev/null || \ + fail "thread.started event is missing from $state_dir/events.jsonl" +grep -Fx "STATUS: $expected_status" "$last_message_path" >/dev/null || \ + fail "expected STATUS: $expected_status in $last_message_path" + +if [[ "$scenario" == "human-gate" ]]; then + grep -F 'Resume command: codex resume --include-non-interactive ' "$log_path" >/dev/null || \ + fail "resume command was not reported; inspect $log_path" +fi + +printf 'PASS: real Codex human-gate %s scenario. State: %s\n' "$scenario" "$state_dir" +if [[ "${START_ISSUE_E2E_KEEP:-}" == "1" ]]; then + printf 'The temporary fixture is preserved at: %s\n' "$fixture_root" +else + git -C "$fixture_dir" worktree remove --force "$worktree_path" + rm -rf -- "$fixture_root" + printf 'Temporary fixture clone and worktree removed. Set START_ISSUE_E2E_KEEP=1 to preserve them.\n' +fi diff --git a/test/start_issue.bats b/test/start_issue.bats index 64c91f7..18657c1 100644 --- a/test/start_issue.bats +++ b/test/start_issue.bats @@ -279,6 +279,20 @@ install_fake_zellij_tab_status() { assert_output_contains ".start-issue/runs//events.jsonl" } +@test "real Codex human-gate E2E runner requires explicit authorization" { + run bash "$REPO_ROOT/test/e2e/human-gate.sh" + + assert_failure + assert_output_contains "set START_ISSUE_E2E=1" +} + +@test "real Codex human-gate E2E runner documents its scenarios" { + run bash "$REPO_ROOT/test/e2e/human-gate.sh" --help + + assert_success + assert_output_contains "--scenario done|human-gate" +} + @test "prompt improvement without issue prints explicit error" { run_start_issue --improve-prompt From 8795a899cd57a367911c9568d187111021606564 Mon Sep 17 00:00:00 2001 From: Danil Pismenny Date: Fri, 31 Jul 2026 00:46:44 +0500 Subject: [PATCH 2/6] Keep Codex E2E smoke prompt tool-free --- test/e2e/human-gate.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/e2e/human-gate.sh b/test/e2e/human-gate.sh index 4b2a7b5..e437b0e 100755 --- a/test/e2e/human-gate.sh +++ b/test/e2e/human-gate.sh @@ -66,8 +66,8 @@ fi prompt=$(cat < Date: Fri, 31 Jul 2026 00:49:45 +0500 Subject: [PATCH 3/6] Verify fixture E2E leaves no source changes --- README.md | 3 ++- README.ru.md | 3 ++- test/e2e/human-gate.sh | 3 +++ 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7130612..ac11140 100644 --- a/README.md +++ b/README.md @@ -281,7 +281,8 @@ START_ISSUE_E2E=1 make e2e-human-gate The script uses the private `dapi/start-issue-e2e-fixture` repository and its control issue, requires authenticated `gh`, rejects the fake Codex binary, and creates an isolated temporary clone and worktree parent. It deletes those after -success; set `START_ISSUE_E2E_KEEP=1` to retain them. To test interactive resume, run: +success; set `START_ISSUE_E2E_KEEP=1` to retain them. It also rejects any +fixture worktree change other than its `.start-issue` state. To test interactive resume, run: ```bash START_ISSUE_E2E=1 \ diff --git a/README.ru.md b/README.ru.md index de507f0..e6aa149 100644 --- a/README.ru.md +++ b/README.ru.md @@ -121,7 +121,8 @@ START_ISSUE_E2E=1 make e2e-human-gate Скрипт использует приватный репозиторий `dapi/start-issue-e2e-fixture` и его control issue, требует авторизованный `gh`, не допускает fake Codex и создаёт отдельный временный clone и worktree parent. После успеха они удаляются; чтобы -сохранить их, задайте `START_ISSUE_E2E_KEEP=1`. Для проверки interactive resume: +сохранить их, задайте `START_ISSUE_E2E_KEEP=1`. Скрипт также отклоняет любые +изменения fixture worktree, кроме своего `.start-issue` state. Для проверки interactive resume: ```bash START_ISSUE_E2E=1 \ diff --git a/test/e2e/human-gate.sh b/test/e2e/human-gate.sh index e437b0e..c6ffa7b 100755 --- a/test/e2e/human-gate.sh +++ b/test/e2e/human-gate.sh @@ -114,6 +114,9 @@ if [[ "$scenario" == "human-gate" ]]; then fail "resume command was not reported; inspect $log_path" fi +unexpected_changes="$(git -C "$worktree_path" status --porcelain | awk '$0 !~ /^\?\? \.start-issue\// { print }')" +[[ -z "$unexpected_changes" ]] || fail "fixture worktree has unexpected changes: $unexpected_changes" + printf 'PASS: real Codex human-gate %s scenario. State: %s\n' "$scenario" "$state_dir" if [[ "${START_ISSUE_E2E_KEEP:-}" == "1" ]]; then printf 'The temporary fixture is preserved at: %s\n' "$fixture_root" From d0540c3ee7d3cdac0f5f8c0336d77dd344f5024f Mon Sep 17 00:00:00 2001 From: Danil Pismenny Date: Fri, 31 Jul 2026 01:00:31 +0500 Subject: [PATCH 4/6] Document cmux E2E workflow and scenarios --- AGENTS.md | 25 +++++++++++++++++++++++++ README.md | 12 ++++++++++++ README.ru.md | 12 ++++++++++++ memory-bank/ops/development.md | 5 +++++ 4 files changed, 54 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 7b7bfef..eb9bd88 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -22,3 +22,28 @@ implementation-plan.md`. Existing legacy packages with `feature.md` and Run `make test` before handoff when changes affect code, tests, release logic, or memory-bank navigation. It includes shell syntax checks, shellcheck, memory-bank link audit, whitespace checks, and the Bats suite. + +## Live E2E Through cmux + +Run a live agent E2E only when explicitly requested. Use a **terminal tab in +the workspace for this repository**, not a new cmux workspace and not a tab in +an unrelated project. From an external shell, `cmux current-workspace` may not +identify the workspace visible to the user; first use `cmux tree --all`, find +the `start-issue` workspace and its pane, then create the surface explicitly: + +```bash +cmux new-surface --type terminal \ + --workspace workspace: \ + --pane pane: \ + --working-directory /absolute/path/to/start-issue-worktree \ + --focus true +cmux rename-tab --workspace workspace: \ + --surface surface: 'human-gate E2E' +cmux send --workspace workspace: \ + --surface surface: \ + 'START_ISSUE_E2E=1 START_ISSUE_E2E_KEEP=1 make e2e-human-gate\n' +``` + +Poll `cmux read-screen` until `PASS` or a terminal failure is visible. Report +the exact terminal status, thread id, and retained artifact path. Do not claim +the suite passed before the tab output contains its terminal result. diff --git a/README.md b/README.md index ac11140..1b35471 100644 --- a/README.md +++ b/README.md @@ -291,6 +291,18 @@ test/e2e/human-gate.sh --scenario human-gate Exit the resumed Codex session to let the script verify the artifacts. +#### Scenarios and checks + +| Scenario | Command | What it verifies | +| --- | --- | --- | +| `done` | `START_ISSUE_E2E=1 make e2e-human-gate` | A real Codex batch run emits `thread.started`, saves `thread-id`, `events.jsonl`, and `last-message.txt`, ends with `STATUS: DONE`, and leaves no fixture change other than `.start-issue` state. | +| `human-gate` | `START_ISSUE_E2E=1 test/e2e/human-gate.sh --scenario human-gate` | The same artifact and clean-worktree checks, plus the reported explicit `codex resume --include-non-interactive ` handoff. The operator exits the resumed interactive session before the script can finish. | + +Both scenarios verify authenticated `gh`, a real rather than fake Codex binary, +and a current `codex exec` interface without the obsolete +`--ask-for-approval` flag. They do not prove application behavior beyond this +human-gate protocol and are intentionally excluded from CI. + Configuration precedence: 1. Agent: CLI `--agent` / `--no-agent`, then project config, user config, `START_ISSUE_AGENT`, then built-in default `claude` diff --git a/README.ru.md b/README.ru.md index e6aa149..c9aeb68 100644 --- a/README.ru.md +++ b/README.ru.md @@ -131,6 +131,18 @@ test/e2e/human-gate.sh --scenario human-gate Выйдите из возобновлённой Codex-сессии, после чего скрипт проверит артефакты. +#### Сценарии и проверки + +| Сценарий | Команда | Что проверяется | +| --- | --- | --- | +| `done` | `START_ISSUE_E2E=1 make e2e-human-gate` | Реальный Codex batch run выдаёт `thread.started`, сохраняет `thread-id`, `events.jsonl` и `last-message.txt`, заканчивается `STATUS: DONE` и не меняет fixture worktree за пределами `.start-issue` state. | +| `human-gate` | `START_ISSUE_E2E=1 test/e2e/human-gate.sh --scenario human-gate` | Те же проверки артефактов и чистоты worktree, а также явный handoff `codex resume --include-non-interactive `. Перед завершением скрипта оператор выходит из возобновлённой interactive session. | + +Оба сценария проверяют авторизованный `gh`, реальный, а не fake Codex binary, и +актуальный интерфейс `codex exec` без устаревшего флага +`--ask-for-approval`. Они не доказывают поведение приложения за пределами +human-gate protocol и намеренно не входят в CI. + ## Использование ```bash diff --git a/memory-bank/ops/development.md b/memory-bank/ops/development.md index 048b9a3..02139a1 100644 --- a/memory-bank/ops/development.md +++ b/memory-bank/ops/development.md @@ -46,6 +46,11 @@ current repository: START_ISSUE_E2E=1 make e2e-human-gate ``` +When a live E2E must be visible in cmux, follow the canonical cmux-tab +procedure in [`../../AGENTS.md`](../../AGENTS.md): find the `start-issue` +workspace using `cmux tree --all`, create a terminal surface in its active pane, +and run the command with `START_ISSUE_E2E_KEEP=1`. + ## Daily Commands ```bash From 95cc23d9499a935257d2816d282278ada47df08b Mon Sep 17 00:00:00 2001 From: Danil Pismenny Date: Fri, 31 Jul 2026 01:11:21 +0500 Subject: [PATCH 5/6] Use cmux caller context for E2E tabs --- AGENTS.md | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index eb9bd88..2aea3cc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,10 +26,22 @@ memory-bank link audit, whitespace checks, and the Bats suite. ## Live E2E Through cmux Run a live agent E2E only when explicitly requested. Use a **terminal tab in -the workspace for this repository**, not a new cmux workspace and not a tab in -an unrelated project. From an external shell, `cmux current-workspace` may not -identify the workspace visible to the user; first use `cmux tree --all`, find -the `start-issue` workspace and its pane, then create the surface explicitly: +the workspace that invoked the agent**, not a new cmux workspace and not a tab +in an unrelated project. Resolve that workspace with `cmux identify`, which +returns both the invocation `caller` and the global `focused` workspace. Use +only `caller.workspace_ref` and `caller.pane_ref`; do not use +`cmux current-workspace`, which reports global focus and can be unrelated. + +```bash +caller_context="$(cmux identify)" +workspace="$(printf '%s' "$caller_context" | jq -r '.caller.workspace_ref // empty')" +pane="$(printf '%s' "$caller_context" | jq -r '.caller.pane_ref // empty')" +test -n "$workspace" && test -n "$pane" +``` + +If `caller` is `null`, do not guess from a workspace name or global focus; ask +for an explicit cmux workspace/pane target. With a resolved caller, create the +surface explicitly: ```bash cmux new-surface --type terminal \ From f8f13c401f54f5fb8296bc7c95b48ae72bf994bc Mon Sep 17 00:00:00 2001 From: Danil Pismenny Date: Fri, 31 Jul 2026 01:34:46 +0500 Subject: [PATCH 6/6] Require real Codex interface for E2E --- README.md | 5 +++-- README.ru.md | 7 ++++--- test/e2e/human-gate.sh | 8 +++++++- 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 1b35471..620249b 100644 --- a/README.md +++ b/README.md @@ -299,8 +299,9 @@ Exit the resumed Codex session to let the script verify the artifacts. | `human-gate` | `START_ISSUE_E2E=1 test/e2e/human-gate.sh --scenario human-gate` | The same artifact and clean-worktree checks, plus the reported explicit `codex resume --include-non-interactive ` handoff. The operator exits the resumed interactive session before the script can finish. | Both scenarios verify authenticated `gh`, a real rather than fake Codex binary, -and a current `codex exec` interface without the obsolete -`--ask-for-approval` flag. They do not prove application behavior beyond this +and the required `codex exec` help interface (`--output-last-message`, without +the obsolete `--ask-for-approval` flag). The selected Codex executable is +printed in the test output. They do not prove application behavior beyond this human-gate protocol and are intentionally excluded from CI. Configuration precedence: diff --git a/README.ru.md b/README.ru.md index c9aeb68..d362085 100644 --- a/README.ru.md +++ b/README.ru.md @@ -139,9 +139,10 @@ test/e2e/human-gate.sh --scenario human-gate | `human-gate` | `START_ISSUE_E2E=1 test/e2e/human-gate.sh --scenario human-gate` | Те же проверки артефактов и чистоты worktree, а также явный handoff `codex resume --include-non-interactive `. Перед завершением скрипта оператор выходит из возобновлённой interactive session. | Оба сценария проверяют авторизованный `gh`, реальный, а не fake Codex binary, и -актуальный интерфейс `codex exec` без устаревшего флага -`--ask-for-approval`. Они не доказывают поведение приложения за пределами -human-gate protocol и намеренно не входят в CI. +обязательный интерфейс справки `codex exec` (`--output-last-message`, без +устаревшего флага `--ask-for-approval`). Выбранный Codex executable печатается +в test output. Они не доказывают поведение приложения за пределами human-gate +protocol и намеренно не входят в CI. ## Использование diff --git a/test/e2e/human-gate.sh b/test/e2e/human-gate.sh index c6ffa7b..7ebccbb 100755 --- a/test/e2e/human-gate.sh +++ b/test/e2e/human-gate.sh @@ -51,7 +51,12 @@ codex_path="$(command -v codex || true)" [[ "$codex_path" != "$repo_root/test/helpers/fake-bin/codex" ]] || fail "PATH resolves codex to the test fake" gh auth status >/dev/null || fail "gh is not authenticated" -if codex exec --help 2>&1 | grep -q -- '--ask-for-approval'; then +codex_exec_help="$(codex exec --help 2>&1)" || fail "codex exec --help failed" +printf '%s' "$codex_exec_help" | grep -Fq 'Run Codex non-interactively' || \ + fail "resolved codex does not expose the real Codex exec interface" +printf '%s' "$codex_exec_help" | grep -Fq -- '--output-last-message' || \ + fail "resolved codex does not support --output-last-message" +if printf '%s' "$codex_exec_help" | grep -q -- '--ask-for-approval'; then fail "installed codex exec still advertises --ask-for-approval; use a current Codex CLI" fi @@ -73,6 +78,7 @@ EOF ) printf 'E2E human-gate scenario: %s\n' "$scenario" +printf 'Codex executable: %s\n' "$codex_path" printf 'Fixture repository: %s (issue #%s)\n' "$fixture_repo" "$fixture_issue" printf 'Worktree parent: %s\n' "$worktree_parent" printf 'Log: %s\n' "$log_path"