Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,40 @@ implementation-plan.md`. Existing legacy packages with `feature.md` and
Run `make test` before handoff when changes affect code, tests, release logic,
or memory-bank navigation. It includes shell syntax checks, shellcheck,
memory-bank link audit, whitespace checks, and the Bats suite.

## Live E2E Through cmux

Run a live agent E2E only when explicitly requested. Use a **terminal tab in
the workspace that invoked the agent**, not a new cmux workspace and not a tab
in an unrelated project. Resolve that workspace with `cmux identify`, which
returns both the invocation `caller` and the global `focused` workspace. Use
only `caller.workspace_ref` and `caller.pane_ref`; do not use
`cmux current-workspace`, which reports global focus and can be unrelated.

```bash
caller_context="$(cmux identify)"
workspace="$(printf '%s' "$caller_context" | jq -r '.caller.workspace_ref // empty')"
pane="$(printf '%s' "$caller_context" | jq -r '.caller.pane_ref // empty')"
test -n "$workspace" && test -n "$pane"
```

If `caller` is `null`, do not guess from a workspace name or global focus; ask
for an explicit cmux workspace/pane target. With a resolved caller, create the
surface explicitly:

```bash
cmux new-surface --type terminal \
--workspace workspace:<start-issue-workspace> \
--pane pane:<start-issue-pane> \
--working-directory /absolute/path/to/start-issue-worktree \
--focus true
cmux rename-tab --workspace workspace:<start-issue-workspace> \
--surface surface:<new-surface> 'human-gate E2E'
cmux send --workspace workspace:<start-issue-workspace> \
--surface surface:<new-surface> \
'START_ISSUE_E2E=1 START_ISSUE_E2E_KEEP=1 make e2e-human-gate\n'
```

Poll `cmux read-screen` until `PASS` or a terminal failure is visible. Report
the exact terminal status, thread id, and retained artifact path. Do not claim
the suite passed before the tab output contains its terminal result.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ This project follows Semantic Versioning.

## [Unreleased]

### Added

- Added an opt-in local E2E smoke suite for the real Codex human-gate workflow.

## [1.13.3] - 2026-07-30

### Fixed
Expand Down
7 changes: 5 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: build install uninstall test print-version bump-patch bump-minor bump-major release-patch release-minor release-major
.PHONY: build install uninstall test e2e-human-gate print-version bump-patch bump-minor bump-major release-patch release-minor release-major

PREFIX ?= $(HOME)/.local
BINDIR ?= $(PREFIX)/bin
Expand Down Expand Up @@ -27,11 +27,14 @@ uninstall:

test:
bash -n scripts/start-issue
shellcheck install.sh scripts/start-issue scripts/build-start-issue scripts/bump-version scripts/prepare-release scripts/lib/start_issue/*.sh
shellcheck install.sh scripts/start-issue scripts/build-start-issue scripts/bump-version scripts/prepare-release scripts/lib/start_issue/*.sh test/e2e/*.sh
python3 scripts/check_memory_bank_index.py --max-depth 4
git diff --check
bats test

e2e-human-gate:
@bash test/e2e/human-gate.sh

print-version:
@awk -F'"' '/^VERSION="/ { print $$2; exit }' scripts/start-issue

Expand Down
35 changes: 35 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,41 @@ State files:
<worktree>/.start-issue/runs/<timestamp>/thread-id
```

### Local real-Codex E2E smoke test

The normal Bats suite uses a fake Codex CLI. To exercise the real local Codex
CLI, run this opt-in test from a `start-issue` checkout:

```bash
START_ISSUE_E2E=1 make e2e-human-gate
```

The script uses the private `dapi/start-issue-e2e-fixture` repository and its
control issue, requires authenticated `gh`, rejects the fake Codex binary, and
creates an isolated temporary clone and worktree parent. It deletes those after
success; set `START_ISSUE_E2E_KEEP=1` to retain them. It also rejects any
fixture worktree change other than its `.start-issue` state. To test interactive resume, run:

```bash
START_ISSUE_E2E=1 \
test/e2e/human-gate.sh --scenario human-gate
```

Exit the resumed Codex session to let the script verify the artifacts.

#### Scenarios and checks

| Scenario | Command | What it verifies |
| --- | --- | --- |
| `done` | `START_ISSUE_E2E=1 make e2e-human-gate` | A real Codex batch run emits `thread.started`, saves `thread-id`, `events.jsonl`, and `last-message.txt`, ends with `STATUS: DONE`, and leaves no fixture change other than `.start-issue` state. |
| `human-gate` | `START_ISSUE_E2E=1 test/e2e/human-gate.sh --scenario human-gate` | The same artifact and clean-worktree checks, plus the reported explicit `codex resume --include-non-interactive <thread_id>` handoff. The operator exits the resumed interactive session before the script can finish. |

Both scenarios verify authenticated `gh`, a real rather than fake Codex binary,
and the required `codex exec` help interface (`--output-last-message`, without
the obsolete `--ask-for-approval` flag). The selected Codex executable is
printed in the test output. They do not prove application behavior beyond this
human-gate protocol and are intentionally excluded from CI.

Configuration precedence:

1. Agent: CLI `--agent` / `--no-agent`, then project config, user config, `START_ISSUE_AGENT`, then built-in default `claude`
Expand Down
35 changes: 35 additions & 0 deletions README.ru.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,41 @@ State files:
<worktree>/.start-issue/runs/<timestamp>/thread-id
```

### Локальный E2E smoke test с реальным Codex

Обычный Bats-набор использует fake Codex CLI. Для проверки с реальным локальным
Codex из checkout `start-issue` выполните opt-in команду:

```bash
START_ISSUE_E2E=1 make e2e-human-gate
```

Скрипт использует приватный репозиторий `dapi/start-issue-e2e-fixture` и его
control issue, требует авторизованный `gh`, не допускает fake Codex и создаёт
отдельный временный clone и worktree parent. После успеха они удаляются; чтобы
сохранить их, задайте `START_ISSUE_E2E_KEEP=1`. Скрипт также отклоняет любые
изменения fixture worktree, кроме своего `.start-issue` state. Для проверки interactive resume:

```bash
START_ISSUE_E2E=1 \
test/e2e/human-gate.sh --scenario human-gate
```

Выйдите из возобновлённой Codex-сессии, после чего скрипт проверит артефакты.

#### Сценарии и проверки

| Сценарий | Команда | Что проверяется |
| --- | --- | --- |
| `done` | `START_ISSUE_E2E=1 make e2e-human-gate` | Реальный Codex batch run выдаёт `thread.started`, сохраняет `thread-id`, `events.jsonl` и `last-message.txt`, заканчивается `STATUS: DONE` и не меняет fixture worktree за пределами `.start-issue` state. |
| `human-gate` | `START_ISSUE_E2E=1 test/e2e/human-gate.sh --scenario human-gate` | Те же проверки артефактов и чистоты worktree, а также явный handoff `codex resume --include-non-interactive <thread_id>`. Перед завершением скрипта оператор выходит из возобновлённой interactive session. |

Оба сценария проверяют авторизованный `gh`, реальный, а не fake Codex binary, и
обязательный интерфейс справки `codex exec` (`--output-last-message`, без
устаревшего флага `--ask-for-approval`). Выбранный Codex executable печатается
в test output. Они не доказывают поведение приложения за пределами human-gate
protocol и намеренно не входят в CI.

## Использование

```bash
Expand Down
7 changes: 6 additions & 1 deletion memory-bank/engineering/testing-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ make test
`make test` runs:

1. `bash -n scripts/start-issue`
2. `shellcheck install.sh scripts/start-issue scripts/build-start-issue scripts/bump-version scripts/prepare-release scripts/lib/start_issue/*.sh`
2. `shellcheck install.sh scripts/start-issue scripts/build-start-issue scripts/bump-version scripts/prepare-release scripts/lib/start_issue/*.sh test/e2e/*.sh`
3. `python3 scripts/check_memory_bank_index.py --max-depth 4`
4. `git diff --check`
5. `bats test`
Expand All @@ -44,6 +44,7 @@ make test
- Shell syntax: `bash -n`
- Static analysis: `shellcheck`
- Behavior/regression tests: Bats under `test/`
- Opt-in real-agent E2E smoke tests: scripts under `test/e2e/`, run manually and never in CI
- Memory-bank navigation: `scripts/check_memory_bank_index.py`
- Whitespace/conflict-marker check: `git diff --check`

Expand Down Expand Up @@ -88,6 +89,10 @@ Manual-only verification is acceptable only for:
- visual/manual review of long help text or docs when no stable assertion is
useful.

Real-agent E2E scripts must require an explicit opt-in environment variable,
avoid fake agent binaries, preserve diagnostic artifacts, and stay outside
`make test` and CI.

For each manual-only gap, record the reason and the manual procedure in the
feature plan or final handoff.

Expand Down
19 changes: 19 additions & 0 deletions memory-bank/features/FT-016/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
title: "FT-016: Real Codex human-gate E2E suite"
doc_kind: feature
doc_function: index
purpose: "Navigation for the opt-in real-Codex human-gate E2E suite feature."
derived_from:
- ../../dna/governance.md
- brief.md
- design.md
- implementation-plan.md
status: active
audience: humans_and_agents
---

# FT-016: Real Codex human-gate E2E suite

- [brief.md](brief.md) — scope and verification contract.
- [design.md](design.md) — selected local E2E design.
- [implementation-plan.md](implementation-plan.md) — execution and test plan.
82 changes: 82 additions & 0 deletions memory-bank/features/FT-016/brief.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
---
title: "FT-016: Real Codex human-gate E2E suite"
doc_kind: feature
doc_function: canonical
purpose: "Canonical problem and verification contract for an opt-in real-Codex human-gate smoke suite."
derived_from:
- ../../flows/feature-flow.md
- ../../engineering/testing-policy.md
status: active
delivery_status: in_progress
audience: humans_and_agents
must_not_define:
- implementation_sequence
- solution_space
---

# FT-016: Real Codex human-gate E2E suite

## What

### Problem

The deterministic Bats suite uses a fake Codex executable and cannot validate compatibility with a locally installed real Codex CLI.

### Outcome

| Metric ID | Metric | Target | Measurement method |
| --- | --- | --- | --- |
| `MET-01` | Operator can run real-Codex smoke validation | One documented opt-in command per terminal state | Script output and saved state artifacts |

### Scope

- `REQ-01` Provide an explicit opt-in local suite for a real Codex `STATUS: DONE` run.
- `REQ-02` Provide a manually completable `STATUS: HUMAN_GATE` resume scenario.
- `REQ-03` Verify state artifacts and reject accidental fake-Codex execution.

### Non-Scope

- `NS-01` Do not run real Codex sessions in CI or `make test`.
- `NS-02` Do not alter the human-gate runtime behavior.

### Constraints / Assumptions

- `ASM-01` The operator has authenticated `gh` and a current `codex`; the private fixture repository owns the control issue.
- `CON-01` A real agent session can have side effects, so explicit opt-in and an isolated worktree parent are required.

## Design Requirement Decision

| Decision | Reason | Downstream owner |
| --- | --- | --- |
| `Design required: yes` | The suite introduces an operator-facing environment and safety contract. | `design.md` |

## Verify

### Exit Criteria

- `EC-01` The `done` scenario validates a real `thread.started` event, state files, and `STATUS: DONE`.
- `EC-02` The `human-gate` scenario validates the reported resume command and `STATUS: HUMAN_GATE` after the operator exits resume.

### Traceability matrix

| Requirement ID | Acceptance refs | Checks | Evidence IDs |
| --- | --- | --- | --- |
| `REQ-01`, `REQ-03` | `SC-01` | `CHK-01` | `EVID-01` |
| `REQ-02`, `REQ-03` | `SC-02` | `CHK-02` | `EVID-02` |

### Acceptance Scenarios

- `SC-01` With explicit authorization and a real Codex CLI, the `done` scenario finishes successfully and reports preserved artifacts.
- `SC-02` With the same prerequisites, the `human-gate` scenario opens resume and validates its state after the operator exits.

### Checks

| Check ID | Covers | How to check | Expected result | Evidence path |
| --- | --- | --- | --- | --- |
| `CHK-01` | `SC-01` | `START_ISSUE_E2E=1 make e2e-human-gate` | `PASS` plus a state path | temporary fixture clone |
| `CHK-02` | `SC-02` | `START_ISSUE_E2E=1 test/e2e/human-gate.sh --scenario human-gate` | `PASS` after resume exits | temporary fixture clone |

### Evidence

- `EVID-01` Preserved `e2e.log`, events, last message, and thread-id from `CHK-01`.
- `EVID-02` The same artifacts plus reported resume command from `CHK-02`.
44 changes: 44 additions & 0 deletions memory-bank/features/FT-016/design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
title: "FT-016: Design"
doc_kind: feature
doc_function: canonical
purpose: "Selected design for the opt-in real-Codex human-gate E2E suite."
derived_from:
- brief.md
status: active
audience: humans_and_agents
must_not_define:
- ft_016_scope
- ft_016_acceptance_criteria
- implementation_sequence
---

# FT-016: Design

## C4 Applicability

| C4 ID | Decision | Trigger / reason | Artifact |
| --- | --- | --- | --- |
| `C4-00` | not required | A local shell test script adds no runtime boundary. | none |

## Selected Solution

- `SOL-01` Use one opt-in Bash runner with `done` and `human-gate` scenarios.
- `SOL-02` Run the source executable from an independently selected target repository and create a unique temporary worktree parent.

## Accepted Local Decisions

- `SD-01` Preserve logs and run state on failure or explicit `START_ISSUE_E2E_KEEP=1`; clean disposable resources after success.
- `SD-02` Keep the suite outside CI and `make test`; the operator explicitly authorizes live execution with `START_ISSUE_E2E=1`.

## Contracts

| Contract ID | Input / Output | Semantics / Constraints |
| --- | --- | --- |
| `CTR-01` | `START_ISSUE_E2E`, issue, optional project dir | Authorization, target issue, and target repo must be explicit or safely defaulted. |
| `CTR-02` | Codex events and last message | Require `thread.started`, all three state files, and the expected terminal status. |

## Failure Modes

- `FM-01` Fake Codex is found on `PATH`; fail before issue work starts.
- `FM-02` Codex, `gh`, issue access, or terminal status is unavailable; fail with the preserved log path.
46 changes: 46 additions & 0 deletions memory-bank/features/FT-016/implementation-plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
---
title: "FT-016: Implementation Plan"
doc_kind: feature
doc_function: derived
purpose: "Execution plan for the real-Codex human-gate E2E suite."
derived_from:
- brief.md
- design.md
status: active
audience: humans_and_agents
must_not_define:
- ft_016_scope
- ft_016_selected_design
- ft_016_acceptance_criteria
---

# FT-016: Implementation Plan

## Current State / Reference Points

| Path | Role |
| --- | --- |
| `scripts/lib/start_issue/agent.sh` | Runtime state and terminal-status contract. |
| `test/helpers/fake-bin/codex` | Deterministic coverage that the new suite must avoid. |
| `Makefile` | Local test command entrypoints. |

## Test Strategy

| Surface | Coverage | Local command | CI |
| --- | --- | --- | --- |
| Runner syntax and integration | `bash -n`, shellcheck, documented target | `make test` | existing test job |
| Real Codex done/resume | Manual opt-in `SC-01`, `SC-02` | `CHK-01`, `CHK-02` | excluded by `NS-01` |

## Work Order

| Step ID | Implements | Goal | Verifies |
| --- | --- | --- | --- |
| `STEP-01` | `REQ-01` - `REQ-03`, `SOL-01` - `SOL-02` | Add the guarded E2E runner and Make target. | Script help and static checks. |
| `STEP-02` | `REQ-01` - `REQ-03`, `SD-01` - `SD-02` | Document prerequisites, commands, and retained artifacts. | README review and `make test`. |
| `STEP-03` | `SC-01`, `SC-02` | Offer manual acceptance commands. | `CHK-01`, `CHK-02`. |

## Stop Conditions / Fallback

| Stop ID | Trigger | Safe fallback |
| --- | --- | --- |
| `STOP-01` | Missing credentials, unavailable issue, or a live Codex failure | Keep deterministic Bats coverage as the release gate and inspect the preserved E2E log. |
5 changes: 5 additions & 0 deletions memory-bank/features/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,8 @@ audience: humans_and_agents
- Базовый формат: `FT-XXX/`
- Вместо `XXX` используй идентификатор, принятый в проекте: issue id, ticket id или другой стабильный ключ
- Один package = одна delivery-единица

## Feature packages

- [FT-016: Real Codex human-gate E2E suite](FT-016/README.md)
Opt-in local smoke suite that validates the real Codex human-gate path.
Loading
Loading