Skip to content

Add the acceptance kit for the secrets path - #41

Closed
czpython wants to merge 1 commit into
mainfrom
dru-462-acceptance-kit
Closed

Add the acceptance kit for the secrets path#41
czpython wants to merge 1 commit into
mainfrom
dru-462-acceptance-kit

Conversation

@czpython

@czpython czpython commented Sep 6, 2026

Copy link
Copy Markdown
Owner

What changes

The acceptance kit for the secrets path moves into the repository, in api-tests/acceptance/.

  • check.py runs one provider: it creates a host with an issuer-backed anthropic secret and an issuer-backed github secret, waits for it, and works from inside the box over SSH. The placeholders in a plain session, claude -p, gh api and gh pr list on a private repository, git clone and a push of a throwaway branch and its delete, the placeholder sent straight to Anthropic, a wrong placeholder, the issuer failing, a restart of the exchange, and a clean delete. On docker it restarts the box too. On docker-sbx it checks sbx's store and the value files as well.
  • issuer.py is the dummy issuer. It answers from files the operator writes, behind a bearer, and takes two control paths: a fetch count and a fail switch. Standard library only, like the check.
  • README.md is the runbook: what to set, how to start the issuer, and what passes. The token files are ignored by git.
  • The deploy doc's Verify section and the layout in AGENTS.md point at the kit.

Where this differs from the ticket

  • The ticket has the issuer serve {"value", "expires_at"} and be told to fail. The stub does both, and it also counts its fetches, so the check needs no access to a log on another host. The check drives both through the issuer's control URL.
  • A wrong placeholder is refused with 401 by Anthropic on docker-sbx, since nothing swaps it, where the exchange answers 403 on the proxy providers. The placeholder sent straight to Anthropic is refused with 401 on every provider, docker-sbx included.
  • The anthropic secret lives 70 seconds in the run and the github secret an hour. The short one lets the run see a refresh and an outage. The long one carries the counts: one fetch on first use, one more after a restart of the exchange.

Names

New names, open to change. In check.py: get_setting, call_api, call_issuer, count_fetches, run_on_host, run_in_box, call_claude, get_status, make_entry, PROVIDER_NEEDS_VALUE. In issuer.py: Issuer, mint, is_authorized, answer.

Gates

uv run ruff check, uv run ruff format --check, uv run pyright, and uv run pytest are green.

Acceptance

The kit itself, on this commit, with the subscription token and a one-hour installation token for a private repository behind the dummy issuer:

  • docker, on the production bed through our proxy: 21 of 21.
  • exe, on the production bed through our proxy over the tailnet, from the Mac: 20 of 20.
  • docker-sbx, on the KVM bed with sbx's own secrets: 25 of 25.

Every run ended with a clean delete, and the proxy log carried no credential. No manual step per box: the operator sets the environment once and the check does the rest.

Review

The adversarial review reported seven findings. All seven are applied:

  • The stub took its bearer on the command line, where every process could read it. It reads ISSUER_BEARER now.
  • A mint path could name a file outside the stub's directory. A service name is plain letters, digits, and dashes now.
  • A lost creation response could leave a permanent host, since the check asked for no lease. The host carries a one hour lease, and the cleanup block starts right after creation.
  • The outage step proved nothing, since the value was still fresh while the issuer failed. The anthropic secret lives 70 seconds now, the stub counts per service, and the step requires a failed attempt while the box keeps working. The github secret keeps an hour and carries the counts of one fetch per secret and of the restart's cost.
  • A failed docker restart could pass the restart step. Its exit code counts now.
  • A secret scoped to another sandbox failed the scope check. Only a global row fails it now.
  • HOST_ACTIVE_TIMEOUT did not apply to POST /hosts, which provisions before it answers. It does now.

@czpython
czpython force-pushed the dru-462-acceptance-kit branch 2 times, most recently from d5784ba to 5f71a52 Compare September 6, 2026 17:06
@czpython
czpython force-pushed the dru-401-per-box-lifecycle branch 2 times, most recently from 2d75103 to f3373ed Compare September 7, 2026 06:01
Base automatically changed from dru-401-per-box-lifecycle to main September 7, 2026 06:03
The first acceptance kit tested the model call only, and that is how
GitHub was missed. This kit covers what an agent does, from inside the
box over SSH: the placeholders in a plain session, claude -p, gh on a
private repository, git clone and a push of a throwaway branch, the
placeholder sent straight to Anthropic, a wrong placeholder, the issuer
failing, a restart of the exchange, and a clean delete. One run covers
one provider, with an issuer-backed anthropic secret and an issuer-backed
github secret. The anthropic value lives 70 seconds in the run, so the
run sees a refresh and an outage. The github value lives an hour and
carries the counts of one fetch per secret.

The dummy issuer answers from files the operator writes, behind a bearer
from the environment, and takes two control paths: a fetch count per
service and a fail switch. A host carries a one hour lease, so a run that
dies leaves no permanent host. Git ignores the token files. The kit
lives in api-tests/acceptance, run by hand against a deployment like the
black-box suite, and not in CI, since it needs a subscription token and
an installation token. The runbook says what to set and what passes.
@czpython
czpython force-pushed the dru-462-acceptance-kit branch from 5f71a52 to 1427d9e Compare September 7, 2026 06:09
@czpython

czpython commented Sep 7, 2026

Copy link
Copy Markdown
Owner Author

Closed unmerged. The kit lives in the operator's local checkout, untracked, not in the repository. The acceptance results on docker, exe, and docker-sbx stay in this PR's body.

@czpython czpython closed this Sep 7, 2026
@czpython
czpython deleted the dru-462-acceptance-kit branch September 7, 2026 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant