Add the acceptance kit for the secrets path - #41
Closed
czpython wants to merge 1 commit into
Closed
Conversation
czpython
force-pushed
the
dru-462-acceptance-kit
branch
2 times, most recently
from
September 6, 2026 17:06
d5784ba to
5f71a52
Compare
czpython
force-pushed
the
dru-401-per-box-lifecycle
branch
2 times, most recently
from
September 7, 2026 06:01
2d75103 to
f3373ed
Compare
The first acceptance kit tested the model call only, and that is how GitHub was missed. This kit covers what an agent does, from inside the box over SSH: the placeholders in a plain session, claude -p, gh on a private repository, git clone and a push of a throwaway branch, the placeholder sent straight to Anthropic, a wrong placeholder, the issuer failing, a restart of the exchange, and a clean delete. One run covers one provider, with an issuer-backed anthropic secret and an issuer-backed github secret. The anthropic value lives 70 seconds in the run, so the run sees a refresh and an outage. The github value lives an hour and carries the counts of one fetch per secret. The dummy issuer answers from files the operator writes, behind a bearer from the environment, and takes two control paths: a fetch count per service and a fail switch. A host carries a one hour lease, so a run that dies leaves no permanent host. Git ignores the token files. The kit lives in api-tests/acceptance, run by hand against a deployment like the black-box suite, and not in CI, since it needs a subscription token and an installation token. The runbook says what to set and what passes.
czpython
force-pushed
the
dru-462-acceptance-kit
branch
from
September 7, 2026 06:09
5f71a52 to
1427d9e
Compare
Owner
Author
|
Closed unmerged. The kit lives in the operator's local checkout, untracked, not in the repository. The acceptance results on docker, exe, and docker-sbx stay in this PR's body. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
The acceptance kit for the secrets path moves into the repository, in
api-tests/acceptance/.check.pyruns one provider: it creates a host with an issuer-backedanthropicsecret and an issuer-backedgithubsecret, waits for it, and works from inside the box over SSH. The placeholders in a plain session,claude -p,gh apiandgh pr liston a private repository,git cloneand a push of a throwaway branch and its delete, the placeholder sent straight to Anthropic, a wrong placeholder, the issuer failing, a restart of the exchange, and a clean delete. On docker it restarts the box too. On docker-sbx it checks sbx's store and the value files as well.issuer.pyis the dummy issuer. It answers from files the operator writes, behind a bearer, and takes two control paths: a fetch count and a fail switch. Standard library only, like the check.README.mdis the runbook: what to set, how to start the issuer, and what passes. The token files are ignored by git.Where this differs from the ticket
{"value", "expires_at"}and be told to fail. The stub does both, and it also counts its fetches, so the check needs no access to a log on another host. The check drives both through the issuer's control URL.Names
New names, open to change. In
check.py:get_setting,call_api,call_issuer,count_fetches,run_on_host,run_in_box,call_claude,get_status,make_entry,PROVIDER_NEEDS_VALUE. Inissuer.py:Issuer,mint,is_authorized,answer.Gates
uv run ruff check,uv run ruff format --check,uv run pyright, anduv run pytestare green.Acceptance
The kit itself, on this commit, with the subscription token and a one-hour installation token for a private repository behind the dummy issuer:
Every run ended with a clean delete, and the proxy log carried no credential. No manual step per box: the operator sets the environment once and the check does the rest.
Review
The adversarial review reported seven findings. All seven are applied:
ISSUER_BEARERnow.docker restartcould pass the restart step. Its exit code counts now.HOST_ACTIVE_TIMEOUTdid not apply toPOST /hosts, which provisions before it answers. It does now.