Skip to content

docs: add the platform repo to the testing comparison (F10, F11) - #164

Merged
jcschaff merged 1 commit into
mainfrom
docs/testing-plan-platform
Sep 11, 2026
Merged

docs: add the platform repo to the testing comparison (F10, F11)#164
jcschaff merged 1 commit into
mainfrom
docs/testing-plan-platform

Conversation

@jcschaff

Copy link
Copy Markdown
Contributor

Summary

platform was missing from docs/plan-testing.md and is the most useful comparison in it: the sibling service, same problem, different choices. Figures from main@75dc795 (2026-09-04); PR biosimulations/platform#107 adds practices main lacks, recorded separately.

Two findings that only appear by comparing

F10 — the two repos fail at gating in opposite directions, and both are wrong.

compose-api platform
Gate SLURM_SUBMIT_KEY_PATH absent in CI none enforced
Effect 16 of 25 tests never run 3 tests hit third-party services on every PR, in a required check

One of those three carries no marker and no skipif, so it survives even the documented -m "not integration". lefthook.yml runs no tests at all, so the drift is never discovered locally. Shared cause: in neither repo does the marker decide where the test runs — which is the argument for F.c. Worth noting platform's own smoke.yaml header states the right policy ("minutes and flaky external state"), which its backend suite then does not follow.

F11 — config shadowing silently disables settings in three repos. pytest takes the first pytest.ini and never merges [tool.pytest.ini_options]. platform, sms-api and biosim-client have both. platform's dead block contains python_files = "main.py" — if that pytest.ini were ever removed as redundant, collection would drop to zero tests with a green suite. This repo has only the pyproject block and is unaffected.

Existing findings refined, not duplicated

  • F2 gains platform as the other end of the same failure: no codecov.yaml at all, no pytest-cov in the lockfile, nothing measured — and two documents advertising a --cov command that cannot run. Four repos carry an unenforced 90% target; the fifth carries none. Neither end produces a number anyone acts on.
  • F3 gains platform as proof branch protection is achievable here: two required contexts, strict: true, zero required approvals. Notably smoke, its only cross-service test, is not required — the check that would catch a broken integration cannot block.

New in Part 1

  • 1.5 how platform tests: 205 backend tests, the same DI singleton-swap pattern executed slightly better (its DB fixtures pair the restore with data cleanup and a comment naming the bug it prevents), real Mongo/Keycloak/Temporal rather than mocks, zero frontend tests.
  • 1.6 what Update dependency pytest to v9 [SECURITY] #107 adds that main lacks: a 4×6 failure matrix over MockTransport, zero-upstream-request assertions, leak assertions, OpenAPI contract tests in pytest. main uses no MockTransport anywhere, which is why F10's live calls get through.

Practice sections

Copying: platform's real-identity RBAC tests (a mocked layer and a real-Keycloak layer over the same endpoints, each labelled in its docstring), its smoke.yaml, fixture teardown that names the bug it prevents, negative-authz assertions inside the happy path.

Avoiding: its release path publishing three images with no test dependency — which our own build-containers.yml shares — and documenting a command that cannot run.

Housekeeping

Old F10 renumbered F12; all twelve findings are referenced by at least one action. make check clean; mkdocs build -s passes.

🤖 Generated with Claude Code

https://claude.ai/code/session_018TgVbCXXWAWhNjtKTzrxvN

platform was missing and is the most useful comparison here: same kind of
service, same problem, different choices. Figures from main@75dc795
(2026-09-04); PR #107 adds practices main lacks, recorded separately in 1.6.

New sections:
- 1.5 how platform tests: 205 backend tests, the same DI singleton-swap
  pattern executed slightly better (its DB fixtures pair the restore with
  data cleanup and a comment naming the bug it prevents), real Mongo,
  Keycloak and Temporal rather than mocks, zero frontend tests, and no
  coverage instrumentation of any kind.
- 1.6 what #107 adds: a 4x6 failure matrix over MockTransport, zero-upstream-
  request assertions, leak assertions, and OpenAPI contract tests in pytest.
  main uses no MockTransport at all, which is why the live calls in F10 get
  through.

Two new findings, both only visible by comparing:

F10: the two repos fail at gating in OPPOSITE directions and both are wrong.
Here everything real is skipped because the credential is absent. In platform
the CI invocation is a bare `pytest`, so three tests hit third-party services
on every PR in a required check -- one of them carrying no marker at all, so
it survives even the documented `-m "not integration"`. Shared cause: in
neither repo does the marker decide where the test runs. That is the argument
for F.c. Note platform's own smoke.yaml header states the right policy, which
its backend suite then does not follow.

F11: pytest takes the first pytest.ini and never merges the pyproject block,
so one config is dead in platform, sms-api and biosim-client. platform's dead
block contains python_files = "main.py", which would collect ZERO tests if
the file shadowing it were ever removed as redundant -- a green suite running
nothing. This repo has only the pyproject block and is unaffected.

Existing findings refined rather than duplicated: F2 gains platform as the
other end of the same failure (no codecov config at all, no pytest-cov, and
two docs advertising a --cov command that cannot run); F3 gains platform as
proof that branch protection is achievable in this org, requiring two build
contexts with strict:true and zero approvals -- while smoke, its only
cross-service test, is not required.

Practice worth copying gains platform's real-identity RBAC tests (a mocked
layer and a real-Keycloak layer over the same endpoints, each labelled), its
smoke job, fixture teardown that names the bug it prevents, and negative-authz
assertions inside the happy path. Practice worth avoiding gains its release
path publishing images with no test dependency -- which our build-containers
workflow shares -- and documenting a command that cannot run.

Old F10 renumbered F12. All twelve findings are referenced by an action.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018TgVbCXXWAWhNjtKTzrxvN
@jcschaff
jcschaff merged commit 78899ed into main Sep 11, 2026
17 of 18 checks passed
@jcschaff
jcschaff deleted the docs/testing-plan-platform branch September 11, 2026 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant