test(frontend): add end-to-end test suite by andre8244 · Pull Request #195 · NethServer/my

andre8244 · 2026-09-08T14:05:20Z

📋 Description

Adds browser-level end-to-end tests. The API is already covered
end-to-end by the authorization suite, but nothing exercised the UI.

The suite drives the real application against a real backend, signing in
through the actual login form as the users the existing test fixture
already provides.

Where each layer runs

Layer	When	Against
Authorization spec check	every push and pull request	nothing external — reads the repo's own files, about a second
Browser suite (`fullstack`)	every push to a pull request, merge to `main`, on demand, weekly	the whole stack built and booted on the runner, with a Logto tenant dedicated to CI
QA smoke (`smoke`)	after each QA deploy	the deployed QA environment, read-only

Production is never contacted. The CI tenant is separate from the
development and QA ones, so provisioning and removing its test users
collides with nobody.

Show tests included

Sign-in (2)

a saved session lands on the dashboard without a round trip to Logto
an unauthenticated visitor is sent to Logto

Permissions (42 — every organization role × technical role pair)

each pair is offered exactly the navigation its permissions allow
each pair reaches the sections it may read and is refused the rest

Action controls (21 — one per pair)

each pair is offered exactly the actions its permissions allow inside a
page, which is where a hidden section still leaves a visible button

Hierarchy scoping (6)

/resellers and /customers list the caller's own branch and nothing
outside it, from a distributor down to a reseller
/users lists a customer organization's own members only

Session (4)

a live session survives a reload without going back to Logto
a tab with no JWT pair re-mints one silently
a refusal offers a way back from the forbidden page
a refusal leaves the rest of the session usable

Organizations (3)

creates a company and lists it
refuses a company with no VAT number
edits a company and keeps the change

Systems (3)

lists a new system and filters down to it
opens the detail view for a system
stops offering to regenerate the secret once the system registers

Users (4)

creates a user with a role and lists it
changes the assigned role and keeps the change
archives the user
refuses a user with no role assigned

QA smoke (4, read-only, against the deployed environment)

serves a healthy backend
sends an anonymous visitor to the identity provider
signs in, loads the dashboard and signs out
renders a list page

Test data

The tests assume no company, user or system exists beforehand. Each one
creates what it needs, named under a reserved prefix, and removes it
afterwards; the cleanup refuses to touch anything outside that prefix,
and each file sweeps leftovers before it starts, so a run that crashed
earlier cannot influence the next. The database is new on every run.

What must be prepared once, per tenant, is the identity provider: an
initialized Logto tenant with its roles and its owner account. The
secrets are listed in .github/workflows/README.md.

Notes

No change to how the application behaves. The only production code
touched is a build-time flag that disables background refetching and the
devtools panel in test builds only, because both interfere with the
tests.

Also adds a cheap authorization check to the existing pipeline: it fails
when an endpoint ships without a declared permission, or when the test
fixture names a role no configuration defines.

github-actions · 2026-09-08T14:05:33Z

🔗 Redirect URIs Added to Logto

The following redirect URIs have been automatically added to the Logto application configuration:

Redirect URIs:

https://my-proxy-qa-pr-195.onrender.com/login-redirect

Post-logout redirect URIs:

https://my-proxy-qa-pr-195.onrender.com/login

These will be automatically removed when the PR is closed or merged.

Covers the browser layer the authorization suite cannot reach: it proves the API refuses the wrong caller, not that the UI hides the button or scopes the table for that same persona. Personas come from the apitool registry, which already fixes a password on every user it creates, so the suite signs in through the real Logto form and needs no test-only auth path in the application. Only storageState is reused; the JWT pair lives in sessionStorage and is re-minted per spec, since the backend rotates refresh tokens and treats a replayed one as theft. Specs cover the saved-session boot, the RBAC matrix across every persona, organizations, systems including the registration handshake, and users including role assignment. Where a spec creates something it asserts what the backend stored, not the rendered row. Everything created is named under a reserved "e2e-" prefix; teardown refuses anything outside it and sweeps leftovers from a crashed run. The user lifecycle runs serially on purpose: creating a user sends a welcome email, so one account covers create, edit and archive. A read-only smoke project targets a deployed environment, catching the configuration failures the local stack cannot show. It runs against /backend/api, since the proxy forwards /api to the separate legacy system. e2e-main.yml runs the full-stack project on merge to main, on demand and weekly, queueing rather than cancelling so a cancelled run never abandons organizations in the tenant. e2e-smoke.yml asks the Render API which commit QA is serving, because Render builds from source and the health endpoint reports no commit. docker-compose.e2e.yml is an additive overlay used only by CI; the developer stack is untouched. VITE_E2E drops query auto-refetch and the Colada devtools panel, both of which race assertions. A setup guard fails fast when the dev server serves a stale module graph, where component-library refs come back undefined and every symptom reads as a product bug. Renames the vitest script to test:unit, so the four test scripts all say what they run, and corrects AGENTS.md 7.1, which documented the removed make gen-tokens. Assisted-by: Claude Code:Opus 5 (1M context)

workflow_dispatch only works for workflows already on the default branch, so the full-stack job cannot be exercised before this lands. A pull_request trigger, scoped to the e2e paths, is the only way to prove the job and its secret wiring against a real runner. Revert before merging: the job provisions and tears down a fixture in the CI Logto tenant on every push to the PR. Assisted-by: Claude Code:Opus 5 (1M context)

sync/configs/*.yml is gitignored, since the per-environment files carry tenant ids and SMTP credentials. A fresh checkout therefore has no config.yml, and `apitool authz provision` reads the role vocabulary from it — so the first run failed with "open ../sync/configs/config.yml: no such file or directory". Track config.ci.yml for exactly this: the roles, permissions and resources of the CI tenant, and nothing else — no tenant ids, no third-party applications, no credentials. Keeping it separate from the annotated example means the job does not break the next time that example is rewritten. Assisted-by: Claude Code:Opus 5 (1M context)

The per-app access_control in sync/configs/config.yml now admits the Support user role on my.nethspot.com, which is what the apps layer evaluates expected portal visibility from. The note beside it still said Owner, Staff or Admin. Assisted-by: Claude Code:Opus 5 (1M context)

Two specs waited for the list request after the navigation that fires it. page.waitForResponse only sees traffic arriving once it listens, so whenever the response came back first the spec waited for a second request that never comes, then timed out at 30s. It surfaced on CI as two flaky tests that passed on retry — the race only loses when the backend is quick. Arm the wait first. openAs takes the pattern and holds both waits open across the goto; apiResponse does the same for a reload or a click. Assisted-by: Claude Code:Opus 5 (1M context)

The owner-level threshold in the fixtures also accepted a Super Admin user role. That role no longer exists: the Staff/Owner model replaced it, and lib/permissions.ts keys the threshold on loginStore.isOwner — the organization role alone, which both Owner and Staff imply. Assisted-by: Claude Code:Opus 5 (1M context)

`apitool authz coverage` is the one subcommand of the authz suite that needs no tenant and no running backend: it reads main.go, the spec and the role vocabulary. Seconds, so it can gate every branch, while `personas` and `run` stay local for want of a provisioned tenant. It fails on a route with no declared intent, and on a fixture user whose role no config defines. The second is what prompted this: a role was added to authz/fixture.yml with no matching definition, and nothing noticed until an unrelated job tripped over it days later. Offline still means a config and a registry must exist, so the step supplies config.ci.yml and a stub registry — assertLocal insists the target is local, and coverage never makes a request. Assisted-by: Claude Code:Opus 5 (1M context)

A file argument does not filter a project's dependencies, so naming one spec still ran the whole setup project — 25 tests, not 2. With --debug pausing before the first action of the first test in the queue, the browser opened on a blank page belonging to a persona login rather than to the spec named on the command line. --no-deps is what narrows it, at the price of reusing whatever sessions e2e/.auth already holds. Assisted-by: Claude Code:Opus 5 (1M context)

The Go problem matcher that actions/setup-go registers scans step output for "<file>.go:<number>: <message>" and turns a hit into an annotation. "Routes in main.go: 196" matched it, so the new coverage step raised a red failure annotation against main.go on a run where every check passed — the annotation message was literally "196". Keep the filename, move it out of that shape. Assisted-by: Claude Code:Opus 5 (1M context)

It existed to prove the job and its secrets before landing, since workflow_dispatch only works for workflows already on the default branch. Three runs on this PR did that, the last two green. Back to the intended triggers: merge to main, manual dispatch, and the weekly drift canary. No more provisioning a fixture in the CI Logto tenant on every push to a branch. Assisted-by: Claude Code:Opus 5 (1M context)

A regression is then attributed to the commit that caused it, while the author still has the change in mind, instead of to a batch of merges. No paths allowlist: the failure worth catching is a backend or component change that breaks the interface, not an edit under e2e/. Docs-only pushes are skipped, since the job holds the CI tenant for minutes. The concurrency block is now load-bearing, so it says why it is shaped this way: one global queue, because the fixture prefix and the persona emails come from authz/fixture.yml with no per-run override, and never cancelled, because a run stopped after provisioning leaves orgs behind that no teardown can find. Assisted-by: Claude Code:Opus 5 (1M context)

The confirmation is type-to-confirm: DeleteObjectModal.vue only emits primary-click when the typed text matches the name exactly, and the spec typed nothing, so every run hit the validation error and archived nobody. It passed because HeadlessUI marks the rest of the page aria-hidden while a modal is open — getByRole('row') therefore matched zero rows, and "the row is gone" was satisfied by the modal still being up. The test asserted that a dialog had opened. Type the name, wait for the modal to close before looking at the table, and ask the backend whether the account is really archived rather than trusting the row's absence. Asserted on the modal's heading, since the element carrying role="dialog" is a zero-size wrapper that Playwright reports as hidden however open the modal is. Once archiving actually worked it exposed a leak: /users appends "AND u.deleted_at IS NULL" unless status=deleted is asked for (entities/local_users.go, deletedClause), so listE2eUsers could not see an archived account and both the teardown and the pre-run sweep would have walked past one, leaving it in the tenant for good. It now asks for all three statuses, and takes a narrower list where a spec wants to ask a question instead of clean up. Also stops filterTo waiting for a row that is already on screen — it passed before the first keystroke had any effect, so the list it handed back was not necessarily filtered — and scopes the confirm click to the dialog rather than picking the last match in the page. Assisted-by: Claude Code:Opus 5 (1M context)

expect(x).toHaveCount(0) after an interaction is satisfied just as well by the interaction not happening, so each one now follows something that must be there. The systems spec's whole point — "Regenerate secret" is withdrawn once an appliance registers — was asserted right after a kebab click, with nothing proving the menu opened. getKebabMenuItems offers the exports in either state, so one of those is asserted first. The RBAC spec deep-linked only the *first* section a persona could not read. Given the NAV order and the role vocabulary that is always /distributors for a partner organization and /users for an Owner-org Support user, so /resellers and /customers were never deep-linked by anybody. It now walks every forbidden section, and opens one it may read first — otherwise a guard that sent everyone to /forbidden would pass. The login spec claimed the saved session "never bounced to /login, and never [went] back out to the Logto origin", which a final-URL assertion cannot see either way. It records the main frame's navigations instead. The smoke spec claimed "a rendered list, not a skeleton that never resolves" while asserting only a 200; a table stuck in its skeleton, or a component throwing on the payload, looks identical on the wire. It now also wants a row or the empty state. The two "a rejected form must not create anything" checks stay, but say what they are: the form is refused in the browser and nothing is submitted, so the backend lookup cannot fail. It guards against a form that starts submitting anyway; it is not evidence the server refuses anything, and server-side rejection remains untested. Also stops the systems filterTo waiting for a row already on screen, which passed before the filter had any effect. Assisted-by: Claude Code:Opus 5 (1M context)

Every selector is resolved through fixtures/i18n.ts, which reads the English catalogue only, while the application takes its locale from getPreference('locale', email) — a localStorage entry, which is exactly what storageState captures and e2e/.auth keeps between runs. One persona that ever had Italian selected would turn every getByLabel into a silent miss, pointing at nothing in particular. Pin locale in the config, and clear the preferences blob before the session is saved; the second also makes the collapsed menu state deterministic. The stale-module-graph guard could not fail. It loaded /login — which immediately navigates away to the Logto form — slept two seconds, and asserted that no "Missing ref owner context" warnings had been logged. The breakage it hunts is component-library refs coming back undefined so that dropdowns and drawers stay shut, and /login mounts no such component. It was also the suite's only waitForTimeout, which the README forbids in bold. So open a drawer. A drawer that appears is the only evidence worth anything here, it costs one page load, and it needs no sleep. Declared after the sign-in loop so the owner's session already exists. t() also fills {name} placeholders now, for labels that are composed rather than literal, and throws on one left unfilled rather than returning a selector that matches nothing. Assisted-by: Claude Code:Opus 5 (1M context)

The owner token was minted once per worker and never refreshed, while the backend's access token is short-lived (stores/login.ts: a 20-minute interval with a one-minute margin) and the job is allowed 45. A run that outlived it would 401 every afterAll, and the destroy helpers treat anything but 200/204/404 as fatal — so the run would end with a teardown failure *and* a leaked fixture. It now drops the cache and re-mints once on a 401, which is the only outcome that does not leave something behind. Tolerating the 401 in the helpers would leak silently. Minting also blocked the worker's event loop for as long as the sign-in took, stalling Playwright's own protocol traffic and timers; it is spawned asynchronously now. runId was per module per worker process, with each worker's counter starting at zero, so two workers spawned in the same millisecond generated the same names and only the `kind` segment kept them apart. The worker index makes that structural rather than a matter of timing. The owner persona no longer states an organization it cannot know: the Owner organization predates the fixture and `authz provision` does not create it, so the registry holds no record and orgName/orgId are empty with a comment saying why. fixtureOrg() exposes the provisioned organizations by their key in authz/fixture.yml, which is where the hierarchy is actually declared — and matrixPersonas' comment now describes what it really selects, every fixture user carrying a role, the sibling-branch accounts included. Assisted-by: Claude Code:Opus 5 (1M context)

The suite's own README says these specs prove the UI "hides the button, scopes the table and renders the right navigation". Navigation was proven thoroughly; scoping was not tested at all. It is the one thing a permission check cannot express — every distributor holds exactly the same read:resellers, and the answer still has to differ per distributor — and it is the direction a leak would actually be noticed in. backend/authz/ proves the API refuses to hand one company another's records; this proves the operator is not shown them. Expectations come from the tree in authz/fixture.yml, which is shaped for the question: a sibling reseller under the same distributor (d1r2) and a second branch entirely (d2), because isolation can only be proved against an organization at the same level on the other side. The second branch is asserted from its own side too — a scope bug that leaked downwards would satisfy every case from d1 and fail there. Rows are found by their detail link rather than by text. Two reasons a substring lies here: a row's cells are concatenated with no separator, so "Company name" and the value run together and a word boundary cannot see the seam; and the fixture's names nest on purpose (authz-d1r1 is a prefix of authz-d1r1c1) while an organization row also carries the name of the organization that created it. Both were live — this was written with text matching first and it found the wrong rows. Assisted-by: Claude Code:Opus 5 (1M context)

The other half of "the UI hides the button" that navigation coverage cannot reach. rbac.spec.ts proves a persona is offered the right sections; this proves that once inside one it is offered the right actions — a Reader may open /systems and must not be invited to create one, a distributor Admin manages resellers and must not be offered Promote, which moves an organization out of its own reach. Expectations are hand-authored, as in the RBAC spec: SURFACES states what each control should require, from the vocabulary in lib/permissions.ts and the documented intent of each gate. Copying the table components' conditions in would prove nothing. Every surface names a control that must be present for anyone who can open it, so a page that failed to render — or a kebab that failed to open — cannot satisfy the absences beneath it. Reachability is a conjunction rather than a single read permission, because the reseller kebab also needs whatever puts an item in the menu: the backend subtracts the organization role's manage:* from anyone holding Reader (authz/model.yml, filterManagePermissionsForReader), so a distributor Reader has no kebab at all and nothing to assert about. What it is denied instead is asserted on the create button on the same page. Impersonation and the add-on catalog are left out and tracked: both need a fixture this spec would have to build, and the second includes canBuyAddons — manage:entitlements AND NOT owner-level, the only gate here where holding more authority has to hide a control. Assisted-by: Claude Code:Opus 5 (1M context)

stores/login.ts and the interceptors in lib/axios.ts are the most breakage-prone code in the application — a rotating refresh chain, a sessionStorage-backed JWT pair, a 401 that replays once and a 403 that redirects — and no spec exercised any of it. Every other spec depends on it and none asserted it. Four questions: a reload re-exchanges in place rather than re-entering the sign-in flow; a tab with no JWT pair re-mints one silently, which is the state every spec actually starts in and was relied on rather than asserted; /forbidden offers a way back and it works; and a 403 leaves the rest of the session usable, since an interceptor that turned one missing permission into a logout is precisely the kind of thing an API-level suite cannot see. No sign-out test, deliberately. Signing out ends the persona's Logto SSO session rather than just this tab's, so the saved storageState every other spec boots from would be dead and which spec noticed first would depend on worker scheduling. It needs a persona of its own, or a project that runs last. Assisted-by: Claude Code:Opus 5 (1M context)

Adding the pull_request trigger left three places describing the previous behaviour: e2e-main.yml's own header, fifteen lines above the block it contradicts, and both READMEs. The concurrency comment also had the cost wrong. GitHub keeps at most one run pending per group and cancels the previously pending one, so the queue never grows and a burst does not serialize. What a burst does is skip the commits in the middle, which is a partial retreat from the per-commit attribution the trigger exists for — worth documenting, unlike a queue that cannot happen. Cancelling instead would trade it for abandoned organizations. paths-ignore was on pull_request only, so a docs-only merge to main still built four images and took the global lock. Four secrets were interpolated straight into a run: body while the rest of the step passes them through env:; they go through env: now too. The tenant was described two ways — "the development tenant" in the workflow README against a tenant "dedicated to CI" in the pull request. Settled in favour of the latter, with the reason it matters: prefix in authz/fixture.yml fixes the organization keys and persona addresses, so CI and a local `authz provision` on one tenant fight over the same Logto users. Note that what each side *deletes* is safely scoped either way — the specs refuse any name outside the e2e- prefix and authz teardown only removes what its own registry records — so the failure mode is a collision while provisioning, not lost data. Records that the job carries no SMTP_* on purpose: creating a user sends a welcome email with a temporary password, the users spec creates one per run, and with no relay configured SendEmail fails closed before touching the network. Also documents that a bare `npm run test:e2e` cannot pass locally, since it includes the smoke project, and adds the two lessons from this round to the gotchas: a negative assertion needs a positive control, and waiting for something already on screen proves nothing. Assisted-by: Claude Code:Opus 5 (1M context)

IS_E2E was imported in a second statement from './lib/config', three lines below the one that already imports from it. Assisted-by: Claude Code:Opus 5 (1M context)

A single config served three projects with incompatible needs, and the variables steering it had grown two meanings each. E2E_BASE_URL named the origin and, by being set at all, suppressed the webServer block. e2e-main.yml set it to http://localhost:5173 — the same string the config already defaulted to — so only the suppression was ever wanted there. E2E_SMOKE_BASE_URL then had to exist because E2E_BASE_URL would have dragged the mutating fullstack project onto a deployed environment along with smoke, leaving three sources for one value: that variable, E2E_BASE_URL, and the QA default in the config. Skipping the dev server for a smoke-only run meant reading --project out of process.argv, which is a workaround for one file serving two targets rather than a thing worth doing. Split instead. playwright.config.ts keeps setup and fullstack, pinned to http://localhost:5173 with no env read at all — the Logto fixture only accepts that origin as a redirect URI, so it was never configurable in the first place. playwright.config.smoke.ts holds the deployed suite, with no webServer and no setup dependency, and is the only place a base URL can still be overridden. Options common to both moved to e2e/playwright.shared.ts, so the split duplicates nothing. What replaces E2E_BASE_URL in CI is reuseExistingServer: true. Compose already publishes the proxy on 5173, so Playwright finds it listening and starts nothing; locally it starts dev:e2e itself. The known cost is that a plain `npm run dev` is reused just as readily and does not carry VITE_E2E — already true before this change, and now stated where the option is set. Both scripts work with no variable set: `npm run test:e2e` for the local suite, `npm run test:e2e:smoke` for QA. The name is playwright.config.smoke.ts rather than playwright.smoke.config.ts so the existing tsconfig.e2e.json include, playwright.config.*, keeps matching. Assisted-by: Claude Code:Opus 5 (1M context)

App.vue rendered PiniaColadaProdDevtools, which is the variant that deliberately ships the devtools panel in production builds — the opposite of what was wanted. Every qa and production bundle carried the whole panel: 97 occurrences of its splitpanes UI and roughly 300 KB of uncompressed assets, on a page no user can open it from. PiniaColadaDevtools is the dev-only export. It resolves to an empty wrapper unless NODE_ENV is development, so the panel is gone from the built bundle rather than merely hidden. The v-if stays. `npm run dev:e2e` is a development build, where the panel does mount and injects itself into the DOM, shadowing e2e selectors — which is the case VITE_E2E exists for. Assisted-by: Claude Code:Opus 5 (1M context)

andre8244 self-assigned this Sep 8, 2026

edospadoni deployed to e2e-testing - my-frontend-qa PR #195 September 8, 2026 14:05 — with Render Active

andre8244 force-pushed the e2e-testing branch from d099a26 to aaa2586 Compare September 8, 2026 15:28

edospadoni deployed to e2e-testing - my-frontend-qa PR #195 September 8, 2026 15:28 — with Render Active

andre8244 force-pushed the e2e-testing branch from 935af0c to 2806126 Compare September 9, 2026 08:03

edospadoni deployed to e2e-testing - my-frontend-qa PR #195 September 9, 2026 08:03 — with Render Active

edospadoni deployed to e2e-testing - my-backend-qa PR #195 September 9, 2026 08:04 — with Render Active

edospadoni deployed to e2e-testing - my-frontend-qa PR #195 September 9, 2026 08:20 — with Render Active

edospadoni deployed to e2e-testing - my-frontend-qa PR #195 September 9, 2026 08:45 — with Render Active

edospadoni deployed to e2e-testing - my-backend-qa PR #195 September 9, 2026 08:45 — with Render Active

edospadoni deployed to e2e-testing - my-backend-qa PR #195 September 9, 2026 09:53 — with Render Active

andre8244 changed the title ~~test(frontend): add Playwright end-to-end suite~~ test(frontend): add end-to-end test suite Sep 9, 2026

edospadoni deployed to e2e-testing - my-frontend-qa PR #195 September 9, 2026 18:12 — with Render Active

edospadoni deployed to e2e-testing - my-frontend-qa PR #195 September 10, 2026 09:43 — with Render Active

andre8244 force-pushed the e2e-testing branch from 6233046 to 90b8127 Compare September 10, 2026 09:53

edospadoni deployed to e2e-testing - my-frontend-qa PR #195 September 10, 2026 09:53 — with Render Active

andre8244 added 13 commits September 10, 2026 12:03

andre8244 added 8 commits September 10, 2026 12:03

chore(frontend): fold the duplicate config import

fdd3266

IS_E2E was imported in a second statement from './lib/config', three lines below the one that already imports from it. Assisted-by: Claude Code:Opus 5 (1M context)

andre8244 force-pushed the e2e-testing branch from 90b8127 to 0da2da8 Compare September 10, 2026 10:03

edospadoni deployed to e2e-testing - my-frontend-qa PR #195 September 10, 2026 10:03 — with Render Active

edospadoni deployed to e2e-testing - my-backend-qa PR #195 September 10, 2026 10:03 — with Render Active

edospadoni deployed to e2e-testing - my-frontend-qa PR #195 September 10, 2026 10:23 — with Render Active

edospadoni deployed to e2e-testing - my-frontend-qa PR #195 September 11, 2026 06:00 — with Render Active

edospadoni deployed to e2e-testing - my-backend-qa PR #195 September 11, 2026 06:00 — with Render Active

Navigation Menu

Uh oh!

test(frontend): add end-to-end test suite - #195

test(frontend): add end-to-end test suite#195
andre8244 wants to merge 22 commits into
mainfrom
e2e-testing

andre8244 commented Sep 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

andre8244 commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📋 Description

Where each layer runs

Test data

Notes

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

andre8244 commented Sep 8, 2026 •

edited

Loading