Skip to content

ui-core's vitest suite times out under load, and it reproduces on main #555

Description

@JArmandoAnaya

What happens

Under machine load, pnpm --filter @visionset/ui-core test fails with a handful of tests
timing out at vitest's 5000ms default. Which tests fail varies run to run, and how many
tracks the load average rather than anything in the diff. Every failure observed so far has
been Error: Test timed out in 5000ms, plus one variant where userEvent.type delivered
only part of a string before the assertion ran.

The files involved are consistently the heaviest ones — the ones mounting a screen under a
real ApiProvider with a stubbed fetch and driving it through userEvent:

  • src/screens/schemaDraft.test.tsx
  • src/screens/screens.test.tsx (the schema editor and version history groups)
  • src/screens/inference.test.tsx
  • src/annotator/addClassProvenance.test.tsx
  • src/annotator/addClassDialog.test.tsx, src/annotator/drawingClass.test.tsx

It is not a change anybody made

Measured on 49a3765, unmodified, on an Apple M-series laptop, with the same command in the
same environment:

 Test Files  4 failed | 45 passed (49)
      Tests  11 failed | 938 passed (949)

Four consecutive runs on one machine over about twenty minutes, as load fell from a load
average of roughly 90 to roughly 7, produced 41, 19, 11 and 5 failures. The set is
nested rather than random: the same files fail, and fewer of their tests fail as the machine
frees up. Running the same three files alone, immediately after a run in which all three had
failed, passes every one of their 75 tests.

CI does not see it. The suite is green on clean runners, which is why this has gone
unnoticed — it costs nothing there and costs a local gate its meaning here.

Why it matters

A local bash scripts/check.sh is what the contribution protocol asks for before a push,
and a gate that fails for reasons unrelated to the diff trains people to read red as noise.
That is the same failure mode #550 and #511 describe, one suite over, and the cost is the
same: the run that finally is about the change looks like the four before it.

It is also actively misleading during a task. A branch under this condition can show fewer
failures than main does, which is what happened while this was being diagnosed.

Likely direction

The 5000ms default is vitest's, and these tests are near it even when they pass: the same
run reports cumulative test time far in excess of wall-clock, so several workers are
contending for one machine. Two things worth measuring before choosing:

  • Raise testTimeout for the suite, which is the smallest change and the one that
    admits what these tests are: several of them mount a screen, wait on TanStack Query, and
    drive a dozen userEvent interactions. A timeout tuned for a pure function is not a
    meaningful bound on that.
  • Cap the worker count, the way perf(e2e): give the worker count its own variable so the local gate uses the machine #530 gave the browser suites their own variable. If the
    contention is between workers rather than with the rest of the machine, this is the fix
    that keeps the timeout honest.

Whichever is chosen, the check is the same and it is cheap: reproduce under synthetic load,
apply, and confirm the suite is stable across several consecutive runs — the repetition
discipline tests/kernel/test_concurrency.py already sets for this repository.

Do not fix it by deleting or skipping the affected tests. They cover the schema draft's
survival across a tab switch and a version published underneath it, which is exactly the
class of defect #389 was.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingfrontendannotator / ui-core / app packagestoolingDeveloper tooling: scripts, checks, local workflow

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions