The TypeScript contracts in lib/forge-contracts.ts describe the JSON each forge concept returns. Nothing verifies that any script actually produces that shape.
So a script can drift from its contract and everything stays green: the TS side compiles against a type nobody checks, and the shell side prints whatever it prints.
Raised by the #13 builder while adding the Ci* contracts. It noted the gap rather than fixing it, correctly, because fixing it only for the CI concepts would be worse than the current state — it would imply the other 18 are covered when they are not.
Scope
All forge concepts, all providers, not just the CI ones. Today that is 22 concepts across github, gitea, gitlab, linear.
Shape
A conformance test per (concept, provider) that runs the real script against a captured fixture and asserts the output parses into its declared contract. Fixtures over live calls, so it runs in CI and offline.
Where a provider disables a concept, the test should assert the disabled envelope rather than skipping — a disabled concept and an untested one should not look the same in the suite. That distinction is the whole point.
Why it matters
Two real defects this week came from output shape rather than logic:
Neither is caught by a type. Both are caught by running the script and checking the result.
Note
This is groundwork, not a fix. It is worth doing before the next provider is added, because each new provider multiplies the untested surface.
The TypeScript contracts in
lib/forge-contracts.tsdescribe the JSON each forge concept returns. Nothing verifies that any script actually produces that shape.So a script can drift from its contract and everything stays green: the TS side compiles against a type nobody checks, and the shell side prints whatever it prints.
Raised by the #13 builder while adding the
Ci*contracts. It noted the gap rather than fixing it, correctly, because fixing it only for the CI concepts would be worse than the current state — it would imply the other 18 are covered when they are not.Scope
All forge concepts, all providers, not just the CI ones. Today that is 22 concepts across
github,gitea,gitlab,linear.Shape
A conformance test per (concept, provider) that runs the real script against a captured fixture and asserts the output parses into its declared contract. Fixtures over live calls, so it runs in CI and offline.
Where a provider disables a concept, the test should assert the disabled envelope rather than skipping — a disabled concept and an untested one should not look the same in the suite. That distinction is the whole point.
Why it matters
Two real defects this week came from output shape rather than logic:
pr-existsreturned nothing on a repo with many PRs; the caller saw an absent value and rendered empty (executeForgeCommand swallows a timeout as null with nothing on stderr, so a broken forge concept renders as empty data #17)status=canceledwhile acceptingcancelled, the spellingtea --helpitself documents (found in Add CI concepts to the forge layer so builders read failure reasons instead of dumping logs into context #13)Neither is caught by a type. Both are caught by running the script and checking the result.
Note
This is groundwork, not a fix. It is worth doing before the next provider is added, because each new provider multiplies the untested surface.