You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Context for a series of PRs that add end-to-end golden tests for the SONiC
config generator. Filed so each PR can stay small without a reviewer having to
reconstruct why it exists or what the ones around it do.
What this adds, and the gap it fills
The SONiC config generator is 12 modules — config_generator.py alone is ~2.5k
lines — and its output is a complete config_db.json that gets deployed to a
switch. Today 25 unit test modules cover the individual helpers, and nothing
exercises the whole path: seeded NetBox → sync_sonic() → generated config.
These PRs add that. NetBox is provisioned from a compose stack, seeded from
in-repo fixtures, and the exported config_db.json for every seeded device is
diffed against a committed golden. Nine golden devices end up covering all 38 config_db tables the generator can emit.
Why golden files rather than more unit tests
Because each golden pins a device's entire emitted config, not one table. The
regressions that matter here are cross-cutting: a change to one helper silently
altering an unrelated table, or a table quietly disappearing from the output.
Unit tests assert what they were written to assert and stay green through both.
That is not hypothetical. Building this surfaced two production bugs that the
existing 25 unit test modules did not catch, and in one case could not have
caught — its fixtures were shaped so the failing condition was unexpressible.
Both went out as separate PRs (see below), since neither is test
infrastructure.
Design decisions worth knowing before reviewing
NetBox comes from docker compose, not kind + Helm. An earlier attempt at this
suite (see Superseded below) used a kind cluster. It was abandoned on measurement: ~1650s of a ~2100s
run was fixed setup — installing kind and Helm, then seeding — so the job used
80-92% of a 2400s budget as a matter of course and any node-local hiccup tipped
it over. Compose brings pre+run to ~1244s and removes python-osism's need for
kind/kubectl/helm entirely. A larger CI nodeset was also tried and dropped: 1235s
at 2 vCPU against 1335s at 8, so the bigger node was slower.
Seed data is synthetic and in-repo, not derived from the testbed or the
netbox-manager example. The motive is independence rather than speed: goldens
built from an input in another repository are hostage to it, and an unrelated
change there would rewrite them. Seeding also drops from 525s to 57.7s, and
coverage improves: the netbox-manager-derived data reached 31 of the 38 emitted
tables, the synthetic fixtures reach all 38.
Scope of the coverage claim
"38 of 38" counts the tables the generator itself can emit, measured as the union
of populated tables across the golden set and cross-checked against the generator
source by a committed script.
It says nothing about tables inherited from a base config, and that limit is
worth stating plainly: every device here is generated from an empty base. In
production the base is files/sonic/config_db.json, which the Containerfile
installs to the hardcoded /etc/sonic/config_db.json. That path does not exist
outside the conductor image, so the harness these PRs add finds no base and
starts each device from {}.
The effect is that 24 tables which are populated in the shipped base and
never emitted by the generator — FEATURE, TELEMETRY, SWITCH, ZTP, the POLICY_* and ECMP_LOADSHARE_* groups among them — pass through to real
switches without any golden covering them. Closing that needs the base config
path to become configurable, which is a separate change listed below rather than
something these PRs leave half-done.
One deliberate trade in the fixtures: they exercise Accton-AS7726-32X and
Accton-AS9726-32D, the latter newly covering the 8-lane 400G path. The
netbox-manager example's Accton-AS5835-54X is therefore no longer exercised — a
real, if small, loss against the churn the synthetic fixtures remove.
The series
Nine PRs, each stacked on the previous, ~9.3k insertions in total of which
roughly 2.7k is generated golden JSON. Refs get added here as they open.
port_config path configurable — SONIC_PORT_CONFIG_PATH setting, so the
generator can be pointed at the in-repo files/sonic/port_config instead of the
container image path. Production change, default unchanged. (41 lines)
Harness scripts — compare.py and generate.py plus their unit tests,
and the regeneration coverage guard. Self-contained Python; no infrastructure
yet. (808)
Base fixtures, coverage report, docs, Zuul job — five synthetic devices
and their goldens, the make sonic-e2e-coverage report, a README section, and
the job definition. This is where the test first runs end to end. (3528, mostly
golden JSON)
The fourth is deliberately the largest: the harness is not independently testable
in smaller pieces, and its Zuul job has to land with the goldens it compares
against or the check is red on arrival. Once that job exists it gates changes
under osism/tasks/conductor/, which makes it the point of no return for the
series.
Split out separately
Three changes came out of this work but are not test infrastructure, so they went
out on their own rather than inside the series. The first two are production bugs
the harness surfaced; the third makes the base config path configurable and
corrects the ownership model's account of where that file comes from.
An earlier version of this work, replaced by the branches above. Closed rather
than retargeted, because their history carries commits that were deliberately
dropped:
The compose half has a green CI run. The synthetic-fixture half has never run
in Zuul — the ~1244s figure above is measured for compose, on a branch that no
longer exists. The Zuul job inside the fourth PR is the first real test of the
fixtures, so that check is the one to watch when the series is pushed.
Context for a series of PRs that add end-to-end golden tests for the SONiC
config generator. Filed so each PR can stay small without a reviewer having to
reconstruct why it exists or what the ones around it do.
What this adds, and the gap it fills
The SONiC config generator is 12 modules —
config_generator.pyalone is ~2.5klines — and its output is a complete
config_db.jsonthat gets deployed to aswitch. Today 25 unit test modules cover the individual helpers, and nothing
exercises the whole path: seeded NetBox →
sync_sonic()→ generated config.These PRs add that. NetBox is provisioned from a compose stack, seeded from
in-repo fixtures, and the exported
config_db.jsonfor every seeded device isdiffed against a committed golden. Nine golden devices end up covering all 38
config_dbtables the generator can emit.Why golden files rather than more unit tests
Because each golden pins a device's entire emitted config, not one table. The
regressions that matter here are cross-cutting: a change to one helper silently
altering an unrelated table, or a table quietly disappearing from the output.
Unit tests assert what they were written to assert and stay green through both.
That is not hypothetical. Building this surfaced two production bugs that the
existing 25 unit test modules did not catch, and in one case could not have
caught — its fixtures were shaped so the failing condition was unexpressible.
Both went out as separate PRs (see below), since neither is test
infrastructure.
Design decisions worth knowing before reviewing
NetBox comes from docker compose, not kind + Helm. An earlier attempt at this
suite (see Superseded below) used a kind cluster. It was abandoned on measurement: ~1650s of a ~2100s
run was fixed setup — installing kind and Helm, then seeding — so the job used
80-92% of a 2400s budget as a matter of course and any node-local hiccup tipped
it over. Compose brings pre+run to ~1244s and removes python-osism's need for
kind/kubectl/helm entirely. A larger CI nodeset was also tried and dropped: 1235s
at 2 vCPU against 1335s at 8, so the bigger node was slower.
Seed data is synthetic and in-repo, not derived from the testbed or the
netbox-manager example. The motive is independence rather than speed: goldens
built from an input in another repository are hostage to it, and an unrelated
change there would rewrite them. Seeding also drops from 525s to 57.7s, and
coverage improves: the netbox-manager-derived data reached 31 of the 38 emitted
tables, the synthetic fixtures reach all 38.
Scope of the coverage claim
"38 of 38" counts the tables the generator itself can emit, measured as the union
of populated tables across the golden set and cross-checked against the generator
source by a committed script.
It says nothing about tables inherited from a base config, and that limit is
worth stating plainly: every device here is generated from an empty base. In
production the base is
files/sonic/config_db.json, which the Containerfileinstalls to the hardcoded
/etc/sonic/config_db.json. That path does not existoutside the conductor image, so the harness these PRs add finds no base and
starts each device from
{}.The effect is that 24 tables which are populated in the shipped base and
never emitted by the generator —
FEATURE,TELEMETRY,SWITCH,ZTP, thePOLICY_*andECMP_LOADSHARE_*groups among them — pass through to realswitches without any golden covering them. Closing that needs the base config
path to become configurable, which is a separate change listed below rather than
something these PRs leave half-done.
One deliberate trade in the fixtures: they exercise Accton-AS7726-32X and
Accton-AS9726-32D, the latter newly covering the 8-lane 400G path. The
netbox-manager example's Accton-AS5835-54X is therefore no longer exercised — a
real, if small, loss against the churn the synthetic fixtures remove.
The series
Nine PRs, each stacked on the previous, ~9.3k insertions in total of which
roughly 2.7k is generated golden JSON. Refs get added here as they open.
SONIC_PORT_CONFIG_PATHsetting, so thegenerator can be pointed at the in-repo
files/sonic/port_configinstead of thecontainer image path. Production change, default unchanged. (41 lines)
compare.pyandgenerate.pyplus their unit tests,and the regeneration coverage guard. Self-contained Python; no infrastructure
yet. (808)
compose.yaml,deploy_netbox.sh,sonic_golden_test.sh, Makefile targets. (423)and their goldens, the
make sonic-e2e-coveragereport, a README section, andthe job definition. This is where the test first runs end to end. (3528, mostly
golden JSON)
pinning the explicit-declaration path rather than the inference fallback. (1711)
PORTCHANNEL/_MEMBER/_INTERFACE. (629)set reaches 38/38. (755)
golden, so the number cannot quietly regress. (30)
The fourth is deliberately the largest: the harness is not independently testable
in smaller pieces, and its Zuul job has to land with the goldens it compares
against or the check is red on arrival. Once that job exists it gates changes
under
osism/tasks/conductor/, which makes it the point of no return for theseries.
Split out separately
Three changes came out of this work but are not test infrastructure, so they went
out on their own rather than inside the series. The first two are production bugs
the harness surfaced; the third makes the base config path configurable and
corrects the ownership model's account of where that file comes from.
Superseded
An earlier version of this work, replaced by the branches above. Closed rather
than retargeted, because their history carries commits that were deliberately
dropped:
Known gap
The compose half has a green CI run. The synthetic-fixture half has never run
in Zuul — the ~1244s figure above is measured for compose, on a branch that no
longer exists. The Zuul job inside the fourth PR is the first real test of the
fixtures, so that check is the one to watch when the series is pushed.