test(azure.ai.agents): validate azure.yaml examples in docs

[glharper]

Fixes #9330

Problem

azure.yaml examples in this extension's docs aren't validated by anything, so they drift out of sync with the code and ship in a state where copying them fails. Two instances were found recently — both by hand, neither catchable by CI:

1. README.md — the "Migrating Legacy Agent Configuration" After: example omitted the required agent name. Copying it failed with name cannot be empty (fixed in fix: name the azure.yaml key in the RAI policy validation error #9328).
2. A Learn article documented rai_config.rai_policy_name on the azure.ai.agent service, which azd doesn't support at all. Because azd ignores unrecognized service properties, the agent deployed with no guardrail and no error (fixed in MicrosoftDocs/azure-ai-docs-pr#13567).

Approach

Implements approach 1 from the issue — validate doc snippets in a test — with one addition, because the two defects above are genuinely different classes.

TestDocExamplesAreValid walks the extension's markdown, extracts every fenced YAML block declaring an azure.ai.agent service, and applies two checks:

1. Resolver. The snippet must survive AgentDefinitionFromService, the same entry point azd uses at deploy time. This catches defect class 1.

2. Vocabulary. Every property must be declared in schemas/azure.ai.agent.json or parsed by azd core. This catches defect class 2, which the resolver cannot — UnmarshalStruct is non-strict, so an unsupported key passes cleanly. That leniency is correct at runtime (forward compatibility for other extensions), but our own docs shouldn't rely on it. The check is docs-only; no runtime behavior changes.

The vocabulary list is read from the schema that already ships with the extension, so it stays current without a second place to maintain.

The partial-snippet caveat

The issue flags that a naive validator false-positives on deliberately incomplete snippets: the three azure.ai.agent entries in docs/private-networking.md omit kind on purpose (the snippet is about the project's network: block; azd falls back to the on-disk agent.yaml).

Rather than inferring intent from the absence of kind — which would silently excuse a complete example that accidentally dropped it — snippets opt out explicitly:

<!-- azd:doc-example partial -->

Default is strict (must fully resolve); the marker only relaxes check 1. The failure message names the marker, so the escape hatch is discoverable at the moment it's needed.

Verification

Beyond the tests passing, I confirmed the test actually fails on each defect it claims to catch, by temporarily reintroducing them:

Injected defect	Result
Drop name from a README example (recreates #9328)	❌ README.md:43: ... - template.name not in valid format: name cannot be empty
Add rai_config: {rai_policy_name: ...} (recreates the Learn defect)	❌ README.md:43: service "my-agent" documents property "rai_config", which azd does not support.
Remove a partial marker	❌ docs/private-networking.md:11: ... did not resolve (is kind missing?)

Failures are reported per snippet as file:line/service, pointing straight at the fence to fix. All docs were restored afterwards — the diff touches no example content.

TestExtractYAMLExamples covers the extractor itself (language filtering, marker scoping, dedenting of indented fences, and CommonMark fence-length nesting), since every other assertion depends on it finding the right blocks.

Validation

From cli/azd/extensions/azure.ai.agents:

• go build ./... — clean
• go test ./internal/project/ -count=1 — ok (full package)
• gofmt -s -l ./internal — no output
• golangci-lint run ./internal/project/... — 0 issues
• cspell lint (Go, repo config) — 0 issues

Currently validates 6 snippets: 3 in README.md, 3 in docs/private-networking.md. New docs are picked up automatically.

Notes

• Test-only plus doc markers; no production code and no core cli/azd changes.
• No new dependencies (gopkg.in/yaml.v3 was already direct).
• No go.mod / go.sum changes.
• AGENTS.md documents the convention so future doc authors know the rule and the escape hatch.
• Approach 2 from the issue (make schemas/examples/*.azure.yaml the single source of truth) is deliberately not taken here — it's a larger docs restructure, and this is complementary to it.

[github-actions]

📋 Prioritization Note

Thanks for the contribution! The linked issue isn't in the current milestone yet.
Thank you for logging this issue; our team is reviewing it. If you need urgent prioritization, tag @RickWinter and @kristenwomack to let us know.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
21 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds automated validation to prevent azure.ai.agent documentation examples from drifting out of sync.

Changes:

• Extracts and validates fenced YAML examples.
• Adds explicit markers for intentionally partial snippets.
• Documents the validation convention.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
internal/project/doc_examples_test.go	Adds extraction and validation tests.
docs/private-networking.md	Marks three examples as partial.
AGENTS.md	Documents validation requirements and opt-out usage.

[Copilot]

Fixed in c40566f. The snippet is now decoded into coreServiceFields, a mirror of core's typed ServiceConfig fields, so a value core would reject (project: [src]) is a yaml type error here too instead of coercing to ""; the proto is mapped from that decode, and everything core doesn't name comes from its ,inline equivalent rather than a hand-built map.

Core's pkg/project isn't imported directly: it isn't in this module's dependency surface today, and go mod tidy with that import pulls azd's provisioning tree (armstorage, gonum, ...) into the extension for a docs test. Verified with project: [src] injected into a README example — the test now fails with service "my-agent" cannot be parsed by azd core.

[Copilot]

Fixed in c40566f. definitionProps is gone. The check now walks the service map itself (every non-core key must be a schema property) and, for the deprecated shape, the config child separately, then recurses through the schema — resolving $ref, descending into items, and honoring additionalProperties (a nested schema is followed; false is strict; absent/true stays permissive per JSON Schema).

Verified with injected defects: rai_config beside config now fails with documents property "rai_config", and a nested policies[0].rai_policy_name fails with the path in the message.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:278

• [azd-code-reviewer] These map fields do not apply core's actual shape rules as the helper claims. For example, core decodes docker into project.DockerProjectOptions, where path is a string, but this mirror accepts docker: {path: [Dockerfile]}; checkVocabulary then skips docker as core-owned, so the broken snippet passes. Mirror the nested core types faithfully (also for K8s/Infra/Hooks), or use core's decoder.

	Docker               map[string]any    `yaml:"docker"`
	K8s                  map[string]any    `yaml:"k8s"`
	Module               string            `yaml:"module"`
	Infra                map[string]any    `yaml:"infra"`
	Hooks                map[string]any    `yaml:"hooks"`

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:407

• [azd-code-reviewer] This assertion only proves that one snippet was discovered. If any of the six existing examples changes its fence label, loses host, or misspells it as azure.ai.agents, that example silently drops out while the test remains green. Assert the current baseline count (and update it for intentional additions/removals), or track the expected snippets individually so losing coverage is visible.

	require.NotZero(t, checked, "no azure.ai.agent doc examples were found — is the extractor still working?")

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:130

• [azd-code-reviewer] Treating absent/true additionalProperties as runtime support leaves the same silent-no-op gap for typed objects. The shipped schema marks agentEndpoint and agentCard with additionalProperties: true, while runtime unmarshals them into fixed Go structs; therefore agentEndpoint: {rai_config: ...} is ignored by json.Unmarshal and still passes this vocabulary check. Tighten those schema nodes (while retaining genuinely free-form maps such as metadata/credentials) or validate them against the runtime types.

This issue also appears on line 274 of the same file.

			if allowed, ok := additional.(bool); !declared || (ok && allowed) {
				continue

cli/azd/extensions/azure.ai.agents/internal/project/doc_examples_test.go:402

• [azd-code-reviewer] The extractor has dedicated tests, but neither advertised validation rule has a committed negative regression test. With only valid live docs as fixtures, a change that makes checkVocabulary permissive or stops enforcing resolver failures can leave this suite green; the temporary defect injections described in the PR will not protect future changes. Add table-driven invalid snippets for missing name, unknown top-level/nested properties, and the partial-marker exception.

				checkVocabulary(t, e, name, svc, schema)

[jongio]

One nit inline. Everything else checks out.

I ran the new test on the branch and mutated the docs to confirm each guard actually fires: dropping name from the README migration example reproduces the name cannot be empty failure from #9330, an undeclared key at the top level and one nested under policies[0] both fail with the file:line message, and project: [src] is a core parse error now instead of a silent coercion to "".

The extractor's nested-fence handling holds up too, so the example block inside AGENTS.md isn't picked up as its own snippet.

[jongio]

These two structpb.NewStruct assertions are the only ones in this file without a message, so a value structpb can't represent fails with a bare proto: invalid type: time.Time and no hint about what to change in the doc.

It's reachable from a snippet: an unquoted date under an extension-owned key, e.g. metadata: with released: 2024-07-18, decodes to time.Time and fails exactly that way. The subtest name still points at the file, but nothing tells the author to quote it.

Suggested change

	require.NoError(t, err)
	props, err := structpb.NewStruct(core.AdditionalProperties)
	require.NoError(t, err,
	"%s: service %q has a value that cannot be represented in azure.yaml "+
	"(quote ambiguous scalars such as dates).", e, name)

Same for the core.Config call a few lines down.