Add data-query category: AL query-generation benchmark

[Onat Buyukakkus (onbuyuka)]

What

Adds a new data-query evaluation category: a benchmark for answering Business Central data
questions through the BC MCP server's Data Query tools. Given a natural-language question, the agent
must retrieve the real data from a live BC environment using the bc_data_* MCP tools and report
exactly what they return — it cannot answer from general knowledge.

The agent writes two files:

• answer.json — the result rows that answer the question (one JSON object per row);
• query.al — the single AL query object it used.

Scoring is execution-based (no LLM judge): build = a parseable answer.json was produced;
resolved (ResolutionRate) = the agent's rows match the gold answer (baked gold_rows, else the
entry's gold_query run live). Rows compare by value — numbers normalized scale-insensitively
(500 == 500.0), Code/No. strings verbatim ("001" != "1"); column names/order ignored;
order-insensitive unless the entry is marked ordered.

Complements the AI Test Toolkit evals in the platform repo: those test the MCP server
end-to-end; this benchmarks models/agents on their ability to use it.

The interesting part: keeping the MCP server the only route to the data

The category is only meaningful if the agent answers through the MCP tools. Making that true was
the bulk of the work — denied one route to the data, the agent kept finding another:

1. The agent read the leaked connection credentials and hit BC's OData /api directly from a
   shell. Fix: agent_subprocess_env() scrubs every BC_SERVER_* / BC_MCP_* / BC_CONTAINER_NAME
   from the launched agent's environment, and a credential-free localhost MCP gateway
   (src/bcbench/agent/shared/mcp_gateway.py) fronts BC — it injects the auth headers upstream (so the
   agent's MCP config carries no secret, and nothing is recoverable from the process command line) and
   path-restricts to /mcp (so /api is unreachable through it).
2. The agent then ran docker exec … sqlcmd against the container's SQL database. Mitigated by
   withholding the container name and removing the incentive (a working, easy MCP path); the durable
   fix is network isolation, documented as the end-state.
3. On the Copilot CLI, custom MCP servers wouldn't load at all — the CLI's MCP registry policy
   fetch returns 403 for the Actions GITHUB_TOKEN, blocking every custom server
   (copilot-cli#4346). Fix: feed a Copilot-licensed
   user PAT via COPILOT_CLI_TOKEN (falls back to github.token). Claude Code is unaffected, so
   the MCP path is validated there first.

Infrastructure

• BC MCP gateway (agent/shared/mcp_gateway.py) — credential-free, /mcp-only reverse proxy on
  localhost. Warms up BC's tool catalog with retries (a cold insider-29 container can take minutes
  to compose it) and caches tools/list so the agent registers the tools on its first call instead
  of racing a cold, sometimes-dropped server; relays streams faithfully, and strips
  capabilities.experimental from the initialize reply — BC advertises x-ms-headerless, which
  otherwise makes Claude's MCP client drop the server.
• Env scrub (agent/shared/env.py) — removes BC connection vars from the agent subprocess.
• MCP config (agent/shared/mcp.py) — points the agent at the gateway; independent --bc-mcp /
  --ms-learn-mcp levers.
• Setup — publishes an AL app (scripts/al/mcp-config-setup/) that provisions the BCBench MCP
  configuration (enables the Data Query tools) and exports the gateway's upstream endpoint. Uses an
  insider BC 29 artifact until the Data Query tools reach a GA artifact (marked TEMPORARY).
• Agent observability — Claude runs with --output-format=stream-json; tool usage (including
  sub-agent and MCP calls) is parsed from the event stream.
• Skill + prompt — the bc-al-query-mcp skill and the data-query prompt pin the exact tool names
  (bc_data_find_tables / bc_data_get_table_schema / bc_data_get_table_relations / bc_data_query)
  and their parameters, which stopped the agent from guessing non-existent tool names.

Results

On the Claude path (claude-sonnet-5) the agent genuinely uses the MCP tools — only the real
bc_data_* tools, no name-guessing.

• A 4-entry test run reaches 4/4 resolved.
• A full-dataset run
  (run 32968838235) reaches
  9/11 resolved (82%), 11/11 build (100%) — the mechanical "no answer.json" failures are gone
  (every entry warms up, queries via the tools, and writes its answer), and the only two misses are
  genuine data mismatches (a values-differ and a row-count-differ), i.e. real model-accuracy
  signal. With strict exact-match scoring this is on par with / above the AI Test Toolkit's grounded
  judge on the same scenarios.

Getting there needed reliable MCP tool registration: the gateway warms up BC's tool catalog with
retries (a cold insider-29 container can take minutes to compose it) and caches tools/list, so the
agent's client registers the tools on its first call instead of racing a cold server.

Dataset

dataset/dataquery.jsonl — each entry has nl_prompt, gold_query, optional baked gold_rows,
environment_setup_version, and ordered.

How to run (no local containers)

Actions → Evaluation with Claude Code → Run workflow → category data-query, a model, enable
bc-mcp / ms-learn-mcp / skills, test-run = true. The self-hosted GitHub-BCBench
runner provisions the sandbox container, publishes the MCP config app, and stands up the gateway; the
agent answers through the tools; the harness compares its rows to the gold answer.

For the Copilot workflow, set the COPILOT_CLI_TOKEN secret first — otherwise the Copilot CLI's MCP
registry policy fetch fails on the Actions GITHUB_TOKEN and blocks all custom MCP servers
(copilot-cli#4346); the Claude workflow is
unaffected.

Docs

• docs/data-query.md — category overview (rewritten for the MCP design).

Follow-ups (tracked, not blocking)

• Set the COPILOT_CLI_TOKEN secret to enable the Copilot CLI path.
• Bake gold_rows for all entries and revert the temporary build_app timeout headroom.
• The insider BC 29 artifact hack is temporary, until the Data Query tools are in a GA artifact.
• Network-isolate the agent (agent-in-a-container) to fully close the docker exec side-door.

[Magnus Hartvig Grønbech (Groenbech96)]

Looks good. Good stuff.

[Copilot]

Pull request overview

Adds the data-query benchmark for deterministic AL query generation and execution against Business Central demo data.

Changes:

• Adds 11 query-generation dataset entries and agent guidance.
• Implements query wrapping, execution, comparison, and result reporting.
• Integrates container setup, workflows, tests, and documentation.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
dataset/dataquery.jsonl	Adds benchmark entries and gold queries.
src/bcbench/evaluate/dataquery.py	Implements evaluation and result comparison.
src/bcbench/operations/bc_operations.py	Adds query wrapping and OData execution.
src/bcbench/dataset/dataset_entry.py	Defines data-query entries.
src/bcbench/dataset/__init__.py	Exports the new entry type.
src/bcbench/types.py	Registers category runtime behavior.
src/bcbench/results/base.py	Adds a general execution-result factory.
src/bcbench/evaluate/__init__.py	Exports the pipeline.
src/bcbench/operations/__init__.py	Exports query operations.
src/bcbench/commands/evaluate.py	Supports mock data-query evaluation.
src/bcbench/agent/shared/config.yaml	Adds the agent prompt template.
src/bcbench/agent/shared/instructions/dataquery-bc/skills/al-query-authoring/SKILL.md	Adds AL query authoring guidance.
scripts/Setup-ContainerAndRepository.ps1	Creates clone-free workspaces.
scripts/BCBenchUtils.psm1	Resolves the new dataset category.
.github/workflows/copilot-evaluation.yml	Enables Copilot runs.
.github/workflows/claude-evaluation.yml	Enables Claude runs.
tests/test_dataquery_evaluation.py	Tests comparison and wrapping logic.
tests/conftest.py	Adds data-query fixtures.
tests/test_type_exhaustiveness.py	Covers category type dispatch.
docs/data-query.md	Documents the benchmark.
docs/index.md	Links the new category.

Comments suppressed due to low confidence (1)

src/bcbench/evaluate/dataquery.py:115

• execute_al_query also raises BuildTimeoutExpired on a gold-query timeout, and it is not a BuildError. This exception escapes instead of taking the intended harness/container failure path.

        except BuildError as e:
            logger.exception(f"Gold query failed to compile/run for {context.entry.instance_id}")
            self.save_result(
                context,

[Onat Buyukakkus (onbuyuka)]

Review feedback addressed (commit 129c5ef)

Thanks for the thorough review. Summary of what changed:

Scoring integrity

• Numeric rounding false-positives — result_sets_match now canonicalizes numbers with Decimal.normalize() (full precision, scale-insensitive) instead of rounding through float to 4 decimals. 1.00001 ≠ 1.00002; 500 == 500.0.
• OData pagination — the fetch now follows @odata.nextLink until exhausted, so large result sets aren't silently truncated to the first page.
• Gold-failure biasing scores — a gold/harness failure is now recorded as unscorable (new scorable flag) and excluded from resolved/total/build/instance_results, so it no longer counts against the agent's ResolutionRate.
• Timeout escaping — BuildTimeoutExpired is now caught around both generated and gold execution (it isn't a BuildError), so a timeout is recorded instead of breaking summarization.
• ValueError on malformed output — wrap_query_as_api now raises BuildError (handled downstream) when there's no query declaration or no object body.

Robustness

• Case sensitivity — the query keyword reassignment and QueryType removal are now case-insensitive and don't require a leading newline, so Query 50123 / { QueryType = Normal; ... } no longer break ID reassignment or create a duplicate QueryType.
• Object-ID conflict on re-run — execute_al_query now uninstalls/unpublishes the throwaway app before and after each run, so re-running locally against the same container doesn't hit a 50100/50101 conflict.

Docs/cleanup

• SKILL.md: OrderBy corrected to a property (OrderBy = descending(Col);), not a block.
• types.py: stale MCP/seed comments removed (folded DATA_QUERY into the existing match arms).
• PR description: validation scope updated to the current 11 gold queries; the 5 new ones are in runner shakeout now.

Added unit tests for the precision fix, the case-insensitive/malformed wrap_query_as_api paths, and the paging/cleanup wiring. Full suite: 635 pass, ruff + ty clean.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/bcbench/operations/bc_operations.py:293

• AL escapes a quote inside a quoted identifier by doubling it ("A ""quoted"" query"), not with a backslash. This regex stops at the first doubled quote, leaves the rest of the original name behind, and turns an otherwise valid query into invalid AL. Match doubled quotes in the quoted-name branch.

    text, replaced = re.subn(
        r'(\bquery\s+)\d+\s+("(?:[^"\\]|\\.)*"|\w+)',
        rf"\g<1>{object_id} {safe_name}",

src/bcbench/evaluate/dataquery.py:31

• This converts every numeric-looking string to a number, so distinct AL text/code values such as "001" and "1" compare equal (and the earlier None conversion similarly equates null with ""). That can award resolution to a query returning the wrong identifier. Preserve string/null identity and normalize only values known to be numeric, or carry type information into comparison.

    try:
        # Canonical decimal form: scale/trailing-zero-insensitive (500 == 500.0) but full precision
        # preserved, so distinct values like 1.00001 and 1.00002 are NOT collapsed. No float rounding.
        return str(Decimal(text).normalize())
    except (InvalidOperation, ValueError):

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

src/bcbench/results/base.py:116

• scorable=False is not propagated to the bc-eval records. category_metrics exports only resolved=False and build=True, and ResolutionRate/BuildRate score those values directly, so the externally reported headline metrics still count a gold-query failure as a resolution failure (and a build success), contrary to the new unscorable semantics. Export scorable and make the downstream evaluators skip these records, or omit unscorable records from the bc-eval export.

    def create_unscorable(cls, context: "EvaluationContext", output: str, error_message: str) -> Self:
        """A harness/dataset failure (not the agent's fault) that must not count toward the resolution rate."""
        return cls(**cls._base_fields(context), output=output, build=True, resolved=False, scorable=False, error_message=error_message)

src/bcbench/operations/bc_operations.py:426

• The OData JSON is parsed through Python float before _normalize_value sees it, so high-magnitude BC Decimal values can lose precision and distinct results can compare equal (for example, adjacent cent values near BC Decimal's upper range). Parse JSON decimal literals directly as Decimal to preserve the deterministic comparison promised by the matcher.

    rows = json.loads(result_file.read_text(encoding="utf-8-sig") or "[]")

src/bcbench/evaluate/dataquery.py:88

• The new pipeline's outcome logic is not covered by the added tests: there are no tests that mock execute_al_query and verify match, mismatch, generated build failure, and gold-query unscorable results. These branches define the benchmark's scores, and the current ordering/export issues are examples that helper-only tests do not catch. Add focused pipeline tests like those used for the existing evaluation pipelines.

    def evaluate(self, context: EvaluationContext[DataQueryEntry]) -> None:

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

src/bcbench/operations/bc_operations.py:305

• The transform is not comment-aware. In a valid query with a preceding comment such as // QueryType = Normal;, this substitution removes the comment occurrence because count=1, leaves the real property, and then injects a second QueryType, causing compilation to fail. Likewise, text.find("{") can select a brace in a leading comment. Locate the declaration/body with comment-aware parsing and remove the actual object-level property rather than the first textual match.

    text = re.sub(r"\bQueryType\s*=\s*\w+\s*;", "", text, count=1, flags=re.IGNORECASE)

    brace_index = text.find("{")
    if brace_index == -1:
        raise BuildError("query-wrap", f"Generated query has no object body ('{{' not found):\n{query_text}")

src/bcbench/agent/shared/instructions/dataquery-bc/skills/al-query-authoring/SKILL.md:27

• This example is effectively the gold solution for dataquery__outstanding-sales-value-by-customer-1: it uses the same Customer → Sales Line join, Order filter, and Outstanding Amount sum. Any run with this skill enabled receives the answer to a benchmark entry (including the first test-run entry), inflating that experiment's score. Replace it with a valid query pattern that is not represented in the dataset.

            dataitem(SalesLine; "Sales Line")
            {
                DataItemLink = "Sell-to Customer No." = Customer."No.";
                DataItemTableFilter = "Document Type" = const(Order);
                column(OutstandingAmount; "Outstanding Amount") { Method = Sum; }

src/bcbench/results/bceval_export.py:53

• This scoring-critical skip path has no regression coverage: tests/test_result_writer.py comprehensively exercises write_bceval_results, but no test creates an execution result with scorable=False. Add mixed and all-unscorable cases to verify these records never reach the bc-eval JSONL output.

            # Unscorable results (harness/dataset failures, e.g. a gold query that didn't compile) must
            # not reach the uploaded/core score, or they'd count against the agent's ResolutionRate.
            if isinstance(result, ExecutionBasedEvaluationResult) and not result.scorable:
                logger.info(f"Skipping unscorable result from bceval export: {result.instance_id}")
                continue

[Sun Haoran (haoranpb)]

[documentation] Isn't the rational behind blocking BC's OData /api is because, in Production (SaaS), a coding agent won't be able to query data this way?

If a coding agent is able to query API in production, it be a very interesting scenario to compare the result between using AL query object VS direct API calls.

---

Only block API endpoints if in production (SaaS), the API path is genuine unavailable.

[Onat Buyukakkus (onbuyuka)]

Good question — the rationale is exactly the prod path. In prod SaaS /api is not disabled, but it requires an OAuth bearer token (app registration or a user token); a coding agent will not have that configured, which is precisely why the MCP server (backed by APIs) exists. So blocking /api in eval approximates the realistic prod scenario where the agent''s only practical route to the data is the MCP tools.

You are right that if an agent did hold API creds in prod, AL-query-via-MCP vs. direct-API would be a genuinely interesting comparison — a nice future experiment, but it needs the agent to carry OAuth creds, which is out of scope here. (See the mcp_gateway.py thread for how the /api block is enforced and the drift analysis.) Leaving open in case you want to shape the wording further.

[Sun Haoran (haoranpb)]

[critical] Is there any other way that we could address this? e.g. disable API in the docker BC env?

Adding the MCP gateway introduce a drift between Production behavior and Evaluation, we would ike the two to be as close as possbile, so the performance gain in evaluation will most likely happen in Production.

[Onat Buyukakkus (onbuyuka)]

Good push — I dug into whether we can drop the gateway or disable /api in the container instead. Summary: the gateway can be trimmed but not eliminated today, and the drift is confined to transport/auth plumbing, not the behavior we measure.

Why not just disable /api in the BC container? The harness computes the gold rows by running the reference AL query over /api from inside the container (Invoke-ScriptInBcContainer -> http://localhost:7048/BC/api). BC on-prem serves /api and /mcp on the same port (7048) behind the same Basic auth, and there is no BC setting to disable only /api at the path level — ApiServicesEnabled=false kills the whole API surface the gold path depends on. A network/firewall block cannot help either, since it cannot distinguish /api from /mcp on one port. So enforcing "agent can reach /mcp but not /api" needs an L7 filter in front -> the gateway.

Is the drift meaningful? The MCP protocol path the agent exercises — tools/list, bc_data_find_tables / get_table_schema / get_table_relations / bc_data_query — is identical to prod. The gateway differs only in transport/auth plumbing: it injects the Basic auth upstream so the agent''s MCP config is credential-free (which also closes the credential-leak side-door). It does not change tool inputs/outputs, so the eval->prod transfer risk on the thing we measure (can the agent use the Data Query tools to get correct data) is low.

Prod realism of the /api block: in prod SaaS /api requires an OAuth bearer token (app registration / user token) a coding agent will not have; the API-backed MCP server is the realistic path, so blocking /api in eval approximates it. (Cross-linked with the docs /api thread.)

Maintainability: the gateway is a single module. Its non-isolation functions each solve a concrete BC-on-insider-29 issue rather than being gratuitous: credential injection (keeps creds out of the agent), the x-ms-headerless strip (a real Claude MCP-client incompatibility with BC''s initialize capabilities.experimental), and the tools/list warm-up cache (BC''s cold per-session catalog composition is slow and drops the connection). Those would still be needed even if isolation were solved another way.

Net: I would keep it, and revisit if BC ever (a) exposes /mcp on a separate port/instance from /api, or (b) we compute gold without /api. Open to a cleaner isolation primitive if you see one.