Skip to content

Start iterating on #1682 On branch edburns/1682-java-tool-ergonomics#1692

Closed
edburns wants to merge 7 commits into
mainfrom
edburns/1682-java-tool-ergonomics
Closed

Start iterating on #1682 On branch edburns/1682-java-tool-ergonomics#1692
edburns wants to merge 7 commits into
mainfrom
edburns/1682-java-tool-ergonomics

Conversation

@edburns

@edburns edburns commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator

new file: 1682-java-tool-ergonomics-prompts-remove-before-merge/20260615-prompts.md

@edburns edburns force-pushed the edburns/1682-java-tool-ergonomics branch from 6c3fe31 to 299c3e4 Compare June 16, 2026 23:52
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@edburns edburns force-pushed the edburns/1682-java-tool-ergonomics branch from 721eeb4 to ac67ecf Compare June 17, 2026 15:26
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generated by SDK Consistency Review Agent for issue #1692 · sonnet46 1.6M

Comment thread dotnet/test/E2E/ToolsE2ETests.cs Outdated
Description = "Custom grep override",
AdditionalProperties = new Dictionary<string, object?>
{
["is_override"] = true,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-SDK consistency: hardcoded internal key "is_override"

This line uses the string "is_override" directly, but that key is declared as internal const string OverridesBuiltInToolKey = "is_override" in CopilotTool.cs — it isn't part of the public API surface.

Every other SDK exposes this intent through a typed, discoverable API:

  • Node.js: overridesBuiltInTool: true (option in defineTool)
  • Python: overrides_built_in_tool=True (kwarg in @define_tool)
  • Go: grepTool.OverridesBuiltInTool = true (struct field)
  • Java: ToolDefinition.createOverride(...) (factory method)
  • Rust: .with_overrides_built_in_tool(true) (builder method)

Two options to bring .NET into alignment:

Option A — Use CopilotTool.DefineTool (the idiomatic .NET helper):

CopilotTool.DefineTool(
    CustomGrep,
    new CopilotToolOptions { OverridesBuiltInTool = true },
    new AIFunctionFactoryOptions { Name = "grep", Description = "Custom grep override" }),

This keeps the AIFunctionArguments-based parameter access (the "low-level" part of the test) while using the typed override flag.

Option B — Make OverridesBuiltInToolKey public so callers who intentionally use raw AIFunctionFactory.Create can reference a typed constant instead of a magic string:

["is_override"] = true,  // ← becomes: [CopilotTool.OverridesBuiltInToolKey] = true

Option A is preferred for consistency, since it still demonstrates the low-level AIFunctionArguments pattern without leaking an internal key.

@github-actions

This comment has been minimized.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generated by SDK Consistency Review Agent for issue #1692 · sonnet46 2.2M

Comment thread dotnet/test/E2E/ToolsE2ETests.cs Outdated
Description = "Custom grep override",
AdditionalProperties = new Dictionary<string, object?>
{
["is_override"] = true,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-SDK consistency note: This hardcodes the raw "is_override" string, which is the internal value of CopilotTool.OverridesBuiltInToolKey (currently internal const). In all other SDKs the low-level override path uses a named, public API element:

SDK Approach
Go grepTool.OverridesBuiltInTool = true (exported struct field)
Java ToolDefinition.createOverride(...) (dedicated factory)
Node.js overridesBuiltInTool: true (named option)
Python overrides_built_in_tool=True (named parameter)
Rust .with_overrides_built_in_tool(true) (builder method)

For .NET, callers doing the raw approach must know the magic string "is_override". Consider making CopilotTool.OverridesBuiltInToolKey public so this test (and any other low-level callers) can write:

[CopilotTool.OverridesBuiltInToolKey] = true,

instead of relying on the private string value. Alternatively, the test could use CopilotTool.DefineTool() with CopilotToolOptions { OverridesBuiltInTool = true } for the override tool — even in a "low-level" test, the other two tools still demonstrate the raw AIFunctionArguments path.

60_000).get(90, TimeUnit.SECONDS);

assertNotNull(response, "Expected a response from the assistant");
String content = response.getData().content().toLowerCase();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor null-safety concern: content() returns String and can be null if the field is absent in the JSON response. Calling .toLowerCase() directly on it would throw a NullPointerException.

All other SDK tests guard against this:

  • Node.js: const content = assistantMessage?.data.content ?? "";
  • Python: content = assistant_message.data.content or ""
  • Go: explicitly checks if content == "" { t.Fatalf(...) }
  • Rust: uses a helper assistant_message_content() that handles null

Suggest:

String content = response.getData().content();
assertNotNull(content, "Expected non-null content in assistant response");
assertFalse(content.isEmpty(), "Expected non-empty content");
assertTrue(content.toLowerCase().contains("analyzing"), ...);

@github-actions

This comment has been minimized.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generated by SDK Consistency Review Agent for issue #1692 · sonnet46 2.6M

Comment thread dotnet/test/E2E/ToolsE2ETests.cs Outdated
Description = "Custom grep override",
AdditionalProperties = new Dictionary<string, object?>
{
["is_override"] = true,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-SDK consistency note — .NET "low-level" override key discoverability

The magic string "is_override" here is intentionally testing the low-level AIFunctionFactory.Create path, and that's correct. However, CopilotTool.OverridesBuiltInToolKey (which equals "is_override") is declared internal, so any .NET consumer who wants to use AIFunctionFactory.Create directly (instead of CopilotTool.DefineTool) has no discoverable public constant to reference.

The equivalent APIs in every other SDK are public and discoverable:

  • Node.js: overridesBuiltInTool: true property on defineTool
  • Python: overrides_built_in_tool=True parameter on @define_tool
  • Go: grepTool.OverridesBuiltInTool = true public struct field
  • Java: ToolDefinition.createOverride() public factory method
  • Rust: .with_overrides_built_in_tool(true) public builder method

Suggestion: consider either (a) making OverridesBuiltInToolKey a public const to allow consumers to reference it without hardcoding the string, or (b) adding a comment here pointing to CopilotTool.DefineTool with new CopilotToolOptions { OverridesBuiltInTool = true } as the idiomatic alternative when AIFunctionFactory.Create isn't required.

@github-actions

This comment has been minimized.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generated by SDK Consistency Review Agent for issue #1692 · sonnet46 2.2M

Comment thread dotnet/test/E2E/ToolsE2ETests.cs Outdated
Description = "Custom grep override",
AdditionalProperties = new Dictionary<string, object?>
{
["is_override"] = true,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-SDK consistency note: All other SDKs expose overridesBuiltInTool as a typed, public API in their low-level tool definition (e.g., Java's ToolDefinition.createOverride(), Go's grepTool.OverridesBuiltInTool = true, Node.js's overridesBuiltInTool: true, Python's overrides_built_in_tool=True, Rust's .with_overrides_built_in_tool(true)). Here the .NET test sets "is_override" via a raw AdditionalProperties dictionary key, which is an internal SDK constant (CopilotTool.OverridesBuiltInToolKey).

Using CopilotTool.DefineTool() with CopilotToolOptions { OverridesBuiltInTool = true } would be the equivalent public SDK API and would align better with how other languages expose this feature:

CopilotTool.DefineTool(CustomGrep, new CopilotToolOptions
{
    OverridesBuiltInTool = true,
}, new AIFunctionFactoryOptions
{
    Name = "grep",
    Description = "Custom grep override",
}),

This is especially relevant since "is_override" is internal const — SDK users reading this test as documentation would copy a magic string that's not part of the documented public surface.

@edburns

edburns commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator Author

PROBLEM STATEMENT: PR 1692 is intractable because of its breadth

I observe that my approach to get this PR merged: #1692 is suboptimal because of its breadth of having changes that span dotnet, go, java, nodejs, python, and rust. Each push to the topic branch of this PR takes over an hour to run all the tests.

I want to break PR 1692 up into several PRs. Work in this session will do this break up.

The topic branch for PR 1692 is checked out at this git worktree:

C:/Users/edburns/workareas/copilot-sdk    150b8417 [edburns/1682-java-tool-ergonomics]

The remainder of this prompt describes how to go about breaking out the language specific elements from that topic branch into 6 new PRs.

❌❌ Do nothing to PR 1692. I will deal with that after our work here is done. ❌❌

SOLUTION APPROACH: BREAK PR 1692 into six other PRs

We have six other worktrees for the other phases of the breakouts, each with its own topic branch on the upstream, as shown next. Each of these has been added as a folder using /add-dir. When this work is done, all the aspects of PR 1692 will have been merged to main, and PR 1692 can be safely closed.

Note: equally important is the parallelism of the PRs. The next section, ORDERING, deals with that aspect.

PR 01: Java

Worktree and topic branch for this PR:

C:/Users/edburns/workareas/copilot-sdk-01-1682-java 8789b28b [edburns/1682-01-java-new-test-and-skill]

This PR will contain the following elements from PR 1692:

  • The skill new-java-e2e-test-yaml-and-test and all supporting files.

  • The net new test java/src/test/java/com/github/copilot/LowLevelToolDefinitionIT.java and all supporting artifacts. Most importantly test/snapshots/tools/low_level_tool_definition.yaml.

PR 02: go test

Worktree and topic branch for this PR:

C:/Users/edburns/workareas/copilot-sdk-02-1682-go   8789b28b [edburns/1682-02-go-new-test]

This PR will contain the following elements from PR 1692:

  • The net new test in go/internal/e2e/tools_e2e_test.go and all supporting files. Note that this test depends on "PR 01: Java" being already merged to main. This aspect will be covered in the section ORDERING.
PR 03: node js test

Worktree and topic branch for this PR:

C:/Users/edburns/workareas/copilot-sdk-03-1682-nodejs 8789b28b [edburns/1682-03-nodejs-new-test]

This PR will contain the following elements from PR 1692:

  • The net new test in nodejs/test/e2e/tools.e2e.test.ts.

  • Any other files to make the node tests run. While working on PR 1692, Copilot felt it necessary to make changes such as nodejs/test/e2e/session_lifecycle.e2e.test.ts and test/snapshots/session_lifecycle/should_list_created_sessions_after_sending_a_message.yaml. I am very skeptical that these are necessary. But it may be that this test addresses a bug that was preventing a clean run. You`ll have to figure it out and do the right thing regarding these other changes. There is an ordering issue here as well.

PR 04: python test

Worktree and topic branch for this PR:

C:/Users/edburns/workareas/copilot-sdk-04-1682-python 8789b28b [edburns/1682-04-python-new-test]

This PR will contain the following elements from PR 1692:

  • The net new test in python/e2e/test_tools_e2e.py.
PR 05: rust test

Worktree and topic branch for this PR:

C:/Users/edburns/workareas/copilot-sdk-05-1682-rust   8789b28b [edburns/1682-05-rust-new-test]

This PR will contain the following elements from PR 1692:

  • The net new test in rust/tests/e2e/tools.rs
PR 06: dotnet test

Worktree and topic branch for this PR:

C:/Users/edburns/workareas/copilot-sdk-06-1682-dotnet 8789b28b [edburns/1682-06-dotnet-new-test]

This PR will contain the following elements from PR 1692:

  • The net new test in dotnet/test/E2E/ToolsE2ETests.cs

ORDERING

For all PRs, you must closely observe the "checks" section and get clean runs an all the checks.

You'll have to wait for the human to perform the merging.

✅✅You must first do PR 01 all the way to merging.✅✅

❌❌❌Do not proceed with PR 02 - PR 06 until PR 01 is done and merged to main.❌❌❌

✅✅For each PR 02 - PR 06, you must fetch and rebase to HEAD of upstream/main before beginnnig work. This is how you get the dependencies you need.

✅✅Once PR 01 is merged, proceed with PRs 02 - PR 06 IN PARALLEL.✅✅ Because you have topic branches for each one, this is acceptable.

General notes

  • ❌❌❌ This entire job is about adding a test. If you find yourself wanting to change core product code, STOP and ask me why. There should be no need for that. ❌❌❌

  • Write a plan for performing the work to copilot-sdk\temporary-prompts\1682-breakout-plan.md. It must include a checklist of the PRs. For each PR, include checklist items such as:

    • Use cherry pick to get the relevant files from copilot-sdk.
    • Create the PR, list the PR number.
    • Push commit(s) containing the cherry picked files.
    • Clean run on all checks.

    Update this file as you go.

  • Use the signed-in gh CLI for all interactions with GitHub. Especially creating PRs.

  • Be reasonably verbose in the PR descriptions. These PRs are related to issue Java: Ergonomics: Defining tools #1682 but they do not fix issue Java: Ergonomics: Defining tools #1682. State that in the PR descriptions.

  • For all PRs 02 - 06, reference the PR for 01.

  • Use the existing topic branches on the worktrees as the topic branches for each PR.

@edburns

edburns commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator Author

edburns and others added 7 commits June 18, 2026 17:34
new file:   1682-java-tool-ergonomics-prompts-remove-before-merge/20260615-prompts.md

Signed-off-by: Ed Burns <edburns@microsoft.com>
Packages the knowledge of creating net-new Java E2E integration tests
with handcrafted replay proxy YAML snapshots into a reusable Copilot skill.

New files:

- .github/skills/new-java-e2e-test-yaml-and-test/SKILL.md
  Main skill instructions covering: YAML snapshot format, proxy matching
  logic, Java IT test template, verification commands, key classes/files
  reference table, and common pitfalls. Includes the critical constraint
  that Java's CapiProxy always sets GITHUB_ACTIONS=true so snapshots must
  be handcrafted rather than recorded.

- .github/skills/new-java-e2e-test-yaml-and-test/examples.md
  Two working examples from the codebase: a simple single-turn test
  (Botanica identity REPLACE) and a multi-turn test with tool calls
  (system message transform with view tool).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the same second prompt (Say hi) as the lifecycle snapshot to avoid replay cache misses in the list-sessions test path on Windows CI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@edburns edburns force-pushed the edburns/1682-java-tool-ergonomics branch from 150b841 to feab734 Compare June 18, 2026 21:34
@github-actions

Copy link
Copy Markdown
Contributor

Cross-SDK Consistency Review

Automated review by the SDK Consistency Agent

Summary

The actual SDK code changes in this draft PR are minimal and do not introduce any cross-SDK consistency issues:

File Change
nodejs/test/e2e/session_lifecycle.e2e.test.ts Test prompt changed: "Say world""Say hi"
test/snapshots/session_lifecycle/should_list_created_sessions_after_sending_a_message.yaml Snapshot updated to match the new prompt and expected response

No public API surface changes were detected. This is a pure test-string update with its corresponding shared snapshot update — no new features, method signatures, or behavioral changes that would require equivalent changes in other SDK implementations.

Pre-merge Reminder 🗑️

The directory 1682-java-tool-ergonomics-prompts-remove-before-merge/ contains 4 planning/notes files that the directory name indicates should be removed before this PR is merged:

  • 1682-low-level-tool-definition.md
  • 20260615-prompts.md
  • 20260616-prompts.md
  • add-tests-that-use-low_level_tool_definition.yaml.md

Verdict

No cross-SDK consistency issues found in the current committed changes. Once the Java tool ergonomics feature work (issue #1682) is implemented, that will warrant a fuller consistency review across all six SDK implementations.

Generated by SDK Consistency Review Agent for issue #1692 · sonnet46 915.1K ·

@edburns edburns closed this Jun 18, 2026
@edburns edburns deleted the edburns/1682-java-tool-ergonomics branch June 18, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant