feat: add Python IntelliSense for generated APIs

[Rana Singh (ranadeepsingh)]

Related Issues/PRs

Closes #1013.

What changes are proposed in this pull request?

SynapseML's generated Python wrappers expose runtime keyword names, but they do
not currently ship PEP 561 typing metadata. Editors therefore infer many
setters and optional parameters as Unknown, and package-level class discovery
is inconsistent.

This PR:

• generates adjacent .pyi files for generated transformers, estimators, and
  models, including typed constructors, setParams, Params, setters, getters,
  estimator model returns, and Python-only generated methods;
• infers precise scalar and collection types, including OpenAI
  ServiceParam values such as concurrency, temperature, and
  deploymentName, without changing runtime converters or the public
  ParamInfo JVM shape;
• generates explicit package stub exports and PEP 561 markers in
  component-owned namespace subpackages, and packages them in both component
  and aggregate wheels;
• adds a manual stub for the exceptional UDFTransformer wrapper, which does
  not inherit its generated base class;
• cleans the aggregate Python merge directory before packaging so removed
  generated files cannot survive into later wheels; and
• documents automatic Pylance/editor support after wheel installation.

The generated runtime .py execution structure remains unchanged; only
documentation type labels become more precise.

How is this patch tested?

• I have written tests and confirmed the proposed feature works.

Validation on the final rebased commit:

• 24 focused Scala codegen and type-inference tests passed.
• The full com.microsoft.azure.synapse.ml.core.** selector passed, including
  the production-stage discovery suite that caught test-only fixture leakage.
• Aggregate compile, Test / compile, scalastyle, and
  Test / scalastyle passed across all six modules.
• black==22.3.0 --check and git diff --check passed.
• Full codegen succeeded for core, deep-learning, cognitive, VW, LightGBM,
  and OpenCV.
• packageSynapseML produced an installable aggregate wheel containing 221
  .pyi files and 17 namespace-safe py.typed markers.
• Pyright reported zero errors for representative APIs, every generated class
  stub, and 140 package-level exports; the same stubs also passed with a
  Python 3.8 target.
• Negative Pyright coverage rejected 10 invalid constructor/setter calls with
  the expected types.
• Jedi resolved OpenAI classes and method completions, and reported all 73
  OpenAIPrompt constructor parameters including concurrency.
• The installed wheel imported successfully, and no .pyi file was loaded by
  the Python runtime.
• The public PythonWrappable trait has no new abstract JVM members compared
  with the target-branch binary surface.
• Normalized AST comparison found zero execution-structure differences across
  all 310 generated runtime .py files.
• Alternating cold import measurements showed no runtime regression
  (typed median 1.495s vs. baseline 1.521s). Comparable dirty-worktree codegen
  was within local noise (60.06s vs. 58.89s).
• Azure build
  232486498
  passed 76 checks, including UnitTests core. Its only failed job is the
  unrelated Fabric E2E cleanup, which aborted before running tests because
  GET https://api.fabric.microsoft.com/v1/workspaces returned HTTP 500 on
  both the initial run and a failed-job-only retry.

Does this PR change any dependencies?

• No. You can skip this section.
• Yes. Make sure the dependencies are resolved correctly, and list changes here.

Does this PR add a new feature? If so, have you added samples on website?

• No new transformer API is added. Installation documentation is updated.
• Yes. Make sure you have added samples following below steps.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Adds PEP 561 typing support for SynapseML’s generated Python wrappers by emitting .pyi stubs and py.typed markers during codegen and ensuring both component and aggregate wheels package these typing artifacts, improving editor IntelliSense for constructors, params, setters/getters, and estimator model returns.

Changes:

• Generate per-wrapper .pyi stub files (including inferred types for ServiceParam) alongside existing runtime .py wrappers.
• Generate __init__.pyi stub exports and py.typed markers for typed subpackages; include *.pyi/py.typed in wheel package_data.
• Add targeted Scala tests for stub/type inference behavior and clean the aggregate merged Python directory before packaging.

Show a summary per file

File	Description
docs/Get Started/Install SynapseML.md	Documents that wheels ship PEP 561 typing metadata and how editors pick it up.
core/src/test/scala/com/microsoft/azure/synapse/ml/codegen/VerifyDefaultParamInfo.scala	Adds tests validating pyi type strings and ServiceParam stub-type inference.
core/src/test/scala/com/microsoft/azure/synapse/ml/codegen/PyCodegenSuite.scala	Adds end-to-end tests asserting generated .pyi, __init__.pyi, and wheel packaging behavior.
core/src/main/scala/com/microsoft/azure/synapse/ml/codegen/Wrappable.scala	Emits .pyi stubs for generated wrappers (typed __init__, setParams, param accessors, and extra methods).
core/src/main/scala/com/microsoft/azure/synapse/ml/codegen/PyCodegen.scala	Generates __init__.pyi exports, creates py.typed markers, and packages typing artifacts.
core/src/main/scala/com/microsoft/azure/synapse/ml/codegen/DefaultParamInfo.scala	Introduces PythonTypeInfo + reflection-based type inference for more precise stub types.
core/src/main/python/synapse/ml/stages/UDFTransformer.pyi	Adds a manual stub for the exceptional UDFTransformer wrapper.
build.sbt	Ensures aggregate wheel includes typing artifacts and cleans merged Python sources before packaging.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 7/8 changed files
• Comments generated: 2
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

• Files reviewed: 7/8 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

core/src/main/scala/com/microsoft/azure/synapse/ml/codegen/PyCodegen.scala:141

• packageFolder becomes values like "/services" in the recursion (packageFolder + "/" + f.getName). Passing that directly into FileUtilities.join(conf.pySrcDir, ..., packageFolder) can cause dir to resolve to an absolute filesystem path (because the last segment starts with "/"), which would make init/stub generation read/write outside conf.pySrcDir. Normalize packageFolder (strip the leading slash) for filesystem paths, while keeping the original packageFolder for the special-case comparisons ("/services", "/cognitive").

      }
    }
    makeInitStub(conf, dir, packageFolder, packageString)
    dir.listFiles().filter(_.isDirectory).foreach(f =>
      makeInitFiles(conf, packageFolder + "/" + f.getName)

• Files reviewed: 7/8 changed files
• Comments generated: 1
• Review effort level: Lite