feat(serve): add opt-in model source tag-based resource reuse by amazeAmazing · Pull Request #5993 · aws/sagemaker-python-sdk

amazeAmazing · 2026-07-07T14:53:50Z

Model Source Tag-Based Resource Reuse

Summary

Adds opt-in resource reuse to the V3 serve builders so redeploying the same model artifacts doesn't create duplicate resources (or exhaust quota-limited Bedrock imported models). Each deploy tags the resources it creates (Model + Endpoint) with a stable model-source identifier; when reuse is requested, a later call for the same source discovers and returns the existing resources instead of creating new ones.

Controlled by reuse_resources: bool = False. On a hit we log a warning and return the existing resource (we do not raise).

Behavior

	BedrockModelBuilder	ModelBuilder (SageMaker)
Flag	`deploy(reuse_resources=)`	`build(reuse_resources=)` + `deploy(reuse_resources=)`
Build discovery	—	`list_models` + model-source tag → reuses existing Model
Deploy discovery	Bedrock `list_custom_models` + tags	SageMaker `list_endpoints` + endpoint tag
Reuses	custom model + active deployment	Model (build) + endpoint (deploy)
Config re-validation	no	yes (env vars, image URI, instance type on model-on-variant endpoints)
On match	warn + return	warn + return

reuse_resources is not inherited: set it on build() to skip Model creation and on deploy() to reuse the endpoint.

Inference components: IC deployments (inference_config is a ResourceRequirements, or a modelbuilder_list build) are not intercepted by reuse_resources at the deploy level. They manage their own reuse by endpoint_name (infrastructure reuse) and inference_component_name (IC update). Passing reuse_resources=True on an IC deploy logs a warning that the flag is ignored. However, build(reuse_resources=True) still works for IC-based models — it finds the existing Model by tag and avoids creating a duplicate.

Nova inference-component deployment

Nova model-customization deploys support inference components. When a ResourceRequirements inference_config is supplied, the Nova checkpoint is hosted as a single inference component referencing the built Model (which carries the Nova image, escrow artifacts, and env); without one, it keeps the direct model-on-variant path.

_is_nova_model identifies Nova from the model package recipe/hub-content name, and also from a package-less source (raw S3 checkpoint or trainer) via base_model_name.
The IC endpoint config sets EnableNetworkIsolation to match the built Model (always True for Nova), fixing a CreateInferenceComponent rejection on mismatched network isolation.
Model-package-dependent logic (restricted-package path, PEFT/recipe metadata, lineage tracking) is guarded so package-less Nova checkpoints deploy cleanly.
Endpoints created on the shared IC path are tagged with the model-source tag so they stay discoverable.

Model-tag reuse

Both Nova and OSS (non-Nova) Models are tagged with the model-source identifier at build time. build(reuse_resources=True) searches existing Models by tag (_find_reusable_model) and skips Model.create on a match, regardless of whether the endpoint is model-on-variant or IC-based.

Checkpoint URI resolution

resolve_nova_checkpoint_uri handles all three Nova output layouts:

HyperPod: <output>/<job>/manifest.json
Serverless: <output>/<job>/output/output/manifest.json
Serverful: <output>/<job>/output/output.tar.gz (manifest inside)

Tries each in turn. If all fail, surfaces every error in the raised exception.

When reuse works and when it does not

Scenario	Reuse supported?	Notes
Same source, same config, model-on-variant endpoint (Nova)	✅	Model found by tag (build skips); endpoint found by tag + config match (deploy returns existing)
Same source, same config, IC-based endpoint (OSS, Nova IC)	✅	Model found by tag (build skips); endpoint found by tag (deploy returns existing)
Same source, different config (env/image/instance)	❌ creates new	Config mismatch detected at deploy → new endpoint created. Model still reused at build if source matches.
Explicit IC deploy (`inference_config=ResourceRequirements(...)`)	Model reuse only	`build(reuse_resources=True)` finds Model by tag. `deploy()` warns flag is ignored and creates the IC normally.
`modelbuilder_list` / `CustomOrchestrator` build	N/A	`is_inference_component_build=True` → reuse gate skipped at build time
Bedrock custom model	✅	Tag match on the custom model; also reuses active deployment

Key changes

New sagemaker/serve/model_reuse.py: tag helpers + service-client discovery (find_existing_bedrock_model, find_active_bedrock_deployment_for_model, find_existing_sagemaker_endpoint) with status gating.
sagemaker/core/training/utils.py: shared Nova manifest/checkpoint helpers (consolidated from duplicated logic), three-layout resolution (HyperPod/serverless/serverful).
ModelBuilder:
- build(): tags the Model with model-source; _find_reusable_model searches by tag (paginated) to skip duplicate creation.
- deploy(): endpoint reuse gate with config validation; IC deploy warns reuse has no effect.
- self.instance_type set from caller's explicit value before _deploy_model_customization.
- EnableNetworkIsolation on IC endpoint configs matches the built Model.
- Raw-S3 model input via model_metadata={"BASE_MODEL_NAME": ...}.
BedrockModelBuilder: reuse of custom model + deployment; consolidated source resolution; response always includes modelArn.

Tests

Unit: test_model_reuse.py, test_model_builder.py (Nova IC routing, network isolation, model-on-variant fallback), test_bedrock_model_builder.py, core test_training_utils.py (3-layout resolution).
Integ (in-repo): model-source tag assertion on OSS deploy (test_model_customization_deployment.py), Nova Bedrock e2e (test_nova_model_customization_deployment.py).
Manually tested: Bedrock reuse, SageMaker endpoint reuse (Nova), and OSS reuse (Model + endpoint),

vaishnavi-kommaraju · 2026-07-07T15:05:30Z

+    """
+    manifest_uri = build_nova_manifest_s3_uri(s3_output_path, training_job_name)
+    try:
+        return read_nova_checkpoint_uri_from_manifest(s3_client, manifest_uri)


what if there's no manifest.json saved in the output folder and it is not a tar.gz file (for hyperpod)? wouldn't we in that case incorrectly throw an error that manifest file not found in the tar.gz output path?

Thanks for the callout! I will adjust

Yup training outputs in SMTJ are tar.gz files by default.

Add reuse_resources to ModelBuilder.build/deploy and BedrockModelBuilder.deploy. On a hit, discover an existing resource by the model-source tag and return it instead of creating a duplicate (warn, do not raise). Honored per call. - New sagemaker/serve/model_reuse.py: tag helpers + service-client discovery - Consolidate Nova manifest/checkpoint reading into sagemaker/core/training/utils.py - SageMaker: build() skips Model creation on reuse (sets built_model to the existing Model); deploy() reuses the endpoint after validating env vars/image/ instance type (PrimaryContainer with Containers[0] fallback for Nova) - Reuse gates are skipped for inference-component builds/deploys so IC create/update (via _deploy_for_ic) is never silently intercepted - Bedrock: reuse custom model + active deployment; response includes modelArn - Reuse discovery uses the cached session/bedrock clients - Support raw S3 URI model input via model_metadata BASE_MODEL_NAME - Unit tests + notebook examples

…euse Route Nova model-customization deploys through the shared single-inference -component path when a ResourceRequirements inference_config is supplied, so each Nova checkpoint (full-rank or LoRA-merged) is hosted as one inference component referencing the built Model. Nova without an inference_config keeps the direct model-on-variant path. - Broaden _is_nova_model to identify Nova from a package-less source (raw S3 checkpoint or trainer) via base_model_name, in addition to the model package recipe/hub-content name. - Set EnableNetworkIsolation on the IC endpoint config to match the built Model (always True for Nova), fixing CreateInferenceComponent rejection on mismatched network isolation. - Guard model-package-dependent logic (restricted-package path, PEFT/recipe metadata, lineage tracking) so package-less Nova checkpoints deploy cleanly. - Apply accumulated tags (including the model-source reuse tag) to endpoints created on the shared IC path so they remain discoverable. - build(reuse_resources=True) only short-circuits when the backing Model can be resolved; IC endpoints and stale/deleted configs fall through and build a real Model, preventing a None built_model on later IC deploys. - deploy() warns that reuse_resources has no effect for inference-component deployments, which manage their own reuse by component name. - Surface both the manifest.json and output.tar.gz errors when Nova checkpoint URI resolution fails, instead of masking the primary failure. Add unit tests covering the Nova IC path (routing, network isolation, IC spec) and the model-on-variant fallback.

mujtaba1747 · 2026-07-08T16:16:21Z

+        Fully-qualified S3 URI to the job's manifest.json.
+    """
+    output_path = s3_output_path.rstrip("/")
+    return f"{output_path}/{training_job_name}/output/output/manifest.json"


The output path for manifest json may be different for serverless and hyperpod.

For serverless: training_job_name/output/output/manifest.json
For SMHP: training_job_name/manifest.json

Can we test this for both flows.

Thanks for calling this out! I will push a new revision to address this gap.

I will test all three flows - Serverless, Serverful and HP

…ayouts Nova training jobs write their checkpoint manifest to different locations depending on the training platform: HyperPod: <output>/<job>/manifest.json Serverless: <output>/<job>/output/output/manifest.json Serverful: <output>/<job>/output/output.tar.gz (manifest inside) resolve_nova_checkpoint_uri previously only tried the serverless manifest path and the serverful tar.gz, so HyperPod jobs (manifest directly under the job directory) failed to resolve. Add build_nova_hyperpod_manifest_s3_uri and try all three layouts in turn, aggregating every failure into the raised error so the real cause is not masked by the last attempt's message. Add unit tests for the HyperPod builder and for resolution from the HyperPod and serverless layouts.

mujtaba1747 · 2026-07-08T23:49:48Z

        )
        assert deployment.get("status") == "Active"

+    def test_nova_bedrock_custom_model_tagged_for_reuse(self, deployed_nova_model, bedrock_client):


can we also invoke this deployed model as part of the test, this would make it e2e.

This file's test already invokes the deployed model in the Bedrock class sequence (TestNovaBedrockDeployment)

test_nova_bedrock_invoke does the invoking.

Simplify reuse discovery by tagging SageMaker Models (not just endpoints) with the model-source identifier, so build(reuse_resources=True) can find and skip recreating an existing Model directly — no IC-state dependency. - Tag non-Nova Models at build time with the model-source tag (matching the Nova path's existing behavior). Both Nova and OSS Models are now discoverable by tag. - Add _find_reusable_model: build(reuse_resources=True) searches Models by source tag, skipping Model creation on a hit. Also discovers the endpoint for deploy() to reuse later. - Simplify _get_model_for_endpoint back to variant-only lookup (returns None for IC endpoints). No longer needs IC-spec resolution since the Model is found directly by tag. - _reused_endpoint_matches_config returns True for IC endpoints (can't read container config from variant; Model was already matched by tag). - deploy() with reuse_resources=True on an IC deploy logs a warning that the flag has no effect (ICs manage reuse by endpoint_name + IC name). - Fix deploy() to set self.instance_type from the caller's explicit value before calling _deploy_model_customization, preventing recipe-resolved defaults from overriding the user's intent. - Add model-source tag assertion to the existing OSS deploy integ test.

mujtaba1747 · 2026-07-10T18:12:44Z

Approved. Tests ran on smoketest (nova) account. Any changes needed for integ tests to run on PySDK acc to be tracked separately.

vaishnavi-kommaraju reviewed Jul 7, 2026

View reviewed changes

amazeAmazing force-pushed the model-reuse-only branch from 99bedb4 to bca9dee Compare July 7, 2026 17:56

vaishnavi-kommaraju approved these changes Jul 7, 2026

View reviewed changes

mujtaba1747 reviewed Jul 8, 2026

View reviewed changes

amazeAmazing force-pushed the model-reuse-only branch 3 times, most recently from 4a88ba8 to 6e8ac17 Compare July 10, 2026 15:36

amazeAmazing force-pushed the model-reuse-only branch from 6e8ac17 to 8ba3b82 Compare July 10, 2026 17:26

Merge branch 'master-nova-follow-ups' into model-reuse-only

a4e8c65

mujtaba1747 approved these changes Jul 10, 2026

View reviewed changes

mujtaba1747 merged commit 37c371e into aws:master-nova-follow-ups Jul 10, 2026
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(serve): add opt-in model source tag-based resource reuse#5993

feat(serve): add opt-in model source tag-based resource reuse#5993
mujtaba1747 merged 5 commits into
aws:master-nova-follow-upsfrom
amazeAmazing:model-reuse-only

amazeAmazing commented Jul 7, 2026 •

edited

Loading

Uh oh!

vaishnavi-kommaraju Jul 7, 2026

Uh oh!

amazeAmazing Jul 7, 2026

Uh oh!

mujtaba1747 Jul 8, 2026

Uh oh!

mujtaba1747 Jul 8, 2026

Uh oh!

amazeAmazing Jul 8, 2026

Uh oh!

mujtaba1747 Jul 8, 2026

Uh oh!

amazeAmazing Jul 9, 2026

Uh oh!

mujtaba1747 commented Jul 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

amazeAmazing commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Model Source Tag-Based Resource Reuse

Summary

Behavior

Nova inference-component deployment

Model-tag reuse

Checkpoint URI resolution

When reuse works and when it does not

Key changes

Tests

Uh oh!

vaishnavi-kommaraju Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

amazeAmazing Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

mujtaba1747 Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

mujtaba1747 Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

amazeAmazing Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

mujtaba1747 Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

amazeAmazing Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

mujtaba1747 commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

amazeAmazing commented Jul 7, 2026 •

edited

Loading

mujtaba1747 commented Jul 10, 2026 •

edited

Loading