Conversation
Adds a read-only operational review for Amazon SageMaker AI workloads -- endpoints, training jobs, pipelines, notebooks, feature store, model registry, and Studio domains -- covering 8 pillars and 20 checks: Security, Performance, Cost Optimization, Service Quotas, Resiliency, Operational Excellence, Sustainability, and Best Practices. Findings use a uniform High / Medium / Low scale, plus Informational for inventory checks with no pass/fail signal. Every High or Medium finding carries exactly one concrete recommendation, and the report leads with a severity-ranked Executive Summary. Findings are keyed per resource, so counts stay comparable between runs. The Best Practices pillar is grounded in the public AWS Well-Architected Machine Learning, Generative AI, and Agentic AI lenses. The skill is strictly read-only: List/Describe/Get control-plane calls and CloudWatch metric reads only, with no data-plane calls, no endpoint invocation, and no inference payload reads. IAM: AIDevOpsAgentAccessPolicy already covers every API the skill calls except savingsplans:DescribeSavingsPlans, which is an optional add-on -- without it the Savings Plan check reports "not evaluated - permission not granted" and the other 19 checks run normally. references/iam-policy.json carries that single statement. Notable check behaviour, each verified against a live account: - Autoscaling detection reads three states. Managed instance scaling, or an Application Auto Scaling target on sagemaker:variant:DesiredInstanceCount with at least one scaling policy, counts as autoscaled. A target registered without a policy only declares capacity bounds and never triggers a scaling action, so it is reported as its own finding rather than passing as healthy. - Serverless variants are scored Informational in checks covering features Serverless Inference does not support (VPC configuration, network isolation, data capture), so the report never emits a recommendation the operator cannot act on. - Service Quotas reads applied limits via servicequotas:GetServiceQuota, never the AWS defaults, and scores utilization from CloudWatch AWS/Usage ResourceCount over a trailing 24 hours at period 3600. SageMaker publishes those metrics roughly every 20 minutes with ingestion lag, so shorter windows return no datapoints. Missing usage data yields Unknown, never an inferred 0%. - The tagging check counts only user-defined tags, since SageMaker auto-injects sagemaker:domain-arn, user-profile-arn and space-arn on every Studio-created resource. Auto-generated model-monitoring-* processing jobs are excluded as they are not operator-taggable. - AWS Health findings are keyed per affected entity and filtered to the in-scope regions. ACTION_REQUIRED events that are open or upcoming carry Medium, since they represent externally-imposed deadlines. - Quotas, limits, prices, costs and percentage savings appear only when an API returned them; the skill does not estimate or recall them. Region discovery uses Cost Explorer with a sagemaker:List* sweep as fallback, so a payer-scoped Cost Explorer miss cannot produce a false "no activity" result. Checks are isolated: an AccessDenied or API error becomes an error row on that check and never aborts the review.
Wires the new sagemaker-ops-review skill into the existing operational review agent rather than shipping a second review agent, so SageMaker AI joins EKS, RDS, Aurora and Bedrock under one entry point. SYSTEM_PROMPT.md: SageMaker AI added to the Goal and to the service identification step, sagemaker-ops-review added to the skill selection list, and a SageMaker artifact naming example added. The skill's report schema is named in the existing "defer to the selected skill's report schema" guidance, alongside the bedrock-operation-review example, because sagemaker-ops-review defines its own eight pillars, a verbatim AI Disclaimer, and a severity-ranked Executive Summary that should not be forced into the generic category set. README.md: SageMaker AI added to Purpose, Key Capabilities, Prerequisites and Related, and the skill selection step now reflects that the skills are chosen per service rather than always both. The Prerequisites entry notes that AIDevOpsAgentAccessPolicy covers every API the skill calls except the optional savingsplans:DescribeSavingsPlans. CHANGELOG.md: bumped to 1.1.0. llms.txt: sagemaker-ops-review added to Available Skills, and the AWS Operation Review entry now lists SageMaker AI. cloudformation/devops-agent-skill-policies.yaml: adds the EnableSageMakerOpsReview parameter, its condition, a PolicySageMakerOpsReview resource granting savingsplans:DescribeSavingsPlans, and a SkillPolicySummary line. Every other API the skill calls is already covered by the managed policy, so this is the only addition required. Note on skill agent type: this skill's README instructs uploading with "Generic" / "All agents" selected rather than narrowing to specific agent types, because a narrowed skill does not appear in the custom agent's skill picker. The aws-operation-review README already documents that as a workaround for the EKS and RDS skills; sagemaker-ops-review states it up front instead, so the caveat is not needed for it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a
sagemaker-ops-reviewskill for read-only operational reviews of Amazon SageMaker AIworkloads, and wires it into the existing
aws-operation-reviewcustom agent rather thanshipping a second review agent.
SageMaker AI was the remaining gap in the operation-review family alongside EKS, RDS/Aurora and
Bedrock. The skill covers 8 pillars / 20 checks — Security, Performance, Cost Optimization,
Service Quotas, Resiliency, Operational Excellence, Sustainability, and Best Practices — over
endpoints, training jobs, pipelines, notebooks, feature store, model registry and Studio domains.
Findings use a uniform High / Medium / Low scale plus Informational for inventory checks, are keyed
per resource so counts stay comparable between runs, and every High or Medium finding carries
exactly one concrete recommendation. The report leads with a severity-ranked Executive Summary. The
Best Practices pillar is grounded in the public Well-Architected ML, Generative AI and Agentic AI
lenses.
Strictly read-only:
List/Describe/Getcontrol-plane calls and CloudWatch metric reads only —no data-plane calls, no endpoint invocation, no inference payload reads.
Design notes for reviewers:
aws-operation-reviewinstead of adding an agent. SageMaker AI joins the existingservice list; the agent's "defer to the selected skill's report schema" clause already
accommodates a skill-specific report structure, so no format conflict.
CHANGELOGbumped to1.1.0.
AIDevOpsAgentAccessPolicycovers every API the skill calls exceptsavingsplans:DescribeSavingsPlans. Verified against the live managed policy (v10) — includingthat
health:Describe*is already present, so the Health check needs no addition beyond aBusiness/Enterprise Support plan. Added as an optional parameter in
cloudformation/devops-agent-skill-policies.yaml; without it the Savings Plan check reports"not evaluated — permission not granted" and the other 19 run normally.
custom agent's skill picker.
custom-agents/aws-operation-review/README.mdalready documents thisas a workaround for the EKS and RDS skills; this skill's README states it up front so the caveat
isn't needed for it. The same latent issue likely affects
agentcore-ops-review, which also shipsa companion agent.
service-quota-checkis general-purpose; this skill's Service Quotas pillar isSageMaker-scoped to seven verified quota codes with utilization-derived severity.
aiml-access-diagnosticsdiagnoses SageMaker access failures during an incident; this reviewsposture. Both are cross-referenced from the skill README.
Type of change
Testing
Five end-to-end runs through DevOps Agent against a test account with real SageMaker resources
(endpoints, notebooks, a Studio domain, pipelines, lifecycle configs, training jobs), across
us-east-1 and us-west-2, on the base
AIDevOpsAgentAccessPolicywithout the optional add-on.Baseline runs without the skill were done first for comparison — unaided, the agent produced
inconsistent structure across iterations and missed the dual-signal autoscaling case entirely.
Verified against live resources:
ManagedInstanceScalingabsent is correctly reported as autoscaled — a managed-scaling-only checkwould false-positive here. A registered target with no scaling policy is reported as its own
finding, since capacity bounds alone never trigger a scaling action.
use_awscall in the invocation trajectory inspected across runs.permission not granted", never a false "no plans found".
empty-state rows, and one-recommendation-per-High/Medium (13 recommendations for 1 High + 12
Medium; none for 7 Low).
GetServiceQuota, utilization fromAWS/UsageResourceCountover a trailing 24 hours. SageMaker publishes those metrics roughly every 20minutes with ingestion lag, so shorter windows return no datapoints; missing data yields Unknown
rather than an inferred 0%.
Serverless Inference doesn't support (VPC configuration, network isolation, data capture), so the
report never emits a recommendation the operator can't act on.
"No SageMaker AI activity detected."
Not reachable in a test account (logic-reviewed only, stated for transparency):
ml.inf*/ml.trn*quota is 0 in a fresh account — empty state onlyOpen question for maintainers: availability of
servicequotasthroughuse_awsappearsintermittent — the same account returned all seven applied limits in one run and
"service unavailable" in the next. The check now retries once and otherwise degrades to
"not evaluated" rather than inventing limits, but is
servicequotasexpected to be reliably exposed?If not, I'm happy to document it as a known limitation in the skill README.
Skill evaluation tool: not run — the tool isn't published.
evals/eval_queries.jsonis includedwith trigger and no-trigger cases. Tagging @aws/tools-for-devops-agent-admins to request an
evaluation run, and happy to iterate on the results.
License confirmation
Two things to check before you submit: