From e0c8e1f0ccd488cd32623037e5eae847bc47e9d2 Mon Sep 17 00:00:00 2001 From: Jack Lunn Date: Fri, 18 Sep 2026 17:20:30 -0500 Subject: [PATCH 1/2] feat(skills): add sagemaker-ops-review skill Adds a read-only operational review for Amazon SageMaker AI workloads -- endpoints, training jobs, pipelines, notebooks, feature store, model registry, and Studio domains -- covering 8 pillars and 20 checks: Security, Performance, Cost Optimization, Service Quotas, Resiliency, Operational Excellence, Sustainability, and Best Practices. Findings use a uniform High / Medium / Low scale, plus Informational for inventory checks with no pass/fail signal. Every High or Medium finding carries exactly one concrete recommendation, and the report leads with a severity-ranked Executive Summary. Findings are keyed per resource, so counts stay comparable between runs. The Best Practices pillar is grounded in the public AWS Well-Architected Machine Learning, Generative AI, and Agentic AI lenses. The skill is strictly read-only: List/Describe/Get control-plane calls and CloudWatch metric reads only, with no data-plane calls, no endpoint invocation, and no inference payload reads. IAM: AIDevOpsAgentAccessPolicy already covers every API the skill calls except savingsplans:DescribeSavingsPlans, which is an optional add-on -- without it the Savings Plan check reports "not evaluated - permission not granted" and the other 19 checks run normally. references/iam-policy.json carries that single statement. Notable check behaviour, each verified against a live account: - Autoscaling detection reads three states. Managed instance scaling, or an Application Auto Scaling target on sagemaker:variant:DesiredInstanceCount with at least one scaling policy, counts as autoscaled. A target registered without a policy only declares capacity bounds and never triggers a scaling action, so it is reported as its own finding rather than passing as healthy. - Serverless variants are scored Informational in checks covering features Serverless Inference does not support (VPC configuration, network isolation, data capture), so the report never emits a recommendation the operator cannot act on. - Service Quotas reads applied limits via servicequotas:GetServiceQuota, never the AWS defaults, and scores utilization from CloudWatch AWS/Usage ResourceCount over a trailing 24 hours at period 3600. SageMaker publishes those metrics roughly every 20 minutes with ingestion lag, so shorter windows return no datapoints. Missing usage data yields Unknown, never an inferred 0%. - The tagging check counts only user-defined tags, since SageMaker auto-injects sagemaker:domain-arn, user-profile-arn and space-arn on every Studio-created resource. Auto-generated model-monitoring-* processing jobs are excluded as they are not operator-taggable. - AWS Health findings are keyed per affected entity and filtered to the in-scope regions. ACTION_REQUIRED events that are open or upcoming carry Medium, since they represent externally-imposed deadlines. - Quotas, limits, prices, costs and percentage savings appear only when an API returned them; the skill does not estimate or recall them. Region discovery uses Cost Explorer with a sagemaker:List* sweep as fallback, so a payer-scoped Cost Explorer miss cannot produce a false "no activity" result. Checks are isolated: an AccessDenied or API error becomes an error row on that check and never aborts the review. --- skills/sagemaker-ops-review/.skilleval.yaml | 9 + skills/sagemaker-ops-review/CHANGELOG.md | 25 ++ skills/sagemaker-ops-review/README.md | 199 ++++++++++ skills/sagemaker-ops-review/SKILL.md | 181 +++++++++ .../evals/eval_queries.json | 12 + .../references/iam-policy.json | 13 + .../references/pillar-checks.md | 359 ++++++++++++++++++ 7 files changed, 798 insertions(+) create mode 100644 skills/sagemaker-ops-review/.skilleval.yaml create mode 100644 skills/sagemaker-ops-review/CHANGELOG.md create mode 100644 skills/sagemaker-ops-review/README.md create mode 100644 skills/sagemaker-ops-review/SKILL.md create mode 100644 skills/sagemaker-ops-review/evals/eval_queries.json create mode 100644 skills/sagemaker-ops-review/references/iam-policy.json create mode 100644 skills/sagemaker-ops-review/references/pillar-checks.md diff --git a/skills/sagemaker-ops-review/.skilleval.yaml b/skills/sagemaker-ops-review/.skilleval.yaml new file mode 100644 index 0000000..fc94439 --- /dev/null +++ b/skills/sagemaker-ops-review/.skilleval.yaml @@ -0,0 +1,9 @@ +# skill-eval audit configuration +# See: https://github.com/aws-samples/sample-agent-skill-eval +audit: + ignore: + # README.md alongside SKILL.md is intentional and required by this repo's + # contribution guide (README carries the non-production disclaimer, + # prerequisites, and upload steps). Matches the convention used by the + # other skills in this repository. + - STR-016 diff --git a/skills/sagemaker-ops-review/CHANGELOG.md b/skills/sagemaker-ops-review/CHANGELOG.md new file mode 100644 index 0000000..a5b4e12 --- /dev/null +++ b/skills/sagemaker-ops-review/CHANGELOG.md @@ -0,0 +1,25 @@ +# Changelog + +## [1.0.0] - 2026-09-18 + +### Added +- Initial release of the `sagemaker-ops-review` skill for AWS DevOps Agent. +- 20 read-only checks across 8 pillars — Security, Performance, Cost Optimization, Service Quotas, Resiliency, Operational Excellence, Sustainability, and Best Practices — defined authoritatively in `references/pillar-checks.md`. +- Security checks: notebook KMS encryption, Studio domain network posture (`VpcOnly` vs `PublicInternetOnly`), and endpoint/model `VpcConfig` coverage including Inference Component endpoints. +- Performance checks: endpoint inference type classification (Real-Time / Serverless / Asynchronous) and 7-day `ModelLatency` / `OverheadLatency` reporting from the `AWS/SageMaker` CloudWatch namespace. +- Cost Optimization checks: resource tagging coverage, Trainium/Inferentia adoption, endpoint autoscaling, SageMaker Savings Plan coverage, lifecycle configuration inventory, Inference Recommender job inventory, and 90-day stale endpoint detection. +- Service Quotas check across seven SageMaker quota codes, each verified against the live `sagemaker` service, with usage read from CloudWatch `AWS/Usage`/`ResourceCount` over a trailing 24-hour window at `period 3600` and the risk tier derived from utilization (≥ 90% High, ≥ 75% Medium, else Low; Unknown when no usage metrics exist). The 24-hour window is deliberate — SageMaker publishes these metrics about every 20 minutes with ingestion lag, so shorter windows return no datapoints and score every quota `Unknown`. +- Resiliency checks: per-variant endpoint instance counts and AWS Health SageMaker lifecycle events. +- Operational Excellence checks: SageMaker Projects, SageMaker Pipelines, and endpoint data capture configuration. +- Sustainability check: Studio domain region inventory. +- Best Practices pillar: advisory recommendations grounded in the public AWS Well-Architected Machine Learning, Generative AI, and Agentic AI lenses. +- Uniform severity model — High / Medium / Low, plus Informational for inventory checks with no pass/fail signal — with exactly one recommendation per High or Medium finding and a severity-ranked Executive Summary that must reconcile with the per-check sections. +- Findings are keyed per resource (`check`, `region`, `resource`), never aggregated into a single row per check, so severity counts stay comparable between runs. +- Serverless endpoints are scored Informational — never Low/Medium/High — in checks covering features Serverless Inference does not support (VPC configuration, network isolation, data capture, Model Monitor), so the report never emits a recommendation the operator cannot act on. +- AWS Health findings are keyed per affected entity and filtered to the in-scope regions, so a single multi-resource event produces one finding per resource and a region-scoped review never reports out-of-region resources. +- The tagging check excludes SageMaker-generated `model-monitoring-*` processing jobs, which are not operator-taggable and accumulate without limit. +- Explicit prohibition on stating any quota, limit, instance price, monthly cost, or percentage saving that an API did not return — including applied-vs-default quota limits, which must come from `servicequotas:GetServiceQuota` and never from `GetAWSDefaultServiceQuota`. +- Dual-signal, three-state autoscaling detection: managed instance scaling, or an Application Auto Scaling target on `sagemaker:variant:DesiredInstanceCount` **with** at least one scaling policy, counts as autoscaled. A target registered without a policy is reported as its own Medium finding, since bounds alone never trigger a scaling action. This avoids both falsely flagging endpoints scaled through Application Auto Scaling alone and falsely passing endpoints that cannot actually scale. +- Per-check failure isolation: an AccessDenied or API error is recorded as an error row and reported as "not evaluated — permission not granted" rather than a false "none found", and never aborts the review. +- Region discovery via `ce:GetCostAndUsage` with a fallback sweep of `sagemaker:List*` calls, so a payer-scoped Cost Explorer miss does not produce a false "no activity" result. +- Sample add-on IAM policy (`references/iam-policy.json`) for the permissions not covered by the AWS-managed `AIDevOpsAgentAccessPolicy`. diff --git a/skills/sagemaker-ops-review/README.md b/skills/sagemaker-ops-review/README.md new file mode 100644 index 0000000..b4d249d --- /dev/null +++ b/skills/sagemaker-ops-review/README.md @@ -0,0 +1,199 @@ +# Amazon SageMaker AI Operational Review — AWS DevOps Agent Skill + +Performs a strictly read-only operational review of Amazon SageMaker AI workloads — endpoints, training jobs, pipelines, notebooks, feature store, model registry, and Studio domains — across **8 pillars and 20 checks**, producing a severity-ranked **Amazon SageMaker AI Operational Review** report with one recommendation per High or Medium finding. + +## Purpose + +Teams running SageMaker AI at scale accumulate posture drift that no single console page surfaces: Studio domains left on `PublicInternetOnly`, endpoints without autoscaling, notebooks without a customer-managed key, endpoints idle for months still billing, quotas quietly approaching their limit. This skill gives AWS DevOps Agent the check definitions, severity model, and report format to assess all of it in one pass from native AWS control-plane and CloudWatch APIs, and to return findings ordered by how much they matter. + +It is designed for recurring review cadences — a weekly or monthly operational review meeting, a pre-launch readiness check, or an ad-hoc audit of a newly inherited account. + +## Key Capabilities + +- **20 checks across 8 pillars** — Security, Performance, Cost Optimization, Service Quotas, Resiliency, Operational Excellence, Sustainability, and Best Practices. `references/pillar-checks.md` is the authoritative definition of each check's APIs, logic, thresholds, and output fields. +- **Uniform severity ranking** — every finding is High, Medium, Low, or Informational (for inventory checks with no pass/fail signal), and the Executive Summary ranks them most-severe first. +- **Exactly one recommendation per High or Medium finding** — concrete and SageMaker-specific, never generic advice. +- **Multi-account and multi-region** — regions are discovered via Cost Explorer with a `sagemaker:List*` sweep as fallback, so a payer-scoped Cost Explorer miss never produces a false "no activity" result. +- **Dual-signal autoscaling detection** — an endpoint counts as autoscaled via managed instance scaling *or* a classic Application Auto Scaling target on `sagemaker:variant:DesiredInstanceCount`. Only endpoints with neither signal are flagged, which avoids falsely flagging endpoints scaled through Application Auto Scaling alone. +- **Per-check failure isolation** — a denied or failing API becomes an error row on that check and the review continues; an AccessDenied is reported as "not evaluated — permission not granted" rather than a false "none found". +- **Well-Architected grounding** — the Best Practices pillar's recommendations cite the public AWS Machine Learning, Generative AI, and Agentic AI lenses. + +## Prerequisites + +### 1. An AWS DevOps Agent Space with the target AWS account + +You need an existing [Agent Space](https://docs.aws.amazon.com/devopsagent/latest/userguide/getting-started-with-aws-devops-agent-creating-an-agent-space.html) with each account you want to review configured as a cloud source, and the `use_aws` tool available to the agent. + +### 2. IAM permissions + +Nearly every API this skill calls is already covered by the AWS-managed [`AIDevOpsAgentAccessPolicy`](https://docs.aws.amazon.com/devopsagent/latest/userguide/aws-devops-agent-security-devops-agent-iam-permissions.html) attached to the DevOps Agent role: + +- `sagemaker:List*`, `sagemaker:Describe*`, `sagemaker:ListTags` +- `cloudwatch:GetMetricData`, `cloudwatch:GetMetricStatistics`, `cloudwatch:ListMetrics` +- `application-autoscaling:DescribeScalableTargets`, `application-autoscaling:DescribeScalingPolicies` +- `servicequotas:GetServiceQuota` +- `ce:GetCostAndUsage`, `ce:GetDimensionValues` (region discovery) +- `health:DescribeEvents`, `health:DescribeAffectedEntities` (Lifecycle Events check) + +`sts:GetCallerIdentity`, used to default the review to the current account, needs no grant — the call cannot be restricted by IAM policy and succeeds for any authenticated principal. + +**One add-on permission** is not in the managed policy: `savingsplans:DescribeSavingsPlans`, used by the Savings Plan check. It is optional — without it that single check reports "not evaluated — permission not granted" and the other 19 run normally. To grant it, either deploy the repo's CloudFormation template with `EnableSageMakerOpsReview=true`: + +```bash +aws cloudformation deploy \ + --template-file cloudformation/devops-agent-skill-policies.yaml \ + --stack-name devops-agent-skill-policies \ + --parameter-overrides ExistingRoleName= EnableSageMakerOpsReview=true \ + --capabilities CAPABILITY_NAMED_IAM +``` + +…or attach the sample policy directly: + +```bash +aws iam put-role-policy \ + --role-name \ + --policy-name DevOpsAgentSkill-SageMakerOpsReview \ + --policy-document file://references/iam-policy.json +``` + +The skill operates strictly **read-only**: no `Create*`, `Update*`, or `Delete*` calls, no endpoint invocation, no job launches, and no data-plane calls of any kind — it never reads an inference payload. + +### 3. Support plan and account placement + +- The **Resiliency → SageMaker Lifecycle Events** check calls the AWS Health API, which requires a **Business or Enterprise Support** plan. Without one, the check reports that it was not evaluated. +- The **Cost Optimization → Savings Plan** check is only meaningful from the **management/payer account**; in a linked account it will legitimately return no plans. + +### 4. SageMaker AI workloads with activity (recommended) + +Latency and Stale Endpoint checks read `AWS/SageMaker` CloudWatch metrics, which only publish after an endpoint receives invocations, and Service Quotas utilization only scores when usage metrics exist. Reviewing an idle account produces "No data" rows rather than false findings. + +## Limitations + +- **Control-plane and metrics only.** The skill reports configuration and CloudWatch signals. It cannot assess model quality, training convergence, data drift, or anything requiring inference payloads or job artifacts. +- **`Check Encryption` covers notebook instances only.** Training jobs and endpoint configs are deliberately excluded to keep the review within the agent's context budget; use the Security pillar's VPC checks and your own KMS policy audit for those. +- **No cost figures.** Cost Explorer is used for region discovery, not spend attribution. The Savings Plan check reports coverage and expiry, not dollar savings. +- **Point-in-time.** Findings reflect state at run time. Service Quotas utilization in particular is scored over a fixed trailing 24-hour window, so a spike outside that window is not visible. +- **Best Practices pillar is advisory.** It emits Well-Architected-grounded guidance, not per-resource findings, and calls no APIs. +- **Large estates may need scoping.** Accounts with many endpoints across many regions can exhaust the agent's run budget; scope to a subset of regions or pillars if a run times out. + +## Agent Types + +**Chat tasks** and **Evaluation**. The skill is intended for on-demand and scheduled review runs, not live incident response — for SageMaker access failures during an incident, use the `aiml-access-diagnostics` skill instead. + +## Uploading to AWS DevOps Agent + +> Reference: [Uploading a skill](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent-devops-agent-skills.html#uploading-a-skill) + +### 1. Package the skill + +Build the archive from **inside** the skill directory so `SKILL.md` sits at the archive root — nesting it under a subdirectory causes `Failed to get skill resource` errors at load time: + +```bash +cd skills/sagemaker-ops-review +zip -qrD ../sagemaker-ops-review.zip . \ + -x 'README.md' 'CHANGELOG.md' '.skilleval.yaml' 'evals/*' +``` + +The resulting `sagemaker-ops-review.zip` contains: + +``` +SKILL.md # frontmatter + skill instructions (required, at root) +references/ +├── pillar-checks.md # the authoritative 20 check definitions +└── iam-policy.json # sample add-on policy (savingsplans:DescribeSavingsPlans) +``` + +`README.md`, `CHANGELOG.md`, `.skilleval.yaml`, and `evals/` are excluded — they are repo and offline-evaluation artifacts, not part of the runtime skill. + +Constraints enforced at upload time: + +- Total zip size ≤ **6 MB**. +- `SKILL.md` is required and must include `name` and `description` frontmatter. +- A `scripts/` directory is **not** allowed — uploads containing scripts are rejected. + +### 2. Upload via the Operator Web App + +1. Navigate to the **Skills** page in your Agent Space Operator Web App. +2. Choose **Add skill** → **Upload skill**. +3. Drag and drop `sagemaker-ops-review.zip` (or browse to it). +4. Select agent type: **Generic** / **All agents**. This is required if you intend to drive the skill from the [`aws-operation-review` custom agent](../../custom-agents/aws-operation-review/README.md) — narrowing the skill to **On-demand** / **Evaluation** keeps it out of the custom agent's skill picker, leaving that agent with no checks to run. Narrow the agent types only if you are driving the skill from Chat alone. +5. Review the validation results. +6. Choose **Upload**. + +## How to Use This Skill + +### Chat tasks + +Full review of the current account: + +> Run an Amazon SageMaker AI operational review for this account. + +Scoped to specific regions and accounts: + +> Run a SageMaker AI operational review for accounts 111122223333 and 444455556666 in us-east-1 and eu-west-1. + +Single pillar: + +> Review just the Security pillar of my SageMaker AI workloads — domains, notebooks, and endpoint VPC configuration. + +Targeted question that still routes through the check definitions: + +> Which of my SageMaker endpoints have no autoscaling and haven't been invoked in 90 days? + +Quota headroom before a launch: + +> Check my SageMaker service quota utilization in us-west-2 before we scale up training. + +### Evaluation + +Point an Evaluation agent at the skill and schedule it — weekly ahead of an operational review meeting, or monthly as a posture check. The report is produced in full each run, so successive runs are directly comparable. + +For a ready-made scheduled configuration, use the [`aws-operation-review` custom agent](../../custom-agents/aws-operation-review/README.md), which loads this skill for SageMaker AI reviews and supports schedule triggers. + +## Report Structure + +The skill produces a single Markdown report with a fixed structure: + +1. `# Amazon SageMaker AI Operational Review` header with Account IDs, Regions, and Date Range. +2. The **AI Disclaimer** blockquote, verbatim. +3. **Executive Summary** — counts by severity, then High and Medium findings most-severe first, each with its recommendation. +4. One `##` section per in-scope pillar, each with a `###` sub-section per check carrying **Guidance**, optional **AI Insights**, **Data** (a table including a `severity` column where the check defines one), and **Recommendations** (one per High/Medium finding, omitted when the check has none). + +If every in-scope check across every in-scope account and region returns no resources, the skill reports the single line "No SageMaker AI activity detected." instead of an empty report. + +## Skill Contents + +| File | Purpose | +|---|---| +| `SKILL.md` | Skill instructions — scope confirmation, check execution rules, severity model, and report format | +| `references/pillar-checks.md` | Authoritative definition of all 20 checks: APIs, logic, thresholds, severity mapping, output fields | +| `references/iam-policy.json` | Sample add-on IAM policy for the permission outside `AIDevOpsAgentAccessPolicy` | + +## Troubleshooting + +| Issue | Resolution | +|---|---| +| "No SageMaker AI activity detected" | Confirm the DevOps Agent role has `sagemaker:List*` / `sagemaker:Describe*` in the target account and region, and that the region is actually in scope | +| A check reports "not evaluated — permission not granted" | Grant the missing permission (see Prerequisites §2). The other checks are unaffected | +| Service Quotas Check shows "Unknown" risk | Usage metrics only publish while a resource is in use; a quota with no recent usage has no utilization to score | +| Latency or Stale Endpoints show "No data" | `AWS/SageMaker` endpoint metrics only publish after invocations — an endpoint with no traffic has no datapoints | +| Lifecycle Events check not evaluated | The AWS Health API requires a Business or Enterprise Support plan | +| Savings Plan check returns nothing in a linked account | Savings Plans are visible from the management/payer account only | +| The run times out | Reduce scope — fewer regions, or a subset of pillars and checks | + +## Customization + +- **Change a check** — edit `references/pillar-checks.md`. It is the single source of truth for APIs, logic, thresholds, and output fields; `SKILL.md` defers to it. +- **Change severity mappings** — also in `references/pillar-checks.md`, per check. Keep the four-level scale (High / Medium / Low / Informational) so the Executive Summary stays coherent. +- **Change scope defaults** — tell the agent which pillars, checks, accounts, and regions you want when you invoke it. + +## Related + +- [`aws-operation-review` custom agent](../../custom-agents/aws-operation-review/README.md) — loads this skill for SageMaker AI operational reviews, on demand or on a schedule. +- [`aiml-access-diagnostics`](../aiml-access-diagnostics/README.md) — diagnoses IAM and access failures for SageMaker and Bedrock calls; use during an incident rather than a posture review. +- [`service-quota-check`](../service-quota-check/README.md) — general-purpose, all-service quota checking. This skill's Service Quotas pillar is SageMaker-specific and scoped to seven verified SageMaker quota codes. +- AWS Well-Architected lenses grounding the Best Practices pillar: [Machine Learning](https://docs.aws.amazon.com/wellarchitected/latest/machine-learning-lens/machine-learning-lens.html) · [Generative AI](https://docs.aws.amazon.com/wellarchitected/latest/generative-ai-lens/generative-ai-lens.html) · [Agentic AI](https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentic-ai-lens.html) + +## Non-production disclaimer + +> ⚠️ This skill is sample code, not intended for production use without additional review and testing. Validate it in a non-production environment first. It performs read-only operational analysis and makes no changes to your AWS resources, but you are responsible for reviewing the IAM permissions you grant and for validating the findings and recommendations it produces before acting on them. diff --git a/skills/sagemaker-ops-review/SKILL.md b/skills/sagemaker-ops-review/SKILL.md new file mode 100644 index 0000000..05f39f4 --- /dev/null +++ b/skills/sagemaker-ops-review/SKILL.md @@ -0,0 +1,181 @@ +--- +name: sagemaker-ops-review +description: Amazon SageMaker AI Operational Review. Use this skill when a user asks to + review, audit, or assess Amazon SageMaker AI workloads (endpoints, training jobs, + pipelines, notebooks, feature store, model registry, domains) for best-practices + posture across Security, Performance, Cost Optimization, Service Quotas, Resiliency, + Operational Excellence, Sustainability, and Best Practices. Triggers on requests like + "SageMaker review", "SageMaker ops review", "SageMaker best practices audit", "ML ops + assessment", "review my SageMaker account", "SageMaker health check", or "ORR for + SageMaker". +metadata: + author: jacklunn + version: "1.0.0" + aws-devops-agent-skills.agent-types: "Chat tasks, Evaluation" + aws-devops-agent-skills.aws-services: "Amazon SageMaker AI, Amazon CloudWatch, AWS Service Quotas" + aws-devops-agent-skills.technical-domains: "AI/ML" +--- + +# Amazon SageMaker AI Operational Review + +Run the Amazon SageMaker AI operational review checks against a customer's Amazon SageMaker AI +resources and produce an **Amazon SageMaker AI Operational Review** report. It evaluates +**8 pillars, 20 checks** using native AWS APIs (via `use_aws`). The Best Practices pillar's +recommendations are grounded in the public AWS Well-Architected lenses — see `pillar-checks.md`. + +This is a strict **READ-ONLY** review: data is collected through native AWS `List*` / +`Describe*` control-plane APIs, CloudWatch metric reads, `servicequotas:GetServiceQuota`, +`health:DescribeEvents`, and `savingsplans:DescribeSavingsPlans`. It performs no model +invocations, launches no jobs, and reads no inference payloads. + +## When to Use + +Activate this skill when the user asks to review, audit, or assess an Amazon SageMaker +workload, check SageMaker best-practices posture, or run a SageMaker operational (readiness) +review — for one pillar, a subset of checks, or the full set. + +## Pillars and Checks + +Run checks grouped by pillar in the order below. **Load `references/pillar-checks.md`** for +each check's APIs, logic, thresholds, and output fields. + +| Pillar | Checks | +|--------|--------| +| **Security** | Check Encryption · SageMaker VPC Check · VPC Configuration Check | +| **Performance** | SageMaker Endpoint Inference Type · SageMaker Endpoint Latency | +| **Cost Optimization** | SageMaker Resource Tagging Check · Trainium and Inferentia Usage · Autoscaling Endpoint Check · Sagemaker Savings Plan · Sagemaker Lifecycle Configurations · Sagemaker Inference Recommender Jobs Check · Sagemaker Stale Endpoints Check | +| **Service Quotas** | Service Quotas Check | +| **Resiliency** | SageMaker Endpoint Instances · SageMaker Lifecycle Events | +| **Operational Excellence** | Sagemaker Project Check · Sagemaker Pipeline Check · SageMaker Endpoint Datacapture Enabled Check | +| **Sustainability** | Domain Region Check | +| **Best Practices** | Well-Architected Recommendations (SageMaker AI) | + +## Step 1: Identify Scope + +Confirm with the user: +- **Account IDs** and **regions** to review (default: current account via `sts:GetCallerIdentity`). If regions are unspecified, discover active regions with `ce:GetCostAndUsage` (SERVICE = "Amazon SageMaker", grouped by REGION); Cost Explorer is payer-scoped, so if it returns nothing, **fall back** to sweeping a default region set with `sagemaker.list-endpoints`/`list-domains`/`list-notebook-instances`. Conclude "no activity" only after both come back empty. +- **Pillars or individual checks** to run (default: all 8 pillars / 20 checks). +- **Date range** for time-windowed checks (Latency = last 7 days, Stale Endpoints = last 90 days, Service Quotas usage = last 60 minutes — these windows are fixed by the checks). + +## Step 2: Run the Checks + +For each in-scope check, call the APIs listed in `references/pillar-checks.md` via `use_aws` +and build the check's result rows. Follow this behavior: + +- **Read-only.** `List*` then `Describe*`; paginate every call that returns a token. +- **Per-check isolation.** Catch and record errors per check as a `{ error }` row — a failed + check never aborts the review. +- **Permissions / graceful degradation.** Nearly all APIs are covered by the AWS-managed + `AIDevOpsAgentAccessPolicy` on the DevOps Agent role. The one exception — + `savingsplans:DescribeSavingsPlans` (Savings Plan check) — is an optional add-on. The AWS + Health APIs used by the Lifecycle Events check are covered by the managed policy but + additionally require a Business/Enterprise Support plan. On AccessDenied for a check, report it + as **"not evaluated — permission not granted"** and continue; never emit a false "none found" + from an access error. +- **Empty results** (permission present, nothing there) produce a single "No found" + row, not a dropped section. +- **Severity-ranked findings.** Assign each finding a severity per `references/pillar-checks.md`: + **High**, **Medium**, **Low**, or **Informational** (inventory checks with no pass/fail signal). + Checks with a compliance signal set severity as defined there — e.g. Studio domain not `VpcOnly` + → High; no autoscaling / stale endpoint / unencrypted notebook / no VPC config / lapsed Savings + Plan → Medium; missing tags / data capture disabled → Low. The Service Quotas Check derives its + tier from utilization (≥ 90% High, ≥ 75% Medium, else Low; Unknown if no usage data). +- **One finding = one non-compliant resource in one check**, keyed by `(check, region, resource)`. + Do **not** aggregate resources into a single finding — three unencrypted notebooks are three + Medium findings, not one. Aggregation breaks the severity counts and makes runs incomparable. +- **One recommendation per High or Medium finding.** Emit exactly one concrete, SageMaker-specific + recommendation for every High and Medium finding. Low and Informational findings do not require one. +- Use only the severities each check defines; do **not** invent thresholds a check does not define. + +## Step 3: Generate the Report + +Produce a single Markdown report titled **"Amazon SageMaker AI Operational Review"**, with the +structure below. + +```markdown +# Amazon SageMaker AI Operational Review + +**Account IDs:** +**Regions:** +**Date Range:** + +> **AI Disclaimer:** The AI-generated insights in this report are provided for informational purposes only. They should be reviewed and validated by qualified personnel before taking any action. AWS is not responsible for any decisions made based on AI-generated content. + +## Executive Summary + + + +## + +### + +**Guidance** + + + +**AI Insights** + + + +**Data** + + + +**Recommendations** + + +``` + +Rules: +- Emit the **AI Disclaimer blockquote verbatim**, immediately after the header. +- **Date Range** is a single short value — the review timestamp, or a date range when the user + scoped one (e.g. `2026-09-18 (point-in-time)`). Do **not** inline every check's window into it; + per-check windows are fixed by the checks and belong in each check's own section. +- One `##` section per **in-scope pillar**, in the table order above; one `###` sub-section per + check in that pillar. Include every in-scope check even when it found nothing (render its + empty-state row). +- The **Executive Summary** ranks findings by severity (High → Medium → Low). Include it whenever + any finding carries a severity; it is what makes the report prioritized and actionable. +- **The Executive Summary must contain every High and Medium finding from every pillar**, and its + severity counts must reconcile exactly with the per-check sections: if the pillar sections contain + 12 Medium findings, the summary says 12 and lists 12 rows. Before emitting the report, count the + High/Medium findings per pillar and check the totals match. Findings from pillars other than + Security and Cost Optimization are the ones most often dropped — Resiliency Health events in + particular. A finding that is scored Medium in its check but missing from the summary is invisible + to the reader, which defeats the point of ranking at all. +- The **AI Insights** block per check is optional; when included, carry the AI-generated / + verify-before-use caveat. +- Render each check's **Data** as a table of the fields defined in `references/pillar-checks.md`, + including the `severity` field for checks that define one. +- Emit a **Recommendations** block for every check that has at least one High or Medium finding — + exactly one recommendation per such finding. Skip the block for checks with only Low or + Informational findings. + +## Constraints + +- READ-ONLY — no resource mutation, no endpoint invocation, no job launches, no payload reads. +- Report only what the APIs return. Do NOT fabricate data or assume unobserved configuration. +- **No invented numbers.** State a quota, limit, instance price, monthly cost, or percentage saving + only if an API call returned it. Never substitute a default limit for an applied one, never + estimate spend from remembered pricing, and never attach "~" or "up to" to a figure you did not + read. If a number would help but was not retrieved, point the reader at the console page or API + that has it. See the "Never state a number the APIs did not return" rule in + `references/pillar-checks.md`. +- Paginate ALL calls that return a pagination token. +- Empty-scope precedence: if **every** in-scope check across **all** in-scope accounts/regions + returns no resources, skip the per-pillar report and instead report the single line + "No SageMaker AI activity detected." Otherwise render the full report — each check that found + nothing gets its own empty-state row (Step 2), never the terse message. +- Keep all guidance and recommendations specific to Amazon SageMaker AI. + +## Data Source Boundaries + +Native AWS APIs only: `sagemaker`, `cloudwatch` (`get-metric-statistics`, `get-metric-data`, +`list-metrics`), `application-autoscaling` (`describe-scalable-targets`, +`describe-scaling-policies`), `servicequotas` (`get-service-quota`), `ce` (`get-cost-and-usage` +for region discovery), `health` (`describe-events`, `describe-affected-entities`), plus the one +optional add-on `savingsplans` (`describe-savings-plans`). All but that add-on are covered by +the AWS-managed `AIDevOpsAgentAccessPolicy`. No data-plane calls and no non-AWS tooling — the +skill is self-contained on the DevOps Agent's cloud-source IAM role. diff --git a/skills/sagemaker-ops-review/evals/eval_queries.json b/skills/sagemaker-ops-review/evals/eval_queries.json new file mode 100644 index 0000000..c4dbfac --- /dev/null +++ b/skills/sagemaker-ops-review/evals/eval_queries.json @@ -0,0 +1,12 @@ +[ + {"query": "Which skill would help me run an Amazon SageMaker AI operational review? Just name it; do not run it.", "should_trigger": true}, + {"query": "Is there a skill for auditing my SageMaker endpoints, training jobs, and Studio domains against best practices? Answer yes or no with the skill name; do not execute it.", "should_trigger": true}, + {"query": "Name the skill that covers SageMaker security, cost optimization, service quota, and resiliency reviews. Do not run any audit.", "should_trigger": true}, + {"query": "I need an ORR for SageMaker. Which skill covers that? Name it only.", "should_trigger": true}, + {"query": "Which skill runs a SageMaker health check across my account? Just name it.", "should_trigger": true}, + {"query": "Why is my Bedrock InvokeModel call returning AccessDeniedException? Name the skill that diagnoses this; do not run it.", "should_trigger": false}, + {"query": "Write a Python script that sorts a list of numbers", "should_trigger": false}, + {"query": "What's the weather forecast for Sydney this weekend?", "should_trigger": false}, + {"query": "Create a CloudFormation template for an S3 bucket", "should_trigger": false}, + {"query": "Review my EKS cluster for upgrade readiness. Name the skill only.", "should_trigger": false} +] diff --git a/skills/sagemaker-ops-review/references/iam-policy.json b/skills/sagemaker-ops-review/references/iam-policy.json new file mode 100644 index 0000000..5feb459 --- /dev/null +++ b/skills/sagemaker-ops-review/references/iam-policy.json @@ -0,0 +1,13 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "SageMakerOpsReviewOptionalSavingsPlans", + "Effect": "Allow", + "Action": [ + "savingsplans:DescribeSavingsPlans" + ], + "Resource": "*" + } + ] +} diff --git a/skills/sagemaker-ops-review/references/pillar-checks.md b/skills/sagemaker-ops-review/references/pillar-checks.md new file mode 100644 index 0000000..4ec3d2d --- /dev/null +++ b/skills/sagemaker-ops-review/references/pillar-checks.md @@ -0,0 +1,359 @@ +# Amazon SageMaker AI — Check Definitions + +**8 pillars, 20 checks.** The Best Practices pillar's recommendations are grounded in public +AWS Well-Architected lenses (see the Best Practices section); all other checks are read-only +`List*`/`Describe*` inventories. + +**Serverless endpoints: never flag a feature serverless does not support.** Per the AWS +[Serverless Inference feature exclusions](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html#serverless-endpoints-how-it-works-exclusions), +serverless endpoints do **not** support: GPUs, AWS Marketplace model packages, private Docker +registries, Multi-Model Endpoints, **VPC configuration**, **network isolation**, **data capture**, +multiple production variants, Model Monitor, and inference pipelines. A variant with +`ServerlessConfig` must be scored **Informational** — with a note that the feature is not +supported for serverless — in any check evaluating one of these, never Low/Medium/High. Flagging +them produces a recommendation the user cannot act on: observed a run instructing the operator to +"re-create the model with a VpcConfig block" for a serverless endpoint, and flagging the same +endpoint for disabled data capture. An unactionable finding is worse than no finding. Autoscaling +is the one nuance: on-demand serverless scales automatically, and serverless with Provisioned +Concurrency supports Application Auto Scaling on `sagemaker:variant:DesiredProvisionedConcurrency` +— but never on `DesiredInstanceCount`. + +**Never state a number the APIs did not return.** Report quotas, limits, instance prices, monthly +costs, and percentage savings **only** when a call in this file returned that value. If a figure was +not read from an API response, omit it — do not estimate, do not recall it from training data, and do +not carry over a "typical" or "default" value. This applies to Guidance, AI Insights, and +Recommendations equally, and it applies even when the number is qualified with "~" or "up to". +Observed fabrications to avoid: a Service Quotas run that substituted AWS default limits for applied +ones and raised a false High; a Project Check claiming "the domain has a limit of 2 projects by +default" (no such quota exists in the `sagemaker` service); per-hour instance prices and derived +monthly idle-cost totals that no pricing API was called to obtain; and blanket "up to 64% savings" / +"10× lower cost" claims with no source. Where a number would help but is unavailable, name the +console page or API the reader can check instead — an unsourced figure that looks authoritative is +worse than no figure, because it gets acted on. + +**What counts as one finding.** A finding is **one non-compliant resource within one check**, +keyed by `(check, region, resource)` — not one row per check. Three unencrypted notebooks are +**three** Medium findings with three recommendations, not one finding reading "3 notebooks are +unencrypted". Never aggregate resources into a single finding or a single recommendation: the +severity counts, the Executive Summary ranking, and the one-recommendation-per-High/Medium rule all +depend on per-resource granularity, and aggregation makes run-to-run counts incomparable. Observed +drifting between runs on an identical account (12 Medium findings vs 5) before this was specified. + +**Severity model.** Findings that carry a best-practice signal are ranked on a uniform scale. +Checks with no pass/fail signal stay **Informational** (they inventory state). Each check below +states which severity a non-compliant finding earns; the Service Quotas Check derives its tier +from utilization. **Every High or Medium finding carries exactly one recommendation** in the +report; Low and Informational findings do not require one. + +| Severity | Meaning | Examples | +|---|---|---| +| **High** | Material risk to security, availability, or spend — act promptly | Studio domain not `VpcOnly`; quota utilization ≥ 90% | +| **Medium** | Best-practice gap that should be remediated | No autoscaling; stale endpoint; unencrypted notebook; no VPC config; no Savings Plan on steady spend; quota 75–90% | +| **Low** | Minor hygiene gap | Missing tags; data capture disabled | +| **Informational** | Inventory / state, no pass/fail | Inference type, latency, lifecycle configs, projects, pipelines, endpoint instances, domain regions, accelerator adoption, recommender jobs, health events | + +**IAM note.** All APIs except one are covered by the AWS-managed **`AIDevOpsAgentAccessPolicy`** +already attached to the DevOps Agent role: `sagemaker` List/Describe/ListTags, `cloudwatch` +GetMetricData/GetMetricStatistics/ListMetrics, `servicequotas:Get*`, +`application-autoscaling:Describe*`, `ce:GetCostAndUsage`/`GetDimensionValues`, and +`health:DescribeEvents`/`DescribeAffectedEntities`. The one exception — +`savingsplans:DescribeSavingsPlans` (Savings Plan check) — is an **optional add-on** not in the +managed policy. When a check's permission is absent, report it as **"not evaluated — permission +not granted"** and continue; never emit a false "none found" on an AccessDenied. + +Each check emits rows keyed by `Region`, `AccountId`, `Check`, plus the fields listed below. +Empty results produce a single "no resources found" row rather than being dropped. + +--- + +## Security + +### Check Encryption +- **APIs**: `sagemaker.list-notebook-instances` → `describe-notebook-instance` +- **Scope**: notebook instances only (training jobs / endpoint configs deliberately excluded to avoid OOM) +- **Logic**: `encrypted = Boolean(KmsKeyId)` +- **Severity**: notebook without a KMS key → **Medium**; encrypted → Informational (OK). Recommendation on Medium: attach a customer-managed KMS key. +- **Fields**: `type` (NotebookInstance), `name`, `encrypted` (bool), `severity`, `kmsKeyId` (or "Not encrypted") + +### SageMaker VPC Check +- **APIs**: `sagemaker.list-domains` → `describe-domain` +- **Pagination**: **list all domains** (no resource cap) — paginate on `NextToken` until exhausted. +- **Logic**: reports network posture; evaluates isolation +- **Severity**: `appNetworkAccessType != VpcOnly` (i.e. `PublicInternetOnly`) → **High**; `VpcOnly` → Informational (OK). Recommendation on High: switch the domain to `VpcOnly` and route through VPC endpoints. +- **Fields**: `domainId`, `domainName`, `domainArn`, `status`, `appNetworkAccessType` (VpcOnly vs PublicInternetOnly), `severity`, `vpcId`, `subnetIds`, `securityGroupIds`; `summary.totalDomains` + +### VPC Configuration Check +- **APIs**: `sagemaker.list-endpoints` → `describe-endpoint` → `describe-endpoint-config` → `describe-model` +- **Logic** (per production + shadow variant): compliant if the variant has `VpcConfig` OR its model has `VpcConfig`; non-compliant if neither. VpcConfig is a property of the model / endpoint config, so evaluate it **regardless of Inference Component usage**: when a variant has no `ModelName` (Inference Component endpoint), resolve the associated model(s) via `sagemaker.list-inference-components` → `describe-inference-component` → `describe-model` and evaluate their `VpcConfig` rather than emitting a "not supported" Warning. Compliant endpoints emit one "All variants have VPC configuration" row. +- **Serverless variants are out of scope.** Serverless Inference does not support VPC configuration + or network isolation at all, so a serverless variant can never be compliant and can never be + remediated. Score it **Informational** with the note "VPC configuration not supported for + Serverless Inference" and emit no recommendation. Only instance-backed variants are eligible for a + finding. +- **Severity**: an **instance-backed** variant with neither variant nor model `VpcConfig` → **Medium**; compliant, or serverless → Informational (OK). Recommendation on Medium: attach `VpcConfig` (subnets + security groups) to the model / endpoint config. +- **Fields**: `isCompliant` (true / false), `severity`, `endpointName`, `variantName`, `instanceType`, `status`, `endpointConfigName`, `modelName`, `variantType` (production/shadow), `hasVariantVpcConfig`, `hasModelVpcConfig`, `isInferenceComponent` + +--- + +## Performance + +### SageMaker Endpoint Inference Type +- **APIs**: `sagemaker.list-endpoints` → `describe-endpoint` → `describe-endpoint-config` +- **Logic**: `serverless` if any variant has `ServerlessConfig`; else `Asynchronous` if `AsyncInferenceConfig`; else `Real-Time` +- **Fields**: `name`, `status`, `inferenceType` + +### SageMaker Endpoint Latency +- **APIs**: `sagemaker.list-endpoints` → `describe-endpoint` → `describe-endpoint-config`; `cloudwatch.get-metric-statistics` +- **Metrics**: `ModelLatency`, `OverheadLatency` (namespace `AWS/SageMaker`, dims `EndpointName`+`VariantName`, stat `Average`, period 86400, **last 7 days**, one datapoint/day) +- **Logic**: report values (2 dp) or "No data"; no pass/fail +- **Fields**: per endpoint `{endpointName, endpointArn, endpointStatus, region, variants:[{variantName, instanceType, dailyMetrics:[{date, ModelLatency, OverheadLatency}]}]}`; `summary.daysAnalyzed=7` + +--- + +## Cost Optimization + +### SageMaker Resource Tagging Check +- **APIs**: `sagemaker.list-models`, `list-endpoints`, `list-training-jobs`, `list-processing-jobs`, `list-transform-jobs`; `sagemaker.list-tags` per resource ARN +- **Exclude SageMaker-generated Model Monitor processing jobs** — those whose name begins + `model-monitoring-`. They are created automatically by a monitoring schedule, cannot be tagged by + the operator after the fact, and accumulate without limit: one account held 240+ of them, which + swamped the check and forced the report to collapse them into a single aggregate row, breaking + per-resource granularity. Tag the *monitoring schedule* instead. Note the count of excluded jobs in + the check's summary so the omission is visible. +- **Logic**: `isCompliant = hasUserDefinedTag` — at least one tag whose key does **not** begin with + a reserved AWS prefix (`sagemaker:`, `aws:`). SageMaker auto-injects `sagemaker:domain-arn`, + `sagemaker:user-profile-arn`, and `sagemaker:space-arn` on every Studio-created resource, so + counting any tag at all marks nearly the whole estate compliant and defeats the check. Cost + Explorer group-by-tag and ownership attribution both require business tags, which is what this + check is for. Report system tags in `existingTags` for context, but do not let them satisfy + compliance. +- **Severity**: resource with no user-defined tag → **Low**; has one → Informational (OK). Recommendation is optional at Low (add cost-allocation / ownership tags). +- **Fields**: `resourceType` (Model / Endpoint / TrainingJob / ProcessingJob / BatchTransformJob), `resourceName`, `resourceArn`, `isCompliant` (bool), `severity`, `existingTags` (array), `userDefinedTags` (array — the subset that determined compliance) + +### Trainium and Inferentia Usage +- **APIs**: `sagemaker.list-notebook-instances`; `list-training-jobs` → `describe-training-job`; `list-endpoint-configs` → `describe-endpoint-config`; `list-apps` +- **Logic**: instance-type prefix match — `ml.inf*` → Inferentia, `ml.trn*` → Trainium. Only matching resources reported. +- **Fields**: `resourceType` (NotebookInstance / TrainingJob / EndpointConfig / App), `resourceName`, `instanceType`, `acceleratorType` + +### Autoscaling Endpoint Check +- **APIs**: `sagemaker.list-endpoints` → `describe-endpoint`; `application-autoscaling.describe-scalable-targets` + `describe-scaling-policies` (ServiceNamespace=`sagemaker`) — both in `AIDevOpsAgentAccessPolicy`. +- **Logic (dual-signal, three states — avoids false positives *and* false negatives):** classic + Application Auto Scaling does not appear on `DescribeEndpoint`, so managed-scaling alone would + falsely flag it. Evaluate both signals, then resolve to one of three states: + + | Signals present | Mechanism | Verdict | + |---|---|---| + | `variant.ManagedInstanceScaling.Status === 'ENABLED'` | `Managed` | autoscaled — Informational | + | `describe-scalable-targets` returns a target on `sagemaker:variant:DesiredInstanceCount` **and** `describe-scaling-policies` returns ≥ 1 policy for that resource | `Application Auto Scaling` | autoscaled — Informational | + | target present but **no** scaling policy | `Application Auto Scaling (target only — no policy)` | **not effectively autoscaled** — see Severity | + | neither signal | `None` | not autoscaled — see Severity | + + A registered scalable target only declares min/max capacity bounds; without a target-tracking, + step, or scheduled policy nothing ever triggers a scaling action, so the endpoint cannot scale + despite appearing configured. Both `describe-scalable-targets` and `describe-scaling-policies` are + covered by `AIDevOpsAgentAccessPolicy`, so the second call is free. If `application-autoscaling` is + denied, fall back to managed-scaling-only and mark the finding lower-confidence. Endpoint status + badge: InService=green, Failed=red, else blue. +- **Serverless variants are out of scope.** A variant with `ServerlessConfig` scales to and from + zero by design and cannot carry an Application Auto Scaling target on + `sagemaker:variant:DesiredInstanceCount`. Report it as `Mechanism = Serverless`, `Autoscaling + Enabled = Yes`, severity **Informational** — never Medium. Only instance-backed variants are + eligible for a finding. +- **Severity**: for an `InService` **instance-backed** variant — + - autoscaled by **neither** signal → **Medium**. Recommendation: register an Application Auto + Scaling target on `sagemaker:variant:DesiredInstanceCount` **and attach a scaling policy**, or + enable managed instance scaling. + - target registered but **no scaling policy** → **Medium**, worded distinctly: "scalable target + registered but no scaling policy attached — the endpoint will not scale". Recommendation: attach + a target-tracking policy (e.g. on `SageMakerVariantInvocationsPerInstance`) to the existing + target. Do not report this variant as autoscaled. + - effectively autoscaled (managed scaling, or target + policy), or serverless → Informational (OK). +- **Fields**: `Autoscaling Enabled` (Yes / No / Target only), `severity`, `Mechanism` (Managed / Application Auto Scaling / Application Auto Scaling (target only — no policy) / Serverless / None), `Policy Count`, `Enabled Variants`, `Total Variants`, `Details` (name, ARN, status, timestamps, config name, failure reason) + +### Sagemaker Savings Plan +- **APIs**: `savingsplans.describe-savings-plans` (filter savings-plan-type=`SageMaker`, maxResults 100) +- **IAM (optional add-on):** `savingsplans:DescribeSavingsPlans` is **not** in `AIDevOpsAgentAccessPolicy`. If the permission is absent, report this check as **"not evaluated — permission not granted"** and continue — never a false "no Savings Plans found" on an AccessDenied. +- **Scope:** Savings Plans data is meaningful only from the management/payer account; in a linked account it may be empty. +- **Logic**: `remainingDays = round((end − now)/day)`; `status = remainingDays > 0 ? 'Active' : 'Expired'` +- **Severity**: a plan expiring soon (`remainingDays` low) **or** no SageMaker Savings Plan on steady inference spend → **Medium**; healthy active coverage → Informational (OK). Recommendation on Medium: renew/purchase a SageMaker Savings Plan sized to steady spend. +- **Fields**: plan fields + `remainingDays`, `status`, `severity`, `utilizationEstimate`, `region`; `summary`: totalSavingsPlans, sagemakerSavingsPlans, activePlans, expiredPlans, totalCommitment + +### Sagemaker Lifecycle Configurations +- **APIs**: `sagemaker.list-notebook-instance-lifecycle-configs` + `sagemaker.list-studio-lifecycle-configs` (concatenated; list only) +- **Logic**: inventory; no pass/fail +- **Fields**: `ConfigName`, `ConfigType`, `ConfigArn`, `CreationTime`, `LastModifiedTime`, `Details`, `RawData.codeString` +- **ConfigType** is `Notebook Instance` for notebook LCCs, or the Studio LCC's `StudioLifecycleConfigAppType` for Studio LCCs — which includes **JupyterServer, KernelGateway, CodeEditor, JupyterLab** (and any future app types). Surface the actual app type per config, not just "Studio". + +### Sagemaker Inference Recommender Jobs Check +- **APIs**: `sagemaker.list-inference-recommendations-jobs` → `describe-inference-recommendations-job` +- **Logic**: inventory of recommender jobs; no pass/fail +- **Fields**: described job fields. Empty → "No Recommendation jobs found" + +### Sagemaker Stale Endpoints Check +- **APIs**: `sagemaker.list-endpoints` → `describe-endpoint`; `cloudwatch.get-metric-data` +- **Metrics**: `Invocations` (namespace `AWS/SageMaker`, dims `EndpointName`+`VariantName`, stat `Sum`, period 86400, **last 90 days**) +- **Logic**: find first non-zero invocation datapoint → `Last Invoked = " days"`; default "Not invoked" if no data +- **Severity**: **instance-backed** endpoint not invoked in ≥ 90 days (or never) → **Medium** + (idle instance-hours are billed continuously); **serverless** endpoint not invoked → **Low** + (hygiene only — serverless scales to zero, so there is no idle compute cost to recover). + Recently invoked → Informational (OK). Recommendation on Medium: delete or right-size the idle + endpoint. Do not recommend a costly remediation on a stale endpoint — prefer deletion over + reconfiguring something with no traffic. +- **Fields**: endpoint describe fields + `Last Invoked`, `inferenceType`, `severity` + +--- + +## Service Quotas + +### Service Quotas Check +- **APIs**: `servicequotas.get-service-quota` (serviceCode `sagemaker`, per quota code); `cloudwatch.get-metric-data` (usage metrics, period 3600, stat Maximum, **trailing 24 hours**) +- **Do NOT apply `FILL(usage,0)`** or any other gap-filling expression. Filling absent datapoints + with zero converts "no usage data" into a confident 0% utilization and scores the quota **Low** + when the correct answer is **Unknown**. Distinguish the two: datapoints returned → compute + utilization; no datapoints → `Max Usage` = "No data", `Risk Level` = **Unknown**. +- **If the Service Quotas API appears unavailable, retry once before degrading.** Availability of + `servicequotas` through `use_aws` has been observed to be **intermittent** — the same account + returned all seven applied limits in one run and "service unavailable" in the next. Retry the + `get-service-quota` calls once, and if the check runs in a subagent, have the parent retry before + accepting the degraded result. Only after a retry fails should the check degrade. +- **If the Service Quotas API is genuinely unavailable** (the `use_aws` tool does not expose + `servicequotas` in the runtime, or the call returns AccessDenied), report the whole check as + **"not evaluated — Service Quotas API unavailable"** with severity Unknown, and continue. Do not emit quota rows + with invented limits, and do not report CloudWatch usage without a limit to score it against — + usage without a denominator is not a utilization finding. +- **Window**: the usage window is fixed at **trailing 24 hours** (`startTime = now − 24 h`, `period 3600`, stat `Maximum`). Keep it fixed for consistent utilization scoring. + **Do not shorten this window.** SageMaker publishes `AWS/Usage` `ResourceCount` roughly **every + 20 minutes**, not per minute, and with ingestion lag — a trailing-60-minute window at `period 60` + returns zero datapoints even when resources are plainly running, which scores every quota as + `Unknown` and silently disables the whole check. Verified 2026-09-18: over 60 min / `period 60` + the endpoint-instance metric returned 0 datapoints, while the same metric over 24 h / + `period 3600` returned the correct maximum of 4 against 4 running instances. +- **Quota codes and usage metrics**: all seven codes below were verified against the live + `sagemaker` service in us-east-1. Usage comes from CloudWatch namespace **`AWS/Usage`**, metric + **`ResourceCount`**, with dimensions `Service=SageMaker`, `Class=None`, `Type=Resource`, and + `Resource` set per row. Do **not** use `AWS/SageMaker` for quota usage — no quota usage metrics + exist in that namespace. + + | Quota code | Quota name | `Resource` dimension | + |---|---|---| + | L-00C91CB5 | Number of instances across all training jobs | `training-job/total_instance_count` | + | L-F311B08F | Number of instances across all processing jobs | `processing-job/total_instance_count` | + | L-60D2A6F0 | Number of instances across all transform jobs | `transform-job/total_instance_count` | + | L-7A3DF611 | Number of instances across active endpoints | `endpoint/total_instance_count` | + | L-04CE2E67 | Total number of notebook instances | `notebook-instance/total_count` | + | L-B683BCB0 | Total domains | `studio/total_domains` | + | L-AC46C40F | Maximum number of Studio user profiles allowed per account | `studio/max_user_profiles_per_domain` | + + If `get-service-quota` returns `NoSuchResourceException` for a code in a given region, skip that + row and continue — quota availability varies by region. +- **Use `get-service-quota` only. Never `get-aws-default-service-quota`.** The former returns this + account's **applied** limit; the latter returns the AWS default, which is dramatically lower once + any increase has been approved. Substituting defaults inverts the utilization maths and + manufactures false High findings. Observed 2026-09-18: a run that fell back to the default API + reported the endpoint-instance limit as **4** and raised a High "quota at 100%, new deployments + will fail" finding, when the applied limit was **200** and true utilization was **2% (Low)**. + Other defaults it reported were equally wrong — training 4 vs 30 applied, notebooks 8 vs 30, + domains 2 vs 500, user profiles 2 vs 6000. +- **Never score a quota from an unverified limit.** If `get-service-quota` does not return an applied + value for a code, emit `Current Value` = "Not retrieved" and `Risk Level` = **Unknown**. Do not + substitute a default, do not guess, and do not raise a High or Medium finding on a limit the check + did not actually read. A quota finding is only as trustworthy as its denominator. +- **Risk thresholds**: `utilization% = maxUsage / currentValue × 100`; **≥ 90 → High** (red), **≥ 75 → Medium** (warning), else **Low** (success); **Unknown** if no usage data +- **Fields**: `Quota Name`, `Account ID`, `Region`, `Current Value`, `Max Usage`, `Current Usage`, `Max Utilization %`, `Risk Level`, `Usage` (time series vs quota limit) + +--- + +## Resiliency + +### SageMaker Endpoint Instances +- **APIs**: `sagemaker.list-endpoints` → `describe-endpoint` +- **Logic**: one row per production variant; no pass/fail +- **Fields**: `Endpoint Name`, `Variant Name`, `Current Instance Count` (or '-'), `Desired Instance Count` (or '-'), `Max Concurrency` (serverless, or '-') + +### SageMaker Lifecycle Events +- **APIs**: `health.describe-events` (services=`SAGEMAKER`, maxResults 100) → `health.describe-affected-entities` per event +- **IAM:** `health:DescribeEvents` and `health:DescribeAffectedEntities` are covered by `AIDevOpsAgentAccessPolicy`, but the Health API requires a Business/Enterprise Support plan. If the permission or support tier is absent, report this check as **"not evaluated — permission not granted"** and continue. +- **Event status scope**: default to **`open` and `upcoming` only**. Do not pull `closed` events — + they are historical noise that crowds out actionable rows (a single account accumulated 20+ + closed maintenance events). Include `closed` only when the user explicitly asks for event history. +- **Logic**: inventory of AWS Health events, with a severity derived from actionability. +- **Severity**: an event with `eventScopeCode`/entity status indicating **ACTION_REQUIRED** and a + status of `open` or `upcoming` → **Medium**; all other events → Informational. Recommendation on + Medium: state the required action and the event's `startTime` as the deadline. + Rationale: these events carry hard externally-imposed deadlines (scheduled notebook maintenance, + platform end-of-support). Leaving them Informational keeps them out of the severity-ranked + Executive Summary, so the most time-critical items in the whole report go unranked — observed in + a live run where maintenance windows 36 and 52 hours out were invisible to the summary. +- **Fields**: `eventArn`, `eventTypeCode`, `eventDescription`, `startTime`, `endTime`, `statusCode`, `actionability`, `severity`, `affectedResources`, `EventDetails`, `ImpactedResources`, `Actions` (console links). Empty → Status "OK" row +- **Row granularity**: one row and one finding **per affected entity**, not per event. An event + returning three affected notebook instances is **three** Medium findings with three + recommendations, because each instance needs stopping individually. Observed a run emitting one + finding covering "test-trn1, test-with-encryption, test", which under-counted Medium by two. Do not + collapse multiple events into an aggregate "(N additional events)" row either. +- **Region scope**: filter events to the **in-scope regions only**. AWS Health returns events across + all regions regardless of the review scope, so a us-east-1-scoped review will otherwise surface + us-west-2 resources — observed a run reporting two us-west-2 notebook findings under a header + reading `Regions: us-east-1`. Either drop out-of-scope events or add their region to the review + scope and header; never report findings for a region the report claims not to cover. Global + (non-regional) SageMaker events may be included, labelled `global`. + +--- + +## Operational Excellence + +### Sagemaker Project Check +- **APIs**: `sagemaker.list-projects` (key `ProjectSummaryList`) → `describe-project` +- **Logic**: inventory; no pass/fail. Empty → "No project found" +- **No project quota exists.** Do not claim projects consume a per-domain or per-account limit — + there is no SageMaker Projects quota, and a `CreateFailed` project occupies no capacity. Report + status and `FailureReason` as returned and stop there. +- **Fields**: described project fields + +### Sagemaker Pipeline Check +- **APIs**: `sagemaker.list-pipelines` (key `PipelineSummaries`) → `describe-pipeline` +- **Logic**: inventory; no pass/fail. Empty → "No Pipeline found" +- **Fields**: described pipeline fields + +### SageMaker Endpoint Datacapture Enabled Check +- **APIs**: `sagemaker.list-endpoints` → `describe-endpoint` +- **Logic**: `Data Capture Enabled = Boolean(DataCaptureConfig.EnableCapture)` (false if config null) +- **Serverless variants are out of scope.** Serverless Inference does not support data capture, so a + serverless endpoint cannot enable it. Score it **Informational** with the note "data capture not + supported for Serverless Inference" — never Low. +- **Severity**: **instance-backed** endpoint with data capture disabled → **Low**; enabled, or serverless → Informational (OK). Recommendation is optional at Low (enable data capture to support model-quality monitoring / evaluation). +- **Fields**: endpoint describe fields + `Data Capture Enabled` (bool), `severity` + +--- + +## Sustainability + +### Domain Region Check +- **APIs**: `sagemaker.list-domains` → `describe-domain` +- **Logic**: inventory of domains and their regions; no pass/fail. Empty → "No Domains Found" +- **Do not assert a region's carbon intensity or renewable-energy mix.** The skill has no data + source for this, and unsourced claims have flipped between runs on the same account — one run + called us-east-1 low-renewable and recommended migrating away, the next called it "strong + renewable energy coverage". State where domains run and, if the user is pursuing a sustainability + goal, point them at the AWS [customer carbon footprint tool](https://aws.amazon.com/aws-cost-management/aws-customer-carbon-footprint-tool/) + and AWS's published regional renewable-energy data rather than ranking regions in the report. +- **Fields**: `domainId`, `domainName`, `region`, `status` + +--- + +## Best Practices + +### Well-Architected Recommendations (SageMaker AI) +- **APIs**: none — advisory content grounded in the public AWS Well-Architected lenses below. +- **Logic**: emit a concise set of SageMaker-AI-specific recommendations, scoped to what the other checks observed where possible. Cite the lens each recommendation draws from. Do not fabricate resource findings — this section is guidance, not per-resource data. +- **Sources** (public): + - Machine Learning Lens — https://docs.aws.amazon.com/wellarchitected/latest/machine-learning-lens/machine-learning-lens.html + - Generative AI Lens — https://docs.aws.amazon.com/wellarchitected/latest/generative-ai-lens/generative-ai-lens.html + - Agentic AI Lens — https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentic-ai-lens.html +- **Recommendation themes** (SageMaker AI only; tailor to observed resources): + - **Model lifecycle & MLOps** — version models in the SageMaker Model Registry, automate build/train/deploy with SageMaker Pipelines, and gate promotions with approval status (ML Lens: MLOps). + - **Endpoint efficiency & scaling** — right-size instances, enable autoscaling, and prefer serverless/async for spiky or latency-tolerant traffic (ML Lens: Performance/Cost). + - **Inference cost** — use Inferentia/Trainium where supported, adopt SageMaker Savings Plans on steady inference spend, and retire stale endpoints (ML Lens: Cost Optimization). + - **Security & isolation** — CMK encryption on endpoints/notebooks, `VpcOnly` Studio domains, network-isolated models, least-privilege execution roles (ML Lens: Security). + - **Generative AI hosting** — for FM/LLM endpoints, monitor `ModelLatency`/token throughput, enable data capture for evaluation, and guard against prompt-injection at the application tier (GenAI Lens). + - **Agentic workloads** — when SageMaker hosts models behind agents, apply tool-access least privilege, observability on agent/tool calls, and human-in-the-loop for high-impact actions (Agentic AI Lens). +- **Fields**: `recommendation`, `pillar`, `lens`, `rationale` From 9ccb457d0498993908d1e4ad9da2541815c09607 Mon Sep 17 00:00:00 2001 From: Jack Lunn Date: Fri, 18 Sep 2026 17:20:50 -0500 Subject: [PATCH 2/2] feat(custom-agents): add SageMaker AI support to aws-operation-review Wires the new sagemaker-ops-review skill into the existing operational review agent rather than shipping a second review agent, so SageMaker AI joins EKS, RDS, Aurora and Bedrock under one entry point. SYSTEM_PROMPT.md: SageMaker AI added to the Goal and to the service identification step, sagemaker-ops-review added to the skill selection list, and a SageMaker artifact naming example added. The skill's report schema is named in the existing "defer to the selected skill's report schema" guidance, alongside the bedrock-operation-review example, because sagemaker-ops-review defines its own eight pillars, a verbatim AI Disclaimer, and a severity-ranked Executive Summary that should not be forced into the generic category set. README.md: SageMaker AI added to Purpose, Key Capabilities, Prerequisites and Related, and the skill selection step now reflects that the skills are chosen per service rather than always both. The Prerequisites entry notes that AIDevOpsAgentAccessPolicy covers every API the skill calls except the optional savingsplans:DescribeSavingsPlans. CHANGELOG.md: bumped to 1.1.0. llms.txt: sagemaker-ops-review added to Available Skills, and the AWS Operation Review entry now lists SageMaker AI. cloudformation/devops-agent-skill-policies.yaml: adds the EnableSageMakerOpsReview parameter, its condition, a PolicySageMakerOpsReview resource granting savingsplans:DescribeSavingsPlans, and a SkillPolicySummary line. Every other API the skill calls is already covered by the managed policy, so this is the only addition required. Note on skill agent type: this skill's README instructs uploading with "Generic" / "All agents" selected rather than narrowing to specific agent types, because a narrowed skill does not appear in the custom agent's skill picker. The aws-operation-review README already documents that as a workaround for the EKS and RDS skills; sagemaker-ops-review states it up front instead, so the caveat is not needed for it. --- .../devops-agent-skill-policies.yaml | 31 +++++++++++++++++++ .../aws-operation-review/CHANGELOG.md | 6 ++++ custom-agents/aws-operation-review/README.md | 9 ++++-- .../aws-operation-review/SYSTEM_PROMPT.md | 13 +++++--- llms.txt | 3 +- 5 files changed, 53 insertions(+), 9 deletions(-) diff --git a/cloudformation/devops-agent-skill-policies.yaml b/cloudformation/devops-agent-skill-policies.yaml index 628ce62..cf7535d 100644 --- a/cloudformation/devops-agent-skill-policies.yaml +++ b/cloudformation/devops-agent-skill-policies.yaml @@ -29,6 +29,7 @@ Metadata: - EnableDmsOperationReview - EnableAgentCoreObservabilitySetup - EnableAgentCoreOpsReview + - EnableSageMakerOpsReview - Label: default: Optional Resource Scoping Parameters: @@ -124,6 +125,11 @@ Parameters: Description: AgentCore Operational Review skill (adds read-only bedrock-agentcore control-plane List/Get and ec2:DescribeSubnets for the multi-AZ check; observability-only mode needs none of these). AllowedValues: ['true', 'false'] Default: 'true' + EnableSageMakerOpsReview: + Type: String + Description: SageMaker AI Operational Review skill (adds savingsplans:DescribeSavingsPlans for the Savings Plan check). + Default: 'false' + AllowedValues: ['true', 'false'] Conditions: CreateNewRole: !Equals [!Ref ExistingRoleName, ''] @@ -136,6 +142,7 @@ Conditions: SkillDmsOperationReview: !Equals [!Ref EnableDmsOperationReview, 'true'] SkillAgentCoreObservabilitySetup: !Equals [!Ref EnableAgentCoreObservabilitySetup, 'true'] SkillAgentCoreOpsReview: !Equals [!Ref EnableAgentCoreOpsReview, 'true'] + SkillSageMakerOpsReview: !Equals [!Ref EnableSageMakerOpsReview, 'true'] HasRegionRestriction: !Not [!Equals [!Join ['', !Ref AllowedRegions], '']] Resources: @@ -428,6 +435,29 @@ Resources: - support.amazonaws.com - ce.amazonaws.com + + # sagemaker-ops-review: adds savingsplans:DescribeSavingsPlans for the Savings Plan check. + # Every other API the skill calls -- sagemaker List/Describe/ListTags, CloudWatch metric reads, + # application-autoscaling:Describe*, servicequotas:GetServiceQuota, Cost Explorer region + # discovery, and health:DescribeEvents/DescribeAffectedEntities -- is already covered by + # AIDevOpsAgentAccessPolicy. Without this policy the Savings Plan check reports + # "not evaluated -- permission not granted" and the other 19 checks run normally. + PolicySageMakerOpsReview: + Type: AWS::IAM::Policy + Condition: SkillSageMakerOpsReview + Properties: + PolicyName: DevOpsAgentSkill-SageMakerOpsReview + Roles: + - !If [CreateNewRole, !Ref DevOpsAgentRole, !Ref ExistingRoleName] + PolicyDocument: + Version: '2012-10-17' + Statement: + - Sid: SageMakerOpsReviewSavingsPlansRead + Effect: Allow + Action: + - savingsplans:DescribeSavingsPlans + Resource: '*' + Outputs: DevOpsAgentRoleArn: Description: Role ARN to use with aws devops-agent associate-service. @@ -453,6 +483,7 @@ Outputs: - database-migration-service-expertise: ${EnableDmsOperationReview} (dms:TestConnection) - agentcore-observability-setup: ${EnableAgentCoreObservabilitySetup} (bedrock-agentcore:Get/ListAgentRuntime, xray:GetTraceSegmentDestination, logs:DescribeDeliveries/DeliverySources/DeliveryDestinations/ResourcePolicies, lambda:GetFunctionConfiguration, ecs:DescribeTaskDefinition/DescribeServices/ListTasks, eks:DescribeCluster) - agentcore-ops-review: ${EnableAgentCoreOpsReview} (bedrock-agentcore read-only List/Get for runtimes/memories/gateways/browsers/code-interpreters/workload-identities, ec2:DescribeSubnets) + - sagemaker-ops-review: ${EnableSageMakerOpsReview} (savingsplans:DescribeSavingsPlans) Skills covered by AIDevOpsAgentAccessPolicy (no extra policy needed): - eks-operation-review, enrich-with-aws-security-agent, crm-production-investigation-guidelines No IAM required: diff --git a/custom-agents/aws-operation-review/CHANGELOG.md b/custom-agents/aws-operation-review/CHANGELOG.md index 0400ff7..9237653 100644 --- a/custom-agents/aws-operation-review/CHANGELOG.md +++ b/custom-agents/aws-operation-review/CHANGELOG.md @@ -1,5 +1,11 @@ # Changelog +## 1.1.0 + +- Added Amazon SageMaker AI support via the `sagemaker-ops-review` skill — endpoints, training jobs, pipelines, notebooks, feature store, model registry, and Studio domains +- Documented that SageMaker AI reviews need `sagemaker-ops-review` uploaded with "All agents" selected, and that `AIDevOpsAgentAccessPolicy` covers every API it calls except the optional `savingsplans:DescribeSavingsPlans` +- Noted the `sagemaker-ops-review` report schema (eight pillars, verbatim AI Disclaimer, severity-ranked Executive Summary) in the report-schema deference guidance, and added a SageMaker artifact naming example + ## 1.0.0 - Initial version diff --git a/custom-agents/aws-operation-review/README.md b/custom-agents/aws-operation-review/README.md index 7ed0bb5..46e49e9 100644 --- a/custom-agents/aws-operation-review/README.md +++ b/custom-agents/aws-operation-review/README.md @@ -2,12 +2,13 @@ ## Purpose -This custom agent performs comprehensive operational reviews of AWS services (EKS clusters, RDS instances, Aurora clusters) against best practices and the Well-Architected Framework. It identifies gaps in security, reliability, performance, cost optimization, and operational excellence, producing actionable recommendations and a structured report artifact. +This custom agent performs comprehensive operational reviews of AWS services (EKS clusters, RDS instances, Aurora clusters, Amazon SageMaker AI workloads) against best practices and the Well-Architected Framework. It identifies gaps in security, reliability, performance, cost optimization, and operational excellence, producing actionable recommendations and a structured report artifact. ## Key Capabilities - Assesses EKS clusters for version currency, security posture, networking, logging, and node group configuration - Evaluates RDS/Aurora instances for engine versions, backup configuration, encryption, Multi-AZ, parameter compliance, and cost optimization +- Reviews Amazon SageMaker AI workloads across eight pillars and twenty checks — endpoint encryption and VPC isolation, Studio domain network posture, endpoint autoscaling and staleness, service quota headroom, AWS Health lifecycle events, data capture, and Well-Architected guidance - Assigns severity levels (critical, high, medium, low) based on security exposure, blast radius, and operational risk - Generates a prioritized report with remediation steps and effort/impact estimates - Produces a persisted Markdown artifact for sharing with stakeholders @@ -15,9 +16,10 @@ This custom agent performs comprehensive operational reviews of AWS services (EK ## Prerequisites - An AWS DevOps Agent space -- IAM permissions for EKS read APIs (`eks:DescribeCluster`, `eks:ListClusters`, `eks:ListNodegroups`, `eks:DescribeNodegroup`, `eks:ListAddons`, `eks:DescribeAddon`) and/or RDS read APIs (`rds:DescribeDBInstances`, `rds:DescribeDBClusters`, `rds:DescribeDBParameterGroups`, `rds:ListTagsForResource`) +- IAM permissions for EKS read APIs (`eks:DescribeCluster`, `eks:ListClusters`, `eks:ListNodegroups`, `eks:DescribeNodegroup`, `eks:ListAddons`, `eks:DescribeAddon`) and/or RDS read APIs (`rds:DescribeDBInstances`, `rds:DescribeDBClusters`, `rds:DescribeDBParameterGroups`, `rds:ListTagsForResource`). For SageMaker AI reviews, the managed `AIDevOpsAgentAccessPolicy` already covers every API the skill calls except `savingsplans:DescribeSavingsPlans`, which is an optional add-on — see the [sagemaker-ops-review prerequisites](../../skills/sagemaker-ops-review/README.md#2-iam-permissions) - The [eks-operation-review skill](../../skills/eks-operation-review/) uploaded to your Agent Space. Important note: for the skill to be used by the custom agent, choose "All agents" in the "Agent Type" field when importing the skill, even that the skill's README file instructs to choose specific agent types - The [rds-operation-review skill](../../skills/rds-operation-review/) uploaded to your Agent Space. Important note: for the skill to be used by the custom agent, choose "All agents" in the "Agent Type" field when importing the skill, even that the skill's README file instructs to choose specific agent types +- For SageMaker AI reviews, the [sagemaker-ops-review skill](../../skills/sagemaker-ops-review/) uploaded to your Agent Space with "All agents" selected in the "Agent Type" field ## Creating the Agent @@ -25,7 +27,7 @@ This custom agent performs comprehensive operational reviews of AWS services (EK 2. Click "Create agent" (on the right side), then on the new menu that popped up, click "Form" (the left-most option) 3. In the "Name" field, use "aws-operation-review" 4. Copy the content of the "SYSTEM_PROMPT.md" file from this directory, and paste it into the "System prompt" field in the custom agent creation form -5. In the "Skills" drop-down list, select both the "eks-operation-review" and "rds-operation-review" skills, and click "Create agent" +5. In the "Skills" drop-down list, select the skills for the services you want to review — "eks-operation-review", "rds-operation-review", and/or "sagemaker-ops-review" — and click "Create agent" 6. Now we need to add the `use_aws` and `use_kubectl` tools - in the new custom agent's window, click "Edit" 7. In the new popped up window, select "Chat". A new chat will start on the left side. Wait for DevOps Agent to finish thinking, and it'll ask you what would you like to change 8. Type "Add the use_aws and use_kubectl tools to this custom agent". Once the chat is finished, verify in the custom agent's page that both `use_aws` and `use_kubectl` are shown under "Tools" for this custom agent @@ -39,4 +41,5 @@ Once finished, the artifact is persisted on the **Artifacts** page in the DevOps - [eks-operation-review skill](../../skills/eks-operation-review/) — domain knowledge for EKS cluster assessments - [rds-operation-review skill](../../skills/rds-operation-review/) — domain knowledge for RDS/Aurora database assessments +- [sagemaker-ops-review skill](../../skills/sagemaker-ops-review/) — domain knowledge for Amazon SageMaker AI operational reviews - [AWS DevOps Agent custom agents documentation](https://docs.aws.amazon.com/devopsagent/latest/userguide/working-with-devops-agent-custom-agents-index.html) diff --git a/custom-agents/aws-operation-review/SYSTEM_PROMPT.md b/custom-agents/aws-operation-review/SYSTEM_PROMPT.md index b007b9a..66ae90f 100644 --- a/custom-agents/aws-operation-review/SYSTEM_PROMPT.md +++ b/custom-agents/aws-operation-review/SYSTEM_PROMPT.md @@ -2,15 +2,16 @@ You are an AWS Operations Review Specialist focused on assessing AWS services ag ## Goal -Perform comprehensive operational reviews of AWS services (EKS clusters, RDS instances, Aurora clusters, Bedrock workloads) to identify gaps in security, reliability, performance, cost optimization, and operational excellence — aligned with AWS best practices and the Well-Architected Framework. +Perform comprehensive operational reviews of AWS services (EKS clusters, RDS instances, Aurora clusters, Bedrock workloads, Amazon SageMaker AI workloads) to identify gaps in security, reliability, performance, cost optimization, and operational excellence — aligned with AWS best practices and the Well-Architected Framework. ## Approach -1. Identify which AWS service the user wants reviewed (EKS, RDS, Aurora, or Bedrock). +1. Identify which AWS service the user wants reviewed (EKS, RDS, Aurora, Bedrock, or SageMaker AI). 2. Load the appropriate skill for the service: - For EKS clusters: use the `eks-operation-review` skill methodology - For RDS/Aurora databases: use the `rds-operation-review` skill methodology - For Bedrock workloads: use the `bedrock-operation-review` skill methodology + - For Amazon SageMaker AI workloads (endpoints, training jobs, pipelines, notebooks, feature store, model registry, Studio domains): use the `sagemaker-ops-review` skill methodology 3. Follow the skill's structured assessment framework to evaluate the resource. 4. For each finding, assess severity (critical, high, medium, low) based on security exposure, blast radius, and operational risk. 5. Generate actionable recommendations with clear remediation steps. @@ -44,12 +45,14 @@ Generate a shareable report artifact as a Markdown document. its own artifact naming and report structure (including its own pillars/categories) in its Step "Generate Report" section — follow that schema exactly when a skill is loaded. For example, the `bedrock-operation-review` skill organizes findings by its five -pillars (Security, Performance, Service Quotas, Cost Optimization, Resilience), not the -generic categories below. Do not force a skill's findings into the generic category set. +pillars (Security, Performance, Service Quotas, Cost Optimization, Resilience), and the +`sagemaker-ops-review` skill organizes them by its eight pillars with a verbatim AI +Disclaimer and a severity-ranked Executive Summary — not the generic categories below. Do +not force a skill's findings into the generic category set. **Artifact naming:** use the naming defined by the selected skill. If the skill does not specify one, fall back to `-review--.md`. -Examples: `eks-review-prod-cluster-2026-06-21.md`, `rds-review-orders-db-2026-06-21.md`, `bedrock-review-1234567890-us-east-1-2026-08-21.md` +Examples: `eks-review-prod-cluster-2026-06-21.md`, `rds-review-orders-db-2026-06-21.md`, `bedrock-review-1234567890-us-east-1-2026-08-21.md`, `sagemaker-review-1234567890-us-east-1-2026-09-18.md` **Report structure (fallback):** use the following only when the selected skill does not define its own report structure. When it does, the skill's structure takes precedence. diff --git a/llms.txt b/llms.txt index 17c6e13..cc93d44 100644 --- a/llms.txt +++ b/llms.txt @@ -40,12 +40,13 @@ Tools can be used with these AWS DevOps Agent types: - [AgentCore Operational Review Skill](skills/agentcore-ops-review/SKILL.md): Read-only operational review of Amazon Bedrock AgentCore resources aligned with the AWS Well-Architected Framework, discovering runtimes, memories, gateways, browsers, code interpreters, and workload identities and assessing runtime resilience, gateway health, memory and knowledge effectiveness, and resource utilization from control-plane and CloudWatch signals, degrading missing signals to documented visibility limits rather than false findings - [RDS/Aurora Database Diagnostics Skill](skills/database-rds-devops/SKILL.md): Runs database-level data-plane diagnostics for Aurora MySQL and Aurora PostgreSQL via predefined read-only health check queries over the RDS Data API, covering buffer pool, connections, locks, replication, storage, performance, and index efficiency, using the rds-aidba MCP server - [Investigation Cost Guardrail Skill](skills/investigation-cost-guardrail/SKILL.md): Estimates and caps the cost of paid API calls during investigations across all AWS services and native agent tools, enforcing per-investigation budgets, flagging expensive operations, requiring time windows, and cancelling when thresholds are exceeded +- [SageMaker AI Operational Review Skill](skills/sagemaker-ops-review/SKILL.md): Performs read-only Amazon SageMaker AI operational reviews across eight pillars and twenty checks — security, performance, cost optimization, service quotas, resiliency, operational excellence, sustainability, and Well-Architected best practices — producing severity-ranked findings with one recommendation per High or Medium finding ## Available Custom Agents - [AWS Health Report](custom-agents/aws-health-report/README.md): Generates a report of AWS Health events (service issues, scheduled changes, account notifications) over a configurable period, grouped by service and category - [Support Cases Report](custom-agents/support-cases-report/README.md): Generates a consolidated report of AWS Support cases over a configurable period, highlighting recurring patterns and items requiring follow-up -- [AWS Operation Review](custom-agents/aws-operation-review/README.md): Performs comprehensive operational reviews of AWS services (EKS, RDS, Aurora) against best practices and the Well-Architected Framework, producing a structured report artifact +- [AWS Operation Review](custom-agents/aws-operation-review/README.md): Performs comprehensive operational reviews of AWS services (EKS, RDS, Aurora, SageMaker AI) against best practices and the Well-Architected Framework, producing a structured report artifact - [Service Quotas Monitor](custom-agents/service-quotas-monitor/README.md): Proactively monitors AWS service quotas across active regions, flags quotas at 85%+ utilization, and requests increases or escalates via support cases - [Redshift Support Specialist](custom-agents/redshift-support-specialist/README.md): Amazon Redshift support agent for query optimization, operational reviews, and cost optimization, paired with the redshift-support-specialist skill