feat: AgentCore Instances compute type (x86_64) for managed environments - #468
matheus-1618 wants to merge 20 commits into
Conversation
Managed environments can now target the AgentCore Instances compute type
(compute: { type: 'instances', architecture: 'x86_64' | 'arm64' }):
- capacity provider per architecture, created lazily by the status
lambda with a deterministic name (immutable resource, create-once)
- runtime created with capacityProviderConfiguration; networking is
inherited from the capacity provider and the workspace moves from
managed session storage to a persistent EBS volume at /mnt/workspace
- x86_64 environments build FROM an amd64 variant of the core image
(opt-in terraform build) on an x86 CodeBuild fleet via per-build
overrides; catalog tools stay arm64-only for now
- everything is gated behind enable_instances_compute (default false)
The previous pin referenced the arm64-specific manifest, which silently ignores --platform and blocks amd64 builds of the core image. The index digest resolves to the exact same arm64 manifest on arm builds (no-op) and selects the amd64 manifest when building for x86_64.
Kiro, OpenCode, bun and uv were pinned to arm64 artifacts, which made --platform linux/amd64 builds impossible. Each download now selects the artifact and its pinned sha256 by TARGETARCH; arm64 builds keep the exact same artifacts as before.
Admins choose between serverless microVMs (default, arm64) and EC2 Instances (x86_64) when creating a managed environment. The choice is immutable after creation, matching the AgentCore contract, so the selector only renders on the create flow.
The verification prologue hardcoded the arm64 assertion, rejecting correctly-built x86_64 images. The expected architecture now follows the recipe (docker inspect reports x86_64 as amd64).
…counts In accounts where ECR enhanced scanning is enabled, DescribeImageScanFindings is backed by Amazon Inspector and the status lambda also needs inspector2:ListCoverage/ListFindings, otherwise every image scan fails with an authorization error.
The first CreateCapacityProvider in an account provisions the AWSServiceRoleForBedrockAgentCoreRuntimeInstances service-linked role on behalf of the caller, which requires iam:CreateServiceLinkedRole scoped to that service.
CreateAgentRuntime with a capacityProviderConfiguration authorizes the association through PassCapacityProvider on the capacity provider ARN.
…device workspace EBS-backed workspace mounts (Instances compute type) come formatted, so /mnt/workspace contains lost+found and git clone refuses the non-empty destination. Remove only the well-known mkfs entries when no checkout exists; any other residue still fails loudly.
JWThewes
left a comment
JWThewes
left a comment
There was a problem hiding this comment.
This adds opt-in EC2-backed AgentCore environments with x86_64 images and persistent EBS workspaces, while retaining arm64 microVMs as the default. The focused compute module and reuse of the existing build pipeline are sensible.
Requesting changes for the five inline findings: persistent-session cleanup, cold-start retry identity, published-core image consistency, capacity-provider replacement after configuration changes, and architecture compatibility for inherited environments. These affect resource lifecycle, build reliability, and established revision semantics.
Validation on b5871c8: all 122 environment tests and 11 workspace/runtime tests passed, and the environment Lambda bundles built successfully. Additional local probes reproduced the retry, image-selection, provider-configuration, and inheritance issues; persistent-volume retention was checked against the official AWS documentation. No AWS deployment or Docker image rebuild was performed during this review.
| volumes: [ | ||
| { | ||
| ebsConfiguration: { | ||
| name: WORKSPACE_VOLUME_NAME, | ||
| sizeGiB: Number(process.env.MANAGED_INSTANCES_WORKSPACE_GIB || 50), | ||
| volumeType: 'gp3', |
There was a problem hiding this comment.
[P1] Add explicit cleanup for persistent Instances sessions
These volumes persist after StopRuntimeSession, idle timeout, and maximum lifetime; they are deleted only by explicit session/provider deletion (AWS lifecycle documentation). The existing validation cleanup in status.js only stops its randomly named session, and intent deletion in lambda/shared/intent-deletion.js also only stops sessions. Consequently, each validation session that provisions storage can leave a default 50-GiB volume behind, and deleting an intent leaves its workspace data and storage charges behind indefinitely.
Please retain the provider/session identifiers needed for cleanup, delete disposable validation sessions after a terminal result, and delete persistent sessions when their owning intent is permanently deleted. Add the necessary IAM permissions and lifecycle tests. Sessions temporarily stopped for park/resume should retain their volumes.
There was a problem hiding this comment.
Addressed in 5c06bb5.
- The capacity provider ARN is now persisted on the revision at runtime creation and carried into the intent's environment snapshot, so both cleanup paths have the identifiers they need.
- Disposable validation sessions are deleted (
DeleteCapacityProviderSession) on every terminal outcome. - Permanent intent deletion deletes the intent's main and lane sessions. An unexpected error aborts the cascade before the DynamoDB partition delete, so the intent still lists and the delete can simply be re-run (sessions that never existed are tolerated as misses). Park/resume is untouched — stopped sessions retain their volumes.
- IAM:
DeleteCapacityProviderSessiongranted to the status, intents and projects lambdas;@aws-sdk/*bumped to ^3.1135.0 (the command only exists in the data-plane client from there) via the repo'sscripts/sync-aws-sdk.mjs. - Lifecycle tests added for both paths (
lambda/shared/test/intent-deletion-sessions.test.js+status-instances.test.js).
| if ( | ||
| RETRYABLE_CONTROL_ERRORS.has(error?.name) || | ||
| (environment.compute?.type === 'instances' && INSTANCES_TRANSIENT_ERRORS.has(error?.name)) | ||
| ) { |
There was a problem hiding this comment.
[P2] Retry provisioning with the same validation session
Returning pending here does not retry the session that was provisioning: the next poll reaches randomUUID() at line 506 and invokes a new session, while the finally block stops the previous agent session. If the initial invocation consistently exceeds the provisioning window, every poll repeats a cold start and the revision can remain VERIFYING indefinitely. A local probe in which the first invocation of each session fails transiently and subsequent invocations can succeed produced three different sessions across three polls, with no revision progress.
Please persist or deterministically derive one validation session ID per revision, reuse it across transient retries, and defer terminal cleanup until validation succeeds or exhausts a bounded retry/deadline policy. Cover a cold start that requires more than one poll.
There was a problem hiding this comment.
Addressed in de4beb2.
The validation session ID is now persisted on the revision and reused across polls, so a transient first-invoke error keeps the provisioning session alive and the next poll reattaches to it instead of repeating the cold start. The session is only stopped on a terminal outcome, and the retry budget is bounded (MANAGED_INSTANCES_VALIDATION_MAX_POLLS, default 30 one-minute polls — exhaustion fails the revision with a clear message). microVM validation keeps the original per-poll semantics. Covered by a multi-poll cold-start test that asserts the same session ID across polls plus the exhausted-budget path.
| const amd64 = amd64CoreImage(); | ||
| return { | ||
| ...recipe, | ||
| architecture: 'x86_64', | ||
| base: { ...recipe.base, imageUri: amd64.imageUri, imageDigest: amd64.imageDigest }, |
There was a problem hiding this comment.
[P2] Resolve the amd64 image from the selected published core revision
amd64CoreImage() reads the latest deployment's environment variables, while this spread retains the selected Standard revision's revisionId. During a platform upgrade, stageCoreRevision deliberately leaves Standard's published pointer on the old revision until publication. Creating/editing/rebuilding an x86 environment during that interval therefore uses the new core image while claiming the old published base. A local probe kept the published base revision unchanged, changed only the configured amd64 digest, and obtained a different base image for that same revision.
Please store architecture-specific image references with the core revision and resolve the amd64 variant belonging to the selected published revision. Add an upgrade test where a newer core has been deployed but has not yet been published as Standard; x86 environments should continue using the published core's variant.
There was a problem hiding this comment.
Addressed in 5724ba0.
The amd64 variant now travels with the core revision: seedSystemEnvironments and stageCoreRevision store amd64Image on the Standard revision, and when the deployed digest matches the already-published revision the variant is backfilled onto it (pre-existing deployments gain their variant exactly). applyComputeBase resolves the amd64 refs from the selected base revision only — the env-var reader is gone — and rejects with AMD64_CORE_IMAGE_MISSING when the revision has no variant. Includes the upgrade-window test: newer core deployed but not yet published as Standard → the x86 environment keeps using the published revision's variant.
| const match = (page.capacityProviders ?? []).find((item) => item.name === name); | ||
| if (match) { | ||
| if (match.status === 'READY') return { capacityProviderArn: match.capacityProviderArn }; | ||
| if (match.status === 'CREATING') return { pending: true }; |
There was a problem hiding this comment.
[P2] Version capacity providers when their configuration changes
The lookup uses only the project/environment/architecture name, so an existing READY provider is returned without checking its configuration. For example, changing the exposed Terraform allowlist from m6i.large to m6i.xlarge updates the Lambda environment but all subsequent runtimes still use the old provider. The same applies to replaced subnets/security groups and storage settings; a local probe changed the requested subnets and instance types and still received the original provider unchanged. A failed provider also permanently occupies the same lookup identity after configuration is corrected.
Please include a fingerprint of the immutable provider configuration in its identity, or implement an explicit replacement flow, so new revisions use the intended configuration while existing revisions retain their provider. Test both configuration changes and recovery after a failed configuration is corrected.
There was a problem hiding this comment.
Addressed in 6ed7916.
The provider name now embeds a short hash of the create-time configuration (instance allowlist, VPC, storage, lifecycle, operator role), so a changed — or corrected-after-failure — configuration produces a fresh provider while runtimes created earlier keep the one they were built with. Superseded providers are intentionally left in place since they may still back existing runtimes (documented in the module header). Tests cover both a configuration change producing a new identity and recovery after a failed configuration is corrected.
| const recipe = compute | ||
| ? applyComputeBase({ recipe: prepared.recipe, compute }) | ||
| : prepared.recipe; | ||
| const flattenedRecipe = compute | ||
| ? applyComputeBase({ recipe: prepared.flattenedRecipe, compute }) | ||
| : prepared.flattenedRecipe; |
There was a problem hiding this comment.
[P2] Validate base architecture for default microVM environments too
Published x86 environments are included in the base selector, but omitting compute normalizes to null and skips the compute/base checks here. A local handler probe created a default microVM environment derived from a published x86 parent and received HTTP 201 with an amd64 base digest and no recipe architecture. Its build is then scheduled as linux/arm64, so this accepted configuration cannot pass the image architecture checks. The edit path has the same validation gap.
Please validate the target architecture against the selected base revision for every creation/revision path, including the default compute type, and reject incompatible combinations before saving or queuing a build. Filter incompatible bases in the UI as well, and add a regression test for an arm64 child of an x86 parent.
There was a problem hiding this comment.
Addressed in fbaa253.
assertBaseArchitecture now runs on every creation/revision path — create, edit, and rebuild-on-latest-base — including the default (compute omitted) case, rejecting an x86_64 base for an arm64 target with a 409 BASE_ARCHITECTURE_MISMATCH before anything is saved or queued. The UI base selector filters incompatible bases as well (x86_64 targets only offer Standard; arm64 targets exclude published x86_64 environments), and switching the compute selector to x86_64 resets the base to Standard. Regression test added for an arm64 child of a published x86_64 parent.
…-start retries The runtime validation generated a new random session on every poll, so a transient first-invoke error (EC2 still provisioning) meant each retry repeated the cold start while the finally block stopped the session that was provisioning — a revision could stay VERIFYING indefinitely. The session ID is now persisted on the revision and reused across polls; the session is only stopped on a terminal outcome, and the retry budget is bounded (MANAGED_INSTANCES_VALIDATION_MAX_POLLS, default 30 one-minute polls). microVM validation keeps the original per-poll semantics.
… its identity Capacity providers are immutable, but the lookup used only the prefix+architecture name: changing the exposed configuration (instance allowlist, subnets/security groups, storage, lifecycle, operator role) had no effect on subsequent runtimes, and a failed provider permanently occupied the lookup identity even after the configuration was corrected. The provider name now embeds a short hash of the create-time configuration, so a changed (or corrected) configuration produces a fresh provider while runtimes created earlier keep the one they were built with. Superseded providers are intentionally left in place — they may still back existing runtimes.
…n path A default microVM environment could derive from a published x86_64 parent: compute normalizes to null, the compute/base checks were skipped, and the revision was accepted with an amd64 base digest — its arm64 build could never pass the image architecture checks. The base architecture is now asserted on create, edit and rebuild-on-latest-base, and the UI base selector only offers compatible bases.
… released Instances sessions keep their persistent workspace volumes across stop, idle timeout and maximum lifetime — only an explicit DeleteCapacityProviderSession releases them. Validation left one default 50-GiB volume behind per revision, and deleting an intent left its workspace volumes (and charges) behind indefinitely. - the capacity provider ARN is persisted on the revision at runtime creation and carried into the intent's environment snapshot - disposable validation sessions are deleted on every terminal outcome - permanent intent deletion deletes the intent's main and lane sessions; an unexpected error aborts before the DynamoDB delete so the cascade stays re-runnable (sessions that never existed are tolerated) - park/resume is untouched — stopped sessions retain their volumes - IAM: DeleteCapacityProviderSession for the status, intents and projects lambdas; @aws-sdk/* bumped to ^3.1135.0 for the new command (scripts/sync-aws-sdk.mjs)
…ision applyComputeBase read the amd64 core refs from the deployment's environment variables while the recipe pinned the selected published Standard revision. During a platform upgrade the staged core is newer than the published one, so an x86 environment created in that window mixed the new image bytes with the old revision identity. The amd64 variant now travels with the core revision (seeded, staged, and backfilled onto a published revision with the matching digest) and applyComputeBase resolves it from the selected base revision only.
|
All five review findings are addressed, one commit per finding:
Each carries the regression tests requested inline (details in the per-thread replies). Note for reviewers: 5c06bb5 bumps Validation: 2871 backend + 569 frontend tests passing, |
Conflicts: the @aws-sdk/* range (main stayed on ^3.1092.0, this branch needs ^3.1135.0 for DeleteCapacityProviderSessionCommand — re-applied via scripts/sync-aws-sdk.mjs) and the validation-session cleanup block in status.js (kept this branch's structure, adopted main's Powertools logger).
2 participants
What
Adds opt-in support for the AgentCore Instances compute type to managed environments, including x86_64 environments — the first non-arm64 path in the platform.
Admins creating a managed environment can now choose its compute:
Everything is gated behind a new terraform variable
enable_instances_compute(defaultfalse). Deployments that don't opt in are unaffected — with the flag off there is no amd64 image build, no operator role, and the compute selector rejectsinstances.Why
/mnt/workspace, lifting the fixed 1 GB session-storage ceiling and surviving session stops.How
compute: { type, architecture }field on the environment (API + UI selector on the create flow; immutable after creation, matching the AgentCore contract).capacityProviderConfiguration+capacityProviderVolume(nonetworkConfiguration— Instances inherits the capacity provider's VPC).environmentTypeOverride/imageOverride+ anIMAGE_PLATFORMvariable in the buildspec).--platform), and Kiro/OpenCode/bun/uv select artifact + pinned sha256 byTARGETARCH. arm64 builds resolve to the exact same artifacts as before.Fixes picked up along the way (apply to microVMs deployments too)
inspector2:ListCoverage/ListFindingsfor the status lambda — in accounts with ECR enhanced scanning,DescribeImageScanFindingsis served by Amazon Inspector and every image scan fails without these.iam:CreateServiceLinkedRole(scoped) — the first capacity provider in an account provisions theAWSServiceRoleForBedrockAgentCoreRuntimeInstancesSLR.bedrock-agentcore:PassCapacityProvider— required byCreateAgentRuntimewhen associating a capacity provider.mkfs.ext4leaveslost+foundat the volume root andgit clonerefuses the non-empty destination. The workspace init now clears the well-known filesystem entries before cloning (any other residue still fails loudly).Notes and limitations
Instance type 't3.large' is not supported); the default allowlist usesm6i.large. m5/m6i/c5/c6i were verified to be accepted.Validation
End-to-end on a fresh us-east-1 deployment with
enable_instances_compute = true:compute: { type: instances, architecture: x86_64 }.--platform linux/amd64, image built FROM the amd64 core and passed the (now architecture-aware) verification script.READY), runtime created withcapacityProviderConfigurationand theworkspaceEBS volume at/mnt/workspace.arch=x86_64, non-root, writable workspace, all four agent CLIs installed.linux/amd64andlinux/arm64builds of the core image verified locally (all six CLI binaries at their pinned versions on both).Tests: 122 backend (18 new covering the compute module and the Instances runtime path) and 569 frontend, all passing;
terraform validateclean.