feat: add dynamodb-latency skill

[aditya-vikram-parakala]

Description

Adds dynamodb-latency, a skill that answers one question with evidence: is DynamoDB actually slow, and if so, why?

The metric everyone reaches for cannot answer that alone. SuccessfulRequestLatency measures time internal to the DynamoDB service — client activity and network trip time are excluded — and it counts successful requests only, so throttled and failed calls never appear in it. That produces a trap in both directions: a clean service-side curve gets read as "no problem found" on a workload that really is slow, and a flat curve beside non-zero throttle events looks healthy when it is actually retry-inflated client latency.

The skill collects the signals that separate those cases, commits to exactly one classification, and — when the service side is clean — attributes the remaining time to the client and names the measurement that would confirm it.

Classification

One primary outcome, chosen in a fixed decision order (indeterminate is evaluated first, so a window without data can never render as healthy):

Outcome	Established by
LT-00 indeterminate	latency published no data, or coverage is materially short of the window
LT-01 service_side	SystemErrors measured above zero
LT-02 throttle_induced	throttle events overlap the elevated-latency periods in time
LT-03 systemic_degradation	Average/p50 sustained above threshold
LT-04 transient_spike	p99/Maximum elevated while Average/p50 stays normal
LT-05 nominal	every signal measured and within threshold

Nine contributing-factor rules (CF-01…CF-09) report alongside the classification, never instead of it.

Three design choices come straight from what the metric does and does not measure:

• Collection is per Operation dimension. A table-wide average blends GetItem with Scan, whose documented expectations differ by an order of magnitude; multi-item operations are judged against their own trailing baseline rather than an absolute threshold.
• LT-02 is established by timestamp overlap, not totals — the elevated-latency periods must intersect the throttle-event periods — so it holds when the successful-request curve stays flat.
• NoData is never coerced to zero. A nominal verdict requires SystemErrors and the throttle metrics to have been measured at zero; an absent metric cannot rule out the causes that outrank it.

Safety posture

Property	Behavior
Mutations	None. No Put*/Update*/Delete*/Create*
Data-plane reads	None. No Scan, Query, GetItem, or BatchGetItem. Mean item size comes from DescribeTable metadata, not from reading items
API surface	sts:GetCallerIdentity, dynamodb:DescribeTable, cloudwatch:GetMetricData — the entire allowlist
IAM	No change to cloudformation/devops-agent-skill-policies.yaml. Those actions are already granted by the AIDevOpsAgentAccessPolicy managed policy (verified against the policy document)
NoData	Never coerced to zero; it blocks the nominal verdict rather than confirming it

Documented facts, and two remediations the docs contradict

references/dynamodb-facts.md carries the mechanics this diagnosis depends on, each cited to AWS documentation, and it records two fixes that circulate widely for DynamoDB latency and that the documentation warns against. Both appear in a mandatory "Not Recommended" report section:

• Setting a very low socket timeout (the often-repeated 50 ms) — AWS gives a 50 ms socket timeout as its own anti-example, warns that overly low timeouts cause client-induced availability issues, and says to prefer request hedging.
• Switching to strongly consistent reads "for latency" — they require twice the throughput, may have higher latency, and are unsupported on GSIs; eventually consistent reads are what AWS lists as a latency reduction.

It also corrects stale guidance that SuccessfulRequestLatency supports only Min/Max/Avg/SampleCount: Percentile is a valid statistic, which is what makes the p50-versus-p99 distinction available at all.

Scope, and why it does not collide with the data-health skill

This skill owns the request path. It does not sample items, identify hot partition keys, classify throttle level (partition/table/account/on-demand-max), recommend data-model or index changes, diagnose DNS/VPC connectivity, or size capacity. It complements dynamodb-data-health-inspection (#87 / PR #94), which owns data-level inspection and disclaims latency, and hands off to the shipped aws-health-events, service-quota-check, and aws-vpc-dns-investigation skills for their branches.

Testing

skill-eval — root "passed": true, overall 0.97 (A):

Test	Result
Audit / structure	100/100 (A) — 0 critical, 0 warning, 0 info
Functional	0.93 (A) — outcome 1.00, style 1.00, efficiency 1.00
Trigger	1.00 (A) — 8/8, 3 runs each
Cost	TRADEOFF (quality +0.24 for +3.9% tokens)

Two defects were found and fixed during the build, not by the eval:

• The description's non-goals, written as a keyword list ("does not … indexes … items"), over-triggered on a data-level question at a measured 0.67 across three runs, because the disclaimer supplied the very terms that attract activation. Rewriting it as a positive scope statement took that to 0.33 while the latency queries held at 1.0. (Single-run trigger eval is noisy enough to invert a verdict — an unrelated negative control flipped 1.0 → 0.0 across runs — so boundary claims rest on 3-run measurements.)
• LT-02's severity was undefined in its own headline branch (throttles present, latency curve flat).

Live AWS + DevOps Agent A/B (us-west-2), because a passing eval is not validation:

• Confirmed the load-bearing mechanics against live CloudWatch: the Percentile statistic returns p50/p99, the Operation dimension resolves per-operation, and a metric that published nothing returns as a distinguishable empty series rather than a zero.
• In a DevOps Agent space, the skill corrected a wrong no-skill diagnosis: on a table whose successful-request latency rose in lockstep with writes against a 1-WCU ceiling, the no-skill agent called it a "client-side artifact"; with the skill it correctly identified DynamoDB-side tail queueing with burst-capacity absorption and no throttling.
• Installed alongside dynamodb-data-health-inspection in the same space, a data-health prompt routed to that skill and not to this one — no cross-activation. Every latency run made zero data-plane calls.

License confirmation

• By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.