Skip to content

feat(genai): add GenAI semantic metrics UI and backend proxy#403

Open
pristley wants to merge 2 commits into
demml:mainfrom
pristley:feat/genai-semantic-metrics-ui
Open

feat(genai): add GenAI semantic metrics UI and backend proxy#403
pristley wants to merge 2 commits into
demml:mainfrom
pristley:feat/genai-semantic-metrics-ui

Conversation

@pristley

@pristley pristley commented Jul 3, 2026

Copy link
Copy Markdown

Summary

Add GenAI semantic metrics support: backend proxy routes for Scouter GenAI endpoints and a Svelte UI for service/trace/span GenAI metrics (timeseries, model usage, token/cost charts), plus a TypeScript API client, strict types, tests, and documentation.

Related: closes #381


Files changed (by component)

  • Backend routes & server

    • crates/opsml_server/src/core/genai_metrics/route.rs
    • crates/opsml_server/src/core/scouter/genai/route.rs
    • crates/opsml_server/src/core/scouter/client.rs (Routes enum additions)
    • crates/opsml_server/src/core/router.rs (router mount)
    • crates/opsml_server/src/core/mod.rs
  • Frontend components

    • crates/opsml_server/opsml_ui/src/lib/components/AgentServiceDashboard/GenAIMetricsNav.svelte
    • crates/opsml_server/opsml_ui/src/lib/components/AgentServiceDashboard/GenAITimeseries.svelte
    • crates/opsml_server/opsml_ui/src/lib/components/TraceDetail/GenAIMetricsTab.svelte
    • crates/opsml_server/opsml_ui/src/lib/components/card/agent/observability/GenAiChartCard.svelte
    • crates/opsml_server/opsml_ui/src/lib/components/card/agent/observability/charts.ts
  • Types & API client

    • crates/opsml_server/opsml_ui/src/lib/types/genai.ts
    • crates/opsml_server/opsml_ui/src/lib/api/genai.ts
    • crates/opsml_server/opsml_ui/src/lib/api/__tests__/genai.test.ts
  • Tests (frontend)

    • crates/opsml_server/opsml_ui/src/lib/components/AgentServiceDashboard/__tests__/GenAITimeseries.test.ts
    • crates/opsml_server/opsml_ui/src/lib/components/TraceDetail/__tests__/GenAIMetricsTab.test.ts
  • Documentation

    • GENAI_METRICS.md

How to test

Prereqs:

Run backend checks:

cargo check -p opsml_server
# or via mise
mise run format
mise run lints

Run frontend tests:

cd crates/opsml_server/opsml_ui
pnpm install
pnpm test

Manual verification:

# full dev stack (recommended)
mise run dev:both
# or frontend only
cd crates/opsml_server/opsml_ui
pnpm run dev
  • Open the Agent Service dashboard -> select "GenAI Metrics".
  • Confirm: Timeseries charts for tokens, latency, model usage, and cost render and respond to time-range changes.
  • In a Trace detail page, open the GenAI tab and confirm per-span metrics load.

API smoke tests (curl):

curl -b "<auth-cookie>" "http://localhost:8080/api/traces/<TRACE_ID>/genai/aggregate"
curl -b "<auth-cookie>" "http://localhost:8080/api/spans/<SPAN_ID>/genai/metrics"
curl -b "<auth-cookie>" "http://localhost:8080/api/services/<SERVICE_ID>/genai/timeseries"

Screenshots / GIFs

  • Timeseries dashboard and Trace GenAI tab updated UI.
  • (Attach screenshots or GIFs here if available.)

Breaking changes

  • None — additions only.

Notes

  • Backend proxies Scouter requests and inserts AuditContext for auditing.
  • Frontend client includes short TTL in-memory caching to reduce repeated fetches.
  • Tests mock the internal API client and Chart.js rendering where appropriate.

Closes: #381

pristley added 2 commits July 3, 2026 15:59
## Summary

Add GenAI semantic metrics support: backend proxy routes for Scouter GenAI endpoints and a Svelte UI for service/trace/span GenAI metrics (timeseries, model usage, token/cost charts), plus a TypeScript API client, strict types, tests, and documentation.

Related: closes demml#381

---

## Files changed (by component)

- Backend routes & server
  - `crates/opsml_server/src/core/genai_metrics/route.rs`
  - `crates/opsml_server/src/core/scouter/genai/route.rs`
  - `crates/opsml_server/src/core/scouter/client.rs` (Routes enum additions)
  - `crates/opsml_server/src/core/router.rs` (router mount)
  - `crates/opsml_server/src/core/mod.rs`

- Frontend components
  - `crates/opsml_server/opsml_ui/src/lib/components/AgentServiceDashboard/GenAIMetricsNav.svelte`
  - `crates/opsml_server/opsml_ui/src/lib/components/AgentServiceDashboard/GenAITimeseries.svelte`
  - `crates/opsml_server/opsml_ui/src/lib/components/TraceDetail/GenAIMetricsTab.svelte`
  - `crates/opsml_server/opsml_ui/src/lib/components/card/agent/observability/GenAiChartCard.svelte`
  - `crates/opsml_server/opsml_ui/src/lib/components/card/agent/observability/charts.ts`

- Types & API client
  - `crates/opsml_server/opsml_ui/src/lib/types/genai.ts`
  - `crates/opsml_server/opsml_ui/src/lib/api/genai.ts`
  - `crates/opsml_server/opsml_ui/src/lib/api/__tests__/genai.test.ts`

- Tests (frontend)
  - `crates/opsml_server/opsml_ui/src/lib/components/AgentServiceDashboard/__tests__/GenAITimeseries.test.ts`
  - `crates/opsml_server/opsml_ui/src/lib/components/TraceDetail/__tests__/GenAIMetricsTab.test.ts`

- Documentation
  - `GENAI_METRICS.md`

## How to test

Prereqs:
- Rust toolchain (cargo) installed via https://rustup.rs
- Node + `pnpm` for the UI

Run backend checks:
```bash
cargo check -p opsml_server
# or via mise
mise run format
mise run lints
```

Run frontend tests:
```bash
cd crates/opsml_server/opsml_ui
pnpm install
pnpm test
```

Manual verification:
```bash
# full dev stack (recommended)
mise run dev:both
# or frontend only
cd crates/opsml_server/opsml_ui
pnpm run dev
```
- Open the Agent Service dashboard -> select "GenAI Metrics".
- Confirm Timeseries charts (Tokens, Latency, Model Usage, Cost) render and respond to range changes.
- On a Trace detail page, open the GenAI tab and confirm per-span metrics load.

API smoke tests (curl):
```bash
curl -b "<auth-cookie>" "http://localhost:8080/api/traces/<TRACE_ID>/genai/aggregate"
curl -b "<auth-cookie>" "http://localhost:8080/api/spans/<SPAN_ID>/genai/metrics"
curl -b "<auth-cookie>" "http://localhost:8080/api/services/<SERVICE_ID>/genai/timeseries"
```

## Screenshots / GIFs

- Timeseries dashboard and Trace GenAI tab updated UI.
- (Attach screenshots or GIFs here if available.)

## Breaking changes

- None — additions only.

## Notes

- Backend proxies Scouter requests and inserts `AuditContext` for auditing.
- Frontend client includes short TTL in-memory caching to reduce repeated fetches.
- Tests mock the internal API client and Chart.js rendering where appropriate.

Closes: demml#381
@pristley pristley requested a review from thorrester as a code owner July 3, 2026 16:17
@pristley

Copy link
Copy Markdown
Author

@thorrester I really loved your codebase and i did my good to contribute can you have a look and give me feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Epic] GenAI Semantic Metrics UI

1 participant