feat(genai): add GenAI semantic metrics UI and backend proxy#403
Open
pristley wants to merge 2 commits into
Open
feat(genai): add GenAI semantic metrics UI and backend proxy#403pristley wants to merge 2 commits into
pristley wants to merge 2 commits into
Conversation
## Summary Add GenAI semantic metrics support: backend proxy routes for Scouter GenAI endpoints and a Svelte UI for service/trace/span GenAI metrics (timeseries, model usage, token/cost charts), plus a TypeScript API client, strict types, tests, and documentation. Related: closes demml#381 --- ## Files changed (by component) - Backend routes & server - `crates/opsml_server/src/core/genai_metrics/route.rs` - `crates/opsml_server/src/core/scouter/genai/route.rs` - `crates/opsml_server/src/core/scouter/client.rs` (Routes enum additions) - `crates/opsml_server/src/core/router.rs` (router mount) - `crates/opsml_server/src/core/mod.rs` - Frontend components - `crates/opsml_server/opsml_ui/src/lib/components/AgentServiceDashboard/GenAIMetricsNav.svelte` - `crates/opsml_server/opsml_ui/src/lib/components/AgentServiceDashboard/GenAITimeseries.svelte` - `crates/opsml_server/opsml_ui/src/lib/components/TraceDetail/GenAIMetricsTab.svelte` - `crates/opsml_server/opsml_ui/src/lib/components/card/agent/observability/GenAiChartCard.svelte` - `crates/opsml_server/opsml_ui/src/lib/components/card/agent/observability/charts.ts` - Types & API client - `crates/opsml_server/opsml_ui/src/lib/types/genai.ts` - `crates/opsml_server/opsml_ui/src/lib/api/genai.ts` - `crates/opsml_server/opsml_ui/src/lib/api/__tests__/genai.test.ts` - Tests (frontend) - `crates/opsml_server/opsml_ui/src/lib/components/AgentServiceDashboard/__tests__/GenAITimeseries.test.ts` - `crates/opsml_server/opsml_ui/src/lib/components/TraceDetail/__tests__/GenAIMetricsTab.test.ts` - Documentation - `GENAI_METRICS.md` ## How to test Prereqs: - Rust toolchain (cargo) installed via https://rustup.rs - Node + `pnpm` for the UI Run backend checks: ```bash cargo check -p opsml_server # or via mise mise run format mise run lints ``` Run frontend tests: ```bash cd crates/opsml_server/opsml_ui pnpm install pnpm test ``` Manual verification: ```bash # full dev stack (recommended) mise run dev:both # or frontend only cd crates/opsml_server/opsml_ui pnpm run dev ``` - Open the Agent Service dashboard -> select "GenAI Metrics". - Confirm Timeseries charts (Tokens, Latency, Model Usage, Cost) render and respond to range changes. - On a Trace detail page, open the GenAI tab and confirm per-span metrics load. API smoke tests (curl): ```bash curl -b "<auth-cookie>" "http://localhost:8080/api/traces/<TRACE_ID>/genai/aggregate" curl -b "<auth-cookie>" "http://localhost:8080/api/spans/<SPAN_ID>/genai/metrics" curl -b "<auth-cookie>" "http://localhost:8080/api/services/<SERVICE_ID>/genai/timeseries" ``` ## Screenshots / GIFs - Timeseries dashboard and Trace GenAI tab updated UI. - (Attach screenshots or GIFs here if available.) ## Breaking changes - None — additions only. ## Notes - Backend proxies Scouter requests and inserts `AuditContext` for auditing. - Frontend client includes short TTL in-memory caching to reduce repeated fetches. - Tests mock the internal API client and Chart.js rendering where appropriate. Closes: demml#381
Author
|
@thorrester I really loved your codebase and i did my good to contribute can you have a look and give me feedback |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add GenAI semantic metrics support: backend proxy routes for Scouter GenAI endpoints and a Svelte UI for service/trace/span GenAI metrics (timeseries, model usage, token/cost charts), plus a TypeScript API client, strict types, tests, and documentation.
Related: closes #381
Files changed (by component)
Backend routes & server
crates/opsml_server/src/core/genai_metrics/route.rscrates/opsml_server/src/core/scouter/genai/route.rscrates/opsml_server/src/core/scouter/client.rs(Routes enum additions)crates/opsml_server/src/core/router.rs(router mount)crates/opsml_server/src/core/mod.rsFrontend components
crates/opsml_server/opsml_ui/src/lib/components/AgentServiceDashboard/GenAIMetricsNav.sveltecrates/opsml_server/opsml_ui/src/lib/components/AgentServiceDashboard/GenAITimeseries.sveltecrates/opsml_server/opsml_ui/src/lib/components/TraceDetail/GenAIMetricsTab.sveltecrates/opsml_server/opsml_ui/src/lib/components/card/agent/observability/GenAiChartCard.sveltecrates/opsml_server/opsml_ui/src/lib/components/card/agent/observability/charts.tsTypes & API client
crates/opsml_server/opsml_ui/src/lib/types/genai.tscrates/opsml_server/opsml_ui/src/lib/api/genai.tscrates/opsml_server/opsml_ui/src/lib/api/__tests__/genai.test.tsTests (frontend)
crates/opsml_server/opsml_ui/src/lib/components/AgentServiceDashboard/__tests__/GenAITimeseries.test.tscrates/opsml_server/opsml_ui/src/lib/components/TraceDetail/__tests__/GenAIMetricsTab.test.tsDocumentation
GENAI_METRICS.mdHow to test
Prereqs:
pnpmfor the UIRun backend checks:
cargo check -p opsml_server # or via mise mise run format mise run lintsRun frontend tests:
Manual verification:
API smoke tests (curl):
Screenshots / GIFs
Breaking changes
Notes
AuditContextfor auditing.Closes: #381