Skip to content

feat(agent-observability): dashboard accuracy fixes + custom model pricing overrides - #12

Merged
Robert0622 merged 1 commit into
mainfrom
feat/agent-observability-dashboard-and-pricing
Sep 15, 2026
Merged

Robert0622 merged 1 commit into
mainfrom
feat/agent-observability-dashboard-and-pricing

Conversation

@alberthaolihai

Copy link
Copy Markdown
Collaborator

What changed

Two commits, both validated against real multi-agent CodeBuddy session data (not just unit tests):

  1. fix: correct SendMessage matcher casing in devflow auto-dispatch hook — the PreToolUse hook for devflow-auto-dispatch.py was registered under matcher send_message, but real CodeBuddy CLI tool calls use SendMessage. The casing mismatch meant this hook never fired on any host, so auto_dispatch/fallback_to_main stats stayed 0 regardless of devflow.defaults.yaml's auto_dispatch.enabled: true. Verified end-to-end on a real medium devflow run after the fix: the Auto-Dispatch health panel showed 5 auto_dispatch / 0 fallback_to_main, matching the 5 real *_completed routing-table transitions in that run.

  2. feat: fix dashboard timeline/dispatch accuracy, add custom model pricing overrides:

    • Timeline events now carry the agent that actually ran them (the field was computed correctly upstream but silently dropped when building timeline segments), so sub-agent dispatch is visible in the UI instead of collapsing into "main".
    • The dispatch panel no longer inflates counts by treating inbox report echoes as fresh dispatches, and no longer shows team-lead (a CodeBuddy team-infra alias for main) as if main had dispatched to a real sub-agent.
    • Daily session count no longer drops sessions that only submitted a prompt with zero tool/usage events.
    • New: users can supply per-model $/token rates via .codebuddy/agent-observability/pricing.overrides.json (or AOBS_PRICING_OVERRIDES_PATH), merged field-by-field on top of the built-in config/pricing.json — missing/invalid override data silently degrades to the built-in table so a bad config file can never break the hook. The dashboard's "建议关注" panel now flags models seen in usage events that still have zero pricing coverage. Delivered via a real medium-sized devflow run (6-role pipeline: architect/developer/code-reviewer/test-engineer/knowledge-engineer/leader), independently re-verified rather than trusting the team's own self-reported acceptance.
    • Documents the raw_response race condition investigation in quickstart.md: three fix attempts (two bounded-retry budgets, one deferred-retry-queue approach) were tried and reverted after real end-to-end verification exposed problems unit tests didn't catch — including a pre-existing shared byte-offset cursor bug between tool-context lookups and usage-event scanning. Left documented as a known, unfixed, low-impact (Bash-only, failure-detection-only) limitation rather than shipping an unverified fix.

Scope

  • Classic
  • Portable
  • Shared behavior contract
  • Installer / CLI
  • Documentation only

Compatibility

Affects the Classic agent-observability skill (.codebuddy/skills/agent-observability/), synced to the generated .claude/.cursor host bundles via build-classic-hosts.py --write. No migration needed: the pricing-overrides mechanism is opt-in (default behavior is byte-identical when no override file is present, verified via before/after snapshot diff), and the dashboard fixes only correct previously-inaccurate aggregation — no schema/config changes required from existing users.

Validation

bash scripts/validate.sh        # passed
bash scripts/smoke-install.sh   # passed (16/16)
bash scripts/scan-secrets.sh    # not run — this helper only supports Linux x86_64, blocked on macOS;
                                 # manually grepped the diff for common secret patterns instead (clean)
python3 scripts/build-classic-hosts.py --check   # OK, bundles match generator
python3 -m pytest .codebuddy/skills/agent-observability/tests/   # 121 passed, 6 pre-existing
                                                                   # unrelated failures (test_cls_sink.py
                                                                   # credential/env deps, one test_collector.py
                                                                   # self-referential flake) — unchanged
                                                                   # baseline, not introduced by this PR

Safety

  • I did not include credentials, internal endpoints, production data, or organization-specific infrastructure.
  • I preserved user files and unrelated changes.
  • I updated generated Classic host bundles when applicable.
  • I documented third-party content and licensing changes when applicable.

🤖 Generated with Claude Code

@alberthaolihai alberthaolihai changed the title fix(agent-observability): dashboard accuracy fixes + custom model pricing overrides fet(agent-observability): dashboard accuracy fixes + custom model pricing overrides Sep 15, 2026
@alberthaolihai alberthaolihai changed the title fet(agent-observability): dashboard accuracy fixes + custom model pricing overrides feat(agent-observability): dashboard accuracy fixes + custom model pricing overrides Sep 15, 2026
@alberthaolihai
alberthaolihai force-pushed the feat/agent-observability-dashboard-and-pricing branch from 1089873 to e63c41c Compare September 15, 2026 06:04
…low runs

Adds a Classic-only, opt-in skill that hooks into SessionStart /
UserPromptSubmit / PreToolUse / PostToolUse / Stop to record tool
calls, token usage, and cost into a local metrics log, plus a script
that aggregates it (together with devflow's workflow-state.json)
into a static dashboard: per-day cost/tokens, session and tool-call
timelines, devflow stage progress, Skill/Rule hit rates, and
Auto-Dispatch health.

Also includes:
- custom per-model pricing overrides for models missing from the
  built-in price table
- accuracy fixes for the dashboard's turn/duration aggregation,
  dispatch attribution, session counting, and devflow stage
  classification (medium/large runs no longer misread as SOLO)
- a fix for build-classic-hosts.py's generated-bundle check
  incorrectly flagging __pycache__/.pyc as unmanaged extras
- cost-by-model chart fixes: long model names no longer get cut off
  by their bar, the chart gets a fairer share of its row's width,
  and the Token composition chart next to it is stretched to fill
  its row's actual height instead of leaving a gap underneath

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@alberthaolihai
alberthaolihai force-pushed the feat/agent-observability-dashboard-and-pricing branch from a2b3cbd to 79c1175 Compare September 15, 2026 09:44
@Robert0622
Robert0622 merged commit 0d7ef79 into main Sep 15, 2026
4 checks passed
@Robert0622
Robert0622 deleted the feat/agent-observability-dashboard-and-pricing branch September 15, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants