Database query details p95 query scans the full trace window and times out

[constantins2001]

Description

Opening a database statement in Services > Database loads the Slowest 5% of Queries tile by calculating p95 across the selected trace window. On a high-volume trace source with a 24-hour window, the generated query times out after 60 seconds.

This is reproducible with the default OpenTelemetry-style trace schema, where Duration is numeric and database statements are read from SpanAttributes.

Environment

• HyperDX 2.35.0
• Self-hosted ClickStack
• Trace source using the standard Duration UInt64 and SpanAttributes Map(...) fields

Steps to reproduce

1. Open Services > Database.
2. Select a 24-hour time range on a high-volume trace source.
3. Open one of the database statement rows.
4. Observe the Slowest 5% of Queries tile in the detail drawer.

Generated query

Sanitized version of the query shown by HyperDX:

SELECT
  quantile(0.95)(
    toFloat64OrDefault(toString(Duration))
  ) AS p95_duration_ns,
  p95_duration_ns / 1e6 AS p95
FROM traces
WHERE
  Timestamp >= fromUnixTimestamp64Milli(...)
  AND Timestamp <= fromUnixTimestamp64Milli(...)
  AND coalesce(
    nullIf(SpanAttributes['db.query.text'], ''),
    nullIf(SpanAttributes['db.statement'], '')
  ) IN ('GET /resource/:id')

The request fails with:

Timeout exceeded: elapsed 60099 ms, maximum: 60000 ms:
While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread).

EXPLAIN indexes = 1 shows timestamp and primary-key pruning, but the generated database-statement predicate does not select the available map-value skipping index. The query therefore evaluates the coalesced map lookup across the selected trace slice.

Duration is already UInt64, so converting it through String and then Float64 also appears unnecessary.

Relevant code

The behavior appears to come from these paths:

• ServiceDashboardDbQuerySidePanel.tsx, which adds the selected database statement as a filter.
• serviceDashboard.ts, which builds the default coalesced statement expression.
• ServiceDashboardSlowestEventsTile.tsx, which calculates the p95 threshold.

Expected behavior

The database query detail drawer should remain usable for common 24-hour ranges without requiring a higher timeout or a full scan of the selected trace slice.

Possible directions include preserving the native numeric type for Duration, allowing the database statement expression to point to an indexed or materialized column, or using a projection/aggregate path for the repeated percentile calculation.