Skip to content

Missed counter samples advance lasttime without oldvalue #815

Description

@somethingwithproof

After a counter, derive, or absolute data source misses a poll, the default poller and daemon paths can carry oldvalue forward while persisting a fresh lasttime. The next valid rate can then use a mismatched interval and create a false alert.

Persist oldvalue and lasttime as one causal pair in both execution paths. Numeric samples advance both; missing, U, nan, empty, stale, and duplicate samples advance neither. A backward clock step discards the rate for that cycle but re-anchors both fields to the current sample, allowing the next in-order poll to recover.

Counter and derive baselines older than the effective RRD heartbeat are treated as unknown; the safe heartbeat floor is twice the greater of the RRD step and configured poller interval. Absolute rates also reject stale intervals, while gauges remain valid.

Unknown rates remain eligible for the next poll but preserve existing alert state instead of producing a false restoral. An operator warning is emitted only on the transition into the unavailable state and respects medium poller verbosity. CDEF, lower/upper, and nested-expression transforms preserve unknown values. Expression sources without their own threshold row resolve from the numeric value in the current poll; unavailable current values fail closed. Unresolved or unknown counter maxima use the conservative reset guard.

Status-only polling batches use prepared parameters, perform the same deleted-threshold cleanup as full sample batches, and fail closed for missing identifiers. Daemon persistence failures are reported.

Linux Docker coverage includes multi-interval rates with zero, blank, null, unknown, unresolved query, and explicit maxima; reset and wrap branches; invalid RRD steps; cadence and heartbeat mismatches; stale, duplicate, out-of-order, and backward-clock samples; unknown-state and transition-log behavior; expression and CDEF propagation; sibling expression sources without threshold rows; daemon bound parameters and failures; status-only cleanup; and pure poller batch rows.

Overlap audit: PR #803 changes poller scheduling but not sample-pair persistence or rate calculations. PR #798 changes configured CDEF threshold conversion and avoids double application; it does not preserve unknown samples or timestamps. PR #788 corrects RPN operator semantics; it does not implement this state or persistence behavior. No open PR addresses this issue.

Metadata

Metadata

Labels

bugconfirmedIssue that has been confirmed by the dev teamdaemonRelated to thold daemon

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions