feat: retry safe reads after database connection loss

[nipsufn]

Summary

Pop now retries generated reads when a database connection closes, allowing a later attempt to use a healthy pooled connection. Retries are always enabled for generated reads outside transactions and cannot be disabled. They are bounded to five total attempts with short, jittered backoff.

Raw SQL remains one-shot unless its caller marks the exact statement with RetryableRead(). Changing that statement invalidates the opt-in. Statements inside transactions are never retried.

The exported API adds RetryableRead(), SQLState, and IsConnectionClosed. SQLState extracts a reported SQLSTATE, while IsConnectionClosed recognizes common connection-loss SQLSTATEs and transport errors. IsConnectionClosed rejects error chains containing cancellation, deadline expiry, or unknown outcomes such as SQLSTATE 08007 and 40003.

When cancellation or a deadline stops a retry, the returned error preserves both the context error and the connection error that caused the retry. An unrelated terminal error is returned unchanged; exhaustion returns the final connection error. Retry outcomes are recorded as OpenTelemetry span events.

Package documentation describes this retry boundary. This also updates pgx/v5 from v5.9.1 to v5.10.0 to recognize pgconn.ErrConnClosed.

Validation

• go mod tidy -diff and gofmt -l .
• go test -count=1 ./...
• go test -race -count=1 ./...
• make test and go vet -tags sqlite ./...
• The full suite and targeted server-close regression pass with the race detector against a live CockroachDB instance.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 644bee23-e813-4026-92a7-bca3f3b31af5

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

🔵 Needs a closer look

It changes core query execution semantics (automatic retries + error wrapping/telemetry) and should receive final human review for correctness and edge-case safety.

Pull request overview

Adds first-class, bounded retry behavior for read-only queries that fail due to a lost database connection, aiming to transparently recover by re-running the read on a healthy pooled connection while preserving cancellation/deadline semantics and emitting OpenTelemetry events.

Changes:

• Introduces a retry loop for generated read paths (and opt-in retry for raw SQL via RetryableRead()), bounded to 5 attempts with jittered backoff.
• Adds exported helpers SQLState and IsConnectionClosed to classify retry-eligible connection-loss errors.
• Updates internal call sites and documentation to reflect retry boundaries; bumps pgx/v5 to v5.10.0.

File summaries

File	Description
query.go	Tracks retry eligibility on Query, adds RetryableRead() and raw-SQL tracking fields.
query_retry.go	Implements the retry loop and telemetry event emission for closed-connection read retries.
query_retry_test.go	Unit tests covering retry behavior, error preservation, and telemetry attributes.
query_retry_integration_test.go	Integration test validating retry behavior against a server-side session close (CockroachDB).
connection_closed.go	Adds IsConnectionClosed and SQLState helpers for retry classification.
connection_closed_test.go	Tests SQLSTATE extraction and connection-closed classification behavior.
finders.go	Wraps generated read executors (First/Last/All/Exists/CountByField) with retry logic; preserves retryability for eager loads.
connection.go	Uses IsConnectionClosed when deciding whether to ignore rollback errors due to lost connections.
doc.go	Documents the retry boundary and raw SQL opt-in behavior.
dialect_sqlite.go	Marks a raw read used by TruncateAll as retryable.
dialect_mysql.go	Marks a raw read used by TruncateAll as retryable.
dialect_cockroach.go	Marks raw reads in TruncateAll and AfterOpen as retryable.
go.mod	Bumps pgx/v5 to v5.10.0 and promotes go.opentelemetry.io/otel to a direct dependency.
go.sum	Updates checksums for the pgx/v5 version bump.

Review details

• Files reviewed: 13/14 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[gaultier]

I think the idea is sound. Just a few minor things.

[alnr]

I think this is probably a good thing.

However, why doesn't the database driver retry for such errors? I'd like to see some research on this.

[nipsufn]

database/sql already retries errors matching driver.ErrBadConn, with up to three attempts. Its driver contract forbids returning that sentinel when the server may have executed the statement, because retrying could duplicate an operation. See the retry loop and driver contract.

The pinned drivers follow that boundary. pgx v5.10.0 translates a closed connection or a failure marked SafeToRetry to ErrBadConn; errors encountered while preloading or streaming rows are returned without that translation. MySQL's markBadConn translates only errBadConnNoWrite, which represents a failure before anything was sent.

The CockroachDB probe used a pooled *sql.DB capped at one connection. Cancelling its session during SELECT pg_sleep(5) produced unexpected EOF, with both errors.Is(err, driver.ErrBadConn) and pgconn.SafeToRetry(err) false. The next read through the same pool succeeded with a different session ID. The pool recovered, but the interrupted read still failed for its caller.

Pop can retry the complete read at its query boundary: generated reads and explicitly opted-in raw statements outside transactions. Raw SQL remains one-shot otherwise, since a query can contain writes or functions with side effects. This also covers failures encountered while consuming rows, after database/sql's retry loop has returned its Rows.