Skip to content

fix: use default pool in prod, not the test SQL sandbox - #88

Merged
ZePedroResende merged 3 commits into
mainfrom
fix/prod-remove-sandbox-pool
Jul 25, 2026
Merged

fix: use default pool in prod, not the test SQL sandbox#88
ZePedroResende merged 3 commits into
mainfrom
fix/prod-remove-sandbox-pool

Conversation

@ZePedroResende

@ZePedroResende ZePedroResende commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Root cause (reproduced + confirmed on the running prod node)

config/prod.exs set Ethui.Repo to pool: Ecto.Adapters.SQL.Sandbox — a verbatim copy of config/test.exs.

The sandbox pool is DBConnection.Ownership. In its default :auto mode it assigns a connection to a process on first DB use and only releases it when that process terminates (manager.ex:203 + the :DOWN handler). Unlike the default DBConnection.ConnectionPool, it does not check the connection back in after each query.

So every long-lived process that queries the DB (the telemetry poller running publish_active_stacks_count, per-stack servers, …) permanently holds one of the pool_size connections. After ~pool_size such processes the pool is exhausted for good.

Reproduced locally

2 long-lived processes that each run ONE query then stay alive, pool_size: 2:

sandbox pool  => fresh query: :checkout_timeout   # connections held per-process
default pool  => fresh query: :ok                 # connections returned per-query

Confirmed on prod

A single Ethui.Accounts.get_api_key_by_token(token) from a fresh iex — zero request load — times out with could not checkout. The token is valid and cached (ETS returns the %ApiKey{}), so cache-served requests still 200'd, masking the dead pool.

Matches the Phoenix template

mix phx.new --database sqlite3 puts pool: Sandbox only in test.exs; prod uses the default pool. This change makes prod match that.

Change

  • Remove pool: Ecto.Adapters.SQL.Sandbox → prod uses the default DBConnection.ConnectionPool, which returns connections after each operation.
  • Drop the dead pool_size in prod.exs (it was schedulers*2 but runtime.exs always overrode it to POOL_SIZE/10). runtime.exs is now the single source. Keep default_transaction_mode: :immediate (recommended for SQLite).

Verify after deploy

Ethui.Repo.config() |> Keyword.take([:pool, :pool_size])   # no :pool key now
Ethui.Accounts.get_api_key_by_token("DST39Q12ceoJxGigeyq4DEkcwgE2g6oqY")  # returns fast, no timeout

SQLite pool_size note (not changed here)

SQLite is single-writer; large pools can cause database is locked under heavy concurrent writes. Current prod pool_size is 10 (runtime, env-tunable via POOL_SIZE); the sqlite template default is 5. Left at 10 — light write load, and WAL + busy_timeout: 2000 + :immediate mitigate. Lower POOL_SIZE if database is locked shows up.

Note

This is the actual fix for the checkout-timeout 500s. The earlier api-key ETS cache (already merged) is a valid optimization but only masked the symptom.

🤖 Generated with Claude Code

prod.exs configured Ethui.Repo with pool: Ecto.Adapters.SQL.Sandbox — a
verbatim copy of config/test.exs. The sandbox pool is DBConnection.Ownership;
in its default :auto mode it assigns a connection to each process on first use
and only releases it when that process terminates (ownership manager
handle_info(:DOWN), db_connection manager.ex:203). It does NOT check the
connection back in after each query, unlike DBConnection.ConnectionPool.

So every long-lived process that touches the DB (telemetry poller, per-stack
servers, ...) permanently holds one of the pool_size connections. After ~10
such processes the pool is exhausted for good: even a single manual
Repo query from a fresh iex session times out with a checkout error, which is
what we observed in production.

Removing the pool: line falls back to the default DBConnection.ConnectionPool,
which returns connections after each operation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 25, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
stacks Error Error Jul 25, 2026 10:29pm

Request Review

The prod.exs pool_size (schedulers*2) was dead — runtime.exs always
overrides it (prod ran pool_size: 10). Drop it so runtime.exs is the single
source; keep default_transaction_mode: :immediate (recommended for SQLite).

Verified with a reproduction: 2 long-lived processes that each run one query
exhaust the sandbox pool permanently (connections held per-process until the
process dies), while the default DBConnection pool returns them after each
query. Matches the Phoenix --database sqlite3 template, which uses the default
pool in prod and the sandbox only in test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ZePedroResende
ZePedroResende merged commit b2165e5 into main Jul 25, 2026
2 of 3 checks passed
@ZePedroResende
ZePedroResende deleted the fix/prod-remove-sandbox-pool branch July 25, 2026 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants