fix: cache api-key auth lookups on the proxy hot path - #87
Merged
Conversation
Every proxied RPC request runs a DB query in ApiKeyAuth to validate the token before forwarding. Under a burst to a single stack this turns N concurrent requests into N concurrent DB checkouts, saturating the pool and producing DBConnection checkout timeouts (500s). Cache successful token lookups in an ETS table (60s TTL) so the hot path skips the DB. Measured: a 200-request burst to one token drops from 400 DB queries to 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
- application.ex: guard :ets.new against re-entrant start/2 (ArgumentError) - api_key_auth.ex: remove dead conn.path_info expression - tests: expired entry is ignored+re-checked; invalid tokens aren't cached Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every proxied RPC request runs a DB query in
ApiKeyAuth(router.ex:proxypipeline →api_key_auth.ex:32) to validate the token before forwarding. No cache.Under a burst to a single stack, N concurrent RPC calls become N concurrent DB checkouts. The pool saturates and requests get dropped from the checkout queue after 0.8–2.4s →
DBConnection.ConnectionError→ 500s.Observed in prod logs: 25 identical checkout-timeout errors, same query, one token, under load.
Why now: config unchanged in recent commits — this is new load (e2e burst), not a regression. Low traffic never saturated the pool; per-request DB does.
Fix
Cache successful token lookups in an ETS table (
:api_key_cache, 60s TTL) so the proxy hot path skips the DB. Invalid tokens are not cached.application.ex: create the ETS table at boot.api_key_auth.ex: ETS-backedcached_api_key/1.api_key_auth_test.exs: test that a warm cache authorizes even after the DB row is deleted, and rejects once evicted.Verification
Reproduced locally with an Ecto-telemetry query counter: a 200-request burst to one token = 400 DB queries without the cache, 0 with it.
mix test test/ethui_web/plugs/api_key_auth_test.exs→ 3 tests, 0 failures.Note (not in this PR)
config/prod.exs:21setspool: Ecto.Adapters.SQL.Sandbox— a verbatim copy of Phoenix's defaultconfig/test.exs. In its default:automode it behaves like a normal pool, so it is not the cause and does not limit connections. But it's a landmine (a test-only pool in prod) and emits the misleading "connections are shared" error text. Optional cleanup, left out here.Trade-off
ponytail:60s TTL means a revoked/deleted key keeps working up to 60s. Fine for stacks; tighten or invalidate-on-delete if that window matters.🤖 Generated with Claude Code