PULP-2385: Route pulp-content reads to RDS replica - #1468
Conversation
Assisted-by: OpenAI
Reviewer's GuideRoutes pulp-content database reads to an optionally configured stage RDS replica through a Django database router, while keeping writes, migrations, app-status reads, and pull-through-related updates on the primary via ContextVar protection and a pulpcore image patch. Sequence diagram for primary protection during pull-throughsequenceDiagram
participant Content as pulp-content
participant Router as ContentReplicaRouter
participant Primary as RDS primary
participant Replica as RDS read replica
Content->>Router: db_for_read(model)
Router->>Replica: return replica
Replica-->>Content: content read
Content->>Router: use_primary_database()
Content->>Router: db_for_read(model)
Router->>Primary: return default
Primary-->>Content: pull-through write-related read
Content->>Router: db_for_write(model)
Router->>Primary: return default
Primary-->>Content: write completed
Content->>Router: ContextVar reset
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've reviewed your changes and they look great!
Sourcery assessment
Needs a human reviewer. If the replica endpoint is unavailable, misconfigured, or too far behind, content reads could fail or return stale data across the service, and pull-through behavior could be disrupted. Reverting restores primary reads, but requests that failed or served stale content during the rollout cannot be undone.
| @@ -0,0 +1,33 @@ | |||
| """Database routing for the content service.""" | |||
There was a problem hiding this comment.
I am wondering if it's not worth it to add a fallback to the primary, in case the replica is unavailable?
Which considering we are on aws, etc etc.. might be over-engineering it
Something like
`import time
from django.db.utils import Error as DjangoDBError
REPLICA_HEALTH_CHECK_TTL = 5 # seconds to trust a health probe before re-checking
_replica_down_until = 0.0
def _replica_is_healthy():
global _replica_down_until
now = time.monotonic()
if now < _replica_down_until:
return False
try:
connections["replica"].ensure_connection()
return True
except DjangoDBError:
_replica_down_until = now + REPLICA_HEALTH_CHECK_TTL
return False
class ContentReplicaRouter:
def db_for_read(self, model, **hints):
if (
"replica" not in settings.DATABASES
or _use_primary.get()
or model._meta.model_name == "appstatus"
):
return "default"
if not _replica_is_healthy():
return "default"
return "replica"`
|
I recommend keeping the current fail-fast behavior when a configured replica is unavailable. Router-level health probing is not reliable for async ORM connections, and transparent retries are unsafe after response streaming or pull-through mutations begin. Silent fallback could overload the primary during a replica incident. Let’s validate stage behavior first and track an observable, bounded circuit-breaker design separately if primary fallback becomes an availability requirement. Assisted-by: OpenAI |
Keep upstream ETag patch 0064 and renumber PULP-2385 routing patch to 0065.\n\nAssisted-by: OpenAI
2 participants
Summary
Route
pulp-contentreads to the stage RDS read replica while keeping writes and pull-through operations on the primary database.Changes
Testing
Jira
PULP-2385
Labels: ai-assisted
Assisted-by: OpenAI
Summary by Sourcery
Route pulp-content reads through the stage RDS replica while preserving primary-database behavior for writes and content pull-through operations.
New Features:
Bug Fixes:
Enhancements:
Build:
Deployment:
Tests: