Retry bank payment import timeouts every 20 minutes - #171
Merged
Conversation
Lengthen BillingPaymentsProcessorJob timeout backoff so overnight BAS outages get more recovery time, and keep that handler on the subclass so it wins over BaseJob's short Exception retries. Co-authored-by: Thibaud Guillaume-Gentil <thibaud@thibaud.gg>
thibaudgg
marked this pull request as ready for review
September 1, 2026 06:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overnight BAS HTTP timeouts (CSA Admin production, incident #667) still exhaust
Scheduled::BillingPaymentsProcessorJobafter 5 attempts. On 1 Sep 2026 for tenantlumieredeschamps, a sample was queued at 04:49 and ran at 05:05 Zurich (~16 minutes) duringBilling::BAS#autologin.Why 20 minutes: the existing 15-minute wait already applies (that 16-minute hop matches 15 minutes plus jitter, not BaseJob’s polynomial ~3s/18s/83s curve). Four intervals still only cover about an hour from the 4am recurring run, and the 5th attempt still timed out. Spacing attempts by 20 minutes (~80 minutes total) gives BAS more time to recover before we discard; the next daily import still picks up any missed payments.
Why the subclass handler must win:
Scheduled::BaseJobdeclaresretry_on Exception, wait: :polynomially_longer, attempts: 5. ActiveJob/rescue_fromsearches last-defined first, so the more specificNet::OpenTimeout/Net::ReadTimeouthandler has to stay on the subclass. That is what selects the long wait instead of BaseJob’s short retries. Other exceptions keep the BaseJob behavior.