feat: CMS-1202 send image to queue to process instead of handling it sync

[baoc75]

Issue

Resolves CMS-1202 https://linear.app/cmsmax/issue/CMS-1202/queue-image-optimization-instead-of-processing-in-http-requests

Overview

https://www.loom.com/share/69df07ef764a4f2aa5812026064533b5

On a cache miss, image transforms no longer run synchronously on the request (blocking PHP-FPM workers with Imagick decode/encode). Instead, the request gets an immediate 302 redirect to the original (short max-age=60 TTL) while a queued job (ProcessImageTransform, Redis) transforms and caches in the background. The next request for the same URL hits the populated cache. Feature-flagged (IMAGE_TRANSFORM_ASYNC_ENABLED, default on) for instant rollback to the old synchronous behavior.

Also closes an abuse gap this uncovered: width/height/quality previously accepted any value, so a client could enumerate distinct params to flood the cache/queue with unbounded variants, bypassing job dedup. Added config whitelists (sizes, qualities, allowed_formats) that clamp requests to the nearest allowed value before the cache/dedup key is computed, plus an IP+path throttle on the dispatch. The throttle gates only the enqueue — an over-limit miss still gets the redirect, so a cache miss never hands the client a broken image (no 429 on the miss path).

Permanent-failure handling is classified: only a genuine decode/encode fault writes the failure sentinel (request path stops re-dispatching); a transient infra fault (S3/network) is left un-sentineled so the next request can re-dispatch and recover.

Requires: a running queue worker (php artisan queue:work redis) in Laravel Cloud, and CACHE_STORE=redis (the dedup lock + cache flags need a cache store; a fresh app ships no cache_locks migration).

Timeframes / TTLs

The feature juggles several independent clocks — here's what each one is and where to tune it:

What	Duration	Config knob	Governs
Temporary miss redirect (CDN)	60s	IMAGE_TRANSFORM_PENDING_MAX_AGE (async.pending_redirect_max_age)	How long the CDN caches the 302 → original after a miss before re-checking origin. Kept short so the CDN picks up the optimized HIT quickly once the job finishes.
Dispatch dedup lock	5 min (300s)	IMAGE_TRANSFORM_JOB_UNIQUE_FOR (async.unique_for)	ShouldBeUnique window — at most one queued job per identical transform while it processes.
Failure sentinel (server)	1h (3 600s)	IMAGE_TRANSFORM_FAILED_LIFETIME (async.failed_lifetime)	After a permanent (decode/encode) failure, how long the request path serves the original and skips re-dispatching. Expires → next request retries.
Permanent redirect + success cache (CDN)	30 days, immutable (2 592 000s)	IMAGE_TRANSFORM_HEADER_CACHE_CONTROL (headers.Cache-Control)	CDN cache for optimized 200s and for the long-cache 302 → original used on the frame-guard and permanent-failure redirects.
Cache flag + frame-count memo (server)	30 days (2 592 000s)	IMAGE_TRANSFORM_CACHE_LIFETIME (cache.lifetime)	Server-side "this transform is cached" flag TTL, and the Imagick frame-count memo (keyed by disk+path+mtime).
Dispatch rate limit	2 req / 60s per IP+path	IMAGE_TRANSFORM_RATE_LIMIT_MAX_REQUESTS / _DECAY_SECONDS	Throttles only the enqueue on a miss (never the redirect) — queue-flood guard.
Job retries	2 attempts	ProcessImageTransform::$tries	Retries a failing transform once before failed() runs.

Two clocks on a permanent failure

A decode failure sets two independent timers that don't match on purpose:

• Server sentinel = 1h → after 1h the server will re-dispatch (in case the source got fixed).
• CDN redirect = 30 days, immutable → a CDN edge won't re-ask origin for 30 days, so it keeps serving the (valid, unoptimized) original at the edge regardless of the 1h sentinel.

Net: a broken image degrades gracefully to its original for up to 30 days at the edge; the 1h sentinel mainly governs direct-to-origin retries. A re-uploaded/fixed source recovers at origin after 1h, but cached edges finish their 30 days first.

Observability note: there is no Sentry in this project — report() and job failures log to storage/logs/laravel.log (+ failed_jobs if that table exists). Permanent decode failures are currently pull-only (grep logs / query failed_jobs), surfaced by the "image keeps 302ing, never optimizes" symptom.

Testing instructions

1. composer install, copy .env.example → .env, set S3 credentials, QUEUE_CONNECTION=redis, CACHE_STORE=redis.
2. Run the suite: vendor/bin/pest — 41 tests should pass (AsyncTransformTest, AllowedSizesAndFormatsTest, AnimatedWebpTransformTest).
3. Manual check:

   • Start a worker: php artisan queue:work redis.

   • Request a not-yet-cached transform URL → expect 302 to the original with Cache-Control: max-age=60. For example: http://image-server.test/production/width=1280,quality=80,format=webp/pyg2bhwsdu01ky1odnkcg/sunsetgrovecopingkit_3.jpg http://image-server.test/production/width=600,quality=80,format=webp/pyg2bhwsdu01ky1odnkcg/sunsetgrovecopingkit_3.jpg

   • http://image-server.test/production/width=800,quality=80,format=webp/pyg2bhwsdu01ky1odnkcg/sunsetgrovecopingkit_3.jpg

   • Repeat the same request after the worker processes the job → expect 200 with X-Cache: HIT and the optimized image.

4. Rollback check: set IMAGE_TRANSFORM_ASYNC_ENABLED=false → misses should transform synchronously again (old behavior).