localis
Waitlist

Describe a service in plain words, or type a CPT/HCPCS code.

5 min read

One Redis, two jobs, and a cache on a collision course with the queue

Every /cpt/*, /hcpcs/*, and /medicaid/* page on the public site is rendered HTML sitting behind PublicCache, a thin wrapper around Laravel's cache facade. That cache lives in the same Redis instance as Horizon's job queue, because standing up a second Redis for a ~10,900-code catalog didn't seem worth it. It still doesn't. But sharing the instance means the cache's memory habits are the queue's problem too, and we found one worth fixing before it became an incident instead of after.

noeviction means "stop," not "degrade"

Production Redis runs with maxmemory unset and the default eviction policy, noeviction. That's the right policy for a queue — you never want Redis quietly dropping a pending job to make room for a cache entry. But it means the cache inherits the same failure mode: once the instance is full, every write anywhere, cache or queue, starts erroring. A cache that grows without bound doesn't get slower or lossier on this setup. It takes the whole box down with it, ingest jobs included.

At the time we looked, that instance held about 440MB on a 1.9GB box. Two things were pushing the cache side of that number up, neither by design.

Growth driver one: warming the whole catalog

A daily scheduled command warmed the public cache with --all, rendering every code page in the catalog in one pass — roughly 435MB of HTML for ~10,900 codes. That's already most of the box on its own, and it ran daily whether or not real traffic justified re-rendering all of it.

The bigger problem was what happened to the previous day's 435MB. PublicCache doesn't track and delete per-page keys when content changes; it prefixes every key with a version integer and bumps the version on each release ingest:

private static function prefix(): string
{
    return 'pfspub:v'.self::version().':';
}

public static function flush(): int
{
    $next = self::version() + 1;
    Cache::forever(self::VERSION_KEY, $next);

    return $next;
}

Old keys under pfspub:v1:* become unreachable the moment the version moves to v2, and they expire on their own TTL rather than needing a bulk delete. That's the right design for correctness — instant invalidation, no tag support required, works identically on file/database/array/redis drivers. It's the wrong design paired with an unconditional daily --all warm: the new generation gets written in full before the old generation's TTL has necessarily expired, so for a stretch of every day Redis was holding two full copies of the catalog. Bounding the warm to --limit=2000 (with --all kept as an explicit manual option, not a default) cut the steady-state footprint without touching the invalidation model at all.

Growth driver two: an unbounded key space

The second driver wasn't the daily warm — it was Open Graph images. Code pages render a ~40KB PNG for social previews, cached per historical release variant so a shared link to last quarter's rate still showed last quarter's number. Crawlers hit those images at will, across every release variant that had ever existed, which is an unbounded key space by construction: it grows with every release forever, driven by traffic we don't control.

Two changes fixed this without giving up per-release accuracy for the cases that need it:

public const IMAGE_STORE = 'og';

public static function rememberImage(string $key, Closure $callback, ?int $ttlHours = null): string
{
    $ttl = now()->addHours($ttlHours ?? self::DEFAULT_TTL_HOURS);

    return ResilientCache::remember(self::prefix().$key, $ttl, $callback, self::IMAGE_STORE);
}

Image blobs now go to og, a store configured to the filesystem instead of Redis:

'og' => [
    'driver' => env('OG_CACHE_DRIVER', 'file'),
    'path' => storage_path('framework/cache/og'),
    'lock_path' => storage_path('framework/cache/og'),
],

And only the canonical (current-release) image variant gets cached at all; historical variants render on demand instead of being persisted forever. Disk isn't free either, so a weekly cache:clear on that store reclaims it — an acceptable tradeoff for images, where a cache miss costs one render instead of an availability problem.

The warm job we deleted entirely

The --limit=2000 fix shipped first. The better fix shipped a day later: delete the scheduled warm. CDN caching already absorbs anonymous traffic on these pages, so the only hits that reach PublicCache at all are CDN-bypassed ones — cookie-bearing sessions, CDN misses. Pre-warming ~2,000 pages a day was spending Redis memory on pages nobody behind the CDN had actually asked for yet, at traffic levels that don't justify speculative rendering. PublicCache::remember() already warms lazily on first real hit:

public static function remember(string $key, Closure $callback, ?int $ttlHours = null): mixed
{
    $ttl = now()->addHours($ttlHours ?? self::DEFAULT_TTL_HOURS);

    return ResilientCache::remember(self::prefix().$key, $ttl, $callback);
}

so the scheduled command wasn't buying steady-state hit rate, only extra memory pressure. Removing it was strictly a subtraction — no replacement logic needed.

The other side of sharing: failing open

Sharing Redis with the queue cuts both ways. A restart puts Redis into a few seconds of LOADING where it rejects every command, and without handling that, a cache read failure during those seconds turns into a 500 on every page that touches PublicCache — including pages that don't need to be fast, just need to not error. ResilientCache wraps every read/write in a catch and degrades to uncached instead:

public static function remember(string $key, mixed $ttl, Closure $callback, ?string $store = null): mixed
{
    try {
        $cached = Cache::store($store)->get($key);
    } catch (Throwable $e) {
        self::logWarning('read', $key, $e);

        return $callback();
    }
    // ...
}

A cache-store outage now costs a slower page, not a broken one.

The pattern

None of these were cache bugs in the usual sense — nothing returned stale or wrong data. The cache was working exactly as designed; it just shared a box with something that fails hard when memory runs out, and nobody had checked what "working as designed" cost in bytes against that ceiling. noeviction is the correct policy for a queue and a trap for anything sharing its Redis instance without a memory budget of its own. Version- prefix invalidation is worth keeping — it's the growth pattern layered on top (unconditional full warms, an unbounded per-variant key space) that needed bounding, not the cache design itself.

← All posts