How would you add a caching layer in front of a read heavy endpoint without serving badly stale data?

Pick the shortest TTL that still moves the needle, then invalidate on write by deleting or overwriting the key in the same path that mutates the row. Key by every input that changes the result, including tenant and a schema version. Add single flight locking so a miss does not stampede the database. Accept explicit staleness where it is harmless, and track hit rate so you know it is working.

Before adding anything, I check whether the query can just be made fast, because a cache in front of a bad query hides the problem until the cache misses at the worst moment. Assuming it genuinely is read heavy, I go cache aside in Redis. The key includes the tenant id, the actual query parameters, and a version prefix I can bump on deploy so a changed response shape never gets served from old entries. TTL is the safety net rather than the mechanism: I invalidate explicitly on write, in the same code path that updates the row, so the window of staleness is milliseconds in the normal case. The failure mode I plan for is the stampede, where a hot key expires and two hundred requests all miss and hit the database at once, so I use a single flight lock and let the losers wait for the winner. Then I export hit rate as a metric, because a cache that dropped from 95 percent to 10 after a refactor goes unnoticed until the database falls over.