What causes a race condition, and how do you find one that only shows up in production?

A race condition happens when two threads or requests touch shared mutable state and the outcome depends on timing. The classic shape is read, modify, write without holding a lock across all three steps. To find one in production, look for state that is checked and then acted on, add correlated logging around that sequence, reproduce under real concurrency rather than in a unit test, and let database constraints catch the violation.

A race needs two things: shared mutable state, and two paths reaching it at once. Nearly every one I have debugged is the same shape, a check followed by an action with a gap in between. Read the balance, decide it is sufficient, write the new balance. Two requests interleave and the number is wrong. Finding them in production is mostly about accepting that you cannot click your way to a repro. I write a script that fires the same request a couple of hundred times concurrently, because the window might be two milliseconds wide. Then I log with a request id either side of the read and the write, so I can see the interleave in the timeline. For the fix I try to move the invariant into the database rather than the app: an update with a where clause on the expected value, or a unique constraint, so the database enforces it. Application level locks only work if every writer goes through your code, and eventually one will not.