Site Reliability Engineer Interview Question

Your primary database fails over at 3am and writes start erroring. What are your first moves?

What the interviewer is probing, how to structure your answer, and a spoken example you can adapt.

Quick answer

Confirm the new primary is actually promoted and accepting writes, then check whether applications reconnected. Stale connection pools and cached DNS are the usual reason errors continue after a healthy failover. Verify replication lag and whether any committed writes were lost against your RPO. Restore write traffic, rule out split brain by making sure the old primary is fenced, then take the postmortem question of why failover was not transparent.

Why interviewers ask this

Interviewers want to see that you do not stop at the database looks fine. The application layer is where most failover incidents actually live, through connection pools that hold dead sockets, DNS TTLs, and drivers that never re resolve. They are also checking whether you think about data loss and fencing rather than only availability.

How to structure your answer

  • Verify promotion and write acceptance on the new primary directly.
  • Move to the client layer: pools, DNS caching, driver behavior.
  • Quantify data loss and lag against the stated RPO.
  • Fence the old primary to rule out double writes.
  • Restore traffic, then capture the transparency gap for the postmortem.

Example answer

Spoken example, first person

Step one is confirming reality: connect to the new primary directly and check it is out of recovery and accepting writes. If the database is healthy and applications are still erroring, the problem is on our side, and in my experience that is where it usually is. Connection pools hold sockets to an address that no longer serves writes, or the JVM cached the DNS entry forever, so the fix is a rolling restart or a pool with proper validation. We had one where a driver honored a 30 second TTL but the pool never revalidated idle connections, so recovery took twenty minutes longer than it should have. Then I check replication lag at the moment of promotion to see if we lost committed writes, because that changes who I need to tell. I also want confirmation the old primary is fenced and cannot accept a write if it comes back. Once writes are flowing, the postmortem question is why this was not automatic.

Walking into this interview soon? GhostPilot listens to your live call, spots the question the moment it is asked, and puts a structured answer on your screen in real time. Try it on your next mock, or grab a $29 Session Pass, no subscription, for the real thing.

See how it works

Follow-up questions to expect

  • How would you make failover transparent to the application next time?
  • How do you detect that writes were lost during promotion?
  • What is the risk of automatic failover, and when would you avoid it?

Related site reliability engineer questions

Your interviewer will ask their own version of this. Paste your actual job description into the free Question Predictor and get the 20 questions that role is most likely to ask, with what each one is really probing.

Predict my questions

Rehearse the hard questions before they are asked

Practise with a live copilot, then walk in ready. A $29 Session Pass gets you through the interview with no subscription and no lock-in.

Get GhostPilot