Site Reliability Engineer Interview Question

What is split brain in a clustered system, and how does quorum prevent it?

What the interviewer is probing, how to structure your answer, and a spoken example you can adapt.

Quick answer

Split brain is when a network partition leaves two halves of a cluster each believing they are the leader, so both accept writes and the data diverges. Quorum prevents it by requiring a majority, N/2 plus 1, to elect a leader or commit a write, and only one side of a partition can hold a majority. Use odd node counts, fencing tokens, and never let a minority partition serve writes.

Why interviewers ask this

This tests whether you understand why consensus systems are built the way they are, which matters the moment you operate etcd, ZooKeeper, Consul, or any replicated database. Interviewers listen for the majority arithmetic, for why an even node count is worse than the odd number below it, and for fencing, since a demoted leader can still be holding a disk or a lock.

How to structure your answer

  • Describe the partition scenario concretely with two halves.
  • Explain the majority rule and why only one side can have it.
  • Point out that even node counts add cost without adding tolerance.
  • Cover fencing so a stale leader cannot keep writing.
  • Note the availability cost: the minority side must refuse writes.

Example answer

Spoken example, first person

Say you have five nodes and a switch fails so you end up with a group of three and a group of two that cannot see each other. Without quorum, both groups notice the leader is gone, both elect their own, and now you have two sources of truth accepting conflicting writes, which is genuinely painful to reconcile afterward. Quorum means you need three of five to elect or commit, so the group of two simply cannot make progress and has to reject writes. That is the trade: you buy consistency by giving up availability on the minority side. Two things I always check in practice. One, use odd numbers, because six nodes still need four for quorum and tolerate the same two failures as five while costing more. Two, fencing. The old leader may not know it was demoted, so you need monotonic fencing tokens or storage level locks, otherwise a paused process wakes up and writes to a volume it no longer owns.

Walking into this interview soon? GhostPilot listens to your live call, spots the question the moment it is asked, and puts a structured answer on your screen in real time. Try it on your next mock, or grab a $29 Session Pass, no subscription, for the real thing.

See how it works

Follow-up questions to expect

  • Why does a five node etcd cluster tolerate only two failures?
  • How would you recover a cluster that has lost quorum permanently?
  • What is a fencing token and where is it enforced?

Related site reliability engineer questions

Your interviewer will ask their own version of this. Paste your actual job description into the free Question Predictor and get the 20 questions that role is most likely to ask, with what each one is really probing.

Predict my questions

Rehearse the hard questions before they are asked

Practise with a live copilot, then walk in ready. A $29 Session Pass gets you through the interview with no subscription and no lock-in.

Get GhostPilot