Data Engineer Interview Question

Is exactly once delivery actually possible in Kafka, and how would you achieve it in practice?

What the interviewer is probing, how to structure your answer, and a spoken example you can adapt.

Quick answer

Exactly once delivery over a network is not possible, but exactly once processing is, within limits. Kafka gives you an idempotent producer plus transactions, so a read process write cycle within Kafka commits offsets and output atomically. End to end, you still need the sink to cooperate, either through a transactional write or an idempotent upsert on a stable key. In practice most teams get at least once plus idempotent sinks.

Why interviewers ask this

This is a depth check on distributed systems reasoning. Weak answers say enable.idempotence equals true and stop there. Interviewers want to hear the distinction between delivery and processing, the scope limit of Kafka transactions (they only cover Kafka topics and consumer offsets), and the pragmatic reality that idempotent sinks are what actually make a pipeline correct end to end.

How to structure your answer

  • Separate exactly once delivery from exactly once processing.
  • Explain the idempotent producer and what duplicate it removes.
  • Describe transactions covering output plus offset commit atomically.
  • State the boundary: external sinks are not inside the transaction.
  • Land on at least once plus idempotent writes as the practical answer.

Example answer

Spoken example, first person

Strictly, no. You cannot guarantee exactly once delivery across an unreliable network, because an acknowledgment can always be lost and the sender has to choose between resending or dropping. What Kafka does give you is exactly once processing semantics within Kafka. The idempotent producer adds a producer id and sequence number so a retried send does not create a duplicate on the partition, and transactions let you commit your output records and your consumer offsets atomically, so a read process write topology does not double count on failure. The limit is scope. The moment you write to Postgres or S3, that write is outside the transaction. So what I actually build is at least once delivery with an idempotent sink, usually a merge on an event id or a natural key. On one clickstream pipeline we deduplicated on event_id with a seven day window, which was cheaper and easier to reason about than trying to make every hop transactional.

Walking into this interview soon? GhostPilot listens to your live call, spots the question the moment it is asked, and puts a structured answer on your screen in real time. Try it on your next mock, or grab a $29 Session Pass, no subscription, for the real thing.

See how it works

Follow-up questions to expect

  • What does transactional.id do and why does it matter on restart?
  • How would you deduplicate a stream without unbounded state?
  • What is the performance cost of enabling transactions?

Related data engineer questions

Your interviewer will ask their own version of this. Paste your actual job description into the free Question Predictor and get the 20 questions that role is most likely to ask, with what each one is really probing.

Predict my questions

Rehearse the hard questions before they are asked

Practise with a live copilot, then walk in ready. A $29 Session Pass gets you through the interview with no subscription and no lock-in.

Get GhostPilot