Cardinality is the number of unique label combinations for a metric, and each combination is a separate time series held in memory. Putting user ids, request ids, email addresses, or raw URL paths into labels can create millions of series and take a Prometheus instance down. Keep labels bounded and low arity, template dynamic path segments, push per request detail into logs or traces, and enforce limits with relabeling and sample caps.
Why interviewers ask this
Cardinality explosions are one of the most common ways monitoring itself becomes the outage, which is a special kind of bad. The interviewer is checking whether you know where the boundary sits between metrics, logs, and traces, and whether you have operated a time series database rather than only queried one.
How to structure your answer
- Define cardinality as unique label sets, each costing a live series.
- List the classic offenders that get added as labels.
- Give the rule for what belongs in a label versus a log field.
- Describe the guardrails: relabel drops, limits, and review of new metrics.
Example answer
Every unique combination of label values is its own time series with its own memory footprint and its own chunk on disk. So the moment someone adds user_id or a raw path as a label, you go from a few hundred series to millions. We had a Prometheus instance OOM repeatedly and it turned out one team had added a full URL including query string as a label on their request counter. The rule I use is that a label value has to come from a small, closed set you could write on a whiteboard: method, status class, endpoint template, region. Anything unbounded goes into a log line or a trace span attribute, where the storage model is designed for it. Practically I enforce it with metric_relabel_configs to drop known bad labels at scrape time, a per target sample limit so one service cannot take down the whole instance, and a quick review whenever a new metric shows up in a pull request.
Walking into this interview soon? GhostPilot listens to your live call, spots the question the moment it is asked, and puts a structured answer on your screen in real time. Try it on your next mock, or grab a $29 Session Pass, no subscription, for the real thing.
See how it worksFollow-up questions to expect
- How would you find which metric is driving your series count?
- When is it right to move a metric into a logs based aggregation?
- What does a sample limit do when a target exceeds it?
Related site reliability engineer questions
Your interviewer will ask their own version of this. Paste your actual job description into the free Question Predictor and get the 20 questions that role is most likely to ask, with what each one is really probing.
Predict my questions