Do it in SQL, in the warehouse. Two billion rows will not fit in pandas memory and moving them over the network is the slow part. Push filtering, joining, and aggregation down to the engine, which is built for it, and pull back only the aggregated result. Use pandas or Polars for the last mile once the data is small enough to sit comfortably in memory.
Why interviewers ask this
This is a practical judgment check about where computation should happen. Interviewers see plenty of candidates who reflexively pull data into a dataframe and then wonder why the job dies. They want to hear push the compute to the data, awareness of what actually costs time (network transfer and shuffles), and a sensible boundary for where the dataframe work begins.
How to structure your answer
- Answer directly: aggregate in the warehouse.
- Explain that transfer and memory, not compute, are the bottleneck.
- Say what belongs in SQL and what belongs in the dataframe.
- Mention partition pruning or incremental computation for cost control.
Example answer
SQL, without hesitating. Two billion rows is not going into a pandas dataframe on any machine I will be given, and even if it were, the transfer over the wire would dominate everything. The engine is designed to scan and aggregate at that scale with parallelism I do not have locally. So I push the filter, the join, and the group by into the warehouse and bring back the aggregate, which for a feature table is usually a few million rows keyed by entity and date, and that sits comfortably in memory. My rough boundary is that anything reducing rows belongs in SQL, and anything shaping a small result for modeling belongs in Python. I also care about cost, so I make sure the query prunes partitions on the date column instead of scanning full history, and if the feature is recomputed daily I build it incrementally, appending yesterday rather than recalculating three years every morning. On one project that change took a nightly job from about forty minutes to under three, and cut the bill by roughly the same factor.
Walking into this interview soon? GhostPilot listens to your live call, spots the question the moment it is asked, and puts a structured answer on your screen in real time. Try it on your next mock, or grab a $29 Session Pass, no subscription, for the real thing.
See how it worksFollow-up questions to expect
- How would you debug a query that suddenly got ten times slower?
- When would you reach for Spark instead of the warehouse?
- What would make you choose Polars over pandas for the last mile?
Related data scientist questions
Your interviewer will ask their own version of this. Paste your actual job description into the free Question Predictor and get the 20 questions that role is most likely to ask, with what each one is really probing.
Predict my questions