How would you choose between Airflow, Dagster, and a managed scheduler for a new platform?

Choose on team skills, integration needs, and how much you care about data awareness. Airflow has the widest ecosystem and hiring pool and treats work as tasks. Dagster models assets and their lineage directly, which improves testing and observability of what data exists. Managed schedulers reduce operational burden but constrain you. For a small team, the lowest operational cost option that your people can debug wins.

I would set criteria first: who maintains it, what it needs to integrate with, and whether the team needs lineage and testing built in. Airflow is the safe default because the ecosystem and the hiring pool are enormous, and almost every data engineer can debug it at 3am, which is worth more than feature lists. Its model is task centric, so it knows a task ran but not what data now exists. Dagster inverts that with software defined assets, so the system knows this table is stale and it is derived from those two, which makes local testing and backfill reasoning genuinely nicer. If I were starting fresh with a team that valued testing, I would lean Dagster. If I inherited existing Airflow DAGs and a team that knows them, migrating is rarely worth the disruption. And unless we have platform engineers to spare, I would take the managed version of whichever we pick, because running the scheduler is not where a small team creates value.