Dagster

Dagster’s core idea is that pipelines exist to produce data assets — tables, files, models — so the orchestrator should model those assets directly rather than just the tasks that build them. Assets are declared in Python with their upstream dependencies, giving you a lineage graph, freshness checks, and partition-aware backfills as first-class features instead of conventions. The dbt integration is the standout for analytics stacks: every dbt model appears as an asset in the same graph as the ingestion and activation jobs around it.

Assets sit on top of a lower-level layer of ops and jobs, and both coexist: you can still write task-shaped jobs where that fits, and partition assets by date or key so backfills operate on exactly the slices that are missing. Automation comes in three flavors — cron-style schedules, sensors that poll for external events, and declarative automation where you attach conditions to assets (“materialize when upstream updates”) and let the framework work out what to run. Asset checks attach data-quality tests directly to the asset they validate, so failures show up in the same lineage graph as the data itself (per docs.dagster.io).

Self-hosting is genuinely production-viable but has moving parts: a webserver (which can run multiple replicas), a daemon that operates schedules, sensors, and the run queue — and which runs as a single instance only, so plan around that — plus one or more code location servers that serve your pipeline definitions. Storage defaults to SQLite and should be swapped for Postgres via the dagster-postgres integration for production; Docker and Kubernetes-via-Helm are the documented deployment targets. Dagster+ removes that operational work in two modes: serverless (fully hosted) or hybrid, where the control plane is hosted but compute runs in your own infrastructure — the usual pick when data can’t leave your network.

Compared with Airflow it trades ecosystem breadth for a stronger data model: typed inputs and outputs, local testability without a running scheduler, and a UI oriented around what data exists and how stale it is. The asset mindset is a real shift, and teams porting task-shaped DAGs sometimes fight it before it pays off. The integration surface is broad — the docs list over 60 libraries covering warehouses (Snowflake, BigQuery, Databricks, DuckDB, ClickHouse), ingestion (Fivetran, Airbyte, dlt, Sling), quality tools (Great Expectations, Soda, Pandera), and BI (Tableau, Power BI, Looker, Sigma) — though it remains smaller than Airflow’s operator ecosystem.

On cost, note that Dagster+ bills a usage component in credits, and a credit is one asset materialization or op executed: fine-grained assets and frequent schedules multiply credits quickly, so model your materialization volume before committing to a paid plan.

Use it when your pipeline is warehouse-centric — event loads, dbt models, reverse-ETL syncs — and you want lineage, partitions, and backfills handled by the orchestrator. Look past it when you depend on Airflow’s much larger operator ecosystem and hiring pool, or your scheduling needs are simple enough that cron and a dbt Cloud job already cover them.