Apache Airflow

Airflow models work as DAGs of tasks written in Python, scheduled by interval or event, and executed by a fleet of workers. Its moat is the provider ecosystem — mature operators exist for practically every warehouse, cloud service, and data tool a tracking pipeline touches — plus a decade of operational lore, hiring pool, and managed offerings (Cloud Composer, MWAA, Astronomer). Airflow 3, released in 2025, modernized the foundations with DAG versioning, a rewritten UI, and a task execution interface that decouples workers from the metadata database; the 3.x line has kept moving since, adding asset partitioning in 3.2 and stateful tasks plus multi-language support in 3.3 (July 2026).

Operationally, a deployment is several cooperating services: a scheduler, a DAG processor that parses your DAG files and serializes them into the metadata database (usually PostgreSQL or MySQL), an API server that serves both the REST API and the UI, and — depending on executor — a pool of Celery or Kubernetes workers plus an optional triggerer that runs deferred tasks in an asyncio event loop. In Airflow 3 the DAG processor is deliberately separated from the scheduler so the scheduler never executes DAG-author code, and task code can no longer touch the metadata database directly — workers talk to the API server instead. That’s a real security and isolation win, but it also means custom operators written against Airflow 2 internals (direct DB sessions, airflow.models imports) need rework toward the airflow.sdk namespace.

Migrating from Airflow 2 is a project, not a version bump: the docs prescribe upgrading to 2.7+ first, linting DAGs with Ruff’s AIR rules (ruff check dags/ --select AIR301), installing the standard provider for operators that used to ship in core, and switching startup from webserver to api-server plus a standalone dag-processor. SubDAGs, the SequentialExecutor, SLAs (replaced by Deadline Alerts), and legacy context variables like execution_date are gone.

The classic gotchas are well documented by the project itself: top-level code in DAG files runs on every parse cycle, so expensive imports or API calls there quietly hammer the scheduler; tasks must be idempotent because retries and backfills re-run them; XCom is for small messages, not data — anything sizable belongs in object storage; and Variable.get() at top level hits the database where a Jinja template would not. The task-centric model is also worth internalizing: assets and data-aware scheduling have narrowed the gap, but asset-first lineage and partitioned backfills as core primitives remain the pitch of newer orchestrators. For steady ELT — load events, run dbt, kick off syncs — Airflow remains the safe, boring choice in the best sense.

Use it when you want the most battle-tested orchestrator, need operators for a sprawling set of systems, or your platform team already runs it. Look past it when you want asset-level lineage and partitioned backfills as built-ins, or a small team would rather not operate a scheduler, metadata database, DAG processor, and workers at all.