{"dataset":{"id":"00000000-0000-0000-0000-000000000059","slug":"airflow-best-practices","name":"Apache Airflow best practices","description":"Source-linked guidance on Apache Airflow fundamentals: DAGs and tasks, operators and sensors, scheduling, connections and variables, and production deployment. Scoped to current Apache Airflow docs.","visibility":"public","version":1,"documentCount":5,"publisher":"Koskamo editorial","license":"Original summaries; linked sources retain their own terms.","reviewedAt":"2026-10-04T00:00:00.000Z","readOnly":true,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},"total":5,"documents":[{"title":"Production deployment","sourceUrl":"https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/production-deployment.html","text":"Deploy production Airflow by replacing the default SQLite backend with PostgreSQL or MySQL via `[database] sql_alchemy_conn` to prevent data loss. Give Airflow's user CREATE/ALTER permissions, then run `airflow db migrate`; `airflow db upgrade` is deprecated in favor of it. For multi-node clusters, replace the default LocalExecutor with remote executors like Celery, Kubernetes, Amazon Batch, ECS, or Edge, ensuring every node holds the DAGs and role-appropriate configuration. Use the Dag Bundle mechanism, including GitDagBundle, for DAG synchronization with versioning. Restrict sensitive configuration — Fernet keys, JWT signing keys, database credentials — to only the components that need them. On disposable nodes, store logs on a distributed file system like S3 or GCS, or external services like Elasticsearch. Configure deployment-varying settings via `AIRFLOW__{SECTION}__{KEY}` environment variables. Scope: Apache Airflow. Reviewed October 4, 2026.","id":"airflow-best-practices:Production deployment","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},{"title":"Managing connections","sourceUrl":"https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html","text":"Airflow's Connection object stores credentials for external services, definable via environment variables, an external secrets backend such as HashiCorp Vault or AWS SSM Parameter Store, or the metadata database. Environment variables follow the AIRFLOW_CONN_{CONN_ID} convention — all uppercase with single underscores — and accept JSON or Airflow URI format values. Use the Connection class's as_json() property to generate JSON representations and to convert URI-format connections to JSON; get_uri() generates URIs, storing JSON structures that cannot be urlencoded losslessly under the __extra__ query param. Prefer get_uri() over manual URI building, since special characters like / in passwords break parsing unless encoded with quote_plus(). For database-stored connections, Airflow encrypts passwords with Fernet. Scope: Apache Airflow. Reviewed October 4, 2026.","id":"airflow-best-practices:Managing connections","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},{"title":"Scheduling with timetables","sourceUrl":"https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timetable.html","text":"For a Dag with a time-based schedule, the internal timetable drives scheduling and determines the data interval and logical date of each created run; cron expressions and timedelta objects are internally converted to timetables. Trigger timetables (CronTriggerTimetable, DeltaTriggerTimetable) do not include a data interval, so data_interval_start equals data_interval_end at trigger time; data interval timetables (CronDataIntervalTimetable, DeltaDataIntervalTimetable) set data_interval_end to the trigger time and data_interval_start to the interval start. Both kinds trigger runs at the same time, but run_id timestamps differ because run_id is based on logical_date. Switching an existing Dag from a trigger to a data interval timetable skips one scheduled run; the reverse does not. AssetOrTimeSchedule combines asset events with time-based schedules. Scope: Apache Airflow. Reviewed October 4, 2026.","id":"airflow-best-practices:Scheduling with timetables","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},{"title":"Operators and sensors","sourceUrl":"https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html","text":"An Operator is conceptually a template for a predefined Task, defined declaratively inside a Dag and chained with the `>>` notation. Core operators include BashOperator, which executes a bash command, and PythonOperator, which calls an arbitrary Python function — the `@task` decorator is the TaskFlow equivalent; community providers add operators such as EmailOperator, HttpOperator, and SQLExecuteQueryOperator. Parameters registered in `template_fields` undergo Jinja substitution just before `pre_execute`; wrap a string in `literal()` to exclude it. Never name a third-party operator field `params`, which is reserved for Dag serialization. Inside f-strings, double the Jinja braces (`{{{{ ds }}}}`) so templates survive interpolation. `pre_execute` and `post_execute` run around each task instance, unlike setup and teardown tasks spanning multiple task instances. Scope: Apache Airflow. Reviewed October 4, 2026.","id":"airflow-best-practices:Operators and sensors","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},{"title":"DAGs: defining workflows of tasks","sourceUrl":"https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html","text":"Use the `LatestOnlyOperator` to skip all tasks downstream of itself when a Dag run is not the latest: directly-downstream `task1` is skipped for all non-latest runs, independent `task2` runs in all scheduled periods, `task3` receives a cascaded skip from `task1` under the default `all_success` trigger rule, and `task4` is not skipped because its `trigger_rule` is `all_done`. Set `depends_on_past` to `True` so a task runs only if its previous Dag run succeeded; the first automated run still runs. Other trigger rules include `all_failed` (upstream tasks failed or upstream_failed), `one_success` (at least one upstream succeeded), and `always` (no dependencies). `removed` is a terminal state that counts toward `all_done`. Teardown tasks default to `all_done_setup_success`, requiring at least one upstream setup success. Scope: Apache Airflow. Reviewed October 4, 2026.","id":"airflow-best-practices:DAGs: defining workflows of tasks","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"}]}