How to self-host Apache Airflow with Docker Compose

What is Apache Airflow?

Apache Airflow is the most widely used open source platform for orchestrating data pipelines. You define workflows as DAGs in Python, and Airflow schedules them, runs tasks on workers, retries failures, backfills history and shows every run in a web UI. It is an Apache Software Foundation project under the Apache-2.0 license. Airflow 3 replaced the old webserver with an API server and runs DAG parsing in a separate DAG processor.

Requirements

• A Linux server with Docker Engine and Docker Compose v2. The official docs ask for at least 4 GB of memory for Docker, ideally 8 GB. Give it 2 or more CPUs and around 10 GB of free disk.
• Basic Python, since DAGs are Python files.

Step 1: Prepare the server

This guide assumes Ubuntu 24.04 with Docker installed from the official Docker Engine guide. Create the project folder and the four directories the Compose file mounts:

mkdir -p ~/airflow && cd ~/airflow
mkdir -p ./dags ./logs ./plugins ./config

Step 2: Create the Docker Compose file

The official file is over 300 lines, pinned to a release and maintained by the Airflow team, so download it rather than retyping it:

curl -LfO 'https://airflow.apache.org/docs/apache-airflow/3.3.2/docker-compose.yaml'

It defines these services, all from the apache/airflow:3.3.2 image except the two databases:

• postgres (postgres:16) for Airflow metadata and redis (redis:7.2-bookworm) as the Celery broker;
• airflow-apiserver (web UI and REST API on 8080), airflow-scheduler, airflow-dag-processor, airflow-triggerer and airflow-worker, using the CeleryExecutor;
• airflow-init, a one-off job that migrates the database and creates the admin user, plus optional airflow-cli and flower profiles.

Two defaults are unsafe on a server: the admin login is airflow / airflow, and the API JWT secret falls back to a fixed string. Create .env with your own values:

echo "AIRFLOW_UID=$(id -u)" > .env
echo "_AIRFLOW_WWW_USER_USERNAME=admin" >> .env
echo "_AIRFLOW_WWW_USER_PASSWORD=CHANGE_ME" >> .env
echo "FERNET_KEY=$(openssl rand -base64 32 | tr '+/' '-_')" >> .env
echo "AIRFLOW__API_AUTH__JWT_SECRET=$(openssl rand -hex 32)" >> .env

Edit .env and replace CHANGE_ME with the output of openssl rand -hex 32. AIRFLOW_UID makes files in dags and logs owned by your user. The Fernet key encrypts connection passwords and variables in the database; keep it, because a new key cannot decrypt old secrets.

Finally, bind the UI to localhost and turn off the example DAGs:

sed -i 's/"8080:8080"/"127.0.0.1:8080:8080"/' docker-compose.yaml
sed -i "s/AIRFLOW__CORE__LOAD_EXAMPLES: 'true'/AIRFLOW__CORE__LOAD_EXAMPLES: 'false'/" docker-compose.yaml

Docker-published ports bypass ufw, so the localhost binding is what keeps the UI off the internet until HTTPS is ready.

Step 3: Start and open the app

Initialise the database and admin user, then start everything:

docker compose up airflow-init
docker compose up -d
docker compose ps

airflow-init should exit with code 0 and print the Airflow version. When all services are healthy, tunnel in with ssh -L 8080:127.0.0.1:8080 user@YOUR_SERVER_IP and open http://localhost:8080. Sign in with the username and password from .env.

Drop a Python file into ./dags and the DAG processor picks it up within a minute or two. New DAGs start paused; unpause one and trigger a run from the UI.

Step 4: Put it behind HTTPS

With Caddy on the host:

airflow.example.com {
    reverse_proxy 127.0.0.1:8080
}

Add AIRFLOW__API__BASE_URL=https://airflow.example.com to .env and run docker compose up -d so links and redirects use the public URL. Anyone with a privileged Airflow account can run code through DAGs and connections, so consider limiting the site to a VPN or known IP addresses.

Backups and upgrades

Keep your DAGs in Git. Back up the metadata database and .env, which holds the Fernet key:

docker compose exec -T postgres pg_dump -U airflow airflow > airflow-$(date +%F).sql

To upgrade, back up first and read the release notes. Then either download the Compose file for the new version (and re-apply the two sed edits) or set AIRFLOW_IMAGE_NAME in .env, and run:

docker compose pull
docker compose up airflow-init
docker compose up -d

airflow-init runs the database migration before the other services start.

Troubleshooting

• Containers restart with exit code 137: out of memory. Give the host 8 GB or lower the worker concurrency.
• Permission denied writing logs: AIRFLOW_UID is missing from .env, so files are owned by root. Set it and chown -R the folders.
• A DAG never appears: run docker compose run --rm airflow-cli airflow dags list-import-errors to see the Python error.
• "Invalid Fernet key" or decryption errors: FERNET_KEY changed after connections were saved. Restore the original key.

Next steps

Build a custom image with your Python dependencies instead of _PIP_ADDITIONAL_REQUIREMENTS, store connections in a secrets backend, ship task logs to S3, and move to the official Helm chart when you need more than one server.