# How to self-host Apache Airflow with Docker Compose

> Apache Airflow Docker Compose setup with the official 3.3.2 file: AIRFLOW_UID, Fernet and JWT secrets, airflow-init, the UI on port 8080, HTTPS and backups.

## Key facts

| Fact | Value |
|---|---|
| App | Apache Airflow (https://appsgit.com/apps/apache-airflow) |
| Difficulty | intermediate |
| Time | about 25 minutes |
| Requirements | 2 vCPU / 4 GB RAM minimum (8 GB recommended); Docker + Docker Compose v2; 10 GB+ free disk; A domain name (optional, for HTTPS) |
| Last updated | 2026-10-06 |

## What is Apache Airflow?

Apache Airflow is the most widely used open source platform for orchestrating data pipelines. You define workflows as DAGs in Python, and Airflow schedules them, runs tasks on workers, retries failures, backfills history and shows every run in a web UI. It is an Apache Software Foundation project under the Apache-2.0 license. Airflow 3 replaced the old webserver with an API server and runs DAG parsing in a separate DAG processor.

## Requirements

- A Linux server with Docker Engine and Docker Compose v2. The official docs ask for at least 4 GB of memory for Docker, ideally 8 GB. Give it 2 or more CPUs and around 10 GB of free disk.
- Basic Python, since DAGs are Python files.

## Step 1: Prepare the server

This guide assumes Ubuntu 24.04 with Docker installed from the [official Docker Engine guide](https://docs.docker.com/engine/install/ubuntu/). Create the project folder and the four directories the Compose file mounts:

```bash
mkdir -p ~/airflow && cd ~/airflow
mkdir -p ./dags ./logs ./plugins ./config
```

## Step 2: Create the Docker Compose file

The official file is over 300 lines, pinned to a release and maintained by the Airflow team, so download it rather than retyping it:

```bash
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/3.3.2/docker-compose.yaml'
```

It defines these services, all from the `apache/airflow:3.3.2` image except the two databases:

- `postgres` (postgres:16) for Airflow metadata and `redis` (redis:7.2-bookworm) as the Celery broker;
- `airflow-apiserver` (web UI and REST API on 8080), `airflow-scheduler`, `airflow-dag-processor`, `airflow-triggerer` and `airflow-worker`, using the CeleryExecutor;
- `airflow-init`, a one-off job that migrates the database and creates the admin user, plus optional `airflow-cli` and `flower` profiles.

Two defaults are unsafe on a server: the admin login is `airflow` / `airflow`, and the API JWT secret falls back to a fixed string. Create `.env` with your own values:

```bash
echo "AIRFLOW_UID=$(id -u)" > .env
echo "_AIRFLOW_WWW_USER_USERNAME=admin" >> .env
echo "_AIRFLOW_WWW_USER_PASSWORD=CHANGE_ME" >> .env
echo "FERNET_KEY=$(openssl rand -base64 32 | tr '+/' '-_')" >> .env
echo "AIRFLOW__API_AUTH__JWT_SECRET=$(openssl rand -hex 32)" >> .env
```

Edit `.env` and replace `CHANGE_ME` with the output of `openssl rand -hex 32`. `AIRFLOW_UID` makes files in `dags` and `logs` owned by your user. The Fernet key encrypts connection passwords and variables in the database; keep it, because a new key cannot decrypt old secrets.

Finally, bind the UI to localhost and turn off the example DAGs:

```bash
sed -i 's/"8080:8080"/"127.0.0.1:8080:8080"/' docker-compose.yaml
sed -i "s/AIRFLOW__CORE__LOAD_EXAMPLES: 'true'/AIRFLOW__CORE__LOAD_EXAMPLES: 'false'/" docker-compose.yaml
```

Docker-published ports bypass `ufw`, so the localhost binding is what keeps the UI off the internet until HTTPS is ready.

## Step 3: Start and open the app

Initialise the database and admin user, then start everything:

```bash
docker compose up airflow-init
docker compose up -d
docker compose ps
```

`airflow-init` should exit with code 0 and print the Airflow version. When all services are healthy, tunnel in with `ssh -L 8080:127.0.0.1:8080 user@YOUR_SERVER_IP` and open `http://localhost:8080`. Sign in with the username and password from `.env`.

Drop a Python file into `./dags` and the DAG processor picks it up within a minute or two. New DAGs start paused; unpause one and trigger a run from the UI.

## Step 4: Put it behind HTTPS

With [Caddy](https://caddyserver.com/docs/) on the host:

```caddyfile
airflow.example.com {
    reverse_proxy 127.0.0.1:8080
}
```

Add `AIRFLOW__API__BASE_URL=https://airflow.example.com` to `.env` and run `docker compose up -d` so links and redirects use the public URL. Anyone with a privileged Airflow account can run code through DAGs and connections, so consider limiting the site to a VPN or known IP addresses.

## Backups and upgrades

Keep your DAGs in Git. Back up the metadata database and `.env`, which holds the Fernet key:

```bash
docker compose exec -T postgres pg_dump -U airflow airflow > airflow-$(date +%F).sql
```

To upgrade, back up first and read the release notes. Then either download the Compose file for the new version (and re-apply the two `sed` edits) or set `AIRFLOW_IMAGE_NAME` in `.env`, and run:

```bash
docker compose pull
docker compose up airflow-init
docker compose up -d
```

`airflow-init` runs the database migration before the other services start.

## Troubleshooting

- **Containers restart with exit code 137:** out of memory. Give the host 8 GB or lower the worker concurrency.
- **Permission denied writing logs:** `AIRFLOW_UID` is missing from `.env`, so files are owned by root. Set it and `chown -R` the folders.
- **A DAG never appears:** run `docker compose run --rm airflow-cli airflow dags list-import-errors` to see the Python error.
- **"Invalid Fernet key" or decryption errors:** `FERNET_KEY` changed after connections were saved. Restore the original key.

## Next steps

Build a custom image with your Python dependencies instead of `_PIP_ADDITIONAL_REQUIREMENTS`, store connections in a secrets backend, ship task logs to S3, and move to the official Helm chart when you need more than one server.

## FAQ

### What is the default Airflow login in Docker Compose?

The official Compose file creates an admin account with username airflow and password airflow. Override both with _AIRFLOW_WWW_USER_USERNAME and _AIRFLOW_WWW_USER_PASSWORD in .env before running airflow-init, as this guide does.

### What port does Airflow use?

The Airflow 3 API server, which also serves the web UI, listens on port 8080. Flower, the optional Celery monitoring tool, uses port 5555 when you enable its profile.

### How much memory does Airflow need in Docker?

The Airflow docs say to give the Docker Engine at least 4 GB of memory and ideally 8 GB. Below that, services restart in a loop. The airflow-init container prints a warning if the host has less.

### Is the official Airflow Docker Compose file production ready?

No. The Airflow project describes it as a quick start for learning and exploration with no production security guarantees, and recommends Kubernetes with the official Helm chart for production. With the hardening in this guide it is workable for a small single-server setup you take responsibility for.

### Is Apache Airflow free?

Yes. Airflow is a free, open source Apache Software Foundation project under the Apache-2.0 license. Managed services such as Astronomer, Amazon MWAA and Google Cloud Composer charge for hosting it.

### Airflow vs n8n?

Airflow is a Python-first orchestrator for data pipelines: you write DAGs as code, and it handles scheduling, retries, backfills and dependencies. n8n is a visual workflow tool for connecting SaaS apps and APIs. Choose Airflow for data engineering, n8n for business automation.

Prefer not to do it yourself? [appsgit installation help](https://appsgit.com/services/install) installs it on your server for a fixed quote.

---

Canonical page: https://appsgit.com/guides/apache-airflow
Source: appsgit (https://appsgit.com), the app store for github. Data from the GitHub API, refreshed nightly.
Machine access: JSON API https://appsgit.com/api/v1/apps (OpenAPI: https://appsgit.com/openapi.json), MCP server https://mcp.appsgit.com/mcp, full index https://appsgit.com/llms-full.txt.