Keywords: Apache Airflow - AWS - How to - Other
I am still learning about Airflow. My objective is to deploy Airflow on Kubernetes (I am using AWS EKS) with
CeleryExecutor and using Helm charts. As a start, I am primarily focused on deploying the example dags.
I have installed the chart like:
helm install airflow bitnami/airflow \
--set airflow.loadExamples=true \
--set web.baseUrl=http://127.0.0.1:8080 \
--set auth.username=$AIRFLOW_USER \
--set auth.password=$AIRFLOW_PASSWORD \
--set auth.fernetKey=$AIRFLOW_FERNETKEY = \
--set postgresql.postgresqlPassword=$POSTGRESQL_PASSWORD \
I have been successful deploying airflow (using the bitnami chart but also had the same issue with other charts), and by running
kubectl get pods I get:
NAME READY STATUS RESTARTS AGE
bitnami-release-airflow-scheduler-774d647447-j6vpd 1/1 Running 0 4m6s
bitnami-release-airflow-web-5897c99754-hq6nr 1/1 Running 0 4m6s
bitnami-release-airflow-worker-0 0/1 Running 0 4m6s
bitnami-release-postgresql-0 1/1 Running 0 4m6s
bitnami-release-redis-master-0 1/1 Running 0 4m6s
I have port-forwarded from the web server's pod to my machine, however, when I try to run the example DAGs, the tasks seem to be stuck in
running. I have later found an error on the UI:
The scheduler does not appear to be running. Last heartbeat was received 1 minute ago. which seems to me that the scheduler is down.
Things I have tried:
- Manually running
init db on the web server via:
kubectl exec -it bitnami-release-airflow-web-5897c99754-hq6nr -- bash -c "airflow initdb" (was successful but did not solve the problem)
- Manually running
airflow scheduler via:
kubectl exec -it bitnami-release-airflow-scheduler-774d647447-j6vpd -- bash -c "airflow scheduler (Tasks started getting added to the queue but then got:
ERROR - Process timed out, PID: 5957
It seems to me that I I am not taking the right steps upon installing the chart. What are the steps do I need to take after installing the Helm chart?
Happy to provide further information if needed.