Skip to content

Latest commit

 

History

History
13 lines (10 loc) · 937 Bytes

README.md

File metadata and controls

13 lines (10 loc) · 937 Bytes

Airflow with Docker Kubernetes

This image used builds upon Apache's Official Airflow image, specialising it for use as controller of tasks being run in the City of Cape Town's Kubernetes cluster.

Please head over to the Kubernetes README to go on the exciting journey of deploying Airflow as a service within the City's Kubernetes cluster (which Airflow will also use to run tasks).

This airflow configuration uses a LocalExecutor to allow for task parallelism, doesn't load the examples, and adds a few utilities as well as localising to South Africa, otherwise the config is left to the defaults.

The reason the LocalExecutor is used is that our intended mode of execution is to offload work directly to Kubernetes, and so avoid the complication of maintaining a separate work broker.