Skip to content

Building upon Puckel's Docker Airflow image, for use in managing the City's data pipelines.

License

Notifications You must be signed in to change notification settings

cityofcapetown/airflow_docker_datapipelines

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

57 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Airflow with Docker Kubernetes

This image used builds upon Apache's Official Airflow image, specialising it for use as controller of tasks being run in the City of Cape Town's Kubernetes cluster.

Please head over to the Kubernetes README to go on the exciting journey of deploying Airflow as a service within the City's Kubernetes cluster (which Airflow will also use to run tasks).

This airflow configuration uses a LocalExecutor to allow for task parallelism, doesn't load the examples, and adds a few utilities as well as localising to South Africa, otherwise the config is left to the defaults.

The reason the LocalExecutor is used is that our intended mode of execution is to offload work directly to Kubernetes, and so avoid the complication of maintaining a separate work broker.

About

Building upon Puckel's Docker Airflow image, for use in managing the City's data pipelines.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published