From Data Pipeline to Airflow - the Obstacles in Our Migration
- Intro
- More Info
- Slido
- Note
Abstract
We had in-house workflow schemas, we had in-house deployment tools, how could we migrate to Airflow with minimum changes. In this talk, I would like to share the obstacles we encountered and how we overcame them.
Description
I will talk about:
- How to deploy simple workflows to Airflow without knowing its concepts.
- How to join instances and clusters in one workflow.
- Beyond the static workflow manager.