Unlocking the Future of Data Pipelines - Apache Airflow 3

Abstract

Apache Airflow 3.0 is here, bringing an enhanced user experience and exciting features like dag versioning, newly designed data-centric syntax, event-driven scheduling, and a brand-new task execution interface that opens up the possibility of writing an Airflow task in a language other than Python! In collaboration with the Ruff team, we have developed linting and auto-fixing rules that simplify the upgrade process, enabling seamless code upgrade with just one command. At the end of this talk, I will share our experience of upgrading from Airflow 1 to Airflow 3 through a real-world open-source use case. I will highlight the challenges we faced and how we utilized the new features to optimize our data workflows. Whether you are a seasoned Airflow user or new to it, this session will provide you with valuable insights to enhance your data orchestration capabilities.

Description