Building large scale data pipelines by Apache Airflow
黃泰瑋 (Tai-Wei Huang)
Tai-Wei Huang is a data engineer at E.SUN Bank. His work mostly focuses on data pipeline, data quality and every thing about data.
Abstract
本演講將說明如何透過 Airflow DAG 大規模擴展 ETL pipeline 以及調控各項參數，達成每日更新 0.7~1T 的資料，並透過 Airflow 蒐集與定義 data downtime 計算出 data SLA，也會佐以講者 ３ 年來辛酸血淚的開發與維運經驗，讓聽眾可以少踩一些坑，安心提早下班
Description
Video
Location
R0
Date
Day 1 • 10:45-11:15 (GMT+8)
Language
Chinese talk w. English slides
Level
Intermediate
Category
Databases