Implementing Layered Data Architecture In Dagster

摘要

A layered data design pattern is a modern data architecture for building ETL/ELT data pipelines comprised of multiple stages so that each stage processes the data and improves the quality of the data progressively. Compared to the imperative way how data engineers build ETL/ELT data pipelines in the last decade, layered data architecture could be of great help in improving data quality steadily and progressively, and reducing data silos while project-specific teams are autonomously producing various data products. We will introduce, in this share, a technical solution based on layered data architecture. The solution is implemented by means of Dagster, a cloud-native data orchestrator with integrated lineage, observability, and a declarative programming model. A simple example will be presented in this talk to demonstrate concepts, principles, and data stack of the solution. In the end, the benefits we have gained from the implementation experience will be conveyed as well.

說明