Link to the talk

The promised three years of waiting are over—toloka.ai is ahead of schedule! A year ago, we shared our approach to integrating dbt and Airflow, and now we’re excited to present our open source library.

This library enables seamless integration between dbt—a relatively new but already popular tool for describing data transformations—and Airflow, the well-known orchestrator of ETL processes.

The talk will be divided into three parts:

• First, we’ll remind you how our system is set up and where data mesh, dbt, and Airflow fit in.
• Next, we’ll review competing solutions—there aren’t many direct competitors to our dbt-af library.
• Finally, we’ll demonstrate how conveniently dbt-af handles its intended tasks.

This talk, like the tool itself, will be useful for:

• DWH teams tired of pressure from analysts;
• Analysts frustrated by slow-moving DWH teams;
• Anyone interested in the practical implementation of data mesh using widely available tools.

Library: https://github.com/Toloka/dbt-af