Github

A functional Data Mesh built on dbt and Airflow

dbt is the de facto standard for data transformations in the data warehouse. Airflow is a powerful orchestrator for any data processing workflows. DMP.AF brings them together and turns the chaos of Data Mesh into a manageable data ecosystem.

Quick start

Has your Data Mesh turned into a complete Mess?

Are you tired of endless approvals, data inconsistencies, and manually triggered pipelines? Are you building a data platform, but every new domain turns into a separate island with its own rules, scripts, and schedules? Data Mesh seems like a vision of the future, but how can you actually implement it—quickly, flexibly, and painlessly?

  • Duplication of transformation and ETL logic
  • No transparency: what was updated, when, and why?
  • Every department writes its own pipeline—making maintenance difficult and changes risky
  • Manual work, errors, wasted time, and loss of trust in your data

Painfully familiar…

Give your data both freedom and reliability.

Integrating dbt + Airflow with the philosophy of Data Mesh is the ultimate way to build a scalable and transparent data platform of the future.

DMP.AF brings these two products together and turns the chaos of Data Mesh into a manageable data ecosystem.

dmp-af runs your dbt models in parallel in Airflow. Each model becomes an independent task, while preserving dependencies between domains. The solution to your data challenges is the dmp.af platform.

Show me solution!!

Domain-Driven Architecture

Separate models by domain into different DAGs, run in parallel, perfect for data mesh architectures.

  • dbt-First Design

    All configuration in dbt model configs. Analytics teams stay in dbt, no Airflow knowledge required.

  • Flexible Scheduling

    Multiple schedules per model (@hourly, @daily, @weekly, @monthly, and more).

  • Enterprise Features

    Multiple dbt targets, configurable test strategies, built-in maintenance, Kubernetes support.

  • Auto-Generated DAGs

    Automatically creates Airflow DAGs from your dbt project, organized by domain and schedule. Handles dependencies across domains seamlessly.

  • Team-Friendly

    Analytics teams stay in dbt. No Airflow DAG writing required. Infrastructure handled automatically.

  • Idempotent Runs

    Each model is a separate Airflow task with date intervals passed to every run. Reliable backfills and reruns guaranteed.

Tired of scrolling?

Deploy dmp.af

“The dmp.af platform simply divided my life into before and after!”

—  Everybody out there

 Simple pricing

The dmp.af library is provided as an open-source solution that anyone can use and modify. Does it seem complicated or unclear? Then the cloud-based solution is right for you!

  • Absolutely free

    Open-source

    Yes, absolutely free. Get started right now.

    £0 (!!!)

    • DAG generation based on dbt
    • Advantages of dbt and Airflow
    • Data Mesh approach out of the box
    • Improvements by the open-source community

    Try it cost free

  • Popular

    Cloud-based solution

    We take care of the infrastructure, so you get quality and reliability.

    от £3.14/month

    • All the functionality of the free library
    • Ready-to-use infrastructure for launching your solution
    • User-friendly web interface
    • Low cost

    Sounds interesting!

  • All Inclusive

    Bespoke solution

    Our team arrives at your site and doesn’t leave until the dmp.af platform is fully implemented.

    £££ upon request

    • All the functionality of the cloud solution
    • But on your own infrastructure
    • Custom modifications
    • Training for your team

    Contact us

Public materials

  • From 1C Chaos to an Incremental DWH on Postgres, Airflow, and dbt

    From 1C Chaos to an Incremental DWH on Postgres, Airflow, and dbt

    Feb 18, 2026

    Original article on denvic.tech (in Russian) Aleksandr Mazalov (Senior Data Engineer & Data Architect) shares a real-world case of migrating from a chaotic architecture built on 1C and K...

    Read more

  • dbt in Data Mesh. With love for the users

    dbt in Data Mesh. With love for the users

    Aug 13, 2024

    Link to the talk Leonid and Nikita are the creators of the rapidly growing dmp-af integration. In their talk, they will discuss working with big data in the data mesh concept, describe their...

    Read more

  • From Hype to Production: Data Mesh with Airflow + dbt

    From Hype to Production: Data Mesh with Airflow + dbt

    Apr 4, 2024

    Link to the talk The promised three years of waiting are over—toloka.ai is ahead of schedule! A year ago, we shared our approach to integrating dbt and Airflow, and now we’re excited to pres...

    Read more

  • dbt is the core of a modern data platform

    dbt is the core of a modern data platform

    Sep 7, 2023

    Link to the talk dbt is one of the fastest-growing tools in the field of data platform and data warehouse development. Its combination of simplicity and functionality won over the Toloka.ai ...

    Read more

Load more posts...

Frequently Asked Questions

dmp-af — General Information

What is dmp-af and what is it for?
dmp-af runs dbt models in parallel in Airflow; each model becomes a separate DAG task with dependencies between models and domains preserved. This speeds up pipelines and scales for large projects.
What are the key advantages?
Domain-oriented architecture with separate DAGs, “dbt-first” configuration directly from models, flexible scheduling (@hourly, @daily, @weekly, @monthly, etc.), support for multiple dbt targets, testing strategies, and Kubernetes integration.
What project sizes is it suitable for?
The project is designed for large dbt installations (1000+ models), but works correctly with projects of any size.

Installation and Getting Started

How do I install dmp-af?
Add the package to your Airflow environment using pip install dmp-af or include dmp-af in your Airflow cluster’s requirements.txt.
How do I create my first DAG?
Define the configuration in your dbt models, then generate the DAG in Airflow; dmp-af will convert the models into tasks considering dependencies and schedules.
Is special configuration needed in Airflow?
A standard Airflow installation with access to your dbt project and profiles is required; dmp-af is installed as a regular Python package and uses configuration from dbt.

Architecture and Capabilities

How does dmp-af integrate with dbt?
The configuration is stored within the dbt models themselves; dmp-af reads parameters and builds tasks and DAGs without manual changes to Airflow.
Does dmp-af support multiple environments and targets?
Yes, multiple dbt targets and corresponding run and test policies are available.
What schedule options are there?
You can set different schedules at the model level, including built-in periods like hourly, daily, weekly, and monthly.
Is Kubernetes supported?
Yes, operation in Kubernetes environments is provided for enterprise scenarios.

Performance and Scalability

How is parallelism and scalability ensured?
Splitting by domains and converting models into independent Airflow tasks allows for parallel computation without breaking dependencies.
Is it suitable for data mesh?
Yes, dmp-af is oriented towards domain-driven design and data mesh architectures.

Quality and Testing

How is testing organized?
dbt testing strategies are supported, along with guidance for running local tests and CI in the contribution documentation.
Can different test strategies be set for different models?
Yes, strategies are configured in dbt and read by dmp-af when building DAGs.

Contribution and Community

How can I participate in development?
Open an issue or submit a Pull Request according to the CONTRIBUTING guidelines; the project is supported by IJKOS & PARTNERS LTD and open to the community.
Where can I see the release and testing process?
The Contributing section contains detailed instructions on releases, code style, testing, and the PR workflow.

Consulting during deployment

Although the dmp.af library is excellent and easy to use solution, we are ready to provide consulting on its deployment.

Contact our team