--

Apologies if this wasn't up to your expectations.

A Framework by definition is a basic structure underlying a system. Yes, the whole purpose of the framework is to stray away from the conventional way of deploying DAGs. I have been doing that for quite some time and find that it clutters the DAG file. Hence, we've built a framework using the BaseOperator (where most custom operators are built) provided by Apache Airflow to establish a better DAG structure (config files, resources, etc).

--

--

Nicholas Leong
Nicholas Leong

Written by Nicholas Leong

Data Engineer — Crunching data and writing about it so you don’t get headaches. 1M+ reads on Medium. https://www.linkedin.com/in/nickefy/

Responses (1)