When you're dealing with resources its important where do you place your qa layer. Since we are in Google Cloud it would be wise to qa there as BQ is faster in that sense. Also, there are xcom variables in Airflow that allows you to transform data loaded from previous step in memory, no re-extraction needed. Check it out! It has its limitations though

Nicholas Leong
Nicholas Leong

Written by Nicholas Leong

Data Engineer — Crunching data and writing about it so you don’t get headaches. 1M+ reads on Medium. https://www.linkedin.com/in/nickefy/

No responses yet