Oct 5, 2021
When you're dealing with resources its important where do you place your qa layer. Since we are in Google Cloud it would be wise to qa there as BQ is faster in that sense. Also, there are xcom variables in Airflow that allows you to transform data loaded from previous step in memory, no re-extraction needed. Check it out! It has its limitations though