Some working performance/cost improvement tips applying to ADF and Databricks recently

While switching to the cloud, we found some pipelines running slowly and cost increased rapidly. To solve the problems, we did flowing steps to optimize the pipelines or data structures. They are all not hard to be implemented. 1. Set the different triggers for different recurring periods. No matter for what reason, it is very… Continue reading Some working performance/cost improvement tips applying to ADF and Databricks recently

How to build a data pipeline in Databricks

For a long term, I thought there was no pipeline concept in Databricks. For the most engineers they will write the whole script into one notebook rather than split into several activities like in Data factory. In this case, we have to rewirte everything in the script when the next pipeline coming. But recently, I… Continue reading How to build a data pipeline in Databricks