OpenSharing and Lakehouse Federation both let Databricks users work with data that is not simply a local managed table, but they solve opposite sides of […]
Data Engineer Professional
Databricks Data Engineer Professional: How Delta Lake Skips and Clusters Data
Databricks Delta Lake performance tuning is often described as a file-size problem, but the more important question is how much data a query can avoid […]
Databricks Data Engineer Professional: Monitoring Data Pipelines
Monitoring a Databricks data pipeline is not the same as checking whether the latest job run is green. A production pipeline can finish successfully while […]
Databricks Data Engineer Professional: Quarantining Bad Data
In Databricks, quarantining bad data is a recovery design, not simply a filter. A robust Databricks pipeline should be able to identify records that violate […]