Latest news & updates_

Dive into these stories to understand how we've partnered with diverse clients to overcome obstacles, implement innovative strategies, and achieve tangible results.

Image of a man using sdp-meta to manage pipelines

Databricks

Delta Live Tables

Spark Declarative Pipelines

SDP-META

SDP-META: Pipelines at Scale

Oct 15th, 2026

•

Eric Grun

Managing pipelines with a metadata-driven framework

Illustration of a person with data tables, databases, and AI agent icons flowing bidirectionally between a secured on-premises server building and a cloud-connected building with AI agents and gears, representing data and asset sharing between platforms.

Sep 28th, 2026

•

Weston Kimbro

OpenSharing: The Natural Evolution of Delta Sharing

Databricks renamed Delta Sharing to OpenSharing and handed governance to the Linux Foundation. Here is what actually changed under the hood, and why the same three-object model still applies.

Tuning Spark Jobs From the Transaction Log

Photon and AQE (Adaptive Query Execution) make the work you already do faster. The Delta transaction log decides how much work exists at all — here are five levers you can read straight out of _delta_log/ and the fixes each one points to.

Genie Ontology Ranks Trust, Not Correctness

Databricks' Genie Ontology and OntoRank solve the problem of trust, not accuracy. Here's why ranking authority isn't the same as checking math.

Inside Lakewatch Training for SOC Teams

A data engineer's first-hand look at Databricks' Lakewatch SIEM training, covering ingestion presets, detection engineering, and how a lakehouse-native architecture closes the coverage gaps that open during SIEM migrations.

W. Kimbro, H. Schumann

Ranking Spark 4.2 Features from S-to-F Tier

A data engineer and an AI/ML engineer tier-rank every major Spark 4.2 feature, argue about it, and reconcile their disagreements into one combined ranking.