EXAMTOPICS / TECHNOLOGY INSIGHTS

Databricks Data Engineer Professional: Delta Sharing vs Lakehouse Federation

OpenSharing and Lakehouse Federation both let Databricks users work with data that is not simply a local managed table, but they solve opposite sides of the boundary. OpenSharing—formerly documented as Delta Sharing—lets a provider expose live, read-only data or other supported assets to recipients across workspaces, accounts, clouds, and even non-Databricks platforms. Lakehouse Federation lets Databricks query data that remains in external databases or catalogs through Unity Catalog foreign catalogs.

The distinction matters for the current Databricks Data Engineer Professional scope, which includes secure sharing and governed federation. A practical architecture question is simple: are you distributing access to data you control, or are you querying data that another system continues to own?

OpenSharing starts with the provider

OpenSharing is provider-centric. A data owner creates a share, adds supported assets, defines recipients, and grants those recipients read access. In Databricks-to-Databricks sharing, Unity Catalog handles the relationship between provider and recipient metastores. In open sharing, recipients can use compatible external tools and protocols without needing to run Databricks.

The provider remains responsible for the shared asset and can add or remove assets, revoke recipients, and audit sharing activity. Updates to the provider’s data can become visible to recipients without the provider building a traditional export pipeline that copies a snapshot into another storage account.

This is especially useful when the business requirement is “give another team or organization governed access to this live dataset.” The provider wants control over what is exposed while avoiding a proliferating collection of CSV exports and bespoke delivery jobs.

Lakehouse Federation starts with the consumer

Lakehouse Federation is consumer-centric. A Databricks team creates a connection to an external source and exposes remote objects through a Unity Catalog foreign catalog. Queries run from Databricks while the underlying data remains in the external system. The remote platform continues to own storage and often remains responsible for primary transaction processing.

Query federation can push supported operations to the source so filtering or aggregation happens remotely instead of pulling every row into Databricks first. Catalog federation provides governed read access to certain external catalogs and object storage. Both patterns let Databricks users discover and query external data through Unity Catalog without first building a full ingestion pipeline.

The result is access rather than ownership transfer. A foreign table is read-only from the Databricks perspective. If the goal is to transform and manage the data as a Databricks-owned asset, ingestion may still be the better architecture.

Choose sharing when you own the data product

If your team owns a curated dataset and another business unit, customer, supplier, or partner needs to consume it, OpenSharing is usually the more natural model. You define the data product once and grant recipients access. Databricks-to-Databricks sharing can also include supported views, volumes, models, and notebooks in addition to tables.

Sharing is also attractive when freshness matters and duplication should be minimized. Instead of creating daily exports, the provider exposes the live asset under controlled read-only semantics. Access can be revoked centrally, and provider/recipient activity can be monitored through audit mechanisms.

Within workspaces attached to the same Unity Catalog metastore, a share may be unnecessary because ordinary Unity Catalog permissions can already govern cross-workspace access. Using sharing inside that boundary can add concepts without adding security value.

Choose federation when the source should stay where it is

Federation is useful when an external database remains the system of record and moving the data would be expensive, unnecessary, or operationally premature. An analytics team may need a small subset of PostgreSQL, SQL Server, Snowflake, or another supported source for a join or exploratory workload. A foreign catalog can expose that data without building a persistent copy first.

This can reduce time to value in migrations. Teams can query legacy or third-party systems from Databricks while progressively deciding which datasets deserve ingestion and optimization in the lakehouse. Governance is centralized through Unity Catalog even though storage remains external.

Federation is not free from source constraints. Query performance depends on the remote system, network path, connector capabilities, and pushdown behavior. A workload that repeatedly scans large external tables may be better served by ingestion into managed Delta or Iceberg tables.

Freshness and performance pull in different directions

Both technologies can provide fresh data without a traditional replication schedule, but the performance model differs. OpenSharing recipients read provider-controlled shared assets. Lakehouse Federation sends queries toward the external source and is therefore sensitive to source-system latency, concurrency, and cost.

If analysts issue many large federated queries against an operational database, they can create load where the business least wants it. Caching, materialization, or ingestion may eventually be necessary. Conversely, sharing a well-optimized Databricks table can preserve the provider’s data layout and avoid forcing every recipient to build a separate ingestion copy.

Architecture reviews should therefore include expected query frequency, scan size, concurrency, and cross-region or cross-cloud data movement. “No copy” is not automatically the cheapest or fastest design.

Governance is different on the provider and consumer sides

With OpenSharing, the provider decides what enters a share and which recipients receive it. The recipient can then govern local access to the shared data within the capabilities of the sharing model. Revocation remains under provider control, which is important when the relationship with a partner changes.

With Lakehouse Federation, Unity Catalog governs access to the foreign catalog, but the external source still has its own authentication and authorization model. The connection identity must have suitable remote privileges, and Databricks principals must have the Unity Catalog privileges required to discover and query the foreign objects.

Interoperability is a governance problem as much as a connectivity problem. A connection that technically works but grants a broad service account unrestricted remote access can undermine the entire least-privilege design.

Do not use federation as a permanent excuse to avoid data engineering

Federation is excellent for live access, exploratory use, migration bridges, and sources that should remain external. It becomes less attractive when the same remote data is queried repeatedly, joined into many pipelines, or transformed heavily. At that point, ingestion can provide stronger performance isolation, data-quality control, and lakehouse-native optimization.

The reverse mistake is copying everything by default. Some reference datasets are small and stable enough to ingest; others change continuously and are better queried at the source. Some partner datasets are contractually shared and should remain provider-owned. Data ownership and lifecycle should drive the choice.

A useful review question is: “Which system should be authoritative for this data six months from now?” Federation is strongest when the answer remains the external system. Ingestion is stronger when Databricks is becoming the managed analytical home. Sharing is stronger when your Databricks environment remains authoritative but other parties need access.

Current terminology matters during design

Databricks documentation now uses OpenSharing for the platform that was previously described as Delta Sharing. Existing exam guides and architecture discussions may still use “Delta Sharing,” “Delta Share,” or “open sharing protocol.” Engineers should recognize those terms as part of the same evolving product family while checking current documentation for supported assets, protocols, and limitations.

Lakehouse Federation remains the term for governed access to external databases and catalogs. Within federation, query federation and catalog federation have different source support and execution behavior, so “federation” should not be treated as a single connector with identical semantics everywhere.

Terminology changes are not cosmetic when they correspond to broader capabilities. A current design should describe what is actually being shared or federated and which protocol or connector owns the access path.

Choose OpenSharing when you own a governed data product and want to grant live, read-only access to recipients without building a separate export for each one. Choose Lakehouse Federation when another system remains authoritative and Databricks users need governed query access in place. Choose ingestion when Databricks should own the analytical copy and optimize it for repeated downstream processing.

Those choices can coexist. A company might federate a legacy operational database during migration, ingest the highest-value tables into Delta for analytics, and then OpenShare curated results with partners. The architecture is coherent because each boundary has a clear owner.

The important skill is not memorizing which feature can reach which source. It is deciding where data ownership, compute, governance, freshness, and cost should live. Once that is explicit, sharing and federation become complementary tools rather than competing product names.

Network and source-system constraints can decide the architecture before either feature is configured. Federation keeps query compute dependent on the remote system’s availability, concurrency limits, and performance characteristics. That may be exactly right for occasional operational lookups but risky for a high-volume dashboard whose service objective is stricter than the source database can support. Sharing avoids that particular dependency for data the provider controls, but it requires the provider to curate and expose an appropriate asset.

Cost attribution differs too. Federated queries can consume Databricks compute while also placing work on a remote system, so optimization has to consider both sides of the connection. Shared consumption shifts the boundary toward the shared data product and the recipient’s read workload. An architecture review should therefore identify not only who owns the data, but who pays for repeated access, who can diagnose a slowdown, and which team has authority to change the physical design when demand grows.

That clarity also makes later migration easier, because teams can revisit one boundary at a time when source ownership, query volume, or sharing requirements change.