attribute-aware access control<\/a>: authorization should be based on trusted policy attributes, not natural-language promises.<\/p>\nBe deliberate about replicated data. If a source table has strong restrictions but a preprocessing job writes all text into a broadly readable chunk table, the governance boundary has moved. The derived table, index source, and any materialized intermediate data must receive protections appropriate to the original sensitivity.<\/p>\n
Lineage makes AI dependencies reviewable before something changes<\/h2>\n
Unity Catalog lineage can show how governed data assets relate to downstream notebooks, jobs, dashboards, models, functions, and other supported objects. In current Databricks capabilities, lineage also extends into governed model services and related AI dependencies. That helps teams answer a question that becomes difficult in mature AI systems: what will break or become exposed if this source, model, or service changes?<\/p>\n
Lineage visibility itself follows permissions. A user who lacks the required access to an object does not automatically gain full knowledge of that asset merely by opening a lineage graph; unauthorized nodes can be masked. This is an important governance property because dependency maps can reveal sensitive project names, datasets, or system relationships even when the underlying content is protected.<\/p>\n
For RAG systems, lineage is most useful when pipelines remain explicit. Preserve the connection from source data to cleaned documents, chunks, indexes, evaluation datasets, and serving applications where the platform supports it. If a policy document is retired or a table changes schema, lineage provides a starting point for identifying dependent AI workloads that may need re-indexing or validation.<\/p>\n
Govern agent tools as capabilities, not just pieces of code<\/h2>\n
An AI agent becomes materially more powerful when it can take actions. A tool may query customer records, update tickets, run SQL, send messages, or invoke an external API. Tool authorization therefore needs the same seriousness as direct user authorization. A prompt such as \u201cdo not modify production\u201d does not provide the protection of a permission boundary.<\/p>\n
Databricks supports governed agent tools through Unity Catalog and Unity Gateway patterns, including Unity Catalog functions and MCP-based services. Current features vary in maturity, and some managed MCP capabilities remain preview or beta, so teams should verify product status before making them mandatory architecture. The durable design principle is clearer: register and permission tool capabilities, authenticate the caller, and limit the actions that the agent can invoke.<\/p>\n
A structured-data lookup with a known query pattern can be a good fit for a governed Unity Catalog function. More general tool access may use MCP services or application-defined logic. Whichever mechanism is chosen, avoid a single all-powerful service credential shared by every agent. Create scoped identities and grant only the functions, services, and data required by that workload.<\/p>\n
Model access is another governed dependency<\/h2>\n
RAG data is only one side of the system. An application may call a hosted foundation model, a custom model endpoint, or an external provider. Current Databricks governance exposes model services as securable objects and uses Unity Catalog privileges together with gateway controls to govern who can invoke them.<\/p>\n
This matters for compliance and cost. A regulated workload may permit only approved models. A high-cost model may be restricted to a small group or protected by rate and budget controls. A model-provider migration may need impact analysis before the old route is retired. Model access should therefore be part of the same architecture review as table access and retrieval permissions.<\/p>\n
Keep secrets out of application code. Prefer managed credentials and service identities. When external providers or tools are used, document what content leaves the Databricks environment, which identity authorizes the call, what retention the provider may apply, and whether the payload contains regulated data.<\/p>\n
Production identity should be explicit and testable<\/h2>\n
Notebook permissions used by a developer are a poor production security model. A deployed endpoint, scheduled ingestion job, evaluation workflow, and background indexing process should run with identities whose privileges can be reviewed independently. Service principals are useful because they separate workload authorization from the changing membership of human teams.<\/p>\n
Test these identities directly. A useful security test verifies that the production principal can read the expected corpus, cannot read a restricted table, can invoke only the intended model or tool services, and cannot write where read-only operation is expected. Negative tests are especially important because a successful happy-path request proves almost nothing about over-permission.<\/p>\n
When end-user identity must affect the result, make that propagation an architectural requirement rather than an afterthought. A single broad application identity that retrieves everything and then attempts to filter in the prompt creates a much weaker boundary than one that enforces permissions before retrieval.<\/p>\n
Trace data is governed production data too<\/h2>\n
LLM traces can contain prompts, retrieved documents, tool arguments, database results, model responses, user identifiers, latency, cost, and evaluator feedback. In other words, observability can recreate much of the sensitive content processed by the agent. Treat raw traces as potentially sensitive even when the original source data is already governed.<\/p>\n
Current MLflow guidance recommends Unity Catalog storage for new and production trace workloads. That allows trace access to be controlled through schema and table permissions and analyzed as Delta data. Row filters, column masking, retention policy, and auditing can then be applied through established governance patterns. Where personally identifiable information is present, use minimization and redaction strategies consistent with broader PII protection practices for machine-learning systems<\/a>.<\/p>\nRedaction must be placed carefully. Client-side redaction can prevent sensitive content from being transmitted into the observability backend at all. Server-side sanitization can create a safer derivative for analysts while raw traces remain tightly restricted. Neither approach changes what the agent itself saw during execution, so trace redaction should not be confused with runtime access control.<\/p>\n
Auditing should connect identity, data, models, and tool calls<\/h2>\n
When an AI incident occurs, investigators need more than the final answer. They need to know who invoked the system, which governed data was available, which model or service handled the request, which tools ran, whether policies allowed the operation, and what changed between the failing and previous versions. Centralized governance and tracing make this reconstruction practical.<\/p>\n
Audit design should focus on answerable questions. Which principal read a sensitive dataset? Which workloads depend on a model scheduled for retirement? Which agents can invoke a write-capable tool? Which evaluation traces contain a certain policy version? Which production identities were granted a new privilege last week? These questions expose gaps that a generic \u201cwe have audit logs\u201d statement does not.<\/p>\n
Retain enough metadata to support investigations, but do not create a second uncontrolled data lake of raw prompts and responses. Access, retention, and deletion rules for observability data should be explicit.<\/p>\n
Governance should shape the RAG architecture before deployment<\/h2>\n
The strongest RAG and agent systems are easier to govern because their architecture makes boundaries visible. Source data is governed, derived corpora preserve appropriate restrictions, application and job identities are separate, model and tool access is permissioned, sensitive traces are controlled, and lineage helps identify downstream dependencies.<\/p>\n
This approach also improves reliability. A precise permission model clarifies who owns data. Trace governance makes debugging evidence available to the right engineers. Lineage reduces the risk of silent downstream breakage. Scoped tools prevent an agent from taking actions it never needed. Governance is therefore not simply a compliance layer placed around the AI application; it is part of the system design that makes production behavior predictable.<\/p>\n","protected":false},"excerpt":{"rendered":"
A RAG application or AI agent expands the security boundary of an ordinary analytics workload. The application may retrieve documents, query structured data, call functions, […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[33,29,65,145],"tags":[],"class_list":["post-3300","post","type-post","status-publish","format-standard","hentry","category-ai-machine-learning","category-certifications","category-databricks-certifications","category-exam-genai-engineer-associate"],"_links":{"self":[{"href":"https:\/\/www.examtopics.biz\/blog\/wp-json\/wp\/v2\/posts\/3300","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.examtopics.biz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.examtopics.biz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.examtopics.biz\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.examtopics.biz\/blog\/wp-json\/wp\/v2\/comments?post=3300"}],"version-history":[{"count":1,"href":"https:\/\/www.examtopics.biz\/blog\/wp-json\/wp\/v2\/posts\/3300\/revisions"}],"predecessor-version":[{"id":3801,"href":"https:\/\/www.examtopics.biz\/blog\/wp-json\/wp\/v2\/posts\/3300\/revisions\/3801"}],"wp:attachment":[{"href":"https:\/\/www.examtopics.biz\/blog\/wp-json\/wp\/v2\/media?parent=3300"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.examtopics.biz\/blog\/wp-json\/wp\/v2\/categories?post=3300"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.examtopics.biz\/blog\/wp-json\/wp\/v2\/tags?post=3300"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}