A retrieval-augmented assistant can produce a fluent, source-cited answer and still be wrong for two independent reasons: the source may be out of date, or the caller may not be allowed to see it. These failures are especially serious in internal policy assistants, where a superseded procedure and a restricted personnel record can both look like relevant search results. Production RAG needs deliberate freshness and permission contracts, not just a vector index and a model.
Imagine a fictional employee assistant that answers questions about expense policies, regional travel rules, and department-only procedures. The finance team publishes a corrected reimbursement policy while an employee moves to a different department. The system must reflect the changed policy and the changed entitlement without leaking another group’s evidence. That requirement spans ingestion, index maintenance, query identity, retrieval configuration, caches, generation, and audit.
A Bedrock knowledge base returning a semantically relevant result does not prove that the source is current or authorized for the user. Test three synchronized document versions: an active equipment policy, a superseded version and a record whose access has just been revoked. Check actual ingestion and retrieval status, the document’s stable source ID, the permitted reader set and the final answer’s citation before accepting a response. A managed connector’s ACL-aware search or application metadata filter can help narrow results, but the trusted application must still authenticate the user and enforce the relevant authorization boundary. Independent AWS AIP-C01 knowledge-base practice scenarios can help identify which layer is responsible for a stale answer or a cross-tenant exposure; they are not evidence that an unexecuted example passed a real AWS permission test. The official AIP-C01 objectives remain the source for exam scope.
Give source documents a durable version and refresh policy
A source record needs an identifier stable across ordinary updates, its effective version, modification time, department classification, owning system, and the reference that permits an operator to inspect the original. A document title alone is insufficient because titles can recur and change. Keep an authoritative source-of-truth location and record when the search system last ingested its contents. A retrieval result should carry enough metadata to distinguish the source’s business-effective date from the technical index date.
Define the freshness objective according to consequence. A public FAQ might tolerate a scheduled refresh. A policy used for spending approval may need a stricter service level and a workflow that blocks high-impact answers until a required update is indexed. Saying “near real time” is not meaningful without a measurable interval and a response to overdue data. The system should expose the age of indexed evidence or refuse an action where the source is outside its permitted staleness window.
Metadata does not enforce the update itself. The pipeline must detect source additions, edits, and deletions and initiate the appropriate synchronization or direct-ingestion operation. An AWS data source can have asynchronous ingestion jobs, and a successful job request does not mean every changed object has already become searchable. Record job identity, status, error counts, and the moment each source version is accepted into the serving index.
Treat deletion and revocation as first-class changes
Many knowledge-base designs carefully ingest new documents but scarcely test removals. A revoked policy, deleted attachment, or expired temporary file can remain visible in an old index, answer cache, or conversation memory if those paths are not considered together. Define how deletion propagates through the data source, derived embeddings, retrieval service, application caches, and retained conversation evidence. The correct behavior may vary by the application’s retention obligations, but it must be explicit.
The Amazon Bedrock Knowledge Base synchronization process describes syncing source changes into a knowledge base. The service can automate ingestion work, while the application still decides what to do while an update is pending. A high-consequence transaction can block or require an operator review if an important revocation has not yet propagated. Do not present asynchronous synchronization as an instantaneous global deletion guarantee.
Test a policy replacement by querying both the old and new phrases before and after the refresh. Test a source deletion using a query that previously retrieved only the deleted item. Record when results change and whether an application-level cache continues serving the older text. Use a version marker in cache keys when appropriate, and give administrators a documented invalidation path. A knowledge base that updates correctly but sits behind a stale response cache does not meet the business freshness contract.
Separate authorization from search filtering
Authentication establishes who is asking; authorization determines which evidence that identity may access. Search filtering helps enforce permitted retrieval, but not every filter is equivalent to a security boundary. A custom vector knowledge base may use metadata attributes such as department or tenant. Those attributes become useful only after the backend independently verifies the caller’s entitlement and constructs a non-bypassable filter. Allowing the user to supply an arbitrary tenant identifier or an editable filter would defeat the design.
Bedrock Managed Knowledge Base offers ACL-aware retrieval for supported, configured data-source connectors. When a caller requests information, the application can send userContext identifying the user it has already authenticated; Bedrock uses the indexed permissions, and where supported additional source verification, to filter retrieved documents. AWS explicitly distinguishes this result filtering from end-user authentication and authorization: the knowledge base cannot establish that a supplied identity is genuine. The application must bind the authenticated caller to that identity, prevent arbitrary user-context substitution and enforce backend authorization on any consequential action. A filtered result still needs freshness checks when source documents or permissions change.
Do not copy the managed-KB request shape into a custom vector KB. AWS distinguishes managedSearchConfiguration for managed retrieval from vectorSearchConfiguration for custom vector search. Managed retrieval also has its own ranking and filter capabilities. A request that happens to use a similarly named property is not guaranteed to implement the same semantics, especially for access-controlled queries.
Handle mixed ACL sources and eventual consistency honestly
ACL-aware retrieval carries conditions that can be surprising. AWS documents that omitting userContext from an ACL-enabled source returns no such source results, while non-ACL sources in the same managed knowledge base can continue returning results. Missing ACL metadata on an ACL-enabled source is treated as inaccessible, not public. If a managed KB mixes protected and unprotected data sources, its operators must understand which documents are intentionally public and verify that the mixture is appropriate.
Permissions can also be eventually consistent. AWS documentation describes caching for some third-party identity-provider credentials and notes that access changes may not take effect immediately. Therefore, a critical revocation might require additional application controls, such as suspending a high-impact action or denying certain queries until the system can demonstrate that the changed entitlement has propagated. Do not promise immediate access revocation simply because a user was removed from a source system minutes ago.
When real-time ACL verification filters some matches, the result set can contain fewer items than the requested limit. A retrieval caller should not assume that receiving two results instead of five means the query failed, nor silently fill the remainder with unfiltered records. Record that result reduction occurred and allow the answer generator to represent insufficient evidence. If the application needs a minimum number of authorized references, that should be an explicit acceptance requirement rather than an invitation to weaken filtering.
Evaluate retrieval separately from generated explanations
The first test layer asks whether the correct authorized source is retrieved. Build positive cases for policy owners, ordinary employees and explicitly denied users. Also include no-access and missing-context cases. Measure document recall and relevance only within the evidence the caller is permitted to access. The retrieval system should not be rewarded for finding the “right” confidential document for an unauthorized user.
Retrieving the correct policy is not the same as generating a faithful answer. When evaluating generative AI on AWS, score whether the response cites the currently approved policy version, respects the caller’s permitted source set and avoids mixing superseded passages into a plausible explanation. A high semantic similarity score cannot establish authorization or factual support. Keep a record of source version and entitlement with each test case so a later index change can be evaluated honestly.
A negative test should ask a question whose answer appears only in a document the user cannot access. The expected response is an absence of authorized evidence, not a synthetic reconstruction. Another case should include two similarly titled policy versions and confirm that the latest effective source is chosen. A third should change department membership and measure the actual revocation delay. These tests reveal security and freshness behavior that conventional semantic search benchmarks miss.
Keep retrieval failures visible in operations and audit
Instrument the path from source update to ingestion job, index version, retrieval request, authorized source IDs, generated answer and any human escalation. Avoid recording raw sensitive document text in broad operational logs; source references and protected audit records are usually more appropriate. Track overdue source synchronizations, failed parsing, partial ACL verification, empty authorized results, stale cache hits and mismatch between intended and returned source versions.
When investigating a failure, classify it before changing a model. A wrong answer with old evidence points toward a source or cache problem. A missing authorized document points toward ingestion, filtering, index, or query composition. A disallowed document in retrieval points toward an authorization boundary failure that requires immediate containment. A correct retrieved passage but unsupported final statement points toward generation behavior. This separation prevents “add a stronger prompt” from becoming an excuse for an access-control defect.
Some security incidents warrant pausing an affected collection while the team validates its permissions, rather than continuing to serve plausible answers. The exact decision depends on sensitivity and business needs, but the application should have a documented degraded mode. An empty retrieval result and a policy violation deserve different user-facing messages and different operator response paths.
Prove freshness and permissions before launch
For the fictional policy assistant, execute a release matrix with at least four identities: policy owner, authorized employee, employee from a different department and suspended user. For each, test an allowed source, a denied source, an updated policy, a deleted source, and a query with no authorized answer. Capture the source identifiers and retrieval configuration used, not just the fluent summary. In a managed ACL design, verify the exact supported connector and identity mapping under the organization’s real authentication system.
Run a synchronization delay scenario and a source-permission change scenario independently. Confirm that high-risk operations are held while freshness or access is uncertain. Then test a mixed-source collection and verify that “public” documents really are meant to be available to everyone. A test that passes for an administrator alone cannot demonstrate least-privilege retrieval.
Reliable RAG is the intersection of current evidence, authorized access and faithful generation. Bedrock Knowledge Bases can perform the managed ingestion and retrieval functions, but the application still owns authentication, freshness policy, permission propagation, cache invalidation, and consequence-aware fallback. Those contracts turn a useful prototype into a service whose answers can be defended and corrected.