A retrieval-augmented generation system is only as dependable as the material it retrieves. When an assistant confidently explains an obsolete policy, the model may not be the first component to fix: the relevant version could be missing from the index, a permission filter could have hidden it, or the retriever might have preferred a semantically similar but wrong passage. The AI-103 blueprint treats retrieval, indexing and agent grounding as implementation skills. Azure AI Search is one of the Azure services that can meet those requirements, but its usefulness comes from document design and evaluation rather than checking an “enable vector search” box.

Imagine a fictional regional parts distributor with repair manuals, product notices and pricing policies. Technicians ask which replacement part fits a given machine model, and support staff ask whether a warranty exception applies. Every document has an owner, product family, publication date, effective date and visibility label. This example is deliberately synthetic. Its value is that you can trace errors from a user question to an index entry and then back to the original document rather than mistaking a fluent response for validated evidence.

Define the answer contract before the index

Write down what the assistant is allowed to answer and which source makes an answer authoritative. A parts-compatibility question might require an exact product code and model revision. A policy question may require the newest approved policy effective on the customer’s service date. If the knowledge set cannot establish those facts, the correct output is an uncertainty statement or a request for human investigation. The retrieval system should return enough metadata to make that distinction, not only a paragraph of text.

Define evaluation queries early. Include exact part identifiers, paraphrased symptoms, outdated manuals, ambiguous model names, two employees with different permissions and a question where no source provides an answer. For each query, record expected document IDs, acceptable citations, wrong-but-plausible documents and whether abstention is required. This test set can be used to compare successive indexing and ranking configurations. It also prevents an engineer from choosing a retrieval method solely because the first demo question happens to work.

Build a document record that survives chunking

Real manuals are not clean blocks of text. A page can include tables, diagrams, document version notes, scanned labels and cross-references. An ingestion stage has to decide whether native text extraction is sufficient or whether layout analysis or OCR is needed. Once text is extracted, split it into chunks small enough for useful retrieval while keeping headings, section labels and the relationships between data fields. Splitting a warning sentence from the part number it qualifies can transform a correct manual into a dangerous answer.

Store the parent document ID and version alongside each chunk. Also preserve page or section location, access groups, effective date, ingestion timestamp and the original storage pointer. If a warranty guide is replaced, mark or remove its obsolete chunks in a controlled update rather than leaving both versions indistinguishable. A source-quality check should compare indexed record counts and sampled extracted passages against the approved corpus; an apparently successful indexer run does not prove that every page was processed correctly.

For illustrated documents, keep image references and extracted text tied to the same source location. An OCR result can be syntactically readable but numerically wrong, especially on a product serial label or tiny diagram. Flag critical fields for verification against the original document rather than letting a model improvise the uncertain digits. Microsoft’s Azure AI Search RAG guidance describes document enrichment, indexing and retrieval as a complete pipeline.

Use hybrid retrieval when language and identifiers both matter

Keyword or full-text retrieval is valuable for precise terms such as a part number, product suffix or security policy reference. Vector search can find relevant language even when the user and the manual express the same concept differently. They fail in different ways: an exact-match query can miss paraphrases, while a vector query may regard two similar but incompatible product revisions as neighbors. For a mixed technical corpus, evaluate both rather than assuming one is universally superior.

In Azure AI Search, a hybrid query can submit full-text and vector components together. The service runs those searches and uses reciprocal rank fusion to merge the results. Semantic ranking can be another optional stage. That architecture explains why an answer may depend on several distinct relevance choices: which fields were searchable, which chunks had embeddings, which filters were active, how candidates were ranked and which citations the application passed to the model.

Use a concrete comparison. Ask for the service procedure for fictional part AX-217-R2. A purely semantic search might surface AX-217-R1 because the surrounding instructions are nearly identical; keyword search may prioritize the exact R2 suffix. A hybrid configuration can benefit from both, but only if the product-revision filter is also correct. Test whether the selected result truly matches the model and effective date. A high semantic similarity score is not itself a compatibility determination.

Make authorization part of retrieval, not an afterthought

A support assistant should not retrieve confidential warranty concessions and then ask the model to avoid mentioning them. The search request must enforce the caller’s access boundaries before text reaches generation. Build permission-aware indexing and retrieval filters around trusted identities. A plain principal string used in a search.in filter is not authentication by itself. Where native query-time token-based ACL enforcement is enabled in preview, the index needs synchronized permission metadata, the request needs an authorized user token, and the calling application still needs its own index-access role. Test with two principals and prove that a denied passage never reaches the model.

Test with two synthetic employees from different branches. Give one employee access to a regional contract and the other access only to the public repair guide. Run identical queries under both identities; verify that both the retrieved documents and final answer honor the permitted set. Also test a stale access grant, a document that was reclassified and a deleted file whose chunks remain in the index. Authorization parity is an ongoing synchronization problem, not a one-time filter checkbox.

Some Microsoft retrieval architectures now provide agentic retrieval and Foundry IQ patterns in addition to more traditional RAG. Choose between them based on available features, network and region constraints, source-access model, citation needs, cost and acceptable latency. The newest portal feature is not necessarily the right production choice. Where preview limitations exist, document them explicitly and test the supported deployment rather than describing a preview-only option as universally available.

Preserve source evidence across the agent boundary

The retriever and language model have different jobs. Retrieval finds candidate evidence; generation explains an answer in the user’s language. Pass the model a bounded, clearly labeled set of passages with stable IDs and source metadata. Require citations to refer to those actual IDs. Do not accept citations the model invents or link an answer to a document that was never in the retrieved set. When two retrieved passages disagree, the system should inspect version and authority rules before summarizing either as settled fact.

Suppose the fictional assistant finds a service notice dated May and another dated August. The August notice replaces a part number only for machines with a particular manufacturing revision. The correct answer might say that an R2 machine uses the newer part while an R1 machine retains the older recommendation. A simplistic “use the latest document” rule would be wrong. This case tests not just retrieval ranking but interpretation of temporal and product-specific qualifications. A human reviewer should be able to find the cited sections without relying on the model’s explanation.

Keep tool-result semantics explicit when an AI agent invokes search. A timeout, missing index, empty result set or access-denied error is not a source passage. The agent should receive a structured failure rather than a blank text string that could be mistaken for “no policy exists.” Depending on the failure, it can retry within a bounded deadline, report the absence of evidence or escalate. It must not convert an unavailable service into a confident answer.

Evaluate retrieval and answers separately

Use two distinct test layers. Retrieval quality asks whether the correct passages appear in the candidate set under the user’s permissions. Track relevant-hit measures, misses and misranked candidates; annotate cases where the corpus itself is incomplete. Answer quality asks whether the generated text is supported by those passages, respects exceptions and abstains when evidence is insufficient. A persuasive answer can be wrong even if retrieval succeeded, and a correct-sounding answer can conceal a serious authorization error.

Run a paired diagnostic for an obsolete policy answer. In test A, retrieval ranks the wrong policy because its metadata lacks an effective date; in test B, the correct policy was retrieved but the model cites an outdated version. Repair indexing and filtering for A, and strengthen evidence packaging and answer verification for B. Someone using third-party AI-103 retrieval scenarios should be able to identify the different traces before choosing a fix. Semantic similarity alone proves neither access control nor source currency.

Changes should be regression tested. A new chunking approach may improve long narrative questions while splitting tables incorrectly. A vector embedding refresh can alter nearest neighbors. A revised semantic configuration may improve general wording but bury rare exact identifiers. Compare before-and-after results on an unchanged holdout set and inspect failure categories; do not report improvement from one aggregate score while critical product-code questions regress.

Operationally, watch ingestion failures, document age, permission synchronization, index size, latency and search-service cost. Pair these with application traces so an operator can distinguish low-quality evidence from a model behavior change. Establish a rollback process for index schema changes and a reindex plan when the source structure changes. If search becomes unavailable, decide whether the assistant should stop, use a safe cached answer with a clearly stated freshness limit, or escalate to human support. For a policy assistant, silence is sometimes safer than a stale answer.

Turn the retrieval design into an exam-ready exercise

Create a small fictional corpus of a dozen policy and equipment records, including two revisions, one restricted manual, a scanned table and an intentionally missing answer. Design fields for text, vector representation, document ID, section, effective date and allowed readership. Run keyword, vector and hybrid queries for the same ten questions. Record the top candidates and whether they were eligible for the user’s identity before sending any passage to a model.

Then ask the application to answer with source identifiers and a confidence limitation. Test a code with one changed suffix, an ambiguous policy date, an expired permission, a corrupt OCR value and a search timeout. Explain whether each failure belongs to ingestion, index construction, retrieval, authorization, evidence packaging or generation. This decomposition builds the judgment that the AI-103 exam’s agentic and information-extraction objectives require without inventing actual exam questions or treating an unexecuted sample as a validated production deployment.