Vora
Which questions must the first version answer?
Ask staff for questions from the channels the assistant will cover. Use authorized, appropriately redacted examples from support conversations, enquiries or call notes. Include questions that currently make staff check with someone else.
Write the expected answer beside each question and name its source. "It is somewhere on the website" is an unfinished entry. Record the actual page or document section, plus any condition that changes the answer.
For a hypothetical maintenance company, the first set might include whether it serves a postcode, which equipment it repairs, and how a customer requests a visit. A request for a guaranteed arrival time may belong outside that initial scope. Identifying that exclusion now keeps it out of the sales promise and the chatbot's answer set.
The first-client guide covers choosing a deliverable you can support. Knowledge review begins with the questions that deliverable creates.
Can the business supply evidence for each answer?
Use a readiness table with the client. The decisions below are recommended review actions, not a validated scoring system.
| Check | Evidence to inspect | If it is missing |
|---|---|---|
| Coverage | A source for each in-scope question, including important exceptions | Add an approved answer or exclude the question |
| Ownership | A named role that can approve and correct the answer | Assign an owner before relying on it |
| Authority | The source the business recognizes as controlling for this topic | Resolve competing documents |
| Applicability | Service, location, audience and effective period | Add the missing scope or ask for clarification |
| Readability | Extracted text retains headings, values and qualifying notes | Repair the document or extraction process |
| Access | The intended user may receive the retrieved information | Separate or restrict the source |
| Maintenance | A route from a business change to the assistant's source | Assign an update process and check it |
| Unanswered requests | Approved wording and a reachable person or queue | Build the fallback before expanding scope |
Do not average these rows into a reassuring score. A source set can perform well on most questions and still expose a restricted record. Treat a failure according to its effect on the proposed service.
Do staff agree on which answer is correct?
Ask two people who regularly handle the same request to identify its controlling source. Disagreement is useful discovery evidence. It may reveal an outdated page, a branch-specific rule or a practice that has never been written down.
Capture the disagreement without choosing the answer that sounds most plausible. A newer file may be a draft. An older policy may still apply to an existing service. The client needs to decide who has authority over that subject and how exceptions are approved.
Keep the detailed resolution in a small source-authority register. For readiness, the decision is simpler: can this question enter the approved answer set, or must it stay out until its owner resolves it?
If staff supply a missing answer verbally, write it down and have the appropriate owner approve it. A meeting note should not silently acquire the same status as an approved customer policy.
Will the source survive extraction?
Open the text the retrieval system will use. Compare it with the original document, especially around tables, footnotes and multi-column layouts. A PDF can look clear to a person while its extracted text places a condition far from the price or service it qualifies.
Consider an illustrative price sheet with "weekdays only" beneath a service table. The readiness check should establish whether that condition remains attached to the relevant service after extraction. If a retrieved passage gives the amount without the condition, the document needs repair or a different processing approach.
Microsoft's Azure AI Search documentation describes a Document Layout method that uses document headings and structure when forming chunks. This establishes that structure-aware processing is a documented approach; it does not establish the quality of a particular client's extracted files. Microsoft's document-layout guide explains the method and points new implementations to its newer Content Understanding approach.
Check ingestion status too. OpenAI documents that some vector-store file operations are asynchronous, so a submitted file can still be processing. Confirm completion before evaluating retrieval. OpenAI's retrieval guide describes the file-operation behavior.
Which information must stay outside public answers?
Inspect source access separately from answer quality. An internal handbook can mix public opening hours with staff contact details. A customer folder can contain both a general brochure and private correspondence.
Classify what the intended user may see before connecting sources. For a public chatbot, use an approved public answer collection where that is practical. For account-specific assistance, define the authenticated user's scope and enforce it in the data-access path.
Use test accounts with synthetic records, including a second account and an unauthenticated visitor where those roles exist. Ask for a known restricted fact and inspect what the system retrieved as well as what it displayed. A prompt that says "do not share private information" is not evidence that retrieval respected access restrictions.
Record a permission failure as a blocking issue for the affected scope. Rephrasing the answer does not settle whether the underlying source remains accessible.
What happens when the business changes its answer?
Choose one ordinary change and follow it through. For example, ask the service owner to describe how a new service-area boundary would reach the website, the approved source and the assistant.
Identify who approves the change, how the previous version leaves the active answer set, and who verifies the new answer. A process that depends on the agency noticing an unrelated email needs a more explicit notification route.
Prices, stock and appointment availability require their own treatment because they may change during a conversation. Mark those questions for the current-facts review, then assess whether the required system access is part of this project.
Also decide how the assistant handles questions with no answer. The fallback and handoff method helps define the boundary. An approved handoff can make a question manageable without adding it to the automated answer set.
What would justify starting the build?
Prepare a review pack containing the scoped question list, approved sources, access rules and unresolved items. Add a small controlled retrieval trial when the format or source structure is uncertain. Keep manual source review, successful ingestion and an acceptable generated answer as separate observations.
Use questions with different conditions: a standard request, an exception, an ambiguous location, a missing answer and a restricted fact. Ask a staff reviewer to compare each response with the agreed source. Record the retrieved passage and the part that passed or failed.
The result can be ready for the agreed scope, ready after narrowing that scope, or waiting for a named repair. Write down the excluded questions and what evidence would allow them back in.
Send the client that list with the build proposal. "Branch opening hours are approved; individualized quotes remain with the sales team" gives them a concrete service to accept and a visible boundary to discuss.