Building a knowledge base that can be trusted

When an assistant gives a wrong answer, the reflex is to look at retrieval. In many of the projects that go wrong, the technical chain is entirely correct and the answer is still false, because the document it cited was wrong when it was written. The failure is silent precisely because every upstream measurement passes: the right passage was found, the citation is accurate, the latency is fine, and the answer is confidently wrong.

The triage that tells you which failure you have

Before touching anything, sort the wrong answers by their citation.

A correct citation with a wrong answer is a material problem. Someone tuned the search engine for a week while the source line had been misprinted for years. A wrong or missing citation is a retrieval problem, and that is the one the models and the index can fix. A correct citation, a correct answer, and a user who still cannot act is a handover problem: the material is right and the workflow around it is missing.

This triage costs an afternoon and prevents the most expensive kind of wasted work, because the three failures look identical from the demo chair and have nothing in common underneath.

The trust pass, before the three steps

Most published playbooks start with role interviews and a directory. That order is wrong. A directory of documents nobody will vouch for is a well-organised way to be wrong.

Take the documents that the intended questions actually need — not the whole folder, a sample of what the first version will cite — and give each one four columns: who wrote it and which version; when it took effect; who signs it as currently correct; and what event would force a re-reading.

The signing rule matters more than the form. The signer must be the role that would be blamed if this answer turned out wrong: the product owner for a specification, the policy owner for a policy. Not IT, not the project sponsor, not "the team". A signature does not certify truth; it certifies that there is a person who can be asked again when the answer looks strange. Documents nobody will sign enter as reference only, are not citable, and the assistant must answer "no basis, transferring to a person" rather than improvise. That single rule is what separates a knowledge base from a search box over an archive.

The signal that this step was skipped is a person, usually the most experienced one, correcting the document in their head without noticing. Ask whether a given line matches what they do and the answer is "well, not exactly, everyone knows that". What everyone knows and no system contains is exactly where the assistant will be confidently, reproducibly wrong.

Then the three steps, with the gate in front of them

With the trust pass done, the familiar sequence works.

Role interviews reconstruct the handover order, not the org chart: who receives a case, who decides, who hands it back. A directory built on the org chart reproduces the org chart and hides the handoffs, which are where the work and the errors live. The directory should follow the flow of business, one folder per step, so that the retrieval unit matches the work unit.

Each role then gets one SOP, and the SOP carries the boundary inside it: who may quote a price, who must escalate, what the assistant may never answer on its own.

In the sixteen-person intellectual property agency, the customer formed an internal AI group and confirmed in writing that the flows ran end to end before any skill was built. That gate is the difference between a delivery and a rebuild. It is also the cheapest insurance available, because the alternative is discovering the disagreement after the code exists.

Two checks belong to the same step. Before adding documents, ask who will use the answer, what they will compare it against, and whether the unit is agreed: an organisation with a large archive and no agreed unit gets a faster way to produce two incompatible answers. And the written process is rarely the file that runs — the executing version is usually in a shadow system, and it is the one people act on.

Permissions belong to the material, not the prompt

Different roles may see different documents, and the boundary has to be enforced before retrieval — by identity and permission, not by a sentence asking the model not to reveal things. Test it with two accounts asking the same question and check the citations and the history as well as the answer. The data and security boundary is part of the knowledge base design, not a later hardening step.

Keeping it trustworthy after the three steps

Two habits keep the thing alive. First, keep a set of real historical questions and re-run them after every material change; a knowledge base is only as good as its answers to questions it was not built around. Second, log every miss as one of three types: not found, misused, expired. Misses that are not logged become the same miss again next quarter.

Anything unconfirmed is written as unconfirmed. Filling the gap with a plausible explanation is the fastest way to convert a known uncertainty into an unknown error. This is the same discipline the delivery method applies to acceptance: a state that has not been checked is recorded as unchecked, not rounded up.

The criterion

Take the five answers the system gives most often and try to find the person who will sign each one. If fewer than five have a signer, the knowledge base is not ready. It is a search engine over an archive, and it will be wrong in ways that no amount of indexing will fix.