What actually belongs in your chatbot's knowledge base (and what doesn't)

The instinct when setting up a new chatbot is to point it at everything — the whole help center, every product page, every internal wiki article that seems remotely related — on the theory that more content means better answers. In practice, this usually backfires. Retrieval finds the most relevant chunks of whatever it's given, and if a third of what it's given is outdated, contradictory, or written for a completely different audience, some fraction of retrieved context on any given question will be exactly that.

Start with what customers actually ask, not what you have written. Your existing support tickets, if you have them, are the single best source for this — they tell you the real distribution of questions, which is usually narrower and more repetitive than a full help center implies. A handful of Q&A pairs written directly for your ten most common questions will outperform an entire lightly-curated wiki, because every one of those ten answers is exact, current, and written for the question it's answering.

Crawled website content earns its place for anything that changes on its own and that you'd otherwise have to remember to re-upload — pricing pages, feature lists, a changelog. The advantage of a crawl over a manual upload is that re-crawling picks up whatever's live on the page right now, so a pricing update or a new feature announcement flows through without a separate step. The tradeoff is that crawled pages often carry marketing language and navigation cruft alongside the actual facts, which can dilute what gets retrieved. Uploaded documents and hand-written Q&A pairs are the tools for content that's precise and doesn't change often — policies, procedures, anything where the exact wording matters.

Internal documentation is worth a harder look before it goes in. A wiki page written for your own support team, full of internal shorthand and assumptions about context a customer doesn't have, can produce answers that are technically drawn from a true source but confusing or wrong-sounding when surfaced directly to a visitor. If internal docs are the best source of truth for something customers ask about, it's usually worth rewriting the relevant parts in customer-facing language rather than uploading the internal version as-is.

Outdated content is the most common quiet failure mode. A knowledge base that hasn't been touched since launch will still retrieve confidently — the model doesn't know the pricing page it's reading from is six months stale, it just answers from what's there. The fix isn't complicated, just easy to neglect: whenever a policy, price, or feature changes, re-crawl or re-upload the relevant source and re-embed. Treat that step as part of shipping the change, not an afterthought.

A useful audit, once a bot has been live for a few weeks: pull the conversations where it clearly struggled or where a visitor asked a follow-up that suggests the first answer missed the mark. That list is a direct map of gaps in the knowledge base — questions being asked that nothing currently answers well. Closing those gaps one at a time, driven by real conversations instead of a guess at what might be useful, is a far better use of time than trying to anticipate every possible question up front.

The overall principle is closer to curating a reference shelf than filling a warehouse: fewer, more precise, more current sources beat a comprehensive but stale and noisy one, every time retrieval has to pick between them.

Want the full picture? Read our guide on ai customer support agent that answers from your own content.