Skip to main content

Knowledge analytics: what Arbyn actually used to answer

Which of your knowledge sources Arbyn actually cites, how confident the matches are, and which content you have that it has never once used.

Written by Joseph

Every other page in your Knowledge Center is about what you have fed Arbyn. This one is about what it actually reached for. It shows which documents, policies, FAQs, web pages, product data, and learned facts got cited in real customer conversations, how well they matched what the customer asked, and just as importantly, what you have that Arbyn has never once pulled from.

What it does

Arbyn logs every real retrieval it performs, not a sample. Each time it searches its knowledge to answer a customer, that search and its result are recorded, so this page can show real usage rather than an estimate. It rolls that log up into four numbers, two charts, and a searchable table of your most-used content.

Where to find it

Go to Knowledge, Analytics.

How it works

The four numbers at the top

  • Tool calls, over the last 7, 30, or 90 days (your choice). How many times Arbyn actually searched its knowledge to answer a customer, in that window.

  • Avg match score, from 0.0 to 1.0. How close the pieces of content Arbyn found were to what the customer actually asked. Higher means Arbyn is finding content that genuinely fits the question; lower means it's often settling for something loosely related.

  • Top source type. Whichever kind of knowledge (documents, FAQs, policies, web pages, your catalog, or facts your team taught it) got cited the most in the window.

  • Source types used, shown as a fraction like "3 of 5". How many distinct kinds of content Arbyn has actually cited, out of how many kinds you have. If you've uploaded documents and policies but only policies ever get cited, this reads 1 of 2, and that's your first clue something in the other type isn't being found.

Match quality distribution

Once you have real citations, a chart breaks them into three bands: Strong (0.70 and above), Moderate (0.40 to 0.69), and Weak (below 0.40). This is what the average score is made of. A store with mostly Strong matches has content that closely fits what customers ask; a store leaning Weak has content Arbyn can find but that doesn't quite answer the question, which is usually a sign the content itself needs to get more specific rather than that anything is broken.

Source-type coverage

A second chart shows every content family you actually have, including any that have never been cited even once, so a real gap in coverage is visible rather than hidden by only showing what's working. This deliberately leaves out Arbyn's own built-in general knowledge (the topics it starts with before it ever reads your store) since that isn't your content; this chart is specifically about what you've added.

The health banner

Right under the top numbers, one of two things shows depending on your data:

  • If your average match score is genuinely low, a warning names it and points you to add more sources or improve your product details.

  • If some content type you have has never been cited, a warning names which ones, so you know exactly where to focus.

  • If neither is true, a plain explainer replaces the warning instead of a false alarm, so a healthy store isn't nagged for no reason.

The most-cited content table

Below the charts, every piece of content that's actually been cited, sorted by how often (you can also sort by average match score, and filter the time window). Each row shows a short preview, which source type it came from, how many times it's been cited, and its average match score. Tap a row's View action to open the full text, along with up to five of the most recent real conversations that cited it, each with a direct link to open that conversation in your inbox.

If a piece's average score is below 0.40, the detail view adds a specific note that it's being used but only loosely matching, and suggests rewriting it or adding a more specific source instead.

Good to know

  • This measures real usage, not potential usage. A document sitting unused in Documents doesn't show up here until Arbyn has actually cited it in a real conversation.

  • A piece of content can be re-indexed or removed and keep its citation history. If the underlying content has since changed, the table still shows the row (so your usage history isn't lost) with a note that the content has been updated rather than the old text.

  • The coverage chart only looks at your own content. Arbyn's built-in general knowledge is excluded on purpose, since the point of this page is to show you what's happening with what you've added.

  • This page needs real conversation volume to say much. A brand-new store with very few conversations will mostly show low or empty numbers here, which reflects volume, not a problem with your setup.

FAQ

Why does the coverage chart show a source type at zero even though I have content there? It means Arbyn has that kind of content available but hasn't been asked a question it matched to, in the time window you're viewing. Try a longer window (90 days), or check whether that content actually answers a question a customer would realistically ask.

Can I see exactly which conversation used a specific answer? Yes. Open any cited row's View action, and if it's been used recently you'll see up to five of the real conversations that cited it, with a link straight into each one.

Does a low average match score mean Arbyn gave bad answers? Not necessarily. It means the content Arbyn found was a loose match for the question, which is a signal to improve your source material. Arbyn still only answers from what it actually retrieves; it doesn't invent an answer when the match is weak.

Did this answer your question?