Skip to main content

Web sources, teaching Arbyn from pages already on your site

Point Arbyn at a URL on your own site, an About page, a sizing guide, a store policy, and it reads and indexes the page, with an option to keep it automatically refreshed.

Written by Joseph

If your store already has a sizing guide, a shipping page, or an About Us that answers questions customers ask, you don't need to retype any of it. Point Arbyn at the URL and it reads the page directly.

What it does

You give Arbyn a URL from your own site, and it reads the page, extracts the actual content, and indexes it the same way an uploaded document would be. You can optionally have it re-check the page weekly, so if you update the page later, Arbyn's copy stays current without you having to re-add it.

Where to find it

Knowledge → Web sources.

Setup (step by step)

  1. Open Knowledge → Web sources.

  2. Click to add a new source and paste the page's full URL.

  3. Choose whether to turn on auto-refresh for that page. On means Arbyn re-checks it roughly weekly and updates its copy if the page has changed. Off means it's a one-time snapshot.

  4. Save. Arbyn fetches the page immediately, and the page's title, detected language, and how many searchable pieces it was broken into all show up in the list right away.

  5. If the fetch fails (the page moved, or returned an error), that's shown clearly in the list rather than silently indexing nothing; you can retry once the URL is fixed.

  6. Click Crawl again on any existing source any time you want to force a fresh read immediately, rather than waiting for the weekly auto-refresh.

How it works

Adding a web source fetches exactly one page, the one at the URL you gave it, not your whole site. If you want several pages covered (a sizing guide and a shipping page and an About page), add each one individually. The content is converted to plain text, broken into the same kind of searchable pieces as an uploaded document, and made available to Arbyn immediately, indexed under that page's URL so a later re-crawl of the same page replaces its content cleanly rather than creating a duplicate.

Auto-refresh keeps a page from going stale without you remembering it. With it on, a background check re-fetches the page on a roughly weekly cadence and replaces the indexed content if anything's changed, so an update to your actual sizing guide reaches Arbyn without you having to revisit this page and manually re-add the URL. With it off, the page is captured once and stays exactly as it was at that moment until you manually crawl it again.

A stale badge tells you when it's worth a fresh look. Any source that hasn't been successfully re-crawled in the last couple of weeks is flagged as stale in the list, whether or not auto-refresh is on, so you always know at a glance which pages might no longer reflect what's actually live on your site.

This is read-only from your side. Once a page is indexed, you're not editing the content here directly, if the page itself needs to change, you edit it on your actual storefront and then either wait for auto-refresh or click Crawl again. This page is a mirror of what's already public on your site, not a separate place to author content.

Good to know

  • One URL is one page, not a whole-site crawl. Add each page you want covered individually.

  • Auto-refresh runs about weekly, not instantly; use Crawl again if you need Arbyn to see a change sooner.

  • A failed fetch is shown clearly, never silently indexed as empty content, so you always know whether a source actually succeeded.

  • This mirrors your live site. To change what Arbyn knows from a web source, change the actual page, this list doesn't let you edit page content directly.

  • Deleting a source here doesn't touch your actual website. It only removes Arbyn's indexed copy.

FAQ

Can I add my whole website at once instead of page by page? No, each web source is one specific URL. Add every page you want Arbyn to know individually.

How often does an auto-refreshed page actually update? Roughly once a week. If you need Arbyn to see a change immediately, use Crawl again on that source rather than waiting.

What happens if a page I added gets moved or deleted? The next crawl attempt (automatic or manual) will fail and that's shown clearly in the list, giving you a chance to fix or remove the source rather than leaving Arbyn silently answering from stale or missing content.

Does turning off auto-refresh mean the page is gone from Arbyn's knowledge? No, it just means the snapshot from when you added it stays as-is until you manually crawl it again; turning auto-refresh off doesn't remove anything already indexed.

Did this answer your question?