Knowledge
Knowledge is the store of business facts — products, locations, people, policies — that grounds every AI feature and the generated llms.txt.
Knowledge is where you record the facts about your business that Spider's AI can't
infer from a crawl. It is the durable context that every AI feature and the generated
llms.txt are grounded in. It is authored once and reused everywhere, so you
describe your business a single time rather than re-explaining it in every prompt.
What it holds
You author facts grouped by category, such as:
- Products and services
- Locations and service areas
- People — the team, authors, and their credentials
- Policies — shipping, returns, guarantees, and the like
Reviewed state
Each entry carries a reviewed state, which keeps the store trustworthy over time. Facts can also be proposed by AI. An analysis or an agent working through the MCP server can suggest an entry it inferred from your site. Those proposals wait for you to accept them before they count as Knowledge. Nothing an AI proposes becomes grounding truth until a human reviews it, so the store stays accurate even as it grows automatically.
Why it matters
AI analyses, content generation, and the site's llms.txt all read from Knowledge.
The more complete it is, the more specific and correct the AI output becomes,
instead of guessing from page text alone. The llms.txt file it generates is the
emerging standard for guiding AI crawlers to your most important content. It
is only as accurate as the facts behind it.
Fill in Knowledge before running Content gaps or AI content workflows. Both read from it directly, so a complete store is the difference between suggestions specific to what you sell and generic topics inferred from page text.
