The Crawler

Spider crawls a whole site from your own machine. It discovers URLs through links and the sitemap, then audits every page it finds.

Two fetch modes

Each crawl can run in one of two modes:

• HTTP — fetches the raw HTML directly. Fast, and closest to how a source-only crawler sees the page.
• JavaScript rendering — loads each page in a real Chromium browser (Playwright), so client-rendered content, injected schema, and JavaScript-dependent issues are all captured.

Rendering mode is the right choice for modern single-page apps and any site where content is assembled in the browser.

Multi-device crawling

Spider can crawl in desktop, mobile, and tablet modes in a single pass. Site-level checks run once per crawl. Per-page checks run for each device, so you can compare how the same page scores across viewports. This is useful for Google's mobile-first indexing.

Cloaking detection

Spider can fetch a page both as an HTTP request and as a rendered browser view, then compare the two. Large differences between what a site serves a bot versus a real browser produce a cloaking score, flagging pages that may be showing search engines something different from users.

Concurrency and page caps

Spider crawls several URLs at once. Both the number of simultaneous requests (concurrency) and how many pages a single crawl can cover are set by your plan:

A crawl configured above your plan's limits is clamped down to them rather than rejected. A request for more pages or higher concurrency simply runs at the cap. You can start, pause, resume, and stop a crawl at any time without losing what has already been collected.

Scope and authentication

Restrict a crawl to a domain, include subdomains, or limit it to a path such as /blog. For sites behind a login, pass authentication as custom HTTP headers under Advanced → Network. An Authorization header (Bearer or Basic) or a Cookie header is sent with every request. Custom headers require the Tracker plan. The headers are stored in the project configuration on your machine and are sent only to the site being crawled.