Screaming Frog SEO Spider: How to Set It Up and Run a Full Audit

The Screaming Frog SEO Spider runs on Windows, macOS and Linux and works from your computer, so its speed depends on your machine's memory. The vendor's user guide documents every setting. This guide covers what matters for a first useful audit.

What does Screaming Frog SEO Spider do?

It starts from a URL, follows every link it finds and records what a bot sees on each page. This process is called crawling. For each URL you get:

• the HTTP status code (200, 301, 404, 500...) and the response time;
• the title tag, meta description, H1 and H2, with their length;
• canonical tags, meta robots and X-Robots-Tag directives, hreflang;
• internal and external links, anchors, crawl depth;
• images, their size and alt text;
• structured data, word count and page size.

The value comes from the view at scale: 10'000 pages sorted by issue in a few minutes, where a manual check would take weeks.

Free version or paid licence: which one do you need?

The free version is enough for a small site or a quick check of a section. As soon as a site passes 500 URLs, or you need to render JavaScript or monitor over time, the licence pays for itself.

Which settings should you change before the first crawl?

1. Choose the crawl mode

Spider mode crawls from a start URL. List mode crawls only the URLs you paste or upload, useful to check a sitemap, a list of old URLs before a migration or a set of backlink targets.

2. Use database storage

In the settings, switch from memory to database storage on an SSD. Large crawls become possible and are saved automatically.

3. Turn on JavaScript rendering if the site needs it

If content or links are injected by JavaScript (React, Vue, some Shopify or Webflow themes), set rendering to JavaScript. The tool then uses a headless Chromium browser, like Google's renderer. Compare the raw HTML and the rendered HTML to spot content that only exists after rendering.

4. Connect the APIs

Connect Google Search Console, GA4 and PageSpeed Insights. Clicks, impressions and Core Web Vitals then appear next to each URL, and URLs found in those sources but not in the crawl reveal orphan pages.

5. Set up custom extraction

With XPath, CSS selectors or regex, extract any element: publication date, author, price, stock status, a schema type. Handy to audit templates at scale.

6. Respect the server

Lower the speed (threads or URLs per second) on fragile hosting, and set the user agent to Googlebot only if the site does not block unknown bots.

How do you run the essential audits step by step?

Google explains how it treats HTTP status codes and canonical URLs: read both before deciding what to redirect or consolidate.

How do you export and share the results?

Every tab exports to CSV, Excel or Google Sheets. Use Bulk Export for combined data (all inlinks to 4xx pages, all images without alt text) and the Reports menu for redirect chains or canonical errors. For a client or a developer, export only the filtered list of issues with the page URL and the source of the problem: a list of 40 broken links with their source pages gets fixed, a 10'000 row file does not. With a licence, schedule a weekly crawl and use crawl comparison to see what changed after each release.

How does a crawl help with SEO and AI visibility?

A crawler sees your site the way Googlebot, Bingbot and the AI crawlers (GPTBot, ClaudeBot, PerplexityBot) do. A page they cannot reach, that returns an error or that sits seven clicks deep wastes crawl budget and has little chance of being indexed by Google or retrieved by ChatGPT and Perplexity. A clean crawl (200 status codes, one canonical, shallow depth, real internal links) is the base of any technical SEO audit, before content and links.

For one page at a time, you do not always need a crawler: the free on-page SEO audit and the free Sorank Chrome extension show the fixes of the page you are viewing in a few seconds.

Screaming Frog SEO Spider finds the technical problems of a whole site in minutes; the work that brings traffic starts when you fix them in order of impact. Once the base is clean, Sorank publishes optimized, internally linked articles on autopilot so the site keeps growing on Google and in the AI.