SEO for Developers: Rendering, Status Codes, Tags, Performance and AI Crawlers

SEO for developers is the part of search optimization that lives in code: templates, routing, rendering, headers and performance. Marketers choose keywords and write content, but a wrong status code, a client-only route or a stray noindex can erase that work in one deploy. Google's own JavaScript SEO basics describe three phases (crawling, rendering, indexing) and each one has failure modes that only developers can fix. This guide covers them in the order they break.

What part of SEO does a developer own?

How should pages be rendered for search engines?

Googlebot renders JavaScript, but in a second step that can be delayed, and Google itself writes that "server-side or pre-rendering is still a great idea" because it is faster for users and crawlers and because not all bots run JavaScript. That last point now matters more than ever: in its December 2024 analysis The rise of the AI crawler, Vercel found that "none of the major AI crawlers currently render JavaScript", naming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot among others. Gemini, which uses Google's infrastructure, is the exception.

Two rules apply whatever the framework. Links must be <a> elements with an href attribute: Google states it can only discover links in that form, so buttons with click handlers and fragment-based routes are dead ends. And each route needs its own URL, without #, that returns its content directly. Our guides to JavaScript SEO, single-page application SEO and Next.js SEO go into framework-specific details.

Which HTTP status codes matter for SEO?

Status codes are instructions to crawlers. Google's documentation on HTTP status codes and network errors describes how each family is handled:

• 200: the page is eligible for indexing. If the content looks like an error or an empty page, Search Console reports a soft 404.
• 301 and 308: permanent redirects, which Google treats as a strong signal that the target should be processed. Use them for moved content and domain changes.
• 302 and 307: temporary redirects, a weak signal. Do not use them for permanent moves.
• 404 and 410: the page does not exist. Google stops using the URL over time. Return a real 404, not a 200 with "not found" text.
• 429 and 5xx: Google's crawlers slow down crawling, and URLs that persistently return server errors are eventually dropped from the index.

Check what your server really sends with a HTTP headers checker, and keep redirect chains to a single hop: our article on SEO redirects covers migration patterns.

Which tags must be in the HTML head?

Every indexable page needs a unique title and description, a self-referencing canonical, and, on multilingual sites, hreflang annotations. Keep them in the server response:

<title>Running shoes for flat feet | Example</title>
<meta name="description" content="Tested picks, sizes and prices.">
<link rel="canonical" href="https://example.com/running-shoes-flat-feet">
<link rel="alternate" hreflang="fr" href="https://example.com/fr/chaussures-pieds-plats">
<meta name="robots" content="index, follow">

The canonical tag must point to the URL you want ranked, with the exact protocol, host and trailing slash convention your site uses.

How should URLs and internal links be built?

Search engines index URLs, so the URL scheme is an SEO decision. Use lowercase, hyphen-separated, readable paths; pick one trailing slash convention and redirect the other; and keep one URL per piece of content. Filters, sorting and tracking parameters multiply URLs without adding content: canonicalize them to the clean version, and do not link to parameter combinations you do not want crawled.

Internal links carry both discovery and importance. Render navigation, breadcrumbs and related-content blocks on the server, give paginated lists plain links to each page, and avoid infinite scroll without paginated URLs behind it. A product or article that is only reachable through site search or a JavaScript filter is, for a crawler, an orphan page.

How do robots.txt and sitemaps fit in?

robots.txt controls crawling, not indexing: a blocked URL can still appear in results if other pages link to it. To keep a page out of the index, let it be crawled and return noindex. Generate the XML sitemap from your routes or database, list only canonical 200 URLs, and update lastmod only when content really changes. Test your rules with a robots.txt checker before each release, and never deploy a staging robots.txt with "Disallow: /" to production.

How does performance affect rankings?

Google uses Core Web Vitals as part of its page experience signals. According to web.dev, a page is good when it meets these thresholds at the 75th percentile of real page loads, on mobile and desktop:

INP replaced First Input Delay as a stable Core Web Vital in 2024. Use field data (Search Console, the Chrome UX Report) to judge, and lab tools to debug. Our glossary entry on Core Web Vitals lists the tools.

How do you add structured data?

Use JSON-LD in a script tag, rendered on the server, describing what is visible on the page:

<script type="application/ld+json">
{"@context": "https://schema.org", "@type": "Article",
 "headline": "Running shoes for flat feet",
 "datePublished": "2026-10-01",
 "author": {"@type": "Person", "name": "Jane Doe"}}
</script>

Generate it from the same data as the page so the two never diverge, validate it in CI, and avoid injecting it through a tag manager. A schema generator gives a starting template for common types.

How do you make a site readable by AI assistants?

AI visibility starts with access. Decide which AI user agents you allow (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and others) and write explicit rules in robots.txt; our guide to robots.txt for AI crawlers explains the difference between training and search bots. Then make sure the text, prices, specs and FAQ are in the initial HTML, use semantic headings and tables rather than images of text, and keep pages light enough to fetch quickly. An llms.txt file can summarize your key pages for AI tools, but no major assistant has confirmed relying on it, so treat it as optional.

Sorank is our product: an autopilot SEO and GEO software. Developers mostly meet it through its integrations: it publishes articles to WordPress, Shopify, Webflow or Wix, or to any custom stack through a webhook, as described on the installation page. Its SEO chat answers questions about your site in conversation, and AI mention tracking checks every week whether ChatGPT, Claude and Gemini cite your brand.

The short version: ship complete HTML, honest status codes and correct head tags from the server, then measure performance on real users. That serves Googlebot and the AI crawlers that do not run JavaScript. To fill a well-built site with content that ranks, see Sorank's plans, from $49 per month with a 3-day free trial.