Technical SEO Basics: Indexing, Canonical Tags and Sitemaps

Content and links get most of the attention in SEO, but none of it matters if search engines cannot find, read and index your pages. Technical SEO is the foundation that makes this possible.

This guide explains the main concepts in plain English. If you are new to SEO, start with does your business website need SEO?.

How Google finds pages: crawling and indexing

Google's process has three broad stages, described in its guide to how Search works:

1. —Crawling: automated software discovers pages, mainly by following links and reading sitemaps.
2. —Indexing: Google analyses the page and decides whether to store it in its index.
3. —Serving: when someone searches, Google selects relevant indexed pages to show.

A page that is not indexed cannot appear in search results. Being crawled does not guarantee being indexed.

Checking indexing

Google Search Console shows which pages are indexed and why others are not. The URL Inspection tool checks individual pages. Common reasons pages are not indexed include duplicate content, noindex instructions, redirects, errors and low-value pages.

robots.txt: controlling crawling

The robots.txt file at the root of your site tells crawlers which areas not to crawl. It is useful for blocking admin areas or search results pages.

Important: robots.txt controls crawling, not indexing. Google explains in its robots.txt introduction that a blocked page can still appear in results if other pages link to it. Never block pages you want to rank.

noindex: keeping pages out of results

A noindex instruction tells search engines not to include a page in results. It suits thank-you pages, internal tools and private drafts. For noindex to work, Google must be able to crawl the page and see the instruction, so do not also block it in robots.txt.

A common error is a development site's noindex being left in place at launch. See our website redesign SEO checklist.

Canonical tags: handling duplicates

The same content can often be reached at several addresses, for example with tracking parameters or with and without a trailing slash. A canonical tag tells search engines which version you prefer.

• —Every important page should usually have a self-referencing canonical.
• —Canonicals are a strong hint, not a command. Google may choose a different version, as its guidance on canonicalisation explains.
• —Pointing many different pages to the homepage as canonical is a mistake.

Redirects: moving pages properly

When a page moves permanently, a 301 redirect sends visitors and search engines to the new address and helps transfer existing signals. Points to remember:

• —Redirect to the most relevant equivalent page.
• —Avoid chains of several redirects.
• —Update internal links to point directly to the new URL.

XML sitemaps: listing your important pages

An XML sitemap lists the URLs you want search engines to know about. Google's sitemaps overview explains that sitemaps help discovery, particularly on larger sites or new sites with few links.

A good sitemap:

• —Includes only live, indexable, canonical pages.
• —Excludes redirected, noindexed and error pages.
• —Updates when pages are added or removed.
• —Is submitted in Search Console.

A sitemap does not guarantee indexing.

Status codes in brief

A "soft 404" is a missing page that returns a 200 status. Search Console flags these.

JavaScript websites

Many modern websites build their pages with JavaScript. Google can render JavaScript, but it adds complexity. It is worth confirming that titles, headings, main content and links are available in the rendered page, using Search Console's URL Inspection tool.

Internal linking and site structure

Search engines discover and understand pages partly through internal links. Important pages should be reachable within a few clicks and linked from relevant content. Our article on local SEO landing pages and search architecture covers structure for multi-location businesses.

Page experience

Speed and stability also form part of technical health. See Core Web Vitals explained.

A technical SEO checklist

• —Important pages indexed in Search Console.
• —robots.txt does not block important content.
• —No unintended noindex tags.
• —Self-referencing canonicals on key pages.
• —Moved pages use single 301 redirects.
• —XML sitemap lists only indexable pages.
• —No broken internal links.
• —Each page has a unique title and one H1.

Next steps

Our Search Engine Optimisation service covers technical foundations alongside content and local search, and our Website & SEO Audit checks these issues across your whole site.