Most SEO advice is tactics without a model. People copy a checklist, the checklist stops working, and nobody knows why. This guide gives you the model first.
Three jobs, one system
A search engine does three things, and almost every mistake comes from confusing them.
- Crawl. A robot fetches URLs it knows about.
- Index. The fetched content is parsed, stored and understood.
- Rank. For a given query, the engine chooses and orders the pages it thinks will help most.
If a page is not crawled, nothing else matters. If it is crawled but not indexed, ranking is impossible. If it is indexed but ranks on page four, you have a relevance problem, not a technical one. Diagnosing which of the three is broken is the whole job.
What a crawler actually does
A crawler starts from a list of known URLs: your sitemap, links from other sites, and links it has seen before. It fetches a URL, reads the HTML, and follows the links it finds.
Three consequences follow:
- Links are still the roads. A page with no internal links is a page nobody finds, including Google.
- Depth matters. Pages four clicks from the homepage are crawled less often than pages one click away.
- robots.txt is a door, not a lock. It asks crawlers not to fetch. It does not remove a URL from the index, and it does not stop anyone else from reading the page.
You can see exactly what a crawler sees. Fetch a URL, strip the JavaScript, and read the HTML. If your main content is not there, you have a rendering problem, not a content problem.
Rendering: where good pages disappear
Modern sites build HTML in the browser with JavaScript. Crawlers can render JavaScript, but rendering costs them time and money, so it happens later and less reliably than a plain HTML fetch.
The practical rules:
- Ship the primary content in the server-rendered HTML.
- Never put navigation, canonical tags or internal links behind a click or a scroll that requires JavaScript.
- If you must render on the client, test with the URL Inspection tool and look at the rendered HTML, not the live page in your browser.
A page that looks perfect to you and blank to a crawler is not a design problem. It is an indexing problem.
How ranking actually works
There is no single ranking factor, and anyone selling you "the algorithm" is selling certainty they do not have. What is well established:
- Relevance. Does the page address the query, in the language and form the searcher expects?
- Quality and trust. Is this a credible source for this subject? Increasingly this is judged through entities and corroboration, not just on-page words.
- Usability. Does the page load, work on a phone, and not fight the reader?
- Links and mentions. Do other credible sources point at this page or this brand?
Notice that three of those four are not things you can "optimise" in a plugin. They are editorial and structural decisions.
What actually moves the needle
In rough order of impact for most sites:
- Fix what blocks indexing. Noindex tags, broken canonicals, robots rules, orphan pages.
- Build pages that answer a real query better than the current results do. Not longer. Better.
- Make the site's structure legible. Clear sections, sensible URLs, internal links that describe what they point at.
- Earn mentions. Links, citations, reviews, directory listings, press. Anything that corroborates that you exist and know the subject.
- Then optimise the details. Titles, descriptions, schema, image alt text.
Most people start at step five. It is the least valuable place to start.
A first-week checklist
If you are starting from nothing, do this before anything clever:
- Verify the site in Search Console and Bing Webmaster Tools.
- Submit an XML sitemap containing only canonical, indexable URLs.
- Confirm robots.txt does not block anything you want indexed.
- Crawl the site with a free crawler and fix every 404 that has internal links pointing at it.
- Check that every important page has a unique title and one H1.
- Make sure the primary content is in the HTML source.
- Pick five queries you genuinely want to rank for and read the current top five results. Ask what they do that you do not.
That is the foundation. Everything else in this encyclopaedia builds on it.