A search engine must find a page, understand it and then decide when to show it. This guide explains each stage and what to check when a page does not appear.
Google describes three stages. Crawling is when automated programs find pages by following links and reading sitemaps, then fetch them. Indexing is when a page is analysed and stored in the index. Serving is when matching pages are chosen for a search. A page can drop out at any stage, and Google does not promise to crawl or index every page.
How search engines work in three stages
How search engines work comes down to three jobs done in order. A crawler finds and fetches pages. An indexing system analyses what was fetched and stores it. A ranking system picks stored pages that match a search and puts them in order. This guide uses Google as the example because its documentation is public and most business owners care about it first. Other search engines publish their own guidance and the details differ, so check each one's current documentation.
| Stage | What happens | What can stop a page | Where to look |
|---|---|---|---|
| Crawling | Google finds a web address and fetches the page | No link or sitemap entry pointing to it, a robots.txt rule, a login, a server error | URL Inspection tool |
| Indexing | Google analyses the page and stores it in its index | A noindex rule, a duplicate of another address, content judged low quality | Page indexing report |
| Serving results | Google chooses and orders matches for a search | Other pages judged more relevant or helpful, or the wrong type of page for the search | Performance report |
Google also says it makes no promise to crawl, index or show a page, even when the page follows its Search Essentials. Payment does not change that. Google states that it does not accept payment to crawl a site more often or to rank it higher.
Stage one: crawling, how a search engine finds pages
There is no central list of every page on the web, so Google keeps looking for new and updated pages. Its documentation calls this URL discovery. Addresses are found mainly in two ways: by following links from pages Google already knows, and from sitemaps that site owners submit. A page that nothing links to and no sitemap lists is easy to miss.
Googlebot is the program that fetches pages. Google says an algorithmic process decides which sites to crawl, how often to return and how many pages to fetch from each. When Googlebot fetches a page it also renders it, running any JavaScript with a recent version of Chrome, so that it sees something close to what a visitor sees.
Some pages are left uncrawled by design or by accident. A site owner can disallow crawling with a robots.txt rule, and pages behind a login are out of reach because Googlebot does not sign in. Links matter too. Google says it can generally follow only links that are ordinary link elements with an address, so a menu that works only through script events may hide pages from it. Every page you care about should be linked from at least one other page on the site.
Crawling is not a verdict on quality. A crawled page has only been read.
Stage two: indexing, how pages are understood and stored
After fetching a page, Google analyses its text, images and video files and stores what it learns in the Google index, a very large database. It also works out whether the page duplicates another. Similar pages are grouped and the most representative one is chosen as the canonical, which is the address that may be shown in results. If one page is reachable at several addresses, Google makes that choice for you unless you give clear signals.
Indexing is never certain. Google lists common blockers: low-quality content, robots meta rules that disallow indexing, and site designs that make indexing difficult. It also uses the mobile version of a page for indexing, so content that is missing on mobile devices may be missing from the index.
Crawling and indexing are separate steps, and mixing them up causes a common mistake. Blocking a page in robots.txt stops crawling but does not remove the address from search, because other sites may still link to it. A noindex rule is the tool for keeping a page out of the index, and it works only if the page is not blocked from crawling, since Google has to fetch the page to read the rule.
Stage three: ranking, how results are chosen
When someone searches, Google's systems look through the index for matching pages and return the results they judge to be the most relevant and of the highest quality. Google says relevance depends on many factors, including the searcher's location, language and device, and that the features on the results page change with the query. Its guide to ranking systems adds that several systems work together, mostly at page level.
Two practical points follow. First, there is no single position that everyone sees. Results in Dubai can differ from results in Tokyo for the same phrase, and Search Console's average position blends many searches, so it is better read as a trend than as a fixed rank. Second, ranking depends on what the searcher wants, which is why the words you choose for each page matter. The keyword research guide shows how to pick the phrases a page should be written for.
Why is my page not showing in search?
Start by working out which stage failed, because the fixes differ. Searching for your site with the site: operator is a quick first look, but Google notes it does not necessarily list every indexed address. A more reliable check is the URL Inspection tool in Search Console. For one address it reports whether crawling is allowed, whether indexing is allowed, what happened when Google fetched the page, and which address Google chose as the canonical.
| What you see | What it usually means | What to do |
|---|---|---|
| Not on Google, and no last crawl date | Google has not found the page | Link to it from other pages and submit a sitemap, then use Test live URL and Request indexing |
| Discovered - currently not indexed | Google knows the address but has not crawled it yet | Make sure important pages link to it and the server responds reliably, then allow time |
| Crawled - currently not indexed | Google fetched the page and has not indexed it for now | Make the page more useful, remove near-copies and link to it. Google says there is no need to resubmit |
| Excluded by noindex tag | A rule tells Google to leave the page out | Remove the rule if the page should be listed |
| Blocked by robots.txt | Crawling is not allowed | Change the rule if the page should be crawled |
| Duplicate, Google chose a different canonical | Another address stands in for this page | Check which address Google picked and whether that suits you |
| Indexed, but missing for your search | The page is stored, so the problem is ranking | Check that the page matches what that searcher wants, then improve it |
Give new pages time. Google says it cannot predict or promise when or whether a URL will be crawled or indexed, and that asking for a recrawl of the same address many times does not make it faster. Its help pages also note that some pages are left out for good reasons, such as redirects and duplicates, so an excluded address is not always a fault.
What you can do to help each stage
You cannot force a page into the results, but you can remove obstacles and make the page clear.
- For crawling, link every important page from at least one other page, and check that robots.txt and logins do not block it.
- For indexing, give each page its own purpose, avoid near-copies, and remove any stray noindex rule left over from the build stage.
- For ranking, write for the searcher's need first, then use a clear title and clear headings. The on-page SEO checklist covers one page at a time, and the guide to heading tags explains how headings give a page an outline.
These steps make a page easier to find, read and judge. Google does not promise a result for any of them.
How to check each stage yourself
- Verify your site in Search Console so you can see Google's own reports.
- Paste a key address into the URL Inspection tool and read the crawl, indexing and canonical fields.
- Open the Page indexing report and read the reasons listed for pages that are not indexed. Decide which are expected and which are faults.
- Open the Performance report to see which searches already show your site and which pages appear.
- Fix the biggest cause first, then look again after Google has had time to return to the page.
The guide to running a basic SEO audit yourself walks through these checks in order. If the findings are more than you want to handle alone, the search engine optimisation service page describes the SEO support Liftup offers.
Questions and answers
How does a search engine find my pages?
Google finds pages mainly by following links from pages it already knows and by reading sitemaps that site owners submit. A page with no links pointing to it and no sitemap entry is easy to miss, so link every important page from at least one other page.
What is the difference between crawling and indexing?
Crawling is fetching a page. Indexing is analysing it and storing it in the search index so it can be shown. A page can be crawled and still not indexed, for example if it carries a noindex rule or Google treats it as a duplicate.
Why is my page not showing in search?
The cause is usually one of a few: Google has not found or crawled the page yet, a rule blocks crawling or indexing, Google treats the page as a duplicate of another address, or the page is indexed but not ranking for the search you tried. The URL Inspection tool in Search Console shows which one applies.
Can I pay to be crawled or ranked faster?
No. Google states that it does not accept payment to crawl a site more often or to rank it higher in its organic results. Paid advertising on Google is a separate product from organic search.
How long does a new page take to appear in search?
Google says it cannot predict or promise when or whether a URL will be crawled or indexed. Requesting indexing in the URL Inspection tool can help Google notice a page, but asking again and again for the same address does not speed it up.
Does being indexed mean my page will rank?
No. Indexing means the page is stored and eligible to be shown. Whether it appears for a search, and where, depends on how relevant and useful Google judges it to be for that searcher, including their location, language and device.