Canonical Tags and Duplicate Content: When and How to Use Them
Canonical tag explained: how Google handles duplicate content, when to use rel=canonical, code examples, store scenarios and mistakes to avoid.
Key takeaways
- Pages not indexed are not always a problem: some are excluded on purpose (noindex, redirects, duplicates with a canonical). Only the pages that should be in Google but are missing matter.
- The Page indexing report in Search Console tells you the exact reason for each URL. Start there, not from guesses.
- The statuses "Crawled – currently not indexed" and "Discovered – currently not indexed" usually point to weak content or signals, not a technical fault.
- Technical causes are the easiest to fix: noindex, robots.txt, wrong redirects, 4xx and 5xx errors, a canonical pointing elsewhere.
- Google does not guarantee that every page will be indexed. Prioritize pages that bring value and improve them before requesting indexing again.
Contents
- Where to find non-indexed pages and how to read the report
- The most common statuses and what to do about each
- "Crawled – currently not indexed" and "Discovered – currently not indexed"
- Technical causes, the easiest to fix
- How to diagnose a page, step by step
- How to prioritize when you have many URLs
- When it is normal for pages not to be indexed
- Common mistakes
- Limits: what you cannot control
- Conclusion: next steps
Pages not indexed are URLs that Google knows about but has not added to its search index, so they cannot appear in results. They are not always a problem: some are excluded on purpose. What matters is the pages that should be visible and are missing. The report in Google Search Console tells you the reason for each one, and below we go through the common ones with fixes.
Where to find non-indexed pages and how to read the report
In Search Console, open Page indexing. The report splits URLs into two groups: indexed and not indexed. For the non-indexed ones, Google shows a reason for each group of URLs, with a list of examples.
Three things to keep in mind:
- The total number of non-indexed pages is not a score. A site with filters, parameters and variants can have thousands of URLs correctly excluded.
- The reason matters more than the number. The same status can be normal for one page and a problem for another.
- The report updates with a delay. Do not expect to see today the effect of a change made today.
If you have not worked with the tool yet, the Google Search Console beginner's guide explains the basic reports. For issues of this kind, which belong to technical SEO, follow the steps below.
The most common statuses and what to do about each
The wording in the interface may vary slightly, but the meaning is the one in Google's documentation.
| Status | What it means | Frequent cause | What you do |
|---|---|---|---|
| Crawled – currently not indexed | Google fetched the page but did not index it | Thin or redundant content, weak signals | Improve the page, link to it internally, then wait |
| Discovered – currently not indexed | Google knows the URL but has not crawled it yet | New site, weak signals, slow server | Internal links, sitemap, check server speed |
| Excluded by 'noindex' tag | The page explicitly asks not to be indexed | Noindex set on purpose or by mistake | Remove noindex if the page should be indexed |
| Blocked by robots.txt | Crawling is forbidden | An overly broad Disallow rule | Fix robots.txt |
| Alternate page with proper canonical tag | The page points to another URL as canonical | Page variants, parameters | Usually nothing: it is correct |
| Duplicate without user-selected canonical | Duplicate with no declared canonical | Same content on several URLs | Declare your preferred canonical |
| Duplicate, Google chose different canonical than user | Google picked a different canonical | Content is not similar enough | Check for contradictory signals |
| Soft 404 | The page looks empty or like an error but returns 200 | Pages with no content, error messages | Return a 404 or add content |
| Not found (404) | The URL does not exist | Broken link, deleted page | Redirect if there is an equivalent |
| Page with redirect | The URL redirects elsewhere | Intentional redirect | Usually nothing: it is correct |
| Server error (5xx) | The server returned an error | Overload, configuration | Fix the server |
| Redirect error | A redirect loop or a chain that is too long | Redirect chain | Fix the chain |
Read the first column as a diagnosis, not as blame. Below we expand on the groups that raise the most questions.
"Crawled – currently not indexed" and "Discovered – currently not indexed"
These are the most common statuses and the most commonly misunderstood.
Discovered – currently not indexed means Google learned of the page (from a sitemap or a link) but has not crawled it yet. The documentation says crawling is usually postponed to avoid overloading the site. On a small, new site, it is often just a matter of time. What you can do: link the page from pages that are already indexed, add it to a clean sitemap and check that the server responds quickly.
Crawled – currently not indexed is more annoying: Google visited the page and decided, at least for now, not to keep it. The documentation notes the page may be indexed later without you resubmitting it. The reason usually lies in the page, not in the technology:
- thin, copied or very similar content;
- a page with no clear role or no answer to a real query;
- an isolated page with no relevant internal link;
- a site with many low-quality pages, which may weaken how Google views the site overall.
The fix is to improve the page: a direct answer, original information, examples, clear structure, internal links from strong pages. The principles are in the article on internal linking and site architecture. If you have many similar pages, consolidate them into one better page.
Technical causes, the easiest to fix
A noindex left by mistake
The tag <meta name="robots" content="noindex"> or the header X-Robots-Tag: noindex excludes the page. It often appears after launches, when test configuration stays behind. Check the page source, not only what the SEO plugin displays.
Blocking through robots.txt
An overly broad Disallow forbids crawling. Note the difference: robots.txt prevents crawling, it does not guarantee exclusion from the index. If you want a page to stay out of results, use noindex and leave the page crawlable, otherwise Google cannot read the tag. For a correct file you can use the robots.txt generator.
A canonical pointing to another URL
If a page declares another URL as canonical, Google treats it as an alternate and does not index it. That is right for parameter variants; it is wrong when the canonical was copied from another template. The full explanation is in the article on canonical tags.
4xx and 5xx errors
URLs returning 404 are not indexed (and should not be). The problem appears when internal links or the sitemap point to them. 5xx errors indicate server problems; if they repeat, Google crawls less often. Clean up broken links and, where an equivalent exists, use a 301 redirect; details are in the redirect guide.
Soft 404
Pages that read like "nothing found" or are nearly empty but return 200. Either make them return 404 or give them real content. It often happens with empty category pages and internal search pages.
Redirects
"Page with redirect" is normal for old URLs. The problems are chains and loops ("Redirect error"): each old address should lead straight to its final destination.
Content loaded through JavaScript
If the text only appears after rendering, Google may see an empty page. See the article on JavaScript SEO.
How to diagnose a page, step by step
- Open URL Inspection for the address and read the reason, the canonical Google selected and the last crawl date.
- Run the live test to see whether the page can be fetched and rendered now.
- Check the page source: meta robots, canonical, headers, main content in the HTML.
- Check robots.txt: the URL must not be blocked.
- Check the sitemap and make sure it contains only indexable URLs returning 200; see the XML sitemap guide.
- Check internal links: how many indexed pages lead to it, and with what anchor text.
- Assess quality: does the page answer the topic better than what already exists?
- Fix, then request validation from the report (the "Validate fix" button), or request indexing for an important page through URL Inspection. According to Google, validation typically takes up to about two weeks, sometimes longer, and you should not click "Validate fix" again until the current validation has succeeded or failed.
How to prioritize when you have many URLs
On a site with hundreds or thousands of URLs, you cannot fix everything at once, and you do not need to. Order them by business impact:
- Pages that make money: services, products, main categories, contact pages. If one of these is missing from the index, it is urgency number one.
- Pages that support the above: articles and guides tied to services, which send internal links to them.
- New pages, recently published: give them a few weeks, then check.
- The rest: technical URLs, archives, variants. Fix them last or exclude them deliberately.
An example of the reasoning
Say the report shows 120 URLs under "Crawled – currently not indexed." You open the list and see that 90 are tag pages with two articles each, 20 are very short old articles, and 10 are important service pages. You do not submit all 120 for indexing. The rational decision: services first (improve them and link them from the homepage), merge or expand the old articles, and set tag pages to noindex if they have no value of their own. The result is not a report with zero non-indexed URLs, but a site where what matters is indexed and what does not matter is excluded on purpose.
This kind of decision is easier when you have defined what you want to measure; see the article on SEO KPIs.
When it is normal for pages not to be indexed
A "clean" report does not mean every URL is indexed. These are normal, among others:
- URLs with a
noindexthat you set (thank-you pages, cart, account, internal search results); - variants with a canonical to the main page;
- old URLs that redirect;
- paginated or filtered pages you do not want indexed;
- test or internal pages.
On an online store, filters and parameters often produce thousands of correctly excluded URLs. What you need to watch is that the important category, product and article pages are indexed. As for crawl budget, just remember that crawl budget problems appear mainly on very large sites.
Common mistakes
- Repeated indexing requests for the same pages, instead of fixing the cause.
- Deleting valuable pages just to shrink the number in the report.
- Blocking in robots.txt pages you want removed from the index.
- Canonicals copied between templates.
- A sitemap with redirected, 404 or noindex URLs.
- Ignoring quality: you fix the tech, but the content stays thin.
- Panicking at every variation in the page count.
- Assuming "indexed" means "will rank."
Limits: what you cannot control
Google states explicitly that it does not guarantee indexing of all pages. Even a technically flawless site may have pages Google does not consider useful enough. Also:
- indexing a new page has no guaranteed timeline;
- you cannot force Google to index a page, you can only remove obstacles;
- the Search Console report has delays and limits the examples it lists;
- on very small sites, a few non-indexed pages are statistically normal.
Conclusion: next steps
A non-indexed page is a symptom, and the reason is in the report. Proceed like this:
- Open the indexing report and sort the groups by the importance of the pages, not by count.
- Fix the clear technical causes first: noindex, robots.txt, canonical, 4xx, 5xx, redirects.
- For "Crawled – currently not indexed," improve the page and link it from other indexed pages, then give it time.
- Request validation or indexing only after you have fixed the cause.
If you have hundreds of URLs in unclear situations, or important pages that will not get into Google, a technical SEO audit identifies and prioritizes the causes. You will find more guides on the blog.