Denver Website Indexing and Google Search Console Guide

Website indexing is the step most Denver business owners never see and most often get wrong by accident. A redesign ships with the staging site's noindex tag, a plugin blocks a folder, or three versions of every URL compete to be the real one. This guide explains how crawling and indexing work, how to set up sitemaps and robots.txt, and how to read Google Search Console without drowning in it.

Crawling and Indexing Are Different Decisions

Crawling is Googlebot fetching a URL. Indexing is Google deciding to store that page and show it in results. A page can be crawled and still left out of the index, and the reasons are usually about duplication or quality rather than technology. Keeping those two ideas separate makes almost every indexing problem easier to diagnose.

robots.txt: Small File, Big Mistakes

The file lives at the root of the host, for example https://example.com/robots.txt, and applies only to that host and protocol. Google reads up to 500 KiB of it. Keep it short. A typical small business site needs very little:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

XML Sitemaps That Help Instead of Hurt

A sitemap is a list of URLs you want indexed. Each file can hold up to 50,000 URLs or 50 MB uncompressed; larger sites use a sitemap index that points to several files. The rule that matters most is simple: list only canonical URLs that return 200 and are meant to be indexed. A sitemap full of redirects, 404s and noindexed pages sends Google mixed signals and makes the Sitemaps report useless as a diagnostic.

• Reference the sitemap in robots.txt and submit it in Search Console.
• Let the CMS or a plugin generate it so it stays current.
• Split large sites by type (services, locations, posts) so the indexing rate of each group is visible.

Canonicals and Duplicate URLs

The same page can often be reached at several addresses: with and without www, with a trailing slash or without, with tracking parameters like ?utm_source=. Google picks one as canonical and treats the rest as duplicates. Your job is to make that choice obvious by agreeing on one version everywhere:

1. Redirect the other host and protocol versions with a single 301 each.
2. Set a self-referencing canonical tag on every indexable page.
3. Link internally to the canonical version only.
4. List only the canonical version in the sitemap.

When all four agree, "Duplicate, Google chose different canonical than user" in Search Console usually shrinks on its own.

The Search Console Reports Worth Your Time

The Messy Middle: "Crawled, Currently Not Indexed"

This is the status that frustrates people most, because nothing is technically broken. Google fetched the page and decided not to store it, at least for now. For local service sites it most often hits pages that look alike: ten suburb pages with the same paragraph and a different city name, or blog posts that repeat what the service page already says.

Requesting indexing again rarely changes the outcome. What tends to work is making the page clearly different and more useful, merging pages that say the same thing, and linking to the survivors from stronger pages on the site. The local keyword mapping guide covers how to decide which area pages deserve to exist.

Indexing and Search Console FAQ