You often come across a website whose sections you’d like to explore, but whose navigation only shows a fraction of its content. The site map, or sitemap, is the file that lists the URLs a website makes available. Knowing where to find it and how to read it gives you access to pages that neither the menu nor the site’s internal search highlights.
Accessing a website’s XML sitemap file
The most direct method is to type the website’s address followed by /sitemap.xml into your browser’s address bar. On most websites built with a CMS such as WordPress, Shopify or Joomla, this URL returns an XML file listing published pages.
If this address returns a 404 error, you can try a few variations: /sitemap_index.xml, /sitemap1.xml, or /sitemap/. The filename is not standardized; each CMS or plugin uses its own convention.
Another reliable option is to open the website’s robots.txt file (accessible via /robots.txt). This file often contains a “Sitemap:” line indicating the exact location of the site map. This is also the method recommended by Google since it retired its old ping notification system in 2023. For example, by viewing the Double Portion site map, you can see what a well-structured sitemap looks like on a real website.
Sitemap indexes and child files: reading a large site map
On large websites, the first file you come across is not the complete list of URLs. It’s a sitemap index, a file that points to several child sitemaps. Each child file contains a group of URLs, often organized by content type (articles, products, categories).
Google sets a limit for each file: no more than 50,000 URLs and no more than 50 MB uncompressed. Beyond that, the website must split its site map into several files linked through this index. To explore every page, you therefore need to open each child file rather than stopping at the first level.
In practice, click each link in the sitemap index in your browser. The XML file appears with loc tags (the page URL) and sometimes lastmod (the date it was last modified). On an e-commerce website with thousands of product pages, reviewing the files one by one is the only way to get a complete overview.
Is the lastmod tag reliable for identifying updates?
Google specifies that this date is only taken into account when it reflects genuine content changes. Many CMS platforms update this date every time the file is regenerated, even when the page itself has not actually changed. A recent lastmod date does not guarantee that the content has been updated. You therefore can’t rely on it to identify updated pages without checking them manually.
Three cases where the sitemap doesn’t show everything
A site map is still only a declared inventory. It lists only what its generator includes. Several situations can create a gap between a website’s actual content and what its sitemap displays.
- Orphan pages (with no internal links) are often missing from the sitemap, especially when it is generated automatically from the website’s navigation structure
- Pages excluded by a rule in robots.txt or by a noindex tag may disappear from the file, even if they remain technically accessible via their direct URL
- Dynamically generated content (filter results, internal search pages) almost never appears in a sitemap, as it would create thousands of redundant URLs
To fill these gaps, you can supplement your sitemap review with a website crawl using a tool such as Screaming Frog or Xenu. These programs follow every internal link and reveal pages the sitemap overlooks.
Cross-checking data with Google Search Console and robots.txt
If you manage your own website, Google Search Console provides a list of indexed pages and flags any issues. The “Sitemaps” section lets you submit the file and check how many URLs have been discovered, then how many have actually been indexed.
The gap between the two figures is revealing. A sitemap may list several hundred pages, while Google indexes only a fraction of them. The reasons vary: thin content, duplicate pages, redirect errors, or simply an insufficient crawl budget for a large website.
Cross-checking the sitemap and robots.txt to detect inconsistencies
The robots.txt file can block access to entire directories while still leaving their URLs in the sitemap. This conflict prevents search engines from processing those pages. Checking that the two files are consistent helps eliminate these contradictions.
In practice, open robots.txt, look for the “Disallow” lines, then compare them with the URLs in the sitemap. If a URL appears in both, there’s an issue to fix: either remove the page from the sitemap or delete the blocking rule.
Reading a sitemap without technical knowledge
The raw XML format displayed in a browser can seem daunting. Some browsers, such as Chrome, display the tags without formatting, making a file with several thousand lines difficult to read.
A few simple solutions make it easier:
- Copy the sitemap URL into an online XML viewer, which organizes the data in a table with sortable columns
- Use a browser extension designed for XML files, which automatically color-codes and indents the content
- Open the file in a spreadsheet by importing the XML, allowing you to filter and sort URLs by date or website section
The site map remains the quickest starting point for mapping a website, as long as you don’t mistake it for a complete inventory. Cross-checking this file with robots.txt, a technical crawl or Search Console gives you a much more accurate picture of what a website actually contains.
