Sitemap & URL Monitoring

A Sitemap monitor reads your sitemap.xml on a schedule and tells you two things:

• What changed in the list — URLs added and removed since the previous run, by name.
• What broke — every URL in the sitemap is requested; a 404 or 5xx, a domain that no longer resolves, a timeout, a TLS error or a redirect loop counts as broken.

Sitemap indexes are followed and .gz sitemaps are decompressed. The monitor's detail page lists every URL of a run with its status code, response time and whether it was added or removed, filterable by broken / added / removed / not judged and searchable.

Not judged is never broken: a bot-protection challenge, a 429 rate limit, a login page (401), an external site's 403, LinkedIn-style 999 answers and private addresses are listed separately under "Not judged" — MonSPHERE couldn't tell whether those URLs work, so it doesn't claim they're broken.

Availability

Requires a plan with Sitemap Monitoring (Professional and above). Your plan also sets how many sitemap URLs one run reads and checks (by default Professional 1,000, Enterprise 10,000); a sitemap with more is checked up to that number and the run says so.

How it works

curl -X POST https://api.monsphere.com/api/v1/organizations/{orgId}/monitors \
  -H "X-Api-Key: msph_live_..." -H "Content-Type: application/json" \
  -d '{
    "name": "Main Sitemap",
    "type": "sitemap",
    "target": "https://example.com/sitemap.xml",
    "intervalSeconds": 21600,
    "config": {
      "urlExcludePatterns": ["/private/", "?utm_"]
    }
  }'

• target — the sitemap (or sitemap index) URL.
• urlExcludePatterns — optional, up to 50: URLs containing any of these are skipped.
• sitemapSkipUrlCheck — optional: only track the URL list (added / removed), don't request each URL.
• The first run records the current state — it doesn't alert on URLs that are already broken. From then on a URL that newly breaks fires the monitor's default "New Broken URL Detected" alert, no setup needed; an already-broken URL doesn't re-alert on every run. For added / removed URLs, create an alert rule on the url_set_changed metric.
• The monitor itself only goes down when the sitemap can't be read (unreachable, not a valid sitemap). Broken URLs are a result, not an outage.
• Runs at most every hour. Changing the target or the URL scope (exclusions, list-only) starts a fresh baseline.

How it runs

Runs on a dedicated crawl worker, isolated from every other monitor's checks. Each URL gets a HEAD request, then GET when HEAD doesn't give a clean answer; at most 4 requests at a time go to one host. Every connection goes through the same network guard as all other checks, so a sitemap can't make MonSPHERE reach private or internal addresses. Requests identify themselves as MonSPHERE-LinkChecker — see Broken Link Monitoring. Per-URL results are kept for your plan's data retention period.