Broken Link Monitoring

A Broken Link monitor starts from a page you choose, crawls your site page by page and checks every link it finds — internal and external links, images, scripts and stylesheets. For each broken URL it shows which of your pages link to it, so you know exactly where to fix it.

A 404 or 5xx, a domain that no longer resolves, a timeout, a TLS error or a redirect loop counts as broken. A 403 on your own site is broken too; on an external site it's usually crawler blocking, so it's listed under "Not judged" along with bot-protection challenges, 429 rate limits, login pages and 999 answers — MonSPHERE never reports a URL as broken when it simply couldn't tell.

Availability

Requires a plan with Broken Link Monitoring (Professional and above). Your plan also sets how many pages one run crawls (by default Professional 500, Enterprise 5,000); a larger site is crawled up to that number and the run says so.

How it works

curl -X POST https://api.monsphere.com/api/v1/organizations/{orgId}/monitors \
  -H "X-Api-Key: msph_live_..." -H "Content-Type: application/json" \
  -d '{
    "name": "Site Links",
    "type": "broken-links",
    "target": "https://example.com/",
    "intervalSeconds": 86400,
    "config": {
      "brokenLinksMaxDepth": 3,
      "urlExcludePatterns": ["/logout", "/cart"]
    }
  }'

• target — the start page. Only pages on the same host are crawled; external URLs are checked but never crawled into.
• brokenLinksMaxDepth — how many link hops from the start page to crawl: 1-10, default 5.
• brokenLinksSkipExternal — optional: don't check links to other sites.
• brokenLinksSkipResources — optional: don't check images, scripts and stylesheets.
• brokenLinksIgnoreRobots — optional: crawl pages your robots.txt disallows (for your own site). By default robots.txt is respected and disallowed URLs aren't requested.
• urlExcludePatterns — optional, up to 50: URLs containing any of these are skipped (use it for logout links, carts and other URLs with side effects).
• The first run records the current state — it doesn't alert on links that are already broken. From then on a link that newly breaks fires the monitor's default "New Broken URL Detected" alert, no setup needed.
• The monitor itself only goes down when the start page can't be loaded. Broken links are a result, not an outage.
• Runs at most every 6 hours. Changing the start page or any crawl option starts a fresh baseline.

Limits: links are read from the page's HTML (<a>, <img>, <script>, stylesheet and icon <link>), so links that only exist after JavaScript runs aren't found, and srcset, <iframe> and CSS url() references aren't checked yet.

About the link checker

Requests from Sitemap and Broken Link monitors identify themselves with this User-Agent:

Mozilla/5.0 (compatible; MonSPHERE-LinkChecker/1.0; +https://monsphere.com/docs/broken-link-monitoring)

It only runs because a MonSPHERE customer set up a monitor for their own site. It respects robots.txt unless the site owner turned that off for their own site, sends at most 4 requests at a time to one host, uses HEAD where it can, and stops a run after 20 minutes. It can't reach private or internal addresses. If you see it on your site and didn't set it up, a link to your site is on a page someone else is monitoring — at most one request per link per run.