robots.txt & Meta Noindex Monitoring

Catching an accidental deindex isn't a separate monitor type — it's an HTTP/HTTPS monitor's existing keyword/forbiddenKeyword body-matching pointed at the right target, checked by Blackbox Exporter on every probe alongside the monitor's normal up/down check. No new infrastructure, no extra request.

Availability

Available on every plan that already includes Advanced HTTP Monitoring (the keyword/forbiddenKeyword fields).

Site-wide: robots.txt disallowing everything

Point the monitor's target directly at the robots.txt file, and use forbiddenKeyword with a regex that only matches a real "block everything" rule:

curl -X POST https://api.monsphere.com/api/v1/organizations/{orgId}/monitors \
  -H "X-Api-Key: msph_live_..." -H "Content-Type: application/json" \
  -d '{
    "name": "Site not fully disallowed",
    "type": "https",
    "target": "https://example.com/robots.txt",
    "intervalSeconds": 300,
    "timeoutSeconds": 10,
    "config": {
      "forbiddenKeyword": "Disallow:\\s*/\\s*$",
      "keywordIsRegex": true
    }
  }'

The anchored regex only matches a bare Disallow: / line — a normal robots.txt with specific disallowed paths (Disallow: /admin, Disallow: /search) never trips it.

Per-page: an accidental noindex tag

Point a monitor at the actual page and check for the noindex meta tag directly:

curl -X POST https://api.monsphere.com/api/v1/organizations/{orgId}/monitors \
  -H "X-Api-Key: msph_live_..." -H "Content-Type: application/json" \
  -d '{
    "name": "Homepage stays indexable",
    "type": "https",
    "target": "https://example.com",
    "intervalSeconds": 300,
    "timeoutSeconds": 10,
    "config": {
      "forbiddenKeyword": "<meta[^>]+name=[\"'"'"']robots[\"'"'"'][^>]+content=[\"'"'"'][^\"'"'"']*noindex",
      "keywordIsRegex": true
    }
  }'

How it runs

Same as every other http/https monitor: a Blackbox Exporter HTTP probe, region-selectable, with the standard uptime/latency analytics — deindex detection just rides along on the same check as a body-content assertion, using fail_if_body_matches_regexp under the hood.