Guidance from running MonSPHERE in production

Alerting hygiene

Alert fatigue is the most common reason teams stop trusting monitoring. A few rules that hold up in practice:

• Alert on symptoms your team acts on, not every metric that moves. A CPU spike that self-resolves in 30 seconds shouldn't page anyone — set a forMinutes window on the alert rule so only sustained conditions fire.
• Route by severity, not by monitor. Critical, customer-facing alerts go to PagerDuty or Opsgenie; everything else goes to a Slack channel your team can triage async.
• Review firing alert rules monthly. A rule that's fired 40 times and been dismissed 40 times isn't monitoring anything — it's just noise with extra steps.

RBAC design

Custom roles are only useful if they map to how your team actually works, not how the permission catalog happens to be organized:

• Start from the narrowest role that lets someone do their job — a compliance reviewer needs audit:read, not org:manage.
• Use the six system roles (Owner, Admin, Operator, Developer, Member, Viewer) before reaching for a custom role — most teams' needs map onto them directly.
• Reserve Platform Admin for the smallest possible set of people; it's cross-organization by design, so it should be treated like root access, not a convenience flag.

Agent rollout

Rolling out the MonSPHERE agent across an existing fleet goes smoothly when it's staged:

1. Install on a handful of non-critical hosts first and confirm exporter health before scripting a fleet-wide rollout.
2. Use a single enrollment token per environment (staging vs. production), not one token shared across everything — it makes revocation surgical instead of all-or-nothing.
3. Set alert rule thresholds after you have a week of real baseline data, not before — an alert tuned against a guess is an alert that pages someone at 3am for nothing.

Dashboard organization

Dashboards rot the same way unused Slack channels do — a few habits keep them useful:

• One dashboard per audience, not per team: an "Executive Overview" and an "On-Call Engineer" dashboard for the same system should look nothing alike.
• Keep the top row for the three or four numbers someone would check first during an incident — status, error rate, latency, active incidents.
• Delete or archive dashboards nobody has opened in 90 days — a stale dashboard someone stumbles into during an incident is worse than no dashboard at all.