Alerts#

Alerts tell you when something needs attention before it becomes an outage. LabTether's alerting system is built around rules (what to watch) and instances (what is currently firing), with notification routing, silences, and enough tuning controls to keep your inbox actionable.

How alerts work#

Rules define the conditions: "CPU above 90% for 5 minutes on any node in the production group." A rule runs continuously against incoming telemetry.

Instances are created when a rule fires. Each instance tracks a specific violation — which asset, when it started, current severity, and whether it has been acknowledged or resolved. Instances clear automatically when the condition is no longer true.

The lifecycle: rule evaluates → condition met → instance fires → notification sent → operator triages → condition clears → instance auto-resolves (or operator manually resolves/deletes).

Creating alert rules#

Navigate to Alerts → Rules to see your current rules and create new ones.

Quick start with templates: LabTether ships with shared templates for common failure modes:

• Agent Offline — agent heartbeat stops
• CPU Saturation — sustained high CPU
• Memory Pressure — memory approaching exhaustion
• Disk Nearly Full — disk usage threshold
• Error Burst — log error rate spike

Enable a template and it starts evaluating immediately. Templates also exist for service health (services.web.down_transition_burst, services.web.uptime_drop) and mobile telemetry (mobile.reconnect_storm, mobile.api_error_burst).

Custom rules: Use New Rule when you need a different metric, pattern, or composite condition. The rule form lets you:

• Pick targets from the live group/asset inventory
• Set the metric and threshold
• Choose severity (critical, warning, info)
• Define the evaluation window

Tip: Start with a small ruleset covering your most critical failure modes. You can always add more later, but starting with too many rules guarantees alert fatigue from day one.

Triaging alerts#

The alerts view shows all active instances, sorted by severity and blast radius. For each instance:

• Acknowledge — signals that someone is looking at it
• Resolve — manually closes the instance (it also auto-resolves when the condition clears)
• Delete — removes stale or duplicate instances
• Silence — suppresses further notifications for a duration
• Escalate to incident — promotes the alert into a tracked incident for coordinated response

Silences#

Silences suppress notifications for matching alerts during a defined window. Use them for:

• Planned maintenance — silence alerts for nodes you are about to reboot or upgrade
• Known noisy conditions — temporarily suppress while you work on a fix

Silences do not stop rules from evaluating or instances from firing — they only suppress notifications. When the silence window expires, notifications resume.

Tuning noise#

The difference between useful alerting and alert fatigue is tuning. Revisit your rules regularly and look for:

• Duplicate rules — multiple rules watching the same condition on the same scope. Consolidate them.
• Flapping signals — alerts that fire and resolve repeatedly in short cycles. Increase the evaluation window so the rule requires sustained violation before firing.
• Over-broad scoping — a rule targeting "all nodes" when only a subset is relevant. Narrow the target to specific groups or assets.
• Threshold drift — thresholds set during initial setup that no longer match your environment. A "disk 80% full" rule on a 10TB array is very different from the same rule on a 128GB boot drive.

Notification routing controls which alerts go where. Configure routes by severity and source:

• Critical alerts → webhook or push notification for immediate attention
• Warning alerts → Slack channel or email digest
• Info alerts → log only, review during weekly check-in

Supported notification channels: webhook, email, slack, apns, ntfy, and gotify.

Tip: Schedule a weekly review of your alert rules. Five minutes of tuning saves hours of noise. Focus on: are the thresholds still right? Are any rules firing but never being acted on? Are there failure modes that have no coverage?