Skip to content
LabTether

Incidents

Alerts tell you something is wrong. Incidents let you track what you did about it. An incident is a structured record of an operational event — the timeline, the investigation, the fix, and the lessons learned — all in one place.

What are incidents

An incident is different from an alert. Alerts fire automatically when a condition is met. Incidents are opened deliberately — by escalating an alert, or by creating one manually — when a situation warrants coordinated tracking.

Use incidents when:

  • Multiple alerts are firing and they share a root cause
  • A problem requires investigation across several nodes or services
  • You want a record of what happened, what was tried, and how it was resolved
  • Multiple people need to coordinate on a response

Not every alert needs an incident. A disk-full alert on a dev VM that you fix in five minutes is probably just an alert. A storage controller failure that takes down three production services is an incident.

Incident workflow

Opening an incident:

  • From the alerts view, escalate an active alert instance to create an incident pre-populated with the alert context.
  • Or create an incident manually from the Incidents page with a title and initial scope.

Scoping impact:

Identify the affected sites, nodes, and services. LabTether's topology and service views help you trace the blast radius — which downstream services depend on the failing component? Which upstream systems might be contributing?

Building the timeline:

Add entries as you investigate and act. The timeline is your operational logbook:

  • What you observed
  • What diagnostics you ran
  • What changes you made
  • What the results were

Each entry is timestamped, so the timeline doubles as an audit trail.

Coordinating remediation:

Assign action steps, execute fixes, and monitor recovery through the incident view. Link to relevant nodes, alerts, and services so everything is in context.

Resolution

When the situation is stabilized:

  1. Verify affected systems have recovered — check node status, service health, and alert state.
  2. Add a resolution summary: root cause, fix applied, and any prevention measures for next time.
  3. Close the incident.

The closed incident remains in the system as a permanent record. Come back to it during post-mortems or when a similar failure mode appears again.

Audit trail

Every incident captures a complete timeline — who did what, when, and what happened next. This is valuable beyond the immediate response:

  • Post-mortems — review the timeline to identify where response time could improve
  • Pattern detection — recurring incident types suggest systemic issues worth addressing with automation or infrastructure changes
  • Compliance — for small teams managing production infrastructure, the incident timeline provides the operational record auditors want to see
  • Onboarding — new team members can read past incidents to understand the environment's failure modes and how the team responds

Tip: Even if you are the only operator, get in the habit of recording incidents for non-trivial outages. Future-you will appreciate having a record of what happened and why, especially when the same symptoms show up six months later.