Skip to content
LabTether

Operations Runbook

Daily

  1. Check the Dashboard for platform-wide anomalies.
  2. Review active alerts by severity.
  3. Review open incidents and owners.
  4. Check node online/offline drift.
  5. Validate connector health status.
  6. Spot-check log and telemetry freshness.
  7. Review failed or partial update runs.
  8. Confirm no recurring session errors (terminal/desktop).

Quick health checks from the CLI:

curl -s http://localhost:8080/healthz
curl -s -H "Authorization: Bearer ${LABTETHER_OWNER_TOKEN}" http://localhost:8080/worker/stats
curl -s -H "Authorization: Bearer ${LABTETHER_OWNER_TOKEN}" http://localhost:8080/agents/connected

Weekly

  1. Review connector sync quality and stale sources.
  2. Review alert noise and tune rules/silences.
  3. Validate retention settings against storage trend.
  4. Validate backup jobs and restore readiness.
  5. Review pending updates and maintenance plans.
  6. Review authentication and enrollment anomalies.

Monthly

  1. Rotate high-risk credentials and integration tokens.
  2. Review dormant assets and connectors; clean up unused entries.
  3. Run a disaster-recovery tabletop exercise and restore drill.
  4. Review unresolved recurring incidents for structural fixes.
  5. Review docs drift and update runbooks based on findings.

Security review

Periodic internal hardening checks:

  1. Confirm LABTETHER_TLS_MODE is auto or external (not disabled).
  2. For auto mode, verify the hub CA is installed system-wide or accepted in browser for both :3000 and :8443.
  3. Verify the labtether-ca Docker volume is read-only for agent/console containers (CA private key only in labtether-data).
  4. Confirm access paths are restricted to private network or Tailscale.
  5. Enforce strong secret/token generation and rotation.
  6. Use least-privilege connector integration accounts.
  7. Audit exposed ports and reverse-proxy rules.
  8. Review auth and session logs for anomalies.
  9. Validate backup encryption and restore access controls.
  10. Keep hub and dependencies patched.