Skip to content
LabTether

Updates

LabTether can orchestrate OS package updates across your Linux fleet — plan what gets updated, preview the changes, execute across targeted nodes, and track the results. No more SSH-ing into each box to run apt upgrade.

Tip: Update orchestration currently covers the os_packages scope (Linux apt/dnf/pacman). Automated postchecks and rollback are not yet implemented — plan for manual verification after updates complete.

Planning updates

  1. Open Updates and create a new update plan.
  2. Select target nodes by group, type, or individual selection. Separate critical infrastructure from non-critical targets — you probably do not want to update your Proxmox host and all its VMs at the same time.
  3. Set the scope to os_packages.
  4. Run dry-run mode to preview what would change on each target. This shows you the package list without touching anything.

Before committing to a plan:

  • Verify all target nodes are online and healthy.
  • Check that no unresolved critical incidents are active.
  • Make sure you have a recent backup or snapshot for critical systems. make db-backup covers the LabTether database; snapshot your VMs separately if needed.

Tip: Start with a canary approach. Update one or two non-critical nodes first, verify they survive, then expand to the rest of the fleet. The dry-run output helps you spot surprising package changes before they land.

Running updates

  1. Open the plan detail view.
  2. Start execution.
  3. Monitor progress as updates are applied to each target node. The plan tracks per-node status so you can see which nodes have completed, which are in progress, and whether any failed.

If a node fails during the update:

  • Check agent connectivity and logs for the failed target.
  • Resolve the issue manually on that node.
  • Re-run the plan for remaining targets.

After updates

Automated postchecks are not yet available, so manual verification is important:

  1. Check telemetry and logs. Look for post-update error spikes, service restart failures, or unexpected resource changes.
  2. Verify nodes are reachable. All updated nodes should show connected status with a recent heartbeat.
  3. Spot-check key services. Open a terminal on a few updated nodes and verify that critical services came back up. Check systemctl --failed for any units that did not survive the update.
  4. Compare before and after. If something looks wrong, compare pre-update and post-update logs and metrics to isolate what changed.
  5. Record outcomes. Add notes to the update plan or open an incident if a node needs remediation.

Warning: If a node is unstable after an update, you may need to manually revert packages or restore from a snapshot. Having a pre-update snapshot is the difference between a 5-minute rollback and a stressful afternoon.