Skip to content
LabTether

Node Won't Connect

A node that was previously online now shows as offline in the console. Terminal and desktop sessions cannot start, and telemetry has stopped updating.

Quick checks (2 minutes)

Before digging deeper, rule out the obvious:

  1. Check last-seen time. Open the node in the console and note the last-seen timestamp. If it was seconds ago, the node may be flapping rather than fully offline (see the intermittent flapping section below). If it was hours or days ago, the problem is likely persistent.

  2. Verify hub health. Hit the hub health endpoint to make sure the hub itself is up:

    curl -s https://<hub-host>:8443/healthz

    If the hub is down, every node will appear offline. Fix the hub first.

  3. Test the network path. From the node (or a machine on the same network), confirm it can reach the hub:

    curl -s https://<hub-host>:8443/healthz

    If this fails, the problem is network-level -- DNS, firewall, or routing -- not agent-level.

Agent not running

The most common reason a single node drops offline is that the agent process stopped.

Linux:

systemctl status labtether-agent

If it is inactive or failed, restart it:

sudo systemctl restart labtether-agent

Then watch the logs for a successful reconnect:

journalctl -u labtether-agent -f --no-pager

macOS:

Check whether the agent process is running:

pgrep -fl labtether-agent

If not, restart it from the menu bar app or via launchctl.

Windows:

Open Services (services.msc) and look for the LabTether Agent service. If it is stopped, start it. Check the Windows Event Log for crash details if it keeps stopping.

Network issues

If the agent is running but the node stays offline, the problem is usually between the agent and the hub.

DNS resolution. On the node, confirm the hub hostname resolves:

dig +short <hub-hostname>

Firewall and proxy. The agent needs outbound HTTPS (port 443 or your configured hub port) and the ability to upgrade to WebSocket on the same connection. Common blockers:

  • Corporate proxies that strip WebSocket upgrade headers
  • Firewall rule changes that block the hub port
  • VPN split-tunnel configurations that changed

Recent infrastructure changes. Ask whether anything changed on the network side -- new firewall rules, proxy reconfiguration, DNS migration, or VPN policy updates. These are the most common culprits for nodes that were working fine yesterday.

Tip: If the node is behind a proxy, make sure the proxy supports WebSocket passthrough. A proxy that works for plain HTTPS but strips the Upgrade: websocket header will cause the agent to connect briefly and then drop.

Auth/enrollment issues

If network connectivity is fine but the agent still cannot connect, the problem may be authentication.

Token expired. Agent enrollment tokens have a limited lifetime. If the token has expired, the agent will fail to authenticate on reconnect. Check the agent logs for authentication errors:

journalctl -u labtether-agent --no-pager -n 50 | grep -i auth

Fingerprint drift. If the node's identity changed -- for example, after a VM clone, OS reinstall, or hardware swap -- the hub may reject the connection because the fingerprint no longer matches. In this case, you need to re-enroll the node:

  1. Remove the old node entry from the console.
  2. Re-run the agent enrollment on the target host.
  3. Approve the new enrollment in the console.

Warning: Re-enrollment creates a new node identity. Historical telemetry and session recordings from the old identity remain associated with the previous node entry.

Host issues

Sometimes the agent is running and the network is fine, but the host itself is in trouble.

Resource pressure. If CPU, memory, or disk is fully saturated, the agent may not get enough resources to maintain its connection. Check the basics on the node:

# CPU and memory
top -bn1 | head -20

# Disk
df -h

If the disk is full, the agent may not be able to write logs or state, which can prevent it from operating correctly.

Clock skew. TLS handshakes and token validation are sensitive to clock drift. If the node's clock is off by more than a few minutes, authentication will fail silently. Check the time:

date -u
timedatectl status   # Linux with systemd

If the clock is significantly off, sync it with NTP and restart the agent.

Intermittent flapping

If the node keeps going online and then offline again every few minutes, the root cause is usually one of:

  • Unstable network path. Packet loss or high latency on the connection between the agent and the hub. Run a sustained ping or traceroute to look for drops.
  • Resource saturation. The host is intermittently overloaded, causing the agent to miss heartbeat deadlines. Watch CPU and memory over time, not just a single snapshot.
  • Auth renewal failures. The agent successfully connects but fails to renew its session token before expiry, causing a disconnect-reconnect cycle. Check agent logs for token renewal errors.

For flapping nodes, the agent logs are the most useful diagnostic. Look for patterns in the timing of disconnects -- if they correlate with a specific interval, that points toward token renewal. If they are irregular, network instability is more likely.