Skip to content
LabTether

Terminal Session Issues

A terminal session fails to start, stalls during use, or saved shell sessions are not behaving as expected.

Session won't start

You open a terminal to a node and get an error or an indefinite spinner.

Is the node online? Terminal sessions require an active agent connection. Check the node's status in the console. If it shows offline, resolve that first -- see Node Won't Connect.

Agent state. Even if the node shows as online, the agent process may be in a degraded state. On the target node, confirm the agent is running and responsive:

# Linux
systemctl status labtether-agent --no-pager

# Check recent agent logs for errors
journalctl -u labtether-agent -n 30 --no-pager

If the agent is running but logs show errors related to session creation or stream handling, restart the agent and retry.

Stream ticket path. Terminal sessions use a one-time stream ticket to establish the WebSocket connection. If the ticket request fails, the session cannot start. Test the ticket path:

# Create a session
curl -s -X POST https://<hub>/api/terminal/sessions \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"asset_id": "<asset-id>"}'

A 201 response means the session was created. A 4xx error usually means an auth or permission problem.

Permission check. If the session creation returns a 403, the current user may not have the session_start permission for the target node under the active policy. Check the user's role and scope in the console.

Tip: If terminal sessions stopped working for a specific user but the node is fine for others, the problem is almost certainly permissions. Check whether a recent policy change removed interactive access for that user's role.

Commands stall

The session opens and you see a prompt, but commands hang or produce no output.

Shell process health. The remote shell process may be stuck. Try pressing Ctrl+C to interrupt. If that does not work, the shell itself may be wedged -- close the session and start a new one.

Resource pressure on the target. If the target host is under heavy CPU or I/O load, commands may take a long time to execute. This is not a LabTether problem but it will look like one. Check resource usage:

top -bn1 | head -15
df -h

If the disk is full, many commands will fail or hang because they cannot write temporary files.

Long-running commands. Some commands (like find / or large tar operations) produce no output for an extended period while they work. This can look like a stall. If you know the command is long-running, wait it out or run it with progress output.

Network quality. Terminal sessions stream data over a WebSocket connection. On high-latency or lossy connections, keystrokes and output may appear delayed. This is especially noticeable over cellular or satellite connections.

Saved shell issues

LabTether supports persistent terminal sessions backed by tmux. If a saved session is not behaving as expected:

Session won't reattach. If a previously saved session fails when you try to reopen it:

  • The tmux server process on the target may have been killed (for example, during a reboot). In this case, the session is gone and cannot be recovered. Start a new session.
  • The agent may have been restarted, clearing its session state. Start a new session.
  • The user's interactive policy may have changed since the session was created. If the user no longer has session_start permission for the target, previously saved sessions will fail to reattach. Check the current policy.

Stale session cleanup. If old saved sessions are accumulating and you want to clean them up:

  • Close sessions individually from the terminal UI.
  • If a session is stuck in a bad state and cannot be closed normally, restart the agent on the target to clear all session state.

Warning: Restarting the agent will terminate all active terminal sessions on that node, not just the problematic one. Coordinate with other users before doing this on a shared node.