Cluster High Availability and Node State Management

IMTerm's cluster is Active-Active: every node is equal, every node can serve sessions, and every node can administer the full cluster. There is no primary/replica distinction and no single point of failure. When a node goes down, users on that node reconnect to any surviving node - the session reconnects within seconds and the host system sees no interruption.

How the Cluster Works

Each IMTerm node runs the same binary and connects to the same NFS data directory and the same set of peers. The peer list is configured in config.yaml and each node announces itself on startup. There is no shared in-memory session state - sessions are stateless at the cluster layer. The NFS directory holds audit logs, print jobs, and configuration; all nodes read and write the same data.

Load balancing is handled by the upstream load balancer (nginx, HAProxy, or a cloud LB). Use cookie-based sticky sessions so that a connected browser always returns to the same node during its session. If that node goes offline, the load balancer redirects the browser to another node and the user is prompted to reconnect.

No shared session state needed. IMTerm maintains session continuity through the host system (AS/400 or z/OS), not through cluster replication. A reconnect after a node failure is a fresh TN5250E or TN3270E negotiation, which typically completes in under two seconds.

Node States

Each cluster node is in one of four states at any given time:

State transitions:

• On startup: STANDBY (if start_in_standby: true) or ONLINE
• Activate: STANDBY or DRAINING -> ONLINE
• Deactivate (drain): ONLINE -> DRAINING -> STANDBY
• Deactivate (force): ONLINE or DRAINING -> STANDBY immediately, terminating sessions
• Stop: any state -> OFFLINE

Shared Storage (NFS)

All nodes must share a common data directory over NFS (or another POSIX-compatible network filesystem such as CephFS). This directory holds:

• User accounts and roles
• Audit log files
• Print jobs and print archive
• Session recordings
• Connection profiles (shared across nodes)

Minimum NFS requirements: NFSv4, 1 Gbps network link between nodes and NFS server, latency under 5 ms. For high-availability NFS, use a replicated NFS cluster or cloud NFS service (AWS EFS, Azure Files).

# config.yaml - same on all nodes, change node_id only
mode: cluster
node_id: node1          # unique per node: node1, node2, node3...
data_dir: /mnt/nfs/imterm

cluster:
  peers:
    - imterm-node1.corp.com:8080
    - imterm-node2.corp.com:8080
    - imterm-node3.corp.com:8080

Activating and Draining Nodes

Admin Console

Open the admin console and go to the Cluster tab. You will see a live table of all known nodes with their current state, session count, and uptime.

Available actions per node:

• Activate - bring a STANDBY node ONLINE to accept new sessions
• Drain - set the node to DRAINING (stops new sessions, waits for existing ones to finish)
• Force stop - move immediately to STANDBY, terminating all sessions on that node

The drain timeout is configurable (default: 10 minutes). If sessions are still active after the timeout, the node moves to STANDBY anyway and those sessions are terminated.

CLI Commands

The same operations are available from the imterm CLI on the node itself or by pointing to a remote node with --node:

# Show cluster status
imterm cluster status

# Activate a node (STANDBY -> ONLINE)
imterm cluster activate

# Drain a node (ONLINE -> DRAINING -> STANDBY)
imterm cluster deactivate

# Force-stop a node immediately (skip drain)
imterm cluster deactivate --force

# Show status of a specific node from any node
imterm cluster status --node imterm-node2.corp.com:8080

Output of imterm cluster status:

NODE                         STATE     SESSIONS  UPTIME
imterm-node1.corp.com:8080   ONLINE    142       5d 3h
imterm-node2.corp.com:8080   DRAINING  12        5d 3h
imterm-node3.corp.com:8080   STANDBY   0         0h 4m

Starting in Standby

Set start_in_standby: true to have a node start in STANDBY state instead of immediately going ONLINE. This is useful when you want to verify the node is healthy before admitting traffic - for example, after an upgrade or a restart following a failure.

cluster:
  start_in_standby: true
  drain_timeout: 600    # seconds to wait for sessions to drain (default 600)

After verifying the node is healthy, activate it from the admin console or with imterm cluster activate.

Zero-Downtime Upgrade Procedure

This procedure upgrades a 3-node cluster with no session interruption. Each node is taken out of service one at a time, upgraded, and returned before moving on to the next.

1. Drain node1 from the admin console (Cluster tab > Drain) or CLI:

   imterm cluster deactivate    # run on node1

   Wait for the session count to reach zero (or for the drain timeout).
2. Upgrade the binary on node1:

   sudo systemctl stop imterm
   sudo cp imterm-v2.3.9 /usr/local/bin/imterm
   sudo systemctl start imterm

3. Verify node1 is healthy (check logs, imterm cluster status, and the health endpoint at /api/health).
4. Activate node1:

   imterm cluster activate

5. Repeat steps 1-4 for node2, then node3.

If start_in_standby: true is set, node1 will start in STANDBY after the restart and you activate it manually in step 4. If it is not set, the node goes ONLINE automatically after starting - make sure it is fully up before sending traffic.

Audit Events

All cluster node state transitions are recorded in the audit log (/var/log/imterm/audit.jsonl). The events include the node ID, the user who triggered the action, and the previous and new state.

All events include: timestamp, node_id, admin username (for manual actions), source IP, and session count at the time of the event. Use these events in Splunk or your SIEM to track rolling upgrades and detect unexpected node failures.

Troubleshooting

Node stays in DRAINING state

Check for long-running sessions on the node. From the admin console (Sessions tab, filter by node), identify any sessions that have been idle or running for a long time. You can terminate individual sessions from the admin console, or use imterm cluster deactivate --force to move to STANDBY immediately.

Peer not appearing in cluster status

Verify that the peer address in config.yaml is reachable from the node: curl http://imterm-node2.corp.com:8080/api/health. Check that firewalls allow TCP on port 8080 between nodes. Nodes discover each other via the peers list - each node must list all other nodes.

IMTE7131W: Cluster quorum degraded

Fewer than 2 nodes are ONLINE. This is a warning, not an error - IMTerm continues serving sessions from the remaining ONLINE nodes. Investigate and activate additional nodes as soon as possible to restore redundancy.