Version: 2.15.0

ECA Cluster Operational Procedures

This page covers routine ECA cluster maintenance and monitoring tasks: safely shutting down and starting the cluster, changing node IP addresses, rotating the management tool password, recovering a single restarted node, day-to-day monitoring commands, and replacing the ECA cluster's self-signed certificate.

note

The ECA cluster remains operational as long as 2 of its 3 nodes are running — see High Availability and Resilience for the underlying quorum requirement.

Cluster shutdown and startup

Restarting the OS on an ECA node does not automatically start the cluster back up — follow the steps below for a controlled shutdown, restart, and boot process.

Shut down the cluster

Log in to the master node (Node 1) over SSH as ecaadmin.
Bring the cluster down and wait until all nodes report down:
```
ecactl cluster down
```
Shut down the OS on each node individually — SSH to each node and run:
```
sudo -s
shutdown
```

Start the cluster

SSH to the master node (Node 1) as ecaadmin.
Bring the cluster up:
```
ecactl cluster up
```
Confirm the boot messages show the user tables and signal table exist — this verifies connectivity to the analytics database over HDFS.
Verify the cluster is fully up:
```
ecactl cluster status
```
Confirm the expected containers and tables are listed in the output.

Change an ECA cluster node's IP address

SSH to the master node (Node 1) as ecaadmin, then bring the cluster down and wait for it to complete:
```
ecactl cluster down
```
Elevate to root and update the network configuration using YaST:
```
sudo -s
yast
```
Navigate to networking and update the IP, DNS, and router settings on the interface. Save and exit YaST. Repeat on every node in the cluster.
Once all nodes are updated, verify network connectivity with ping and nslookup.

On the master node (Node 1), edit the ECA environment file:

nano /opt/superna/eca/eca-env-common.conf

Update the IP address of each node to match the new settings:

export ECA_LOCATION_NODE_1=x.x.x.x
export ECA_LOCATION_NODE_2=x.x.x.x2
export ECA_LOCATION_NODE_3=x.x.x.x3

Save and exit (Ctrl+X).

Update the NFS export permissions on every PowerScale cluster managed by this ECA instance to include the new node IP addresses. Replace the IPs below with your actual ECA node IP addresses (the example shows 3 nodes — adjust for your deployment):
```
isi nfs exports modify --id 3 -f --add-root-clients="x.x.x.x, y.y.y.y, z.z.z.z" --add-clients="x.x.x.x,y.y.y.y,z.z.z.z"
```

Update the HDFS access zone with the new ECA node IP addresses:

isi hdfs rack list --zone=eyeglass
isi hdfs rack modify igls-hdfsrack0 --zone=eyeglass --client-ip-ranges="x.x.x.x, y.y.y.y, z.z.z.z"

From the master node (Node 1), clear the ZooKeeper ramdisk and restart Docker, then bring the cluster up:
```
ecactl cluster exec "sudo rm -rf /opt/superna/mnt/zk-ramdisk/* && sudo systemctl restart docker"
ecactl cluster up
```
Confirm the boot messages look as expected.
Validate the Eyeglass appliance's /etc/hosts file:
- Log in to Eyeglass over SSH as admin and run cat /etc/hosts.
- Confirm the new ECA node IP addresses are present. If not, edit /etc/hosts and correct them.
In the Eyeglass web UI, open Manage Services and confirm all active ECA nodes show as Active and green. Any entries still showing the old node IP addresses as inactive should be removed by clicking the red X next to each one.

Rotate the ECA management tool password

Release 2.5.7 and later protects the ECA cluster's management tool UIs (HBase, Kafka, Spark) behind a username/password login over HTTPS, accessible from the Manage Services icon in the Eyeglass GUI. The default login is ecaadmin with password 3y3gl4ss.

Run the following, replacing <password> with the new password:

ecactl cluster exec "htpasswd -b /opt/superna/eca/conf/nginx/.htpasswd ecaadmin <password>"

The new password takes effect immediately on all nodes.

Single ECA node restart or rejoin after a host crash

Use this procedure when a single ECA node needs to rejoin the cluster after a restart — for example, after the underlying host was rebooted for maintenance. Under normal conditions, a single node should not be restarted unless directed by Superna Support.

On the master node, log in over SSH as ecaadmin.
Re-integrate the node into the cluster:
```
ecactl cluster refresh
```
This re-checks access to database tables on all nodes.
Verify there are no dead servers:
```
ecactl db shell
status
```
If no dead servers are listed, open the Eyeglass GUI, check Manage Services, and confirm all nodes show green.

Monitoring commands

Task	Command
Check ECA database status	`ecactl db shell` then `status`
Check overall cluster status	`ecactl cluster status`
Check container memory/CPU stats on a node	`ecactl stats` (auto-refreshes)

Replace the self-signed certificate for the ECA Nginx proxy

SSH to ECA Node 1.

Back up the existing certificate and key:

cd /opt/superna/eca/conf/nginx
mv nginx.crt nginx.crt.bak
mv nginx.key nginx.key.bak

Generate a new self-signed certificate, replacing the domain name, IP address, and validity period as needed for your environment:

openssl req -new -x509 -sha256 -newkey rsa:2048 -nodes -keyout nginx.key -days 365 -out nginx.crt -subj "/CN=<your-hostname>" -addext "subjectAltName=DNS:<your-hostname>,IP:<eca-node-ip>"

Push the new certificate to all nodes and restart the Nginx container:

ecactl cluster push-config
ecactl cluster services restart --container nginx --all

Cluster shutdown and startup​

Shut down the cluster​

Start the cluster​

Change an ECA cluster node's IP address​

Rotate the ECA management tool password​

Single ECA node restart or rejoin after a host crash​

Monitoring commands​

Replace the self-signed certificate for the ECA Nginx proxy​

See Also​