How to Run Commands Across All Nodes

To execute a command on all nodes, use the ecactl cluster exec command followed by the desired command.

• Get the hostname of each node:

  ecactl cluster exec hostname

• List running containers on each node:

  ecactl cluster exec ecactl containers ps

• Restart the turboaudit container on all three nodes:

  ecactl cluster exec "ecactl containers stop turboaudit && ecactl containers rm -r turboaudit && ecactl containers up -d turboaudit"

You can run commands that require sudo, but you will be prompted for the ecaadmin password each time.

• Restart the Docker service on all nodes:

  ecactl cluster exec sudo systemctl restart docker

Commands

CLI Command	Function
ecactl cluster <command>	Up: Bring up the cluster across all nodes. Down: Bring down the cluster across all nodes. Status: Get the status of all processes and the connection to the HDFS database. Refresh: Use this command on an ECA node when it has restarted and needs to rejoin an existing cluster.
ecactl cluster remove-docker-networks	Remove all Docker networks to allow ecactl cluster up to recreate them. Use this when the Docker subnet has changed and you need to delete the networks and recreate them with the correct IP information.
ecactl cluster down --hard	Use this only if the normal down process hangs or a container will not shut down correctly. This command shuts down the database without a clean shutdown process. Use it only if directed by support.
ecactl cluster up --clean	Clean up Zookeeper data during the cluster up process. WARNING: If you are using a SQL Database this will also wipe out the database configuration. Use this only if directed by support
ecactl cluster down --bg	Use the --bg flag for a rapid, parallel cluster shutdown during upgrades. Requires version 2.5.7 or later.
ecactl stats	Return container memory and CPU statistics to view the health of all containers.
ecactl containers ps	List all running containers.
ecactl logs --follow iglssvc	Tail the Eyeglass agent logs on a node for debugging. Other services include rmq and fastanalysis.
ecactl cluster exec <command>	Execute the specified command across all ECA nodes. Run this from the ECA master node. Use only with single commands (not multiple concatenated commands). Example: ecactl cluster exec ecactl containers ps

ZooKeeper and Turbo Audit Diagnostics

• Open a ZooKeeper shell, which lets you browse the underlying tree that tracks which ECA node owns which audited PowerScale folder:

  ecactl zk --help
  ecactl zk shell

• Once in the shell, list the top-level Turbo Audit nodes:

  ls /superna/eca/turboaudit
  ls /superna/eca/turboaudit/ecanodes
  ls /superna/eca/turboaudit/auditfolders

  auditfolders lists each audited cluster folder together with its owning node. To find which ECA node currently owns a specific audit folder, get that folder's owner key:

  get /superna/eca/turboaudit/auditfolders/<folder-id>/owner

Kafka Diagnostics

ecactl kafka topics --help
ecactl kafka topics --describe

--describe returns details for every Kafka topic. If neither command errors out, Kafka is considered healthy.

To see which services are running on which node in the distributed cluster:

ecactl cluster services list

HBase Health Check

Run an HBase consistency check (hbck) from inside the hbase-master container:

ecactl containers exec hbase-master /bin/bash
bin/hbase hbck

Run an ECA Container in Debug Mode

ecactl containers stop <container_name>
ecactl containers rm -f <container_name>
ecactl containers up -d <container_name> --debug
ecactl logs --follow <container_name>

Sync NTP Settings from the Master Node (ECA 2.9.0+)

Add the NTP server list to /opt/superna/eca/eca-env-common.conf on the master node:

export NTP_SERVERS=<ip-or-fqdn>

Then bring the cluster up with the config-sync flag, which propagates the NTP_SERVERS value and syncs the chronyd service across all nodes:

ecactl cluster up --config-ntp-server

Schedule Automatic Container Restarts (Cron)

Use restart-cron to schedule periodic restarts of specific containers — commonly fastanalysis, evtarchive, and turboaudit:

sudo -E ecactl components restart-cron set fastanalysis,evtarchive,turboaudit 0 0,6,12,18 '*' '*' '*'

This is the recommended default schedule and can be run on a single node. The outer double quotes are required when this command is wrapped in ecactl cluster exec.

To set a custom schedule for one or more containers:

sudo -E ecactl components restart-cron set <container>[,<container>...] <cron interval>

To remove a scheduled restart:

sudo -E ecactl components restart-cron remove <container>[,<container>...]