Version: 2.9.0

Syslog Alarm Forwarding

Introduction

This guide will take you through the different steps required to set up Alarm Forwarding to a Syslog Server, as well as how to filter Alarms and Troubleshoot.

Creating the Alarm Forwarding File

The location of the Logs will show the Alarms that have been triggered. If this file is empty or doesn't exist, you will receive an error. To create the file:

SSH into your Eyeglass VM using a terminal.
Execute the following command to switch to the root user:
```
sudo su
```
Run the following command to create the file:
```
nano /etc/syslog-ng/conf.d/superna.conf
```

Once the file is open, write its contents:

@version:X.XX
 filter  f_superna {
    message("RSW0002") or message("RSW0011") ;
    };
 source igls_src {file("/opt/superna/sca/logs/igls_alarms.log"); };  
 destination logserver { udp("X.X.X.X" port(514)); };  

 log {
    source(igls_src);
    filter(f_superna);
    destination(logserver);
 }

Replace the X's with the correct version and the Syslog server IP address you want the messages to be forwarded to.

info

To consult the correct version, run the command:

    syslog-ng --version

You will see the number next to "Config Version" in the results.

After changes are saved, restart Syslog to ensure that they take effect. To do so, run the following command:
```
    systemctl restart syslog-ng
```
Finally, to check that it's running, run the following command:
```
    systemctl status syslog-ng
```

From this moment, the associated Syslog server will begin receiving the alarm messages.

tip

The following command:

    iglstest alarm test

Allows you to trigger test alarms, to make sure that the system is working. When you run it and go consult the alarm in the Eyeglass UI, the source will be set to Undefined.

To see the contents of the Alarms Log at any moment:

SSH into your Eyeglass VM as admin
Enter the following command:

cat /opt/superna/sca/logs/igls_alarms.log

How to Filter Alarms

To filter the alarms that are sent to your chosen Syslog server, you must add a piece of code to the Syslog config file which was created above. Depending on the characteristic you wish to filter by, the code will change.

TO FILTER BY SEVERITY

Add the following to the filter section of the file:

    filter f_superna {

    message("Severity:CRITICAL") or message("Severity:MAJOR") ;

    };

TO FILTER BY ALARM CODE

Add the following to the filter section of the file:

    filter f_superna {

    message("RSW0002") or message("RSW0011") ;

    };

Be sure to consult the Alarm Codes guide for the specific codes you can select for this filter.

Troubleshooting

The following section will outline the steps for resetting the forwarding statistics in your Eyeglass VM, and then using packet capture to see sent messages. If the original issue persists, contact Support.

How to Reset the Forwarding Statistics

Enable verbose logging

SSH to your Eyeglass VM as admin by using the following command:
```
    sudo -s
```
You will be prompted to enter the admin password.
Run the following command:
```
    syslog-ng-ctl verbose --set=on
```

Check the current statistics

Run the Statistics Command:

    syslog-ng-ctl stats | grep logserver

IMPORTANT

If the counters are not incrementing, or show zeroes, it means that nothing has matched your filter and nothing was forwarded to your destination.

Reset the stats to zero

Run the following command:
```
    syslog-ng-ctl stats --reset 
```

How to Use Packet Capture to See Outgoing Messages Sent to the Target Syslog Server

SSH to your Eyeglass VM as admin by using the following command:
```
    sudo -s
```
You will be prompted to enter the admin password.

Execute the following procedure to monitor any udp Syslog messages sent based on matching alarms:

Run the following command, replacing X.X.X.X with the IP address of the Syslog server configured in the Syslog config file mentioned above.
```
tcpdump -nnAs0 -i eth0 udp port 514 -v | grep -A 2 "X.X.X.X" 
```

warning

This command will not display any data until a packet is sent to the target Syslog server based on the matching logic configured in your filter.

Leave this command running, and continue to the next steps

Open a new SSH session as the admin user. Make sure to leave the previous session running.
Run the test alarm command:
```
    iglstest alarm test
```

IMPORTANT

The random alarm may not match your filter logic. If you need to, edit the config file and restart syslog-ng to continue with this procedure.

The test alarm command should be run as many times as required for a packet to appear in the first SSH session
You may also run the Statistics Command to verify any matches processed by the Syslog-ng filter logic, and repeat as necessary.

info

Check the target Syslog server after you have run all the checks, to verify the messages are appearing.

How to Read an Alarm Message

Syslog messages sent by Eyeglass are based on the information recorded in the /opt/superna/sca/logs/igls_alarms.log file, as defined in the configuration file above. When they are forwarded to the target Syslog server, they appear as messages with several sections separated by comma.

The format is as follows (all sections separated by a comma):

What is between the commas?	Description
[DEBUG] IGLS_ALARMS:168 - Eyeglass	Always exactly as is
[empty]	Always empty
Event: value	Value is the timestamp of the event
AID: value	Value is the entity of the alarm, and it depends on the type of alarm. (It can range from being a username, to an affected cluster, etc.)
Port: Nil	Value is always NIL
Type: null	Value is always null
Entity Type: [empty]	Value is always empty
Extra Data: value	Extra data. Value depends on the type of alarm. Consists of multiple possible subsections including but not limited to: (user, affected zone, affected isilon, detectors, number of affected files, status, etc.)
Description: value	Value depends on the type of alarm, but is a basic description of the issue.
NSA	Always NSA
Severity: value	Value depends on the severity of the alarm.
Impact: false	Value is always false
Category: value	Value is Alarm Code

Next Steps

At this point, if you haven't done so already: run some test alarms. You may also head to Alarm Codes to verify which codes will be relevant to you.

Introduction​

Creating the Alarm Forwarding File​

How to Filter Alarms​

Troubleshooting​

How to Reset the Forwarding Statistics​

How to Use Packet Capture to See Outgoing Messages Sent to the Target Syslog Server​

How to Read an Alarm Message​

Next Steps​