Description of sensorAlarm

Program sensorAlarm polls the database filled by readsensor and checks for alarm conditions. Alarms are given based on criteria and are provided in a file. The table with all criteria and actions is given below.

The code for sensorAlarm can be found in CVS, in the module wsrt_mac/alarms. The following files are used by the sensorAlarm program:

• alarmActions.py → class to execute shell-commands and send mail to the observer on duty.
• class_sensorAlarm.py → class that does most of the things
• DBConnection.py → class for interfacing with a MySQL database
• log.py → defines a function to print a log message with date/time stamp.
• sensorAlarm.py → Main program

The program can be started by executing sensorAlarm.py; no commandline options are needed for a default run. It should be started by user monitor on the WSRT monitor host (now wop10). The files should be present in directory /wsrt/bin on wop10.

The alarmActions class handles the execution of shell-commands. It provides a mechanism to stop a command after it has been running for a certain amount of time (timeout). The default timeout is one second.

In case of problems with executing a command (timeout, or a non-zero exit status) the observer will be notified by email/SMS. This uses the sendMail program.

Logmessages are written to a logfile (see class log.py).

This class is described in the file class_sensorAlarm.py. This class holds all the methods that do the actual checking and alarming.

The class reads the file holding the alarmconditions. At predefined intervals (to be set at startup of the main program) it will go through the list of alarmconditions and evaluate each condition to be true or false. In case a condition is true, an alarm must be raised. That involves:

• Executing the action that is described in the alarmconditions file
• Setting the alarmstatus of all the associated sensors to on in the Sensor database(s).

Before obtaining the values, we first check if the sensor database values are recent enough to trust. This limit is set at 10 minutes, i.e., only if the database contains values less than 10 minutes old, the alarmconditions will be evaluated. All values are taken from the local MySQL database (the one running on wop10, that is).

In the alarmconditions.cfg-file is a column that gives the amount of individual samples/measurements of a sensor value to take into account. This is done in the following way:

• In case of 1 or 2 measurements, we just take the sample (or the mean of the two)
• in case of more than 2 samples: we determine the time interval between the latest two samples, and allow only values that are younger than this time interval multiplied by the amount of samples to take into account. This is done so that when the database is not filled for a while (e.g., due to maintenance, or a software problem), only the newer values from after the interruption are included.
• Of all the 'valid' values, the median is calculated (instead of the average) so we are less vulnerable to accidentally large deviations from the mean (i.e., in case of a sensor readout problem).

This class is the interface with a MySQL table. A default constructor connects to the local MySQL server and the database Sensor, but by providing other parameters in the constructor other servers and databases can be reached as well. It has methods to connect, to check if a connection is still valid, to run a 'select'-type query and to run a 'update'-type query.

The main program can be found in file sensorAlarm.py.

It handles the following things:

• Interpret commandline options.
• Creation of a logfile with a unique name in directory /log/alarms.
• Daemonize itself so the user can close the shell after starting the program.
• Run a never-ending loop in which alarmconditions are checked at regular intervals.

To handle the commandline options we use the module Optionparser. This module defines simple functions so that commandline options can be defined and taken along. The available options are:

  -h, --help            show this help message and exit
  -iINTERVAL, --interval=INTERVAL
                        Check interval time in seconds (prefer multiples of 60)
  -fFILE, --inputfile=FILE
                        Name of input alarmconditions config file
  -n, --nodaemonize     Don't run as a daemon process

The default interval is 120 seconds. The default alarmconditions config file is /wsrt/config/alarmconditions.cfg.

The logfile is written when the process is daemonized, only. Otherwise the log messages are written to to terminal. The logfile ends up in directory /log/alarms. Logfiles are named: sensorAlarm.yyyymmddThhmmss, so with a date/time stamp of creation attached. A softlink to the latest logfile is always created as well and named sensorAlarm.log.

Also, there is a file called sensorAlarm.pid that holds the Process PID of the daemonized process. This can be used to kill the process if needed.

Before we start the loop, we read the alarmconditions file, set up the database connection handlers for two databases (two, as the alarm flag needs to be updated both locally and on the WSRT MySQL server), and reset all alarms in these two databases.

In the main loop, we do the following things:

• Check if the local database can still be reached. If not, raise an alarm by sensing a email/SMS to observer.
• Check if there is recent (less than 10 minutes old) data in the database. If not, raise an alarm by sensing a email/SMS to observer.
• Check all alarmconditions. If an alarmcondition is met, it will be reported and acted upon only once. If the alarmcondition is over, this will be reported as well (and the alarm flag in the database will be retracted).
• Wait until the next requested checktime.

For sensorAlarm, we have used the following Python modules:

• For string handling: string
• For date and time issues: time and datetime
• For system calls and such : os, sys and popen2
• For signal handling (timeouts): signal
• For parsing commandline options: optparse (see here).
• For MySQL connections: MySQLdb (see the manual and the project page).

The alarmconditions.cfg-file has lines like this:

...
CONSwarm > 27.0 # 5 # sendMail obs 'Console too warm'
CONSwarm > 30.0 # 5 # sendMail obs 'Console meltdown'
CONSwarm < 15.0 # 5 # sendMail obs 'Console too cold'
...
Sup-He-RT0 + Ret-He-RT0 < 15 # 5 # sendMail stiepel 'He-pressure RT0 too low'
Sup-He-RT1 + Ret-He-RT1 < 15 # 5 # sendMail stiepel 'He-pressure RT1 too low'
Sup-He-RT2 + Ret-He-RT2 < 15 # 5 # sendMail stiepel 'He-pressure RT2 too low'
...

This is the Alarm conditions table that is used in the software. It originates from the file /wsrt/config/alarmconditions.cfg on wop10.