Housekeeping and maintenance procedures @ LBTO

On each AO server at LBTO a number of procedures are regularly scheduled to perform housekeeping tasks and help in maintenance operations.

Gathering and archiving system performance data

In order to monitor the global performance of the AO servers the sysstat package has been installed on adsecdx, wfsdx, adsecsx, wfssx.

The package sets up the sadc data gathering procedure which samples the main performances indicators every 10 minutes.

A procedure to save sar data from adsecdx, wfsdx, adsecsx, wfssx has been installed on the machines. The procedure is named sardata.py (see attachment) and is started from cron (user: aoroot) a little after midnight.

Data files are saved on directory: obelix.arcetri.astro.it:/mirror0/SAR using the account archiver.

The related data files are to be found in: /var/log/sa.

Performance data can be analyzed with sar, sadc, and other system utilities.

Python procedures to read performance data files (sarf.py) and generate plots (sarplot.py) have been developed (see attachments)

Crontab settings to archive sar data:

10 0 * * * python /home/aoroot/sardata.py > /home/aoroot/sardata.log 2>&1

Disk space low alarm

A procedure to check the free disk space and send alarm via e-mail has been installed. It is based on a python program (disk-alarm.py: see attachments) which is launched daily as user AOeng by cron:

0 8 * * * /home/aoacct/scripts/disk-alarm.py > disk-alarm.log 2>&1

Here is a brief description (as generated by disk-alarm.py -h running on wfsdx)):

disk-alarm.  Vers 2.6 - L.Fini Dec. 2011 on: wfsdx.lbt.as.arizona.edu

This procedure executes the command 'df -ka' and parses the output to find
the data occupation of disks.

When any disk capacity is under the configured treshold, a warning message 
is sent to a list of e-mail addresses.

Usage: 
         python disk-alarm.py [-c] [-h] [-t] [-q]
         
Where: 
        -q     Quiet mode (do not show info on stdout)
        -t     Send only test email to configured addresses
        -c     Create or modify configuration
        -h     Print this page and exit

Usually it should be run periodically (e.g.: from cron) with no options.


Current configuration:
  Config file : /home/aoacct/.disk-alarm
  Config.Vers.: 15
     Disk list: /dev/sdb1, /dev/mapper/VolGroup01-LogVol00
      Treshold: 10%
  E-mail addrs: lfini@arcetri.astro.it, puglisi@arcetri.astro.it
   Mail server: localhost
   Sender addr: disk-alarm@lbto.org

Note: the procedure stores configuration data in a file on the user's home directory: $HOME/.disk-alarm.