Housekeeping and maintenance procedures @ LBTO

On each AO server at LBTO a number of procedures are regularly scheduled to perform housekeeping tasks and help in maintenance operations.

Gathering and archiving system performance data

In order to monitor the global performance of the AO servers the sysstat package has been installed on adsecdx, wfsdx, adsecsx, wfssx.

The package sets up the sadc data gathering procedure which samples the main performances indicators every 10 minutes.

A procedure to save sar data from adsecdx, wfsdx, adsecsx, wfssx has been installed on the machines. The procedure is named (see attachment) and is started from cron (user: aoroot) a little after midnight.

Data files are saved on directory: using the account archiver.

The related data files are to be found in: /var/log/sa.

Performance data can be analyzed with sar, sadc, and other system utilities.

Python procedures to read performance data files ( and generate plots ( have been developed (see attachments)

Crontab settings to archive sar data:
10 0 * * * python /home/aoroot/ > /home/aoroot/sardata.log 2>&1

Disk space low alarm

A procedure to check the free disk space and send alarm via e-mail has been installed. It is based on a python program ( see attachments) which is launched daily as user AOeng by cron:

0 8 * * * /home/aoacct/scripts/ > disk-alarm.log 2>&1

Here is a brief description (as generated by -h running on wfsdx)):

disk-alarm.  Vers 2.6 - L.Fini Dec. 2011 on:

This procedure executes the command 'df -ka' and parses the output to find
the data occupation of disks.

When any disk capacity is under the configured treshold, a warning message 
is sent to a list of e-mail addresses.

         python [-c] [-h] [-t] [-q]
        -q     Quiet mode (do not show info on stdout)
        -t     Send only test email to configured addresses
        -c     Create or modify configuration
        -h     Print this page and exit

Usually it should be run periodically (e.g.: from cron) with no options.

Current configuration:
  Config file : /home/aoacct/.disk-alarm
  Config.Vers.: 15
     Disk list: /dev/sdb1, /dev/mapper/VolGroup01-LogVol00
      Treshold: 10%
  E-mail addrs:,
   Mail server: localhost
   Sender addr:

Note: the procedure stores configuration data in a file on the user's home directory: $HOME/.disk-alarm.

I Attachment Action Size Date Who Comment manage 8 K 30 Dec 2011 - 10:03 UnknownUser Disk space checking procedure manage 4 K 21 Oct 2011 - 13:04 UnknownUser Sar data gathering and store procedure manage 3 K 29 Dec 2011 - 15:34 UnknownUser Performance data file reader (needed by manage 11 K 29 Dec 2011 - 15:35 UnknownUser Performance data plotting utility
Topic revision: r2 - 30 Dec 2011, LucaFini
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback