Housekeeping and maintenance procedures @ LBTO
On each AO server at LBTO a number of procedures are regularly scheduled to perform housekeeping tasks and help in maintenance operations.
Gathering and archiving system performance data
In order to monitor the global performance of the AO servers the
package has been installed on adsecdx, wfsdx, adsecsx, wfssx.
The package sets up the
data gathering procedure which samples the main performances indicators every 10 minutes.
A procedure to save sar data from adsecdx, wfsdx, adsecsx, wfssx has been installed on the machines. The procedure is named
(see attachment) and is started from cron (user:
) a little after midnight.
Data files are saved on directory:
using the account
The related data files are to be found in:
Performance data can be analyzed with
, and other system utilities.
Python procedures to read performance data files (
) and generate plots (
) have been developed (see attachments)
Crontab settings to archive sar data:
10 0 * * * python /home/aoroot/sardata.py > /home/aoroot/sardata.log 2>&1
Disk space low alarm
A procedure to check the free disk space and send alarm via e-mail has been installed. It is based on a python program (
: see attachments) which is launched daily as user
0 8 * * * /home/aoacct/scripts/disk-alarm.py > disk-alarm.log 2>&1
Here is a brief description (as generated by
running on wfsdx
disk-alarm. Vers 2.6 - L.Fini Dec. 2011 on: wfsdx.lbt.as.arizona.edu
This procedure executes the command 'df -ka' and parses the output to find
the data occupation of disks.
When any disk capacity is under the configured treshold, a warning message
is sent to a list of e-mail addresses.
python disk-alarm.py [-c] [-h] [-t] [-q]
-q Quiet mode (do not show info on stdout)
-t Send only test email to configured addresses
-c Create or modify configuration
-h Print this page and exit
Usually it should be run periodically (e.g.: from cron) with no options.
Config file : /home/aoacct/.disk-alarm
Disk list: /dev/sdb1, /dev/mapper/VolGroup01-LogVol00
E-mail addrs: firstname.lastname@example.org, email@example.com
Mail server: localhost
Sender addr: firstname.lastname@example.org
: the procedure stores configuration data in a file on the user's home directory: