TCS Merge Logs Tool

There are a number of log and telemetry files from disparate systems that record information relating to various aspects of the telescope system. Consumers of these log files have a desire to generate a combined log file containing information from selectable source logs and telemetry over a given frame of time. The Merge Logs application attempts to solve this by allowing users to pull together the various log and telemetry files and construct a single, sorted file with which they can investigate problems or other issues.

See details in the Merge Logs Users Guide on the CAN
Other notes by Juan Carlos are here

GUI

There is a GUI associated associated with the command line program. It can be executed from the adsec machines from /lbt/tools/bin/mergelogsgui :

Usage

There are examples at the end of the help output from the command line:

> /lbt/tools/bin/mergelogs --help
usage: mergelogs [-h] [-s start] [-e end] [-i input-file] [-o output-file] [--list-files]
                 [--adl] [--afb] [--convert] [--keep-temp-files] [--ents] [--all-tel] [--mcspu-log]
                 [--syslog] [--events]
                 [--aosl] [--aosr] [--ecs] [--env] [--gcsl] [--gcsr] 
                 [--iif] [--mcs] [--mcs_az] [--mcs_el] [--mcspu_az] [--mcspu_el] 
                 [--oss] [--oss_adsc_dx] [--oss_adsc_sx] [--oss_swa] [--oss_terc] [--ovms] 
                 [--pcs] [--pcs_dx] [--pcs_sx] [--pmcl] [--pmcr] [--psfl] 
                 [--psfr] 

CONTROL OPTIONS:
  -h, --help          Print this message and exit
  -s, --start         The time at which to start sorting in the input files
  -e, --end           The time at which to stop sorting in the input files
                      Start and end times are any one of the following forms:
                        YYYYMMDD:HH:MM:SS[.mmm[uuu]]
                        YYYY-MM-DDTHH:MM:SS[.mmm[uuu]]
                        NNNNN.N[NNNNNNNNNNN] - MJD format
                        NNNNNNNNNNNNNNNN - Telemetry format
                      Default start time is 00:00:00.000 of the current day
                      Default end time is 23:59:59.999 of the current day
                      Times are given in UTC.
  -i, --input-file    An input file that contains a list of files
                      to be parsed and sorted.  These files are in addition
                      to any other files included using their inclusion switches.
  -o, --output-file   An output file to write the resultant sorted
                      data.  If this option is omitted the sorted
                      data will be written to standard out.
  --list-files        Lists the input files that will be parsed and 
                      sorted, then exits.
  --adl               Specifies the directories from which to pull adsec log files
                      This is a space separated list of directories.  If more than
                      one directory is specified, they must be surrounded by quotes.
                      If no directories are specified, this defaults to the current directory.
  --afb               Specifies the list of partial file names to match when searching
                      for AdSec log files.  This is a space separated list of partial
                      names.  If more than one name is specified, they must be surrounded
                      by quotes.
  --convert           Converts any HDF5 files in the input files list and 
                      converts them to CSV files.  Also sorts events logs. 
                      These intermediate files are stored in the tmp directory 
                      created in the current working directory.
  --keep-temp-files   Keeps CSV and sorted events log temporary files created 
                      in tmp during the sort.  The default deletes the tmp 
                      directory upon completion.
  --ents              Excludes normalized time stamp and source file from sorted output.

INCLUDE OPTIONS:
  --all-tel           Includes all telemetry files
  --events            Includes events log
  --syslog            Includes system log
  --mcspu-log         Includes mcspu log (not telemetry)
  --aosl              Includes aosl telemetry files
  --aosr              Includes aosr telemetry files
  --ecs               Includes ecs telemetry files
  --env               Includes env telemetry files
  --gcsl              Includes gcsl telemetry files
  --gcsr              Includes gcsr telemetry files
  --iif               Includes iif telemetry files
  --mcs               Includes mcs telemetry files
  --mcs_az            Includes mcs/az telemetry files
  --mcs_el            Includes mcs/el telemetry files
  --mcspu_az          Includes mcspu/az telemetry files
  --mcspu_el          Includes mcspu/el telemetry files
  --oss               Includes oss telemetry files
  --oss_adsc_dx       Includes oss/adsc/dx telemetry files
  --oss_adsc_sx       Includes oss/adsc/sx telemetry files
  --oss_swa           Includes oss/swa telemetry files
  --oss_terc          Includes oss/terc telemetry files
  --ovms              Includes ovms telemetry files
  --pcs               Includes pcs telemetry files
  --pcs_dx            Includes pcs/dx telemetry files
  --pcs_sx            Includes pcs/sx telemetry files
  --pmcl              Includes pmcl telemetry files
  --pmcr              Includes pmcr telemetry files
  --psfl              Includes psfl telemetry files
  --psfr              Includes psfr telemetry files

EXAMPLES:
Display this help message
  mergelogs -h

Parse and sort all telemetry files with a start time of Feb 3, 2013 7:20:20pm and end time of Feb 3, 2013 9:20:20pm
  mergelogs -s 20130203:19:20:20.000 -e 20130203:21:20:20.000 --all-tel

Parse and sort event log files with default start and end times
  mergelogs --events

Parse and sort custom input files with a start time of Feb 3, 2013 7:20:20pm and end time of Feb 3, 2013 9:20:20pm
  mergelogs -s 20130203:19:20:20.000 -e 20130203:21:20:20.000 -i /my_path/my_file.my_ext

Parse and sort custom input files with a start time of Feb 3, 2013 7:20:20pm and end time of Feb 3, 2013 9:20:20pm
with the resultant sorted data written to an output file
  mergelogs -s 20130203:19:20:20.000 -e 20130203:21:20:20.000 -i /my_path/my_file.my_ext -o /my_out_path/my_out_file.my_ext

Parse and sort adsec log files from directories with a start time of Feb 3, 2013 7:20:20pm and end time of Feb 3, 2013 9:20:20pm
with the resultant sorted data written to an output file
  mergelogs --afb "adam house keeper fastdiagn.L.0000" --adl "/local/aolog/current/" -s 20130203:19:20:20.000 -e 20130203:21:20:20.000 -i /my_path/my_file.my_ext -o /y_out_path/my_out_file.my_ext

NOTES:
  A start time greater than an end time will immediately exit the application.
  If an input file (default or custom) cannot be found, it will not be added, but execution will continue.

Potential Updates

• The application uses a local tmp directory to unpack/merge files. If the user enters a timescale accidentally that is very large, the files can be huge and fill the disk. Juan Carlos thought a sanity check could be issued to the user if it looks like the case.

Build and Installation

In SVN at: https://svn.lbto.org/view/tools/trunk/mergelogs/

The adsecdx and adsecsx boxes are CentOS 5, so it is built there. The Dec-2014 build was done in /home/aoacct/ks/mergelogs-build