Mergelogs

The mergelogs is an application developed by Christopher Cox and Juan Carlos Guerra in C++ to help to read the multiple logs and to merge them in intervals of times. The application can be used from the command line or a GUI. The mergelogs has been built to be used for any TCS logs. The application with the TCS logs are not yet tested.

The IT person in contact about the mergelogs application is Kellee Summers.

Important

The application use the tmp directory to unzip the files and it is hardly recomended to set the timestamps in a maximum of two days intervals. It can be applied to more days but this will take too long and memory from the Adsec running system.

Setup

The application is not set in the bin of the adsec machines, but it can be launched from a local directory.

1.- Go to a local directory in AOeng account (eg:/ home/aoacct/merge/)

2.- From the directory execute: /lbt/tools/bin/mergelogs []

Syntaxis

An example of using the mergelogs application on AdsecDX machine:

[AOeng@adsecdx jcg]$ /lbt/tools/bin/mergelogs --afb "adsecarb housekeeper fastdiagn" --adl "/local/aolog/" -s 20130203:19:20:20.000 -e 20130203:21:20:20.000 -o /home/aoacct/jcg/test.txt
Decompressing: /local/aolog/adsecarb.R.00001386472519.log.gz
Decompressing: /local/aolog/housekeeper.R.00001386519120.log.gz
Decompressing: /local/aolog/fastdiagn.R.00001386438066.log.gz
Finished: tmp/adsecarb.R.00001386472519.log
Finished: tmp/housekeeper.R.00001386519120.log
Finished: tmp/fastdiagn.R.00001386438066.log
Start parsing...
End parsing.

The option -i will give you the possibility to have a text file with the logs that you want to search. The logs files have to exist in the time frame that you want to merge. The file has to have for each line the full path of the log name.

Example of list file (file.list)

Filename: file.list

/local/aolog/fastdiagnostic.R.log

/local/aolog/adsecarb.R.log

[AOeng@adsecdx jcg]$ /lbt/tools/bin/mergelogs -s 20141208:17:20:20.000 -e 20141208:21:20:20.000 -i /home/aoacct/jcg/list.list -o /home/aoacct/jcg/test.txt

Help

[AOeng@adsecdx jcg]$ /lbt/tools/bin/mergelogs -help
usage: mergelogs [-h] [-s start] [-e end] [-i input-file] [-o output-file] [--list-files]
[--adl] [--afb] [--convert] [--keep-temp-files] [--ents] [--all-tel] [--mcspu-log]
[--syslog] [--events]
[--aosl] [--aosr] [--ecs] [--env] [--gcsl] [--gcsr]
[--iif] [--mcs] [--mcs_az] [--mcs_el] [--mcspu_az] [--mcspu_el]
[--oss] [--oss_adsc_dx] [--oss_adsc_sx] [--oss_swa] [--oss_terc] [--ovms]
[--pcs] [--pcs_dx] [--pcs_sx] [--pmcl] [--pmcr] [--psfl]
[--psfr]

CONTROL OPTIONS:
-h, --help Print this message and exit
-s, --start The time at which to start sorting in the input files
-e, --end The time at which to stop sorting in the input files
Start and end times are any one of the following forms:
YYYYMMDD:HH:MM:SS[.mmm[uuu]]
YYYY-MM-DDTHH:MM:SS[.mmm[uuu]]
NNNNN.N[NNNNNNNNNNN] - MJD format
NNNNNNNNNNNNNNNN - Telemetry format
Default start time is 00:00:00.000 of the current day
Default end time is 23:59:59.999 of the current day
Times are given in UTC.
-i, --input-file An input file that contains a list of files
to be parsed and sorted. These files are in addition
to any other files included using their inclusion switches.
-o, --output-file An output file to write the resultant sorted
data. If this option is omitted the sorted
data will be written to standard out.
--list-files Lists the input files that will be parsed and
sorted, then exits.
--adl Specifies the directories from which to pull adsec log files
This is a space separated list of directories. If more than
one directory is specified, they must be surrounded by quotes.
If no directories are specified, this defaults to the current directory.
--afb Specifies the list of partial file names to match when searching
for AdSec log files. This is a space separated list of partial
names. If more than one name is specified, they must be surrounded
by quotes.
--convert Converts any HDF5 files in the input files list and
converts them to CSV files. Also sorts events logs.
These intermediate files are stored in the tmp directory
created in the current working directory.
--keep-temp-files Keeps CSV and sorted events log temporary files created
in tmp during the sort. The default deletes the tmp
directory upon completion.
--ents Excludes normalized time stamp and source file from sorted output.

INCLUDE OPTIONS:
--all-tel Includes all telemetry files
--events Includes events log
--syslog Includes system log
--mcspu-log Includes mcspu log (not telemetry)
--aosl Includes aosl telemetry files
--aosr Includes aosr telemetry files
--ecs Includes ecs telemetry files
--env Includes env telemetry files
--gcsl Includes gcsl telemetry files
--gcsr Includes gcsr telemetry files
--iif Includes iif telemetry files
--mcs Includes mcs telemetry files
--mcs_az Includes mcs/az telemetry files
--mcs_el Includes mcs/el telemetry files
--mcspu_az Includes mcspu/az telemetry files
--mcspu_el Includes mcspu/el telemetry files
--oss Includes oss telemetry files
--oss_adsc_dx Includes oss/adsc/dx telemetry files
--oss_adsc_sx Includes oss/adsc/sx telemetry files
--oss_swa Includes oss/swa telemetry files
--oss_terc Includes oss/terc telemetry files
--ovms Includes ovms telemetry files
--pcs Includes pcs telemetry files
--pcs_dx Includes pcs/dx telemetry files
--pcs_sx Includes pcs/sx telemetry files
--pmcl Includes pmcl telemetry files
--pmcr Includes pmcr telemetry files
--psfl Includes psfl telemetry files
--psfr Includes psfr telemetry files

EXAMPLES:
Display this help message
mergelogs -h

Parse and sort all telemetry files with a start time of Feb 3, 2013 7:20:20pm and end time of Feb 3, 2013 9:20:20pm
mergelogs -s 20130203:19:20:20.000 -e 20130203:21:20:20.000 --all-tel

Parse and sort event log files with default start and end times
mergelogs --events

Parse and sort custom input files with a start time of Feb 3, 2013 7:20:20pm and end time of Feb 3, 2013 9:20:20pm
mergelogs -s 20130203:19:20:20.000 -e 20130203:21:20:20.000 -i /my_path/my_file.my_ext

Parse and sort custom input files with a start time of Feb 3, 2013 7:20:20pm and end time of Feb 3, 2013 9:20:20pm
with the resultant sorted data written to an output file
mergelogs -s 20130203:19:20:20.000 -e 20130203:21:20:20.000 -i /my_path/my_file.my_ext -o /my_out_path/my_out_file.my_ext

Parse and sort adsec log files from directories with a start time of Feb 3, 2013 7:20:20pm and end time of Feb 3, 2013 9:20:20pm
with the resultant sorted data written to an output file
mergelogs --afb "adam house keeper fastdiagn.L.0000" --adl "/local/aolog/current/" -s 20130203:19:20:20.000 -e 20130203:21:20:20.000 -i /my_path/my_file.my_ext -o /my_out_path/my_out_file.my_ext

NOTES:
A start time greater than an end time will immediately exit the application.
If an input file (default or custom) cannot be found, it will not be added, but execution will continue.

GUI

The GUI is recomended to be used at Steward or in the telescope, the pop up and control of the GUI is a bit slow from other places.

The GUI can be launched from the home directory as:

[AOeng@adsecdx jcg]$ /lbt/tools/bin/mergelogsgui &

Screen_Shot_2014-12-08_at_12.25.46_PM.png

Upgrade

  • A grep option. The application can launch a grep keywords or combination of them to search and output the file.
  • Timestamp control. it wouldn't allow a large set of time intervals (years or months).
  • Color code. A color code can be applied for keyword WAR,ERR,FAT,FUNCTEMERGENCYSTOP,CHDISTAVERAGE,Coils disabled ,etc.. in the output of the grep option or the merging.

-- %USERSIG{JuanCarlosGuerra - 2014-12-08}%

Comments


 
Topic revision: r2 - 08 Dec 2014, JuanCarlosGuerra
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback