You are here: Foswiki>Main Web>TWikiUsers>KelleeSummers (06 Sep 2018, KelleeSummers)Edit Attach

Kellee Summers' Page

Troubleshooting Main Pages

After reboot of web machine in Tucson, check:
  • telemetry data needs to be available at /lbt/telemetry_data
  • cron table is sometimes a link, sometimes a file - make sure /var/spool/cron/web matches /web/server/cron/web.tucson
  • make sure the teladmin account is available - it's used to rsync between the mountain and Tucson

Responsibilities

Application(s) Description Status
Status Server web application for overall observatory status needs real-time graphs (hourly, daily, etc.)
INDI interfaces for ALH, instruments, etc. still in very early stages of development; some use by new instruments
LBC we have complete control of the LBC software new BB controllers, upgrade firmware on iBBs
MODS instrument software handover someday - currently still in OSU's purview
GCS Guiding Control Subsystem of the TCS
Xianyu is responsible for the wavefront sensing portion, I am just guiding 		interface to AzCam needs to be more robust - lots of ITs associated with that
AzCam PEPSI folks take care of their machines/software for this, but we control LUCIs, MODS;
PEPSI does use the same version, etc. 		upgrade someday to latest version from ITL
OAC off-axis control server that does motion control for LUCI/PEPSI-PFU guide probe in AGw HK upgrade in progress
Telemetry software lib used by TCS, OVMS, LUCI, future instruments support to new users of the lib; mostly just file maintenance duties now
FACSUM
WEATHERSTATION 		big display GUIs maintenance, FACSUM has a memory leak, so it is restarted daily;
AO state is incorrect when using ARGOS
should be web interfaces someday?
ALH EPICS IOCs for TCS/instruments and IT maintenance mode
Weatherstation low-level software that interfaces to the Vaisala systems maintenance; needs to be rehosted from Windows Borland C to Linux
DIMM seeing measurements we need auto-focus implemented to prevent operators from playing with the focus
OVMS vibration monitoring system OVMS+ has another
particle monitors telemetry implementation for the portable particle monitors done - waiting to be used

Project Notes

Task List

Visualization
bug in telem_vis when selecting fields, when both is selected - you cannot get left (both selections give you right)
dictionary creation doesn't work to include the new luci subsystems (since they are softlinks)
h5csv generates "" for empty descriptions instead of just blank - used to break the web page, but DS fixed it - should fix it anyway
Re-generate the collection-doc doxygen so it has all the right units; delete references to diagnostic streams and speeds
Need a nicer telemetry client for instruments to use; a python or Java example would be good
script to clean up telem-vis-xxx.csv files in tmp
auto create the operations stream CSV file daily so it's not re-created for each person that looks at it?
maybe this is handled by the canned plots?
Need a telem conversion tool and associated telemetry library that accepts a date as an argument so that we can process other data and create HDF5 telemetry files from other sources. This will be used for TBAD output and ARGOS vibration data.
Take one of the simple programs I wrote to rebuild weather and DIMM telemetry streams and make it more generic - accepting a date for the created stream. This way we can use it when we want to put ARGOS vibration and TBAD data into HDF5 files.
INDI
change the timing of the indi.fcgi to be >1 second - maybe up to 30secs? right now it's sub-second and hammering the apache logs with POST messages
started cleanup of AO drivers/server to be consistent with the rest; ready to test the exes and scripts for startup
Weather
Fix weather station use of client connections - get rid of connection on client error (http://www.borlandtalk.com/esocketerror-asynchronous-socket-error-10053-vt12190.html)
update the weather doc on the CAN
What config is running on the front weather station? Cannot access it with the COM0 port.
what's necessary to make the ENV PC a spare for the weatherstation PC?
oacontrol
updates for new housekeeping
add telemetry and INDI interfaces; IOC interface - same as what we have for PMC
AzCam
update agw7-cam to same 5.1 version as the other AzCam servers - this will allow us to get logging working
make cam machines auto-login like CB did to the weatherstation PC
GCS should stop when it gets azcam errors like on the resetController
GCS should have a client program to do the resetController
mine the GCS logs for timing of the MODS2 and LUCI2 probe motions
create diagrams for all the AGW configurations like the AGW1/AGW2 diagram
Misc
the MODS email and EPICS threshold uses 1, but is the script not getting decimal values? So, triggering an alarm when the value (glycol supply pressure) was really 1.7, and not really 1?
add igpower and/or dewar pressure to MODS alarm channels
what's left to test for the startup/shutdown script?
put the old OVMS installation docs in the VCAN from here: https://zero.as.arizona.edu/wiki/pages/h509e6b7/OPD_and_Vibration_Monitoring_System.html
Big Displays
port to web!
build/release procedures are not up to date
clean up the installation areas and SVN to delete unused stuff (old WDA)
port to 64-bit?
Document Updates
update the telem docs on the CAN with OBSOLETE in the watermark or something
update the DIMM docs on the CAN with OBSOLETE in the watermark or something - or delete the Kastubh docs
build procedures are not up to date for Alarm Handler - separate for "EPICS from scratch" and "ALH" (32-bit) and "iocs" (64-bit)

Notes

Cron jobs on ssh machine: 

### run a check of the virtual mem used by all TCS systems twice an hour
### (emails if over the threshold)
00,30 00,01,02,03,04,05,06,07,08,09,10,11,12,13,14,15,16,17,18,19,20,21,22,23 * * * /home/ksummers/checkVirtual.sh 
### log the virtual mem used by all TCS systems twice a day
10 00,15 * * * /home/ksummers/logVirtual.sh >> /home/ksummers/tcsVirtualMem.log

Transitioned LBC and OBS3 tkloggers to use a GUI-less version on 24-Feb-2017. So, now LBC and OBS3 have cron jobs to restart their tcllogger programs each day after rollover of the file.

LBC 
00 14 * * * /home/lbccontrol/bin/emailLogErrors.sh > /dev/null 2>&1
10 14 * * * /home/lbccontrol/bin/moveTechFITSFiles.sh > /dev/null 2>&1
## restart tcllogger on LBC daily - kills the currently running one.
10 19 * * * /home/lbccontrol/bin/start_tcllogger.sh  > /dev/null 2>&1

OBS3 
## twice an hour check the MODS temps, set alarm conditions
15,45 * * * * /home/telescope/bin/emailMODSErrors.sh  > /dev/null 2>&1
20,50 * * * * /home/telescope/bin/emailMODS2Errors.sh  > /dev/null 2>&1
## each day check ALH log for instrument complaints
10 14 * * * /home/telescope/bin/emailInstrAlarms.sh  > /dev/null 2>&1
## restart tcllogger on daily to look for telemetry errors 
10 00 * * * /home/telescope/bin/start_tcllogger.sh  > /dev/null 2>&1


TCS Notes


Testing GCS

moved to Software/GCSSoftwareMainPage#Testing_GCS

AOS States and FACSUM

aos.side[].ao.sl_ready aos.side[].ao.ao_ready aos.side[].ao.loopon STATE TEXT
0 0 0 REST REST, if AOS is running, ADSEC-OFF if AOS not running
0 0 1 REST REST, if AOS is running, ADSEC-OFF if AOS not running
0 1 0 REST REST, if AOS is running, ADSEC-OFF if AOS not running
0 1 1 REST REST, if AOS is running, ADSEC-OFF if AOS not running
1 0 0 FLAT blank, Paused, Lost, Guiding based on other flags
1 0 1 REST REST, if AOS is running, ADSEC-OFF if AOS not running
1 1 0 OPEN blank, Paused, Lost, Guiding based on other flags
1 1 1 CLOSED AO-LOCK

Topic attachments
I Attachment Action Size Date Who Comment
NUCPicture.jpgjpg NUCPicture.jpg manage 601 K 26 Jan 2015 - 20:15 UnknownUser NUC power button
Edit  | Attach | Print version | History: r134 < r133 < r132 < r131 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r134 - 06 Sep 2018, KelleeSummers
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback