Seeing Limited Trouble Shooting Guide

The following procedures list suggested steps to understand problems encountered during seeing limited operation. The error messages displayed by the AOSGUI are usually not very informative and something incorrect. Thus, the steps listed are given as a recipe to find most problem encountered.

Note, below the steps to try to find the problem will be listed for the SX AdSec (ie login to AOeng@adsecsx, etc) but the trouble shooting step for the DX Adsec are the same, with dx substituted for sx.

Engineering GUI Startup:

Usually, for the errors displayed on the AOSGUI:

• "No IDL License Available"
• "Generic Error"
• "Unknown Exception"

General Steps for Trouble Shooting:

• connect to the mountain network: lbtssh or VPN
• Login to AOeng@adsecsx and start adsceng GUI:
  
• Start AdSec Control GUI from adsceng GUI:
  
• Check "AO Arbitrator" and "AdSec Arbitrator" processes:
  
• Check that all processes are running and connected to the Msgd with adsc_check command:
  

General Solutions

1. Last resort:
   • Power Off Adsec on AOSGUI or AdSec Control GUI
   • Stop all process with adsc_stop:
     
   • Start all processes with adsc_start:
     
   • Power On AdSec on AOSGUI or AdSec Control GUI

No Elevation Information

• No Elevation information available so communication problem between the TCS/AOS and AO Supervisor:
  

Solution

1. Stop and Start AOSL from TCSGUI (Telescope Operator can do this)
2. Have TO check that MCS is running. This is the subsystem where elevation information comes from.
3. Check "AO Arbitrator" and "AdSec Arbitrator" processes. If either AO or AdSec Arbitrator is down, then restart with button click:
   
4. If a process is running but not connected to Msgd (see adsc_check above) then stop and start via System Process GUI:
   
5. As a last resort, the entire AdSec can be powered off, processes stopped and restarted and powered back on (see for instructions)

No Wind Information

* If no wind information is available then there is a problem with the Anemometer Process:

Solution

• Run adsc_check command to see if Anemometer Process is down or not connect to Msgd.
  
• If there is a problem then stop and start the Anemometer Process via the System Process GUI:
  

No IDL License Available

* If the AOSGUI pops up an error message box, there is a 50\% chance that there really is an IDL License problem.

IDL Zombie Process

* If adsc_check shows idlctrl to be a Zombie Process, IDL must be stopped and restarted:

Solution

1. Stop the IDL Control via the System Process GUI (see above)
2. Start the IDL Control via the System Process GUI
3. Start an IDL Terminal via the adsceng GUI (see above)
4. At the idl prompt type AdOpt> print, fsm_load_program(/auto) (note, you will see output scrolling on the AdSec Control GUI. This take more than a minute)
5. The AdSec should be in a Ready state
6. Stop the AdSec Arbitrator via the System Process GUI
7. Start the AdSec Arbitrator via the System Process GUI
8. The AdSect is ready ready to "SET"

This solution disconnects the IDL controlling program that communications to the AdSec by Stopping the IDL Control Process. When IDL is restarted, id does on reconnect to the AdSec. The IDL> command fsm_load_program(/auto) reconnects to the AdSec and reloads the DSP code. IDL will then have full communication to the AdSec. The AdSec Arbitrator process usually does not reconnect automatically to the IDL process, so this arbitrator must be stopped and restarted to force a reconnection. In addition, the *fast diagnostic* process will usually reconnect to the AdSec when the DSP codes are loaded via the *fsm_load_program(/auto)* command. Be sure the *fast diagnostic* process is running and connect to the Msgd before the DSP load code request is executed.

• As a last resort, the entire AdSec can be powered off, processes stopped and restarted and powered back on (see for instructions)

IDL License "Really" Not Available

• To determine if IDL really has lost its connection to the license server:
  1. The adsc_check command will show the idlctrl as "NOT running"
  2. In an adsecsx terminal type idl. If a license is available then IDL will start:
     

Solution

• SX: The SX AdSec computer has an INAF stand alone license, so a Stop and Start of the IDL Control Process should solve the problem. The DSP code must be reloaded via the IDL> *print, fsm_load_program(/auto)* command. If the Stop and Start IDL Control Process does not solve the problem then there is some larger IDL problem
• DX: The DX AdSec computer uses the LBTO license server. Thus, if a Stop and Start of the IDL Control Process does not solve the problem then there is a larger IDL problem. The starting of IDL in an adsecdx terminal will fail (ie go in to demonstration mode) if there is a problem with the LBTO IDL server.

IDL License Not The Problem

In some cases error unrelated to IDL are displayed on the AOSGUI as an IDL License Not Available. In these cases, the Engineering GUIs must be examined to see if there are other process that are stop or in a Zombie state. If so, then usually restarting the process will correct the problem

*NOTE* if the *fast diagnostic* process has to be restarted, then the command *fsm_load_program(/auto)* must be executed at an IDL> prompt so a reconnection is made.

-- DougMiller - 18 Dec 2012