Adam

Adam server

An adam server has been proposed to replace the adam library and have a single point of communication with the Adam.

1) Manages the Adam watchdog preventing the mirror from turning off
2) Acts as a server for all Adam communications from other processes
3) Checks the overall software status and turns off the mirror if some required process is not alive. To be implemented using a variable to which each process writes regularly to signal that it is still alive

This server was postponed, just updating the library is probably enough.

Adam watchdog

Points 1) and 3) of the adam server (the watchdog) must be implemented anyway somewhere else, probably in the AdSec Arbitrator, and removed from the MirrorCtrl

Assigned to: LF - status: OPEN

Adam simulation mode

Several adsec-related programs (adsec arbitrator, GUI) have an instance of the Adam library that will immediately try to connect with the Adam hardware. A simulation mode is needed to allow testing these programs offline.
Assigned to: none - status: OPEN

AdSec Arbitrator

thSendMsg bug

AdsecArbitrator sometimes dies because of a thSend msg error. Cause unknown.

Assigned to: none - status: OPEN

Arbitrator failure when ripping shell.

A RIP request to the mirror causes a Failure to AdSecArb. (AdSecArb \x{2192} Fast \x{2192} IDL \x{2192} Fast). Must be avoided temporarilty changing the FastDiagnostic safety limits.

A new command will be available via the FastDiagnSystem library in order to switch to a custom set of thresholds (in this case more relaxed) and then to switch back to the previously backed up values (this because the thresholds were not necessary the same of the configuration file as they might have been changed). The Adsec arbitrator will be in charge of calling this function before and after completion of the RIP sequence (care should be taken in order to avoid leaving the relaxed thresholds during normal operation)

Assigned to: JA - status: CLOSED -

Lab mode

AdSec arb has a "lab mode" where it doesn't check environment parameters (elevation, wind speed, etc) and this avoids ripping the shell. Both engineering gui and AOS gui show a big red warning label when in lab mode.

status: CLOSED

Switch BCU command

Implement:

- AO arbitrator (in response to a PresetAO command)
- AdSec arbitrator command to MasterD to set the tube IP address (move cfg. of IP from masterd. conf to adsec conf. Need a list of possible IP addresses).
- IDL command to BCU sent by the same arbitrator command
- Aggiungere all'adsec arbitrator una variabile che dice qual'รจ la stazione focale corrente.

Assigned to: MX (IDL) AP (C++) - status: CLOSED

Dump

apparently is pretty well defined but on random paper. Write it on twiki.

Dumpimg mechanismo foreseen for diagnostics and troubleshooting, not the same as operational logging.

A dump is triggered when:
  • The shell is ripped due to safety reasons (see below)
  • Anomalie (i.e. values out of range but not necessarily under an emergency or malfunction condition)
  • On demand (via the library)
The dump should be saved under $ADOPT_LOG/Adaptive_Secondary_ timestamp.dump

The useful information belongs to all the FastDiagnostics (actually the MirrorCntrl), HouseKeeper and AdamHouseKeeper.

Two new parameters are suggested for the library:
  • Frame number specification
  • Text comment in order to specify any comment and the reason of the dump
Conditions for the shell ripping:
Assigned to: JA - status: OPEN -

Safe RIP command

The fsm_rip() command goes through IDL, and may stall for a lot of time if IDL is busy. This must be avoided for safety reasons.
The Arbitrator routine that sends this command must someway detect if IDL is not answering fast enough, and execute a RIP bypassing IDL (e.g. sending a command directly to Adam).

New rip procedure:

- change fastdiagn. thresholds etc.
- send command to Adam to rip the shell quickly
- call IDL command fsm_rip()
- restore fastdiagn thresholds etc.

Assigned to: none - status: CLOSED

Detect IDL crash

IDL sometimes crashes. This must be detected, either in the IDL controller or somewhere else, and a new IDL must be started, together with the necessary safety action (e.g. ripping the shell if it was set, opening the loop if it was closed, etc). This might be implemented as a sudden Alarm coming from the IdlSubsystem.

Assigned to: none - status: OPEN

Flat selection

Flat saving/loading from AdSec engineering interface has a few bugs in name generation that should be resolved. Also to consider whether the current scheme with a new directory for each flat, with only one file inside, should be kept or another scheme considered.

Assigned to: none - status: OPEN

AO Arbitrator

New FSM

Implement new FSM for AOarbitrator, as described in AoArbitratorFsm
Assigned to: AP - status: CLOSED

Error recovery

Implement new AO arbitrator error recovery scheme, as described in AoArbitratorFsm.

Assigned to: AP - status: OPEN

Progress bars

Arbitrators should report to AOS progress information during long commands (e.g. set mirror flat, powering on WFS, etc) so that a progress bar can be shown to the operator. Can be implemented using variables that the AOS reads.

Assigned to: none - status: OPEN

Libraries

Logger

Luca implemented new logger together with new AOApp, to be integrated into all software one piece at a time.

UPDATE: In order to reduce the porting overhead and minimize the chances introducing compatability problems while solving other problems it was agreed to fix the current Logger instead of using a new library, with the following requirements:
  • Fix the rename method so it work across different file systems.
  • Use syslog in case no file can be opened (specified one or temporal), stderr shall not be used.
  • Logger should not read the ADOPT_LOG environment variable, shoult use '/tmp' unless otherwise is specified (to be discussed).
  • Fix individual and global logging level mechanism.
  • Telemetry log files will have a '.tel' extension which will contain only telemetry values.
  • Although it was previously agreed that there will be only one log file per application, the multiple files ability is kept but it's adviced to avoid it's use (i.e. avoid adding an extra name to the Logger:get)
  • General code clean up, specially related to the current path, file name and level.
Assigned to: JA - status: UNDER TEST - to be tested

New configuration

Change the current conf/ and calib/ directories to allow for multiple subsystems in the future (FLAO, LBTI, etc). Revise the process configuration scheme, removing dependencies from Python stuff and adopting the new naming standard described in CommonFeatures .

Assigned to: AP - status: CLOSED

IDL

Non-interactive server and display

The current IDL server is used interactively and cannot function as a real server because it will try to open windows etc. on the local machine. This is the suggested replacement:
  1. Use idl RPC to communicate with IDL server instead of the current idlCtrl process that uses stdin/out/err. The idl RPC library will forbit any graphics operation.
  2. replace all graphics calls in IDL calibration procedures (plot, tvscl, etc) with wrappers that will send the data to another IDL process that will display this data.
  3. In most cases, this display is enough. In some rare cases, real interaction with IDL is needed (e.g. Xmanager with widgets, buttons, etc). For these cases, a non-rpc idl modality must be foreseen, with the understanding that it will not used during normal observation. The current use case for this non-rpc modality is shell mounting/dismounting, which is done in the clear room
ROADMAP

1. Implement an IDL RPC server with an interface identical to the current IdlCtrl (takes a command string, executes IDL, and returns stdout and stderr).
2. Implement the switch between IDL RPC and IdlCtrl so that Arbitrators and terminals can send commands to either one or the other
- Switching is done by IdlCtrl that spawns an idl or an idlrcp depending on a config. file setting or command line option
3. Test all Arbitrator IDL commands (fsm_*) until they completely work with IDL RPC
4. tbd

Assigned to: AP - status: OPEN

Conf/calib/meas data

Write a document detailing what must be stored and where, and the data retain policy.

Assigned to: AP - status: CLOSED see PathChanges

Implement it

Work started on branch "newpaths"

GUI

AI#2: GUI AdSecArbitrator \x{2192} show zern applied. (side+ADSEC.ZERN_APPLIED)
AI#20. Embed AdoptControl in AOSGUI \x{2192} Guido has to write email AI#21. Embed PANIC actions in AOS GUI \x{2192} Guido has to write email
AI#24a The logs has to be removed from GUIs (AdsecArb and AdoptControl) and only alert should be displayed. Think also about “INFO” for execution success (TCS style). Show also in a tab of AdSecArbGUI the variable side+ADSEC.LAST_FAULT_ANALYSIS. AdSecArb must null that variable aft er a fault_recovery button pressed.
AI#26. All buttons has to be grayed when accepted and running (AdOptControl?)
AI#30. Adjust powerbackplane temperatures on Housekeeper GUI
AI#36. Check dew point variable non displayed in AdSecArbGUI
AI#39. Housekeeper GUI has to show time/min instead time sec
AI#45. Some force not correctly displayed in AdSecMirGui
AI#47. AOSGUI: large fields and scrolling horiz

Other

AI#4. DONE Argument removal from conf files. The side and some other parameter has to be removed from them. (PAUSE: Alfio in charge on that)
AI#5. Telemetry. Luca started an http server for data collecting and packaging. The idea is to have the same server for local and remote queries.
AI#5a: Backup telemetry data on Telescope. (ADSEC->housekeeper, fastdiagnostic, dumps sav, flat and WFS->?)
AI#6: Dump of Fastdiagnostic and Housekeeper: add to param file one field asking for dump in case of alarm or create a FunctEmergenctyStopandDump
AI#7. Guido requested a slow process for following actuators “jumping” and acting nulling ICC gain (marco)
AI#9. Guido requested a workaround for only 1 actuators at maximum force: it has to not reset the shell
AI#12. AOS and MSGD: always up and every command pass through MSGD, even process start/kill and restart.
AI#17. AOS and unit B: think about how to manage all the tests in solar tower/clean room where the AOS can be not available (LAB mode to be tested)
AI#24. Logging and log policy: Javier suggested VLT like log style. Every morning either log and telemetry are packed and stored in a database. During the night you should not use log file. If you need you can access them manually without specific tools (or f.e. with a refurbished version of logviewer.py).
AI#27. Check if logger of Housekeeper act as the others backing up the old log file \x{2192} seems has problems with file archiving. AI#32-33-34: Housekeeper/Fastdiagnostic has to update mirror general status .
AI#38. idlctrl.conf: remove from conf file the names embedded
AI#40: Aoarbitrator: has to be aware of AdSec and WFS status, even particular conditions: f.e. wind/elevation.
AI#41. adsc_stop not kill GUIs and all stuff.
AI#42. !!!! IDL ROUTINE TSS_ENABLE(), TSS_DISABLE() but EMPTY!!!!!!!!!!!!!!!!!!!!!!
AI#46. flat selection: done with “Applicable”
AI#52: HO offload computation of fastdiagnostic: reset the “history” when a new reference or matrix is set.
AI#53: thread slow of fastdiagnostic to be tested and explore possibility of a thread slow with timer events (ex: mean of positions @5hz for 2 minutes) AI#54: IDL has to act only as slave (no commands to other processes)
AI#57 develop plan for always start IDLCTRL on vncserver or NX and implement it.
AI#58 detect idl crash (without crashing idlctrl)
AI#59: automatically acqure accelerolmeter with mirrorctrl o with slow process....
AI#60: embed new I4D trigger managed with Pyro
Marco has proposed an additional safe mode state, to which the FSM goes automatically when something prevents setting the shell (e.g. elevation less than 20).
Topic revision: r47 - 05 Oct 2010, AlfioPuglisi
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback