YYYY-MM-DD - test gain ramp failure
Software versions used:
stable, test.YYYYMMDD.reorder-gain-rampSoftware version after test:
Test 1: Gain ramp IDL procedure
Servers and software versions used:
soul-sxwfs: stable (no change)
The previous addition of the gain ramp performed ramping in the order of: 1) set gain (ramped), 2) set gain (original), 3) set AdSec
state to AORunning. If it completed successfully, the AdSec
Arbitrator state would also advance to AORunning. Jenny noted in the nightly report for 2022-02-20
that this should be reordered so that the state changes before the gains are updated. However, this could result in a problem if the second set gain fails (such as from a communication timeout) before the state is updated, because the AdSec
Arbitrator states could remain in LoopSuspended
with non-zero gains if the second set gain were to fail.
New code: (GH #106
Xianyu's updated IDL procedure takes care of this by first attempting to set the gains back to zero if the fsm_set_gain fails, and if this also fails, causing the shell to RIP. If setting the gains back to zero succeeds, it will return an IDL_GAIN_RAMP_ERROR, which the AdSec
Arbitrator will then respond to by moving to a failure state, which a person can then manually recover from as necessary (Panic, or in the future, recover fail.)
To test that the gain ramp works as intended, we just need to close the loop and execute Pause and Resume commands as normal, while tailing the IDL logs to see that the command execution is working. We may also want to record optical loop data to verify that the frames are being affected by the scaled gains.
- To test if the error handling performs properly for the condiiton that the first fsm_set_gain fails, we can create a FITS file with the incorrect number of modes, and before a PauseAo, update the RTDB variable ADSEC.L.G_GAINS_A to the new FITS filename. The number of modes are checked at the beginning of fsm_set_gain, so this will cause the first fsm_set_gain to fail, which should cause fsm_resume_ao to set the gains back to zero and the AdSecArbitrator to update the state to Failure.
- This does not test the case that the second fsm_set_gain fails. Open to suggestions if we feel it's necessary to test this as well.