2022-04-25 - make AO arbitrator commands, including OffsetXY, "uninterruptible"
Software testing queue
Servers and software versions used:
sxadsec: stable, running test aoarbitrator.py from ~/soul/test.20220425.offsetxy-uninterruptible
Problem (IT 8556):
During offsets, if there are safe skips, the SkipFrame
Alert from the AdSecArbitrator
can cause the AO Arbitrator to set an "_interrupt" flag and interrupt the OffsetXY
command, which will cause the AO Arbitrator to respond to AOS that the OffsetXY
command has failed after execution. However, the interrupt flag doesn't actually stop the movement of the bayside stages or in any other way interrupt OffsetXY
. When OffsetXY
returns an error, AOS won't update the focal plane position of the star, which will cause all the subsequent offsets to be incorrect, because the positions of the baysides are not synced between AOS and the AO/WFS arbitrators.
2021-2022 interrupted offset logs
shows a log of interrupted OffsetXY
commands resulting in AOS errors from 2021-2022 for both LUCI and LBTI.
See the pull request
for a more detailed discussion of the problem above and a diff for the code changes. This code introduces an "_interruptible" flag to the AO Arbitrator, which defaults to False and will be set to True by commands that actually observe the "_interrupt" flag during their execution (see the pull request for a full list.) In the current version of the AO Arbitrator, after a command has executed, if the "_interrupt" flag is set, it will report an error that the command has been interrupted and return an error to AOS. This will modify the logic so that an error is returned only if both the "_interrupt" and "_interruptible" flags are set.
These tests will verify that commands that should be interruptible can be interrupted and that OffsetXY
is not interruptible:
Daytime Results (SW checklist log, AO log):
- Tail the logs for AO Arbitrator command requests and results:
tail -f $ADOPT_LOG/`date +%Y/%m/%d`/AOARB* | egrep 'Request|interrupted|successful|after'
- We can run the commands that are interruptible, i.e. AcquireRefAO (both in intervention and auto mode), CenterPupils, and CloseLoop, and make sure that they can still be interrupted by the "Interrupt" button in FLAOGUI.
DEC on sky results: (Log)
- Everything worked as intended. We manually sent AOARB::Interrupt commands with a Python script while executing OffsetXY (open loop), OffsetXY (closed loop), CenterPupils, CloseLoop, and OptimizeGain.
- The only command that "interrupted" and resulted in an exception being raised was OffsetXY during closed loop. Other commands logged a "Interruption of uninterruptible command [...] was ignored" message.
- We also observed a SkipFrameAlert during an OptimizeGain, which did not "interrupt" the command, but did raise an exception (IDL_SAFE_SKIP_ACTIVE, a bit of a misnomer, produced by autogain.py) due to skip frames being present, which correctly got sent to AOS.
Recommended future testing:
- OffsetXY correctly reports that it's uninterruptible.
- AOS correctly does not report an error.
- Out-of-FoV-limits OffsetXY still correctly returns an error.
- 25 Apr 2022