2022-04-25 - make AO arbitrator commands, including OffsetXY, "uninterruptible"

Software: BM
AO: AV, JCG

Software testing queue

Servers and software versions used:
sxadsec: stable, running test aoarbitrator.py from ~/soul/test.20220425.offsetxy-uninterruptible
lbti-sxwfs: stable

Problem (IT 8556): During offsets, if there are safe skips, the SkipFrame Alert from the AdSecArbitrator can cause the AO Arbitrator to set an "_interrupt" flag and interrupt the OffsetXY command, which will cause the AO Arbitrator to respond to AOS that the OffsetXY command has failed after execution. However, the interrupt flag doesn't actually stop the movement of the bayside stages or in any other way interrupt OffsetXY. When OffsetXY returns an error, AOS won't update the focal plane position of the star, which will cause all the subsequent offsets to be incorrect, because the positions of the baysides are not synced between AOS and the AO/WFS arbitrators.

2021-2022 interrupted offset logs shows a log of interrupted OffsetXY commands resulting in AOS errors from 2021-2022 for both LUCI and LBTI.

New code: See the pull request for a more detailed discussion of the problem above and a diff for the code changes. This code introduces an "_interruptible" flag to the AO Arbitrator, which defaults to False and will be set to True by commands that actually observe the "_interrupt" flag during their execution (see the pull request for a full list.) In the current version of the AO Arbitrator, after a command has executed, if the "_interrupt" flag is set, it will report an error that the command has been interrupted and return an error to AOS. This will modify the logic so that an error is returned only if both the "_interrupt" and "_interruptible" flags are set.

Test: These tests will verify that commands that should be interruptible can be interrupted and that OffsetXY is not interruptible:
  1. Tail the logs for AO Arbitrator command requests and results:
    tail -f $ADOPT_LOG/`date +%Y/%m/%d`/AOARB* | egrep 'Request|interrupted|successful|after'
  2. We can run the commands that are interruptible, i.e. AcquireRefAO (both in intervention and auto mode), CenterPupils, and CloseLoop, and make sure that they can still be interrupted by the "Interrupt" button in FLAOGUI.
Daytime Results (SW checklist log, AO log):
  • Everything worked as intended. We manually sent AOARB::Interrupt commands with a Python script while executing OffsetXY (open loop), OffsetXY (closed loop), CenterPupils, CloseLoop, and OptimizeGain.
  • The only command that "interrupted" and resulted in an exception being raised was OffsetXY during closed loop. Other commands logged a "Interruption of uninterruptible command [...] was ignored" message.
  • We also observed a SkipFrameAlert during an OptimizeGain, which did not "interrupt" the command, but did raise an exception (IDL_SAFE_SKIP_ACTIVE, a bit of a misnomer, produced by autogain.py) due to skip frames being present, which correctly got sent to AOS.
DEC on sky results: (Log)
  • OffsetXY correctly reports that it's uninterruptible.
  • AOS correctly does not report an error.
  • Out-of-FoV-limits OffsetXY still correctly returns an error.
Recommended future testing: -- BrandonMechtley - 25 Apr 2022
Topic revision: r4 - 29 Apr 2022, BrandonMechtley
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback