2022-04-25 - optical loop diagnostic data during pause, uninterruptible offsets, bayside out of range movement timeouts, verify idlsystem thread safety bug

Software: BM

Software testing queue

Servers and software versions used:
sxadsec: stable, with custom aoarbitrator.py run from ~/soul/test.20220425.offsetxy-uninterruptible
lbti-sxwfs: test.20220425.interrupted-offsets-and-bayside-timeouts

Test 1: Collect optical loop data during open loop

Test 2: Test change that makes the bayside stages immediately return an error on out-of-range movements instead of timing out

Test 3: Test change that makes certain AO arbitrator commands "uninterruptible"

Problem (IT 8556): During offsets, if there are safe skips, the SkipFrame Alert from the AdSecArbitrator can cause the AO Arbitrator to set an "_interrupt" flag and interrupt the OffsetXY command, which will cause the AO Arbitrator to respond to AOS that the OffsetXY command has failed after execution. However, the interrupt flag doesn't actually stop the movement of the bayside stages or in any other way interrupt OffsetXY. When OffsetXY returns an error, AOS won't update the focal plane position of the star, which will cause all the subsequent offsets to fail, because the positions of the baysides are not synced between AOS and the AO/WFS arbitrators.

New code: See the pull request for a more detailed discussion of the problem above and a diff for the code changes. This code introduces an "_interruptible" flag to the AO Arbitrator, which defaults to False and will be set to True by commands that actually observe the "_interrupt" flag during their execution (see the pull request for a full list.) In the current version of the AO Arbitrator, after a command has executed, if the "_interrupt" flag is set, it will report an error that the command has been interrupted and return an error to AOS. This will modify the logic so that an error is returned only if both the "_interrupt" and "_interruptible" flags are set.

Test: These tests will verify that commands that should be interruptible can be interrupted and that OffsetXY is not interruptible:
  1. Tail the logs for AO Arbitrator command requests and results:
    tail -f $ADOPT_LOG/`date +%Y/%m/%d`/AOARB* | egrep 'Request|interrupted|successful|after'
  2. Daytime: We can run the commands that are interruptible, i.e. AcquireRefAO (both in intervention and auto mode), CenterPupils, and CloseLoop, and make sure that they can still be interrupted by the "Interrupt" button in FLAOGUI.
  3. DEC time: Same as above, but we should also try to interrupt an offset during closed loop and an offset during open loop. The offset should only interrupt when the loop is closed. We can also try to produce conditions that will cause a SafeSkip Alert during OffsetXY in open loop and see that the offset completes normally doesn't report that it's been interrupted.

Test 4: Verify existence of IdlSystem race condition

-- BrandonMechtley - 25 Apr 2022
Topic revision: r1 - 25 Apr 2022, BrandonMechtley
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback