2022-05-07/08 - (adsec + soul sx wfs + lbti sx wfs) release checkout: upgrade on LBTI and SOUL-LUCI DEC nights

  • Software: Brandon: I'll have notification volume up for email/Slack, available at 480-326-6838.
  • AO: Amali (LBTI), Barry and Juan Carlos (LUCI-AO)
  • Software versions used:
    • sxadsec: latest
    • lbti-sxwfs: latest
    • soul-sxwfs: latest
  • Software version after test:
    • sxadsec: latest
    • soul-sxwfs: latest
    • lbti-sxwfs: latest
Note: LBTI will do a test Saturday night, and LUCI-AO will do a test Sunday night. If everything works, we can keep software in "latest" and I will upgrade "stable" Monday morning.


  • adsec: AdsecArbitrator: zero-pad LN reconstructor matrix at the beginning of columns
    • updates the loadRecMat Ice command to zero-pad the reconstructor matrix if it is smaller than the expected SOUL-sized matrix.
    • This only affects LINC-NIRVANA.
  • adsec: FLAOGUI and AOArbitrator: add "Force open loop button.
    • adds a "Force open loop" button to the FLAOGUI which will trigger a Skip Frame Alert to be sent to the AO Arbitrator, which will cause AO Arbitrator to go into a LoopFault state from closed loop.
  • adsec: AdSecArbitrator and MasterDiagnostic: log skip frame percent, which ICE client can't receive an alert during dispatch errors, and wrong filotto error frame bytes
    • minor changes to logging for debugging.
  • adsec: aoarbitrator: implement "interruptible" flag to AO arbitrator commands to allow them to return successfully if they ignore interruptions from FLAOGUI or SkipFrame Alerts.
  • wfs: WfsArbitrator, CopleyCtrl: return error immediately on out of range bayside movements instead of waiting for a timeout to expire
    • previously, if an out-of-range bayside movement was received, the WfsArbitrator would wait up to a minute for it to timeout instead of returning an error immediately.
  • wfs: WfsArbitrator: wait for RelayCtrl status to become "online" before attempting to power on BCUs.
    • previously, if the RelayCtrl process (powerboard) wasn't ready in an "online" state before attempting to power on the BCUs, the BCU turnon command would time out.
    • BM 5/27: This has been rolled back until we can figure out why we're having problems with the Ocam starting up.
  • wfs: WfsArbitrator: optimize gain bugfix: max_iteration parameter was ignored
    • allows max_iterations to be configured in autogain.conf.
  • wfs: WfsArbitrator: use bestEffort=true when centering pupils on LBTI
    • was previously done by manually editing the WfsArbScripts.py on lbti-sxwfs.
  • wfs: WfsArbitrator: drawlib: free invalid draw objects
    • free invalid draw objects in optimize gain to prevent some possible memory leaks.
  • scripts: add move_log_files.sh
    • This is the script that is currently used by cron jobs on sxadsec, dxadsec, and soul-sxwfs to move log files to mt-synha (/ao-data on the adsec machines).


We've tested all the changes above during the day and during DEC nights. Here's a checklist of extra tests you can do to verify the release:
  1. Close the loop.
  2. Pause the loop and make an offset that will result in out-of-range bayside movements.
    1. Make sure the offset fails immediately, allowing you to immediately do another preset instead of needing to wait for three attempts to time out.
    2. tail -f $ADOPT_LOG/`date +%Y/%m/%d`/{wfsarb,bayside}*
      1. You should not see: "[AOException] RETRY: Stages position not reached (-5001) TIMEOUT_ERROR"
      2. You should see:
        CopleyCtrl_XXXXXX |ERR| [...] MAIN > Error: movement out of range
        WfsArbitrator_XXXXXX |INF| [...] COMMANDHANDLER > Command execution failed: Fsm forced to Failure
        WfsArbitrator_XXXXXX |ERR| [...] MAIN > presetAO: FATAL - Error waiting for target to be reached. Message from controller: Error in MoveTo(): movement out of range (-5002) VALUE_OUT_OF_RANGE_ERROR
        WfsArbitrator_XXXXXX |ERR| [...] MAIN > Error waiting for target to be reached. Message from controller: Error in MoveTo(): movement out of range (-5002) VALUE_OUT_OF_RANGE_ERROR


  • UT 2022-05-08 LBTI test checklist
    • LBTI tests of bayside out of range errors successful. No timeouts from out of range movements. Received new "movement out of range" error.
    • No other new errors reported.
  • UT 2022-05-09 LUCI night log (See "Bayside Stage Test" ca. UT 09:16.)
    • LUCI tests successful. No timeouts from out of range movements. Received new "movement of out of range" error.
    • 40s timeout reported during an offset that was due to failure to reach the target, rather than a movement error. A new IT (8635) was created to report the issue.

-- BrandonMechtley - 02 May 2022
Topic revision: r11 - 28 May 2022, BrandonMechtley
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback