17:23 Powering up after adsec software update:

[AOeng@dxadsec ~]$ use_soul to-be-master && use_soul
Switched live version to /home/aoeng/soul/to-be-master.
Usage: use_soul [version]
Currently running: to-be-master (soul)
Available versions in /home/aoeng/soul:
dev-mmc_with_freeze
mixed_mode_control.20221028
mixed_mode_control.20230124
mixed_mode_control.20230201
stable
stable.20211124
stable.20230305
stable.old
test.20220829.restart
to-be-master

Flat set with 623 modes

17:27 Updating WFS software and executing a w_start_all. Failed first time to bring up processes cleanly so stop_all/start_all

[dxwunit@lbti-dxwfs source]$ use_soul to-be-master && use_soul
Switched live version to /home/dxwunit/soul/to-be-master.
Usage: use_soul [version]
Currently running: to-be-master (soul)
Available versions in /home/dxwunit/soul:
failed.20230304
stable
stable.20211124
stable.20221117
test.20230310.ncpa-threshold
to-be-master

17:40 Hexapod moved to position: -5.85, -3, 1.4, 53, -888

17:47 Closing the loop with 1700Hz s1.5 distrubance
  • Center Pupils put OCAM not live 17:48
  • Closed with zero gains and OCAM went not live. WFS Frame Counter not incrementing.
  • Cycled the BCU2K, rebinned. Close loop zero gain and OCAM remained live and WFS frame counter is incrementing. ok
17:58 Closing loop with 1700Hz, s1.5 disturbance
  • Optimize gain 2.3 0.39 0.51, frequency 1470Hz and bin1
  • Fake pie shape seen seen between modes and commands
  • GOpt is 0.45
  • Pie chape in commands and modes persists.
18:04 execute the following on obs-32bit on the mountain: ssh LBTO@obs-32bit.mountain.lbto.org (PW will be supplied) % idl IDL> iif_register, RCBG IDL> irc_offsetxyao, 0.1,0.1, /DEBUG
  • Start positions for bayside X Y -38.003, -48.997 moved to -38.1, -48.1, hexapod thin shell offload RX -1.551, RY 0.187 to RX-2.401, RY -0.662
18:09 IDL> irc_offsetxyao, -0.1,-0.1, /DEBUG
  • Start positions for bayside X Y -38.1, -48.1 moved to -38.0, -48.0, hexapod thin shell offload RX-2.401, RY -0.662 to RX -1.679 RY 0.238
18:14 IDL> irc_offsetxyao, -0.3,-0.3, /DEBUG
  • Start positions for bayside X Y -38.0, -48.0 moved to -37.7, -47.7, hexapod thin shell offload RX -1.679 RY 0.238 to RX 0.403 RY 2.9
18:16 IDL> irc_offsetxyao, 0.3,0.3, /DEBUG
  • Start positions for bayside X Y -37.7, -47.7 moved to -37.998, -48.016, hexapod thin shell offload RX 0.403 RY 2.9 to RX -1.916 RY 0.211
There were 209 fake pie shape events 18:06-18:23. We are switching to a 900Hz ACE table to monitor frequency

Roughly a 100 mode starting 17:58

18:25 Loading

ln -sib table_LBTI_ACE-AO_20221014_500m_900Hz.txt table_LBTI_ACE-AO.txt

and closing loop with 900Hz s1.5 disturbance
  • 4.26, 0.74, 0.56
  • Applying GOpt ~0.45
Guido is tailing to monitor for Fake Pie shape

18:31 Paused through AOS
  • Thin shell offload z -0.046, starting Bayside -5.205
  • irc_offsetzao, 1.0, /DEBUG Bayside at -4.194
  • irc_offsetzao, -1.0, /DEBUG Bayside -5.209
  • IDL> irc_offsetzao,-1.5,/DEBUG
    => IRCR OffsetZAO -1.50000 right ; echo $?


    ** irc_offsetzao **
    request = IRCR OffsetZAO -1.50000 right ; echo $?
    result = ---------------------------------------------- result = Exit 1
    result = 1
    err = Resmsg[0] = OffsetZAO result status: Error
    err = Resmsg[1] = OffsetZ{FLAO}[71] - Illegal command for state LoopSuspended

    ===> Error in irc_offsetzao
    err = Resmsg[0] = OffsetZAO result status: Error
    err = Resmsg[1] = OffsetZ{FLAO}[71] - Illegal command for state LoopSuspended

  • IDL> irc_offsetzao,1.5,/DEBUG
    => IRCR OffsetZAO 1.50000 right ; echo $?


    ** irc_offsetzao **
    request = IRCR OffsetZAO 1.50000 right ; echo $?
    result = ---------------------------------------------- result = Exit 1
    result = 1
    err = Resmsg[0] = OffsetZAO result status: Error
    err = Resmsg[1] = OffsetZ{FLAO}[72] - Illegal command for state LoopSuspended

    ===> Error in irc_offsetzao
    err = Resmsg[0] = OffsetZAO result status: Error
    err = Resmsg[1] = OffsetZ{FLAO}[72] - Illegal command for state LoopSuspended

18:35 Z stage tests with 900Hz. The failures yesterday were likely from fake pie.
  • irc_offsetzao, 1.0, /DEBUG, Bayside moved from -5.2 to -4.2. Thin shell offload in z changed from -0.046 to -0.044
  • Getting about 1 safeskip per second.
  • IDL> irc_offsetzao, -1.0, /DEBUG
    • irc_offsetzao, 1.0, /DEBUG, Bayside moved from -4.2 to -5.2. Thin shell offload in z -0.044 (no change)
  • IDL> irc_offsetzao,15.0,/DEBUG
    => IRCR OffsetZAO 15.0000 right ; echo $?


    ** irc_offsetzao **
    request = IRCR OffsetZAO 15.0000 right ; echo $?
    result = ---------------------------------------------- result = Exit 1
    result = 1
    err = Resmsg[0] = OffsetZAO result status: Error
    err = Resmsg[1] = OffsetZ{FLAO}[76] - [Exception] 'NoneType' object is not callable

    ===> Error in irc_offsetzao
    err = Resmsg[0] = OffsetZAO result status: Error
    err = Resmsg[1] = OffsetZ{FLAO}[76] - [Exception] 'NoneType' object is not callable
  • IDL> irc_offsetzao,-15.0,/DEBUG
    => IRCR OffsetZAO -15.0000 right ; echo $?


    ** irc_offsetzao **
    request = IRCR OffsetZAO -15.0000 right ; echo $?
    result = ---------------------------------------------- result = Exit 1
    result = 1
    err = Resmsg[0] = OffsetZAO result status: Error
    err = Resmsg[1] = OffsetZ{FLAO}[77] - [Exception] 'NoneType' object is not callable

    ===> Error in irc_offsetzao
    err = Resmsg[0] = OffsetZAO result status: Error
    err = Resmsg[1] = OffsetZ{FLAO}[77] - [Exception] 'NoneType' object is not callable

  • IDL> irc_offsetzao,-10.0,/DEBUG
    => IRCR OffsetZAO -10.0000 right ; echo $?


    ** irc_offsetzao **
    request = IRCR OffsetZAO -10.0000 right ; echo $?
    result = Resmsg[0] = OffsetZAO result status: OK
    result = Resmsg[1] = OffsetZ{FLAO}[78]
    result = ---------------------------------------------- result = 0
    err =

    => irc_offsetzao: Command Successful

    • Thin shell offload moved in z from -0.044 to -0.099
    • Task took 3-4 min to complete
    • No fake pie shapes during process
    • bayside moved from -5.2 to -15.2
  • IDL> irc_offsetzao,10.0,/DEBUG
    => IRCR OffsetZAO 10.0000 right ; echo $?


    ** irc_offsetzao **
    request = IRCR OffsetZAO 10.0000 right ; echo $?
    result = ---------------------------------------------- result = Exit 1
    result = 1
    err = Resmsg[0] = OffsetZAO result status: Error
    err = Resmsg[1] = OffsetZ{FLAO}[79] - Illegal command for state LoopClosed

    ===> Error in irc_offsetzao
    err = Resmsg[0] = OffsetZAO result status: Error
    err = Resmsg[1] = OffsetZ{FLAO}[79] - Illegal command for state LoopClosed

    • Failed after 8 steps (last 2 offset was outside threshold limits)
    • Thin shell offload -0.099 to -0.057, baysides went from -15.2 to -7.2
  • IDL> irc_offsetzao,2.0,/DEBUG
    => IRCR OffsetZAO 2.00000 right ; echo $?


    ** irc_offsetzao **
    request = IRCR OffsetZAO 2.00000 right ; echo $?
    result = Resmsg[0] = OffsetZAO result status: OK
    result = Resmsg[1] = OffsetZ{FLAO}[80]
    result = ---------------------------------------------- result = 0
    err =

    => irc_offsetzao: Command Successful

    • Return from -7.2 to -5.2 (last 2 steps of previous test). No issue.
These test show the software functionality is ok, but the thresholds may need to be tuned. SHARK will need 3mm of offset so testing just their requirements:
  • IDL> irc_offsetzao,3.0,/DEBUG
=> IRCR OffsetZAO 3.00000 right ; echo $?


** irc_offsetzao **
request = IRCR OffsetZAO 3.00000 right ; echo $?
result = Resmsg[0] = OffsetZAO result status: OK
result = Resmsg[1] = OffsetZ{FLAO}[81]
result = ---------------------------------------------- result = 0
err =
=> irc_offsetzao: Command Successful
  • IDL> irc_offsetzao,-3.0,/DEBUG
    => IRCR OffsetZAO -3.00000 right ; echo $?


  • ** irc_offsetzao **
    request = IRCR OffsetZAO -3.00000 right ; echo $?
    result = Resmsg[0] = OffsetZAO result status: OK
    result = Resmsg[1] = OffsetZ{FLAO}[82]
    result = ---------------------------------------------- result = 0
    err =

    => irc_offsetzao: Command Successful

The failure issues may have to do with the hexapod taking too long to reach position but the movements of the size expected by SHRAK NIR were ok everytime.

At 900Hz we did not get the Fake PIE shape. But we have ruled out as a cause for the failed offsetzao.

Reverting DX LBTI WFS to stable:

[dxwunit@lbti-dxwfs source]$ use_soul stable && use_soul
Switched live version to /home/dxwunit/soul/stable.
[dxwunit@lbti-dxwfs source]$ use_soul
Usage: use_soul [version]
Currently running: stable (soul)
Available versions in /home/dxwunit/soul:
failed.20230304
stable
stable.20211124
stable.20221117
test.20230310.ncpa-threshold
to-be-master
to-be-master-and-fwFix

19:11 Testing software on SOUL-LUCI DX

[AOeng@soul-dxwfs source]$ use_soul to-be-master && use_soul
Switched live version to /home/AOeng/soul/to-be-master.
Usage: use_soul [version]
Currently running: to-be-master (soul)
Available versions in /home/AOeng/soul:
stable
stable.20211125
stable.20221117
stable.20230201.tocheck
stable.20230202
test.badocam
to-be-master
to-be-master-and-fwFix

Powering on the DX WFS. Baysides failed to power. Cycling main 24V power and homing.

19:19 Changing software on SX adsec and WFS for test

[AOeng@sxadsec source]$ use_soul to-be-master && use_soul
Switched live version to /home/aoeng/soul/to-be-master.
Usage: use_soul [version]
Currently running: to-be-master (soul)
Available versions in /home/aoeng/soul:
stable
stable.20211011.pause-resume
stable.20211012
stable.20211124
stable.20221117
stable.20230201
stable.20230305
test.20220419.hk-partial-timeout
test.20220518.shark-setncpa
test.20220602.setz0-bank-a
test.20220929.hk-log-telemetry-on-sbcu-timeout
test.20221031.hk-timeout
test.20230302.offsetZ
to-be-master

Flat set with 639 modes

19:36 Verified on DX LUCI WFS after the recompile yesterday.
  • Set focal station. Closed loop with zero gains. The frame counter incremented without issue.
19:43 Verified on SX LUCI WFS after the recompile yesterday.
  • Set focal station. Closed loop with zero gains. The frame counter incremented without issue.
Reverting Software
  1. SX LUCI WFS:

    AOeng@soul-sxwfs source]$ use_soul stable && use_soul
    Switched live version to /home/AOeng/soul/stable.
    [AOeng@soul-sxwfs source]$ use_soul
    Usage: use_soul [version]
    Currently running: stable (soul)
    Available versions in /home/AOeng/soul:
    stable
    stable.20221117
    stable.20230201
    stable.20230202
    test.20230302.offsetZ
    test.20230310.ncpa-threshold
    test.20230322.ncpa-threshold-and-fwFix
    test.20230322.rerotator-and-fwFix
    to-be-master
    to-be-master-and-fwFix

    w_start_all/ w_idle
  2. DX LUCI WFS

    AOeng@soul-dxwfs source]$ use_soul stable && use_soul
    Switched live version to /home/AOeng/soul/stable.
    [AOeng@soul-dxwfs source]$ use_soul
    Usage: use_soul [version]
    Currently running: stable (soul)
    Available versions in /home/AOeng/soul:
    stable
    stable.20211125
    stable.20221117
    stable.20230201.tocheck
    stable.20230202
    test.badocam
    to-be-master
    to-be-master-and-fwFix

    w_start_all and w_idle after adsec software change

  3. DX Adsec:
    Rest shell, power off, adsc_stop

    [AOeng@dxadsec ~]$ use_soul stable && use_soul
    'Switched live version to /home/aoeng/soul/stable.
    [AOeng@dxadsec ~]$ use_soul
    Usage: use_soul [version]
    Currently running: stable (soul)
    Available versions in /home/aoeng/soul:
    dev-mmc_with_freeze
    mixed_mode_control.20221028
    mixed_mode_control.20230124
    mixed_mode_control.20230201
    stable
    stable.20211124
    stable.20230305
    stable.old
    test.20220829.restart
    to-be-master

    Flat set with 632 modes

  4. SX Adsec

    AOeng@sxadsec source]$ use_soul stable && use_soul
    Switched live version to /home/aoeng/soul/stable.
    [AOeng@sxadsec source]$ use_soul
    Usage: use_soul [version]
    Currently running: stable (soul)
    Available versions in /home/aoeng/soul:
    stable
    stable.20211011.pause-resume
    stable.20211012
    stable.20211124
    stable.20221117
    stable.20230201
    stable.20230305
    test.20220419.hk-partial-timeout
    test.20220518.shark-setncpa
    test.20220602.setz0-bank-a
    test.20220929.hk-log-telemetry-on-sbcu-timeout
    test.20221031.hk-timeout
    test.20230302.offsetZ
    to-be-master

    Housekeeper disconnected during power on. Manually restarted in time and connected. Still getting errors in housekeeper log so restarting from scratch

  5. 20:04 adsc_start for SX again. No problems with power up sequence this time. Flat set with 639 modes

-- JenniferPower - 31 Mar 2023
Topic revision: r2 - 31 Mar 2023, JenniferPower
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback