Details.

[AOeng@soul-sxwfs source]$ use_soul
Usage: use_soul [version]
Currently running: stable (soul)
Available versions in /home/AOeng/soul:
latest
stable
stable.20211125
test.20220304.aoice-default-ports
test.20220419.agw-power-wait-for-operating-rebase-onto-stable

It will be deployed the latest version to run tonight.

AOeng@soul-sxwfs source]$ use_soul latest
Switched live version to /home/AOeng/soul/latest.
[AOeng@soul-sxwfs source]$

Adsec is running th latest version used last night in LBTI.

[AOeng@sxadsec source]$ use_soul
Usage: use_soul [version]
Currently running: latest (soul)
Available versions in /home/aoeng/soul:
latest
stable
stable.20211011.pause-resume
stable.20211125
test.20220222.reorder-gain-ramp
test.20220304.aoice-default-ports
test.20220329.hk-partial-timeout
test.20220419.hk-partial-timeout
[AOeng@sxadsec source]$
[AOeng@sxadsec source]$

The swithc to soul latest come up with a wfsarbitrator down.

w_stop/start_all

all processes up.

2:54PM Operate.

7.935418| MAIN > Loaded devices: ['stagex', 'stagey', 'stagez', 'luci']
WfsArbitrator_190005|DEB| 10|2022-05-08 21:53:50.334554| MAIN > arbFunc TurnSystemOn (, 'FLAOWFS', '', 'WFS_complete_(with_ccd47)', '', False)
WfsArbitrator_190005|INF| 11|2022-05-08 21:53:50.334624| MAIN > Setting configuration WFS_complete_(with_ccd47)
WfsArbitrator_190005|INF| 12|2022-05-08 21:53:50.334654| MAIN > Turning WFS on
WfsArbitrator_190005|DEB| 13|2022-05-08 21:53:50.334691| MAIN > Devices: ['ts8sx', 'ts8dx', 'bcu2k', 'bcu47', 'fw', 'tt', 'stages', 'camera', 'cameralens', 'ccd47', 'rerot', 'adc', 'cubestage']
WfsArbitrator_190005|DEB| 14|2022-05-08 21:53:50.334960| MAIN > Initializing Python modules...
WfsArbitrator_190005|INF| 15|2022-05-08 21:53:50.335004| MAIN > ['ts8sx', 'ts8dx', 'bcu2k', 'bcu47', 'fw', 'tt', 'stages', 'camera', 'cameralens', 'ccd47', 'rerot', 'adc', 'cubestage']
WfsArbitrator_190005|INF| 16|2022-05-08 21:53:50.335077| MAIN > Loading power
WfsArbitrator_190005|INF| 17|2022-05-08 21:53:50.340631| MAIN > Loading fw1
WfsArbitrator_190005|INF| 18|2022-05-08 21:53:50.342002| MAIN > The filterwheel1 controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_190005|INF| 19|2022-05-08 21:53:50.342059| MAIN > filterwheel1 interface loaded
WfsArbitrator_190005|INF| 20|2022-05-08 21:53:50.342096| MAIN > Loading fw2
WfsArbitrator_190005|INF| 21|2022-05-08 21:53:50.343140| MAIN > The filterwheel2 controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_190005|INF| 22|2022-05-08 21:53:50.343189| MAIN > filterwheel2 interface loaded
WfsArbitrator_190005|INF| 23|2022-05-08 21:53:50.343226| MAIN > Loading adc
WfsArbitrator_190005|INF| 24|2022-05-08 21:53:50.344382| MAIN > The adc1 controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_190005|INF| 25|2022-05-08 21:53:50.344429| MAIN > adc1 interface loaded
WfsArbitrator_190005|INF| 26|2022-05-08 21:53:50.345421| MAIN > The adc2 controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_190005|INF| 27|2022-05-08 21:53:50.345463| MAIN > adc2 interface loaded
WfsArbitrator_190005|INF| 28|2022-05-08 21:53:50.345892| MAIN > ADC interface loaded
WfsArbitrator_190005|INF| 29|2022-05-08 21:53:50.345957| MAIN > Loading camera
WfsArbitrator_190005|INF| 30|2022-05-08 21:53:50.347464| MAIN > Loading sc
WfsArbitrator_190005|INF| 31|2022-05-08 21:53:50.349824| MAIN > Loading ccd47
WfsArbitrator_190005|INF| 32|2022-05-08 21:53:50.351522| MAIN > Loading tv
WfsArbitrator_190005|INF| 33|2022-05-08 21:53:50.352498| MAIN > Loading rerot
WfsArbitrator_190005|INF| 34|2022-05-08 21:53:50.353492| MAIN > The rerotator controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_190005|INF| 35|2022-05-08 21:53:50.353545| MAIN > rerotator interface loaded
WfsArbitrator_190005|INF| 36|2022-05-08 21:53:50.353586| MAIN > Loading cuberot
WfsArbitrator_190005|INF| 37|2022-05-08 21:53:50.354944| MAIN > The mercury controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_190005|INF| 38|2022-05-08 21:53:50.354992| MAIN > mercury interface loaded
WfsArbitrator_190005|INF| 39|2022-05-08 21:53:50.355030| MAIN > Loading cubestage
WfsArbitrator_190005|INF| 40|2022-05-08 21:53:50.356139| MAIN > The cubestage controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_190005|INF| 41|2022-05-08 21:53:50.356189| MAIN > cubestage interface loaded
WfsArbitrator_190005|INF| 42|2022-05-08 21:53:50.356228| MAIN > Loading tt

TTCtrl_189667 |INF| 16|2022-05-08 21:53:07.276305| MAIN > ------------------------------- LOW-LEV APPLY ----------------------------------
TTCtrl_189667 |INF| 17|2022-05-08 21:53:07.276327| MAIN > Default phases (0-120-240) in TT Mirror units: 0 21845 43691
TTCtrl_189667 |INF| 18|2022-05-08 21:53:07.276338| MAIN > Freq: 55 - Amp: 0 - Offset: 0 - Phase: 0
TTCtrl_189667 |INF| 19|2022-05-08 21:53:07.276348| MAIN > Freq: 55 - Amp: 0 - Offset: 0 - Phase: 21845
TTCtrl_189667 |INF| 20|2022-05-08 21:53:07.276359| MAIN > Freq: 55 - Amp: 0 - Offset: 0 - Phase: 43690
TTCtrl_189667 |INF| 21|2022-05-08 21:53:07.946822| MAIN > BCU47 unreachable! TT fate unknown
TTCtrl_189667 |WAR| 22|2022-05-08 21:53:10.276782| MAIN > [WavesException] Applying waves failed
TTCtrl_189667 |ERR| 23|2022-05-08 21:53:10.276815| MAIN > TipTilt start/restart failed

Stop/start wfsarb

itrator_190005|INF| 33|2022-05-08 21:53:50.352498| MAIN > Loading rerot
WfsArbitrator_190005|INF| 34|2022-05-08 21:53:50.353492| MAIN > The rerotator controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_190005|INF| 35|2022-05-08 21:53:50.353545| MAIN > rerotator interface loaded
WfsArbitrator_190005|INF| 36|2022-05-08 21:53:50.353586| MAIN > Loading cuberot
WfsArbitrator_190005|INF| 37|2022-05-08 21:53:50.354944| MAIN > The mercury controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_190005|INF| 38|2022-05-08 21:53:50.354992| MAIN > mercury interface loaded
WfsArbitrator_190005|INF| 39|2022-05-08 21:53:50.355030| MAIN > Loading cubestage
WfsArbitrator_190005|INF| 40|2022-05-08 21:53:50.356139| MAIN > The cubestage controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_190005|INF| 41|2022-05-08 21:53:50.356189| MAIN > cubestage interface loaded
WfsArbitrator_190005|INF| 42|2022-05-08 21:53:50.356228| MAIN > Loading tt

Process stuck at this point.


WfsArbitrator_13301|DEB| 1|2022-05-08 22:04:29.037033| MAIN > Connected as pyarg
WfsArbitrator_13301|INF| 2|2022-05-08 22:04:29.037211| MAIN > Loading stagex
WfsArbitrator_13301|INF| 3|2022-05-08 22:04:29.038494| MAIN > baysidex interface loaded
WfsArbitrator_13301|INF| 4|2022-05-08 22:04:29.038901| MAIN > Loading stagey
WfsArbitrator_13301|INF| 5|2022-05-08 22:04:29.040078| MAIN > baysidey interface loaded
WfsArbitrator_13301|INF| 6|2022-05-08 22:04:29.040483| MAIN > Loading stagez
WfsArbitrator_13301|INF| 7|2022-05-08 22:04:29.041587| MAIN > baysidez interface loaded
WfsArbitrator_13301|INF| 8|2022-05-08 22:04:29.041979| MAIN > Loading luci
WfsArbitrator_13301|INF| 9|2022-05-08 22:04:29.042298| MAIN > Loaded devices: ['stagex', 'stagey', 'stagez', 'luci']
WfsArbitrator_13301|DEB| 10|2022-05-08 22:04:44.301293| MAIN > arbFunc TurnSystemOn (, 'FLAOWFS', '', 'WFS_complete_(with_ccd47)', '', False)
WfsArbitrator_13301|INF| 11|2022-05-08 22:04:44.301372| MAIN > Setting configuration WFS_complete_(with_ccd47)
WfsArbitrator_13301|INF| 12|2022-05-08 22:04:44.301409| MAIN > Turning WFS on
WfsArbitrator_13301|DEB| 13|2022-05-08 22:04:44.301453| MAIN > Devices: ['ts8sx', 'ts8dx', 'bcu2k', 'bcu47', 'fw', 'tt', 'stages', 'camera', 'cameralens', 'ccd47', 'rerot', 'adc', 'cubestage']
WfsArbitrator_13301|DEB| 14|2022-05-08 22:04:44.301753| MAIN > Initializing Python modules...
WfsArbitrator_13301|INF| 15|2022-05-08 22:04:44.301806| MAIN > ['ts8sx', 'ts8dx', 'bcu2k', 'bcu47', 'fw', 'tt', 'stages', 'camera', 'cameralens', 'ccd47', 'rerot', 'adc', 'cubestage']
WfsArbitrator_13301|INF| 16|2022-05-08 22:04:44.301876| MAIN > Loading power
WfsArbitrator_13301|INF| 17|2022-05-08 22:04:44.308931| MAIN > Loading fw1
WfsArbitrator_13301|INF| 18|2022-05-08 22:04:44.311556| MAIN > The filterwheel1 controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_13301|INF| 19|2022-05-08 22:04:44.311618| MAIN > filterwheel1 interface loaded
WfsArbitrator_13301|INF| 20|2022-05-08 22:04:44.311665| MAIN > Loading fw2
WfsArbitrator_13301|INF| 21|2022-05-08 22:04:44.313120| MAIN > The filterwheel2 controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_13301|INF| 22|2022-05-08 22:04:44.313197| MAIN > filterwheel2 interface loaded
WfsArbitrator_13301|INF| 23|2022-05-08 22:04:44.313242| MAIN > Loading adc
WfsArbitrator_13301|INF| 24|2022-05-08 22:04:44.314724| MAIN > The adc1 controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_13301|INF| 25|2022-05-08 22:04:44.314780| MAIN > adc1 interface loaded
WfsArbitrator_13301|INF| 26|2022-05-08 22:04:44.316093| MAIN > The adc2 controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_13301|INF| 27|2022-05-08 22:04:44.316146| MAIN > adc2 interface loaded
WfsArbitrator_13301|INF| 28|2022-05-08 22:04:44.316359| MAIN > ADC interface loaded
WfsArbitrator_13301|INF| 29|2022-05-08 22:04:44.316409| MAIN > Loading camera
WfsArbitrator_13301|INF| 30|2022-05-08 22:04:44.318298| MAIN > Loading sc
WfsArbitrator_13301|INF| 31|2022-05-08 22:04:44.321850| MAIN > Loading ccd47
WfsArbitrator_13301|INF| 32|2022-05-08 22:04:44.324114| MAIN > Loading tv
WfsArbitrator_13301|INF| 33|2022-05-08 22:04:44.325402| MAIN > Loading rerot
WfsArbitrator_13301|INF| 34|2022-05-08 22:04:44.326706| MAIN > The rerotator controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_13301|INF| 35|2022-05-08 22:04:44.326763| MAIN > rerotator interface loaded
WfsArbitrator_13301|INF| 36|2022-05-08 22:04:44.326807| MAIN > Loading cuberot
WfsArbitrator_13301|INF| 37|2022-05-08 22:04:44.328200| MAIN > The mercury controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_13301|INF| 38|2022-05-08 22:04:44.328271| MAIN > mercury interface loaded
WfsArbitrator_13301|INF| 39|2022-05-08 22:04:44.328318| MAIN > Loading cubestage
WfsArbitrator_13301|INF| 40|2022-05-08 22:04:44.329566| MAIN > The cubestage controller process does not provide MOVESTATUS RTDB variables. Movement errors will result in timeouts instead of aborting immediately.
WfsArbitrator_13301|INF| 41|2022-05-08 22:04:44.329620| MAIN > cubestage interface loaded
WfsArbitrator_13301|INF| 42|2022-05-08 22:04:44.329664| MAIN > Loading tt
WfsArbitrator_13301|INF| 43|2022-05-08 22:04:44.344288| MAIN > Loading thorlabs
WfsArbitrator_13301|INF| 44|2022-05-08 22:04:44.344425| MAIN > Loading lamp
WfsArbitrator_13301|INF| 45|2022-05-08 22:04:44.345026| MAIN > Loading lens
WfsArbitrator_13301|INF| 46|2022-05-08 22:04:44.346592| MAIN > Warning: interface with derotator not working. Camera lens will not follow pupil rerotator position
WfsArbitrator_13301|INF| 47|2022-05-08 22:04:44.346655| MAIN > Loading flowerpot
WfsArbitrator_13301|INF| 48|2022-05-08 22:04:44.347303| MAIN > Loading temps
WfsArbitrator_13301|INF| 49|2022-05-08 22:04:44.348568| MAIN > Loading irtc
WfsArbitrator_13301|INF| 50|2022-05-08 22:04:44.349066| MAIN > Exception loading device irtc: BackgDir () takes exactly 3 arguments (2 given)
WfsArbitrator_13301|INF| 51|2022-05-08 22:04:44.349111| MAIN > Loading tvoptics
WfsArbitrator_13301|INF| 52|2022-05-08 22:04:44.349200| MAIN > Loaded devices: ['power', 'fw1', 'fw2', 'adc', 'camera', 'sc', 'ccd47', 'tv', 'rerot', 'cuberot', 'cubestage', 'tt', 'thorlabs', 'lamp', 'lens', 'flowerpot', 'temps', 'tvoptics']
WfsArbitrator_13301|INF| 53|2022-05-08 22:04:44.349271| MAIN > Waiting for network connection
WfsArbitrator_13301|INF| 54|2022-05-08 22:04:44.349321| MAIN > setup unit W SOUL #1
WfsArbitrator_13301|INF| 55|2022-05-08 22:04:44.349380| MAIN > Waiting for network connection (left box)
WfsArbitrator_13301|INF| 56|2022-05-08 22:04:44.349810| MAIN > Turning off RESET_BCU_47
WfsArbitrator_13301|INF| 57|2022-05-08 22:04:45.867760| MAIN > Turning on PROG_BCU_2K
WfsArbitrator_13301|INF| 58|2022-05-08 22:04:48.871053| MAIN > Setting up BCU 2K
WfsArbitrator_13301|INF| 59|2022-05-08 22:04:48.871131| MAIN > Turning on RESET_BCU_2K
WfsArbitrator_13301|INF| 60|2022-05-08 22:04:49.872534| MAIN > Turning on BCU2K _POWER
WfsArbitrator_13301|INF| 61|2022-05-08 22:05:05.238571| MAIN > BCU 2K OK
WfsArbitrator_13301|INF| 62|2022-05-08 22:05:05.239240| MAIN > Turning on INS_TEST_OUT_1
WfsArbitrator_13301|INF| 63|2022-05-08 22:05:06.255795| MAIN > Turning on INS_TEST_OUT_2
WfsArbitrator_13301|INF| 64|2022-05-08 22:05:07.923836| MAIN > Turning on INS_TEST_OUT_3
WfsArbitrator_13301|INF| 65|2022-05-08 22:05:09.541891| MAIN > Turning on INS_TEST_OUT_4
WfsArbitrator_13301|INF| 66|2022-05-08 22:05:10.458537| MAIN > Turning on main
WfsArbitrator_13301|INF| 67|2022-05-08 22:05:12.326978| MAIN > Turning on FAN_2K_AXIAL
WfsArbitrator_13301|INF| 68|2022-05-08 22:05:13.945205| MAIN > Turning on FAN_2K_TANG
WfsArbitrator_13301|INF| 69|2022-05-08 22:05:15.613544| MAIN > Turning on FAN_47
WfsArbitrator_13301|INF| 70|2022-05-08 22:05:16.930923| MAIN > Turning on FAN_LN47

ocam frames
  • ok
power on
  • OK

no open at sunset high wind.

12:30 am high wind gusts >20m/s

1:27am still close due high wind gust

wind_20220508.png

wind2_20220508.png

1:55 am going to Open

Still some wind gusts above 20m/s

GCS shows bad psf...

gcs_1_20220508.png

2:11 am moving to bright star ~10 mag

seeing variable 2"-1.7"

No Bin1, seeing too variable and high.

changing star to a bin2 to have a loop more stable.

manual gain just to get the loop close.

Skip check flux.

mag.14.9

gain:1.15,0.1,0.05

MAIN > offsetXY(): starting offset of -60.148, 60.161 mm
WfsArbitrator_5719 |INF| 271|2022-05-09 09:28:52.900663| MAIN > offsetXYZ(): Focus correction (absolute): 0.000 mm
WfsArbitrator_5719 |ERR| 272|2022-05-09 09:28:52.900986| MAIN > RETRY: Offset outside FoV limits. Target xyz is -137.28 7.67 52.89 (-5002) VAL

05-09 09:30:24.897924| MAIN > arbFunc offsetXYinternal (, 'FLAOWFS', 'LUCIFER', 0.0022557499999999999, 0.60305500000000001, True)
WfsArbitrator_5719 |INF| 274|2022-05-09 09:30:24.898277| MAIN > offsetXY(): starting offset of -0.001, 0.603 mm
WfsArbitrator_5719 |INF| 275|2022-05-09 09:30:24.899248| MAIN > offsetXYZ(): Focus correction (absolute): 0.000 mm
WfsArbitrator_5719 |INF| 276|2022-05-09 09:30:26.805076| MAIN > offsetXY(): done
WfsArbitrator_5719 |DEB| 277|2022-05-09 09:30:26.805161| MAIN > arbFunc offsetXYinternal done.
WfsArbitrator_5719 |DEB| 278|2022-05-09 09:30:32.383371| MAIN > arbFunc doResumeLoop (, 'FLAOWFS', 'LUCIFER')
WfsArbitrator_5719 |INF| 279|2022-05-09 09:30:32.502246| MAIN > Total counts/subap: 35.175152 - framerate: 900.0 - Flux: 303
WfsArbitrator_5719 |DEB| 280|2022-05-09 09:30:32.503369| MAIN > arbFunc doResumeLoop done.

2-05-09 09:32:00.334240| MAIN > arbFunc offsetXYinternal (, 'FLAOWFS', 'LUCIFER', -74.823099999999997, 59.858800000000002, True)
WfsArbitrator_5719 |INF| 285|2022-05-09 09:32:00.334550| MAIN > offsetXY(): starting offset of -75.115, 59.492 mm
WfsArbitrator_5719 |INF| 286|2022-05-09 09:32:00.335711| MAIN > offsetXYZ(): Focus correction (absolute): 0.000 mm
WfsArbitrator_5719 |ERR| 287|2022-05-09 09:32:00.336277| MAIN > RETRY: Offset outside FoV limits. Target xyz is -152.24 7.61 52.89 (-5002) VALUE_OUT_OF_RANGE_ERROR

09:32:00.336277| MAIN > RETRY: Offset outside FoV limits. Target xyz is -152.24 7.61 52.89 (-5002) VALUE_OUT_OF_RANGE_ERROR
WfsArbitrator_5719 |DEB| 288|2022-05-09 09:33:31.149819| MAIN > arbFunc offsetXYinternal (, 'FLAOWFS', 'LUCIFER', 0.59958999999999996, -0.59685900000000003, True)
WfsArbitrator_5719 |INF| 289|2022-05-09 09:33:31.150928| MAIN > offsetXY(): starting offset of 0.602, -0.594 mm
WfsArbitrator_5719 |INF| 290|2022-05-09 09:33:31.151902| MAIN > offsetXYZ(): Focus correction (absolute): 0.000 mm
WfsArbitrator_5719 |INF| 291|2022-05-09 09:33:35.160972| MAIN > offsetXY(): done
WfsArbitrator_5719 |DEB| 292|2022-05-09 09:33:35.161042| MAIN > arbFunc offsetXYinternal done.
WfsArbitrator_5719 |DEB| 293|2022-05-09 09:33:40.342812| MAIN > arbFunc doResumeLoop (, 'FLAOWFS', 'LUCIFER')
WfsArbitrator_5719 |INF| 294|2022-05-09 09:33:40.455875| MAIN > Total counts/subap: 27.465549 - framerate: 900.0 - Flux: 236
WfsArbitrator_5719 |DEB| 295|2022-05-09 09:33:40.456908| MAIN > arbFunc doResumeLoop done.

98|2022-05-09 09:35:05.693775| MAIN > arbFunc doPauseLoop done.
WfsArbitrator_5719 |DEB| 299|2022-05-09 09:35:08.512511| MAIN > arbFunc offsetXYinternal (, 'FLAOWFS', 'LUCIFER', -60.456299999999999, 75.423500000000004, True)
WfsArbitrator_5719 |INF| 300|2022-05-09 09:35:08.512774| MAIN > offsetXY(): starting offset of -60.824, 75.127 mm
WfsArbitrator_5719 |INF| 301|2022-05-09 09:35:08.513556| MAIN > offsetXYZ(): Focus correction (absolute): 0.000 mm
WfsArbitrator_5719 |ERR| 302|2022-05-09 09:35:08.513936| MAIN > RETRY: Offset outside FoV limits. Target xyz is -137.35 22.65 52.89 (-5002) VALUE_OUT_

09:36:41.265057| MAIN > offsetXY(): starting offset of -1.201, 1.194 mm
WfsArbitrator_5719 |INF| 305|2022-05-09 09:36:41.265825| MAIN > offsetXYZ(): Focus correction (absolute): 0.000 mm
WfsArbitrator_5719 |INF| 306|2022-05-09 09:36:45.174972| MAIN > offsetXY(): done
WfsArbitrator_5719 |DEB| 307|2022-05-09 09:36:45.175043| MAIN > arbFunc offsetXYinternal done.
WfsArbitrator_5719 |DEB| 308|2022-05-09 09:36:50.995911| MAIN > arbFunc doResumeLoop (, 'FLAOWFS', 'LUCIFER')
WfsArbitrator_5719 |INF| 309|2022-05-09 09:36:51.105760| MAIN > Total counts/subap: 29.608689 - framerate: 900.0 - Flux: 255
WfsArbitrator_5719 |DEB| 310|2022-05-09 09:36:51.106697| MAIN > arbFunc doResumeLoop done.

2:37am seeing 1.35"

09 09:38:20.556094| MAIN > arbFunc offsetXYinternal (, 'FLAOWFS', 'LUCIFER', -59.259500000000003, 74.226799999999997, True)
WfsArbitrator_5719 |INF| 315|2022-05-09 09:38:20.556374| MAIN > offsetXY(): starting offset of -59.622, 73.936 mm
WfsArbitrator_5719 |INF| 316|2022-05-09 09:38:20.557146| MAIN > offsetXYZ(): Focus correction (absolute): 0.000 mm
WfsArbitrator_5719 |ERR| 317|2022-05-09 09:38:20.557487| MAIN > RETRY: Offset outside FoV limits. Target xyz is -137.35 22.65 52.89 (-5002) VALUE_OUT_OF_RAN

2:45am Done !

2:45 move to Bin1 - osco-

bin1_20220508_1.png

optimize skip frames in ho2

902|2022-05-09 09:52:04.659529| ALERTHANDLER > Received from masterdiagnostic.L a DumpSaved alert with filename /local/aolog/2022/05/09/ADAPTIVE-SECONDARY_00001652089924_20220509_095204.bin
AdSecArbitrator_97878|INF| 903|2022-05-09 09:52:04.659562| ALERTHANDLER > Processdump is skipped
20220905 09:52:04.6605 Alert: DumpSaved: DumpSaved vector ready
AdSecArbitrator_97878|WAR| 904|2022-05-09 09:52:52.063010| MAIN > Skip frame detected! (0.33% of frames skipped.)
AdSecArbitrator_97878|WAR| 905|2022-05-09 09:52:53.170201| MAIN > Skip frame detected! (0.67% of frames skipped.)
AdSecArbitrator_97878|WAR| 906|2022-05-09 09:52:56.273799| MAIN > Skip frame detected! (1.22% of frames skipped.)
AdSecArbitrator_97878|WAR| 907|2022-05-09 09:52:59.587884| MAIN > Skip frame detected! (7.00% of frames skipped.)
AdSecArbitrator_97878|WAR| 908|2022-05-09 09:53:00.592889| MAIN > Skip frame detected! (7.11% of frames skipped.)
AdSecArbitrator_97878|WAR| 909|2022-05-09 09:53:01.765437| MAIN > Skip frame detected! (14.00% of frames skipped.)
AdSecArbitrator_97878|WAR| 910|2022-05-09 09:53:02.772112| MAIN > Skip frame detected! (5.33% of frames skipped.)
AdSecArbitrator_97878|WAR| 911|2022-05-09 09:53:03.779305| MAIN > Skip frame detected! (4.33% of frames skipped.)
AdSecArbitrator_97878|WAR| 912|2022-05-09 09:53:04.792056| MAIN > Skip frame detected! (6.44% of frames skipped.)
AdSecArbitrator_97878|WAR| 913|2022-05-09 09:53:05.982757| MAIN > Skip frame detected! (1.78% of frames skipped.)
AdSecArbitrator_97878|WAR| 914|2022-05-09 09:53:09.249025| MAIN > Skip frame detected! (28.22% of frames skipped.)
AdSecArbitrator_97878|WAR| 915|2022-05-09 09:53:10.260465| MAIN > Skip frame detected! (54.78% of frames skipped.)
AdSecArbitrator_97878|WAR| 916|2022-05-09 09:53:11.266665| MAIN > Skip frame detected! (36.89% of frames skipped.)
AdSecArbitrator_97878|WAR| 917|2022-05-09 09:53:12.272654| MAIN > Skip frame detected! (55.00% of frames skipped.)
AdSecArbitrator_97878|WAR| 918|2022-05-09 09:53:13.287114| MAIN > Skip frame detected! (38.22% of frames skipped.)
AdSecArbitrator_97878|WAR| 919|2022-05-09 09:53:14.348201| MAIN > Skip frame detected! (49.00% of frames skipped.)
AdSecArbitrator_97878|WAR| 920|2022-05-09 09:53:15.555885| MAIN > Skip frame detected! (41.00% of frames skipped.)
AdSecArbitrator_97878|WAR| 921|2022-05-09 09:53:16.580835| MAIN > Skip frame detected! (48.00% of frames skipped.)
AdSecArbitrator_97878|WAR| 922|2022-05-09 09:53:17.585705| MAIN > Skip frame detected! (34.33% of frames skipped.)
AdSecArbitrator_97878|WAR| 923|2022-05-09 09:53:18.590406| MAIN > Skip frame detected! (27.33% of frames skipped.)

It failed during the optimize gain for a TT.

Adsec in failure.

9 09:53:22.238219| MAIN > HO2 optimization completed: Acquisition skipped too many frames (threshold is 30%)
WfsArbitrator_5719 |INF| 510|2022-05-09 09:53:22.238290| MAIN > TT: range 0.20 - 5.00
WfsArbitrator_5719 |INF| 511|2022-05-09 09:53:28.875742| MAIN > TT minimum was not deep enough, forcing iteration (threshold is 0.700000)
WfsArbitrator_5719 |INF| 512|2022-05-09 09:53:32.899975| MAIN > TT minimum was not deep enough, forcing iteration (threshold is 0.700000)
WfsArbitrator_5719 |INF| 513|2022-05-09 09:53:38.651474| MAIN > TT minimum was not deep enough, forcing iteration (threshold is 0.700000)
WfsArbitrator_5719 |INF| 514|2022-05-09 09:53:46.059538| MAIN > TT minimum was not deep enough, forcing iteration (threshold is 0.700000)
WfsArbitrator_5719 |INF| 515|2022-05-09 09:53:51.183044| MAIN > Received exception with code -10045
WfsArbitrator_5719 |ERR| 516|2022-05-09 09:53:51.184225| MAIN > Safe skip active - cannot proceed (-10045) IDL_SAFE_SKIP_ACTIVE
WfsArbitrator_5719 |DEB| 517|2022-05-09 09:53:51.184950| MAIN > arbFunc doStopLoop (, 'FLAOWFS', 'LUCIFER')
WfsArbitrator_5719 |DEB| 518|2022-05-09 09:53:51.188706| MAIN > arbFunc doStopLoop done.

3:00 am preset again

ncpa on triggered an open

9 10:07:42.847203| MAIN > arbFunc updateNcpaInternal (, 'FLAOWFS', 'LUCIFER', 151.13800000000001, 0)
WfsArbitrator_5719 |DEB| 714|2022-05-09 10:07:42.867596| MAIN > arbFunc updateNcpaInternal done.
WfsArbitrator_5719 |DEB| 715|2022-05-09 10:07:44.827847| MAIN > arbFunc doStopLoop (, 'FLAOWFS', 'LUCIFER')
WfsArbitrator_5719 |DEB| 716|2022-05-09 10:07:44.829733| MAIN > arbFunc doStopLoop done.

reclose.

Pause didn't went through

Mon May 9 10:14:03.669 2022 aos.Pause{FLAO}.started - Pause{FLAO}[99]
Mon May 9 10:14:03.683 2022 aos.Pause{FLAO}.failed - Pause{FLAO}[99] - Illegal command for state AOArbitrator.Ready

3:16 skip frames and adsec failure.

adsc_stop/start

ncpa on

optimize_bin1_20220508.png

3:18 am preset again.

seeing variable 1.5"

gcs2_20220508_bin1_3try.png

e detected! (0.22% of frames skipped.)
AdSecArbitrator_120830|WAR| 205|2022-05-09 10:27:41.110413| MAIN > Skip frame detected! (0.56% of frames skipped.)
AdSecArbitrator_120830|WAR| 206|2022-05-09 10:27:42.134910| MAIN > Skip frame detected! (0.22% of frames skipped.)
AdSecArbitrator_120830|WAR| 208|2022-05-09 10:27:44.175305| MAIN > Skip frame detected! (3.56% of frames skipped.)
AdSecArbitrator_120830|WAR| 209|2022-05-09 10:27:45.184565| MAIN > Skip frame detected! (0.67% of frames skipped.)
AdSecArbitrator_120830|WAR| 210|2022-05-09 10:27:47.215195| MAIN > Skip frame detected! (4.00% of frames skipped.)
AdSecArbitrator_120830|WAR| 211|2022-05-09 10:27:48.231566| MAIN > Skip frame detected! (17.00% of frames skipped.)
AdSecArbitrator_120830|WAR| 212|2022-05-09 10:27:49.266774| MAIN > Skip frame detected! (16.56% of frames skipped.)
AdSecArbitrator_120830|WAR| 213|2022-05-09 10:27:51.288550| MAIN > Skip frame detected! (11.33% of frames skipped.)
AdSecArbitrator_120830|WAR| 214|2022-05-09 10:27:52.297365| MAIN > Skip frame detected! (11.56% of frames skipped.)
AdSecArbitrator_120830|WAR| 215|2022-05-09 10:27:53.328669| MAIN > Skip frame detected! (10.67% of frames skipped.)
AdSecArbitrator_120830|WAR| 216|2022-05-09 10:27:54.338159| MAIN > Skip frame detected! (10.11% of frames

HO2 more stable.

ncpa on

triggers the skip frames.

ncpa off

optimize_try3_20220508.png

3:38 bin2 forced.mag 14.8

skip frame ho2

tt scan too high

TT too high

possible skip the last steps.

Doing offsets.

10:48:27.638173| WFS > Executing offsetXY...
WfsArbitrator_5718 |INF| 1829|2022-05-09 10:48:27.638296| WFS > Python command: WfsArbScripts.offsetXY(self, sensor = 'FLAOWFS', instr = 'LUCIFER', offsetx = 0.603083, offsety = 0.013972, brake = True)

2022-05-09 10:48:27.639484| MAIN > offsetXYZ(): Focus correction (absolute): 0.000 mm

10:49:13.553211| WFS > Command returns success .
WfsArbitrator_5718 |INF| 1833|2022-05-09 10:49:14.130240| WFS > Executing resumeLoop...

WfsArbitrator_5719 |INF| 1138|2022-05-09 10:49:07.709286| MAIN > Stage movement timeout after 40 seconds. Retry 1 of 3
WfsArbitrator_5719 |INF| 1139|2022-05-09 10:49:13.552993| MAIN > offsetXY(): done

Luci rotators to grab ->298.

2-05-09 10:49:46.086225| MAIN > Total counts/subap: 38.582622 - framerate: 900.0 - Flux: 332
WfsArbitrator_5719 |DEB| 1146|2022-05-09 10:49:46.086515| MAIN > arbFunc doPauseLoop done.
WfsArbitrator_5719 |DEB| 1147|2022-05-09 10:49:48.980734| MAIN > arbFunc offsetXYinternal (, 'FLAOWFS', 'LUCIFER', -1.1962200000000001, 0.0015662600000000001, True)
WfsArbitrator_5719 |INF| 1148|2022-05-09 10:49:48.980884| MAIN > offsetXY(): starting offset of -1.196, -0.004 mm
WfsArbitrator_5719 |INF| 1149|2022-05-09 10:49:48.981650| MAIN > offsetXYZ(): Focus correction (absolute): 0.000 mm
WfsArbitrator_5719 |INF| 1150|2022-05-09 10:49:51.087076| MAIN > offsetXY(): done

repeat the series of offset and it didn't fail

9 10:52:46.594625| MAIN > arbFunc offsetXYinternal (, 'FLAOWFS', 'LUCIFER', 0.000514095, -0.59671200000000002, True)
WfsArbitrator_5719 |INF| 1185|2022-05-09 10:52:46.594900| MAIN > offsetXY(): starting offset of 0.003, -0.597 mm
WfsArbitrator_5719 |INF| 1186|2022-05-09 10:52:46.595705| MAIN > offsetXYZ(): Focus correction (absolute): 0.000 mm
WfsArbitrator_5719 |INF| 1187|2022-05-09 10:52:48.701350| MAIN > offsetXY(): done
WfsArbitrator_5719 |DEB| 1188|2022-05-09 10:52:48.701440| MAIN > arbFunc offsetXYinternal done.

skip frames

Offset.

10:55:00.243366| MAIN > arbFunc offsetXYinternal (, 'FLAOWFS', 'LUCIFER', -60.456899999999997, 75.422799999999995, True)
WfsArbitrator_5719 |INF| 1211|2022-05-09 10:55:00.243596| MAIN > offsetXY(): starting offset of -60.825, 75.126 mm
WfsArbitrator_5719 |INF| 1212|2022-05-09 10:55:00.244305| MAIN > offsetXYZ(): Focus correction (absolute): 0.000 mm
WfsArbitrator_5719 |ERR| 1213|2022-05-09 10:55:00.244629| MAIN > RETRY: Offset out

925|2022-05-09 10:55:42.415126| MAIN > Skip frame detected! (106.44% of frames skipped.)
AdSecArbitrator_120830|WAR| 926|2022-05-09 10:55:42.415277| MAIN > Maximum skip frames reached: 106.44% of frames skipped. Will alert ICE clients.
AdSecArbitrator_120830|WAR| 927|2022-05-09 10:55:43.420142| MAIN > Skip frame detected! (107.11% of frames skipped.)
AdSecArbitrator_120830|WAR| 928|2022-05-09 10:55:43.420312| MAIN > Maximum skip frames reached: 107.11% of frames skipped. Will alert ICE clients.
AdSecArbitrator_120830|WAR| 929|2022-05-09 10:55:44.424986| MAIN > Skip frame detected! (106.56% of frames skipped.)
AdSecArbitrator_120830|WAR| 930|2022-05-09 10:55:44.425170| MAIN > Maximum skip frames reached: 106.56% of frames skipped. Will alert ICE clients.
AdSecArbitrator_120830|WAR| 931|2022-05-09 10:55:45.550525| MAIN > Skip frame detected! (107.44% of frames skipped.)
AdSecArbitrator_120830|WAR| 932|2022-05-09 10:55:45.550674| MAIN > Maximum skip frames reached: 107.44% of fr

loop open.

skip frames.

reclose

5-09 10:58:33.647400| MAIN > arbFunc offsetXYinternal done.
WfsArbitrator_5719 |DEB| 1248|2022-05-09 10:58:42.597010| MAIN > arbFunc doResumeLoop (, 'FLAOWFS', 'LUCIFER')
WfsArbitrator_5719 |INF| 1249|2022-05-09 10:58:42.700772| MAIN > Total counts/subap: 35.396037 - framerate: 900.0 - Flux: 305
WfsArbitrator_5719 |DEB| 1250|2022-05-09 10:58:42.701784| MAIN > arbFunc doResumeLoop done.

4:05am seeing 1.6"
  • change to bin2
  • ocam not live.
  • masterdiagnostic down/
  • restart process
  • power cycle bcu2k and reconfigure ocam
719 |INF| 1394|2022-05-09 11:15:58.007442| MAIN > calibHODarkInternal(): starting measure, 100 frames
WfsArbitrator_5719 |INF| 1395|2022-05-09 11:15:59.640491| MAIN > ccd.equalize_quadrants(): nothing to do for ocam2
WfsArbitrator_5719 |ERR| 1396|2022-05-09 11:16:02.781483| MAIN > Reading shared buffer masterdiagnostic.L:OPTLOOPBUF (-5001) TIMEOUT_ERROR
  • finished the optimize gain with a TT around 35....Gopt was getting in stable mode.
Not much time left

4:00 am stop. getting into twilight.

forecast.

seeing_20220508.png tau_20220508.png

Cn2_20220508.png

-- JuanCarlosGuerra - 08 May 2022
I Attachment Action Size Date Who Comment
Cn2_20220508.pngpng Cn2_20220508.png manage 697 K 08 May 2022 - 21:36 JuanCarlosGuerra  
bin1_20220508_1.pngpng bin1_20220508_1.png manage 110 K 09 May 2022 - 09:51 JuanCarlosGuerra  
gcs2_20220508_bin1_3try.pngpng gcs2_20220508_bin1_3try.png manage 135 K 09 May 2022 - 10:25 JuanCarlosGuerra  
gcs_1_20220508.pngpng gcs_1_20220508.png manage 163 K 09 May 2022 - 09:11 JuanCarlosGuerra  
optimize_bin1_20220508.pngpng optimize_bin1_20220508.png manage 260 K 09 May 2022 - 10:07 JuanCarlosGuerra  
optimize_try3_20220508.pngpng optimize_try3_20220508.png manage 280 K 09 May 2022 - 10:37 JuanCarlosGuerra  
seeing_20220508.pngpng seeing_20220508.png manage 219 K 08 May 2022 - 21:35 JuanCarlosGuerra  
tau_20220508.pngpng tau_20220508.png manage 273 K 08 May 2022 - 21:36 JuanCarlosGuerra  
wind2_20220508.pngpng wind2_20220508.png manage 875 K 09 May 2022 - 08:28 JuanCarlosGuerra  
wind_20220508.pngpng wind_20220508.png manage 879 K 09 May 2022 - 07:32 JuanCarlosGuerra  
Topic revision: r3 - 09 May 2022, JuanCarlosGuerra
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback