20220906 OSCO

08:40 LUCI script for PresetAO
  • Josh step though and start NCPA
  • LUCI start taking images
  • modal.jpg:
    modal.jpg
  • Popup box with Communication lost to Slope Computer
  • wfs control GUI says AOSet
  • AdSec Control GUI says AORunning
  • OCAM shows loop it closed (flat pupils)
WfsArbitrator_50984|DEB|       332|2022-09-07 08:51:39.008233|             MAIN > arbFunc doSaveStatus (, 'FLAOWFS', 'LUCIFER', '/local/aomeas/adsec_data/20220907/Data_20220907_085137/wfs.fits')
WfsArbitrator_50984|DEB|       333|2022-09-07 08:51:39.013511|             MAIN > getSetupFits(): fits header created
WfsArbitrator_50984|DEB|       334|2022-09-07 08:51:39.043309|             MAIN > arbFunc doSaveStatus done.
WfsArbitrator_50984|DEB|       335|2022-09-07 08:51:53.021487|             MAIN > arbFunc updateNcpaInternal (, 'FLAOWFS', 'LUCIFER', 255.23500000000001, 0)
WfsArbitrator_50984|ERR|       336|2022-09-07 08:51:54.021918|             MAIN > thWriteVar error(-5001): Timeout error
WfsArbitrator_50984|DEB|       337|2022-09-07 08:51:58.587979|             MAIN > arbFunc doStopLoop (, 'FLAOWFS', 'LUCIFER')
WfsArbitrator_50984|ERR|       338|2022-09-07 08:51:59.588940|             MAIN > Timeout waiting for slopecompctrl.L.FASTLINK.ENABLE.CUR (-5001) TIMEOUT_ERROR
20220709 08:51:59.8145 Alert: Error: LOST COMMUNICATION WITH SLOPE COMPUTER
WfsArbitrator_50984|DEB|       339|2022-09-07 08:52:06.445720|             MAIN > arbFunc updateNcpaInternal (, 'FLAOWFS', 'LUCIFER', 255.441, 0)

08:54 StopAO from AOSGUI
  • failed
Wed Sep 7 08:54:59.327 2022 aos.info - TTF Offload request (FLAO) Z: 0.293669 -0.10824 -0.000951671
Wed Sep 7 08:55:00.086 2022 aos.Stop{FLAO}.started - Stop{FLAO}[75] (msg:Stopped from operator console)
Wed Sep 7 08:55:00.494 2022 aos.warning - FLAO: error on status request from AO Arb. Excp: src/Ice/OutgoingAsync.cpp:1049: ::Ice::UnknownException:unknown exception:Traceback (most recent call last): File "aoarbitrator.py", line 997, in RequestAOStatus fstat=self._fsm.getState().getName().split('.') File "/home/aoeng/live/aoroot/lib/python/statemap.py", line 104, in getState raise StateUndefinedExceptionStateUndefinedException
Wed Sep 7 08:55:00.996 2022 aos.warning - FLAO: error on status request from AO Arb. Excp: src/Ice/OutgoingAsync.cpp:1049: ::Ice::UnknownException:unknown exception:Traceback (most recent call last): File "aoarbitrator.py", line 997, in RequestAOStatus fstat=self._fsm.getState().getName().split('.') File "/home/aoeng/live/aoroot/lib/python/statemap.py", line 104, in getState raise StateUndefinedExceptionStateUndefinedException
Wed Sep 7 08:55:01.061 2022 aos.info - HO offload command end
Wed Sep 7 08:55:01.108 2022 aos.Stop{FLAO}.failed - Stop{FLAO}[75] - [AOException] Timeout waiting for slopecompctrl.L.FASTLINK.ENABLE.CUR (-5001) TIMEOUT_ERROR
Wed Sep 7 08:55:01.378 2022 aos.info - HO offload: previous 8 requests ignored
Wed Sep 7 08:55:01.398 2022 aos.info - HO Offload request (FLAO). Z: 0 0 0 0 -35.774 23.3426 1.19959 8.45217 -6.38527 -4.32412 -7.10089 0 0 0 0 -0 0 0 -0 -0 -0 -0
Wed Sep 7 08:55:03.031 2022 aos.info - TTF offload command end

08:56 BCU Interface was down
  • restarted, no change
  • bcu_down.jpg:
    bcu_down.jpg

08:56 Rebin OCAM from HW GUi
  • over illumination
  • Rebin from wfs Control GUI to bin 1 and EMGain to 1, OK
  • Rebin CCD47 from HW GUI. Running now

WfsArbitrator_50984|DEB|       731|2022-09-07 09:14:24.123312|             MAIN > arbFunc updateNcpaInternal (, 'FLAOWFS', 'LUCIFER', 263.779, 0)
WfsArbitrator_50984|DEB|       732|2022-09-07 09:14:24.142357|             MAIN > arbFunc updateNcpaInternal done.
WfsArbitrator_50984|DEB|       733|2022-09-07 09:14:45.741316|             MAIN > arbFunc doStopLoop (, 'FLAOWFS', 'LUCIFER')
20220709 09:14:46.9506 Alert: Error: LOST COMMUNICATION WITH SLOPE COMPUTER
WfsArbitrator_50984|DEB|       734|2022-09-07 09:14:45.743034|             MAIN > arbFunc doStopLoop done.
WfsArbitrator_50984|DEB|       735|2022-09-07 09:14:45.933126|             MAIN > arbFunc updateNcpaInternal (, 'FLAOWFS', 'LUCIFER', 263.88999999999999, 0)
WfsArbitrator_50984|DEB|       736|2022-09-07 09:14:45.947996|             MAIN > arbFunc updateNcpaInternal done.
WfsArbitrator_50984|DEB|       737|2022-09-07 09:14:55.956491|             MAIN > arbFunc updateNcpaInternal (, 'FLAOWFS', 'LUCIFER', 263.94099999999997, 0)

09:14 Lost communication with Slope Computer
  • WFS Control GUI flashed Offline and AOSet
  • AdSec Control GUI StopAO successful
  • BCU Interface still up
  • WFS Control GUI set to bin 2, EMGain 1 OK

09:20 AOSGUI StopAO. Success but HW GUI shows all Red boxes
  • Kill HW GUI and start new one. Now shows green power on

09:23 LUCI script for PresetAO
  • Do all steps to close the loop but did not turn on NCPA
  • GOPT not start. Seeing is 1.3-1.6"
WfsArbitrator_50984|DEB|       985|2022-09-07 09:27:57.395927|             MAIN > arbFunc optimizeGainInternal done.
WfsArbitrator_50984|DEB|       986|2022-09-07 09:27:59.291574|             MAIN > arbFunc startGoptInternal (, 'FLAOWFS', 'LUCIFER')
WfsArbitrator_50984|INF|       987|2022-09-07 09:27:59.291923|             MAIN > Waiting for stable gopt (relative threshold is 15%)
WfsArbitrator_50984|ERR|       988|2022-09-07 09:28:52.925859|             MAIN > RETRY: Gopt did not stabilize within 10 iterations (-5001) TIMEOUT_ERROR
WfsArbitrator_50984|DEB|       989|2022-09-07 09:29:26.305469|             MAIN > arbFunc startGoptInternal (, 'FLAOWFS', 'LUCIFER')
WfsArbitrator_50984|INF|       990|2022-09-07 09:29:26.305814|             MAIN > Waiting for stable gopt (relative threshold is 15%)
WfsArbitrator_50984|ERR|       991|2022-09-07 09:30:20.333550|             MAIN > RETRY: Gopt did not stabilize within 10 iterations (-5001) TIMEOUT_ERROR

09:30 Can not save AO telemetry.
MasterDiagnostic_99348|WAR|       119|2022-09-07 09:47:18.775542|           BCU39 > FramesDownloader BCU39: [InvalidDiagnosticPacketException] Unexpected packetId 2 for frameId 614 (instead of packetId 0 for frameId 614)
MasterDiagnostic_99348|WAR|       120|2022-09-07 09:47:18.775569|           BCU39 > FramesDownloader BCU39: packet successfully ignored !
MasterDiagnostic_99348|WAR|       121|2022-09-07 09:47:21.535682|           BCU39 > FramesDownloader BCU39: [InvalidDiagnosticPacketException] Unexpected packetId 2 for frameId 992 (instead of packetId 13 for frameId 991)
MasterDiagnostic_99348|WAR|       122|2022-09-07 09:47:21.535733|           BCU39 > FramesDownloader BCU39: packet successfully ignored !
MasterDiagnostic_99348|WAR|       123|2022-09-07 09:47:21.775841|           BCU39 > FramesDownloader BCU39: [InvalidDiagnosticPacketException] Unexpected packetId 1 for frameId 268 (instead of packetId 10 for frameId 267)
[telescope@to1 ~]$ ./get_ao_telemetry
Traceback (most recent call last):
 File "/home/AOeng/doug/get_telemetry/get_telemetry.py", line 23, in 
   self = thAOApp(myname='wobble')
 File "/home/AOeng/live/aoroot/lib/python/AdOpt/thAOApp.py", line 175, in __init__
   thrdlib.thStart( IP=addr)
thrdlib.error: thStart error(-2002): The client is already connected
  • Stop start Masterdiagnostics process, but no go
  • Stop Start all WFS processes
  • Now can save Telemetry
  • Still getting above MasterDiagnostics error above

09:50 LUCI PresetAO
  • Loop Closed with NCPA On
  • TN 20220907_095612, 095652,095715 From Optical Loop Data GUI
  • TN from OSA Script
[telescope@to1 ~]$ ./get_ao_telemetry
  => Command line arguements processed
   Set  0   Tracknum  20220907_095953   LuciName  unknown
   Set  1   Tracknum  20220907_100005   LuciName  unknown
   Set  2   Tracknum  20220907_100024   LuciName  unknown
  -> wrote file /home/AOeng/doug/get_telemetry/OSCO/20220907_095952.log
  • gopt_0957.jpg:
    gopt_0957.jpg
  • modal_0957.jpg:
    modal_0957.jpg
  • ttjitter_0957.jpg:
    ttjitter_0957.jpg

10:12 Bin 2 preset
  • 74 elevation
  • 0.9" seeing
[telescope@to1 ~]$ ./get_ao_telemetry
  => Command line arguements processed
   Set  0   Tracknum  20220907_102254   LuciName  unknown
   Set  1   Tracknum  20220907_102304   LuciName  unknown
   Set  2   Tracknum  20220907_102315   LuciName  unknown
  -> wrote file /home/AOeng/doug/get_telemetry/OSCO/20220907_102254.log
  • modal_1033.jpg:
    modal_1033.jpg

10:34 Slope computer error
WfsArbitrator_99442|DEB|       855|2022-09-07 10:34:32.594823|             MAIN > arbFunc updateNcpaInternal done.
WfsArbitrator_99442|DEB|       856|2022-09-07 10:34:46.603627|             MAIN > arbFunc updateNcpaInternal (, 'FLAOWFS', 'LUCIFER', 182.84700000000001, 0)
WfsArbitrator_99442|ERR|       857|2022-09-07 10:34:47.604104|             MAIN > thWriteVar error(-5001): Timeout error
WfsArbitrator_99442|DEB|       858|2022-09-07 10:34:53.277611|             MAIN > arbFunc doStopLoop (, 'FLAOWFS', 'LUCIFER')
20220709 10:34:53.7980 Alert: Error: LOST COMMUNICATION WITH SLOPE COMPUTER
WfsArbitrator_99442|DEB|       859|2022-09-07 10:34:53.281514|             MAIN > arbFunc doStopLoop done.
WfsArbitrator_99442|DEB|       860|2022-09-07 10:35:00.578532|             MAIN > arbFunc updateNcpaInternal (, 'FLAOWFS', 'LUCIFER', 182.69800000000001, 0)
WfsArbitrator_99442|DEB|       861|2022-09-07 10:35:00.593684|             MAIN > arbFunc updateNcpaInternal done.

10:42 LUCI Preset bin 2
  • Close loop but No NCPA
  • No Errors. Observation completed

-- DougMiller - 07 Sep 2022
I Attachment Action Size Date Who Comment
bcu_down.jpgjpg bcu_down.jpg manage 99 K 07 Sep 2022 - 10:20 DougMiller  
gopt_0957.jpgjpg gopt_0957.jpg manage 58 K 07 Sep 2022 - 10:20 DougMiller  
modal.jpgjpg modal.jpg manage 72 K 07 Sep 2022 - 10:21 DougMiller  
modal_0957.jpgjpg modal_0957.jpg manage 69 K 07 Sep 2022 - 10:21 DougMiller  
modal_1033.jpgjpg modal_1033.jpg manage 63 K 07 Sep 2022 - 10:34 DougMiller  
ttjitter_0957.jpgjpg ttjitter_0957.jpg manage 64 K 07 Sep 2022 - 10:22 DougMiller  
Topic revision: r2 - 07 Sep 2022, DougMiller
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback