SX SOUL Commissioning

https://wiki.lbto.org/Commissioning/Com20200930

01:02
  • Getting Ready
02:09
  • Doing an AGN first. We're doing it with an off-axis star.
  • NCG 6240
  • Measured magnitude 13.6, catalog, 12.1...
  • GOPT: ~0.9 and stable.
  • Bin 2, 854 Hz
  • g_tt_1.52_ho1_0.11_ho2_0.10.fits
  • PSF Star: Showing 14.3 magnitude
  • GOPT 0.065
  • g_tt_1.20_ho1_0.06_ho2_0.20
  • Highly Elongated. GRRRR
02:45
  • OSCO
  • Full AUTO!
  • Safe skip, when the loop was opening, before we loaded the script. I don't think that it was a problem, just some odd errors with ICE.
  • I think that we were getting the stuff ready for OSCO.
  • OSCO Successful.
AdSecArbitrator _24445|INF| 3613|2020-09-30 02:48:05.655230| COMMANDHANDLER > Command Stop Ao (code 2015) successfully completed
AdSecArbitrator_24445|WAR| 3614|2020-09-30 02:48:05.669857| MAIN > Cannot dispatch alert: ConnectionI.cpp:2924: Ice::CloseConnectionException:
AdSecArbitrator_24445|WAR| 3614|2020-09-30 02:48:05.669857| MAIN > . protocol error: connection closed
AdSecArbitrator_24445|WAR| 3615|2020-09-30 02:48:07.003652| MAIN > Skip frame detected!
20203009 02:48:25.0411 Alert: HighOrderOffload: HighOrderOffload vector ready
  • Wed Sep 30 02:48:00.249 2020 aos.warning - FLAO: error on status request from AO Arb. Excp: src/Ice/OutgoingAsync.cpp:1049: ::Ice::UnknownException:unknown exception:Traceback (most recent call last): File "aoarbitrator.py", line 993, in RequestAOStatus fstat=self._fsm.getState().getName().split('.') File "/home/aoeng/soul/aoroot/lib/python/statemap.py", line 104, in getState raise StateUndefinedExceptionStateUndefinedException
    Wed Sep 30 02:48:00.749 2020 aos.warning - FLAO: error on status request from AO Arb. Excp: src/Ice/OutgoingAsync.cpp:1049: ::Ice::UnknownException:unknown exception:Traceback (most recent call last): File "aoarbitrator.py", line 993, in RequestAOStatus fstat=self._fsm.getState().getName().split('.') File "/home/aoeng/soul/aoroot/lib/python/statemap.py", line 104, in getState raise StateUndefinedExceptionStateUndefinedException
    Wed Sep 30 02:48:01.250 2020 aos.warning - FLAO: error on status request from AO Arb. Excp: src/Ice/OutgoingAsync.cpp:1049: ::Ice::UnknownException:unknown exception:Traceback (most recent call last): File "aoarbitrator.py", line 993, in RequestAOStatus fstat=self._fsm.getState().getName().split('.') File "/home/aoeng/soul/aoroot/lib/python/statemap.py", line 104, in getState raise StateUndefinedExceptionStateUndefinedException
    Wed Sep 30 02:48:01.750 2020 aos.warning - FLAO: error on status request from AO Arb. Excp: src/Ice/OutgoingAsync.cpp:1049: ::Ice::UnknownException:unknown exception:Traceback (most recent call last): File "aoarbitrator.py", line 993, in RequestAOStatus fstat=self._fsm.getState().getName().split('.') File "/home/aoeng/soul/aoroot/lib/python/statemap.py", line 104, in getState raise StateUndefinedExceptionStateUndefinedException
    Wed Sep 30 02:48:02.250 2020 aos.warning - FLAO: error on status request from AO Arb. Excp: src/Ice/OutgoingAsync.cpp:1049: ::Ice::UnknownException:unknown exception:Traceback (most recent call last): File "aoarbitrator.py", line 993, in RequestAOStatus fstat=self._fsm.getState().getName().split('.') File "/home/aoeng/soul/aoroot/lib/python/statemap.py", line 104, in getState raise StateUndefinedExceptionStateUndefinedException
    Wed Sep 30 02:48:02.750 2020 aos.warning - FLAO: error on status request from AO Arb. Excp: src/Ice/OutgoingAsync.cpp:1049: ::Ice::UnknownException:unknown exception:Traceback (most recent call last): File "aoarbitrator.py", line 993, in RequestAOStatus fstat=self._fsm.getState().getName().split('.') File "/home/aoeng/soul/aoroot/lib/python/statemap.py", line 104, in getState raise StateUndefinedExceptionStateUndefinedException
    Wed Sep 30 02:48:03.251 2020 aos.warning - FLAO: error on status request from AO Arb. Excp: src/Ice/OutgoingAsync.cpp:1049: ::Ice::UnknownException:unknown exception:Traceback (most recent call last): File "aoarbitrator.py", line 993, in RequestAOStatus fstat=self._fsm.getState().getName().split('.') File "/home/aoeng/soul/aoroot/lib/python/statemap.py", line 104, in getState raise StateUndefinedExceptionStateUndefinedException
    Wed Sep 30 02:48:03.751 2020 aos.warning - FLAO: error on status request from AO Arb. Excp: src/Ice/OutgoingAsync.cpp:1049: ::Ice::UnknownException:unknown exception:Traceback (most recent call last): File "aoarbitrator.py", line 993, in RequestAOStatus fstat=self._fsm.getState().getName().split('.') File "/home/aoeng/soul/aoroot/lib/python/statemap.py", line 104, in getState raise StateUndefinedExceptionStateUndefinedException
    Wed Sep 30 02:48:04.252 2020 aos.warning - FLAO: error on status request from AO Arb. Excp: src/Ice/OutgoingAsync.cpp:1049: ::Ice::UnknownException:unknown exception:Traceback (most recent call last): File "aoarbitrator.py", line 993, in RequestAOStatus fstat=self._fsm.getState().getName().split('.') File "/home/aoeng/soul/aoroot/lib/python/statemap.py", line 104, in getState raise StateUndefinedExceptionStateUndefinedException
    Wed Sep 30 02:48:04.752 2020 aos.warning - FLAO: error on status request from AO Arb. Excp: src/Ice/OutgoingAsync.cpp:1049: ::Ice::UnknownException:unknown exception:Traceback (most recent call last): File "aoarbitrator.py", line 993, in RequestAOStatus fstat=self._fsm.getState().getName().split('.') File "/home/aoeng/soul/aoroot/lib/python/statemap.py", line 104, in getState raise StateUndefinedExceptionStateUndefinedException
    Wed Sep 30 02:48:05.252 2020 aos.warning - FLAO: error on status request from AO Arb. Excp: src/Ice/OutgoingAsync.cpp:1049: ::Ice::UnknownException:unknown exception:Traceback (most recent call last): File "aoarbitrator.py", line 993, in RequestAOStatus fstat=self._fsm.getState().getName().split('.') File "/home/aoeng/soul/aoroot/lib/python/statemap.py", line 104, in getState raise StateUndefinedExceptionStateUndefinedException
    Wed Sep 30 02:48:05.656 2020 aos.info - FLAO Alert [ code:177] SkipFrame
    Wed Sep 30 02:48:09.799 2020 aos.info - HO offload command end
03:15
  • Palomar 10
  • Rerotator Error.
  • I'm homing the rerotator, then we will send a new preset with a different PA.
  • The X position was railed at 100 in bin1
  • It's now at about 50 in bin 2.
  • Guide star shows 13.3
  • Bin2 928 Hz
  • GOPT about 0.1
  • g_tt_1.16_ho1_0.11_ho2_0.08
  • DIMM problems, again...................
08:34
  • Wed Sep 30 03:33:36.206 2020 aos.Pause{FLAO}.started - Pause{FLAO}[599]
    Wed Sep 30 03:33:36.255 2020 aos.Pause{FLAO}.failed - Pause{FLAO}[599] - [Exception] exception ::Ice::UnknownUserException{ unknown = AODefs::AOException}
    Wed Sep 30 03:33:40.025 2020 aos.info - FLAO Alert [ code:0] Error
    Wed Sep 30 03:33:52.182 2020 aos.info - FLAO Alert [ code:0] Error
    Wed Sep 30 03:34:04.338 2020 aos.info - FLAO Alert [ code:0] Error
  • MAIN > Timeout - Will emergencyReact next loop [FastDiagnostic.cpp:1090]
    fastdiagn_24391 |WAR| 1602476|2020-09-30 03:33:15.702015| MAIN > Timeout - Will emergencyReact now [FastDiagnostic.cpp:1085]
    fastdiagn_24391 |ERR| 1602477|2020-09-30 03:33:15.702050| FUNCTEMERGENCYSTOP > emergencyReact = 1 [Funct.cpp:95]
    fastdiagn_24391 |INF| 1602478|2020-09-30 03:33:15.702087| ADAM-MODBUS > AdamModbus: disabling coils...
    fastdiagn_24391 |INF| 1602479|2020-09-30 03:33:15.718320| FUNCTEMERGENCYSTOP > [FunctEmergencyStop] try to send Alert to AdSec Arbitrator [Funct.cpp:124]
    fastdiagn_24391 |INF| 1602480|2020-09-30 03:33:15.718426| FUNCTEMERGENCYSTOP > [FunctEmergencyStop] Alert to AdSec Arbitrator succesfully sent! [Funct.cpp:134]
    fastdiagn_24391 |INF| 1602481|2020-09-30 03:33:15.733010| MAIN > dumpadsecdiagnhistory succesfully sent to masterdiagnostic [FastDiagnostic.cpp:1300]
    fastdiagn_24391 |WAR| 1602482|2020-09-30 03:33:17.757146| MAIN > Timeout - Will emergencyReact next loop [FastDiagnostic.cpp:1090]
    fastdiagn_24391 |WAR| 1602483|2020-09-30 03:33:19.758021| MAIN > Timeout - Will emergencyReact now [FastDiagnostic.cpp:1085]
    fastdiagn_24391 |ERR| 1602484|2020-09-30 03:33:19.758043| FUNCTEMERGENCYSTOP > emergencyReact = 1 [Funct.cpp:95]
    fastdiagn_24391 |INF| 1602485|2020-09-30 03:33:19.758073| ADAM-MODBUS > AdamModbus: disabling coils...
    fastdiagn_24391 |ERR| 1602486|2020-09-30 03:33:19.758209| ADAM-MODBUS > [TcpReceiveException] Error receiving data (errno=11, error message means: Resource temporarily unavailable, received Bytes = 0 ), socket closed. (code -15003) TcpConnection receive error File TcpConnection.cpp line 133
    fastdiagn_24391 |ERR| 1602487|2020-09-30 03:33:19.758229| ADAM-MODBUS > SetSingleCoil: failed to RecvForceSingleCoil
    fastdiagn_24391 |INF| 1602488|2020-09-30 03:33:19.785001| FUNCTEMERGENCYSTOP > [FunctEmergencyStop] try to send Alert to AdSec Arbitrator [Funct.cpp:124]
    fastdiagn_24391 |INF| 1602489|2020-09-30 03:33:19.785084| FUNCTEMERGENCYSTOP > [FunctEmergencyStop] Alert to AdSec Arbitrator succesfully sent! [Funct.cpp:134]
    fastdiagn_24391 |INF| 1602490|2020-09-30 03:33:19.793898| MAIN > dumpadsecdiagnhistory already sent. [FastDiagnostic.cpp:1311]

  • OCAM not live.
  • Shell ripped and everything stopped.
  • The images before the rip looked like the offset started before the loop paused.
  • It looks like the pause/resume weren't syncronized with the moving of the telescope and re-sending the preset fixed it?
  • The resuming of the loop prematurely, or pausing it late, caused the adsec to attempt to apply too much force and the shell ripped.
  • Actuator 164 is jumping, but it wasn't during the event.
04:19
  • Pal 10 finished OK, doing another OSCO
  • OK
04:39
  • HR 8799
  • ADI
  • Bin 1
  • 1700Hz
  • GOPT 4.57
  • g_tt_0.79_ho1_0.17_ho2_0.21
  • Shows 7.4, should be 5.94
  • Running OK.
06:42
  • Finishing up with ADI
  • Finished

-- GregoryTaylor - 30 Sep 2020
Topic revision: r3 - 30 Sep 2020, GregoryTaylor
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback