LUCI1/AO Commissioning

01:50 ACE preset failed because both Ref and Guide star out of patrol field. Want to take sky flat 10' off target

01:58 Now LUCI software not communicating to the IIF.

02:10 ACE Preset + RunAO: Failed
  • Stage moves when I move manually
Wed Oct 19 02:10:45.950 2016 left RunAO alert for LUCIFER: AcquireRef{FLAO}[25] - [AOException] centerStar: FATAL - Timeout error on baysidey (-5001) TIMEOUT_ERROR
Wed Oct 19 02:10:45.960 2016 left RunAO command failed from LUCIFER, type NORMAL, leftRePoint 0, rightRePoint 0

02:16 Preset failed because ADC2 was spinning and camera lens could not center the pupils because they were continuously moving
Wed Oct 19 02:16:38.078 2016 left RunAO alert for LUCIFER: AcquireRef{FLAO}[28] - Camera lens centering did not converge after 180 seconds
Wed Oct 19 02:16:38.087 2016 left RunAO command failed from LUCIFER, type NORMAL, leftRePoint 0, rightRePoint 0

02:23 Too much flux on pyramid
  • Set fw1_acq and a d fw1_obs
Wed Oct 19 02:23:45.638 2016 left RunAO alert for LUCIFER: AcquireRef{FLAO}[31] - [AOException] RETRY: Error in acquire ref: Star too bright: ccd39 is saturated, AO loop cannot work. - code: 0 NO_ERROR (-1000) GENERIC_SYS_ERROR
Wed Oct 19 02:23:45.647 2016 left RunAO command failed from LUCIFER, type NORMAL, leftRePoint 0, rightRePoint 0

02:28 Bayside X stage timeout
Wed Oct 19 02:28:33.915 2016 left RunAO alert for LUCIFER: AcquireRef{FLAO}[34] - [AOException] centerStar: FATAL - Timeout error on baysidex (-5001) TIMEOUT_ERROR
Wed Oct 19 02:28:33.925 2016 left RunAO command failed from LUCIFER, type NORMAL, leftRePoint 0, rightRePoint 0

02:31 CCD39 goes to Not Live
  • set binning to 2x2, but did not come to live
  • Set Loop parameters to bin 2 and reset properly
  • Preset work fine now
Wed Oct 19 02:31:10.507 2016 left Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: PresetAO{FLAO}[36] - [AOException] presetAO: FATAL - Reading shared buffer masterdiagnostic.L:OPTLOOPBUF (-5001) TIMEOUT_ERROR
Wed Oct 19 02:31:10.521 2016 left Preset command failed for LUCIFER mode ADAPTIVEACE_ACTIVE with guidestars, offset, AO reference, Sound=presetFailed.wav

02:41 FOUND BUG:
  • preset on-axis
  • offset outside of patrol field
  • When try to offset anywhere (in or out of patrol field) the offset fails (see below)
  • These offsets should return return a warning that loop is Paused, but continue with offset
    • if outside the patrol field, loop remains Paused
    • if inside the patrol field, loop is Resumed

Wed Oct 19 02:44:53.178 2016 left Offset alert for LUCIFER: Pause{FLAO}[7] - Illegal command for state AOArbitrator.LoopSuspended
Wed Oct 19 02:44:53.188 2016 left Offset command failed for LUCIFER, Sound=offsetFailed.wav

02:44 send iif_offset from IDL to back on-axis
  • Error
      IRCL: offset, 0, 0, ROT=0, COORD=DETXY, MOVE=ABS     (iif_offset)
  
% Compiled module: IRC_OFFSETPOINTING.
===> Error in irc_offsetpointing
        err = Resmsg[0]  = OffsetPointing2 result status: Error
        err = Resmsg[1]  = Pause{FLAO}[7] - Illegal command for state AOArbitrator.LoopSuspended

Wed Oct 19 02:41:29.576 2016 left Offset alert for LUCIFER: Pause{FLAO}[7] - Illegal command for state AOArbitrator.LoopSuspended
Wed Oct 19 02:41:29.602 2016 left Offset command failed for LUCIFER, Sound=offsetFailed.wav

025341 Loop Data, bin 1, 990 Hz

02:54 Offset outside patrol field: OK
Wed Oct 19 02:54:20.407 2016 left Offset command complete with warnings for LUCIFER, Sound=offsetWarning.wav

02:56 Offset back inside patrol field fails because AOS received a PauseAO request from IIF and rejects it with a failure because the system was already Paused
Wed Oct 19 02:54:10.449 2016 aos.OffsetXY{FLAO}.started - OffsetXY{FLAO}[51] (ofsx:359.855[359.855],ofsy:4.82107)
Wed Oct 19 02:54:10.475 2016 aos.OffsetXY{FLAO}.failed - OffsetXY{FLAO}[51] - [AOException] RETRY: Offset outside FoV limits. Target xyz is -445.88 -46.92 54.60 (-5002) VALUE_OUT_OF_RANGE_ERROR
Wed Oct 19 02:55:28.903 2016 aos.warning - FLAO: Variable iif.DIMM.seeing expired
Wed Oct 19 02:56:41.121 2016 aos.Pause{FLAO}.started - Pause{FLAO}[7] 
Wed Oct 19 02:56:41.138 2016 aos.Pause{FLAO}.failed - Pause{FLAO}[7] - Illegal command for state AOArbitrator.LoopSuspended
Wed Oct 19 02:56:43.996 2016 aos.info - FLAO: Variable iif.DIMM.seeing is valid
Wed Oct 19 02:59:48.000 2016 aos.Pause{FLAO}.started - Pause{FLAO}[7] 
Wed Oct 19 02:59:48.016 2016 aos.Pause{FLAO}.failed - Pause{FLAO}[7] - Illegal command for state AOArbitrator.LoopSuspended

03:07 Failed because I had put in FW1=4 not 3 so I moved it to 3 manually. Caused this timeout. User error not AO System error
pyarg              |ERR|      1400|2016-10-19 03:07:48.481642|             MAIN > Timeout error on filterwheel1 (-5001) TIMEOUT_ERROR

03:12 Failure
  • Skip Frame
  • Loop Opens
  • AdSec Arbitrator died
  • Pause is requested from IIF and fails
Wed Oct 19 03:14:09.200 2016 left Offset alert for LUCIFER: Pause{FLAO}[7] - [Exception] exception ::Ice::UnknownUserException{ unknown = AODefs::AOException}
Wed Oct 19 03:14:09.211 2016 left Offset command failed for LUCIFER, Sound=offsetFailed.wav

035517 Loop Data, bin 1, 400 Hz, Seeing ~0.7"

04:17
Wed Oct 19 04:17:02.074 2016 left Offset alert for LUCIFER: Pause{FLAO}[7] - [Exception] Ice.ConnectionRefusedException:Connection refused
Wed Oct 19 04:17:02.101 2016 left Offset command failed for LUCIFER, Sound=offsetFailed.wav
)

adsecarb.L         |INF|       499|2016-10-19 04:17:01.405645|   COMMANDHANDLER > FSM (status: AORunning) has received command 2018 (SetGain)
adsecarb.L         |ERR|       500|2016-10-19 04:17:01.405699|   COMMANDHANDLER > adsecarb: Fsm is discarding command 2018 (SetGain)...

AOARB.L            |ERR|       569|2016-10-19 04:17:01.633566|             MAIN > [Exception] Ice.ConnectionRefusedException:
AOARB.L            |ERR|       569|2016-10-19 04:17:01.633566|             MAIN >  . Connection refused
AOARB.L            |WAR|       570|2016-10-19 04:17:01.633729|             MAIN > Communication with AdSec lost

04:22 Shel Ripped
  • Skip Frame
  • High forces
  • Shell Ripped
  • Failure
  • Recover Fail fixes it
adsecarb.L         |WAR|       123|2016-10-19 04:22:08.735834|             MAIN > Skip frame detected!
adsecarb.L         |INF|       125|2016-10-19 04:22:12.955534|             MAIN > Received cmd: Offloading(0)
adsecarb.L         |INF|       126|2016-10-19 04:22:12.955605|   COMMANDHANDLER > FSM (status: AORunning) has received command 2029 (TTOffload)
adsecarb.L         |INF|       127|2016-10-19 04:22:12.986983|   COMMANDHANDLER > Command TTOffload (code 2029) successfully completed
adsecarb.L         |INF|       128|2016-10-19 04:22:12.987212|             MAIN > Received cmd: PauseAo()
adsecarb.L         |INF|       129|2016-10-19 04:22:12.987247|   COMMANDHANDLER > FSM (status: AORunning) has received command 2027 (Pause Ao)
adsecarb.L         |INF|       130|2016-10-19 04:22:12.995655|   COMMANDHANDLER > Command Pause Ao (code 2027) successfully completed
adsecarb.L         |INF|       131|2016-10-19 04:22:28.013024|             MAIN > Received cmd: ResumeAo()
adsecarb.L         |INF|       132|2016-10-19 04:22:28.013091|   COMMANDHANDLER > FSM (status: AOPause) has received command 2028 (ResumeAo Ao)
adsecarb.L         |INF|       133|2016-10-19 04:22:28.015732|   COMMANDHANDLER > Command ResumeAo Ao (code 2028) successfully completed
adsecarb.L         |INF|       134|2016-10-19 04:22:28.211303|             MAIN > Received cmd: Offloading(1)
adsecarb.L         |INF|       135|2016-10-19 04:22:28.211360|   COMMANDHANDLER > FSM (status: AORunning) has received command 2029 (TTOffload)
adsecarb.L         |INF|       136|2016-10-19 04:22:28.442627|   COMMANDHANDLER > Command TTOffload (code 2029) successfully completed
adsecarb.L         |WAR|       137|2016-10-19 04:22:28.754815|             MAIN > Skip frame detected!
adsecarb.L         |INF|       143|2016-10-19 04:22:34.818370|             MAIN > Received cmd: StopAo(1)
Last message repeated
adsecarb.L         |INF|       144|2016-10-19 04:22:34.818424|   COMMANDHANDLER > FSM (status: AORunning) has received command 2015 (Stop Ao)
adsecarb.L         |INF|       145|2016-10-19 04:22:34.818445|    SERIALIZATION > StopAoImpl(): restoreShape = 1
adsec.L            |INF|      1711|2016-10-19 04:22:34.818701|             MAIN > fsm_stop_ao start
adsec.L            |INF|      1712|2016-10-19 04:22:34.818760|             MAIN > fsm_skip_recovery FORCE set
adsec.L            |INF|      1713|2016-10-19 04:22:35.033502|             MAIN > Null Gain application and start sending null slope...
adsec.L            |INF|      1714|2016-10-19 04:22:35.159527|             MAIN > Cleaned last slopes command.
adsec.L            |INF|      1715|2016-10-19 04:22:35.159608|             MAIN > Modes out of safe range    ID: # -1
adsec.L            |INF|      1716|2016-10-19 04:22:35.159669|             MAIN > Commands out of safe range ID: # 89 # 91 #116 #117 #119
adsec.L            |INF|      1717|2016-10-19 04:22:35.159718|             MAIN > Forces out of safe range   ID: # -1
adsec.L            |INF|      1718|2016-10-19 04:22:35.159753|             MAIN > SafeSkipFrames caused by commands out of range detected.
adsec.L            |INF|      1719|2016-10-19 04:22:35.159787|             MAIN > Recovery from any SafeSkipFrames started...
adsec.L            |INF|      1720|2016-10-19 04:22:35.159816|             MAIN > Getting info on actual mirror position...
adsec.L            |INF|      1721|2016-10-19 04:22:35.163882|             MAIN > Reconstructor filter cleaning...
adsec.L            |INF|      1722|2016-10-19 04:22:35.169736|             MAIN > Control current integrator cleaning up...
adsec.L            |INF|      1723|2016-10-19 04:22:35.189673|             MAIN > Null new delta command contribution
adsec.L            |INF|      1725|2016-10-19 04:22:35.192225|             MAIN > Restore old oversampling...
adsec.L            |INF|      1726|2016-10-19 04:22:35.198920|             MAIN > Done.
adsec.L            |INF|      1727|2016-10-19 04:22:35.264230|             MAIN > Restore control current integrator action....
adsec.L            |INF|      1728|2016-10-19 04:22:35.309748|             MAIN > Done.
adsec.L            |INF|      1729|2016-10-19 04:22:35.309781|             MAIN > Reload last shape selected
adsecarb.L         |ERR|       146|2016-10-19 04:22:36.011451|     ALERTHANDLER > Received an ERROR alert! Forcing Failure state.
adsecarb.L         |ERR|       147|2016-10-19 04:22:36.011523|     ALERTHANDLER > Alert message: COILS DISABLED from housekeeper.L
20161910 04:22:36.0124 Alert: Error: COILS DISABLED
adsec.L            |INF|      1730|2016-10-19 04:22:36.196521|             MAIN > Loading shape: /local/aomeas/adsec_calib/flat/Applicable/LUCIFER.sav
adsec.L            |INF|      1731|2016-10-19 04:22:36.198673|             MAIN > Selected flat: /local/aomeas/adsec_calib/flat/Applicable/LUCIFER.sav
adsec.L            |INF|      1732|2016-10-19 04:22:37.114165|             MAIN > The threshold has been moved to 0.2 as the total forces are included in the computation of the modal filtering.
adsec.L            |INF|      1733|2016-10-19 04:22:37.114211|             MAIN > The threshold has been additionally moved to 0.5 to try flattening better the external edge.
adsec.L            |INF|      1734|2016-10-19 04:22:37.114276|             MAIN > Delta command modal filtering for reducing peak force to 0.30 [N]
adsec.L            |INF|      1735|2016-10-19 04:22:37.148943|             MAIN > total current added in the force vector for the computation of the modal filtering
adsec.L            |INF|      1736|2016-10-19 04:22:37.159387|             MAIN > Maximum mode index used to apply flat -> # 0
adsec.L            |INF|      1737|2016-10-19 04:22:37.160659|             MAIN > Min/Max delta command to apply for flattening -> #  -1.36189e-06 /    1.30812e-06 [m]
20161910 04:22:37.1614 Alert: Warning: stopAo failed or syntax error occured
adsecarb.L         |ERR|       148|2016-10-19 04:22:37.161082|              IDL > Command returns error : IDL Safe params (current absolute or rms or delta, position absolute o delta o rms) out of bounds
adsec.L            |INF|      1738|2016-10-19 04:22:37.160740|             MAIN > Out of threshold request. Not possible to apply a flat command
adsec.L            |INF|      1739|2016-10-19 04:22:37.160763|             MAIN > IDL Safe params (current absolute or rms or delta, position absolute o delta o rms) out of bounds
adsec.L            |INF|      1740|2016-10-19 04:22:37.160796|             MAIN > FSM_LOAD_SHAPE:  -----------> Out of threshold request. Not possible to
adsec.L            |INF|      1740|2016-10-19 04:22:37.160796|             MAIN >  .                    apply a flat command -10046
adsecarb.L         |INF|       149|2016-10-19 04:22:37.218057|   COMMANDHANDLER > Command execution failed: Fsm forced to Failure
adsecarb.L         |ERR|       150|2016-10-19 04:22:37.218154|             MAIN > StopAo: FATAL - IDL Safe params (current absolute or rms or delta, position absolute o delta o rms) out of bounds
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN > Cannot dispatch alert: Outgoing.cpp:493: Ice::UnknownException:
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  . unknown exception:
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  . Traceback (most recent call last):
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  .   File "aoarbitrator.py", line 184, in alert
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  .     self._loop._fsm.SkfAlert()
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 597, in trans_sm
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  .     getattr(self.getState(), attrib)(self, *arglist)
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 444, in SkfAlert
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  .     AOArbitrator_Default.SkfAlert(self, fsm)
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 78, in SkfAlert
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  .     self.Default(fsm)
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 89, in Default
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  .     raise statemap.TransitionUndefinedException(msg)
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  . TransitionUndefinedException: 
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  . 	State: AOArbitrator.LoopClosed
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  . 	Transition: SkfAlert
adsecarb.L         |WAR|       151|2016-10-19 04:22:37.219405|             MAIN >  . 
adsecarb.L         |INF|       152|2016-10-19 04:22:37.446647|     ALERTHANDLER > Received from masterdiagnostic.L a DumpSaved alert with filename /home/aoeng/aolog/2016/10/19/ADAPTIVE-SECONDARY_00001476850957_20161019_042237.bin
adsecarb.L         |INF|       153|2016-10-19 04:22:37.446674|     ALERTHANDLER > Processdump is skipped
20161910 04:22:37.4475 Alert: DumpSaved: DumpSaved vector ready
adsecarb.L         |ERR|       154|2016-10-19 04:22:37.473536|     ALERTHANDLER > Received an ERROR alert! Forcing Failure state.
adsecarb.L         |ERR|       155|2016-10-19 04:22:37.473590|     ALERTHANDLER > Alert message: COILS DISABLED from housekeeper.L
20161910 04:22:37.4742 Alert: Error: COILS DISABLED
adsecarb.L         |WAR|       156|2016-10-19 04:22:38.472849|             MAIN > Skip frame detected!

04:27 AdSec Arbitrator died while fsm_set_flat was running.
  • restarted AdSec Arb process and Shell was flat
  • Log messages listed below when Arb is restarted
adsec.L            |INF|      2146|2016-10-19 04:26:06.469681|             MAIN > fsm_set_flat() procedure done.
adsecarb.L         |INF|        41|2016-10-19 04:27:09.476883|             MAIN > Creating arbitrator... serialize method: TEXT
adsecarb.L         |INF|        42|2016-10-19 04:27:09.477101|             MAIN > AdSecArbitrator version 3.1 64b [TH](Built: Sep 17 2016 08:53:36) 
adsecarb.L         |INF|        43|2016-10-19 04:27:09.477143|             MAIN > AdSecArbitrator starting from state PowerOff (from configuration file)
adsecarb.L         |INF|        44|2016-10-19 04:27:09.477159|             MAIN > Wind lower,upper thresholds: 3.000000,6.000000 m/s
adsecarb.L         |INF|        45|2016-10-19 04:27:09.477171|             MAIN > Elevation threshold for resting: 25.000000 deg
adsecarb.L         |INF|        46|2016-10-19 04:27:09.477183|             MAIN > Min elevation threshold for setting the shell: 25.000000 

044243 Loop Data, bin 2, 400 Hz, Seeing ~0.8 (1.2" on dimm :()

04:46 Skip Frame, Loop Open
adsecarb.L         |WAR|       270|2016-10-19 04:46:58.504331|             MAIN > Skip frame detected!
adsecarb.L         |INF|       273|2016-10-19 04:47:00.926049|             MAIN > Received cmd: StopAo(1)
Last message repeated
adsecarb.L         |INF|       274|2016-10-19 04:47:00.926125|   COMMANDHANDLER > FSM (status: AORunning) has received command 2015 (Stop Ao)
adsecarb.L         |INF|       275|2016-10-19 04:47:00.926149|    SERIALIZATION > StopAoImpl(): restoreShape = 1
adsec.L            |INF|      2248|2016-10-19 04:47:00.926469|             MAIN > fsm_stop_ao start
adsec.L            |INF|      2249|2016-10-19 04:47:00.926525|             MAIN > fsm_skip_recovery FORCE set
adsec.L            |INF|      2250|2016-10-19 04:47:01.141871|             MAIN > Null Gain application and start sending null slope...
adsec.L            |INF|      2251|2016-10-19 04:47:01.268510|             MAIN > Cleaned last slopes command.
adsec.L            |INF|      2252|2016-10-19 04:47:01.268599|             MAIN > Modes out of safe range    ID: # -1
adsec.L            |INF|      2253|2016-10-19 04:47:01.268640|             MAIN > Commands out of safe range ID: # -1
adsec.L            |INF|      2254|2016-10-19 04:47:01.268677|             MAIN > Forces out of safe range   ID: # 39
adsec.L            |INF|      2255|2016-10-19 04:47:01.268706|             MAIN > SafeSkipFrames caused by forces out of range detected.
adsec.L            |INF|      2256|2016-10-19 04:47:01.268734|             MAIN > Recovery from any SafeSkipFrames started...
adsec.L            |INF|      2257|2016-10-19 04:47:01.268767|             MAIN > Getting info on actual mirror position...
adsec.L            |INF|      2258|2016-10-19 04:47:01.272913|             MAIN > Reconstructor filter cleaning...
adsec.L            |INF|      2259|2016-10-19 04:47:01.278943|             MAIN > Control current integrator cleaning up...
adsec.L            |INF|      2260|2016-10-19 04:47:01.298937|             MAIN > Null new delta command contribution
adsec.L            |INF|      2262|2016-10-19 04:47:01.301471|             MAIN > Restore old oversampling...
adsec.L            |INF|      2263|2016-10-19 04:47:01.307621|             MAIN > Done.
adsec.L            |INF|      2264|2016-10-19 04:47:01.372569|             MAIN > Restore control current integrator action....
adsec.L            |INF|      2265|2016-10-19 04:47:01.419207|             MAIN > Done.
adsec.L            |INF|      2266|2016-10-19 04:47:01.419338|             MAIN > Reload last shape selected
adsec.L            |INF|      2267|2016-10-19 04:47:02.246847|             MAIN > Loading shape: /local/aomeas/adsec_calib/flat/Applicable/default.sav
adsec.L            |INF|      2268|2016-10-19 04:47:02.249120|             MAIN > Selected flat: /local/aomeas/adsec_calib/flat/Applicable/default.sav
adsec.L            |INF|      2269|2016-10-19 04:47:03.196355|             MAIN > The threshold has been moved to 0.2 as the total forces are included in the computation of the modal filtering.
adsec.L            |INF|      2270|2016-10-19 04:47:03.196421|             MAIN > The threshold has been additionally moved to 0.5 to try flattening better the external edge.
adsec.L            |INF|      2271|2016-10-19 04:47:03.196466|             MAIN > Delta command modal filtering for reducing peak force to 0.30 [N]
adsec.L            |INF|      2272|2016-10-19 04:47:03.231586|             MAIN > total current added in the force vector for the computation of the modal filtering
adsec.L            |INF|      2273|2016-10-19 04:47:03.241093|             MAIN > Maximum mode index used to apply flat -> # 629
adsec.L            |INF|      2274|2016-10-19 04:47:03.242223|             MAIN > Min/Max delta command to apply for flattening -> #  -4.79906e-06 /    5.86345e-06 [m]
adsec.L            |INF|      2275|2016-10-19 04:47:03.361976|             MAIN > COMM  MAX [m] 6.89624e-05
adsec.L            |INF|      2276|2016-10-19 04:47:03.362033|             MAIN > FORCE MAX [N] 0.264161
adsec.L            |INF|      2277|2016-10-19 04:47:04.034443|             MAIN > Done.
adsec.L            |INF|      2278|2016-10-19 04:47:04.036014|             MAIN > All Done. Succefully recovered from SafeSkipFrames.

04:57 RunAO fail during Optimize gain because seeing increased from 1.2 to 1.8 on DIM
Wed Oct 19 04:57:21.811 2016 left RunAO alert for LUCIFER: AcquireRef{FLAO}[188] - [AOException] OptimizeGain: FATAL - Safe skip active - cannot proceed (-10045) IDL_SAFE_SKIP_ACTIVE
Wed Oct 19 04:57:21.822 2016 left RunAO command failed from LUCIFER, type NORMAL, leftRePoint 0, rightRePoint 0

05:00 Handover to ARGOS

baysidex.L         |INF|       444|2016-10-20 11:43:20.738765|             MAIN > Stage pos (mm): -84.414, current (A): -2.110
baysidex.L         |INF|       445|2016-10-20 11:44:45.003047|             MAIN > Moving to: -92.326 mm
baysidex.L         |INF|       446|2016-10-20 11:44:45.838973|             MAIN > Stage pos (mm): -84.636, current (A): 0.000
baysidex.L         |INF|       447|2016-10-20 11:44:46.200328|             MAIN > Stage pos (mm): -86.358, current (A): -2.270
baysidex.L         |INF|       448|2016-10-20 11:44:46.583119|             MAIN > Stage pos (mm): -88.136, current (A): -2.170
baysidex.L         |INF|       449|2016-10-20 11:44:46.954254|             MAIN > Stage pos (mm): -89.977, current (A): -2.490
baysidex.L         |INF|       450|2016-10-20 11:44:47.305782|             MAIN > Stage pos (mm): -91.723, current (A): -2.990
baysidex.L         |INF|       451|2016-10-20 11:44:47.667088|             MAIN > Stage pos (mm): -92.219, current (A): -3.510
baysidex.L         |INF|       452|2016-10-20 11:44:48.028416|             MAIN > Stage pos (mm): -92.421, current (A): -2.830
baysidex.L         |INF|       453|2016-10-20 11:44:48.360662|             MAIN > Stage pos (mm): -92.421, current (A): -2.340
baysidex.L         |INF|       454|2016-10-20 11:44:48.712139|             MAIN > Stage pos (mm): -92.421, current (A): -1.940
baysidex.L         |INF|       455|2016-10-20 11:45:26.914024|             MAIN > Moving to: -92.326 mm
baysidex.L         |INF|       456|2016-10-20 11:45:27.331994|             MAIN > Stage pos (mm): -92.404, current (A): -0.010
baysidex.L         |INF|       457|2016-10-20 11:45:27.683493|             MAIN > Stage pos (mm): -92.279, current (A): -0.620
baysidex.L         |INF|       458|2016-10-20 11:45:28.046855|             MAIN > Stage pos (mm): -92.279, current (A): -0.810
baysidex.L         |INF|       459|2016-10-20 11:45:28.429623|             MAIN > Stage pos (mm): -92.279, current (A): -1.060
baysidex.L         |INF|       460|2016-10-20 11:45:28.789000|             MAIN > Stage pos (mm): -92.279, current (A): -1.280
baysidex.L         |INF|       461|2016-10-20 11:45:29.152291|             MAIN > Stage pos (mm): -92.279, current (A): -1.510
baysidex.L         |INF|       462|2016-10-20 11:45:29.503880|             MAIN > Stage pos (mm): -92.280, current (A): -1.740
baysidex.L         |INF|       463|2016-10-20 11:45:29.867128|             MAIN > Stage pos (mm): -92.280, current (A): -1.960
baysidex.L         |INF|       464|2016-10-20 11:45:30.228411|             MAIN > Stage pos (mm): -92.281, current (A): -2.160
baysidex.L         |INF|       465|2016-10-20 11:45:30.587834|             MAIN > Stage pos (mm): -92.285, current (A): -2.350
baysidex.L         |INF|       466|2016-10-20 11:45:30.970624|             MAIN > Stage pos (mm): -92.292, current (A): -2.530
baysidex.L         |INF|       467|2016-10-20 11:45:31.302636|             MAIN > Stage pos (mm): -92.301, current (A): -2.630
baysidex.L         |INF|       468|2016-10-20 11:45:31.633731|             MAIN > Stage pos (mm): -92.306, current (A): -2.730
baysidex.L         |INF|       469|2016-10-20 11:45:31.928880|             MAIN > Stage pos (mm): -92.311, current (A): -2.780
baysidex.L         |INF|       470|2016-10-20 11:45:32.241150|             MAIN > Stage pos (mm): -92.315, current (A): -2.790

adsecarb.L         |WAR|      1673|2016-10-20 12:13:39.689331|             MAIN > Skip frame detected!
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN > Cannot dispatch alert: Outgoing.cpp:493: Ice::UnknownException:
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  . unknown exception:
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  . Traceback (most recent call last):
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  .   File "aoarbitrator.py", line 184, in alert
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  .     self._loop._fsm.SkfAlert()
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 601, in trans_sm
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  .     getattr(self.getState(), attrib)(self, *arglist)
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 78, in SkfAlert
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  .     self.Default(fsm)
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 89, in Default
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  .     raise statemap.TransitionUndefinedException(msg)
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  . TransitionUndefinedException: 
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  . 	State: AOArbitrator.Ready
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  . 	Transition: SkfAlert
adsecarb.L         |WAR|      1674|2016-10-20 12:13:39.690847|             MAIN >  . 
adsecarb.L         |WAR|      1675|2016-10-20 12:13:40.698514|             MAIN > Skip frame detected!

adsec.L            |INF|      1376|2016-10-20 12:29:09.309544|             MAIN > fsm_stop_ao start
adsec.L            |INF|      1377|2016-10-20 12:29:09.309604|             MAIN > fsm_skip_recovery FREEZE set
adsec.L            |INF|      1378|2016-10-20 12:29:09.309639|             MAIN > fsm_skip_recovery FORCE set
adsec.L            |INF|      1379|2016-10-20 12:29:09.309672|             MAIN > fsm_skip_recovery NOLOADSHAPE set
adsecarb.L         |WAR|      1980|2016-10-20 12:29:09.498376|             MAIN > Skip frame detected!
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN > Cannot dispatch alert: Outgoing.cpp:493: Ice::UnknownException:
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  . unknown exception:
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  . Traceback (most recent call last):
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  .   File "aoarbitrator.py", line 184, in alert
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  .     self._loop._fsm.SkfAlert()
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 601, in trans_sm
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  .     getattr(self.getState(), attrib)(self, *arglist)
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 78, in SkfAlert
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  .     self.Default(fsm)
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 89, in Default
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  .     raise statemap.TransitionUndefinedException(msg)
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  . TransitionUndefinedException: 
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  . 	State: AOArbitrator.Ready
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  . 	Transition: SkfAlert
adsecarb.L         |WAR|      1981|2016-10-20 12:29:09.513313|             MAIN >  . 

-- %USERSIG{DougMiller - 2016-10-19}%
Topic revision: r5 - 09 Nov 2016, DougMiller
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback