SX and DX On-Sky Checkout

03:40 Startup AO systems

04:00 SX Bayside stage homing error

04:15 Started Optical Loop process on both sides

04:20 After ~10 ties at 4 different rotator angle, finally homed - Last night and tonight seems that 160 rotator angle

04:30 Tried to start goptgui.py on both
  • both failed
[AOeng@ao-wfsdx2 source]$ goptgui.py 
Configuration directory: /home/aoeng/aoroot/conf/wfs/current/processConf
Traceback (most recent call last):
  File "/home/aoeng/aoroot/bin/goptgui.py", line 411, in 
    main = gopt(qApp)
  File "/home/aoeng/aoroot/bin/goptgui.py", line 148, in __init__
    self.getz0()
  File "/home/aoeng/aoroot/bin/goptgui.py", line 402, in getz0
    self.z0 = self.asm.get_z0()
  File "/home/aoeng/aoroot/lib/python/AdOpt/asm.py", line 59, in get_z0
    cmd = 'print, m2z4adsec("'+self.getKLname()+'", findgen(n_elements(kl)), kl, %d%s)' % (maxMode, lucikeyword)
  File "/home/aoeng/aoroot/lib/python/AdOpt/asm.py", line 29, in getKLname
    return m2c.split('/')[-2]
IndexError: list index out of range

04:56 DX AdSec Arbitrator died
  • No commands were being executed
  • AdSec was just flat
adsecarb.R         |WAR|       599|2016-09-18 04:56:40.879780|             MAIN > Cannot read coils status ([AOVarException] thGetVar errorVariable: adamhousekeeper.R.COIL_STATUS (type INT_VARIABLE, len 1)
adsecarb.R         |WAR|       599|2016-09-18 04:56:40.879780|             MAIN >  . )
adsecarb.R         |ERR|       600|2016-09-18 04:56:40.879831|             MAIN > AbstractArbitrator::Run() Caught exception: [AOVarException] Error writing variable (code -3008) Operation allowed to variable owner only File RTDBvar.cpp line 496Variable: adsecarb.R.SAFESKIP_PERCENT (type REAL_VARIABLE, len 1)
adsecarb.R         |ERR|       600|2016-09-18 04:56:40.879831|             MAIN >  .
adsecarb.R         |ERR|       601|2016-09-18 04:56:40.879960|             MAIN > Error in AdSecArbitrator: [AOVarException] Error writing variable (code -3008) Operation allowed to variable owner only File RTDBvar.cpp line 496Variable: adsecarb.R.SAFESKIP_PERCENT (type REAL_VARIABLE, len 1)
adsecarb.R         |ERR|       601|2016-09-18 04:56:40.879960|             MAIN >  .
adsecarb.R         |INF|       605|2016-09-18 04:56:40.897797|             MAIN > Closing ICE server
adsecarb.R         |INF|       606|2016-09-18 04:56:40.898680|             MAIN > Communication with MsgD stopped
  • Restart process
  • Shell still set

05:24 LUCI binoc script (twined)
  • AO727
  • Bin 1, 9.09 mag
pyarg              |DEB|       258|2016-09-18 05:24:48.232126|             MAIN > ('Delay base 949, delay inc 1, repetitions 4127',)
pyarg              |DEB|       259|2016-09-18 05:24:48.232148|             MAIN > ('Real framerate 196.986112 Hz',)
pyarg              |INF|       260|2016-09-18 05:24:48.413055|             MAIN > Setting tiptilt to amp= 0.00, freq= 0.00, offx= 0.00, offy= 0.00
pyarg              |INF|       261|2016-09-18 05:24:48.477155|             MAIN > ccd39 change binning done
pyarg              |DEB|       262|2016-09-18 05:24:48.513926|             MAIN > ('selected program 1x1, 400 kpix/sec',)
pyarg              |DEB|       263|2016-09-18 05:24:48.513991|             MAIN > ('Delay base 5074, delay inc 10, repetitions 493',)
pyarg              |DEB|       264|2016-09-18 05:24:48.514025|             MAIN > ('Real framerate 99.955020 Hz',)
pyarg              |INF|       265|2016-09-18 05:24:48.945958|             MAIN > Setting tiptilt to amp= 0.52, freq= 0.00, offx= 0.00, offy= 0.00
pyarg              |INF|       266|2016-09-18 05:24:48.948523|             MAIN > calibHODarkInternal(): starting measure, 1000 frames
pyarg              |INF|       267|2016-09-18 05:24:49.451068|             MAIN > ccd.equalize_quadrants(): Current settings: bin 1, readout speed 400 kpix/sec
pyarg              |INF|       268|2016-09-18 05:24:49.451307|             MAIN > ccd.equalize_quadrants(): Starting values: (700, 700, 700, 700)
pyarg              |ERR|       269|2016-09-18 05:24:51.931985|             MAIN > Reading shared buffer masterdiagnostic.R:OPTLOOPBUF (-5001) TIMEOUT_ERROR

Sun Sep 18 05:25:17.874 2016 right Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: Stop{FLAO}[175]
Sun Sep 18 05:25:17.883 2016 right Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: PresetAO{FLAO}[176] - [AOException] presetAO: FATAL - Reading shared buffer masterdiagnostic.R:OPTLOOPBUF (-5001) TIMEOUT_ERROR
Sun Sep 18 05:25:17.913 2016 right Preset command failed for LUCIFER mode ADAPTIVEACE_ACTIVE with guidestars, AO reference, Sound=presetFailed.wav

05:28 I killed the wrong CCD39

05:32 SX Preset failed due to AGW
  • DX side continued to Closed loop
  • Would be nice to be able to only one side preset.
  • Instead, has to send preset to both sides
  • Jochen says can lock mirror on side that completed
Sun Sep 18 05:33:15.589 2016 left Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: Stop{FLAO}[365]
Sun Sep 18 05:33:15.600 2016 left Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: AGw command status: OK
Sun Sep 18 05:33:15.614 2016 left Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: error: setProbePosition: couldn't move probe to position (SFP) (x,y,z): 85.6564, 157.594, 15.4604 mm. Patrol field limits ?
Sun Sep 18 05:33:15.628 2016 left Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: guidestar list successfully changed but probe couldn't be moved to position.
Sun Sep 18 05:33:15.642 2016 left Preset command failed for LUCIFER mode ADAPTIVEACE_ACTIVE with guidestars, AO reference, Sound=presetFailed.wav

05:56 SX then DX Offset failed within 20 seconds - had performed 10-12 preset on each side

adsecarb.R |INF| 307|2016-09-18 05:56:53.955384| COMMANDHANDLER > FSM (status: AORunning) has received command 2027 (Pause Ao) 
adsecarb.R |INF| 308|2016-09-18 05:56:53.966876| COMMANDHANDLER > Command Pause Ao (code 2027) successfully completed 
adsecarb.R |WAR| 309|2016-09-18 05:56:54.350456| MAIN > Skip frame detected! 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > Cannot dispatch alert: Outgoing.cpp:493: Ice::UnknownException: 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . unknown exception: 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . Traceback (most recent call last): 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . File "aoarbitrator.py", line 184, in alert 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . self._loop._fsm.SkfAlert() 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 597, in trans_sm 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . getattr(self.getState(), attrib)(self, *arglist) 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 78, in SkfAlert 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . self.Default(fsm) 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 89, in Default 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . raise statemap.TransitionUndefinedException(msg) 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . TransitionUndefinedException: 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . State: AOArbitrator.LoopSuspended 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . Transition: SkfAlert 
adsecarb.R |WAR| 310|2016-09-18 05:56:54.356405| MAIN > . 
adsecarb.R |WAR| 311|2016-09-18 05:56:55.366957| MAIN > Skip frame detected! 
adsecarb.R |WAR| 312|2016-09-18 05:56:55.371410| MAIN > Cannot dispatch alert: Outgoing.cpp:493: Ice::UnknownException: 
adsecarb.R |WAR| 312|2016-09-18 05:56:55.371410| MAIN > . unknown exception: 
  • systems shows AO Paused on both sides
  • Optical gain become large
  • Click Open Loop on WFS Control GUI and everything OK
  • Next preset is fine

06:03 Preset OK

06:11 Safe Skip on Left side
  • Same error as above
  • Tried to reclose the loop but couldn't

06:20 RunAO
  • DX success. Stay in Bin 1
  • SX failed. Switched to Bin 2
    • Skip frame and same error as above
  • Found that Bin 2 reconstructor were ones measured at incorrect angle, but rotator was moving the correct rotator value
  • Coplied table_LUCIFER_ACE-AO.txt from flao@

06:50 Closed loop
  • SX bin 1
  • DX bin 2
  • Opt loop date
    • SX 065148
    • DX 065151
    • SX 065348
    • DX 065350

06:56 DX Loop Opened
  • TCS received loop open signal
  • Reclosed loop manually on engineering GUI
  • Clicked Re-close Loop on FLAO GUI. Nothing happened
  • Stop/Start AO Arbitrator: nothing
  • Stop/Start AdSec Arbitrator: nothing
  • Stop/Start WFS Arbitrator: loop open

07:13 During DX center pupil routine, correct information was received from the AOS (swing arm not deployed)
  • Shell Ripped
  • AOS right reports shell is Safe
adsecarb.R         |ERR|       126|2016-09-18 07:13:11.726511|             MAIN > Swing arm not deployed! Resting the shell.
adsecarb.R         |INF|       127|2016-09-18 07:13:11.849577|   COMMANDHANDLER > forceRest(): disabling coils
adsecarb.R         |INF|       128|2016-09-18 07:13:11.849609|      ADAM-MODBUS > AdamModbus: disabling coils...
adsecarb.R         |ERR|       129|2016-09-18 07:13:11.849739|      ADAM-MODBUS > [TcpReceiveException] Error receiving data (errno=0 ), socket closed. (code -15003) TcpConnection receive error File TcpConnection.cpp line 131
adsecarb.R         |ERR|       130|2016-09-18 07:13:11.849773|      ADAM-MODBUS > SetSingleCoil: failed to RecvForceSingleCoil
adsecarb.R         |INF|       131|2016-09-18 07:13:11.877041|   COMMANDHANDLER > FSM (status: AORunning) has received command 2010 (Rest)
adsecarb.R         |INF|       132|2016-09-18 07:13:12.007608|      ADAM-MODBUS > AdamModbus: disabling coils...
adsec.R            |INF|      3023|2016-09-18 07:13:12.022513|             MAIN > Set  WFS port #0 on switch BCU
adsec.R            |INF|      3024|2016-09-18 07:13:12.023333|             MAIN > WFS port #0 on switch BCU successfully set.
adsec.R            |INF|      3025|2016-09-18 07:13:12.024285|             MAIN > Settings:
adsec.R            |INF|      3026|2016-09-18 07:13:12.024328|             MAIN > Skip threshold: MAX DELTA MODE   0.000260000
adsec.R            |INF|      3027|2016-09-18 07:13:12.024356|             MAIN > Skip threshold: MAX ABS MODE     0.000130000
adsec.R            |INF|      3028|2016-09-18 07:13:12.024396|             MAIN > Skip threshold: DELTA COMMAND    1.00000e-05
adsec.R            |INF|      3029|2016-09-18 07:13:12.024420|             MAIN > Skip threshold: MAX COMMAND      0.000120000
adsec.R            |INF|      3030|2016-09-18 07:13:12.024445|             MAIN > Skip threshold: MIN COMMAND          0.00000
adsec.R            |INF|      3031|2016-09-18 07:13:12.024466|             MAIN > Skip threshold: MAX FORCE           0.770000
adsec.R            |INF|      3032|2016-09-18 07:13:12.024487|             MAIN > Skip threshold: MIN FORCE          -0.770000
adsec.R            |INF|      3033|2016-09-18 07:13:12.034838|             MAIN > Safe skip thresholds successfully set.
adsec.R            |INF|      3034|2016-09-18 07:13:12.034873|             MAIN > Installing error handling code
adsec.R            |INF|      3035|2016-09-18 07:13:12.034917|             MAIN > Reset optical loop gain to zero
adsec.R            |INF|      3036|2016-09-18 07:13:12.039280|             MAIN > Disable actuators coils with ADAM
adsec.R            |INF|      3038|2016-09-18 07:13:12.156332|             MAIN > Coils disabled. Cleaning currents
adsec.R            |INF|      3039|2016-09-18 07:13:13.245384|             MAIN > Recontructor cleaning started...
adsec.R            |INF|      3040|2016-09-18 07:13:13.962888|             MAIN > Recontructor cleaning done.
adsec.R            |INF|      3041|2016-09-18 07:13:13.963077|             MAIN > Start sending diagnostic frames...
adsec.R            |INF|      3042|2016-09-18 07:13:13.964067|             MAIN > Settings:
adsec.R            |INF|      3043|2016-09-18 07:13:13.964108|             MAIN > Skip threshold: MAX DELTA MODE   0.000260000
adsec.R            |INF|      3044|2016-09-18 07:13:13.964135|             MAIN > Skip threshold: MAX ABS MODE     0.000130000
adsec.R            |INF|      3045|2016-09-18 07:13:13.964171|             MAIN > Skip threshold: DELTA COMMAND    1.00000e-05
adsec.R            |INF|      3046|2016-09-18 07:13:13.964199|             MAIN > Skip threshold: MAX COMMAND      0.000120000
adsec.R            |INF|      3047|2016-09-18 07:13:13.964222|             MAIN > Skip threshold: MIN COMMAND          0.00000
adsec.R            |INF|      3048|2016-09-18 07:13:13.964244|             MAIN > Skip threshold: MAX FORCE           0.770000
adsec.R            |INF|      3049|2016-09-18 07:13:13.964266|             MAIN > Skip threshold: MIN FORCE          -0.770000
adsec.R            |INF|      3050|2016-09-18 07:13:13.995606|             MAIN > Done
adsec.R            |INF|      3051|2016-09-18 07:13:13.998984|             MAIN > Trigger disabled
adsec.R            |INF|      3052|2016-09-18 07:13:13.999487|             MAIN > Offset disabled
adsec.R            |INF|      3053|2016-09-18 07:13:13.999523|             MAIN > IF Trigger successfully set
adsec.R            |INF|      3054|2016-09-18 07:13:14.002503|             MAIN > Accelerometers successfully set
adsec.R            |INF|      3055|2016-09-18 07:13:14.002642|             MAIN > Update progress successfully completed (2)
adsec.R            |INF|      3056|2016-09-18 07:13:14.002670|             MAIN > Change RMS thresholds to avoid spurious diagnostic dumps when the shell is REST.
adsec.R            |INF|      3057|2016-09-18 07:13:14.022954|             MAIN > RMS thresholds successfully changed.
adsec.R            |INF|      3058|2016-09-18 07:13:14.029403|             MAIN > Oversampling frames disabled.
adsec.R            |INF|      3059|2016-09-18 07:13:14.029443|             MAIN > Disabling DSP WATCHDOG...
adsec.R            |INF|      3060|2016-09-18 07:13:14.032426|             MAIN > ... done.
adsec.R            |INF|      3061|2016-09-18 07:13:14.033368|             MAIN > Uploading reference wave paramenters...
adsec.R            |INF|      3062|2016-09-18 07:13:14.218552|             MAIN > ... done.
adsec.R            |INF|      3063|2016-09-18 07:13:14.218619|             MAIN > Enabling DSP WATCHDOG...
adsec.R            |INF|      3064|2016-09-18 07:13:14.222219|             MAIN > ... done.
adsec.R            |INF|      3065|2016-09-18 07:13:14.406133|             MAIN > Oversampling frames disabled.
adsec.R            |INF|      3066|2016-09-18 07:13:14.406209|             MAIN > Reference wave decreased
adsec.R            |INF|      3067|2016-09-18 07:13:14.571847|             MAIN > No misfunctioning capacitive sensors found.
adsec.R            |INF|      3068|2016-09-18 07:13:14.572121|             MAIN > No gap contamination found.
adsec.R            |INF|      3069|2016-09-18 07:13:14.572335|             MAIN > Dust test successfully completed
adsec.R            |INF|      3070|2016-09-18 07:13:14.572458|             MAIN > Update progress successfully completed (2)
adsec.R            |INF|      3071|2016-09-18 07:13:14.572601|             MAIN > FSM Rip done.
adsecarb.R         |INF|       133|2016-09-18 07:13:15.814516|   COMMANDHANDLER > Command Rest (code 2010) successfully completed
20161809 07:13:16.6897 Alert: Error: COILS DISABLED
20161809 07:13:33.6270 Alert: HighOrderOffload: HighOrderOffload vector ready

07:25 Could not put FLAO GUI into Intervention mode
  • GUI had Iterrupt Command button active
  • Sent new preset and it was performed in Auto mode.
  • At the end of preset, Set Intervention button was available
  • Clicked it for RunAO

07:32 Close loop on both side
  • Now offset happening on both sides to start with
  • Before sent 5 or 6 offset to left first and then started doing both
  • Olga use a different twin routine to make the LUCI script

07:40 Bayside stages are not turning off after Offset
  • See chevron noise on DX CCD39
  • Alfio?

07:41 DX Safe Skip the Loop Open
adsecarb.R         |WAR|       604|2016-09-18 07:47:41.256407|             MAIN > Skip frame detected!
20161809 07:47:56.7331 Alert: HighOrderOffload: HighOrderOffload vector ready
adsecarb.R         |INF|       625|2016-09-18 07:48:13.709723|             MAIN > Received cmd: StopAo(1)
Last message repeated
adsecarb.R         |INF|       626|2016-09-18 07:48:13.709799|   COMMANDHANDLER > FSM (status: AORunning) has received command 2015 (Stop Ao)
adsecarb.R         |INF|       627|2016-09-18 07:48:13.709836|    SERIALIZATION > StopAoImpl(): restoreShape = 1
adsec.R            |INF|      3237|2016-09-18 07:48:13.710225|             MAIN > fsm_stop_ao start
adsec.R            |INF|      3238|2016-09-18 07:48:13.710334|             MAIN > fsm_skip_recovery FORCE set
adsec.R            |INF|      3239|2016-09-18 07:48:13.926468|             MAIN > Null Gain application and start sending null slope...
adsec.R            |INF|      3240|2016-09-18 07:48:14.056447|             MAIN > Cleaned last slopes command.
adsec.R            |INF|      3241|2016-09-18 07:48:14.056522|             MAIN > Modes out of safe range    ID: # -1
adsec.R            |INF|      3242|2016-09-18 07:48:14.056563|             MAIN > Commands out of safe range ID: # -1
adsec.R            |INF|      3243|2016-09-18 07:48:14.056612|             MAIN > Forces out of safe range   ID: # 87 #115
adsec.R            |INF|      3244|2016-09-18 07:48:14.056640|             MAIN > SafeSkipFrames caused by forces out of range detected.
adsec.R            |INF|      3245|2016-09-18 07:48:14.056662|             MAIN > Recovery from any SafeSkipFrames started...
adsec.R            |INF|      3246|2016-09-18 07:48:14.056684|             MAIN > Getting info on actual mirror position...
adsec.R            |INF|      3247|2016-09-18 07:48:14.062313|             MAIN > Reconstructor filter cleaning...
adsec.R            |INF|      3248|2016-09-18 07:48:14.068447|             MAIN > Control current integrator cleaning up...
adsec.R            |INF|      3249|2016-09-18 07:48:14.088001|             MAIN > Null new delta command contribution
adsec.R            |INF|      3251|2016-09-18 07:48:14.090835|             MAIN > Restore old oversampling...
adsec.R            |INF|      3252|2016-09-18 07:48:14.098000|             MAIN > Done.
adsec.R            |INF|      3253|2016-09-18 07:48:14.151834|             MAIN > Restore control current integrator action....
adsec.R            |INF|      3254|2016-09-18 07:48:14.196619|             MAIN > Done.

07:48 Safe Skip and Loop Opened

adsecarb.L         |WAR|      3696|2016-09-18 07:48:16.019756|             MAIN > Skip frame detected!
adsecarb.L         |INF|      3708|2016-09-18 07:48:29.285627|             MAIN > Received cmd: StopAo(1)
Last message repeated
adsec.L            |INF|      4481|2016-09-18 07:48:29.286172|             MAIN > fsm_stop_ao start
adsec.L            |INF|      4482|2016-09-18 07:48:29.286234|             MAIN > fsm_skip_recovery FORCE set
adsec.L            |INF|      4483|2016-09-18 07:48:29.501523|             MAIN > Null Gain application and start sending null slope...
adsec.L            |INF|      4484|2016-09-18 07:48:29.627325|             MAIN > Cleaned last slopes command.
adsec.L            |INF|      4485|2016-09-18 07:48:29.627417|             MAIN > Modes out of safe range    ID: # -1
adsec.L            |INF|      4486|2016-09-18 07:48:29.627469|             MAIN > Commands out of safe range ID: # -1
adsec.L            |INF|      4487|2016-09-18 07:48:29.627508|             MAIN > Forces out of safe range   ID: #140
adsec.L            |INF|      4488|2016-09-18 07:48:29.627633|             MAIN > SafeSkipFrames caused by forces out of range detected.
adsec.L            |INF|      4489|2016-09-18 07:48:29.627675|             MAIN > Recovery from any SafeSkipFrames started...
adsec.L            |INF|      4490|2016-09-18 07:48:29.627713|             MAIN > Getting info on actual mirror position...
adsec.L            |INF|      4491|2016-09-18 07:48:29.631630|             MAIN > Reconstructor filter cleaning...
adsec.L            |INF|      4492|2016-09-18 07:48:29.636918|             MAIN > Control current integrator cleaning up...
adsec.L            |INF|      4493|2016-09-18 07:48:29.653485|             MAIN > Null new delta command contribution
adsec.L            |INF|      4495|2016-09-18 07:48:29.655566|             MAIN > Restore old oversampling...
adsec.L            |INF|      4496|2016-09-18 07:48:29.662029|             MAIN > Done.

08:03 Send binoc ACE Preset
  • IIF waiting on PresetAO on Both sides
  • Click Cancel Preset on FLAO GUI
  • IIF show Preset Complete then RunAO Failed
  • This is OK. Just Alfio and Luca's was to stop after PresetAO

08:20 DX Bayside X Stage Timeout
  • FLAO GUI not allowing me to change from Intervention to Auto modes
  • Next preset runs in intervention
  • After Preset OK, can now set Auto mode
Sun Sep 18 08:20:49.111 2016 right Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: Stop{FLAO}[337]
Sun Sep 18 08:20:49.120 2016 right Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: PresetAO{FLAO}[338] - [AOException] presetAO: FATAL - Timeout error on baysidex (-5001) TIMEOUT_ERROR
Sun Sep 18 08:20:49.134 2016 right Preset command failed for LUCIFER mode ADAPTIVEACE_ACTIVE with guidestars, AO reference, Sound=presetFailed.wav

baysidex.R |INF| 653|2016-09-18 08:18:42.700770| MAIN > Moving to: -100.162 mm 
baysidex.R |INF| 654|2016-09-18 08:18:43.981089| MAIN > Stage pos (mm): -76.455, current (A): -1.580 
baysidex.R |INF| 655|2016-09-18 08:18:44.332672| MAIN > Stage pos (mm): -77.498, current (A): -1.820 
baysidex.R |INF| 656|2016-09-18 08:18:44.684232| MAIN > Stage pos (mm): -78.520, current (A): -2.600 
baysidex.R |INF| 657|2016-09-18 08:18:45.047628| MAIN > Stage pos (mm): -79.571, current (A): -2.890 
baysidex.R |INF| 658|2016-09-18 08:18:45.408839| MAIN > Stage pos (mm): -80.648, current (A): -3.720 
baysidex.R |INF| 659|2016-09-18 08:18:45.770104| MAIN > Stage pos (mm): -81.739, current (A): -4.280 
baysidex.R |INF| 660|2016-09-18 08:18:46.131454| MAIN > Stage pos (mm): -82.079, current (A): -4.640 
baysidex.R |INF| 661|2016-09-18 08:18:46.492780| MAIN > Stage pos (mm): -82.083, current (A): -2.310 
baysidex.R |INF| 662|2016-09-18 08:18:46.852114| MAIN > Stage pos (mm): -82.083, current (A): -2.310 
baysidex.R |INF| 663|2016-09-18 08:18:47.203677| MAIN > Stage pos (mm): -82.083, current (A): -2.320 
baysidex.R |INF| 664|2016-09-18 08:18:47.566974| MAIN > Stage pos (mm): -82.083, current (A): -2.310 
baysidex.R |INF| 665|2016-09-18 08:18:47.918528| MAIN > Stage pos (mm): -82.083, current (A): -2.300 
baysidex.R |INF| 666|2016-09-18 08:18:48.271039| MAIN > Stage pos (mm): -82.083, current (A): -2.310 
baysidex.R |INF| 667|2016-09-18 08:18:48.603086| MAIN > Stage pos (mm): -82.083, current (A): -2.310 
baysidex.R |INF| 668|2016-09-18 08:18:48.935091| MAIN > Stage pos (mm): -82.083, current (A): -2.320 

08:24 Testing

baysidex.R |INF| 698|2016-09-18 08:24:18.291683| MAIN > Moving to: -100.000 mm 
baysidex.R |INF| 699|2016-09-18 08:24:18.594379| MAIN > Stage pos (mm): -84.996, current (A): -2.660 
baysidex.R |INF| 700|2016-09-18 08:24:18.944010| MAIN > Stage pos (mm): -85.022, current (A): -3.700 
baysidex.R |INF| 701|2016-09-18 08:24:19.307297| MAIN > Stage pos (mm): -85.549, current (A): -4.440 
baysidex.R |INF| 702|2016-09-18 08:24:19.666684| MAIN > Stage pos (mm): -86.620, current (A): -3.480 
baysidex.R |INF| 703|2016-09-18 08:24:20.018243| MAIN > Stage pos (mm): -87.654, current (A): -3.440 
baysidex.R |INF| 704|2016-09-18 08:24:20.369800| MAIN > Stage pos (mm): -88.174, current (A): -2.310 
baysidex.R |INF| 705|2016-09-18 08:24:20.721413| MAIN > Stage pos (mm): -88.174, current (A): -2.320 
baysidex.R |INF| 706|2016-09-18 08:24:21.072918| MAIN > Stage pos (mm): -88.174, current (A): -2.310 
baysidex.R |INF| 707|2016-09-18 08:24:21.426480| MAIN > Stage pos (mm): -88.174, current (A): -2.320 
baysidex.R |INF| 708|2016-09-18 08:24:21.776047| MAIN > Stage pos (mm): -88.174, current (A): -2.320 
baysidex.R |INF| 709|2016-09-18 08:24:22.127636| MAIN > Stage pos (mm): -88.174, current (A): -2.320 
baysidex.R |INF| 710|2016-09-18 08:24:22.490874| MAIN > Stage pos (mm): -88.174, current (A): -2.320 

08:35 Closed loop on 9.4 mag star Optical Loop Data
  • SX NCPA ON 309
  • SX NCPA OFF 310
  • DX NCPA OFF 351
  • DX NCPA ON 352

08:43

adsecarb.L         |WAR|      4161|2016-09-18 08:43:42.530966|             MAIN >  . 
adsecarb.L         |WAR|      4162|2016-09-18 08:43:43.871003|             MAIN > Skip frame detected!
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN > Cannot dispatch alert: Outgoing.cpp:493: Ice::UnknownException:
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  . unknown exception:
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  . Traceback (most recent call last):
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  .   File "aoarbitrator.py", line 184, in alert
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  .     self._loop._fsm.SkfAlert()
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 597, in trans_sm
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  .     getattr(self.getState(), attrib)(self, *arglist)
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 78, in SkfAlert
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  .     self.Default(fsm)
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 89, in Default
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  .     raise statemap.TransitionUndefinedException(msg)
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  . TransitionUndefinedException: 
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  .    State: AOArbitrator.LoopSuspended
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  .    Transition: SkfAlert
adsecarb.L         |WAR|      4163|2016-09-18 08:43:44.039450|             MAIN >  . 

08:42 DX in Pause and not changing state
  • Sending Offset PCS is waiting on the RIGHT due to failure on Left did not allow this step
  • Steve released sycn offset
  • Offset failed on both side (via IIF)
  • Click Open Loop on WFS Control GUI on both sides
  • Click Cancel Preset on FLAO GUIs
adsecarb.R         |INF|       841|2016-09-18 08:42:41.263332|             MAIN > Received cmd: PauseAo()
adsecarb.R         |INF|       842|2016-09-18 08:42:41.263365|   COMMANDHANDLER > FSM (status: AORunning) has received command 2027 (Pause Ao)
adsecarb.R         |INF|       843|2016-09-18 08:42:41.275969|   COMMANDHANDLER > Command Pause Ao (code 2027) successfully completed
20161809 08:43:00.6285 Alert: HighOrderOffload: HighOrderOffload vector ready

08:54 CCD39 goes Not Live during Close Loop phase
  • cycle slope computer,
pyarg              |INF|      1273|2016-09-18 08:54:13.433299|             MAIN > ccd39 change binning done
pyarg              |ERR|      1274|2016-09-18 08:54:14.467408|             MAIN > Timeout waiting for slopecompctrl.R.MASTERD.DECIMATION.CUR (-5001) TIMEOUT_ERROR

09:01 SX Received StopAO te * Why??

adsecarb.L         |INF|      4797|2016-09-18 09:01:40.413918|             MAIN > Received cmd: StopAo(1)
Last message repeated
adsec.L            |INF|      4767|2016-09-18 09:01:40.414392|             MAIN > fsm_stop_ao start
adsec.L            |INF|      4768|2016-09-18 09:01:40.414480|             MAIN > fsm_skip_recovery FORCE set
adsec.L            |INF|      4769|2016-09-18 09:01:40.630330|             MAIN > Null Gain application and start sending null slope...
adsec.L            |INF|      4770|2016-09-18 09:01:40.759213|             MAIN > Cleaned last slopes command.
adsec.L            |INF|      4771|2016-09-18 09:01:40.759281|             MAIN > Modes out of safe range    ID: # -1
adsec.L            |INF|      4772|2016-09-18 09:01:40.759322|             MAIN > Commands out of safe range ID: # -1
adsec.L            |INF|      4773|2016-09-18 09:01:40.759350|             MAIN > Forces out of safe range   ID: #  9 # 11 # 15
adsec.L            |INF|      4774|2016-09-18 09:01:40.759371|             MAIN > SafeSkipFrames caused by forces out of range detected.
adsec.L            |INF|      4775|2016-09-18 09:01:40.759389|             MAIN > Recovery from any SafeSkipFrames started...
adsec.L            |INF|      4776|2016-09-18 09:01:40.759411|             MAIN > Getting info on actual mirror position...
adsec.L            |INF|      4777|2016-09-18 09:01:40.763115|             MAIN > Reconstructor filter cleaning...
adsec.L            |INF|      4778|2016-09-18 09:01:40.768878|             MAIN > Control current integrator cleaning up...
adsec.L            |INF|      4779|2016-09-18 09:01:40.787440|             MAIN > Null new delta command contribution
adsec.L            |INF|      4781|2016-09-18 09:01:40.790112|             MAIN > Restore old oversampling...
adsec.L            |INF|      4782|2016-09-18 09:01:40.797723|             MAIN > Done.

adsecarb.L         |WAR|      4863|2016-09-18 09:14:51.043769|             MAIN > Skip frame detected!
20161809 09:15:50.8145 Alert: HighOrderOffload: HighOrderOffload vector ready
20161809 09:16:10.6401 Alert: HighOrderOffload: HighOrderOffload vector ready
20161809 09:16:30.8006 Alert: HighOrderOffload: HighOrderOffload vector ready
adsecarb.L         |INF|      4883|2016-09-18 09:16:45.777197|             MAIN > Received cmd: StopAo(1)
Last message repeated
adsec.L            |INF|      4883|2016-09-18 09:16:45.777656|             MAIN > fsm_stop_ao start
adsec.L            |INF|      4884|2016-09-18 09:16:45.777727|             MAIN > fsm_skip_recovery FORCE set
adsec.L            |INF|      4885|2016-09-18 09:16:45.992954|             MAIN > Null Gain application and start sending null slope...
adsec.L            |INF|      4886|2016-09-18 09:16:46.122979|             MAIN > Cleaned last slopes command.
adsec.L            |INF|      4887|2016-09-18 09:16:46.123048|             MAIN > Modes out of safe range    ID: # -1
adsec.L            |INF|      4888|2016-09-18 09:16:46.123080|             MAIN > Commands out of safe range ID: # -1
adsec.L            |INF|      4889|2016-09-18 09:16:46.123113|             MAIN > Forces out of safe range   ID: #  0 #  1 #  9 # 14
adsec.L            |INF|      4890|2016-09-18 09:16:46.123140|             MAIN > SafeSkipFrames caused by forces out of range detected.
adsec.L            |INF|      4891|2016-09-18 09:16:46.123161|             MAIN > Recovery from any SafeSkipFrames started...
adsec.L            |INF|      4892|2016-09-18 09:16:46.123182|             MAIN > Getting info on actual mirror position...
adsec.L            |INF|      4893|2016-09-18 09:16:46.126997|             MAIN > Reconstructor filter cleaning...
adsec.L            |INF|      4894|2016-09-18 09:16:46.133463|             MAIN > Control current integrator cleaning up...
adsec.L            |INF|      4895|2016-09-18 09:16:46.151752|             MAIN > Null new delta command contribution
adsec.L            |INF|      4897|2016-09-18 09:16:46.154272|             MAIN > Restore old oversampling...
adsec.L            |INF|      4898|2016-09-18 09:16:46.160922|             MAIN > Done.

09:50 Force bad seeing on both sides
  • Hopefully not so many safe skips

09:58 SX bayside X Stage timeout
pyarg              |INF|      4213|2016-09-18 09:58:36.570095|             MAIN > source_acq.acquire(): current error X=-0.134 mm, Y=0.346 mm
pyarg              |ERR|      4214|2016-09-18 09:58:46.661112|             MAIN > Timeout error on baysidex (-5001) TIMEOUT_ERROR

10:31 About 2 hours ago the DX Optical Gain routine died
  • When restart it, one set of data are saved by Optical Diag routine
  • Optical Gain routine does not calculate a gain value and dies
  • I deleted the directory Data_20160918_250000, but did not change behavior
  • How do I get the Optical Gain routine running?

10:42 Safe Skip, Loop Opens, AdSec Arbitrator dies
adsecarb.R         |WAR|      2212|2016-09-18 10:42:25.868686|             MAIN > Skip frame detected!
adsecarb.R         |INF|      2227|2016-09-18 10:42:40.454585|             MAIN > Received cmd: StopAo(1)
Last message repeated
adsec.R            |INF|      4095|2016-09-18 10:42:40.455033|             MAIN > fsm_stop_ao start
adsec.R            |INF|      4096|2016-09-18 10:42:40.455135|             MAIN > fsm_skip_recovery FORCE set
adsecarb.R         |INF|      2228|2016-09-18 10:42:40.454660|   COMMANDHANDLER > FSM (status: AORunning) has received command 2015 (Stop Ao)
adsecarb.R         |INF|      2229|2016-09-18 10:42:40.454700|    SERIALIZATION > StopAoImpl(): restoreShape = 1
adsec.R            |INF|      4097|2016-09-18 10:42:40.672197|             MAIN > Null Gain application and start sending null slope...
adsec.R            |INF|      4098|2016-09-18 10:42:40.802342|             MAIN > Cleaned last slopes command.
adsec.R            |INF|      4099|2016-09-18 10:42:40.802497|             MAIN > Modes out of safe range    ID: # -1
adsec.R            |INF|      4100|2016-09-18 10:42:40.802600|             MAIN > Commands out of safe range ID: # -1
adsec.R            |INF|      4101|2016-09-18 10:42:40.802699|             MAIN > Forces out of safe range   ID: # 22 #166 #167
adsec.R            |INF|      4102|2016-09-18 10:42:40.802768|             MAIN > SafeSkipFrames caused by forces out of range detected.
adsec.R            |INF|      4103|2016-09-18 10:42:40.802811|             MAIN > Recovery from any SafeSkipFrames started...
adsec.R            |INF|      4104|2016-09-18 10:42:40.802847|             MAIN > Getting info on actual mirror position...
adsec.R            |INF|      4105|2016-09-18 10:42:40.812553|             MAIN > Reconstructor filter cleaning...
adsec.R            |INF|      4106|2016-09-18 10:42:40.820351|             MAIN > Control current integrator cleaning up...
adsec.R            |INF|      4107|2016-09-18 10:42:40.844447|             MAIN > Null new delta command contribution
adsec.R            |INF|      4109|2016-09-18 10:42:40.847741|             MAIN > Restore old oversampling...
adsec.R            |INF|      4110|2016-09-18 10:42:40.855971|             MAIN > Done.

10:53 During Center Pupils routine on SX
  • Loop Closed, Gains 0
  • CCD39 goes to Not Live
  • WFSFrameCounter increments, so frames arriving
  • Finally Safe Skip and Loop Opens
  • CCD39 goes back to Live

11:24 While in closed loop, the AdSec Arbitrator died.
  • Restarted the
adsecarb.R         |INF|       520|2016-09-18 11:24:06.700962|   COMMANDHANDLER > FSM (status: AORunning) has received command 2029 (TTOffload)
adsecarb.R         |INF|       521|2016-09-18 11:24:06.706577|   COMMANDHANDLER > Command TTOffload (code 2029) successfully completed
adsecarb.R         |WAR|       522|2016-09-18 11:24:12.503435|             MAIN > Cannot read coils status ([AOVarException] thGetVar errorVariable: adamhousekeeper.R.COIL_STATUS (type INT_VARIABLE, len 1)
adsecarb.R         |WAR|       522|2016-09-18 11:24:12.503435|             MAIN >  . )
adsecarb.R         |ERR|       523|2016-09-18 11:24:12.503500|             MAIN > AbstractArbitrator::Run() Caught exception: [AOVarException] Error writing variable (code -3008) Operation allowed to variable owner only File RTDBvar.cpp line 496Variable: adsecarb.R.SAFESKIP_PERCENT (type REAL_VARIABLE, len 1)
adsecarb.R         |ERR|       523|2016-09-18 11:24:12.503500|             MAIN >  .
adsecarb.R         |ERR|       524|2016-09-18 11:24:12.503624|             MAIN > Error in AdSecArbitrator: [AOVarException] Error writing variable (code -3008) Operation allowed to variable owner only File RTDBvar.cpp line 496Variable: adsecarb.R.SAFESKIP_PERCENT (type REAL_VARIABLE, len 1)
adsecarb.R         |ERR|       524|2016-09-18 11:24:12.503624|             MAIN >  .
adsecarb.R         |INF|       525|2016-09-18 11:24:12.503657|      ADAM-MODBUS > AdamModbus: enabling tss...
adsecarb.R         |ERR|       526|2016-09-18 11:24:12.503768|      ADAM-MODBUS > [TcpReceiveException] Error receiving data (errno=0 ), socket closed. (code -15003) TcpConnection receive error File TcpConnection.cpp line 131
adsecarb.R         |ERR|       527|2016-09-18 11:24:12.503798|      ADAM-MODBUS > SetSingleCoil: failed to RecvForceSingleCoil
adsecarb.R         |INF|       528|2016-09-18 11:24:12.524383|             MAIN > Closing ICE server
adsecarb.R         |INF|       529|2016-09-18 11:24:12.525600|             MAIN > Communication with MsgD stopped

11:45 SX RunAO Bayside X timeout
pyarg              |ERR|      6387|2016-09-18 11:45:39.058963|             MAIN > Timeout error on baysidex (-5001) TIMEOUT_ERROR

11:45 Interrupt command button works on FLAO GUI even when not in Intervention Mode.
  • After SX RunAO failed, I click the DX Interrupt Button and the DX RunAO failed

11:48 DX Preset fail
Sun Sep 18 11:48:47.162 2016 right Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: Stop{FLAO}[609]
Sun Sep 18 11:48:47.172 2016 right Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: PresetAO{FLAO}[610] - [AOException] presetAO: FATAL - Reading shared buffer masterdiagnostic.R:OPTLOOPBUF (-5001) TIMEOUT_ERROR
Sun Sep 18 11:48:47.186 2016 right Preset command failed for LUCIFER mode ADAPTIVEACE_ACTIVE with guidestars, AO reference, Sound=presetFailed.wav

11:58 Collect Optical Loop + LUCI data
  • Rotator 269
  • Bin 1, 9.1 mag
  • Take dark from WFS Control GUI
  • Error during command calibratePISCESDark
  • CalibratePISCESDarkFrame: WARNING - RETRY: Error reading LUCI (o) NO_ERROR

12:18 Opt Loop + LUCI
  • NCPA ON
    • 121912 20
    • 122227 50
  • No NCPA
    • 122321 50 (401)
    • 122347 20 (402)
  • NCPA ON
    • 122434 20 (403)
    • 122453 50 (404)
  • No NCPA
    • 122528 50 (405)
    • 122621 20 (406)

-- %USERSIG{DougMiller - 2016-09-18}%
Topic revision: r5 - 09 Nov 2016, DougMiller
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback