SX CentOS 6 On-Sky tests

Summary

  • Spent about 4 hours On-Sky
  • Lost 2 hours to clouds and humidity
  • Seeing was 0.5-0.8
  • Did not get the loop closed on neither the DX nor SX side.
  • Both SX and DX
    • Closed loop was very unstable against Safe Skip
      • Optimize Gain values seemed a bit higher than the values found by CentOS version, but not too much
      • Get Safe Skips even during low order closed loop to center the pupil.
      • Copied autogain.conf from flao@wfssx to both CentOS 6 and 7. Lowered gains in auto gain, but still a lot of safe skip
      • Rerotator was on and offset values were correct.
      • The initial "All" gains during Optimize gain seem very high: 1.7, 0.8, 0.8. If these values are near overdriving then during ranging of other gains it may overdrive the loop. Can we change these "All" gains
      • Alfio, any ideas?
  • SX
    • WFS Arbitrator not connected to msgd
      • Happened every time the the loop opened due to Safe Skip, whether during pupil centering or Optimize Gain
      • Did not happened on DX side
      • Alfio, ideas?
    • CCD 39 go "Not Live". Stopping starting Slope Computer fix it.
  • DX
    • CCD 39 go "Not Live" restarting CCD39 controller, powering little joe and restarting slope computer did not help. w_stop/start
    • WFS Arb die on w_start and have to w_stop/start again

Details

02:30 Stop SX CentOS 5 AdSec software
  • login at tcs@tcs1 and change link aos.conf -> aos.conf.uao
  • Stop and Start AOSL and R
  • No problems

03:03 DX AdSec Control GUI died

03:05 AdSec failure while rested
adsec.L            |INF|       694|2016-09-15 03:05:16.641246|             MAIN > No actuators were removed on the fly
adsec.L            |INF|       695|2016-09-15 03:05:16.644543|             MAIN > Distance threshold check analysis result:
adsec.L            |INF|       696|2016-09-15 03:05:16.644611|             MAIN > Found bad capacitive sensors on CH ID: 137
adsec.L            |INF|       698|2016-09-15 03:05:16.651753|             MAIN > Temperature check passed with no issues found.
adsec.L            |INF|       699|2016-09-15 03:05:16.651860|             MAIN > -- PROCESS DUMP END --
adsec.L            |INF|       700|2016-09-15 03:05:16.672621|             MAIN > Try to decrease reference wave to 1/8 of std amplitude
  • Power cycle and Set no problems

03:50 DX PresetAO failed
Thu Sep 15 03:50:22.869 2016 right Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: PresetAO{FLAO}[8] - [AOException] presetAO: FATAL - IDL variable undefined 'tn'
Thu Sep 15 03:50:22.883 2016 right Preset command failed for LUCIFER mode ADAPTIVEACE_ACTIVE with guidestars, AO reference, Sound=presetFailed.wav

03:59 DX PresetAO failed while trying to create disturbance file with IDL
pyarg              |INF|       234|2016-09-15 03:59:32.441740|             MAIN > selectDisturbance(): searching for disturbance file, please wait...
pyarg              |ERR|       235|2016-09-15 03:59:32.504745|             MAIN > IDL variable undefined 'tn'
  • determined that KL_v21 files from AdSec machine were not available. sinusmode_generate() IDL routines failed. No proper error message.
  • Xianyu mounted the needed directory on WFS from AdSec

04:12 SX RunAO failed between Check Pupils and Check flux
  • Shell Rested
  • CCD39 ended up Not Live
adsecarb.L         |INF|       153|2016-09-15 04:12:06.445135|             MAIN > Received cmd: StopAo(1)
adsec.L            |INF|      1164|2016-09-15 04:12:06.445670|             MAIN > fsm_stop_ao start
adsec.L            |INF|      1165|2016-09-15 04:12:06.445732|             MAIN > fsm_skip_recovery FORCE set
adsecarb.L         |INF|       154|2016-09-15 04:12:06.445239|   COMMANDHANDLER > FSM (status: AORunning) has received command 2015 (Stop Ao)
adsecarb.L         |INF|       155|2016-09-15 04:12:06.445294|    SERIALIZATION > StopAoImpl(): restoreShape = 1
adsec.L            |INF|      1166|2016-09-15 04:12:06.660828|             MAIN > Null Gain application and start sending null slope...
adsec.L            |INF|      1167|2016-09-15 04:12:06.785264|             MAIN > Cleaned last slopes command.

pyarg              |DEB|       317|2016-09-15 04:12:19.449852|             MAIN > selectSlopenull(): starting
pyarg              |INF|       318|2016-09-15 04:12:19.450922|             MAIN > selectSlopenull(): selected slopenull zero.fits
pyarg              |INF|       319|2016-09-15 04:12:19.450949|             MAIN > calibHODarkInternal(): starting measure, 1000 frames
pyarg              |INF|       320|2016-09-15 04:12:27.263889|             MAIN > ccd.equalize_quadrants(): Current settings: bin 1, readout speed 2500 kpix/sec
pyarg              |INF|       321|2016-09-15 04:12:27.264159|             MAIN > ccd.equalize_quadrants(): Starting values: (672, 709, 690, 695)
pyarg              |ERR|       322|2016-09-15 04:12:29.737328|             MAIN > Reading shared buffer masterdiagnostic.L:OPTLOOPBUF (-5001) TIMEOUT_ERRO

04:22 Can not get CCD39 to restart. Shows not live
wfsarb.L           |INF|       185|2016-09-15 04:22:14.752114|              WFS > Python command: WfsArbScripts.modifyAO(self, sensor = 'FLAOWFS', instr = 'LUCIFER', freq = 990, binning = 1, TTmod = 0, Fw1Pos = -1, Fw2Pos = -1, decimation = 1, checkCameralens = 'False')
wfsarb.L           |ERR|       186|2016-09-15 04:22:17.457340|              WFS > Command returns error : Reading shared buffer masterdiagnostic.L:OPTLOOPBUF (-5001) TIMEOUT_ERROR
wfsarb.L           |INF|       187|2016-09-15 04:22:17.457370|   COMMANDHANDLER > Command execution failed: Fsm forced to Failure
  • Stopped and Started CCD39 Controller. Still fails
  • Power off/on LittleJoe CCD 39. Still fails
  • Stop and start Slope Computer. Click 1x1. OK

06:16 Preset to AO53
  • 8.8 mag

06:21 RunAO failed
  • safe skip
  • Loop open
  • wfsarb running but not connected to msgd
    • Can not restart from GUI
    • Kill process
    • Status is Operating
  • adsecarb die
    • Restart and State is AORunning
  • Open loop on WFS Control GUI.
adsecarb.L         |WAR|       323|2016-09-15 06:21:22.660450|             MAIN > Skip frame detected!
adsecarb.L         |WAR|       324|2016-09-15 06:21:23.663052|             MAIN > Skip frame detected!
adsecarb.L         |INF|       325|2016-09-15 06:21:23.736427|   COMMANDHANDLER > FSM (status: AORunning) has received command 2024 (Save status)
adsecarb.L         |INF|       326|2016-09-15 06:21:23.993476|             MAIN > Received cmd: RunAo(1,990,860)
adsecarb.L         |INF|       327|2016-09-15 06:21:23.993553|   COMMANDHANDLER > FSM (status: AORunning) has received command 2014 (Run Ao)
adsecarb.L         |ERR|       328|2016-09-15 06:21:23.993588|   COMMANDHANDLER > adsecarb: Fsm is discarding command 2014 (Run Ao)...
adsecarb.L         |WAR|       329|2016-09-15 06:21:23.993626|             MAIN > RunAo: WARNING - adsecarb:Illegal command for state AORunning (Run Ao)
adsecarb.L         |INF|       330|2016-09-15 06:21:23.994053|             MAIN > Received cmd: RunAo(1,990,860)
adsecarb.L         |INF|       331|2016-09-15 06:21:23.994140|   COMMANDHANDLER > FSM (status: AORunning) has received command 2014 (Run Ao)
adsecarb.L         |ERR|       332|2016-09-15 06:21:23.994188|   COMMANDHANDLER > adsecarb: Fsm is discarding command 2014 (Run Ao)...
adsecarb.L         |WAR|       333|2016-09-15 06:21:23.994236|             MAIN > RunAo: WARNING - adsecarb:Illegal command for state AORunning (Run Ao)

wfsarb.L           |INF|       283|2016-09-15 06:21:04.548908|   COMMANDHANDLER > FSM (status: LoopClosed) has received command 2020 (Save status)
wfsarb.L           |ERR|       284|2016-09-15 06:21:04.548963|   COMMANDHANDLER > wfsarb: Fsm is discarding command 2020 (Save status)...
wfsarb.L           |INF|       285|2016-09-15 06:21:10.917669|   COMMANDHANDLER > FSM (status: LoopClosed) has received command 2020 (Save status)
wfsarb.L           |ERR|       286|2016-09-15 06:21:10.917709|   COMMANDHANDLER > wfsarb: Fsm is discarding command 2020 (Save status)...
wfsarb.L           |INF|       287|2016-09-15 06:21:17.287559|   COMMANDHANDLER > FSM (status: LoopClosed) has received command 2020 (Save status)
wfsarb.L           |ERR|       288|2016-09-15 06:21:17.287587|   COMMANDHANDLER > wfsarb: Fsm is discarding command 2020 (Save status)...
wfsarb.L           |INF|       289|2016-09-15 06:21:23.734692|   COMMANDHANDLER > FSM (status: LoopClosed) has received command 2020 (Save status)
wfsarb.L           |ERR|       290|2016-09-15 06:21:23.734721|   COMMANDHANDLER > wfsarb: Fsm is discarding command 2020 (Save status)...
wfsarb.L           |ERR|       291|2016-09-15 06:21:23.737587|              WFS > Command returns error : Safe skip active - cannot proceed (-10045) IDL_SAFE_SKIP_ACTIVE

06:41 Change /home/aoeng/aoroot/calib/wfs/current/fov.txt
  • Change initial X Stage position (xc) to -77 from -73.5
  • Restart wfsarb

06:42 WFS Arbitrator not connected to msgd during autogain
pyarg              |ERR|       191|2016-09-15 06:42:22.730342|             MAIN > Safe skip active - cannot proceed (-10045) IDL_SAFE_SKIP_ACTIVE

06:43 UAO GUI report actuator 469
  • Clicked Remove
  • OK

06:51 WFS Arbitrator not connected to msgd
  • happened at the end of Optimize gain
  • Loop gains seem a little high, but get a lot of safe skips during optimize gain. In CentOS 5 version we hardly ever get safe skip.

07:13 WFS Arbitrator not connected to msgd
wfsarb.L           |ERR|       164|2016-09-15 07:13:00.231737|   COMMANDHANDLER > wfsarb: Fsm is discarding command 2020 (Save status)...
wfsarb.L           |INF|       165|2016-09-15 07:13:02.329398|             MAIN > AdSec is skipping frames, camera lens not moving
wfsarb.L           |INF|       165|2016-09-15 07:13:02.329398|             MAIN >  . 
wfsarb.L           |ERR|       166|2016-09-15 07:13:03.682550|              WFS > Command returns error : Safe skip active - cannot proceed (-10045) IDL_SAFE_SKIP_ACTIVE

07:30 Modified RotatorOffsetBin2 from 225.4 to 258.9 in
  • /home/aoeng/aoroot/conf/wfs/current/processConf/wfsarb.conf

07:48 Preset to AO125
  • 13 mag measured
  • bin 2

07:52 RunAO fail
  • safe skip during pupil centering. Why, gains are low
  • Loop open
  • WFS arb not connected to msgd

07:58 DX Safe skip during optimize gain
adsecarb.R         |WAR|       666|2016-09-15 07:58:41.479224|             MAIN > Skip frame detected!
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN > Cannot dispatch alert: Outgoing.cpp:493: Ice::UnknownException:
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  . unknown exception:
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  . Traceback (most recent call last):
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  .   File "aoarbitrator.py", line 184, in alert
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  .     self._loop._fsm.SkfAlert()
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 597, in trans_sm
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  .     getattr(self.getState(), attrib)(self, *arglist)
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 78, in SkfAlert
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  .     self.Default(fsm)
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 89, in Default
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  .     raise statemap.TransitionUndefinedException(msg)
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  . TransitionUndefinedException:
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  . 	State: AOArbitrator.Ready
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  . 	Transition: SkfAlert
adsecarb.R         |WAR|       667|2016-09-15 07:58:41.483668|             MAIN >  .

-- %USERSIG{DougMiller - 2016-09-15}%
Topic revision: r2 - 15 Sep 2016, DougMiller
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback