LUCI/AO commissioning

01:35 Preset, RunAO
  • Interrupted the RunAO becasue the 4 Pyramid pupils were vignetted (looked like a vertical eye)
  • Filter Wheel 1 was not positioned correctly.
  • Home FW1 from HW GUI
  • FW homed when system was power up at 01:06
  • I do not see any errors in filterwheel1.L.log
  • I do not see any information about homing, at 01:06 or at 01:38

01:42 Preset and RunAO success in Bin 2
  • testing AFC
  • K band
  • 5 minute image integration with AFC ON
  • Jochen see elongation in N30 images

02:06 IIF Error

Mon Oct 24 02:06:09.655 2016 left Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: PresetAO{FLAO}[900] - [AOException] presetAO: FATAL - [Errno 4] Interrupted system call

02:06 WFS Arbitrator Error

pyarg              |INF|       631|2016-10-24 02:05:55.943719|             MAIN > Setting tiptilt to amp= 0.00, freq= 0.00, offx= 0.00, offy= 0.00
pyarg              |ERR|       632|2016-10-24 02:05:55.965035|             MAIN > [Errno 4] Interrupted system call
  • From pyarg log:
pyarg              |INF|       626|2016-10-24 02:05:55.363642|             MAIN > Setting tiptilt to amp= 0.00, freq= 0.00, offx= 0.00, offy= 0.00
pyarg              |INF|       627|2016-10-24 02:05:55.363822|             MAIN > Setting tiptilt to amp= 0.00, freq= 0.00, offx= 0.00, offy= 0.00
pyarg              |DEB|       628|2016-10-24 02:05:55.746273|             MAIN > ('selected program 2x2, 800 kpix/sec',)
pyarg              |DEB|       629|2016-10-24 02:05:55.746305|             MAIN > ('Delay base 809, delay inc 5, repetitions 853',)
pyarg              |DEB|       630|2016-10-24 02:05:55.746335|             MAIN > ('Real framerate 197.079285 Hz',)
pyarg              |INF|       631|2016-10-24 02:05:55.943719|             MAIN > Setting tiptilt to amp= 0.00, freq= 0.00, offx= 0.00, offy= 0.00
pyarg              |ERR|       632|2016-10-24 02:05:55.965035|             MAIN > [Errno 4] Interrupted system call

Submitted IT #6218

02:08 Clicked on PresetAO on AOS Command

Mon Oct 24 02:08:11.163 2016 left Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: Busy executing PresetAO{FLAO}

02:08 Seems that the LUCI scheduler is confused. Jochen reset

02:10 Preset and RunAO arrive.
  • I interrupted the RunAO to save time because Jochen will resend preset
  • Jochen cannot access LUCI machine: Opps, human error

02:16 Instrument manager failed when script is executed
  • Preset is running
  • Script is dead
  • Now RunAO is executing

02:17 RunAO failed

Mon Oct 24 02:17:33.929 2016 left RunAO alert for LUCIFER: AcquireRef{FLAO}[907] - Command interrupted
  • Had I doubled click on interrupt a couple minutes ago? I don't think so and it should have

02:20 AFC OFF
  • 1 minute integrations
  • 4.5 pixels, look round
  • K band

03:17 AFC ON
  • 3x5 minute integrations
  • H band

04:24 Errors start in AdSec Arbitrator
  • First, edge of shell started to "roll under"
  • Skip frame counter at 90%
  • PauseAO received
  • While in Pause State had a few more "Skip frame detected" then ICE:UnknownException error
adsecarb.L         |WAR|       989|2016-10-24 04:24:34.941504|             MAIN > Skip frame detected!
adsecarb.L         |INF|       990|2016-10-24 04:24:35.291457|             MAIN > Received cmd: Offloading(0)
adsecarb.L         |INF|       991|2016-10-24 04:24:35.291529|   COMMANDHANDLER > FSM (status: AORunning) has received command 2029 (TTOffload)
adsecarb.L         |INF|       992|2016-10-24 04:24:35.477205|   COMMANDHANDLER > Command TTOffload (code 2029) successfully completed
adsecarb.L         |INF|       993|2016-10-24 04:24:35.477458|             MAIN > Received cmd: PauseAo()
adsecarb.L         |INF|       994|2016-10-24 04:24:35.477509|   COMMANDHANDLER > FSM (status: AORunning) has received command 2027 (Pause Ao)
adsecarb.L         |INF|       995|2016-10-24 04:24:35.481302|   COMMANDHANDLER > Command Pause Ao (code 2027) successfully completed
adsecarb.L         |WAR|       996|2016-10-24 04:24:36.312587|             MAIN > Skip frame detected!
adsecarb.L         |WAR|       997|2016-10-24 04:24:37.363428|             MAIN > Skip frame detected!
adsecarb.L         |WAR|       998|2016-10-24 04:24:38.441579|             MAIN > Skip frame detected!
adsecarb.L         |WAR|       999|2016-10-24 04:24:39.484294|             MAIN > Skip frame detected!
adsecarb.L         |WAR|      1000|2016-10-24 04:24:40.554713|             MAIN > Skip frame detected!
adsecarb.L         |WAR|      1001|2016-10-24 04:24:41.558092|             MAIN > Skip frame detected!
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN > Cannot dispatch alert: Outgoing.cpp:493: Ice::UnknownException:
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  . unknown exception:
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  . Traceback (most recent call last):
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  .   File "aoarbitrator.py", line 184, in alert
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  .     self._loop._fsm.SkfAlert()
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 601, in trans_sm
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  .     getattr(self.getState(), attrib)(self, *arglist)
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 78, in SkfAlert
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  .     self.Default(fsm)
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  .   File "/home/aoeng/aoroot/lib/python/AOArb_sm.py", line 89, in Default
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  .     raise statemap.TransitionUndefinedException(msg)
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  . TransitionUndefinedException: 
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  . 	State: AOArbitrator.LoopSuspended
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  . 	Transition: SkfAlert
adsecarb.L         |WAR|      1002|2016-10-24 04:24:41.560248|             MAIN >  . 
adsecarb.L         |WAR|      1003|2016-10-24 04:24:42.796786|             MAIN > Skip frame detected!
  • Failure was sent to LUCI
  • AO System left in Pause state
  • I sent a ResumeAO from the AOS Command GUI
  • Loop Resumed, but no corrections applied because still skip frame
  • This error repeated ever second or so until a StopAO was finally received (Skip frame counter went to 100%)
  • IssueTrak # 6219 submitted

04:46 Test NCPA ON -vs- OFF for bin 2. J band
  • elevation 37
  • Guider = 1.0, WFS = 0.98
  • Gopt Gain = 0.17, very stable +-0.1
  • Loop gains originally 2.55, 2.0, 1.5
  • off luci # 67
  • on 68
  • off 69
  • on 70
  • off 71
  • on 72

05:00 Hand over to ARGOS

  • 2 bayside X stage timeouts in 67 offsets requested. Routine to retry successfully retry, no errors to LUCI
  • no bayside Y stage timeouts

-- %USERSIG{DougMiller - 2016-10-24}%

Comments

 
Topic revision: r1 - 24 Oct 2016, DougMiller
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback