Closed Dome DX reconstructor Checkout. 20180222

DX Closed Loop

07:10 Closed loop

07:26 While the loop was close the shell was RIPPED by housekeeping. (At least that's how I interpret the log below).
20182202 07:26:42.5120 Alert: HighOrderOffload: HighOrderOffload vector ready
AdSecArbitrator_37203|ERR|    123592|2018-02-22 07:26:52.595067|     ALERTHANDLER > Received an ERROR alert! Forcing Failure state.
AdSecArbitrator_37203|ERR|    123593|2018-02-22 07:26:52.595102|     ALERTHANDLER > Alert message: COILS DISABLED from housekeeper.R
20182202 07:26:52.5957 Alert: Error: COILS DISABLED
AdSecArbitrator_37203|INF|    123594|2018-02-22 07:26:53.937659|     ALERTHANDLER > Received from masterdiagnostic.R a DumpSaved alert with filename /local/aolog/2018/02/22/ADAPTIVE-SECONDARY_00001519284413_20180222_072653.bin
AdSecArbitrator_37203|INF|    123595|2018-02-22 07:26:53.937711|     ALERTHANDLER > Processdump is skipped
20182202 07:26:53.9384 Alert: DumpSaved: DumpSaved vector ready
AdSecArbitrator_37203|ERR|    123596|2018-02-22 07:26:53.966207|     ALERTHANDLER > Received an ERROR alert! Forcing Failure state.
AdSecArbitrator_37203|ERR|    123597|2018-02-22 07:26:53.966228|     ALERTHANDLER > Alert message: COILS DISABLED from housekeeper.R
20182202 07:26:53.9666 Alert: Error: COILS DISABLED
20182202 07:27:04.3962 Alert: HighOrderOffload: HighOrderOffload vector ready

07:30 Try to recover
  • Stop/Start AdSecArb
  • Loop Closed State with skip frame
  • Open Loop on WFS Control GUI.
  • Now AOSet State, but position is 24-39 um, not ~60 um
  • Rest on AdSec Control GUI
  • SetFlatAO: OK
  • Load recons, set focal station, etc
  • Close Loop on WFS Control GUI: OK

07:44 HouseKeeping disable coils again
AdSecArbitrator_122064|ERR|       226|2018-02-22 07:44:22.055357|    ALERTHANDLER > Received an ERROR alert! Forcing Failure state.
AdSecArbitrator_122064|ERR|       227|2018-02-22 07:44:22.055409|    ALERTHANDLER > Alert message: COILS DISABLED from housekeeper.R.20182202 07:44:22.0563 Alert: Error: COILS DISABLED
AdSecArbitrator_122064|INF|       228|2018-02-22 07:44:23.405914|    ALERTHANDLER > Received from masterdiagnostic.R a DumpSaved alert with filename /local/aolog/2018/02/22/ADAPTIVE-SECONDARY_00001519285463_20180222_074423.bin
AdSecArbitrator_122064|INF|       229|2018-02-22 07:44:23.405940|    ALERTHANDLER > Processdump is skipped
20182202 07:44:23.4066 Alert: DumpSaved: DumpSaved vector ready
AdSecArbitrator_122064|WAR|       230|2018-02-22 07:44:23.946062|            MAIN > Skip frame detected!

07:46 State of DX AdSec after HouseKeeping disable coils
  • After the RIP, the positions are from 24 to 39 um and there are 15 actuators with -0.8 N of force. Note, these are all actuators that are in the no-pos list. This may be an artifact of the disable coil and not real. The position from 24-39 um is strange. As seen in the next image, the usually rested position range is from 3 to 34 um. May this is just due to the signal is still at full wave and after the Rest the signal is 1/4th wave
    After_Rip.jpg
  • After the disable coil, the AdSec is in a failed state. stop/start the AdSec Arbitrator puts the AdSec back into Closed Loop state. The Loop is opened from the WF Control GUI and the AdSec goes to AOSet State (with positions from 24-39 um). The Rest button is then pushed and the position go from 334 um (usual rest position range)
    After_Rest.jpg

07:50 Movies showing the state of the AdSec after the HouseKeeping disabled coils
  • The AO loop is still closed and deltas are being sent from the WFS to the AdSec. Note the NumCrcErr is increasing, PendingSkipFrameCnt is increasing and WFSFrameCounter is changing but not in a steadily increasing manner.
  • After_Rip_Loop_Closed.mov: HouseKeeping disable coils
  • Here with Loop is open and the actuator positions are from 24-39 um. Half way thought the movie the shell is Rested and the position range goes from 3-34 um
  • After_Rip_to_Rest.mov: HouseKeeping disable coils

08:00 Decide to do a full adsc_stop/start
  • Stop WFSArbitrator so it will not be talking to AdSec when starting up
  • adsc_stop/start
  • Power On, Load Programs, SetFlatAO all OK
  • Start WFSArbitrator: troubles

08:12 Start wfsarbitrator process
  • Failure State
  • Msgd and several processed dead
[AOeng@flao-dxwfs UAO]$ date
Thu Feb 22 08:14:27 UTC 2018
AOeng@flao-dxwfs UAO]$ w_check
Configuration directory: /home/aoeng/aoroot/conf/wfs/current/processConf

msgdrtdb             NOT running
mirrorctrl           running but NOT connected to Msgd
pinger               running but NOT connected to Msgd
masterdiagnostic     running but NOT connected to Msgd
slopecompctrl        running but NOT connected to Msgd
techviewctrl         running but NOT connected to Msgd
powerboard           running but NOT connected to Msgd
picctrl              running but NOT connected to Msgd
ccd39                running but NOT connected to Msgd
ccd47                running but NOT connected to Msgd
ttctrl               NOT running
filterwheel1         running but NOT connected to Msgd
filterwheel2         running but NOT connected to Msgd
adc1                 running but NOT connected to Msgd
adc2                 running but NOT connected to Msgd
rerotator            running but NOT connected to Msgd
mercury              running but NOT connected to Msgd
cubestage            running but NOT connected to Msgd
baysidex             running but NOT connected to Msgd
baysidey             running but NOT connected to Msgd
baysidez             running but NOT connected to Msgd
cameralensctrl       NOT running
housekeeperWFS       NOT running
optloopdiag          running but NOT connected to Msgd
od_gopt              running but NOT connected to Msgd
truesense            running but NOT connected to Msgd
wfsarb               running but NOT connected to Msgd
dbfiller             running but NOT connected to Msgd
gopt                 running but NOT connected to Msgd
No AdOpt processes are running

08:20 Restart WFS
  • perform w_stop
  • w_start
  • start wfsarb (Operating State) and HW GUI (all power listed as off)
    • however, I could move the bayside stages

08:30 Clicked "OFF" on WFS Control GUI: success
  • w_stop/start
  • startup WUnit as usual. success

12:33 Start TT data
  • bin 2, 200 Hz
  • 13.4 mag
  • light 89
  • clear, HKSpec filters
  • 90 elevation

12:34 TT LUT On, bayside stages off, 90 elevation
  • 450, (.26, -0.7), 4, 5, 6, 123438, 123503, 123525
  • 420, (0.26, -0.7) open loop

13:54 TT LUT On, bayside stages off, 90 elevation
  • 420, (0.05, -0.26) 9, 10, 11, 125444, 124606, 124629
  • 390, (0.15, 0.04), 12, 13, 14, 124724, 124746, 124808
  • 360, (0.35, 0.01), 15, 16, 17, 124903, 124926 (no HO1,2), 124948
  • 330, (0.22, 0.01), 18, 19, 20, 125044, 125106, 125128
  • 300, (0.12, -0.10), 21, 22, 23, 125224, 125345, 125307
  • 270, (-0.34, -0.59), 24, 25, 26, 125403, 125425, 125447
  • 240, (-0.24, -0.74), 27, 28, 29, 125542, 125604, 125627
  • 210, (-0.11, -1.05), 30, 31, 32, 125722, 125744, 125806
  • 180, (0.11, -1.28), 33, 34, 35, 125900, 125923, 125945
  • 150, (0.22, -1.40), 36, 37, 38, 130043, 130105, 130129
  • 120, (0.32, -1.25), 39, 40, 41, 130224, 130245, 130308
  • 90, (0.42, -0.74), 42, 43, 44, 130404, 130426, 130448
  • 60, (0.45, -0.47), 45, 46, 47, 130544, 130605, 130628
  • stop script

13:07 TT LUT Off, bayside stages On
  • 30, 48, 49, 50, 131239, 131301, 131323
  • 60, 51, 52, 53, 131419, 131441, 131503
  • 90, 54, 55, 56, 131559, 131622, 131645
  • 120, 57, 58, 59, 131741, 131803, 131826
  • 150, 60, 61, 62, 131922, 131944, 132007
  • 180, 63, 64, 65, 132104, 132127, 132149
  • 210, 66, 67, 68, 132246, 132309, 132331
  • 240, 69, 70, 71, 132428, 132450, 132512
  • 270, 72, 73, 74, 132610, 132633, 132655
  • 300, 75, 76, 77, 132749, 132812, 132835
  • 330, 78, 79, 80, 132931, 132953, 133015
  • 360, 81, 82, 83, 133113, 133136, 133158
  • 390, 84, 85, 86, 133256, 133320, 133342
  • 420, 87, 88, 89, 133439, 133500, 133523
  • 450, 90, 91, 92, 133625, 133647, 133710

13:39 LUCI2 background (light off) 93, 94 133931, 134015

13:41 LUCI2 dark (blind/blind) 95, 96, 134246, 134329

Check SX Focus on LUCI1

-- %USERSIG{DougMiller - 2018-02-22}%
I Attachment Action Size Date Who Comment
After_Rest.jpgjpg After_Rest.jpg manage 160 K 22 Feb 2018 - 09:02 DougMiller HouseKeeping disable coils
After_Rip.jpgjpg After_Rip.jpg manage 168 K 22 Feb 2018 - 09:02 DougMiller HouseKeeping disable coils
After_Rip_Loop_Closed.movmov After_Rip_Loop_Closed.mov manage 10 MB 22 Feb 2018 - 09:02 DougMiller HouseKeeping disable coils
After_Rip_to_Rest.movmov After_Rip_to_Rest.mov manage 9 MB 22 Feb 2018 - 09:02 DougMiller HouseKeeping disable coils
Topic revision: r3 - 22 Feb 2018, DougMiller
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback