Summary.

temperature 7 degrees celcius.

Starting with the pupil checks. It took too long to perform all the 4 pupils, a shift applied to B1,B2 and B3. B3 checked and it looks good. Also a close loop check at B3 with turbulence ON showed ot be stable.

Also a talk with Guido and Xianyu about the errors in the adsec and recover procedure that it should be implemented with a better informaiton from the error message. A topic to discuss and to find a possible solution to work with it.
  • updated issue track #8325

Details

  • Power On WFS.
  • Housekeeper process was down.
  • Ocam2k not live.
  • troubleshooting worked.Power cycle bcu2k and change binning.
WfsArbitrator _76149|INF| 95|2021-04-02 17:35:36.705497| WFS > Executing presetAO..

WfsArbitrator _76149|WAR| 111|2021-04-02 17:38:09.636924| WFS > Command returns retryable error : RETRY: No stars found in TV frame (-20001) WFSARB_STAR_NOTFOUND

WfsArbitrator _76149|INF| 112|2021-04-02 17:38:09.636985| COMMANDHANDLER > Command not executed (status RETRY) (errstr RETRY: No stars found in TV frame (-20001) WFSARB_STAR_NOTFOUND)

WfsArbitrator _76149|WAR| 113|2021-04-02 17:38:09.637012| MAIN > centerStar: WARNING - RETRY: No stars found in TV frame (-20001) WFSARB_STAR_NOTFOUND
  • recenter the star
  • found in a different hexapod position.
  • center star from FLAO gui
  • The system perform the Center pupil without problem, the next step Close loop the system goes in Loop Close but the ocam2k goes not live.
  • Checking the Mirr frame counter goes to 65536 and a count again.
  • At this point decided to start both WFS and adsec from zero. Clean Startup
  • email to aoops, guido and xianyu about power off and restart of the system.
mirr_framecounter_wrong_april2.png

AdSecArbitrator _146397|ERR| 9988|2021-04-02 18:12:11.484169| IDL > Command returns error : IDL Dust detected in the mirror
20210204 18:12:11.4850 Alert: Warning: recoverFailure failed or syntax error occured
AdSecArbitrator_146397|INF| 9989|2021-04-02 18:12:11.484567| COMMANDHANDLER > Command execution failed: Fsm forced to Panic
AdSecArbitrator_146397|ERR| 9990|2021-04-02 18:12:11.484686| MAIN > RecoverFailure: FATAL - IDL Dust detected in the mirror
20210204 18:12:30.6894 Alert: HighOrderOffload: HighOrderOffload vector ready
AdSecArbitrator_146397|INF| 9991|2021-04-02 18:12:47.363148| MAIN > Received cmd: RecoverFailure ()
AdSecArbitrator_146397|INF| 9992|2021-04-02 18:12:47.363231| COMMANDHANDLER > FSM (status: Panic) has received command 2002 (Recover failure)
AdSecArbitrator_146397|ERR| 9993|2021-04-02 18:12:47.363262| COMMANDHANDLER > adsecarb: Fsm is discarding command 2002 (Recover failure)...
AdSecArbitrator_146397|WAR| 9994|2021-04-02 18:12:47.363301| MAIN > RecoverFailure: WARNING - adsecarb:Illegal command for state Panic (Recover failure)
20210204 18:12:50.9495 Alert: HighOrderOffload: HighOrderOffload vector ready

troubleshoot..

V V V

WfsArbitrator _76149|INF| 230|2021-04-02 18:17:11.922345| WFS > Python command: WfsArbScripts.updateModulation(self, sensor = 'FLAOWFS', instr = 'LUCIFER', freq = 200)

WfsArbitrator _76149|INF| 231|2021-04-02 18:17:34.258654| WFS > Command returns success .

WfsArbitrator _76149|INF| 232|2021-04-02 18:17:34.258714| COMMANDHANDLER > Command PresetAO (code 2041) successfully completed

WfsArbitrator _76149|INF| 233|2021-04-02 18:17:58.025766| MAIN > Received cmd: centerStar()
  • start/adsec
  • wfs stop/start.
  • Call Guido and Xianju. Discussion about the failure mode, error message and a possible procedure to recover/poweroff/on after a Panic status in the Adsec Arb.
  • time to power on and cool down the ocam.
alignment for rotation test bin1.

BIN1 ROT

  • moving up clockwise rotaion the pupils are well in the middle.

WfsArbitrator _190796|INF| 357|2021-04-02 19:26:52.431744| OPTLOOPDIAG > Executing save...

WfsArbitrator _190796|INF| 358|2021-04-02 19:26:54.002562| WFS > Executing saveStatus

BIN1_APRIL2_POS.png

BIN2 ROT

  • moving up clockwise rotaion the pupils are well in the middle.

WfsArbitrator _190796|INF| 1558|2021-04-02 20:09:43.872596| WFS > Python command: WfsArbScripts.updateNcpa(self, sensor = 'FLAOWFS', instr = 'LUCIFER', rotAngle = 0, zero = 1)

WfsArbitrator _190796|INF| 1559|2021-04-02 20:09:43.884148| WFS > Executing presetAO...

WfsArbitrator _190796|INF| 1560|2021-04-02 20:09:43.884270| WFS > Python command: WfsArbScripts.presetAO(self, sensor = 'FLAOWFS', instr = 'LUCIFER', mode = 'ACE-AO', starxpos = -0, starypos = 0, telElevation = 0, derotAngle = 0, starMag = 13.6, starColor = 0, r0 = 0, badseeing = 0, v0 = 0)

WfsArbitrator _190796|INF| 1561|2021-04-02 20:09:43.885353| MAIN > LUT A

bin2_april2.png
  • My internet conneciton faster.. GUIs more responsive

BIN3 ROT

  • shutter 45
  • mag 16.3

bin3_april2.png

BIN4 ROT

  • shutter 41
  • mag 17.5
  • it was visible in the CC47.
WfsArbitrator _190796|INF| 3301|2021-04-02 21:25:36.284921| MAIN > LUT Applied: deg deg V V mm mm V V V V

WfsArbitrator _190796|WAR| 3302|2021-04-02 21:26:02.391598| WFS > Command returns retryable error : RETRY: No stars found in TV frame (-20001) WFSARB_STAR_NOTFOUND

WfsArbitrator _190796|INF| 3303|2021-04-02 21:26:02.391661| COMMANDHANDLER > Command not executed (status RETRY) (errstr RETRY: No stars found in TV frame (-20001) WFSARB_STAR_NOTFOUND)

WfsArbitrator _190796|WAR| 3304|2021-04-02 21:26:02.391699| MAIN > centerStar: WARNING - RETRY: No stars found in TV frame (-20001) WFSARB_STAR_NOTFOUND
  • cancel preset and sent again.
  • Star visible again in the CCD47
  • center star ok
bin4-april2.png

bin4_april2_puperr.png

PUPIL ROTATION GRAND SLAM

PUPROT_APRIL2_GRANSSLAM.png

BIN1 ROT SHIFT ONE PIXEL DOWN.

  • Executed the preset, but it showed an overilluminaiton triggered. Recover and abort the bin1 checked

WfsArbitrator _190796|INF| 3824|2021-04-02 22:27:15.636423| ADSEC > Executing openLoop...

WfsArbitrator _190796|INF| 3825|2021-04-02 22:27:15.636465| ADSEC > Requesting StopAo to AdSecArbitrator...

WfsArbitrator _190796|INF| 3826|2021-04-02 22:27:16.946739| COMMANDHANDLER > Command StopLoopAdsec (code 2033) successfully completed

WfsArbitrator _190796|INF| 3827|2021-04-02 22:27:49.305415| MAIN > Processing alert 4001 (IlluminationErr) from ocam2.L: OCAM

WfsArbitrator _190796|INF| 3828|2021-04-02 22:28:08.524083| WFS > Executing stopLoop...

WfsArbitrator _190796|INF| 3829|2021-04-02 22:28:08.524180| WFS > Python command: WfsArbS

WfsArbitrator _190796|INF| 3834|2021-04-02 22:28:19.563590| COMMANDHANDLER > FSM (status: AOSet) has received command 2041 (PresetAO)

WfsArbitrator _190796|INF| 3835|2021-04-02 22:28:19.563717| OPTLOOPDIAG > Executing enableAntiDrift(true)...

WfsArbitrator _190796|INF| 3836|2021-04-02 22:28:19.563829| WFS > Python command: WfsArbScripts.updateNcpa(self, sensor = 'FLAOWFS', instr = 'LUCIFER', rotAngle = 0, zero = 1)

WfsArbitrator _190796|INF| 3837|2021-04-02 22:28:19.574660| WFS > Executing presetAO...

WfsArbitrator _190796|INF| 3838|2021-04-02 22:28:19.574754| WFS > Python command: WfsArbScripts.presetAO(self, sensor = 'FLAOWFS', instr = 'LUCIFER', mode = 'ACE-AO', starxpos = -0, starypos = 0, telElevation = 0, derotAngle = 0, starMag = 10, starColor = 0, r0 = 0, badseeing = 0, v0 = 0)

WfsArbitrator _190796|INF| 3839|2021-04-02 22:28:19.575667| MAIN > LUT Applied: Rotang Ele X_LUT Y_LUT X_Enc Y_Enc X_Enc Y_Enc X_Tot Y_Tot

WfsArbitrator _190796|INF| 3840|2021-04-02 22:28:19.575720| MAIN > LUT Applied: deg deg V V mm mm V V V V

WfsArbitrator _190796|INF| 3841|2021-04-02 22:28:20.285533| MAIN > LUT Applied: Rotang Ele X_LUT Y_LUT X_Enc Y_Enc X_Enc Y_Enc X_Tot Y_Tot

WfsArbitrator _190796|INF| 3842|2021-04-02 22:28:20.285584| MAIN > LUT Applied: deg deg V V mm mm V V V V

WfsArbitrator _190796|INF| 3843|2021-04-02 22:28:20.290540| MAIN > LUT Applied: Rotang Ele X_LUT Y_LUT X_Enc Y_Enc X_Enc Y_Enc X_Tot Y_Tot

WfsArbitrator _190796|INF| 3844|2021-04-02 22:28:20.290579| MAIN > LUT Applied: deg deg V V mm mm V V V V

WfsArbitrator _190796|ERR| 3845|2021-04-02 22:28:45.964230| WFS > Command returns error : exception ::AODefs::AOException

WfsArbitrator _190796|ERR| 3845|2021-04-02 22:28:45.964230| WFS > . {

WfsArbitrator _190796|ERR| 3845|2021-04-02 22:28:45.964230| WFS > . code = -1

WfsArbitrator _190796|ERR| 3845|2021-04-02 22:28:45.964230| WFS > . what = [AOException] /home/AOeng/aoroot/calib/wfs/current/ocam2/LUTs/current/pixelGating.txt (code -1010) File open/read/write error File BCUlbt.cpp line 485

WfsArbitrator _190796|ERR| 3845|2021-04-02 22:28:45.964230| WFS > . }

WfsArbitrator _190796|INF| 3846|2021-04-02 22:28:45.964325| COMMANDHANDLER > Command execution failed: Fsm forced to Failure

WfsArbitrator _190796|ERR| 3847|2021-04-02 22:28:45.964423| MAIN > presetAO: FATAL - exception ::AODefs::AOException

WfsArbitrator _190796|ERR| 3847|2021-04-02 22:28:45.964423| MAIN > . {

WfsArbitrator _190796|ERR| 3847|2021-04-02 22:28:45.964423| MAIN > . code = -1

WfsArbitrator _190796|ERR| 3847|2021-04-02 22:28:45.964423| MAIN > . what = [AOException] /home/AOeng/aoroot/calib/wfs/current/ocam2/LUTs/current/pixelGating.txt (code -1010) File open/read/write error File BCUlbt.cpp line 485

WfsArbitrator _190796|ERR| 3847|2021-04-02 22:28:45.964423| MAIN > . }

WfsArbitrator _190796|ERR| 3860|2021-04-02 22:30:04.045165| WFS > Command returns error : exception ::AODefs::AOException

WfsArbitrator _190796|ERR| 3860|2021-04-02 22:30:04.045165| WFS > . {

WfsArbitrator _190796|ERR| 3860|2021-04-02 22:30:04.045165| WFS > . code = -1

WfsArbitrator _190796|ERR| 3860|2021-04-02 22:30:04.045165| WFS > . what = [AOException] /home/AOeng/aoroot/calib/wfs/current/ocam2/LUTs/current/pixelGating.txt (code -1010) File open/read/write error File BCUlbt.cpp line 485

WfsArbitrator _190796|ERR| 3860|2021-04-02 22:30:04.045165| WFS > . }

WfsArbitrator _190796|INF| 3861|2021-04-02 22:30:04.045250| COMMANDHANDLER > Command execution failed: Fsm forced to Failure

WfsArbitrator _190796|ERR| 3862|2021-04-02 22:30:04.045294| MAIN > presetAO: FATAL - exception ::AODefs::AOException

WfsArbitrator _190796|ERR| 3862|2021-04-02 22:30:04.045294| MAIN > . {

WfsArbitrator _190796|ERR| 3862|2021-04-02 22:30:04.045294| MAIN > . code = -1

WfsArbitrator _190796|ERR| 3862|2021-04-02 22:30:04.045294| MAIN > . what = [AOException] /home/AOeng/aoroot/calib/wfs/current/ocam2/LUTs/current/pixelGating.txt (code -1010) File open/read/write error File BCUlbt.cpp line 485

WfsArbitrator _190796|ERR| 3862|2021-04-02 22:30:04.045294| MAIN > . }

  • RESET ILLUMINATION PROTECTION.
  • OCAM LIVE
  • BCU2K OFF/ON
  • RESET BINNING.
  • THE NEXT PRESET IS STILL FAILING.
WfsArbitrator _190796|ERR| 3878|2021-04-02 22:33:52.724743| WFS > . what = [AOException] /home/AOeng/aoroot/calib/wfs/current/ocam2/LUTs/current/pixelGating.txt (code -1010) File open/read/write error File BCUlbt.cpp line 485

WfsArbitrator _190796|ERR| 3878|2021-04-02 22:33:52.724743| WFS > . }

WfsArbitrator _190796|INF| 3879|2021-04-02 22:33:52.724826| COMMANDHANDLER > Command execution failed: Fsm forced to Failure

WfsArbitrator _190796|ERR| 3880|2021-04-02 22:33:52.724873| MAIN > presetAO: FATAL - exception ::AODefs::AOException

WfsArbitrator _190796|ERR| 3880|2021-04-02 22:33:52.724873| MAIN > . {

WfsArbitrator _190796|ERR| 3880|2021-04-02 22:33:52.724873| MAIN > . code = -1

WfsArbitrator _190796|ERR| 3880|2021-04-02 22:33:52.724873| MAIN > . what = [AOException] /home/AOeng/aoroot/calib/wfs/current/ocam2/LUTs/current/pixelGating.txt (code -1010) File open/read/write error File BCUlbt.cpp line 485

WfsArbitrator _190796|ERR| 3880|2021-04-02 22:33:52.724873| MAIN > . }

WfsArbitrator _190796|INF| 3881|2021-04-02 22:35:55.549820| MAIN > Camera l
  • SET SAFE OCAM.
  • slope computer stop/start
  • bcu2k stop/start
  • reconfigure bin.
  • background taken
  • check bias, fine.
  • cool down

BIN3 ROT SHIFT

  • The worst point shows only a few saturation points and it is recoverd rot angle 280 to 240.
  • looking good.
WfsArbitrator _190796|INF| 4144|2021-04-02 23:11:26.457415| MAIN > Camera lens tracking: moving by x= -1.2 y= 3.0 (um)

WfsArbitrator _190796|INF| 4145|2021-04-02 23:11:28.094390| MAIN > Camera lens tracking: moving by x= 1.2 y= 12.1 (um)

WfsArbitrator _190796|INF| 4146|2021-04-02 23:11:29.757985| MAIN > Camera lens tracking: moving by x= -4.3 y= 24.3 (um)

WfsArbitrator _190796|INF| 4147|2021-04-02 23:11:31.489947| MAIN > Camera lens tracking: moving by x= -0.8 y= 8.7 (um)

WfsArbitrator _190796|INF| 4148|2021-04-02 23:11:33.104913| MAIN > Camera lens tracking: moving by x= -2.4 y= 5.4 (um)

WfsArbitrator _190796|INF| 4149|2021-04-02 23:11:34.741588| MAIN > Camera lens tracking: moving by x= -1.2 y= 3.0 (um)

bin3_apr3_shift.png

bin3_apr2_shift-err.png
  • done good correction.

bin3 close loop check s1.5

LOO

WfsArbitrator _190796|INF| 4306|2021-04-02 23:21:44.203712| WFS > Executing presetAO...

WfsArbitrator _190796|INF| 4307|2021-04-02 23:21:44.203890| WFS > Python command: WfsArbScripts.presetAO(self, sensor = 'FLAOWFS', instr = 'LUCIFER', mode = 'ACE-AO', starxpos = -0, starypos = 0, telElevation = 0, derotAngle = 0, starMag = 16.3, starColor = 0, r0 = 0, badseeing = 0, v0 = 0)

WfsArbitrator _190796|INF| 4308|2021-04-02 23:21:44.205114| MAIN > LUT

WfsArbitrator_190796|INF| 4380|2021-04-02 23:27:23.002731| WFS > Executing saveStatus...

WfsArbitrator_190796|INF| 4381|2021-04-02 23:27:23.002830| WFS > Python command: WfsArbScripts.saveStatus(self, sensor = 'FLAOWFS', instr = 'LUCIFER', outputFile = '/local/aomeas/adsec_data/20210402/Data_20210402_232721/wfs.fits')

  • gopt on
WfsArbitrator_190796|INF| 4389|2021-04-02 23:29:01.007207| WFS > Executing saveStatus...

WfsArbitrator_190796|INF| 4390|2021-04-02 23:29:01.007279| WFS > Python command: WfsArbScripts.saveStatus(self, sensor = 'FLAOWFS', instr = 'LUCIFER', outputFile = '/local/aomeas/adsec_data/20210402/Data_20210402_232859/wfs.fits')

WfsArbitrator_190796|INF| 4391

23:39 STOP

WfsArbitrator_190796|INF| 4414|2021-04-02 23:39:35.627686| WFS > Executing off...

WfsArbitrator_190796|INF| 4416|2021-04-02 23:39:52.547428| MAIN > Processing alert 4001 (IlluminationErr) from ocam2.L: OCAM

WfsArbitrator_190796|ERR| 4417|2021-04-02 23:40:21.180737| WFS > Command returns error : Cannot turn off OCAM2K. Reason: Timeout waiting for powerboard.L.OCAM2K_POWER.CUR - TIMEOUT_ERROR (-1000) GENERIC_SYS_ERROR

WfsArbitrator_190796|INF| 4418|2021-04-02 23:40:21.180823| COMMANDHANDLER > Command execution failed: Fsm forced to Failure

WfsArbitrator_190796|ERR| 4419|2021-04-02 23:40:21.180872| MAIN > Off: FATAL - Cannot turn off OCAM2K. Reason: Timeout waiting for powerboard.L.OCAM2K_POWER.CUR - TIMEOUT_ERROR (-1000) GENERIC_SYS_ERROR
  • it can be power off manually all the devices. but trying to power On and OFF.
WfsArbitrator_190796|INF| 4429|2021-04-02 23:46:51.498371| COMMANDHANDLER > FSM (status: AOSet) has received command 2004 (Operate)

WfsArbitrator_190796|INF| 4430|2021-04-02 23:46:51.498451| OPTLOOPDIAG > Executing enableAntiDrift(true)...

WfsArbitrator_190796|INF| 4431|2021-04-02 23:46:51.498677| WFS > Executing operate...

WfsArbitrator_190796|INF| 4432|2021-04-02 23:46:51.498736| WFS > Python command: WfsArbScripts.setOperating(self, sensor = 'FLAOWFS', instr = 'LUCIFER', config = 'WFS_complete_(with_ccd47)', boardSetup = '', opticalSetup = False)

WfsArbitrator_190796|INF| 4433|2021-04-02 23:47:12.899318| WFS > Python command: WfsArbScripts.updateModulation(self, sensor = 'FLAOWFS', instr = 'LUCIFER', freq = 2000)

WfsArbitrator_190796|INF| 4441|2021-04-02 23:52:56.651423| WFS > Python command: WfsArbScripts.setPowerOff(self, sensor = 'FLAOWFS', instr = 'LUCIFER')

WfsArbitrator_190796|INF| 4442|2021-04-02 23:53:46.123390| WFS > Command returns success .

WfsArbitrator_190796|INF| 4443|2021-04-02 23:53:46.123625| COMMANDHANDLER > Command Off (code 2005)

  • to investigate a better way for a power off when a failure occurs....>. The sequence of power off can be changed..???

-- JuanCarlosGuerra - 02 Apr 2021
I Attachment Action Size Date Who Comment
BIN1_APRIL2_POS.pngpng BIN1_APRIL2_POS.png manage 57 K 02 Apr 2021 - 20:04 JuanCarlosGuerra  
PUPROT_APRIL2_GRANSSLAM.pngpng PUPROT_APRIL2_GRANSSLAM.png manage 81 K 02 Apr 2021 - 23:03 JuanCarlosGuerra  
bin2_april2.pngpng bin2_april2.png manage 46 K 02 Apr 2021 - 20:50 JuanCarlosGuerra  
bin3_apr2_shift-err.pngpng bin3_apr2_shift-err.png manage 70 K 02 Apr 2021 - 23:28 JuanCarlosGuerra  
bin3_apr3_shift.pngpng bin3_apr3_shift.png manage 54 K 02 Apr 2021 - 23:24 JuanCarlosGuerra  
bin3_april2.pngpng bin3_april2.png manage 45 K 02 Apr 2021 - 21:30 JuanCarlosGuerra  
bin4-april2.pngpng bin4-april2.png manage 76 K 02 Apr 2021 - 22:11 JuanCarlosGuerra  
bin4_april2_puperr.pngpng bin4_april2_puperr.png manage 88 K 02 Apr 2021 - 22:13 JuanCarlosGuerra  
mirr_framecounter_wrong_april2.pngpng mirr_framecounter_wrong_april2.png manage 149 K 02 Apr 2021 - 20:41 JuanCarlosGuerra  
Topic revision: r2 - 03 Apr 2021, JuanCarlosGuerra
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback