CentOS 6.7 tests

04:59 DX Load Program failed
adsec.R            |INF|       284|2016-09-12 04:59:56.071399|             MAIN > fsm_reset() procedure done.
Last message repeated
adsecarb.R         |ERR|       201|2016-09-12 04:59:56.074935|              IDL > Command returns error : IDL crate id from power backplane error
adsecarb.R         |INF|       202|2016-09-12 04:59:56.075218|   COMMANDHANDLER > Command execution failed: Fsm forced to Failure
20161209 04:59:56.0767 Alert: Warning: loadProgram failed or syntax error occured

4:49 SX Load Program Failed
adsec.L            |INF|       284|2016-09-12 04:59:51.696171|             MAIN > fsm_reset() procedure done.
Last message repeated
adsecarb.L         |ERR|       110|2016-09-12 04:59:51.699394|              IDL > Command returns error : IDL crate id from power backplane error
20161209 04:59:51.7002 Alert: Warning: loadProgram failed or syntax error occured
adsecarb.L         |INF|       111|2016-09-12 04:59:51.699829|   COMMANDHANDLER > Command execution failed: Fsm forced to Failure

05:07 DX Power on success but error on:
adsec.R            |INF|       199|2016-09-12 05:07:57.485147|             MAIN > fsm_power_on() command done.
adsecarb.R         |INF|       105|2016-09-12 05:07:57.485721|   COMMANDHANDLER > Command Power on (code 2004) successfully completed
adsec.R            |INF|       200|2016-09-12 05:07:57.487631|             MAIN > Setting overcurrent protection for 76.49 [deg]
adsecarb.R         |INF|       106|2016-09-12 05:07:57.487344|             MAIN > Setting piston current for elevation 76.489244 degrees
adsecarb.R         |WAR|       107|2016-09-12 05:07:57.495053|              IDL > Syntax error found. Catching...
adsecarb.R         |WAR|       108|2016-09-12 05:07:57.495425|              IDL > Syntax error successfully catched

idlctrl.R |INF| 133|2016-09-12 05:07:57.487455| MAIN > Executing command: errcode= setPistonCurrent( 76.4892) & print, errcode 
idlctrl.R |INF| 134|2016-09-12 05:07:57.487527| MAIN > StartCmd 
idlctrl.R |INF| 135|2016-09-12 05:07:57.487632| MAIN > Setting overcurrent protection for 76.49 [deg] 
idlctrl.R |INF| 136|2016-09-12 05:07:57.494891| MAIN > EndCmd 
idlctrl.R |INF| 137|2016-09-12 05:07:57.494935| MAIN > % Attempt to subscript PBGAINMATRIX with is out 
idlctrl.R |INF| 137|2016-09-12 05:07:57.494935| MAIN > . of range. 
idlctrl.R |INF| 137|2016-09-12 05:07:57.494935| MAIN > . % Error occurred at: SET_CURR_LIMITS 66 
idlctrl.R |INF| 137|2016-09-12 05:07:57.494935| MAIN > . /home/aoeng/aoroot/idl/adsec_lib/adsec/level2/set_curr_limits.pro 
idlctrl.R |INF| 137|2016-09-12 05:07:57.494935| MAIN > . % SETPISTONCURRENT 18 
idlctrl.R |INF| 137|2016-09-12 05:07:57.494935| MAIN > . /home/aoeng/aoroot/idl/adsec_lib/adsec/level2/setpistoncurrent.pro 
idlctrl.R |INF| 137|2016-09-12 05:07:57.494935| MAIN > . % $MAIN$ 
idlctrl.R |INF| 137|2016-09-12 05:07:57.494935| MAIN > . % Execution halted at: $MAIN$ 
idlctrl.R |INF| 137|2016-09-12 05:07:57.494935| MAIN > . % Program caused arithmetic error: Floating illegal operand 
idlctrl.R |INF| 137|2016-09-12 05:07:57.494935| MAIN > . 

05:13 DX Load Program error
adsec.R            |INF|       207|2016-09-12 05:13:37.417802|             MAIN > fsm_reset() procedure started.
adsec.R            |INF|       208|2016-09-12 05:13:37.418241|             MAIN > Disabling timeout on diagnostic frames accepted by fastdiagnostic...
adsec.R            |INF|       209|2016-09-12 05:13:37.418501|             MAIN > Ok: timeout on frames disabled.
adsec.R            |ERR|       210|2016-09-12 05:13:37.435506|      ADAM-MODBUS > [TcpReceiveException] Error receiving data (errno=0 ), socket closed. (code -15003) TcpConnection receive error File TcpConnection.cpp line 131
adsec.R            |INF|       211|2016-09-12 05:13:39.595018|             MAIN > Waiting for Crate Bcus ready...
adsec.R            |INF|       212|2016-09-12 05:13:39.595146|             MAIN > Checking for BCUs 5-10 ready...
adsec.R            |WAR|       213|2016-09-12 05:13:39.695430|             MAIN > ...waiting 29.9 s
adsec.R            |WAR|       214|2016-09-12 05:13:40.795981|             MAIN > ...waiting 28.8 s

Later...

adsec.R            |INF|       283|2016-09-12 05:15:54.211145|             MAIN > fsm_reset() procedure done.
Last message repeated
adsecarb.R         |ERR|       110|2016-09-12 05:15:54.214342|              IDL > Command returns error : IDL crate id from power backplane error
20161209 05:15:54.2150 Alert: Warning: loadProgram failed or syntax error occured
adsecarb.R         |INF|       111|2016-09-12 05:15:54.214672|   COMMANDHANDLER > Command execution failed: Fsm forced to Failure

05:18 SX Load program failure
idlctrl.L          |INF|       214|2016-09-12 05:18:21.644448|             MAIN > Disabling DSP WATCHDOG...
idlctrl.L          |INF|       215|2016-09-12 05:18:21.647805|             MAIN > done.
idlctrl.L          |INF|       216|2016-09-12 05:18:21.674431|             MAIN > fsm_reset() procedure started.
idlctrl.L          |INF|       217|2016-09-12 05:18:21.674503|             MAIN > % WHO_IS_MASTER: Only one BCU can be used as master. Fix the configuration
idlctrl.L          |INF|       217|2016-09-12 05:18:21.674503|             MAIN >  .                   problem and reset
idlctrl.L          |INF|       217|2016-09-12 05:18:21.674503|             MAIN >  . % PROC_STARTUP: Interrupt disabling fails!
idlctrl.L          |INF|       217|2016-09-12 05:18:21.674503|             MAIN >  . 
idlctrl.L          |INF|       218|2016-09-12 05:18:21.674797|             MAIN > Disabling timeout on diagnostic frames accepted by fastdiagnostic...
idlctrl.L          |INF|       219|2016-09-12 05:18:21.674942|             MAIN > Ok: timeout on frames disabled.
idlctrl.L          |INF|       220|2016-09-12 05:18:23.844876|             MAIN > Waiting for Crate Bcus ready...
idlctrl.L          |INF|       221|2016-09-12 05:18:29.347909|             MAIN > Ok. All Crate Bcus are ready
idlctrl.L          |INF|       222|2016-09-12 05:18:29.347975|             MAIN > Waiting for Switch Bcu ready...
idlctrl.L          |INF|       223|2016-09-12 05:18:31.549712|             MAIN > Ok.  Switch Bcu is ready
idlctrl.L          |INF|       224|2016-09-12 05:18:31.549985|             MAIN > housekeeper.L test started.

Popup Box after failure

Command Load program failed with status EXECUTION FAILED
IDL ctrate id from power backplane error

adsec.L            |INF|       284|2016-09-12 05:19:07.562208|             MAIN > fsm_reset() procedure done.
Last message repeated
adsecarb.L         |ERR|       110|2016-09-12 05:19:07.565025|              IDL > Command returns error : IDL crate id from power backplane error
20161209 05:19:07.5657 Alert: Warning: loadProgram failed or syntax error occured
adsecarb.L         |INF|       111|2016-09-12 05:19:07.565329|   COMMANDHANDLER > Command execution failed: Fsm forced to Failure

05:45 Active Preset to AO819
  • success

05:47 Power on both WFS's

05:50 DX Error with bayside Y stage

WfsArbitrator_11587|ERR|       109|2016-09-12 05:49:22.572646|             MAIN > Errors for these devices: ['setupXYZstages', 'setupADC', 'setupCCD47']
WfsArbitrator_11587|ERR|       110|2016-09-12 05:49:22.572665|             MAIN > 
WfsArbitrator_11587|ERR|       111|2016-09-12 05:49:22.572680|             MAIN > setupXYZstages: Timeout waiting for baysidey.R.STATUS
WfsArbitrator_11587|ERR|       112|2016-09-12 05:49:22.572707|             MAIN > setupADC: Timeout waiting for adc1.R.STATUS
WfsArbitrator_11587|ERR|       113|2016-09-12 05:49:22.572722|             MAIN > setupCCD47: Timeout waiting for ccd47.R.STATUS
WfsArbitrator_11587|INF|       114|2016-09-12 05:49:22.572753|             MAIN > Elapsed time: 161.74 seconds
WfsArbitrator_11587|ERR|       115|2016-09-12 05:49:22.574300|             MAIN > baysidey in wrong state for requested operation. Current state: STATE_NOCONNECTION. (-11005) CTRL_WRONG_STATE_ERROR
WfsArbitrator_11587|ERR|       116|2016-09-12 05:49:22.574320|             MAIN > Traceback (most recent call last):
WfsArbitrator_11587|ERR|       117|2016-09-12 05:49:22.574336|             MAIN >   File "/home/flao/TEST/lib/python/AdOpt/WfsArbScripts.py", line 2993, in arbFunc
WfsArbitrator_11587|ERR|       118|2016-09-12 05:49:22.574352|             MAIN >     errstr = func(*args)
WfsArbitrator_11587|ERR|       119|2016-09-12 05:49:22.574367|             MAIN >   File "/home/flao/TEST/lib/python/AdOpt/WfsArbScripts.py", line 792, in TurnSystemOn
WfsArbitrator_11587|ERR|       120|2016-09-12 05:49:22.574383|             MAIN >     initStagePos(app)
WfsArbitrator_11587|ERR|       121|2016-09-12 05:49:22.574399|             MAIN >   File "/home/flao/TEST/lib/python/AdOpt/WfsArbScripts.py", line 1967, in initStagePos
WfsArbitrator_11587|ERR|       122|2016-09-12 05:49:22.574414|             MAIN >     setStagePos( app, app.stagex.getPos(), app.stagey.getPos(), app.stagez.getPos())
WfsArbitrator_11587|ERR|       123|2016-09-12 05:49:22.574430|             MAIN >   File "/home/aoacct/aoroot/lib/python/AdOpt/hwctrl/simplemotor.py", line 139, in getPos
WfsArbitrator_11587|ERR|       124|2016-09-12 05:49:22.574445|             MAIN > InvalidStateException: baysidey in wrong state for requested operation. Current state: STATE_NOCONNECTION. - code: -11005 CTRL_WRONG_STATE_ERROR

baysidex.R         |INF|        51|2016-09-12 05:54:46.682485|             MAIN > Connecting to ts847dx, 2101
baysidex.R         |ERR|        52|2016-09-12 05:54:46.685289|             MAIN > Error configuring network: (-1) Return from search routine (item not found) - Errno
: Connection refused
baysidex.R         |INF|        53|2016-09-12 05:54:48.686435|             MAIN > Connecting to ts847dx, 2101
baysidex.R         |INF|        54|2016-09-12 05:56:32.156268|             MAIN > Connecting to ts847dx, 2101

baysidey.R         |INF|       554|2016-09-12 05:55:35.446649|             MAIN > Connecting to ts847dx, 2102
baysidey.R         |ERR|       555|2016-09-12 05:55:35.449975|             MAIN > Error configuring network: (-1) Return from search routine (item not found) - Errno
: Connection refused
baysidey.R         |INF|       556|2016-09-12 05:55:37.451659|             MAIN > Connecting to ts847dx, 2102
baysidey.R         |ERR|       557|2016-09-12 05:55:37.453943|             MAIN > Error configuring network: (-1) Return from search routine (item not found) - Errno
: Connection refused

baysidez.R         |INF|       612|2016-09-12 05:57:20.167028|             MAIN > Connecting to ts847dx, 2103
baysidez.R         |ERR|       613|2016-09-12 05:57:21.169235|             MAIN > Error configuring network: (-1) Return from search routine (item not found) - Errno
: No route to host
baysidez.R         |INF|       614|2016-09-12 05:57:23.172963|             MAIN > Connecting to ts847dx, 2103

05:59 w_stop/start
  • Power On
  • Bayside stage do not power on.
  • Power Supply problems?

06:10 presetAO fail. fw1 not responding
   MAIN > Executing: self.fw1.moveTo(-0.000026)
WfsArbitrator_32579|ERR|       137|2016-09-12 06:10:15.273393|             MAIN > Controller in wrong state for requested operation. (-11005) CTRL_WRONG_STATE_ERROR
WfsArbitrator_32579|ERR|       138|2016-09-12 06:10:15.273424|             MAIN > Traceback (most recent call last):
WfsArbitrator_32579|ERR|       139|2016-09-12 06:10:15.273444|             MAIN >   File "/home/flao/TEST/lib/python/AdOpt/WfsArbScripts.py", line 2993, in arbFunc
WfsArbitrator_32579|ERR|       140|2016-09-12 06:10:15.273463|             MAIN >     errstr = func(*args)
WfsArbitrator_32579|ERR|       141|2016-09-12 06:10:15.273482|             MAIN >   File "/home/flao/TEST/lib/python/AdOpt/WfsArbScripts.py", line 928, in doPrepareAcquireRef
WfsArbitrator_32579|ERR|       142|2016-09-12 06:10:15.273500|             MAIN >     boardSetup.load( app, setupFile)
WfsArbitrator_32579|ERR|       143|2016-09-12 06:10:15.273533|             MAIN >   File "/home/aoacct/aoroot/lib/python/AdOpt/boardSetup.py", line 107, in load
WfsArbitrator_32579|ERR|       144|2016-09-12 06:10:15.273553|             MAIN >   File "/home/flao/TEST/lib/python/AdOpt/thAOApp.py", line 762, in execfile
WfsArbitrator_32579|ERR|       145|2016-09-12 06:10:15.273571|             MAIN >     eval(bytecode)
WfsArbitrator_32579|ERR|       146|2016-09-12 06:10:15.273589|             MAIN >   File "", line 1, in ?
WfsArbitrator_32579|ERR|       147|2016-09-12 06:10:15.273607|             MAIN >   File "/home/aoacct/aoroot/lib/python/AdOpt/hwctrl/simplemotor.py", line 186, in moveTo
WfsArbitrator_32579|ERR|       148|2016-09-12 06:10:15.273626|             MAIN > InvalidStateException: Controller in wrong state for requested operation. - code: -11005 CTRL_WRONG_STATE_ERROR

06:12 Cycle Filter Wheel power on HW GUI
  • fw2 power off
  • fw2 powers on and communicates
  • fw1 unknown
  • fw1 no change

06:15 Stop/Start Filter Wheel ctrl 1
  • no change

06:22 Power off, w_stop, w_start, Power On
  • Now fw1 works
  • Was the terminal server ts8 port 3105 the problem

06:28 PresetAO failed. Could not communicate to CCD47
Mon Sep 12 06:28:11.134 2016 left Preset alert for LUCIFER mode ADAPTIVEACE_ACTIVE: PresetAO{FLAO}[5] - Timeout waiting for techviewctrl.L.PIXELLUT.CUR (-5001) TIMEOUT_ERROR
Mon Sep 12 06:28:11.162 2016 left Preset command failed for LUCIFER mode ADAPTIVEACE_ACTIVE with guidestars, AO reference, Sound=presetFailed.wav
  • connection problem to ts8dx, port 3501

06:32 Power CCD47 Little Joe off/on
  • No change

06:33 Stop CCD47 Controler and start
  • No change

06:37 Power Off, w_stop, w_start, Power On
  • fw1 OK
  • CCD47 OK
  • bayside X ERROR
  • problem with ts8dx port 3101?
WfsArbitrator_11192|ERR|       109|2016-09-12 06:42:41.616398|             MAIN > Errors for these devices: ['setupXYZstages', 'setupRerotator']
WfsArbitrator_11192|ERR|       110|2016-09-12 06:42:41.616429|             MAIN > 
WfsArbitrator_11192|ERR|       111|2016-09-12 06:42:41.616449|             MAIN > setupXYZstages: Timeout waiting for baysidex.L.STATUS
WfsArbitrator_11192|ERR|       112|2016-09-12 06:42:41.616482|             MAIN > setupRerotator: Timeout waiting for rerotator.L.STATUS
WfsArbitrator_11192|INF|       113|2016-09-12 06:42:41.616519|             MAIN > Elapsed time: 203.39 seconds
WfsArbitrator_11192|ERR|       114|2016-09-12 06:42:41.618542|             MAIN > baysidex in wrong state for requested operation. Current state: STATE_NOCONNECTION. (-11005) CTRL_WRONG_STATE_ERROR
WfsArbitrator_11192|ERR|       115|2016-09-12 06:42:41.618570|             MAIN > Traceback (most recent call last):
WfsArbitrator_11192|ERR|       116|2016-09-12 06:42:41.618590|             MAIN >   File "/home/flao/TEST/lib/python/AdOpt/WfsArbScripts.py", line 2993, in arbFunc
WfsArbitrator_11192|ERR|       117|2016-09-12 06:42:41.618609|             MAIN >     errstr = func(*args)
WfsArbitrator_11192|ERR|       118|2016-09-12 06:42:41.618627|             MAIN >   File "/home/flao/TEST/lib/python/AdOpt/WfsArbScripts.py", line 792, in TurnSystemOn
WfsArbitrator_11192|ERR|       119|2016-09-12 06:42:41.618646|             MAIN >     initStagePos(app)
WfsArbitrator_11192|ERR|       120|2016-09-12 06:42:41.618664|             MAIN >   File "/home/flao/TEST/lib/python/AdOpt/WfsArbScripts.py", line 1967, in initStagePos
WfsArbitrator_11192|ERR|       121|2016-09-12 06:42:41.618682|             MAIN >     setStagePos( app, app.stagex.getPos(), app.stagey.getPos(), app.stagez.getPos())
WfsArbitrator_11192|ERR|       122|2016-09-12 06:42:41.618700|             MAIN >   File "/home/aoacct/aoroot/lib/python/AdOpt/hwctrl/simplemotor.py", line 139, in getPos
WfsArbitrator_11192|ERR|       123|2016-09-12 06:42:41.618719|             MAIN > InvalidStateException: baysidex in wrong state for requested operation. Current state: STATE_NOCONNECTION. - code: -11005 CTRL_WRONG_STATE_ERROR

06:45 power off/on Bayside stages
  • No Change

06:47 Stop/Start Bayside x,y process
  • No Change

07:20 Per John's suggestion, check and found wfs routines running on both new WFS machines (CentOS 6,7)
  • Stopped process and everything worked.

07:30 SX Bayside X stage did not home. Had to stop process and power cycle and move stage back and forth manually. Not sure which solved the problem

08:00 Close AO Loop on both sides on a 9.6 mag star
  • Opt Loop Data with Gopt Running, no NCPA
    • SX 080200 (gain = .33)
    • DX 080313 (gain = .29)

08:05 NCPA on
  • DX 080502 (~60 nm rms)
  • SX 080559 (~50 nm rms)

08:10 DX loop open
  • safe skip
  • AdSec go into fault
  • Recover fail worked

08:15 SX loop open
  • Could see high order correction was too high. Stripes on pupil
  • tried to lower high order gain, but at each gopt loop the ho2 gain went back to original

08:34 Centering of star on tech viewer
WfsArbitrator_24373|INF|       754|2016-09-12 08:34:18.826767|             MAIN > source_acq.acquire(): starting source acquisition
WfsArbitrator_24373|INF|       755|2016-09-12 08:34:23.131090|             MAIN > Star found at x=20.7, y=27.4, flux=18173.3
WfsArbitrator_24373|INF|       756|2016-09-12 08:34:23.132523|             MAIN > source_acq.acquire(): star found at X=331.33,Y=437.87
WfsArbitrator_24373|INF|       757|2016-09-12 08:34:23.132562|             MAIN > source_acq.acquire(): current error X=-2.469 mm, Y=0.465 mm
WfsArbitrator_24373|INF|       758|2016-09-12 08:34:33.128872|             MAIN > Star found at x=38.9, y=31.0, flux=18194.0
WfsArbitrator_24373|INF|       759|2016-09-12 08:34:33.131099|             MAIN > source_acq.acquire(): star found at X=622.06,Y=496.68
WfsArbitrator_24373|INF|       760|2016-09-12 08:34:33.131141|             MAIN > source_acq.acquire(): current error X=-0.143 mm, Y=-0.005 mm
WfsArbitrator_24373|INF|       761|2016-09-12 08:34:33.131168|             MAIN > source_acq.acquire(): target reached.
WfsArbitrator_24373|INF|       762|2016-09-12 08:34:33.131523|             MAIN > offsetXYZ(): Focus correction (absolute): 0.000 mm

Mon Sep 12 08:37:42.588 2016 right RunAO alert for LUCIFER: StartAO{FLAO}[20] - Safe skip active - cannot proceed (-10045) IDL_SAFE_SKIP_ACTIVE
Mon Sep 12 08:37:42.621 2016 right RunAO alert for LUCIFER: StartAO{FLAO}[20] - Safe skip active - cannot proceed (-10045) IDL_SAFE_SKIP_ACTIVE
Mon Sep 12 08:37:42.636 2016 right RunAO command failed from LUCIFER, type NORMAL, leftRePoint 0, rightRePoint 0

-- %USERSIG{DougMiller - 2016-09-12}%
Topic revision: r3 - 09 Nov 2016, DougMiller
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback