Starting in a few min with CentOS 5 first.


SX homing problem fixed, beginning soon.


/local/towerdata not mounted. local/elab not mounted.: Fri Sep 16 02:44:20.213 2016 aos.PresetAO{FLAO}.failed - PresetAO {FLAO}[15] - IDL variable undefined 'tn'


Starting preset.

Fri Sep 16 03:02:00.388 2016 aos.AcquireRef{FLAO}.failed - AcquireRef {FLAO}[18] - IDL variable undefined 'nstars'

Restarting wfs arbitrator


Acquire Ref step seems to have hung, but there are no errors.

Fri Sep 16 03:15:54.628 2016 aos.AcquireRef{FLAO}.failed - AcquireRef {FLAO}[22] - Command timeout - -13107: Arbitrator interface generic error

Boost issue, should be fixed in UAO.


loop stuck closed in safe skip. Recoverd by stopping the slope computer. CCD39 display "Not Live" again.


Could not recover, had to do w_stop/start


Preset successful.


GOPT and NCPA functional.


Changed successfully to CentOS7, flat set.

Fri Sep 16 04:39:26.094 2016 aos.PresetAO{FLAO}.failed - PresetAO {FLAO}[2] - [AOException] presetAO: FATAL - thGetVar gopt.R.GOPT.REQ error(-3006): Variable not found error

Starting the GOPT process fixed this.


Preset working. Went to bin 2 to with magnitude of 10.2 on WFS CCD39.

Skip frame upon completing Optimize gain. Gains were very high 2.4 2.0 1.6

These gains seem larger, by a lot, than the minima on the graphs that the optimize gain utility generates.


New preset

Fails just after Optimize gain with Safe Skip


Tried to manually set gains, CCD39 froze, couldn't recover performed w_stop/start.

WFS arbitrator process down upon w_start, this is the second time that's happened tonight. I was able to restart it manually.


Seeing variable and bad 1.5-1.8.

New Preset. Loop opened with safe skip, just like before.


Attempted to do manual gain. Got safe skip with a rolled left edge of pupil at gain 0.5 0.05 0.0.


Attempting new preset in Intervention Mode.


I turned off "Bad Seeing Mode" on AOS, the "Good seeing reconstructor" works better.

Almost worked, loop opened, when I tried to reclose the loop, the shell ripped with "Safe Skip"


Lost CCD 39 again. had to perform w_stop/start


Stuck powering on at BCU47 OK step.


Finally powered on fully.

Fri Sep 16 06:53:19.369 2016 aos.PresetAO{FLAO}.failed - PresetAO {FLAO}[43] - [AOException] presetAO: FATAL - thGetVar gopt.R.GOPT.REQ error(-3006): Variable not found error

Had to start GOPT process and restart arbitrator.


It looks like when I do the Apply optical gain step, GOPT doesn't work properly, there is no gain ratio applied because it sees all commands as 0. Then when the the NCPA is applied, the NCPA is way too high and the shell rips.

Xianyu confirms that the NCPA vector applied was out of bounds. This I think confirms my suspicion.

It looks like the Optical Gain routine was not on, We'll see if turning it on helps.


New Preset.

Optical Gain Process keeps crashing. The Optical Diagnostic for GOPT log gives the following errors:

od_gopt.R |INF| 6718|2016-09-16 07:27:23.969235| MAIN > Starting save of 2000 frames
od_gopt.R |INF| 6719|2016-09-16 07:27:23.969282| MAIN > >> Sending command Save status request...
od_gopt.R |DEB| 6720|2016-09-16 07:27:23.969304| ARB-INTERFACE > Requesting command Save status to wfsarb.R...
od_gopt.R |DEB| 6721|2016-09-16 07:27:23.969461| ARB-INTERFACE > Message succesfully sent
od_gopt.R |DEB| 6722|2016-09-16 07:27:23.969474| ARB-INTERFACE > Waiting message reply: sender=wfsarb.R, code=2020
od_gopt.R |DEB| 6723|2016-09-16 07:27:23.979551| ARB-INTERFACE > Error in thWaitMsg: -5001 (Timeout error)
od_gopt.R |ERR| 6724|2016-09-16 07:27:23.979613| MAIN > Command Save status error
od_gopt.R |ERR| 6725|2016-09-16 07:27:23.979643| MAIN > Error saving WFS status: (-1) Return from search routine (item not found)
od_gopt.R |INF| 6726|2016-09-16 07:27:23.979670| MAIN > >> Sending command Save status request...
od_gopt.R |DEB| 6727|2016-09-16 07:27:23.979680| ARB-INTERFACE > Requesting command Save status to adsecarb.R@M_ADSEC...
od_gopt.R |DEB| 6728|2016-09-16 07:27:23.979765| ARB-INTERFACE > Message succesfully sent
od_gopt.R |DEB| 6729|2016-09-16 07:27:23.979780| ARB-INTERFACE > Waiting message reply: sender=adsecarb.R@M_ADSEC, code=2024
od_gopt.R |DEB| 6730|2016-09-16 07:27:23.989848| ARB-INTERFACE > Error in thWaitMsg: -5001 (Timeout error)
od_gopt.R |ERR| 6731|2016-09-16 07:27:23.989892| MAIN > Command Save status error
od_gopt.R |ERR| 6732|2016-09-16 07:27:23.989929| MAIN > Error saving AdSec status: (-1) Return from search routine (item not found)
od_gopt.R |INF| 6733|2016-09-16 07:27:23.990251| MAIN > Getting frame 1 of 2000
od_gopt.R |INF| 6734|2016-09-16 07:27:23.992222| MAIN > Getting frame 2 of 2000
od_gopt.R |INF| 6735|2016-09-16 07:27:23.994788| MAIN > Getting frame 3 of 2000
od_gopt.R |INF| 6736|2016-09-16 07:27:23.996797| MAIN > Getting frame 4 of 2000
od_gopt.R |INF| 6737|2016-09-16 07:27:23.998713| MAIN > Getting frame 5 of 2000
od_gopt.R |INF| 6738|2016-09-16 07:27:24.000766| MAIN > Getting frame 6 of 2000
od_gopt.R |INF| 6743|2016-09-16 07:27:24.002970| MAIN > Getting frame 7 of 2000


od_gopt.R |INF| 11259|2016-09-16 07:29:26.416894| MAIN > Writing on disk....
od_gopt.R |ERR| 11260|2016-09-16 07:29:26.417371| MAIN > Error saving dimm data: (-1) Return from search routine (item not found)
od_gopt.R |ERR| 11261|2016-09-16 07:29:26.417757| MAIN > Error saving wind speed data: (-1) Return from search routine (item not found)
od_gopt.R |ERR| 11262|2016-09-16 07:29:26.418088| MAIN > Error saving wind direction data: (-1) Return from search routine (item not found)
od_gopt.R |ERR| 11263|2016-09-16 07:29:26.418480| MAIN > Error saving guide camera data: (-1) Return from search routine (item not found)
od_gopt.R |ERR| 11264|2016-09-16 07:29:26.418789| MAIN > Error saving slope data
od_gopt.R |ERR| 11265|2016-09-16 07:29:26.419133| MAIN > Error saving modes data
od_gopt.R |ERR| 11266|2016-09-16 07:29:26.419461| MAIN > Error saving frames counter data
od_gopt.R |ERR| 11267|2016-09-16 07:29:26.419771| MAIN > Error saving frames counter data
od_gopt.R |ERR| 11268|2016-09-16 07:29:26.420128| MAIN > Error saving command data
od_gopt.R |ERR| 11269|2016-09-16 07:29:26.420522| MAIN > Error saving skip frame counter data
od_gopt.R |ERR| 11270|2016-09-16 07:29:26.420834| MAIN > Error saving pending frame counter data
od_gopt.R |ERR| 11271|2016-09-16 07:29:26.421160| MAIN > Error saving wfs global timeout data
od_gopt.R |ERR| 11272|2016-09-16 07:29:26.421469| MAIN > Error saving fastlink timeout data
od_gopt.R |ERR| 11273|2016-09-16 07:29:26.421789| MAIN > Error saving crc errors data
od_gopt.R |ERR| 11274|2016-09-16 07:29:26.422180| MAIN > Error saving timestamp data
od_gopt.R |ERR| 11275|2016-09-16 07:29:26.422506| MAIN > Error saving loop closed data
od_gopt.R |ERR| 11276|2016-09-16 07:29:26.422809| MAIN > Error saving antidrift data
od_gopt.R |INF| 11277|2016-09-16 07:29:26.423677| MAIN > All data saved correctly
od_gopt.R |INF| 11930|2016-09-16 07:32:09.212872| MAIN > Loading dark: 20160916_073209_antidrift.fits


New Preset

GOPT process stayed running, but the GOPT gui says that it is suspended because the loop is not closed, but the loop is closed.

Fri Sep 16 07:50:45.936 2016 aos.Stop{FLAO}.failed - Stop{FLAO}[56] - Command interrupted

Lost Azimuth Drive.


New Preset


Another Preset. Autogain worked, but when reclosing the loop after it opened, it did not remember the gains that were found during Autogain.

There are a lot of safe skips. I wonder if the threshold is lower than it is in CentOS 5.

Alfio says that the force threshold is in the firmware.

We can adjust how many safe skips it takes to open the loop, however, but only in autogain.conf

Doug and Xianyu think there must be a threshold somewhere in the configuration files, though, because we changed it for LBTI


We commented out a line in the AO arbitrator. Now it should skip trying to apply NCPA.


New Preset. Full preset worked.

Tried Offset

Fri Sep 16 09:40:13.551 2016 aos.OffsetXY{FLAO}.started - OffsetXY {FLAO}[80] (ofsx:2.97718[2.97719],ofsy:0.00810851)
Fri Sep 16 09:40:16.739 2016 aos.OffsetXY{FLAO}.failed - OffsetXY {FLAO}[80] - Command interrupted

We think that the X offset might be in the wrong direction.

Loop stayed paused.


New Preset with attempted offset. Y offset worked.

X offset:

Fri Sep 16 09:49:54.208 2016 aos.Resume{FLAO}.failed - Resume{FLAO}[12] - [AOException] RETRY: Star not found - loop still paused (-20002) WFSARB_STAR_WRONGFLUX

There is a bug in which the x flip flag has no effect. When we have a stage transform it will take care of this, but Alfio will fix it in CentOS6, and Xianyu will do so in 7.


Alfio fixed the bug, we are moving the telescope to a higher star. It should be ready to go.


New Offset. X offset works. Since we changed the code, we are going to check y. Y worked.


Off axis presets working, both in X and Y.


GoPT fixed gui working NCPA seems to be applied. No safe skips. !!!!!!


Shell ripped for unknown reason. WFS arb disconnected from msgd. I killed the orphan process and restarted. Reset shell.


NCPA still works. We made the OK and KO work and show up green and red, respectively, in the gopt gui.


NCPA Changes with rotator angle, as expected. We still need to verify it, but it seems to function properly, in a qualitative sense.


Doing offsets with NCPA and Gopt going.

Excessive gopt traffic in ICE caused adsec arbitrator to die. Pressing resumeAO in adsec gui re starts ao.

Fri Sep 16 11:36:42.743 2016 aos.Pause{FLAO}.failed - Pause{FLAO}[12] - [Exception] ConnectionRefusedException:Connection refused


Everything seems stable.


did off axis preset. Loop closed and gopt worked well


Shell set and rested from AOS gui successfully.

-- %USERSIG{GregoryTaylor - 2016-09-16}%
Topic revision: r1 - 16 Sep 2016, GregoryTaylor
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback