AO Commissioning on 16 June 2011

Observer(s): G. Brusa, E. Pinna, J. Brynnel, F. Quiros-Pacheo, J. Argomedo, K. Boutsia (LBTO)

Telescope Operator: D. Gonzales (LBTO)

SW Support: N. Cushing, D. Cox (Tucson)

Telescope Support: J. Little (LBTO)

Instrument Support: D. Officer (LBTO)

Summary

At the beginning of the night it took us 30min to correct pointing which is unusually long. Pointing was way off, as well as focus. We found out that there is a problem with the AZ cam and is loosing the guiding star. Anemometer and AZ cam had to be restarted after standards.
Variable seeing between 0.7'' - 1.1''.
Thin clouds, getting worse during the second half of the night, clearing up before dawn

Details

02:30 Start sky flats JHKs

02:47 Pointing is way off - we need to do pointing correction

03:04 the shell was ripped, it did a ''recover failure'' but the AOS did not realize
  • the shell rip was caused by an anemometer crash

03:21 iif_preset,BS9156, MODE=ACTIVE, TS=0, GS=1, PA=-180
  • preset failed
  • it does find the guide star and then it looses it - strange!

03:24 list of offsets
  • iif_offset, 4, -11, COORD='DETXY'
  • iif_offset, 4, -2, COORD='DETXY'
  • iif_offset, -5, -2, COORD='DETXY'
  • iif_offset, -5, -11, COORD='DETXY'
  • executed for J,H,Ks see summary bellow for list of images
  • elevation 50

03:35 DIMM 1.00''

03:46 iif_preset,BS9158, MODE=ACTIVE, TS=0, GS=4, PA=-90
  • preset failed because we have no guide star
  • send the preset again
  • pointing correction is needed!

03:52 iif_preset,BS9158, MODE=ACTIVE, TS=0, GS=4, PA=-90
  • the guide star is there and then it disappears - have to send the preset again
  • it is an AZ cam problem
  • at the beginning of the collimation cycle the focus is way off and it takes a long time to collimate

03:57 start offseting
  • iif_offset, 5, -1, COORD='DETXY'
  • iif_offset, 5, -11, COORD='DETXY'
  • iif_offset, -5, -11, COORD='DETXY'
  • iif_offset, -5, -1, COORD='DETXY'
  • executed for J,H,Ks see summary bellow for list of images
  • elevation 77

04:17 T.O went up to restart AZ cam and anemometer

04:29 T.O is back but we still get the same error for the anemometer
  • the anemometer was rebooted but the shell did not rip

04:33 Ready to go
  • pisces temperature 74.67K

04:35 iif_preset, AO517, MODE=ACQUIRE,GS=0, ROTMODE=IDLE

04:38 iif_preset, AO517, MODE=ACE, GS=1, RS=0, PA=180
  • the preset failed
===> Error in irc_presetTelescope
        err = Resmsg[0]  = PresetTelescope result status: Error
        err = Resmsg[1]  = Stop[163] complete
        err = Resmsg[2]  = did not receive full CCD readout from guide camera within timeout period of 9460 ms and for 5 consecutive attempts.
  • this happened because the GCS had to be restarted after the AZ cam reboot.

04:40 iif_preset, AO517, MODE=ACE, GS=1, RS=0, PA=180

04:42 irc_runao - success!

05:25 DIMM 0.7''

06:01 DIMM 0.9''

06:03 Obtain psf for strehl measurement - 45% (good)- the same value as last year

06:10 going on the dome to switch off the colling at the treehouse

06:18 the treehouse was already off - there are some clouds

06:23 iif_preset, AO517, MODE=ACE, GS=1, RS=0, PA=90

06:25 irc_runao

06:31 iif_preset, AO517, MODE=TTM, GS=1, RS=0, PA=90
  • failed - the shell ripped - all errors come from the GCS
 
 ===> Error in irc_presetTelescope
        err = Resmsg[0]  = PresetTelescope result status: Error
        err = Resmsg[1]  = IDL Mirror stroke requested is too large
        err = Resmsg[2]  = Stop[177] failed - IDL Mirror stroke requested is too large
        err = Resmsg[3]  = couldn't adjust exposure time to meet low counts requirement. star too dim or guide rate too high.
        err = Resmsg[4]  = maximum exposure time is 2100 ms at the given guide rate, estimated counts are 283
        err = Resmsg[5]  = adjusted exposure time to 2100 ms
        err = Resmsg[6]  = no guide star found in guide box after resumeGuiding * 
 

06:33 iif_preset, AO517, MODE=TTM, GS=1, RS=0, PA=90
  • it was successful this time

06:34 irc_runao

06:47 shell ripped - timeout of diagniostic frames

06:49 iif_preset, AO517, MODE=TTM, GS=1, RS=0, PA=90

06:50 irc_runao - failed
  • err = Resmsg[0] = RunAO result status: Error
  • err = Resmsg[1] = err = Resmsg[2] = AcquireRef[188] failed
  • BCU 39 stopped

06:56 iif_preset, AO517, MODE=TTM, GS=1, RS=0, PA=90

06:57 irc_runao

07:03 the guide star was lost - BUT it should not be guiding!!!

07:10 iif_preset, AO517, MODE=TTM, GS=1, RS=0, PA=90

07:13 irc_runao - failure
  • err = Resmsg[0] = RunAO result status: Error
  • err = Resmsg[1] = err = Resmsg[2] = AcquireRef[198] failed
  • the shell is ripped

07:18 iif_preset, AO517, MODE=TTM, GS=1, RS=0, PA=90

07:19 irc_runao
  • we started runao with 2900nm of rms and the shell ripped
  • we should wait

07:24 iif_preset, AO517, MODE=TTM, GS=1, RS=0, PA=90

07:26 irc_runao - failed
  • err = Resmsg[0] = RunAO result status: Error
  • err = Resmsg[1] = err = Resmsg[2] = AcquireRef[204] failed

07:31 iif_preset, AO517, MODE=TTM, GS=1, RS=0, PA=90

07:32 we lost the guiding star - although the preset returned no errors!

07:35 iif_preset, AO517, MODE=TTM, GS=1, RS=0, PA=90
  • now we see the star - and then we lost! - looks like the AZ cam problem
  • the preset returned with no error

07:37 iif_preset, AO517, MODE=TTM, GS=1, RS=0, PA=90
  • same error - the guide star is drifting away

07:40 iif_preset, AO517, MODE=TTM, GS=1, RS=0, PA=90
  • preset failed

07:42 iif_preset, AO517, MODE=TTM, GS=1, RS=0, PA=90 - OK

07:44 - more clouds

07:45 we switched off guiding and sending zernikes before irc_runao
  • irc_runao - failed giving error
  • err = Resmsg[1] = right Preset mode not adaptive
  • this happened because by stopping the guiding before runao we passed to track mode.

07:47 irc_runao
  • this time we only stopped sending zernikes
  • failed again
  • err = Resmsg[1] = err = Resmsg[2] = AcquireRef[217] failed

07:49 we need to wait because there are clouds - can not find the guide star

07:52 iif_preset, AO643, MODE=ACQUIRE, GS=0, ROTMODE=IDLE
  • correcting pointing

07:58 iif_preset, AO643, MODE=TTM, GS=3,RS=0,PA=-90
  • star verification failed - star was in the aquisition image but does not show in the guide box

08:04 iif_preset, AO643, MODE=ACTIVE, GS=3, PA=-90
  • this works
  • the TTM preset does not find the guide star for AO corrections

08:08 iif_preset, AO643, MODE=ACE, GS=3,RS=0,PA=-90

08:09 irc_runao
 
===> Error in irc_runao
        err = Resmsg[0]  = RunAO result status: Error
        err = Resmsg[1]  = RETRY: No stars found in TV frame (-20001) WFSARB_STAR_NOTFOUND
        err = Resmsg[2]  = AcquireRef[226] failed - RETRY: No stars found in TV frame (-20001) WFSARB_STAR_NOTFOUND

08:17 iif_preset, AO643, MODE=ACE, GS=3,RS=0,PA=-90

08:21 irc_runao

08:33 shell ripped during offsetting

08:38 iif_preset, AO643, MODE=ACE, GS=3,RS=0,PA=-90

08:40 irc_runao
  • the shell ripped

08:49 irc_runao
  • irc failed because we did not send the preset
  • error : err = Resmsg[1] = No focal station selected

08:50 irc_runao
  • shell rip - reason unknown!

08:57 irc_runao
===> Error in irc_runao
        err = Resmsg[0]  = RunAO result status: Error
        err = Resmsg[1]  = AOARB:Illegal command for state Operational (AcquireRefAO)
        err = Resmsg[2]  = AcquireRef[239] failed - AOARB:Illegal command for state Operational (AcquireRefAO)

08:58 irc_runao - shell ripped

09:03 Call Marco in Italy

09:09 irc_runao - failed because "Preset mode not adaptive"
  • it probably needs a preset before...

09:11 irc_runao - failed for the same reason
  • clouds getting thicker

09:15 DIMM 1.1''

09:16 iif_preset, AO643, MODE=ACE, GS=3,RS=0,PA=-90
  • disable "report centroids to PCS" - "report zernikes to PSF"

09:18 irc_runao - failed - no star found in TV frame
  • center it manually

09:18 irc_runao
  • err = Resmsg[1] = RETRY: source_acq error: Star centering procedure failed to converge after 5 iterations (-20003) WFSARB_STAR_NOTCONVERGED
  • err = Resmsg[2] = AcquireRef[245] failed - RETRY: source_acq error: Star centering procedure failed to converge after 5 iterations (-20003) WFSARB_STAR_NOTCONVERGED

09:20 irc_runao - same error

09:23 irc_runao - same error

09:24 irc_runao
  • failed again but this time the error was " No stars found in TV frame (-20001) "

09:28 iif_preset, AO722, MODE=ACQUIRE, GS=0, ROTMODE=IDLE
  • going to a bright star (R= 7.32)

09:32 iif_preset, AO722, MODE=ACE, GS=46,RS=0, PA=-160

09:35 irc_runao - failed
  • the guider was on

09:39 irc_runao - failed
  • err = Resmsg[1] = No focal station selected

09:40 irc_runao
  • right after sending runao we stop guiding
  • ripped all the same

The problem is most probably in the software according to Marco. Need to understand what has changed since yesterday in TCS The tracking is very bad

09:56 DIMM - 1.3''

10:06 iif_preset, AO722, MODE=ACE, GS=46,RS=0, PA=-160
  • clouds have cleared

10:07 irc_runao - shell rip
  • this time it ripped at a different point of the procedure - before centering the pupils

10:12 iif_preset, AO722, MODE=ACE, GS=46,RS=0, PA=-160

10:14 irc_runao
  • trying to close loop manually

10:20 iif_preset, AO722, MODE=ACE, GS=46,RS=0, PA=-160

10:22 irc_runao - success
  • this time the guiding turned off on it own and the shell did not rip
  • successful for unknown reasons...

10:29 shell ripped - the Adsec arbitrator is lost
  • the loop is not open after all
  • there is something wrong...

10:33 We restart IIF and GCS

10:37 iif_preset, AO722, MODE=ACE, GS=46,RS=0, PA=-160
  • failed right away
  • err = Resmsg[0] = PresetTelescope execution failed: Requested side not allowed for this instrument / focal station combination
  • it failed because after restarting the GCS, idl procedures have to be restarted so that they connect to the iif again

10:38 iif_preset, AO722, MODE=ACE, GS=46,RS=0, PA=-160
  • after restarting the idl

10:40 irc_runao
  • guiding disactivated
  • success!

10:48 DIMM 1.3''

11:09 iif_preset, AO722, MODE=ACE, GS=46,RS=0, PA=-160
  • now it seems to be working fine
  • either the oofload was wrong or the guider was sending wrong offsets ... (?)
  • they restarted AO some days ago - maybe they should have restarted the TCS....

11:11 irc_runao
  • shell ripped

11:17 iif_preset, AO722, MODE=ACE, GS=46,RS=0, PA=-160

11:18 irc_runao - shell ripped
  • it seems not to follow the exact procedure when in disenables guiding

11:25 iif_preset, AO722, MODE=ACE, GS=46,RS=0, PA=-160
  • the guiding is oscilating between two stars

11:28 iif_preset, AO722, MODE=ACE, GS=26,RS=0, PA=-90
  • we change PA in order to take another guide star.
  • failed because the CCD 39 was blocked and had to be initialized

11:29 iif_preset, AO722, MODE=ACE, GS=26,RS=0, PA=-90
  • we see the guiding star but the AO team does not see the RS
  • they look manually for the star

11:32 irc_runao
  • the guiding was disabled before the closed loop
  • shell ripped!

11:35 We restart IIF and GCS
  • restart idl procedures
  • restart also AOS

11:38 iif_preset, AO722, MODE=ACE, GS=26,RS=0, PA=-90
  • no star in the guider - it was there and then it drifted away

11:41 iif_preset, AO722, MODE=ACE, GS=26,RS=0, PA=-90
  • the wfs is saturated! Too much background

11:43 iif_preset, BS9183, MODE=ACTIVE, GS=0, PA=0
  • trying for a last standard
  • preset failed - PCS failed because the shell is ripped

11:46 iif_preset, BS9183, MODE=ACTIVE, GS=0, PA=0
  • preset failed - because background too high
  • err err = Resmsg[2] = full CCD readout is fully saturated. Processing impossible.
  • no standard

11:52 start flats J,H,Ks

12:25 End of night - closing dome

summary flats and standards
BS9156 - elevation 50  
pisces.20110616.0025.hdr.fits - .0038.fits J
pisces.20110616.0039.hdr.fits - .0047.fits Ks
pisces.20110616.0048.hdr.fits - .0056.fits H
BS9158 - high elevation 77  
pisces.20110616.0057.hdr.fits - .0064.fits H
pisces.20110616.0065.hdr.fits - .0073.fits Ks
pisces.20110616.0074.hdr.fits - .0083.fits J
evening flats - good counts FILTER
pisces.20110616.0006.hdr.fits - *.0015.fits Ks
pisces.20110616.0016.hdr.fits - *.0020.fits H
pisces.20110616.0021.hdr.fits - *.0024.fits J
morning flats - good counts FILTER
pisces.20110616.0170.hdr.fits - *.0173.fits Ks
pisces.20110616.0164.hdr.fits - *.0168.fits H
pisces.20110616.0159.hdr.fits - *.0163.fits J

-- KonstantinaBoutsia - 16 Jun 2011

In the afternoon we checked the cooling hoses at the hub level and found no evidence of any kinks or bending. We also removed the cover in the back of the hub and looked inside to the connection to the AS and also found nothing wrong.

Around 21:50 UT the AO supervisor shutdown the ASM power supply because of high temps, what happened was that the pump of the instrument cooling had shut down because of overpressure. We decided to change the threshold of the ALARM for the coolant flow from 0 (l/min) to 3 (l/min) in fact we realized that when there is no flow the reading is slightly above zero thus defeating the purpose of the alarm. The WARNING level was left at 5 (l/min). NOTE: after discussion with Armando later on it seems that this threshold is in fact not used by the system. I think that based on the new requirement of keeping the unit as cold as possible we need to revise the decision and re-introduce the low cooling flow as a trigger of unit power supply shutdown.

We did some tests with shell set and AO loop closed and gain = 0 (dome closed). We found the general expected behavior of temperatures of BCU's, drivers and Stratix chips. It appears that going down with elevation makes things a bit worse but mainly we found that Crates 2 and 3 tend to be 8 degrees warmer than the other crates, this at least at 30 to 60 deg elevation. Now testing 90 deg elevation…

UT 2:04 telescope dome closed, telescope at zenith no WARNINGS and temps for hub and external same as before BUT temps for Crates are better, all the Crates are below 35C with the exception of Crate 2 with Power driver ~ 40 and Crate 3 Stratix at 38 C. It appears that at Zenith the unit has a more uniform temperature distribution.

UT 2:07 Rest command from AOS GUI failed… The AdSec Control GUI shows failure of the command, sent a Recover Fail. Starting acquisition of flats and standard stars.

UT 4:17 The flats and standard stars acquisition was completed. Now we are fixing: * rebooting AZ-cam * power cycling the anemometer

UT 4:38 preset AO517 sent

UT 4:40 preset failed to received cam read-out, the AZ cam was rebooted so GCS should have been restarted

UT 4:41 preset AO517 sent again

UT 4:42 preset successful

UT 4:42 runao

UT 4:43 AO closed loop: success (magWFS ~ 12.0) BIN 2

UT 4:45 we run the auto-gain manually after runao, to check how well it works

UT 4:53 we found that the search for the minimum did not converge because the rms slope signal does not increase significantly even after the gain is increased significantly (we ramped to above 7!). It appears that at BIN2 the behavior of the merit function (wfs signal rms) is alost constant at high gains, why?

UT 4:55 deltay -3mm to center the star on PISCES acquiring the star in FeII with 3s exposure (pisces.20110616.0088)

UT 4:59 looking at the AO telemetry to understand what is happening (we have the moment a gain: 3.75, 1.30)

UT 5:03 auto-gain gives 5.6, 4.0,4.0 (after stopping it before going in skip frames)

UT 5:05 looking at telemetry and SR to see the performances

UT 5:08 trying the old method and found 1.3 and 0.6 (very different from before).

UT 5:12 we will modify the auto-gain routine so that we will continue the search for the minimum in the following case: the last point of the window is only 20% or less than the minimum AND the found minimum is lower than the one found in the previous window (the last condition was added now to prevent a run away of the gain). For the moment the new routine will not be used.

UT 5:17 manually optimized gain: 1.9 1.1 and now we acquire some optical gain data

UT 6:04 completed the optical gain data acquisition (we had to repeat several times because the DIMM data were not always available…)

UT 6:05 acquiring some SR data pisces.20110616.0112-0116 (41% SR) seeing from AO telemetry 0.5", DIMM ~1".

UT 6:18 we check that the tree house fan is off so the SR should be valid.

UT 6:27 acquire ref e runao successful

UT 6:30 testing the new version of the automatic gain, this simply uses a different threshold, it does not work yet

UT 6:31 sent a preset for TTM mode but the shell 'ripped'

UT 6:33 sent a preset for TTM mode again

UT 6:34 runao BIN 4

UT 6:37 successful: loop closed

UT 6:39 sending the automatic gain procedure (old version), got: 1.56 0.64

UT 6:40 offsetting the WFS stages, deltay -3mm (as usual, this is a constant offset that should be put in the preset)

UT 6:43 acquiring ao telemetry and sr in this condition

UT 6:46 shell 'ripped' (06:45:51 timeout of frames from diagnostics)

UT 6:52 shell 'ripped' (06:51:50) it looks like fastdiagnostics disable coil but not sure

UT 8:28 we went back to work on ACE mode after determining that the TTM is not working properly. We are setting the gain manually since the automatic gain is not getting the optimal gain, it looks like the wfs signal rms is almost independent from the gain for high gains, we are using g_tt = 4.02 and g_ho = 2.44.

UT 8:33 shell 'ripped' during the offsetting to center the star on PISCES

UT 8:44 (~) shell 'ripped' again but we are not able to find out why.

UT 8:50 (~) shell 'ripped' again but we are not able to figure out why, we called Italy…

UT 9:29 after several attempts we decided to go to a bright star

UT 9:36 after preset a runao was sent and the shell was 'ripped' right away

we found that the runao does not disable the guiding we did the following tests: a) disable guiding right after runao but the shell still ripped b) setting the gain to zero right after sending runao neither of these things seem to have fixed the problem

UT 11:09 a meeting with Roberto about the problems had yesterday (high temperatures) with the ASM concluded the following: a) we need to know which sensor is showing a higher temperature, we would need to have a log of the ~180 temp sensors for the 'critical' events b) the 'external' temperature sensors are read by components on the backplanes of the crates, we need to know which sensors were giving 'bogus' data to possibly pinpoint to a single crate c) the VCCP is also read by components on the backplanes, the value going to 0 V also needs to be linked to a specific crate, again this could be related to an intermittent malfunctioning of a single crate.

UT 11:09 the AO seems to work again, several TCS sub-systems were restarted

-- GuidoBrusa - 16 Jun 2011
Topic revision: r4 - 16 Jun 2011, JoarBrynnel
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback