UT 20160531 LUCI AO Commissioning Interrupt

Executing 01_WunitTransform

Handover from observers 12:38 MST (07:38 UT)

JHill @ telescope, GTaylor running AO system

Config both sides with N30_FieldStop mask, BrG, Mirror, FC off.

Script modified this afternoon to includes the CU position and lamps off


- 0001 0003 LUCI1 3x2.75s LIR:INT

- 0093 0095 LUCI2 3x2.75s LIR:INT

LUCI1 Field Stop was adjusted yesterday

LUCI2 Field Stop needs to be moved left 280 pix, up 270 pix

FM4m1 2430 -1314

FM4m2 3920 -1078

Preset sent 8:06 AM

AO system problems...

Both AGws on the same off-axis guide star and John reports that their XY positions are within half a millimeter. However, on the N30 camera images I see the on-axis reference star on SX at 505,124 and the DX 817,1364. That is 23.7 arcsec apart!

Preset complete @ 08:31 (both sides)

RunAO SX failed (timeout), DX failed (camera lens problem)

SX shell ripped

Preset sent 08:47 SX failed (timeout waiting for slope computer)

RunAO DX failed (camera lens problem)

Preset sent 08:55

RunAO SX & DX failed (camera lens problem)

Greg says these are slope computer issues?

Telescope returned to science observing 09:30 UT

Report from DMiller on AO problems:

This morning after looking at the logs from last night
and running some tests it was found that the AO systems
suffered from two problems last night.  These problems
have been fixed and both AO systems are ready for 
on-sky close loop operation.

Permissions problem with a temporary file:
 Neither the SX nor the DX AO system could calculate
the pupil position on the pyramids because the analysis writes
to a temporary file that was owned by the user argos.  The files,
one on SX and one on DX, were leftover from the ARGOS run
last week.  On both wfs machines I logged in as argos and
removed these files and the pupil analysis worked properly.

The log error message is in pupilcheck39.R.log

python_14019       |INF|       246|2016-05-31 18:32:13.754787|             MAIN > Exception: [Errno 1] Operation not permitted: '/tmp/acqpupils.fits'

This check for correct pupil calculation has not been part of our
daytime check that the AO systems is ready to operation on-sky,
but will be include now.

IssueTrak # 6013 was submitted.

Fiber connection from SX WUnit to SX AdSec:
 When the loop was closed last night slope from the WUnit slope computer never
arrived at the AdSec.  This problem was reproduced this morning and it was
concluded that the the fibers had changed since the last time the AO Loop
was closed (last Friday).  Apparently on Sunday some fiber connections in the ULTH
where changed in order to correct some problems with LN fiber communication to
the SX AdSec.  Once these fiber connections were corrected today the SX AO FLAO
Loop could be closed.

Neither of these problems were found on Sunday night.
 - When we tried to collect LUCI1/AO data because on the first binocular ACE preset
   a bayside X timeout occurred on the SX side and the preset failed before the loop 
   as closed.  By the time the second preset was sent clouds had rolled in and not AO
   Reference star was found and the SX preset failed.
 - The DX AO Loop closed on the first preset even with the problem of the
   permissions on the temporary file.  Sometimes the initial position of the pupil
   is close enough to correct that the procedure to close the AO loop proceeds past
   the pupil alignment step.  

The problem with the temporary files could have been found and corrected last night.
The fiber problem could not have been fixed last night, and this was the problem which
Greg tried to debug until it was decided to hand the telescope back to the observers.

Again, both AO systems are ready for closed loop operation.


### OSA Night Log:

Created with: LBTO OSA Night Report Tool v6.3
Submitted by: John Hill < [[mailto:jhill@lbto.org][jhill@lbto.org]]>
Submitted on: Tue May 31 11:48:55 2016


UTdate:  2016-05-31

OSA:  John Hill
ISA:  Olga Kuhn
ISp:  John Morris
Tel:  Michael Wagner
Soft: Kellee Summers
AO:   Greg Taylor
Eng:  Tom McMahon

Partner: LBTB - Germany
Names:   Alessandra Contursi
 Sirio Belli

L_Start: None

Evening:  O:Spectroscopic
Eve_IQ:   0.6
Midnight: O:Spectroscopic
Mid_IQ:   0.55
Morning:  O:Spectroscopic
Morn_IQ:  0.8

UThandover: 23:27
UTopened:   02:17
UTclosed:   11:35
UTSciStart: 03:24
UTSciEnd:   11:10

For the night of: Mon, 2016 May 30 ---> Tue, 2016 May 31
 Local midnight = 2016 May 31,  7 hr UT, or JD 2457539.792
 Local Mean Sidereal Time at midnight =  16 17 17.1
 Sunset ( 1000 m horizon):   19 27 MST; Sunrise:   5 07 MST
 Evening twilight:  21 01 MST;  LMST at evening twilight:  13 18
 Morning twilight:   3 33 MST;  LMST at morning twilight:  19 51
 12-degr twilight: 20 24 MST -->  4 10 MST; night center:   0 17 MST
 Moonrise:   1 56 MST   
 Moon at civil midnight: illuminated fraction 0.300
 1.8 days after last quarter, RA and dec:   0 17 45,  -0 23.1
 The sun is down for  9.7 hr;  6.5 hr from eve->morn 18 deg twilight.
  4.9 dark hours after end of twilight and before moonrise.

07:55to08:00 = 00:05 > Adaptive_Optics.Adaptive_Secondary > Shell RIP'd while restarting things
08:38to08:46 = 00:08 > Adaptive_Optics.Adaptive_Secondary > SX Shell RIP'd lost elevation data
09:07to09:11 = 00:04 > Adaptive_Optics.Adaptive_Secondary > Both shells RIP'd for Operator error
09:32to09:55 = 00:23 > Collimation_or_Pointing.LUCI2 > Active Optics runaway in Z9,Z10,Z11
07:46 Length of Night (12deg-12deg Twilight)
00:40 Total Downtime  (Series)
00:40 Total Downtime  (Parallel)
91.4% On-Sky Efficiency



The night started out nicely with clear sky and good seeing.  We finished the telluric standard, and we were taking science spectra before astronomical twilight.  Things degraded after that as some thin clouds moved in, and the seeing got a bit worse.

DaveT and GregT interupted LBTB science for 1.5 hours, but AO problems prevented them from getting any AO calibrations.

Back to science observing after that on the same galaxy.  We lost 23 minutes to an active optics runaway in Z11.


Spent the afternoon sorting out latching of the Left Shutter Door,
a failed test preset, LUCI issues, and a variety of other problems left over from this
morning's power bump. Fortunately we got it all fixed before we opened.

00:58UT reset encoders
00:59UT Authorized: NONE / LUCIFER
00:59UT Tam=9.0 Tch=10.2 Tgl(ref):SX=7.2(9.6),DX=7.2(9.5) Hum=29.2% Wind=5.5@231\xB0
01:50UT DaveT is working on a LUCI2 MOS error.
01:58UT ALH is warning about HBS output cooling which is -5.2 degC - why?
Are input and output cooling temperatures reversed?
ALH is also warning about HBS coolant pump #2.
(HBS GUI says both coolant pumps are off, but they are in fact running.)
(HBS GUI says 2 main pumps are running, but only shows a flow for pump 4.)
02:17UT Shutter doors open
02:18UT Tam=8.0 Tch=10.2 Tgl(ref):SX=7.2(9.7),DX=7.3(9.5) Hum=27.4% Wind=3.8@217\xB0
02:23UT Open side vent doors
02:23UT Close Left Shutter Door
02:26UT Start Mirror Ventilation
02:29UT Left Shutter Door closed and latched correctly on its own.
02:30UT Open Left Shutter Door
02:31UT Open Rear Vent Doors
02:40UT ACQUIRE preset RA 11:46:03.0 DEC +47:46:45.9 AZ -6.5 EL 74.9 Acquisition
02:47UT Pointing model change - IE/CA: DX => 10.0/-258.0
02:51UT ACTIVE preset RA 11:46:03.0 DEC +47:46:45.9 AZ -13.6 EL 74.5 Collimation
02:51UT ACTIVE preset RA 10:43:43.3 DEC +57:11:57.1 AZ -24.3 EL 61.7 Standard
02:58UT Starting DIMM
03:04UT SEEING: DIMM 0.92" @62.9\xB0, GCSR 0.64" (LBT @60.6\xB0)
03:04UT Tam=7.5 Tch=8.4 Tgl(ref):SX=7.6(8.3),DX=7.6(8.0) Hum=25.2% Wind=3.9@288\xB0
03:32UT ACTIVE preset RA 14:19:39.5 DEC +52:52:33.6 AZ 32.2 EL 64.1 Science
03:47UT starting the science spectroscopy exposures
03:59UT Tam=6.6 Tch=7.7 Tgl(ref):SX=7.7(7.7),DX=7.6(7.6) Hum=27.8% Wind=2.1@314\xB0
04:02UT SEEING: DIMM 0.89" @72.8\xB0, GCSR 0.58" (LBT @67.1\xB0)
04:30UT Connected DIMM Focuser (John didn't know this trick.) but it remains at 2020.
05:10UT Thin clouds sneaking in from the North - losing up to 0.1 mag on guide star.
05:25UT Tam=7.4 Tch=7.9 Tgl(ref):SX=7.6(7.8),DX=7.4(7.6) Hum=31.2% Wind=4.4@188\xB0
05:25UT SEEING: DIMM 0.98" @70.9\xB0, GCSR 0.52" (LBT @69.5\xB0)
05:44UT Thin clouds have thickened to 0.2 mags in a remarkably uniform layer.
06:00UT Reduce all vent doors to 50%.
06:08UT We see the flashes from a thunderstorm down in northern Chihuahua.  Can monsoon season be far away?
06:10UT Clouds are thickening to 0.5 mag.
06:17UT Reduce all vent doors to 20%.
07:17UT Tam=6.8 Tch=7.4 Tgl(ref):SX=7.4(7.2),DX=7.2(7.2) Hum=29.3% Wind=5.4@190\xB0
07:17UT SEEING: DIMM 1.01" @55.3\xB0, GCSR 0.52" (LBT @58.7\xB0)
07:18UT Clouds have moved off, so we are observing in clear sky again - Dave pointed out that Clear Sky Clock predicted this band of clouds.

07:44UT ACQUIRE/ACQUIRE preset RA 17:34:56.0 DEC +12:33:38.1 AZ 156.9 EL 68.4 Acquisition
07:44UT Authorized: LUCIFER / LUCIFER
07:51UT Pointing model change - IE/CA: SX => -13.0/-28.0; DX => 25.0/-250.0
07:52UT ACTIVE/ACTIVE preset RA 17:27:22.3 DEC -00:19:23.5 AZ 171.5 EL 56.7 Collimation
07:55UT ASM_L ripped while Greg is restarting something
08:00UT ASM_L set
08:06UT ACE_ACTIVE/ACE_ACTIVE preset RA 18:03:47.7 DEC -00:17:23.7 AZ 161.7 EL 55.7 Science
08:09UT right failed with techviewctrl, left failed with bayside-z
08:11UT ACE_ACTIVE/ACE_ACTIVE preset RA 18:03:47.7 DEC -00:17:23.7 AZ 163.8 EL 56.0 Science
08:12UT now left failed with techviewctrl ??, right succeeded.
08:19UT SX shows two stars in wfs, but DX does not.  Moving hotspot doesn't help much.
08:20UT Tam=6.3 Tch=6.9 Tgl(ref):SX=7.2(6.9),DX=7.0(6.9) Hum=33.4% Wind=5.2@168\xB0
08:20UT SEEING: DIMM 1.08" @53.1\xB0, GCSL 0.83", GCSR 0.75" (LBT @56.4\xB0)
08:28UT Dave reports that target stars are in different places in the two LUCIs by 23 arcsec. Why? 
08:30UT ACE_ACTIVE/ACE_ACTIVE preset RA 18:03:47.7 DEC -00:17:23.7 AZ 172.2 EL 56.8 Science succeeds
08:32UT RunAO on both sides - right failed on camera lens, left for generic error
08:38UT ASM_L ripped
08:44UT Stop and restart AOSL to recover elevation data.
08:46UT ASM_L set
08:47UT ACE_ACTIVE/ACE_ACTIVE preset RA 18:03:47.7 DEC -00:17:23.7 AZ 180.3 EL 57.0 Science left failed for slope computer timeout, right succeeded
08:55UT ACE_ACTIVE/ACE_ACTIVE preset RA 18:03:47.7 DEC -00:17:23.7 AZ 184.0 EL 56.9 Science - preset succeeds both sides
08:57UT RunAO - failed on camera lens convergence on both sides.
Giving up on AO for tonight......
09:03UT Authorized: NONE / LUCIFER
09:04UT ACTIVE preset RA 12:13:12.0 DEC +64:28:56.3 AZ 330.5 EL 28.3 Collimation
09:07UT ASM_L and R ripped - John's star was too low
09:11UT ASM_L set
09:11UT ACTIVE preset RA 13:58:40.2 DEC +52:06:25.8 AZ 313.2 EL 38.6 Collimation
09:11UT ASM_R panic'd so I continued observing at EL=38 in that state. Wind is low.
09:12UT END OF INTERUPT TIME (not counting the new LUCI acquisition)

09:13UT ACTIVE preset RA 14:19:39.5 DEC +52:52:33.6 AZ 313.9 EL 41.5 Science
09:20UT adsc_stop and adsc_start on left to stop the panic.
09:30UT Tam=6.3 Tch=6.7 Tgl(ref):SX=6.8(6.4),DX=6.7(6.8) Hum=28.9% Wind=4.5@149\xB0
09:32UT Collimation error on right? WFS is vignetted strangely?  I should call JohnH!
It seems to be a runaway of Z9,Z10,Z11.  Pupil has been too high on wfs cutout all night. guide star is at 125,94.
09:43UT Clear active optics on DX primary
09:46UT Remove active optics corrections from DX secondary
09:49UT Set Z9, Z10 active optics gains to 0.
09:51UT Guide image is back to normal
09:57UT Note that DX Primary 119 has disabled itself. Was that the start of the runaway? It seems unlikely, but we can check the time later.
10:21UT ACTIVE preset RA 22:41:13.5 DEC -01:21:08.8 AZ 118.8 EL 35.2 Science - didn't find the guide star
10:24UT Enabled DX M1 actuator 119 during the preset, but it disabled again a few minutes later.
10:26UT ACQUIRE preset RA 22:32:46.2 DEC +03:00:18.2 AZ 117.5 EL 40.5 Acquisition
10:27UT Pointing model change - IE/CA: DX => 12.0/-249.0
10:27UT ACTIVE preset RA 22:41:13.5 DEC -01:21:08.8 AZ 120.1 EL 36.3 Science
10:28UT Enabled DX M1 actuator 119 again.
10:55UT adsec_stop on sx, adsec_start, Power On.
11:04UT ACTIVE preset RA 19:54:41.0 DEC -15:54:49.0 AZ 188.5 EL 41.0 Standard
11:07UT Tam=5.9 Tch=6.3 Tgl(ref):SX=6.4(6.3),DX=6.3(6.3) Hum=27.5% Wind=9.0@135\xB0
11:07UT ASM_L set
11:08UT Close all vent doors as wind is picking up.
11:09UT SEEING: DIMM 1.14" @42.8\xB0, GCSR 0.94" (LBT @40.8\xB0)
11:23UT ALH is complaining about DX Primary Cell State, but PMCGUI shows no errors or warnings. ?? 
11:31UT stop reporting Zernikes as the sky is getting bright. wfs spots are low in the lenslet apertures by 0.4 arcsec.  Was this a contributor to the runaway?
11:35UT Closing shutter doors
11:41UT Both shutter doors are closed AND LATCHED! Vent doors already closed.
11:42UT All rotators are parked.  Telescope is stowed at zenith. Enclosure off.
11:43UT AdSec shells rested at zenith (power still on)
11:47UT Alessandra says "best night I ever had at LBT" as we go off to bed!

-- %USERSIG{DavidThompson - 2016-06-02}%


Topic revision: r2 - 06 Jun 2016, DavidThompson
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback