LUCIFER Com 3, Night 5
UT 2009 January 14

Observers: N.Ageorges, W.Seifert, M.Juette, V.Knierim, J.Hill (@LBT)
Telescope Operator: D.Gonzalez-Huerta
SW Support: T.Leibold in Tucson until 00:30
Telescope Support: B.Curtis and D.Officer


We exercised nearly all of the TCS-GCS-IIF hanging types of problems in the first hour after twilight. So John got to see them one after another in person! Torsten was able to solve at least one of these problems related to the guide star checking after an offset.

We used some time in twilight to test Jesper's new transformation live. However, too many other problems kept interupting, so we didn't get a definitive result. Evidence from one star seems to indicate a sign reversal in the zeropoint of the X correction.

After the first hour, things settled down and were fairly smooth. There was some kind of strange TCS communication error just after midnight. We spent about 10 minutes studying it.

Details (Times in MST):

Weather continues photometric. T=2 degC. D=-21degC. Wind 5 m/sec.

17:10 Open Dome. Fighting some problems with the AGW 1-wire network in parallel.

17:20 Starting LUCIFER flats

17:30 With help from Torsten, AGW is ready to go again. We implemented the same workaround as found the other night. We are running check4nport.20080716a rather than the daemonized version.

18:10 flats are finished

18:12 With some time before it is dark, we decide to try offsets using Jesper's transformations. Changed internal_transformation from true to false in AIP_L.cfg.

18:15 Preset in track mode to adjust pointing. Adjust IE=32.4 CA=2.84

18:18 Most of David's windows disappear on lbtmu01, but the telescope keeps running

18:21 Preset ACTIVE from LUCIFER - failed at first because sky too bright relative to star. I adjusted manual exposure time to 1 sec.

18:28:59 Preset ACTIVE fails to start guiding, but GCS is taking guide images. No star is visible in the images. Preset never finished. We pressed "stop Guiding" to end it at 18:36. Seems like the star not found counter is broken. See Issue # 1950 - Torsten identified the problem later in the evening.

18:36 Preset ACTIVE failed with 1 sec ExpTime (plus noise on acquisition image).

18:38 Preset ACTIVE to a brighter star. Succeeded. Then collimate.

18:48 Preset ACTIVE to off-axis star

18:54 Start "BENDING mode" with FCin=1 for Z11 and Z22

18:55 AzCam stopped delivering guider images. I added a note to Issue # 1723.
Jan 13 18:55:37 lbtmu107 LBT_left_GCS: AzCam: IST reported error while receiving image.
Jan 13 18:55:37 lbtmu107 LBT_left_GCS: IST: error when receiving image
Jan 13 18:55:37 lbtmu107 LBT_left_GCS: WFS: z(08) = 87.7654
Jan 13 18:55:37 lbtmu107 LBT_left_GCS: error: GuidingThread: guide image didn't arrive within timeout of 2888 ms.
Jan 13 18:55:37 lbtmu107 LBT_left_GCS: IST: closing open socket

18:56 Stop and restart GCS

19:01 Send Active Corrections to "Default".

19:02 Making the first actual offset with Jesper transformation. But star was out of range - observer error.

19:02 The corresponding stepFocus is hung.

19:08 Stop and restart PSF. (stepFocus still hung)

19:09 Stop and restart OSS. (stepFocus still hung)

19:10 Stop and restart GCS (this released the blocked stepFocus)

19:11 Preset ACTIVE but no star in ACQUISITION image

19:14 Preset TRACK PARALLACTIC failed because GCS failed to park probe.

19:16 Stop and restart GCS

19:18 Preset TRACK PARALLACTIC failed because motor already in motion
Jan 13 19:18:04 lbtmu107 LBT_left_GCS: [E] AIP AGW unit error: AGW1: Motor already in motion
oacserver did not report any error.

David gets the same message when he tries to home the motors.

19:20 stopAGW, then startAGW, home motors, and that cures the problem.

19:23 Preset TRACK PARALLACTIC - succeeds. IE=65.2 CA=2.84

Re-apply Zglobal= +0.5 mm since active optics were cleared when we restarted PSF at 19:08.

19:27 Preset ACTIVE - Why was there so much more coma than we saw earlier? Something seems to have changed position during that last killing of PSF. ?? Pointing-free coma correction is applying -3.3 mm of Y to the primary at EL=69 deg.

19:35 focus 0 (stepFocus) from IRAF, but we hit -Y limit. Add +1 mm Yglobal to both M1 and M2. But after we untangled that, stepFocus was still hanging.

19:36 Stop and restart GCS to release stepFocus.

19:42 Preset ACTIVE, hit Y limit again.

19:46 another stepFocus hangs in coincidence with -Y limit. We add another +1 m Y global.

19:50 Stop and restart GCS to release stepFocus.

19:56:56 At last we try the offset again. Seeing is about 1.1 arcsec FWHM on the guider image.

20:06 Preset ACTIVE to another star (didn't find any star - just off the edge of acquisition image)

20:11 Preset TRACK PARALLACTIC - Adjust IE=62.1CA=-1.5 (wasn't a big enough change to explain it?)

20:16 Preset ACTIVE finds star in acquisition image at 480,263 pixels near the right edge of the frame. Does this mean we still have a transformation problem?

Jan 13 20:10:33 lbtmu107 LBT_left_GCS: AGWUnitAIP: SFP position to move to: 0.000000, 192.200000 mm
Jan 13 20:10:34 lbtmu107 LBT_left_GCS: AGWUnitAIP: rotation of the AGW coords system in relation to SFP: 179.70 degrees
Jan 13 20:10:34 lbtmu107 LBT_left_GCS: AGWUnitAIP: internal transformation to AGW native coords: 2.199127, 419.994243 mm
Jan 13 20:10:34 lbtmu107 LBT_left_GCS: AGWUnitAIP: external transformation to AGW native coords: 4.198235, 420.172773 mm (oacontrol)
Jan 13 20:10:34 lbtmu107 LBT_left_GCS: AGWUnitAIP: moving probe to native coords: 4.198235, 420.172773 mm using external transformations

20:21 LUCIFER stops getting DD data. Torsten thinks it is the Common SW blocking problem again.

20:24 Disconnect LUCIFER, restart IIF and irs per Torsten's advice.

I started looking at the oil temperatures, since the collimation seems a bit unstable. WinCC reports that the EL pad temperatures are +5.5 degC. While ECS is reporting that the coolant (now oil) return temperature is -5.5 deg (note opposite sign) (this sensor reads 3-4 degC too low) with the valve CV0423 open 50%. Ambient and telescope steel temps are in the range 0-1 degC.

20:40 LUCIFER putting a slit mask in.

20:44 We've sent an expedition (Doug/Brian/David) upstairs to check the temperature of HBS oil in the tank. They report that the tank is actually -1.8 degC and that everything else is in order.

20:57 Adjust HBS setpoint offset from -7 to -8 degC. Stop and restart ECS.

21:08 Adjust HBS setpoint offset to -9 degC. (After talking to Jonathan, we explain this value for tank below ambient by +5 degC rise in capillaries/pads, and -4 degC offset of the sensor.) Stop and restart ECS again. After a few minutes the apparent Coolant (oil) Return Temperature is -7.6 degC (actually -3.6 degC).

21:10 Taking calibration spectra with LUCIFER.

21:14 Preset ACTIVE

21:44 Switch back to internal native GCS transformations. Now we find star on the edge of the acquisition field at 515,178. (But we haven't recently adjusted the pointing, so this isn't a certain result.)

22:00 Having problems centering star in the slit. Small offsets don't seem to be behaving well. Guide corrections are running +-0.2 arcsec in each axis, so atmospheric motion would appear to be the explanation. Guider is showing 1 arcsec FWHM in 4.6 sec exposures.

22:15 HBS pad temperatures are down to +4 degC.

LUCIFER continuing to take stellar spectra without too many interruptions.

23:20 Polaris monitor has spiked from 2 arcsec to 4 arcsec in the last half hour. Guider is still seeing 1 arcsec images at EL=42 deg.

23:38 Preset ACTIVE - no star in acquisition image (which is much less convenient when you have the slits in place)

23:43 Preset TRACK PARALLACTIC - Adjust IE=49.9 CA=-15.2

23:46 Preset ACTIVE - succeeds

After the pointing correction (and still in the same area of the sky), we found the guide star at 405,200 on the acquisition image 000082. Does anybody understand why the AGW no longer has the same centering on LUCIFER as it used to? (I note hopefully that reversing the sense of Jesper's corrections would improve this situtation.) This off-axis star is at x=0 y=122.6 mm.

00:15 Preset ACTIVE - preset hangs in a new and different way that Torsten has never seen before. Note that VENT_LOGGED is not a GCS message.

Jan 14 00:15:24 lbtmu107 LBT_left_GCS: No data passed to ArgumentList::deserialize
Jan 14 00:15:25 lbtmu107 LBT_left_GCS: No arguments
Jan 14 00:15:25 lbtmu107 LBT_left_GCS: No arguments
Jan 14 00:16:19 lbtmu107 LBT_left_GCS: [E] AIP AGW unit error: AGW1: RPC Communication Error
Jan 14 00:16:19 lbtmu107 LBT_left_GCS: SFPtoAGWgeneric: -0.000012, 0.001357 (rad) to AGWgeneric failed with 201
Jan 14 00:16:20 lbtmu107 LBT_left_GCS: parseXMLBuf error in exerces VENT_LOGGED
Jan 14 00:16:20 lbtmu107 LBT_left_GCS: No arguments
Jan 14 00:16:20 lbtmu107 LBT_left_GCS: No arguments
Jan 14 00:16:20 lbtmu107 LBT_left_GCS: error: setProbePosition: couldn't move probe to position (SFP) (x,y,z): -1.508777, 168.848935, 21.402673 mm.
Jan 14 00:16:20 lbtmu107 LBT_left_GCS: parseXMLBuf error in exerces K
Jan 14 00:16:20 lbtmu107 LBT_left_GCS: No arguments
Jan 14 00:16:20 lbtmu107 LBT_left_GCS: No arguments

00:26 Kill and restart GCS as a start.

00:28 Preset ACTIVE - succeeded. Seeing fluctuating between 1.2 and 2 arcsec at EL=39 deg, while Polaris monitor is reporting 4-5 arcsec.

00:55 stepFocus command hung for 5 minutes

01:00 kill and restart GCSL (twice)

01:03 the next challenge is to get out of the LUCIFER observing script so we can send the preset again.

01:33 Preset ACTIVE, FCin=0 for Z11, Z22, Clear C00 to unload the forces that were getting up to 68# on Act 101.

01:41 FCin=1 Wavefront got down to 110 nm rms with forces around 10#, but seeing is still 1.4-1.8 arcsec at EL=53 deg.

02:18 Chiller CH0401 LWT is up to -11 degC, so we turn off right side mirror ventilation to reduce the load. Valve CV0423 was full open trying to control the oil temperature.

02:25 Preset ACTIVE - LUCIFER observing continues taking various kinds of spectroscopic standards.

02:30 Polaris monitor is reporting 7 arcsec seeing! However we are seeing 1.4 to 1.8 arcsec FWHM on the guider. Maybe we should hang a sheet over the Polaris display.? Wind (LBTO) is reporting 10 m/sec, so nothing that you would expect to make the result this bad.

02:42 IIF is hanging so LUCIFER is not getting DD entries.

02:44 Kill and restart IIF, but not yet irs.

02:52 Had to send a new preset, even though the previous preset was still happily running. Nancy had trouble getting the LUCIFER observing script to reconnect - complaining of empty buffer.

03:30 Chiller CH0401 LWT is down to -15.7 degC

03:35 HBS Coolant (Oil) Return Temperature reads -8.0 degC. Elevation Pad Input temps range from 3.2 to 5.0 degC. Chamber ambient is +1 degC. Seeing has improved at bit to 0.8 arcs FWHM at EL=52.

04:00 My OSSGUI died with these messages:
X Error: BadRequest (invalid request code or no such operation) 1
  Major opcode:  0
  Minor opcode:  0
  Resource id:  0xb0042
X Error: BadLength (poly request too large or internal Xlib length error) 16
  Major opcode:  104
  Minor opcode:  0
  Resource id:  0xb0042
OSSGUI: Fatal IO error: client killed

Paul: I confirm my previous observation that each small motion of the M2 hexapod requested by PSF is taking OSS 9 seconds to complete. The typical size of these moves is 20 microns.
Wed Jan 14 04:05:31.439 2009  secondary mirror adjustMirrorCollimation started for 0 seconds in the future
Wed Jan 14 04:05:40.088 2009  secondary mirror adjustMirrorCollimation complete for 0 seconds in the future

05:18 Preset ACTIVE - failed

Jan 14 05:20:03 lbtmu107 LBT_left_GCS: TCPSocket::send error: Bad file descriptor
Jan 14 05:20:03 lbtmu107 LBT_left_GCS: full CCD readout requested, now waiting for image to arrive.
Jan 14 05:20:15 lbtmu107 LBT_left_GCS: fatal error: 5. reception of image failed to arrive within timeout period of 7659 ms, giving up.
Jan 14 05:20:15 lbtmu107 LBT_left_GCS: GCSL:startGuiding execution time: 73008 ms at 1231935615488

05:21 Preset ACTIVE - aborted - astronomer error

05:23 Preset ACTIVE - succeeded

06:03 Preset ACTIVE - succeeded

06:36 Preset ACTIVE - succeeded

06:43 Offset command hangs. GCS log stopped updating at 06:43:46.

06:54 kill and restart GCS

07:01 end of observing T=0.3degC, D=-18degC still photometric

TCS Things to test for tomorrow or the next day

  • More work on Jesper's revised transformations
  • Torsten's GCS patch 2.05 for Issue #1950
  • Active Optics mode 3c (ignoring focus)

-- JohnHill - 13 Jan 2009
Topic revision: r6 - 14 Jan 2009, JohnHill
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback