Lucifer February Com Night 8 February 13-14 2009
Observers: N.Ageorges, W.Seifert, V.Knierem (LUCIFER), DThompson (@LBT)
Telescope Operator: W.Wack
SW Support: T.Edgin until 2:30 am, on call thereafter.
Telescope Support: J.Rousseau
- Tonight's Super Blah game, final score: Weather 1, Telescope 2, Astronomers 0
- Lost to winds at start of night: (18:16 to 23:15) 5 hours
- Functional failure of hardpoint 3 time lost: 5 hours
- Issue Trak #2011 updated.
- LUCIFER Rotator/Cable chain drive failure. 2.5 hours lost (IT# )
- Spare part on mountain, Jon and Jay to swap in out in the morning and test.
- The one bright note was that we managed to verify that the AGW transformation is at least basically correct although this would have been a 30 minute test if everything worked well.
Backed out patch to GCS that allowed use of the newest AGW transform from Jesper. We are running on the GCS version from yesterday, with the AIP_L.cfg parameter set to use the internal GCS probe coordinate transformation.
Sustained winds of 20m/s, gusts to 22 since before sunset. As of 22:00 Tair=-7.1 Tmirror=-5.4 RH=53.6% Dewpt=-15 Wind(LBT)=20m/s Wind(SMT)=2.24m/s and it is changing so it is active.
23:15 Winds have been below 20m/s (and gusts below 22) with a downward trend on the LBT weather station for a while now. SMT wind reads 5.81m/s. We tested pointing into and away from the wind with the dome closed and into the wind is indeed a factor of 2 (more like 2.5) below that facing away from the wind. We will open now. LUCIFER team advised to stay to the north.
Pointing correction: IE changed from -31.0 to 11.0, CA changed from 10.0 to -28.0. Also, when the temps are lower than the previous day, focus may need to be adjusted more positive. Collimating took ~8 cycles, but converged well in 1.5 arcsec seeing.
23:55:47 Preset because M1 failed to make chase. One "collimation" cleared it. And it has happened again.
LUCIFER testing the off-axis guide star acquisition with last night's version of GCS. Works at multiple off-axis guide stars at different PAs and distance from AGW_X axis. The AGW transformation has the opposite sign for AGW_X than the internal transformation.
00:04 Switching to the new GCS, first tests to use the same internal GCS transformation as above. Transformations agree in sign. Pointing off, first source at top of detector and was found, but we should correct again.
Feb 14 00:09:12 lbtmu107 LBT_left_GCS: AGWUnitAIP: internal transformation to AGW native coords: -15.952150, 519.575503 mm
Feb 14 00:09:13 lbtmu107 LBT_left_GCS: AGWUnitAIP: external transformation to AGW native coords: -14.586384, 520.100016 mm (oacontrol)
Coordinates differ by the expected ~2mm because of the offset of the hotspot, but are basically the same.
At EL=82, IE changed from 11.0 to 4.52, CA from -28.0 to -34.84. Ouit of collimation a bit, but star found and initial WFS was 1726nm rms, 1223, 904, 660, 629, 299. Guider was running 4.6s exposures, WFS 30 sec.
00:30:53.484 Preset again after previous failed to make chase. EL~80. AGW native coords: 58.297914, 510.580224 mm, it was only a few arcsec off from the hotspot. FWHM on guider around 1.3. WFS collimation: 948, then preset sent at a different PA that worked.
00:36:23.96 Preset failed to make chase. Collimate and resend, worked.
00:39:51.12 Preset failed to make chase, took several collimations to recover.
00:41:38.26 Preset successful but: 00:42:53.987 2009 setting none mode on left side
00:44:15.54 Preset failed...GCS died, GUIs turned red
00:44:25.723 2009 Preset alert for left LUCIFER mode ACTIVE: Invalid buffer: RPCERROR
00:44:25 lbtmu107 LBT_networkserver: GCSL died unexpectedly
00:46:53.945 Preset failed, no guide star found, but there is one there. See #000013. Star is 80 peak counts above background.
00:50:21.280 Preset to new target, EL~45. Failed, no guide star found but one there, see #00014. Again, star is 80 peak counts above the background.
00:55:30.692 Preset to same target, different guide star. Failed, no guide star found but one there, see #00015. Star is only 40 peak counts above background.
00:57:59.399 Preset to new target with brighter guide stars, EL~50. Chase failed to make tolerance. Collimate & retry.
00:59:47.360 Preset to brighter guide star, successful. First WFS image taking a long time to come through.
Feb 14 01:02:00 lbtmu107 LBT_left_GCS: WFS: error: triggerWFS: WFS image didn't arrive within timeout of 37716 ms.
Feb 14 01:03:35 lbtmu107 LBT_left_GCS: WFS: error: triggerWFS: WFS image didn't arrive within timeout of 37716 ms.
Feb 14 01:05:04 lbtmu107 LBT_left_GCS: WFS: error: triggerWFS: WFS image didn't arrive within timeout of 37716 ms.
Feb 14 01:06:36 lbtmu107 LBT_left_GCS: WFS: error: triggerWFS: WFS image didn't arrive within timeout of 37716 ms.
GCS restarted to clear the AZCam communication error to WFS. TCS log says GCSL still running when restarting, but we verified on 107 that it was not.
01:12:26.400 Preset successful.
Wind rising and we are seeing wind buffeting of the secondary. PVT is reading +/-0.01 and drive torques look normal.
Wayne checking direction and speed by hand. Mirror is failing to make chase and it is not recovering with multiple collimates. Wayne reports the wind is mostly from the west and in the parking lot is ~13m/s.
01:25:53.170 Preset to new source to east to try to face more away from the wind. M1 failed to make chase and did not recover until the 5th try. Re-preset again, failed to receive acquisition image within timeout period (01:32:49.677).
01:33:46 Restarted GCSL to clear AZCam problem. 01:33:48 lbtmu107 LBT_networkserver: GCSL died unexpectedly
01:37:22.200 Preset failed, M1 failed to make chase. Wayne "The Collimator" Wack keeping busy tonight.
01:39:47.653 Preset, chase failed.
01:43:14.781 Preset EL~55 successful. Recovered star at right edge of acquisition image. We had not collimated for a while, first WFS ~3100nm, second 1445, 588, 625 (mostly Z4), 297. Correct pointing
01:53:05.310 Preset TRACK mode failed: parkProbe() command failed. Restarting AGW and homing the motors.
01:57:20.012 Preset failed: Chase failed to make tolerance
02:01:08.510 Preset TRACK mode: Chase failed to make tolerance. After collimating, we measured the pointing offset anyway. IE now -6.26 CA now -32.84
02:08:34.308 Preset failed: Chase failed to make tolerance.
02:24:09 Switched AIP_L.cfg internal_transformation to false to use the new AGW transformation from Jesper.
02:28:02.40 Preset, same source but different PA failed: because Chase state not IDLE. There were a couple of presets in rapid succession just before this that failed.
02:30:55.386 Preset again, Failed because Chase failed to make tolerance
02:36:20.180 Preset again after collimation re-established. Successful. Same star at a different PA - the star was in exactly the same place on the acquisition image! See acquire000022 and 23.
02:38:18.382 Preset: Failed because Chase failed to make tolerance
02:40:24.563 Preset: Failed because Chase failed to make tolerance
02:43:58.762 Preset: successful!
02:45:34.697 Preset: Failed because Chase state not IDLE
02:48:43.438 Preset: Failed because Chase failed to make tolerance
02:52:08.920 Preset: Failed because Chase failed to make tolerance
02:56:32.77 Preset: Failed because Chase failed to make tolerance
03:00:55.325 Preset: Failed because Chase failed to make tolerance
OK, desperation sinks in - we restart PSFL even though this should not work. The Collimator says: "When all else fails, resort to superstition." The active optics were cleared when we did this, so the mirror had to move a long way, but it got to position OK.
03:04:50.502 Preset successful.
03:07:19.861 Preset: successful. Collimated back to <200nm in a few cycles.
03:14:56.969 Starting a dithering script to see how it goes. Offset failed because Chase failed to make tolerance
03:17:17.461 Offset Failed because Chase failed to make tolerance
03:22:42.967 Preset to new field EL~60 successful!
03:25:08.180 I noticed on the IIF GUI that the WFS is showing "paused" but GCS GUI shows it is running (so does getStatus) and the WFS is updating. Is this a problem on the IIF?
I also noticed throughout the night that the initial guider image was consistently off by ~2 arcsec or so on pretty much every acquisition. I do not recall that happening before, but perhaps the initial images were not displayed before.
03:31:52.411 Offset started: Failed because Chase failed to make tolerance
03:40:08.871 Offset started: from IIF log:
Sat Feb 14 03:41:10.096 2009 Offset alert for left LUCIFER: couldn''t move probe to position (SFP) -31.7928, 127.687, focus: 22.4958 :AGw command status: timed out waiting for probe to reach destination
Sat Feb 14 03:41:10.149 2009 Offset alert for left LUCIFER: couldn''t move probe to position (SFP) -31.7928, 127.687, focus: 22.4958 :AGw command status: timed out waiting for probe to reach destinationerror: setProbePosition: couldn''t move probe to position (SFP) (x,y,z): -31.7928, 127.687, 22.4958 mm.
Sat Feb 14 03:41:10.196 2009 Offset alert for left LUCIFER: oacontrol indicated an error moving to new position
Sat Feb 14 03:41:10.226 2009 Offset alert for left LUCIFER: timed out waiting for probe to reach destination
Sat Feb 14 03:41:10.259 2009 Offset alert for left LUCIFER: Ok
Sat Feb 14 03:41:10.287 2009 Offset command complete with warnings for left LUCIFER
and from GCS log:
Feb 14 03:40:09 lbtmu107 LBT_left_GCS: AGWUnitAIP: SFP position to move to: -31.792840, 127.686983 mm
Feb 14 03:40:09 lbtmu107 LBT_left_GCS: AGWUnitAIP: rotation of the AGW coords system in relation to SFP: 179.70 degrees
Feb 14 03:40:09 lbtmu107 LBT_left_GCS: AGWUnitAIP: internal transformation to AGW native coords: 31.123833, 484.348300 mm
Feb 14 03:40:09 lbtmu107 LBT_left_GCS: AGWUnitAIP: external transformation to AGW native coords: 32.453145, 485.692464 mm (oacontrol)
Feb 14 03:40:09 lbtmu107 LBT_left_GCS: AGWUnitAIP: moving probe to native coords: 32.453145, 485.692464 mm using external transformations
Feb 14 03:40:09 lbtmu107 LBT_left_GCS: AGWUnitAIP::doSetProbePosition: calling oacontrol::setxy(10.144.0.64, 0, x:32.453145 y:485.692464, 23.000000, 0) ...
Feb 14 03:40:15 lbtmu107 LBT_left_GCS: Setting up block for handle 3204: block:7:1
Feb 14 03:41:10 lbtmu107 LBT_left_GCS: AGWUnit: cancelling worker thread: -1680524368
Feb 14 03:41:10 lbtmu107 LBT_left_GCS: AGWUnit: cancellation succeeded.
Feb 14 03:41:10 lbtmu107 LBT_left_GCS: error: setProbePosition: couldn't move probe to position (SFP) (x,y,z): -31.792840, 127.686983, 22.495842 mm.
These look like reasonable coordinates to me, why did it not go there?
03:49:06.093 Preset successful
03:53:07.125 Offset: Failed because Chase failed to make tolerance
Winds have been staying a solid 15-17m/s. Tair=-9.1, Tmirror=-7.75 RH=52.1% Dewpt=-17.2
04:00:06.081 Preset successful. Star on acquisition frame close to upper right corner.
04:03:11.641 Offset failed, couldn''t move probe to position (SFP) 4.26277, 236.114
These should be warnings for now, the assumption being that the astronomer meant to do that.
04:06:14.683 Preset failed: because Chase state not IDLE
Pointing update: IE changed from, -6.26 to -18.02, CA from -32.84 to -37.52
14 04:12:49.070 Preset started. Hung: the rotator faulted, again when Wayne tried to ready it. Bus voltages on GUI are red. Drive fault for the cable chain. Cable chain drive 0 (zero) is complaining about a "heatsink temperature". The preset timed out in 300 seconds for left LUCIFER mode ACTIVE from IIF. Wayne cycled power and now the chain is not able to achieve a "ready" state. Telescope went into a "holding" mode when the cable chain power was cycled.
Waking JRousseau in the hopes he will have some better insight. We cycled power up in the right treehouse. Jon said that the problem looks to be in the same unit as before, with the module showing F06 and then n32 errors.
04:50 Wayne reports that the rotator still will not power up. On the GUI we see: "Error: rotator is faulted or busy turning on or off."
04:57 Calling JHill. He talked us through trying a couple things from old ITs, but they did not work so we called DAshby. He said the drive module needs to be swapped out. Since it is 5:15am we are shutting down in favor of a more careful fix by Jon and Jay in the daytime. End of night 5:30am.
- 14 Feb 2009