Lucifer February Com Night 7 February 12-13 2009
Observers: N.Ageorges, W.Seifert, V.Knierem (LUCIFER), DThompson (@LBT)
Telescope Operator: W.Wack
SW Support: T.Edgin until 1 am, on call thereafter.
Telescope Support: J.Rousseau
- Overall poor seeing.
- Left door latch pin "failure to retract"
- Very strong oscillations seen when facing directly into 10m/s wind
- Overall guiding quite jumpy, frequently landing outside the D=1.08 arcsec circle.
- Something on AGW changed, coords from external transform now flipped in X
- Hardpoint 3 semi-failure (see CRITICAL IT#2011, but also see notes below) - Lost ~5 hours
Open before sunset to clear skies, but windy.
None of the SW changes below went in. Probably not until B21, March7.
Closed 7:08 for high winds (gusts to 25m/s). Tair=-6.2C Tmirror-2.8C RH=46.3% Dewpt = -16.6.
21:52 Re-opening, winds below 20m/s for the last 45 minutes. Tair=-7.6 RH=50.6 Dewpt=-17.0.
There was a latch-pin "failure to retract" error (BD07-04) on the left door. JRousseau awakened, JLittle called
22:35 Fixed, re-opening again (seemed to be OK except for a limit switch?).
We found that the "external" transformation numbers in the GCS syslog entries has the opposite sign in X from the internal numbers, so I switched back to using the internal transformations. Something changed on the mountain, as this was working fine yesterday.
04:12:21 lbtmu107 LBT_left_GCS: AGWUnitAIP: internal transformation to AGW native coords: 49.376779, 457.065928 mm
04:12:21 lbtmu107 LBT_left_GCS: AGWUnitAIP: external transformation to AGW native coords: -48.514097, 458.001210 mm
23:11 Repeated "hardpoint 3 is not showing any motion" warnings followed by "failed to make chase"
23:26 We are seeing very large jumps and strong vibrations. Telescope is pointing almost directly into the wind, with 10m/s average and gusts to 15m/s. See wfscimage000025.fits (jpeg attached here). Later (05:12) we were again pointing directly into the wind (6m/s with gusts to 10) and did not see any evidence of vibrations.
- wfscimage000025 from UT20090212:
23:45 Repeated "hardpoint 3 is not showing any motion" warnings followed by "failed to make chase". This is causing the presets to fail and not initiating the guiding thread. John suggests clicking "collimate" until M1 makes chase and then re-sending the preset, but M1 may do this.
00:07 Lowering M1 to cycle power on the motor controller. Same hardpoint 3 messages for a bit, but then the mirror appears to have successfully lowered itself.
01:20 Waking Jon Rousseau. JHill would like us to reseat the motor controller and check the stepper motor on hardpoint 3.
02:10 Things seem fine in the cell, but it now works. Back to LUCIFER...nope. The problems continue. We can get M1 collimated eventually by repeatedly clicking "collimate" on the PSF GUI. But then sending another preset results in the hardpoint 3 warnings and M1 failing to make chase. Guiding and WFSing seem to start and run, but with M1 not making chase. IT# submitted 2011.
Sometime after this we tried acquiring a star for a pointing correction at EL~43 and it was not in the LUCIFER field of view. At this point we re-loaded the pointing model and cleared all active optics and offloads on PSF for both mirrors. Following preset was found to be in decent collimation and the pointing off by ~40 arcseconds.
03:52 Now GCS is repeatedly failing to receive the acquisition image. It is trying 5 times. We restarted GCSL.
I mis-remembered something I read in IT#494 earlier tonight and we restarted PSFL but did not need to. IT#494 describes something similar to what we are seeing tonight, and it says to restart PMCL. We did not do this because LUCIFER was trying to take data when we realized my mistake and were not sure what the consequences would be in restarting PMCL with the mirror up.
04:09 Wayne reports that collimation "seems better" (making chase more frequently). Could the restart of PSFL have done this?
It has run now for an hour without missing a collimation cycle. Could this be a symptom of the need to periodically restart some/all of TCS?
04:30 /Repository disk is over 91% full. Tony earlier advised that over 90% could cause some of the slowdown issues that we have seen so I deleted all October LBC data from /Repository and gzipped some of the IRTC data. A df now shows /Repository at 413GB free, 85% full. This will rapidly become a problem, as LBC can generate 60GB/night, LUCIFER a few GB/night.
Guiding looks very jumpy, even in 2 arcsec seeing.
05:28 Failure to find guide star in acquire000016.fits, but there are 3 usable stars in the image.
Lost WFSing to the GCS-AZCam communication issue.
Feb 13 06:12:40 lbtmu107 LBT_left_GCS: WFS: error: triggerWFS: WFS image didn't arrive within timeout of 37716 ms.
Feb 13 06:14:10 lbtmu107 LBT_left_GCS: WFS: error: triggerWFS: WFS image didn't arrive within timeout of 37716 ms.
Feb 13 06:15:42 lbtmu107 LBT_left_GCS: WFS: error: triggerWFS: WFS image didn't arrive within timeout of 37716 ms.
Feb 13 06:17:13 lbtmu107 LBT_left_GCS: WFS: error: triggerWFS: WFS image didn't arrive within timeout of 37716 ms.
Stopped and restarted GCS only. Re-sent preset.
06:29:54 Warning from GCS that the probe timed out on its motion. I have not seen this before, what does it mean?
06:34 guiding lost - sky is getting too bright for the WFS.
06:59 Closing dome. Tair=-6.6 RH=6.6% Dewpt=-36.9C Wind 10m/s, gusts to 17m/s
Implemented TCS Changes that need the following tests:
- AGW Transformation upgrade
- Check internal_transformation flag in AIP_L.cfg
- Run closed-dome presets and see if the internal and new external transformations agree
- If so, go with Jesper's new transformation
- New IRS incorporating UpdatePointingReference() command
- Test with iraf script in the afternoon
- Test with new LUCIFER build
- Test PSF patch that removed problem when presetting from horizon
- Preset telescope, and note the hopefully modest amount of Secondary Offload on the PSF Primary GUI.
- Apply some active optics corrections to M2 (either by really collimating on a star, or by sending small amounts of Z4 or Z7 manually on the PSF Secondary GUI if dome is closed). If there are big temperature gradients in the telescope, you might have to send a stepFocus to offload most of the active correction before doing the test.
- Have operator drive the telescope to horizon
- Issue another preset to a different target.
- Observe the amount of Secondary Offload on the PSF primary GUI. If it is only slightly (<0.5 mm) different than step 1) then test has succeeded. If the amount of Secondary Offload has increased alot (2-3 mm), then the test has failed.
- 12 Feb 2009
- wfscimage000025 from UT20090212: