LUCIFER Com 3, Night 4
UT 2009 January 13
Observers: N.Ageorges, W.Seifert, M.Juette, V.Knierim; D.Thompson (@LBT), JHill in Tucson at the beginning
Telescope Operator: D.Gonzalez-Huerta
SW Support: T.Sargent in Tucson until 2:00am
- New AGW transformation updated, results will be logged but not used.
- PCS patch to fix REL RADEC offsets implemented, verified to be OK
- LUCIFER switch problems delayed start by 1:15.
- TCS restart: IRS-IIF-GCS to clear persistent communication problem (see IT#1945)
- Cable chain error on the rotator. We recovered OK, but see IT#1954
17:15 Open and ready. We are at horizon with the dome open. The drives are off to prevent accidental slews.
Diagnosing a switch problem in LUCIFER. Peter called. Problem at least solved for the current time. 1:15 lost.
Mon Jan 12 19:16:30.184 2009 Preset sent from LUCIFER.
Mon Jan 12 19:22:07.973 2009 subsystem shut down. We stopped IIF to clear out the memory leak, stopped and restarted GCS, and then IIF. We are having again (or still) the problem ty
Mon Jan 12 19:52:54.011 2009 We stopped GCS, then killed it. It looked like the request came through but things got stuck at the point where GCS was supposed to tell the AGW to park the probe. Tom noted "invalid handle in blockstub" messsages in the syslog just before we stopped GCS.
20:00 IE changed from -5.32 to -10.12, CA from 1.84 to -1.96
20:14 LUCIFER seeing donuts while the AGW is in focus. It looks like this is internal.
20:16 Tout=1.1 and rising. T(M1)=0.7 RH=23.3 Wind is about 11m/s. There were thin clouds around at sunset.
In the last acquisition image (#0047) that immediately followed a pointing correction, the guide star was offset in X only (see attached image). The GCS reported in the logs the following stage coordinates:
Jan 12 20:02:21 lbtmu107 LBT_left_GCS: AGWUnitAIP: internal transformation to AGW native coords: -27.100503, 462.438663 mm
Jan 12 20:02:21 lbtmu107 LBT_left_GCS: AGWUnitAIP: external transformation to AGW native coords: -25.047492, 462.394739 mm (oacontrol)
Note that X differs by ~2mm but Y agrees closely. This has been consistent over several previous presets as well. The star is offset from the hotspot by closer to 7 arcsec (4mm). In the image below, the green circle marks the location of the WFS hotspot.
Mon Jan 12 20:47:43.643 2009 Offset command did not return to LUCIFER for more than 5 minutes. The GCS appears to be cycling correctly, but does not see a star on the images. Verified that the guider images are indeed empty. It should timeout if the guide star is accessible (i.e. should be in the patrol field) but lost or not seen. GCS stopped at 20:56:49 and restarted. Note added to IT#1945.
20:59:08.10 New preset from LUCIFER. The acquisition (#0048) is about the same as before, oac coords offset by 2mm, guide star offset from hotspot by about 4mm:
Jan 12 20:59:16 lbtmu107 LBT_left_GCS: AGWUnitAIP: internal transformation to AGW native coords: -27.040681, 462.487127 mm
Jan 12 20:59:16 lbtmu107 LBT_left_GCS: AGWUnitAIP: external transformation to AGW native coords: -24.987537, 462.443133 mm (oacontrol)
Mon Jan 12 23:15:15.657 2009 A stepfocus completed on the LUCIFER side, but the query they send for telescope parameters that immediately follows did not return. The IIF syslog entries for the DD queries stopped at 23:13:32 (also the time LUCIFER's UI stopped updating). However, after
this time the IIF shows an offset and a stepfocus that came in and completed without problem:
Mon Jan 12 23:14:29.233 2009 Offset command started for left LUCIFER from IIF
Mon Jan 12 23:14:58.903 2009 Offset command complete for left LUCIFER from IIF
Mon Jan 12 23:14:58.956 2009 StepFocus command started for left LUCIFER for 0.000000000000000e+00
Mon Jan 12 23:15:15.657 2009 StepFocus command complete for left LUCIFER for 0.000000000000000e+00
But we are not sure where this came from. We then issued a stepfocus from iraf at 23:31:24.413 that completed. An initial getparameter request from iraf failed, but two subsequent requests worked. LUCIFER restarted their telescope service, no changes were done on the TCS side.
23:38:29 the LUCIFER getmultiparam requests again stopped. I again tried requesting DD data through iraf and it worked. LUCIFER's telescope service restarted.
23:48:41.069 2009 Stepfocus command complete for left LUCIFER logged on IIFGUI.
23:56:35 LUCIFER finally received the return from IRS. Why did it take so long to go through.
Tue Jan 13 00:00:08.863 2009 subsystem shut down. I stopped IIF and when I restarted GCS the IIF also restarted?
00:04 We did the following shutdown and restart sequence: stop LUCIFER's telescope service, IRS, IIF, GCS, start GCS, IIF, IRS, LUCIFER's telescope service. We noticed that when LUCIFER sent its first preset after this, it went through without problems. Prior to that during this run they have been seeing the first (and second?) request return an "invalid buffer: error: no such handle" message before subsequent commands went through. Perhaps this is a symptom of an impending problem in IRS? That is the last place to look, since we have been restarting IIF and GCS over the last couple of nights but have not done anything to IRS. Note added to Issuetrak #1945.
Note that at the beginning of the night (after restarting IIF and GCS), LUCIFER's first IIF request tonight saw the same error, at UT 02:24:39 (19:24:39 local).
00:27:38 Cable chain error on the rotator. An "rcready lfbg" request on MCSPU said "rotator is faulted or busy turning on or off". On the MCSGUI rotator page clicked "on" to restore the power and this fixed the rotator. Tom said it looks like a servo reset. ISSUETRAK #1954
Could this be related to the rotator work done the other night? Tom looked through the logs and said the cause was that the signal "CBOK" (cable chain bus power OK) became false, so the system faulted/shut down.
Remainder of the night was pretty quiet.
- 13 Jan 2009