UT 2008 December 07
Observers: N.Ageorges, W.Seifert, M.Juette, F.Eisenhauer several others from LUCIFER team. D.Thompson (@LBT)
Telescope Operator: D.Gonzalez-Huerta
SW Support: M. De La Pena in Tucson until 11:20, plus a phone call a bit later.
Data will eventually be copied to /Repository/LUCIFER_Data/20081206/
HW Issues during the night
- AGW WFS blockage cleared (IT#1883 closed)
- Smoother startup (and night). Telescope ready at 4:53
isCollimated logic agreed with LUCIFER Team (see below)
- Request for
absorb_to_base IIF command
- AGW-PCS transform adjusted (179.7 degree rotation) but note that there
is an indication that the AGw axes are non-orthogonal (~0.15 degrees).
- More offset, scale tests, NB filter zero points
- Attempted focus again, seeing too variable for solution
- Diagnosed continuing image motion on WFS collimation, solution suggested.
: none (IT#1883 closed)
TCS/SW Issues during the night
(2h lost, most of the time spent understanding the issue)
- GCS hang (only GCS restarted): 19:48:53
- OSS died:16:37:05, 20:12:19
- IRS/IIF restart: 00:11:49.792, 23:58:45
HW change from last night: obstructing piece removed from WFS light path. IT#1883 closed.
SW change from last night: Jose fixed bug in IRS multi-client.
- 3:40 Started walkthrough, telescope checkout.
- Encoders reset
- 16:03:40 IRS restarted (Jose fixed the multi-client bug)
- 16:00 LBC Blue filled with LN2
- 16:34:07 Test preset completed successfully.
- 16:35:34 AGW stopped, restarted, and motors homed
- 16:37:05 Left M2 PSF GUI turned red: OSS died unexpectedly. We did restart PSFL before realizing it was OSS.
- 4:53 Telescope at zenith, turned over to LUCIFER For sky flats
- GCS-AGW communications should not require restarting AGW and homing motors, only a GCS restart.
17:21:31 First preset from LUCIFER sent in TRACK mode; completed successfully.
17:38:35 Preset TRACK/PARALLACTIC to adjust pointing
- IE changed from -72.5 to -9.9, CA changed from 6.75 to 4.35 from end of last night
M1 temp 1.9C, outside temp 4.6C, RH 7%, wind 8m/s (SMT 4m/s)
17:53:30 Preset to first program field, done. Guider sees 1.4 arcsec FWHM, WFS working well now. Despite being ~5 degrees warmer than yesterday (0C), the initial collimation tonight had much lower coma than yesterday's startup. LUCIFER continuing with offset tests.
19:19:16.413 Preset failure for left LUCIFER mode ACTIVE: couldn''t find a star in any GS location (acquire000117)
Guide star present at (186,277) and ~100 counts above background. There is a brighter cosmic ray hit at (58,232), but the logs look like nothing was detected.
NOTE: It was decided in a meeting on Dec 10, 2008 between Chris B., Norm C., Torsten L., Dave T., and Doug M. that the GuidingFlag should not be examined by any instrument as the flag is an internal GCS flag and only has context when utilized by the GCS. Rather the TCS will be modified so that a preset and offset will delay returning to the instrument until after the GCS has verified the guidestar is within a configurable distance from the hot spot for a minimum of 2 (configurable) guide adjustments. If after 5 (configurable) guide adjustments the guidestar is still outside the configurable distance the preset or offset will fail.
flag logic (embedded in acquisition sequence):
- Unset GuidingFlag on all preset and offset commands
- Take acquisition image
- ID guide star, issue pointing correction offset to mount to move to WFS hotspot
- Guide loop exposure time scaling/threshholding logic goes here
- Start guide loop
- Take guide exposure
- Monitor GS flux, etc...
- If (GS-hotspot) > 0.54 arcsec (10 pixels):
issue guide offset, set internal counter to zero, repeat loop
- If (GS-hotspot) < 0.54 arcsec (10 pixels) once:
issue guide offset, set internal counter to 1, repeat loop
- If (GS-hotspot) < 0.54 arcsec (10 pixels) a second consecutive time:
issue guide offset, set GuidingFlag, set internal counter back to zero, repeat loop
The request is for two consecutive guide frames that are within the 10 pixel limit from the hotspot. We suggest this be parameterized in the config file so it can be adjusted later as needed.
NOTE: It was decided in a meeting on Dec 10, 2008 between Chris B., Norm C., Torsten L., Dave T., and Doug M. that the following algorithm (modified from the original description) is what will be implemented.
- On all presets and offsets
- Clear isCollimated flag
- Leave other values in place
- For each new WFS cycle at this position:
- Set isCollimated if Zernikes.RMS is less than 400nm
- Unset isCollimated if Zernikes.RMS is more than 400nm
- Update Zernikes.RMS and Zernikes.lastUpdate values
The threshhold for setting the isCollimated flag (the 400nm above) should also be a config file paramerter so it can be easily changed if necessary (note: apparently it already is in AIP_L.cfg).
19:48:53 GCS appears to have hung. Telescope and probe are in the correct position, but the guiding thread failed to resume.
From the 107 log:
Dec 6 19:48:53 lbtmu107 LBT_left_GCS: offsetGuiding succeeded, new position reached.
Dec 6 19:48:53 lbtmu107 LBT_left_GCS: GCSL:offsetGuiding execution time: 9949 ms at 1228618133836
Dec 6 19:48:53 lbtmu107 LBT_left_GCS: Invalid handle in blockstub: ...(odd characters removed)
Dec 6 19:48:53 lbtmu107 LBT_left_GCS: Invalid handle in blockstub: ...(odd characters removed)
We tried stop
from the TCSGUI but the process remained on 107:
1033 16433 1.1 5.8 1281748 60204 ? Sl 04:01 11:03 GCSL
So I issued a kill -9 16433. GCSGUI immediately turned red and the LUCIFER connection resumed with an "invalid buffer RPCERROR" but everything works.
20:12:19 OSS died unexpectedly. LUCIFER was executing a dither script and issued a step focus while OSS was in the middle of starting up.
Dec 6 20:12:19 lbtmu105 LBT_networkserver: OSS died unexpectedly
Dec 6 20:12:19 lbtmu105 LBT_networkserver: OSS is not active
Dec 6 20:12:19 lbtmu105 LBT_networkserver: dropping OSS
Dec 6 20:15:26 lbtmu105 LBT_networkserver: OSS starting on lbtmu105
Dec 6 20:15:26 lbtmu105 LBT_networkserver: OSS started
Dec 6 20:15:27 lbtmu105 LBT_networkserver: Server 5/6: name lbtmu105, load 24.000000, subsystem count 2 ( OSS PMCR ), is active true, last update 1228619726474
Dec 6 20:15:27 lbtmu105 LBT_networkserver: update 1228619727436 lbtmu105 1 10.144.0.105 10.10.0.105 10.30.0.105 5 36 2 OSS 0 PMCR 1
Dec 6 20:15:28 lbtmu105 LBT_IIF_LUCIFER_left: OSS not running
21:17:17.396 Preset to astrometric field for scale/orientation tests.
Considerable discussions with LUCIFER Team and Michele about Guide and Collimated flags (see above) as well as implementation of dithering schemes for spectroscopy and how to implement them.
Official request from LUCIFER Team to provide absorb to base command through the IIF
23:58:45 We are having unexpected behavior. The multiparameter calls from LUCIFER as showing up in the IRS log every 15-20 seconds. The result of their offset request did not come back although the completion of the offset is noted in the IIFGUI at 23:52:02.530. I did a top
on 105 and found PMCR taking up 40+% of the CPU. The load average is also high (up to 8 on the first number). Michele called.
4415 telescop 15 0 337m 29m 6044 S 40.6 2.9 2079:49 PMCR
Sun Dec 7 00:11:49.792 2008 subsystem started (stopped IRS, stopped IIF, start IIF, start IRS). Still the polling from LUCIFER looks odd, coming through in blocks every ~10 seconds rather than once per second. Now it looks OK. Michele suggested we continue with observing while she watches the logs.
00:24:16 The SFP_rotation parameter was changed to 179.7.
Walter checked some data from eariler and found a rotation of 0.3 +/- 0.05 degrees. One quick measurement with LUCIFER shows we went the right way, but it is still off by 0.04 degrees.
01:03:13.057 2008 Preset command failed for lack of guide star in acquisition image - need to check/correct the pointing.
IE changed from -9.9 to -28.6, CA changed from 4.35 to -1.65.
01:13:37.472 Preset ACTIVE after correcting pointing...successful. First WFS above 900nm, second below 300. Guider shows 0.45 arcsec FWHM.
Start focus sequence. AIP_L.cfg shows focus=31.6mm
- Data: luci_20081206_ 15x2s exposures saved integrated
| LUCI# || Focus || FWHM |
| 0375 || 25.121 || 5.15 |
| 0376 || 19.122 || 7.16 |
| 0377 || 22.121 || 5.48 |
| 0378 || 25.122 || 4.26 |
| 0379 || 28.121 || 4.59 |
| 0380 || 31.122 || 5.11 |
| 0381 || 25.121 || 5.47 |
- Results: Seeing too variable, not a clean fit.
02:37:17.746 Check if the AGW-PCS transform is OK.
03:07:38.115 Offset command completed, but the resumption of guiding appeared to spiral in to the hotspot in a few iterations.
03:12:43.239 Preset failed, guide star is at the right edge of the acquisition image (acquire000134), but this is at the same PA as the prior preset and the guide star was much closer to the hotspot.
03:40:48.230 Preset completed.
Check of functionality of secondary offload
Image motion on the guider images during WFS collimation updates continue to be seen. To check this we did the following...preset to a field in ACTIVE mode toi establish the starting position, then monitor the M1 total collimation values during a number of WFS cycles. Coincidentally, we went through a zenith passage (at ~88.5 degrees) during this test that helped to figure out what is going on. The telescope was in "default" mode where coma and focus are sent to the secondary. The data are:
04:05:36.604 preset to current position
M1 total collimation (the last column is the rms WFE for that cycle):
-0.929 0.661 -0.821 -15.751 -0.769
-0.937 0.670 -0.825 -15.790 -0.685 133.36
-0.937 0.670 -0.825 -15.790 -0.685 132.69
-0.953 0.688 -0.838 -15.868 -0.516 300.59
-0.961 0.697 -0.848 -15.906 -0.434 235.21
-0.968 0.705 -0.857 -15.942 -0.358 213.20
-0.975 0.713 -0.865 -15.977 -0.284 228.65
-0.980 0.719 -0.867 -16.005 -0.225 249.98
-0.985 0.725 -0.874 -16.029 -0.174 253.21
-0.989 0.729 -0.878 -16.048 -0.134 233.86
-0.991 0.732 -0.879 -16.060 -0.109 259.63
-0.991 0.732 -0.879 -16.060 -0.109 234.07 note: zenith passage around here
-0.992 0.733 -0.887 -16.063 -0.102 281.53
So the primary does not
strictly stay stationary during the WFS cycles and this can be responsible for the image motion seen in LUCIFER images and on the guider. I suspect that every WFS cycle triggers an update from the lookup table line on PSFL that does go to the primary, causing the image motion. Note that there are two lines with identical M1 positions around the zenith passage, when there was no change in EL and thus no change to the M1 position, which suggested the lookup table as the source of motion.
We noted that there is a "disable" button under the lookup table line on PSFL, but did not get a chance to test this. If disabled, this would force the WFS to detect any collimation offsets coming from the collimation model or temperature effects. The solution would be to send all M1 lookup table and temperature corrections also to the secondary during science observations, and continue the offload back to the primary on presets and offsets. This should make a significant improvement in the stability of the images on LUCIFER.
Continued with LUCIFER tests (offsets, calibrations).
06:40 slew to zenith in the west for sky flats.
- 05 Dec 2008