AGw Commissioning with LUCIFER 13-14 October 2008

Observers: J. Hill , with LUCIFER folks (N.Ageorges,W.Seifert, et al) and DIMM folks (J.Schmidt,T.Herbst)
Software Support: T. Sargent and N. Cushing in Tucson
Telescope Operators: A. Ceranski

Summary

The evening started at 5PM with the telescope facing east for a National Geographic photo shoot. We took some astrometric data on M92. We took some 180x1 sec data cubes to study the jitter problem. We worked on tracking down some bugs that are causing GCS to hang at various times. The challenge is that these bugs are rare, so it is hard to pin down what causes them. Had one crash of Jet on MCSPU. Had one HBS alarm shutdown. Seeing was approaching 1 arcsec at 10 PM, but then got worse after that. Conditions were too poor to do field alignment data. We spent the rest of the night collection some 0 degC collimation data, making some DIMM co-pointing tests, and taking practice field alignment data in poor seeing. Nancy took some LUCIFER sky flats at the end.

Details

Times reported in UT below.

00:00 Open for National Geographic photo shoot. Sky is clear. T=1degC, D=-26degC, Wind 10 m/sec.

01:00 Restarted GCS to get the latest version (1.9.40) from Torsten.

01:35 Lights back on for the photo guys to pack up.

01:42 Preset to the alignment star

01:47 Apply -5 arcsec Tip to M1 to get us off the +Y limit.

01:51 Adjust IE=-159.2 CA=+50, but star was drifting away (at rates of arcmin/minute) at EL=89 deg

01:55 Preset to another star at lower elevation, IE=+25.655 CA=6.75 Why is IE shifted the opposite direction from last night?

02:08 Preset to BS9169 on-axis. GCS died unexpectedly, and repeatedly.

Oct 13 19:12:37 lbtmu107 LBT_left_GCS: GCSL:stopGuiding execution time: 27 ms at 1223950357609
Oct 13 19:12:38 lbtmu107 LBT_left_GCS: GCSL:setGuidestarList executed
Oct 13 19:12:45 lbtmu107 LBT_Networkserver: GCSL died unexpectedly
Oct 13 19:12:45 lbtmu107 LBT_Networkserver: GCSL is not active
Oct 13 19:12:45 lbtmu107 LBT_Networkserver: dropping GCSL

02:15 Rolling back to yesterday's version of GCS. Changing the symbolic link of GCSL to GCS.08101301

02:22 Preset again.

02:30 Photographers are gone and the foot lights are off. Active Optics is slowly converging, but having some trouble dialing out the strong Z7 term.

02:40 Got almost collimated, and then ran into -Y limit at EL=46 degrees, so the collimation correction ran away. Clear Active Optics and remove the -5 arcsec Tip that we installed earlier.

I think the strong Z7 correction means that our M1 collimation model made at +9 deg has inaccurate temperature corrections so we need this large initial correction for 1 degC. The challenge is to measure the temperature corrections over a span of weeks, and ideally to make them pointing free.

02:43 Restart GCSL to pick up some new exposure time parameters. I reduced the WFS low water mark from 1000 to 400 and perhaps it should be even lower to avoid 90sec WFS exposures.

02:43: Preset again to BS9169 on-axis. We added in a Yglobal=-0.6 mm to reduce the total Y needed for the collimation. This will require us to rezero the pointing. I think this means that the Y thermal term needs more work.

02:54 Preset in Track Parallactic mode to adjust pointing. CA=+6.75 IE=+6.3

M92

02:59 Preset to M92 with off-axis guide star, EL=+43 deg

03:03 Nancy adjusts the guide star position to get more of M92 in the field. Guider thinks the seeing is 1.29 arcsec FWHM.

03:05 Starting a series of LUCIFER exposures on M92. The first group has WFS still running sending corrections to M1. Ambient Temperature and mirror temperature are fairly stable at 1 degC. I'm supposing that telescope temperature is still changing.

03:25 Jet crashes

03:42 Jet restarted after Aaron presses the reset button, but Nancy's old preset is still hung.

03:44 Update WFS hotspot in AIP_L.config and wfsc_init.pro per Doug's recommendation from the Denver airport.

WFScam_hotspot_x 388.0 double # ccd x coordinate of pixel that represents leftmost x-center of WFS spots
WFScam_hotspot_y 225.0 double # ccd y coordinate of pixel that represents leftmost y-center of WFS spots

Using agw_post_collimate on wfsc 0511 gives results consistent with those from the previous hotspot.

03:46 Aaron restarts IIF, but has to force it to run on lbtmu101 (this was an operational error but it was not caught until 04:14). Then IIF hangs when Aaron tries to authorize Lucifer. Norm has arrived to help debug this. Norm's Notes: IIF must always run on lbtmu102 as defined in the lbt.conf file. The instrument teams are aware of this constraint (LBC, Lucifer, DIMM, IRTC, Dave's iraf scripts, MAT).

04:10 Restart a bunch of subsystems PCS, IIF, OSS, etc. PSF was sending WFS to Default (M2) after this restart.

04:14 Determined the IRS was not running on any computer. No idea why it was not running. It should have been running on lbtmu105. This accounts for some of the communication problems between TCS and clients (instruments). Start IRS on lbtmu105 as root, "service irs start". Also, restarted IIF which was running on lbtmu101 to run on lbtmu102.

Lost the better part of an hour.

04:18 pointto BS9188 Active. Updated Yglobal to -0.8 mm.

04:27 Active Optics had runaway astigmatism, so we clear C00. Clearing C00 seems to be a very effective fix when the image is highly elongated on the guider. Collimated here at BS9188 through wfsc 0520.

04:32 Nancy preset to another field - StoneP astrom. Recentering IE=-1.57

04:42 Move off-axis

04:47 start astrometry data with WFS running with 30 sec exposures.

05:05- Lost the guide star for unknown reason. Stop Guiding and WFS from GUI, then resend preset.

05:06:07 New Preset - Long time for GCS to get the guide image following the start of a new preset. What was it doing?

05:43 Struggling to do WFS with faint star and moonlight.

05:46 Preset failed, but the PA=0 didn't work because target was off the cluster center.

05:48 Preset on-axis

Oct 13 22:59:59 lbtmu107 LBT_left_GCS: PROFILING: WFS: triggerWFS() entered.
Oct 13 22:59:59 lbtmu107 LBT_left_GCS: PROFILING: WFS: requesting image...
Oct 13 22:59:59 lbtmu107 LBT_left_GCS: WFS: requesting a WFS image...
Oct 13 23:00:12 lbtmu107 LBT_left_GCS: WFS: timeout while waiting to issue a requestImage command. trying again...
Oct 13 23:00:13 lbtmu107 LBT_left_GCS: PROFILING: WFS: triggerWFS() entered.
Oct 13 23:00:13 lbtmu107 LBT_left_GCS: PROFILING: WFS: requesting image...
Oct 13 23:00:13 lbtmu107 LBT_left_GCS: WFS: requesting a WFS image...
Oct 13 23:00:26 lbtmu107 LBT_left_GCS: WFS: timeout while waiting to issue a requestImage command. trying again...
Oct 13 23:00:26 lbtmu107 LBT_left_GCS: PROFILING: WFS: triggerWFS() entered.

06:02 Stop guiding, stop WFS, stop and restart hung GCS. It required a couple kills to really kill it.

06:03 Resend preset

06:05 HBS alarm - Pad Pressure E22 - Aaron added this value to the force list in WinCC and we were able to continue. Lost 8 minutes.

Note that we still seem to have the hot return channel problem in the OCS. ECS is cooling the oil to 0 degC, but the oil going to the pads is 11 degC.

06:12 resend preset off-axis

Data cubes to study the jitter problem

06:16 Start 170s cube of 1 sec images with guiding and WFS running. Jump at time of updating WFS. Seeing falls apart, so this data cube isn't going to be worth much except looking for bad seeing. Didn't save the data.

06:21 Try another cube with guiding and wfs, seeing is a little better now. Wind seems to be increasing.

06:30 pauseGuiding (and WFS), and take another cube. Lots of apparently atmospheric focus variation in this one. The jitters are not caused by guiding or WFS, they exist in the open loop tracking.

06:34 resumeGuiding and WFS

06:36 take another cube with guiding and wfs.

06:38 Send Active Corrections to M1 and take another cube. We see the 1 arcsec jumps when M1 gets a collimation correction.

Guider thinks the seeing is 1.5 - 1.8 arcsec, and Tom/Johannes report 1.7 arcsec from the DIMM. Polaris monitor is varying between 1.5 and 2 arcsec. Unfortunately this is not good enough to do the field aberration measurements.

06:49 pauseGuiding, Clear Active Optics on M2, resumeGuiding so we are reapplying the coma correction to M1.

06:53 Marcus is testing the isCollimated flag with a new version of his LUCIFER software.

06:58 Marcus issues another preset. Star has an astigmatic tail.

07:00 Clear C00 on M1 and the astigmatic tail is gone.

23:58:59 GCS WFS gets stuck

Oct 13 23:58:56 lbtmu107 LBT_left_GCS: getWFImage: _wt.getLastWFSImage() returned NULL, no WFS image available
Oct 13 23:58:59 lbtmu107 LBT_left_GCS: WFS: resuming loop.
Oct 13 23:58:59 lbtmu107 LBT_left_GCS: PROFILING: WFS: triggerWFS() entered.
Oct 13 23:58:59 lbtmu107 LBT_left_GCS: PROFILING: WFS: requesting image...
Oct 13 23:58:59 lbtmu107 LBT_left_GCS: WFS: requesting a WFS image...
Oct 13 23:59:37 lbtmu107 LBT_left_GCS: WFS: timeout while waiting to issue a requestImage command. trying again...
Oct 13 23:59:37 lbtmu107 LBT_left_GCS: PROFILING: WFS: triggerWFS() entered.
Oct 13 23:59:37 lbtmu107 LBT_left_GCS: PROFILING: WFS: requesting image...
Oct 13 23:59:37 lbtmu107 LBT_left_GCS: WFS: requesting a WFS image...
Oct 14 00:00:16 lbtmu107 LBT_left_GCS: WFS: timeout while waiting to issue a requestImage command. trying again...
Oct 14 00:00:16 lbtmu107 LBT_left_GCS: PROFILING: WFS: triggerWFS() entered.
Oct 14 00:00:16 lbtmu107 LBT_left_GCS: PROFILING: WFS: requesting image...
Oct 14 00:00:16 lbtmu107 LBT_left_GCS: WFS: requesting a WFS image...

07:04 Abort preset, but didn't work. Kill and restart GCS

07:05 Marcus resends preset. GCS seems to wait before actually starting WFS image. What happened between 00:06:16 and 00:06:45?

Oct 14 00:06:15 lbtmu107 LBT_left_GCS: getWFImage: _wt.getLastWFSImage() returned NULL, no WFS image available
Oct 14 00:06:16 lbtmu107 LBT_left_GCS: WFS: resuming loop.
Oct 14 00:06:16 lbtmu107 LBT_left_GCS: PROFILING: WFS: triggerWFS() entered.
Oct 14 00:06:16 lbtmu107 LBT_left_GCS: PROFILING: WFS: requesting image...
Oct 14 00:06:16 lbtmu107 LBT_left_GCS: WFS: requesting a WFS image...
Oct 14 00:06:45 lbtmu107 LBT_left_GCS: WFS: intending to wait 37716 ms for the WFS image.

07:11 With DDViewer we observe the isCollimated bit to come on we got the RMS=385 nm.

07:12 LUCIFER sends preset again. isCollimated bit is still on. Is that the correct behaviour? Probably we should unset this bit when a preset is issued, or at least when a preset is issued to a different star.

07:15 Aaron sends Z4=+1000 to M1 to see if we can unset the collimation. The corrections seems to be split over several WFS corrections, so we never unset the bit.

07:18 Aaron sends Z4=+2000 and the isCollimated bit turns off after the next measurement.

07:28 The next time we sent Z4 +2000, the WFS never saw it and we had to send another 2000.

07:30 Marcus sends another preset. This time he sees the bit, but the software is hung.

07:38 SX M1 is 2degC when ambient is 1.5 degC since the Mirror Ventilation system has run out of headroom as the chilled water warms up. We stop DX M1 ventilation to unload the chiller a bit. Butch has worked on the chiller some more today and its drop from in to out is -2.5 degC.

Start collecting data for a Collimation Model while Marcus and Volker are working on their code to display isCollimated.

These numbers include Xglobal=-1 and Yglobal=-0.8 and Zglobal=+0.3.
Present Active Optics corrections are X=-1.358 Y=-2.110 Z=+0.214 RX=-22.7 RY=14.6

Xtot Ytot Ztot RXtot RYtot RZ EL Temp
Total -0.373 -1.165 +0.185 17.83 -2.12 0 61.6 2.02
Total -0.426 -1.266 +0.200 16.76 -1.43 61.3 1.98
Total -0.327 -1.281 +0.203 16.56 -2.26 0 60.7 1.88
Total -0.571 -1.008 +0.145 18.417 -0.115 0 63.6 1.62

GCS repeated one of the errors from last night.

Oct 14 00:49:18 lbtmu107 LBT_left_GCS: 1. reception of image failed to arrive within timeout period of 7659 ms, trying again in 4 seconds...
Oct 14 00:49:18 lbtmu107 LBT_left_GCS: autoAcquire(): waiting 4716 ms for IST to finish receiving image.
Oct 14 00:49:23 lbtmu107 LBT_left_GCS: autoAcquire(): waiting 4716 ms for IST to finish receiving image.
Oct 14 00:49:27 lbtmu107 LBT_left_GCS: autoAcquire(): waiting 4716 ms for IST to finish receiving image.
Oct 14 00:49:32 lbtmu107 LBT_left_GCS: autoAcquire(): waiting 4716 ms for IST to finish receiving image.
Oct 14 00:49:37 lbtmu107 LBT_left_GCS: PROFILING: full readout received.
Oct 14 00:49:37 lbtmu107 LBT_left_GCS: full CCD read out received.
Oct 14 00:49:37 lbtmu107 LBT_left_GCS: hotspot is 289.000000,255.000000.
Oct 14 00:49:37 lbtmu107 LBT_left_GCS: acquire000xxx.fits couldn't be saved
Oct 14 00:49:37 lbtmu107 LBT_left_GCS: PROFILING: searching for star in full CCD readout...
Oct 14 00:49:37 lbtmu107 LBT_left_GCS: windowsize around hotspot is 80.
Oct 14 00:49:37 lbtmu107 LBT_left_GCS: windowsize around hotspot is 120.

07:51 A second preset didn't help, so we had to stop and restart GCSL. Again the IIF Abort button didn't end the preset.

08:00 Marcus sends a preset.

08:15 Marcus presets to BS9104 on-axis, but we didn't find a star.

08:19 setExpTime to 2000 and preset again. This causes GCS to hang, but it recovers. Still no star.

LUCIFER image confirms that point is off. Why is it off by many arcsec now when it had been stable in the previous couple hours. Is it because we moved from North to South? CA = 1.9 IE=3.9

08:41 BS9104 was apparently too faint, so we move to BS9108. We find that star easily, but WFS seems to have a glint from the moon. How can it have glint when it is behind the pinhole? See wfsc 0644, 0645

0849 Preset again, but the stray light is making bogus aberrations.

08:53 pointto BS9108 gs=21 ACTIVE PA=0 (couldn't move probe)

08:54 pointto BS9108 gs=21 ACTIVE PA=INDEF -> 95.22 No glint here, so we've dialed out the bogus aberrations.

09:13 Seeing still up near 2 arcsec, but WFS rms for this off-axis star is about 170 nm rms.

DIMM Pointing (Co-Pointing) Test

09:28 pointto M5_0050 on-axis trying to acquire same star on DIMM. GCS is hung looking for acquisition image again. DIMM is vibrating many arcsec in the wind, so it isn't any help in isolating the source of the F/15 jitter.

09:35 Stop and restart GCS, but it didn't restart cleanly.

09:37 pointto M5_0094 on-axis EL=74, the guider locked on OK.

09:43 pointto M5_0202 on-axis EL=70

Oct 14 02:46:45 lbtmu107 LBT_left_GCS: autoAcquire(): waiting 4716 ms for IST to finish receiving image.
Oct 14 02:46:50 lbtmu107 LBT_left_GCS: PROFILING: full readout received.
Oct 14 02:46:50 lbtmu107 LBT_left_GCS: full CCD read out received.
Oct 14 02:46:50 lbtmu107 LBT_left_GCS: hotspot is 289.000000,255.000000.
Oct 14 02:46:50 lbtmu107 LBT_left_GCS: acquire000xxx.fits couldn't be saved
Oct 14 02:46:50 lbtmu107 LBT_left_GCS: PROFILING: searching for star in full CCD readout...
Oct 14 02:46:50 lbtmu107 LBT_left_GCS: windowsize around hotspot is 80.

09:49 pointto WT10_314 GCS hung

09:53 pointto WT10_265 on-axis EL=70 Guide OK

09:57 pointto WT10_157 on-axis EL=43, Guide OK, WFS saturated in 30 sec

10:00 pointto WT10_108 on-axis EL=32, Guide OK, WFS hung

Oct 14 03:03:01 lbtmu107 LBT_left_GCS: WFS: requesting a WFS image...
Oct 14 03:03:20 lbtmu107 LBT_left_GCS: WFS: timeout while waiting to issue a requestImage command. trying again...
Oct 14 03:03:20 lbtmu107 LBT_left_GCS: PROFILING: WFS: triggerWFS() entered.
Oct 14 03:03:20 lbtmu107 LBT_left_GCS: PROFILING: WFS: requesting image...
Oct 14 03:03:20 lbtmu107 LBT_left_GCS: WFS: requesting a WFS image...
Oct 14 03:03:38 lbtmu107 LBT_left_GCS: WFS: timeout while waiting to issue a requestImage command. trying again...
Oct 14 03:03:38 lbtmu107 LBT_left_GCS: PROFILING: WFS: triggerWFS() entered.

10:03 pointto WT10_065 on-axis EL=15

10:07 pointto WT10_065 on-axis EL=15 after restarting GCS. Guide OK, but image is moving around alot. WFS saturated in 10 sec.

10:11 pointto M5_0094 on-axis EL=74,

10:16 Aaron presets to really bright star for DIMM test

Johannes measures seeing of 1.5 - 1.6 arcsec on this star.

Back to collecting collimation data

10:47 stop and restart GCS to pick up new minimum WFS exposure

10:48 pointto M5_0094 on-axis EL=59.7, set WFSExpTime 5000 but still saturated

10:53 pointto WT10_299 on-axis EL=84.9

Oct 14 03:53:34 lbtmu107 LBT_left_GCS: error: setProbePosition: couldn't move probe to position (SFP) (x,y): 0.143419, 0.011064 mm.

10:55 pointto WT10_299 on-axis EL=84.9 Same preset worked the second time. Image has significant coma and focus

11:02 pointto BS9116 on-axis EL=56.9

Total -0.287 -0.731 +0.221 +21.65 -1.02 0 57.2 0.98

Practice Field Alignment Data

11:12 pointto BS9108 on-axis EL=54.4 with some moon glint on WFS.

11:22 agw_collimate on-axis wfsc 001-004

11:26 pointto BS9108 gs=16 PA=INDEF GUIDE -> PA=181.09

11:28 agw_multiple, 3 (but default exposure time was 3 sec)

11:30 agw_multiple, 3, EXP=30 gs=16 PA=181.09 EL=51.5 wfsc 007-010

11:33 pointto BS9108 gs=21 PA=INDEF GUIDE -> PA=95.2

11:33 agw_multiple, 3, EXP=20 gs=21 PA=95.2 wfsc 011-013

11:37 pointto BS9108 gs=18 PA=INDEF GUIDE -> PA=30.2

11:38 agw_multiple, 3, EXP=30 gs=18 PA=30.2 EL=51.5 wfsc 014-016 BUT spots are too faint

11:40 pointto BS9108 gs=7 PA=INDEF GUIDE -> PA=272

11:41 agw_multiple, 3, EXP=30 gs=7 PA=272 EL=49.8 wfsc 017-019

11:44 pointto BS9108 gs=9 PA=INDEF GUIDE -> PA=227.9

11:45 agw_multiple, 3, EXP=30 gs=9 PA=227.9EL=49.8 wfsc 020-022

11:47 pointto BS9108 on-axis EL=48.5 PA=0 wfsc 023-025

11:52 pointto BS9108 gs=9 PA=INDEF GUIDE -> PA=227.9

11:54 agw_multiple, 3, EXP=30 gs=9 PA=227.9 EL=47.3 wfsc 026-028

11:56 pointto BS9108 gs=20 PA=INDEF GUIDE -> PA=58.2 wfsc 029 (oops mistakenly sent these results to primary, and bad seeing)

11:59 pointto BS9107 on-axis PA=0 GUIDE EL=56.8

Seeing is bad enough to dilute the gain of the WFS.

12:01 pointto BS9105 on-axis PA=0 GUIDE EL=32.2 wfsc 030-033

12:07 pointto BS9105 gs=21 PA=INDEF GUIDE -> PA=77.4 EL=31.1

12:09 agw_multiple, 3, EXP=30 gs=21 PA=77.4 wfsc 034-036

12:13 pointto BS9105 on-axis PA=77.4 GUIDE EL=30.0 GCS died unexpectedly

12:14 restart GCSL

12:15 pointto BS9105 on-axis PA=77.4 GUIDE EL=30.0 wfsc 037-039

Total +0.494 -0.644 +0.972 14.25 1.90 0 28.7 0.57

12:21 pointto BS9105 gs=6 PA=INDEF GUIDE -> PA=337.5 but guider didn't lock onto the guide star

12:23 pointto BS9107 gs=0 ACQUIRE PAMode- (same orientation as LUCIFER picture) IE=-9 CA=7

12:27 pointto BS9105 gs=6 PA=INDEF GUIDE -> PA=337.5 GCS died because I forgot to kill the manual readGuideCamera

Restart GCSL

12:29 pointto BS9105 gs=6 PA=INDEF GUIDE -> PA=337.5

12:30 agw_multiple, 3, EXP=30 gs=6 PA=337.5 wfsc 040-042

12:33 pointto BS9122 gs=0 ACTIVE PA=0 EL=74

12:38 clear C00, to give it some help, set Zglobal=0 to improve focus

Total -1.312 +0.359 -0.655 +26.86 +7.19 0 74.2 0.39

Morning Sky Flats

12:48 Over to Nancy for some sky flats

13:20 Close Dome. Sky is photometric. Wind about 10 m/sec T=-1degC D=-14degC.

-- JohnHill - 14 Oct 2008
Topic revision: r9 - 02 May 2009, JohnHill
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback