AGW Commissioning with IRTC -- 29-30 October 2008 MST

Observers: J. Hill on mtn and A. Rakich in Tucson
Software Support: N. Cushing and J. Borelli in Tucson
Telescope Operator: D. Gonzalez Huerta


Another photometric night with good seeing. Startup went significantly smoother tonight. Relatively few telescope problems although we are having problems with the test version of irs. The new collimation model was verified. Also had some problems with subsystems having memory leaks and using too much virtual memory (this is what happens when they don't crash on a regular basis!).

The main data collection activity was collecting intra-/extrafocal cubes of IRTC data so that we (Doug) can run EF to verify the collimation results for guide stars in different places in the field.

An important thing we learned tonight is that slow moving seeing (in the dome) does not average out in 30 second WFS integrations. This is certaintly a cause of poor convergence (~300 nm rms) of the active optics loop in conditions like we had tonight (relatively good seeing dominated by a slow ground layer).

We made a preliminary 3 mm adjustment of the AGw-IRTC focus. More data analysis is needed to confirm the precise answer.


New Stuff

Tom S. has a new version of MCSPU code for us to test.

I've used the data we collected last night to create a new SX Primary Mirror collimation model for IRTC. (It is installed for IRTC, but not for LUCIFER.)

Times reported in UT below


00:45 Open Dome T=7degC D=-7degC Wind light, Sky photometric except some low clouds approaching from South of Tucson.

01:00 Testing the new MCSPU code. It seems to work fine in the short term (staying in strip encoder mode during slews). (We had no problems with it for the rest of the night.)

PCS had to be restarted as polynomials were not arriving at the mcspu for the rotator. Oct 29 18:00:42 lbtmu102 LBT_PCS: ERROR rotator is not movable (Not 'READY' for some reason).

IIF was not accepting commands, would not die, had to be murdered with a kill -9 that barely took effect.
Oct 29 18:00:36 lbtmu102 LBT_IIF: IIF Thread Stopped
Oct 29 18:00:36 lbtmu102 LBT_IIF: IIF Exiting
Oct 29 18:00:43 lbtmu102 LBT_Networkserver: IIF refused to stop
Oct 29 18:01:40 lbtmu102 LBT_Networkserver: IIF died unexpectedly
Oct 29 18:01:41 lbtmu102 LBT_Networkserver: IIF is not active
Oct 29 18:01:41 lbtmu102 LBT_Networkserver: dropping IIF
Oct 29 18:02:32 lbtmu102 LBT_IIF: IIF Version 5.13
Oct 29 18:02:33 lbtmu102 LBT_IIF: IIF Thread Started
Oct 29 18:02:34 lbtmu102 LBT_Networkserver: IIF says it is now running

01:03 The slew ended in a peculiar way. But then another slew worked fine.

01:06 Increased AZ and EL velocity limits to 1.5 deg/sec (with John L.'s agreement). (This worked fine all night.)

Is one preset failing 9 times counted as 9 failures?

01:08 pointto M5_0881 gs=0 ACTIVE (rotator wasn't ready)

01:09 pointto M5_0881 gs=0 ACTIVE (no polys received ??)

01:12 pointto M5_0881 gs=0 ACTIVE (still no rotator polys, although the preset didn't immediately fail)

01:16 Stop and restart PCS a second time

01:18 Aborted the preset from 01:12 but irs didn't return to LBTtools

01:20 pointto M5_0881 gs=0 ACTIVE (failed because GCS saw AGW probe at Z=-0.001)

irs needed to be restarted following the PCS problem. ,pre> Oct 29 18:20:03 lbtmu105 irs: irs shutdown succeeded Oct 29 18:20:03 lbtmu105 su(pam_unix)[21904]: session opened for user LBTO by (uid=0) Oct 29 18:20:03 lbtmu105 LBT: Reading LBT.conf with locale mountain. Oct 29 18:20:16 lbtmu105 su(pam_unix)[21861]: session closed for user root Oct 29 18:20:53 lbtmu105 su: Starting DIRTI server V.1.0.11

GCS had trouble getting the AGw set the filter postion or focus read back.
Oct 29 18:20:57 lbtmu107 LBT_left_GCS: AGWunit init(): _filterPosition or _focus read back from AGW server is negative indicating an error.

01:22 pointto M5_0881 gs=0 ACTIVE (didn't find star, since we didn't set IE/CA after restarting PCS)

01:25 Replaced IE=-23 CA=-29 (last night's values)

01:26 pointto M5_0881 gs=0 ACTIVE (LBT_left_GCS: TCPSocket::recv error: Connection reset by peer)

GCS had socket problems connecting to the agw-control computer or the AzCam server.
Oct 29 18:26:55 lbtmu107 LBT_left_GCS: TCPSocket::recv error: Connection reset by peer.

01:30 The previous preset ends with a failure when I issued a getStatus command to GCS, but then my getStatus command hung.

01:32 Stop GCSL subsystem and restart

01:33 pointto M5_0881 gs=0 ACTIVE (didn't find star)

01:36 pointto M5_0881 gs=1 ACQUIRE - succeeded without star

01:37 cl < didn't find star

01:42 David finds 6-degree discrepancy in Az encoder 0 (which makes a 1.5 deg pointing error), so we stop to azencreset again.

01:52 pointto M5_0881 gs=1 ACQUIRE (hiccup in the slewing algorithm) - succeeded without star

01:54 Found star at offset 30 0. Adjust IE=+2 CA=-35
The initial pupil showed negligible coma, but need Z=+0.4 mm correction of focus.

02:01 pointto M5_0881 gs=1 ACTIVE found the off-axis star and started active optics. Initial seeing is 0.6 arcsec FWHM on the guide camera.

02:12 Star image has the jitters for a minute or so.

02:37 pointto M5_0881 gs=1 GUIDE - succeeded

02:40 Andrew running agw_collimate in IDL

02:44 Seeing is worse, or did we get a bad focus correction? It turned out the the IDL script had hung, so there hadn't been a collimation update in 5 min. A manual collimate put a core back in the image.

02:55 agw_collimate still hangs on get DD parameter

03:01 agw_collimate works again after restarting irs. Norm and Jose are puzzling over what is causing irs to hang.

We also note that pressing "collimate" on the PSF GUI to update the mirror position doesn't not cause a significant jump of the image (not more than a few tenths of an arcsec). But when Andrew sends an active optics correction, the image often jumps by 1 arcsec. Is this because the active correction is making a larger motion of the mirror, or are we not making our coma correction in a completely pointing free way? Indeed, if I wait a few minutes so that a collimation correction builds up a significant motion, then I do see ~1 arcsec jumps. (See Issue #1148)

03:21 pauseGuiding

03:24 resumeGuiding

EF Confirmation of Off-Axis Collimation

03:29 pointto M5_0881 gs=1 ACTIVE PA=346.3

03:40 200x4msec cube 0035 to test writing to a subdirectory of /Repository. The "Path FITS" on IRTC says: r:\IRTC_Data\081030\ .

The following will be 360x90msec cubes. Moving intra/extra with +/-0.7 mm on M2.

03:48 intrafocal 0036 extrafocal 0037 but there is a surplus cube 0038? irtc.20081030.035259.fits

03:54 pointto M5_0881 gs=1 ACTIVE PA=346.3 x=-0.4 y=-116.5

03:55 pointto M5_0881 gs=1 ACTIVE PA=301.3 x=81.9 y=-82.4 (Guiding and WFSing start, but my preset never returns to IRAF.)

03:59 Andrew restarts irs

04:01 intrafocal 0039 extrafocal 0040 irtc.20081030.040419.fits

04:05 pointto M5_0881 gs=8 ACTIVE PA=INDEF=268.79 - succeeded x=0.2 y=-168.7

04:11 intrafocal 0041 extrafocal 0042 irtc.20081030.041400.fits

04:16 pointto M5_0881 gs=8 ACTIVE PA=223.79 - succeeded x=119.4 y=-119.4

04:22 intrafocal 0043 extrafocal 0044 irtc.20081030.042547.fits

04:27 pointto M5_0881 gs=7 ACTIVE PA=INDEF=188.13 x=0.1 y=-156.0

04:36 intrafocal 0045 extrafocal 0046 irtc.20081030.043918.fits

04:40 pointto M5_0881 gs=7 ACTIVE PA=INDEF=143.13 x=110.4 y=-110.3

04:41 irs hangs just as Norm comes back into the room

04:42 Norm restarts irs

04:44 Norm restarts irs again (testing something about the syslog messages)

Guider is reporting 0.4 to 0.5 arcsec FWHM images in 4.6 sec exposures, while Polaris monitor is reporting 1.4 arcsec. WFS focus convergence seems to be bouncing around a bit at this location.

04:53 intrafocal 0047 extrafocal 0048 irtc.20081030.045552.fits

04:56 pointto M5_0881 gs=10 ACTIVE PA=INDEF=83.34 x=-0.1 y=-175.5

05:01 intrafocal 0049 extrafocal 0050 irtc.20081030.050424.fits

05:06 pointto M5_0881 gs=10 ACTIVE PA=128.34 (offset to the hotspot didn't get to the guide field in time)

05:09 pointto M5_0881 gs=10 ACTIVE PA=128.34 x=-124.2 y=-124.1 - (failed to make the correct offset)

05:12 Adjust IE=+5 CA=-25

05:13 pointto M5_0881 gs=10 ACTIVE PA=128.34 x=-124.2 y=-124.1 - succeeded

05:19 intrafocal 0051 extrafocal 0052 irtc.20081030.052333.fits

05:23 irs hung on " irc GetParameter /home/dthompson/iraf/LBTtools/IRTC/std_irtc_dd.dat"

Note that the corrections often change by a few hundred nanometers when we switch to a different guide star in the field. It may be that the WFS still has some residuals that are pupil position and/or aberration dependent.

Collimating at EL=44deg with GCS

Now moving to a lower elevation star.

05:25 pointto M5_0036 gs=9 Tel="ACTIVE" PA=INDEF (not low enough, need more oxygen)

Problems finding M5 stars in the catalog that have proper motions for the guide stars. Otherwise they fail with coercing '/par' in the script.

05:39 pointto M5_0893 gs=10 Tel="ACTIVE" PA=INDEF=37.63

05:42 offset 30 30 to get blank sky cube 360x90msec 0053 (for the previous star)

05:45 offset 0 0

Convergence is not so good down here at EL=44 deg, although the seeing is still only 1 arcsec. GCS WFSing is struggling to get below 300 nm rms. We let it run for a while during which time we are discussing with Doug about the results from the focus data we took last night.

06:30 stopWFS in GCS to test new pupil location algorithm in IDL

Collimating at EL=43deg with IDL

06:32 restart irs

06:40 restart irs again

wfsc 0027 has a big jump compared to 0025-0026, 0028

06:47 restart irs again

wfsc 0033 gave big number (didn't send them), then wfsc 0034 had small numbers again. Then wfsc 0035 shows another big jump (sent).

06:55 restart irs again

Updated new parameters from Doug in AIP_L.cfg
WFScam_hotspot_x   393     (previously 388)
WFScam_hotspot_y   220     (previously 225)
WFS_pupil_shift_x   0.20      (previously 0.21)
WFS_pupil_shift_y   0.19      (previously 0.163)

06:57 startWFS without restarting GCS. This collected wfsc 0147 - 0152 with the old parameters.

07:03 pointto M5_0893 gs=10 Tel="ACTIVE" PA=INDEF=37.63 in order to force GCS to read the AIP_L.cfg file. First wfsc image is 0153.

Now taking 500x20msec cubes.

07:09 intrafocal 0054 extrafocal 0055 irtc.20081030.071140.fits

07:12 pointto M5_0893 gs=10 Tel="ACTIVE" PA=82.63 (acquire offset went to the right place)

07:15 pointto M5_0893 gs=10 Tel="TRACK" then IE=+5 CA=-38. Note that our initial manual focus correction is almost gone now (cancelled by the active optics).

07:19 pointto M5_0893 gs=10 Tel="ACTIVE" PA=82.63 x=-128.3 y=-128.3

wfsc 0159 - 0163 is impressively vignetted.

07:26 intrafocal 0056 extrafocal 0057 irtc.20081030.072746.fits

07:28 resumeWFS starting with wfsc 0164

07:30 pointto M5_0893 gs=11 Tel="ACTIVE" PA=INDEF=311.80 (star too far out of the field)

07:32 pointto M5_0893 gs=3 Tel="ACTIVE" PA=INDEF=301.65 x=0 y=-106.43

no vignetting on this star starting with wfsc 0166. Note a focus jump near wfsc 0171.

07:43 intrafocal 0058 extrafocal 0059 irtc.20081030.074613.fits

07:45 irs hung during IRAF irc GetParameter

07:47 pointto M5_0893 gs=3 Tel="ACTIVE" PA=346.65 x=-75.3 y=-75.3

07:52 intrafocal 0060 extrafocal 0061 irtc.20081030.075439.fits

07:56 pointto M5_0893 gs=9 Tel="ACTIVE" PA=INDEF=203.87 x=0 y=-178.4

wfsc 0186 has a big jump in focus and Z8, but maybe because we hit the +X limit. Or did we hit the X-limit because we had a big jump? It seems like it was a spontaneous divergence. (Can we sell tickets for that?)

08:08 Clear Active Optics, and remove the +0.3 Zglobal offset.

08:10 pointto M5_0893 gs=9 Tel="ACTIVE" PA=INDEF=203.87 x=0 y=-178.4

08:22 intrafocal 0062 extrafocal 0063 irtc.20081030.082500.fits

08:26 pointto M5_0893 gs=9 Tel="ACTIVE" PA=INDEF=248.87 x=126.2 y=126.0

08:30 intrafocal 0064 extrafocal 0065 irtc.20081030.083433.fits

08:34 restart irs

offset 30 30 for blank sky of 500x20msec cube 0066

AGW probe version of the focus test

08:38 pointto M5_0162 gs=3 ACTIVE PA=INDEF=77.52 x=0 y=-179.5 Focus=13.251

08:42 Clear Active Optics, but the star still looks fairly ugly. EL=73.9

These are 500x60msec cubes.

08:48 intrafocal 0067 extrafocal 0068 irtc.20081030.085146.fits

08:52 pointto M5_0182 gs=1 ACTIVE PA=INDEF=321.56 (picked up the wrong guide star because we were defocussed)

08:58 pointto M5_0182 gs=1 ACTIVE PA=INDEF=321.56 (succeeded, but says it failed in IRAF, star too faint)

09:00 pointto M5_0182 gs=5 ACTIVE PA=INDEF=56.37 x=0.1 y=-110.4 focus=22.988

Poor convergence wfsc 0215 - 0221

GCS also seems to be taking a lot of time to write Zernike info to the syslog.

ECS and GCS both have memory leaks so they are:
 3795 telescop  15   0  724m  53m 7600 S  7.3  5.3  16:19.48 GCSL
 3310 telescop  15   0  6696 2148 1244 S  4.3  0.2   9:42.18
 6342 telescop  15   0 1540m  69m 4596 S  3.7  6.9 700:56.85 ECS
16275 telescop  15   0 24084 7668 2724 S  2.0  0.7   4:24.17 ds9
22995 telescop  15   0  8596 2336 2220 S  1.7  0.2 134:27.10 gshmserver
16244 telescop  15   0  9292 2420 2144 S  1.3  0.2   3:01.45 sshd
11652 telescop  17   0  8096 2812 1384 S  1.0  0.3   0:00.70
  241 root      15   0     0    0    0 D  0.3  0.0  20:58.33 kjournald
16382 telescop  15   0  8576 2448 2148 S  0.3  0.2   0:40.76 sshd
    1 root      16   0  2896  444  416 S  0.0  0.0   0:51.52 init

Kill PCS (by accident), GCS and ECS (the later two on lbtmu017).

08:20 to-station spontaneously goes black while David is typing. It looks like X may have died.

09:24 pointto M5_0182 gs=5 ACTIVE PA=INDEF=56.37 x=0.1 y=-110.4 focus=22.988 (IE/CA were cleared)

09:26 pointto M5_0182 gs=5 ACTIVE PA=INDEF=56.37 x=0.1 y=-110.4 focus=22.988

Now the WFS cycle time is shorter by 4-5 sec.

Convergence is still poor. Is this slow turbulence at low altitude?

Note the greatly reduced sizes of ECS and GCSL virtual memory.
12669 telescop  15   0  426m  27m 6416 S  5.0  2.7   0:23.96 ECS
 3310 telescop  16   0  6696 2156 1244 S  4.3  0.2  10:19.65
12055 LBTO      15   0  6424 2820 1384 S  4.0  0.3   0:29.54
12273 telescop  15   0  358m  56m 8376 S  2.7  5.6   0:16.65 GCSL
11652 telescop  16   0  8096 2824 1384 S  2.0  0.3   0:13.06
12076 LBTO      17   0  7600 2820 1384 S  2.0  0.3   0:10.31
22995 telescop  15   0  8596 2340 2220 S  1.7  0.2 134:40.65 gshmserver
 9802 LBTO      15   0  9528 2948 2196 S  1.7  0.3   2:59.93 sshd
12062 LBTO      16   0  6356 2820 1384 S  1.7  0.3   0:09.50
12069 LBTO      15   0 22936  11m 5212 S  1.7  1.1   0:03.25 ds9
16275 telescop  16   0 24168 7692 2724 S  1.0  0.7   4:26.92 ds9
16244 telescop  16   0  9292 2420 2144 S  0.7  0.2   3:03.17 sshd
  241 root      15   0     0    0    0 S  0.3  0.0  20:59.55 kjournald
10002 telescop  17   0  2600  916  696 R  0.3  0.1   0:04.46 top
11895 ncushing  16   0  3436  916  692 S  0.3  0.1   0:03.28 top
    1 root      16   0  2896  444  416 S  0.0  0.0   0:51.52 init
    2 root      RT   0     0    0    0 S  0.0  0.0   0:02.11 migration/0

These are 500x60msec cubes.

09:23 intrafocal 0069 extrafocal 0070 irtc.20081030.093701.fits

09:28 Collimating again on-axis

09:45 setFocus 25.898 intrafocal 0071 extrafocal 0072 irtc.20081030.094814.fits

09:49 setFocus 22.988 and resumeGuiding

09:54 setFocus 28.988 intrafocal 0073 extrafocal 0074 irtc.20081030.095751.fits

09:58 setFocus 22.988 and resumeGuiding

10:02 setFocus 19.988 intrafocal 0075 extrafocal 0076 irtc.20081030.100552.fits

The astute and the awake may noticed that nothing changed in the previous pairs of EF cube. !!! Revised procedure: pauseGuiding, move stage, resumeGuiding, collimate, pauseGuding, move M2 intrafocal. take cube, move M2 extrfocal, take cube, takepic

10:09 Open all 4 vent doors as wind is dropping.

10:12 setFocus 25.988 and resumeGuiding

10:16 intrafocal 0077 extrafocal 0078 irtc.20081030.101904.fits

10:20 setFocus 28.988 and resumeGuiding

10:22 intrafocal 0079 extrafocal 0080 irtc.20081030.102441.fits

10:25 setFocus 31.988 and resumeGuiding

10:30 intrafocal 0081 extrafocal 0082 irtc.20081030.103452.fits

Convergence is much better now, so the vent doors must be doing their job of flushing out the slow dome seeing.

10:34 restart irs

10:37 setFocus 34.988 and resumeGuiding

10:39 intrafocal 0083 extrafocal 0084 irtc.20081030.104244.fits

10:44 setFocus 30.988 and resumeGuiding (The estimated focus from post-it note optics.) 1.5 sec guider image is reporting 0.4 arcsec FWHM.

10:49 intrafocal 0085 extrafocal 0086 irtc.20081030.105243.fits

10:53 500x60msec blank sky cube 0087

10:55 restart irs - Norm says dying in a different way without the socket problem.

11:00 setFocus 33.988 and resumeGuiding (post-it note optics isn't too efficient this time of day, because I screwed him up by offsetting the probe to take the blank sky image. And we all know that offsetting the probe adjusts the probe focus.)

11:04 intrafocal 0088 extrafocal 0089 irtc.20081030.110813.fits

11:10 setFocus 30.988 and resumeGuiding (This -3 mm probe move needed +Z4 to refocus.)

11:25 update focus in AIP_L.cfg from 29 to 37 to make this 8 mm change.

11:26 pointto M5_0182 gs=5 ACTIVE PA=INDEF=56.37 x=0.1 y=-110.4 focus=30.971

11:39 pointto M5_0182 gs=8 ACTIVE PA=INDEF=298.01

But Andrew's post-it analysis forgot to compensate for the spherical introduced by the intra-extra focal shift. This has important consequences for LBCFPIA as well.

Note added later: The sign of spherical reverses between the intra and extrafocal positions, so to first order the effect cancels out when you compare the diameters of the two pupils.

11:45 update focus in AIP_L.cfg from 29 to 32 (via 37) to make Doug's estimated focus adjustment.

Start of some field aberration experiments

11:46 pointto M5_0182 gs=8 ACTIVE PA=INDEF=298.01 starting with wfsc 0306

11:56 3000x0.6msec cube 0090 of guiding and active optics

12:07 intrafocal 0091 extrafocal 0092 irtc.20081030.121120.fits

12:12 pointto M5_0182 gs=8 ACTIVE PA=253.01 x=114.75 y=-115.25 focus=19.051 wfsc 0325

12:26 intrafocal 0093 (note slow currents) extrafocal 0094 irtc.20081030.123053.fits

12:30 restart irs

No correction experiment to characterize the atmospheric fluctuation

12:32 sendZernikesToPSF 0, resumeGuiding (and WFS) starting with wfsc 0336 Andrew is updating the primary once per minute. Doug: please make a plot of these unsent results (not urgent). The focus term is surprisingly steady compared to the other terms.

Low-gain active optics experiment

12:50 edit AIP_L.cfg for temporary gain of 0.1 instead of 0.8. sendZernikesToPSF 1,

12:51 pointto M5_0182 gs=8 ACTIVE PA=253.01 Hmmm....this looks like it might work to make more stable corrections in less stable conditions.

Why are the rightmost full column of spots not centered in their subaps (which are now visible in twilight)? See the stars versus subap circles in wfsc 0369. Could the focus be drifting too fast for this low active optics gain to keep up (resulting in steady state errors that need an integrator)?


13:05 Close as morning is upon us. T=5.4 degC D=-4.2 degC wind light.

-- JohnHill - 30 Oct 2008
Topic revision: r9 - 11 Nov 2008, JohnHill
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback