Telescope and AGw4 Commissioning at LDG - 04-05 May 2009 MST

Observers:: J. Hill, J. Storm (mtn), A. Rakich (Tucson)
Telescope Operator: W. Wack
Software Support: N. Cushing (mtn) Telescope Support: J. Rousseau, J. Morris


We got guiding to work. The main trick was to use PCS with X-flip=True (what we expect for DG) in combination with reversing the sign of X values in the oacserver. Those two flips effectively provided GCS with an X-flip that was needed for reporting the guiding centroids back to PCS. This gave us the ability to guide, but with the spiraling-in problem when guiding off the y-axis.

To quote Michele:
 IRTCDG should have the same XFLIP and YFLIP settings as LUCIFER
    has *currently*.  These are XFLIP=true, YFLIP=false.  The PCS
    which should be used for direct Gregorian testing is
    /lbt/tcs/current/bin/PCS.090503_01a; This is PCS_Version-3.34a
    where current=B25.centos4.

We confirmed that LEFTSCALE for LDG is -1.0 (appropriately opposite the rotation direction of LFBG).

We don't yet understand completely why LEFTZEROPOINT wants to be +5.5 and not something near -180 degrees. That needs some more work.

Note Michele's other warning which explains some difficulties last night:
 If LEFTZEROPOINT is near -180. then it is possible for PCS to generate
rotator polynomials with A0 values which are large and negative.
The MCSPU conditions the polynomials.  If the A0>360, then newA0 = A0 - 360.
If A0<0.0, then newA0 = A0 + 360.0.  The final newA0 must be in the
range of 0-360, or the MCSPU will not accept the poly. we did
the other night, the IRTCDG LEFTZEROPOINT may have to be +180.0.

We patched GCS to remove the minus sign we had added to the camera rotation to correct the spiraling at LFBG. This needs to be added later as a configurable item for different instruments and focal stations. (Torsten had already added the corresponding flag for Zernike rotations.) So Guiding now works correctly at all field positions.

We had terrible problems with guider images getting intermixed with WFS images and vice-versa. We eventually traced that down to a configuration problem on the AzCam machine. See Issue 2163. This same problem also exposed another bug in GCS that caused GCS to crash if a WFS image was not the expected size (ie. it was a guider image). Norm, Chris and Michele patched that bug. John speculates that this might fix a few of the rare GCS crashes that had not been previously explained.

We found and corrected another configuration file issue where GCS was using the LFBG rotator angle to correct the LDG Zernikes.

Lost the second half of the night to clouds. No time was lost to telescope hardware issues. Some time was spent fighting software, but that's what commissioning is expose the bugs.

The next step on-sky is to get Active Optics working. That will be the priority for tomorrow night. After that, we'll go back and precisely measure the hotspot, rotator center and LEFTZEROPOINT.

Used /Repository/AGW_Data/090504/wfsc000019_dd.fits to set initial values for Left Direct Gregorian wfsc. These values should be checked/improved when the telescope is collimated. DLM 090505
;               Values set 090505  DLM
        wfsc_x_center_on_axis = 390.2
        wfsc_y_center_on_axis = 251.1
        agw_x_lenslet_edge[0] = 381.65
        agw_y_lenslet_edge[0] = 240.45
        wfsc_rot_offset = 9.0


Test preset with dome closed to see if TCS network is still intact.
18:33 pointto M5_1246 Tel="GUIDE" gs=0 PAM+ PA=0

18:40 Switch PCS to PCS.090503_01a (X-flip=true, Y-flip=false) which is the version that theoretically should work if we X-flip the AGW transformation compared to what LUCIFER uses at LFBG.

lrwxrwxrwx  1 root     root       14 May  4 18:40 PCS -> PCS.090503_01a
-rwxr-xr-x  1 delapena lbt  11588880 May  3 22:07 PCS.090503_01a  x-flip=true yflip=false
-rwxr-xr-x  1 delapena lbt  11588880 May  3 22:20 PCS.090503_02b  x-flip=false yflip=false
-rwxr-xr-x  1 delapena lbt  11588880 May  4 00:23 PCS.090503_03c  x-flip=true yflip=true
-rwxrwxr-x  1 root     tcs  11588196 Apr 22 00:04 PCS.090503B25
18:41 pointto M5_1246 Tel="GUIDE" gs=0 PAM+ PA=0

18:44 Wayne authorizes IRTC at LFBG.

18:46 Wayne authorizes IRTC at LDG.

18:48 readGuideCam -e 10 - failed (why?, we rebooted agw-control this afternoon while Norm and Kyle were working in the computer room, but Wayne had already turned AGW4 back on) Did the selectAGW command fail when we changed instrument authorizations?

18:55 selectAGW junk

18:56 Updated AIP_IRTC_DG_L_RM_sub11.cfg so that all three of Z5, Z8, Z10 have signs flipped compared to LFBG.



19:05 Open Dome, T=+7.9 degC, D=-8.3 degC, Wind ~ 5-8 m/sec from South. Sky with scattered cirrus.

19:15 Switched over to summer chiller as forecast is for warm weather all week. (It takes a while for the loop to warm up and the summer chiller to come on.)

19:18 Changed pcsmx from +1 to -1 in oacontrol.conf (This should follow the PCS X-flip installed above and still allow us to find stars.) stopAGW -u 1, stop and restart oacserver.

19:29 pointto M5_1246 Tel="ACQUIRE" gs=0 PAM- - star landed at bottom of acquisition on first try! Adjust IE=-62 CA=+5.

19:42 pointto BS9145 Tel="ACQUIRE" gs=0 PAM- - Adjust *IE=-52 CA=+2 CA= *

19:49 Changed GCS default guide rate in lbt.conf from 0.2 to 0.4.

19:50 pointto BS9145 Tel="ACQUIRE" gs=8 PAM+ PA=INDEF - found the star about 8 arcsec lower right.

19:54 offset 0 0 ROT=0.5 (wrong way)

19:55 offset 0 0 ROT=-1.0 then -2.0 but these are moving orthogonal to the hotspot.

19:58 pointto BS9145 Tel="ACQUIRE" gs=8 PAM+ PA=60 to move off the y-axis. And we see a star, 8 arcsec to lower left. Star only moved about 1 arcsec open loop in the 5 minutes it took the rotator to track 1 degree, so we think the rotation scale (direction) is correct.

20:05 offset 0 0 ROT=+1. (star moved down, mostly orthogonal to hotspot)

20:07 offset 0 0 ROT=-2. (closest approach to hotspot)

Test guiding

20:08 pointto BS9145 Tel="GUIDE" gs=8 PAM+ PA=60 - The star spiraled in nicely, and it guiding off-axis.

20:13 Changed WFS_probeRotationFactor -1.0 in AIP_IRTC_DG_L.cfg.

20:15 pointto BS9145 Tel="GUIDE" gs=8 PAM+ PA=60 - Star still spiraled in.

20:16 Stop and restart GCS.

20:17 pointto BS9145 Tel="GUIDE" gs=8 PAM+ PA=60 - Star still spiraled in.

20:18 pointto BS9145 Tel="GUIDE" gs=8 PAM+ PA=INDEF=344.29 - Acquisition offset worked correctly on the y-axis.

Issue: CCD rotation factor doesn't seem to work in either direction (and there are only two choices). But, wait, the name is:
WFS_ZrotatorDirection  1.0             float   # multiplication factor for rotator angle (to change direction use -1.0)
WFS_probeRotationFactor -1.0             float   # used to change how the probe rotation is added to the rotator angle (use -1.0 to subtract it from rotator)
These are for the Zernike rotations, not for the guide corrections.

Test active optics

This is good old GCS active optics with Z5, Z8, Z10 flipped in sign.

20:21 pointto BS9145 Tel="ACTIVE" gs=8 PAM+ PA=INDEF

20:24 Clicking on Cam View for WFS crashed GCS.

20:25 pointto BS9145 Tel="ACTIVE" gs=8 PAM+ PA=INDEF - The initial impression was the WFS is not converging.

20:26 GCS crashed again.
May  4 20:26:12 lbtmu107 LBT_networkserver: GCSL died unexpectedly

20:50 We've discovered that we don't have any syslog messages from anybody on lbtmu107 this evening. Is that why GCS is crashing?

20:57 pointto BS9145 Tel="GUIDE" gs=8 PAM+ PA=60 - didn't get star from edge of field.

21:02 pointto BS9145 Tel="ACQUIRE" gs=0 PAM- Adjust IE=-58 CA=+9

21:08 Spinning rotator 360 deg in Acquire mode shows that the rotator center is about 1.5 arcsec left of hotpsot. We'll repeat this later once we've gotten active optics and guiding both under control.

21:13 readWFSCam -e 15000 100 while manually guiding (it starts out making star images)

21:20 Michele has delivered a patched version of GCS with the CCD probe rotation correction restored back to the traditional +rotradians rather than the -rotradians that made the correction work with LFBG. This is subversion 12 (GCS v2.1.12) which has the value hardwired.

21:25 Jim D. points out that syslog messages have been redirected from /var/log/messages to /var/log/debug

21:32 Stop and restart GCSto pick up v2.1.12.

Test off-axis guiding with patched GCS

21:34 pointto BS9145 Tel="GUIDE" gs=8 PAM+ PA=60 - There was no spiraling, so the patch worked.

21:45 pointto BS9145 Tel="ACTIVE" gs=0 PAM+ PA=0

21:46 GCS crashed again. Norm and Chris are still working this problem. It has something to do with creating a WFS sub-image.

21:48 Set PSF M1 gains to zero for Z5-Z10 but not Z4, Z11.

21:49 pointto BS9145 Tel="ACTIVE" gs=0 PAM+ PA=0

GCS crashes again.

22:03 pointto BS9145 Tel="ACTIVE" gs=0 PAM+ PA=0

22:16 change guideWindowSize from 100 to 512 in AIP_IRTC_DG_L.cfg (really an experiment)

22:03 pointto BS9145 Tel="ACTIVE" gs=0 PAM+ PA=0

But GCS crashed again......even though it guided for a few times with a large guide image. Norm Chris and Michele have worked out that GCS is crashing on a null pointer that happens when GCS tries to extract a sub-frame of the WFS image. The null pointer happens when the requested sub-frame overfills the original image. This started happening tonight because Guider and WFS images from AGW4 were getting interchanged on the AzCam server.

22:21 pointto BS9145 Tel="ACTIVE" gs=0 PAM+ PA=0

22:22 Dropped into track mode for ?? reason.

22:24 pointto BS9145 Tel="ACTIVE" gs=0 PAM+ PA=0

22:27 GCS crashed again, so I guess my large guide image cheat doesn't work. This is because my image is 512x512 and not 536x528.

22:36 Updated system clock time on the AGW4 AzCam server. It was running 20 minutes behind. Then restarted Windows.

22:40 Updated WFScam_hotspot x and y - 388, 244 is Jesper's measured center of the pupil image. Which I'm not so confident that this is exactly the right number to put here.
WFScam_hotspot_x	388.0		double	# ccd x coordinate of pixel that represents leftmost x-center of WFS spots
WFScam_hotspot_y	244.0		double	# ccd y coordinate of pixel that represents leftmost y-center of WFS spots

22:43 pointto BS9145 Tel="ACTIVE" gs=0 PAM+ PA=0

22:46 change guideWindowSize from 512 to 536, but that didn't work since 536 isn't a legal value.

 ay  4 22:48:38 lbtmu107 LBT_left_GCS: AzCam: bad reply from server while trying to set ROI: ERROR "Could not set vskip"
May  4 22:48:38 lbtmu107 LBT_left_GCS: fatal error setting subframe!

23:00 Updated WFScam_hotspot x and y - 384, 244 (still trying to dodge the GCS crashing bug)

23:01 pointto BS9145 Tel="ACTIVE" gs=0 PAM+ PA=0 - This was working.

Fixing the GCS crashing problem

23:05 Michele installs a GCS patch to trap the null pointer that was causing this crash.

23:06 Stop and restart GCS to get the new patch. Putting hotspot_x back to 388.

23:07 pointto BS9145 Tel="ACTIVE" gs=0 PAM+ PA=0

This trapped what was crashing GCS, so the patch was a glorious success. The new entertainment is that when the two images get swapped, the guider analyzes the WFS image and chooses one of them. This makes a whole new flavor of telescope jump- jumping by about 10 arcsec.

23:20 pointto BS9145 Tel="ACTIVE" gs=0 PAM+ PA=90

Z7 went from +600 to -454, and Z8 went from -300 to -700.

23:26 changed Zrotation direction from +1 to -1.
WFS_ZrotatorDirection  -1.0             float   # multiplication factor for rotator angle (to change direction use -1.0)

23:27 pointto BS9145 Tel="ACTIVE" gs=0 PAM+ PA=0 - now with rotating the Zernikes the other way.

23:33 pointto BS9145 Tel="ACTIVE" gs=0 PAM+ PA=90

23:49 Change WFS_rotatorAngle mcs.rotatorSide[0].rotators[5].actualPositionASec

23:49 pointto BS9145 Tel="ACTIVE" gs=0 PAM+ PA=0 - clouds have started moving in. This makes some very interesting WFS images with pupil views of the secondary.

00:21 Having trouble holding the guidestar on the hotspot. The star is well-detected even though it is fairly cloudy.


00:40 Close for clouds.

Fixing the exchanged images bug

After several hours of snooping around, Norm, John and Jesper finally worked out what was making the guider and WFS images get interchanged. If you start the PEPSI1 and PEPSI4 servers in the same directory on the AzCam PC, they share the same temporary FITS image "tempfile.fts". The eureka moment was about an hour ago when we found in the AzCamServer.bat file on the AGW3 AzCam machine the telling comment "rem set different working directories for parallel processes". After that it was mostly a matter of learning the tricks of "ewfmgr" to unlock the flash disk and make permanent changes to it. We changed the AGW4 AzCamServer bat files by adding a line "cd C:\Program Files\AzCam" and a similar but different path for the other one (the same directories used for AGW3.

02:27 Stop and restart GCS for testing with revised AzCam configuration.

02:30 pointto BS9152 Tel="ACQUIRE" gs=0 PAM+ PA=0

The fake preset with dim dome lights clearly shows that we have fixed the problem. All WFS images have lenslets and vice versa.

-- JohnHill - 05 May 2009
Topic revision: r6 - 07 May 2009, JohnHill
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback