BG performance verification with IRTC - 17-18 November 2008

Observers: D.Thompson, J. Hill (at beginning) in Tucson, with N.Ageorges, W.Seifert, J. Brynnel
Telescope Operator: W. Wack
SW Support: T.Leibold

Summary

Details (Times reported in MST below)

Open

Checked health of IRTC, set up configuration, etc. Everything looks OK from iraf/IRTC. Note: save button on IRTC GUI will save a single exposure as a 2D fits named as cubes like sequence images. Sequence images are saved automatically to /Repository/20081118/

Switching to LUCIFER control...can't run LUCIFER SW, some network/connection error. Will control from IRTC while Nancy retrieves needed info from her hotel.

Starting IE=-4.00 CA=-69.00 (in use at end of last night)

07:09 pointto ACT0240 gs=0 Tel="ACQUIRE" ... failed (GCSL and OSS not running)
07:12 pointto ACT0240 gs=0 Tel="ACQUIRE" ... succeeded (but in PA mode)
  • source on chip, out of focus and with some coma.
07:12 pointto ACT0240 gs=0 Tel="ACQUIRE" ... succeeded PAM-
  • Clear all active optics, etc.
  • M1 Z offset 0.4
  • IE=-5.0 CA=-69.0

The interesting thing that happened here was that David accidentally sent a Yglobal offset off 0.4 mm which simultaneously corrected the pointing (by ~8 arcsec) and the residual coma that we saw at the beginning. (We undid this after we noted how interesting it was.) This seems to imply that the temperature induced pointing errors are caused by Y-translation.

07:25 pointto BS9186 gs=0 ... succeeded. Allow to collimate.

07:36 pointto BS9186 gs=5 ... succeeded. 0.9 arcsec FWHM on guider.

07:45 cl < largeoff.cl
  • dice-5 on source 30x2.0s (11-15_cube, 024934-.fits)
  • dice-5 on sky 400"E 10x2.0s (16-20_cube)
  • repeat dice-5 on source 30x2.0s (21-25_cube)

IRS appears to have hung on the 4th offset command in the script (three cycles of offset-focus-takeseq completed). IIF shows both the command coming through and completing, but the IRS log entries only show the incoming request, no completion reply. Called Norm, calling Stephen (unavailable).

08:09:43 Shutdown/restart IRS. Sent the following offset commands not in a script: offset 0.0 0.0
offset 5.0 7.0
offset -5.0 -7.0
offset 7.0 -5.0
offset -7.0 5.0
offset 0.0 0.0
20:17:24 IIF recieved offset request 20:17:37 IIF reported complete 20:17:37 Socket connection reset by peer 20:20:53 IRS reported complete We called VATT/Jeff and asked him to shut down the DIMM for ~20 minutes.

08:38 cl < largeoff.cl
Continues working despite socket errors. The 400 arcsec offset E worked, guiding paused through the whole 5 point dither at the "sky" position and guiding resumed when we offset back to the starting position.

20:39:01 hung on an offset again, but an exception caught on a getFilter. Repeating "TCP Socket Error: Connection reset by peer" messages in the logs. The IRS seems to hang for 2.5-3.0 minutes whenever the IIF log reports this socket error but they do return if you are patient enough to wait.

09:18 David restarted IRS and IIF. LUCIFER seeing if their SW will now work. Sending an offsetting script. 24_cube taken at one of the dither positions during the LUCIFER script. The script completed with no TCP socket errors despite 1Hz getMultiParameter request. Perhaps the solution when the socket errors are seen is to restart both the IIF and the IRS? Confirm with SW group.

09:37 DIMM/Jeff called so he could restart his DIMM program, although we are planning on pointing tests with LUCIFER for the next couple of hours.

Kyle, Jim, please note Nancy is running LUCIFER from the "lbti" wireless network on her laptop and getting updates every ~second while running from the linux box next to the TO station downtown is only updating every 8 seconds or so.

09:50 Started LUCIFER script that should run for 15 minutes, plus 450x2.0sec on IRTC GUI (25_cube). Test failed: the drives of the telescope (HBS) failed during the test (motor #3 overload).

This was about the time of the HBS problem:
From more /var/log/messages | grep MCS | grep -v Networkserver on lbtmu107:
Nov 17 22:05:47 lbtmu107 LBT_MCS: MCS:AZDRIVE:ready executed
Nov 17 22:05:55 lbtmu107 LBT_MCS: MCS:AZDRIVE:ready execution time: 8090 ms at 1226984755798

Solution/bypass was to reset the breaker for pump #3 to clear the alarm and left the pump off.

10:52 Back up...sending a preset from LUCIFER SW. HBS crashed again? Problem in WinCC computer? Pump #3 put back in operation.

11:07 Back up...sending a preset from LUCIFER SW. Collimated. Starting script. Stopping script after a few minutes to start IRTC data taking in parallel.

LUCIFER scripts are testing if an offset that moves the guide star off the patrol field will recover guiding once a subsequent offset brings the guide star back onto the patrol field. We are getting frequent instances of "probe failed to move to offset position" but it is not clear in the logs why this is happening. This test did work earlier with a larger (400 arcsec) offset. See below, probably solved.

23:30:39 An offset from a sky position (offset 35 35 from center) back to the center (0 0) failed. GCS reporting "probe failed to move to offset position". A getStatus shows AGW probe is at (x,y): 35.662, 187.475 mm

Usually I see a request in the log for an X coordinate in the focal plane (PCS_XY coords) that is well beyond the patrol field limit of X~+/-170 or so. Here's a block from the GCS log messages in syslog:

Nov 18 00:12:49 lbtmu107 LBT_left_GCS: guiding paused
Nov 18 00:12:52 lbtmu107 LBT_left_GCS: setGuidestarList (and conversion):
Nov 18 00:12:52 lbtmu107 LBT_left_GCS: offset = 0.000000, 612.200000 mm.
Nov 18 00:12:52 lbtmu107 LBT_left_GCS: RA/DEC = 0.480485, 0.849575
Nov 18 00:12:52 lbtmu107 LBT_left_GCS: converted into SFP by PCS (x,y) = -350.751629, -156.878739 mm
Nov 18 00:12:52 lbtmu107 LBT_left_GCS: converted into native AGW coordinates by GCS (x,y) = 350.751629, 769.078739 mm
Nov 18 00:12:52 lbtmu107 LBT_left_GCS: best focus determined to be -40.07 mm (including off-axis offset)
Nov 18 00:12:52 lbtmu107 LBT_left_GCS: off axis focus compensation is -71.67 mm
Nov 18 00:12:52 lbtmu107 LBT_left_GCS: AGWUnitAIP::doSetProbePosition: calling oacontrol::setxy(10.144.0.64, 0, x:350.751629 y:769.078739, 23.000000, 0) ...
Nov 18 00:12:52 lbtmu107 LBT_left_GCS: [E] AIP AGW unit error: AGW1: Requested position is Out of Range
Nov 18 00:12:52 lbtmu107 LBT_left_GCS: AGW::setxy(350.8, 769.1) returned 255
Nov 18 00:12:52 lbtmu107 LBT_left_GCS: error: setProbePosition: couldn't move probe to position (SFP) (x,y,z): -350.751629, -156.878739, -40.070851 mm.
Nov 18 00:12:52 lbtmu107 LBT_left_GCS: offsetGuiding command failed as probe couldn't be moved on new position.
Nov 18 00:13:53 lbtmu107 LBT_left_GCS: guide loop already paused

And another example?
Nov 18 00:08:14 lbtmu107 LBT_left_GCS: converted into SFP by PCS (x,y) = -99.049541, 76.815271 mm
Nov 18 00:08:14 lbtmu107 LBT_left_GCS: converted into native AGW coordinates by GCS (x,y) = 99.049541, 535.384729 mm
Nov 18 00:09:36 lbtmu107 LBT_left_GCS: converted into SFP by PCS (x,y) = -183.050174, -1.183114 mm
Nov 18 00:09:36 lbtmu107 LBT_left_GCS: converted into native AGW coordinates by GCS (x,y) = 183.050174, 613.383114 mm

01:08 David reports 20m/s winds fairly consistent and wants to close.

In poking through the syslog on lbtmu107 (more messages | grep GCS | grep -v WFS | grep 'Nov 18 00:' | grep 'Guiding executed') I am seeing "resumeGuiding"s followed immediately (0-1 second later) by a "pauseGuiding":

Nov 18 00:05:40 lbtmu107 LBT_left_GCS: GCSL:resumeGuiding executed
Nov 18 00:05:40 lbtmu107 LBT_left_GCS: GCSL:pauseGuiding executed

in a larger pattern of pause-offset-resume-pause-resume. Is this the expected behavior? As near as I can tell, we were getting similar messages in the logs earlier when this test worked.

OK, LUCIFER was sending relative DETXY offsets and IRTC was sending absolute RADEC offsets so some of the above behavior is explained. Question for Torsten: when an unreachable offset is requested (guide star off the patrol field), are the stages moved at all, or kept where they are until a new valid position is given?

04:52 Re-opening. Wind down to 10.0/17.1 m/s and David thinks it is safe. IE=-15 CA=-63. LUCIFER to work on pointing tests. Guider is reporting highly variable seeing between 2+ and 4+ arcsec FWHM. WFS reports

05:10 EL=16 no guide star found. acquire000050.fits - source is there but very large FWHM

05:13 EL=22 no guide star found.

05:34 we are guiding/wfsing off axis. EL=68. Large offset pattern to check recovery of guiding. 600x2.0sec 32_cube

05:55 Repeat with inverted signs in the offsets. 600x2.0s 33_cube

06:18 Repeat with subset of dither positions. 150x2.0s 34_cube

06:24 Repeat with subset of dither positions. 150x2.0s 35_cube. Offset did take GS off patrol field this time. Returned and there was a guide star on the guidebox, but it looks like it failed to pick it up as it walked off in a few iterations. Sky rapidly getting brighter, strong gradients in guider images.

06:30 Closing, AGw no longer able to continue as the sky is too bright.

Torsten: Can you check why the guiding apparently failed to pick up the star after the return from a long offset? The first image after the return is 7793 (see also /Repository/guiderimage00779*.fits). The log looks like it finds it, but if I read this correctly the star should be at 65,64 in the guider image and it is more like 60,46. Here is some of the log entries (I deleted some that did not seem to be related).

Nov 18 06:26:45 lbtmu107 LBT_left_GCS: guide image file name = /tmp/guiderimage007793.fits
Nov 18 06:26:45 lbtmu107 LBT_left_GCS: WWPA centroid, verifying star does exist...starpix = 1293.00
Nov 18 06:26:45 lbtmu107 LBT_left_GCS: 9 times stddev of bkgrd noise is 1630.7
Nov 18 06:26:45 lbtmu107 LBT_left_GCS: centx: 304.573105, centy: 266.202423, xf: 65.573105, yf: 64.202423,peak: 24132.000000, mbg: 22839.250000
Nov 18 06:26:45 lbtmu107 LBT_left_GCS: star offset to hotspot exceeds 0.750000 arcsecs (1.141413)
Nov 18 06:26:46 lbtmu107 LBT_left_GCS: PROFILING: sent guide correction to PCS.

Also note:
Nov 18 06:26:45 lbtmu107 LBT_left_GCS: guide image file name = /tmp/guiderimage007793.fits
But the timestamp in the image header is 13:24:47.68 so the azcam computer still seems to be ~2 minutes off.

Did not get to this:
Take 3000 frames at 100Hz on M5 star to investigate vibrations.

-- DavidThompson - 18 Nov 2008
Topic revision: r7 - 19 Nov 2008, JoarBrynnel
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback