Left Direct Gregorian Technical Observing with IRTC: UT 2009 November 25

Observer: JHill (LBTO)
Telescope Operator: ACeranski
Instrument Support:
Telescope Support: J. Little
SW support: NCushing, PGrenz, DCox, TSargent, SHooper (Tucson, first half)


Spent the first half of the night working thorugh various hardware, dspware, software and computer problems. The majority of these seem on indirectly related to this month's engineering work on the telescope. You can read about the details of what we fixed below.

Weather was spectacular, so we spent the 4 hours in the second half collecting AGW transformation data in seeing ranging from 0.36 to 0.66 arcsec on the guider. Sky was photometric all night long. These cubes need to be stacked and centroided so Jesper can do the transformation calculations.

We see a quite strong azimuth pointing change as a function of elevation which doesn't seem to match the pointing model from Oct 26. Possibly this is caused by the temperature gradients in the C-ring, but I don't understand why the change in pointing is so purely azimth. Could the azimuth zeropoint have changed so that we need to adjust IA rather than CA? We confirmed that an IA adjustment was needed on the following night.

We also are having problems with GCS sending bogus corrections in higher order Zernikes, and then not being able to dial them back out. We've seen this before, but never in superior conditions like we have tonight.

Tried to take some field data in the later part of the night. But IDL has the same problems with modfied DD names, plus something else that I can't figure out.

Spent 4 hours on commissioning data, and all the rest fighting problems or features.


We are using TCS software build 35 installed yesterday.

Engineers (Ashby and friends) are finishing up work on the telescope.

00:15 Rebooted IRTC Windows PC for good measure. IRTC connects without problems.

00:28 Open vent doors as telescope is 8 degC above ambient temperature.

John L. reports that SX Direct Rotator has a kinked air line, but the brakes are working at the moment.



01:05 Open shutter doors, although Dan C. is still working on DSP software for MCSPU. T=2.3 degC, D=-23.6 degC, Wind light at 2 m/sec. Sky is clear.

01:18 sent test preset via IRAF pointto, but was unable to communicate with irc. Called Norm.

01:20 Dan is finished, so telescope is up and running. Engineers used the first hour of twilight.

IRC configuration problem

IRTC> !irc Register 'directGregorian left' IRTC
irc - InfraRed Camera Client V.1.4.4
Connection refused by IIF. The IIF must be running to use irc.

The problem is that the /lbt/irtc/etc/irc_client.config file has a Tucson IP address in it rather than the local "iif" machine.
Factory.Proxy=Factory:tcp -p 10000 -h #-t timeout(milliseconds)
should be
Factory.Proxy=Factory:tcp -p 10000 -h iif #-t timeout(milliseconds)

Lost 20 minutes to this IRC configuration problem.

Finding first star

01:42 pointto M5_0881 TRACK PA=0 - The star miraculously appears on the IRTC, BUT rotator is not working correctly, and the secondary mirror failed to collimate.

01:47 Aaron restarts rotator.

01:48 pointto M5_0881 gs=0 TelM="ACQUIRE" PAM- - We see the star, but M2 is still out of position at X=-7.01 Y=-3.879 Z=0 when PSF wants M2 at Y=-1.317. Note that there are no IIF events that warn that M2 isn't moving, only some yellow on PSF M2 GUI.

01:53 pointto M5_0881 gs=2 TelM="ACTIVE" PA=INDEF

01:54 Stop and restart OSS, M2 still won't move.

01:58 Aaron did move absolute by hand, but PSF GUI didn't report any motion. And neither did the OSS GUI, but we know the mirror moved because we saw the star image move on IRTC.

02:00 Init M2 on OSS GUI, but then it doesn't want to home after that.

02:03 Paul arrives in his office at a highly opportune time. They restart OSS again (a couple times).

02:09 M2 is back in the proper position. Lost 25 minutes plus 5 minutes of debug time later.

02:10 pointto M5_0881 gs=0 TelM="ACQUIRE" PAM- John sees the star on his GCS movie GUI, but Aaron doesn't see it on his. Aaron restarts his GCSGUI, but that doesn't help. Norm can see the image OK on his GCSGUI displayed in Tucson from obs3.

02:15 IE=-24 CA=+48

02:16 pointto M5_0881 gs=2 TelM="ACTIVE" PA=INDEF - Seeing is about 0.9 arcsec.

02:18 Paul and Norm kill OSS again to try to understand why they aren't getting any log messages from OSS.

02:24 Moon mysteriously disappears from all-sky camera image.

02:26 Paul and Norm stop OSS and move it from tcs3 to tcs4, but the first attempt didn't shutdown properly.

02:28 OSS now running on tcs4. Apparently tcs3 is not writing to the combined syslog properly.

02:30 offset 0 10 DETXY ABS - The image got a bit worse as there was a large accumulation to offload from M2.

02:31 offset 0 -10

02:32 offset 5 0

02:37 Seeing degrades to 1.4 arcsec. This discourages us from taking much useful AGW transformation data.

TCS Test B35 Test 1

02:40 pointto BS9107 gs=0 TelM="ACTIVE" PA=0

Note that one of the telescope air temperature sensors changed by 5 degC as we slewed from SW to NE part of sky. Could this be warm air blowing out of the inside of the C-Ring? See the attached plot below.

02:44 pointto BS9107 gs=0 TelM="ACTIVE" PA=-19

02:45 pointto BS9107 gs=0 TelM="ACTIVE" PA=+19

02:47 Aaron slews rotator to 430, then slewtotrack. But rotator angle is decreasing. Need to be south of meridian for working the +450 limit, so will try the -90 limit.

02:51 pointto BS9107 gs=0 TelM="ACTIVE" PA=+196

02:53 Aaron slews rotator to -70 then slewtotrack.

DG Rotator stopping

02:56 pointto BS9107 gs=0 TelM="ACTIVE" PA=+194, but now rotator is stopping unexpectedly. Call Dan C and he restarted the DSP. Used 15 minutes.

03:18 pointto BS9107 gs=0 TelM="ACTIVE" PA=+191

03:20 pointto BS9107 gs=0 TelM="ACTIVE" PA=+189 Rotator stopped again during slewtohold.

03:29 Norm and Paul stop and restart OSS to test the logging again after restarting the syslog daemon on tcs3.

03:33 Norm figures out that the GCSGUI problem on obs1 is because /Repository (and friends) are not mounted on the to-station (obs1). Stephen fixes that a few minutes later.

03:50 Rotator stops at -89:56:12 after making warning messages on MCSPU for 5 minutes. Test successful.

03:51 pointto BS9107 gs=0 TelM="ACTIVE" PA=+189 - Rotator stopped again while slewing.

03:54 Stop for Dan C. and Tom to restart jet. But it was more tricky than he first thought. Apparently the problem was a stuck polynomial.

04:22 pointto BS9107 gs=0 TelM="ACTIVE" PA=+189 - OSS seems to be hung again.

04:27 Norm does a Stop and restart OSS, but it wouldn't stop.

04:29 OSS is working now, but we need a different guide star as the WFS is saturated.

04:34 pointto BS9107 gs=82 TelM="ACTIVE" - Now with a little collimation, the seeing is 0.4 arcsec on the guider, and even better in short H-band images on the IRTC.

GCS Data Dictionary Problem

04:41 takeseq 0.02 3000
IRTC> !irc GetParameter /home/dthompson/iraf/LBTtools/IRTC/std_irtc_dd.dat
Resmsg[0]  = GetParameter result status: Error
Resmsg[1]  = Offset not found : gcs.side[0].AGW.genericX
Resmsg[2]  = Offset not found : gcs.side[0].AGW.genericY

Which needs to change to this: (I changed it in /home/LBTO/iraf/std_irtc_dd.dat only.)
gcs.side[0].AGW.genericXrequested,AGW_X_AC,TDOUBLE,Requested X position of the AGw probe
gcs.side[0].AGW.genericYrequested,AGW_Y_AC,TDOUBLE,Requested Y position of the AGw probe
gcs.side[0].AGW.genericXachieved,AGW_X_AC,TDOUBLE,Achieved X position of the AGw probe
gcs.side[0].AGW.genericYachieved,AGW_Y_AC,TDOUBLE,Achieved Y position of the AGw probe

Lost 10 minutes. (Dave later fixed my typo (above) where I created redundant header entries.)

04:52 takeseq 0.02 3000 irtc.20091125.045201.fits irtc.20091125_00001_cube.fit

Note: /Repository/20091125/ has a load of old IRTC files from 20091027 in it. Can we get rid of these? AND the IRTC cube that I just took isn't getting copied there. I can see my new cube in /newdata/.

The first cube wound up in /newdata/inder/warning/20091025/ (The month is not a typo in the wiki log.)

Pointing and acquisition tests

05:00 focus 0 to force M2 offload

05:03 pointto BS9107 gs=0 TelM="ACQUIRE" PAM- - Adjust IE=-38 CA=33

05:10 pointto BS9107 gs=82 TelM="ACTIVE" - The probe is not well-centered on this guide star. Is the transform screwed up again? Or did we have the wrong guide star? Based on the next 15 minutes, it was the wrong guide star.

05:17 pointto BS9107 gs=0 TelM="ACQUIRE" PAM- - Adjust IE=-36 CA=26

05:21 pointto BS9107 gs=47 TelM="ACTIVE" PA=INDEF=335.73 - No star found.

05:23 pointto M5_1133 gs=0 TelM="ACQUIRE" PAM- Adjust IE=-35 CA=105 (so we had the wrong star above)

05:30 pointto BS9107 gs=0 TelM="ACQUIRE" PAM- Now we have a star on the right edge and one on the left edge but none in the middle ???

05:34 pointto M5_0151 gs=0 TelM="ACQUIRE" PAM- Adjust IE=-30 CA=+116

Firefox crashed, but it remembered what I had in the wiki edit buffer.

05:40 pointto StoneCagw gs=0 TelM="ACQUIRE" PAM- - Star arrives within 2 arcsec.

05:42 pointto StoneCagw gs=17 TelM="ACTIVE" - seeing is 0.4 to 0.5 arcsec FWHM on the guider at EL=55 deg.

StoneC Transformation Data - Destroyed by disk mounting problem.

05:48 cl < tran_StoneC_00.cl with 1 sec x 45 IRTC images
transtar (10) - Probe motion is out of range
transtar (14) irtc.20091125.054956.fits /newdata/irtc.20091125_00002_cube.fit
transtar (15) irtc.20091125.055213.fits /newdata/irtc.20091125_00003_cube.fit (15 is the on-axis star, so this script file is confused, but we got the background cube)
transtar (12) - Probe motion is out of range.
transtar (18) irtc.20091125.055559.fits irtc.20091125_00005_cube.fit

What is making this error from IRAF>?
TRANSTAR: Taking IRTC sequence 1. 45 
std_irtc_dd: Undefined variable.
   Image obtained: irtc.20091125.054956.fits

JMH: Figured out later that this problem is caused by irc not accepting IRAF indirect path references. So for IRTC.takepic and IRTC.takeseq you can specify the keyword file as "/home/LBTO/iraf/myfile.dat", but you can't use the IRAF reference "home$myfile.dat". The takepic.cl script is happy with the IRAF indirect reference, and happily passes the filepath on to irc, but irc isn't happy with the IRAF reference. So the script needs to expand the reference with "path" before passing it to irc.

Now the cubes are appearing in /Repository/20091125/ but where are the single images going?

cd /OldRepository/IRTC_Data/20091125/
cl < home$tran_StoneC_00.cl (new version)
transtar (20) irtc.20091125.061555.fits /newdata/irtc.20091125_00006_cube.fit
transtar (19) irtc.20091125.061843.fits
transtar (14) irtc.20091125.062132.fits /newdata/irtc.20091125_00008_cube.fit
transtar (17)
transtar (18) irtc.20091125.062716.fits
transtar (5) irtc.20091125.062942.fits

I still can't find the single images.

06:34 takepic 1.0 after setting keyword parameter to the correct local filename.
   irc GetParameter home$std_irtc_dd.dat
std_irtc_dd: Undefined variable.
   irc GetImage

      Image obtained: irtc.20091125.063527.fits
Archive image not found: /newdata/irtc.20091125.063527.fits

Copied the std_irtc_dd.dat file into dthompson's /home/dthompson/iraf/LBTtools/IRTC directory as root.

06:39 takepic 1.0
This gets rid of the undefined variable problem in takepic. But I still can't find the single images. Called Dave T. to see if he can figure it out. In the meantime, Dave found another problem in std_irtc_dd.dat and fixed it.

06:45 cl < home$tran_StoneC_90.cl - seeing is now 0.8 arcsec
transtar (21) irtc.20091125.064737.fits /newdata/irtc.20091125_00012_cube.fit
transtar (20)

06:52 cl < home$tran_StoneC_180.cl -
transtar (13)
transtar (16)
transtar (17)
transtar (7) irtc.20091125.070000.fits /newdata/irtc.20091125_00017_cube.fit

The problem seems to be that machine "iif" (aka "tcs2") does not have /newdata mounted. Become root on tcs2 and say:
mount /newdata
mount /Repository
mount /OldRepository
and now it works OK. The single images appear in /newdata/ as they are supposed to. All the header information in the single images of the previous transformation data was lost. Perhaps something can be reconstructed from the getdat log file.

StoneC Transformation Data

The image was rather triangular looking on the first preset. What was going on here? Did we have an M2 offload problem? Did we get some bad correction forces? It looks better after an iteration of active optics.

The getdat information was copied to /home/agwuser/agw-control/logbook.1.20091125 at the end of the night.

07:22 cl < home$tran_StoneC_180.cl -
transtar (13)
transtar (16)
transtar (17)
transtar (7)

07:34 cl < home$tran_StoneC_90.cl - seeing is now 0.8 arcsec
transtar (21) irtc.20091125.073530.fits
transtar (20) /newdata/irtc.20091125_00022_cube.fit

07:39 cl < home$tran_StoneC_00.cl
transtar (20)
transtar (19)
transtar (14)
transtar (17)
transtar (18)
transtar (5) irtc.20091125.075234.fits /newdata/irtc.20091125_00029_cube.fit

07:53 Clear C00 on M1, and this improved the image in a significant way to 0.6 arcsec. Then some active optics got us down below 0.5 arcsec on the guider.

07:55 cl < home$tran_StoneC_190.cl with 0.45 arcsec seeing
transtar (13)
transtar (16) (probe can't move there)
transtar (17) (probe can't move there)
transtar (7)
transtar (4)
transtar (8)

John must have had the picture upside down when he made all these StoneC scripts last month. I've had to go back and edit each one of them.

08:07 cl < home$tran_StoneC_200.cl
transtar (13)
transtar (10) irtc.20091125.081109.fits /newdata/irtc.20091125_00037_cube.fit
transtar (11)
transtar (12)
transtar (14)
aborted script here to check headers.

08:21 cl < home$tran_StoneC_225.cl
transtar (13)
transtar (10) star at edge of acquisition field, but looks OK
transtar (11) star at edge of acquisition field, but looks OK
transtar (12) star at edge of acquisition field, but looks OK
transtar (14)
transtar (21)
transtar (7)
transtar (8)
transtar (6)
transtar (3)
transtar (2)
transtar (1)

08:42 Active Optics is putting in very large correction forces so that the force on actuator 101 is changing by 40# in a single iteration. This is clearly a problem, although the problem is easy to counter once you

08:49 Clear C00 again. Image doesn't improve in a noticeable way. Maybe 0.60 to 0.55.

08:52 pointto M5_0151 gs=0 TelM="ACQUIRE" PAM- Adjust IE=-37 CA=113

08:56 pointto StoneCagw gs=14 ACTIVE PA=INDEF=264.97 but why is guide star way off in the corner of the acq image? I just recentered.

08:59 pointto M5_0151 gs=0 TelM="ACQUIRE" PAM- - Star is still centered within ~2 arcsec.

09:00 pointto M5_0154 gs=0 TelM="ACQUIRE" PAM- Star is off the right edge of the acquisition field.

09:03 pointto M5_0162 gs=0 TelM="ACQUIRE" PAM- Star is off the left edge of the acquistion field.

09:05 pointto M5_0151 gs=0 TelM="ACQUIRE" PAM- Star is still centered within ~3 arcsec. So it seems that we have a strong azimuth dependence of the pointing with elevation that we didn't notice earlier in the night. Pointing model is irtcdg20091026.

09:08 pointto StoneCagw gs=0 TelM="ACQUIRE" PAM- - Star is not seen on the acq movie. Adjust IE=-36 CA=134 which is a huge 17 arcsec change in AZ direction. AZ=237 EL=40

09:15 pointto StoneCagw gs=14 ACTIVE PA=INDEF=264.97 which confirmed we had the correct star.

StoneE AGW Transformation Data

09:19 pointto M5_0258 gs=0 TelM="ACQUIRE" PAM- - Adjust CA=96

09:22 pointto StoneEagw gs=15 TelM="ACTIVE" - seeing is 0.36 arcsec on guider at EL=56

09:27 cl < home$tran_StoneE_00.cl
transtar (15) irtc.20091125.092838.fits /newdata/irtc.20091125_00053_cube.fit but star is saturated on IRTC because it is very small.
Stopped to adjust exposure time from 1 sec to 0.3 sec. Note that there is an active optics update about 70% of the way through these sequences.

09:32 cl < home$tran_StoneE_00.cl
transtar (15)
transtar (27)
transtar (29)
transtar (30) grabbed wrong guide star, this is the bkgnd frame. cube 57
transtar (14)
transtar (13)
transtar (12)
transtar (1) irtc.20091125.095056.fits /newdata/irtc.20091125_00061_cube.fit

09:52 cl < home$tran_StoneE_90.cl
transtar (27)
transtar (29)
transtar (30) This time it seems to be guiding on star 30 even with two fainter stars in the field.
transtar (28)
transtar (23)
transtar (24) grabbed wrong guide star way out in the corner, more bkgnd frames.
transtar (21)
transtar (22) Field is crowded, it used a star from the wrong edge, more bkgnd. I adjusted CA=90 which was the wrong way.
transtar (19)

10:13 Adjust CA=102 so we are well-centered again.

10:15 cl < home$tran_StoneE_180.cl
transtar (20)
transtar (18)
transtar (11)
transtar (10)
transtar (9)
transtar (7)
transtar (23)
transtar (24)
transtar (21)
transtar (22)
transtar (19)

03:30 Firefox crashes repeatedly when viewing finder charts.

03:42 cl < home$tran_StoneE_270.cl
transtar (11)
transtar (10)
transtar (9)
transtar (7)
transtar (8)
transtar (6)
transtar (17)
transtar (5)
transtar (12) binary? suspect coords
transtar (13)
transtar (2) irtc.20091125.110859.fits /newdata/irtc.20091125_00092_cube.fit

11:10 Clear C00

11:14 pointto M5_0269 gs=0 TelM="ACQUIRE" PAM- - Adjust IE=-40 CA=+63

11:17 pointto BS9126 gs=0 TelM="ACQUIRE" PAM- - Adjust CA=+27 at EL=79

11:22 cl < home$field_BS9126.cl from /OldRepository/AGW_Data/091125/
on-axis wfsc 1-4, but getting an error in irc_getparameter, so we didn't send any active corrections.

Acquiring WFSC Image: should take about 30 seconds....
    ...Done: Filename = /OldRepository/AGW_Data/091125/wfsc000003.fits
   Acquiring Data Dictionary Information...
    ** Error in irc_getparameter **
gcs.side[0].AGW.SFP_X,SFP_X,TDOUBLE,current position of the probe
gcs.side[0].AGW.SFP_Y,SFP_Y,TDOUBLE,current position of the probe

change SFP_X to SFPachieved_X in /home/LBTO/idl/wfsc_new/irc/irc_dd.dat

Lost 20 minutes.

11:27 Also we see that the IDL license daemon is down. Spent 5 minutes restarting that. Instructions in Issuetrak #2310.

11:45 cl < home$field_BS9126.cl from /OldRepository/AGW_Data/091125/
Too bad that didn't fix the problem. Fixed some more names in OSS section of irc_dd.dat.
oss.side[0].secondary_mirror.abs_pos[0],LM2X,TDOUBLE,Left secondary mirror X position (mm)
oss.side[0].adsc.abs_pos[0],LM2X,TDOUBLE,Left secondary mirror X position (mm)

12:05 cl < home$field_BS9126.cl from /OldRepository/AGW_Data/091125/
But we lost the star by changing elevation.

12:09 pointto BS9126 gs=0 TelM="ACQUIRE" PAM- - Adjust IE=-32 CA=+56 at EL=68

12:12 cl < home$field_BS9126.cl from /OldRepository/AGW_Data/091125/
STILL reporting an error in irc_getparameter. Did I edit the wrong version? Now change the version in wfsc rather than wfsc_new . (and put wfsc_new back the way it was)

12:18 cl < home$field_BS9126.cl from /OldRepository/AGW_Data/091125/
BUT THAT STILL DIDN'T REMOVE THE ERROR, so there must be something more subtle in irc_getparameter.pro. It works OK from the command line using the file that I editted.

IRTC> !irc GetParameter /home/LBTO/idl/wfsc/irc/irc_dd.dat
Resmsg[0]  = GetParameter result status: OK
Resmsg[1]  = 07:30:34.500000000
Resmsg[2]  = +29:51:12.000000000
Resmsg[3]  = 0.00000000000000E+00
Resmsg[4]  = 07:30:34.566232815
Resmsg[5]  = +29:51:11.989419535
Resmsg[6]  = 0.00000000000000E+00
Resmsg[7]  = 0.00000000000000E+00
Resmsg[8]  = 0.00000000000000E+00
Resmsg[9]  = 7.59752384025119E+01
Resmsg[10]  = -1.73944120001421E+00

Sent an email to vacationing Doug to see if he can figure it out. Doug immediately recognized when he read his email the next day that the problem was a missing configuration file that got removed from /tmp when the workstations were rebooted. Doug's IDL code uses a copy of irc_client.config in /tmp to control various parameters and file destinations.

12:37 pointto BS9126 gs=0 ACTIVE - seeing is 0.6 on guider

12:45 clear C00, but again the image didn't improve.


12:54 Close dome for exhaustion, and twilight is coming.

-- JohnHill - 19 Nov 2009
I Attachment Action Size Date Who Comment
daq.20091125.gifgif daq.20091125.gif manage 47 K 26 Nov 2009 - 04:03 JohnHill Plot of Telescope and Mirror Temperatures
pmcc.dx.thermal.20091125.gifgif pmcc.dx.thermal.20091125.gif manage 28 K 26 Nov 2009 - 04:11 JohnHill Plot of DX Primary Mirror Temperatures
pmcc.sx.thermal.20091125.gifgif pmcc.sx.thermal.20091125.gif manage 30 K 26 Nov 2009 - 04:10 JohnHill Plot of SX Primary Mirror Temperatures
Topic revision: r7 - 27 Nov 2009, JohnHill
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback