AGw Commissioning with IRTC - 16-17 November 2008

Observers: D.Thompson, J. Hill (at beginning) in Tucson, with N.Ageorges, W.Seifert, and J.Brynnel
Telescope Operator: W. Wack
SW Support: P.Grenz

Summary

  • Mostly preset and offset tests tonight.

  • Please sync IRTC computer with our time server - it was 2 minutes behind as of the beginning of the night!

  • 20:22:53 no EL track polynomials, no obvious reason for them to stop. EL drive went to hold mode.

  • 01:12 and 01:48 had presets complete with a very out of focus image on the first guider image. WFS converged, but why were we so far off when every other preset started out in decent collimation (i.e. collimation model is good)?

  • Multiple apparent hangs of the IRS towards the end of the night. All appeared to hang on the irc getParameter command.

  • At least three acquisitions failed claiming "no guide star" although at least one was visible. In two cases the star was only 30-50 counts above sky, but one (000031) was 80-90 above sky and thus should have been detected.

  • I'm shutting down all windows on lbtdu16 and logging out in hopes of getting a fresh start tonight.

Details (Times reported in MST below)

Data in /Repository/20081117/ for now, will be moved to /Repository/IRTC_Data later.

Open

T=8.6 degC summer!, with clear sky.

18:20 pointto M5_0881 gs=0 ACQUIRE PAM- (failed for rotator not running)

18:22 pointto M5_0881 gs=0 ACQUIRE PAM-
Nov 16 06:20:04 lbtmu107 LBT_left_GCS: [E] AIP AGW unit error: AGW1: Network Error
Nov 16 06:20:04 lbtmu107 LBT_left_GCS: AGW::setxy(-17.9, 462.6) returned 202

18:24 Wayne stopAGW and startAGW, but gets an error when he tries to home it. Needed another 30 sec for UMAC to be alive.

18:28 kill and restart GCS since it was hung on the above error.

18:29 pointto M5_0881 gs=0 ACQUIRE PAM- ... IE=-32 CA=-78

JMH hands over to DJT

Coordinating with SCIDAR people on VATT. Will try to call them (through Wayne) before presetting.

18:51 Lpoint BS9101
Preset successful. WFS collimated from well out-of-focus with some coma. Current seeing 1.6 arcsec on guide camera. WFS shows ~250nm rms per cycle.

19:07 Lpoint BS9101_GS03
Had to bypass file permissions issues on irtc images in /Repository.

19:34 Lpoint BS9101
Back to using off-axis guide star. Guider sees ~1.1 arcsec FWHM, WFS sending ~220nm.

verified BS9101 on IRTC, using off-axis guide star #3, PA=307.24 205 arcsec off axis

Test dithering as IRTC

Get a sky set before starting:
offset 5.0 0.0
takeseq 2.0 30 take1+ (irtc.20081117.023850.fits +)
offset 0.0 0.0
Set IRTC path to R:\20081117\ and prefix to irtc.20081117

Note: Filter on IRTC is J, temp set to 240. Running at PA=

Start dither60 script (takeseq 2.0 30 take1+ with offsets and focus 0.0's).

07:43 cl < dither60.cl (first set, script in /Repository/20081117)
First seq image = irtc.20081117_00122_cube.fit
Last sequence image failed?

07:55 repeat dither60.cl (second set)
Failed again to take the last sequence image.

The IRTC windows control computer is ~2 minutes ahead of the other computers. This needs to be synced to our time server (currently it does it off some remote windows time server) and more often than once per week.

08:07 repeat dither60.cl (third set) at EL~63

08:18 repeat dither60.cl (fourth set) EL~63.5

Check out this error:

20:22:53 no EL track polynomials
20:22:54 EL reverted to a "hold" state
Sun Nov 16 20:22:58.960 2008 Offset command started for left IRTC from IIF
Last offset in script, prior takeseq was successful.
20:28 offset command timed out in 300 seconds!

Paul reports nothing obvious in the syslogs.

COMMENT
Michele checked the SYSLOGs, event database, the MCSPU log, and the PCS diagnostics file for clues. From the perspective of the PCS, the polynomials are being generated. The timestamp and coefficients for the elevation polynomials are within specification required by MCSPU. There are no missing polynomials in the timeframe stated. In summary, Michele cannot find anything that would indicate missing or bad elevation polynomials except for the initial MCSPU error. Tom will add more thorough diagnostic information in the MCSPU at an allowed time to aid in this situation for the future. The Offset command failed because once the MCSPU elevation axis went into HOLD mode, it was no longer accepting any polynomials.

Test dithering authorized as LUCIFER

08:55 Authorize as LUCIFER on IIF GUI and IRTC.
Preset to BS9101 successful with warnings, but the guider sees a VERY out of focus image!
We were running with the LUCIFER collimation model from October. We copied IRTC's over the default LUCIFER files and initialized PSFs to read them in. Also adjusted IE=-26 CA=-68

/home/telescope/TCS/Configuration/PSF/new
cp SXPMIRTCFCollimation.dat SXPMLUCIFERCollimation.dat
cp SXSMIRTCFCollimation.dat SXSMLUCIFERCollimation.dat

09:16 Lpoint BS9101 cat="Persson.cat" succeeded. Reasonable starting collimation, 370nm WFerror on first collimation cycle. FWHM on guider ~0.75 arcsec.

09:20 cl < dither60.cl while authorized as LUCIFER
EL=64.5 at start. Script completed.

Testing dithering through the actual LUCIFER interface

Turning over to Nancy to repeat from the LUCIFER interface.

preset from LUCIFER again, same source. Take a sequece with IRTC during this (240x2.0sec).

Offset was taking a very long time. PCS GUI reports only "error". We believe that the IRTC sequence (authorized as LUCIFER) was blocking LUCIFER from sending any offset requests through the IRS. ^C in iraf stopped the task but did not abort the IRTC image sequence. Will set the IRTC to not send commands to the telescope and try again.

10:09 Started 240x2.0sec sequence from IRTC GUI directly, did not go through IRAF interface. LUCIFER script controlling the dithering.

10:19 Preset to FS111 at PA=90 through LUCIFER interface, successful.

GCS GUI suggestions:
Show guider integration time on guider "cam view" as well as actual guide rate?
Show WFS countdown timer on WFS subpanels.
Move green status bars to the appropriate guider and wfs subpanels?
Note: GCS reporting FWHM(R) incorrectly - it is displaying sqrt(FWHMx^2 + FWHMy^2) but it should calculate this as sqrt[(FWHMx^2 + FWHMy^2)/2.0] (the geometric mean, not sum).
Zernikes displayed in GUI should be zeroed out on a new preset.

10:25 Started script through LUCIFER and IRTC data sequence from IRTC GUI (240x2.0sec). Guider seeing ~1.0 arcsec FWHM.

10:33 Started script again as LUCIFER and IRTC now taking 210x2.0sec to equalize times.

10:44 Preset to same source, different guide star (PA=-90).
10:53 Started script again as LUCIFER, IRTC sequence 210x2.0sec started on IRTC GUI.
11:09 same again.

IRTC data sequences taken from GUI not saved, my fault (DT: unfamiliar with GUI controls).
Set the default to "Shot & Save"!!!

IIF Authorization changed back to IRTC

11:40 Starting presetting/pointing tests...

11:38 Lpoint FS106_PA45 cat="Lsource.cat" Tel="ACQUIRE", IE=-31 CA=-74

11:43 Lpoint FS106_PA45 cat="Lsource.cat", the preset hung on the "irc ClearStars". Nothing appears to be coming through the IIF, so it looks like the IRS is hung. Restart apparently needs root priveleges ("/sbin/service irs restart" as telescope failed, Permission denied). Note: the TO should be able to restart all subsystems as needed. Calling Stephen at 12:02. Paul restarted irs (twice before it seemed to be working) after getting root password from Stephen. At second restart the LUCIFER getMultiParameter polling started up at ~1/sec. Paul says it looks like it hung on one of the getMultiParameter calls (at 23:41:00) from LUCIFER, but note the timestamp was about when we sent the ACQUIRE mode preset before (although that succeeded as far as I could tell).

Note added later from JMH: If you send any commands from IRAF (authorize, pointto, etc) while LUCIFER is also sending commands to irs, these can collide and cause irs to hang. My understanding of Jose's explanation is that this version of irs only has one command object that talks to IIF, so two programs talking to it (i.e. LUCIFER and IRAF as IRTC) can confuse things. The version of irs that we were testing earlier in the month has a separate command object for each instrument so multiple instruments (plus DIMM etc) can talk to irs simultaneously. Note that two LUCIFERs talking to irs could still cause problems in the newer version.

12:12 Lpoint FS106_PA45 cat="Lsource.cat" ... succeeded. EL62, LST=3:50
takeseq 2.0 300 take1+ (153_cube)

12:27 Lpoint FS120_PA135 cat="Lsource.cat" ... succeeded. (30x2.0s 154_cube). Some vignetting of pupil on wfs.

12:36 Lpoint FS153_PA-115 cat="Lsource.cat" ... failed (no star in acquisition image, EL=9 degrees!)

12:42 Lpoint FS153_PA-90 cat="Lsource.cat" ... failed (no star in acquisition image, EL=9 degrees!)

12:43 Lpoint BS9101 cat="Persson.cat" ... succeeded (30x2sec 155_cube).

Why are there 2577 defunct ds9's on lbtmu107? The ds9's on the acquisition images just started appearing after I used ds9 from the command line to look at an acquisition image. There were 2 Acquire.py running, one from Nov 14 and another from Nov 15.

Note added later from JMH: There should definitely not be two versions of Acquire.py running at the same time on any machine. That may or may not have been the cause of the defunct ds9's.

12:49 Lpoint BS9138 cat="Persson.cat" ... failed, EL=8!

12:55 Lpoint BS9185 cat="Persson.cat" ... failed (below horizon, my fault (DT), should have been 9135).

12:56 Lpoint BS9135 cat="Persson.cat" ... failed, EL=16.

12:58 Lpoint BS9134 cat="Persson.cat" ... succeeded, EL=28 (30x2.0sec 156_cube).

1:04 Lpoint FS101_PA180a cat="Lsource.cat" ... succeeded EL=35 (30x2.0sec 157_cube).

1:12 Lpoint FS106_PA45 cat="Lsource.cat" ... succeeded EL=54 LST=4:41 Star was a ~2 arcsec diameter donut! 2858nm sent on first wfs image, -2208nm Z4, everything else small. Second WFS=771, with -388 Z4, 393 Z6. Guider shows ~0.7" FWHM. (30x2.0 158_cube).

Note added later by JMH: The collimation model may not be especially accurate at elevations below 35 degrees. If you apply active optics corrections at a very low elevation, and then preset back to a higher elevation, the extra corrections made down low will still be in effect. I think this is the explanation of why the initial focus was off by as much as 2 arcsec. Especially if you have adjusted the initial focus with Zglobal, this would be a good time to Clear Active Optics and start afresh.

01:20 Lpoint BS9123 cat="Persson.cat" ... failed EL=20 LST=4:51 There is a star in acquire000021.fits at 68,401, about 30-50 counts peak above background in the 2.0 sec exposure (should be found with improved algorithm).

01:26 Lpoint BS9118 cat="Persson.cat" ... succeeded EL=50 LST=4:54 (30x2.0sec 159_cube).

01:30 Lpoint FS119_PA-120 cat="Lsource.cat" ... succeeded EL=55 LST=5:02 (30x2.0sec 160_cube)

01:38 Lpoint FS1 cat="quick.cat" ... failed, no guide star, EL=10!

01:42 Lpoint FS106_PA45 cat="Lsource.cat" ... succeeded EL=49 LST=5:12 (30x2.0sec 161_cube).

01:48 Lpoint FS111_PA00 cat="Lsource.cat" ... succeeded but G* is out of focus ~1.5" donut. First WFS sent -1107nm Z4, same direction as before. (30x2.0sec 162_cube).

Single IRTC images are not getting the full additional header information...why not? Paul does not see any errors coming back from the GetMultiParameter request, and this has worked in the past.

01:20 Lpoint BS9123 cat="Persson.cat" ... failed, no guide star detected. Source in acquisition image 000027, but only 30-50 counts above background. Closer to hotspot than earlier attempt.

T_out=7.2 T_M1=6.6

02:05 Lpoint FS122 cat="quick.cat" ... succeeded EL=67 LST=5:35 (30x2.0sec 163_cube, 164_cube). The second cube is definitely after convergence, I may have started the first one too soon.

02:14 Lpoint BS9116 cat="Persson.cat" ... succeeded EL=57 LST=5:43 (150x2.0sec 165_cube).

02:24 Lpoint FS101_PA180a cat="Lpoint.cat" ... failed, no guide star found, EL=19 LST=5:55.

02:29 Lpoint BS912 cat="Persson.cat" ... failed, no guide star found, EL= 25 LST=6:00. Sound threshhold is 50 in AIP_L.cfg and this star has ~80-90 counds above background, some coma (see acquire000031.fits) - it should have been found.

02:37 Lpoint FS106_PA45 cat="Lsource.cat" ... succeeded EL=40 LST=6:08 (150x2.0sec 166_cube).

02:50 Starting Offset/IQ tests...

dither60.cl modified to omit initial on-center integration and take 150x2.0 sec exposures at each position. See fivept.cl.

02:50 Lpoint FS123_PA00 cat="Lsource.cat" ... succeeded EL=49 LST=6:22. 03:00 started fivept.cl. 167-171_cube. Script took 40 minutes to take 25 minutes of data. 15 minutes/5 = 3 minutes for overheads of offsetting, focus 0.0, and any IRTC readout overheads.

03:30 Lpoint FS123_PA170 cat="Lsource.cat" ... succeeded EL=57 LST=7:00. 03:34 started fivept.cl. 172-176_cube. Done at 04:05, five minutes faster than the last set (same source, different guide star)???

04:07 Lpoint FS123_PA35 cat="Lsource.cat" ... succeeded EL=62 LST=7:36. 04:10 started fivept.cl. 177-181_cube. Done at 4:45. IRTC data frequently looked out of focus (2sec exposures) while the guider looked fine (4.6sec exposures).

04:46 Lpoint FS123_PA205 cat="Lsource.cat" ... succeeded EL=67 LST=8:14. 04:48 started fivept.cl. Looks like the first offset in the script hung again. Paul is restarting IRS. ^C'ed out of the fivept.cl script. Lucifer window is also locked on the screen saver? There were 2 copies of the IRS running. Several more restarts, and I looked out of the iraf cl as well. Seems to be back up now (5:16).

Note added later by JMH: Don't be fooled by two apparent instances of "irs" running when you do a ps. irs has a script file and the actual program which seem to have the same name (in the middle of the night). This should be changed, since even experts like Paul can apparently be fooled along with the rest of us hackers.
root     27165  0.0  0.1  6168 1236 ?        S    06:06   0:00 /bin/bash /etc/init.d/irs restart
root     27166  0.0  0.0  4556  768 ?        S    06:06   0:00 initlog -q -c /lbt/irtc/bin/irs
irs      27168  0.0  2.5 74468 25924 ?       Sl   06:06   0:01 /lbt/irtc/bin/irs

05:16 Lpoint FS123_PA205 cat="Lsource.cat" ... succeeded EL=69 LST=8:47. 05:18 looks like it hung again, this time on a GetParameter command from the IRTC on the first dither position. The offset and focus commands before the takeseq worked, as did the GetFilter, GetFOV, and SetIRParam commands that preceded the GetParameter. Several restarts again, and

05:30 Lpoint FS123_PA205 cat="Lsource.cat" ... succeeded EL=68 LST=9:00. Sent fivept_v2.cl at 05:33 (modified to only take 90x2.0sec per pointing because dawn approacheth). Only 182_cube obtained, IRS hung on the GetParameter command just before taking the second exposure.

No preset (GCS continued guiding through this). Just sent fourpt_v2.cl (even though there is no version 1. Hung again at the GetImage in the second position. Restarted IRS and sent threept_v2.cl (last three positions in fivept.cl). Again...twopt_v2.cl. And again...onept_v2.cl. All four of the last times IRS appears to have hung on a GetParameter command associated with a GetImage. I did not have to exit iraf, just sent a new authorize command. 182-186_cube in this set.

Focus seq by hand 187_cube = sky offset 0.0 0.0 focus -6 (failed, max is -3, so I sent that but should have moved AGw stage directly instead of focus....)

Resend preset to clear things out from a "good" starting position, but it hangs again on ClearStars.

That is it, closing at 6:17am.

-- JohnHill - 17 Nov 2008
Topic revision: r8 - 18 Nov 2008, DavidThompson
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback