AGw Commissioning for 30th September 2008 - 1st October 2008 (Day 2)

Observers from Tucson: Doug Miller, John Hill, Andrew Rakich (with a couple Kiwi)
Software Support from Tucson: N. Cushing, T. Sargent, D. Cox
Telescope Operator: A. Ceranski

Summary

Seeing of 2-2.5" for the first three hours of the night. Norm, Tom and Chris B. worked on the irs. Still some troubles with IRS/IIF communication in the form of Socket Error and others, and GCS issues. Search for the word "ERROR" for problem specific information and for the word "NOTE" for important details about the nights activities.

Seeing of 0.8 to 1.5 the rest of the night. Off-axis active optics seems to be making reliable corrections on-axis as seen by the AGw.

We collected one set of field aberration data (thee off-axis stars) before the WFS camera failed. Andrew reports that from first set of data that the position of the secondary can not explain the aberrations seen by Lucifer. It looks like only about 15" of secondary rotation is needed minimize the binodal astigmatism, where more than 1' of rotation would be needed to make the aberrations seen by Lucifer.

The WFS Camera looks to have failed the same as it did 2 weeks ago. Power cycle did not help. Started up AzCam gui on the AzCam computer on take images with WFSC. See the same image as acquired the image with GCS. Everything looks exactly like two weeks ago when the camera cable was broken. Norm reopened IssueTrak #1640.

Nighttime Activities

All times in UTC.

2:15:55 - Authorization took several minutes to return. Not sure why it took so long. Stopped IRS to turn on debugging in server configuration file, /lbt/irtc/etc/irtc_server.config, IRS.debug=true, and than restarted irs.

2:55 - Had to init and home the secondary mirror to get collimation to work.

2:59 - Manually tweaking up focus.

NOTE: IE = -60, CA=-65

3:01:13 - Preset successful - Active Mode.

Guiding and WFS working. Probe on axis. Seeing is estimated at 2.5 arcsec.

Arron reported mirror is not at thermal equilibrium (2+ degree difference between it and ambient).

Collimated using WFS images 1 - 20, guider images 1-236

guide rate is 0.2 Hz. Star jumping around.

3:19 - Setexp to guide camera integration time set to 1000 ms

3:24 - Seeing is getting better. Sky is better, mirror temp is better. Seeing is 0.9 arcsec.

3:28:50 - guide rate changed to 1 Hz and integration time 1000ms. Star still jumping around.

3:33:39 - guide rate changed to 2 Hz and integration time 1000ms.

3:39:45 - guide rate changed to 2 Hz and integration time 2000ms.

3:47:35 - Active optics loop turned off by having PCS GUI ignore guiding corrections.

3:55:40 - New Preset, BS9177 on axis. Preset failed twice because could not find guide star which was not in the field of view.

3:59:16 - ERROR - From IIF log, lbtmu103 LBT_IIF: TCPSocket::recv error: Connection reset by peer.

Communication between IIF and IRS seemed to stop. Could not determine what happened.

4:11:24 - Restarted IRS.

ERROR - Determined the previous preset was still active. TCS was under the impression guiding was active but GCS did not.

4:16:02 - Preset. Searching for a star.

ERROR - Offset pointing is frozen.

4:28 - Restarted IRS to solve problem.

4:30 - Hung again. Calling Chris.

5:20 - We have been wrestling with IRS issues. Moved all subsystems off of 105 to other servers so IRS is the only thing on the machine. IRS restarted.

ERROR - irs.bin log messages are not being completely written out.

Sep 30 22:20:54 lbtmu105 irs.bin: [I] PresetTelescope command status: SUCCESS
Sep 30 22:20:54 lbtmu105 irs.bin:
Sep 30 22:20:54 lbtmu105 irs.bin: [I] PCS set new target to RA: 4.885009315612145e+00 Dec: 8.568500037193717e-01 complete

Noticed several times when IRS was hung we had the following message in the IIF log but this did not occur everytime.

Sep 30 22:23:24 lbtmu103 LBT_IIF: IIF:Offset execution time: 3555 ms at 1222838604898
Sep 30 22:24:41 lbtmu103 LBT_IIF: IIF:Offset executed
Sep 30 22:24:45 lbtmu103 LBT_IIF: IIF:Offset execution time: 3895 ms at 1222838685388
Sep 30 22:24:45 lbtmu103 LBT_IIF: TCPSocket::recv error: Connection reset by peer

Notice the seconds are the same. Is the handling of the command return object contributing to this problem?

5:32:38 Presetting.

5:38:23 - Presetting to star at 80 elevation. Found guide star which was out of focus (doughnut) then lost it.

5:41:59 - Preset in track mode to insure finding star.

At elevation approximately 82

Tweaking up the pointing for better pointing to off axis star.

IE -70, CA -28 BS9188 on axis

5:47:26 - Preset in ACTIVE mode, primary mirror into +Y limit, way out of focus, guiding failed.

Doug and John H. determining how to proceed.

5:55:55 - New preset. Did not acquire guide star due to lack of signal.

WFSC image #58 is first image for star BS9188.

Primary mirror once gain ran into +Y limit.

6:13 - ERROR - There is a correlation between Doug performing a from iraf while the IRC is talking to the IRS and the socket problem;

TCPSocket::recv error: Connection reset by peer

reported in the IIF log.

6:24:12 - Preset to star BS9101, IE-125, CA-28

6:36:03 - Preset. Possible ERROR - Doug claims this preset failed. Looked to be up against Y limit on primary. Is this a warning or error. If so, it is not reflected in the IIF log or the IRS log.

Sep 30 23:36:05 lbtmu103 LBT_IIF: IIF:Preset executed
Sep 30 23:36:18 lbtmu103 LBT_IIF: IIF:Preset execution time: 13891 ms at 1222842978904
Sep 30 23:38:34 lbtmu103 LBT_IIF: IIF:Preset executed
Sep 30 23:39:06 lbtmu103 LBT_IIF: IIF:Preset execution time: 31220 ms at 1222843146095

Sep 30 23:36:03 lbtmu105 LBT_IIF: [I] ClearStars: Status OK
Sep 30 23:36:03 lbtmu105 irs.bin: [I] ClearStars: Status OK
Sep 30 23:36:03 lbtmu105 LBT_IIF: [I] SetHotspot: Status OK
Sep 30 23:36:03 lbtmu105 irs.bin: [I] SetHotspot: Status OK
Sep 30 23:36:04 lbtmu105 LBT_IIF: [I] SetOffset: Status OK
Sep 30 23:36:04 lbtmu105 irs.bin: [I] SetOffset: Status OK
Sep 30 23:36:04 lbtmu105 LBT_IIF: [I] SetGuideStars: Status OK
Sep 30 23:36:04 lbtmu105 irs.bin: [I] SetGuideStars: Status OK
Sep 30 23:36:05 lbtmu105 LBT_IIF: [I] [D] Calling PresetTelescope IIF::statusInf->Block()
Sep 30 23:36:05 lbtmu105 irs.bin: [I] [D] Calling PresetTelescope IIF::statusInf->Block()
Sep 30 23:36:18 lbtmu105 LBT_IIF: [I] PresetTelescope command status: SUCCESS
Sep 30 23:36:18 lbtmu105 irs.bin: [I] PresetTelescope command status: SUCCESS
Sep 30 23:36:18 lbtmu105 LBT_IIF: [I] PCS set new target to RA: 1.067777891959698e-01 Dec: 1.364362661378456e-01 complete
Sep 30 23:36:18 lbtmu105 LBT_IIF: [I] left PMC Mirror GoToPositionAbs Command Complete
Sep 30 23:36:18 lbtmu105 LBT_IIF: [I] Ok
Sep 30 23:36:18 lbtmu105 LBT_IIF: [I] No focal station defined
Sep 30 23:36:18 lbtmu105 LBT_IIF: [I] success
Sep 30 23:36:18 lbtmu105 irs.bin: [I] PCS set new target to RA: 1.067777891959698e-01 Dec: 1.364362661378456e-01 complete
Sep 30 23:36:18 lbtmu105 irs.bin: [I] left PMC Mirror GoToPositionAbs Command Complete
Sep 30 23:36:18 lbtmu105 irs.bin: [I] Ok
Sep 30 23:36:18 lbtmu105 irs.bin: [I] No focal station defined
Sep 30 23:36:18 lbtmu105 irs.bin: [I] success
Sep 30 23:36:18 lbtmu105 LBT_IIF: [I] [D] IRS PresetTelescope execution done
Sep 30 23:36:18 lbtmu105 irs.bin: [I] [D] IRS PresetTelescope execution done

6:37 Applied -30 arcsec Tip (coma free) to primary mirror in order to roll us off the +Y limit.

6:42 - Began collimating at WFSC image 61 through 74, Seeing is 0.7 arcsecs.

6:44 - set guide camera exposure to 2000 ms, guide rate is set at 2 Hz.

6:51:51 - BS9101, guide star #3

WFSC to 74 through 87

7:02 - BS9101, guide star #7

7:05:22 - GCS ERROR - WFSC #88 was acquired while we were moving to the guide star and the corrections were sent to the primary.

collimation on 89-93 guide star #7

Oct  1 00:02:41 lbtmu103 LBT_IIF: IIF:Preset executed
Oct  1 00:03:41 lbtmu103 LBT_IIF: IIF:Preset execution time: 60237 ms at 1222844621887
Oct  1 00:03:41 lbtmu103 LBT_IIF: TCPSocket::recv error: Connection reset by peer

ERROR - Is the command return object causing this failure (see above and actual IIF log for many examples).

WSFC image 97 guide star #7

7:12:06 - Preset failed. Inst had not issued authorize command to irs. We suspect this led to the problem.

Oct  1 00:12:06 lbtmu105 LBT_IIF: [I] PresetTelescope command status: RUNNING
Oct  1 00:12:06 lbtmu105 irs.bin: [I] PresetTelescope command status: RUNNING
Oct  1 00:12:06 lbtmu105 LBT_IIF: [I] Invalid buffer: RPCERROR
Oct  1 00:12:06 lbtmu105 irs.bin: [I] Invalid buffer: RPCERROR
Oct  1 00:12:06 lbtmu105 LBT_IIF: [I] [D] IRS PresetTelescope execution done
Oct  1 00:12:06 lbtmu105 irs.bin: [I] [D] IRS PresetTelescope execution done

7:13:18 Preset reissued following an inst initiated authorization.

on-axis, wfsc 98-103

7:16 Seeing is 0.9 arcsec

7:18:45 - Preset BS9107, on-axis

7:45 Finally found on-axis star IE=-68 CA=-60

7:47 pointto BS9107, ACTIVE, gstar=0 wfsc 109-112 Collimated!

7:53 pointto BS9107, GUIDE, gstar=0 wfsc 2 ... irs problems

7:59 - ERROR - IRS/IFF got all messed up.

Oct  1 00:59:00 lbtmu103 LBT_IIF: IIF:SendWavefront execution time: 8122 ms at 1222847940689
Oct  1 00:59:00 lbtmu103 LBT_IIF: TCPSocket::recv error: Connection reset by peer
Oct  1 01:00:39 lbtmu103 LBT_Networkserver: Server 2/4: name lbtmu103, load 15.720000, subsystem count 4 ( IIF MCS OSS PSFL ), has rules false, last update 1222845185966
Oct  1 01:00:39 lbtmu103 LBT_Networkserver: update 1222848039675 lbtmu103 10.144.0.103 10.10.0.103 10.30.0.103 2 16.33 4 IIF 1 MCS 1 OSS 1 PSFL 1
Oct  1 01:00:39 lbtmu103 LBT_Networkserver: update 1222848039676 lbtmu103 10.144.0.103 10.10.0.103 10.30.0.103 2 16.33 4 IIF 1 MCS 1 OSS 1 PSFL 1
Oct  1 01:00:39 lbtmu103 LBT_IIF: parseXMLBuf error in exerces nameStringargumentsString92left]]>senderString
Oct  1 01:00:39 lbtmu103 LBT_IIF: No arguments
Oct  1 01:00:39 lbtmu103 LBT_IIF: No arguments

Oct  1 00:58:52 lbtmu103 LBT_IIF: IIF:SendWavefront executed
Oct  1 00:59:00 lbtmu103 LBT_IIF: IIF:SendWavefront execution time: 8122 ms at 1222847940689
Oct  1 00:59:00 lbtmu103 LBT_IIF: TCPSocket::recv error: Connection reset by peer
Oct  1 01:00:39 lbtmu103 LBT_Networkserver: Server 2/4: name lbtmu103, load 15.720000, subsystem count 4 ( IIF MCS OSS PSFL ), has rules false, last update 1222845185966
Oct  1 01:00:39 lbtmu103 LBT_Networkserver: update 1222848039675 lbtmu103 10.144.0.103 10.10.0.103 10.30.0.103 2 16.33 4 IIF 1 MCS 1 OSS 1 PSFL 1
Oct  1 01:00:39 lbtmu103 LBT_Networkserver: update 1222848039676 lbtmu103 10.144.0.103 10.10.0.103 10.30.0.103 2 16.33 4 IIF 1 MCS 1 OSS 1 PSFL 1
Oct  1 01:00:39 lbtmu103 LBT_IIF: parseXMLBuf error in exerces nameStringargumentsString92left]]>senderString
Oct  1 01:00:39 lbtmu103 LBT_IIF: No arguments
Oct  1 01:00:39 lbtmu103 LBT_IIF: No arguments

-bash-3.00$ tail -f /var/log/messages | grep -i iif
Oct  1 01:00:39 lbtmu103 LBT_IIF: parseXMLBuf error in exerces nameStringargumentsString92left]]>senderString
Oct  1 01:00:39 lbtmu103 LBT_IIF: No arguments
Oct  1 01:00:39 lbtmu103 LBT_IIF: No arguments
Oct  1 01:01:45 lbtmu103 LBT_IIF: parseXMLBuf error in exerces nameStringargumentsString92left58pcs.pointingStatus.achieved.achieved_RA.RAStringpcs.pointingStatus.achieved.achieved_DEC.DECStringpcs.pointingStatus.achieved.papcs.pointingStatus.achieved.parallacticAnglepcs.pointingStatus.achieved.azimuthpcs.pointingStatus.achieved.elevationpcs.mcsDemand.rotatorDemand.anglepcs.pointingStatus.time_UT.UTStringpcs.pointingStatus.time_SD.SDStringecs.weatherStation.smt.ambientTemperaturecsq.side[0].authorizedInstrumentcsq.side[0].authorizedFocalStationpcs.side[0].pointingStatus.guider.delta_XnameStringargumentsString92left]]>senderString
Oct  1 01:02:35 lbtmu103 LBT_IIF: No arguments
Oct  1 01:02:35 lbtmu103 LBT_IIF: No arguments
Oct  1 01:03:18 lbtmu103 LBT_IIF: parseXMLBuf error in exerces nameStringargumentsString92left58pcs.pointingStatus.achieved.achieved_RA.RAStringpcs.pointingStatus.achieved.achieved_DEC.DECStringpcs.pointingStatus.achieved.papcs.pointingStatus.achieved.parallacticAnglepcs.pointingStatus.achieved.azimuthpcs.pointingStatus.achieved.elevationpcs.mcsDemand.rotatorDemand.anglepcs.pointingStatus.time_UT.UTStringpcs.pointingStatus.time_SD.SDStringecs.weatherStation.smt.ambientTemperaturecsq.side[0].authorizedInstrumentcsq.side[0].authorizedFocalStationpcs.side[0].pointingStatus.guider.delta_X

8:05:05 - NOTE: Restarted IIF on 105 so it is running on same machine as IRS. Had to reset focus via PSF.

8:09 - pointto BS9107 GUIDE gstar=80 PA=160 wfsc 9-10 (added corrections, ignore) wfsc 11-13

8:19 - on-axis PA=160 wfsc 14-15

8:23 - gstar=89 PA=60 No star found

8:25 - on-axis PA=60 TRACK (probe park!)

8:33 - on-axis PA=0 GUIDE wfsc 16-17

8:37 - gstar=89 PA=60 No star found

8:38 - gstar=91 PA=92 No star found

8:42 - pointto M5_1133 IE=-97 CA=-35

8:53 - pointto BS9107 TRACK PA=0, star found on hotspot

9:00 - pointto BS9107 GUIDE PA=0, wfsc 18-23

9:07 - gstar=80 very dim, poor guiding

9:12- gstar = 97, PA=124, wfsc 24-26

9:15 - on-axis PA=124 wfsc 27-29

9:19 - gstar = 82 PA=27 wfsc 30-32

9:27 - on-axis PA=27 wfsc 33-35

9:30 - gstar = 20 PA=293 wfsc 36-38

9:35 - on-axis PA=293 wfsc 39-42

9:40 - gstar = 39 PA=226

ERROR - No WFSC Image ~80 counts per pixel....Another broken cable??

9:50 - Aaron powered off and on the AGw, aocserver and check4nport

10:00 - Still only ~80 counts per pixel. Looks like no communication to WFS camera

10:10 - Started up AzCam gui on the AzCam computer on take images with WFSC. See the same image as acquired the image with GCS. Everything looks exactly like two weeks ago when the camera cable was broken. Could there be a short or open in the connector on the camera or controller and changing the cable wiggled it enough to work again? Now with time and operation the connector has failed.

10:30 Andrew reports that from first set of data that the position of the secondary can not explain the aberrations seen by Lucifer. It looks like only about 15" of secondary rotation is needed minimize the binodal astigmatism, where more than 1' of rotation would be needed to make the aberrations seen by Lucifer.

11:00 NOTE: Aaron powered off the AGw unit so mountain staff can start working on the AGw.

Notes added the following day

a) JMH has updated the IRTC and LUCIFERSX pointing model to change IE from the old value of -60 to the new value of -97 after the -30 arcsec of tip was applied. The new files are named irtc.20081001.ptmod, etc in /home/telescope/TCS/Configuration/PCS/.

b) JMH has also updated the IRTCF and LUCIFER collimation models so that the global offsets include the -30 arcsec Tip of the primary (Y=-1.396 mm and RX=-30 arcsec), plus at combined shift of -0.5 mm Y for both M1 and M2 (no net alignment change). Files are dated 20081001 in /home/telescope/TCS/Configuration/PSF/.

-- NormCushing - 01 Oct 2008
Topic revision: r8 - 02 Oct 2008, JohnHill
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback