Telescope and AGw Commissioning at LFBG with IRTC - 1-2 April 2009 MST

Observers: J. Hill (mtn), A. Rakich (standby), J. Storm (Potsdam)
Telescope Operator: A. Ceranski
Telescope Support: J. Little and friends

Summary

Closed at the beginning for high winds.

Completed software tests 9, 13, 14, 15. Struggled with something (that might be OSS) causing LSS to hang.

Tested new MCSPU code.

Finished updating the pupil parameters for GCS and IDL.

Got IRS working and talking to IRTC.

Opened dome at 3AM is the wind slacked a little. But had telescope problems almost as soon as we got to the star. The servos appear to stop reading the strip encoders, and then the telescope drifts away. We ran an old version of MCSPU code and had the same problem. See Issue 2108.

Closed dome at 04:15 in the face of servo problems and rising wind.

Preparations

Tom S. provided a new version of MCSPU code that patches some problems encountered last week.

Dave A. updated the Azimuth scale in the DSP code of MCSPU.

Doug M. provided the correct WFS pupil position values based on the on-axis pupils we took last night.

;  from January 2009
;        wfsc_x_center_on_axis = 393.0
;        wfsc_y_center_on_axis = 220.0

;  090401 DLM today -- this is the center of the pupil
        wfsc_x_center_on_axis = 403.0
        wfsc_y_center_on_axis = 212.0

;  from January 2009
;        agw_x_lenslet_edge[0] = 368.35
;        agw_y_lenslet_edge[0] = 227.15

;  090401 DLM today - this is the corner of the lenslet where the algorithm starts to search
        agw_x_lenslet_edge[0] = 381.65
        agw_y_lenslet_edge[0] = 213.85
        wfsc_rot_offset = 9.0

Fortunately or unfortunately, John didn't realize that Doug hadn't put these numbers in IDL on the mountain until 07-Apr-2009 at 4AM. So we used them all week in GCS, but not in IDL. See the continuing discussion there where the old numbers might have been better.

Details

18:00 We are closed for high wind, although the wind has dropped significantly in the last hour.

18:20 Michele rolled back ECSGUI to the old version to sure an apparent rift in subspace.

18:30 Started a 540 degree Azimuth scan to check the motor encoder / strip encoder calibration for Dave. But it only lasted about 200 secs until a bug in one of Tom's new mjd checks shut it down. Tom is working on the problem.

18:52 Getting ready to open vent doors and start mirror ventilation, but then the wind came back up so vents were only open a few minutes.

18:58 Restart the Azimuth slew scan with patched MCSPU code.

19:24 Got irs working with some coaching from Norm. Jose had changed the IP address in /lbt/irtc/etc/irtc_server.config but not the port address. The correct port address 8081 was written in the comments lines in this file (and confirmed in the lbt-irtccs.ini file on the WindowsPC). Until just now irs had been using port 6565.

19:30 Found a hint about why IRTC images written to /Repository have wrong ownership. See Issue 1790. Should be "LBTO".
Apr  1 19:24:03 lbtmu05 automount[1950]: lookup(file): key "lbto" not found in map.
Apr  1 19:24:03 lbtmu05 automount[1950]: failed to mount /home/lbto

19:50 Found Issue 1839 with explicit details on how to mount the NFS disk on the IRTC windows machine. However, the problem seems to be that user LBTO has been wiped off the NFS machine in the incident 2 weeks ago.

21:12 Editted both AIP_IRTC_L.cfg and AIP_L.cfg to include the values indicated by Doug above.

TCS Software Tests

21:22 TCS SW Test 9 - Verify that the Secondary Mirror in the OSS can be re-Initialized Pressed "Close Brakes" and then "Init" for M2 on OSSGUI. Works as advertised except that hexapod continues moving for more than 1 minute after the OSSGUI reports: "oss.side[0].secondary_mirror:[init] command.complete.

21:30 Authorize as IRTC, pointto BS9177. Preset failed because of an EL-limit problem (which wasn't the test we were doing. Telescope got stuck at elevation 3:17 degrees. This provides the opportunity to test the manual control of HBS oil temperature in ECS (although I don't recommend that as a cure for this particular problem). This correctly adjusted the oil temperature as advertised.

21:49 Restart MCSPU, an Aaron is surprised to find the telescope Elevation 27 degrees higher than before. ???

* In hindsight, we weren't stuck. This was a precursor of the telescope problems

21:52 Run another full Azimuth scan for Dave A.

22:03 Stop and restart OSS to continue TCS SW Test 13.

22:05 pointto BS9177 GUIDE gs=0 - failed because rotator not ready

22:08 pointto BS9177 GUIDE gs=0 - This reported as a failure, but the IIFGUI doesn't show any events reporting failure of the preset. ???

22:12 pointto BS9177 GUIDE again -- this is running 10x slower than normal. IIF showed "Calculating Offload" for nearly a minute. Fails with zero events reported on IIFGUI.

22:14 "top" report from lbtmu102. Note 950m virtual memory use by IIF
 5687 jkraus    17   0 53604  42m 9.8m S  4.0  4.2  31:31.76 3DWGUI
 7752 telescop  15   0  318m  29m 6220 S  3.0  2.9   5:38.52 PCS
 5486 telescop  15   0  786m 184m 9732 S  2.0 18.3  23:16.04 IIF
 5459 telescop  15   0  6292 2624 2384 S  1.7  0.3  25:50.11 gshmserver
  241 root      15   0     0    0    0 D  0.3  0.0  19:52.25 kjournald
    1 root      16   0  2220  464  436 S  0.0  0.0   0:00.85 init
    2 root      RT   0     0    0    0 S  0.0  0.0   0:00.27 migration/0

22:17 "top" report from lbtmu104. Note 99% CPU use by PMCL (because of telemetry?) and 869m virtual memory use. Suspect this is why we aren't getting any events back from LSS.
top - 22:17:18 up 18 days, 55 min,  2 users,  load average: 3.45, 2.95, 3.17
Tasks:  81 total,   1 running,  80 sleeping,   0 stopped,   0 zombie
Cpu(s): 24.3% us, 42.9% sy,  0.0% ni, 32.7% id,  0.0% wa,  0.2% hi,  0.0% si
Mem:   1034552k total,   268540k used,   766012k free,      740k buffers
Swap:  2621432k total,    44076k used,  2577356k free,    19060k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23804 telescop  15   0  687m 182m 4692 S 99.9 18.1   2335:01 PMCL
23627 telescop  15   0  7124 1276 1160 S  2.0  0.1  30:58.87 gshmserver
23650 telescop  15   0 78048  25m 3424 S  0.3  2.5  10:16.54 LSS
25605 telescop  16   0  2480  868  692 R  0.3  0.1   0:00.06 top
    1 root      16   0  2896  492  468 S  0.0  0.0   0:01.60 init
    2 root      RT   0     0    0    0 S  0.0  0.0   0:00.30 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.74 ksoftirqd/0
    4 root      RT   0     0    0    0 S  0.0  0.0   0:00.27 migration/1
    5 root      34  19     0    0    0 S  0.0  0.0   0:00.63 ksoftirqd/1
    6 root       5 -10     0    0    0 S  0.0  0.0   0:00.03 events/0
    7 root       5 -10     0    0    0 S  0.0  0.0   0:00.03 events/1
    8 root       5 -10     0    0    0 S  0.0  0.0   0:00.01 khelper
    9 root      15 -10     0    0    0 S  0.0  0.0   0:00.00 kacpid
   38 root       5 -10     0    0    0 S  0.0  0.0   0:00.00 kblockd/0
   39 root       5 -10     0    0    0 S  0.0  0.0   0:00.00 kblockd/1

22:20 killed both PMCR and PMCL

22:24 turned off telemetry for PMC by editting lbt.conf (with vi). Restart PMCL and PMCR but they didn't start in the correct amount of time.

22:28 killed both PMCR and PMCL

22:29 Restart PMCL and PMCR Called Chris.

22:34 PMCL comes alive after 6 minutes.

22:40 Stop LSS, but LSS wouldn't stop.

22:41 Kill LSS, reports LSS not running.

22:41 PMCR comes alive

22:42 Start LSS.

22:44 pointto BS9177 GUIDE again - fails without reporting events.

Last real event in /lbt/log/events was at 22:08.

[telescope@lbtmu104 log]$ ps aux |grep LSS 1033 23650 0.5 2.6 78048 27044 ? Sl Mar31 10:16 LSS 1033 25765 0.0 0.0 5060 644 pts/3 S+ 22:50 0:00 grep LSS

22:52 Kill LSS, reports LSS murdered.

22:53 Start LSS on lbtmu105

22:53 pointto BS9177 GUIDE again - this time events are being reported by IIF, but LSS gets stuck again partway through the preset.

/lbt and /Repository seem to be mounted twice. Is that OK?

[telescope@lbtmu105 ~]$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      67829336  24893724  39490060  39% /
/dev/sda1               101086     25683     70184  27% /boot
none                    517276         0    517276   0% /dev/shm
nfs:/exports/homes           -         -         -   -  /n/nfs/exports/homes
nfs:/exports/homes/admin/ftp/pub/
                             -         -         -   -  /opt/admin
nfs:/exports/HDF             -         -         -   -  /opt/HDF
nfs2:/exports/Repository
                     2879867392 2150989472 582589056  79% /Repository
nfs:/exports/lbt     2879867392 2150989472 582589056  79% /lbt
nfs:/exports/lbt     2879867392 2150989472 582589056  79% /lbt
nfs2:/exports/Repository
                     2879867392 2150989472 582589056  79% /Repository
nfs.lbt.as.arizona.edu:/exports/homes/telescope
                     2879867392 2150989472 582589056  79% /home/telescope
nfs.lbt.as.arizona.edu:/exports/homes/cjb
                     2879867392 2150989472 582589056  79% /home/cjb

23:01 killed LSS. ECS starts updating its private flat file as expected.

23:02 start LSS and it starts writing events to the "events" flatfile again.

23:06 another preset hangs LSS again at about the time GCS starts looking for a guide star.

23:07 kill LSS, start LLS

23:08 pointto BS9177 TRACK succeeds, but events have hung up again. Last events are:
Wed Apr  1 23:08:31.877 2009 54923.25592450998 5 PSFL psf.primaryMirror.command.adjustMirrorCollimation.started left left PSF primary mirror adjustMirrorCollimation started for 0 seconds in the future
Wed Apr  1 23:08:31.882 2009 54923.25592456822 5 PSFL psf.tertiaryMirror.command.adjustMirrorCollimation.started left left PSF tertiary mirror adjustMirrorCollimation started with rzColl 0 for 0 seconds in the future
Wed Apr  1 23:08:31.883 2009 54923.25592458122 5 PSFL psf.secondaryMirror.command.adjustMirrorCollimation.started left left PSF secondary mirror adjustMirrorCollimation started for 0 seconds in the future

Try to authorize IRTC Direct

23:12 kill LSS, restart LSS

23:15 pointto BS9177 TRACK but it is rejected by Jose.

23:17 LBC pointto BS9177 - succeed with events and warnings.

Didn't get the "complete" event for authorizing left LUCIFER.

23:19 LUCIFER pointto BS9177 TRACK - LSS hangs again

23:22 stop OSS - wouldn't stop; kill OSS - not running, restart OSS. Running on lbtmu101.

23:24 kill LSS; start LSS

23:25 authorize LUCIFER/LBC, didn't get the complete command again for LUCIFER.

23:26 stop OSS. kill LSS. start LSS.

23:27 authorize LUCIFER/LBC, completed on both sides

23:33 roll back to OSS version of build 23 from Monday afternoon. start OSS but it failed to run.

23:37 undo my rollback, start the MArch 31 version of OSS. Moved M3 selector a few degrees to test aliveness.

23:41 pointto BS9177 TRACK - preset completed without freezing LSS. OSS did not move M3 to correct location.

23:45 Stop and restart OSS to continue TCS SW Test 13 a second time.

23:47 pointto BS9177 TRACK without any reauthorizations. Preset succeed except didn't move M3. So TCS SW Test 13 has failed because it didn't move the mirror, although it did eliminate the error message about no focal station.

23:48 pointto BS9177 GUIDE - worked correctly but didn't like guiding on dome lights.

High wind continues with gusts above 20 m/sec.

00:15 TCS SW Test #14

00:16 Authorize IRTC; irc GetFilter works

00:18 Authorize LUCIFER; irc GetFilter still works

00:19 service irs stop

00:20 irc GetFilter correctly fails and reports irs not running.

00:21 pointto BS9177 GUIDE gs=0 - succeeded up to finding a guide star.

00:28 service irs start

SW Test #14 is a success.

00:30 open vent doors halfway and start mirror ventilation. VATT is reporting 1.3 arcsec seeing. Mirrors and telescope are actually pretty close to ambient temperature.

01:23 Jesper has restored the pcsmx flag back to the original value of +1 as it was before our experimenting at the end of last night.

01:25 Stop GCSL. Stop and restart oacserver for the above change to take effect.

01:26 pointto BS9177 GUIDE gs=0 for practice. Invalid buffer RPC error = GCS has stopped.

Apr  2 01:25:39 lbtmu107 LBT_left_GCS: brightest star @ 2.260000, 0.000000. FWHM = 3.290000, Elipticity = 0.734000, Flux = 267.141907
Apr  2 01:25:39 lbtmu107 LBT_left_GCS: PROFILING: end of search.
Apr  2 01:25:39 lbtmu107 LBT_left_GCS: found guide star in image at 2.260000,0.000000.
Apr  2 01:25:43 lbtmu107 LBT_networkserver: GCSL died unexpectedly
Apr  2 01:25:43 lbtmu107 LBT_networkserver: GCSL is not active
Apr  2 01:25:43 lbtmu107 LBT_networkserver: dropping GCSL

01:27 Restart GCSL

A few probe tests as LUCIFER

01:29 pointto BS9177 GUIDE gs=7 PA=285 - succeeds up until not finding a useful star

Apr  2 01:29:25 lbtmu107 LBT_left_GCS: AGWUnitAIP: internal transformation to AGW native coords: -0.805557, 429.435528 mm
Apr  2 01:29:25 lbtmu107 LBT_left_GCS: AGWUnitAIP: external transformation to AGW native coords: -1.717290, 428.566186 mm (oacontrol)

01:33 pointto BS9177 GUIDE gs=7 PA=0 - succeeds up until not finding a useful star

Apr  2 01:33:17 lbtmu107 LBT_left_GCS: AGWUnitAIP: internal transformation to AGW native coords: -176.013656, 563.001505 mm
Apr  2 01:33:18 lbtmu107 LBT_left_GCS: AGWUnitAIP: external transformation to AGW native coords: -177.160332, 562.762536 mm (oacontrol)

A probe test as IRTC

01:36 Authorize IRTC

01:37 pointto BS9177 GUIDE gs=7 PA=0 - Note that coordinates are appropriately the same for both instruments.
Apr  2 01:37:17 lbtmu107 LBT_left_GCS: AGWUnitAIP: internal transformation to AGW native coords: -176.008512, 563.006161 mm
Apr  2 01:37:17 lbtmu107 LBT_left_GCS: AGWUnitAIP: external transformation to AGW native coords: -177.155184, 562.767194 mm (oacontrol)

Open

03:00 Open dome with wind down to 16 m/sec. T=-3.7 degC D=-12 degC.

03:09 pointto M5_1416 ACQUIRE gs=0 PAM- - telescope and buidling collided. Was this wind or was it the standard problem that we were having last night?

03:14 hbsstart won't turn the oil on, so it seems to be the same problem we had last night. Stop MCSPU and restart.

03:17 pointto M5_1416 ACQUIRE gs=0 PAM-

03:20 Restore CA=-59 from yesterday as a guess. Clear M2 offload that had accumulated. But still no star.

03:22 spiral found star at offset 30 30. We actually did have a star image for about 4 minutes.

Issue 2108

03:26 wierd building event when I sent an offset. AZ=363 EL=80 Azimuth servo stuck at 363 but encoders reporting 297.

03:31 Polys stop ????????? after Aaron restarts MCSPU.

03:32 resend preset.......to recover polys, but MCSPU complaining that polys are in past.

03:33 Restart PCS and MCSPU this time.

03:34 pointto M5_1416 ACQUIRE gs=0 PAM- telescope OK now.

MCSPU complains of bad polynomials, but then stops reporting telescope position and it runs away about the same time we stopped PCS.

03:45 Close shutter doors while we sort this out. Open vent doors.

03:50 Restarted MCSPU with same failutrre

03:56 repoint mcstemp to the /home/tsargent/feb11MCSPU/ directory.

03:57 Start this old MCSPU

03:58 pointto M5_1416 ACQUIRE gs=0 PAM-

04:00 Reopen shutter doors.

04:05 Both left shutter door drives trip off (probably a gust of wind). Send Aaron upstairs.

04:06 El axis fails as Az axis was failing a few minutes ago. Strip encoder stops updating and telescope drifts away

Close

04:10 Close both shutter doors after Aaron resets SD0501.

04:15 Put EL axis in motor encoder mode as it is willing to track OK.

04:20 Wind is back up to 20 m/sec, so we decide that we are done for the night, and send Jesper off to lunch.

04:24 Telescope runs away in Azimuth as strip encoders fail again.

04:26 Restore mcstemp link to the new version of the code that Tom installed today, as the MCSPU code does not appear to be the issue.

04:27 Restart MCSPU to test telescope in motor encoder mode.

See Issue #2108.

04:46 Park telescope after 20 minutes of successful tracking with the motor encoders.

The very short plan for tomorrow

1) Verify GCS WFS convergence on sky with new pupil parameters

2) Take intra and extrafocal images with guider to check the alignment of the focus stage.

3) Adjust the AGw probe encoder zeropoints to get probe on the rotator center, and measure the position of the rotator axis at the same time.

4) Check IRTC position angle, apply an offset to LEFTZEROPOINT to make measurement unambiguous [Verified that IRTC is not mounted upside down on 05-Apr-2009 UT]

5) Collect AGw transform data with IRTC (good seeing)

6) Check the Azimuth pointing in the wrap/unwrap state in the north (seeing not critical) [Minimal measurement done on 03-Apr-2009 UT]

See Joar's document for the long plan.........

-- JohnHill - 02 Apr 2009
Topic revision: r7 - 07 Apr 2009, JohnHill
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback