Telescope Performance Verification
with IRTC at LFBG - 22-23 May 2009 MST

Observers: J. Hill (Tucson)
Telescope Operator: D. Gonzalez Huerta
Software Support:
Telescope Support:

Summary

Not raining, but still cloudy on Mt. Graham. So we start closed-dome tests of Build 27 under CentOS5. Found one error in the lbt.conf file, otherwise it was boringly functional.

Then we switched back to the CentOS4 TCS to test LBC with Build 27. Had a problem with PMCL not wanting to connect to VxWorks from lbtmu103.

LBC seems happy with B27 under either CentOS4 or CentOS5, so I think the transition could happen any day next week.

IRAF/IRTC also seems happy with B27. Stephen rolled back the irc changes this afternoon until Dave T. is in town to sync LBTtools with them.

Preparations:

Details (Time in UTC)

Dome-closed tests of B27 under CentOS5

Dome closed because of clouds - despite being in the clearest county in Arizona.

Problem 1

03:11 authorize IRTC, followed by pointto M5_1246 doesn't raise any activity in IIF. It looks like the irc authorize command isn't connecting.

May 23 03:12:38 lbtmu302 LBT_IIF_IRTC_bentGregorianFront: No RPC server found on  10.144.0.102 at port 50021 for function IIF:RotNextPosition:execute

It was a problem in /lbt/tcs/current.centos5/etc/lbt.conf
iifIP                                   string  10.144.0.102  
changed to 192.168.0.17.

03:26 Issued a guide preset, but GCS crashed instantly.

03:28 restart GCS, and selectAGW. David trying to restart the AGW.

03:38 pointto M5_1246 Tel="GUIDE" gs=0 PAM+ PA=0 - succeeded up until acq image saturated. Turn down lights.

03:41 pointto M5_1246 Tel="GUIDE" gs=0 PAM+ PA=0 - succeeded up until there was no star.

GCS.starOverlay bool true

03:45 pointto M5_1246 Tel="GUIDE" gs=0 PAM+ PA=0 - didn't get the fake stars. Stop GCSL. selectAGW.

03:47 pointto M5_1246 Tel="GUIDE" gs=0 PAM+ PA=0 - happily guiding on a fake star with 0.19 arcsec seeing.

03:53 focus 0

03:54 Remove corrections of M2 offload on M1 in PSF.

04:14 focus 0 - This adds some stray secondary offloads. -0.104 0.005 -0.119 1.485 -0.573. Maybe we want to completely suppress this calculation when there have been no active updates to M2?

04:20 offset 0 10

04:21 Open vent doors to cool down the dome. Inside=12 degC, Outside=2 degC

04:24 offset -10 0

top - 04:29:57 up 11 days,  7:40,  1 user,  load average: 16.90, 13.80, 13.38
Tasks: 113 total,   1 running, 112 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.1%us,  1.1%sy,  0.0%ni, 93.3%id,  0.2%wa,  0.1%hi,  0.2%si,  0.0%st
Mem:   8313508k total,  2047208k used,  6266300k free,   210632k buffers
Swap:  4194296k total,        0k used,  4194296k free,  1235116k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19563 telescop  22   0  406m  27m 5744 S 22.0  0.3  82:49.80 PMCL
19516 telescop  15   0  6456 2712 2476 S  1.0  0.0   3:33.03 gshmserver
20138 telescop  18   0  346m  43m 7932 S  0.7  0.5   0:25.38 GCSL
23631 telescop  15   0  2368  964  756 R  0.3  0.0   0:00.06 top

04:32 setGuideRate 0.8

04:33 pointto M5_0531 Tel="GUIDE" gs=0 PAM+ PA=0

top - 04:39:13 up 11 days,  7:49,  1 user,  load average: 16.20, 16.49, 15.16
Tasks: 113 total,   1 running, 112 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.3%us,  1.2%sy,  0.0%ni, 92.7%id,  0.3%wa,  0.1%hi,  0.3%si,  0.0%st
Mem:   8313508k total,  2058796k used,  6254712k free,   210636k buffers
Swap:  4194296k total,        0k used,  4194296k free,  1242188k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19563 telescop  22   0  406m  27m 5744 S 23.0  0.3  84:57.52 PMCL
19516 telescop  15   0  6456 2712 2476 S  1.7  0.0   3:38.48 gshmserver
20138 telescop  18   0  358m  46m 7952 S  1.7  0.6   0:30.57 GCSL
19544 telescop  19   0  939m 414m 3460 S  0.3  5.1   0:09.94 SDB
24214 telescop  15   0  2368  964  756 R  0.3  0.0   0:00.02 top

04:40 Close Vent Doors as humidity is increasing

04:43 takepic 0.1 - irtc.20090523.044328.fits

top - 04:51:35 up 11 days,  8:02,  1 user,  load average: 15.98, 14.88, 14.96
Tasks: 113 total,   1 running, 112 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.0%us,  1.1%sy,  0.0%ni, 93.3%id,  0.3%wa,  0.1%hi,  0.2%si,  0.0%st
Mem:   8313508k total,  2065484k used,  6248024k free,   210640k buffers
Swap:  4194296k total,        0k used,  4194296k free,  1248924k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19563 telescop  22   0  406m  27m 5744 S 22.3  0.3  87:49.06 PMCL
19516 telescop  15   0  6456 2712 2476 S  0.7  0.0   3:45.80 gshmserver
20138 telescop  18   0  358m  46m 7952 S  0.7  0.6   0:39.81 GCSL
 2871 root      16   0  1720  580  488 S  0.3  0.0   0:03.32 syslogd
24214 telescop  15   0  2368  964  756 R  0.3  0.0   0:01.00 top

04:52 pointto M5_0531 Tel="GUIDE" gs=0 PAM+ PA=0

top - 04:53:26 up 11 days,  8:04,  1 user,  load average: 5.14, 11.95, 13.94
Tasks: 113 total,   1 running, 112 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.2%us,  1.2%sy,  0.0%ni, 92.8%id,  0.3%wa,  0.1%hi,  0.2%si,  0.0%st
Mem:   8313508k total,  2067104k used,  6246404k free,   210640k buffers
Swap:  4194296k total,        0k used,  4194296k free,  1250316k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19563 telescop  22   0  406m  27m 5744 S 23.0  0.3  88:14.54 PMCL
20138 telescop  18   0  368m  47m 7952 S  1.3  0.6   0:41.00 GCSL
19516 telescop  15   0  6456 2712 2476 S  1.0  0.0   3:46.87 gshmserver
   14 root      10  -5     0    0    0 S  0.3  0.0   8:09.65 events/0
 2871 root      15   0  1720  580  488 S  0.3  0.0   0:03.40 syslogd

It seems that the memory leak in GCS is happening very near (within 15 sec) of the time that the acquisition image is being taken. I saw a jump of VIRT by 10m at each preset.

22:35 stopGuiding, stopAGW, Authorize as LBC.

LBC Closed-dome tests with CentOS5 and B27

05:44 lbcstart.sh, turn on

05:51 preset PC1643+4631A - Note that IIF issued warning about hitting a M1 position limit, but LBC did not.05:56

05:54 restored GCS star overlay to false.

05:56 stop the OB.

05:58 Stop all subsystems from TCSGUI.

06:01 Stop network services on lbtmu301, lbtmu302, lbtmu303, lbtmu304.

06:50 Switch to lbtmu01 and start normal CentOS4 TCS. (Norm had cleared reflective memory on the CentOS4 cluster earlier in the evening.)

06:07 lbckill; Switch iifIP back to 10.144.0.102 on /lbt/tcs/current/etc/lbt.conf on lbccontrol.; lbcstart

Problem 2

06:13 PMCL didn't connect to VXWorks. David rebooted VxWorks.

06:18 Stop and restart PMCL on lbtmu103 (where it was). Events are issued for threads connecting, but PMCGUI LEFT never got out of the Red state.
May 22 23:19:40 lbtmu103 LBT_PMCL: Actuator telemetry: No telemetry since telemeter registration failed.

06:21 Stop PMCL and restart on lbtmu101. It worked OK there.

LBC Closed-dome tests with CentOS4 and B27

06:28 preset PC1643+4631A with LBC

06:32 stop the OB

06:34 preset to virgocluster

06:43 stop the OB

06:45 Park the telescope for the night as the moist airmass from New Mexico has moved back over Mt. Graham (maybe it never moved off).

-- JohnHill - 22 May 2009
Topic revision: r4 - 23 May 2009, JohnHill
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback