19 December 2008 UT

morning Doug & Mark brought the new F972N20 filter back to Tucson to get a better scan of it.

afternoon reviewing logs from last night. Suspicious of filter moves from/to B-BESSEL.

14:30 Started an OB to switch back and forth from U-BESSEL to B-BESSEL. 30 dithers, 0" telescope disconnected and at horizon. Starts with 213156 (20081218 still UT date). Completed with no error.

afternoon Work on loading new firmware to Red controller to reset after a intermittent problem.

15:30 Vincenzo & Fernando running testfilter and testrotator to check filters and rotator movement on LBC-Red. The Red cryostat is still in the lab. The filter alignment is good still - no need to edit the positions in the configuration file. One wheel does end up a few millimeters off - this causes the filter name to be followed by a ? in the User Interface.

16:00 Running an OB with many dithers which switches between B-BESSEL and V-BESSEL and takes a 20s exposure at each. Starts with 221312. Completed without error.

16:30 Going to mount LBC-Red

17:30 Turning on Red for first time. Camera & Trackers did not come up, but explained because PC was already on. Need to update software & drivers. Turn off/on 2x to accomplish this. PCI link not working.

17:45 Taking biases with both red and blue. Blue look fine. Red need adjustment of levels. There is pickup noise in the red chips: Chip #1 seems to be worst, in a 25 pix square box, std/median = 6.5/310 ADU for chip 1 and is 5/560, 5.5/380 and 5/340 for chips 2, 3 and 4, repsectively on image number 005258.

18:00 New OB to take dome flats". Both red and blue camera controller errors - I think this is related to the software change(?)

2008/12/18 01:05:44.822000 E B CAMERA CCDCTRL "ok" (code:1) controller error [ccdctrl.c:1739]

2008/12/19 01:05:44.826971 W B CAMERA CCD Controller error errcode:8 [src/camera/camera.c:1194]

2008/12/19 01:05:44.827014 S B CAMERA reinitializing CCD Controller start

2008/12/19 01:05:46.244000 E B CAMERA CCDCTRL "LCA not loaded" (code:48) controller error [ccdctrl.c:2248]

2008/12/19 01:05:47.341000 E R CAMERA CCDCTRL "ok" (code:1) controller error [ccdctrl.c:1739]

2008/12/19 01:05:47.344767 W R CAMERA CCD Controller error errcode:8 [src/camera/camera.c:1194]

2008/12/19 01:05:47.344839 S R CAMERA reinitializing CCD Controller start

Andrea made some changes and we turned off/on camera to reload.

18:30 While comparing header info (rotator angle, filter, dither, elevation) with wiki notes for 17 December UT, noticed that rotator angle reported for the first integration after a slew differs significantly from the angle reported for the rest of the integrations. Parallactic angle also differs. Does this mean that the parallactic angle is read during the slew and reported incorrectly for the first observation? Is the rotator in the correct position and the feedback incorrect?- we cannot check this on the 'inside of dome' observations.

18:50 Running OB again - this time successful. It has 8 dithers and uses V,B (B) and V,Y(R)

19:19 Red rotator encountered a problem, but playing the OB again recovered.

20:15 Telescope ready. So we slewed to take a test OB, but each time we sent "play" to slew, it reported an "invalid buffer RPC error", PCS crashed and the subsystem was killed. All worked fine last night and Andrea did not change anything. But a difference is that tonight we are using red channel.

Error in /var/log/messages for PCS is:

Dec 18 20:11:17 lbtmu102 LBT_IIF: IIF:Preset execution time: 3492 ms at 1229656277567

Dec 18 20:11:17 lbtmu102 LBT_networkserver: PCS died unexpectedly

Dec 18 20:11:17 lbtmu102 LBT_networkserver: PCS is not active

Dec 18 20:11:17 lbtmu102 LBT_networkserver: dropping PCS

and in lbc.log we have:

2008/12/19 03:11:14.078328 N - TCS IIF_PresetTelescope to equatorial (27.500000deg,+25.000000deg) and equinox 2000.0 started

2008/12/19 03:11:14.081278 N - starting dithering step 1/1

2008/12/19 03:11:14.081461 N - SLAVE left

but then

2008/12/19 03:11:18.145602 W R TCS IIF_PresetTelescope or IIF_OffsetTelescope error: Invalid buffer: RPCERROR [src/tcs/tcs.c:827]

which is what the TCS reports - an "invalid buffer: RPCERROR".

Andrea says that LBC in the past has sent presets to BOTH sides. The PCS code was 'hardwired' to interpret this only as LEFT. When LBC-Red was disabled he changed the code to sent the preset (SetNewTarget) only to the LEFT. Sometime while the LBC code was chnaged to send presets only to LEFT, the 'hardwired' features of the PCS were removed, so today after Andrea had changed the LBC code to send presets again to BOTH, the PCS could not interpret this correctly. We called Michelle and she is changing some of the TCS code and sending a new patch.

21:45 new PCS version. This works. Michelle clarified that the 'hardwired' features were not removed from the PCS but they were hardwired in a more global way, which was tested but not as fully as was possible today with both sides of LBC up and running and the LBC code changed accordingly. Issue #1907 covers this.

21:55 left mirror went down when we slewed. Nevertheless, OB is going along OK. It slews and takes 3-sec images at all filters on Blue & Red. Only 1 dither position.

22:00 noticed lbcarchive was full. /data directory was 100%. Deleted some files after consulting with Dave T. and sent an issue trak #1908 about this to action Riccardo & Federico to check the archive.

22:30 sending a new OB to do 5 dithers, cycling through all filters at each position: Blue --- V,B,U,r,g,Us and Red --- V, Y, I, R, r, i, z. Blue controller error occurred at ~22:35 between the U and r-SLOAN filter. Last saved was 053133 at B. Elev=57 degrees.

2008/12/19 05:33:38.857000 E B CAMERA CCDCTRL load FPGA on SPC-A200 error (CCD Controller fiber link failure) [ccdctrl.c:860]

2008/12/19 05:33:39.669000 E B CAMERA CCDCTRL "FPGA not loaded" (code:48) controller error [ccdctrl.c:2251]

22:40 Powered off/on LBC - up now. Try OB again - it worked ok.

23:30 Slew to another elevation, rotate camera by 180 degrees and repeat the OB. Failed with Blue Controller error. Last image saved was 062844.

2008/12/19 06:41:34.125050 E B CAMERA reinitializing CCD Controller failed

2008/12/19 06:41:34.125128 E B CAMERA timeout on CCD Controller [src/camera/camera.c:1093]

23:45 Turn off/on LBC and try OB again. It failed with same error after not long.

00:00 Stopped tracking to maintain telescope at elevation=83 degrees since that seems to be a critical position. Crashed at about 00:08.

00:20 Moved down to lower elevation.

01:15 Calling it a night as there is enough information on the failure of Blue communications. It seems to be related to a combination of rotator & elevation angles causing stress to the fibers. In the morning, we will need to move the telescope to investigate this further. VIncenzo, Fernando & Andrea will need use of the telescope during the day, starting around 8 or 9 am.

-- OlgaKuhn - 18 Dec 2008
Topic revision: r3 - 19 Dec 2008, OlgaKuhn
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback