BG February Com Night 1, February 1 2009 UT
Observers: J. Hill and A. Rakich in Tucson plus O. Guyon (visiting for a while), and J. Storm in Potsdam (second half)
Telescope Operator: A. Ceranski
SW Support: P. Grenz
Telescope Support: J. Rousseau
- Winter Chiller is still broken, so no mirror ventilation. This is a serious impediment to observing both because of mirror seeing and 5th order spherical aberration caused by thermal gradients in the mirror.
- Right shutter door is only openable under local control upstairs.
- LSS was throttling down all of TCS so we had to stop it. This needs some daytime TCS debugging.
- Automatic M2 Offloading and Collimation Update seems to work fine when an offset is issued.
- There is a bug in our new algorithm for offloading M2 when a preset is issued. We will want to patch PSF if Chris can figure out a fix. The main problem is that is needs more accounting when the new preset is to a different elevation angle. It works OK if you preset back to the same position.
- Moving M2 on a sphere has some big error in it. Back to the drawing board.
- Moving M3 works as expected in tip and tilt. The selector angle has units issues.
- We took some fast WFS data in poor seeing.
- Our two attempts to remove the X-Flip both failed. (and only the first one we understand; the second one acquires the star, but guides in the wrong direction)
- We got decent data to verify the field rotation on the Guider CCD. Jesper is analyzing it this afternoon.
- We tested guiding at some slightly different rates. No significant increase was achieved. We haven't investigated where the time is going. I think Torsten should put his delays back in (although it is not a solid result due to variable seeing).
- We exercised the new version of oacserver, and we want to keep it.
TCS Requests for Sunday (somewhere in between critical and high, and in order of decreasing priority)
- Roll back to Build 18 so we can use LUCIFER (according to plan)
- Keep the X-Flip True in PCS (or optionally give us two versions of PCS in Build 18)
- Replace the delays that Torsten removed from GCS
- Patch PSF for smarter Offload at preset (if possible)
- Check the signs on tip and tilt for Tertiary Mirror global offsets (Paul)
I'm assuming that we are doing all these things to Build 18 later today. Although, I'm not completely clear on what Build 18 means after all that.
TCS Requests for Thursday (priority medium to get them in the stable Feb build for LUCIFER)
- Patch PSF for using Z11/Z22 gains when calculating focus compensation
- Adjust which subsystems can run on lbtmu107 if that is what Torsten meant (see his note just below)
- Make the OSS M3 selector angle commanded and reported both in the same units (arcseconds).
- make sure GCS is running on lbtmu107 without any other CPU intensive subsystems running on the same machine. I strongly recommend that neither PSF nor PCS nor PMC are running on lbtmu107 along with GCS as I've seen GCS being very slow when this is the case and the subsystems are being exercised.
- JMH: This is not the case. GCSL, PSFL and SDB are running on lbtmu107. Isn't this a TCS configuration issue?
- get a software support person to correct for this if the previous condition is not met.
- Once GCS is guiding, watch how quickly it determines stable guiding after the acquisition. Previously, this happened after two guide iterations but that was with delays to match the guide rate. I took these delays out which means there is a potential of overcorrecting because of the mount still moving when guide images are being taken. If this is the case it might take longer to ensure stable guiding and in the worst case it might fail during that check. If this is the case, I can quickly add the delays again and ship a patched GCS on Saturday evening that does not have this speed-up attempt in it. (Torsten)
Details (Times in MST):
Startup pointing & collimation correction
17:55 Paul and John home the Tertiary Mirror and Tertiary Selector.
Running on the Summer Chiller, with no mirror ventilation.
18:00 Open Dome (left shutter only plus vent doors), T=-1.3 degC. D=-12 degC, Wind (SMT) 1 m/sec.
Startup pointing & collimation correction
18:08 pointto failed - rotator not ready
18:09 Aaron notices alarm in rotator section of MCSGUI. Are these functional? Tom S. says the thresholds may be incorrect.
18:10 pointto M5_0202 gs=0 Tel="ACQUIRE" PAM- - preset succeeded, but we didn't find the star.
18:12 Home Left M2. The OSS seems to be reporting the wrong bit for "Moving" versus "In Position". Then recollimate.
18:23 Why does PSF M1 have values in the Secondary Offload register? We aren't sure where these non-zero numbers came from.
18:28 Adjusting IE,CA to look for the star.
18:40 Verified that M2 cover is off.
18:42 Set M3 position to -615 -20 0.0 8.35 deg. M3 is live, and this is why we couldn't find the star initially. Note that Selector accepts commands in degrees, but reports result in arcsec. This confuses both the observer and the PSF. Now, we can see the star. These need to be added to the SX M3 Collimation file.
(Because we took them out in December?) I updated /home/telescope/TCS/Configuration/PSF/new/SXTMLUCIFERCollimation.dat to have these global offsets. However: Note that the tip value is now -615 arcsec, while it was +615 in the August version of the file.
18:47 Adjust IE=-42 CA=-7
(Encoders were last reset yesterday.)
18:52 Reduce Zglobal by 0.15 mm to +0.05 mm.
18:53 pointto BS9108 gs=0 ACTIVE -- GCSGUI displayed the Acquisition image (with a star) for a few seconds, but did not show the initial
guider images while checking for stability. Preset failed because guiding not stable for 6 cycles.
18:55 Put active corrections on M1 for initial collimation.
18:57 pointto BS9108 gs=0 ACTIVE --
19:04 Preset has not ended (according to IIFGUI events). GCSL is taking one guide exposure every 90 sec, and not finding the star.
19:06 Stop SDB on lbtmu107, and restart on lbt. Had to stop and restart PCSGUI and PRESETGUI to get rid of unknown RPC calls in lbtmu107 syslog.
19:10 Preset has not ended (according to IIFGUI events). getStatus command on to-station runs for minutes without finishing.
19:15 Kill and restart GCS.
19:18 Stop IIF without success. Kill and restart IIF.
19:22 stop and start irs
19:26 Stopped PSFR (to see if LSS is alive - apparently not)
19:28 Stop LSS, and restart
19:35 Paul does battle with and conquers a zombified LSS. This gets events working again, but we still can't authorize from IRAF.
19:40 Redistributing subsystems to different servers to have less load on lbtmu105.
19:48 stop and start irs again.
20:00 irc Authorize command on lbtmu05 returns a string of dashes.
20:05 irc Authorize command on lbtmu01 prints a message, but neither of them work to the point of making an event.
20:10 pointto sort of works, but in 100x slow motion.
20:20 Kill LSS, and then things speed up and the preset from 10 minutes ago finishes unsuccessfully. So it seems that LSS is somehow constipated and holding up the entire TCS.
20:23 Paul changes lbt.conf to write events to a flat file. Then restart LSS.
20:26 pointto now runs at full speed.
20:33 pointto BS9108 ACQUIRE
20:35 Things have slowed down again (but not completely stopped), so back to Paul for more LSS debugging. Things seem to be OK without LSS running at all.
Trying to guide with X-flip false.
20:40 M1 Zglobal=0.65 for new focus.
20:42 pointto BS9108 ACTIVE fails. Star is centered in Acq image, but doesn't appear on GCSGUI. Why?
20:52 pointto BS9108 ACQUIRE - Adjust IE=-41 CA=-11
for precise centering.
20:56 pointto BS9108 ACTIVE -
20:56:21 Update Origin was X=-1.167 Y=0.209, but no star is found in the guide image.
21:05 pointto BS9108 gs=14 ACQUIRE - no star
Conclude that Xflip was changed in PCS, but not in oacontrol. Called Michele and got her to give us a Build 19 version of PCS with Xflip true.
Change PCS with Xflip true
21:28 PCS Xflip=true
21:30 pointto BS9108 gs=0 ACQUIRE PAM- Adjust IE=-43 CA=-18
21:34 pointto BS9108 gs=14 ACQUIRE
21:35 pointto BS9108 gs=14 ACTIVE - Now the guiding starts correctly, and the acquisition image stays displayed until I kill it.
21:40 focus 0
21:44 pointto BS9126 gs=15 ACTIVE - didn't find star - was grossly out of focus.
Maybe the offloading of M2 on preset has a problem?
Clear the Secondary Offload from EL=47, and the image substatially improves at EL=73.
This is a bug in the new preset algorithm that we figured out later.
22:07 Send Z22=+60 nm.
22:20 Send M1 Z11=+1000. This made a big error in focus, because by design M2 focus compensation doesn't work when you apply the Z11 on the PSF primary GUI. Thus, it work as it should have, even though it surprised us.
22:25 Put focus compensation back to M1. And focus exploded to fill the guide box. We don't understand why this happened.
Put focus compensation back to M2.
22:31 Put focus compensation back to M1.
22:36 Send M1 Z11=+1000
22:40 Put focus compensation to M2.
22:56 Put focus compensation to M1. Image jumps at second correction.
23:03 Put focus compensation to M2
We think that we were getting fooled by variable local seeing messing up the WFS. At the end of the story we can't really see any particular problem with the convergence of Z4 or Z11.
23:15 Tracking and Guiding at 87 deg EL.
M2 Hexapod Move-on-Sphere Tests
Andrew and Paul are sending commands from the OSS GUI.
23:21 pointto M5_0269 ACQUIRE, then remove M2 offload corrections. Adjust IE=-37
23:26 Radius = 1979 mm
23:28 Theta = +10 arcsec, and image moves a bunch (off the acquire field)
23:29 Theta = -10 arcsec, but we didn't get image back. Was there a conflict between OSS / PSF? Eventually we recovered the collimation.
23:33 Radius = -1979 mm, star moves the other way. We had the wrong radius. Go back and recollimate.
23:39 Radius = +1067 Theta=+10 star moves. And we see that the hexapod is moving many millimeters (like from X=7 to X=14 mm). So there is a scale error someplace.
23:52 Radius +1067 Theta=+0.017. Mirror still moves alot.
23:59 Radius +1067 Theta=0, mirror still moved a little bit.
Conclude that the Move-on-Sphere code is seriously broken. Need to go back to the drawing board. Or should we abandon it?
M3 motion tests
Andrew and Paul are sending commands from the OSS GUI. We are still watching the image in ACQUIRE mode.
00:10 Move selector +0.04 degrees, but that moves out of Acq field. Need to move relative to 8.35 deg.
00:11 Move selector +0.00 degrees. Then to nominal +8.35 deg. Repeatability is good (within telescope drift of an arcsec).
00:14 Move selector to +8.39 degrees, and it seems to have moved the star the expected 6 arcsec. Star moved toward 5 o-clock.
00:16 Move RX from -615 to -687 arcsec. This moved toward 7 o-clock.
00:21 Move RY from -20 to -92 arcsec. This moved toward 5 o-clock.
11.7 arcsec of M3 tip-tilt per arcsec of focal plane pointing change.
00:30 pointto BS9139 gs=0 ACTIVE
Set all active corrections going to primary.
Fast WFS Tests
00:35 pointto BS9122 gs=0 ACTIVE
Seeing is fluctuating 0.8 to 2.0 arcsec, at EL=60, so too poor for field aberration data. Thus we are trying this fast WFS experiment to characterize the fluctuating seeing.
00:49 Collimating in IDL up through wfscimage 4 with 20 sec exposures.
00:52 Fast data starting at wfscimage 5 - 10 Six images of 5 sec exposure time.
00:54 New Collimation wfscimage 11 - 14
00:57 Fast data starting at wfscimage 15 - 26 Twelve images of 5 sec exposure time.
01:01 New Collimation wfscimage 27 - 30
01:05 Fast data starting at wfscimage 31 - 42 Twelve images of 5 sec exposure time.
01:08 New Collimation wfscimage 43 - 44
01:10 Fast data starting at wfscimage 45 - 56 Twelve images of 5 sec exposure time.
01:1x New Collimation wfscimage 57 - 59
01:16 Fast data starting at wfscimage 60 - 71 Twelve images of 5 sec exposure time.
01:20 Jesper joins us from Potsdam. We are all connected to Polycom 126.96.36.199 in the conference room.
01:xx New Collimation wfscimage 72 - 75
01:23 Fast data starting at wfscimage 76 - 87 Twelve images of 5 sec exposure time.
01:28 New Collimation wfscimage 88 - 92
01:31 Fast data starting at wfscimage 93 - 104 Twelve images of 5 sec exposure time.
X-Flip Tests - the Sequel
01:36 pointto BS9148 gs=0 ACTIVE - This preset wound up with very poor collimation, but I'm not sure why since all the corrections were going to the primary previously. Maybe even a zero secondary offload messes it up. Set Secondary Offload to no.
01:50 Stopped and restarted PCS using PCSB19XFLIPFalse
. Jesper has changed X_GP to +1 in
01:55 Aaron has to use kill -9 to get the previous oacserver to stop. (Issue 1944)
02:00 Stop and restart GCSL.
02:01 pointto BS9148 gs=0 ACTIVE - failed because primary mirror panic'd - maybe because of big Z11 corrections?
02:05 Clear Active Optics and M2 Offload while M1 is coming back up.
02:10 pointto BS9148 gs=0 ACTIVE - hung because of AGW1: RPC Communication Error from GCS.
02:13 stopAGW, startAGW, home the probe fails
02:!5 Kill GCSL, stopAGW, startAGW, home the probe, restart GCSL.
02:20 pointto BS9148 gs=0 ACTIVE - succeeded, but far from focus
02:24 pointto BS9148 gs=0 ACQUIRE
02:25 ACQUIRE mode stops unexpectedly.
02:26 pointto BS9148 gs=0 ACQUIRE PAM- Adjust IE=-37 CA=-12
02:30 pointto BS9148 gs=0 ACTIVE - guide star appears and then guides right out of the field.
02:37 Change M_X_CCD to +1
02:38 Stop GCSL, stopAGW, kill -9 on the oacserver, startAGW, home stages, start GCSL
02:43 pointto BS9148 gs=0 ACTIVE - guide star appears and then guides out of the field.
Back to X-Flip true mode
02:45 Stop GCSL, PCS, stopAGW, restart oacserver, startAGW, home stages, restart PCS, set IE,CA, restart GCSL
02:51 pointto BS9148 gs=0 ACTIVE - succeeds
02:58 guide star pops to a 3 arcsec donut, and then quickly goes back. ?? (Later at 04:16 I saw this again and figured out that it was a big slug of Z22 being applied.)
Checking AGW CCD Alignment
03:00 pointto M5_0441 ACQUIRE
03:10 pointto WT10_292 ACQUIRE
03:12 pointto WT10_292 TRACK - but that is no good because it parks the probe.
03:16 pointto WT10_292 GUIDE and then stop guiding. Star was still too bright for this experiment
03:25 pointto BS9145 gs=0 ACTIVE
03:31 pointto BS9145 gs=0 TRACK
setxy -x 0 -y 612.5
setfocus -p 31.6
03:36 readGuideCam -e 50000 while Aaron slews rotator = guider000008 in /Repository
03:38 readGuideCam -e 70000 = guider000009
03:43 Slew to M53 with PresetGUI
: 13 12 36.26 +18 11 13.4 PA=0
03:46 10 sec = guider000010
03:47 15 sec = guider000011
03:48 15 sec x 5 = guider000012 - 16
03:50 Preset with PA=30
03:52 15 sec x 6 = guider000017 - 22
03:53 Preset with PA=90
03:54 15 sec x 6 = guider000023 - 28
Guiding, Offloading, Offsetting Tests
04:00 pointto BS9148 gs=0 ACTIVE - failed because SX primary mirror panic'd.
04:05 Clear C00 and recover panic.
04:10 pointto BS9148 gs=0 ACTIVE - succeeds with really strong spherical aberration in the image.
04:16 The "ring" guide star appeared again. This time I saw it coincide with -412 nm of Z22 being applied (essentially our maximum possible Z22 correction). We probably needed to suppress Z22 in the unventilated mirror even though this is when we need it most.
04:18 Set the Z22 gain to zero in order to avoid another panic.
04:20 Seeing was briefly below 0.5 arcsec, but it didn't last long.
04:23 Send active corrections to "split", and focus compensation fro Z11 to M2.
04:24 setGuideRate 1.0 (Exposures are 647 msec), EL=81.7 degrees. getStatus says the Effective Guide rate is still 0.18 Hz.
04:31 setGuideRate 0.4 , now the Effective Guide rate says 0.15 Hz.
04:45 setGuideRate 0.2, now the Effective Guide rate says 0.19 Hz, so it seems that setting the Guide rate isn't really changing anything with this varying short exposure.
The image has been bouncing around a fair bit, so I conclude that we should put the delays that Torsten removed back in.
The image seems to be popping back and forth between in-focus and defocussed. Chris: Are you sure that the Z11 and Z22 focus compensation is
checking the gain? Or is the gain too deep inside PSF? GCS is still sending Z22 corrections even though I have the gain set to zero in PSF-M1. I suspect that the new higher level focus compensation is still sending the Z22 focus correction even when the Z22 gain is zero.
04:54 Set Z11/Z22 focus compensation to M1. This caused a huge collimation update since M1 hadn't been updated in quite some time. And we lost the guide star.
04:58 pointto BS9148 gs=0 ACTIVE - succeeded, although star is off in the corner of acq image.
05:02 This is the nicest guided image we have seen in a while, even though the primary collimation is updating. It does have the traditional primary jumps.
05:10 offset 10 0 (Still with M2 Offload turned off, so it only updates collimation without offloading from M2)
05:17 offset 30 0
05:22 Turn On Offloading of Active Corrections on M2
05:23 offset -30 0 - This did the expected M2 offload as part of the offset command. (We don't have events, but timing should still be in the syslogs.)
05:26 offset 0 60 (can't move probe there)
05:27 offset 0 -60
05:37 focus 0
05:54 Set the Z11/Z22 focus compensation to M2
05:58 For sure the image jumps out of focus when a Z22 is sent from GCS, but not used due to zero gain. We can suppress Z22 from the GCS configuration files if need be.
06:15 Close Left Shutter Door. Sky was photometric all night.
LUCIFER will be available via remote operations in the second half of the nights as needed. If the weather looks bad (but the current forecast is mostly clear) email Nancy by 9:00pm to let her know we won't need LUCIFER support.
- Check AGW software auto-start
Check if the AGW software (check4nport and oacontrol) auto-start after a normal shutdown and reboot of the agw-control computer. If it does not work, then we must continue doing a manual restart with the old file.
Tests that can be done without LUCIFER:
First half of the night:
- Startup pointing & collimation correction (10 minutes)
This may require a spiral search to find the star. Use the pointing model stars (WT10 or ACT) as the potential for confusion is less than with Persson standards for the pointing correction, then any usable guide star for collimation.
- Fast Wavefront Sensing on Bright Star (30min each iteration; Doug, John, Andrew)
It would be useful to gather multiple sets of data in different conditions (equilibrium temps or not, bad seeing vs. good seeing). This test is repeated below.
- TCS verification and other SW group tests (Norm)
- Wakeup checks
- PCS Xflip check w.r.t. AGW transformations (postponed). This feature is dependent on Lucifer software being upgraded to work with the newest version of the IIF. The software to support this feature will not be included in the B18 version of the TCS.
- PCS (non)filtering check (closed dome)
- Speed up of maximum guide rate to 1Hz (low priority)
- Test of new TCS features on-sky (2 hr, John)
- Test new GCS and GCSGUI image displays
- Sign test Z4 offload to M2 in PCS mode 3d (stable or runaway?)
- Verify that M2 offload happens automatically on presets and offsets. This feature is dependent on Lucifer software being upgraded to work with the newest version of the IIF. The software to support this feature will not be included in the B18 version of the TCS.
- Verification of M3 motions (Andrew, Paul, 30 min)
- Calibration of coordinate system
Second half of the night:
- Verification of OSS/M2 move-on-sphere or coordinated motion (Andrew, Paul) (30 min, low priority)
- Field Aberration Stability Test (2 hr, Andrew)
M5 Stars Sky Maps
Persson's Star =
Start time =
| Guide Star
|| Doug's Angle
End time =
- AGW CCD Orientation recalibration (Jesper, 30m)
Recalibrate the CCD to SFP transformation now that the LUCIFER true north has been determined. This would require a number of exposures of an astrometric field with different values of PA. This would also require 30min of observing time.
- AGW Non-orthogonality test (Jesper, 30min, details below, requires LUCIFER)
- Fast Wavefront Sensing on Bright Star (Doug, John, Andrew) - details below
- login as LBTO on lbtmu107
- make a directory for the data and cd there ("%" is shell prompt)
% mkdir /Repository/AGW_Data/090116w_fast
% cd /Repository/AGW_Data/090116w_fast
- determine the current WFSC integration time
**Note, getStatus reports exposure time in milliseconds
- Start IDL (">" is the idl prompt)
> agw_wfsc_multiple, 1, EXP=1 (the first image will probably fail to display: bug in my IDL code)
> agw_wfsc_multiple, 1, EXP=10 (2nd wfsc image will display)
**Note, 1 is number of exposures to take and EXP=10 is integration time in seconds
- collimate the telescope
> agw_wfsc_collimate, EXP=30
> agw_wfsc_collimate, EXP=30
> ... As many collimations as is needed to get WFE<400 nm rms
- if exposure time is good then take at least 60 seconds worth of integration time
> agw_wfsc_multiple, 6, EXP=10
***Note, Crtl-c will stop collection of data and leave everything in happy state
- repeat step 5-6 above three times
- exit IDL
> exit or Ctrl-d
- when finished set wfsc exposure time back to original (say 30 seconds)
% setWFSExpTime 30000
Doug's email from Jan. 18 about collecting fast WFSC data
It would be useful to gather multiple sets of data in different conditions (equilibrium temps or not, bad seeing vs. good seeing). This test is repeated above.
- Collimation model (1-2 hr, low priority, could be skipped)
Tests that need LUCIFER imaging:
- Re-measure AGW-LUCIFER focus (1 hour, postpone to LUCIFER Com4)
- Use several different off-axis guide stars are different rotator angles.
- Use LUCIFER internal focus motion to speed the data acquisition and avoid drifts of temperature or seeing.
- Active optics: Delivered IQ Tests (2 hours)
- Mode 3d (deprecates 3c): Z11 and Z22 applied with focus portion applied to M2.
- Standard mode: All active optics sent to M1
- Useful to repeat 3b (Z11, Z22 FCIN=1) for comparison
Take 15 minutes data in ACTIVE mode in these configurations. If possible, take a series of 5s LUCIFER exposures during each run to compare centroid offsets on LUCIFER to the same data on the guider images. Compare to results of 3a and 3b tests in January.
- AGW Non-orthogonality test (Jesper, 30min)
We will need a number of LUCIFER images with a probe offset to different positions to confirm that the resulting offsets in the LUCIFER focal plane are indeed linear and not S-shaped.
- AGW Coordinate Transformation Calibration (Jesper, 2 hours)
Repeat coordinate transform calibrations (2hr): There seems to be a pointing offset in prior data of the order 5mm which I still have not been able to pin-point. It would be very useful to get a new dataset after we have confirmed the rotation axis location in the AGW coordinate system, on LUCIFER, and with a freshly reset telescope pointing.
- Alignment of chief ray to LUCIFER pupil with M3 (Andrew, Walter, John, 2 hours)
- Point to bright star
- Insert LUCIFER pupil viewer
- Take Pupil image
- Offset rotator 180 deg
- Take Pupil image
- Iterate M3 tip-tilt to minimize the pupil motion
- Compensate pointing with IE,CA
- Pointing Model (low priority, 3 hours)
- 15 Jan 2009