Observing Log for PISCES night of 02 March 2012 UT

AZ / OSURC mini-queue with Hill et al., Close et al., McGreer et al., Woodward et al.,

Observer: JHill (LBTO wearing his AZ observer hat)
AO Support: DMiller (Tucson), JCGuerra (LBTO)
Telescope Operator: SAllanson (LBTO)
Telescope Support: JLittle (LBTO)
Instrument Support: TShih (LBTO)

Summary

We opened 1hr late because of high winds. Sky is mostly clear. Seeing continues to be poor (1.5 - 3 arcsec on guider) or worse, and this is giving alot of trouble for the AO loop. We got 28-minutes of data on J0946+1835, but nothing useful at all after that. The AO loop can't handle the seeing jumping to 3 arcsec or worse. We've used ~8 hours as an opportunity to practice sending a large number of ADAPTIVE presets, and to experiment with keeping the loop closed in poor conditions. With Alfio's assistance we've worked out the 100-mode reconstructor that can keep the loop closed on bright stars in seeing up to 3.5 or 4 arcsec. This was very educational.

Here's what John learned about AO in bad seeing:

• For bright stars with 400-mode correction, instances of bad seeing send huge slope corrections to M2 and skip frames cause the shell to RIP.
• We can keep the loop closed by zeroing the gain of the high -order modes and thus reducing the high forces to the shell.
• The trick described by Simone yesterday from SX is to zero-gain the 25 high-stress modes, and then increase the number of modes to 500. This makes a 500-mode reconstructor more resilient to bad seeing (we were only using 400 modes).
• For faint stars, there is a second problem in bad seeing. The Pyramid has a 1.5 arcsec field stop, so that light is lost when the PSF is larger than 1.5 arcsec. This can turn the faint star into a much too faint star. In fact we see this manifested in the CCD39 reporting a fainter magnitude as the seeing gets worse. In this case the loop opens for lack of photons rather than severe wavefronts.

Data Summary for Tonight

• Took 28 minutes of Ks data on J0946+1835 for McGreer & Fan (~1:00)
• Took tens of Ks ADI frames on HD92788 (not enough to be useful)

Bad Images

Images 10048, 10332, 10570 appear to be all zeros on ds9. However the corresponding raw images w/o full headers seem to be OK. See Issue 3980

Preparations

Details

02:00 UT - We are closed for high winds. Winds are presently gusting to 22 m/sec from the SW, which is an improvement from the 30+ m/sec winds a couple hours ago. T=-4.3 degC, D=-17.8 degC. Sky is clear but not photometric.

Open

02:30 Wind is down to 18 m/sec, so we are open. We will need to work in the east until the wind drops more. We've opened too late to hit any of the RC targets in the West. We'll start with a McGreer et al. target since they don't have any data yet.

02:35 ACQUIRE preset to a pointing star

02:38 ACTIVE preset - seeing is 2 arcsec on the guider

Observations are working from /OldRepository/PISCES_Data/Feb12_02/ even though it is March 02

J0946+1835 for McGreer & Fan

02:46 ADAPTIVE ACE ACTIVE preset for J0946 - failed for a bug in IDL reading target_new.dat (We had switched from AOeng to LBTO account and were missing some updates.)

02:49 ADAPTIVE ACE ACTIVE preset for J0946 PA=-71 gs=7

02:55 Loop opened before RunAO finished. "wfsarb: Illegal command for state AOSet" It was skipping frames before this.

02:58 resend the preset -- estimated magnitude 15.1 (catalog 14.4) 4x4 binning

03:05 Loop is closed

60 s Ks images start at 012

03:08 Loop opened before we finished the first image. so images 014-17 are open loop

03:10 resend preset

Seeing on the DIMM is 2 arcsec.

03:17 Loop is closed - starting with image 018

We couldn't see the QSO in a single 60 sec Ks image, but differencing two of the dither positions shows that the binary structure is clearly there and well centered. The image isn't especially sharp, but it is lots better than 2 arcsec seeing.

AO parameters: binning=4x4, TT modulation=6, 36 modes, 100 Hz, 15.78 mag, tt_gain=3.6, ho_gain 2.7,

03:47 Shell RIP'd during image 044 on the final dither position

03:52 ADAPTIVE ACE ACTIVE preset to J0946 gs=7 PA=264 (starting a new PA and dither pattern)

03:57 StartAO failed (apparently slope computer stuck, but no error message yet)

04:00 Command timeout error arrives

CCD39 is stuck, and resists JC attempts to get it running again.

In the meantime, JH is taking 60 sec dark frames. Image 048 is a bad readout. Image 049 is a dark. Some confusion after that. dark60.obs in PISCES_Data/Feb_Calib/scripts/

(suggestion: don't check for loop open when filter=dark)

It seemed different to me last night, but here is how it is working tonight:
PISCES writes an image pisces.20120302.0048.fits. If IDL "observe" thinks this is a good image (with loop closed), it copies it and adds header information as pisces.20120302.10048.fits. If not, then only the original image is in /newdata/

04:20 Resend the preset, but now the rerotator failed. Seeing on guider is 1.4 arcsec.

04:29 Resend the preset again.....

BCU47 temperature is 7.7 degC. Ambient is -4.5 degC. Now AO reports the magnitude as 14.8 (better seeing) - momentarily.

04:35 Loop closed - first Ks image here is 058.

04:37 Loop opened - skip frames when seeing blew up. Loop opened in the last few secs on image 058. Images through 061 are open loop.

04:38 Resend the preset again.....

04:43 Shell was closed for ~30 sec because shell RIP'd.

04:49 resend the preset: observe, 'scripts/J0946_presetK_v2.obs'

04:52 RunAO failed with timeout error

04:54 resend the preset: observe, 'scripts/J0946_presetK_v2.obs'

05:00 RunAO failed in auto gain. We think the BCU47 may be getting too cold as it gets colder outside.

Object: Feb12_J0946
Dither
X	Y	frames	Exposure	Filter	PISECS Image
0.00000	0.00000
"	"	1 of 6	60.0000	Ks	pisces.20120302.10012.fits	SCIENCE
"	"	2 of 6	60.0000	Ks	pisces.20120302.10013.fits	SCIENCE
Object: Feb12_J0946
Dither
X	Y	frames	Exposure	Filter	PISECS Image
0.00000	0.00000
"	"	1 of 6	60.0000	Ks	pisces.20120302.10018.fits	SCIENCE
"	"	2 of 6	60.0000	Ks	pisces.20120302.10019.fits	SCIENCE
"	"	3 of 6	60.0000	Ks	pisces.20120302.10020.fits	SCIENCE
"	"	4 of 6	60.0000	Ks	pisces.20120302.10021.fits	SCIENCE
"	"	5 of 6	60.0000	Ks	pisces.20120302.10022.fits	SCIENCE
"	"	6 of 6	60.0000	Ks	pisces.20120302.10023.fits	SCIENCE
-3.00000	3.00000
"	"	1 of 6	60.0000	Ks	pisces.20120302.10024.fits	SCIENCE
"	"	2 of 6	60.0000	Ks	pisces.20120302.10025.fits	SCIENCE
"	"	3 of 6	60.0000	Ks	pisces.20120302.10026.fits	SCIENCE
"	"	4 of 6	60.0000	Ks	pisces.20120302.10027.fits	SCIENCE
"	"	5 of 6	60.0000	Ks	pisces.20120302.10028.fits	SCIENCE
"	"	6 of 6	60.0000	Ks	pisces.20120302.10029.fits	SCIENCE
-3.00000	-3.00000
"	"	1 of 6	60.0000	Ks	pisces.20120302.10030.fits	SCIENCE
"	"	2 of 6	60.0000	Ks	pisces.20120302.10031.fits	SCIENCE
"	"	3 of 6	60.0000	Ks	pisces.20120302.10032.fits	SCIENCE
"	"	4 of 6	60.0000	Ks	pisces.20120302.10033.fits	SCIENCE
"	"	5 of 6	60.0000	Ks	pisces.20120302.10034.fits	SCIENCE
"	"	6 of 6	60.0000	Ks	pisces.20120302.10035.fits	SCIENCE
3.00000	-3.00000
"	"	1 of 6	60.0000	Ks	pisces.20120302.10036.fits	SCIENCE
"	"	2 of 6	60.0000	Ks	pisces.20120302.10037.fits	SCIENCE
"	"	3 of 6	60.0000	Ks	pisces.20120302.10038.fits	SCIENCE
"	"	4 of 6	60.0000	Ks	pisces.20120302.10039.fits	SCIENCE
"	"	5 of 6	60.0000	Ks	pisces.20120302.10040.fits	SCIENCE
"	"	6 of 6	60.0000	Ks	pisces.20120302.10041.fits	SCIENCE
3.00000	3.00000
"	"	1 of 6	60.0000	Ks	pisces.20120302.10042.fits	SCIENCE
"	"	2 of 6	60.0000	Ks	pisces.20120302.10043.fits	SCIENCE
"	"	3 of 6	60.0000	Ks	pisces.20120302.10044.fits	SCIENCE

Close

05:02 Close dome and point to horizon so we can valve off the cooling water to that box.

Trying to Winterize BCU47

We tried to find to valve to close off the glycol to the BCU47, but JC was unsuccessful in figuring out where the valve was as seen from the front side. Or maybe it is better to say that we know where the valve is, but we can't figure out how to see or reach it from the front (mirror) side. The instructions are further complicated by the PISCES restriction on rotator angle at horizon.

06:00 Increase the Right IC offset to +1.5 degC.

Open

ULAS1120+0641 for McGreer & Fan

06:15 Open with Wind 15 m/sec T=-5.1 degC D=-15.6 degC. Sky still clear.

06:16 ADAPTIVE ACE ACTIVE preset: observe, 'scripts/ULAS1120_presetK.obs' with PA=-60 gs=3 rs=1
Seeing is varying 1.7 to 2.4 arcsec on the guider.

06:22 RunAO failed for "no stars found on TV frame". Star is visible on CCD47, but seeing is varying alot.

06:23 Resend the preset -- this time it finds the star and centers it correctly.

06:25 RunAO fails with "IDL Safe/Pending skip frame occurred" (a new error message for this run)

06:27 JC is trying to reproduce this closing the loop manually. Then he killed the AO arbitrator.

06:29 Resend the preset. RunAO failed to find the star because seeing 2.5

06:31 Sitting here running Active Optics waiting for seeing to get better. But that seems fairly pointless, so we switch to one of the low priority ADI targets that is a bright reference.

HD92788 for Skemer & Close

06:40 ADAPTIVE ACE TRACK preset in ADI mode with PA=-44 gs=3 rs=0

Estimated magnitude is 9, but should be 5.

06:44 Loop closed and then opened in ~1 sec. DIMM seeing says 3.5. Turbulence is moving across the pupil at ~15-20 m/sec, so it is outside the dome and not mirror/dome seeing.

06:46 Resend the preset .... but PISCES is still taking 200 images from the previous failed script.

(suggestion: Do not use a large number of frames, because you can't stop if the AO loop opens.)

Not this: 
camera, EXP=4, FRAMES=40, COADDS=1, FILTER=5
getimage, IMAGE=SCIENCE

Rather this:  
camera, EXP=4, FRAMES=10, COADDS=1, FILTER=5
getimage, IMAGE=SCIENCE
getimage, IMAGE=SCIENCE
getimage, IMAGE=SCIENCE
getimage, IMAGE=SCIENCE

iif_offset, -3, +7, COORD="DETXY"

06:54 loop opens

06:55 Resend the preset .... failed for no stars found in TV frame.

Change catalog Feb12_HD92788.catalog to make the magnitude 9 rather than 5.

07:00 Resend the preset ......

Failed because we needed to restart IDL.......

07:03 Resend the preset.....

07:08 Loop opens after only a few frames......

07:09 Resend preset .....

07:13 Taking data starting with image 270. Rotator angle is 138.425

Loop opens after image 277. The diffraction image doesn't look to bad, so the turbulence must be mainly low altitude.

Restart IDL with F7 in emacs

07:15 resend the preset now with mag=9 still PA=316

07:19 loop closed

Data starting with image 291 Rotator angle is 138.4258 Some images have diffraction rings, some do not.

12:21 Loop opens at image 308

Need to move farther east as AZ axis is showing wind effects.

Object: Feb12_HD92788
"	"	1 of 20	0.800000	Ks	pisces.20120302.10270.fits	SCIENCE
"	"	2 of 20	0.800000	Ks	pisces.20120302.10273.fits	SCIENCE
"	"	3 of 20	0.800000	Ks	pisces.20120302.10274.fits	SCIENCE
"	"	4 of 20	0.800000	Ks	pisces.20120302.10275.fits	SCIENCE
"	"	5 of 20	0.800000	Ks	pisces.20120302.10276.fits	SCIENCE
"	"	6 of 20	0.800000	Ks	pisces.20120302.10277.fits	SCIENCE
"	"	7 of 20	0.800000	Ks	pisces.20120302.10278.fits	SCIENCE
Object: Feb12_HD92788
"	"	1 of 20	0.800000	Ks	pisces.20120302.10290.fits	SCIENCE
"	"	2 of 20	0.800000	Ks	pisces.20120302.10291.fits	SCIENCE
"	"	3 of 20	0.800000	Ks	pisces.20120302.10292.fits	SCIENCE
"	"	4 of 20	0.800000	Ks	pisces.20120302.10293.fits	SCIENCE
"	"	5 of 20	0.800000	Ks	pisces.20120302.10294.fits	SCIENCE
"	"	6 of 20	0.800000	Ks	pisces.20120302.10295.fits	SCIENCE
"	"	7 of 20	0.800000	Ks	pisces.20120302.10296.fits	SCIENCE
"	"	8 of 20	0.800000	Ks	pisces.20120302.10297.fits	SCIENCE
"	"	9 of 20	0.800000	Ks	pisces.20120302.10298.fits	SCIENCE
"	"	10 of 20	0.800000	Ks	pisces.20120302.10299.fits	SCIENCE
"	"	11 of 20	0.800000	Ks	pisces.20120302.10300.fits	SCIENCE
"	"	12 of 20	0.800000	Ks	pisces.20120302.10301.fits	SCIENCE
"	"	13 of 20	0.800000	Ks	pisces.20120302.10302.fits	SCIENCE
"	"	14 of 20	0.800000	Ks	pisces.20120302.10303.fits	SCIENCE
"	"	15 of 20	0.800000	Ks	pisces.20120302.10304.fits	SCIENCE
"	"	16 of 20	0.800000	Ks	pisces.20120302.10305.fits	SCIENCE
"	"	17 of 20	0.800000	Ks	pisces.20120302.10306.fits	SCIENCE
"	"	18 of 20	0.800000	Ks	pisces.20120302.10307.fits	SCIENCE

ULAS1120+0641

07:22 ADAPTIVE ACE ACTIVE preset for ULAS1120 - RunAO failed with no stars found.

AO449 = LHS 2724

AO449 = LHS2724 happens to be in the east where we can point. This is testing the 300-mode reconstructor in really bad seeing.

07:38 ADAPTIVE ACE TRACK preset to AO449 in ADI mode with PA=-100 gs=4

failed for no stars found in TV frame.

PA was wrong for PARALLACTIC mode!

Steve sent ACQUIRE preset to center up the pointing.

07:48 ADAPTIVE ACE TRACK preset to AO449 in ADI mode with PA=-70 gs=4

(suggestion: Pop-up box for PA has to be different for PARALLACTIC mode presets. Need Parallactic angle+90-EL or something like that rather than position angle.)

Failed for star not found

07:53 ADAPTIVE ACE ACTIVE preset to AO449 PA=-120 gs=4 - seeing is 1.8 arcsec on guider

This star has bad coordinates relative to gs=4. Refstar is just outside the upper left corner of CCD47.

07:57 Resend preset with gs=1 PA=-120

Coordinates of the star are also bad relative to gs=1

AO459 HD 119214

08:00 ADAPTIVE ACE ACTIVE preset to AO459 gs=3 PA=-140

08:04 Loop closed, but then skip frames when seeing explodes from 2 to 3 arcsec on guider. This was the "new" 300-mode reconstructor.

08:09 AO459_adi.obs - shell rRIP'd

08:12 AO459_adi.obs gs=3 PA=-65

Shell RIP'd

08:20 JC experimenting with closing a 300-mode low gain loop by hand

Excerpts from a Skype discussion with Alfio and JC

[1:44:30 AM] John M Hill: If you guys want to send us a low-stress reconstructor for DX, we have the perfect bad seeing conditions to test it out.
[1:45:19 AM] John M Hill: The 300-mode reconstructor you sent us yesterday does better than the standard reconstructor for bright stars, but the loop still opens when the seeing is worse than 3 arcsec.
[1:46:53 AM] Alfio Puglisi: hello john, I can try to produce a 100 modes or so reconstructor, but it will take me a while
[1:48:03 AM] John M Hill: Yesterday -  Simone was proposing a standard 400 modes reconstructor with the 25 high-stress modes zeroed out as was done for SX.
[1:49:19 AM] Alfio Puglisi: the 25 modes are mostly over mode 400th, i think only two or three or them are in the 300-400 range
[1:50:43 AM] Alfio Puglisi: Marco has identified the modes for the DX side. These modes are not removed from the reconstructor, rather, they are assigned a gain of zero during the autogain procedure
[1:50:59 AM] Alfio Puglisi: so the 400-modes rec you have is OK, but I have to update the DX autogain script

[1:54:36 AM] Alfio Puglisi: OK, looking at the DX high-force modes, the modes that would be removed are:
[1:54:41 AM] Alfio Puglisi: 398         434         467         468         502         503
         536         537         540         568         569         575
[1:54:51 AM] Alfio Puglisi: that's only mode #398 in your case
[1:56:03 AM] Alfio Puglisi: so if the 300-modes rec. did not work in bad seeing, I do not think that the low-stress rec will work either
[1:56:28 AM] John M Hill: Yes, that seems like the reasonable conclusion.

[1:57:15 AM] Alfio Puglisi: I can try to make a 100 or 200 modes rec. in the next half hour or so
[1:57:30 AM] John M Hill: OK, we can try that.

09:03 ADAPTIVE ACE ACTIVE preset on AO459 (not what I was intending to do, wrong script name?) This was AO459_rot.obs

Images starting with 330 - well saturated in 4s

09:10 loop opened at image 350. The next few images after that are seeing-limited up to 369.

What Alfio did

[2:07:40 AM] Alfio Puglisi: I have some problems in making RECs from here due to the network connection... if you want we can try a quick hack: i am adding a line to the autogain script, which just sets the gain for all modes higher than 100 to zero
[2:08:03 AM] Alfio Puglisi: the procedure to update this value is the following:
[2:08:27 AM] Alfio Puglisi: 1) modify $ADOPT_SOURCE/PyModules/AdOpt/autogain.py, line 311 (on wfsdx)
[2:08:39 AM] Alfio Puglisi: 2) type "make install" from $ADOPT_SOURCE/PyModules/
[2:08:42 AM] Alfio Puglisi: 3) restart the WFS arbitrator
[2:08:53 AM] Alfio Puglisi: then run the autogain as usual
[2:09:07 AM] Alfio Puglisi: right now I installed the version with 100 modes
[2:10:05 AM | Edited 2:10:09 AM] Alfio Puglisi: of course I would need something more flexible with a conf. parameter, etc. I can work on that this afternoon, and I will put back the original version before we all forget about it :)

[2:13:00 AM | Edited 2:13:18 AM] Alfio Puglisi: yes, but in order to activate them, a restart of the WFS arb. is needed

09:13 Restart the WFS arbitrator to get Alfio's new version of autogain.py with zero gain above 100-modes.

09:15 AO459_rot.obs again

09:18 Loop closed - starting with image 370. Not an obvious diffraction pattern, but much better concentration than seeing limited

Image 415 rode through a patch of 4 arcsec seeing and the loop remained closed.

09:46 Loop finally opened with the "supernova" of seeing (4.5 arcsec on guider). Somewhere near image 538

09:50 AO459_adi.obs with PA=-65 (puts rotator at 117) gs=3 exp=2 sec

But that script had "pause" instead of "runao" so we have to wait for PISCES to take 20 frames.

09:58 obsaerve, Start=3 actually starts at position 4 as listed by /check/ - Doug says this is a known bug that needs to be fixed.

10:00 loop closed starting with image 570 which is a bad read. The star is not well centered in the frame.

10:03 losing 50% of the flux on the pyramid.

10:11 loop opened.

10:16 Wind is touching 21 m/sec, and seeing is bouncing above 4 arcsec, so we've reverted to the vegetative mode of observing -- sitting here doing mostly nothing.

More discussion with Alfio that seems to explain what we have been experiencing.

[2:27:13 AM] John M Hill: Will this same trick work for a fainter refstar like 12th magnitude?
[2:27:51 AM] John M Hill: At image 415 the loop survived some seconds of 4 arcsec seeing, so we are impressed!
[2:28:47 AM] Alfio Puglisi: it should work down to mag. 13 or so. Fainter than that, we switch to bin3 which already has only 66 modes, and this trick has no effect
[2:29:20 AM] Alfio Puglisi: 12th mag is bin2, which has 150 modes corrected, so this will remove only the last 50 modes. The relative decline in performance should be lower :)
[2:30:15 AM] John M Hill: I'm thinking that we'll go and try the comet C/2009 P1 Garradd
[2:30:52 AM] Alfio Puglisi: The problem with fainter stars is that, with bad seeing gusts, the sensor might become light-starved and the loop will open even if you are only correcting a few modes. It won't happen in the bright end like now, since there is plenty of light.
[2:31:45 AM] John M Hill: Yes, I think you are right.  I think we saw that earlier.
[2:33:43 AM] Alfio Puglisi: our hypothesis is that the psf become bigger than the 1.5" pupil stop in front of the pyramid, so some light is lost. At least, that's what Simone told me :)
[2:34:52 AM] John M Hill: Yes, we see that also.  When the seeing is worse than 2 arcsec, the magnitude reported by AO is 0.5-1 magnitude fainter than the expected magnitude of the refstar.  Thanks for explaining why that happens.

(JC suggestion: have a task that watches pyramid counts and pauses the AO loop when the seeing goes up and the counts go down.)

10:35 AO459_rot.obs so we can watch the seeing on the guider. But wrong PA.

10:40 AO459_rot_v2.obs PA=+80 with gs5

images starting at 650 in POSITION mode (rotator tracking)

10:52 DIMM is reporting 2.5 - 3 arcsec seeing. AO corrected image on the guider is as good as 1.7 arcsec.

AO500 BD+3 3043 (10th mag)

11:04 AO500_rot.obs PA=-60 gs=4 (gs 4 is a binary - will have bad coords)

Failed with "No star found in TV frame" - too faint to center at mag=10.00 with filters. CCD39 reporting 12th mag.

11:12 JC closing loop manually after removing ND filters.

(suggestion: would be good to have a way to modify the magnitude of the refstar without having to edit the catalog file. Is there a flag for that?)

11:17 resend the preset

11:22 JC: Autogain doesn't seem to be working for this fainter star with bin=2

11:27 It worked briefly on a second try.

11:30 reclose the loop manually

11:32 taking images starting with 770

bin=2 153 modes (100)

This image really looks good. That's because the seeing has improved to 1.3 arcsec.

11:35 Shell RIP'd so the good seeing didn't last long. image 790.

11:39 resend preset

JC Question: Why does 100 modes work for bin=1? but not for bin=2 when we might expect the S/N per subaperture/pixel to be the same?

11:44 Steve clears active optics and reflattens the shell since we had a big defocus.

11:45 resend the preset - and things as still ~1 mm out of focus

11:47 ACQUIRE preset to AO500 for focussing manually.

11:50 resend the preset AO500_rot.obs

12:00 restart IDL because out of free LUNs.

12:03 resend preset - no stars in TV frame.

12:xx manual close loop

12:24 start image at 810 (which was a bad readout)

Loop opened at the end of image 811 next 18 are seeing limited

To start IDL in emacs: F7
To break IDL program in emacs: ^C^C

Close

12:33 Close T=-5.7 D=-15 Wind 20m/sec from WNW. BCU47=7.3 degC

12:40 ten x 60s dark frames starting with image 820

"

"

1 of 10

60.0000

blank/dark

pisces.20120302.10048.fits

DARK

-- JohnHill - 29 Feb 2012