0:00 Adsec brought up in soul latest. 636 modes for the flat
  • adsec power off
  • adsec stop
  • use_soul latest
  • exit terminal and restart, edited the adsecarb.conf ->gainRampDelay from 1000 to 2000
  • adsc_start and power on as per usual in new end
0:23 wfs brough up in soul latest

1:25 Just during Steve's pointing (Guide Preset) we see several filotto errors. Seeing 0.6"

1:39 FIrst OSCO target HD38515 Catalog mag R 7.49, preset sent. First preset failed because guide star was too close to the edge. Second one successful. Seeing 0.6-0.7"
  • Steve execute. Rotator 337. NCPA off luci.20220220.0008-
  • WFS magnitude 8.8, 90.2 phtons/subapp, 0.92 slope rms
  • resume gain happened before resume gainAO

    AdSecArbitrator _81362|INF| 219|2022-02-20 01:56:59.686635| MAIN > Received cmd: Offloading(0)
    AdSecArbitrator_81362|INF| 220|2022-02-20 01:56:59.686748| COMMANDHANDLER > FSM (status: AORunning) has received command 2029 (TTOffload)
    AdSecArbitrator_81362|INF| 221|2022-02-20 01:57:00.299035| COMMANDHANDLER > Command TTOffload (code 2029) successfully completed
    AdSecArbitrator_81362|INF| 222|2022-02-20 01:57:00.299342| MAIN > Received cmd: PauseAo ()
    AdSecArbitrator_81362|INF| 223|2022-02-20 01:57:00.299406| COMMANDHANDLER > FSM (status: AORunning) has received command 2027 (Pause Ao)
    AdSecArbitrator_81362|WAR| 224|2022-02-20 01:57:00.299763| IDL > Tip-tilt gain is set to zero. Offload disabled until next setGain command
    AdSecArbitrator_81362|INF| 225|2022-02-20 01:57:00.712673| COMMANDHANDLER > Command Pause Ao (code 2027) successfully completed
    AdSecArbitrator_81362|INF| 226|2022-02-20 01:57:14.496032| MAIN > Received cmd: ResumeAo ()
    AdSecArbitrator_81362|INF| 227|2022-02-20 01:57:14.496174| COMMANDHANDLER > FSM (status: AOPause) has received command 2028 (ResumeAo Ao)
    AdSecArbitrator_81362|INF| 228|2022-02-20 01:57:16.510349| COMMANDHANDLER > Command ResumeAo Ao (code 2028) successfully completed
    AdSecArbitrator_81362|INF| 229|2022-02-20 01:57:16.630737| MAIN > Received cmd: Offloading(1)
    AdSecArbitrator_81362|INF| 230|2022-02-20 01:57:16.630815| COMMANDHANDLER > FSM (status: AORunning) has received command 2029 (TTOffload)
    AdSecArbitrator_81362|INF| 231|2022-02-20 01:57:16.635585| COMMANDHANDLER > Command TTOffload (code 2029) successfully completed
    20222002 01:57:35.3766 Alert: HighOrderOffload: HighOrderOffload vector ready

    1:57Sequence ok in adsec arb but in IDL set gain before resumeAO .
idl_ctrl_81208     |INF|      5500|2022-02-20 01:57:14.496624|             MAIN > Executing command: errcode= fsm_set_gain('/local/aomeas/adsec_calib/M2C/KL_v20/gain/g_tt_1.30_ho1_0.28_ho2_0.29.fits', BANK=1, CLIP_N_MODES=10, SCALE=0.1) & print, errcode
..
idl_ctrl_81208     |INF|      5505|2022-02-20 01:57:14.502265|             MAIN > Executing command: errcode= fsm_resume_ao() & print, errcode

Tracking numbers for OSCO

NO NCPA, 1st POSITION:
   => Command line arguements processed
    Set  0   Tracknum  20220220_014905   LuciName  unknown
    Set  1   Tracknum  20220220_014917   LuciName  unknown
    Set  2   Tracknum  20220220_014929   LuciName  unknown
   -> wrote file /home/AOeng/doug/get_telemetry/OSCO/20220220_014905.log

NO NCPA, 2nd POSITION:
   => Command line arguements processed
    Set  0   Tracknum  20220220_015307   LuciName  unknown
    Set  1   Tracknum  20220220_015318   LuciName  unknown
    Set  2   Tracknum  20220220_015330   LuciName  unknown
   -> wrote file /home/AOeng/doug/get_telemetry/OSCO/20220220_015306.log

01:57UT manual offset abs detxy -5 +5, NCPA on
luci.20220220. --0046

NCPA ON, 1st POSITION:
   => Command line arguements processed
    Set  0   Tracknum  20220220_015818   LuciName  unknown
    Set  1   Tracknum  20220220_015829   LuciName  unknown
    Set  2   Tracknum  20220220_015841   LuciName  unknown
   -> wrote file /home/AOeng/doug/get_telemetry/OSCO/20220220_015817.log

NCPA ON, 2nd POSITION:
   => Command line arguements processed
    Set  0   Tracknum  20220220_020218   LuciName  unknown
    Set  1   Tracknum  20220220_020229   LuciName  unknown
    Set  2   Tracknum  20220220_020244   LuciName  unknown
   -> wrote file /home/AOeng/doug/get_telemetry/OSCO/20220220_020218.log

2:08 Slewing to Xianyu's target, R=14.64. Rotator 214 to start. Seeing 0.5 0.6" on the guider to start.

Probe position may have drifted error. Probe position happy with a second preset
  • Estimaged OCAM magnitude: 14.3. Bin2, frequency 767, modulation 3, OCAM gain 600, 3.4 photons/subapp, 18
  • Optimize gain ramps up to tip tilt to 35, and ramping HO2 to 15. excessive. 12.54 2.33 0.84 applying an optical gain Optical gain ~0.1, 1.28.0.24 0.09. NCPA on.
2:30 The loop keeps opening. Soon as HO2 is applied some weird shape is applies. It looks like when the loop opened the Gopt did not reset to zero. Clicking optimize gain did reset it so thats fine.
  • Reoptimize, Gopt still at ~0.1, gains 0.68, 0.21, 0.11. Image a bit elongate with these updated gains. TT jitter?
  • TN024254
2:56 The power spectrum suggested we were underdriving tt. drove up 15 (even with optical gain). The correction is still pretty ratty on the luci images in H and J. The seeing still look

3:01 reoptimizing. Optical gain on and NCPA on - gains set to 0.82, 0.14,0.09 with optical gain ~0.1.
  • Optimize gain raps way to high. Tt rang 0-50, ho1 0-10, ho2 0-15 on round1 with values at minimums on faily challow curves found around tt 21, ho1 ~1.8, ho2 ~0.2. On round 2 TT 11, Ho2 2, Ho2 1. Ranges need to be refined to ovaoid safe skipping
  • Even after this optimize images are elongated on luci. Rotator position 244, on axis guide star.
3:15 loop opened. Unstable correction. Consitions are very stable so unsure what happened. tt 1.06 0.15 0.06 with optical gain of 0.1

3:23 Middle of an exposure and we opened
  • Infuriating. The only way to turn off Optical gain is to reoptimize. can;t just send held gains and reapply optical gain.
  • reoptimized, optical gain on 0.98,
3:30. Script send and offset but we remained paused. Resumed through AOS. IIF claimed offset failed upon resuming in AOS but all looked ogood through WFS, AOS and adsec so reumed script. No issues on the next offset.

3:35. The gain vector is setting in the adsec before the WFS resumes

3:39 Safeskipping and opened again. When safeskipping, the gain verctor looked empty but it might have been my timing. Need to look later. We were not offsetting at the time

3:48 Offsets taking an exhorbanant amount of time at times. It looks like this is a result of the gain ramp (maybe because of our 2000ms adjustment at sunset).
  • GOpt is only 0.1 throughout the observation with bin2 target in 0.6" stable seeing... that seems low.
This previous science field had a companiosn 0.6" away that was 1.5 mag different. Perhaps that caused some instabilities. Xianyu's theory, and it seems tohold wieight, especially given the stability of the AOref.

3:58 Slewing to PSF calibrator. R 14.38 catalog magnitude. Not running check flux. Rotator 250
  • 14.5 OCAM OCAM mag, 2.8 photons/subapp
  • Optimize gain range tt up to 35 first pass and 22, ho1 up to 6.5 first pass and 4.5, ho2 10 both passes. Gains 14, 1.3, 1.1
  • Gains 1.46 0.13 0.11 with 0.1 optical gain and NCPA on.
  • Seeing 0.7" on the DIMM
  • TN#041815
4:41 Reset the ->gainRampDelay from 2000 to 0, restart adsec arb, flat set with 555

4:41 Preset for Full AO N375 Full field test at M67. Catalog magnitude of AO ref ia 10.73

The lower left and upper right quadrants on OCAM are noisy in bin1. Rebin to bin2 and back to bin1 and noise is gone.

bin1_OCAM noise.png:
bin1_OCAM noise.png

Noise in those quadrants still but does not seem to be impacting center pupils or closing the loop. Handover is in 20 min so working with the noise.
  • Bin1 optimize gain ramps tt 0-7.5 first time and second, ho1 0-3 both first and second, 2.1 for ho2 first and second.
  • GOpt ~0.35, gains 1.09, 0.22, 0.14 with GOpt, NCPA on. OCAM mag 12.3, 3.9 photons/subapp,
  • TN 050356
  • The gain verctor loaded essecially simultaneously with the gain vector loading
  • idl_ctrl_81208 |INF| 75793|2022-02-20 05:08:41.668136| MAIN > Executing command: errcode= fsm_set_gain('/local/aomeas/adsec_calib/M2C/KL_v20/gain/g_tt_1.09_ho1_0.22_ho2_0.14.fits', BANK=1) & print, errcode
    ...
    idl_ctrl_81208 |INF| 75797|2022-02-20 05:08:41.673941| MAIN > Executing command: errcode= fsm_resume_ao() & print, errcode
    
    

5:20 Returning from small offset failed. Offset while safeskipping. I tried to resume, but loop opened. Its handover so we are stopping

5:22 Safed WFS and am warming. BCU2K cycled when warm to resolve the OCAM noise issue. Looks ok

9:40 Cooling OCAM for second half operations.

10:12 Reconfiguring for mono LUCI

10:18 POinting and collimation near first target. Seeing 0.44" on the guider.

10:22 First target HS 1338-0807, bin 4 target, catalog magnitude 16.7. 15 degrees from the moon.
  • GCS failed to start WFSing. Probe drift. Cancel and resend preset works
  • OCAM looks noisy again after switching from bin1 to bin4. Noise similar to what was seen earliler in the night with the lower left and upper right sectors showing high noise after changing binnings.
  • bin4Noise.png
  • Center Pupils claimed to have succeeded but star wandered off. Concerned the noise is the issue.. Taking new dark did not resolve so noise and it is close to the lower left pupil. Switching binnings to see if that helps, it did not. If anything its worse.
10:34 Noise on OCAM is persistent so am warming and Cycling BCU2K. Noise is not present after cycling the OCAM. Verified in Bin1 configuration and bin4. Cooling again.

10:51. Resending preset for moon test. HS 1338-0807, bin 4 target, catalog magnitude 16.7. 15 degrees from the moon. No noise any more in OCAM.
  • Star wandering off in center pupils again. Sky background is very high here.
  • I executed center star again but the gude star was far from the hot spot. This would be a problem if off axis guiding had to take over. resend preset
  • Center pupils failed again. The realtime pupils are far left of the real pupils, by almost one full pupil. Managed to grab with some finessing.
  • OCAM magnitude 16.8, bin4 freq 388 modulation 3, OCAM gain 600, lslope rms is 0.87, 2.9 photons/subapp, 16 average pupil intensity
  • rotator is 124, gains 3.55, 0.16 0.10
  • TN#110550, 110605
  • Sky offsets set with guide star in partol field so fell to active mode. Offsetting back to source, as exepcted the AOref was far off the tip of the pyramid and we safeskipped and opened. We did not have the the grainRampDelay in place because of increased overheads but this would have been a good place to test that.
  • GCS guiding kicked in after an offset and again the AOref was not ont eh tip of the pyramid. This time though guiding should not have started since the AOref was available....
11:27 Slewing to M92, AOref catalog magnitude 9.43. Pointing check needed. No star in CCD47.
  • the star is much fainter than expected. Barry claims to have done this in ESM before so very odd. But even changing filters this target is pretty faint.
  • UCAC mag ~ 12.5 and not in UCAC5 so moving on.
11:35going to nearby star with confirmed UCAC magnitude 11.7
  • OCAM magnitudem12.9, 10.5 photons/subapp, 50 average pupil intensity, bin2 900hz modulation 3, OCAM 600.
  • Gains 0.5, 0.16, 0.21 with Optical gain just under 1 and NCPA on
  • Seeing up to 1.3
  • There was noise in the lower left and upper right sectors when executing the bin 1 center pupils, but this disappeared when switching to bin2
CopleyCtrl_125898 |ERR| 768|2022-02-20 11:50:53.284917| MAIN > Error: movement out of range
CopleyCtrl_125898 |ERR| 769|2022-02-20 11:50:53.284996| MAIN > Request: -108.038329 Limits: 0.000000 -108.000000
CopleyCtrl_125898 |ERR| 770|2022-02-20 11:50:53.294144| MAIN > Variable handler error: [AOException] Error in MoveTo(): movement out of range (code -5002) Value is out of range
CopleyCtrl_125898 |ERR| 771|2022-02-20 11:51:36.376514| MAIN > Error: movement out of range
CopleyCtrl_125898 |ERR| 772|2022-02-20 11:51:36.376601| MAIN > Request: -108.038329 Limits: 0.000000 -108.000000
CopleyCtrl_125898 |ERR| 773|2022-02-20 11:51:36.376697| MAIN > Variable handler error: [AOException] Error in MoveTo(): movement out of range (code -5002) Value is out of range
CopleyCtrl_125898 |ERR| 774|2022-02-20 11:52:19.474298| MAIN > Error: movement out of range
CopleyCtrl_125898 |ERR| 775|2022-02-20 11:52:19.474366| MAIN > Request: -108.038329 Limits: 0.000000 -108.000000
CopleyCtrl_125898 |ERR| 776|2022-02-20 11:52:19.474455| MAIN > Variable handler error: [AOException] Error in MoveTo(): movement out of range (code -5002) Value is out of range
  • Barry claims the star is in the AOref in the OT, close to teh edge but should have worked. Issues with the OT AOref field? Need to characterize to make sure accurate for next OT release.
  • bin2 tracking data: TN120806
  • Opened after a large offset from sky, safe skips. Last exposure. Most offsets were fine
12:17 Slewing to Cat's Eye Nebula for pretty picture time. Catalog magnitude: 11.38. 0.72" seeing.
  • OCAM magnitude 13.0, bin2, 900Hz, 9.6 photons/subapp, 50 average pupil intensity
  • 1.3-1.8" on the DIMM
  • rotator 129 to start
  • TN122643
12:38 Failure of adsec. The failure occurred when moving to large offset, where we should have been pauseAO for a sky position. We were on sky offset with loop paused (WFS showed paused).

AdSecArbitrator_117347|INF|      1444|2022-02-20 12:36:45.371295|            MAIN > Received cmd: PauseAo()
AdSecArbitrator_117347|INF|      1445|2022-02-20 12:36:45.371426|  COMMANDHANDLER > FSM (status: AORunning) has received command 2027 (Pause Ao)
AdSecArbitrator_117347|WAR|      1446|2022-02-20 12:36:45.372007|             IDL > Tip-tilt gain is set to zero. Offload disabled until next setGain command
AdSecArbitrator_117347|INF|      1447|2022-02-20 12:36:45.767070|  COMMANDHANDLER > Command Pause Ao (code 2027) successfully completed
AdSecArbitrator_117347|ERR|      1448|2022-02-20 12:36:51.978803|    ALERTHANDLER > Received an ERROR alert! Forcing Failure state.
AdSecArbitrator_117347|ERR|      1449|2022-02-20 12:36:51.978858|    ALERTHANDLER > Alert message: COILS DISABLED from fastdiagn.L
AdSecArbitrator_117347|INF|      1450|2022-02-20 12:36:52.030410|    ALERTHANDLER > Received from masterdiagnostic.L a DumpSaved alert with filename /local/aolog/2022/02/20/ADAPTIVE-SECONDARY_00001645360611_20220220_123651.bin
  • adsc_stop/start. flat set with 499 modes
  • Looks like PauseAO command was never successfully completed?
12:46 preset resent
  • OCAM 13.0, bin2, freq 900, rotator 136
  • Gains 1.16, 0.19, 0.11
  • TN130024
13:05 OCAM not live and adsec in failure. Moving to a large sky offset again. OCAM not live beacuse WFS had loop closed, but adsec in failure. mismatch state.
  • recovered adsec, restarted wfs arbitrator this time for good measure.
  • AdSecArbitrator_29594|INF|       220|2022-02-20 13:04:08.768788|   COMMANDHANDLER > FSM (status: AOPause) has received command 2028 (ResumeAo Ao)
    AdSecArbitrator_29594|INF|       221|2022-02-20 13:04:08.776562|   COMMANDHANDLER > Command ResumeAo Ao (code 2028) successfully completed
    AdSecArbitrator_29594|INF|       222|2022-02-20 13:04:08.885933|             MAIN > Received cmd: Offloading(1)
    AdSecArbitrator_29594|INF|       223|2022-02-20 13:04:08.886044|   COMMANDHANDLER > FSM (status: AORunning) has received command 2029 (TTOffload)
    AdSecArbitrator_29594|INF|       224|2022-02-20 13:04:08.998393|   COMMANDHANDLER > Command TTOffload (code 2029) successfully completed
    AdSecArbitrator_29594|ERR|       225|2022-02-20 13:05:30.239250|     ALERTHANDLER > Received an ERROR alert! Forcing Failure state.
    AdSecArbitrator_29594|ERR|       226|2022-02-20 13:05:30.239311|     ALERTHANDLER > Alert message: COILS DISABLED from fastdiagn.L
    AdSecArbitrator_29594|INF|       227|2022-02-20 13:05:30.290592|     ALERTHANDLER > Received from masterdiagnostic.L a DumpSaved alert with filename /local/aolog/2022/02/20/ADAPTIVE-SECONDARY_00001645362330_20220220_130530.bin
    AdSecArbitrator_29594|INF|       228|2022-02-20 13:05:30.290623|     ALERTHANDLER > Processdump is skipped
    AdSecArbitrator_29594|ERR|       229|2022-02-20 13:05:34.284523|     ALERTHANDLER > Received an ERROR alert! Forcing Failure state.
13:13 Preset resent
  • TN132109 (bin2 in twilight)
  • gains 1.39, 0.17, 0.16 with Optical gain and NCPA, seeing 1"
13:33 We took it until the sky caused some safe skips. It was almost all sky. Closing up. OCAM in safe config.

13:45 Reverting to soul stable. Power off, adsc_stop start, use_soul stable. And starting back up. Flat with 636. Rested when complete. Also reverted wfs reverted to soul _stable

-- JenniferPower - 20 Feb 2022
Topic revision: r5 - 20 Feb 2022, JenniferPower
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback