20210130 OSCO and GRB

01:30 Tried OSCO but had OCAM troubles
  • When close loop for Center Pupil the OCAM went not live
  • Stop start Masterdiagnostics but still not live
  • While loop closed and not live the errors were
[AOeng@sxadsec 11]$ grep "Request: AcquireRefAO()\|Request: CenterStar()\|Request: CenterPupil()\|Request: CloseLoop()\|Request: OptimizeGain()\|Request: ApplyOpticalGain()\|ReadyForStartAO" */AOARB.L.202011*.log >/tmp/acq_11.dat
[AOeng@sxadsec 11]$ rep "Request: AcquireRefAO()\|Request: CenterStar()\|Request: CenterPupil()\|Request: CloseLoop()\|Request: OptimizeGain()\|Request: ApplyOpticalGain()\|ReadyForStartAO" */AOARB.L.2020*.log >/tmp/acq_10.dat
-bash: rep: command not found
[AOeng@sxadsec 11]$ cd ../10
[AOeng@sxadsec 10]$ rep "Request: AcquireRefAO()\|Request: CenterStar()\|Request: CenterPupil()\|Request: CloseLoop()\|Request: OptimizeGain()\|Request: ApplyOpticalGain()\|ReadyForStartAO" */AOARB.L.2020*.log >/tmp/acq_10.dat
-bash: rep: command not found
[AOeng@sxadsec 10]$ grep "Request: AcquireRefAO()\|Request: CenterStar()\|Request: CenterPupil()\|Request: CloseLoop()\|Request: OptimizeGain()\|Request: ApplyOpticalGain()\|ReadyForStartAO" */AOARB.L.2020*.log >/tmp/acq_10.dat
  • Power cycle BCU2k, stop/start Master Diag. OCAM2k Contr, slope com
  • OK now

02:00 Preset to GRO, Rmag=16.5
  • bin 4 OCAM image has high noise in outter quadrants
    • TN 021069
    • Bin4_noise_1.png:
      Bin4_noise_1.png
  • after closed loop then open loop, reset OCAM to bin 2, reset OCAM to bin 4, no noise now
  • Issue #8288, note added: AO: OCAM Bin 3 images have bright quadrants
  • reload recon and disturb
  • Close loop
  • Optimize gain
  • gain 1.3, 0.5
  • GOPT = 0.11
  • Barry reports the LUCI images look round and with similar or better peak
  • TN 022207 024007
  • accidently asked for 400012 frames of Opt Loop Data. Process got stuck. Stop start did not fix it.
  • bin4_modal_1.jpg:
    bin4_modal_1.jpg
  • bin4_jitter_1.jpg:
    bin4_jitter_1.jpg

06:45 Bin 3 Closed loop
  • 358 Hz
  • gains 6, 1.6, .6
  • GOPT 0.13

07:20 Barry see Trefoil on image
  • turn on NCPA
  • -19 -6 -6 -5 -11 -8 -5
  • Image looks better. Don't see obvious three lobes. Not sure if strehl would be better. Still halo of light
  • loop open
  • re-close, optimize gain, gopt, ncpa

08:38 Started to get moonlight on the PWFS.
  • PWFS measured mag increased by 1 mag in a few minutes.
  • Kept getting brighter so in 3 minutes the background level was higher than the AO Ref Star
  • Need Baffle
  • Maybe could have changed position angle, but script was specific to move the AO reference star off the LUCI image.
  • Moved the next target
  • We were ~37 degrees from the moon. Moonlight was illuminating the inside of the chamber
  • COPY Issue #8267, note added: SOUL: Spill light on OCAM2K
  • moonlight.jpg:
    moonlight.jpg

08:50 Preset to 10.9 mag start
  • PWFS measure 11.5
  • X camera lens maxed at -100
  • xdiff was 0.2. Waited and xdiff decreased to 0.1 in about 3 minutes as rotator moved
  • Pupil Center completed
  • closed loop, optimized gain, gopt, NCPA
  • Barry report that star in the field, 23" from AO Ref Star was "squished"
  • Notice that different offset positions slightly change the xdiff error. Enrico suggest that the offset of the bayside stages may slightly move the camera lens frown, sad smile
  • Issue #8253, note added: AO: The SX camera lens hits its motion limit for some Instrument Rotator Angles

08:57 The GOPT loop stop updating
  • I stopped and start the Optical Loop Gain process and all was well
  • Except the GOPT value is not display in AuxLoop, but ratio is.
  • Is the TERMINATE from the new "process check daemon"? (ie w_start_all)
python_97594       |INF|        29|2021-01-31 08:52:29.666935|             MAIN > Tracknum: 20210131_250000 demodDelta/demodCmd: 1.41e-08/1.36e-08 gopt: 0.430
python_97594       |INF|        30|2021-01-31 08:52:33.994117|             MAIN > Tracknum: 20210131_250000 demodDelta/demodCmd: 1.36e-08/1.36e-08 gopt: 0.430
python_97594       |INF|        31|2021-01-31 08:53:33.298727|             MAIN > TERMINATE received. Its TimeToDie.
python_122372      |DEB|         1|2021-01-31 08:53:49.988354|             MAIN > Connected as gopt.L
python_122372      |INF|         2|2021-01-31 08:54:49.443996|             MAIN > TERMINATE received. Its TimeToDie.
python_128587      |DEB|         1|2021-01-31 08:55:06.040146|             MAIN > Connected as gopt.L
python_128587      |INF|         2|2021-01-31 08:56:06.227564|             MAIN > TERMINATE received. Its TimeToDie.
Process killed!!
python_128587      |INF|         3|2021-01-31 08:56:08.755130|             MAIN > TERMINATE received. Its TimeToDie.
python_134029      |DEB|         1|2021-01-31 08:56:14.681820|             MAIN > Connected as gopt.L
python_134029      |INF|         2|2021-01-31 08:56:19.895656|             MAIN > Closed loop iterations=1, waiting for the next one
python_134029      |INF|         3|2021-01-31 08:56:24.815271|             MAIN > Tracknum: 20210131_250000 demodDelta/demodCmd: 1.39e-08/1.35e-08 gopt: 1.000
python_134029      |INF|         4|2021-01-31 08:56:29.741052|             MAIN > Tracknum: 20210131_250000 demodDelta/demodCmd: 1.38e-08/1.36e-08 gopt: 1.000
python_134029      |INF|         5|2021-01-31 08:56:34.503702|             MAIN > Tracknum: 20210131_250000 demodDelta/demodCmd: 1.38e-08/1.37e-08 gopt: 1.000
  • After restart of Optical Loop Gain process, the ratio was calculated but two buttons on the GOPT GUI were not pressed. "Apply to GOPT" and "Apply to Slope Computer"
  • When I clicked these on, the process was TERMINATED again and restarted now with these button clicked and GOPT applied properly
  • Ratio and GOPT are now displayed in auxLoops
  • Issue #8306, submitted on your behalf: AO: Optical Loop Gain process TERMINATED and when restarted GOPT not applied

10:08 GOPT stop updating again
  • Process had not been TERMINATE (I didn't see it in the log) but two buttons were not clicked.
  • I clicked them and this time GOPT was applied correctly (not TERMINATED and restarted like last time)
python_61942       |INF|       101|2021-01-31 10:11:13.414632|             MAIN > Tracknum: 20210131_250000 demodDelta/demodCmd: 1.36e-08/1.36e-08 gopt: 0.395
python_61942       |INF|       102|2021-01-31 10:11:17.655772|             MAIN > Tracknum: 20210131_250000 demodDelta/demodCmd: 1.31e-08/1.34e-08 gopt: 0.392
python_61942       |INF|       103|2021-01-31 10:11:21.965018|             MAIN > Tracknum: 20210131_250000 demodDelta/demodCmd: 1.27e-08/1.34e-08 gopt: 0.385
python_61942       |INF|       104|2021-01-31 10:12:21.372396|             MAIN > TERMINATE received. Its TimeToDie.
python_109445      |DEB|         1|2021-01-31 10:12:37.889436|             MAIN > Connected as gopt.L
python_109445      |INF|         2|2021-01-31 10:12:43.022603|             MAIN > Closed loop iterations=1, waiting for the next one
python_109445      |INF|         3|2021-01-31 10:12:47.413686|             MAIN > Tracknum: 20210131_250000 demodDelta/demodCmd: 1.25e-08/1.31e-08 gopt: 1.000
python_109445      |INF|         4|2021-01-31 10:12:52.373149|             MAIN > Tracknum: 20210131_250000 demodDelta/demodCmd: 1.31e-08/1.36e-08 gopt: 1.000
python_109445      |INF|         5|2021-01-31 10:13:13.731782|             MAIN > Closed loop iterations=1, waiting for the next one
python_109445      |INF|         6|2021-01-31 10:13:18.606355|             MAIN > Tracknum: 20210131_250000 demodDelta/demodCmd: 1.19e-08/1.31e-08 gopt: 1.000
python_109445      |INF|  
  • From the Optical Loop Gain log we see that GOPT was not updating for 1 minute so the new daemon process did the correct thing and TERMINATE and restart. However, the two "Apply" buttons are not clicked. This should be corrected.

11:02 OffsetXY failed and the next ResumeAO failed.
  • Looks like "11:02:57 Communication with FLAOWFS lost"
  • Then next offsetXY at 11:02:57 disappears into the ether and returns an error to the AOS
  • The ResumeAO is not requested because OffsetXY failed. Error is passed to LUCI
  • At 11:06:20 I send the same OffsetXY from the AOS command GUI and stages moved. light is on PWFS
  • At 11:07:10 I sent a ResumeAO from the AOS Command GUI and loop is closed
  • The next offset failed because the AO Ref star moved off the hotspot (at 11:08:11)
  • at 11:08:11 I used the AOS Command GUI to Pause the loop, and then offsetXY the opposite X,Y values of the last offset
  • The AO Ref Star was now on the PWFS. ResumeAO from AOS
  • Next OffsetXY from LUCI was successful.
  • Issue #8306, note added: AO: Optical Loop Gain process TERMINATED and when restarted GOPT not applied
AOARB.L.202101310000.log:python_61935       |INF|      2455|2021-01-31 11:02:54.565300|             MAIN > Request: Pause()
AOARB.L.202101310000.log:python_61935       |DEB|      2456|2021-01-31 11:02:54.812337|             MAIN > Status after command: AOArbitrator.LoopSuspended
AOARB.L.202101310000.log:python_61935       |WAR|      2457|2021-01-31 11:02:57.384667|             MAIN > Communication with FLAOWFS lost
AOARB.L.202101310000.log:python_61935       |INF|      2458|2021-01-31 11:02:57.648476|             MAIN > Request: OffsetXY((-1.2004691362380981, -2.3921847343444824))
AOARB.L.202101310000.log:python_61935       |DEB|      2459|2021-01-31 11:02:57.648556|             MAIN > Status after command: AOArbitrator.LoopSuspended
AOARB.L.202101310000.log:python_61935       |ERR|      2460|2021-01-31 11:02:57.648568|             MAIN > No connection to WFS [FLAOWFS]
AOARB.L.202101310000.log:python_61935       |INF|      2461|2021-01-31 11:02:58.389028|             MAIN > Communication with FLAOWFS established
AOARB.L.202101310000.log:python_61935       |INF|      2462|2021-01-31 11:06:20.560059|             MAIN > Request: OffsetXY((-1.200469970703125, -2.3921799659729004))
AOARB.L.202101310000.log:python_61935       |DEB|      2463|2021-01-31 11:06:25.171768|             MAIN > Status after command: AOArbitrator.LoopSuspended
AOARB.L.202101310000.log:python_61935       |INF|      2464|2021-01-31 11:07:10.124075|             MAIN > Request: Resume()
AOARB.L.202101310000.log:python_61935       |DEB|      2465|2021-01-31 11:07:10.393937|             MAIN > Status after command: AOArbitrator.LoopClosed
AOARB.L.202101310000.log:python_61935       |INF|      2466|2021-01-31 11:08:00.420360|             MAIN > Request: Pause()
AOARB.L.202101310000.log:python_61935       |DEB|      2467|2021-01-31 11:08:00.881063|             MAIN > Status after command: AOArbitrator.LoopSuspended
AOARB.L.202101310000.log:python_61935       |INF|      2468|2021-01-31 11:08:03.688848|             MAIN > Request: OffsetXY((-1.2076243162155151, -2.3896543979644775))
AOARB.L.202101310000.log:python_61935       |DEB|      2469|2021-01-31 11:08:07.799784|             MAIN > Status after command: AOArbitrator.LoopSuspended
AOARB.L.202101310000.log:python_61935       |INF|      2470|2021-01-31 11:08:11.139909|             MAIN > Request: Resume()
AOARB.L.202101310000.log:python_61935       |DEB|      2471|2021-01-31 11:08:11.644706|             MAIN > Status after command: AOArbitrator.LoopSuspended
AOARB.L.202101310000.log:python_61935       |ERR|      2472|2021-01-31 11:08:11.644737|             MAIN > [AOException] RETRY: Star not found - loop still paused (-20002) WFSARB_STAR_WRONGFLUX
AOARB.L.202101310000.log:python_61935       |INF|      2473|2021-01-31 11:09:31.247701|             MAIN > Request: Pause()
AOARB.L.202101310000.log:python_61935       |DEB|      2474|2021-01-31 11:09:31.247755|             MAIN > Status after command: AOArbitrator.LoopSuspended
AOARB.L.202101310000.log:python_61935       |INF|      2475|2021-01-31 c.475616|             MAIN > Request: OffsetXY((1.2076200246810913, 2.3896501064300537))
AOARB.L.202101310000.log:python_61935       |DEB|      2476|2021-01-31 11:09:45.786401|             MAIN > Status after command: AOArbitrator.LoopSuspended
AOARB.L.202101310000.log:python_61935       |INF|      2477|2021-01-31 11:09:50.817199|             MAIN > Request: Resume()
AOARB.L.202101310000.log:python_61935       |DEB|      2478|2021-01-31 11:09:51.002927|             MAIN > Status after command: AOArbitrator.LoopClosed

-- DougMiller - 31 Jan 2021
I Attachment Action Size Date Who Comment
Bin4_noise_1.pngpng Bin4_noise_1.png manage 31 K 31 Jan 2021 - 02:54 DougMiller  
bin4_jitter_1.jpgjpg bin4_jitter_1.jpg manage 89 K 31 Jan 2021 - 02:54 DougMiller  
bin4_modal_1.jpgjpg bin4_modal_1.jpg manage 81 K 31 Jan 2021 - 02:54 DougMiller  
moonlight.jpgjpg moonlight.jpg manage 39 K 31 Jan 2021 - 09:58 DougMiller  
Topic revision: r4 - 31 Jan 2021, DougMiller
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback