UT Nov 24 2021 - D/E Night

Night Info

  • Sunset 17:20
  • Twilight Ends 18:41
  • Moonrise 20:36
  • Twilight Starts 05:32
  • Sunrise 06:53

Night Plan

Rollover of plans from previous night and catch up with Eng/Sci observations

AO Plan:

  • Starting Configuration bino-LUCIs - AO
    • Conditions:
      • thin clouds
      • 1.2" or better for full AO, 1.5" or better for ESM
    • OT files:
      • /home/lbto/Service_Observing/2021B_D_E/LUCI/AO_ESM
      • ESM is ESM/ESM_V26.xml
      • AO is AO/AO_Testing_V32.xml
        • Star on-axis testing use folder "AO STARS - FULL DITHERS *NEW*"
        • Science-like targets: Einstein's Cross is in folder "AGN/ULIRGs/LIRGs"
        • ADI tests - separate OT file (should be merged later with the AO Testing OT)
    • Currently Bin 4 is NOT available - may change after discussion with Arcetri
  • OSCO at twilight
  • Full AO testing on-axis
    • Bin 1 stars
    • Bin 2 stars
    • Bin 4 stars
  • AO Science-like Observations
    • Einstein's Cross (Bin 4) - twilight-22:45hr
    • ADI testing on HR8799 or IP Tau (Bin 1) - transits ~midnight
  • ESM testing
    • M52
      • Bin 1 (R=10.52) - dither sequence (off field) - 30deg limit at 01:15 - KHJ
      • Bin 4 (R=16.51 forced) - dither sequence (off field) - 30deg limit at 01:15 - KHJ
    • HCG 31
      • Bin 1 (R=10.93) - dither sequence (off field) - visible 22hr-04:30hr - KHJ
    • M51
      • Bin 4 (R=16.51 forced) - dither sequence (off field) - visible 04:30hr-twilight - Ks only
  • Test of full AO w/N375 camera (lower priority - if time)
  • Poor Condition Backup
    • Local Mergers:
      • OT File: /home/lbto/Service_Observing/2021B_D_E/PoorWeather/LocalMergers/COBandhead_Mergers_V30.xml
      • LUCIs (to keep at same focal station)
      • Arp 230 - twilight-23:45 (telluric above 30deg 19:45hr)
      • NGC 1210 - 22:30-00:30hr
    • Reconfigure for TMS/LBC testing at 21:15hr if conditions are deemed too poor (e.g. contintued trend of poor seeing) to try AO
    • Continue with LBC TMS testing until 02:45hr or 03hr - unless conditions improve for AO - then resume AO testing
      • Switch back to AO if 30min consistently at or better than min requirements for AO (and trends suggest no further degradation in weather beyond that)
    • If at LUCI focal station and AO cannot be executed (or seeing >1.5" and >thin clouds)
      • Dwarf Merger SDSSJ1148+2149 - 1.5hr
        • OT: /home/lbto/Service_Observing/2021B_D_E/LUCI/Rothberg_Science/Dwarf_Mergers/DwarfNergers_OT_V4.xml
        • up to ~1mag extinction, seeing up to 2"
        • Fraternal L1 zJsepc, L2 HKspec - if one LUCI is offline, then observer HKspec only
        • Telluric star first (rises before target)
        • Target hits 30deg elevation 03:33hr
        • Run script 2x (script is 30min shutter time more if time permits)

If no AO available:

  • Configure for bino-MODS
  • SpecStd Feige 110 in twilight
  • M31 Nova observations - bin-MODS - 2.5hr - until 21:15hr
    • File Location: /home/lbto/Service_Observing/2021B_D_E/MODS/RMWagner_M31_Nova
      • m31n2008_pa4pt29_1pt2.acq
      • m31n2008_1pt2_extended1800.obs
      • Updated scripts to put bright star N of nova into target (PA = 4.29 based on SX and DX r-band acquisition images)
    • up to 0.5 mag cloud extinction, seeing up to 1.5"
    • acq image may need 180sec
    • 1800sec exposures
    • hits 30deg elevation limit at 02hr
  • Reconfigure to LBCs 21:15-21:45hr
  • LBC/TMS Testing 21:45-02:45hr
    • any conditions
  • Reconfigure to LUCIs 02:45-03:15hr
  • Dwarf Merger SDSSJ1148+2149 - 1.5hr
    • OT: /home/lbto/Service_Observing/2021B_D_E/LUCI/Rothberg_Science/Dwarf_Mergers/DwarfNergers_OT_V4.xml
    • up to ~1mag extinction, seeing up to 2"
    • Fraternal L1 zJsepc, L2 HKspec - if one LUCI is offline, then observer HKspec only
    • Telluric star first (rises before target)
    • Target hits 30deg elevation 03:33hr
    • Run script 2x (script is 30min shutter time more if time permits)
  • Backup Programs - if conditions too poor for M31 Nova
    • Start of night - MODS
      • OT File: /home/lbto/Service_Observing/2021B_D_E/PoorWeather/LocalMergers/COBandhead_Mergers_V30.xml
      • MODS LS - Arp 230 until 21:15hr - then swtich to LBCs
        • Note: There is also a LUCI LS OB for Arp 230 - that can be executed as a backup another time - OR - if already configured for LUCIs
    • If conditions are worse than limits for Dwarf Mergers by 02hr during LBC/TMS, reconfigure at 03hr to LUCIs
    • Local Mergers - NGC 4004
      • OT File: /home/lbto/Service_Observing/2021B_D_E/PoorWeather/LocalMergers/COBandhead_Mergers_V30.xml
      • LUCI LS observations of NGC 4004
      • Target hits 30deg elevation at 03:30hr, Telluric at 04hr


Closed for winter storm

1:30 Field stop alignment checked. Stow pins in, deploying ARGOS swing arm.

Closing on Bin1:
  • Center pupils failing - the camera lens was railing on the far limit. Doug tried to center the camera lens but continued to fail. Doug railed the camera lens to the opposite limit then and the camera lens was then able to increment into the correct position.
  • 05:01:22 in WFS arbitrator
  • Optimize gain plots do not appear on the FLAO GUI
  • Currrent Gains with no distrubance 1.88 0.18 0.12
  • Closed without NCPA TN#011343. luci1.20211124.0014. Round psf with some trefoil
  • with NCPA TN#011543. luc1..0015, looks better without the trefoil
  • with NCPA TN#011628 luci1.0017
  • loop paused with lamp off, dark 10x2.5 cube luci..018, 2x2.5 cube luci1.0019
Closing on Bin2
  • Preset sent for 13.0 mag with light source set to 58
  • Center star failed because the magnitude was a hair off supposedly, although at 56 when closed lamp the magnitude was 13.4 which is in the middle o fhte bin 2 range so that should not have failed.
  • Center pupils was doing nothing again. Had to manually slide the camera lens in the positive direction to start motion to get it to find the correct position.
  • Re ran center pupils. Railed in X, but within limits on the diff.
  • We have 6.3 photon/subapp at 13.5 mag. which seems too low
  • Gains 6.89, 0.18, 0.41 for bin2
  • Closed without NCPA TN#013913. luci1..0019. Some trefoil.
  • GOpt on TN#014340 luci1.0020
  • With NCPA on TN#014522 luci1.0021
  • Taking darks 10x2.5 cube luci1.0022

  • Turning up light source for bin2 to 58 (60 was too bright). Estimated magnitdue 13. 10.1 photons/subapp. Reoptimizing after loop opened. 7.14, 0.18, 0.58
  • Closed with NCPA off TN#015529 (no luci images), TN#015551 luci1..0023
  • With Optical Gain, TN#015747 luci1.0024
  • With NCPA on TN#02004. luci1..0025
Bin 4
  • Sending preset with lamp set to 45 for center star to grab it, then will up to 55 so we can get some counts on OCAM. Center star is too restrictive. Estimated magnitude at 55 is 14.0.
  • Center star failed to center (just a hair from center) on first attempt, but grabbed on second
  • Camera lens is failing to move with center pupils, required manual intervention to start the camera lens motion to find the pupils
  • In closed loop we see the same instabilities as yesterday in bin4. the flat looks bad
  • we loaded yesterdays flat and it still looks bad
We loaded the flat in bin4 20211123.sav. It looks bad. We went to bin1 again and the flat looks bad there too. So why?

GOpt gui is not coming up, error can not find adsec.L.mc2

2:33 We are going to restart adsec to see if we can fix the shape on the adsec.
  • did not fix shape
2:41 Setting up in bin1 to build new flat
  • Bin 1 optimized with gains 1.92, 0.18, 0.18. Saving new flat 20211124.sav
2:51 Bin4 preset with new flat
  • lamp set to 45 (around 17th mag)
  • center star failed 253|2021-11-24 02:51:02.111993| MAIN > RETRY: source_acq error: mag 8.964 - expected: 14.693 difference: 5.729
    that is balogna!
  • Looks like pacman when closed. Adding any gains looks like its crunching the mirror.
3:03 Bin4 ACE with old pupils
  • rotator moved to 158. Doug loaded ace table from 20211025 with pupils 20200916_173857
  • And don't those pupils look nice and flat
  • Pupils look flat in center pupils and closed loop but we are raining in Y. Set gains to 0.75, 0.25 0.25, TN#031432, luci1..0026
  • reloading 20211124.sav flat, Juan Carlos saved some TN's
  • Juan Carlos is adjusting the pupil
    • pupil 20211124_010445 as soon as ho1 was added the mirror crunched, but looked flat prior to ho1 added.
  • Light appears to be spilling from one pupil intot he next. Going to try adjusting the lightsource intensity down. Setting to 45 for an estimated magnitude ~17
  • 0.75, 0.25 0.25 TN 035409, luci1.0031 were the old pupils 20200916_173857
  • 0.75, 0.25, 0.25 TN 040120 luci1..0032 with pupils 20211124_010445, aberrations with ho1. Offload may not have been enabled.
  • reapply flat.
  • pupils 20211124_010445, starts to build shape with ho1 >0.1. TN#040722 luci1..034
    There are only a few pixels between pupils so there are things Juan Carlos will work on.
4:13 Softare work with Brandon
  • Resting shell and safing AO for adsec restarts
  • Warming OCAM and safing wfs for wfs software restarts.
  • powering off adsec, stop and restarting in new env:


  • setflat default at 638 modes
  • Power off WFS, w_stop_all. Starting back up in source dir
  • w_check didn't work...

    Warning: No xauth data; using fake authentication data for X11 forwarding.

    Last login: Wed Nov 24 04:31:20 2021 from



    [AOeng@soul-sxwfs source]$ w_check

    Traceback (most recent call last):

    File "/home/AOeng/soul/latest/aoroot/bin/w_check", line 5, in <module>

    from AdOpt import cfg, processControl, thAOApp

    File "/home/AOeng/soul/latest/aoroot/lib/python/AdOpt/cfg.py", line 273, in <module>

    exec('from cfg_'+subsystem_name+' import *')

    File "<string>", line 1, in <module>

    ImportError: No module named cfg_

  • config.py is not getting the subsystem name even though its not in the environment. The symlink had a typo
  • housekeeper failed to start with first w_start_all, stop_all, then start_all again. Housekeeping was successful in coming up.
  • Error during power up: Timeout wating for techviewctrl.L.STATUS (-5001) TIMEOUT_ERROR

    WfsArbitrator _37386|INF| 70|2021-11-24 04:44:22.360351| MAIN > Timeout waiting for techviewctrl.L.STATUS - TIMEOUT_ERROR
    WfsArbitrator_37386|ERR| 71|2021-11-24 04:44:22.360437| MAIN > Exception: Timeout waiting for techviewctrl.L.STATUS
    WfsArbitrator_37386|ERR| 72|2021-11-24 04:44:22.360485| MAIN > Timeout waiting for techviewctrl.L.STATUS (-5001) TIMEOUT_ERROR

  • Can't bring up the hardware gui :

    Traceback (most recent call last):

    File "/home/AOeng/soul/latest/aoroot/bin/wfshw.py", line 89, in <module>

    main = GUIMain()

    File "/home/AOeng/soul/latest/aoroot/bin/wfshw.py", line 74, in init


    File "<string>", line 1, in <module>

    File "/home/AOeng/soul/latest/aoroot/lib/python/AdOpt/WfsGUI/restoreSetup.py", line 13, in init


    File "/home/AOeng/soul/latest/aoroot/lib/python/AdOpt/WfsGUI/restoreSetup.py", line 43, in refreshList

    lst = os.listdir( prefix)

    OSError: [Errno 2] No such file or directory: '/home/AOeng/soul/latest/aoroot/calib/wfs/current/setups/'

  • Looks like powerup may have failed because it was looking for that setup directory. Once brandon copied over hardware gui came up and we were able to proceed with power up.
  • Error during command powerOn
    Operate:FATAL - timeout waiting for for baysidex.L.STATUS (-50001) timeout

    WfsArbitrator _37386|INF| 137|2021-11-24 04:53:31.895060| MAIN > Timeout waiting for baysidex.L.STATUS - TIMEOUT_ERROR
    We are at PA158 from earlier testing. Going to 341 and retrying.

  • OCAM and CCD47 not live. CCD47 just required a rebin. OCAM required a bigger kick requiring a restart of the BCU interface, Slope ctrl, OCAM, Master Diagnostic
  • Closing on bin1 for Optical loop diagnotic test
    • Center pupils failed. The camera lens railed in teh neg limits in both X and Y initially, manually adjusted but then it railed again in the positive in ZX and negative in Y. Pupils 20211023_215958. Tried to apply some gain to get the pupils to catch but safe skipping immediately.
    • Loading the flat from pre restart. Some defocus so removing
    • Safe skipping as soon as we stars center pupils.
    • loooks like it was just because pupils were so far off and lens was not converging
    • manual alignment of the lens and reclose worked and looked flat
    • Filotto errors in teh adsec log correspond to wfs stop loop commands.
    • Optimize gain 1.15, 0.11, 0.11 TN20211124_05733, luc1..034 (OCAM warm), blind filter in place. H filter NT055938, luci1..0035
  • Closing bin1 for filotto error diagnotstic test and pause pause test with cooled ocam
    • Gains 1.92, 0.18, 0.23 but skipping frames. Wrong filotto errors are still streaming. The pupils weren't perfectly alignmed when optimizing so reoptimizing: gains 1.56, 0.18, 0.23
    • Pause through AOS no filotto errors, pause through the wfs no filotto, pause through Adsec control gui no filotto errors
    • TopAO through adsec control gui to help determine source of filotto errors
    • Open in WFS control guide = filotto errors... OCAM not live when reclosing caused a RIP. Adsc stop/start to recover
  • Bin 1 Pause pasue test
    • Gains 1.11, 0.20, 0.28, optical gain & NCPA on, TN 065031
    • pause through the AOS sends the pause to both the adsec and wfs
      • pause resume - gain vector is in adsec and gains applied
      • pause pause pause pause resume gain vector is in adsec and gains applied
    • Pause through the adsec pause the loop, sets gain0 , but doesn't tell the wfs anything. the state on teh wfs arbitrator state doesn't change to match adsec state.
    • Pause through the WFS control gui is not working. Pause will show in the WFS control gui. doLoop Pause is showing in the WFS commands gui and not shown in the Adsec Gui. Pause in WFS still continues to send slopes to adsec.
7:30 Swapping Housekeeper versions. rested the shell. Stopped the housekeeper. Barnsdone started his version but not in time. Adsec went into failure. Going to adsc_stop and start in his new environment.

7:49 Looks like it didn't fully update teh files. We are going to rest, power off the adsec and update the housekeeper again.
  • 7:55 adsc stop/start
  • housekeeper not running after load program
    • idl_71662 |INF| 223|2021-11-24 07:57:26.320475| MAIN > IDL Housekeeper is not yet fully initialized or gives bad reading...
      AdSecArbitrator_71779|WAR| 78|2021-11-24 07:57:26.320806| IDL > Syntax error found. Catching...
      Last message repeated
      AdSecArbitrator_71779|WAR| 79|2021-11-24 07:57:26.321106| IDL > Syntax error successfully catched

  • 8:03 Going back to pause-resume env
8:08 handing to Doug for NCPA checks
  • Setup subframe. Subframe SW 1 905 1082 256 256 after rotating around with low gains and finding midpoint of rotation. No disturbance, 1 sec exposusures
  • luci.0039 5x1sec cube
  • Dark luci1.20211124.0040 TN#085533
  • Unexpected fall out from the RTD update: doug looks directly on the lucifer machine for last image so is reading back the 50## image name rather than the 00##.
9:27 Starting data collection in modes 7 & 8
  • Doug mentioned that there is some weird vibration at 330deg
  • 10:55 Doug completed the collection of coma NCPA coeffients and is inputting them
Starting collection of modes 9 & 10
  • at rotator position of 245 we railed the camera lens in Y and opened loop. had to go to 275 to reconverge. Gains 2.37, 0.18, 0.23. Apply optical gain. 1.43, 0.11, 0.13. GOpt value 0.417
  • going back to 245 after convergence y 0.85. At 215 we can't converge with the same correction. Setting gains to half and applying optical gain: 0.72, 0.5, 0.5 with optical gain on of 0.502
  • Back to full strength gains of 1.43, 0.11, 0.13 with optical gain of 0.5 at rotator angle of 155 again.
  • The loop opened , some higher forces (angle 95). reapplied optical gain 0.541.
  • all but the last 2 angles ccompleted for modes 9&10. People are up and there are some vibrations now (and tired observers). calling it a night.
See attached plots for Dougs NCPA coefficient results.

13:00 ARGOS cal unit parked with light source off, disable rr mode and look up tables, rested shell, safed luci,

13:06 Powering off the WFS timeout error:

WfsArbitrator _102759|INF| 211|2021-11-24 13:02:23.596001| MAIN > Turning off main
WfsArbitrator_102759|INF| 212|2021-11-24 13:02:53.596873| MAIN > Cannot turn off Main power 24V switch. Reason: Timeout waiting for powerboard.L.main.CUR - TIMEOUT_ERROR - GENERIC_SYS_ERROR
WfsArbitrator_102759|ERR| 213|2021-11-24 13:02:53.596978| MAIN > Cannot turn off Main power 24V switch. Reason: Timeout waiting for powerboard.L.main.CUR - TIMEOUT_ERROR (-1000) GENERIC_SYS_ERRO

Failed to turn off BCU2K before powering off the main power 24V power switch. To be safe powering on and powering off again.

Start up failed x2 FATAL - cannot turn on Cube stage Reason. Timeout waiting for picctrn.L.cubestage.CUR - TIMEOUT_ERROR (1000) GENERIC_SYS_ERROR

-- BarryRothberg - 23 Nov 2021
  • Mode 7 NCPA Coefficient plots:

  • Mode 8 NCPA Coefficient plots:

  • Mode 9 NCPA Coefficient plots:

  • Mode 10 NCPA Coefficient plots:

  • Old pupils used to create reconstructor, 20211025 ACE bin4 20210916_173857 pupil:
    20200916_173857_bin4 ACE.png

  • Old pupils used to create reconstructor, 20211123 ACE bin4 20211117_192657 pupil:
    20211117_192657_bin4 ACE.png
I Attachment Action Size Date Who Comment
20200916_173857_bin4 ACE.pngpng 20200916_173857_bin4 ACE.png manage 235 K 24 Nov 2021 - 21:01 JenniferPower Old pupils used to create reconstructor, 20211025 ACE bin4 20210916_173857 pupil
20211025pupils_bin4 ACE.pngpng 20211025pupils_bin4 ACE.png manage 235 K 24 Nov 2021 - 20:56 JenniferPower 20211025 pupils with bin4 with gains
20211117_192657_bin4 ACE.pngpng 20211117_192657_bin4 ACE.png manage 242 K 24 Nov 2021 - 21:03 JenniferPower Old pupils used to create reconstructor, 20211123 ACE bin4 20211117_192657 pupil
20211125pupils_bin4 ACE.pngpng 20211125pupils_bin4 ACE.png manage 235 K 24 Nov 2021 - 20:51 JenniferPower bin4 using 20211125 pupils
Mode_10.jpegjpeg Mode_10.jpeg manage 213 K 24 Nov 2021 - 13:34 JenniferPower Mode 10 NCPA Coefficient plots
Mode_7.jpegjpeg Mode_7.jpeg manage 189 K 24 Nov 2021 - 13:33 JenniferPower Mode 7 NCPA Coefficient plots
Mode_8.jpegjpeg Mode_8.jpeg manage 125 K 24 Nov 2021 - 13:33 JenniferPower Mode 8 NCPA Coefficient plots
Mode_9.jpegjpeg Mode_9.jpeg manage 209 K 24 Nov 2021 - 13:34 JenniferPower Mode 9 NCPA Coefficient plots
Topic revision: r11 - 24 Nov 2021, JenniferPower
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback