UT 25 Jun 2021 - AO Engineering Night



Night Info

  • Sunset 19:37
  • -18 Twilight ends 21:13
  • Night Center 00:22 - nominal handover from LBTI to LUCI 00:17
  • -18 Twilight starts 03:31
  • Sunrise 05:08
  • Moon up all night

Night Plan

  • LBTI Testing first half, use science scripts, no NCPA
    • LBTI conditions: seeing < 1.3", thin clouds (1-1.5mags)
  • LUCI testing 2nd half, AO and ESM scripts used for engineering, no NCPA
    • Conditions: seeing < 1.3" for full AO, seeing < 2" for ESM, thin clouds (1-1.5mags)
  • AO Guidelines from J. Power
  • LUCI-AO Script Locations
    • Full AO: /home/lbto/Service_Observing/2021A_D_E/May_Jun_Jul_2021/AO_Testing/AO_Testing_V28.xml
    • ESM: /home/lbto/Service_Observing/2021A_D_E/May_Jun_Jul_2021/ESM_Testing/ESM_Testing_V24.xml
  • Morning Twilight: PEPSI follow-up (04:15 MST) to monitor the newly discovered Nova V1674 Her - 20min

Backups

If AO is halted for technical issues - OR - conditions too poor to continue (seeing >1.3" first half, seeing > 2" 2nd half; clouds 2+mags extinction)
  • Conditions Good (AO halted for technical reasons)
    • MODS field collection data - (doug completed IDL code checks) - 1-1.5 hrs
      • Conditions: seeing < 1.2" - clouds OK
    • DONE LBC Trail test - 2hrs
  • Conditions too Poor
    • DONE LBC trail test - 2hrs
      • Script location: /home/lbto/Service_Observing/2021A_D_E/May_Jun_Jul_2021/LBC_MayJuneJul2021
    • Pointing model data collection (PtStars - LUCI) - 2hrs
    • Photometric standards (LBC, MODS, or LUCI)
      • ANY seeing conditions, but must be clear
      • MODS scripts: /home/lbto/Service_Observing/2021A_D_E/May_Jun_Jul_2021/MODS_Checkout/SpectroPhotStds or PhotFields
        -MODS1 DONE , MODS2 had IMCS related scripting issues
      • LUCI scripts: /home/lbto/Service_Observing/2021A_D_E/May_Jun_Jul_2021/LUCI_PhotStds - OT file: LUCI_Phot_ZP_OT_V2.xml
    • Backup Science - Local Mergers - bino LUCI
      • scripts are 1hr bino, but continue for 90min or longer depending on how poor
      • /home/lbto/Service_Observing/2021A_D_E/May_Jun_Jul_2021/Rothberg_Local_Mergers
      • OT File: COBandhead_Mergers_V14.xml
      • NGC 6598
      • UGC 10607
      • UGC 10675
      • NGC 7585

Night Summary

1:00 Setup of LBTI, LUCI's, and AO logging. LUCI1 field stop verified.

2:00 LBTI is fully up and checked out. Starting the LUCI wfs.
wfs_init
all processes came up without issue
Operate
WFS Camera looks good, everything came up fine. Leaving warm.

LBTI Log:

https://docs.google.com/spreadsheets/d/12I39t3YglkeRd-hHyHg2tCZnJKl71KCuRmnO1pmfXD8/edit?usp=sharing

LUCI SOUL Log template:

https://docs.google.com/spreadsheets/d/1fopABkYtWEmZrznF_hMoKByDf1y4s3hYRaE5-_-FDeY/edit?usp=sharing

LBTI SOUL Checkout:

2:41 we are open, sending preset to Checkout star Eta Uma. Refining Lmircam pupils while Jared refines pointing.

Updated tertiary pglocbal offsets:

During center pupils:

Error start #2

Timeout of IDL controller triggering a failure

02:58:53.661811| MAIN > FSM_LOAD_SHAPE: -----------> Error in low level procedure CMD #6 -5001
03:03:40.749983| MAIN > Executing command: errcode= fsm_fault_recovery(/FORCE) 
03:03:44.941973| MAIN > Error reading BCUs: [CommandSenderTimeoutException] Timeout expired [HouseKeeper.cpp:2730] 
03:05:37.950108| MAIN > Timeout - Will emergencyReact next loop 
AdSecArbitrator_49349|ERR| 9209|2021-06-25 02:58:53.850105| IDL > Command returns error : Timeout error
20212506 02:58:53.8506 Alert: Warning: stopAo failed or syntax error occured
AdSecArbitrator_49349|INF| 9210|2021-06-25 02:58:53.857384| COMMANDHANDLER > Command execution failed: Fsm forced to Failure
AdSecArbitrator_49349|ERR| 9211|2021-06-25 02:58:53.857673| MAIN > StopAo: FATAL - Timeout error

Block errors noted in the logs:
M_ADSEC |ERR| 24901719|2021-06-25 02:58:53.649281| BLOCK_ERROR [-1025]: A message cannot be sent for network congestion. - sending notification for variable "ADSEC.L.LOOPGAINS" to auxloops.L@M_LBTIWFS
M_ADSEC |ERR| 24901730|2021-06-25 02:58:53.749461| M_LBTIWFS: BLOCK_ERROR [-1025]: A message cannot be sent for network congestion. - sending reply
M_ADSEC |ERR| 24901732|2021-06-25 02:58:53.849644| BLOCK_ERROR [-1025]: A message cannot be sent for network congestion. - sending notification for variable "ADSEC.L.B0_A" to auxloops.L@M_LBTIWFS
M_ADSEC |ERR| 24901848|2021-06-25 02:58:54.214351| M_LBTIWFS: EXCD_BLOCK_ERROR [-1026]: Block error number treshold exceeded - Client: M_LBTIWFS. Closing it

Josh is restarting LSS, commuication issues

Error state #1:

AdSecArbitrator_49349|WAR|      9242|2021-06-25 03:06:51.634079|             MAIN > Cannot read coils status ([AOVarException] thGetVar error (code -5001) Timeout errorVariable: adamhousekeeper.L.COIL_STATUS (type INT_VARIABLE, len 1)
 

From Xianyu:
These errors occured during recovery from failure state #2

02:58:53.661811| MAIN > FSM_LOAD_SHAPE: -----------> Error in low level procedure CMD #6 -5001 
03:03:40.749983| MAIN > Executing command: errcode= fsm_fault_recovery(/FORCE) 
03:03:44.941973| MAIN > Error reading BCUs: [CommandSenderTimeoutException] Timeout expired [HouseKeeper.cpp:2730] 
03:05:37.950108| MAIN > Timeout - Will emergencyReact next loop
03:03:43.927843| DSPPosCheckCnt > 467 467 0 0 0 0 0 0 467 467 467 467 467 467 0 467 467 467 467 0 0 0 0 0 467 467 467 467 467 467 467 0 0 0 0 467 0 0 0 0 467 467 467 467 468 468 468 468 469 469 469 468 469 468 468 468 468 468 468 468 468 468 468 468 469 469 469 469 469 469 469 468 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 0 0 0 0 0 0 0 0 467 467 467 467 468 468 468 468 0 0 0 0 0 0 0 0 468 468 468 468 468 468 0 0 468 468 468 0 0 468 0 468 468 468 468 468 468 468 468 468 469 469 0 469 468 0 468 468 468 469 469 469 468 468 468 468 469 469 469 469 469 469 469 468 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 469 470 470 0 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 468 0 468 468 0 0 0 0 468 468 468 468 468 468 468 468 468 468 468 0 0 0 0 0 468 468 468 468 468 468 0 0 0 0 0 0 0 0 0 0 468 468 468 468 468 468 468 468 469 469 469 469 469 0 469 469 469 469 469 469 469 469 469 469 470 470 470 470 470 470 470 469 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 0 470 470 470 0 470 470 470 467 0 0 0 0 0 0 0 468 468 468 468 468 468 468 468 0 0 468 0 0 0 0 0 468 468 468 468 468 0 468 0 0 0 0 0 0 0 0 0 468 468 468 468 468 468 0 0 469 468 468 468 468 468 468 468 468 468 468 468 468 468 468 468 469 469 469 469 469 469 469 468 469 0 469 469 469 469 469 469 470 470 470 470 470 470 469 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 470 0 0 0 0 0 0 0 0 471 471 471 471 471 471 471 471 0 0 0 0 0 0 0 0 472 472 472 472 0 0 0 0 0 0 472 472 0 0 0 0 472 472 472 472 472 472 472 472 473 473 473 473 472 472 472 472 472 472 472 472 472 472 472 0 473 473 473 473 473 473 473 472 473 473 473 473 473 473 473 473 473 473 473 473 473 473 473 473 473 473 473 0 474 474 474 474 474 474 0 474 474 474 474 474 0 474 474 474 474 474 474 474 470 470 470 470 0 0 0 0 470 470 470 470 470 470 470 470 470 470 0 0 0 0 0 0 470 470 470 470 471 0 471 0 0 0 0 0 0 0 0 0 471 471 471 471 471 471 471 0 472 471 471 471 471 471 471 471 471 471 0 471 471 471 471 471 472 472 472 472 472 472 472 471 472 472 472 472 472 472 472 472 472 472 472 472 472 472 472 472 472 472 472 472 0 472 472 472 472 472 472 472 473 473 473 473 473 473 473 473 473 473 473 473
03:03:43.931075| MeanDSPCoilCurrent > 0.0363548
03:03:41.974629| FAST > Processed 1 cycles of 23 vars in 1.04 s ( 0.964 Hz). HW frame rate 10 
03:03:41.983385| MAIN > Adam in status: 1:1:0:0:0:1:1:1
03:03:42.001263| MAIN > Adam out status: 1:1:1:1:1:0:0:1
03:03:43.001722| FAST > Processed 1 cycles of 23 vars in 1.03 s ( 0.974 Hz). HW frame rate 10 
03:03:43.019289| MAIN > Adam in status: 1:1:0:0:1:1:1:1
03:03:43.037348| MAIN > Adam out status: 1:1:1:0:1:0:0:1
03:03:42.572376| MAIN > Disable coils for avoiding any dangerous reset.
03:03:43.070483| MAIN > Reference wave reduced to 1/8 of std amplitude

This was during the fault recovery from the Error state #2,
This instance of Error state #1 did not cause an overcurrent because the shell was already rested.but it does look like we were communicating to the mirror when this happened so we aren't dead in the water yet....

3:09 is executing a an Adsec Stop Start.

3:14 Josh and Stephen Hooper will be restarting AOS. They have rested the shell after the setflat AO was completed after recovery.

3:18 Setting the shell again and resending preset. Messages are not coming through in IIF.

Hooper and Josh are restarting IIF.

3:31 Resending preset, still nothing in the AOS or IIF logs.

3:35 We are going to take a few minutes to analyze the various issues: the failures and log issues

3:39 We are going to close up in preparation for restarting TCS. The humidity is 87% and rising and we want the system in a safe state.

Xianyu and Guido are continuing investigations in the logs:
Further investigations show that we cannot trust adsec fault recovery. The fault recovery assumes the shell is rested but this is not always the case. The code must check that the coils are infact disabled before changing the square wave. Another problem is the message deamon doesn't handle the communication properly. adsec stop start is the safest recovery in that case since that will trigger the moxa recovery (which has a different path for recovery) will disable coils.

4:12. TCS restart complete. Sending a closed dome test preset. Humidity at 89.3%

We have reauthorized as LUCI and tried a test preset but logs are still not populating. Humidity 91%

Guido:

There is an issue on the ASM fault recovery.
The disable coil was issued but never executed. The theory is that when we operated the system we wont be able to stop. We need to understand why this command was lost.
These commands all go through the msgdmn. If that is not working correctly then we have a critical issue.
There were hard forces applied to the shell sucking the shell to the reference body. The housekeeper stopped for ~10seconds. This was all wrapped in teh recover fail button.

In normal operation the arbitrator protects from these operation. But there are indications that the RIP and fault recovery have issues at the software level at the low level IDL routine and high level software.

We are stopping all AO operations for today

5:25 humidity has dropped. We are switching to LBC's. We are going to power off the 2 WFS's before we rest the shells and recofigure.

5:37 recofigure to LBC started

LBCTrailing test

6:05 Preset to 18h +29d for rotational trailing tests. El 77:04 deg
6:07 Running dohybrid
6:17 blue looks a little soft after convergence so running dofpia again. DIMM 0.9"
Red converged 1.28" seeing, blue in 1.46:31

6:32 Preset to 18:10hr 29deg. Elevation 79:50 to start. lbcb.20210625.063244 & lbcr.20210625.0632400

We see a dark spot on middle of red images. Possible condensation. It changes throughout image set. Humidity ~80%. IT#8407
slight alteration to scheme. We are staying on the same field because of timing. and for simplicity.:
6:55 running dofpia and copointing. Collimation took a long time, likely because of the high elevation. There also was a pretty significant size copointing off set in place, IE 10.55" and 4.34" despite copointing recently only 10 minutes away at an elevation of about 77 deg instead of 85deg.
 
7:17 Preset to 18:15 at elevation 85degrees. Whoops we missed transit.

7:32 executing lbcrangebal only for 18:40, dec 29 target, IE -3.11", CA -3.18"
7:36 science started at an elevation of 85:37 9 minutes before transit.
 

MODS SpecPhot

7:54 Reconfiguring to MODS.
Josh is having issues with the right side swing arms but resolved.
8:07 Mods is awake and simSnap taken. Conditions are clear so we are executing spectrophotometric standars with MODS starting with a low airmass one followed by a higher airmass one.
8:20 Pointing and collimation presets
8:27 Preset to BD33. Seeing 1.2" on the guiders.
MODS2:
Computed Slit Alignment Offset:
dX = 3.343 arcsec
dY = 8.226 arcsec

MODS2 Offset Command:
offsetxy 3.343 8.226 rel

MODS1:

Computed Slit Alignment Offset:
dX = -0.986 arcsec
dY = 11.624 arcsec

MODS1 Offset Command:
offsetxy -0.986 11.624 rel

8:36 Science started bd332642.obs script running through all inst configs. Elevation 44 degrees. Seeing 1.2-1.6"

7:58 Mods2 IMCS lock failed on RED grating config, retry successful. Working on a second desktop so missed the retry needed for about 2minutes.
ERROR: MODS2 had an IMCS error persisting during the red config.
>> imcslock
** ERROR: IMCSLOCK disallowed during exposures
** Abort, Retry, or Ignore? >


tried ignoring and the blue exposes started to take. Noticed that the inst config changed to the blue config on the second red exposure. I did not trust the Red images although said locked (0008 -0009) 7:10 Retry: happened again!! identically. I wonder if this is a ROBS issue. I'm going to try and run this script on obs2.
7:21 Starting. It happened again running the script:
DONE: RED EXPTIME [[ExpTime][ExpTime]] =120 seconds. -CLAMP EXPSTATUS=DONE
>> red go
(Note that red is still exposing and MODS reconfigured to blue only)
DONE: RED IMCSLOCK Red-Channel IMCS locked-on [[RedIMCS][RedIMCS]] =LockedOn
>> instconfig blue grating
DONE: INSTCONFIG completed
>> imcslock
** ERROR: IMCSLOCK disallowed during exposures
** Abort, Retry, or Ignore? >
<b>* bd332642.obs aborted with errors at line 14 command 'imcslock'
 ... aborting IMCS lock request ...
*</b> ERROR: imcsabort disallowed during exposures
 ... abort sequence terminated with errors

MODS1 is completing without error. Trying to comment all out except the red grating configuration. Humidity is 91.6% and rising and we are at 33 in elevation on this target nearly complete on MODS1. We will not be able to complete MODS2. Only dual grating obtained with MODS2.

Seeing up to 2" on the guiders

7:39 I commented out everythign except the red grating section. The script showed ** Script bd332642.obs done immediately upon starting the first of the 3 exposures. IMCS lock issue each time though, but we are at a low elevation so not surprising. Succeeds each time with a retry, I think there may be an issue with the red go command. We are closing for high humidity. MODS 1 specphot complete in all configurations.

The issue was investigated further and appears to be related to IMCS lock. I repeated the issue on MODS1 and saw it on both ROBS and OBS operations. IT 8416 was writtedn up to describe the MODS scripting issue.

Calibrations

Taking mods cals, LBC biases

modscals: all biases,

5" slit flats for dual grating, red grating, blue grating.
arcs for dual grating, red grating, blue grating, dual prism, red prism, blue prism
slitless for dual grating

PENDING: 5" slit flats for dual prism, red prism, and blue prism,
slitless flats red grating, blue grating

-- BarryRothberg - 24 Jun 2021
Topic revision: r6 - 25 Jun 2021, JenniferPower
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback