UT Sep 03 2022 - Night #2 of Restart



Night Info

  • Sunset 18:43
  • Nautical Twilight Ends 19:37
  • Astro Twilight Ends 20:08
  • Moonset 22:00
  • Astro Twilight Starts 04:31
  • Nautical Twilight Starts 05:01
  • Sunrise 05:56
  • Moon illumination ~44%

Staff

  • OSA: D.G.Huerta
  • ISp: E.Solheid
  • Mtn Manager: B.Waggoner
  • ISA/SO: B. Rothberg, A.Becker, R.T.Gatto
  • Technical Expert: J. Hill


Night Plan

Rollover of any tasks not completed previous night

LUCI AGW Wakeup and tests

  • Starting Configuration bino-LUCIs
  • AGW1/2 Alignment - 3hrs total
    • Files needed:
      • ssh -X into TO1 as telescope; then ssh - X as tcs@tcs1
      • /home/telescope/TCS/Configuration/GCS/LUCI_L.cfg or LUCI_R.cfg
      • image files: /lbt/data/share/tcs/lbto.YYYMMDD
      • oac@oac: /lbt/oacontrol/current/etc/oacontrol.conf
    • Pupil alignment check - find a star and center it - need Bent Gregorian (1 hr) - need to do before AGW checkout
    • verify SH spots illuminated - critical - (0.5 hr)
    • verify guiding hotspot center - critical - (1 hr)

Telescope Wakeup

  • SX M3 Alignment LFBG/DX M3 Alignment RFBG - Critical (3 hrs) - requires LUCI AGWs - both sides can be done in parallel - requires SH spot and hotspot check completed
    • Pupil alignment check - find a star and center it - need Bent Gregorian (1 hr) - need to do before AGW checkout, but active optics needs to be working
    • Pupil wobble (2 hrs) - requires LUCI-1/AGW-1 to guide, collimate, SH spots well illuminated, and hotspot centered - can be done before AGW Rot ZP and Transform
  • Field Aberration (2 hrs) - Critical (1hour) (SX more important than DX due to SX removal)
    • Can be done in parallel (data collection, analysis in serial)
    • need both AGW1 and AGW2 on the telescope and checked to execute
    • must complete SH spot centering and hotspot check before executing
    • 1.2" or better seeing (1" or better ideal)
    • Target Fields:
      • BS9177 18:39:33.7 +49:05:38.0
      • StoneP1 22:37:45.0 -01:13:40.45
      • BS9188 23:30:33.4 +38:18:57.0
      • BS9107 03:32:03.0 +37:20:40.0

PEPSI-Pol AGW Checkout

  • Reconfigure from LUCI-PEPSI-Pol (0.25 hr)
  • AGW 3/4 Alignment
    • verify SH spots illuminated - critical - (0.5 hr)
    • PEPSI-Pol collimation check - critical - (0.5hr)
    • PEPSI-Pol Alignment Check for M3 - critical - (0.5hr)

PEPSI-PFU AGW Checkout

  • Reconfigure from PEPSI-Pol to PEPSI PFU (0.25hr)
  • AGW 7/8
    • DONE verify SH spots illuminated - critical - (0.5 hr)
    • DONE PEPSI-PFU collimation check - critical - (0.5hr)
    • DONE PEPSI-PFU Alignment Check for M3 - critical - (0.5hr)
    • DONE Take Standard star and compare fluxes between SX and DX

Outstanding LBC tasks

  • Superfocus sequence (1.2" seeing - stable - or better) - 1hr
  • Verification of filters in LBC-Blue after daytime physical inspection

Closed Dome Work

  • AO tasks if closed for night

Night Log

Tonight's restart theme....

Closed Dome

  • UT 01:07 - made backup copies of LUCI_L.cfg and LUCI_R.cfg as LUCI_L.cfg.20220903, LUCI_R.cfg.20220903
  • UT 01:08 - J.C.Guerra done with WFS check, free to reconfigure
  • UT 01:35 - test preset with LUCIs in the beam
  • UT 01:59 - dome is open

Pointing Check

  • UT 02:01 - pointing check AGWs - will wait until its a bit darker
  • UT 02:11 - pointing check
  • UT 02:18 - sending active preset for collimation

Alignment of AGWs - Light into lenslets

  • https://wiki.lbto.org/Commissioning/BGTechProcedures#Procedure_for_adjusting_the_guiding_hotspot_so_the_light_from_the_star_goes_into_the_S_45H_wfs
  • made backup copies of /home/telescope/TCS/Configuration/GCS/LUCI_L.cfg and LUCI_R.cfg
  • changed maxOffsetToHotspot and maxHotspotFromProbe to 2.54
  • UT 02:22 - moving to another star for collimation (first one was a double star) - will do a first collimation then restart GCS
  • UT 02:27 - having some active optics issues - clearning them and blockng zernikes - trying again
  • UT 02:30 - hotspot is off to the bottom left of the normal positon on both sides - the CFG files are from Jan 2 2022 for LUCI_L and Feb 2021 LUCI_R - backed them up at 01:07 UT today
  • UT 02:32 - restarting GCS to see if changes take effect, resending preset with Zernikes blocked
  • UT 02:34 - hotspot is bigger but still shifted bottom left of normal
    • using Controls on guide control - orignal is 249.196
    • moving +5 pixels X to 254
    • moving +5 pix X to 259
    • Note: For AGW1@LFBG: Decreasing guiding_hotspot_y moves star up in lenslet
    • moving -5 pix Y to 191
    • moving -5 pix Y to 186
    • moving-5 pix Y to 181
    • moving -5 pix Y to 176
    • moving +5 pix X to 264
    • moving +5 pix X to 269
    • moving -5 pix Y to 171
    • moving +10 X, -10 Y to 279, 161
  • UT 02:46 - re-starting GCS left, resending preset
  • after preset the hotspot X,Y is in the same spot and spots llluminated the same
  • UT 02:52 - BR got this backwards, moving +Y to move the star DOWN in lenslet
    • moving Y to 201 pix (+5)
    • moving Y to 206 pix (+5) - error message on facsum "GCS command sethotspot failed set coordiante outside of CCD"
    • trying this from the terminal
    • [tcs@tcs1 GCS]$ gcsclient setHotspot left -x 249.0 -y 206.0
      Hotspot setting didn't succeed!
      hotspot coordinates outside of CCD
    • this is similar to what A.Becker saw on FACSUM
    • trying this on GCS right moving from 270,238 to 270,233
    • [tcs@tcs1 GCS]$ gcsclient setHotspot right -x 270.0 -y 233.0
      Hotspot setting didn't succeed!
      hotspot coordinates outside of CCD
    • looking at oacontrol.conf , but no evidence of change since Nov 9 2021
  • UT 03:06 - giving up for now - will submit critical issueTrak - reconfiguring for PEPSI-Pol

PEPSI-Pol AGW Checkout

  • UT 03:25 - sending pointing preset
  • PEPSIPOL_L and _R.cfg last update Jan 7, 2022, made backup copies
  • maxOffsetToHotspot and maxHotspotFromProbe already set to 2.54, and 3.5, respectively
AGW4 Problems
  • UT 03:29 - error moving AGW4 motors - responding from GCS engineering GUI
  • UT 03:31- I.Ilyn cannot home AGW4, powering off and on, restarting OAC service
  • UT 03:35 - no error on AGW4, sending preset again
  • UT 03:38 - restarted GCS right several times
  • UT 03:41 - restarting azcam server and GCS right again
  • UT 03:42 - guidecam init failed on AGW4
    • I.Ilyn can read the cameras, seems to be TCS communication problem to the OSA station
  • UT 03:47 - restarting OAC server
  • UT 03:55 - AGW4 initialized
  • UT 03:59 - AGW4 in funky state - I.Ilyn may send a preset from PEPSI GUI to test
  • UT 04:00 - I.Ilyn sees a failed to communicate error message for AGW4
  • UT 04:01 - restarting OAC service again
  • UT 04:05 - failure to communicate with MOXA error again
  • UT 04:07 - rebooting OAC service again
    • the repeated MOXA error implies its a hardware problem?
    • still same error - AGW4 unavailable
    • do we need to powercycle AGW4 on the telescope?
  • UT 04:13 - GCS right restarted yet again.....
  • getstatus
    • [tcs@tcs1 GCS]$ gcsclient getStatus right
      GCS Version: 5.6a
      GS is invalid or not yet set.
      GCS status: "OK"
      Selected AGW unit: 4
      Selected AGW name: "NONE"
      Selected AGW type: none
      Associated rotator angle (deg): -56.4547
      GCS isAcquiring: 0
      GCS isGuiding: 0
      GCS isWFSing: 0
      GCS Guide Rate: 0.4
      Guide camera exposure time (ms): 9999999
      WFS camera exposure time (ms): 9999999
      AGW position (mm): 0.922433, 193.383 (native: -0.02, 418.55)
      AGW Focus: -1
      AGW Filter: -1 : B
      AGW Status:
      AGW Power: ON
      AGW Homed State - Position: YES Filter: YES Focus: YES
  • J.Hill notes log file is filled with messages "can't connect to azcam server"
  • I.Ilyn can take images from the camera and WFS on AGW4
  • UT 04:18 - GCS right restarted yet again
    • [oac@oac ~]$ getdata -u4
      general state: 0x201
      general errors: 0
      THETA state: 0x3
      THETA errors: 0
      R state: 0x3
      R errors: 0
      FOC state: 0x3
      FOC errors: 0
      FIL state: 0x3
      FIL errors: 0
      Home Status: 0x17
      UMAC Power Status: 1
      OAC Version: 5.2.1
      === onewire data unavailable =
    • UT 04:23 - restarting GCS right again
    • [oac@oac ~]$ getdata -u4 -n
      general state: 0x201
      general errors: 0
      THETA state: 0x3
      THETA errors: 0
      R state: 0x3
      R errors: 0
      FOC state: 0x3
      FOC errors: 0
      FIL state: 0x3
      FIL errors: 0
      Home Status: 0x17
      UMAC Power Status: 1
      OAC Version: 5.2.1
      === 8051 housekeeping data unavailable =
  • UT 04:26 - D.G.Huerta getting Elliott to see if he can go on the lift to restart AGW4 (this was suggested 30min ago)
  • UT 04:28 - E.Solheid will go up and unplug/replug in AGW4
  • UT 04:30 - moving to zenith, turning off HBS and locking out telescope
  • UT 04:43 - this might have fixed it - apparently unplugging and plugging back in worked....
    • [oac@oac ~]$ getdata -u4 -n
      general state: 0x201
      general errors: 0
      THETA state: 0x100
      THETA errors: 0x20000
      R state: 0
      R errors: 0
      FOC state: 0
      FOC errors: 0
      FIL state: 0
      FIL errors: 0
      Home Status: 0
      UMAC Power Status: 1
      OAC Version: 5.2.1
      Ambient Temp: 12.6 degC
      PC Enclosure Temp: 25.0 degC
      Guide Controller Temp: 13.0 degC
      WFS Controller Temp: 13.5 degC
      Guide Camera Temp: 9.9 degC
      WFS Camera Temp Temp: 10.6 degC
      UMAC Enclosure Temp: 28.1 degC
      -15 Voltage: -12.0 volt
      +15 Voltage: 12.0 volt
      Peltier Voltage: 11.3 volt
      HK Power Status: 1
      HK Flow Status: 0
      HK Alarm Status: OK
      HK Warning Status: OK
    • failed on init guidecamera, rebooting
    • UT 04:46 - restarting GCS right
  • UT 04:46 - azcam NOW rebooted, restarting GCS right again
  • UT 04:47 - failed, homing is OK, but getting "guidecam init failed, in an error state" - could we be connected to virtual azcam?
  • UT 04:50 - homing seems to be taking too long for AGW4, got "guidecam init failed" again
  • UT 04:51 - GCS doesn't appear connected to the AGW, I.Ilyn rebooting azcam
  • UT 04:53 - azcam restarted, restarting GCS right again
  • UT 04:54 - again "guidecam init failed"
  • UT 04:55 - rebooting azcam server computer, then restarting GCS right yet again
  • UT 04:59 - J.Hill figured out we have been talking to the virtual azcam computer the WHOLE time! Calling P.Grenz to fix this.... IT 8700
  • UT 05:11 - can't get ahold of P.Grenz, calling S.Hooper
  • UT 05:19 - J.Hill found that P.Grenz had put the wrong IP number in the config file for the original azcam server
  • UT 05:32 - S.Hooper and J.Hill working on restoring old configs
  • UT 05:32 - restarting GCS right
  • UT 05:34 - error message now points to .199, and "connection reset error...." is now gone, but in the GCS GUI still shows "guide cam init failed"
  • UT 05:35 - rebooting azcam server
  • UT 05:38 - stopping GCS right and rebooting oac
  • UT 05:43 - P.Grenz commented out the guidecamera IPs and put in the AZCAM IP
  • UT 05:46 - now stop/restart GCS left, GCS right now shows the correct AGW# and AGw unit "ok"
  • AGWs 3 and 4 don't use AZCAM server, they use agw cam machine, P.Grenz changed the config file for ALL AGws to point at virtual azcam server
    • Could this explain AGW 1 and 2 errors?
    • HK card on AGW4 not readable again
    • if new azcam server had a subtle difference in the readout of the detector and AGW1 and AGW2 were pointing at it, might have caused IT8699 issues
Resuming PEPSI-POL AGW Checkout
  • UT 05:56 - sending pointing preset - error moving motors...new problem AGW3 failed
  • UT 05:59 - recovering from SX hexapod error, then resending pointing preset
  • UT 06:00 - more errors
    • Sat Sep 3 06:00:40.208 2022 right Preset alert for PEPSIPOL mode ACQUIRE: AGw command status: AGW4: Error moving the motor(s)
      Sat Sep 3 06:00:40.225 2022 right Preset alert for PEPSIPOL mode ACQUIRE: isPosAccessible: probe cannot move to requested position (SFP) (x,y): 0.0221107, -0.0484252 mm.
      Sat Sep 3 06:00:40.243 2022 right Preset alert for PEPSIPOL mode ACQUIRE: guidestar list successfully changed but failed moving probe to position.
  • UT 06:04 - resending preset with PA = 180 to see if out of position error is fixed
    • failed again
    • Sat Sep 3 06:05:47.093 2022 right Preset alert for PEPSIPOL mode ACQUIRE: AGw command status: AGW4: Error moving the motor(s)
      Sat Sep 3 06:05:47.110 2022 right Preset alert for PEPSIPOL mode ACQUIRE: isPosAccessible: probe cannot move to requested position (SFP) (x,y): -0.0227113, 0.0524087 mm.
      Sat Sep 3 06:05:47.129 2022 right Preset alert for PEPSIPOL mode ACQUIRE: guidestar list successfully changed but failed moving probe to position.
    • I.Ilyin attempting to recover
  • UT 06:09 - need to restart oac service, then GCS left and right
  • UT 06:12 - AGW4 failed to communicate with moxa
    • [oac@oac ~]$ getdata -u 4 -n
      general state: 0x201
      general errors: 0
      THETA state: 0x100
      THETA errors: 0x30000
      R state: 0
      R errors: 0
      FOC state: 0x3
      FOC errors: 0
      FIL state: 0x3
      FIL errors: 0
      Home Status: 0x14
      UMAC Power Status: 1
      OAC Version: 5.2.1
      === 8051 housekeeping data unavailable =
    • theta stage was moving in the tent on 3L, and on the telescope several days ago (when PEPSI-Pols were mounted)
    • J.Hill testing chanigng filter, returned "motor already in motion"
    • UT 06:19 - J.Hill trying to stop AGW4 - AGW4 refuses - meaning we cannot power cycle
    • same error also appeared on theta stage AFTER E.Solheid power cycled AGW4
    • THETA errors: 0x30000 is a warning that stage is at llmit switch, not that it has crashed
  • AGW4 is offline and cannot be used - Critical IT 8701
  • UT 06:25 - reauthorizing SX only
  • UT 06:33 - J. Hill declares SH spots are acceptable illuminated

PEPSI-Pol Checkout

  • I.Ilyin will move hotspot by a pixel to make sure we don't get the same errors we saw with AGW1 and AGW2
  • PEPSI pinhole is not set correctly (fiber needs to be turned to a hard stop - needs to be done tomorrow by ISp) - thus we cannot collect usable data tonight - opening another IT 8702 critical
  • Will take a quick test exposure to make sure system is working
  • First semi-light on SX! more to be done tomorrow!

PEPSI-PFU Checkout

  • UT 06:50 - Reconfigure to PEPSI-PFU
  • UT 07:06 - tertiary problem
    • Sat Sep 3 07:05:26.154 2022 TCS authorize alert for right PEPSIPFU at bentGregorianRearFiberFeed focus: right tertiary: Following error
      Sat Sep 3 07:05:26.171 2022 TCS authorize alert for right PEPSIPFU at bentGregorianRearFiberFeed focus: AGW init failed with: AIPAGW8 wfscam: couldn't connect to AzCam server 192.168.2.38:2412. AzCam process dead or not accessible, AzCam computer frozen or connectivity issues.
      Sat Sep 3 07:05:26.190 2022 TCS authorize alert for right PEPSIPFU at bentGregorianRearFiberFeed focus: AGW selection failed for instrument: PEPSIPFU, FS: bentGregorianRearFiberFeed, config: PEPSIPFU_R.cfg
      Sat Sep 3 07:05:26.208 2022 TCS authorize alert for right PEPSIPFU at bentGregorianRearFiberFeed focus: PresetFlat [79] - [AOException] LoadShape: WARNING - adsecarb:Illegal command for state Ready (Load shape)
      Sat Sep 3 07:05:26.226 2022 TCS authorize command failed for left PEPSIPFU at bentGregorianRearFiberFeed focus and right PEPSIPFU at bentGregorianRearFiberFeed focus
    • AGW 7/8 connect errors
  • I.Ilyin rebooting azcam comptuters
  • UT 07:15 - still waiting for M3 DX - cannot move the mirror selector
  • UT 07:18 - M3 DX now in place, moving to focal station
  • Now getting AGW 7/8 failures
    • Sat Sep 3 07:08:06.098 2022 right SetAGWFilter alert for PEPSIPFU: no AGW unit selected.
      Sat Sep 3 07:08:06.120 2022 right SetAGWFilter command complete for PEPSIPFU for filter 1
      Sat Sep 3 07:08:06.173 2022 left SetAGWFilter alert for PEPSIPFU: no AGW unit selected.
      Sat Sep 3 07:08:06.185 2022 left SetAGWFilter command complete for PEPSIPFU for filter 1
    • Restarting GCS left and right, and restarting azcam
    • I.Ilyin was restarting AGW7 at same time,, now restarting GCS left again
  • UT 07:22 - AGWs recovered, ready for preset, sending pointing preset
  • UT 07:35 - still spiral searching for star
  • UT 07:51 - can't seem to focus or find anything with PFU
  • UT 07:54 - found the star on SX!
  • UT 07:58 - still spiraling for star on DX
  • UT 08:01 - DX was REALLY out of focus, but its found(ish) now
  • UT 08:10 - we are at Elevation 22degrees........
  • UT 08:11 - fleeing to higher elevation, setting both shells
  • UT 08:22 - new star aligned on SX
  • UT 08:24 - active preset resent on DX
  • UT 08:27 - 1.2"-1.3" seeing on SX
  • UT 08:29 - sending active preset to another target
  • UT 08:39 - DX having issues, star falls out of pinhole, no one seems to be touching anything
    • putting in filter to stop saturations
    • SH spots o DX not well illuminiated
  • UT 08:42 - taking 40sec test exposures
  • no degradation between SX and DX 300micron fiber
  • UT 08:49 - preset to take 100micron fiber data
  • UT 08:58 - 100 micron looks OK - although SX has MORE light?
  • PEPSI PFU Released for Science

AGW 1/2 Checkout Part Deux

  • UT 09:07 - new error on re-authorization
    • Sat Sep 3 09:03:19.935 2022 TCS authorize alert for left LUCIFER at bentGregorianFront focus: AGW selection failed for instrument: LUCIFER, FS: bentGregorianFront, config: LUCI_L.cfg
      Sat Sep 3 09:03:19.990 2022 TCS authorize alert for right LUCIFER at bentGregorianFront focus: AGW selection failed for instrument: LUCIFER, FS: bentGregorianFront, config: LUCI_R.cfg
    • on GCSGUI the error is unable to communicate with umac for both sides
    • Restarting GCS left and right
  • UT 09:12 - restarting everything (oac, gcs, etc)
  • rebooting agw1 and agw2 cam server seems to have worked
  • UT 09:25 - moving hotspot worked, it updated no errors
  • zernikes off on right
  • AGW2
    • move light up so changing Y=238 to 233
    • moving X=265 from 270
    • moving to X=260
    • moving to Y=228
    • letting Zernikes through to let it collimate a bit
  • Star is R=6.24 - WAY too bright (no wonder I was having trouble!)
  • UT 09:31 - moving to R=10.9 star
  • AGW1
    • SH spots look good, we are not changing hotspot
  • AGW2
    • SH spots look good, not checking

AGW Rotator Center

  • Stop Guiding
  • Slew to hold 0 or 360 (chose 0) on MCSGUI left front
  • [tcs@tcs1 GCS]$ ptincrement 0 CA -5
  • [oac@oac ~]$ setxy -u 1 -x 0 -y 612.5
  • [tcs@tcs1 GCS]$ gcsclient readGuideCam left -e 10000 - should fail
  • [tcs@tcs1 GCS]$ gcsclient readGuideCam left -e 90000
    • readGuideCam: acquiring full guider image ...
    • GCS side: left
    • new image (1/1) received in 90.988secs : /home/telescope/TCS/Configuration/GCS/left_guider000001.fits
      seconds needed for 1 images: 90.988
      this is 90.988 seconds per image
  • Can't find the image on the OBS machine, i took it on the TCS machine - and why did it put it in the config directory!?
  • My mistake I should not have run it on the TCS machine!
    • [lbto@obs1 ~]$ gcsclient readGuideCam left -e 90000
      readGuideCam: acquiring full guider image ...
      GCS side: left
      new image (1/1) received in 90.9535secs : /lbt/data/share/tcs/lbto.20220903/left_guider000002.fits
      seconds needed for 1 images: 90.9535
      this is 90.9535 seconds per image
  • X = 115, Y= 80
  • correct for binning x2 --> X=230, Y=160
  • correct for DETSEC (x+15, y+39) --> X=245, Y=199
    • [lbto@obs1 ~]$ gcsclient getHotspot leftHotspot is (x,y): 249.00, 196.00
  • X offset +4pix, Y offset -3 pix
  • Since this is so close re-measuring it again
  • UT 09:57 - resending preset
  • UT 09:59 - Closing for humidity (94.8%)
  • reverting from 2.54" to 0.54 in /home/telescope/TCS/Configuration/GCS/LUCI_L.cfg and LUCI_R.cfg (copied the backups made before opening to the original file)
  • NEED TO VERIFY THE +4,-3 pix offset by taking this data again - Rollover task to next night or later if we open
UT 11:45 - calling it a night - at 99.9% humidity and clouds - mid-point between astro and nautical twilight

-- BarryRothberg - 29 Aug 2022
Topic revision: r8 - 03 Sep 2022, BarryRothberg
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback