UT 02 Sep 2021 - Night #1 of Restart



Night Staff:

OSA: S. Allanson

ISp: C. Bolyard

ISA: B. Rothberg, R.T. Gatto (Tucson secret Outpost)

Mtn Manager: P. Hartley

Sunset: 18:49

Twlight Ends: 20:09

Twilght Starts: 04:30

Sunrise: 05:50

Night Log:

Night called at Daily Planning Meeting

UT 02:15 - change of plans - bladders being removed on roof, powering up LBCs on ROBS - issues with previous session left running and unable to type in terminals

UT 02:21 - LBCs started up, taking test biases -biases OK

UT 02:43 - still removing and stowing the tarps, brought up MODS2 and GCS GUIs in case we work on MODS AGW alignments

UT 03:00 - bladders and tarps removed

UT 03:10 - preparing to open - will attempt LBC collimation and co-point on Dumbbell - then play it by ear

UT 03:20 - humidity close to limit (pre-opening) - Track mode test LBC preset successful.

UT 03:27 - TMS issues - S.Allanson checking hardware - TMS will be offline tonight

UT 03:36 - humidity still at 87% - opening enclosure

UT 03:42 - at 92% humidity - it hit 90% just as we were about to open...

UT 04:09 - TMS rebooted - running closed dome checks

UT 04:25 - will run MODS2 focus scripts

UT 04:33 - running all three facility instruments (or just LBCs, MODS-2 and LUCI GUIs really slows down ROBS - load average went up to 6+)

UT 04:37 - tried to bring up LUCIs - instrument manager busy - can't connect - will investigate - tried to stop and start IM on LUCI computer - won't restart

LUCI Computer Woes

UT 04:42 - can readout detector with GEIRS1 - can't access anything via LMC - luci webpage shows nothing reporting on presseure.temps, etc since 2021-09-01 23:49:03.3180

-unable to bring up the luci computer on X2GO

when ssh in and bing up LMC get errors:

Java HotSpot (TM) 64-Bit Server VM warning: Insufficient space for shared memory file:
18984
Try using the -Djava.io.tmpdir= option to select an alternate temp location

UT 04:51 - D. Thompson just joined polycom with same problems, also tried to restart IMGUI. Manually trying to restart the software (LMC reports everything off now).

UT 04:58 - /dev/md126 which maps to "/" says out of space (error message out of space also on luci.luci.lbto.org webpage) -somehow all 129GB got used up and perhaps this is causing the software issues?

UT 05:14 - D. Thompson shut down luci, S. Allanson rebooted luci computer

UT 06:01 - IT 8454 - able to operate with 73M free on "/" - barely enough - but if anything eats up more space LUCI-1 is unusable.

UT 06:12 - decided to not pursue G150 flexure and wavelength cals tonight in case we lose the 73M of free space and the software becomes unresponsive.

MODS2 Focus Scripts

UT 04:38 - IMCS lock on M2R failed on 2nd image - retried and it locked on

UT 05:31 - IMCS lock failure on M2R 0019 - retried locked on

UT 05:35 - IMCS lock failure on M2R 0021 - retry worked

UT 05:38 - IMCS lock failure on M2R 0022 - retry worked

-seems a lot of failures all of a sudden

UT 06:07 - IMCS lock failure on M2R 0034 - aborted (retry didn't work) - seemed to be a race condition "** ERROR: instconfig instrument configuration already in progress - command disallowed"

UT 06:08 - restarting this again

UT 06:27- had to run red fitsflush for the last few exposures to pop out
  • moving to zenith, parking MODS2 rotator
  • dualgr2.pro
  • redgr2.pro
    • M2R 0012-0022
  • dualim2.pro
  • redim2.pro
    • M2R 0034-0044
  • bluim2.pro
    • M2B 0023-0033
  • blugr2.pro
    • M2B 0034-0044

UT 06:57 - humidity has dropped to 78%, SMT is at 95% - going to complete the MODS2 focus sequence then evaluate. Would do LBC first then switch to MODS AGWs after depending on seeing/collimation struggles, etc.

UT 07:08 - humidity hit 90% and climbing

UT 07:23 - humidity hit 79% but SMT 96% at SMT

-yoyo humidity is a bit troubling

UT 07:58 - now it looks like clouds are rolling in...

UT 08:02 - humidity dropped again but is 20 percentage points higher at the SMT as the clouds roll in, can only see stars in the west now

UT 09:01 - clouds have thinned out, humidity 83% - will attempt to open

-humidity is doing the limit dance again

LBC On-Sky Checks

UT 09:08 - enclosure open sending preset to HCG 98 for collimation

-mirror temps are about delta 3C from ambient

UT 09:12 - dohybrid started

-fpia on blue struggling with z22

-adding +1000 z4 on blue

-adding +1000 z4 on blue

-making z4 +5000 on blue (in total)

-red coma seems to be running away, last correction was -1891

UT 09:26 - shift+S out of dohybrid, clearing active optics, will add +5000 z4 on both - seems like the WFS part of dohybrid starting in a bad place

UT 09:27 - running dohybrid again

-WRS backed out our iniital offset

UT 09:32 - backed out of dohybrid - will run dofopia only

UT 09:33- clearing active optics, running dofpia only

DIMM 1.2"

-adding +z7 and +z8 manually

-adding +1000 z11 on red

UT 09:43 - DIMM 1.4"

-backed out dofpia

Red is running away on z4, clearing active optics on red

manually putting in +2000 z7

UT 09:44 - running dofpia

-adding z11

LBC-Red added tons of coma

-clearing out red active optics

+z7, +z4 and -z11

UT 09:54 - running dofpia

-clearning adding +1000 z11

Red looks good, but now z11 runaway on blue

backing out - we think we got Red ok, will add +1000 z11 for blue as well

Note: Given conditions and instrument status - only other items to be done are MODS AGW hotspot - other tasks require better seeing for LBCs or should be MODS-bino

UT 10:04 -looking good thanks to Steve (my only contribution was suggesting +z11) - but now humidity is rising

"....Blue leader stay on target....."

...was looking good....then Blue z4 ran away...

Would be nice if we had an "Accept Current Collimation" Option - often it gets close then runs away

UT 10:13 - ran away...both sides

UT 10:15 - stopping dofpia

UT 10:16 - one last attempt with some z22 for fun

-S. Allanson disabling any additional changes to z4,z5,z6,z7,z8,z11,z22 (gains are set to 0 in PSFGUI right)

stars seem to be fading.....

...blue running away again

-set gains to 0 to keep acceptable collimation values

UT 10:29 - sending co-pointing scripts

-setting gains to 1 to allow tek chip to do its thing



COPOINTING: B=102949 R=102948
Pointing updates: delta_IE = -10.46", delta_CA = 11.39"
Mirror updates: dX(mm) dY(mm) dRX(") dRY(")
SX: 1.00 0.40 8.61 -21.50
DX: -0.96 0.85 18.13 20.69
-------------------------------------------------------------------------
UT 10:32 - going to HCG 98 - cycle through Usgr, zir and 4pt dither

co-point images fwhm blue: 1", red 0.88"

UT 10:46 - closing up, we hit 95% humidity. Got 4 images each Blue and Red, managed to do 1 dither offset in first set of filters.

-allsky can't really see anything now except hazy Moon

-last exposure about 1.2" in Uspec, 1" in z-band

LBC Cals

UT 10:58 - taking 25 biases each LBC

- first blue bias lots of electronic noise

-some horizontal noise on red first few images

UT 11:05 - lost connection to ROBS (my internet is fine)

-logged back in and ROBS is down to a crawl - load average 1.5, logged into OBS2 - everything snappy

UT 11:12 - lost connection to ROBS again

UT 11:15 - putting MODS2 to sleep

UT 11:41 - reconfigured for MODS - AGW5 weird error, focus says -99999, TCS seems to indicate its not powered on - services not running on MODS - AGW services and all services stopped on MODS-1

-could we have used AGW5 if everything else was powered off?

UT 11:54 - powering off LBCs

UT 11:58 - done for tonight


Proposed Plan of Action:

  • Telescope Wake-up ~0.5hr
    • test presets (sidereal and NS)
  • LBC Wake-up - contingent on release of LBC-Blue
    • Collimation - DoHybrid, then DOFPIA (~1hr)
    • Superfocus Sequence, PA=0, 180 (1hr)
    • Rotator Center Checks (pairs of 0,180; 90, 270) ~0.5hr
    • Test Science OB ~ 1hr
      • Sidereal
      • Non-Sidereal
    • Photometric Fields and Spectrophotometric Stars (to establish ZPs) ~ 1-1.5hrs
      • Requires multiple airmass visits (clear conditions, any seeing)
      • Cycle through all filters
      • This is now a high priority because we need to define ZPs
    • Twilight Flats
  • MODS -
    • Basic Checkout
      • Verify collimation
      • Verify AGW 5/6 Hotspots are centered
      • Observe a standard star in all modes (30-45 min for 3 images). If clear, obtain the full set of 3 images per mode; if not, make a note in the object name and obtain only 1 image per mode.
      • Dice-5 imaging offsets
    • Test Science OB
      • Sidereal
      • Non-Sidereal
  • Contingencies in case LBCs and MODS/MODS AGWs are not available
    • PEPSI
      • PFU alignment
      • PEPSI pointing model (poor weather)
    • LUCI-1 restart activities
      • Check Rotator Center
      • AGW1 Alignment (LUCI-1) (1 night) Overview of AGW procedure
      • Imaging check
        • dither in field (0.5hr) -
        • dither out of field (0.5hr) - M52 K and some J-band

Current Status

  • LBCs
    • LBC-Blue - available
    • LBC-Red - available
  • MODS offline
    • MODS-1 - M1RC offline - hardware issue communicating with HEB (not a fiber issue)
    • MODS-2 - OK (IMCS check TBD), ready for daytime checkout
    • AGW5/6 - available
  • LUCI-1 available
    • AGW-1 available
  • LUCI-2 offline
    • AGW-2 off telescope - earliest reinstall Thurs. Sep 02 local time
  • PEPSI
    • AGW-7/8 available
    • Software environmental system issues

Closed Dome Activities

  • Test Preset before opening (any config)
  • ARGOS Cal-Unit test for M3 alignment and pupil wobble checkout - postponed
  • LUCI-1 Only Calibrations
    • Flexure data for G150
      • /home/dthomspons/Luci/FlexureCompensation
      • README - Calib15 - 2hrs
      • Ks filter, Argon lamp, optical sieve mask
      • take dark frame
      • Manual Control MCSGUI for elevations
    • Wavelength data for G150
      • G150 Ks, Argon, take image through slit, then put in grating, line should show up where the slit is located (within a few pixels of the image of the slit)
      • Check a few lines (2 or 3)
  • MODS-2 Only Calibrations
    • Imaging Flats
      • Dual
      • Red
      • Blue
    • Spectroscopy (flats, slitless flats, arcs)
      • Dual
      • Red
      • Blue
    • Bias frames
      • 2x1 and 2x2 bias
      • 1x1 Binned Biases
    • Focus Scripts:
      • on robs: /home/lbto/modseng/modsScripts/Testing/MODSfocus
      • on obs#: /home/modseng/modsScripts/Testing/MODSfocus
      • The highest priority ones are the following:
        • Dual Grating: dualgr#.pro
        • Red Grating: redgr#.pro
        • Dual Imaging: dualim#.pro
        • Red Imaging: redim#.pro



-- BarryRothberg - 27 Aug 2021
Topic revision: r9 - 02 Sep 2021, BarryRothberg
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback