Prev Next

SW&IT Meeting Minutes 2022-05-02

Attendees: Brandon, David, Glenn, Matthieu, Paul, Stephen, Xianyu, Yang

Outlook

• Weekly schedules [Telescope/Schedule/Support]
• Team calendar
• IssueTrak summary
• open PR
• open issues

Labor Reporting

Report auto-generated every Monday at 6:30am, Do not edit this table, Update your entries in smartsheets.

20220418

LR20220418 - Bec, Matthieu	72.0
LR20220418 - Mechtley, Brandon	80.0
LR20220418 - Eychaner, Glenn	76.0
LR20220418 - Edmeades, David	40.0
LR20220418 - Grenz, Paul	38.0
LR20220418 - Hooper, Stephen	0.0
LR20220418 - Zhang, Yang	80.0
LR20220418 - Zhang, Xianyu	83.0

News from management

Previous Action Items

Owner	Action	Due Date	Completion Date

Open topics

• Re: OBS Stations (JP emails and follow up)
• Career Conversations, due June 30.
• TCS/OSS Argos cal source user manual and notification
• TCS waffleAO rpc vs. ice - dead code and obsolescence
• SSD plan https://wiki.lbto.org/Software/SummerShudown2022

Hi All, 

We were discussing the status of the observing stations on both the mountain and remote observing stations at today's science operations meeting.  A few points have risen.
With C-19 mode we do not make it up to the summit for operations often or to the Tucson remote room very often, but we realize it is still critical that these stations remain operational and up to date.  We are having our partner observers coming now, as well as having had a number of instances that have required us to observe in person (internet outages, testing, etc).  
We would like to consider optimizing the mountain obs station configurations and ensuring it is operationally ready for us and our users.  
obs2-4 are 32 bit machines and do not have the latest and greatest software.  LUCI rtd has only been kept up to date on 64bit, RB_science is much better on 64 Bit, etc.
 obs2-3 are the only machines that reliably have monitors connected in the summit observing room (besides obs1 of course). 
obs5-6 did not have monitors connected recently although those are ideal for regular operations.  I had connected monitors during previous summit trips, but these large monitors are often disconnected to be used for other uses.  Would be ideal to have a 64 bit computer for observer operations with monitors connected.  Perhaps obs2-3 (32bit) without monitors just accessible remotely, obs5-6 (64bit)  with monitors?  
Telescope operators use obs2 as a backup for obs1 so would a 64bit be a better backup anyways to match obs1?
obs5 has reported slow downs/freezing up. Memory issue? 
obs2-4 swap (/scratch) space is used so it still requires the 32bit machines.  disk space and memory for 64 bit machine options?
With the instrument teams that are coming, partner observers, etc, having the 64 bit machines for operations would be great.  Please let us know your thoughts and what we might be able to do.  Thanks!!!

• 32bits, plans to remove in 507 never concluded. obs2-4, could we just keep one 32bits. setup obs2 (next to obs1/OSA) as 64bits
  • we're stuck with 32bits (or is it centos7 = gnome2) for x2go
• osa vs obs workstation. obs1=tops1, create a new tops2 64bits, replace the obs2 (physically) both have 6x screens
• obs3-4 use 2x 4K screens, keep one (at least) 32bits, maybe headless?
• obs5-6 as they are
• do we really need all those machines
  • TO(OSA=Steve, etc) obs1 + backup = tops1/2
  • Observer/ISA(Olga, etc.) obs2-4 + obs5-6 + robs
  • s/w vs h/w issues?
• do we keep robs/VM? use obs5-6 instead (with hw)
• shark-NIR (when team is at mountain) will remote to their servers from one of our obs station
• summary
  • obs1 stays, possibly rename tops1 (or whatever)
  • obs2-3 goes away
    • obs2 space used for tops2
    • personal user accounts (staff at lbto) should go away, user: lbto or whatever and migrate their home into ~lbto/
    • any sw from AO going into obs3? we'll figure it out if needed
  • obs4 stays, 32bits
  • obs5-6 get 2x 4K, reliably connected screens
  • robs goes away, or call it swgdev11 (i.e. not operation, but development or such)
  • no real plan to support 32bits app anymore

Team reports

MB

• Last week
  • OC, Management, AOSWSYS
  • DMS v0.1-49
  • aorev_git lbti-dx
  • LBTI nimble data
• This week
  • ATT, GMT M1 report, Luci-gui WG
  • SOUL comm plan EPinna
  • LBTI nimble ctd.
  • ARGOS github synch (Lorenzo?)

YZ

• Previous:
  • Updated TCS releasing document (RELEASING.md in the master branch).
  • Worked with Martina on integrating the MODS WCS and new image service into archive workflow.
  • Study LBC code compilation and changes.
  • TMS software update and test on Friday (with Heejoo and Trenton).
• Next:
  • Evaluate MODS WCS service with Olga.
  • Investigate adding new image support into DMS Python APIs.
  • Continue LBC code study.
  • TMS software work (and wiki work) and possibly new tests.
  • Some misc issuetraks.

DE

• Last week:
  • IT ticket re: null info in critical ticket notification. They found a bug, but didn't elaborate. Subsequent notifications have been good.
  • Backup server rescue work, looks like some firmware got updated but some was not, which caused the system to be very unhappy.
  • LBTI Google Workspace ingestion
  • User stuff, VPN accounts
  • New M1 Mac Mini set up and users confirm access
• This week:
  • Finish backup server recovery
  • LBTI transition work

GE

• Last week:
  • Tested vncserver (TigerVNC) and X2Go on isola-luci. See my notes in the LUCI GUI Plan of Intent.
  • More work on IT 8619 (problem centering slit), which turns out to be far more pervasive and insidious than I first thought.
  • Searched IssueTrak for old LUCI ITs, and bookmarked them for future reference.
  • Created an inventory of observer computers so that I could intelligently discuss JP's emails.
  • Terrible allergies early in the week.
• This week:
  • Close IT 8619 (problem centering slit).
  • Create a test case in my simulated TCS for IT 8573.
  • Work on Observer Scheduler GUI (IT 8571, 8529).
  • I will be taking half days off work Monday May 2 (afternoon) and Tuesday May 3 (morning).

SH

• SCADA system with Dan R. (network, etc.)
• Solidworker
• Tucson network issues
• Windows server replacement strategy
• Room 505
• Mountain network for chiller
• Synology rsync

next:

• Fedora 35 docker and system's RPMs
• Core switches
• Zabbix

PSG

• Last week
  • AZCAM code inspection.
  • SHARK-NIR testing.
  • Fixing SHARK-NIR bug.
• This week
  • Finish SHARK-NIR bug fix.
  • Talk to users of new ARGOS swing arm software.
  • AZCAM integration?

BM

• Last week
  • LBTI daytime and SOUL-LUCI DEC testing Monday for IT 8602 and 8556
  • LBTI DX meeting - Fabio will work in separate branches, PR after comissioning done. He doesn't suspect many, if any changes.
  • check_elevation / ao-supervisor release discussion
  • Worked on simulating adsecarbitrator for gain ramp testing.
• This week
  • set_z0 test (Doug making test plan)
  • IT 8628: New issue from Guido (connection to AOS lost) Maybe similar to issue reported by Jenny/Juan Carlos 2022-04-14
  • ao-supervisor 109 - dropped rtdb messages when loading disturbances
  • Testing gain ramp in VM
  • Lower priority grab bag:
    • WfsArbitrator: AcquireRefAo, set CCD viewer overlay message only after baysides/filterwheel have moved
    • AbstractFsm thread safety bug fix
    • Disable FLAOGUI "interrupt" button for uninterruptible commands.