Prev Next

SW&IT Meeting Minutes 2020-04-06

Attendees: Carl, David, Igor, LeRoy, Matthieu, Stephen, Xianyu, Yang


Labor Reporting

Report auto-generated every Monday at 6:30am, Do not edit this table, Update your entries in smartsheets.

LR20200309 - Bec, Matthieu 80.0
LR20200309 - Holmberg, Carl 41.0
LR20200309 - Edmeades, David 80.0
LR20200309 - Sola, Igor 72.0
LR20200309 - Hooper, Stephen 80.0
LR20200309 - Zhang, Yang 75.0
LR20200309 - Zhang, Xianyu 98.0

LR20200323 - Bec, Matthieu 80.0
LR20200323 - Holmberg, Carl 80.0
LR20200323 - Edmeades, David 72.0
LR20200323 - Sola, Igor 80.0
LR20200323 - Hooper, Stephen 80.0
LR20200323 - Zhang, Yang 77.0
LR20200323 - Zhang, Xianyu 80.0

News from management

  • Closing of ITs guidelines

Outcome of Meeting

  • The Admin for the Group will review the closure to verify it meets the content requirements as follows:
    • What was the issue?
    • How was it detected / identified (symptoms that could help identify the same or similar problem)?
    • What was the root cause(s) of the issue? (how / why did it occur)?
    • What was the corrective action/ resolution taken?
    • What was the corrective action/ resolution taken?
    • How were the open questions/ recommendations resolved or addressed?
    • What follow-up, additional actions were taken (respective procedures updated, configuration changes, like items inspected/ updated as able, other related ITs, specs changed, etc)?
    • What is the expected improvement from the solution (reduce the failure rate, eliminate the problem, reduce risk, improve performance – including quantitative data when possible).
    • How can this be prevented from occurring in the future?
    • Are there other applicable documents that can be linked (procedures, wiki page, google drive)?
    • Are there other applicable documents that can be linked (procedures, wiki page, google drive)?
    • Does the originator of the IT approve of the solution?
  • Managers will convey the closure content requirements (above) to their staff – so that everyone gets the same message.
  • Managers will convey to their staff that if they have an instrument IT they want closed, they need to submit a “request to close” to Robert for discussion in his weekly meeting (before they close the IT).
  • Bucki will have a training presentation on “troubleshooting tools & methods” in the near future for engineering - others can be included if desired

Previous Action Items

Owner Action Due Date Completion Date

Open topics

  • tcs test for OSAs - use for testing, training, etc. MB to check if it's beeing used, or has any value for OSA but maybe also SO's.
  • zoombombing, zoom numbers in private wiki - setting to password or use @lbto.orf - for now wait and see.
  • LBTObs slack
  • AO telemetry - SH follow with Pat, ao-analysis in Tucson
  • CR temperatures
  • Power shutdown discussion - need to be ready by Apr 17

DE get back to Gab/Chad for details and share with IT.

Subject: Updated: SO power outage impact on observatory sites

There will be three outages.

The first is tentatively scheduled for April 17; we are working with the building contractor to firm that up in the next few days. This outage will involve replacing a electrical panel that supplies rooms in Bldg 65 (old wing), and 65B (North wing, 1st & 2nd floors), and will nominally last 4-5 hours during the day. None of the affected rooms identified by the contractor collide with your equipment or with the main Steward network hardware. However, the panel being replaced is poorly labeled and so we expect that some unidentified circuits will be affected. To avoid damage to computer systems, many Steward servers will be powered off the afternoon before the outage and restarted the afternoon following the outage (~24 h down). We suggest that you do the same for your systems.

Please let myself, Carlos, and Mark (at a minimum) know if you will choose to power off systems, and if you will need personnel on site to do so or if they can be shutdown remotely. It is possible, but not certain, that the relay links to Bigelow/Lemmon/Hopkins sites will go down. Our hope is that mountain sites are not so dependent on internet access that they stop functioning over a single night outage, but from some of your earlier responses this may be a concern. If you feel short term loss of these links is unacceptable, please let me know ASAP with justification.

The 2nd and 3rd events will both be building wide, and both last for ~16 hours. FM will be replacing the main electrical supply switches and breakers to the building. The 2nd event will likely be in May. The 3rd event will likely be in July. I'll be following up about those as we firm up the dates.

Best regards,


  • Talk to Pat about scheduling a task for LeRoy /other to photograph luci.luci internals: motherboard, motherboard model number, capture cards (for bus type ID)

Team reports


  • Last week
    • dms allsky - follow flushing with IT
  • This week
    • cobbler / kickstart
    • ims f30
    • ROWG#5
    • student ad
    • dms weather loss



  • AO-Cravat meeting with Al and Matthieu (reached some sort of a plan, I will be working on C/Python Strehl code and general automated report generation, while Al will initially look into the E-Lab code).
  • IT#8021, LBC rotator trailing issues. Analyzed some LBC and TCS code. In discussion with Olga. We've suspected some timing related issues. May schedule a meeting with Olga for an in-depth analysis. Currently working on analyzing LBC log files myself.
  • IT#7854, the LBC Web GUI sometimes loses track of the true internal state of LBC. Looking at LBC PHP frontend and C backend interaction.
  • 64-bit MODS Python program work.
  • Registered a free Wolfram account for 15-day Mathematica trial. (Tried Mathematica online, but unable to upload/import local file, what a bummer!)
  • Did some safety training.


  • Last week
    • AHS, created Jenkins INDI-core project, began moving C++ code from C7/g++ 4.8 to F30/g++ 9.2.
    • Safety training.
  • This week
    • ARGOS, fix build on Jenkins, initial work to isolate GUI from back end.
    • AHS, finish repackaging INDI library and commit.
    • Distance Learning
      • Per the CS advisors office, the UA CS school doesn't offer classes outside of the regular schedule, and the only relevant C++ class isn't going to be offered this year. Baring other suggestions, I'll give the Udemy class a go.


  • Last week
    • PB SW plan document
    • PB lab test HW plan document
    • SOUL SW requirements for power backplane SW
    • AO telemetry requirements
    • Meetings
  • This week
    • PB action items
    • PB SOUL meeting
    • AO telemetry document
    • Actuators status analysis


  • Broke phone/new phone
  • Rack and re-wire ovirt cluster, changes to network from UITS
    • KVM not working very well
    • PDUs not capable of switching ports on and off
    • hv02 has bad memory DIMM B2 (firmware upgrade)
  • Updated SSL certificates on
  • meetings


  • upgrade vm1.tucson
  • allsky
  • ovirt (plug in install media)
  • Tucson VPN routes
  • MGIO cameras


Last week:
  • PIT
    • Merged italian and general PIT backend CGIs in to just one that work for both of them.
    • Since we are not offering AO/ARGOS for 2020B, and we are well into proposal season submission, we work on a way to fail a proposal and send an informative message to the user from the backend.
    • Working on a couple of Italian PIT issues. Most notably a glitch that seems to be triggered only in MacOS, on the latest OS.
    • Working on a google drive API V3 python upload and permission setting tool that allows me to automatically, upload and share links on builds. The python package that works around the google api v3 restful api sucks. The google API although powerful is cumbersome when compared to a UNiX file system.

  • LUCI
    • CalUnit Web I/O will be sent to my home so I can configure it.

This week:
  • PIT
    • Will keep working on some of the PIT issues discussed above. Mostly on a GUI issue. A dialog box with an unexpected behavior.
    • The Italians are testing the release and there might be a couple of issues to deal with.
    • Finish the google drive upload/permission setting tool
    • Keep working on the PIT admin tool backend.
Topic revision: r22 - 10 Apr 2020, MatthieuBec
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback