SW&IT Meeting Minutes 2018-05-07
Attendees: Andrew, David, Doug, Igor, Joe, Kellee, Matthieu, Petr, Stephen, Xianyu, Yang
Previous Action Items
- Tech-talk #2 TCS (PK)
- MODS handover update (MB)
News from management
- MW looking into ccd controllers upgrade (e2v, archon) discuss at SPIE - potential common architecture for all LBC and GWS
- SSD planning moving forward
- SHARK-VIS/NIR monthly meeting, lead to attend
- mirror cover, late? needs update
- consolidated support calendar
- Added tests for satellite closure and status gui
- Added parts of IT section to handover document
- Added logging information for LAT
- Update network diagram, hostnames, state machine diagrams, and host configuration instructions
- Determine spares strategy
- Last week
- Testing/breaking/learning Veeam with good final results. Recommended going ahead with purchase
- This week
- Focusing IssueTrak duplicate email problem
- Big monitor install for Laura
- Jim webcam
- Laura PC monitoring: all has been well (so far) since last driver update
- Still working current view for the LUCI scheduler, early estimate was too optimistic. Hope to have it done and ready for testing by the end of the week.
- Discussed with Dave T. #6234: focus offsets for camera and gratings
- Added some documentation regarding PIT and LUCI simulation environment.
- Engineering night last Friday
- It was very interesting to see, in particular how the telescope is operated and how software we can help or hinder at operation at times.
- Things that jump to mind from that night were:
- LUCI software
- If one side is broken for a few minutes, it was unclear to the users how to continue on one side only.
- (What the software can and can\x92t do even for a experience user is not that clear)
- NonSiderial and absorbs
- Addition of absorbs by the LUCI Real Time Display. What exactly are we adding and is it clearly displayed in the LUCI GUI?
- Scripting software
- Generating a script on the OT and then editing it manually. Although in this case might have been justified, it was striking. Certainly some things could have been improved if the user had a more commanding knowledge of the OT.
- For instance the ability to change exposure times.
- Some confusion on multislit absorb offsets when absorb observation and science are paired.
- AOs complex initialization
- Watching all the bits and turns makes me wonder if a decision making software wouldn\x92t be more helpful than documentation.
- Some steps need to be carried serially while some can be done in parallel.
- What if the step fails? that in itself might carry a bunch of new steps or back out of previous steps. Even for a experience operator navigate all these and expect then to read documentation while a crisis is happening is daunting.
- E.g. Homing of the Front Wave Sense three stage, where one stage gets stuck, and won\x92t home if not aid by gravity, (telescope at zenith) and even so it has to be manually offset by a negative count before homing, etc.
- I can imagine that some expert system with decision tree/scheduling software can be more helpful than a documents and links to documents. Some software that helps weave the different tasks and it\x92s dependencies. As a first approach it doesn\x92t have to do anything other than manage the information, later on, it could help making things more automatic.
- Get luci-sim2 simulation in order but Petr or other people in the group to use.
- Finalize current view for LUCI Scheduler
- Continued documentation of Planner requirements. This will be a living document.
- Implemented single preset model
- Filter user choices (should have same instrument and target)
- Validate acquisition sequence logic for single preset LUCI
- OAC HK upgrade
- handle errors from EPICS writes
- support empty values in threshold strings
- made it the default version in Tucson environment
- Released GCS patch for logging issue (IT7206) and centroiding crash (IT7045)
- TCS training - review and discussion
- management OC, WPC
- support calendars
- Ketiv'Vault follow up
- PBP upgrade technical meeting #2
- Argos post run
- Loggin, packaging YZ
- Labor record keeping
- monthly report
- review Argos handover
- SSD wrap up
- VCan wrap up
- discuss ETC with Mark and Al
- 'activate' first migrations
- responsibilities and assignments draft
- work on 7218 - hopefully last fix for co-pointing (separating trajectory calculation from trajectory usage)
- TCSTroubleshooting presentation, please comment on it if something is missing
- finish 7218 (test, mountain)
- work on 7190 (LSS anomaly)
- vacations 11th (Friday, probably not full) - 28th (Monday), I can be reached on email or +420 737 500 268 (CEST preffered, UT +2, e.g. MST + 9)
- upgraded zabbix to newer version, resolved agent death notification issues with mounted filesystems -- less e-mail from zabbix.
- move VMs to 10.130.142.0 subnet if Paul Hart has resolved IDL licensing issues (adsec, wfs).
- grafana for prettier dashboard -- how does it work?
- weather station work with Leroy.
- kickstart for pacemaker controlled bind servers, glusterfs distributed zone files?
- dockerized TCS -- refresh on work that I left over from last year?
- Technical discussion with Roberto (MG)
- Reviewed and changed the fix for IT7240
- Wrote another patch (move most shell safety from AdSecArbitrator to adamhousekeeper) for IT7230
- Tested the two patches for IT7230 (both are sucessful)
- Fond the WFS arbitrator does not use the ICE calls to AdSecArbitrator
- Solved the MOXA issues, MOXA works with simulator
- Implement all shell safety in AdamhouseKeeper
- Make the AdSecArbitrator and AdamhouseKeeper work together on the shell safety
- Change the AdSec piston current and square wave directly from AdamhouseKeeper if nessary
- Write the fail recovery table for the powerbackplane
- Another Technical discussion with Roberto (MG)
- Integrate OVMS+ to AdSec meeting
- Working on the "mergelogs" tool build (rpm build + Jenkins integration).
- After some investigation, the "syslog" facility and architecture is considered suitable for local logging use:
- Refactor the AO logging code to use the "syslog" facility.
- Study a proper "syslog" server configuration that works with local demands.
- Investigate potential performance implications using "syslog" architecture (e.g., will high frequency logging be a performance bottleneck).
- Investigate whether small backend tools needed to further process the log outputs (e.g., categorization, sorting, and searching), or will it be better off to use some existing log management tools (preferably open sourced).
- Study the components and architecture of the AOSupervisor code in general.