MODS Software Handover

The ideal handover scenario includes planning, presentation, training, a build/run exercise and a support agreement with the developer.
The non-ideal handover scenario includes a lot of reverse-engineering. MODS should be close to an ideal handover - we have time to plan and be part of the commissioning and the MODS team will be around for awhile to support.

Handover Materials

• Rick Pogge and others contact info
• MODS Software Handover document (March-2018, version b) - also in the Vault
• MODSSoftwareDescriptionDocument - RP's suggestion for the overall document
• MODS Network Infrastructure Requirements v1.1 (April-2014)
• Instrumentation/SciInstrMODS - our wiki page describing operations, mostly written by Rick Pogge
• Instrumentation/MODSStartUp - how LBTO support personnel (not observers) bring up the instrument
• MODS Quick Reference - Olga's notes on our wiki
• MODS Troubleshooting - on our wiki, for observers
• more MODS Troubleshooting - on our wiki, for instrument techs, OSAs
• Instrument Support Scripts
• post-Shutdown Startup Notes - under construction by Olga
• MODS Observer's Cookbook
• http://www.astronomy.ohio-state.edu/MODS/ - MODS page at Ohio State
• http://www.astronomy.ohio-state.edu/MODS/Docs/index.html - Online MODS document archive at Ohio State
• Kellee's Software Notes
• MODS for the LBT, SPIE 2010 (also on the system here)
• MODS for the LBT SPIE 2006
• Interface Definition between off-axis AGW units and the Guiding Control System of the LBT (481s222)
• http://www.astronomy.ohio-state.edu/~rgonzale/modsMechSoft.html - Ray's old software page - this is probably not where the most up to date info is (see above link to rgonzale). But, there's a lot of info here. It may be all historical though.
• http://www.astronomy.ohio-state.edu/~rgonzale/doc/html/index.html - Ray's doxygen main page for Mechanism Software and MicroLYNX Documentation
• MODS CAN repository (no software stuff) Category: 600 Telescope Auxiliaries
  Note: a few have been added since I created this list - I should update, or delete all the non-software links
  • MODS1 Instrument Commissioning Report (updated May-2013)
  • MODS AGW Stage Progress Report (26-May-2004)
  • MODS AGW Stage Functional Description and Observing Use Cases (15-Nov-2005)
  • MODS AGW Stage Powerpoint (16-Aug-2009)
  • MODS1 Acquisition, Guide and Wavefront Sensing (AGw) Unit Commissioning Report (26-June-2011)
  • Mirror Baffle - Cell Fit Close Iso 1
  • Mirror Baffle - Cell Fit Med Iso 1
  • Mirror Baffle - Cell Fit View 1
  • MODS Instrument & Cart Transload Plan - PDF
  • MODS Instrument & Cart Transload Plan - PPT
  • MODS Instrument Dismount Preparations (25-June-2012)
  • Progress Review Report April 22-23, 2004
  • MODS Rigging Plan for mounting instrument on telescope - PDF
  • MODS Rigging Plan for mounting instrument on telescope - PPT

• Commissioning/AzCamServer - notes about azcamserver

• docs are also in SVN - probably they should be deleted from SVN and put in the CAN
  • top-level doc/MODSAGW_Stage_Use_Cases_v1.3.pdf is CAN doc 603o703
  • MODSAGW_oacserver.pdf from January-2008 is not in the CAN, and is not complete
  • agwSim and agwServers both have these docs in SVN: AGWsim.pdf - AGWSim v0.2 MODS AGW Stage Simulator Agent (16-Feb-2006)
    IMPv2.pdf ICIMACS Messaging Protocol Version 2 (21-April-2004)
    MODSAGWSpec.pdf - AGW Specifications for MODS (28-April-2004)
    MODSAGWStage.pdf - MODS AGW Stage Progress Report (26-May-2004), CAN doc 603o701a

LBTO engineer should spend time going through all the existing documentation (that is current) available before the walkthrough.

Presentation

The current responsible engineer should provide a presentation to the LBTO responsible engineer that includes:

• Software Overview
  • Requirements
  • Use Cases
  • Reference Documentation
  • Development Metrics (LOC, overall complexity, etc.)
• Design Concepts
  • Software Architecture and Strategy
  • 3rd party products and interfaces
  • Communications Strategy
  • Languages/Scripts
  • User Interface (GUIs)
  • Performance / Design Contraints
• Build/Run-time Concepts
  • Source Control
  • Makefiles, scripts
  • Configuration files
  • Start/Stop/Monitoring tools
  • Logging / Analysis concepts

Would be nice to walk through this info, rather than just have it presented.

Training

• Documents Review / Feedback / Questions
• Build Exercise
  • Source Management / Extraction
    https://svn.lbto.org/repos/mods/trunk - svn repository for the library used by GCS for MODS
    AGWSim ?
  • Build
    Ray: The AGW also works with MODS, and when we make changes, it means the changes have to be done in two places on local mods and on /trunk/mods. 'mods' builds all of mods system including the agw - GCS on a Centos 5.x computer. These libraries are built in mods computers under /home2/mods with 'make' and all the libraries are kept under /home2/mods/ulibs including the agw-GCS libraries.
    My understanding is that you build the agw-GCS code under Centos 6.x which might not be compatible with our mods.
  • Deploy/Install
• Runtime Exercise ( accomplished in Jan-2014)
  • Start/Stop
  • Monitoring logs, errors, and performance
  • Post-run analysis (ensure performance and products are as expected)
• Engineering Tools?
• Test/Debug Procedures & Tools?
• Simulation capabilities?
• Off-Line Data Analysis Tools?

Source Code

The only thing LBTO builds is the library libagwutils.a used by GCS.

In our repository, the top-level makefile ( Makefile.build ) builds only the API and app directories - the app directory builds the libagwutils.a used by GCS.

• The checkout in /home2/mods/trunk on the mountain is not up to date - it's got the wrong SVN address (using arizona.edu)
• After talking with Ray about the changes for the SFPToCCD transformation changes, I cleaned up the repository: deleted bin , obj , lib , libs directories, etc. It still doesn't exactly match what's in their /home2/mods/agw_1.2.0 though. (May-2013)

Building the libagwutils.a from SVN

Log in to a 64-bit CentOS machine in Tucson (last build used tcs@tcs-test ), export/checkout to the version-number you are building, make and install.
For instance, for the last build version 1.2.3:

ssh tcs@tcs-test
cd modsagw
svn checkout https://svn.lbto.org/repos/mods/trunk/agw mods-1.2.3
cd mods-1.2.3
edit Makefile.build to set the VERSION, SUBLEVEL, PATCHLEVEL numbers for what you want (top of file)
make
make install   (copies to /lbt/tcs_devel)

ssh tcs@tcs1.mountain.lbto.org
cd /lbt/tcs_devel
scp -pr tcs@tcs-test.tucson.lbto.org:/lbt/tcs_devel/mods-1.2.3 .

• Note that the GCS links in the static library, so the TCS file .build/environment has to be modified for the path of the latest MODS library.

Building the Instrument Control Applications from SVN

TBD

System Administration

• The FITS files are written to newdata via a mount on the modsNdata machines in /archive/data :

  mods1data:mods% df .
  Filesystem           1K-blocks      Used Available Use% Mounted on
  mt-archive:/mnt/newdata
                       103212288   3412160  94557256   4% /archive/data

• Power down procedures:
  • All instruments, including MODS: Instrumentation/InstrumentsShutdownForPowerOutage
  • MODS system shutdown procedure on Instrumentation page
  • MODS1 Computer Power-Down Procedure - pdf captured from RP

• Identify all accounts and logins
  all the accounts on the mods1 machine use tcsh

account	machine	description
mods	mods1 mods1data	owns the mods processes
islprog	mods1	owns mods source
support	mods1	owns support dir in source area
dts	mods1
isluser	mods1
observer	mods1
LBTO lbto	mods1 mods1data obs
OSURC osurc	mods1 mods1data obs
LBTB lbtb	mods1 mods1data obs
INAF inaf	mods1 mods1data obs
AZ az	mods1 mods1data obs
MODSeng modseng	obs machines	support scripts

• Identify all equipment makes, models, OS, and versions
  See: MODS Network Infrastructure Requirements v1.0 (Oct-2013)
• Document how computers were set up and configured
• 3rd party interfaces/equipment
• Installed programs/packages
• Required Licenses & Maintenance agreements
• Periodic procedures (cron jobs) identified

• How we set up SNMP for Zabbix to monitor the FC3 mods data boxes

Architecture

Port mapping, Switch map, Etc.

MODS has its own VLAN 192.168.139. Within that, they have assigned IPs like so:
192.168.139.0xx are common resources
192.168.139.1xx are MODS1
192.168.139.2xx are MODS2

Given a base IP address: 192.168.139.xyz and the above x values, the y is: red IEB (0) blue IEB (1) MODSx LLB and IUB (2) MODSx Instrument Computers (3) MODSx KVM console appliances (4)

MODS1 Computers
mods1	192.168.139.130
mods1data	192.168.139.131
blue ccd 1	DOS machine
red ccd1	DOS machine
MODS1 WAGOs
Instrument Utility Box (IUB)	192.168.139.122
Blue Instrument Electronic Box (IEB)	192.168.139.110
Red IEB..	192.168.139.100
Lamp/Laser Box..	192.168.139.120
MODS1 COMTROL Servers (for MicroLynx devices)
Blue IEB Comtrol 1	192.168.139.111
Blue IEB Comtrol 2	192.168.139.112
Blue IEB IMCS Comtrol.	192.168.139.113
Red IEB Comtrol 1	192.168.139.101
Red IEB Comtrol	192.168.139.102
Red IEB IMCS Comtrol	192.168.139.103
MODS1 AGw	mods1-cam 192.168.2.35 (CRB #13, serves both cameras via azcamserver 192.168.2.199)
MODS2 AGw	mods2-cam 192.168.2.36 (CRB #13, serves both cameras via azcamserver 192.168.2.199)

There's a KVM for the computers (MODS1 and MODS2)

As-built Network Architecture diagram

• MODS computer architecture diagram from MODS Network Infrastructure Requirements v1.0 (Oct-2013):
  

Internal equipment
Connectivity to backbone

how does MODS talk to the archive?
it is mounted, but it doesn't show up until you go there (because of the automount ?):

mods1data:mods% more /etc/mtab 
/dev/md1 / ext3 rw 0 0
none /proc proc rw 0 0
none /sys sysfs rw 0 0
none /dev/pts devpts rw,gid=5,mode=620 0 0
usbfs /proc/bus/usb usbfs rw 0 0
/dev/md0 /boot ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
/dev/md3 /lhome ext3 rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
automount(pid3050) /archive autofs rw,fd=5,pgrp=3050,minproto=2,maxproto=4 0 0
nfsd /proc/fs/nfsd nfsd rw 0 0

mods1data:mods% ps -ef | grep 3050
root      3050     1  0 Jul13 ?        00:00:00 /usr/sbin/automount --timeout=60 --ghost /archive file /etc/auto.archive ..... 
mods      4854  4553  0 22:48 pts/1    00:00:00 grep 3050

mods1data:mods% more /etc/auto.archive 
# $Id: MODSSoftwareHandover.txt,v 1.35 2014/01/08 19:53:54 KelleeSummers Exp $
# This is an automounter map and it has the following format
# key [ -mount-options-separated-by-comma ] location
# Details may be found in the autofs(5) manpage
#
# LBT archive machine archive1 = 192.168.38.101 [rwp/osu 2010-06-12]
# automounts archive:/newdata as /archive/data
#
data archive:/newdata

mods1data:mods% ls /archive
data/

mods1data:mods% df 
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md1              15116744   9690760   4658084  68% /
/dev/md0                101018      9041     86761  10% /boot
none                   1037384         0   1037384   0% /dev/shm
/dev/md3             136599440  92640808  37019768  72% /lhome

mods1data:mods% cd /
/

mods1data:mods% ls -al
total 270
drwxr-xr-x   28 root root  4096 Jul 13 00:22 ./
drwxr-xr-x   28 root root  4096 Jul 13 00:22 ../
drwxr-xr-x    3 root root     0 Jul 13 00:22 archive/
-rw-r--r--    1 root root     0 Jul 13 00:22 .autofsck

mods1data:mods% cd /archive/data; pwd
/archive/data

mods1data:mods% df .
Filesystem           1K-blocks      Used Available Use% Mounted on
archive:/newdata     246076176  58100744 175475440  25% /archive/data

mods1data:mods% df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md1              15116744   9690764   4658080  68% /
/dev/md0                101018      9041     86761  10% /boot
none                   1037384         0   1037384   0% /dev/shm
/dev/md3             136599440  92640808  37019768  72% /lhome
archive:/newdata     246076176  58100744 175475440  25% /archive/data

Spares

Strategy and vendors

They currently have a hot spare for the data server and the instrument control boxes in the rack. The boxes are not rsync 'ed, like I originally thought. They are manually updated every 6 months to 1 year.

• MODS_SW_HO_opkcomments.rtf: Olga's comments on the current Software Handover Document (March-2018)