Computer Startup Sequence

This procedure assumes that power was lost on the mountain, hosts are down, but clean power is now back On-Line. Do NOT begin this procedure if "dirty" power is suspected. This procedure is sequential; Do each step in order, and only proceed to the next section after instructions for the prior section have completed!

Manual actions are required to begin. Once these actions are done, computer restarts will proceed automatically (mostly). Some follow-on manual actions are also required.
Do NOT just start pushing random power buttons! SW/IT or the SW/IT manager will be needed during the procedure.

Core Network and Computer Room Electricity

  1. Turn on main circuit breakers located in CRA circuit breaker panel PP22-UPS-1 (Right wall CRA)
    PP22-UPS-1-sm.jpgPP22-UPS-1Breakers-sm.jpg

  2. Turn on main circuit breakers located in CRB in circuit breaker panel PP22-UPS-2 (located on wall near MODS rack, CRB).
    PP22-UPS-2-sm.jpgPP22-UPS-2Breakers-sm.jpg

  3. Turn on the REAR switches to the three power supplies on core-b (Rear R2 CRA, U1-U21).
    Core_B-sm.jpg

Wait 5 minutes - Cisco switches need time to startup


Before bringing up the computers, check that the core routing is up (green check) in the list below:
Note: only one core is necessary (doesn't matter which). One VATT switch is too (doesn't matter which).

No permission to view NetworkStatus

Infrastructure machines

  1. Power up the Synology disk arrays in CRA-R3 slots 1-8 by pushing the power buttons on just the two Synology units labeled MAIN (the non-main units are labeld EXP for expansion).
    SynologyPower-sm.jpg SynologyLabels-sm.jpg

  1. Push grey button on face of the Tape Unit labeled LBTLIBRARY1, (CRA-R6 slots U12-15)
    20170912_TapeUnit.jpg

monitor.mountain (CRA-R3, U24)

  1. Push power button on monitor.mountain.lbto.org.
    20170912_monitor.jpg

    After monitor.mountain boots, it will auto-start the script that restarts most computers.
  2. Call Tucson for IT help. An intentional "wait state" in the script may need to be cleared (after verification of network, DNS, and disk mounts). Then, the script will bring up many hosts automatically. To clear the wait state, IT personnel must delete the verify-mounts file in /home/ups/servers. Afterwards, all remaining computers that can be started via PDU control will be started.

Call Tucson IT support for help. They will need to verify the following (NFS, DNS):

Verification of a NFS server can be carried out with two commands (on a standard networked Linux install):

rpcinfo -p 192.168.39.20

Should show "nfs" as one of the RPC programs registered. If "nfs" is not returned, the NFS server is not running.
showmount -e 192.168.39.20

Should show the exported NFS filesystems. An example (which may vary):
# showmount -e 192.168.39.20
Export list for 192.168.39.20:
/volume6/lbto_archive        *
/volume3/lbto_flao           *
/volume3/lbto_homes          *
/volume3/lbto_data           *
/volume3/lbto_lbtshare       *
/volume3/lbto_argos          *
/volume6/lbto_telemetry_ovms *
/volume6/lbto_repository     *
/volume6/lbto_telemetry_tcs  *
/volume3/lbto_logs           *
/volume3/mt-syn-iso          192.168.38.26,192.168.38.25,192.168.38.22
/volume1/mt-syn-datastore2   192.168.34.0/24,192.168.38.0/24

DNS can be verified by issuing a query command (here using "dig"):
  dig -t any mountain.lbto.org @192.168.34.11
  dig -t any mountain.lbto.org @192.168.34.12

The command should return an ";;ANSWER" section filled with information. If a blank ";;ANSWER" section is returned, or if nothing is returned, DNS is not up properly.

Finish Manual Reboot Actions (for computers without PDU control):

  1. Push power buttons on OBS hosts (control room, OBS 1 through OBS 6)
  2. Push the power buttons on the NUC computers driving the large displays in the control room
  3. Push power button on DIMM computer (LLTH)

Additional Software Actions to complete startup

TCS / Software personnel need to perform the checks in the PostPowerHitCheckout procedures/notes to get the rest of the software running - this includes VMs, weatherstation, TCS/WDA, Alarm Handler IOCs, etc.

Send out an email to tucson@lbto.org and to mountain@lbto.org to let them know the infrastructure is back up

Send out an email to instruments@lbto.org (which includes partner instrument teams) to let them know that TCS is running so any non-automated instrument computers and software can be restarted

I Attachment Action Size Date Who Comment
20170912_TapeUnit.jpgjpg 20170912_TapeUnit.jpg manage 311 K 13 Sep 2017 - 22:31 UnknownUser CRA tape unit
20170912_monitor.jpgjpg 20170912_monitor.jpg manage 83 K 13 Sep 2017 - 22:37 UnknownUser monitor computer server
Core_B-sm.jpgjpg Core_B-sm.jpg manage 80 K 07 Jul 2017 - 20:08 UnknownUser core-b power supplies
PP22-UPS-1-sm.jpgjpg PP22-UPS-1-sm.jpg manage 113 K 07 Jul 2017 - 19:54 UnknownUser PP22 breaker box, CRA
PP22-UPS-1Breakers-sm.jpgjpg PP22-UPS-1Breakers-sm.jpg manage 343 K 07 Jul 2017 - 19:49 UnknownUser Main breaker switch, CRA PP22-UPS-1
PP22-UPS-2-sm.jpgjpg PP22-UPS-2-sm.jpg manage 37 K 07 Jul 2017 - 20:14 UnknownUser PP22 UPS-2, CRB breaker box
PP22-UPS-2Breakers-sm.jpgjpg PP22-UPS-2Breakers-sm.jpg manage 143 K 07 Jul 2017 - 20:14 UnknownUser PP22 UPS-2, CRB breaker box
SynologyLabels-sm.jpgjpg SynologyLabels-sm.jpg manage 347 K 14 Sep 2017 - 16:07 UnknownUser Synology rack labels
SynologyPower-sm.jpgjpg SynologyPower-sm.jpg manage 273 K 14 Sep 2017 - 15:41 UnknownUser Synology power buttons
Topic revision: r19 - 14 Sep 2017, KelleeSummers
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback