You are here: Foswiki>Instrumentation Web>InstrumentsShutdownForPowerOutage (17 Jun 2022, OlgaKuhn)Edit Attach

Facility Instruments In the Event of a Power Outage

Before computers are shutdown, ensure the instrument is in a safe state and shutdown processes. The following describes how to put the facility instruments into a safe state and shut down their associated computers in preparation for possible total loss of power.

This procedure is meant to be used in conjunction with the IT computer shutdown/startup procedures: Software/ComputerShutdownStartup.

Overview

-Before shutting down Facility instruments, send a note to telescope work

-Contact Instrument Teams regarding other instruments:
  1. ARGOS, AO, LBC, MODS, and PEPSI machines are taken care of by the overall computer shutdown script that is driven by UPS power states. No manual intervention should be required - unless we are in the middle of observing, which would require someone to safe the instrument (see below).
    See Software/ComputerShutdownStartup.
  2. AO
    Doug Miller
    cell: 520-609-8663
    work: 520-626-6945
  3. ARGOS
    Gustavo Rahmer
    cell: 520-247-8229
    work: 520-626-7152

-Safe & Shutdown Facility Instruments:

LUCI 1 & 2 in the event of a Power Outage

Safing LUCI1 & 2

If LUCI1 or LUCI2 is actively being used for testing or science, there is a possibility the MOS may not be in a safe state for the software to be shutdown. It is not safe to shut off the instrument SW if a mask is in turnout or moving. Check the message log for recent activity. If necessary, confirm with the observer interface that there is no mask in use (all masks are in storage or a mask in the FPU).

To confirm no mask is in a dangerous state look at a LUCI observer GUI. If one is not available on one of the computers in the control room, open one:
  1. On any obs machine, open a terminal. Type:
    >open_observer luci all
    This will open all of the components normally used to observe with either or both LUCI's. The main one we care about is the Observer GUI.
    LUCI2ObserverGUI.png

  2. If an oberver's script is currently excuting, it should be stopped by using the "Other..." button to select "ABORT". ABout takes 30 sec. or so to complete, so give it some time. If the script was actually running, some objects on screen will become ringed in red when it aborts. If a mask is in motion between cabinet and turnout or FPU, wait for it to finish. It can take a minute or so. If it just says "Turnout" or "in FPU" or "no mask used", then proceed with next step.
  3. In the upper left panel for LUCI1 and upper right panel for LUCI2 of the LUCI Observer GUI, read the state of the MOS and if a mask is in use. The dangerous state is if a mask is in "turnout" or in motion between the cabinet and the focal plane (FPU). If mask is in FPU (as shown below), you don't have to do anything. If it is necessary to change the state of the mask, bring up the Instrument GUI from the "LUCI1" and/or "LUCI2" button at the bottom or each status panel on the observer GUI. You will see open this:
    LUCI2InstrumentGUI.png
  4. The right side column shows the state that the instrument is in now. The left side is use to tell it what change to make. Use the drop down menus in the left column of the Instrument GUI to select the MOS configuration desired (either mask in FPU or "no mask in use") and select "blind" filters. Then hit "submit". (If a mask is in turnout, select FPU - it's faster than selecting "no mask in use", which puts the mask back into the cabinet.) Clicking the "refresh" button is necessary to see any changes. (It's also harmless.)
  5. When done, close all of the GUI's.

Power down the LUCI1 and LUCI2 MCE/ROE

This is necessary to avoid overheating of the electronics if instrument cooling is lost. Logged in to the luci.luci.lbto.org, open firefox. One can log in to luci.luci.lbto.org via NXclient as any user and open the firefox window using the desktop icon. Or use ssh by typing the following in a terminal to log in: 'ssh -X lucifer@luci.luci.lbto.org', then enter the password, then type firefox.

In the firefox window, go the the LUCI web I/O page. Type the following in the address bar to be directed to the corresponding webI/O:
192.168.0.11 for LUCI1 on the telescope
192.168.0.21 for LUCI2 on the telescope
192.168.0.31 for either LUCI when they are in the cleanroom

LUCIMCE.png

Select "Config" from the tab on the left to login. Select Expert mode. It will ask for an engineering pasword.

Once logged in, click on the green "ON" button next to MCE/ROE remote power to power them off. The green button should reload as grey \x93OFF\x94

Logout by selecting the Config => Session Control => Logout in the left bar. Select \x93Abort\x94 to exit without saving (which is the desired config).

Shutdown LUCI Software

Once again, there are 2 ways to go about this: NX Client or via ssh.
  1. Open an NXclient session as lucifer to the luci.luci.lbto.org computer OR from a terminal "ssh -X lucifer@luci.luci.lbto.org"
  2. Start up the software. If using NXclient, double click the "LMC" Icon. If opening from a terminal ssh'd into luci.luci.lbto.org as user "lucifer", type "/home/lucifer/lcsp/bin/startLuci.sh". Make sure you used the -X flag when you ssh'd or else the GUI will not come up. Note, it reacts slower if you open it via "ssh".
  3. Select the \x93all services\x94 tab. Select the STOP button next to "System Status" (top row). It will ask for confirmation. Confirm that you want to stop the software. After about 20 seconds hit "refresh table" button at the bottom of the GUI. You should see all green dots have turned red, indicating processes stopped. If any are still green, hit refresh again until all are red. It can take a minute. This will stop services for both LUCI1 and LUCI2. Exit GUI.
The software for both LUCI's has now been stopped. Normally you would not need to shut down the Linux operating system. The normal power failure shutdown scripts that are triggered when the main power fails will do that automatically if it becomes necessary.

MODS in the event of a Power Outage

MODS software services and instrument computers (except the DOS machines) will go down with the automatic computer shutdown script when the UPS reaches a specific threshold.

If MODS is in use when a power outage is imminent, you can safe the mechanisms with the following commands:
  • If you're on an obs machine (or remote, etc.), login as lbto in a mountain xterm and type modsSafeMechanisms
  • If you're on the KVM on mods1data or mods2data , type safeSystem.sh

The computers will go down with the UPS monitor script. No manual intervention should be required.
See Software/ComputerShutdownStartup.
Also see MODS Server Shutdown Script - Rick's documentation of the MODS shutdown script used by the facility shutdown script.

OLD (manual) MODS Safing and Shutdown Procedure

  1. Select 'mods1data' on the kvm and open a terminal on the desktop (right click on the background of the desktop and select 'terminal from the menu')
    or
    ssh into mods1data as user mods
  2. Type the following isis commands to 'safe' the instrument, closing the hatch, turning of the laser, and turning off the calibration lamp:
    >isisCmd m1.ie hatch close
    >isisCmd m1.ie irlaser disable
    >isisCmd m1.ie lamp off
  3. Turn off the HEB's, AGW, and WFS. Particularly important in times when instrument cooling could be lost.
    >isisCmd m1.ie util heb_b off
    >isisCmd m1.ie util heb_r off
    >isisCmd m1.ie util agc off
    >isisCmd m1.ie util wfs off
  4. Repeat steps 1-3 for mods2, logging in to mods2data, and substituting 2 in for 1 in all isis commands.

Shutting down processes and computers

  • Quit IC's. If performing remotely, bring up the remote kvm access though a browser. Enter the address 192.168.139.19 in a compatible browser and enter the username and password. The interface will look similar to the kvm in CRB. This is the only way to quit the IC's. Select m1.bc on kvm and type quit (no prompt needed to type). Once process has quit a prompt should appear.
    Repeat for m1.rc, m2.bc, and m2.rc.
  • Stop the software services:
    login as lbto in a mountain xterm and type modsKillServices
    or
    If you're on the KVM on mods1data or mods2data , type killServices.sh or the manual process here...
  1. In a terminal logged in as mods on mods1data, type the following:
    >mods1 stop imcs
    >mods1 stop cb_red
    >mods1 stop cb_blue
    >mods1 stop lbttcs
    >mods1 stop     env
    >mods1 stop agw
    >mods1 stop ie
    >mods1 stop isis
    1 In a terminal logged in as mods on mods2data, type the following:
    >mods2 stop imcs
    >mods2 stop cb_red
    >mods2 stop cb_blue
    >mods2 stop lbttcs
    >mods2 stop env
    >mods2 stop agw
    >mods2 stop ie
    >mods2 stop isis

  • Check the Status and confirm all processes stopped, with: mods1 status and mods2 status
  • Power off the four computers by logging in
    as root on mods1 and typing poweroff
    as mods on mods1data and typing poweroff
    as mods on mods2 and typing poweroff
    as mods on mods2data and typing poweroff
  • If on-site, power off the DOS boxes using the front panel power switch.
    If off-site, it is OK for the DOS boxes to get powered down when the rack power turns off.

LBC R & B in the event of a Power Outage

If it is possible that LBC is powered up (during observing), power off via the web interface. Or using a telescope xterm on the mountain.

  1. As user telescope, in an xterm, type lbc off
    this will effectively send the GUI Turn OFF for "Other Systems" and "House Keeping"
or
  1. Open a web browser and go to http://lbccontrol/ui/main.html
  2. Select the "Power Control" from the left side panel and "Turn OFF" other systems and "Turn OFF House Keeping"/

LBCControl.png

The CMU computer will be shutdown by the overall computer shutdown script that is driven by UPS power states. No manual intervention should be required.
See Software/ComputerShutdownStartup.

AGW in the event of a Power Outage

The agwN-cam, modsN-cam, and azcamserver machines are taken care of by the overall computer shutdown script that is driven by UPS power states. No manual intervention should be required.
See Software/ComputerShutdownStartup.

LBTI in the event of a Power Outage

  1. Go to the LBTI Web page lbti-web:8080

    and select "Engineering tools --> Power"
    LBTIPowerControl.png
  2. IS THIS STILL VALID? DON'T SEE THE OPTION WHEN THE STATE IS OFF.
    Click the "Initiate Complete LBTI Shutdown"
    it performs the following, in order:
    1. shut off NOMIC camera
    2. shut off LMIRCAM camera
    3. shut off PhaseCam camera
    4. shut off cryomech compressor
    5. close all valves
    6. shut off all pumps
    7. clean shutdown of ubc, dxwfs and sxwfs computers
    8. shuts off ALL pulizzi power strip outlets
    9. clean shutdown of nomic, lmircam, housekeeper computers
    10. clean shutdown of mountain Nimble SAN
    11. clean shutdown of web computer

PEPSI in the event of a Power Outage

PEPSI software and computers are taken care of by the overall computer shutdown script that is driven by UPS power states. No manual intervention should be required.
See Software/ComputerShutdownStartup.

If you need to manually shutdown PEPSI servers:
in a terminal, type: 
 ssh pepsi@alpha.pepsi.lbto.org ses/pepsi/misc/shutdown
or 
 ssh pepsi@192.168.164.14 ses/pepsi/misc/shutdown

This command shuts down all of the PEPSI computers (in CRB and elsewhere). The hardware can go down with the UPS.

If glycol is not running, then the AGWs 7 & 8, the SX/DX PFU and the bridge UMAC need to be shutdown. To do this, type: 
 ssh pepsi@alpha.pepsi.lbto.org ses/pepsi/misc/shutdownUMAC

If issued from any of two PEPSI desk computers, then no password is required.
If issued from any other LBT computer (like a terminal on an obs machine), the password is necessary.
Then the confirmation is asked:
Are you sure to shutdown all PEPSI servers (y/n)?
type y or n. No further dialogs or passwords are required.

Linc-Nirvana in the event of a Power Outage

  1. Go to the LN Master Switch Panel Gui

    lnPowerPanel.png
  2. Select the off button in the upper right-hand corner. This button will power off all 6 cabinets by software control.
  3. Send email to LN@lbto.org notifying of outage/status.

See also: Instrumentation/LnPowerOffPage

-- %USERSIG{JenniferPower - 2015-06-17}%

Comments

 
I Attachment Action Size Date Who Comment
LBCControl.pngpng LBCControl.png manage 153 K 06 Sep 2016 - 17:03 UnknownUser LBC Power Control web page
LBTIPowerControl.pngpng LBTIPowerControl.png manage 94 K 01 Nov 2016 - 15:41 UnknownUser LBTI Power Control page
LNMasterGUI.pngpng LNMasterGUI.png manage 257 K 01 Nov 2016 - 15:29 UnknownUser LN master GUI snapshot
LUCI2InstrumentGUI.pngpng LUCI2InstrumentGUI.png manage 85 K 01 Nov 2016 - 15:30 UnknownUser LUCI2 Instrument GUI
LUCI2ObserverGUI.pngpng LUCI2ObserverGUI.png manage 187 K 01 Nov 2016 - 15:29 UnknownUser LUCI2 Observer GUI
LUCIMCE.pngpng LUCIMCE.png manage 130 K 01 Nov 2016 - 15:30 UnknownUser LUCI MCE/ROE GUI
LUCIMgmntConsole.pngpng LUCIMgmntConsole.png manage 35 K 01 Nov 2016 - 15:31 UnknownUser LUCI Management Console
lmc.pngpng lmc.png manage 135 K 16 Dec 2016 - 22:03 UnknownUser  
lnPowerPanel.pngpng lnPowerPanel.png manage 419 K 28 Jun 2018 - 17:57 JenniferPower  
modsShutdown_v1.1.pdfpdf modsShutdown_v1.1.pdf manage 196 K 28 Nov 2016 - 20:02 UnknownUser MODS Server Shutdown Script
Edit  | Attach | Print version | History: r34 < r33 < r32 < r31 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r34 - 17 Jun 2022, OlgaKuhn
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback