MODS1 Instrument Startup Procedure
This document describes how to start the MODS1 computers and data-acquisition software, and ready the instrument
for observing and calibration. It includes options for a full startup from the point where the instrument and all computers
are powered off.
The instructions are meant primarily for LBTO support personnel. Observers would only follow these procedures
under the direct supervision of LBTO or MODS team personnel under extraordinary circumstances.
- NOTE:
- These instructions assume that the MODS1 instrument is already powered up and connected to the network with glycol coolant
flowing.
Instrument Computer Startup
The 4 computers that operate MODS1 are located in the rack in Computer Room B on Level 2, as follows:
- mods1data is the instrument Data Server in a black 4U enclosure. It runs FC3 Linux
- M1.BC is the Blue CCD control computer in a black 4U enclosure. It runs DOS
- M1.RC is the Red CCD control computer in a black 4U enclosure. It runs DOS
- mods1 is the Instrument Server in the Dell 2U-height enclosure. It runs CentOS Linux
All four computers share the fold-out keyboard, mouse, and LCD monitor via a KVM switch. They
are located next to each other in the rack in the order (top to bottom) above.
Note that there is a spare CCD computer in a 4U case in this picture immediately above the mods1
Dell computer which is in the bottom-most position in the rack. The exact number and position of
the computers in the rack will change over time, so pay attention to the labels on the front panels.
Elsewhere in the rack are hot spare computers, the CISCO network switch, and two power distribution units. You should
not have to do anything with these.
Note that like many things in the computer rooms, the rack may not be in the same place as when the photo above
was taken.
Power Up the MODS Rack and Ethernet Switch
Verify that rack power is on before you begin. In normal circumstances this is left on all the time.
Make sure that the CISCO Catalyst 2960G network switch is powered on and blinking. Note that the switch is currently mounted rear-facing, so you can only check for blinking lights through the rear door of the rack.
Verify that the Instrument Server Computer is on
The mods1 Dell instrument server should automatically power up and boot when AC power is applied. If not, press the
power button located at the upper left corner of the front panel.
This computer typically takes 3-4 minutes to boot from power up. When it is powered up and booted the tiny 1-line LCD panel should read "mods1".
Power up the Data Handling and CCD Control Computers
Pull out and unfold the LCD monitor and keyboard and turn it on.
Two taps on the
Scroll Lock key will bring up the KVM port selection menu. If the Raritan KVMoIP switch has been power cycled, you will have
to login. Username is
mods and the password is the current support password.
Use the touch pad at the lower right of the keyboard to select the machine you want to monitor, left click, and then left click the "Connect" button that appears on the screen.
The main power switches for the 4U boxes are the black rocker switch labeled "o" located at the top center of the front panel. To power up one of the 4U computers, press and release this switch.
Power up,
in order:
- the Blue CCD computer (M1.BC) and wait for the DOS prompt
- the Red CCD computer (M1.RC) and wait for the DOS prompt
- the mods1data computer, and wait until it boots and X windows is launched. mods1data is setup to automatically login as the mods instrument support user.
Instrument Control Software Startup
In this step you start up the low-level instrument control, telescope interface, and data handling system software on the mods1 and mods1data computers.
These programs must be started by the
mods support user on the mods1data console in the MODS equipment rack. For practical reasons these programs
are never to be started on the general observing workstations (obs2 et al.), nor from partner observer accounts.
Instrument Control and Interface Services
On the mods1data Linux machine, open a terminal window and startup the instrument services in order as follows:
mods1 start isis
mods1 start ie
mods1 start agw
Now type
mods1 status
and you should see the following:
MODS1 Status at 2014 Jan 08 14:26:14
MODS1 Instrument Server:
Service State Owner
--------------------------------------
IE Running mods
AGw Running mods
redIMCS Stopped
blueIMCS Stopped
modsenv Stopped
lbttcs Stopped
modsUI Stopped
--------------------------------------
MODS1 Data Server:
Service State Owner
--------------------------------------
isis Running mods
Red Caliban Stopped
Blue Caliban Stopped
--------------------------------------
At this step the modsUI, lbttcs, modsenv, and red-/blue-channel Caliban and IMCS programs should
not be running.
MODS Environmental Monitor Agent (modsenv)
In the same xterminal as above, type
mods1 start env
to start the MODS instrument environmental sensor monitor and logging program.
MODS system temperatures and pressures will be logged in the
/Logs/Env
folder on mods1data, stored by UTC date.
LBT Telescope Interface Agent (lbttcs)
Now type
mods1 start lbttcs
to start the MODS LBT telescope control system (TCS) and instrument interface (IIF) agent. This will launch no windows or produce any output on the screen, unless
for some reason the TCS goes offline, which kills the lbttcs agent.
If the TCS has to be restarted during the night, the support astronomer will have to restart the lbttcs agent before observing resumes.
Red and Blue CCD Data-Acquisition Software
- IMPORTANT:
- Do not perform this step if glycol coolant is not flowing to the instrument. The onboard CCD electronics boxes
will overheat and turn themselves off within about 30 minutes of power up if no coolant is flowing.
First, verify that the CCD Head Electronics Boxes (HEBs) are powered on. In an xterminal on mods1data, type:
isisCmd m1.ie util heb_b status
isisCmd m1.ie util heb_r status
These commands should return
HEB_B=ON
and
HEB_R=ON
, respectively.
If either box is off, type
isisCmd m1.ie util heb_b on
to power on the Blue HEB and/or
isisCmd m1.ie util heb_r on
to power on the Red HEB. Both should reply with
DONE: UTIL HEB_x=ON
.
When the HEBs are powered on, start the CCD control programs as follows:
- Select
M1.RC
on the KVM switch
- Type
IC
at the DOS prompt and wait for the detector status screen
- Select
M1.BC
on the KVM switch
- Type
IC
at the DOS prompt and wait for the detector status screen
Data Handling Programs ("caliban")
MODS uses the "caliban" data-handling program developed at OSU to transfer raw FITS images from the CCD control
computers to the
/lhome/data
data storage drive on host
mods1data
. The initials "CB" refer to caliban, qualified
by the _red or _blue suffixes for the red and blue channels, respectively.
Once the IC programs are running on
M1.RC
and
M1.BC
, use the KVM switch to go back to the
mods1data
X console
and type:
mods1 start cb_red
mods1 start cb_blue
These will launch two xterm windows running the red- and blue-channel Caliban data-transfer agents.
In each screen you should see a bunch of startup chatter.
The system should now be ready to start the IMCS and user interfaces.
IMPORTANT: the MODS Control Panel GUI should never be started on mods1data or the rack console!
Image Motion Compensation System
Once the HEBs are powered up and the IC programs are started, start the IMCS (Image Motion Compensation System, aka Flexure Compensation) control programs.
Open up a
new terminal on the mods1data console and type
mods1 start imcs
Leave this terminal up after the IMCS startup. If the IMCS is interrupted and needs to be stopped and restarted, it will spew messages in this window.
You are now ready to start the observing programs on the observer workstations in the control room and initialize the instrument.
Leave the programs running on the mods1data computer console. You can check their status at any time while logged into the observing workstations by typing
mods1 status
in an xterm window.
Instrument Start Up and Initialization
These steps are performed by the support astronomer before the observer can use MODS for calibrations or observing. On the first night of an observing run, the support
astronomer should do a warm restart of the instrument and run a couple of test images to check instrument health before turning it over to the observers.
This is the only time that the support astronomer will start and operate the MODS control panel GUI.
The MODS engineering and support scripts for these steps will be found in
/home/MODSeng/modsScripts/Support/
. Regular observers should not
run any of the scripts in this folder except at the direction of LBTO or MODS team personnel.
All MODS computers and control programs must be running, and MODS powered on for this operation.
Start the MODS Control Panel GUI
Login to one of the observing workstations as the
LBTO user and open an xterminal window.
Startup the MODS User Interface control panel (aka "the GUI") by typing
mods1 start gui
This will launch the MODS control panel GUI. Go to the MODS1 dashboard panel by clicking on the MODS1 icon on the left.
Finally, open a terminal window and cd into
/home/MODSeng/modsScripts/Support/
. This is where you will execute
the various support scripts in subsequent steps.
Which Startup Should I Do?
There are two types of instrument startup:
- Warm Startup - Done when MODS is still powered up, but after a computer system reboot (crash or routine), when MODS has been in a low-power sleep mode between runs, or at the start of a partner observing block or technical night to put the instrument into a known starting state.
- Cold Startup - Done after MODS has been powered off completely (e.g., immediately after installation on the telescope or after a site-wide power failure).
In general, a warm startup is more common than a cold startup. For example, during MODS commissioning in 2010 and 2011, we typically went 3-4 months between cold starts, usually following instrument service or site power outages.
Instrument Warm Start
An instrument warm start checks the instrument state, queries each mechanism, turns on the AGw cameras, and puts the instrument
into dual imaging mode.
A warm start is done routinely on an instrument that has been powered up for a while, and should be done at the start of a partner
block just to make sure there is no delinquent setup from the previous block.
Start the gui and open a terminal window as described above. Type
execMODS modsWarmStart.pro
to start the warm start procedure. Watch the progress on the MODS control panel and in the term as it runs. A typical warm start requires about 1 minute to complete.
If faults occur, you will be asked to assess the severity and Abort, Retry, or Ignore the problem and move on. The correct response during a startup is to
retry ("R"). If one retry fails, or you see an error about the mechanism being power cycled (PWRFLR fault), go into another terminal (leave the Abort, Retry, Ignore prompt waiting), and manually reset the mechanism and then do a "retry". For example, if you get a PWRFLR fault on the red collimator (rcolfoc) you should abort and address the problem before starting over. The usual way to address a fault on a single mechanism (e.g., rcolfoc) is to do a
manual reset, e.g.,
modsCmd rcolfoc reset
Then tell the script to retry and continue. If this fails to clear the mechanism, consult the trouble guide or contact MODS team personnel for assistance.
At the end of the warm startup, MODS will be left in dual-channel imaging mode with the hatch closed.
Instrument Cold Start
An instrument cold start needs to be done after one of these situations:
- The instrument has been completely powered off
- When recovering from site-wide power failures that include loss of power to the instruments
- One or both of the Instrument Electronics Boxes (IEBs) have been powered off
- The Instrument Utility Box (IUB) has been powered off (this powers off the entire instrument)
- When any part of the instrument has been powered down for safe servicing
A cold start should
not be done as a matter of course (i.e., re-initialize an already powered up instrument that had been running normally).
Start the gui and open a terminal as described above. Type
execMODS modsColdStart.pro
to start the cold-startup procedure. Watch the progress on the MODS control panel and in the terminal window while it runs. Typical cold-starts take 6-8 minutes to complete (it depends on the starting point).
If faults occur, you will be asked to assess the severity and Abort, Retry, or Ignore the fault. The usual correct response during a startup is to
retry ("R"). If one retry fails, you should try to address the problem with the control panel at the command level, and then resume execution of the cold startup script, unless the problem is so severe you need to seek assistance.
At the end of the cold startup, MODS will be left in dual-channel imaging mode with the hatch closed. Type
execMODS modsWake.pro
to run a short procedure to fully wake up MODS.
Post-Startup Tasks
After successful startup (warm or cold), take an image of the sieve mask in both channels to do a quick
check of the instrument.
First, open a new terminal and type
modsDisp
to launch the MODS raw image display program on your console.
Then, in your script terminal window, type
execMODS sieveSnap.pro
to instruct MODS to put in the pinhole sieve mask and calibration tower, turn on the VFLAT lamp, and take images of the mask with the blue and red cameras using SDSS g and r filters, respectively. The images should show the pinhole grid well-centered on the images, a few pixels away from the center boundary of the four quadrants (not exactly at "four corners").
Another good test is to change the CCD readout to 8Kx3K (full frame) and take bias frames in each channel and look for any signs of noise or other new artifacts.
When you are satisfied that the instrument is up and running, close your copy of the MODS control panel GUI, but you may keep the modsDisp windows running so you can see the data as they arrive in the archive staging disk (
/newdata
).
You are now ready to help the observers startup the MODS controls for their observing run.
--
RichardPogge - 2012 Nov 12