MODS1 Instrument Startup Procedure

This document describes how to start the MODS1 computers and data-acquisition software, and ready the instrument for observing and calibration. It includes options for a full startup from the point where the instrument and all computers are powered off.

The instructions are meant primarily for LBTO support personnel. Observers would only follow these procedures under the direct supervision of LBTO or MODS team personnel under extraordinary circumstances.

NOTE:

These instructions assume that the MODS1 instrument is already powered up and connected to the network with glycol coolant flowing.

Instrument Computer Startup

The 4 computers that operate MODS1 are located in the rack in Computer Room B on Level 2, as follows:

mods1data is the instrument Data Server in a black 4U enclosure. It runs FC3 Linux

M1.BC is the Blue CCD control computer in a black 4U enclosure. It runs DOS

M1.RC is the Red CCD control computer in a black 4U enclosure. It runs DOS

mods1 is the Instrument Server in the Dell 2U-height enclosure. It runs CentOS Linux

All four computers share the fold-out keyboard, mouse, and LCD monitor via a KVM switch. They are located next to each other in the rack in the order (top to bottom) above.

Note that there is a spare CCD computer in a 4U case in this picture immediately above the mods1 Dell computer which is in the bottom-most position in the rack. The exact number and position of the computers in the rack will change over time, so pay attention to the labels on the front panels.

Elsewhere in the rack are hot spare computers, the CISCO network switch, and two power distribution units. You should not have to do anything with these.

Note that like many things in the computer rooms, the rack may not be in the same place as when the photo above was taken.

Power Up the MODS Rack and Ethernet Switch

Verify that rack power is on before you begin. In normal circumstances this is left on all the time.

Make sure that the CISCO Catalyst 2960G network switch is powered on and blinking. Note that the switch is currently mounted rear-facing, so you can only check for blinking lights through the rear door of the rack.

Verify that the Instrument Server Computer is on

The mods1 Dell instrument server should automatically power up and boot when AC power is applied. If not, press the power button located at the upper left corner of the front panel.

This computer typically takes 3-4 minutes to boot from power up. When it is powered up and booted the tiny 1-line LCD panel should read "mods1".

Power up the Data Handling and CCD Control Computers

Pull out and unfold the LCD monitor and keyboard and turn it on.

Two taps on the Scroll Lock key will bring up the KVM port selection menu. If the Raritan KVMoIP switch has been power cycled, you will have to login. Username is mods and the password is the current support password.

Use the touch pad at the lower right of the keyboard to select the machine you want to monitor, left click, and then left click the "Connect" button that appears on the screen.

The main power switches for the 4U boxes are the black rocker switch labeled "o" located at the top center of the front panel. To power up one of the 4U computers, press and release this switch.

Power up, in order:

1. the Blue CCD computer (M1.BC) and wait for the DOS prompt
2. the Red CCD computer (M1.RC) and wait for the DOS prompt
3. the mods1data computer, and wait until it boots and X windows is launched. mods1data is setup to automatically login as the mods instrument support user.

Instrument Control Software Startup

In this step you start up the low-level instrument control, telescope interface, and data handling system software on the mods1 and mods1data computers. These programs must be started by the mods support user on the mods1data console in the MODS equipment rack. For practical reasons these programs are never to be started on the general observing workstations (obs2 et al.), nor from partner observer accounts.

Instrument Control and Interface Services

On the mods1data Linux machine, open a terminal window and startup the instrument services in order as follows:

   mods1 start isis
   mods1 start ie
   mods1 start agw

Now type

   mods1 status

and you should see the following:

MODS1 Status at 2014 Jan 08 14:26:14

MODS1 Instrument Server:
  Service       State           Owner
  --------------------------------------
  IE            Running         mods
  AGw           Running         mods
  redIMCS       Stopped
  blueIMCS      Stopped
  modsenv       Stopped
  lbttcs        Stopped
  modsUI        Stopped
  --------------------------------------
  
MODS1 Data Server:
  Service       State           Owner
  --------------------------------------
  isis          Running         mods
  Red Caliban   Stopped
  Blue Caliban  Stopped
   --------------------------------------

At this step the modsUI, lbttcs, modsenv, and red-/blue-channel Caliban and IMCS programs should not be running.

MODS Environmental Monitor Agent (modsenv)

In the same xterminal as above, type

   mods1 start env

to start the MODS instrument environmental sensor monitor and logging program.

MODS system temperatures and pressures will be logged in the /Logs/Env folder on mods1data, stored by UTC date.

LBT Telescope Interface Agent (lbttcs)

Now type

   mods1 start lbttcs

to start the MODS LBT telescope control system (TCS) and instrument interface (IIF) agent. This will launch no windows or produce any output on the screen, unless for some reason the TCS goes offline, which kills the lbttcs agent.

If the TCS has to be restarted during the night, the support astronomer will have to restart the lbttcs agent before observing resumes.

Red and Blue CCD Data-Acquisition Software

IMPORTANT:

Do not perform this step if glycol coolant is not flowing to the instrument. The onboard CCD electronics boxes will overheat and turn themselves off within about 30 minutes of power up if no coolant is flowing.

First, verify that the CCD Head Electronics Boxes (HEBs) are powered on. In an xterminal on mods1data, type:

   isisCmd m1.ie util heb_b status
   isisCmd m1.ie util heb_r status

These commands should return HEB_B=ON and HEB_R=ON, respectively.

If either box is off, type

   isisCmd m1.ie util heb_b on

to power on the Blue HEB and/or

   isisCmd m1.ie util heb_r on

to power on the Red HEB. Both should reply with DONE: UTIL HEB_x=ON.

When the HEBs are powered on, start the CCD control programs as follows:

1. Select M1.RC on the KVM switch
2. Type IC at the DOS prompt and wait for the detector status screen
3. Select M1.BC on the KVM switch
4. Type IC at the DOS prompt and wait for the detector status screen

Data Handling Programs ("caliban")

MODS uses the "caliban" data-handling program developed at OSU to transfer raw FITS images from the CCD control computers to the /lhome/data data storage drive on host mods1data. The initials "CB" refer to caliban, qualified by the _red or _blue suffixes for the red and blue channels, respectively.

Once the IC programs are running on M1.RC and M1.BC , use the KVM switch to go back to the mods1data X console and type:

   mods1 start cb_red
   mods1 start cb_blue

These will launch two xterm windows running the red- and blue-channel Caliban data-transfer agents. In each screen you should see a bunch of startup chatter.

The system should now be ready to start the IMCS and user interfaces.

IMPORTANT: the MODS Control Panel GUI should never be started on mods1data or the rack console!

Image Motion Compensation System

Once the HEBs are powered up and the IC programs are started, start the IMCS (Image Motion Compensation System, aka Flexure Compensation) control programs.

Open up a new terminal on the mods1data console and type

    mods1 start imcs

Leave this terminal up after the IMCS startup. If the IMCS is interrupted and needs to be stopped and restarted, it will spew messages in this window.

---

You are now ready to start the observing programs on the observer workstations in the control room and initialize the instrument.

Leave the programs running on the mods1data computer console. You can check their status at any time while logged into the observing workstations by typing mods1 status in an xterm window.

Instrument Start Up and Initialization

These steps are performed by the support astronomer before the observer can use MODS for calibrations or observing. On the first night of an observing run, the support astronomer should do a warm restart of the instrument and run a couple of test images to check instrument health before turning it over to the observers.

This is the only time that the support astronomer will start and operate the MODS control panel GUI.

The MODS engineering and support scripts for these steps will be found in /home/MODSeng/modsScripts/Support/. Regular observers should not run any of the scripts in this folder except at the direction of LBTO or MODS team personnel.

All MODS computers and control programs must be running, and MODS powered on for this operation.

Start the MODS Control Panel GUI

Login to one of the observing workstations as the LBTO user and open an xterminal window.

Startup the MODS User Interface control panel (aka "the GUI") by typing

   mods1 start gui

This will launch the MODS control panel GUI. Go to the MODS1 dashboard panel by clicking on the MODS1 icon on the left.

Finally, open a terminal window and cd into /home/MODSeng/modsScripts/Support/. This is where you will execute the various support scripts in subsequent steps.

Which Startup Should I Do?

There are two types of instrument startup:

1. Warm Startup - Done when MODS is still powered up, but after a computer system reboot (crash or routine), when MODS has been in a low-power sleep mode between runs, or at the start of a partner observing block or technical night to put the instrument into a known starting state.
2. Cold Startup - Done after MODS has been powered off completely (e.g., immediately after installation on the telescope or after a site-wide power failure).

In general, a warm startup is more common than a cold startup. For example, during MODS commissioning in 2010 and 2011, we typically went 3-4 months between cold starts, usually following instrument service or site power outages.

Instrument Warm Start

An instrument warm start checks the instrument state, queries each mechanism, turns on the AGw cameras, and puts the instrument into dual imaging mode.

A warm start is done routinely on an instrument that has been powered up for a while, and should be done at the start of a partner block just to make sure there is no delinquent setup from the previous block.

Start the gui and open a terminal window as described above. Type

    execMODS modsWarmStart.pro

to start the warm start procedure. Watch the progress on the MODS control panel and in the term as it runs. A typical warm start requires about 1 minute to complete.

If faults occur, you will be asked to assess the severity and Abort, Retry, or Ignore the problem and move on. The correct response during a startup is to retry ("R"). If one retry fails, or you see an error about the mechanism being power cycled (PWRFLR fault), go into another terminal (leave the Abort, Retry, Ignore prompt waiting), and manually reset the mechanism and then do a "retry". For example, if you get a PWRFLR fault on the red collimator (rcolfoc) you should abort and address the problem before starting over. The usual way to address a fault on a single mechanism (e.g., rcolfoc) is to do a manual reset, e.g.,

     modsCmd rcolfoc reset

Then tell the script to retry and continue. If this fails to clear the mechanism, consult the trouble guide or contact MODS team personnel for assistance.

At the end of the warm startup, MODS will be left in dual-channel imaging mode with the hatch closed.

Instrument Cold Start

An instrument cold start needs to be done after one of these situations:

1. The instrument has been completely powered off
2. When recovering from site-wide power failures that include loss of power to the instruments
3. One or both of the Instrument Electronics Boxes (IEBs) have been powered off
4. The Instrument Utility Box (IUB) has been powered off (this powers off the entire instrument)
5. When any part of the instrument has been powered down for safe servicing

A cold start should not be done as a matter of course (i.e., re-initialize an already powered up instrument that had been running normally).

Start the gui and open a terminal as described above. Type

    execMODS modsColdStart.pro

to start the cold-startup procedure. Watch the progress on the MODS control panel and in the terminal window while it runs. Typical cold-starts take 6-8 minutes to complete (it depends on the starting point).

If faults occur, you will be asked to assess the severity and Abort, Retry, or Ignore the fault. The usual correct response during a startup is to retry ("R"). If one retry fails, you should try to address the problem with the control panel at the command level, and then resume execution of the cold startup script, unless the problem is so severe you need to seek assistance.

At the end of the cold startup, MODS will be left in dual-channel imaging mode with the hatch closed. Type

    execMODS modsWake.pro

to run a short procedure to fully wake up MODS.

Post-Startup Tasks

After successful startup (warm or cold), take an image of the sieve mask in both channels to do a quick check of the instrument.

First, open a new terminal and type

    modsDisp

to launch the MODS raw image display program on your console.

Then, in your script terminal window, type

    execMODS sieveSnap.pro

to instruct MODS to put in the pinhole sieve mask and calibration tower, turn on the VFLAT lamp, and take images of the mask with the blue and red cameras using SDSS g and r filters, respectively. The images should show the pinhole grid well-centered on the images, a few pixels away from the center boundary of the four quadrants (not exactly at "four corners").

Another good test is to change the CCD readout to 8Kx3K (full frame) and take bias frames in each channel and look for any signs of noise or other new artifacts.

When you are satisfied that the instrument is up and running, close your copy of the MODS control panel GUI, but you may keep the modsDisp windows running so you can see the data as they arrive in the archive staging disk (/newdata).

You are now ready to help the observers startup the MODS controls for their observing run.

-- RichardPogge - 2012 Nov 12