MODS Post-Shutdown Startup

The following is a procedure for restarting MODS1 or MODS2 after it is installed, hooked up and released to science operations.

Are the IUBs, IEBs and COMTROL servers online? (pingMODSN)

• Run pingMODSN to check the status of the IUB, IEB and COMTROL ports. pingMODSN can be run from the lbto account on obs1, obs2 or robs and from the mods account on modsNdata. If any ports are reported as offline, check:
  • that the IUB is plugged in at the instrument. There is no remote switching of the IUB.
  • The IEBs may be off. Check the IEB status by isisCmd --modsN mN.ie util ieb_b or isisCmd --modsN mN.ie util ieb_r, and if one or the other of the IEBs is off, you'll need to turn it on by (in this example for MODS1 and ieb_b, but replace 1 with 2 and b with r for other combinations). (I always turn it off explicitly before turning it on, to clear any BRK=FAULT messages).

             isisCmd --mods1 m1.ie util ieb_b off
             isisCmd --mods1 m1.ie util ieb_b on

    After turning on the IEB, the IE server will need to be restarted.

             mods1 start ie

Turn on HEBs and start IC programs to monitor dewar temperatures and pressures during cooldown

1. Start up the isis, ie, agw and env services.
2. After confirming that glycol is flowing, turn on the HEBs. One way to double-check the glycol flow at MODS is by checking the glycol supply/return pressures and temperatures in the env log which is on the modsN computer in ~/Logs/Env/. The glycol temperature should be just a couple degrees below ambient, and the supply/return pressures are ~40 psi for both MODS (the supply pressure sensor on MODS2 is broken). Samples of mods@mods1:~/Logs/Env/mods1.20171025.log and mods@mods2:~/Logs/Env/mods2.20171025.log are below:

            Sample of mods@mods1:~/Logs/Env/mods1.20171025.log:
            # UTC Date/Time      Tamb  Psup  Pret  Tsup  Tret  Tiub  Tagw  Bair  Bret  Rair  Rret  AirT  AirB  ColT  ColB
            2017-10-25T00:00:07  12.8  47.2  45.9  12.6  13.1  13.5 -101.6  16.1  12.8  15.6  12.6  11.3  11.7  11.6  10.3
            2017-10-25T00:01:07  12.7  47.2  45.8  12.7  13.1  13.6 -102.2  16.1  12.8  15.6  12.6  11.2  11.8  11.6  10.3
   
            Sample of mods@mods2:~/Logs/Env/mods2.20171025.log:
            # UTC Date/Time      Tamb  Psup  Pret  Tsup  Tret  Tiub  Tagw  Bair  Bret  Rair  Rret  AirT  AirB  ColT  ColB
            2017-10-25T00:00:49  13.3 100.0  43.4  12.5  15.3 -10.4  16.0  16.4  12.8  16.0  12.9  12.3  11.8  13.2 -14.1
            2017-10-25T00:01:49  13.3 100.0  43.4  12.4  15.3 -10.4  16.1  16.4  12.8  16.0  12.8  12.3  11.8  13.3 -14.0

3. Start the IC programs on M1.BC, M1.RC, M2.BC and M2.RC.
4. isisCmd --modsN mN.rc estatus

            lbto@obs4:2 % isisCmd --mods1 m1.rc estatus
            DONE: ESTATUS TEDPOWER=ON LEDPOWER=OFF IGPOWER=ON DEWPRES=7.1600E-06 CTEMPIN=9.35 CTEMPOUT=9.81 HSTEMP=12.51      HEBTEMP=17.11 +24V=23.81 +5V=4.96 CCDTEMP=-99.47 -15V=-14.98 DEWTEMP=-196.73 +12FAN=11.77 BOGHTRV=2.44 +3.3V=3.29

5. Are TEDPOWER and IGPOWER ON? Is LEDPOWER OFF? If not, turn them on and off. The example below for MODS1 where the isisCmd is run on obs1, obs2 or robs (if run from mods1data, then --mods1 is not needed).
   

            isisCmd --mods1 m1.rc tedpower on
            isisCmd --mods1 m1.rc igpower on
            isisCmd --mods1 m1.rc ledpower off

6. Turn back on the MODS alerts, to record temperatures every minute during cooldown and every 30 minutes during operation.
   1. ssh ioc@hydrogen.mountain.lbto.org
      1. if you login from telescope@robs, then no password is required
   2. Execute the following command: sudo systemctl stop lbto-ahs-modsalert.timer

      It should not report anything back.

   3. To verify it is working, type: systemctl status lbto-ahs-modsalert.timer

      It should report back something like the following, where the Active: field shows active.

      ● lbto-ahs-modsalert.timer - MODS alerts
      Loaded: loaded (/usr/lib/systemd/system/lbto-ahs-modsalert.timer; enabled; vendor preset: disabled)
      Active: active (waiting) since Tue 2022-12-06 22:33:56 GMT; 12s ago
      Trigger: Tue 2022-12-06 23:00:00 GMT; 25min left

Can we communicate with the mechanisms? (pokeMODSN)

Execute the command, pokeMODSN from the lbto account on obs1, obs2 or robs, or the mods account on mods1data or mods2data. Following is a sample of the output when communication is fine. The messages refer to the initialization that is needed and is the next step.

mods1data:mods% pokeMODS1
Poking MODS1 Mechanisms...
Common focal plane:
DONE: HATCH HATCH=CLOSED
DONE: CALIB CALIB=OUT
DONE: AGWX AGWXS=0.000
DONE: AGWY AGWYS=0
DONE: AGWFOC AGWFS=0.000
DONE: AGWFILT AGWFILT=1 AGWFNAME='Clear'
*** ERROR: MSELECT MSELECT=PWRFLR MSELECT has been power cycled and must be reset to initialize
*** ERROR: MINSERT MINSERT=PWRFLR MINSERT has been power cycled and must be reset to initialize
DONE: DICHROIC DICHROIC=3 DICHNAME='Blue'
...
Done - if any errors or timeouts, correct/reset as needed.

If there are any timeouts

    mods1data:mods% isisCmd m1.ie hatch
    *** ERROR: HATCH HATCH=TIMEOUT cannot write to 192.168.139.101:8001

you may need to reboot the comtrol port. After rebooting any of the comtrol ports relating to the blue or red IEBs, you will need to restart the IE service. (Presumably after rebooting the comtrol ports relating to the IMCS, the IMCS service will need to be restarted.) If there is still a timeout, perhaps there is a faulty microLynx controller. The first less invasive procedure to recover functionality may be to swap the cable to a spare microLynx controller. This page which records the current status of the MODS MicroLYNX controllers and service log has a table of the controller assignments. The MODS team must be involved in this work. As of August 2023, the microLYNX service procedures (uploading code to the MLC) have not be documented so this is something that Rick Pogge or Dan Pappalardo needs to do.

Run the coldStart script to initialize mechanisms

This takes a few minutes, so make sure the telescope will not be moving for the duration. Run /lbt/lbto/mods/modsScripts/Support/modsNcoldStart.pro where N = 1 or 2 to initialize and set mechanisms inside modsN.

Check the AGw guide and WFS cameras

Make sure the guide and WFS camera controllers are on (example commands to query and turn on or off are as follows):

1. isisCmd --mods1 m1.ie agc to query the status of guide camera in MODS1.
2. isisCmd --mods2 m2.ie wfs on will turn on the WFS camera in MODS2.
3. isisCmd --mods1 m1.ie agc off will turn off the guide camera in MODS1.

To take guide and WFS images directly from the azcamserver, independently of which instrument is currently authorized:

1. vncviewer azcamserver. This will open a window to the azcamserver PC where you'll see 16 terminal windows with the titles 1g,1w,... one for each of the 8 AGws. For 5g and 5w are the guide and WFS cameras for MODS1 and 6g and 6w are for MODS2.
2. User the exposure.expose() command to take an image:
   • exposure.expose() will use the previous exposure time and imagetyp settings.
   • exposure.expose(2.0,'object') will take a 2-sec exposure with imagetyp 'object'.
   • exposure.expose(0.0,'zero') will take a bias.

Some error messages you may see (more at Running the AzCam Server):

1. Controller Reset error, check power and fibers
   • Is the camera on? Are the fibers OK?

Basic Functionality

1. Hatch open/close, Calib In/Out;
2. Mask select, Dichroic select, Blue Grating Turret, Blue Filter Wheel, Red Grating Turret, Red Filter Wheel
3. AGw
   • Home
   • Using "gprobe" commands in the GUI command window, move it and watch. Current position only updates when the Update button at the lower left is clicked.
     • Probe operational limits: X = +/- 87 mm and Y = 0 to -180mm (see IT 7736)
       • on-axis: gprobe 0,0
       • gprobe -87,0
       • gprobe 87,0
       • gprobe 0,-180

sieve/imSnap checks

1. sieveSnap
2. imSnap (same as sieveSnap but with imaging mask)
   • any sign of condensation, dust, debris?

IMCS TTF ZeroPoints

1. Running imcsTools (ssh -X mods@modsNdata; imcsTools start) will allow you to watch as IMCS lock is achieved.
2. Follow this procedure for determining and updating MODS IMCS TTF zeropoints

Check camera focus settings using sieve and pinhole masks.

Collect Basic Calibrations for Reference

It is valuable to collect calibration data for all modes, to have these available in the archive for observers who may not have the chance or may miss taking these calibrations during their run. Telescope needs to be stationary and at zenith for flats and comparison lamp spectra and the dome needs to be dark dome for flats and biases. Not all need to be collected at once and this is not a prerequisite for nighttime checkout.

1. Biases for all ROIs: 8Kx3K, 3Kx3K, 4Kx3K and 1Kx1K
2. Slitless pixel flats: Run grpixflat scripts for dual, red-only and blue-only modes: check that exposure times are OK, especially if any of the VFLAT or QTH lamps have been replaced.
3. Comparison lamps (grating): Run grlamps scripts for dual, red-only, blue-only. Again, check count levels (a few lines may saturate).
4. Imaging flats: Run imflats scripts for dual, red-only, blue-only.
5. Comparison lamps (prism) Run the direct and dual mode prism comparison lamp scripts.
   • The direct-mode scripts use just one lamp at a time and are, therefore, a good check that all the lamps are working.
     • /lbt/lbto/mods/modsCalibs_OT/
     • prlamps_b_m1.txt and prlamps_b_m2.txt: Hg, Ar, Kr, Xe
     • prlamps_r_m1.txt and prlamps_r_m2.txt: Ne, Ar, Kr, Xe
     • prlamps_m1.txt and prlamps_m2.txt: Hg+Ne, Hg+Ar, Kr, Xe

Basic on-sky checkout

1. AGw
   • Check/adjust position of hotspot on guide camera
     • BGTech->adjust hotspot
   • Confirm/adjust AGw on-axis position to align center of rotation on guide camera with hotspot
     • BGTech->align hotspot to rotator center
2. Rotator Centers (on MODS detectors)
   • Collimate and then send track presets to a dense field at PA=0, 180 and PA=90, 270. Use the pointing mask to allow a correction for flexure.
     • OLD: Analyze data with modsrc. modsrc works on both MODS and LBC images and it computes rotator centers for pairs of images which do not have to be taken at diametrically opposed position angles but can be, say, 90 deg apart instead.
     • NEW: For now, I assess flexure and, if it has only been a few pixels, I coadd the PA=0,180,90,270 images and determine the center of rotation on the detector by displaying the coadded image in ds9 and drawing lines between the same stars at diametrically opposite PAs.
     • The rotator centers x, y are entered as CRPIX1,2 in the WCS addition to the MODS headers, so Y Zhang should be informed of any changes.
3. Rotator Zero Points
   • Acquire a MOS mask and note the delta theta offset needed to align it. Do this for 2 astrometric masks. The rotator zero point will be set (in PCSInstrument.conf) so that the mask or longslit is oriented at PA=0. The delta theta by which the instrument needs to be rotated to align the mask and field is the error in the rotator zeropoint. If the rotator zeropoint is updated, then a new transform needs to be determined.
4. Transform:
   • At minimum, check the existing transform by making a series of DETXY offsets with large (60") throw to compare orientation of deltaX, deltaY with slit.
     • Reduce these data with a script, (modsdotcircle.py, currently only in Olga's ~/Dropbox/MODSdither)
   • Full transform data collection (https://wiki.lbto.org/bin/view/Commissioning/ComTools#Guide_Probe_Transformation)
5. Observe a standard star in all modes, grating and prism (30-45 min for 3 images). If clear, obtain the full set of 3 images per mode; if not, make a note in the object name and obtain only 1 image per mode.
6. Observe a photometric imaging field in dual and direct modes. M13, M71, M92, (add other good ones from OT ...)
7. Dice-5 imaging offsets
   • low priority check on orientation and scale of DETXY offsets, repeatability of these and also an opportunity to gather data on image quality.