MODS Post-Shutdown Startup (Under construction!)

The following is a procedure for restarting MODS1 or MODS2 after it is installed, hooked up and released to science operations.

Are the IUBs, IEBs and COMTROL servers online? (pingMODSN)

  • Run pingMODSN from modsNdata to check the status of the IUB, IEB and COMTROL ports. If they report that they are offline, check:
    • that the IUB is plugged in at the instrument. There is no remote switching of the IUB.
    • The IEBs may be off. Check the IEB status by isisCmd --modsN mN.ie util ieb_b or isisCmd --modsN mN.ie util ieb_r, and if one or the other of the IEBs is off, you'll need to turn it on by (in this example for MODS1 and ieb_b, but replace 1 with 2 and b with r for other combinations). (I always turn it off explicitly before turning it on, to clear any BRK=FAULT messages).
               isisCmd --mods1 m1.ie util ieb_b off
               isisCmd --mods1 m1.ie util ieb_b on
      After turning on the IEB, the IE server will need to be restarted.
               mods1 start ie

Turn on HEBs and start IC programs to monitor dewar temperatures and pressures during cooldown

  1. Start up the isis, ie, agw and env services.
  2. After confirming that glycol is flowing, turn on the HEBs. One way to double-check the glycol flow at MODS is by checking the glycol supply/return pressures and temperatures in the env log which is on the modsN computer in ~/Logs/Env/. The glycol temperature should be just a couple degrees below ambient, and the supply/return pressures are ~40 psi for both MODS (the supply pressure sensor on MODS2 is broken). Samples of mods@mods1:~/Logs/Env/mods1.20171025.log and mods@mods2:~/Logs/Env/mods2.20171025.log are below:
             Sample of mods@mods1:~/Logs/Env/mods1.20171025.log:
             # UTC Date/Time      Tamb  Psup  Pret  Tsup  Tret  Tiub  Tagw  Bair  Bret  Rair  Rret  AirT  AirB  ColT  ColB
             2017-10-25T00:00:07  12.8  47.2  45.9  12.6  13.1  13.5 -101.6  16.1  12.8  15.6  12.6  11.3  11.7  11.6  10.3
             2017-10-25T00:01:07  12.7  47.2  45.8  12.7  13.1  13.6 -102.2  16.1  12.8  15.6  12.6  11.2  11.8  11.6  10.3
    
             Sample of mods@mods2:~/Logs/Env/mods2.20171025.log:
             # UTC Date/Time      Tamb  Psup  Pret  Tsup  Tret  Tiub  Tagw  Bair  Bret  Rair  Rret  AirT  AirB  ColT  ColB
             2017-10-25T00:00:49  13.3 100.0  43.4  12.5  15.3 -10.4  16.0  16.4  12.8  16.0  12.9  12.3  11.8  13.2 -14.1
             2017-10-25T00:01:49  13.3 100.0  43.4  12.4  15.3 -10.4  16.1  16.4  12.8  16.0  12.8  12.3  11.8  13.3 -14.0
  3. Start the IC programs on M1.BC, M1.RC, M2.BC and M2.RC.
  4. isisCmd --modsN mN.rc estatus
             lbto@obs4:2 % isisCmd --mods1 m1.rc estatus
             DONE: ESTATUS TEDPOWER=ON LEDPOWER=OFF IGPOWER=ON DEWPRES=7.1600E-06 CTEMPIN=9.35 CTEMPOUT=9.81 HSTEMP=12.51      HEBTEMP=17.11 +24V=23.81 +5V=4.96 CCDTEMP=-99.47 -15V=-14.98 DEWTEMP=-196.73 +12FAN=11.77 BOGHTRV=2.44 +3.3V=3.29
  5. Are TEDPOWER and IGPOWER ON? Is LEDPOWER OFF? If not, turn them on and off:
             isisCmd --mods1 m1.rc tedpower on
             isisCmd --mods1 m1.rc igpower on
             isisCmd --mods1 m1.rc ledpower off
  6. Turn back on cron jobs --- to record temperatures every minute during cooldown and every 30 minutes during operation.
    1. log in to telescope@obs3.mountain.lbto.org (must be obs3)
    2. "crontab -e" to edit the cron jobs in "vi".
               ## twice an hour check the MODS temps, set alarm conditions
               15,45 * * * * /home/telescope/bin/emailMODSErrors.sh  > /dev/null 2>&1
               20,50 * * * * /home/telescope/bin/emailMODS2Errors.sh  > /dev/null 2>&1
               ## while cooling, we checked the MODS1 temps every 1 minute
               #*/1 * * * * /home/telescope/bin/emailMODSErrors.sh  > /dev/null 2>&1
               #*/1 * * * * /home/telescope/bin/emailMODS2Errors.sh  > /dev/null 2>&1
               ## each day check ALH log for instrument complaints
               10 14 * * * /home/telescope/bin/emailInstrAlarms.sh  > /dev/null 2>&1 

Can we communicate with the mechanisms? (pokeMODSN)

Following is a sample of the output when communication is fine. The messages refer to the initialization that is needed and is the next step.
mods1data:mods% pokeMODS1
Poking MODS1 Mechanisms...
Common focal plane:
DONE: HATCH HATCH=CLOSED
DONE: CALIB CALIB=OUT
DONE: AGWX AGWXS=0.000
DONE: AGWY AGWYS=0
DONE: AGWFOC AGWFS=0.000
DONE: AGWFILT AGWFILT=1 AGWFNAME='Clear'
*** ERROR: MSELECT MSELECT=PWRFLR MSELECT has been power cycled and must be reset to initialize
*** ERROR: MINSERT MINSERT=PWRFLR MINSERT has been power cycled and must be reset to initialize
DONE: DICHROIC DICHROIC=3 DICHNAME='Blue'
...
Done - if any errors or timeouts, correct/reset as needed.
If there are any timeouts
    mods1data:mods% isisCmd m1.ie hatch
    *** ERROR: HATCH HATCH=TIMEOUT cannot write to 192.168.139.101:8001
you may need to reboot the comtrol port. After rebooting any of the comtrol ports relating to the blue or red IEBs, you will need to restart the IE service. (Presumably after rebooting the comtrol ports relating to the IMCS, the IMCS service will need to be restarted.) If there is still a timeout, perhaps there is a faulty microLynx controller. The first less invasive procedure to recover functionality may be to swap the cable to a spare microLynx controller. This page which records the current status of the MODS MicroLYNX controllers and service log has a table of the controller assignments. The MODS team must be involved in this work. As of November 2017, the microLYNX service procedures have not be documented but will be in the near future.

Run the coldStart script to initialize mechanisms

This takes a few minutes, so make sure the telescope will not be moving for the duration. Run ~modseng/modsScripts/Support/modsNcoldStart.pro where N = 1 or 2 to initialize and set mechanisms inside modsN.

Check the AGw guide and WFS cameras

Make sure the guide and WFS camera controllers are on (example commands to query and turn on or off are as follows):
  1. isisCmd --mods1 m1.ie agc to query the status of guide camera in MODS1.
  2. isisCmd --mods2 m2.ie wfs on will turn on the WFS camera in MODS2.
  3. isisCmd --mods1 m1.ie agc off will turn off the guide camera in MODS1.
To take guide and WFS images directly from the azcamserver, independently of which instrument is currently authorized:
  1. vncviewer azcamserver. This will open a window to the azcamserver PC where you'll see 16 terminal windows with the titles 1g,1w,... one for each of the 8 AGws. For 5g and 5w are the guide and WFS cameras for MODS1 and 6g and 6w are for MODS2.
  2. User the exposure.expose() command to take an image:
    • exposure.expose() will use the previous exposure time and imagetyp settings.
    • exposure.expose(2.0,'object') will take a 2-sec exposure with imagetyp 'object'.
    • exposure.expose(0.0,'zero') will take a bias.
Some error messages you may see (more at Running the AzCam Server):
  1. Controller Reset error, check power and fibers
    • Is the camera on? Are the fibers OK?

Basic Functionality

  1. Mask select, Dichroic select, Blue Grating Turret, Blue Filter Wheel, Red Grating Turret, Red Filter Wheel, calib unit, hatch, agw
    • Move by hand using script commands. Does it go to position?
      • agw range (home position is at one corner of patrol field).
    • Lamps: Do these all go on? A check of this would be to run the scripts to obtain comparison lamp spectra; one uses two lamps, but we could separate this. (Running the already set lamp scripts will be OK for now, but we could make a script specifically for checkout).
  2. with imcsTools running, does the IMCS lock?

Basic Calibrations for checks

  1. sieveSnap
  2. imSnap (same as sieveSnap but with imaging mask)
    • any sign of condensation, dust, debris?

IMCS TTF ZeroPoints

  1. Running imcsTools will allow you to watch as IMCS lock is achieved.
  2. Follow this procedure for determining and updating MODS IMCS TTF zeropoints

Collect Requested Calibrations

Obtain calibrations that have been specifically requested.
  • 2017 October: Need MODS2 calibrations for the pipeline.
    • Scripts are in ~modseng/modsScripts/Testing/
      • pinholeSlit.cal
      • sudokuMask.cal
      • pr_pinhole.cal
      • pr_sudoku.cal

Basic Calibrations for Reference

It is valuable to collect calibration data for all modes, to have these available in the archive for observers who may not have the chance or may miss taking these calibrations during their run. Telescope needs to be stationary and at zenith for flats and comparison lamp spectra and the dome needs to be dark dome for flats and biases. Not all need to be collected at once and this is not a prerequisite for nighttime checkout.
  1. biases in all modes: 8Kx3K, 3Kx3K, 4Kx3K and 1Kx1K
  2. Run grpixflat scripts for slitless pixel flats: in order of priority, dual, red, blue: check that exposure times are OK, especially if any of the VFLAT or QTH lamps have been replaced.
  3. Run grlamps scripts for comparison lamps: in order of priority: dual, red, blue. Again, check count levels.
  4. Run imflats scripts (time estimate...)
  5. Check camera focus settings using sieve and pinhole masks (lengthy task...done for the first time after commissioning last Spring. Plans to check in Nov 2017).

Basic on-sky checkout

  1. Rotator Centers
    • Collimate and then send track presets to the same field at PA=0, 180 and PA=90, 270. Analyze data with modsrc. modsrc works on both MODS and LBC images and it computes rotator centers for pairs of images which do not have to be taken at diametrically opposed position angles but can be, say, 90 deg apart instead.
  2. Rotator Zero Points
    • Acquire 2 MOS masks and note the delta theta offset needed to align the mask. Ideally, the rotator zero point will be set (in PCSInstrument.conf) so that the mask or longslit is oriented at PA=0. The delta theta by which the instrument needs to be rotated to align the mask and field is the error in the rotator zeropoint. If the rotator zeropoint is updated, then a new transform needs to be determined. A 0.7 deg error in the rotator zeropoint translates to 0.73" at a distance 60" from the slit center. (If one of two stars separated by 60" is perfectly centered in a long-slit slit, the other will be on one edge of the slit unless the field is rotated by 0.7 degrees; since MOS mask alignment computes a rotational offset as well as a shift, the error in teh rotator zeropoint is corrected during a MOS alignment).
  3. Transform:
  4. Observe a standard star in all modes (30-45 min for 3 images). If clear, obtain the full set of 3 images per mode; if not, make a note in the object name and obtain only 1 image per mode.
  5. Dice-5 imaging offsets
    • low priority check on orientation and scale of DETXY offsets, repeatability of these and also an opportunity to gather data on image quality.

-- %USERSIG{OlgaKuhn - 2017-10-24}%


 
Topic revision: r13 - 20 May 2019, OlgaKuhn
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback