Running experimental TMS software

This page outlines the steps needed to run the TMS (Telescope Metrology System) experimental software developed beginning from the summer of 2020 until the end of 2022.

Description

The current (circa end of 2022) TMS software system is a collection of command-line programs and scripts. It allows taking TMS raw measurements and then it calculates the appropriate corrections taking into consideration various TCS and LBC instruments parameters. The final correction can then be sent to the TCS to be applied. There are various modes and configurations to run the TMS software collection and this page explains them and details the steps to do that.

As a note, this is a temporary setup for TMS operation. We are in the process of redesigning the entire TMS software system, which will be better in many ways.

TMS Strategies for LBC Observing

There are three different strategies for how to use TMS to improve your LBC observing: prepositioning M1 to shorten your initial collimation time; extending the interval before running FPIA again; extending a single FPIA to cover many positions in the sky. As of end of 2022, all three of these modes are working with reasonable reliability, and have been used on-sky for science observations.

Using TMS to pre-position M1 to shorten your initial collimation

Prepositioning the optics can be a significant savings of time at the start of LBC observation, but using a reference from a previous night may not always give perfect collimation. The idea is to get close, so that FPIA has very little work to do. Beware that the pre-positioning will not work if the TMS collimators have been realigned since the previous reference. The basic sequence is:
  • have someone turn on the TMS Etalon laser (when no infrared instruments are working)
  • copy 4 reference files from a previous night (instructions below)
    • DO NOT take a new reference offset.
  • start the active TMS loop (instructions below) This can be done with the dome closed, or in bright twilight before sky flats.
  • preset to your collimation field (as normal, after flats)
    • allow time for TMS to make one or two corrections at the elevation position of your target
  • Press ^C (Ctrl-C) to stop your active TMS loop
  • run your initial FPIA (as normal) - DOFPIA should be sufficient, as the large coma that requires DOHYBRID should have been taken care of.
  • set a new TMS reference offset (instructions below) if your FPIA gave a good result
  • run the TMS loop again to enable TMS corrections (instructions below)

Using TMS to extend the time between running FPIA

This is the simplest and most standard mode of running TMS with LBCs. The basic sequence is:
  • have someone turn on the TMS Etalon laser (when no infrared instruments are working)
  • preset to your collimation field (as normal)
  • run FPIA (as normal)
  • set a TMS reference offset (instructions below)
  • start the active TMS loop (instructions below, note that TMS will not apply a correction when the primary is in the extrafocal position.)
  • copoint and observe as normal, except you need to do FPIA less often.
  • if you decide to run FPIA again, then you must enter ^C (Ctrl-C) to stop your active TMS loop (at least by the time FPIA finishes)
  • after a new FPIA collimation, you MUST set a new TMS reference offset see instructions below (or TMS will remove the corrections that FPIA just put in)
  • restart the TMS correction loop after the new FPIA (see instructions below)

TMS measures the distance between the primary mirror and the LBC hub. TMS software also measures backplate-faceplate temperature of the primary mirror to estimate the change in focus since the time of the reference offset (if that reference offset was from today). TMS does not yet predict Z11/Z22 spherical aberration that may be needed on the primary (we are working on it), but it does tolerate any Z11/Z22 that you may apply.

Using TMS to extend a single FPIA to multiple positions on the sky

This is particularly useful for flux standards, or for nights with poor seeing (where you want decent collimation, but do not need the absolute best collimation). If active TMS is already running, the basic sequence is:
  • preset to your new field (as normal)
  • allow time for TMS to make one or two corrections at the elevation position of your target (TMS is faster than running FPIA, but is not infinitely fast. This probably takes 1-1.5 minutes.)
    • We recognize that staring at the PSFGUI or at LBTplot to learn when TMS applied corrections is inconvenient. We've created watch_tms_left.py and watch_tms_right.py to tail the TMS log files and show you when new corrections happened.
  • copoint, and continue with your observations

Prerequisites

In order to run the TMS software, you will need the following items:
  1. Have access to the machine "robs.mountain.lbto.org" as user "telescope"
  2. Have both (or one of the) LBCs authorized and in beam (depending on which side you use).
  3. Have the primary mirror in a "collimated" position.
  4. Have the Etalon multiline TCP server turned on.
  5. Have the Etalon laser on and ready.
  6. Know that scripts live in /lbt/lbto/supportscripts/MetrologySupport/

The main TMS programs to use

There are two TMS programs that you would normally use. One for setting a reference offset, and the other for starting the TMS correction loop.

Setting TMS reference offset

The program lbc_set_tms_offset.py under the directory /lbt/lbto/supportscripts/MetrologySupport/ on machine telescope@robs.mountain.lbto.org is the one you use to set a new TMS reference offset. It is relatively simple to use. It accepts the option -s, which specifies the sides of TMS you want a reference to be set. Use "-s 2" for bino TMS, "-s 0" for SX TMS and "-s 1" for DX TMS ".

The following examples show how you would normally use it to set TMS references. Suppose you are logged in as "telescope@robs.mountain.lbto.org", and you have opened a terminal windows, "$" is the Linux shell prompt:

$ /lbt/lbto/supportscripts/MetrologySupport/lbc_set_tms_offset.py -s 2 # set a new TMS reference for both sides
$ /lbt/lbto/supportscripts/MetrologySupport/lbc_set_tms_offset.py -s 0 # set a new TMS reference for SX only
$ /lbt/lbto/supportscripts/MetrologySupport/lbc_set_tms_offset.py -s 1 # set a new TMS reference for DX only

Occasionally you may want to disable certain channels from the reference offset (maybe because those channels do not work very well and reliably for the moment). To do so, you will need to edit the script lbc_set_tms_offset.py. Open this file and find the line that looks exactly like:

#tmspy += '--drop 8:1,11:1,22:1'

First remove the starting "#" character (to uncomment the line so that it takes effect). And then change the "--drop ..." string to disable the channels you want. The part "8:1" means channel 8 will be disabled. If you have multiple channels to disable, then place additional ones there separated by a comma "," (make sure there is no space in-between the components). For example, if you want to disable channel 5 and 23, then it will look like this:

tmspy += '--drop 5:1,23:1'

Remember to put back the "#" character at the start of the line if you don't want to use this feature. A safe way is to perhaps duplicate the line and make your changes in the new line:

#tmspy += '--drop 8:1,11:1,22:1'
tmspy += '--drop 5:1,23:1' # possibly include a reason to disable them, and remember to remove it when no longer needed

You can simply delete your line when you no longer want this feature. In this way, the original line is always there and always the same.

Running the TMS correction loop

The program tms_loop.py under the directory /lbt/lbto/supportscripts/MetrologySupport/ on machine telescope@robs.mountain.lbto.org is the current TMS correction loop entry program. Normally you can simply just run the program as is (assuming you have logged in as telescope@robs.mountain.lbto.org and in a terminal window):

$ /lbt/lbto/supportscripts/MetrologySupport/tms_loop.py # start TMS active loop with default options

If you run the above, then it starts the TMS active loop (i.e., TMS corrections sent to the telescope) on both sides with missing channel processing mode enabled (the missing channel mode refers to a configuration where the TMS software automatically corrects a TMS measurement that has missing channel data, with this mode disabled, TMS simply refuses to calculate a correction when receiving a measurement with missing channel data and only calculates a correction on TMS measurements with full channel data compared to the reference). Also TMS correction will not be sent while LBC is taking images. If you need more control, you should specify options on the command line. The program tms_loop.py supports the following options:

  • -h, --help: remind you what options you can have
  • --version: show the current version number and exit (current version is v0.2 dated Dec 16, 2022)
  • -s: specify the side TMS will run, 0 represents SX, 1 represents DX, 2 represents both side, default is 2.
  • -o: if specified, TMS correction will ignore the LBC shutter state (i.e., TMS corrections may be sent during LBC exposure)
  • -e: if specified, TMS correction will ignore elevation on source
  • -x: if specified, TMS correction will ignore extrafocal position
  • -p: turn on passive mode, passive mode is a mode where TMS only measures and calculates corrections, but will NOT send them
  • --stdevpop: should be followed by an integer to specify the standard deviation population size (used in filtering successive TMS measurements), default is 3
  • --stdevbnd: should be followed by a floating point number to specify the standard deviation filtering threshold in micron (used in filtering successive TMS measurements), default is 5.0
  • --mcp: specify missing channel mode, the following options are supported:
    • none: disable missing channel mode
    • sx: only enable missing channel mode on SX side
    • dx: only enable missing channel mode on DX side
    • both: enable missing channel mode on both sides (both is the default)
  • -v: turn on verbose mode, this is a stacked option, i.e., using more "v"s will increase the verbosity level, e.g., -vvv is more verbose than -vv. Normally you should NOT change this option. The default level is -vvv, which should be adequate for most cases.

You can always press Ctrl-C to stop the TMS loop.

Here are some of the examples you can run:

  • If you only want to observe the TMS corrections but do not want them applied to the telescope, you should run in passive mode:
    $ /lbt/lbto/supportscripts/MetrologySupport/tms_loop.py -s 1 -p
       
    The above command runs the TMS correction loop in passive mode on the DX side. Currently running passive mode on one side and active mode on the other side is not supported. However a software patch can make this happen. We plan to support this feature in the future TMS software.

  • If you want to apply TMS corrections regardless of whether LBC is taking images:
    $ /lbt/lbto/supportscripts/MetrologySupport/tms_loop.py -o
       
    Using the "-o" option would enable this. However normally this is only useful during engineering tests. In normal science operations, you should not use this option (just to be safer).

  • If for some reason, you suspect the missing channel mode is giving you lots of troubles (e.g., LBC images have unexplainable jumps or trailing), you can disable the missing channel mode:
    $ /lbt/lbto/supportscripts/MetrologySupport/tms_loop.py --mcp none  # disables missing channel mode on both sides
    $ /lbt/lbto/supportscripts/MetrologySupport/tms_loop.py --mcp sx    # only enable missing channel mode on SX
       

  • You can also change the standard deviation filtering parameters:
    $ /lbt/lbto/supportscripts/MetrologySupport/tms_loop.py --stdevpop 5 --stdevbnd 3.5
       
    The above options set the standard deviation filtering population to be five (i.e., five successive TMS measurements will be used to compute a standard deviation), and the threshold to be 3.5 microns (i.e., TMS channel data will be filtered if their standard deviation is above 3.5 microns). Normally you should not change them as the default values are tuned to be working well in most cases.

Reusing a previous TMS reference position

If you have run TMS software previously and would like to start your current session based on a previous reference position, you can also do so (although in a slightly cumbersome way). First of all, you will need to identify the reference position you want to use. This is typically the last reference position on one of your previous TMS runs. When you identify the date, you will then go to your TMS data collection on that night (currently in the directory: /home/telescope/TMS_logs/yyymmdd/) and find these files:
  • Reference position files for left side (SX):
    • telescope_left_tmsoffset.dat
    • tms_sx_ref.dat
  • Reference position files for right side (DX):
    • telescope_right_tmsoffset.dat
    • tms_dx_ref.dat

Make sure you are logged in as telescope@robs.mountain.lbto.org as these files come with specific user permissions. You may have trouble reading/copying them if you are under a different user.

There are basically two reference position files for each TMS side. When you find them, copy them to the /tmp directory on robs.mountain.lbto.org. And then when you start your TMS session, remember that you DO NOT need to run the reference step anymore. Then everything else is just the same as before. Of course, you can also reuse the reference position just for one side. To do so, just copy the files for the side you want to /tmp, and then remember to skip the reference step for the side you want the ref position to be reused. E.g., if you want to reuse the SX side ref position, then copy telescope_left_tmsoffset.dat and tms_sx_ref.dat from the date of your choice to /tmp. And then run the reference step for the DX side only (e.g., lbc_set_tms_offset.py -s 1).

Plots

If you run LBTplot while observing with LBC, you will see three new (since Oct 2021) plot buttons.

sxTMS and dxTMS are rather busy plots that show what TMS is doing. These only work if LBTplot is running on the same machine as TMS (usually robs), because they read log data files from /tmp/.

Things that look more like discreet points (diamonds or squares) are the (active or passive) TMS corrections (Group Rotation), and things that look more like continuous lines (many dots close together) are from PSF telemetry (Instrument Offsets, and Mode3 Pointing).

Back-F plots the backplate-faceplate temperature of the mirrors converted to a focus correction. The value of the Z4 correction is relative to the beginning of the plot. TMS uses the routine that makes this plot to apply the focus correction since the last reference was set (different than the timescale of your plot).

Collecting data on TMS

When you are done with running the TMS software, you should copy the data files produced during the run for later analysis and study. These data files are being placed in the "/tmp" directory and therefore are not guaranteed to always be there. Since Oct 2021, this is automatically handled by the automated Daily Transfer Service.

Here is a brief description of the important files produced by the TMS software (just for your reference, if you do not analyze TMS results and performance, then you do not need to look into any of them):

  • telescope_left_tmsoffset.dat and tms_sx_ref.dat: the TMS reference position for the SX side
  • telescope_right_tmsoffset.dat and tms_dx_ref.dat: the TMS reference position for the DX side
  • tms_left_yyyymmdd_telescope.txt: the final TMS corrections calculated for the SX side
  • tms_right_yyyymmdd_telescope.txt: the final TMS corrections calculated for the DX side
  • tms_tcp.log: the raw TMS measurement and pre-processing data recorded
  • tms_screen.out: the entire screen copy of what is produced by the program tms_loop.py

Daily data transfer service details

Currently an automatic data transfer service is set up on the machine robs.mountain.lbto.org. The service runs each day at 8:00AM Tucson time (or 15:00 UTC). It basically tries to search for all the TMS related files in the /tmp directory. And if it finds any, it will attempt to transfer them to the location: /home/telescope/TMS_logs/d, where d is a new directory created by the transfer service. d is usually named using the current UT date (e.g., 20211015).

If for any reason, d already exists inside /home/telescope/TMS_logs, then the transfer service will attempt to look inside of it and try to figure out if there are any files inside that are equivalent to the ones in the /tmp directory. It will skip transferring these files. If for any reason, a file f in /tmp also appears in existing d directory, and that the two versions of f (the one in /tmp and the one in d) have different contents, then the transfer service will attempt to transfer f to d with a new name. The new name is typically f.v2 (however if that is taken, then it will try f.v2.v2, etc. until the name is unique inside d). This renaming transfer of f also applies to the case when f inside d is not readable to the transfer service (e.g., perhaps due to permission issues).

If for any reason, d already exists inside /home/telescope/TMS_logs, and it is entirely not accessible to the transfer service (e.g., due to permission issues), then the transfer service will attempt to transfer everything to d.v2 (likewise, if d.v2 is already taken, it will try d.v2.v2 etc. until the name is unique).

So in short, the transfer service will never attempt to overwrite any files already inside /home/telescope/TMS_logs. Also the service is a transfer service, meaning after it is done, the original files inside /tmp will be removed. This is to give the next TMS run a fresh start. If for some reason, you wish to reuse a previous TMS reference position, then you will have to manually pick the ones you like to use, and then place the necessary files back into the /tmp directory before your TMS run. Please see the section on running TMS with a previous reference position for details.

Lastly, the location /home/telescope/TMS_logs is perhaps just marginally safer than /tmp. The long term storage of any data on robs.mountain.lbto.org is not guaranteed. Therefore you should still try to back up your TMS data somewhere else (that is safer) whenever you have the chance.

Extracting data

The data you collected (as described in the previous section) are in a condensed form. They may not be immediately useful to you. There is a utility program you can use to extract and reformat the data as CSV files that will be much simpler to process and analyze. The most important data files you collected will be tms_<left|right>_<timestamp>_telescope.txt (which contains the final TMS corrections calculated) and tms_tcp.log (which contains all the information from the Etalon multiline server as well as local pose calculation related information). The extraction utility program is tms_extract.py located under the same TMS directory /lbt/lbto/supportscripts/MetrologySupport on robs.mountain.lbto.org. It will accept these two types of input files.

It is simple to use. There are two main options: '-t' for extracting "tms_tcp.log" type information; and '-c' for extracting TMS final corrections contained within "tms_<left|right>_<timestamp>_telescope.txt". And an optional option '-o' lets you specify a string to be used as the prefix for all the output files. We will give some examples how you can use it.

$ /lbt/lbto/supportscripts/MetrologySupport/tms_extract.py -t /path1/tms_tcp.log /path2/tms_tcp.log -o tms
$ /lbt/lbto/supportscripts/MetrologySupport/tms_extract.py -c tms_right_20201109_telescope.txt tms_right_20201030_telescope.txt
$ /lbt/lbto/supportscripts/MetrologySupport/tms_extract.py -c tms_right_20201109_telescope.txt tms_right_20201030_telescope.txt -t tms_tcp.log -o tms.test

The '-o' option is optional and it accepts an arbitrary string. If specified, the string is used as the prefix of the output files. For example if '-o tms' is given, then the output files will be named as "tms.raw", "tms.correction" and so on. If not specified, then the output file names will be automatically prefixed with 'tms.<b>-<e>' where '<b>' is the timestamp of the first record and '<e>' is the timestamp of the last record in the files given. An example of the automatically generated file name could be: "tms.20201109.200423.104043-20201110.120402.763123.raw". The auto-generated names tend to be quite long and verbose. So if you don't like them, then you should supply the '-o' option and name them yourself.

Both options '-t' and '-c' take a list of corresponding files (can be one as well). As shown you can also use either option alone. The "tms_tcp.log" files contain full timestamp with every record inside so you can actually rename the input files however you like. But the "tms_<left|right>_<timestamp>_telescope.txt" files only contain partial timestamps inside and with the rest of the timestamps encoded in their names (e.g., "tms_right_20201030_telescope.txt"). The extraction program will need both types of timestamps to properly sort the data and so do not remove the timestamps in the file name, or the utility program will complain about that and refuse to proceed. Also please do not remove the original "left" or "right" strings embedded in the file names as they indicate the side channel group that they represent. All supplied input files will be read and all valid data contained inside will be collected and sorted on the timestamps. The order of files on the command line does not matter since the program will do the sorting itself.

The '-t' option generates the following files (assuming with '-o tms' option):
  • tms.raw: the raw channel measurement length from the Etalon multiline server.
  • tms.etalon: the Etalon corrected channel length data.
  • tms.ciddor: our own corrected channel length data (with LBT weather telemetry).
  • tms.pose.sx and tms.pose.dx: the calculated pose data (i.e., x,y,z,rx,ry,rz). Depending on which TMS side you use, you may only get one of these, or both.
  • tms.stdev: the calculated standard deviation on each channel. The standard deviation is currently calculated on three most recent channel measurements at each given timestamp. The unit is in micron.
  • tms.stdev.filter: the channel filtering based on standard deviation for each vector. Basically if a channel's value is 0, then it is not filtered; if instead a channel has 1, then it is being filtered because of exceeding the standard deviation threshold. For convenience, the first data row in the file is a summation of each column (i.e., how many times each channel has been filtered by standard deviation).
  • tms.stdev.filter.sx and tms.stdev.filter.dx: these are side specific standard deviation filtering results. They only record all the pose calculation channels on each side when they are not all filtered out. Basically if all pose channels on a side are filtered out, then perhaps it is due to errors in the eTalon software, or intended large mirror movement. And when no pose channels are filtered out, then it is basically not interesting to look at them. So these files give an idea of the stability of all the pose channels. Basically the more a channel is filtered out (the first row number), then the less stable it is. You can also use these information to decide whether or not to manually disable certain channels.
  • tms.drop: the intentional exclusion of particular channels. If a channel in a vector is 0, then it is not excluded intentionally; if a channel has value 1 in its vector, then it is being dropped intentionally. For convenience the first data row in the file calculates the summation of each column (i.e., total excluded times for each channel).
  • tms.ref.sx and tms.ref.dx: these are timestamps when a reference is made. You can cross reference these timestamps in other files to find out informations on the references (e.g., reference the tms.raw file to find out the raw values and the tms.pose.sx and tms.pose.dx to find out the reference pose calculated).
  • tms.weather.lbt: the LBT weather telemetry we used.

The '-c' option generates the following files (assuming the '-o tms' prefix option given as well as that you supplied correction data from both sides):
  • tms.correction.sx and tms.correction.dx: the final TMS correction vectors computed (the ones actually sent to the telescope).
  • tms.thermalz4.sx and tms.thermalz4.dx: the thermal Z4 compensation on the primary mirrors (in mm unit).

All of these generated files are CSV style data with a timestamp followed by corresponding data items (the first line is a CSV header that explains the names of each field). If there are any data items that are not valid or present, then 'none' will be displayed in these files. Notice if a particular file is empty, then that is because there is no information related.

If you wish to do the extraction on your own machine, then you can copy the "tms_extract.py" and the "config.py" files to your own machine (and put them in the same directory). You will need a recent Python 3 environment installed to be able to run it (recommended to be at least Python 3.6). Other than the standard Python 3 environment, it does not depend on anything else.
Topic revision: r40 - 27 Apr 2023, OlgaKuhn
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback