Running experimental TMS software

This page outlines the steps needed to run the TMS experimental software developed during the summer and fall of 2020, and spring of 2021.

Description

The current (circa summer and fall 2020, and Spring 2021) TMS software system is a collection of command-line programs and scripts. It allows taking TMS raw measurements and then it calculates the appropriate corrections taking into considerations of various TCS and LBC instruments parameters. The final correction can then be sent to the TCS to be applied. There are various modes and configurations to run the TMS software collection and this page explains them and details the steps to do that.

Prerequisite

In order to run the TMS software, you will need the following items:
  1. Have access to the machine "robs.mountain.lbto.org" as user "telescope"
  2. Have LBCR authorized and in beam.
  3. Have the primary mirror in an intended reference position.
  4. Have the TMS laser on and ready.
  5. Have the Etalon multiline TCP server turned on.

TMS Modes

There are several different modes you can run the TMS software and most modes have their own shortcut convenience script associated. When you decided to run a particular mode, then you should invoke the corresponding script (more on the steps in later sections, this section explains the modes and their scripts).

  1. Active loop mode: this is the normal mode where we continuously take TMS measurements and apply the corrections to the telescope. The script associated with this mode is: tms_loop_rightonly.pl
  2. Active/Drop loop mode: this is the same as the previous normal mode, i.e., taking TMS measurements and send corrections to telescope, except that we allow artificially induced channel dropping. The script associated with this mode is: tms_loop_rightonly_drop.pl (Note: this mode is currently not used in production.)
  3. Passive loop mode: this mode continuously takes TMS measurements and computes corrections, but does not actually send the corrections to the telescope. The script corresponding to this mode is: tms_loop_rightonly_passive.pl (Note: this mode is currently not used in production.)
  4. Passive/Drop loop mode: this is the passive mode with channel being dropped. The corresponding script is: tms_loop_rightonly_passive_drop.pl (Note: this mode is currently not used in production.)
  5. Active oneshot mode: this mode reads the TMS measurement once and then calculates and sends the corrections once based on the measurement reading. The corresponding script is: lbc_tms.py (Note: this mode is currently not used in production.)
  6. Passive oneshot mode: this mode is similar to the previous one, only that it does not send the corrections to the telescope. The same script lbc_tms.py handles this mode as well (with additional options). (Note: this mode is currently not used in production.)

Currently all modes require that the real measurement data from the Etalon software match the reference vector's channel informatioin (e.g., if the reference vector has nine channels, then all subsequent vectors require the same nine channels to be present). Otherwise they will refuse to proceed to the pose calculation step. The "active/drop" and "passive/drop" modes still allow you to specify which channels that you do no wish to use. However if you drop any channels present in the reference vector, then it will fail to calculate a pose. You can also use the drop option to specifically disable particular channels in the reference vector. This is more involved as editing some of the scripts may be needed. Currently we do not outline this feature here in detail. Also note the two oneshot modes do not use artificial channel droppings (although you could do it, but it is currently not needed and we do not document it in detail here).

Running a Particular Mode

The following steps are needed to run the TMS software in a particular mode. Note that you can only run one mode at a time. Please do not launch multiple instances of the TMS software.

  1. Check to see that you have met all the prerequisites.
  2. Login to the machine "robs.mountain.lbto.org." You can do either of the following:
    • On a Linux/Unix system, open a terminal window and type the following command and press enter: ssh telescope@robs.mountain.lbto.org If you are asked to enter passwords, then please contact LBTO IT for help. Normally you should be able to login without a password from most LBTO machines.
    • If you are using Windows system, or prefer a remote desktop, then you can login using a remote desktop application such as "x2go." Please configure your remote desktop for the machine "robs.mountain.lbto.org" and user "telescope" and desktop "MATE". Note if you are outside of the LBTO network, then you may need to turn on VPN before being able to make connections to the machine "robs.mountain.lbto.org"
  3. Enter the following command and press enter (if you use remote desktop to login to "robs.mountain.lbto.org", then please open a terminal window first): cd /lbt/lbto/supportscripts/MetrologySupport
  4. Once you've done FPIA enter the following command and press enter: ./lbc_set_tms_offset.py -s 1 This step sets a reference pose for the TMS system. It will output some text in the terminal window and will typically take around 30 seconds to complete. If you have failure reported in this step, then please repeat it multiple times. If there is still failure, then you won't be able to continue. Please contact software support for help.
  • Running loop mode:
    1. Then enter the following command and press enter: ./[mode_script] Where [mode_script] is the name of the script given in the previous section. When running, it will output lots of texts in the terminal window and will run forever (for example to run the TMS active mode script, you will enter: ./tms_loop_rightonly.pl). If you see errors, then you can let it run for a while and see if it will correct itself. If not, then please proceed to the next step and contact software support for help.
    2. Press Ctrl-C (both the "Control" and "C" keys together) to stop the program. If it does not respond, then please press "Ctrl-C" multiple times.
  • Running the oneshot mode:
    1. If running the active mode, then do this in a terminal window: ./lbc_tms.py -s 1 --tmspy '--verbose'
    2. If running the passive mode, then do this in a terminal window: ./lbc_tms.py -s 1 -p --tmspy '--verbose'
    3. Either one should just take one measurement and then calculate the results and quit (or send corrections before quiting).
    4. Note: if you are running in the oneshot mode, then you need to supply all the parameters desirable, otherwise, it will use the default values. The most important of these are the standard deviation based filtering threshold and population size, which default to 2 microns and 3 vectors. If you want to change them, then you will have to supply the values yourself, e.g., ./lbc_tms.py -s 1 --tmspy '--verbose --stdevpop 10 --stdevbnd 5' will use a population size of 10 vectors and use 5 microns as the filtering threshold. There are other parameters to change as well. But these two are the most important ones to remember.

In any mode, the screen outputs should give you ample information on what it is doing.

Note if you want to make a new reference pose (e.g., after an active optics cycle) and you are running in loop mode, then you will need to stop the program first by executing step 2 in the loop mode section, and then repeat loop mode steps 1 and 2.

Also note if running in any of the loop modes, you will need to keep your terminal window and remain logged in robs.mountain.lbto.org to keep the TMS software running. If you close your terminal window and/or logout of robs.mountain.lbto.org, then the TMS software running in the terminal will be terminated. See the next section for how to run the software in detached mode so you can close your terminal window and logout.

Running in Detached Loop Mode

If you are running in any of the loop modes, then sometimes you may run the TMS software for long time, and you may wish to close your terminal window and logout of the machine robs.mountain.lbto.org and come back later when it is time to stop the program and collect data. By default the steps given in the previous section runs the TMS software attached to the lauching terminal window. There are various ways you can use to detach the TMS process from the terminal window. If you have a preferred way to do that, then use your own way. Otherwise, here we will give a method that uses the "nohup" utility.

The "nohup" utility is a standard UNIX and POSIX to ignore the hangup signal, thereby allowing you to essentially detach your program from its terminal window. "nohup" stands for no hang up. To launch the TMS loop with "nohup," replace step 5 in the previous section with this command:
nohup ./[mode_script] > [path]/[screen] 2>&1 &
The [mode_script] is the name of the TMS script given previously; the [path] part is a standard UNIX path name, the [screen] part is a file name you wish to use to store the screen output from the TMS program. By default "nohup" will use "nohup.out" for [screen] and will use the directory from where you launched the TMS software as [path]. We do not typically want to use the default behavior (since the "supportscripts" directory is automatically synced to github) and therefore you should specify the [path] and [screen] parts.

For example, if you want to lauch the TMS active/drop mode detached and redirect the screen output to the "/tmp/screen.out" file, then you will do this in step 5:
nohup ./tms_loop_rightonly_drop.pl > /tmp/screen.out 2>&1 &

After this, you should be able to close your terminal window and logout of the machine robs.mountain.lbto.org. When you come back later to robs.mountain.lbto.org, you cannot use "Ctrl-C" to stop the TMS program since you lost the original terminal window and the TMS process became detached. Instead you will need to find out the process ID (PID) of the running TMS software and then kill it using the PID. Do the following steps:
  1. Open a terminal window and run the following command: ps -A | grep tms
  2. Depending on the result of this command: a If the command returns nothing, then the TMS software has already been terminated. You do not need to do anything. a If the command shows a single line starts with a number, then you will run the command: kill -9 [PID] to stop the TMS process, where "[PID]" is the first number you saw from the previous command (for example: kill -9 6728 where "6728" is the PID of the reported TMS process ID). a If the command shows multiple lines, then that means you have multiple TMS processes running. This normally should not occur. You can of course run the "kill" command to stop each of them. When in doubts, you can consult the software team for help.

Collecting Data

When you are done with running the TMS software, you should copy the data files produced during the run for later analysis and study. These data files are being placed in the "/tmp" directory and therefore are not guaranteed to always be there. Therefore you should copy the data immediately after you finish running the TMS software. Here is a list of the files you should copy and collect:
  • /tmp/telescope_right_tmsoffset.dat
  • /tmp/tms_error_correction.dat
  • /tmp/tms_left_<timestamp>_telescope.txt
  • /tmp/tms_right_<timestamp>_telescope.txt
  • /tmp/tms_raw_<timestamp>_telescope.txt
  • /tmp/tms_tcp_measurement.log

Depending on the mode you are running, you may not see all of these files, which is normal. After you have copied these files, you should probably delete them from the '/tmp' directory so that next time when you run the software, you will get a fresh copy of them (in case the old ones are still there).

Extracting Data

The data you collected (as described in the previous section) are in a condensed form. They may not be immediately useful to you. There is a utility program you can use to extract and reformat the data as CSV files that will be much simpler to process and analyze. The most important data files you collected will be tms_<left|right>_<timestamp>_telescope.txt (which contains the final TMS correction vectors sent or intended to be sent to the telescope) and tms_tcp_measurement.log (which contains all the information from the Etalon multiline server as well as local pose calculation related information). The extraction utility program is tms_extract.py located under the same TMS directory /lbt/lbto/supportscripts/MetrologySupport on robs.mountain.lbto.org. It will accept these two types of input files.

It is simple to use. There are two main options: '-t' for extracting "tms_tcp_measurement.log" type information; and '-c' for extracting TMS final corrections contained within "tms_<left|right>_<timestamp>_telescope.txt". And an optional option '-o' lets you specify a string to be used as the prefix for all the output files. We will give some examples how you can use it.

/lbt/lbto/supportscripts/MetrologySupport/tms_extract.py -t /path1/tms_tcp_measurement.log /path2/tms_tcp_measurement.log -o tms
/lbt/lbto/supportscripts/MetrologySupport/tms_extract.py -c tms_right_20201109_telescope.txt tms_right_20201030_telescope.txt
/lbt/lbto/supportscripts/MetrologySupport/tms_extract.py -c tms_right_20201109_telescope.txt tms_right_20201030_telescope.txt -t tms_tcp_measurement.log -o tms.test

The '-o' option is optional and it accepts an arbitrary string. If specified, the string is used as the prefix of the output files. For example if '-o tms' is given, then the output files will be named as "tms.raw", "tms.correction" and so on. If not specified, then the output file names will be automatically prefixed with 'tms.<b>-<e>' where '<b>' is the timestamp of the first record and '<e>' is the timestamp of the last record in the files given. An example of the automatically generated file name could be: "tms.20201109.200423.104043-20201110.120402.763123.raw". The auto-generated names tend to be quite long and verbose. So if you don't like them, then you should supply the '-o' option and name them yourself.

Both options '-t' and '-c' take a list of corresponding files (can be one as well). As shown you can also use either option alone. The "tms_tcp_measurement.log" files contain full timestamp with every record inside so you can actually rename the input files however you like. But the "tms_<left|right>_<timestamp>_telescope.txt" files only contain partial timestamps inside and with the rest of the timestamps encoded in their names (e.g., "tms_right_20201030_telescope.txt"). The extraction program will need both types of timestamps to properly sort the data and so do not remove the timestamps in the file name, or the utility program will complain about that and refuse to proceed. All supplied input files will be read and all valid data contained inside will be collected and sorted on the timestamps. The order of files on the command line does not matter since the program will do the sorting itself.

The '-t' option generates the following files (assuming with '-o tms' option):
  • tms.raw: the raw channel measurement length from the Etalon multiline server.
  • tms.etalon: the Etalon corrected channel length data.
  • tms.ciddor: our own corrected channel length data (with LBT weather telemetry).
  • tms.pose: the calculated pose data (i.e., x,y,z,rx,ry,rz).
  • tms.stdev: the calculated standard deviation on each channel. The standard deviation is currently calculated on three most recent channel measurements at each given timestamp. The unit is in micron.
  • tms.stdev.filter: the channel filtering based on standard deviation for each vector. Basically if a channel's value is 0, then it is not filtered; if instead a channel has 1, then it is being filtered because of exceeding the standard deviation threshold. For convenience, the first data row in the file is a summation of each column (i.e., how many times each channel has been filtered by standard deviation).
  • tms.drop: the intentional exclusion of particular channels. If a channel in a vector is 0, then it is not excluded intentionally; if a channel has value 1 in its vector, then it is being dropped intentionally. For convenience the first data row in the file calculates the summation of each column (i.e., total excluded times for each channel).
  • tms.missing: this file contains the overall channel missing information. If a channel is 0 in its vector, then it is not missing in that vector at that moment; if a channel is 1 then it is missing from the final vector. This file is the overall missing count. The missing channel could be due to standard deviation based filtering, intentional drop, or failing the final sanity check because the entire vector does not meet the minimal condition. For convenience, the first data row in the file calculates the summation of each column (i.e., total missed times for each channel). Note: this file only counts those channels that contribute to the final pose calculation. So the diameter channels are always counted as missing in this file.
  • tms.ref.raw: reference channel length vectors (the raw version).
  • tms.ref.etalon: reference channel length vectors (the Etalon corrected version).
  • tms.ref.ciddor: reference channel length vectors (our own corrected version).
  • tms.ref.pose: poses that became references.
  • tms.ref.channel: the reference vector's channel composition. That is, which channels are considered in the reference vector. For each channel, there are four possible values:
    1. 0: the particular channel is present in the reference vector taken at the specified timestamp.
    2. 1: the particular channel is not present in the reference vector, but it is intensionally excluded.
    3. -1: the particular channel is not present in the reference vector because it is being rejected (due to bad measurements).
    4. none: the particular channel is not present in the reference vector because it is not relevant (e.g., the diameter channels).
  • tms.ref.comp: this file reports the compatibility between each vector v and the reference vector. Each channel in a vector could have three values:
    1. 0: this means the particular channel is compatible with the same channel in the reference vector (i.e., if the ref vector has this channel, then v also has it, and if the ref vector has the channel missing for whatever reason, then v also has it missing).
    2. 1: this particular channel in v is a surplus channel, i.e., it is in v but not in the ref vector. Currently surplus channels do not participate in the pose calculation.
    3. -1: this particular channel is in the ref vector, but not in v. Basically whenever there is -1 in a vector v in this file, then it is not compatible with the reference vector, i.e., v will be discarded and does not result in a final pose.
  • tms.weather.etalon: the Etalon weather telemetry as returned by the Etalon multiline server.
  • tms.weather.lbt: the LBT weather telemetry we used.

The '-c' option only generates one file: tms.correction (assuming the '-o tms' prefix option given). This is the final TMS correction vectors computed.

All of these generated files are CSV style data with a timestamp followed by corresponding data items (the first line is a CSV header that explains the names of each field). If there are any data items that are not valid or present, then 'none' will be displayed in these files. Notice if the reference files (.ref.*) are empty then this is because the given input files do not have reference vector information (e.g., a run that does not set a reference and instead used a previous reference).

If you wish to do the extraction on your own machine, then you can copy the "tms_extract.py" file to your own machine. You will need a recent Python 3 environment installed to be able to run it (recommended to be at least Python 3.6). Other than the standard Python 3 environment, it does not depend on anything else.

Specifying Drop Channels

The Active/Drop and Passive/Drop modes allow you to specify which channels will be dropped and their probability. The current setup for both of these modes specifies that we will drop channel 11 at 5% probability and channel 23 at 2% probability. To change that, you will need to edit their respecitve script (i.e., edit "tms_loop_rightonly_drop.pl" for Active/Drop mode and edit "tms_loop_rightonly_passive_drop.pl" for Passive/Drop mode).

You will see the following string in both of these files:
'--drop 11:0.05,23:0.02'

As you can guess the "11:0.05" part is to specify channel 11 being dropped at 5% probability and the "23:0.02" part is to specify channel 23 being dropped at 2% probability (the comma in between is needed to separate them). You can add or remove these parts. Valid channels are 4,8,10,11,13,14,15,17,23. Probability is specified by a number between 0 and 1 (both inclusive) with 0 meaning absolutely NOT dropped and 1 meaning absolutely will be dropped. Please do not remove the single quotes around the "--drop" clause. For example, if you want to specify to drop channel 4 definitively, then you should change it to:

'--drop 4:1'
If you want to specify multiple channels, then do not forget to separate each of the blocks by a comma.

The "--drop" clause actually supports a much more sophisticated syntax to allow very flexible channel and probability fixing. Since this feature is not routinely used. We will not specify the entire syntax here. If you are interested, then you can make a request to the TMS mailing list and if there are sufficient interestes, then maybe we will write a tutorial at some point.

Selecting Specific Channels in the Reference Vector

The "drop" option can also be used to select certain specific channels to use in a reference vector. This effectively limit the entire TMS system to run on only these selected channels (remember that now the reference vectors sets the usable channels, if a normal vector has less channels than the reference vector, it will be rejected and no pose will be calculated, but if a normal vector has more channels than the reference vector, then those surplus channels will be ignored in the pose calculation).

To do so, you will need to edit the script lbc_set_tms_offset.py. Open this file and find the line that contains the content com = lbto.path.support + '/MetrologySupport/tms.py --makev0 --verbose --drop 11:1 --stdevbnd 5 --stdevpop 3' (currently on line 280, but it may change). You can see it already includes the "--drop 11:1" option. This means the current reference vector will be made without channel 11. You can add more channels to be excluded if you want to. For example, if you want to exclude channel 4 from the reference vector, then just replace the --drop 11:1 part with --drop 11:1,4:1 (this effectively uses a seven channels reference vector). The syntax used in the "--drop" option is just the same as described in the previous section.

Specifying (or Changing) the Standard Deviation Based Filtering Parameters

The current standard deviation based filtering system has been configured to use a five micron threshold, and a population of three vectors (i.e., standard deviation is computed on three successive measured vectors). Both of these can be reconfigured. To reconfigure them, you need to edit the following scripts:
  • lbc_set_tms_offset.py this script controls the reference vector setting. Find the line com = lbto.path.support + '/MetrologySupport/tms.py --makev0 --verbose --drop 11:1 --stdevbnd 5 --stdevpop 3' and edit it (currently line 280, but it may change).
  • tms_loop_rightonly.pl this script controls the normal measurement setting. Find the line system("${lbto::path::support}/MetrologySupport/lbc_tms.py -s 1 --tmspy '--verbose --stdevpop 3 --stdevbnd 5'"); and edit it (currently line 10, but it may change).

If you want to change the settings for both of the reference vector and normal vectors, then please edit both of these files. Locate the line described above in each of the files (or the file you want to edit). The --stdevpop option controls the standard deviation population (you can see it is currently set to three). And the --stdevbnd option controls the filtering threshold (in micron) and it is currently set to five. You can change it to any other positive integer values. Be careful and do not change anything else (especially the quotation marks).
Topic revision: r15 - 30 Mar 2021, YangZhang
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback