TCS Configuration

C. Biddick, July-2017

Networking

The TCS uses both UDP and TCP for various network operations. There are six IP addresses of importance, which are kept in TCS configuration files. Note that some of the IP address manipulation is IPv4 specific. The ports used by the network servers are different for different locales, so all activity on a cluster must use the same locale.

The TCS maintains a shared memory segment that holds information about the subsystems. It is 4KB in size with key 0x14EB (see command ipcs for displaying information about shared memory segments). It has room for 43 subsystems. Each entry contains the subsystem name, the host name, the commandIP, the commandBRDIP, the gshmIP, the gshmBRDIP, its PID, and its running flag.

The commandIP is the host specific IP on the commandfilter network (see below). The gshmIP is the host specific IP on the gshmfilter network (see below).

There is a list of subsystems in the tcs.conf variable subsystems. This defines an ID for each subsystem, and the order of the list is used when starting/stopping all the subsystems as a group.

A subsystem is active when it has been started by the networkserver and has a valid PID. A subsystem is running when it is active and has informed the networkserver it is fully operational.

The list of host names for a cluster is in the tcs.conf variable "servers". A host can be active or passive. An active host can run subsystems, a passive host cannot. The active list is kept in the tcs.conf variable active.

A subsystem can be forced to run on a specific host using the "start_on" parameter in tcs.conf, or by using

netconfig start <sub> on <host>

This will bypass the active/passive check mentioned above. For the mountain the IIF and DDS are forced to run on tcs2.

DNS

On the mountain we have DNS entries for the iif, dds, and irs (at least). iif is the IP of the IIF host, and is currently tcs2. dds is the the IP of the DDS host, and is currently tcs2. irs is the IP of the IRS host, and is currently obs2. IRS is the server process that communicates with the IRTC Windows interface for reading out the IRTC (InfraRed Test Camera).

tcs.conf

commandfilter
This is the broadcast address for the command network in the cluster. From it is found the machine specific address (called commandIP inside the TCS). (This fails to work if the netmask is 255.255.255.255.) This is used by the rpcserver on each computer in the cluster for all command routing, and the networkserver for keeping track of running subsystems. This address can be either a real or dummy address. If it is a dummy address, none of the networking is visible to the outside world. If the TCS cluster has more than one computer, this address has to be a real address. This does not have to be the same address as the gshmfilter.

RPCs are not usually allowed to span networks. This is done with a software check that requires the first IPv4 octet number to be the same as the commandIP. This is primarily a safety feature to prevent a TCS running downtown from affecting the mountain. However, there is a configuration variable ("rpcnet") that lists first octet values that are allowed to be used by RPC to other networks.

gshmfilter
This is the broadcast address for the reflective memory ring. From it is found the machine specific address (called gshmIP inside the TCS). It is used by the gshmserver to update reflective memory. This address can be either a real or dummy address. If it is a dummy address, none of the networking is visible to the outside world. If the TCS cluster has more than one computer, this address has to be a real address. This does not have to be the same address as the commandfilter.

iifIP
This is the IP address for the IIF on the command network, and is used by the IIF C++/C interface, which is currently only used by the LBC. This IP address must be the same as the commandIP for the computer that is running the IIF. It can be a DNS name.

mcs.conf

mcspuIP
This is the IP address for the MCSPU. The MCSPU computer has to run an rpcserver, and if the MCSPU is run on a computer in the cluster, it must use the same rpcserver as the TCS, and so the IP must be the same as the commandIP on that computer. If the MCSPU is run on a computer not in the cluster, it can use any address, as long as the networks are routed properly. It can be a DNS name.

iif.conf

Factory.Host
This is the ICE listening IP address for the IIF. It can be different from the iifIP. It can be a real or dummy address, but it must be a real address if access is desired from another computer. It can be different from the commandIP if there is more than one IP address for the IIF computer. It can be a DNS name.

dds.conf

Factory.Host
This is the ICE listening IP address for the DDS. It can be a real or dummy address, but it must be a real address if access is desired from another computer. It can be different from the commandIP if there is more than one IP address for the DDS computer. It can be a DNS name.

Examples:

mountain cluster: (tcs1 - 192.168.3.16, tcs2 - 192.168.3.17)
commandfilter: 192.168.3.255
gshmfilter: 192.168.3.255
iifIP: iif (tcs2)
mcspuIP: 192.168.18.170
IIF Factory.Host: iif (tcs2)
DDS Factory.Host: dds (tcs2)

developer desktop: (rm580d-1 - 10.0.0.1, 150.135.245.33)
commandfilter: 10.0.0.255
gshmfilter: 10.0.0.255
iifIP: 10.0.0.1
mcspuIP: 10.0.0.1
IIF Factory.Host: 10.0.0.1
DDS Factory.Host: 10.0.0.1

developer vm: (xxxxx - 10.0.0.1, <real IP>)
commandfilter: 10.0.0.255
gshmfilter: 10.0.0.255
iifIP: 10.0.0.1
mcspuIP: 10.0.0.1
IIF Factory.Host: <real IP>
DDS Factory.Host: <real IP>

Important file information

Syslog

The TCS Syslog is written by the TCS server process syslogserver, which only runs on one computer in the TCS cluster. That computer is determined by the environment variable TCSSYSLOGRUNON. On the mountain we are currently using tcs2. If the variable TCSSYSLOGRUNON is not defined, the server will always run. The developer environments normally do not define the variable. The variable is read by the netconfig command, which is used to start the syslogserver if needed.

If the environment variable TCSSYSLOGTIMESTAMP is defined (as anything), then extra timestamp data is written. This extra data is a now value designed to see if there has been a delay since the original entry was sent. It is not currently being used.

The path for the log file is defined by the environment variable TCSSYSLOGPATH. We always want the Syslog and Event Log written in the same place, so on the mountain the path is /lbt/log. If the variable is not defined, the path is /data/tcs/LSS, which is the desired location on the developer environment. If the variable is defined, then subdirectories will be used for the year and month. Otherwise not.

We use environment variables because historically the syslogserver was not part of the TCS and had no access to the TCS configuration. That could now be changed.

The Syslog file name is always of the form yyyymmdd.log, e.g., 20171005.log, but there is always a symbolic link pointing to the active file: current.log in the top level directory. If subdirectories are being used, the link is at the top level, and the actual file is two levels down: /lbt/log/current.log -> /lbt/log/2017/10/20171005.log.

Event log

The Event Log is written by the LSS subsystem, and configuration file entries are used to manage it. The configuration entry LSS_EVENT_LOG_PATH, defined in tcs.conf, specifies the path. It should be the same as the Syslog path. On the mountain the path is /lbt/log/, while on a developer box the path is /data/tcs/LSS/.

The boolean configuration entry lss.log_to_sub_dir, defined in lss.conf, determines if subdirectories are used for the year and month. On the mountain we use them, on a developer box usually not.

The Event Log file name is always of the form yyyymmdd.events, e.g., 20171005.events, but there is always a symbolic link pointing to the active file: current.events in the top level directory. If subdirectories are being used, the link is at the top level, and the actual file is two levels down: /lbt/log/current.events -> /lbt/log/2017/10/20171005.events.

The string configuration entry LSS_CLIENT_LIST_PATH, defined in lss.conf, specifies the path where the LSS keeps some state information. It should normally have the same value as LSS_EVENT_LOG_PATH discussed above.

Telemetry data

The TCS telemetry collection library must be given a path specifying the location of the HDF5 files. The string configuration entry TEL_STORE_DIR, defined in tcs.conf, is used to specify the path. On the mountain it is /lbt/telemetry_data/tcs/. For developers it is /data/telemetry_data/.