FACSUM and WEATHERSTATION Troubleshooting

The FACSUM and WEATHERSTATION UIs on the overhead monitors function in identical ways. The only difference is the name of the script you call to start them and the name of the process when they are running.

In theory, you shouldn't ever have to restart either the GUI or web side even when TCS goes up and down. They will all pick back up after TCS restarts (specifically the DDS subsystem - always check if DDS is running before starting any troubleshooting).

• Notes on the Operators wiki for ENV and weather
• Details for the software that reads the raw weather data from the weather station

To (re)start DDS

• ssh tcs@tcs1.mountain.lbto.org (this is an auto-login from Tucson)
• TCSGUI
• If DDS is not "Started", start it by clicking on "Start" next to DDS
• If you want to restart DDS (not recommended) click on "Stop" and the on "Start" next to DDS and answer "Yes" to all the dialogue questions. Most likely, this will not fix anything.

To restart the Displays

• To stop/restart, use the http://info.mountain.lbto.org/displays/displays.php OR http://info.tucson.lbto.org/displays/displays.php pages.
  Press Stop and then (after several seconds), press Start .
  

• You can vnc into the displays using the command vncviewer using the IP address noted above on the restart displays page.

• If the displays page is not working, you can log in and bounce the displays manually:

   ssh root@IPAddress
   stop display
   stop display-manager
   start display-manager
   start display

If all the displays are showing ovms as their startup software, then it means the displays.csv file is corrupted. To fix it, log on to the info machine and copy the saved file to the current one:

1. ssh web@info.mountain.lbto.org (this is an auto-login from Tucson)
2. cd /web/info/displays
3. cp displays.csv.good displays.csv
4. restart the GUI you want to via the web page above

More notes about this issue are in IT 5459

To restart the web services

The web service is a script running on one of the tcs machines on the mountain. Log in, kill the currently running script, start the new one

    ssh tcs@tcs1.mountain.lbto.org
    pkill -9 wda.sh 
    nohup /home/tcs/bin/wda.sh & 

If the AllSky Image is not updating

The AllSky image is grabbed with the curl command in the wda.sh script on tcs1 :

curl http://192.168.18.40/AllSkyCurrentImage.JPG > /web/info/AllSky/AllSkyCurrentImage.JPG 

/web on tcs1 is mounted from web1.mountain.lbto.org

If the AllSky image is updating on the weatherstation-pc (see notes here ), then make sure the file on web.mountain.lbto.org in /web/info/AllSky is up to date. If not, then wda.sh is not working on tcs1 .

If the GCS Guide images are not updating

FACSUM looks for the GCS images using the TCS DD entries gcs.side[].GuideCam.imagePath, gcs.side[].GuideCam.imageFilename (and likewise WFSCam). It uses the configuration variable WEB_SERVER_BASE_URL and the path read from GCS to get the particular files using an http request. As of Fall-2017, this was using http://info as the base url and /Repository/GCSfile/dailyimages/ as the path. These images are "http requested" on the NUCs from info . The Tucson web server info is proxied to the mountain.
/Repository is an alias on the mountain web server to the mount point /lbt/data/repository

From SH, about the setup:

The http request is proxied from the web server down here to the web server on the mountain. When I was trying to resolve the issue with excessive bandwidth consumption by clients accessing web cam images on the mountain, I had to change the fairly straight-through proxy statements on the web server down here to understand the difference between requests that should be directed to the mountain, and requests to resources that should be held down here: if an image is requested on the mountain it should come from a mountain web server, and if an image is requested down here it should come from the web server down here.

X Resources

There is a tool called xrestop, and it's not so much a profiler as it is a top equivalent for X resources (thus the name, XRESourcesTOP). One of the things you can see with the tool is how many pixmaps an application is using as well as the amount of memory used by those pixmaps. The number of pixmaps (Pxms in the tool output) is what Chris was watching when debugging cpu issues with FACSUM. That number should be pretty small soon after FACSUM and WEATHERSTATION start and should grow slowly over the next 3 hours as the AllSky movie is built. At this point, the number of pixmaps should be more or less stable. It may increase or decrease by 1 or 2 from time to time but those should balance out and not grow unbounded.

In Feb-2014, FACSUM is hovering around 114 Pxms and 160 Misc after having been run all night. WEATHERSTATION is currently at 109 and 153. We see different numbers from machine to machine for the same application, but in general they should not be increasing in an unbounded way. When FACSUM was exhibiting the pixmap leak, it was growing at about 4 pixmaps per second and reached as high as 23000. This is very obvious when watching the xrestop output. In the case of FACSUM, it was found that a call to setStyleSheet on two QLineEdit objects caused this issue. The fix was to alter the objects by adjusting the object palettes.

See WeatherStationSoftware for details on the software that reads the raw weather data, all-sky images.

See Build and Deploy Big Display Software for how to run FACSUM or WEATHERSTATION in Tucson