FACSUM and WEATHERSTATION Troubleshooting
The FACSUM and WEATHERSTATION UIs on the overhead monitors function in identical ways. The only difference is the name of the script you call to start them and the name of the process when they are running.
In theory, you shouldn't ever have to restart either the GUI or web side even when TCS
goes up and down. They will all pick back up after TCS
restarts (specifically the DDS subsystem - always check if DDS is running before starting any troubleshooting).
To (re)start DDS
ssh firstname.lastname@example.org (this is an auto-login from Tucson)
- If DDS is not "Started", start it by clicking on "Start" next to DDS
- If you want to restart DDS (not recommended) click on "Stop" and the on "Start" next to DDS and answer "Yes" to all the dialogue questions. Most likely, this will not fix anything.
To restart the Displays
If all the displays are showing
- You can vnc into the displays using the command
vncviewer using the IP address noted above on the restart displays page.
ovms as their startup software, then it means the
displays.csv file is corrupted. To fix it, log on to the info machine and copy the saved file to the current one:
More notes about this issue are in IT 5459
ssh email@example.com (this is an auto-login from Tucson)
cp displays.csv.good displays.csv
- restart the GUI you want to via the web page above
To restart the web services
The web service is a script running on one of the tcs machines on the mountain. Log in, kill the currently running script, start the new one
pkill -9 wda.sh
nohup /home/tcs/bin/wda.sh &
If the AllSky Image is not updating
The AllSky image is grabbed with the curl command in the
curl http://192.168.18.40/AllSkyCurrentImage.JPG > /web/info/AllSky/AllSkyCurrentImage.JPG
is mounted from
If the AllSky image is updating on the weatherstation-pc (see notes here
), then make sure the file on
is up to date. If not, then
is not working on
If the GCS Guide images are not updating
FACSUM looks for the GCS
images using the TCS
). It uses the configuration variable
and the path read from GCS
to get the particular files using an http request. As of Fall-2017, this was using
as the base url and
as the path. These images are "http requested" on the NUCs from
. The Tucson web server
is proxied to the mountain.
is an alias on the mountain web server to the mount point
From SH, about the setup:
The http request is proxied from the web server down here to the web
server on the mountain. When I was trying to resolve the issue with
excessive bandwidth consumption by clients accessing web cam images on
the mountain, I had to change the fairly straight-through proxy
statements on the web server down here to understand the difference
between requests that should be directed to the mountain, and requests
to resources that should be held down here: if an image is requested
on the mountain it should come from a mountain web server, and if an
image is requested down here it should come from the web server down
There is a tool called
, and it's not so much a profiler as it is a top equivalent for X resources (thus the name, XRESourcesTOP). One of the things you can see with the tool is how many pixmaps an application is using as well as the amount of memory used by those pixmaps. The number of pixmaps (
in the tool output) is what Chris was watching when debugging cpu issues with FACSUM. That number should be pretty small soon after FACSUM and WEATHERSTATION start and should grow slowly over the next 3 hours as the AllSky movie is built. At this point, the number of pixmaps should be more or less stable. It may increase or decrease by 1 or 2 from time to time but those should balance out and not grow unbounded.
In Feb-2014, FACSUM is hovering around 114 Pxms and 160 Misc after having been run all night. WEATHERSTATION is currently at 109 and 153. We see different numbers from machine to machine for the same application, but in general they should not be increasing in an unbounded way. When FACSUM was exhibiting the pixmap leak, it was growing at about 4 pixmaps per second and reached as high as 23000. This is very obvious when watching the xrestop output. In the case of FACSUM, it was found that a call to
objects caused this issue. The fix was to alter the objects by adjusting the object palettes.
See WeatherStationSoftware for details on the software that reads the raw weather data, all-sky images.