Warning: All this may or may not work

If things anywhere are acting weird check disk space by typing df -h.

Can check out all the system processes logs with cd ADOPT_LOG then ls and by date

Troubleshooting

WFS Arbitrator not starting

If the Wfs Arbitrator refuses to start with this message:

Error in WfsArbitrator: [AOException] Previous Python message queue cannot be removed: Permission denied (code -1)

It means that the internal queues used to communicate with the Python process are not working, probably because someone started the system with user root. The Wfs Arbitrator uses the mq_queue mechanism, and queues are implemented by the operating system in a hidden filesystem. You need to become root, create a directory and mount the filesystem, at this point the queues are visible:

[root@lbti-sxwfs ~]# sudo mkdir /dev/mqueue
[root@lbti-sxwfs ~]# mount -t mqueue none /dev/mqueue
[root@lbti-sxwfs ~]# ls -l /dev/mqueue/
total 0
-rw-r--r--. 1 root root 80 Feb  6 00:27 reply_wfsarb_sxwunit
-rw-r--r--. 1 root root 80 Feb  6 00:27 request_wfsarb_sxwunit

Notice that the queues are owned by user root.

Delete the queues, after this you can unmount the filesystem:
[root@lbti-sxwfs mqueue]# rm reply_wfsarb_sxwunit request_wfsarb_sxwunit
rm: remove regular file `reply_wfsarb_sxwunit'? y
rm: remove regular file `request_wfsarb_sxwunit'? y
[root@lbti-sxwfs ~]# umount /dev/mqueue/
[root@lbti-sxwfs ~]# exit

and now the Wfs Arbitrator should be able to start.

Bad port servers

Symptom: multiple components red on HWGUI.

Try:

  1. Cycle port servers from Pulizzi page (WFS rack, outlets 1 and 13). Note, they take ~1 minute to boot up.
  2. If that doesn't work, manually reset the relevant connections as follows:
    1. Figure out which one you want. They're called #1 and #2, or a name of the form ts8 (name of the device) + either 39 or 47 + either sx or dx. Example: ts839dx
    2. Find its IP address by typing cat /etc/hosts on the wfs computer
    3. Open Firefox from the wfs computer
    4. Type the IP address into the url bar. It will ask you for a login, which you should get from Amali/Paul if needed
    5. Go to "serial ports" under "Management" on the sidebar
    6. Click "connections" next to the relevant component
    7. Click "disconnect" next to the connection that comes up
    8. Turn the relevant WFS system process off and then back on. This should fix your problem (if it doesn't, you have more problems :D)
    9. Repeat for each relevant process

OCAM image bad on start-up

a. IF BAD DISPLAY, power off the OCAM outlet from the Pulizzi page, turn it back on, and try again
    1. IF STILL BAD DISPLAY, power off both BCU2K and OCAM outlets, and try again, first BCU and then OCAM (turn back on BCU first, OCAM second)
    2. IF STILL BAD DISPLAY, call for help (after trying multiple times)
  1. In the HW GUI status list, check that both "CCD47" and "Tip-Tilt" have green checks
    1. IF BAD CCD47/TIP-TILT power cycle the CCD47 outlet from the Pulizzi page, wait for it to re-initialize, and check again
    2. IF STILL BAD, try again - can take a few cycles
    3. IF STILL BAD after 2-3 tries, call for help

OCAM image goes bad mid-observing

If bad display: OPEN THE LOOP then
  1. Take optical loop data - 5000 frames, all boxes checked
  2. Take Phone screenshot
  3. Note your loop params
  4. Change operating mode to anything different from 1x1 cropped
  5. Take more optical loop data with new mode
  6. Then switch back to 1x1 cropped
  7. Reset freq to whatever was noted

Adsec - Red Terminal

  1. type adsc_stop, then adsc_check, then adsc_start

Adsec Control GUI

  1. Mid tabs - "Focal Station" load "bentGregorianCenter"

Manual Loop Closing Issues

If optimize gain keeps running and re-running try lowering the gain on the WFS GUI, or playing with the options around there (failing on the name of this tab/section) try 1000Hz Mod 3 gains 10, 1, 600 HZ, mod 6, 600 gain. Gain too high will cause likely OCAM Illumination Error, try lowering.

Other

LBTI Webpage has ?? for bayside status = in wfs terminal do a kill the indie server with killall indieserver or sudo killall indieserver if killall doesn't work then type ps -aef | grep indi and check the indie server line for a timestamp of "now" indicating it just restarted. There is an indie server script that will restart indie server if it as been killed. Things are good again if the bayside stages are showing green with OFF or ON, depending on if they are moving or not.

Weird =w_check== - may need to simply type =w_check again...otherwise not sure

Filter Issues - star looks like lentil instead of circle - magnetic home sensor too strong. So sometimes they register "home" when they're actually like half a position past home, it's easiest to recognize in twilight, because it's really obvious that the pupils are shaped like lentils instead of little circles. FIX - re-home the filter wheels from a filter that's not next to the home position, choose one of the ones in the middle of the list, home from there, go back to where you wanted to be. Home position is between the first and last filter in the list. Cause - depends on the last filter used previously + if there were any power bumps etc during the day

Weird half/full Shell Rip - Failure means full shell rip, yellow box and it's obvious on the pupils/47 that the star looks like a spiderweb but in general, the symptom of that one is that the adsec goes into Failure when you/FLAOGUI tries to open the loop. Usually means that the connection from WFS to adsec is wonky, most commonly if the adsecarb was started up after the wfsarb. wfsarb forges its adsec comm connections when its process starts up, so if the adsec doesn't exist when the wfs starts, it forges connections to a lonely void and for some reason can do everything except open the loop, in that void Fix - stop and start the wfs arb process

FLAO in Auto mode - AOarb process is the one that controls FLAOGUI and its intervention-vs-auto mode so anytime you stop either the AOarb process alone, or as part of adsc_stop, if will default FLAOGUI back into auto also, because we were sharing these nights with Argos, who use Auto mode, the first time you open it for the night will also be in Auto. Fix - Click set intervention on FLAO GUI

instead of just left in LBTI-ready state from the night before

High disk usage on SxWFS - During checkout, if the disk usage on the SxWFS is high, a gzip of the logs could be performed. First, ssh into the WFS machine with the IP: sxwunit@192.168.148.248. Then, do the following:
  1. cd $ADOPT_LOG
  2. cd 2021
  3. du -h to find dates with lots of logs
  4. cd to those dates
  5. gzip *.log

List

Shell failing - click "Recover Fail"

OCAM Illumination Error - watch for pop-up, usually just reset illumination protection but sometimes makes entire camera go bad and need to cycle Pulizzi power or a w_stop_all

Reclose loop if safe skipping, may need to open a refine pointing, reoptimize etc.

Watch out for pop-ups!

Flat Default - 250 modes bad, 500/600 modes good Restart Adsec - Rest, off, adsc_stop, wait to finish, adsc_start

Overdriving/Underdriving - could be effect of seeing

Try Resume AOS Commands

If STOP AO before new preset then click PRESET AO before ACQUIRE REF.

-- EmilyMailhot - 25 Sep 2019
Topic revision: r15 - 19 Feb 2021, JaredCarlson
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback