Warning: All this may or may not work
If things anywhere are acting weird check disk space by typing
Can check out all the system processes logs with cd ADOPT_LOG then ls and by date
WFS Arbitrator not starting
If the Wfs Arbitrator refuses to start with this message:
Error in WfsArbitrator: [AOException] Previous Python message queue cannot be removed: Permission denied (code -1)
It means that the internal queues used to communicate with the Python process are not working, probably because someone started the system with user root. The Wfs Arbitrator uses the mq_queue mechanism, and queues are implemented by the operating system in a hidden filesystem. You need to become root, create a directory and mount the filesystem, at this point the queues are visible:
[root@lbti-sxwfs ~]# sudo mkdir /dev/mqueue
[root@lbti-sxwfs ~]# mount -t mqueue none /dev/mqueue
[root@lbti-sxwfs ~]# ls -l /dev/mqueue/
-rw-r--r--. 1 root root 80 Feb 6 00:27 reply_wfsarb_sxwunit
-rw-r--r--. 1 root root 80 Feb 6 00:27 request_wfsarb_sxwunit
Notice that the queues are owned by user root
Delete the queues, after this you can unmount the filesystem:
[root@lbti-sxwfs mqueue]# rm reply_wfsarb_sxwunit request_wfsarb_sxwunit
rm: remove regular file `reply_wfsarb_sxwunit'? y
rm: remove regular file `request_wfsarb_sxwunit'? y
[root@lbti-sxwfs ~]# umount /dev/mqueue/
[root@lbti-sxwfs ~]# exit
and now the Wfs Arbitrator should be able to start.
Bad port servers
Symptom: multiple components red on HWGUI.
- Cycle port servers from Pulizzi page (WFS rack, outlets 1 and 13). Note, they take ~1 minute to boot up.
- If that doesn't work, manually reset the relevant connections as follows:
- Figure out which one you want. They're called #1 and #2, or a name of the form ts8 (name of the device) + either 39 or 47 + either sx or dx. Example: ts839dx
- Find its IP address by typing
cat /etc/hosts on the wfs computer
- Open Firefox from the wfs computer
- Type the IP address into the url bar. It will ask you for a login, which you should get from Amali/Paul if needed
- Go to "serial ports" under "Management" on the sidebar
- Click "connections" next to the relevant component
- Click "disconnect" next to the connection that comes up
- Turn the relevant WFS system process off and then back on. This should fix your problem (if it doesn't, you have more problems :D)
- Repeat for each relevant process
OCAM image bad on start-up
a. IF BAD DISPLAY, power off the OCAM outlet from the Pulizzi page, turn it back on, and try again
- IF STILL BAD DISPLAY, power off both BCU2K and OCAM outlets, and try again, first BCU and then OCAM (turn back on BCU first, OCAM second)
- IF STILL BAD DISPLAY, call for help (after trying multiple times)
- In the HW GUI status list, check that both "CCD47" and "Tip-Tilt" have green checks
- IF BAD CCD47/TIP-TILT power cycle the CCD47 outlet from the Pulizzi page, wait for it to re-initialize, and check again
- IF STILL BAD, try again - can take a few cycles
- IF STILL BAD after 2-3 tries, call for help
OCAM image goes bad mid-observing
If bad display:
OPEN THE LOOP then
- Take optical loop data - 5000 frames, all boxes checked
- Take Phone screenshot
- Note your loop params
- Change operating mode to anything different from 1x1 cropped
- Take more optical loop data with new mode
- Then switch back to 1x1 cropped
- Reset freq to whatever was noted
Adsec - Red Terminal
Adsec Control GUI
- Mid tabs - "Focal Station" load "bentGregorianCenter"
Manual Loop Closing Issues
If optimize gain keeps running and re-running try lowering the gain on the WFS GUI, or playing with the options around there (failing on the name of this tab/section) try 1000Hz Mod 3 gains 10, 1, 600 HZ, mod 6, 600 gain. Gain too high will cause likely OCAM Illumination Error, try lowering.
LBTI Webpage has ?? for bayside status
= in wfs terminal do a kill the indie server with
sudo killall indieserver
if killall doesn't work then type
ps -aef | grep indi
and check the indie server line for a timestamp of "now" indicating it just restarted. There is an indie server script that will restart indie server if it as been killed. Things are good again if the bayside stages are showing green with OFF or ON, depending on if they are moving or not.
- may need to simply type =w_check again...otherwise not sure
- star looks like lentil instead of circle - magnetic home sensor too strong. So sometimes they register "home" when they're actually like half a position past home, it's easiest to recognize in twilight, because it's really obvious that the pupils are shaped like lentils instead of little circles.
- re-home the filter wheels from a filter that's not next to the home position, choose one of the ones in the middle of the list, home from there, go back to where you wanted to be. Home position is between the first and last filter in the list.
- depends on the last filter used previously + if there were any power bumps etc during the day
Weird half/full Shell Rip
- Failure means full shell rip, yellow box and it's obvious on the pupils/47 that the star looks like a spiderweb
but in general, the symptom of that one is that the adsec goes into Failure when you/FLAOGUI tries to open the loop. Usually means that the connection from WFS to adsec is wonky, most commonly if the adsecarb was started up after the wfsarb. wfsarb forges its adsec comm connections when its process starts up, so if the adsec doesn't exist when the wfs starts, it forges connections to a lonely void and for some reason can do everything except open the loop, in that void
- stop and start the wfs arb process
FLAO in Auto mode
- AOarb process is the one that controls FLAOGUI and its intervention-vs-auto mode so anytime you stop either the AOarb process alone, or as part of adsc_stop, if will default FLAOGUI back into auto also, because we were sharing these nights with Argos, who use Auto mode, the first time you open it for the night will also be in Auto.
- Click set intervention on FLAO GUI
instead of just left in LBTI-ready state from the night before
High disk usage on SxWFS
- During checkout, if the disk usage on the SxWFS
is high, a gzip of the logs could be performed. First, ssh into the WFS machine with the IP: email@example.com
. Then, do the following:
- cd $ADOPT_LOG
- cd 2021
- du -h to find dates with lots of logs
- cd to those dates
- gzip *.log
Shell failing - click "Recover Fail"
OCAM Illumination Error - watch for pop-up, usually just reset illumination protection but sometimes makes entire camera go bad and need to cycle Pulizzi power or a w_stop_all
Reclose loop if safe skipping, may need to open a refine pointing, reoptimize etc.
Watch out for pop-ups!
Flat Default - 250 modes bad, 500/600 modes good
Restart Adsec - Rest, off,
, wait to finish,
Overdriving/Underdriving - could be effect of seeing
Try Resume AOS Commands
If STOP AO before new preset then click PRESET AO before ACQUIRE REF.
- 25 Sep 2019