WarmUp
if temp is below freezing (wait...up to about 20 minutes for the controller electronics to warm up)
Turn ON
button (status goes to initializing
and then ready
) /home/lbcobs/supportscripts/ObserverSupport
and /home/lbto/iraf/LBTtools
FITS files are written to the Windows PCs in the D:\
directory and ftp'ed to the repository. /lbt/data/logs/lbc/tec/RTech
and BTech
in dated directories. Should be stored up to a year or so.
Observers use the scripts lbckill.sh
and lbcstart.sh
(aliased as lbckill
and lbcstart
) from the CMU to restart the control software. These have been added into the sudoers
file so that they can be run as telescope
and lbccontrol
users.
enabled
) admin
)
There is no dataprobe device for the CMU. To shut it down, run the shutdown
command as telescope
, lbccontrol
or root
There is also a replacement to the testpower3
command line and dataprobe power control. If you run power.py
as root
on the CMU, you can see the power state of all the electronics. You can toggle the power from this GUI as well. If only one side is powered up, use the --channel
option : power.py --channel b
powerdump
for all five of the power devices. powerdump: 192.168.59.113 outlets owners 20110000 last access 123 sec agorepresents that port 1 of 59.113 (CCD controller power line) has two users (makes sense, the TRACKER and the CAMERA share the CCD controller power), port 3 has one user (HK sensors are only used by the HKEEP subsystem), and port 4 has one user (port 4 is the shutter, only the CAMERA subsystem uses that).
/lbccontrol/current/conf
: redchannel.conf
, bluechannel.conf
(filter wheel positions, IP addresses, channel/rotator/tracker/camera configuration...), and lbc.conf
(mostly paths)
For testing purposes, there is a Windows XP IBM machine. It is on the bottom shelf of one of the LBC carts on 3L, the spare power supply is on the other LBC handling cart. Jenny created a document that describes roughly how they are put back together in the lab. This document has pictures which should show exactly their location.
Date | Description | ||
---|---|---|---|
CMU | Sep-2018 | 2.15: IT 5715 and IT 6800 updates - do dither on first offset if DEC=-90, mutex updates for error conditions in controllers | |
Feb-2018 | 2.14a: TCS 2018A with moon almanac information | ||
Sep-2017 | 2.14: Support SYNCOFFSET flags in Q scripts; a few INDI properties published | ||
Sep-2016 | 2.13: Update for trackers stuck on STOP/RESUME and IT 5715 - wait for TCS offsets, even if -90 DEC is given | ||
Feb-2016 | 2.12: Check tec FITS image for zero in first/next instead of checking background; write guide fits thumbnail to repository dir, timestamp in DD, fix guiding status on FACSUM, fix crash in camera preset if fails 15 times. | ||
Jan-2016 | 2.11: Bug fixes to make sure we close FITS files in all cases (good and error); protect file output errors; delete AO references and files (chip1/2.gif files); cancel HK watchdog when turned OFF | ||
Dec-2015 | 2.1: Image analysis and tracking on the CMU, use target PA to calculate rotator angle, not "achieved" | ||
Sep-2015 | 2.03: Rebuilt with TCS 2015C for TCS release. | ||
Feb-2015 | 2.03: Small changes, just rolling some of the 2.1 logging, etc. fixes to the current version: - Make stopped/disabled systems WARNING instead of NOTEs. - Add function name to RPC error log messages. - Fix core when TCS is not available (NULL instrument names). - Ice version of TCSSendWavefront in /home/lbcobs/LBCFPIA. - Rollover log file at noon local time, 19:00UTC. |
||
Nov-2014 | 2.02 modified housekeeping to not "recover" when a sensor timeout occurs; increased the sleep time in power interface for new iBootBar compatibility | ||
Oct-2014 | 2.01 is 2.0 version compiled with TCS2014D and three updates for core dump fixes: TCS read/write vs command, camera preset failure recover, TURNON | ||
Sep-2014 | New CMU version (2.0), OS is CentOS 6.5, new drivers, new PCIe profibus board, updated PHP. | ||
Mar-2014 | Query and log the shutter motor status and positions when zero measured expo time error occurs. | ||
Feb-2014 | Update for IT 4985 - catch a zero "measured exposure time" when non-zero was requested, stop the OB fix for trajectory load errors when STOP is requested (IT 4948) See Release1.7.txt |
||
Nov-2013 | Updates for IT 3346 debugging; IT4315, IT4411; added OB status string to DD. For full details, see: Release1.6.txt |
||
Sep-2013 | Populating many DD items; TCS 2013C; added more RPC error recovery; logging updates for rotator messages and RPC errors. SVN version 244 tagged as 1.5 |
||
Aug-2013 | Software updates for filter wheel mechanism changes and backlash offset; updates for ITs: 3289, 3759, 4411, 4642, 4728; updates for sending DD entries to TCS; built with TCS 2013B release; documentation and logging updates. For full details, see: Release1.4.txt |
||
26-April-2013 | Added 5sec sleep in housekeeping when resetting before checking for bad reading again; Clear fStop flag when starting a new command - fix for IT 4316 documentation updates |
||
26-March-2013 | Red HK Use it should be gray when HK is ON, like Blue HK ; web fixes for some errors, typos Added FILE, LINE logging in macros, more logging in HK on restarts, delete unused variables; Updates for balance function and to make testfilters not have to be restarted to run a different function; documentation updates |
||
Sep-2012 | Updates to config for new CCD controller hardware ID; delete Balluff device; CMU software built by KS; checkin of software updates by Andrea DiPaola for the ccd controller software on the windows machines |
||
Oct-2011 | Last version built and released by Andrea DiPaola. | ||
Windows software |
Aug-2018 | Updates to offset files in lbc.zip for all controllers (Fernando and Gustavo) | |
Aug-2017 | Built on Windows 7 virtual machine, deleted image analysis/tracking executables | ||
Mar/Apr-2017 | Updated offset file for blue science (see ITs 6542 and 6549) | ||
Jan-2016 | Save the FITS files for "too few" or "too many" downloaded pixels | ||
Sep-2014 | Log shutter open telemetry on failure. | ||
Mar-2014 | Don't check for background save errors during preset or expose - no reason to abort. Also released Baruffolo's source/my build of lbciaRun.exe . |
||
Feb-2014 | Check for zero measured exposure time and return SHUTTER error to user. | ||
Nov-2013 | Sept-2013 lbc.zip released: fix for stale guide image (IT 4411) when track_first fails; update to all chips bias values from summer shutdown checkout; LBTO build of all DiPaola source. Baruffolo's source of lbciaRun.exe not released because lbcia failed during startup testing (we ran with it during startup until it was reverted during science) |
||
Sep-2013 | new source from Gallozzi for track_first and track_next - built by LBTO (not part of lbc.zip so this is released) |
||
29-Aug-2012 | Last version built and released by Andrea DiPaola |
BOTH
- probably still need it, since we don't want to change LBC to send two presets. BOTH
as async.
BOTH
.
ActionPlay
function of actions.c
, doesn't call TCSPresetStart
if DEC > -89.
TCSMotionWait
. It is called in a PTHREAD_CREATE
fTelescope
flag when setting the string it sends to FACSUM, so it can report that it's guiding when it's not really sending anything to TCS.
watchdog on StatusThread started a new instance
and no response on gui sem_trywait
return and make it think it needs to cancel the thread and restart. lbckill/lbcstart
but I've tested just closing/opening the socket and it doesn't work.
no delay, late by xxx secs
messages lbc.c
fails and says "unknown command" : 2016/10/06 20:17:54.895906 N - unknown command >:1:TURNONHK 1 1<
upsmon
to turn off systems and HK before poweroff.lbcoff.sh
script that is used by the LBC shutdown script in /etc/rc0.d
Telescope
and Guiding
and Keep Current Pointing
according to Dave T
OBExecutionConfigFile
? this file must be written when you pause an OB (see actions.c
) OBExecution = TRUE OBFileName = lbc.070516.182516.ob OBFullFileName = /home/lbccontrol/obs/lbc.070516.182516.ob CompletedImages = 4 CompletedDitheringSteps = 0 CompletedBlueFilters = 0 ActualFilterCompletedBlueImages = 4 CompletedRedFilters = 0 ActualFilterCompletedRedImages = 0
track_first
to let you know the guide star it chooses
/lbt/data/logs
partition now, mounted on the CMU.
go.bat
script that logs when restarting the RPC servers; redirect Windows side output; check all timeouts are long enough; send DD entries in separate thread?
RPC error due to PC startup delay
message in the log because to even decide the power is on, it does a clnt_create
- why do we ever see this message?
ErrorCheckAndRecovery
function is only for CCD controller errors - not RPC error recovery.
testfilters
should let you startup the system again if fails during init; why does testfilters
set the "new" flag for configuration so that it writes and complains on shutdown?
testrotator
to have absolute angle commands, without auto-unwrapping: Subject: absolute angles in test rotator? Date: Thu, 11 Sep 2014 15:17:15 -0700 From: John Hill <jhill@as.arizona.edu> To: ksummers@lbto.org Kellee, Have you thought about whether it would be possible to give absolute angle commands in testrotator? By absolute, I mean angles that are not automatically unwrapped. Presently it applies unwrap algorithms to whatever command you send, so that you can only explore -270 to +89 and not the whole range of the rotator. The range is slightly different for red and blue presumably because they have slightly different zeropoints. I don't have any use for this other than making testrotator a bit more useful for hardware and cable tests. Ciao, John
testfilters
fails trying to uninitialize when the system has an error and uninitializes on its own - need to check before blinding calling uninit to shutdown
testrotator
capture ^C
and stop tracking
lbc\bin
to find the sci headers
IT | Detail | Status |
---|---|---|
7032 | Red HK not coming up (timeout talking to dataprobe) | the iBootBar dataprobes have sometimes required someone to log in and type reboot and we haven't figured out what gets it into this state. One thing to do is to make sure we are cleaning up the socket to the device everywhere, even on errors. |
6800 | OB hung with R (8.0s guiding failed) and B (GOYA rotator warning) | no issue with the rotator, this problem is red controller had a hardware failure but somehow missed the STOP command that the software issued to recover. The observer issued a STOP command, but didn't wait until it was done to issue another STOP. The hang should be fixed in version 2.15, released Sep-2018. |
6756 | LBC pause/resume behavior | UI reported number of exposures taken as greater than total requested why is this only showing up when using one channel? |
5715 | stars trailed on first LBC image after a dither | John says this is likely due to LBC not checking for collimation update to finish; this is a one-sided LBC problem Should be fixed in version 2.15, released Sep-2018. |
5302 | Suggested changes/reorg of LBC image headers | enhancement |
5205 | OB progress is incorrect after pause/resume | bug |
5065 5548 |
Timeout error on motion of filter wheel | We see this with wheel #2 for both red and blue. Seems like mostly it's a move command that doesn't get received on the motor because we timeout and then check and it hasn't moved at all. |
4705 | Add pointing offsets to LBC headers | enhancement Waiting on Olga for specifics, then CB can add to the TCS public interface. |
4642 | Add notification on failures | enhancement pop up on tracking stopped - no, we do NOT want to stop operations added ERROR message when tracking background is < 1 should it be identically zero? - cannot do a pop-up without stopping the OB. Fixed so it waits til it has tried up to the max exposure time before it complains Added Z4 to GUI. Make the background of the ? red -- needs testing. Olga also added the "shutter seems to be not working" to this IT - changed this to return an ERROR - so it does not go to Ready for the camera anymore -- tested it. how do we test the science image? don't have to, they don't have this problem |
4566 | Red Rotator communication problem | bug : When the rotator failed in a preset, it tried to retry indefinitely - it re-tried ~680 times before they stopped it. It is not really obvious to the operator that is happening, because it automatically retries and claims rotator system recovered from a problem ; The one error they see is: >>> preset/expose/save interrupted at preset time lbcstart they did after this problem (this is not uncommon, often core's on startup) |
4236 | hung at unitialization of blue HK | this is the HK timeouts - cannot seem to fix this within the software, but a lbckill/lbcstart fixes it! |
3833 | audible alarm on errors | enhancement |
3816 | Scale R/B exposures separately | enhancement |
3759 | Blue tracker images suddenly all zeroes | bug Happens on both sides. Goes away with an OFF/ON toggle. Sometimes it's not zero, but just funky. Also look at adding a "reset" that power-cycles the electronics, but not the windows PCs. |
3595 | Rotator position on first dither | still seeing problems as of Jan-2012 |
2765 | tick marks are sticky on the UI | I did some testing - this is all controlled by the browser settings and cookies and what you submit to the GUI. If you check mark, but do not "submit" by pressing PLAY or TURN ON or TURN OFF, then the settings will reset to the last time that happened in this browser environment. If you clear and restart, nothing is sticky - if you kill and restart, it remembers. Olga notes in the IT that running dofpia and the check marks erasing? DOFPIA "submits" the http form so the checkmarks will fill in based on that. |
2631 | too few downloaded pixels error | This is not mine, but I am monitoring the problem. |
2573 | LBCB/LBCR First Image of a Sequence has small Rotator Offset (~0.5 deg.) | reopened in Sep-2013; AdP says maybe to track Parallactic angle to open shutter delay. |
2546 | WCS info in LBC tech chip headers | enhancement Waiting on Olga to provide |
2397 | RPC Client/Host Comm Error | bugs multiple problems documented here - guiding lapses, missing images, inhibit counter on restart, etc. This became a big issue coming out of summer shutdown 2013. Added better logging, retry during preset. Need to: release go.bat script that logs when restarting the RPC servers; redirect Windows side output; check all timeouts are long enough; send DD entries in separate thread? |
Task | Detail | Status |
---|---|---|
add a reset button | allow the user to reset all the electronics, but not the PCs to save time recovering can be set with config file setting WindowsPCPowerLine values for emulated |
|
filters GUI | simple GUI over the filters test command line tool | |
error email | would be nice to have scripts daily grep out interesting info from the logs and send an email to Fernando/Mauro | currently sending email to myself with errors |
status email | would be nice to monthly run diagnostics on the logs like we ran in January for filter performance and send an email | |
testfilters | add options to: move specified amount; write the current filter positions to a file | |
CCD test | automatically calculate bias values from the oversampling pixels and write to ini file |
|
|
|
power.py created - it requires you to refresh after changing the state of something because it's not threaded - maybe sufficient? |
|
|
complete |
|
|
complete in lbccontrol1.2 version of testfilters |
Task | Detail | Status |
---|---|---|
Rehost | Replace CMU and Windows PCs with newer hardware | See the rehost notes for status |
non-sidereal updates to GUI | implement changes to OB gui to access JPL horizons to look up objects and download the ephemeris for TCS | |
STOP failures | sometimes the STOP command actually fails and when this happens you have no recourse but to do the kill/start. But, the system is fine - it just thinks a command is running. We should make the TURNOFF available anytime to get out of this state. |
AdP says I should figure out when STOP failures occur. The only thing I've found so far is comm failure (5) on a stop during a stop. Saw the same thing in Jan-2014 again (see note in IT 2397 on 3-Jan-2014). |
blue tech time off? | perusing logs in May, found blue tech 5 seconds off from the other machines | 15-may-2013 checked and blue tech was sync'ed Monthly, checking sync on each of the boxes |
software to extract CCD controller temps from logs | to monitor whether overheating with covers on; also related to IT 4101 | I have implemented tklogger to look for housekeeping errors in the lbc.log but it does not pay attention to the values - doesn't do any parsing; we should make it parse then we wouldn't get false complaints on 0 values; and we could check for CCD detector temps |
I | Attachment | Action | Size | Date | Who | Comment |
---|---|---|---|---|---|---|
![]() |
9152-88Final.pdf | manage | 1 MB | 11 Sep 2018 - 15:33 | UnknownUser | LBC Software SPIE papers |
![]() |
CMU-reboot-13May.txt | manage | 9 K | 17 May 2013 - 17:24 | UnknownUser | Notes on Michelle's reboot of CMU 13-May-2013 |
![]() |
HKElectronics.jpg | manage | 2 MB | 21 Oct 2014 - 16:11 | UnknownUser | Diagram of HK electronics |
![]() |
LBC_Lab_setup.pdf | manage | 534 K | 16 Sep 2014 - 20:00 | UnknownUser | Description of how to set up the lab environment for CCD controller testing |
![]() |
PowerControl.png | manage | 155 K | 17 May 2013 - 17:22 | UnknownUser | Power control snapshot |
![]() |
Release1.4.txt | manage | 3 K | 14 Nov 2013 - 21:41 | UnknownUser | Details for the Summer-2013 LBC release |
![]() |
Release1.6.txt | manage | 3 K | 12 Dec 2013 - 22:03 | UnknownUser | Detailed release notes for Nov-2013 release |
![]() |
Release1.7.txt | manage | 855 bytes | 20 Feb 2014 - 21:58 | UnknownUser | LBC release 1.7 notes |
![]() |
SPIE2004DiPaola.pdf | manage | 332 K | 11 Sep 2018 - 15:33 | UnknownUser | LBC Software SPIE papers |
![]() |
powerGUI.png | manage | 27 K | 13 Nov 2013 - 20:32 | UnknownUser | Power GUI screenshot |