DIMM Software Upgrades
Current working issues
- At high LBT elevations, the X axis gets stuck and moves sent to the mount don't do anything; this may also be apparent in all the
Warning: Stars close to the edge log messages - they seem all to be in the
maxX value which means that it's only the one direction we're failing to track in.
The X axis on the cameras is the up-down axis of the mount, so that makes sense. It's usually trying to move up and cannot (+X).
- Steve requested an option to make the spiral search (really a raster) not stop after N iterations: just keep going forever, hopefully until it finds it. It would be easy to make a config value that a negative number or something means don't ever stop.
- Add an alarm so the operators know when auto-acq has exited - then they don't have to watch the DIMM GUI.
- Need to re-arrange the star list that AA uses when measuring fails so that it doesn't keep handing DIMM the same star that won't work. This should be implemented with a timeout, because you don't want the star out of the list forever.
- Saw a crash of the server when a camera disconnect was pushed during measuring. The image capture crashed. The
connectCamera method in the server should not just close the camera when it gets the disconnect, it should cleanly shut down the camera thread first! That's why the server crashed, then the client crashed afterward. (20170209)
- It is sharing memory and not protecting it all over the place. Tried to add protection to the dimm settings that are being pushed to the clients (GUI and auto-acq), but it caused a lot of slow down. The dimm settings include the star list mostly, then just flags for status. Need to come up with a better way to protect and still share data between the camera thread, auto-acquisition, and the GUI.
With the addition of the acquisition thread, separated the config and "status" type of info and protected the status info. Still need to rearrange the use of the
DimmCommLBT and protect that better.
- We need a mount state that is an exception case, not just "not tracking". When we catch exceptions in the mount thread, it stops tracking, but cannot notify the server or camera, so they keep trying to measure/guide/slew/etc. Likewise, there should be an overall state of acquiring before measuring so that auto-acq can be modified to move to the next star if DIMM never can get started measuring.
- Add a hot pixel map to the web cam centroiding.
- Add background subtraction image to DIMM centroiding.
- What do we do about the laser on the web cam??
- client should intercept a cntl-C and exit nicely -- and should take the server with it
- Server exits if TCS connection goes away; need to kill the client at the same time. Otherwise it's there and when they try to do something it cores in Ice exception
- centroiding - what about ghosts? See FITS file 20150707044153
- Add to telemetry: commanded position offsets and associated encoder counts so we can determine if it's making the move we ask for (commanded vs achieved position); whether dimm thinks LBT is tracking (?)
- camera has a line in the init sequence that looks like it should be setting the gain to 128, but somehow it comes out 97?
- Saw a core of the server and client (see notes in
/home/ksummers/dimm/troubleshooting/20150612-CameraCore.txt) when TO did a "Disconnect" of the camera while in preview mode. It actually core's when you try to reconnect. If you are in preview mode, the GUI shouldn't let you disconnect.
- There is still a memory problem with the image sometimes - it's caught, but I don't know what's causing it. The total image size is not the same as xsize-x-ysize (i.e., full image size, but only 200x200). the
sendImage method would core dump if we didn't ignore it: looks like it starts with full-frame image, but then at some point the height/width get reset to 200x200, but it's still working on the big image, (see
numLines complaints in logs)
- Sometimes I see the std:out complaints:
horizontalToSlit: DTZ out of bounds:-3.35832e-59 horizontalToSlit: DTZ out of bounds:-3.35832e-59
- Make sure catch blocks are releasing locks.
Did this in the camera thread.
- Need to delete the throws in the
DimmCommSerial class. This is used in an implementation of an abstract base class that uses status codes, not exceptions to convey errors. Need to delete the
There are still fixed
throws that are not caught -
DimmCommLBT is throwing, for instance when it cannot construct and when it fails reading params (maybe other times). These should be caught or handled, not just ignored.
- Make the telemetry test program read from a file and write several data sets.
Why does it take so long sometimes to see the first seeing measurement?
It just takes that long usually to process 300 images. Didn't find any timing issues when debug added, or any correlation to guide corrections, etc.
There's a race condition for the first seeing calculation - as soon as the buffer is full, sometimes we still have a -1 for seeing so the async thread stops measuring even though we didn't really lose it - we see the "out of region-of-interest" message basically at the same time as the first "Seeing:" measurement.
In 3.3, added a sleep after the buffer is full before checking the seeing value.
Auto acquisition needs to be merged into the server.
A mount exception while measuring stops the camera thread, but does not communicate back to the server that the camera has stopped, so the preview hangs and auto-acq doesn't know. Need to check for camera running in the async task. Also need to check if
m_DimmSettings.measuring is true, then if the camera is not running or the mount is in an exception, turn off measuring.
Fixed this by just turning off guiding when there's a mount exception rather than stopping the camera - this seems like the right answer because the star will drift out of the region and a new star will clear the mount exception. The mount exceptions are just soft limits, not indications of hardware problems.
Why do we see the message from the telemetry collection library out of the blue
acquisition collection connected to telemetry - shouldn't that only be when we start up?
It's when we first write the data, so only on the first acquisition.
why do we have to see a seeing value of -1 to say the star is out of the ROI? Shouldn't we know before that?
This is losing centroids. It goes through the list of image infos and if centroids==2, it's used, otherwise it increments problemcount. It returns -1 for seeing if the list size is not bigger than problemcount.
The bigger problem here is the that the
DimmSettings is not a singleton, there are copies around, so the camera thread, for instance, cannot set measuring to false when something goes wrong to notify the server. Right now, the server asks for the seeing value from the image buffer class to decide if there's a problem.
spiral search does not have the same throw up/down and left/right - it's a rectangle, because of the full-frame being rectangle?
They have noted in the docs that it's really a raster, not a spiral - raster is a rectangle.
exposure time calculation - should that go back in? based on ROI - needs to be tested; but the exposure times in there were way too high anyway, I think we can ignore this.
- Steve/Tim took a movie of a DIMM bootup - does it do this every time, or was this a problem boot? It was during a period of a lot of problems with the mount, so it could have been an error scenario. I don't know the exact time, so I cannot map it to a log. I also don't know if they did it via the software, or just turned on the mount.
- I have updates to DIMM docs that were never put on the CAN - I should update again and get them out there: 116s101 - req'ts status, etc and 116s103 - user guide; design doc should be deleted or watermarked or something - it's not really a design
- Why does the
logrotate fail all the time on DIMM? every day
Jun 4 14:02:04 dimm logrotate: ALERT exited abnormally with 
Jun 5 14:02:04 dimm logrotate: ALERT exited abnormally with 
DimmCoordinates : these classes should all be singletons, created in
DimmServiceI and then available for use by all classes. Currently they are a pointer attribute in all the classes.
Cleaned this up a lot when the acquisition thread was added.
|| DIMM measurement of the plate scale
Added logging of sigma parallel and sigma perpendicular for use in measuring the DIMM plate scale.
|| FWHM reported is larger than actual images
|| enhancement Do we need an automatic focus feature?
Auto-acquisition was implemented in March-2016 see notes here
Sometimes it would struggle to acquire, but then no problem after a restart - we know the FOV is reset, is it possible the exposure time makes a difference? It's ~4 millisecs on init, then set to ~10 millisecs when you do a start measurement. Never changes after that. however, I think it is meant to - there is code there to set it quite a bit higher (~144 millisecs) if the FOV is 200x200. But there's a bug in it and it won't get triggered - I don't think. Also, fixed a problem where the FOV was not reset in the spiral search. So, if you started the search after measuring ok, it was only 200x200 (Oct-2015).
What does it actually fail on when acquiring?
- Calculate a threshold based on the image statistics
m_SrcImageInfo.threshold = ( ushort ) ( stats.median() + 3 * m_SrcImageInfo.deviation );
- Create 3x3 pixel regions of entire image, averaging the flux in each region
- If a 3x3 pixel region average is > threshold, add
X,Y to list of peaks
- For each of the peaks
if a 30x30 pixel region doesn't fit around the center of the pixel, dump it
calculate the background by adding all the pixels not within a 4x4 area around the center (but this looks wrong)
get mean and std of background pixels
subtract (mean - deviation) (called bias) from each pixel of the image into reduced image
for each x in the 30x30 image
for each y in the 30x30 image
add x * (flux of reduced image) to sum_x
add y * (flux of reduced image) to sum_y
add (flux of reduced image) to sum_intensity
centroid.x = sum_x/sum_intensity
centroid.y = sum_y/sum_intensity
* can't we just clip the region instead of dumping it if it's not in the area?
method of the
local CameraImage img
set up Monitor on m_Monitor (attribute of DimmCameraThread)
if running (will be running if measuring was set to true)
img.data filled in from m_Camera -> getImage()
calculate image statistics into img.info
img.info = m_ImageHandler -> findStars
if (img.info.centroids.size() == 2) this image has 2 centroids
get mount -> position
if min/max X/Y values are within one of the guide limits, update counter
if counter == 4 (configurable)
apply guide correction using mount -> slew
end centroids == 2
m_imageHandler -> pushImage (img)
if ( it's been 5 secs since last measurement && seeing buffer is filled )
m_ImageHandler -> calculateSeeing()
get mount -> position
get m_DimmCommLBT -> getLBTinfo
end if time to calculate seeing
m_imageQueue -> put(img)
end if measuring
end if running
end while !m_Terminate
Tracking / Guiding
- When DIMM is tracking, the stars often drift, but not always the same direction. When it's guiding and sends a guide update, the stars do NOT go to the center of the image. Why not??
When we lose the star, I recently changed it to turn off guiding on the mount, but not the camera . The GUI is using the mount setting to enable/disable the option to guide, but the camera setting is the only one doing anything. The operators want guiding to stay enabled, even when we lose the star or move.
In an effort to track down how we're tracking into hard limits, one of the changes I made was to make the mount stop "tracking" and "guiding" when we lose the star, or change targets. (The mount automatically starts tracking again when you move to a star, or on a move via the handpaddle.) The application actually has separate "camera" and "mount" guiding flags. The mount guiding flag doesn't do anything in the code. Yet, that's what the client GUI is using to enable/disable the option to guide - so when we lose the star. The mount AND camera guiding flags are set when the user selects it from the GUI. The camera guiding flag is never disabled, unless the user disables guiding via the GUI.
The bottom line is that right now, once you select "Guiding" the first time, it stays enabled on the camera, which is where it is really doing something. So, you don't have to worry about reselecting it, even though the GUI makes it seem that it's not ON (because the mount flag is off). Really what should happen is that we should move the "Guiding" icon on the GUI to be with the camera, and it would enable/disable based on the camera guiding flag, not the mount.
Also, we should make guiding the default when they start a measurement.
Steve agrees: "Guiding has been working really well as of late, more often than not, and so I think guiding by default is a great idea!"
Implemented in version 2.5 - end of November-2012.
- DMiller says DIMM should recognize when LBT is doing an offset; JH says DIMM needs to stop and restart
JH notes on offsets: DIMM should stop tracking
- SAllanson says guiding works about 50% of the time, and cannot pinpoint why (particular area of sky, elevation, what? there seems to be no correlation)
- Need to create a picture of what's happening - maybe a GUI showing DIMM camera FOV, mount limits, ROI
- Once saw a ROI calculated far from the center of the camera - didn't center correctly after choosing star? Haven't seen this again, but there may be problems in the centering.
- Need to add a gain parameter to guiding
There are separate flags in the code for camera and mount guiding
. If guiding is selected from the GUI, both are enabled. But, only the camera's guiding actually does anything. The camera's guiding checks if your min/max centroid values are getting close to the guiding edge (specified in the config file as
in percentage). If this occurs more than the configured
times, then it sends an adjustment to the mount:
double distanceX = ( centerStarsX - centerFrameX ) * m_DimmSettings.pixelscale / 3600.;
double distanceY = ( centerStarsY - centerFrameY ) * m_DimmSettings.pixelscale / 3600.;
m_Mount->slew ( currentPointing.upDown + distanceX, currentPointing.leftRight + distanceY );
Tuning the Servos
Bob Meeks and Mike Gusick found that the mount servos have no way to be tuned. They suggested I look at the settings for accelerations because the manual says these settings should be modified if more weight is added to the mount. I added some debug to the mount driver. I found
some info in the mount manual (page 61 of 116x020a.pdf
) that shows you a
dump of the mount settings from the windows software. I tried to print
out the same info. It's pretty similar, so I think it's the same info.
The only difference between RA/DEC axes are index angle, guide speed and
the encoder limits.
RA Axis DEC Axis
IndexAngle: 1786 1705
GuideSpeed: -672926 438102
MinPosLimit: -4960000 -5460000
MaxPosLimit: 1120000 1700000
is 1080000000 (hardcoded to be 1200000000*0.90) and the
Acceleration is hard-coded to 400, the same values as the example in the
manual - so likely these are not customized values. I suggested the next time we
have some technical time with DIMM
on the sky, we can play around with
OK, the 90% and 400 values correspond to a table in the Paramount manual for a payload of 75 lb or less. I think whoever coded them in the first place got those values from the manual. The telescope itself weighs 45 lbs and I suspect with all the stuff hanging on it that it is very close to 75 lbs or even much higher if we include the counterweights. If the payload is 100 lb we should reduce the speed to 75%. We should also adjust the preload on the worm gear a little. This is done with some spring plungers inside the unit. The knob on the outside is only for releasing the worm for balancing.
There is also a recommendation to reduce the speed in lower temperatures.
While temperature should not be a problem now, it could explain issues
we've had in the past in cold weather. We also can't ignore supercooling
of the DIMM at night.
Maybe if we backed the numbers down to 50% and 250 (which are the lowest
values suggested in the manual) and also tighten the worm tension at some
point we might improve things enough to help.
implemented these updates the end of November-2012, version 2.5
Auto-acquisition was implemented in March-2016 see notes here. It fixes the problems noted below with offsets.
Steve had mentioned a long time ago that telescope offsets caused a problem with short integrations and long dithers. Doug mentioned the same thing. So for instance, it loses its star when LUCI offsets the telescope. However, DIMM does pay attention to the the telescope state.
Is this logic defective?
- the DIMM software will stop tracking its star only if the data dictionary entry
IIFOnSource returns FALSE AND the difference in telescope az or el position since its last check (every half second) is more than the configurable value 0.3333
Chris says this is the correct data dictionary entry to be reading - it does not include the rotators, just az/el state.
IIFOnSource is TRUE, DIMM will continually try to track its star
IIFOnSource is FALSE, but the difference in position of the LBT is less than the 0.3333 value for both az and el, DIMM will continue to try to track its star
Is the logic OK, but the configurable 0.3333 arcseconds should be modified?
Is there other telescope info DIMM can get to make the decision easier? A dithering flag? something in the guiding parameters?
Tom says: The IIF
and the PCS
are aware when an offset occurs, but the MCS
are not aware of offsets. It just follows the polynomial stream that the PCS
is sending. The PCS
builds the offset into those polynomials, so it is transparent to MCS
I think the proper operation when the telescope is offset is that the DIMM realizes there is an offset in progress, pauses, reads from the DD (I don't know where, or if available) the offset amount, moves its own mount the "opposite" amount (quotes meaning there will be geometry involved because the DIMM will almost always be pointing some place different from the telescope) and then resumes operation.
This suggestion is short on specific information, but
if this seems like a plausible setup, then we can go
into more details.
One problem, I think, is that when an offset occurs,
there is not an event or any notification to instruments
that an offset is about to take place. The PCS, AOS and
GCS are informed, and they do the appropriate things,
but they are part of the TCS, the DIMM is not. The
DIMM would have to monitor the appropriate DD
variable to be prepared for the offset.
I think the logic that you described is the correct logic. Modulo the 1 sec delay of MCSPU setting the flag, the on/off source flag is a reliable indicator of whether the telescope is tracking stars.
When DIMM sees the telescope go OFF SOURCE, then it should stop what
it is doing and WAIT until the telescope goes ON SOURCE. At that
point DIMM should begin acquiring a new star with preference given to
the previous star (since dithers are small).
If the DIMM acqusition sequence takes longer than the LUCI dither
time/interval, then we are out of luck.
I don't have a good idea of what to do with the 1 sec delay other than
to throw away the last 1 sec of data when you see the OFF SOURCE.
The ROI of 200x200 pixels amounts to about 1arcminutex1arcminute - if we're not on the edge in the field, we should be able to keep the star on dithers.
Right now, DIMM is very manual. When the operator goes to a new position in the sky, they have to choose a new star from DIMM's list and acquire it.
- eventually, the star drifts out of the ROI and the operator has to re-acquire
- If the telescope slews, DIMM stops tracking - goes to home position.
This is fixed with auto-acquisition
Questions / Notes