DIMM Software Upgrades

• DIMMSoftwareDebug notes
• Release Notes

Current working issues

• At high LBT elevations, the X axis gets stuck and moves sent to the mount don't do anything; this may also be apparent in all the Warning: Stars close to the edge log messages - they seem all to be in the maxX value which means that it's only the one direction we're failing to track in.
  The X axis on the cameras is the up-down axis of the mount, so that makes sense. It's usually trying to move up and cannot (+X).
• Steve requested an option to make the spiral search (really a raster) not stop after N iterations: just keep going forever, hopefully until it finds it. It would be easy to make a config value that a negative number or something means don't ever stop.
• Add an alarm so the operators know when auto-acq has exited - then they don't have to watch the DIMM GUI.
• Need to re-arrange the star list that AA uses when measuring fails so that it doesn't keep handing DIMM the same star that won't work. This should be implemented with a timeout, because you don't want the star out of the list forever.
• Saw a crash of the server when a camera disconnect was pushed during measuring. The image capture crashed. The connectCamera method in the server should not just close the camera when it gets the disconnect, it should cleanly shut down the camera thread first! That's why the server crashed, then the client crashed afterward. (20170209)
• It is sharing memory and not protecting it all over the place. Tried to add protection to the dimm settings that are being pushed to the clients (GUI and auto-acq), but it caused a lot of slow down. The dimm settings include the star list mostly, then just flags for status. Need to come up with a better way to protect and still share data between the camera thread, auto-acquisition, and the GUI.
  With the addition of the acquisition thread, separated the config and "status" type of info and protected the status info. Still need to rearrange the use of the DimmCommLBT and protect that better.
• We need a mount state that is an exception case, not just "not tracking". When we catch exceptions in the mount thread, it stops tracking, but cannot notify the server or camera, so they keep trying to measure/guide/slew/etc. Likewise, there should be an overall state of acquiring before measuring so that auto-acq can be modified to move to the next star if DIMM never can get started measuring.
• Add a hot pixel map to the web cam centroiding.
• Add background subtraction image to DIMM centroiding.
• What do we do about the laser on the web cam??
• client should intercept a cntl-C and exit nicely -- and should take the server with it
• Server exits if TCS connection goes away; need to kill the client at the same time. Otherwise it's there and when they try to do something it cores in Ice exception
• centroiding - what about ghosts? See FITS file 20150707044153
• Add to telemetry: commanded position offsets and associated encoder counts so we can determine if it's making the move we ask for (commanded vs achieved position); whether dimm thinks LBT is tracking (?)
• camera has a line in the init sequence that looks like it should be setting the gain to 128, but somehow it comes out 97?
• Saw a core of the server and client (see notes in /home/ksummers/dimm/troubleshooting/20150612-CameraCore.txt) when TO did a "Disconnect" of the camera while in preview mode. It actually core's when you try to reconnect. If you are in preview mode, the GUI shouldn't let you disconnect.
• There is still a memory problem with the image sometimes - it's caught, but I don't know what's causing it. The total image size is not the same as xsize-x-ysize (i.e., full image size, but only 200x200). the sendImage method would core dump if we didn't ignore it: looks like it starts with full-frame image, but then at some point the height/width get reset to 200x200, but it's still working on the big image, (see numLines complaints in logs)
• Sometimes I see the std:out complaints:

  horizontalToSlit: DTZ out of bounds:-3.35832e-59 horizontalToSlit: DTZ out of bounds:-3.35832e-59

• Make sure catch blocks are releasing locks.
  Did this in the camera thread.
• Need to delete the throws in the DimmCommSerial class. This is used in an implementation of an abstract base class that uses status codes, not exceptions to convey errors. Need to delete the CommExceptions completely.
  There are still throws that are not caught - DimmCommLBT is throwing, for instance when it cannot construct and when it fails reading params (maybe other times). These should be caught or handled, not just ignored.    fixed
• Make the telemetry test program read from a file and write several data sets.
• Why does it take so long sometimes to see the first seeing measurement?
  It just takes that long usually to process 300 images. Didn't find any timing issues when debug added, or any correlation to guide corrections, etc.
• There's a race condition for the first seeing calculation - as soon as the buffer is full, sometimes we still have a -1 for seeing so the async thread stops measuring even though we didn't really lose it - we see the "out of region-of-interest" message basically at the same time as the first "Seeing:" measurement.
  In 3.3, added a sleep after the buffer is full before checking the seeing value.
• Auto acquisition needs to be merged into the server.
• A mount exception while measuring stops the camera thread, but does not communicate back to the server that the camera has stopped, so the preview hangs and auto-acq doesn't know. Need to check for camera running in the async task. Also need to check if m_DimmSettings.measuring is true, then if the camera is not running or the mount is in an exception, turn off measuring.
  Fixed this by just turning off guiding when there's a mount exception rather than stopping the camera - this seems like the right answer because the star will drift out of the region and a new star will clear the mount exception. The mount exceptions are just soft limits, not indications of hardware problems.
• Why do we see the message from the telemetry collection library out of the blue acquisition collection connected to telemetry - shouldn't that only be when we start up?
  It's when we first write the data, so only on the first acquisition.
• why do we have to see a seeing value of -1 to say the star is out of the ROI? Shouldn't we know before that?
  This is losing centroids. It goes through the list of image infos and if centroids==2, it's used, otherwise it increments problemcount. It returns -1 for seeing if the list size is not bigger than problemcount.
  The bigger problem here is the that the DimmSettings is not a singleton, there are copies around, so the camera thread, for instance, cannot set measuring to false when something goes wrong to notify the server. Right now, the server asks for the seeing value from the image buffer class to decide if there's a problem.
• spiral search does not have the same throw up/down and left/right - it's a rectangle, because of the full-frame being rectangle?
  They have noted in the docs that it's really a raster, not a spiral - raster is a rectangle.
• exposure time calculation - should that go back in? based on ROI - needs to be tested; but the exposure times in there were way too high anyway, I think we can ignore this.
• Steve/Tim took a movie of a DIMM bootup - does it do this every time, or was this a problem boot? It was during a period of a lot of problems with the mount, so it could have been an error scenario. I don't know the exact time, so I cannot map it to a log. I also don't know if they did it via the software, or just turned on the mount.
• I have updates to DIMM docs that were never put on the CAN - I should update again and get them out there: 116s101 - req'ts status, etc and 116s103 - user guide; design doc should be deleted or watermarked or something - it's not really a design
• Why does the logrotate fail all the time on DIMM? every day

  Jun  4 14:02:04 dimm logrotate: ALERT exited abnormally with [1]
  Jun  5 14:02:04 dimm logrotate: ALERT exited abnormally with [1] 

• DimmSettings , DimmMount, DimmCoordinates : these classes should all be singletons, created in DimmServiceI and then available for use by all classes. Currently they are a pointer attribute in all the classes.
  Cleaned this up a lot when the acquisition thread was added.

Issue Traks

IT	Detail	Status
5051	DIMM measurement of the plate scale	enhancement Added logging of sigma parallel and sigma perpendicular for use in measuring the DIMM plate scale.
2897	FWHM reported is larger than actual images	enhancement Do we need an automatic focus feature?

Acquisition

Auto-acquisition was implemented in March-2016 see notes here.

Sometimes it would struggle to acquire, but then no problem after a restart - we know the FOV is reset, is it possible the exposure time makes a difference? It's ~4 millisecs on init, then set to ~10 millisecs when you do a start measurement. Never changes after that.
however, I think it is meant to - there is code there to set it quite a bit higher (~144 millisecs) if the FOV is 200x200. But there's a bug in it and it won't get triggered - I don't think.
Also, fixed a problem where the FOV was not reset in the spiral search. So, if you started the search after measuring ok, it was only 200x200 (Oct-2015).

What does it actually fail on when acquiring?

1. Calculate a threshold based on the image statistics

   m_SrcImageInfo.threshold = ( ushort ) ( stats.median() + 3 * m_SrcImageInfo.deviation );

2. Create 3x3 pixel regions of entire image, averaging the flux in each region
3. If a 3x3 pixel region average is > threshold, add X,Y to list of peaks
4. For each of the peaks

 if a 30x30 pixel region doesn't fit around the center of the pixel, dump it
 calculate the background by adding all the pixels not within a 4x4 area around the center (but this looks wrong)
 get mean and std of background pixels
 subtract (mean - deviation) (called bias) from each pixel of the image into reduced image
 for each x in the 30x30 image
    for each y in the 30x30 image
        add x * (flux of reduced image) to sum_x
        add y * (flux of reduced image) to sum_y
        add (flux of reduced image) to sum_intensity
  centroid.x = sum_x/sum_intensity
  centroid.y = sum_y/sum_intensity

* can't we just clip the region instead of dumping it if it's not in the area?

Camera Thread

The run method of the DimmCameraThread

local CameraImage img

while !m_Terminate

   set up Monitor on m_Monitor (attribute of DimmCameraThread)

   if running             (will be running if measuring was set to true)

      img.data filled in from m_Camera -> getImage()
      calculate image statistics into img.info

      if measuring
         img.info = m_ImageHandler -> findStars
         if (img.info.centroids.size() == 2)         this image has 2 centroids
 
            if guiding
               get mount -> position
               if min/max X/Y values are within one of the guide limits, update counter
               if counter == 4  (configurable)
                   apply guide correction using mount -> slew
            end guiding
         end centroids == 2

         m_imageHandler -> pushImage (img)

         if ( it's been 5 secs since last measurement && seeing buffer is filled )
            m_ImageHandler -> calculateSeeing()
            get mount -> position
            get m_DimmCommLBT -> getLBTinfo
            send telemetry
         end if time to calculate seeing

         m_imageQueue -> put(img)

      end if measuring

   end if running

end while !m_Terminate

Tracking / Guiding

• When DIMM is tracking, the stars often drift, but not always the same direction. When it's guiding and sends a guide update, the stars do NOT go to the center of the image. Why not??
• When we lose the star, I recently changed it to turn off guiding on the mount, but not the camera . The GUI is using the mount setting to enable/disable the option to guide, but the camera setting is the only one doing anything. The operators want guiding to stay enabled, even when we lose the star or move.
  In an effort to track down how we're tracking into hard limits, one of the changes I made was to make the mount stop "tracking" and "guiding" when we lose the star, or change targets. (The mount automatically starts tracking again when you move to a star, or on a move via the handpaddle.) The application actually has separate "camera" and "mount" guiding flags. The mount guiding flag doesn't do anything in the code. Yet, that's what the client GUI is using to enable/disable the option to guide - so when we lose the star. The mount AND camera guiding flags are set when the user selects it from the GUI. The camera guiding flag is never disabled, unless the user disables guiding via the GUI.
  The bottom line is that right now, once you select "Guiding" the first time, it stays enabled on the camera, which is where it is really doing something. So, you don't have to worry about reselecting it, even though the GUI makes it seem that it's not ON (because the mount flag is off). Really what should happen is that we should move the "Guiding" icon on the GUI to be with the camera, and it would enable/disable based on the camera guiding flag, not the mount.
  Also, we should make guiding the default when they start a measurement.
  Steve agrees: "Guiding has been working really well as of late, more often than not, and so I think guiding by default is a great idea!"
  Implemented in version 2.5 - end of November-2012.
• DMiller says DIMM should recognize when LBT is doing an offset; JH says DIMM needs to stop and restart
  JH notes on offsets: DIMM should stop tracking
• SAllanson says guiding works about 50% of the time, and cannot pinpoint why (particular area of sky, elevation, what? there seems to be no correlation)
• Need to create a picture of what's happening - maybe a GUI showing DIMM camera FOV, mount limits, ROI
• Once saw a ROI calculated far from the center of the camera - didn't center correctly after choosing star? Haven't seen this again, but there may be problems in the centering.
• Need to add a gain parameter to guiding

There are separate flags in the code for camera and mount guiding. If guiding is selected from the GUI, both are enabled. But, only the camera's guiding actually does anything. The camera's guiding checks if your min/max centroid values are getting close to the guiding edge (specified in the config file as limitGuide in percentage). If this occurs more than the configured minFramesToGuide times, then it sends an adjustment to the mount:

                                double distanceX = ( centerStarsX - centerFrameX ) * m_DimmSettings.pixelscale / 3600.;
                                double distanceY = ( centerStarsY - centerFrameY ) * m_DimmSettings.pixelscale / 3600.;
                                m_Mount->slew ( currentPointing.upDown + distanceX, currentPointing.leftRight + distanceY );

Tuning the Servos
Bob Meeks and Mike Gusick found that the mount servos have no way to be tuned. They suggested I look at the settings for accelerations because the manual says these settings should be modified if more weight is added to the mount. I added some debug to the mount driver. I found some info in the mount manual (page 61 of 116x020a.pdf) that shows you a dump of the mount settings from the windows software. I tried to print out the same info. It's pretty similar, so I think it's the same info. The only difference between RA/DEC axes are index angle, guide speed and the encoder limits.

RA Axis                          DEC Axis
    IndexAngle: 1786               1705
    BaseRate: 0
    SlewSpeed: 1080000000
    Accel: 400
    GuideSpeed: -672926            438102
    CreepRate: 0
    MinPosLimit: -4960000         -5460000
    MaxPosLimit: 1120000           1700000
    PECRatio: 10
    MaxPosError: 1000
    UnitId: 100
    BackEMF: 72
    HomeVelHigh: 660000000
    HomeVelMed: 110000000
    HomeVelLow: 3900000
    HomeDirection: 1
    HomeSense1: 1
    HomeSense2: 0
    Home2Index: 0
    PecCutoffSpeed: 0

The SlewSpeed is 1080000000 (hardcoded to be 1200000000*0.90) and the Acceleration is hard-coded to 400, the same values as the example in the manual - so likely these are not customized values. I suggested the next time we have some technical time with DIMM on the sky, we can play around with these values.

Bob responded:

OK, the 90% and 400 values correspond to a table in the Paramount manual for a payload of 75 lb or less. I think whoever coded them in the first place got those values from the manual. The telescope itself weighs 45 lbs and I suspect with all the stuff hanging on it that it is very close to 75 lbs or even much higher if we include the counterweights. If the payload is 100 lb we should reduce the speed to 75%. We should also adjust the preload on the worm gear a little. This is done with some spring plungers inside the unit. The knob on the outside is only for releasing the worm for balancing.

There is also a recommendation to reduce the speed in lower temperatures. While temperature should not be a problem now, it could explain issues we've had in the past in cold weather. We also can't ignore supercooling of the DIMM at night.

Maybe if we backed the numbers down to 50% and 250 (which are the lowest values suggested in the manual) and also tighten the worm tension at some point we might improve things enough to help.

implemented these updates the end of November-2012, version 2.5

Telescope Offsets

Auto-acquisition was implemented in March-2016 see notes here. It fixes the problems noted below with offsets.

Steve had mentioned a long time ago that telescope offsets caused a problem with short integrations and long dithers. Doug mentioned the same thing. So for instance, it loses its star when LUCI offsets the telescope. However, DIMM does pay attention to the the telescope state.
Currently:

• the DIMM software will stop tracking its star only if the data dictionary entry IIFOnSource returns FALSE AND the difference in telescope az or el position since its last check (every half second) is more than the configurable value 0.3333
  Chris says this is the correct data dictionary entry to be reading - it does not include the rotators, just az/el state.
• if IIFOnSource is TRUE, DIMM will continually try to track its star
• if IIFOnSource is FALSE, but the difference in position of the LBT is less than the 0.3333 value for both az and el, DIMM will continue to try to track its star

Is this logic defective?
Is the logic OK, but the configurable 0.3333 arcseconds should be modified?
Is there other telescope info DIMM can get to make the decision easier? A dithering flag? something in the guiding parameters?

Tom says: The IIF and the PCS are aware when an offset occurs, but the MCS and MCSPU are not aware of offsets. It just follows the polynomial stream that the PCS is sending. The PCS builds the offset into those polynomials, so it is transparent to MCS or MCSPU.

Doug says:

I think the proper operation when the telescope is offset is that the DIMM realizes there is an offset in progress, pauses, reads from the DD (I don't know where, or if available) the offset amount, moves its own mount the "opposite" amount (quotes meaning there will be geometry involved because the DIMM will almost always be pointing some place different from the telescope) and then resumes operation.

This suggestion is short on specific information, but if this seems like a plausible setup, then we can go into more details.

One problem, I think, is that when an offset occurs, there is not an event or any notification to instruments that an offset is about to take place. The PCS, AOS and GCS are informed, and they do the appropriate things, but they are part of the TCS, the DIMM is not. The DIMM would have to monitor the appropriate DD variable to be prepared for the offset.

John says:

I think the logic that you described is the correct logic. Modulo the 1 sec delay of MCSPU setting the flag, the on/off source flag is a reliable indicator of whether the telescope is tracking stars.

When DIMM sees the telescope go OFF SOURCE, then it should stop what it is doing and WAIT until the telescope goes ON SOURCE. At that point DIMM should begin acquiring a new star with preference given to the previous star (since dithers are small).

If the DIMM acqusition sequence takes longer than the LUCI dither time/interval, then we are out of luck.

I don't have a good idea of what to do with the 1 sec delay other than to throw away the last 1 sec of data when you see the OFF SOURCE.

The ROI of 200x200 pixels amounts to about 1arcminutex1arcminute - if we're not on the edge in the field, we should be able to keep the star on dithers.

Autonomous Operation

Right now, DIMM is very manual. When the operator goes to a new position in the sky, they have to choose a new star from DIMM's list and acquire it.

• eventually, the star drifts out of the ROI and the operator has to re-acquire
• If the telescope slews, DIMM stops tracking - goes to home position.

This is fixed with auto-acquisition.

Questions / Notes

• The AO system shouldn't be used if seeing is worse than 1.5.
• when I use the handpaddle remotely, the click/release Qt stuff does not work - it processes the endMove from the release BEFORE the startMove a lot. This causes the GUI to think the status is moving when it's not.
• Why is there a dimm-mount IP address ?
  The mount code uses this

  mksdriver.cpp:   #define DEF_PORT_NAME               "tcp://192.168.18.51:4002" 

• should we turn off tracking when spiral search fails?
  No - the mount can remain in track, without a centroid.
• exposure time is hard-coded to 10 when you start a measurement (milliseconds)
  modified to be configuration setting - separate value for measuring from spiral search.
• why are we performing calculateSeeing in two threads? It's being done in the camera thread, with the "New seeing value" message every 5 seconds. It's also being done from the main runInfoService_async thread that's 2Hz.
• Steve had it get stuck a few times - the camera didn't get the stop. Server said "Stop measuring", but camera didn't stop - kept reporting -1 seeing value. < br> I see this in the current version as well.
• why do we get "not tracking" and "motion out of range" messages at the same time? should only 1 of these matter to DIMM software?
  As long as the mount is connected, if LBT is not OnSource, it checks if motion is out of range, slews home and sets tracking off
• what log message can I look for in the log to KNOW we are connected and happy vs. stuck?
• why do we get many, many motorStatus::Entry messages sometimes? stuck in the while loop waiting for home position reached -- but shouldn't it timeout?
• why do we see msgs from trackedIntoLimit with the encoder count after we called abortSlew ? it's called from isSlewing from DimmMountBisque::slewRaw , IsHomed2 and =startSlewRaw=