LBC Windows PC Software Rehost

Incorporating Guiding/Active Optics into the Tracker object

The 2.1 version was released in January-2016.
These notes are now just historical.

Version 2.1 of the LBC control software has tracking and image analysis software in the TRACKERS code, not on the Windows PC. To run this version
1. log on to the cmu as lbccontrol or root
2.  lbckill
3.  cd /lbccontrol
4.  rm current
5.  ln -s 2.1 current
6.  lbcstart
7.  service httpd restart    (use "sudo service httpd restart" if using lbccontrol account)

Version 2.02 is the version we've been running since 18-Nov-2014. To return to Version 2.02:
1. log on to the cmu as lbccontrol or root
2.  lbckill
3.  cd /lbccontrol
4.  rm current
5.  ln -s 2.02 current
6.  lbcstart
7.  service httpd restart    (use "sudo service httpd restart" if using lbccontrol account)

Note: We don't need an updated lbc.zip file when we release 2.1 with tracking/ia on the CMU. It's just the go.bat file that needs to be modified to not start the IA program. But, don't need to do anything for testing - it simply will not use the IA that's running on the Windows boxes.

Testing

  • A good night's use on 20150928 (see IT5697 and IT5700)
  • Barry used the new version on Christmas-2014. Several core files (mostly in TCS changes - used the Ice interface). see LBCRehostTesting20141224

Issues

  • There is a lot of cleanup that needs to be done to the tracking code. For the first release, it was maintained as close to the current version as possible so it would be easy to make sure it ported correctly. But, there's still a lot of stuff that's being done that's not used, etc.
  • This star used to be filtered because of the flag 4, why do we get it if the flux is so huge?
    currently I have a flux check in trackfirst for 1million to get rid of it (they used to have a flux check in there at some point of 60K)
    test filename is: lbcbtec.20140701.082846_1.fits
    From imageRef1.cat in old and new:
      ID   X           Y           fwhm  flux             fluxmax   mag        magerr      background class  flag
       25   1845.555    160.373     5.95  2.28824e+06      54238.4   -15.8988   0.0006      712.895    0.01   4
    
       ID   X           Y           fwhm  flux           fluxmax     mag        magerr      background  class  flag
       25   1845.555    160.373     5.87  2288228        54238.41    -15.8987   0.0006      712.9218    0.97   0 
    
  • why does the "no GOOD pupils found" message from IDL turn in to "no corrections required" on trackers?
    Because it exits lbcfpia but writes all zeroes to the output file, including the error code of zero, when ref is 1.
  • make the tracking gif files readable by apache - saw permission denied messages for /lbccontrol/current/www/ui/dynamic/Rguide.gif during testing, which is a link to /images/tftp/RTech/guide.gif which doesn't have world read
  • why am I doing a hard-coded rotate 90 when creating gif files?
  • make sure to test error situations, if we STOP during IDL, etc.
  • update null.gif -- not doing that anymore - need to figure out the right place
    no, this should be fine - shouldn't change based on Windows vs Linux - this is in the instrument control (lbc) code which one to display

longer-term
  • merge DOFPIA and LBCFPIA IDL files?
  • should make the log files not hard-coded; when I try to run standalone, it wants to write to /images/IA/FPIA and root owns the files - it should be overridable
  • clean up configuration file afterwards - some stuff not used anymore, never used ...
  • documentation "groups" ? ia trackfirst tracking - what do we want these to really be (these are the Modules link in the doxygen documentation)? Are the right files being used in doxygen (don't use the old pc files, don't use old power files, etc.)
  • does anything use the chip*.gif files? yes, built into 4.html in the GUI
  • does anything read AoParam.txt ?
    it is written to /images/tftp... and linked to www/ui/dynamic so I have link to it from the wiki since it's available via web server, but anything else?
  • maybe make the .out files get appended to? add timestamp?
  • make log files roll over; make out files roll over?

done
  • move logging to lbc log
  • separate lbcia logging between red/blue
  • why do we have to be in the pro directory? why can't it just find that stuff?
  • do we need environment variables for anything? seems like maybe I can get rid of all the env var checking
  • sextractor parameters - can we set some of these from the command line instead of it finding them in relative path (where you are now) ?
    http://mensa.ast.uct.ac.za/~holwerda/SE/Manual_files/Guide2source_extractor.pdf

    gave them absolute path in the param file used by sextractor
  • found a few bugs - have to determine how they would affect things
    nothing is using the offx value read from the files, nothing is using flux/fluxmax in trackers code
  • have to separate all the files read and written to be red and blue -- also first and next ??
  • hard-coded values in the sextractor configuration for paths, etc. - doesn't even use the paths set up in the structure
    for instance: default.ass is used as the hardcoded ParamSEXFileName in tracknext, which means the filename that WriteOutNEXTSEXParamFile uses. but it's also hard-coded in the WriteOutNEXTSEXConfFile - not using the variable/constant.
    this is true of the NNW files as well, CONV
  • check all the strings to make sure I'm using them right
  • check which of the input files are required, which are created dynamically, which are junk (not using flats for instance)

To test during the day, with local FITS files

  • trackers.c can be modified to be a no-power, no-RPC, hard-coded FITS files version
  • trackerstest.c is set up to instantiate two trackers and run guiding in separate threads
  • with daytime testing, have to use hard-coded FITS files and date
    • ia.c using the today keyword to find FITS files
    • get_last_tech_files.pro ignores the date checkBCSoftwareRehostNotes

What Has to Change with New CCD Controller Software

  • right now the tracking code is looking in /Repository - after the archive processes move it from newdata - will it be the same?
    Thought we were going to have the ccdctrl write to Repository for tech and newdata for sci, but it would have to write to YYYYMMDD directories. Maybe it needs to go to newdata for both and let the archive software move it
    Have to use newdata - it takes minutes for files to be moved to Repository

    Use local CMU disk, not mounted disk.
  • power control for new CCD controller, and delete power of windows PCs

Changes

  • changed a lot of things that would be errors to ignore - don't care if we cannot write a file, for instance
  • moved redundant code into a shared file
  • renamed the SEX file writing methods to be what type of file they're writing rather than 1,2,3,4
  • no longer writing several files that are constant - the default.param , default.ass, default.conv, and default.nnw files were written on every call even though they are exactly the same every time
  • IDL_Cleanup was exiting - changed to set flag to just cleanup
  • took some significant IDL changes to make the logging use r or b named file
  • changed the call to LBCFPIA to include /NoDisplay and hard-coded the config file name
  • set some paths, etc. to make the config file found from the /lbccontrol/current/conf instead of locally (that's why they were CD'ing to pro directory) and not to log in the pro directory
  • logging Z1-Z22 instead of P0 to P21 now in the LBC log -- didn't change the AO params outputs

now the source files (in addition to the standard trackers.c and trackerstest.c ) are:
Source Files
ia.h/ia.c IAActiveOptics function; The wrapper for IDL - sets up and calls DoTecIA procedure, then reads results from file.
trackfirst.h/.c Reads FITS files, calls sextractor, writes ref catalog for tracknext.
tracknext.h/.c Reads FITS files, calls sextractor, calculates offsets sent back to be sent to guiding.
track_shared.h/.c shared constants, routines
Input Files
scan1.dat in conf directory; 3 constants for scan area of chip 1.
default.param
default.ass
in conf/sex directory; Sextractor file that defines what fields you want in the catalog file - slightly different between first and next - trackfirst uses default.param, tracknext uses default.ass. (specified in the .sex file below)
default.conv in conf/sex directory; Constant sextractor file of convolution values.
default.nnw in conf/sex directory; Constant sextractor neural network values.
Output Files
all track output files prepended with r_ or b_
x_track_first.out
x_track.out
 
x_default.sex
x_assoc1.sex
List of options to sextractor including paths to all the input files. default used by trackfirst, assoc used by tracknext
x_imageRef1.cat Built by trackfirst on call to sextractor (CATALOG_NAME in sextractor config)
x_reducedRef_1.cat Built by trackfirst - "reduced". Used by tracknext as ASSOC_NAME in sextractor config.
x_image1.cat Built by tracknext on call to sextractor (CATALOG_NAME in sextractor config)
x_reduced_1.cat Built by tracknext as "reduced". Not used by anything?
all IA output files postscripted with r or b or red, blue
lbciaCoeffs_xxx.dat Written by the IDL procedures, this is the results. Read by the IAActiveOptics function of trackers
lbciaRefValuesxxx.dat Results when it is recording reference values (doRef arg is 1). Read by LBCFPIA if doRef is 0.
lbcia_x.log IDL log output for errors/debug/warning/.... goes to this log. This is the IDL InfoLog, DebugLog, ExtDebugLog, etc.
lbciaRun.log Output from the stdout of IDL programs goes here (compile statements). I deleted the dump of the cfg file since it never changes. This file is shared between blue/red because there is really just one IDL session.
AOParam_xxx.txt Same as what is logged in the lbc.log for the zernikes. Copied data from the lbciaCoeffs file
does anything read this?
xxxYYYY-MM-DD.TecLog This file is written to by LBCFPIA as the default log (unless the logfile keyword is sent). The values are the Z4,Z5,Z6,Z7,Z7,Z11,Z22, estimated seeing and wavefront image estimation

Guiding Software

Guiding software uses 3rd party products cfitsio , ImageMagick , SExtractor

Here at LBTO, cfitsio is used by IRS software (and DIMM). The include files are copied to /lbt/tcs_devel/include and library in /lbt/tcs_devel/lib
looks like it's using version 3.31 from the include file
Currently, LBC is using 3.26 (windows PC version)

The windows PCs are using sextractor version 2.2.2 (Sep-2001); the mountain and new CMU are using 2.8.6 (Apr-2013)

Modified the track_first and track_next code and makefile to use this version of cfitsio

ImageMagick (convert is the executable) is already installed in Tucson and on the mountain.

Copied four FITS files that I've used in the past for tracking tests to my machine for testing.

Lots of standalone testing done in Aug-2014 on the new CMU. Used files from the real system for comparison.
See the notes and logs in /home/ksummers/lbc/CMURehost/2.1TrackingTests/

Active Optics Software

Testing when Active Optics moved in to Trackers object of instrument control

The active optics can be run completely standalone to test. The only problem is that it wants to find FITS files based on the current time. I commented out this check in get_last_tech_files and ran with a path to my files instead of /Repository.

From the pro subdirectory, you can run manually like so:
IDL> DoTecIA, 1, 'r', 30.0, datadir='/lbccontrol/test'  

are there separate lbcfpia_tec.cfg files for red and blue?? not currently

Modified the path to the log file - it was /home/lbceng . I'm using /images/IA/FPIA since the track logs/catalogs/etc go into /images/IA/track

Had to install mpfit library - we don't have this in SVN for LBC - maybe it's in SVN for DOFPIA?

make sure the guide.fits file has the right header info
guide.fits:SIMPLE  =                    T / File conforms to FITS standard                 
guide.fits:BITPIX  =                  -32 / Bits per pixel                                 
guide.fits:NAXIS   =                    2 / Number of axes                                 
guide.fits:NAXIS1  =                   38 / Axis length                                    
guide.fits:NAXIS2  =                   38 / Axis length                                    
guide.fits:BZERO   =                  0.0 / Offset data range to that of unsigned short    
guide.fits:BSCALE  =                  1.0 / Default scaling factor                         
guide.fits:BLANK   =                    0 / Value to use for undefined pixels              
guide.fits:EXTEND  =                    T / FITS dataset may contain extensions            
guide.fits:END

See notes in and logs in /home/ksummers/lbc/CMURehost/2.1IDLTests/

Testing in Tucson

The lbciaRun.exe program is just a wrapper to call the IDL programs to do the work.
   export LM_LICENSE_FILE=/lbt/astronomy/stow/idl81/license/license.dat  
but it looks like I don't have this defined right now - how is it finding the license??

Need to make sure we have the right config file(s) and we pick up the right ones (it may default and I don't notice...)

When you just bring up the windows PC, it unpacks lbcfpia.zip to the C:\lbcfpia directory. In there are the configuration files:
      3712  04-27-10 10:44   lbcfpia/src/lbcfpia.cfg
      3592  05-07-10 08:04   lbcfpia/src/lbcfpia_tec.cfg  

The IDL program DoTecIA is defaulting the DataDir to D:\
need to fix this by setting it in the executable or changing the default in the IDL

The C program calls the IDL with no argument for /tec , but the IDL program hardcodes the /tec onto the configuration file read. The config file reader is tacking on src , so since it's already in src from the CD command, it's not getting the file from src/src which is what you would have if it used the environment variable and added src .

The environment variable LBCFPIA_HOME is set to c:\lbcfpia

LBCIA_HOME is d:\

I think it's getting it because it uses the . directory as the default:

    defcfg = 'lbcfpia.cfg'
    If KeyWord_Set(Tec) Then defcfg = 'lbcfpia_tec.cfg'
    ;
    ; Cfg file search order:
    ;   1. FileName parameter (CandidateFiles[0])
    ;   2. lbcfpia.cfg in current directory (CandidateFiles[1])
    ;   3. $LBCFPIA_HOME/src/lbcfpia.cfg (CandidateFiles[2])
    ;
    If KeyWord_Set(FileName) Then CandidateFiles = [FileName,'',''] $
       Else CandidateFiles = ['', '', '']
    CandidateFiles[1] = '.'+ps+defcfg
    HomeDir = GetEnv("LBCFPIA_HOME")
    If HomeDir NE "" Then CandidateFiles[2] = HomeDir+ps+'src'+ps+defcfg 

And in the lbciaRun.txt file on the tech machines, it says:
1cfg file - .\lbcfpia_tec.cfg 

lbciaRun-BTech-Mar2013.txt: Sample log file from lbciaRun executable from Blue Tech machine - there are 101 executions in the log file

There's an IDL command executed in the C program before it runs the DoTecIA to CD c:\\lbcfpia\src
that's how it finds the pro files. This is changed to pro subdirectory in the linux stuff.
We will have to figure out if we want this to remain separate from dofpia or not.


Original Message --------
Subject: Re: DoTecIA on tech LBC tech PC is not running
Date: Thu, 03 Oct 2013 12:34:22 +0200
From: andrea.dipaola@oa-roma.inaf.it
....

Hi, the blue/red qualifier is used to distinguish the two channels and is managed inside Windows PCs software that pass to IDL code the proper value...

Andrea

Olga Kuhn <okuhn@lbto.org> ha scritto:

> Hi,
>
> I just wanted to note that I don't find an lbciaRefValues.dat file, but
> instead (now looking on the Blue Tech PC), it is lbciaRefValuesBlue.dat.
> Both the lbciaCoeffs.dat and lbciaRefValues.dat files seem to be qualified
> by the channel name. As these are not the expected file names, I am not
> sure if this is new or not, but just wanted to note it.
>
> Olga

To run it on CentOS correctly:
  1. set LBCIA_HOME to the source directory that has a pro subdirectory with a lbcfpia_tec.cfg file
  2. run the executable like: lbciaRun 1 b 120.0
  3. it will write to the lbciaRun.txt file in LBCIA_HOME

  • WindowsPCVersion.txt: 22-Aug-2014 testing notes - including PIXSCALE differences between tech/science
I Attachment Action Size Date Who Comment
WindowsPCVersion.txttxt WindowsPCVersion.txt manage 3 K 18 Mar 2015 - 16:30 UnknownUser 22-Aug-2014 testing notes - including PIXSCALE differences between tech/science
lbciaRun-BTech-Mar2013.txttxt lbciaRun-BTech-Mar2013.txt manage 288 K 11 Mar 2014 - 18:06 UnknownUser Sample log file from lbciaRun executable from Blue Tech machine
test.txttxt test.txt manage 4 K 11 Mar 2014 - 20:38 UnknownUser Track testing on my machine
Topic revision: r50 - 18 May 2016, KelleeSummers
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback