LBC Software Rehost Notes

Initially, we were on a path to rehost the LBC software to newer machines and operating systems. However, SkyTech is working on new CCD controller drivers that are ethernet based and run on Linux. That solution gets rid of the Windows machines completely and the PCI interface boards to the controllers.

  • see CAN 602s201 for the LBTO software and Information Technology description.
  • see CAN 602f051 for the LBC CCD Controllers CTR_A1600 details.

New Ethernet CCD Controllers

The ethernet CCD controllers plan will upgrade the CCD controller boards to have the optical interface replaced by a standard Ethernet module and a local PIC microcontroller (a BeagleBone). This will make the Windows PCs obsolete. So, the gray areas in the following diagram will be moved -- some to the CMU and the CCD processing to the BeagleBone onboard the CCD controller.
LBCSoftwareArchitecture.jpg

Strategy

The software strategy is to change as little as possible to keep the testing required to a minimum. We would like to try to keep the CMU to CCD controller interface identical to what it is now -- using RPC on the BeagleBone to replace the RPC interface on the Windows PC.

Plan / Status

The plan is documented in CAN 602s201 (last updated Nov-2014 in the CAN, but updated May-2016 here).

Get whatever API documentation we can from SkyTech at least for the calls we make, for instance
SPC_A200_Expo_Time_Read_by_PCI_A300 , etc. any error code descriptions
We got the original doc (23-Sep-2014): CTR_PROCEDURES Library shared object for CTR-A2100 software description .
The document was updated in June-2016 with error codes and what-to-do information: CTR-D2100_DS_0.03.doc

July/Aug 2016 The INAF CMU is back in Tucson (accessible via 150.135.245.170).
Fernando, Kellee, Allesandro C perform checkout on the mountain with the latest BBB see LBCBBBMountainTestingJuly2016 and LBCBBBIntegrationTesting
See IT 6056 about IT setup of the Tucson/mountain testing.
May 2016 Fernando, Bellesi, and Kellee performed testing at SkyTech. See notes
Jan 2016 Tracking and image analysis code running on the CMU (version 2.1) was released
Summer 2014 Switched to the new CMU, with the updated PCIExpress profibus card.

To Do

BBB CCD Controller Integration

Below is a breakout of the software processes that run on the BeagleBone Black (BBB). The green processes at the top are on the CMU. The diagram is an attempt to make clear the separate test programs, libraries, etc., how they communicate, and software responsibility. The lowest-level software is the SkyTech API, in the diagram as \x93CTR API\x94. The highest level software is run on the LBC CMU (in green, mostly not shown) and communicates via RPC with the processes on the BBB.


    CCDControllerSoftwareArchitecture-BBB.jpg

Integration Testing of the BBB Controller - Summer 2016 integration testing notes



Image Analysis Port
This code was released in January-2016
  1. Got some errors about fits2gif failing and a create FITS failure - need to track down why these might fail - the fits2gif is already checking the file exists first
  2. Sometimes it fails finding the FITS file for tracking - this is somewhat benign because you just lose a guide cycle
    Mitigated by writing the tracking FITS files to local CMU disk instead of mounted disk.
  3. as of the Sep-2015 testing, the biggest problem is a crash we saw once in the IDL execution. The core says it was in a compilation:
    #0  0x088dde65 in IDL_RBtreeSearch () from /lbt/lbc_runtime/idl81/idl81/bin/bin.linux.x86/libidl.so.8.1
    #1  0x088dded5 in IDL_RBtreeSearch_ID () from /lbt/lbc_runtime/idl81/idl81/bin/bin.linux.x86/libidl.so.8.1
    #2  0x087c1031 in _IDL_enter_proc_name () from /lbt/lbc_runtime/idl81/idl81/bin/bin.linux.x86/libidl.so.8.1
    #3  0x087c2354 in sem_pro_callx () from /lbt/lbc_runtime/idl81/idl81/bin/bin.linux.x86/libidl.so.8.1
    #4  0x087c23de in _IDL_rul_pro_call_plist () from /lbt/lbc_runtime/idl81/idl81/bin/bin.linux.x86/libidl.so.8.1
    #5  0x088c5c56 in _IDL_parse () from /lbt/lbc_runtime/idl81/idl81/bin/bin.linux.x86/libidl.so.8.1
    #6  0x08748c24 in sparse () from /lbt/lbc_runtime/idl81/idl81/bin/bin.linux.x86/libidl.so.8.1
    #7  0x08748a92 in compile_stmt () from /lbt/lbc_runtime/idl81/idl81/bin/bin.linux.x86/libidl.so.8.1
    #8  0x08748b5c in _IDL_compile_single_stmt () from /lbt/lbc_runtime/idl81/idl81/bin/bin.linux.x86/libidl.so.8.1
    #9  0x08757b1e in IDL_Executive () from /lbt/lbc_runtime/idl81/idl81/bin/bin.linux.x86/libidl.so.8.1
    #10 0x08b3bb6b in IDL_ExecuteStr () from /lbt/lbc_runtime/idl81/idl81/bin/bin.linux.x86/libidl.so.8.1
    #11 0x0809bb9b in IAActiveOptics (paod=0x972ec790) at src/trackers/ia.c:114
    #12 0x08094c48 in TrackersExpose (arg=0xa088040) at src/trackers/trackers.c:1775
    #13 0x001ddb39 in start_thread () from /lib/libpthread.so.0
    #14 0x005d8d6e in clone () from /lib/libc.so.6

    Made the IDL code a separate executable.
  4. complaints from the ob.php and mainob.html in http log
  5. need a detailed walkthrough of the first/next code for anyplace it can crash, in particular through the FITS stuff; this wouldn't have mattered if it crashed here before as it was a separate exe
    this is done, think it's ok now, haven't crashed here again

Issues

  • Keep in mind the issues about separation of red and blue processing. For instance, right now, they can do guiding and AO corrections on both sides at the same time. But, they cannot do DOFPIA on red and blue at the same time. But they would like to. It's a limitation of being on the obs machines.
  • The waveform editor program was written by SkyTech for Windows. How will that work in the new environment? Will we run it directly on the beagle? make it a python program?
    Fernando's note (25-Sep-2014): I never thought that people of LBTO would like to program the clocks of LBC ctr ,usually i am the only one to do this unpleasant task (never found other volunteers...:))and i felt comfortable and safe to do this on a laptop Windows PC transferring then the files with sftp. ... i will ask Skytech, today, if the BBB may export a graphic display that is mandatory to the porting of wfe...
  • how will the beagle get the software at boot time? maybe it will be part of the build/install to load on the beagle? So it does not get it everytime it boots?
    We discussed this during testing in La Spezia - we decided the directory structure on the BBB will be frozen for controller configuration. It will not reload its data from the CMU each time it boots.
  • the source code is built assuming the profibus driver is installed in devices/Applicom.... - update notes to include updates to makefile if the driver is installed somewhere else
  • Will there be issues if we put the FITS files on the same machine with the rest of the instrument control processes?
    should we mount the repository disk on the CMU? yes
  • should we use a mounted /lbt on the new CMU for run time? like IDL license, convert program, sextractor using lbc_runtime setup
  • do we want to share the LBCFPIA pro files? or keep our own copy?
  • Without the Windows environment, what will the test environment be? For instance, in room 116, when INAF guys come to test
    Do we suggest a separate test CMU?? Or a spare that is rsync 'ed like MODS?
    access the BBB directly to test

Updates for INAF CMU

probably moot since we can just make it a mirrored copy

New CMUs - Configuration and Testing

new CMU was operational Summer 2014

Profibus Testing on new CMU

a new board was required with the new CMU hardware

Rehosting Tracking and IA Software

Testing

Windows Software moved to CMU

The tracking and image analysis code is running on the CMU as of the end of December-2015 (LBC instrument control version 2.1).

We can test the tracking/IA ported code before the CCD controller port is complete. This is a good thing, since the CCD controller code port is waaay overdue .

More testing was done in the extra E-time of Sep-2015 (20150928, 20150930). See notes in /home/ksummers/lbc/2.1/20150930.txt
  • On 20150930, we had about an hour of use by John and then later by Barry when LBC was using tracking. Barry used LBC a lot more, but not doing tracking. During both John's and Barry's testing, there were no crashes in the IDL, there was only 1 guide FITS file not found, active optics corrections were sent.
  • On 20150928, Barry used it a lot with tracking (almost 3 hours early, 2+ hours later). There was one crash in IDL (see above issues) and many times it failed finding the FITS file for tracking, several failures finding the FITS file for fits2gif, and a failure creating a FITS file for the guide thumbnail.
  • Abandoned the Ice TCS interface. It was crashing in the destroy of the result structure, even when checking NULL and mutex around it - not worth chasing.

See commissioning wiki, 11-18-2014 for details of the first sky tests.
See LBCTrackingTests20141210 for details of the next set of sky tests.

See MODIFIED FOR DAYTIME TESTING - for what to put back for nighttime testing.

Specifically, need to verify:
the tracking code finds the FITS files in the repository dir verified 20141118, moved to newdata
guiding offsets are communicated to the TCS verified 20141210
IA IDL code finds the FITS files in the repository newdata dir verified 20141210
focus corrections are communicated to the TCS verified 20150928
STOP commands are handled correctly in the IDL  
lbckill/start handles clean up and restart of IDL ok
the GUI shows the correct info when guiding/failed/done, including the GIF file verified 20141210
check permissions if problems with the GUI files verified 20150930
revert TCS code if comm problems with TCS abandoned the Ice interface, not worth the effort


To revert to the non-Ice TCS version in 2.1:
  1. change top-level makefile to TCS older version (called makefile-newTrackers)
  2. mv tcs dir to backup
  3. checkout from SVN tcs dir (or copy from 2.02)
  4. rebuild


To test the new CCD controller while keeping the existing Windows PC CCD controllers in place:
  1. assign IP addresses to BBB, different from the Windows PC IPs
  2. verify:
    • power up / TURN ON
    • images written to newdata and moved to Repository
    • error conditions / STOP
  3. no changes should be required for DOFPIA


CMU Rehost Checkout -- completed Summer-2014

The CMU was upgraded without the upgrade to the Ethernet CCD controllers.

For all of the following tests - watch the LBC log (individual logs for test programs) and watch the LBC GUI; check /var/log/messages for TCS complaints.

1. Run the standalone programs to verify we're talking to the right systems, logging, etc.
testfilters , testrotator , testprofibus , testpower3 , testcamera , testtrackers , testob , testconfig , testhousekeeping , testlog, testtcs , testupload , testtag
many tests can be performed - home devices, move to positions, ... verify no errors
verify power control
verify logging
verify encoders are read correctly on filters and rotators
verify communication to TCS
verify communication to CCD controllers
2. Bring up the LBC software on the CMU more notes here on full-up testing
3. Turn on housekeeping -- verify it's reading red/blue temps and vacuum pressures correctly and logging  
4. Turn on other systems verify power is enabled in the correct way
5. Create OBs and play them - verify no complaints, FITS files are created, GUI updates,  
6. Bring up the web pages at lbccontrol - check the stuff listed here  
7. Before sky -- make sure all the config files match operational - telescope trajectories, debug, tracking time, etc.
8. Sky testing. Use the notes here: LBC Quick Ref Wiki Page , LBC Science Prep Checklists  

RPC Communication Within New CMU

There are two potential next steps to the phased testing, if we are moving to the Ethernet CCD controllers.
  • If we keep the trackers and camera code identical to what is currently running, need to rehost the image analysis RPC server that is running on the Windows PC - ia.c
  • If we modify the trackers and camera code to call the tracking and lbcia executables directly, we do not need the IA RPC server


BeagleBone Checkout - July-2014

During the July visit, INAF guys brought the prototype CCD board. Initially used a cross-over ethernet cable to connect the beagle directly to Fernando's laptop. Then, KS modified the IP settings on the beagle so it could be seen on our network, so we plugged it in to a port SH configured as static in room 116. KS captured all the code from the Ubuntu machine in Tucson: /home/ksummers/lbc/BBB . The SkyTech test program is named CTR-S100.out from the directory /home/ubuntu/CTR-S100

user: ubuntu
pass: ubuntu



How long does the tftp take from the Windows PCs to the archive?

tracking FITS files are about 1.2M
science FITS files are about 82M

The times recorded for tftp between the Windows tech PCs to the archive are reported below. The granularity is not great, he's only recording one digit.
   17020 TRACKER image samples
      3269    0.0 secs
     13395    0.1 secs
       206    0.2
        64    0.3
        18    0.4
        19    0.5
         9    0.6
         4    0.7
        10    0.8
         5    0.9
only 20 of 17020 were >= 1.0 secs 

In 3856 science images, the range went up to 8.5secs.


Original INAF Proposal

They came up with a proposal to implement during summer shutdown 2014.
From Andrea to Christian (6-June-2013):
> Dear Christian,
>
> We would like to give you a short summary about a possible upgrade
> for the LBC camera control system. The goal is to make easier the
> architecture of the cameras changing the present
> SkyTech-optical-fiber-CCD-controller data link with a standard
> Ethernet connection.
>> This is realized changing one of the controller board with an evolved one
> where the optical interface is replaced by a standard Ethernet module
> and a local PIC microcontroller.
>
> This solution improves the data transfer efficiency towards the
> LBTO archive using the SD memory of the new controller boards as
> data buffer for the acquired images. So built in fits images can be
> sent directly to the LBTO archive without any interaction with the
> CMU or other PCs.
>
> Each CCD controller will have its own I.P. address for its
> identification in the instrument network. In this new configuration
> the 4 win PCs are obsolete and can be removed from the LBC PCs
> architecture leaving the CMU (1 Linux PC) as the only
> PC for LBC.
>
> We have already funded the project for the prototype development.
> Skytech has just delivered the Hardware and we are beginning to
> test the prototype in the second half of the year and we plan to
> complete the lab commissioning by the end of the year.
> Skytech could produce 4 (+1 spare) boards during the first semester
> 2014. The total cost is 25000 euros.
>
> If you agree with the funding and schedule of the upgrade, we are
> available for installation of the new hardware during the 2014
> summer stop.
>
> Concerning the PC architecture, this implies to operate with the
> present 4PC+CMU configuration until summer 2014.
>
> Ciao,
> Emanuele, Andrea, Fernando

After telecon with Andrea when I was on the mountain with Fernando and Mauro in Aug-2013, I sent this:
Ciao Fernando and Andrea,

Thank you for the discussion this morning. I would like to make sure we all know what the status is and the next steps. Please let me know if any of this is incorrect.

The status is that SkyTech is currently testing the new hardware. Fernando suggested they would be done by the end of 2013.

I understand that there are a lot of outstanding issues to be determined, like the command set for the new DLL, image handling to the repository, tracking. These items should be worked out in the next few months. I would like to be included in some of that discussion, if possible.

The next step is for INAF to produce a proposal to LBTO including the proposed hardware architecture (multiple NICs in single machine, CentOS machine?), the preliminary software architecture, resources required from INAF and LBTO, and proposed schedule.

Does that sound reasonable?

Ciao,
Kellee

But, we still have no proposal (11-Dec-2013).

LBCWindowsRehostPlan

notes from when we were just rehosting the Windows machines to Windows 7, not getting rid of them
I AttachmentSorted ascending Action Size Date Who Comment
TripReportMay2016.txttxt TripReportMay2016.txt manage 4 K 20 Jul 2016 - 17:42 UnknownUser Details of trip to La Spezia, testing
CTR-D2100_DS_0.00.pdfpdf CTR-D2100_DS_0.00.pdf manage 600 K 03 Dec 2014 - 23:22 UnknownUser CTR_PROCEDURES Library shared object for CTR-A2100 software description
CTR-D2100_DS_0.03.docdoc CTR-D2100_DS_0.03.doc manage 212 K 20 Jul 2016 - 17:44 UnknownUser Updated CTR_PROCEDURES software description with error handling
602s201d.docdoc 602s201d.doc manage 12 MB 20 Jul 2016 - 18:22 UnknownUser CAN document 602s201 - July 2016 version
CCDControllerSoftwareArchitecture-BBB.jpgjpg CCDControllerSoftwareArchitecture-BBB.jpg manage 758 K 05 Aug 2016 - 16:24 UnknownUser BBB software architecture
LBCSoftwareArchitecture.jpgjpg LBCSoftwareArchitecture.jpg manage 114 K 16 Jan 2014 - 23:13 UnknownUser Software Architecture
pc-purchase.pdfpdf pc-purchase.pdf manage 26 K 07 May 2013 - 18:14 UnknownUser Initial PC purchase specifications
Topic revision: r59 - 29 Aug 2016, KelleeSummers
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback