Operating the AzCam Guider and WFS units in the AGWs.

See MountainOperations/AGWCheatsheet for quick reference.

This document summarizes configuration and communication information for the various guider and wfs cameras at LBT.

Note that cooling should be enabled any time the AGws are powered up.
It is also the case that cooling from the ICS should not cause condensation, so if cooling is on and the AGw is powered down, that is not a problem - for the AGws on the ICS system (1,2,5,6,7,8).
But the PEPSI polarimeter AGws (3 and 4) are not on ICS, they use the other cooling and it can cause condensation.

Power for MODS1/MODS2 AzCams

See MODSAGWControl for power and control info for the MODS mounted AzCams.
The computers are in CRB.

Power for Leach AzCams

Power for the AGW1/AGW2 Leach controllers is driven by the AGW. startAGW via the oac system turns power on to these controllers.
=oac= does not power the computers.

Power for Magellan AzCams

PEPSI has Magellan AzCams, using computers mounted in AGW3/4 and the treehouse for AGW7/8. Power for all of Magellan controllers is driven by the AGW. startAGW via the oac system turns power on to these controllers. For AGW3/AGW4, the oac control of the power relay also turns on the AzCam computers.
There is no remote power control for the AGW7/AGW8 AzCam computers in the treehouse.

AGWs and AzCams

Every AzCam has an image capture card in a Windows PC and talks to an azcamserver software application. The azcamserver application is what GCS talks to.

The following diagram outlines which AzCam servers are connected to which guide and WFS cameras of the AGW units, so it is easier to find the name and the location of a computer for troubleshooting. Note that AGW1, AGW2, AGW5, AGW6 with Leach-style (ARC) controllers have Windows PCs in the computer rooms, while AGw3 and AGw4 with PEPSI-style (Magellan) controllers have a single Windows PC inside the AGw unit.

All of computers, MOXAs, UMACS are available on the network. The only connectivity shown on the diagram is true communication paths (camera control and motion control/power).

 

lbt-gcs-agw-config.jpg


The diagrams below provide the LBTO system architecture (in this case AGW1/AGW2-LUCI and for AGW5/AGW6-MODS) for communicating from GCS to the AGW AzCam.

 

OACAGWArch.jpg




 

AGWArch-MODS.jpg

Where are the AzCam and OAC computers located? Computer Room A and B, and LTH, and AGW#3/4 rings.

All AGWs with Leach (ARC) camera systems have their AzCam computers sitting in CRB where the fibers coming from the telescope are connected to their respective Frame Grabber cards. The LEDs on the frame grabber cards should be OFF if correctly communicating through the fibers. If you see a red LED on these cards, it means a bad connection to the camera.

All Computers with Leach camera interface cards are labeled in CRB/rack #13: agw1-cam, agw2-cam, mods1-cam, mods2-cam. Power for the computers is driven by PDUs. Or they can be rebooted via VNC.
We almost never have to hard power-cycle them anymore. But, if required, keep in mind that each machine has two power ports.
(See the IT document for PDU ports: NT-PDU-Layout)
AGW unit            instruments       system     Label       IP              associated machines & location
-------------------------------------------------------------------------------------------------------------------------------------------------
#1 #2 #3 #4 #7 #8  LUCIs, PEPSIs      AIP        oac         192.168.2.150   in CRA, rack 3   for motion control of AGW probe, power for camera and controller

#1 #2 #5 #6        ALL-LEACH          Leach      azcamserver 192.168.2.199  for image acquisition for Guider and WFS cam, CRB/rack #13
                   LUCIs, MODS

AGW #1             LUCI1 (L)          Leach      agw1-cam    192.168.2.31    in CRB/rack #13.   Serves both guide and WFS cameras for LUCI1
AGW #2             LUCI2 (R)          Leach      agw2-cam    192.168.2.32    in CRB/rack #13.   Serves both guide and WFS cameras for LUCI2
MODS #1            MODS1 (L)          Leach      mods1-cam   192.168.2.35    in CRB/rack #13.   Serves both guide and WFS cameras for MODS1
MODS #2            MODS2 (R)          Leach      mods2-cam   192.168.2.36    in CRB/rack #13.   Serves both guide and WFS cameras for MODS2

AGW #3             PEPSIPOL (L)       Magellan   agw3-cam    192.168.2.33    in the Agw unit, runs a local azcamserver and serves both cameras
AGW #4             PEPSIPOL (R)       Magellan   agw4-cam    192.168.2.34    in the Agw unit, runs a local azcamserver and serves both cameras
AGW #7             PEPSIPFU (L)       Magellan   agw7-cam    192.168.2.37    LTH. runs a local azcamserver and serves both guide and WFS cameras
AGW #8             PEPSIPFU (R)       Magellan   agw8-cam    192.168.2.38    LTH, runs a local azcamserver and serves both guide and WFS cameras

If you require a restart, you can reboot via logging in through VNC (see details below), or you can find the PDUs and the ports in this list.
(The user name for all four PDUs is apc.)

Each machine has two power supplies, mapping to two ports on the PDU. The web page is labeled, so we do not include the port numbers here.


machine location PDU
azcamserver CRB PDU2
azcamserver-spare CRB PDU2
agw1-cam CRB PDU2
agw2-cam CRB PDU1
mods1-cam CRB PDU2
mods2-cam CRB PDU2
oac CRA PDU in R3 CRA (only one power port)

Connection to the mountain machines

The OA control machine is linux, and so is accessed via ssh.

The other machines are Windows, and so are accessed via vncviewer or rdesktop from Tucson or an obs machine on the mountain.
The VNC connection to the AzCam Windows machines for guiding and wavefront sensing is not for routine observing.
This connection should only be used for debugging purposes by a Software Engineer or Mike Lesser. (ITL).

This VNC only allows one user at a time and does not kick the other VNC user off, rather it fails. If VNC doesn't work, connection through rdesktop and a correct Windows login can be used.


OA control tucson -> ssh -X oac@oac.mountain.lbto.org Linux
AzCamServer obs1-> vncviewer azcamserver
KVM VNC to r12-kvm.mountain.lbto.org (192.168.52.86)
Win7
agw1-cam obs1 -> vncviewer agw1-cam
KVM VNC to r12-kvm.mountain.lbto.org (192.168.52.86)
Win7
agw2-cam obs1 -> vncviewer agw2-cam
KVM VNC to r12-kvm.mountain.lbto.org (192.168.52.86)
Win7
agw3-cam obs1 -> vncviewer agw3-cam Win7
agw4-cam obs1 -> vncviewer agw4-cam Win7
mods1-cam obs1 -> vncviewer mods1-cam
KVM VNC to r12-kvm.mountain.lbto.org (192.168.52.86)
Win7
mods2-cam obs1 -> vncviewer mods1-cam
KVM VNC to r12-kvm.mountain.lbto.org (192.168.52.86)
Win7
agw7-cam obs1 -> vncviewer agw7-cam Win7
agw8-cam obs1 -> vncviewer agw8-cam Win7

Screenshot of the azcamserver VNC session, running eight AzCamServer instances:

azcamserverScreenshot.png


PEPSI AzCams

See PepsiAzCamNotes for some of the details associated with the PEPSI AzCams (AGws 3,4,7,8)


AzCamServer Architecture

LBTO has a different architecture for the AzCamServer than typical observatories. The ITL documentation lumps the AzCamServer with the camera controller hardware and software. The following diagram attempts to illustrate the different components and where they are in the LBTO configuration for LUCI and MODS.
AzCamArchitecture_LBTO_Guiders.gif

Error Messages

  • AzCamServer does not handle bad returns from temp requests. For instance, when the MODS agc and wfs cameras are turned OFF, but GCS still has MODS selected, GCS keeps asking for temps and AzCamServer gets a TIMEOUT, but the command doesn't fail - it just tries to convert it to a float:
    ERROR processing command:could not convert string to float: TIMEOUT
    This gets passed to GCS and then complains in the log file.

  • In Sept-2016, we had a Leach controller power supply failure that caused the AzCamServer complaint on a PowerOn (see below): agw2guidecamfailure20160920.PNG

  • In Sept-2016, we had a flat image from the guide controller after replacing a power supply. The counts were ~250-254, everywhere in the image. This was traced to the connection in the back not being plugged in.
    Mike had also suggested you might see this if cooling were unplugged, or if the controller had taken a power spike and was now bad.

  • In Sept-2016, a message about a reset failure - reset error, check power and fibers ended up being a fiber problem.

  • getaddrinfo errors
    controller server not opened: [Errno 11004] getaddrinfo failed
    Sometimes a reboot fixes it. Sometimes multiple reboots.

  • This is another socket error, saying the service is gone on the cam machine:
    ValueError("invalid literal for int() with base 10: 'controller server not opened: [Errno 10061] No connection could be made because the target machine actively refused it'",)

  • Crashes of python and the driver code (CMagControllerServer32.exe) can sometimes be found in the event viewer. Here's a snapshot of an "Application Error" from agw8-cam:
    agw8-cam-EventViewer.png

  • To take an AzCam exposure to test communication without GCS:
    In an AzCamServer window (not AzCamConsole), type: exposure.Expose() to get a bias (0 second) image, or
    exposure.Expose(2.2) to get a 2.2 second exposure.
    Note that the AzCamServer Expose command takes seconds, whereas the gcsclient command for reading an image takes milliseconds for expo time.
    You do not need to worry about shutter commands since there are no shutter parameters for the LBT guiders.
    Typing help(exposure.Expose) should display the code comments and arguments.
    Note that you get an error from this when it cannot send to the LBT guider (since you're not using one). The error is
     LBT guider ImageServer not opened: [Errno 10049] The requested address is not valid in its context 

  • See more notes on this page for other command-line options for AzCamServer.

  • The following example uses LUCI-2 to take AzCam exposures with GCS, for standard checkout, assuming GCS-R is running:
    ssh -X telescope@obs3.mountain.lbto.org
    gcsclient selectAGW right LUCI_R.cfg              
    gcsclient readGuideCam right -e 100                    sets exposure to 100ms and takes exposure
                                                           you should see the fits file in /lbt/data/share/tcs/telescope.YYYYMMDD/right_guider000001.fits
    
    gcsclient readWFSCam right -e 100                      sets exposure to 100ms and takes WFS exposure
                                                           you should see the fits file in /lbt/data/share/tcs/telescope.YYYYMMDD/right_wfsc000001.fits
    
    gcsclient selectAGW right MODS_R.cfg           MAKE SURE TO SET THE AGW BACK TO THE CURRENT INSTRUMENT AUTHORIZED, FOR EXAMPLE, MODS2

BIOS Settings

BIOS settings for agw1-cam were captured in Sep-2015 and are attached below.
Topic attachments
I Attachment Action Size Date Who Comment
AGWArch-MODS.jpgjpg AGWArch-MODS.jpg manage 164 K 27 Jul 2017 - 16:30 UnknownUser GCS-azcam-AGW architecture diagram for MODS
AzCamArchitecture_LBTO_Guiders.gifgif AzCamArchitecture_LBTO_Guiders.gif manage 21 K 08 Jun 2015 - 17:07 UnknownUser LBTO AzCamServer architecture (M. Lesser)
OACAGWArch.jpgjpg OACAGWArch.jpg manage 1 MB 26 Sep 2016 - 15:18 UnknownUser GCS-AzCam-OAC-AGW architecture example
agw1-cam-BIOS-AdvancedProcessorOptions.jpgjpg agw1-cam-BIOS-AdvancedProcessorOptions.jpg manage 355 K 24 Sep 2015 - 21:33 UnknownUser Snapshot of agw1-cam BIOS settings
agw1-cam-BIOS-BootSettings.jpgjpg agw1-cam-BIOS-BootSettings.jpg manage 396 K 24 Sep 2015 - 21:33 UnknownUser Snapshot of agw1-cam BIOS settings
agw1-cam-BIOS-ControllerInfo.jpgjpg agw1-cam-BIOS-ControllerInfo.jpg manage 141 K 24 Sep 2015 - 21:33 UnknownUser Snapshot of agw1-cam BIOS settings
agw1-cam-BIOS-PolicyControl.jpgjpg agw1-cam-BIOS-PolicyControl.jpg manage 176 K 24 Sep 2015 - 21:33 UnknownUser Snapshot of agw1-cam BIOS settings
agw1-cam-BIOS2.jpgjpg agw1-cam-BIOS2.jpg manage 340 K 24 Sep 2015 - 21:33 UnknownUser Snapshot of agw1-cam BIOS settings
agw2guidecamfailure20160920.PNGPNG agw2guidecamfailure20160920.PNG manage 70 K 26 Sep 2016 - 16:48 UnknownUser azcamserver screenshot of power supply failure
agw8-cam-EventViewer.pngpng agw8-cam-EventViewer.png manage 254 K 09 Sep 2015 - 18:10 UnknownUser Event viewer window from agw8-cam
azcamserverScreenshot.pngpng azcamserverScreenshot.png manage 120 K 01 Jun 2015 - 22:21 UnknownUser AzCamServer screenshot
guide_wfs_capture1.jpgjpg guide_wfs_capture1.jpg manage 135 K 01 Jul 2020 - 21:35 CarlHolmberg Blank WFS image vs. Guider image noise.
lbt-gcs-agw-config-20120212.pdfpdf lbt-gcs-agw-config-20120212.pdf manage 165 K 13 Feb 2014 - 00:48 UnknownUser LBTO GCS AGw Configuration
lbt-gcs-agw-config-20150126.gifgif lbt-gcs-agw-config-20150126.gif manage 34 K 02 Jun 2015 - 21:54 UnknownUser lbt-gcs-agw-config-20150126.gif File
lbt-gcs-agw-config-20150126.vsdvsd lbt-gcs-agw-config-20150126.vsd manage 149 K 02 Jun 2015 - 21:54 UnknownUser lbt-gcs-agw-config-20150126.vsd Visio File
lbt-gcs-agw-config.gifgif lbt-gcs-agw-config.gif manage 476 K 02 Jun 2015 - 22:12 UnknownUser AGw AzCam configuration
lbt-gcs-agw-config.jpgjpg lbt-gcs-agw-config.jpg manage 332 K 07 Nov 2017 - 22:39 UnknownUser LBTO AzCam configurations
mod1-cam-fiber-cables.jpgjpg mod1-cam-fiber-cables.jpg manage 1 MB 01 Jul 2020 - 21:38 CarlHolmberg WFS and Guider camera fiber optic cable labeling.
mod1-cam-rear.jpegjpeg mod1-cam-rear.jpeg manage 1 MB 01 Jul 2020 - 21:36 CarlHolmberg WFS and Guider camera fiber optic connections into mod1-cam.
Topic revision: r78 - 24 Aug 2020, IgorSola
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback