You are here: Foswiki>Commissioning Web>Logs>25Oct2012_Log (30 Oct 2012, JuanGuerra)Edit Attach

UT 20121054 AO#1

Observer:JCGuerra (LBTO),D.Miller (LBTO),K.Boutsia(INAF) A.Puglisi remote(INAF)
Telescope Operator: Geno.B.(LBTO)
Additional AO Support: JCGuerra (LBTO)
Telescope Support: Elliot (LBTO)
Instrument Support: John M (LBTO)


  • On sky check of the AO_DX with the 2012F.

  • Startup of the DX size. New supervisor on Build 2012F.

  • Some bugs of the new supervisor has been debugged and fixed.

  • AO_DX side is validated for operations on 2012F.

  • Change build to 2012H. Starting to validate the AO for DX.

  • Problem with the telescope; Pointing and collimation and the OSS subsystem went down many times. Lost elevation telemetry. The AO_DX system couldn't be tested on sky satifactory. Instability of the OSS subsystem.


Start 18:00

The system started up from the AOS without any problem. The new supervisor is installed and checked during the daytime and is decided to keep this version. This is the final version and it has to be debugged.

  • Preset to AO722.

Prest to a bright source through the IIF. The preset failed due to bayside stage in wrong position. The preset is resent.

The Preset failed several times due to a problem of a slopenull file called zero.fits. In the directory of configuration for the bin1 to bin 4 the file zero.fits is added.

Fri Oct 26 01:43:02.530 2012 aos.PresetAO.failed - PresetAO[43] failed - RETRY: File zero.fits does not exists (-1010) FILE_ERROR

Change the rotator PA to allow the camera lens to converge. We are encountering with the wobble problem.

Failure of the preset due to CCD39 in wrong state. The CC39 is rebooted.

Fri Oct 26 02:23:29.339 2012 aos.PresetAO.failed - PresetAO[50] failed - ccd39 in wrong state for requested operation. Current state: STATE_CONFIGURING. (-11005) CTRL_WRONG_STATE_ERROR

Preset resend. Failure. Lost communication with slope computer. Seeing many errors in the system, the WFS is power off and software restarted.

Fri Oct 26 02:31:39.569 2012 aos.PresetAO.failed - PresetAO[54] failed - Reading shared buffer masterdiagnostic.R:OPTLOOPBUF (-5001) 

The system is power on from the AOS. The same is sent and the loop is close using the engineering gui. The AO system closed the loop without any problem.

  • Preset to AO722

Preset failed due to equalize quadrant problem, Alfio is debugging this problem and is fixed.

Resend the preset. And Preset is succesfull. Taken some data.

With this successful preset the system is keeping at close loop during 10 minutes, applying offsets without any problems. The offsets has been inside of the patrol field and outside with a success result.

Equalize quadrnat effect. See the picture below, this failure stopped the process to close the loop due to a saturation of the CCD39 and the way that the system equalize the quadrants adn take the dark. This bug is analysed by Alfio, and he started to debug and to test a few fixes. The problem still unresolved.

  • CCD39, unbalance quadrants and RIP the shell:

New preset:

Fri Oct 26 04:46:59.769 2012 aos.PresetAO.started - PresetAO[101] (aomode:ACE-AO,focalst:bentGregorianFront,instr:IRTC,wfs:FLAOWFS,so1:0.6,so2:-3,ro1:0.6,ro2:-3,mag:7.32,cindex:1.81,elev:nan,rotang:nan,gravang:nan,r0:0,skybrgt:nan,windspeed:nan,winddir:nan)

Fri Oct 26 04:55:32.983 2012 aos.StartAO.started - StartAO[107] Problem to close the loop.

DECIMATION PROBLEM during the transfer of the frames at 1kHz, the system has to decimate the frames, aparently this bug was fixed, this has to be checked on SX... Problem debugged and Fixed.

Resend preset.

Fri Oct 26 05:02:03.850 2012 aos.PresetAO.started - PresetAO[120] (aomode:ACE-AO,focalst:bentGregorianFront,instr:IRTC,wfs:FLAOWFS,so1:0.6,so2:-3,ro1:0.6,ro2:-3,mag:7.32,cindex:1.81,elev:nan,rotang:nan,gravang:nan,r0:0,skybrgt:nan,windspeed:nan,winddir:nan)


good close the taken_20121026_051612,_051800

TrackNo RefStar Mag El Wind DIMM/OL Rec bin #mod freq gain mod nph AntiDrift SR filter exp #frames disturb SN notes
20121026_051612 ??? 8.2 50 5 NaN 0.66 092604 1 400 990 0.8 1.1 0.8 3 310 ON 45.7 H 10 100 ONSKY zero.fits - Warning: Selected IRTC dark +2h old!
20121026_051800 ??? 8.2 50 4 NaN 0.56 092604 1 400 990 0.8 1.1 0.8 3 312 ON 58.6 H 10 100 ONSKY zero.fits - Warning: Selected IRTC dark +2h old!

After open the loop we can see a large focus and stigmatism in the guider star, See image below.
Stopped th preset till. Doug is working on the PSFGUI to get a better PSF in the guider. The subsystem is restarted several times.

  • Astig+Focus after open loop:

Resend the preset.

Fri Oct 26 05:38:02.055 2012 aos.PresetAO.started - PresetAO[152] (aomode:ACE-AO,focalst:bentGregorianFront,instr:IRTC,wfs:FLAOWFS,so1:0.6,so2:-3,ro1:0.6,ro2:-3,mag:7.68,cindex:2.23,elev:nan,rotang:nan,gravang:nan,r0:0,skybrgt:nan,windspeed:nan,winddir:nan) Fri Oct 26 05:38:02.066 2012 - WFS source set to: FLAOWFS

optimize gain procedure stack somewhere, the system is loop closed with optimized gain executing, the AOS is not reporting anything. Alfio is debugging...........

Fri Oct 26 05:50:51.974 2012 - HO Offload request. Z: 0 0 0 0 13.9846 9.5819 6.37051 -34.1605 85.5674 42.7536 30.9417 0 -0 -0 -0 -0 -0 0 0 0 0 0
Fri Oct 26 05:50:54.223 2012 aos.StartAO.failed - StartAO[154] failed - [AOException] Command timeout (code -13107) Arbitrator interface generic error

The time out of rhte Autogain has been updated to 10 minutes. Apparently when the system is doing the autogain procedure the preset can't be cancel till the autogain finish or end with a timeout.

Resend the preset.

Fri Oct 26 06:03:39.156 2012 aos.PresetAO.started - PresetAO[156] (aomode:ACE-AO,focalst:bentGregorianFront,instr:IRTC,wfs:FLAOWFS,so1:0.6,so2:-3,ro1:0.6,ro2:-3,mag:7.68,cindex:2.23,elev:nan,rotang:nan,gravang:nan,r0:0,skybrgt:nan,windspeed:nan,winddir:nan) Fri Oct 26 06:03:39.164 2012 - WFS source set to: FLAOWFS

Fri Oct 26 06:04:23.652 2012 aos.PresetAO.complete - PresetAO[156] complete - dx:nan dy:nan slNull: f1:Filter #0 f2:Filter #0 freq:990 mag:0 nBins:1 nModes:400 r0:0 snMode:0 strehl:0 ttMod:3

Fri Oct 26 06:09:18.910 2012 aos.StartAO.complete - StartAO[158] complete

Applying offsets during the close loop. The offsets are inside adn outside of the patrol field. The AO system responded well. Data taken: 20121026_072134

Tracknum 20121026_051800
Closed Loop Yes
Is OK? No - Warning: Selected IRTC dark +2h old!
AntiDrift ON
nphotons/sub/fr 312.391937
Magnitude 8.195597
# Modes 400
Modal rec Rec_20101024_092604.fits
Binning 1
Frequency [Hz] 989
Modulation 2.996182
FW1 Dichroic 600-1000 nm
FW2 FB850-10
Gain minmax 0.000000 1.085000
Telescope elevation 50.009090
Wind speed NaN
seeing from OL modes 0.559111
lambda [um] 1.600000
exptime [s] 0.010000
framerate [Hz] 66.000000
no. frames 100
SR SE 0.586142
IRTC dark 20121026_072703.fits_cube.fits

The images looks a bit elongated adn we can see the same effect in the Guider. We are applying the lookup table for the seeing limited mode when we are running AO??, for seeinglimited mode we applied Z6:-4 and we can see that this value is applied during the AO. Marco and Doug are investigating this issue..

Several preset were done closeing the loop and applying offsets. Success.


Fri Oct 26 09:26:37.164 2012 aos.PresetAO.started - PresetAO[313] (aomode:TTM-AO,focalst:bentGregorianFront,instr:IRTC,wfs:FLAOWFS,so1:0.3,so2:-1,ro1:0.3,ro2:-1,mag:15.9,cindex:2,elev:nan,rotang:nan,gravang:nan,r0:0,skybrgt:nan,windspeed:nan,winddir:nan) Fri Oct 26 09:26:37.177 2012 - WFS source set to: FLAOWFS

Still updating.

Fri Oct 26 09:27:42.971 2012 aos.PresetAO.complete - PresetAO[313] complete - dx:nan dy:nan slNull: f1:Filter #0 f2:Filter #0 freq:200 mag:0 nBins:4 nModes:5 r0:0 snMode:0 strehl:0 ttMod:6 Fri Oct 26 09:34:26.110 2012 aos.Stop.started - Stop[316] (msg:Preset canceled)

Autogain is not set corrected for the TTM and the recosntructor are not right. Alfio debug and test new autogain for TTM mode.

Fri Oct 26 09:52:07.619 2012 aos.PresetAO.started - PresetAO[327] (aomode:TTM-AO,focalst:bentGregorianFront,instr:IRTC,wfs:FLAOWFS,so1:0.3,so2:-1,ro1:0.3,ro2:-1,mag:15.9,cindex:2,elev:nan,rotang:nan,gravang:nan,r0:0,skybrgt:nan,windspeed:nan,winddir:nan)

The TTM is tested applying offset inside and outside of the patrol field and the result is a success. Some data were taken:


The AO system on DX has been tested with the new 2012H build and we have found an instability of the OSS susbystem which went down many times and the secondaries lost the elevation telemetry forcing the shell to rip and to stop the loop. The OSS problem couldn't be solved and we had this problem for the rest of the night.

Fri Oct 26 12:15:49.765 2012 aos.PresetAO.started - PresetAO[85] (aomode:ACE-AO,focalst:bentGregorianFront,instr:IRTC,wfs:FLAOWFS,so1:0.3,so2:-1,ro1:0.3,ro2:-1,mag:11.7,cindex:-2.76,elev:nan,rotang:nan,gravang:nan,r0:0,skybrgt:nan,windspeed:nan,winddir:nan) Fri Oct 26 12:15:49.781 2012 - WFS source set to: FLAOWFS

Fri Oct 26 12:42:16.255 2012 - Updated variable R.IDL_STAT: 1 Fri Oct 26 12:42:20.206 2012 aos.error - setZernikes(). PSF says: OSS subsystem not running Fri Oct 26 12:42:20.227 2012 - TT Offload request. Tip:0.198908 Tilt:0.456529 Foc:-0.014252 Fri Oct 26 12:42:20.278 2012 aos.error - setShellOffload(). PSF says: OSS subsystem not running Fri Oct 26 12:45:19.025 2012 aos.error - Stopping AdSec Fri Oct 26 12:45:19.182 2012 aos.StartAO.failed - StartAO[6] failed Fri Oct 26 12:45:19.905 2012 aos.error - setZernikes(). PSF says: OSS subsystem not running.

The night end with the system in failure and left in this state for the Software group to see the failure and diagnose and fix for the next nights.

-- JuanGuerra - 26 Oct 2012

I Attachment Action Size Date Who Comment
Screenshot_astigmNOTapplied2.pngpng Screenshot_astigmNOTapplied2.png manage 97 K 30 Oct 2012 - 19:00 UnknownUser Astig+Focus after open loop
Screenshot_ccd39_rip2.pngpng Screenshot_ccd39_rip2.png manage 287 K 30 Oct 2012 - 18:50 UnknownUser CCD39, unbalance quadrants and RIP the shell
Topic revision: r3 - 30 Oct 2012, JuanGuerra
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback