DX on Sky

1:12

Two errors

Fri Nov 9 01:08:13.345 2018 aos.info - WFS source set to: FLAOWFS

Fri Nov 9 01:08:18.514 2018 aos.PresetAO{FLAO}.failed - PresetAO{FLAO}[2898] - [AOException] presetAO: FATAL - Timeout waiting for slopecompctrl.R.CONFIG.CUR (-5001) TIMEOUT_ERROR

ARGOS was running a msgdb process that was interfering with our system.

Fri Nov 9 01:16:29.231 2018 aos.PresetAO{FLAO}.failed - PresetAO{FLAO}[2900] - [AOException] presetAO: WARNING - RETRY: Star position outside limits: position -2751457.7,-7774568.5 Limits: from -37.5,-29.6 to 74.5,50.4 (-5002) VALUE_OUT_OF_RANGE_ERROR

6:40
  • Getting ready to go.
  • When trying to do the "Close Loop Step, The pupilcheck ccd39 process failed and the step failed, then we got the following errors, upon retrying.
    -13006 - Illegal command for state AOArbitrator.ManualAcquire 13006 - Illegal command for state AOArbitrator.ManualAcquire 13006 - Illegal command for state AOArbitrator.ManualAcquire 13006 - Illegal command for state AOArbitrator.ManualAcquire
  • "Center Pupils" failed when I retried, but then "Close Loop" worked.
  • GOPT and NCPA are running.
  • It's a double star.
7:08
  • Everything seems to be proceeding normally, the star looks pretty good.
7:15
  • Safe skip, not sure why.
  • Optimize Gain drove Ho2 gains until the loop opened several times. I don't know why, it worked the first time that I used it on this target.
  • Manual gains. /local/aomeas/adsec_calib/M2C/KL_v22/gain/g_tt_1.70_ho1_1.00_ho2_0.30.fits
  • On reclosing the loop, I forgot to turn NCPA off, and it applied 3 times the gain, as GOPT was not on yet.
  • Forgot to take datasets. 073127, 073141 (loop paused) 073218, 073314
  • 073407(loop paused), 073425, 073525, 073627
  • 074036
07:45
  • An edge was rolling and the loop opened, I didn't get a dataset when it happened as too many things were going on.
  • Adsec arbitrator died. 074533
  • Offloading went inactive. and the loop opened.
7:50
  • This thing is really acting up tonight, but I can't tell if an actuator is bad because there is so much junk in the fast diagnostic log about actuator 70, which is not in use.
  • Did the following to not print the actuator warning for number 70: tail -F fastdiagn.R.201811090010.log | grep WARN | perl -ne 'if($_ !~ /CHDISTRMS-0070/){print $_}'
8:04
  • Datasets: 080441, 080456, 080603, 081132
  • Going to a MOS target, non AO
08:28
  • 559 jumped again.
fastdiagn_22442   |INF|  9105211|2018-11-09 08:27:17.570217|   NOTIFYACTFAIL > Failing actuator detected N. 559 Value: 0.000156814 on position (CHDISTAVERAGE-0559)
fastdiagn_22442 |INF| 9105212|2018-11-09 08:27:17.570300| NOTIFYACTFAIL > [FunctNotifyFailingActuator] Alert to AdSec Arbitrator succesfully sent! [Funct.cpp:235]
fastdiagn_22442 |WAR| 9105213|2018-11-09 08:27:17.570463| FUNCTWARNING > FunctWarning CHCURRAVERAGE-0559 0.8
fastdiagn_22442 |INF| 9105214|2018-11-09 08:27:17.572807| MAIN > dumpadsecdiagnhistory succesfully sent to masterdiagnostic [FastDiagnostic.cpp:1294]
fastdiagn_22442 |WAR| 9105215|2018-11-09 08:27:17.575708| FUNCTWARNING > FunctWarning CHCURRAVERAGE-0559 0.8
fastdiagn_22442 |WAR| 9105216|2018-11-09 08:27:17.582381| FUNCTWARNING > FunctWarning CHCURRAVERAGE-0559 0.8
fastdiagn_22442 |WAR| 9105217|2018-11-09 08:27:17.617912| FUNCTWARNING > FunctWarning CHCURRAVERAGE-0559 0.8

8:32
  • New command to get all of the messages for an actuator that is acting up: tail -F fastdiagn.R.201811090010.log | grep 'WARN\|NOTIFYACTFAIL' | perl -ne 'if($_ !~ /CHDISTRMS-0070/){print $_}'
9:04
  • New target
  • We did a teluric star first.
9:21
  • Now moving to AO
  • Bin 4, here we go, PSF star
    -13006 - [Exception] exception ::Ice::UnknownUserException { unknown = AODefs::AOException } -13006 - Illegal command for state AOArbitrator.ManualAcquire -13006 - [Exception] exception ::Ice::UnknownUserException { unknown = AODefs::AOException } -13006 - [Exception] exception ::Ice::UnknownUserException { unknown = AODefs::AOException } -13006 - Illegal command for state AOArbitrator.ManualAcquire Fri Nov 9 09:37:04.846 2018 aos.AcquireRef{FLAO}.failed - AcquireRef{FLAO}[3083] - TcpTransceiver.cpp:284: Ice::ConnectionLostException:connection lost: recv() returned zero Fri Nov 9 09:37:04.979 2018 aos.warning - FLAO: Arbitrator (NT) communication error
  • Stupid problem with the close loop again. I restarted the AO arbitrator, it seems to have done the trick.
9:45
  • Another problem with Close Loop, I re-did Center pupils again and it seems to have worked.
  • Gains 2.3 2.0 1.65
  • 100637
10:14
  • On the science target.
  • Same gains as before
  • Shows mag 16.3
  • 101616, 101710, 101802, 102224
  • Fri Nov 9 10:23:18.445 2018 aos.OffsetXY{FLAO}.failed - OffsetXY {FLAO}[3114] - [AOException] RETRY: Offset outside FoV limits. Target xyz is -73.08 -80.31 55.50 (-5002) VALUE_OUT_OF_RANGE_ERROR
  • It should have ignored that, but the preset keeps running
  • The next offset brought it back
10:26
  • Getting fainter, we must have a thin cirrus layer: 16.6
  • 103135, 103225 (offsetting), 103314, 103427
  • Network out.
10:38
  • 16.8,
  • Pause command stuck
  • Fri Nov 9 10:44:00.252 2018 aos.Resume{FLAO}.failed - Resume{FLAO}[2938] - Illegal command for state AOArbitrator.Ready
  • Fri Nov 9 10:44:21.508 2018 aos.Pause{FLAO}.started - Pause{FLAO}[2938]
  • Fri Nov 9 10:44:21.531 2018 aos.Pause{FLAO}.failed - Pause{FLAO}[2938] - Illegal command for state AOArbitrator.Ready
10:49
  • Checking pointing and re-sending the preset.
  • The loop was closed manually and so things got stuck.
  • Back to magnitude 16.5
  • 105955, 110238,
11:07
  • Back to PSF star
  • Why does the "Close Loop" step take so long?!?!?!?!
  • 112702, 112745
11:32
  • Done

-- GregoryTaylor - 09 Nov 2018
Topic revision: r3 - 09 Nov 2018, GregoryTaylor
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback