ARGOS DX Swing Arm Incident 2015/04/24

24.04.2015 around 17:00 local time, with the Handover from ARGOS day time activities to the Telescope night time operation LBC red moved into the deployed DX ARGOS swing arm. The T-gusset of the ARGOS swing arm got damaged.The swing arm got moved from absolute encoder position 5872 to 61xx (the proper number was not recorded). This movement corresponds to the radius of the LBC (~500mm).

First Actions

• LBC got retracted
• ARGOS swing arm got retracted
• Telescope was put to horizon
• Visible inspection of:
  • LBC: scratches at the LBC housing was found. No further visible damage could be seen. Rotator tests and camera test have been successful. The Camera was used during night time and showed no strange behavor. (Jeff Urban, Elliott Solheid)
  • ARGOS swing arm (Wolfgang Gaessler):
    • The T-gusset was badly deformed.
    • The two digital calibers have been damaged and the mounting was bent/broken.
• The CalUnit was removed and the broken calibers have been removed.
• It was identified, that some parts of the broken calibers were missing.

#Image(IMG_.JPG, 25%)

• An inspection of MODS followed to check if the missing parts fall into MODS. Nothing could be found.
• As the swing arm didn't show the park position reached, the CalUnit holder was removed. With that the swing arm electronics showed the park position reached.

Further investigations

• [Done] Interlock breaker termination check and fix
  • 2015.04.25 morning: The termination in the upper right tree house was inspected (Gustavo Rhamer, Wolfgang Gaessler)

• 2014.04.27 evening:
  • The termination in the upper right tree house was set to the proper configuration (Kevin Newton).
  • DX: At relay K9H13 line was connected to terminal 14 and now moved to terminal 54
  • SX: At relay K9H12 line was connected to terminal 11 and now moved to terminal 14
  • The DX side was tested.
    • DX PFC could not move when DX ARGOS SW was not in park position.
    • DX PFC could move when DX ARGOS SW was in park position.
  • The SX side was tested.
    • SX PFC could not move when SX ARGOS SW was not in park position.
    • As the SX ARGOS SW was on a spacer and could not be fully retracted. Test open!
  • It was seen, that the software does not show that PFC is not moving in case of interlock and no indication why. Movement command succeeds without movement!!!
  • It is also still possible to move ARGOS swing arms while the swing arms on one side are locked out with key in upper right tree house. The ARGOS enable signal is not pulled down!! Further investigation on that showed, that on the LBTO side the signal is given properly up to the ARGOS junction box.
• [Done] Investigation on the removement of the gusset
  • Is the swing arm stable without gusset attached? Visible inspection gave the impression, YES.
  • How to take apart?
  • 2015.04.25 evening: The broken gusset was taken off. (Lorenzo Busoni, Wolfgang Gaessler)
• [Done] Visible inspection of the carbon tubes
  • 2015.04.25 evening: The DX carbon tubes where inspected. No obvious damage seen. (Wolfgang Gaessler)
• [Done] Check of the gear
  • Is there any damage of the gear possible by moving the arms with force?
  • The Harmonic Drive used can in principal be used from both sides (Ralf-Rainer Rohloff). Therefore, no damage is expected and was also not seen retracting the swing arm after the incident.
• [Done] Check of breaks
  • Are the breaks released in working position or are the engaged?
    • Yes, the break is engaged. Verified by acoustic feedback driving into working position.
  • If breaks are engaged does what does a forced movement do on it?
    • Discussion with M. Lehmitz: The motor has a permanent magnet break. The break will have a force slip with a moment larger than/equal to 22Nm.
• [Done] Check switching of the SX gusset to the DX side
  • Can the SX gusset be moved to the DX? YES
  • Are there any asymetries? Only the calibers.
  • 2015.04.26 during day: The SX gusset was moved to the DX side (Lorenzo Busoni, Wolfgang Gaessler)

Summary

1.1 History:

• e-mail from 20120207 inserted in the e-mail from 20131213 states that in November 2011 the interlock were by-passed the first time and power to the ARGOS swing arms were disabled as measure to a switch problem by Michael Gusick and Bruce Bingham.
• The ARGOS swing arm was brought back into operation for the December 2014 calibration unit alignment. e-mail from 20141216 inserted in the e-mail from 20131217 reports the successful functional tests of the interlock. Further findings of proper functions in the email from 20131217.
• Issue Track 4989:
  • 04.01.2014 again a mail function of ARGOS park position limit switch. The schematics attached to the Issue track show that bridging the interlock was the solution.
  • 14.04.2014 another mail function. The intermediate enabling of the interlock was redone again and the interlock was disabled.
  • 15.04.2014 the relevant ARGOS personal (Wolfgang Gaessler) was informed that the latest wiring changes have been redone. (e-mail G. Rahmer to W. Gaessler 15.04.2014)
  • It looks like the wiring was not set back again from than on.

1.2 The reason for ARGOS limit switch mail function:

• The reason was finally found in Nov/Dec 2013. The initial assumption of a misaligned position switch was proven wrong. The selected switch does not initialize correctly when the voltage after a power cut is to long below 5V.

1.3 Solution to the mail function proposed by MPIA:

1. We proposed to put the controller on permanent 24V. This was done 28.03.2014 (e-mail answer to G. Rahmer)
2. Replace the switches with simple mechanical ones without electronic switch point setting. Replacement happened on SX either February run 2014 or May run 2014 (no documentation to this could be found).

2.0 Incident analyses

• DX gusset complete damaged.
• DX swing arm no visible damages.
• DX swing arm motor and gear.
  • The swing arm gear is bi directional. The ratcheting torque is 5600Nm (e-mail 20100224 T. Bluemchen). A collision torque of 3430Nm was assumed. The gear was projected to withstand such torque. The breaking torque of the motor is given by Siemens data sheet with 22Nm. The gear transmission ration is 160. This would give a torque of 3520Nm, which was applied. Still far below the ratcheting torque. Therfore, it is assumed, that the gear is fine. As result of the restricted use of the swing arm with slower speed as well as from the principal how such magnetic motor breaks work we also do not assume any damage or major performance loss of the motor break. Anyway the break is not safety relevant as in working as well as in parking position it is a redundant measure to keep the swing arm in position. The telescope must not be moved to horizon when ever the swing arm is not in park position or working position. This was already like that before.

Resolution

A spare motor and gear combination will be purchased.

A new gusset will be produced and installed.

The DX switch will be changed to the manual version to be more fail safe.

Action Items

AI1. A new gusset has to be ordered (WG, in progress)

AI2. A spare gear has to be ordered (WG, in progress)

AI3. A spare motor has to be ordered (WG, in progress)

AI4. Some fixation has to be designed and built to change in case the motor and gear (WG, in progress)

AI5. Verification of the interlock (WG, KN, done 20150427)

AI6. Final documentation of the interlock verification (WG, with handover)

AI7. Software feedback in TCS, when ARGOS swing arm is deployed (LBOT, open)

AI8. Change of the switch to a simple mechanical. (WG, open)

Anyway to do:

• IP issue
• firmware issue
• Integration in ARGOS sw

TBD:

• work to be done to make the Calunit as facility. (ARGOS independent software, remote power of calunit light sources)
• do we want to make a test of the break?