AO System Software Weekly Meeting

YYYY-MM-DD

2023-02-03

   AC, BM, XZ, GB, MB, JCG

   Schedule wise, SHARK-NIR Mar 6, LBTI Mar 9 - ECD time in between

   1) ncpa phase 1 (BM)
      * PR created
      * BM will test it on a VM next
      * Description in the PR https://github.com/LBTO/ao-supervisor/pull/174

   2) offsetZAO phase 1 - connection between AOS and ao-supervisor (XZ)
      * no chance to make progress, power outage, DX issues, MG document
      * activity will start week of Feb 13 when GB travels to Italy
      * impact on efficiency, safety etc. (allows not to have human errors)
      * failure mode for Z and XY are not handled the same at asm level. Z=over-current, XY=skip-frames (MG only check for global TT, not other modes)

   3) one step "Fourier" command (XZ, GB)
      * need to review the document shared by SHARK-NIR
      * may require more iterations with them
      * may include a short LBTO document to explain what we understand, something GB may review
      * impact even bigger, because currently need GB+XZ to allow coronograp mode

   For XZ, by consensus - start with 2) and then 3)

   4) AdSec soul-dev/SX vs mmc/DX (XZ)
      * within the context of a possible split of 3 repos: common/asm/wfs
      * general agreement we should merge asm sides first, then split second
   
   5) Bring back the discussion about Elevation limit from GB.
      * GB will write a couple of senses.

2023-01-27

    JH, AC, GB, JP, BM, XZ, MB, GT, AV, JCG
   
   * update on day-time/ecd-time plans and readiness (BM, XZ)
      * https://wiki.lbto.org/AdaptiveOptics/AOSWTest-20230201-checkout
      * hk+rr patches on both asm sides. SX is ready, DX is WIP.
         * if possible, try daytime - but not absolutely necessary. Possibly Monday. Some git/github/deploy work mostly for BM and XZ, but straightforward.
            * but ECD could be on Saturday ... skip daytime if so.
            * possibly: release SX asap, do DX later
         * schedule tbc given the outage yesterday (LBTI probably not available), Weather forecast isn't looking great for AO.
         * we'll know more at 3pm, but there will be ECD time over the 3 nights. JP is coordinating with OK and AB after this meeting.

   * ncpa phase 1 - updates? (BM)
      * BM shouldn't be an issue. Working on it.

   * shark-nir offsets
      * Z-AO, ways forward (all)
      * https://github.com/LBTO/ao-supervisor/issues/173
      * phase 1: (XZ will take this)
         * offsetZAO connection between AOS and ao-supervisor z-stage. fairly easy to implement
            * allows "engineering level" control for shark-nir (they'll sequence it themselves)
      * phase 2:
         * 2.1 implement a new offsetFocus IIF command: will be high-level sequencing/checking similar to offsetPointing2, phase 2.1 is the "plumbing" only (check with YZ or someone)
         * 2.2 make it work, that will use offsetZAO internally, but possibly no changes in ao-supervisor (will discuss internally, maybe PG or GE, or MB)
            * likely to use the same/similar logic that exists for XY, just add Z
            * reference https://wiki.lbto.org/Software/IIFOffsetSequence
               * look at step 12 which is probably (currently) only XY, and just needs to add Z
         * shark-nir hasn't expressed any concerns (yet!) about moves in Z affecting XY ...

   * one step "Fourier" command
      * currently requires support from GB and XZ.
         * currently "ramped manually", would be best to have a "smart/fool proof" command
      * GB and XZ will review the report.
      * next run is around March 6
 
   * generic web-based iif a-la-IRC (lessons from LBTI, new frameworks, utilize code generation)
      * new project for TCS, not LBTI (MB)

2023-01-20

* IT8809 https://lbt.issuetrak.com/CSIssue_View.asp?IssueNbr=8809

   Phase1
   - ncpa is scaled by gopt, but it happens in the RTC
      ncpa from the instrument pov is "constant", in the sense we dont care about the scaling - scaling happens in RTC
   - use a threshold scheme, if less than, no need to apply, covers the (LBTI) case when it doesn't change
      one threshold per mode = vector
      one threshold config per instrument
      one same file as the current table
   - keep the current 10sec cycle, just add the threshold check

   Phase2
   - rate should not exceed 10 sec update, but we could check every 1sec
      log diving for luci
   - add a check for amplitude(s) requested

   PhaseX
   - iLocater "may" want an elevation dependent calculation, who will figure the maths
   
   PhaseY
   - why is it failing in the first place, can we fix it ourselves, bank swapping?

   * AO scientist will be responsible to update the new thresholds
   * BM will take Phase1 and we'll roll it by GB/XZ as usual
      * submit our intention and plan to Arcetri for good measure and feedback if any

* offsetZAO https://github.com/LBTO/ao-supervisor/issues/173

   * AOS implemented, wfs stage implemented
      * need to connect both, but that's the easy part (TaskA)
      * DLM says he recalls even the offsetZAO worked with IRC ???

   * there will be deeper considerations (TaskB)
      * offload to hexapod, speed of the stage, etc.

   * XZ points out the stage plane of motion is not "flat" (a real Z may involves X and Y)
      * this needs to be followed up with Shark-NIR, do they care or not? see TaskB
      * commands should "give" X, Y, Z and use an internal model to drive the stages
      * Argos needed Z due to the ngs vs lgs focus, but it was complicated and didn't happen

   * offsetXYAO has another issue. it's limited to a range, but the AOS / DD reports the demand, not the achieved
      * XZ pointed out the stage plane of motion is not "flat"

   * TaskB small project on its own
      * who could lead this effort?
      * AC will put it on next SHARK-NIR agenda - Thu Jan 26
         * XZ will send a list of questions to AC
         * Maria's email "Re: TCS commands to interact with SOUL" (originally started by Davide)

* ECD patch readiness 
      * BM pressed the send button, JP got it
      * Fabio's changes are already deployed and tested, so no need to add to "our" test activity

2023-01-13

MB, BM, XZ, GB, AC

   * review https://wiki.lbto.org/AdaptiveOptics/AOSWTest-20230125-cameralens
      * keep 1, 3, 4, 5


2022-12-16

MB, GB, BM, DLM, GT, JCG, BR, AC

coming next (TODO: unprioritized):
   * MB - LN issue and patch, follow up - https://lbt.issuetrak.com/CSIssue_View.asp?IssueNbr=8784
      * they have a patch. next run is tbd (LBTB nights)
   * JP - AO config for SHARK/LBTI - same or not?
      * JCG: they are different. but AOS loads the AO table(s) and knows about the instruments:
         * Thu Dec 15 18:31:38.054 2022 aos.info - WFS source set to: LBTIWFS
         * Thu Dec 15 18:31:38.705 2022 aos.info - FLAO: WFS: loading setup for SHARK
      * need to follow up with Jenny (how does AOS tells the WFS it's LBTI or SHARK?)
      * DLM: it is working.
         * to be understood by SWG
   * DLM: LN writing of memory BANKS
      a) we do not have any evidences overloading a small vector on the same bank is an issue (SOUL)
      b) bank swapping makes sense for a big recon matrix, and that is what LN is doing already
      c) similarly to our SOUL z0 patch (both-banks > bankA only), it makes sense for LN to only load its gain to the bank it's using.
         c.1) our SOUL experience shows LN does not need to switch bank to set gains
         c.2) our SOUL experience suggests LN set gains on "used bank" rather than "both banks"
      d) there are more questions about the banks and swapping we will plan to elucidate. And indeed, it seems like it could be hidden from the user.
      * Carmelo had some issue with gains. To be investigated (on their end).
      * Al will follow up about skip frames episode, safe for adsec. There will be an LN post-run meeting.
      * Note with skip frames, they wont be able to load new recons ...
      * Arcetri s/w checks the skip frame counter, check again few seconds later, and if the same, proceed
      * this seems to hint that if he system is skipping, it doesn't want to load
         * check with Arcetri what was the logic/thinking behind that
         * currently at the FSM level, it's fatal, should probably be an error.
   * MB - bank swapping improvement and automation (MG support?)
      * priority is probably medium or low for LN
      * but interesting for us
   * GB - telescope elevation limit with shell set (30, 26 exception)
      * already in place
         * AoArbitrator level
         * "Adam" housekeeper level
      * add a third level: catch it before sending the demand (rather than react to it) - "best effort" because the target could start above but move below the limit.
   * GB - telescope elevation limit with LBC and ASM both deployed or both retracted (90)
      * enforce telescope at zenith if they are both facing each other (something falls and bounce on LBC, back onto the shell - they're 6" apart)
      * use case for both retracted? tbd, maybe a bit more efficient if mixed LBC/M2 nights. Also when aluminizing M1 (but here they have covers - at horizon the bouncing is less likely to happen)
      * use case for both deployed? daytime maybe when using the crane, etc.
   * MB - AO telemetry (e.g. skip frame %, loop rate, *any* important parameters we know about already, etc, = designed such it can grow backward and forward)
      * future development for us, currently nothing in DMS for AO.
      * SOUL has some msql, but "mostly" limited to WFS
   * SE - automate NCPA
   * SE - automate Flex Comp - understand imaging/spectro use cases and Luci built-in AFC
      * DLM has some routines (for both). Will make them available in github. Used once or twice a year.
      * priority tbd. uses IRC automation, Luci image acquisition, etc.
   * SE - DIMM stats
      * AO related or site related, Maybe both. Inform on conditions when to use AO or not. Average, variability, seasonal, etc.
      * what do we want to extract? AO domain discussion, different dimensions
   * JCG - Argos MG support?
      * JCG working on edits for the request document
      * GB will look at the doc
         * careful about overloading MG (ticket #1 not concluded ...)
         * maybe limited to re-return the card for repair
   * MB - auto revision control
      * what does it means, in general and for ao-ops (.gitignore)
      * MB will ingest /local/aomeas/adsec_calib per DLM document (request for feedback? none received) - conclusion of AOG 2022-13-15
   * MB - ingest some of our discussion into IssueTrack or github

2022-12-02

MB, BM, GB, JP, XZ
------------------

Action Item:
   - BM "hiding" w_start/w_stop access so people don't use it by mistake

- rebuild of spare soul wfs server
   - setup and ready to test on Monday, using soul-sx
      - BM check and make a telescope daywork entry
      - GB: stopping the WFS arbitrator affect the ASM
         - rtdb/msgd meltdown?
         - XZ: user manual states that restarting WFS, user should restart ASM, JP will recheck/confirm
            - check with Arcetri on Monday, manuals are D10 and D11
            - with Luci, the restart keeps msgd/rtdb up
            - with LBTI, not so
   - kick start almost done, just some network config to finalize
   - mysql tables creation pending, TBD init script

- set_z0 patch
   - needs to be deployed on DX (already on SX)
      - use BM method (avoids a system restart - see next point)
   - MB will get the approval from EPI because they're commissioning on DX

- discussion with MG about issues on asm DX
   - investigating and diagnosing with Mario, GB, XZ
      - Mario initially suggested a "debug" new f/w, risky, thinking about other options
      - failing single DSP board, or bigger issue? TBD. Gathering more diagnostics or updating HK to collect new data.
      - DX commissioning is proceeding, but everyone aware of the issue and will report and stop if occurs.

- spare adsec machine
   - IT buying 2TB SSD, x2 per server for mirroring, x3 for sx/dx/spare
   - spare machine in XZ office, IT will provide a drive to XZ to get a backup of what's there
   - machine will go to the mountain after that
      - cloning is fine (as opposed to rebuild from scratch)
      - discussion about the procedure 

- database for ASM spare
   - check with Pat Hartley about "e-maint" https://www.emaint.com
   - MG delivered a spare spreadsheet and parts spreasheet, but it's a mess to figure the history

2022-11-18

MB, DLM, BR, AC, GB, XZ, BM
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Action Item
  MB: setup the "template", using either the spreadsheet or doc.
    > MB to be done next week.

  GB/XZ: DX asm, mmc/freeze patch tests and schedule
    > carried over
    > patch for skip frame recovery
    > add as a new activity when Arcetri visits - dicsuss when discussing their visit schedule next week (short week)

  DLM: triage /local/aomeas/adsec_calib (***)
    > 1 or 2 weeks, pending still
    > check email sent by DLM during the meeting https://docs.google.com/document/d/1cIHD2d3IvN4St5UHivFPwBK5y79LDX9Rtxz6M0wp7u0/edit

  GB maybe discuss some new AO / TCS / system integration items

  XZ: decide on step 4.1, new drives, clones, reinstall from scratch, etc.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

> discuss spare adsec machine (GB)
  * readiness, location, etc.
    * in XZ's office
    * originally used for PB development, also used for bcu2k reflashing
    * need a "real" spare at the mountain
       1) reconfigure (if needed) to be a true spare (install latest ASM s/w)
         RAID1 mirroring
         build (make) = 1 hour to build    -
         ao-confcalib is in github         - these two are on the mirrored partition
         what is missing is the /local/aomeas/adsec_calib (see ***) - this partition is not mirrored
         KL29 is the one we use / want for operation, possibly 28 also
         Each side has its own config.
          > ao "can work" as far as tss/safety without adsec_calib
          > anything else (AO, seeing limited, etc) requires it
       2) if PB development comes back, we can get a new machine for that
       3) check level of disk spare (e.g. rebuild the mirror), something for IT
       4.a) XZ check the machine first in Tucson, make sure ready to be handed over
          > possibly get new drive and keep the existing one
       4.b) IT will take the spare from XZ's office, bring it to mountain
          > set it up to current level of s/w
          > need to have both dx/sx available
          > maybe use a tar scheme where we skip the need to rebuild
          > hostname, IP, etc.


> Simone's powerpoint about failure vs rate vs. etc.
  * not shared yet
  * probably will need to be closely scrutinized when available
  * DLM can cross check the results
  * where is the information about loop rate in ASM ??
    DLM does a timestamp diff and computes it
    XZ there is an entry (runAO), but rate can change after runAO (e.g. binning, check flux, etc)
      > only adjusted to lower value? if more flux, could go faster in fact (if same bin) - TBC
      > BR reports seeing it increase
      > XZ, check flux can change it up or down
      > closeloop command then set things up
      > BR: value in the luci headers somehow
    XZ will double check the code
    if not present, we should make it trivial to find (new printf or what not)
    BM: pyarg.202211170000.log:3791:WfsArbitrator_146186|DEB|      3539|2022-11-17 13:07:42.452433|             MAIN > ('set framerate: 1000.000000 Hz',)
    Anyone who knows the recipe, share the recipe
      > XZ fast diagnotic data (use HW frame rate = 1kHz here)
        astdiagn_75329    |INF|   3367133|2021-02-27 03:43:41.559159|             FAST > Processed   315 cycles of  7702 vars in   1.03 s (   305 Hz). HW frame rate   1e+03 [DiagnApp.cpp:832]
      > carefull:
        > 500 Hz is the dummy rate when open loop
        >  50 Hz is the dummy rate when shell rested
      > we'll use this as our first pass

> discuss MG email next week ???
  * run down from GB
    > system aging
    > proposed a procedure for troubleshooting, suspect TBC h/w issue
    > follow up actions for GB/XZ

> handover excercise. spare WFS should be centos6 ready and at the mountain
    > Fabio will send updated instructions
    > people might be on holidays and we'll recschedule as needed



==== meeting ended here =====

> set_z0 release to operation strategy
  > GB will circulate a recap proposal
  > DLM need to check with Jenny and LBTI team

--------------------------------------

- JCG has a new form for end of day check, will circulate to AO-Support, looks neat!
- data handling, pending
- AO s/w multiple version convergence plan, update?
- anemometer test-rig, update?
- bcu2k in Tucson JTAG, update?
  Alessandro reports some issues (cables? bad connection for Mario) will get back to it on Monday
- AOB:
   - Argos
      - workstations at the mountain?
      - SOUL compat using “Alfio’s trick” plans?

2022-11-04

MB, GB, BM, DLM, GT, XZ, JCG
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Action Item
  MB+GB: setup the "template", using either the spreadsheet or doc.
    > carried over again

  GB/XZ: mmc/freeze patch tests and schedule
    > carried over

  DLM,AO: triage /local/aomeas/adsec_calib
    > 1 or 2 weeks

  GB maybe discuss some new AO / TCS / system integration items

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

> set_z0 release to operation strategy

  > caveat
    > daytime tests passed, did not show over-currents
    > may not be related to loop rate
    > similar policy as was done for LBTI at 1700Hz
      > e.g. stop on 1st overcurrent event (stop = no ncpa, or 900, or both)
      > LBTI doesn't re/de rotate, use constant ncpa
        > success at night with LBTI is not an indicator for LUCI
        > LBTI has 'soft' and 'hard' ncpa config
        > both were causing skip-frames, although small acceptable number
          > but the impact of hard, vs soft, vs soft-soft is noticeable (worse to better)
        > was tested night of Nov 3 (MST)
          > softer soft ncpa
          > did they use default values from SX ? doesn't make sense
          > even very small values (10nm) were causing issues - need to check with Jenny and LBTI team
    > possible future plans
      > ncpa updates based on re-rotator angle, not fixed time rate (10 sec)
      > ncpa values are logged? yes
      > skip frames:
        > caused by excessive forces
        > but, the NCPA were small enough it should not happen
          > to be investigated
      > check config SX vs DX for thershold, skip frames, etc (+/- 0.77), are they the same?
    > LBTO can support LBTI comm. what is the plan?
      > the values they tried last night seemed irrelevant (very small) so why apply them?
      > maybe it was a debug effort (as opposed to optics/physicsi/perfomance)

  > GB will circulate a recap proposal
  > DLM need to check with Jenny and LBTI team

--------------------------------------

- JCG has a new form for end of day check, will circulate to AO-Support, looks neat!
- data handling, pending
- AO s/w multiple version convergence plan, update?
- anemometer test-rig, update?
- bcu2k in Tucson JTAG, update?
  Alessandro reports some issues (cables? bad connection for Mario) will get back to it on Monday
- AOB:
   - Argos
      - workstations at the mountain?
      - SOUL compat using “Alfio’s trick” plans?

2022-10-28

MB, GB, JCG, BR, BM, DLM, JP, XZ, GT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Action Item
  MB+GB: setup the "template", using either the spreadsheet or doc.
    > carried over again

  GB/XZ: mmc/freeze patch tests and schedule
    > carried over

  DLM,AO: triage /local/aomeas/adsec_calib
    > 1 or 2 weeks

  GB maybe discuss some new AO / TCS / system integration items

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

- MG support contract
    > MG support starts Nov 1st

- catchup on recent activities
   - z0
    > Reply from Enrico, use 1700 Hz
    > Schedule tests Tue/Wed next week
      > 1.7kHz + NCPA + new housekeeper on SX on TUESDAY
      > same test plan as earlier
      > DLM will coordinate and organize the work
      > parallel tests ocam-bad-frames on the other side

  - new new hk
    > ready to go

- data handling, pending
  > some files have been flushed to the NAS by XZ

- AO s/w multiple version convergence plan, update?
  * set_z0 failed on DX
    > IDL error with BANK keyword on DX adsec
    > tested with LUCI SX at night https://wiki.lbto.org/AdaptiveOptics/20220608_SX_Night
    > ASM DX mmc does not have the patch
    > LBTI needs ncpa @900 on DX for its commissioning plan
      > LBTI SX is using the patched that support the BANK keyword set_Z0, and it is working
      > JP will propose to Arcetri to use Monday daytime for that
      > MMC version needs to be patched for set_z0 on DX on MONDAY
        > if it works, we keep at night
        > if it fails, we revert both ASM and WFS
      > BM will take care of the merge https://github.com/LBTO/ao-supervisor/pull/126/files

  * LBTI-SOUL-comm meeting Monday at 8.00am
  * test start Monday at 9.00am, Fabio to be online figure the issue with config issue

---------------------------

  * LBTI run
  * modification for freeze keyword has been implemented by XZ for mmc
    * XZ contacted Enrico about deploying the change, no answer yet
  * needs to be tested. Prepare test plan, schedule, etc.
    * only needed for calibration (not "typical" regular night operation)
    * freeze vs flat

- anemometer test-rig, update?
  * MW2 worked on it, will produce a document to detail the hardware
  * XZ setup device IP, config, etc.
    > XZ working on the sw modifications
  * it's alive and available IP 192.168.18.162
  * looking at default baud rate, work in progress

- bcu2k in Tucson JTAG, update?
  * Alessandro cable should arrive Nov, may get some free sample. TBD
  * tbd, MB will ask Ale

---------------------------

- stats on AO failures related to hexapod and wfs stages, how big a contributor?
  * keep for later when DLM is here, could be a long discussion
  * other aspects (e.g. patrol field in OT, skip frames, resilience ... etc.
  > hexapod faults occur daytime, configuration is different from nighttime
    > nightime maybe once or twice a year
    > IT6326
  > wfs stages
    > info was collected, used to have sticky motor/timeeout, but addressed during SSD maintenance
    > probably not a big contributor

- AOB:
   - Argos
      - workstations at the mountain?
      desktop shortcuts, simplify operations
      IT recommends an OS update.
      Sebastian is going to visit LBT? part of formal handover, iterating at the moment
      - SOUL compat using “Alfio’s trick” plans?

- power bump -> hk should restore
  - someone checks GB or XZ
  - power bump more likely during stormy weather
  - asm UPS ? MW2 will discuss/review, not sure yet
    system to cover short power bumps in the plan

  MB auto-rev control presentation next week, round up with GT later this afternoon for Luci/sx



> We (Guido and Xianyu) have discussed an issue that was brought up today at the AOG meeting, the configuration control for the ASMs. Our basic structure to achieve this control is the following:
> 1 - identify the list of files required to run the ASM
> 2 - identify the actual files that correspond to this list, this is the task of the person who calibrates the system and creates a configuration tagged with "to be used"
> 3 - develop a mechanism that allows the following:
>       1 - a way backup the current configuration (with a tag)
>       2 - a way to check the current configuration against the "to be used" configuration
>       3 - a way to restore a saved configuration
> 
> These points can be further discussed with the goal of getting a 'configuration control' underway. 
> 
> After we decide on a final method, do you think that the SW group can implement this kind of configuration control?


DLM
  SX adsec, anything before Aug 2022 can be ignored ?
  DX adsec, anything before Mar 2022 can be ignored ?

> conclusion:
  AOscientist should look at 1. and 2. and provide "simple rule"

/local/aomeas/adsec_calib

2022-10-21

MB, AC, BR, BM, DM, JCG, XZ
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Action Item
  MB+GB: setup the "template", using either the spreadsheet or doc.
    > carried over again

  GB/XZ: mmc/freeze patch tests and schedule
    > carried over

  DLM,Ao: triage /local/aomeas/adsec_calib
    > 1 or 2 weeks

  GB maybe discuss some new AO / TCS / system integration items

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

- MG support contract
    > Roberto sent the document, in the hand of Joe and Gene

- catchup on recent activities
   - z0
    > Simone suggested to try 1700 + ncpa
    > Doug will coordinate daytime for tests (with BM, XZ, GR)
  - new new hk
    > XZ working on the changes today, could be ready for next week.
         will check with Arcetri, github review, etc.

- data handling, pending
  > some files have been flushed to the NAS by XZ
- AO s/w multiple version convergence plan, update?
  * LBTI run
  * modification for freeze keyword has been implemented by XZ for mmc
    * XZ contacted Enrico about deploying the change, no answer yet
  * needs to be tested. Prepare test plan, schedule, etc.
    * only needed for calibration (not "typical" regular night operation)
    * freeze vs flat

- anemometer test-rig, update?
  * MW2 worked on it, will produce a document to detail the hardware
  * XZ setup device IP, config, etc.
    > XZ working on the sw modifications
  * it's alive and available IP 192.168.18.162

- bcu2k in Tucson JTAG, update?
  * Alessandro cable should arrive Nov, may get some free sample. TBD


- stats on AO failures related to hexapod and wfs stages, how big a contributor?
  * keep for later when DLM is here, could be a long discussion
  * other aspects (e.g. patrol field in OT, skip frames, resilience ... etc.
  > hexapod faults occur daytime, configuration is different from nighttime
    > nightime maybe once or twice a year
    > IT6326
  > wfs stages
    > info was collected, used to have sticky motor/timeeout, but addressed during SSD maintenance
    > probably not a big contributor

- AOB:
   - Argos
      - workstations at the mountain?
      desktop shortcuts, simplify operations
      IT recommends an OS update.
      Sebastian is going to visit LBT? part of formal handover, iterating at the moment
      - SOUL compat using “Alfio’s trick” plans?

- power bump -> hk should restore
  - someone checks GB or XZ
  - power bump more likely during stormy weather
  - asm UPS ? MW2 will discuss/review, not sure yet
    system to cover short power bumps in the plan

  MB auto-rev control presentation next week, round up with GT later this afternoon for Luci/sx



> We (Guido and Xianyu) have discussed an issue that was brought up today at the AOG meeting, the configuration control for the ASMs. Our basic structure to achieve this control is the following:
> 1 - identify the list of files required to run the ASM
> 2 - identify the actual files that correspond to this list, this is the task of the person who calibrates the system and creates a configuration tagged with "to be used"
> 3 - develop a mechanism that allows the following:
>       1 - a way backup the current configuration (with a tag)
>       2 - a way to check the current configuration against the "to be used" configuration
>       3 - a way to restore a saved configuration
> 
> These points can be further discussed with the goal of getting a 'configuration control' underway. 
> 
> After we decide on a final method, do you think that the SW group can implement this kind of configuration control?


DLM
  SX adsec, anything before Aug 2022 can be ignored ?
  DX adsec, anything before Mar 2022 can be ignored ?

> conclusion:
  AOscientist should look at 1. and 2. and provide "simple rule"

/local/aomeas/adsec_calib

2022-10-14

MB, AC, BR, BM, GT, GB, XZ
~~~~~~~~~~~~~~~~~~~~~~~~~~

Action Item
  MB+GB: setup the "template", using either the spreadsheet or doc.
    > carried over for next week. MB create a template and submit to GB for review

  JCG: add "s/w (reverted to/set to) night-time version" in the daytime sign-off email
    > also discussed at AOG yesterday. It is better now, with point of contact for daytime tests.

  JCG: z0 test, available Wed and Thu, not Fri - create telescope time request
    > done

  GB/XZ: mmc/freeze path tests and schedule
    > carried over

  MB auto-rev control presentation next week, round up with GT later this afternoon for Luci/sx
  GB maybe discuss some new AO / TCS / system integration items

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


- MG support contract
    > MB will remind Roberto to send us a final version of the contract at Monday's ASM meeting

- catchup on recent activities
   - z0/hk test update
    > 4000 iterations at 900Hz, no issue
    > Simone asked 1.5kHz, SOUL acceptance "spec"
    > *only closing the loop* (i.e., no set_z0 tests yet)
      > found out 1700 Hz is ok
                  1500 Hz troublesome
                  1350 Hz troublesome
    > thus, next step ???
      > discuss with ATT (see GB email summary for more complete details)
      > 127Hz t/t resonnance, "fake pie-shape" (see IT 7697 and maybe others - JHill mention he could see it with OVMS)
      > IIR filter implemented by Arcetri (Guido Agapito)
      > we had configs for 1350 / 1700, not 1500
      > should we run 1700 + set_z0 ?
        - maybe discuss at ATT
        - how is the loop rate determined, AO guide star magnitude, lookup table = discrete or continuous ???
          - probably but __TBC__
            > filter(/s) is discrete ?
            > wfs rate is continuous ?
          - how does the IIR "tune" itself
    > do we keep the patches for ECD?
      > GB/XZ looking into another improvement for HK (timeout on telemetry data)
      > set_z0 is a good patch to have, but probably not the cause of all the issues
      > GB/XZ will make a proposal
      > next ECDs are Oct24-26, then Nov 5 onward


- ASM meeting Monday 16 at 12:30

#
#

- data handling, update?
- AO s/w multiple version convergence plan, update?
  * modification for freeze keyword has been implemented by XZ for mmc
  * needs to be tested. Prepare test plan, schedule, etc.
    * only needed for calibration (not "typical" regular night operation)
    * freeze vs flat

- anemometer test-rig, update?
  * MW2 working on it

- bcu2k in Tucson JTAG, update?
  * Alessandro getting some cables, schedule next week

- stats on AO failures related to hexapod and wfs stages, how big a contributor?
  * keep for later when DLM is here, could be a long discussion
  * other aspects (e.g. patrol field in OT, skip frames, resilience ... etc.

- AOB:
   - Argos
      - workstations at the mountain?
      desktop shortcuts, simplify operations
      IT recommends an OS update.
      Sebastian is going to visit LBT? part of formal handover, iterating at the moment
      - SOUL compat using “Alfio’s trick” plans?

- power bump -> hk should restore
  - someone checks GB or XZ
  - power bump more likely during stormy weather
  - asm UPS ? MW2 will discuss/review, not sure yet
    system to cover short power bumps in the plan

2022-10-07

MB, GB, CV, XZ, JP, BR, JCG
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Action Item
  MB+GB: setup the "template", using either the spreadsheet or doc.
  JCG: add "s/w (reverted to/set to) night-time version" in the daytime sign-off email
  JCG: z0 test, available Wed and Thu, not Fri - create telescope time request
  GB/XZ: mmc/freeze path tests and schedule

- MG support contract and its implications
  - email group e.g. microgate-support@lbto.org
    * LBTO staff only
    * first request to MG sent by GB or MB, but includes the microgate-support@lbto.org
    * cc'd on replies to it's available for anyone to follow.
  - form for requests ?
    - send questions / requests to LBT tech reps (GB/MB), filter what we may already know
    - ESO uses a form scheme
      - reply and follow up from MG, came in the same form?
    - keep track of activities:
      - GB like google doc, but use an easy to find location (e.g. AOG drive / microgate)
      - use a template (structured and easier to describe the request)
      
      e.g. use google form -> generates spreadsheet - allows attachments?

      sheet: pro, easy to get a summary. con, lose the document aspect
      google: search very fast, better than wiki

- improving communication with sciops, day time tests and notification about rolling things back
  - explicit better than implicit, send email to telescopework@lbto
  - maybe use a template (check out list, similar to JCG email)

- catchup on recent activities
   - z0/hk test update? plans for Fri. (or Sat / Mon) ECD?
    - hk test successful, but new features could be desirable
      - timeout set to 1 second, could be a config option? no
        when iterating over the bcus
      - could be released, but other developements considered.
        Best wait next iteration GB/XZ are working on.
      - better planning for "night" release
    - z0 no entirely completed (3000 vs 10000 updates)
      release for ECD? or continue with the test
      DLM is away
      SOUL+DX next week (Wed, Thu, Fri, cont Sat)
      JCG will coordinate the tests next week

  * Conclusion: we'll try schedule for next week ECD, next Saturday Oct 15 = split night with PETS.

- data handling, update?
- AO s/w multiple version convergence plan, update?
  * modification for freeze keyword has been implemented by XZ for mmc
  * needs to be tested. Prepare test plan, schedule, etc.
    * only needed for calibration (not "typical" regular night operation)
    * freeze vs flat

- anemometer test-rig, update?
  * MW2 will be back Mon and start on it
- bcu2k in Tucson JTAG, update?
  * Alessandro getting some cables

- stats on AO failures related to hexapod and wfs stages, how big a contributor?
  * keep for later when DLM is here, could be a long discussion
  * other aspects (e.g. patrol field in OT, skip frames, resilience ... etc.

- AOB:
   - Argos
      - workstations at the mountain?
      desktop shortcuts, simplify operations
      IT recommends an OS update.
      Sebastian is going to visit LBT? part of formal handover, iterating at the moment
      - SOUL compat using “Alfio’s trick” plans?

- power bump -> hk should restore
  - someone checks GB or XZ
  - power bump more likely during stormy weather
  - asm UPS ? MW2 will discuss/review, not sure yet
    system to cover short power bumps in the plan

2022-09-30

MB, GB, BR, BM, DE, DLM, GT, JP, SH, XZ

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- update on set_z0 / hk logging patch / loop rate /test and schedule (XZ, BM, GB)
  - hk patch has been reviewed and test plan in the work
  - daytime test (gains=0) scheduled for next Tue, Oct 4 + Wed, Oct 5
    already on telescope daytime calendar
    team: BM, XZ, GB, DLM
    nb: SOUL team working on DX, happens in parallel
  - if pass
    - set_z0 test will continue on (gains > 0)
    - if pass:
      - option 1) revert, but plan for release at next ECD night
      - option 2) keep
      - half ECD Fri, Oct 7 - more options Oct 8, Oct 10
        - we'll use option 1) and wait ECD


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- update on moxa and anemometer test rig (XZ)
  - new unit should be at the mountain, Cindy tracking its location
  - MW2 will set it up, back from vacation on Oct 10
  - tcpdump capture script from BM
    - running currently, automatic email and pcap captured files
    - runs on both sides
  - XZ has a new new patch (ops s/w). Create a PR so we can review it.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- update on open AI wrt. AO confcalib, DLM code in github (DLM)
  - already discussed during AOG

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- update on bcu2k / JTAG (if needed, MB)
  - device has been ordered and being expedited, should arrive next Mon Oct 3
  - Alessandro will update Mario about our way forward

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- disk space, log and data management, uniform system services (IT, dns, background jobs, etc) - recycled from 20220902
   - document from XZ https://docs.google.com/document/d/1fWAWboZ2nZoJbF_1I5zK25fii2_Y6NNv/edit
    - only covers the ASM
    - expand to include all WFS, Argos, etc.
    - *** XZ will lead the implementation with help from IT/SW ***
      - e.g., SH suggest we could run the job on the synologies rather than the machines/VMs
    - review how the email alarms are generated (e.g. if level at 80, alarm if it exceeds)
    - guideline on using the data (currently done on the adsec machine, future = use VMs)
  - DE moving flao_data on a new share.
    - Initial rsync about to complete
    - DE will coordinate with XZ about swapping the mounts today Sep 30
    - the export will be tighten (replace * to the ones actually used = asm + soul-luci)
    - LBTI isn't using the NAS mount (they only mount the adsec)
      - data handling process is manually done on LBTI, should be automated
    - investigate 2-steps copy
      e.g. copy wfs to adsec, and later copy adsec to NAS
      follow up with Fabio et al. - could be historical
  - mount situation, available to world? needs to be discussed
  - how are we doing with space
    - 49 TB total
    - 26 TB free after rsync (flao-data = 12TB)
    - SX/DX asm each 5.5TB used, possibly 10TB more to offload on NAS
        - mostly AO calibration and AO loop data
    - do we have data that is already > 5 years? probably
      - could use a leveling strategy ("pruning") using disk space rather than age
    - are sx/dx following the same structure ?
    - wfs soul/lbti should also be aligned

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- AOB

    - PB and ATT
      - GB working on presentation with XZ
      - meeting will be post ATT, GB will send a meeting invitation, week of Oct. 10
      - GB/MB will catch up (this afternoon 3:00-3:30)

2022-06-24

MB, JP, AC, AV, BR, BM, GT, GB, JCG, 

ESM on DX for ECD Jul12-13 ?
  - TBD daytime work needed before can be tried at night.
    - DX asm for restart
    - some actuators disabled recently
    - ongoing discussion (check flat with 4D, etc)
    - probably need more time to calibrate, etc > 2 weeks
      - ESM is 12 modes
    - to be iterated during AOG

BM, MB, XZ review comments in PR120
  - pending, not critically urgent, waiting for Jacopo report
  - phase1 = allow the mode they use, and the amplitude < they used.

GB iterate TSS remote power switch with JR and ENG
  - om-going, in the hands of GB and others

JC, DM, BM review comment in github Issue115 and disable ncpa while optimize discussion
  - comments from Alfio in the wiki QA
  - will iterate

AOB
---
  - comments on test plan sent by BM?
    - GB overall good but busy this week
    - nothing critically urgent
    - for restart, 1-2 nights for Luci
      - BM send input to BR by mid-Aug to include in the plan
  - ao.sys.sw meeting will be suspended until Fri - Aug 12
    - other restart planning meeting usually planned mid-Aug
    - M3 out (sx) until Aug 24th
    - mmc work before shutdown w. new actuators removed (GB will follow with a meeting)
  - during SSD
    - SX removed by Jul12-13, put rigid (XZ will be here, he and others in SWG are familiar with the procedure)
      - update the PDF instructions to inc. the f/w change (GB)
    - DX powered off, complete shutdown, cover, etc (usual procedure)
  - MB off/remote Jun25-Jul31

2022-06-17

MB, JP, CV, BR, JC, GT, DM, BG, BM, XZ, ??


++++++++++++
Action Items
++++++++++++
  - BM runtime, upgrades recap
  - BM, MB, XZ  review comments in PR120
  - GB iterate TSS remote power switch with JR and ENG
  - JC, DM, BM review comment in Issue115 and disable ncpa while optimize discussion



++++++++++
Discussion
++++++++++


- dx mmc freeze         https://github.com/LBTO/ao-supervisor/pull/140
    GB - modification ready in IDL. Waiting for Armando to review.
         writing the rationale pending (other power outage)
         only needed for AO, not SL
         no Luci-AO on DX, no LBTI until restart (early September)
         LBTI impact (to have the patch ready) is not certain, see Jun runs
    XZ - PR129 is mmc (that we are running now)
         PR140 patch is ready to go (deployed, but not used)

- dx mmc + "latest" (patches we have on SX)
    merge ASM
    merge WFS (LBTI handled by Fabio)
    daytime test, etc. Update Wiki AO Software Test. BM will make an update, interested parties can look at it.
    BM will take the lead for this activity (schedule, liaise with FR, AO support, etc)
    in fact, SX stable == SX latest

- housekeeper changes
    being reviewed in github, BM test on VM

- ncpa ICE              https://github.com/LBTO/ao-supervisor/pull/120
    review other comments BM, XZ, MB, APU in the PR
    range check missing. Logic to implement?
    allow IIF interface for SHARK-NIR. Note: they only use mode XX (xx TBC)
    phase 1: allow the mode they use and val < the max (35?) they used
    phase 2: more complete solution (lots of work)
      note: LUCI/LBTI currently have some checks for ncpa? probably not
            LBTI doesn't use ncpa that often, iLocater use slightly different ones but
            LUCI has a table + rotator dependent
            The GUI allows to type in numbers? no
    this feature is about IIF/ICE, existing instruments use lower level directly from WFS
    Al shared an update (email) from Jacopo, will be discussed Mon Jun 20 (although mostly about shipping)
      need more iteration how they intend to use NCPA/asm vs. Internal/DM
      GB thinks it is internal DM
    The IIF should be generic

- fault recovery        https://github.com/LBTO/ao-supervisor/pull/79
    Patch ready since end of 2021
    May need more work, keep on the back burner or phase it. GB and XZ will decide
    Disable button is good for now

- revision control      Recap table what *runs* where, realignments pending/needed?
    BM will create a recap / table
    disable button: on SX, missing on DX? To be checked
    also add what could be tested next (day) and future (latest)

- IDL license server    status?
    XZ: back to lbto license server
    GB: remote vs local server, dependencies
    XZ: local version is old, only one seat
        IDL checker actually spawn an instance to check if it's working ... so would defeat the single seat
          may work for 1 minute ... but dies when the ping kicks in
        Harris said our license is not "upgradable" within the current arrangement
        Things might get better as we upgrade OS, IDL version, etc.
        Steward license server could be used if we had a complete failure of our own one

- lbti wfs INDI         PG looking into it
- log analysis          wfs freq/mode, from where?
                        maybe in the fastdiagnostic.log > can be use if not available for history
                        JC will share info by email (update modulation ...)
                          WFS > Python command: WfsArbScripts.updateModulation(self, sensor = 'FLAOWFS', instr = 'LUCIFER', freq = 900)
                        IDL log may have some too, runAO possibly not used during the day (AOS)
                        Ocam2k may have limited discrete freq. (or exp time)
                        params are "tuned" only if you do AoCheck (check flag on the AO GUI)
                        Processed    50 cycles of  7702 vars in  0.996 s (  50.2 Hz). HW frame rate    50.2 [DiagnApp.cpp:832]

- TSS                   remote power switch, James Reidl email
                        GB will join the discussion
                        moxa/adam process? e.g. running from a VM - for now, no action

- mmc on SX             restart schedule?
                        to be discussed next week? will be more clear once we have BM recap.
                          SX will be in the lab!
                          idea to create a timeline for convergence

- AOB

    JC - disable ncpa during optimize gain.
       - is it implemented?
        DM: yes it is, done by APU long time ago
        BM, DM and JC will regroup and let us know - (BM has some info from APU)
          https://github.com/LBTO/ao-supervisor/issues/115

2022-06-10

- z0 daytime test outcome and schedule for next ECD (5 minutes)
  complete and passed

- AOS/RTDB patch (5 minutes)
  complete

- ocam2k lost frame, network, LBTI INDI branch out discussion (5 minutes)
  MB will send an email, reboot dxunit as a catch all
  EPI confirmed they would reboot on next test, but meanwhile system was used at night and no issue (uptime is 135 days)
  MB talked to PGR to check it out

- AO ops logging OSA tool/APU scripts update (5 minutes)
  MB looking at APU script

- steps to go > 900Hz (40 minutes)
    - preliminary stats, to be refined, MB provided a snapshot, APU working on it separatly
    - different oc causes - some fake, some true. hard to extract from log
      - e.g. shall is rested, cannot have any real oc (physically)
      - e.g. seeing limited OC, some correlated to wind and facing into it
    - change close loop count vs. close loop duration
    - Steve's student collected and annotated LBTI stats, google spreadsheet
    - 1-wk close scrutinity data, XZ shared link to google drive:
      https://drive.google.com/drive/u/0/folders/19CGOUwFXbBcOIO9ToZGNj9-3ZuRbK1FL
    - APU's script to be extended to include OC events, but they come from housekpeer/fasdiag, he looks at arbitraor logs (I think), easy enough
    - currently, the conclusion is that it's a statistcak exercise, daytime may not reveal much because it's so sparse


- AOB
    - "bug" in StopAO with mmc.
    GBR and XZH will provide a doc, there is a flag "hold"
    normally will flat the shell. Hold would leave it where it is.
    existing code does not work well with mmc.
      implementation might have been problematic anyways
    patch submited, allow freeze and which modes to freeze, discussion with Armando and Marco, pending
      candidate to test next week.
    in parallel, purpose of the hold and deos it really work
      used as a diagnostic feature (e.g. before it skips to understand the issue)
      need more iteration with Enrico to understand his use case
      history: might have been a feature requested by LBTI, restart with less modes
      note the current pause does that (freeze in place)
      theory: it was meant to keep t/t, possibly because the offload is not working properly
    JC will be available for test
      XZ is taking some time off...
      We need to accept the change on soul-mixed-control, Brandon will check/fix the github and PR situation
      GB will coordinate the test/date/plan - 2 hours minimum
      if it works, next ENG is July 4
        day time test to check
        not needed at night, needs more iteration

    GB will open the discussion with Arcetri
    WFS eng gui has an option to freeze or not (checkbox)
    is it related to the pause+skip and we discussed to flatten during pause (possibly keeping t/t to offload itself)
    offload to hexapod is 1Hz, but applied porbably less.
      offload goes to PSF and the TCS decides what goes where (TBC)
    offload to M1 probably even slower (10-30sec?) - but probably not an issue (see below)
    "issue" with ncpa offset makes the job harder on the wfs
    LBTI has sot/hard ncpa, maybe we should do that with Luci also
      bin1 + ncpa is definteyl a bad combo
      ncpa with luci is to be defined, may cause the psf to be same if not worse
      need some more discussion and evaluation - extreme perf vs stability, usual discussion
        possibly an option in PIT/OT, like a check box - many use case perf is not essential
        OSCO always failing at bin1 (bright because twilight)
        does bin1 work at all (even without ncpa)? i.e. reliability
        who's allowed to edit the ACE/AOS tables
        need to confirm LBTI 400 modes, LUCI 560?

    - SOUL-comm
      RIXes ...
        SoW is in most part h/w spec
        there is some tables/requirements for FLAO, Soul is required to operate "as well"
        FLOA was accepted based on requirement (on sky) - where is the doc?

google link wiht SOUL documents from Enrico:  https://drive.google.com/drive/folders/1_YSZBX1MBahkttMcrwgV9b-PKvUqBI1a
memo of understanding: https://drive.google.com/file/d/1q2YNl4IwhUwrmuV2ZbTb4zOUZXxcxvdw/view

2022-06-03

- z0 daytime test outcome and schedule for next ECD (5 minutes)
  date ? http://doc.lbto.org/web/LBT_2022A.pdf
    JC: test ok, BM sent email instructions
    Mon 6 June (Tue 7 + Jul 4 also available)
    ECD: regular observation using NCPA, second night probably. JP will be on AO
    JP: start with dev version from the start of the night - all agree
    BR: writing the sciops OT scripts and test plan, use "known to be stable AO" scenario

- AOS/RTDB patch (5 minutes)
    June 6 or 7, when Yang comes back
    try again (once) may work. will tell us more and if it still fails, more intrusive reap/re-establish link
    msgs/rtdb is what we have, has occasional issues (in general, not limited to AOS/AO), using established 3rd party middleware (e.g. custom RPC that became ICE) would bring a lot more refactoring, avoid if possible or longer term
    IT8397, IT8528 are different. but close enough, timeout -5001 on read or write/ack-handshake

- ocam2k lost frame, network, LBTI INDI branch out discussion (5 minutes)
    ocam2k going out live (display, but also gaps in TN) , every 10s or so
    reboot lbti-wfs-dx as pro-active measure, pass it by Fabio
    remove INDI ? LMircam might be using it (other than fill some fits keyword)
    used for other housekeeping tasks (e.g. power on/off equipment etc), should use 200% CPU or coredump in a loop
    ITs with ocam not live, 8250, 3957 - remind Enrico or others to enter ITs
      not same as badframe, not same at ocam2k not live from ITs, does not prevent operation
      no need to make ITs for now
      MB will send an email, reboot dxunit as a catch all

- AO ops logging OSA tool/APU scripts update (5 minutes)
    in progress

- steps to go > 900Hz (40 minutes)
    - preliminary stats, to be refined, MB provided a snapshot, APU working on it separatly
    - different oc causes - some fake, some true. hard to extract from log
      - e.g. shall is rested, cannot have any real oc (physically)
      - e.g. seeing limited OC, some correlated to wind and facing into it
    - change close loop count vs. close loop duration
    - Steve's student collected and annotated LBTI stats, google spreadsheet
    - 1-wk close scrutinity data, XZ shared link to google drive:
      https://drive.google.com/drive/u/0/folders/19CGOUwFXbBcOIO9ToZGNj9-3ZuRbK1FL
    - APU's script to be extended to include OC events, but they come from housekpeer/fasdiag, he looks at arbitraor logs (I think), easy enough
    - currently, the conclusion is that it's a statistcak exercise, daytime may not reveal much because it's so sparse


- AOB
Topic revision: r22 - 03 Feb 2023, MatthieuBec
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback