AO System Software Weekly Meeting

 


NEXT

   XX, XX, XX

   Add any items you would like to be discussed here, with your initials in brackets [XX]

   XZ, BM, MB, GB

   - NCPA/Fourier mode application requirements [MB]
     https://docs.google.com/document/d/1V3jLOMD9XAtqL2M09T00W0hSgguQTNWyPtkKRwnRhXg/edit

     - currently, there's a command line script (for fourier/shark-nir), and some GUIs (for luci), but nothing coordinated
     - needs some kind of MVC (model view controller), e.g. what is currently applied, etc.
     - need to iterate with all instruments on requirements
     - currently, to enable ncpa:
         1) "spare" slot in the WFS slopes is set to 0/1 to enable/disable
         2) "spare" vector of 672 (line or column = vector of modes) in the 672 (modes) x 2848 ("slopes" == "2x supabs", but in fact has some "spares", e.g. for z0 values)
            B0 (in MG ref document) bankA = Reconstructor
     - currently not using the bank swapping
         - did we patch the code to write only one bank ?
            - adsec has option to do A, B, or both. Currently (TBC) we use single bank=1

   - hightlights from AO4ELT7
      - all big telescope getting delayed
         - GMT maybe getting closer, maybe 2028
         - TMT still struggling with site decision
      - extreme AO hot topic
      - DMs developments.
         - TNO, still some challenges
      - nice presence from LBT, talks from Enrico, Steve, OSU
      - it was hot and humid, no AC !!!
      - meeting(s) with ADS/MG
      - XZ shared picture of the posters https://drive.google.com/drive/u/0/folders/1hJkMBPniSoadnEbaplmUuFQHy2H1yklu

   DLM, GB, XZ, SR, MB, BM, JP

   - adsec /ao-meas files to bring back from cleanup exercise
      - only used for tests, if only a few files, can go in github. Can be removed in the future.
      - we could add a Readme,md

   - what is the general plan for Magellan unit? [MB]
      - it's TBD, need to check with MarkS. We know the current plan is to ship to Italy.
         - ADS + MG will take the job (more bandwidth)
            - ADS has all the mechanical/optical capability
            - MG electronics vs controls
               - the Magellan is using a different version of the s/w (a very old branch?)
                  - there will be some s/w and config effort to check all this
               - if we ship the electronics to Italy, we should include a clause with MG to validate the "electronics" are good (dsp/fpga, etc)
                  - the s/w was developed by Arcetri, unknown if MG has capability to test the shell (on their own)
                  - we (LBTO) have inherited the Arcetri s/w side, so we could do it. This could be a consequent amount of work (/time).
         - history:
            - TS1 (rejected by LBTO) was "loaned" to Magellan / Phil Hinz (reduced diameter about 1" because there was crack)
            - ref body was also the first LBTO iteration, and slightly damaged at the edge.
               - repair vs replace will be discussed (cost/time/etc)
            - all this was not an issue for Magellan
            - should be compatible for LBTO both side, just s/w / calib config
               - cabling/fiber to be dx/sx ? not straightforward, but can be managed
               - optical prescription slightly different, but can be handled by new calibrations
               - spacing of actuators? shouldn't be different, maybe the gap at rest will be a bit different
                  - alignment of actuators/magnet TS/refbody? will be discussed further. the magnets should be the same, the actuator holes could be slightly off, but maybe within tolerance
      - there will be some follow-up meetings during AO4ELT

   - BM patches, when to test?
      - will there be AO before end of semester (see LUCI2 issue)
         - stuck detector focus stage (inside Luci), but within AO parameters
            - z-range of stage? should be ok for AO/luci2, some filters, but not everything
         - being discussed as we speak, will be more clear next week. GT doing some tests today. Stay tuned _after_ the 3:30 dpm
         - how often do we recalibrate?
            - for IM/Recon, done when system removed and comes back to telescope (thus, infrequent)
            - focus offsets etc. more frequently (e.g work on the AGWs)
   
   - volunteer to run the meeting during July [MB]
      - MB off/remote till Aug. There will be a local POC. Available 1h/day or so.
      - next week skip (AO4ELT)
      - then SSD mountain activity
      - no meeting until Aug or thereabout (same as AOG)

   AC, BM, GT, MB, DLM, GB, JP, JCG, SR

   - bcu2k, test new dsp firmware for zero slopes fix [MB]
      - history:
         - JCG data was collected and shared with Arcetir in the past
         - they came up with a s/w fix, but also worked with MG to get the issue addressed in DSP
         - we now have the DSP fix (new ldr file), want to test it
         - there is another issue (separate not discussed here, about negative pixel) Alessandro running test in the lab
            - suspicion there could be EMI issue on the telescope (why they test it in the Lab Tucson)
      - we should have a test plan
         - we can recycle a previous one.
         - JCG will create one, available today so it can be reviewed for test on Sat/Sun
         - we want to see if the new DSP does not break anything.
            - knowing if it really fixes anything could take a lot of time.
            - if the issue repeats, we can discuss the s/w fix
         - GB
            - we don't know the internal of the DSP code internal
               - we do have a switch-bcu description (where ??), but not the slope-bcu (bcu2k)
                  - or rather, only "conceptual" / algorithmic. No details.
                  - the patch dsp is basically coming from the same people who created the original bcu2k
                  - original bcu2k (SOUL upgrade) was tested extensively in lab
               - this was developed mostly between Arcetri and MG
            - we should follow the same / similar acceptance plan
               - difference is h/w on telescope
            - experience has shown MG f/w can take a lot of iteration. But that was switch-bcu (ovms-tt, much more complicated and really new, as opposed to a slope-bcu fix)
         - SR: insist on MG providing a report / factory acceptance test
         - MB: experience with MG support contract (link at top of page)
         - JP: happens about once every 10 hours of operation
         - DLM: system was delivered. Trust our vendor to know what they do. We can move forward with our test plan, we know the issue quite well, and familiar with the system.
         - MB: MG own knowledge of the LBT system is quite old now
         - GB: not about trust, trust and verify. Knowing what they changed will help determine what we want to test.
         - JP: it's a blackbox. Impact science. Timeline for the process to complete?
         - JCG: does MG even have a bcu2k in Italy?
         - JCG: he has a description of the changes, but verbal only
         - SR: down the road, we should formalize the work with MG.
         - DLM: if slopes go crazy, worst case the asm will take care of itself (as it does when the loop blows up during operation)
         - SR: run open loop. apply known modes, check slopes are what we expect. build confidence.
               allow to move on the next step, low gains, etc.
         - JP: LUCI2 is likely dead for semester (trouble shooting still ongoing), LBTI wont come back until semester. So the schedule could be favorable for us.
      - Plan B ? Use the s/w patch on short time scale, get the dsp sorted out properly in parallel
         - would rather not

      *** Plan ***
         - no rush for tests this weekend
         - JCG create the high-level test plan, circulate to AO@lbto
         - JCG ask Arcetri a document describing the changes he heard about.
            - ask for anything that could be tested internally, or to watch for (unlikely)
         - next week we'll know if LUCI2 is back or not

   - adsec /ao-meas files to bring back from cleanup exercise [MB]
      - give above plan, no rush. XZ will be there next week.
      - JP sent an email to Guido, etc. about what she did.

GT, DM, GB, JCG, BM, XZ, ACo

   Add any items you would like to be discussed here, with your initials in brackets [XX]

- IT 8893 - Got response from Alfio, Timeout can be reduced. BM will make a test plan. - AO (ASM, SOUL, Argos) cabinets for computers, electronics, etc.
- there is space.
- XZ/GB: Would prefer to keep computers in computer room.
- *AI: JCG/ACa will check next week where everything can be stored.* - AO laptop, Argos laptop, etc. - merge into one? - win-xp? backward compatible win-10/11? virtualize?
- AO Laptop: MG engineering panel (ASM+WFS, BCUs) - just needs ethernet
- XZ: IT says virtualization would be a problem because Eng. Panel would be incompatible with new HW. Was never tested, though.
- *AI: GB to check with MG if Eng. Panel can be abandoned and only Matlab is necessary. Maybe they can give us Eng. Panel source code?*
- *AI: XZ/ACa to verify that our only two needs are changing IPs and flashing firmware.*
- Needs stable location with check-in/check-out
- If we use a laptop, the current laptop has limited shelf life -- will need to be replaced. We could have a backup.
- ARGOS: 2 laptops, 1 for swing arm controllers, 1 for PLCs, laser tuning
- DM: If we need a laptop for AO, then we can decide if we want to/can merge with ARGOS laptops.
- VOILA (/Grafana?) - how helpful?
- Can discuss in a future meeting.
- SOUL DX master-and-fwFix has three parts:
- fwFix ("filterwheel 2", for LBTI): JP: this should be released
- OCAM bad frame notification (colors in GUI, message in logs): XZ: not too important
- zero slope GOPT value update (reset_gopt): DM: could upgrade, but just disable in config.
- XZ: Continuous loop is bad. GB: Maybe it can run only when the loop is closed.
- Make clear that even if we deploy this, it is not an acceptable longterm solution.
- *AI: Meet with Arcetri in 2 weeks before AO4ELT. XZ gone next week. Al will send an email. Perhaps 6/19 or during AO4ELT, or can be resolved over email.*

   JP, XZ, MB, DLM, BM, AC

   - ECD time Sunday night. Plans for AO s/w upgrades?
      - nothing major at the moment, email alarms scripts, etc.
         - PR for anemometer email alarm (already running)
      - Arcetri has been testing a "new" (tbc) fix for zero-slopes. FwFix ?
         - They recommend to deploy it all SOUL WFS stations (SOUL progress meeting from Thursday)
         - fw fix from MG, not guaranteed to work = test daytime
            - if it works, no need to change the s/w
         - follow-up discussion needed at system level with LBTO
         - there are two tests
            - zero intensity frames (collect data, driven by Arcetri, Alessandro is POC - could be EMI from telescope or what not)
            - zero slopes (the issue we are discussing now)
               - unclear if Fabio coordinated yet, but they are the drivers. He provided a timeline (powerpoint in the AOG minutes)
               - https://docs.google.com/presentation/d/1B4kKBPjH_PNqatqx-wgHXW6nBVIFI95UJX2gJ9SD5gI/edit?usp=sharing (slides p15 and on)
   - IT8893
      - last commit indicates timeout was increase to account for longer exposure time
      - come up with a 'smart' scheme?
      - DLM thinks the exp time is usually 1sec or so.
      - BM will check with Alfio why he increased the value
      - 2 aspects:
         - shared memory rtdb buffer (if this fails, don't need to wait timeout)
            - this could be more involved and require changes to msgd internal
            - DLM occurs very seldom, not worth the effort
         - frame grabber
            - timeout is not a config, but simple one liner
               - per frame, or for the entire operation (e.g. take multiple frames)
               - DLM will check, recalls 1Hz was pretty much "normal"
            - use AO ref star mag and use some fuzzy logic
   - IT created a backup of one ASM machine. DE ordered a new one for the other side.
      - keep in CRB. Maybe use a dedicated cabinet. AO is in 3 upper, but mostly heavy h/w stuff.
   - crosshair position for WFS luci vs lbti. Currently in Alfio's home directory???
      - JP will circulate what she changed. BM to follow up.
   - AO patrol field for DX, to update the OT (happen during summer)

   AC, BM, MB

   - SHARK-VIS coming
      - will need network in the clean room, starting June 20th. AC will send an email to IT. TBD subnet, need to able to talk to IIF (and maybe the simulator also, same as shark-nir or new instance?)
      - DE is working on specifying a KVM for them
      - we'll follow up how they want their equipment racked (CRB or lab?)
   - IT8893 work in progress.
   - IT8841 (nudges) / IT8819 (GPS) investigation between MB and DCox continues. Atomicity of reading time in DSP, other ideas in mcspu.
   - work on SX ASM on-going (moving to the lab)

   DLM, AC, MB, BM

   - MB to check with Michelle about MG bi-annual report
   - JP (TBC) got some input from iLocater about NCPA
   - ao-asm-meas for DX (TBC), test happening now (no cals, next week with cal + close loop and gains)
   - on-going effort with repoint and psf stabilization
   - IT8893 (CCD47 timeout)
      - BM looking into it with JCG. Could be easy, TBD
      - possibly due to change of binning happening while the operation occurred
      - investigation on-going
   - shark-nir alarm issue has been resolved by Davide Ricci.
   - people will get busy on the TS removal from May 25th
      - short meeting today, we'll decide if we regroup next week.

   AC, BM, GB, XZ, MB, DLM

   - MG support contract, 6 months hours usage recap is (TBC) due. [MB]
      - MB will talk to Michelle or send an email to Roberto.
   - NCPA requirement document from JP [MB]
      - XZ will add some comments to the document.
   - ao-asm-meas
      - done on SX.
      - almost ready for DX, run a check to make sure it's fine. XZ will try to schedule something next week
      - external USB, should be here or on their way. XZ will check with DE
      - spare adsec in Tucson. Will configure/check after ao-asm-meas and send to the mountain
   - master-fwFix
      - Fabio cleaned up his PR.
      - pending system level discussion, not used for shared-risk science for now
   - Misc
      - IRC rpm on AO machines working fine
      - do we need to keep/backup the disturbance files?
         - location to be refined and consistent sx/dx. DLM will send info to XZ
      - SA was asking about remote viewers for AO: OCAM, CCD47, ASM
         - web solution.
         - viewer only. doesn't interact with AO per say. show slopes, pixels, bg, etc.
         - AOPM SOUL but not the highest priority.
      - PSF stability development
         - DLM will generate an IT, and follow up with Brandon
            - tip/tilt offset voltages
      - repoint on acquisition
         - BM added some notes to the document
            - "check-ref with manual acquire" seems to execute a repoint + stage move
            - DLM will review
      - Timeout for CCD47 can be reduced IT8893
         - BM will check
      - IT8899, Alessandro will check the h/w side
      - IT8897, known issue (continuous dump). To be prioritized.
           - GB will add to AOPM
           - probably not critical. Fix possible, but why does it happen is more complex.

   BM, DLM, GB, XZ, MB, JP, JCG

   - MG support tickets (linked above): accurate? needs updates? [MB]
      - 003 (ARGOS DX BCU, JCG)
      - 005 (Power Backplane spares, GB)
         - ping MG again.
         - remind them to use the microgate-support@lbto.org
            - forward responses if MG replies when they fail to.
         - POC can create IT related to MG support and compule the information there
            - alternatively Mario and Roberto have IT usernames, but maybe less cumbersome (for them)
   - NCPA requirement document from JP [MB]
      - draft in progress (JCG, JP): https://docs.google.com/document/d/1V3jLOMD9XAtqL2M09T00W0hSgguQTNWyPtkKRwnRhXg/edit#heading=h.y7spz5ibuii5
   - https://github.com/LBTO/ao-asm-meas vs ao-confcalib [MB]
      - tests performed on Wed. It works, but ... options:
         a) remote mounted partition shared between WFS and ASM
            - we could use the mirrored disk for everything, and nfs share the mirrored disk.
            - this is the simplest. Think about logical partitions, not physical disks.
            - currently there is only one parition on the mirrored disk. could use two, but it's not really a relevant details. Just set nfs server exports to point to the right place.
            - flats (SL) could be shuffled right away. Takes care of failures and let SL still work.
            - ${rootdir}/flat -> new location
         b) change s/w to send the data instead of a filename on a share mount (as currently)
         c) change s/w to keep all the data (recon, etc) on the WFS machines
            - few 100MB, disk space ok.
            - the change would move the capability from asm-GUI to wfs-GUI, but otherwise transparent.
            - this could be better, but not super high-priority, requires some work
            - NB: Linc uses this scheme already (sending recon via ICE), altho it dumps stuff in /tmp and another process takes care of it (iirc)
      - two types of files: flats, recons
   - JCG sent a screenshot of a "new GUI"
      - https://wiki.lbto.org/pub/Software/AOSysSwWeekly/2BB9F7F5-A151-419F-BEBE-0504B9F2F026.png
      - new feature on ??? by ??? for ???
         - things should be discussed as some of us do not know about it
      - will be deployed for this weekends SOUL comm ???
         - was used during last comms?
            - tested on sky, but (probably) not day
         - DX luci
         - XZ checked the PR, there is still issue (the 100x commits one). We asked about hithub/PR last week.
            - see https://github.com/LBTO/ao-supervisor/pull/184/commits (100 commits)
            - BM will reiterate with Fabio, on the previous email thread
               - this is probably more "system" than "software" (which is what we also pointed earlier, see GB email)
               - may arrange a follow up meeting, may disrupt their "workflow", but may be what we want from them.

   AC, GB, JP, XZ, BM

   - ncpa
      - news from JCG ASM WBS 1.1.1 req document
         - busy week working at night / soul-comm
   - /aomeas backup / git development
      - JP, initial list of recon for LUCI. DLM will double check
      - XZ, gave it a first pass.
      - IT, David sent an order for 14TB drives, on-going
   - Fw-patch
      - Fabio sent a link to github
      - reconcile to master? FR issued a PR, maybe he was on old SOUL-devel. Needs to clean it up.
   - MG support contract
      - answered the BCU-47, different rev but compatible. XZ asked about documentation
      - track status by other means than email
         - MB will add a recap wiki (see header)
   - AOB
      - Daytime tests? busy week
         - test recover from ASM failures
      - SHARK-NIR run, plans to run the Fourier mode (still needs XZ or GB support)
      - TCS repoint (AOPM-202), PSF stabilization (AOPM-201)
      - Sam R. will start on July 12, will be at the AO4ELT7

  AC, DLM, GT, GB, XZ, MB, JP, BM, JCG

   - adding topics = open to all (e.g. following AOG discussions)
   - NCPA fix readiness
      - daytime tested with LBTI
         - are we limited to 10 nm ??? only 2 hours on DX last year
         - second part; s/w fix allows to run at 1700Hz with NCPA
      - pending:
         - test with LUCI / changing rotator angle
         - multiple sources sending NCPA
            - Gopt-GUI vs. outer-loop, see notes from 2023-03-24 (ASM issue = hk interrupted, not sure if related or not) => need to be re-tested.
      - Shark-Vis (forerunner, really) may have interest to use NCPA
      - LMIRCAM may also have interest to use NCPA
         - "complete" target date = November 30, "official" science readiness (when Taurus is observed)
      - Shark-Nir has its own DM, not needed or less important (that we know for now)
         - but they use it for their acquisition (waffle/Fourier)
         - Comm activities back about October
      - Create an AOPM WBS
         - it is already: ASM WBS 1.1.1
            - create a requirement document
               - e.g. currently there could be multiple processes, that need to be managed (TBC ???)
                  - there is a GUI that, if open, keeps writing values
                  - possible issue when switching instrument (e.g. LUCI to LBTI)
                  - in conclusion, it kind of work, but poorly implemented / quick hack
               - JCG will work on the first draft = the new phase #1
                  - compile all the information from IT (8809, etc), instruments, emails, these notes, etc.
         - it will be needed by 2023B

   - FwFix
      - is it available for next week ?
         - GB and XZ are reviewing the information received from Fabio and Enrico (INAF has a long holiday weekend) run is 26/27 April
         - JCG booked daytime for Tuesday 25 April (pre checks) but can include additional tests

   MB, DLM, AC, BM, GB, JP, XZ   

   - SSD network upgrade, 2 days of possible outage
      - no problem
         - SX ASM in the lab (tentative last week of July = 7/25)
         - DX ASM fully shutdown, potential work
   - AO server backup of /local
      - AOPM activity (schedule and priority)
         - IT can do the USB dumps meanwhile
      - that was done for all AO machines
      - on the new machines:
         - sxadsec /dev/sdb1             7.2T  5.7T  1.2T  84% /local
         - dxadsec /dev/sdb1             7.3T  5.4T  1.5T  79% /local
         - single USB drive with everything, no preferences
         - cleanup the NAS?
            - currently clean
      - subsequently
         - XZ is working on figuring the "1%" we need for operation (about/estimate couple 100 directories, vs. current 1000's)
         - we'll have automated backup for these (github or TBD)
            - new flats may need some s/w handling
      - spare adsec server
            - AOPM activity (schedule and priority)
         - first, "cleanup" files
         - then copy them over the new server
         - 3 cases (XZ will write a short document)
            - 1 disk fails: move drives
            - 2 disk fail: use the spare
            - computer fails: use spare
         - new server becomes the operation machine? NO / NOT currently
            - spare has SSDs (faster, but nothing as big as the current /local)
            - spare will have a single, mirrored partition
            - aim for Kickstart to automate the setup
            - can be setup for both sides (same code) but config/etc to be a hot spare, needs both SX and DX

   - LBTI wfs machines
      - manual shuffle of logs (LUCI wfs is automated)
         - BM will check it out, JP available to help. Maybe some IT needed.

   - AO test plans, some links to update
      - BM will sort it out

   AC, BM, GB, XZ, MB, JCG, GT (briefly)

   - ao-supervisor merged and moved back into github as 'master'
      - it's currently running everywhere as "stable"
      - policy (draft / discussion):
            - SOUL comm progress update meeting, good platform to follow the issues and fixes they are trying to address
               - anything affecting s/w should be discussed there
               - Jenny sends the invites.
               - review Apr 18
               - comments due by next Tuesday Apr 11, RIX table, AC will resend the link.
                  -  Reviewers are AC and BM, Jenny P, Steve E, Doug. Send comments to them (formal)
                  - next iteration, include an ASM person in the reviewers
            - we need TAGs
               - stable-luci-sx, stable-asm-dx etc. stable-A-B where A=asm,luci,lbti B=sx,dx
               - do we call "stable" or "stable-A-B" on the server ?
               - use_soul should only show the stable, to avoid mistakes (add a -f to allow developers, for example)
               - what is called stable* is in github, on the master branch
               - anything else is daytime test, must be other branches
            - create new branches for tests
            - test during the day
               - PASS:
                  rebase-merge to master as release-candidate-A-B (that will be used for nest step, night test)
               - FAIL: keep working on dev branch
            - prior to night time test
               - select the features that will be tested
               - create a release-candidate-A-B, merging the selected features
               - daytime test the combo, _at the discretion of the developer and circumstances_ (e.g. simple python typo)
               - 
            - outcome of the test, after the night tests
               - PASS: almost nothing to do (cosmetic naming and taging, and directory)
                  - rename realease-candidate-A-B to stable-A-B
                  - rename stable-A-B to stable-A-B-YYYY-MM-DD:
                     - in git (tag)
                     - on the server (directory / softlink)

E.G (use as a baseline - above text was working draft)

      dev cycle (label #1)
         feature T tested during the day      
         feature U tested during the day      
         feature V tested during the day      
      prior to ECD night time      
         select from T, U, V - create realease-candidate-A-B
         switch: test combo directly at night or repeat a combo daytime test (depending on feature and complexity, risk of interaction etc)
      ECD night time
         run realease-candidate-A-B
               - PASS: almost nothing to do (cosmetic naming and taging, and directory)
                  - rename realease-candidate-A-B to stable-A-B
                  - rename stable-A-B to stable-A-B-YYYY-MM-DD:
                     - in git (tag)
                     - on the server (directory / softlink)
                  - there could be asm restart because of full paths ...
               - FAILS
                  bring back stable-A-B (on the server)
                  if infinite loop (unable to address quickly) cleanup github   
                  goto #1

   AC, GT, MB

   Meeting adjourned, lack of quorum.

   GB, XZ, BM, AC, MB, JCG

   - ncpa phase1
      - ready for night time test with LBTI
      - ready for daytime test with LUCI (check rotation angle)
      - there was an issue with AdSec, coincidental or not ???
   - BUT ... adsec issue that happened while the ncpa was being tested
      - GB created IT8851 https://lbt.issuetrak.com/CSIssue_View.asp?IssueNbr=8851
      - redo the tests with GB and XZ to monitor in real-time
         - internal discussion
         - review PR https://github.com/LBTO/ao-supervisor/pull/174
      - GB and XZ still looking at the logs (more or less between 11:30 and 13:30)
      - time window seems correlated with the ncpa tests
      - not critical, but not the normal behavior, something changed ?!
         - masterdiagnostic unable to read the switch-bcu
         - the effect is not readily reported by the UI/software, so BM/JCG were not aware (it comes by email alerts, or if you follow what the logs record...)
         - JCG: there was/were OCAM not-live event(s)
            - frame rate went weird (on the GUI viewer)
            - don't we have a monitoring patch from Fabio ???
               - "aux task" checks images, and weird slopes (e.g. neg slopes from bad bias / illumination) but maybe not the rate ??
            - do we keep a "user log" of daytime test like a wiki page?
               - yes; https://wiki.lbto.org/AdaptiveOptics/20230322_SX_Day
         - JCG: there were also gaps (~40sec) in the housekeeper log (that runs at 1Hz)
         - JCG: AdSec was restarted only once
         - BM: some bugs were being fixed while testing
   - XZ is working on merge the AdSec SX/DX
      - how to preserve the commit timestamps and authors? we'll check later today
      - this will need a lot of testing
      - SX goes off on May 26 ...
   - IT8612 https://lbt.issuetrak.com/CSIssue_View.asp?IssueNbr=8612
      - BM would start to look into it, look at the logs etc. TCS simulator as needed
      - Organize a follow up meeting with DLM, JP, JH, AO people   - all 5 patches all wfs are deployed as "stable" for AdSec and SOUL-LUCI
      - LBTI-DX deployed but stable needs to be "commissioned"
      - LBTI-SX not deployed yet
   - GUI bin fix to CCD47
      - everything deployed on WFS machines and "stable"
   - offsetZAO
      - must be on both AdSec and WFS
      - only deployed as a test version, on SX AdSec and LUCI WFS (the one used by SHARK-NIR)
      - it's in the test versions (but no chance to test during ECD, closed for weather)
      - met SHARK-NIR and they have a scheduled day time test (April 13)
         - XZ will update WFS-LBTI-SX
         - they can play with the TCS simulator, the IIF/ICE command exists, but the simulation behavior is TBD ...
   - offsetXYAO (patch fixed at the same time as offsetZAO)
      - must be only on AdSec
      - present only on SX at the moment
   - rerotator fix IT8762 https://lbt.issuetrak.com/CSIssue_View.asp?IssueNbr=8762
      - tested during the day with JCG (after the adsec issue mentioned above)
      - schedule time with other patches (ncpa, etc)
   - general git tip:
      - squash useless commits (like "fix typo")
      - keep relevant "milestones" to go from A to Z
      - github changed their key for auto commit, but it's been taken care of on AO machines by BM

   BM, DLM, GT, GB, AC, JP, JCG, XZ, MB

   - we ***really*** need to merge the AdSec SX/DX
      > MMC on SX, update mmc act config with -1's
      > target date ECD April 2nd
   
   - tested all 5 patches all wfs (LBTI was lagging) during the day, ready for next ECD (w/AO) night
   - tested ncpa, only LBTI wfs
      - BM issued the ECD time request, including the test instructions
      - instructions to swap release have been updated (there was an issue last time)
      - unlikely to have LBTI during ECD, if no access can be redone later
   - include the WFS bin fix
      - most relevant for LBTI.
      - LUCI is automated so mostly gets around it
   - offsetZAO
      - tested Ok during the day
      - release to shark-nir for their own day time test, doesn't need night time <<<
         - if they're happy, we leave it on sky during their run in May
         - LUCI may eventually use the new feature, but later (able to defocus with the Luci detector internally)
      - the patch includes a bug fix to offsetXY, so it should be validated during ECD
         - add to ECD form, JCG with have Luci/AO during ECD
         - XZ to create the test plan (or reuse the one from daytime)
            - good use case to exercise OT LUCI-AO
            - only SX or DX+SX. If works, will be deployed on DX the following day. On the other hand, could be binocular that night, and less hassle.
               > only SX because of the mmc/soul-devel different branches, shark-nir so that works
            - JCG and XZ will iterate about good pattern/OT
   - upgraded monitoring script was tested today
      - keep ? Yes, not AO operation. only adsec machines, both
      - push the change to git/hub
      - there anemometer email runs in a "screen" session, currently has to be started manually, 
         - use cron or timer.d/service.d - it is running both sides
   - git merge commit vs rebase

   DLM, MB, AC, JP
   
   XZ, BM running tests.
   Lack of quorum, meeting canceled.

   Action Items:
      * discuss NCPA thresholds with BM, DLM
      * Tuesday next week tests - BM to file the time if needed.
      * discuss IT8612
         Many IT's, from an email with John: First you have to fix IT 8612 where somebody is ignoring the repoint flag.
         Then your are back to IT 7640 where Petr put in a patch that was supposed
         to have fixed it in Nov 2019.
         For LBTI its more tricky where one side works and one side doesn't.
         See Issues 6827 and 6360. Petr thought 6827 might be fixed by the patch
         for 7640, but I don't see how that explains the sided behaviour.   

   DLM, JP, BM, JCG, XZ, AC, MB

   - Things being tested today
     Limited daytime telescope availability due to other issues - LBTI DX BCU47 (now fixed, reset box was dangling), AGW1 vignetting, etc)
      1.dx RR mode (contact XZ)
         - TCS/AOS needs to be updated (Yang is in the loop)
         - need zenith (cal source)
      1.sx ncpa "filtering" (contact BM, JCG coordinate for LBTI/mountain)
         - test plan below
         - LBTI stationary, needs to catch up on release consistency (the infamous "5 patches" from Feb)
            - ECD has some LBTI on-sky engineering time (optimize gains), use the time for _both activities_
            - BM will be available and send instructions if need to revert.
         - LUCI would need rotation
      2. offsetZ (use IRC/IDL wrapper from DLM) (contact XZ)
         - probably wont have time to test today, Monday ?
         - already in the telescope plan for Monday, ask entire day, rather than 4h
      3. OCAM over illumination IT8831. patch to be tested possibly Monday, or later, not super high priority

   - BM took action on the server filling up, changed the core.d / ulimit -c 0
      - are the LBTI machine rollover as LUCI ???
      - action item for BM next week ...

Here are the two other test plans I mentioned. For the test checklists, I might need some help with the details of what will need to be done telescope-wise. And of course let me know if anything else looks suspicious.

(wfs) lbti-*wfs: ncpa threshold phase 1: apply NCPA only if the difference is greater than a threshold applied in NCPA FITS files
For the NCPA test, it'd be good if in addition to "no change", we can test a non-zero but sub-threshold NCPA change, which I assume will mean just rotating the telescope by a small amount, maybe with some unrealistically high threshold values.
https://wiki.lbto.org/AdaptiveOptics/AOSWTest-202303XX-lbti-ncpa-phase1

(wfs) *-wfs: prevent temporary out of range rerotator tracking error from causing a PresetAO to fail
For the rerotator test, we'll want to test starting out at an angle out of the rerotator's range and then sending a preset that will move it in-range. 
https://wiki.lbto.org/AdaptiveOptics/AOSWTest-202303XX-rerotator-out-of-range

   Phase1.5 = 
      Phase 1
   - NCPA is scaled by gopt, but it happens in the RTC
      ncpa from the instrument pov is "constant", in the sense we dont care about the scaling - scaling happens in RTC
   - use a threshold scheme, if less than, no need to apply, covers the (LBTI) case when it doesn't change
      one threshold per mode = vector
      one threshold config per instrument
      one same file as the current table
   - keep the current 10sec cycle, just add the threshold check
      Plus:
   - rate should not exceed 10 sec update, but we could check every 1sec
      log diving for luci

   GT, XZ, BM, MB, AC

   * IT8830 = SOUL-mix-control vs SOUL-devel
      * should eventually be updated on both sides
      * m1 panics from 2023-02-23 related? check with JCG
   * aorev_git is now setup on dxadsec
   * BM split of ao-supervisor into adsec/wfs/common WIP
      * git subrepos? could be confusing git wise
   * GB was in Italy discussing new ASM
      * preliminary comments
   * Offset-Z is moving along (XZ email, pending a reply or some interaction with shark-nir)
      * will make a PR, coordinate day time test before ECD
      * no comments is probably fine, XZ will reissue a notice just for confirmation
         * note the comment about accuracy on the z stage
   * "fourrier" mode
      * will be coming next

cancelled due to lack of quorum.

cancelled due to lack of quorum.

   AC, BM, XZ, GB, MB, JCG

   Schedule wise, SHARK-NIR Mar 6, LBTI Mar 9 - ECD time in between

   1) ncpa phase 1 (BM)
      * PR created
      * BM will test it on a VM next
      * Description in the PR https://github.com/LBTO/ao-supervisor/pull/174

   2) offsetZAO phase 1 - connection between AOS and ao-supervisor (XZ)
      * no chance to make progress, power outage, DX issues, MG document
      * activity will start week of Feb 13 when GB travels to Italy
      * impact on efficiency, safety etc. (allows not to have human errors)
      * failure mode for Z and XY are not handled the same at asm level. Z=over-current, XY=skip-frames (MG only check for global TT, not other modes)

   3) one step "Fourier" command (XZ, GB)
      * need to review the document shared by SHARK-NIR
      * may require more iterations with them
      * may include a short LBTO document to explain what we understand, something GB may review
      * impact even bigger, because currently need GB+XZ to allow coronograp mode

   For XZ, by consensus - start with 2) and then 3)

   4) AdSec soul-dev/SX vs mmc/DX (XZ)
      * within the context of a possible split of 3 repos: common/asm/wfs
      * general agreement we should merge asm sides first, then split second
   
   5) Bring back the discussion about Elevation limit from GB.
      * GB will write a couple of senses.

    JH, AC, GB, JP, BM, XZ, MB, GT, AV, JCG
   
   * update on day-time/ecd-time plans and readiness (BM, XZ)
      * https://wiki.lbto.org/AdaptiveOptics/AOSWTest-20230201-checkout
      * hk+rr patches on both asm sides. SX is ready, DX is WIP.
         * if possible, try daytime - but not absolutely necessary. Possibly Monday. Some git/github/deploy work mostly for BM and XZ, but straightforward.
            * but ECD could be on Saturday ... skip daytime if so.
            * possibly: release SX asap, do DX later
         * schedule tbc given the outage yesterday (LBTI probably not available), Weather forecast isn't looking great for AO.
         * we'll know more at 3pm, but there will be ECD time over the 3 nights. JP is coordinating with OK and AB after this meeting.

   * ncpa phase 1 - updates? (BM)
      * BM shouldn't be an issue. Working on it.

   * shark-nir offsets
      * Z-AO, ways forward (all)
      * https://github.com/LBTO/ao-supervisor/issues/173
      * phase 1: (XZ will take this)
         * offsetZAO connection between AOS and ao-supervisor z-stage. fairly easy to implement
            * allows "engineering level" control for shark-nir (they'll sequence it themselves)
      * phase 2:
         * 2.1 implement a new offsetFocus IIF command: will be high-level sequencing/checking similar to offsetPointing2, phase 2.1 is the "plumbing" only (check with YZ or someone)
         * 2.2 make it work, that will use offsetZAO internally, but possibly no changes in ao-supervisor (will discuss internally, maybe PG or GE, or MB)
            * likely to use the same/similar logic that exists for XY, just add Z
            * reference https://wiki.lbto.org/Software/IIFOffsetSequence
               * look at step 12 which is probably (currently) only XY, and just needs to add Z
         * shark-nir hasn't expressed any concerns (yet!) about moves in Z affecting XY ...

   * one step "Fourier" command
      * currently requires support from GB and XZ.
         * currently "ramped manually", would be best to have a "smart/fool proof" command
      * GB and XZ will review the report.
      * next run is around March 6
 
   * generic web-based iif a-la-IRC (lessons from LBTI, new frameworks, utilize code generation)
      * new project for TCS, not LBTI (MB)

* IT8809 https://lbt.issuetrak.com/CSIssue_View.asp?IssueNbr=8809

   Phase1
   - ncpa is scaled by gopt, but it happens in the RTC
      ncpa from the instrument pov is "constant", in the sense we dont care about the scaling - scaling happens in RTC
   - use a threshold scheme, if less than, no need to apply, covers the (LBTI) case when it doesn't change
      one threshold per mode = vector
      one threshold config per instrument
      one same file as the current table
   - keep the current 10sec cycle, just add the threshold check

   Phase2
   - rate should not exceed 10 sec update, but we could check every 1sec
      log diving for luci
   - add a check for amplitude(s) requested

   PhaseX
   - iLocater "may" want an elevation dependent calculation, who will figure the maths
   
   PhaseY
   - why is it failing in the first place, can we fix it ourselves, bank swapping?

   * AO scientist will be responsible to update the new thresholds
   * BM will take Phase1 and we'll roll it by GB/XZ as usual
      * submit our intention and plan to Arcetri for good measure and feedback if any

* offsetZAO https://github.com/LBTO/ao-supervisor/issues/173

   * AOS implemented, wfs stage implemented
      * need to connect both, but that's the easy part (TaskA)
      * DLM says he recalls even the offsetZAO worked with IRC ???

   * there will be deeper considerations (TaskB)
      * offload to hexapod, speed of the stage, etc.

   * XZ points out the stage plane of motion is not "flat" (a real Z may involves X and Y)
      * this needs to be followed up with Shark-NIR, do they care or not? see TaskB
      * commands should "give" X, Y, Z and use an internal model to drive the stages
      * Argos needed Z due to the ngs vs lgs focus, but it was complicated and didn't happen

   * offsetXYAO has another issue. it's limited to a range, but the AOS / DD reports the demand, not the achieved
      * XZ pointed out the stage plane of motion is not "flat"

   * TaskB small project on its own
      * who could lead this effort?
      * AC will put it on next SHARK-NIR agenda - Thu Jan 26
         * XZ will send a list of questions to AC
         * Maria's email "Re: TCS commands to interact with SOUL" (originally started by Davide)

* ECD patch readiness 
      * BM pressed the send button, JP got it
      * Fabio's changes are already deployed and tested, so no need to add to "our" test activity

  MB, BM, XZ, GB, AC

   * review https://wiki.lbto.org/AdaptiveOptics/AOSWTest-20230125-cameralens
      * keep 1, 3, 4, 5

  MB, GB, BM, DLM, GT, JCG, BR, AC

coming next (TODO: unprioritized):
   * MB - LN issue and patch, follow up - https://lbt.issuetrak.com/CSIssue_View.asp?IssueNbr=8784
      * they have a patch. next run is tbd (LBTB nights)
   * JP - AO config for SHARK/LBTI - same or not?
      * JCG: they are different. but AOS loads the AO table(s) and knows about the instruments:
         * Thu Dec 15 18:31:38.054 2022 aos.info - WFS source set to: LBTIWFS
         * Thu Dec 15 18:31:38.705 2022 aos.info - FLAO: WFS: loading setup for SHARK
      * need to follow up with Jenny (how does AOS tells the WFS it's LBTI or SHARK?)
      * DLM: it is working.
         * to be understood by SWG
   * DLM: LN writing of memory BANKS
      a) we do not have any evidences overloading a small vector on the same bank is an issue (SOUL)
      b) bank swapping makes sense for a big recon matrix, and that is what LN is doing already
      c) similarly to our SOUL z0 patch (both-banks > bankA only), it makes sense for LN to only load its gain to the bank it's using.
         c.1) our SOUL experience shows LN does not need to switch bank to set gains
         c.2) our SOUL experience suggests LN set gains on "used bank" rather than "both banks"
      d) there are more questions about the banks and swapping we will plan to elucidate. And indeed, it seems like it could be hidden from the user.
      * Carmelo had some issue with gains. To be investigated (on their end).
      * Al will follow up about skip frames episode, safe for adsec. There will be an LN post-run meeting.
      * Note with skip frames, they wont be able to load new recons ...
      * Arcetri s/w checks the skip frame counter, check again few seconds later, and if the same, proceed
      * this seems to hint that if he system is skipping, it doesn't want to load
         * check with Arcetri what was the logic/thinking behind that
         * currently at the FSM level, it's fatal, should probably be an error.
   * MB - bank swapping improvement and automation (MG support?)
      * priority is probably medium or low for LN
      * but interesting for us
   * GB - telescope elevation limit with shell set (30, 26 exception)
      * already in place
         * AoArbitrator level
         * "Adam" housekeeper level
      * add a third level: catch it before sending the demand (rather than react to it) - "best effort" because the target could start above but move below the limit.
   * GB - telescope elevation limit with LBC and ASM both deployed or both retracted (90)
      * enforce telescope at zenith if they are both facing each other (something falls and bounce on LBC, back onto the shell - they're 6" apart)
      * use case for both retracted? tbd, maybe a bit more efficient if mixed LBC/M2 nights. Also when aluminizing M1 (but here they have covers - at horizon the bouncing is less likely to happen)
      * use case for both deployed? daytime maybe when using the crane, etc.
   * MB - AO telemetry (e.g. skip frame %, loop rate, *any* important parameters we know about already, etc, = designed such it can grow backward and forward)
      * future development for us, currently nothing in DMS for AO.
      * SOUL has some msql, but "mostly" limited to WFS
   * SE - automate NCPA
   * SE - automate Flex Comp - understand imaging/spectro use cases and Luci built-in AFC
      * DLM has some routines (for both). Will make them available in github. Used once or twice a year.
      * priority tbd. uses IRC automation, Luci image acquisition, etc.
   * SE - DIMM stats
      * AO related or site related, Maybe both. Inform on conditions when to use AO or not. Average, variability, seasonal, etc.
      * what do we want to extract? AO domain discussion, different dimensions
   * JCG - Argos MG support?
      * JCG working on edits for the request document
      * GB will look at the doc
         * careful about overloading MG (ticket #1 not concluded ...)
         * maybe limited to re-return the card for repair
   * MB - auto revision control
      * what does it means, in general and for ao-ops (.gitignore)
      * MB will ingest /local/aomeas/adsec_calib per DLM document (request for feedback? none received) - conclusion of AOG 2022-13-15
   * MB - ingest some of our discussion into IssueTrack or github

  MB, BM, GB, JP, XZ


Action Item:
   - BM "hiding" w_start/w_stop access so people don't use it by mistake

- rebuild of spare soul wfs server
   - setup and ready to test on Monday, using soul-sx
      - BM check and make a telescope daywork entry
      - GB: stopping the WFS arbitrator affect the ASM
         - rtdb/msgd meltdown?
         - XZ: user manual states that restarting WFS, user should restart ASM, JP will recheck/confirm
            - check with Arcetri on Monday, manuals are D10 and D11
            - with Luci, the restart keeps msgd/rtdb up
            - with LBTI, not so
   - kick start almost done, just some network config to finalize
   - mysql tables creation pending, TBD init script

- set_z0 patch
   - needs to be deployed on DX (already on SX)
      - use BM method (avoids a system restart - see next point)
   - MB will get the approval from EPI because they're commissioning on DX

- discussion with MG about issues on asm DX
   - investigating and diagnosing with Mario, GB, XZ
      - Mario initially suggested a "debug" new f/w, risky, thinking about other options
      - failing single DSP board, or bigger issue? TBD. Gathering more diagnostics or updating HK to collect new data.
      - DX commissioning is proceeding, but everyone aware of the issue and will report and stop if occurs.

- spare adsec machine
   - IT buying 2TB SSD, x2 per server for mirroring, x3 for sx/dx/spare
   - spare machine in XZ office, IT will provide a drive to XZ to get a backup of what's there
   - machine will go to the mountain after that
      - cloning is fine (as opposed to rebuild from scratch)
      - discussion about the procedure 

- database for ASM spare
   - check with Pat Hartley about "e-maint" https://www.emaint.com
   - MG delivered a spare spreadsheet and parts spreasheet, but it's a mess to figure the history

MB, DLM, BR, AC, GB, XZ, BM


Action Item
  MB: setup the "template", using either the spreadsheet or doc.
    > MB to be done next week.

  GB/XZ: DX asm, mmc/freeze patch tests and schedule
    > carried over
    > patch for skip frame recovery
    > add as a new activity when Arcetri visits - dicsuss when discussing their visit schedule next week (short week)

  DLM: triage /local/aomeas/adsec_calib (***)
    > 1 or 2 weeks, pending still
    > check email sent by DLM during the meeting https://docs.google.com/document/d/1cIHD2d3IvN4St5UHivFPwBK5y79LDX9Rtxz6M0wp7u0/edit

  GB maybe discuss some new AO / TCS / system integration items

  XZ: decide on step 4.1, new drives, clones, reinstall from scratch, etc.

> discuss spare adsec machine (GB)
  * readiness, location, etc.
    * in XZ's office
    * originally used for PB development, also used for bcu2k reflashing
    * need a "real" spare at the mountain
       1) reconfigure (if needed) to be a true spare (install latest ASM s/w)
         RAID1 mirroring
         build (make) = 1 hour to build    -
         ao-confcalib is in github         - these two are on the mirrored partition
         what is missing is the /local/aomeas/adsec_calib (see ***) - this partition is not mirrored
         KL29 is the one we use / want for operation, possibly 28 also
         Each side has its own config.
          > ao "can work" as far as tss/safety without adsec_calib
          > anything else (AO, seeing limited, etc) requires it
       2) if PB development comes back, we can get a new machine for that
       3) check level of disk spare (e.g. rebuild the mirror), something for IT
       4.a) XZ check the machine first in Tucson, make sure ready to be handed over
          > possibly get new drive and keep the existing one
       4.b) IT will take the spare from XZ's office, bring it to mountain
          > set it up to current level of s/w
          > need to have both dx/sx available
          > maybe use a tar scheme where we skip the need to rebuild
          > hostname, IP, etc.


> Simone's powerpoint about failure vs rate vs. etc.
  * not shared yet
  * probably will need to be closely scrutinized when available
  * DLM can cross check the results
  * where is the information about loop rate in ASM ??
    DLM does a timestamp diff and computes it
    XZ there is an entry (runAO), but rate can change after runAO (e.g. binning, check flux, etc)
      > only adjusted to lower value? if more flux, could go faster in fact (if same bin) - TBC
      > BR reports seeing it increase
      > XZ, check flux can change it up or down
      > closeloop command then set things up
      > BR: value in the luci headers somehow
    XZ will double check the code
    if not present, we should make it trivial to find (new printf or what not)
    BM: pyarg.202211170000.log:3791:WfsArbitrator_146186|DEB|      3539|2022-11-17 13:07:42.452433|             MAIN > ('set framerate: 1000.000000 Hz',)
    Anyone who knows the recipe, share the recipe
      > XZ fast diagnotic data (use HW frame rate = 1kHz here)
        astdiagn_75329    |INF|   3367133|2021-02-27 03:43:41.559159|             FAST > Processed   315 cycles of  7702 vars in   1.03 s (   305 Hz). HW frame rate   1e+03 [DiagnApp.cpp:832]
      > carefull:
        > 500 Hz is the dummy rate when open loop
        >  50 Hz is the dummy rate when shell rested
      > we'll use this as our first pass

> discuss MG email next week ???
  * run down from GB
    > system aging
    > proposed a procedure for troubleshooting, suspect TBC h/w issue
    > follow up actions for GB/XZ

> handover excercise. spare WFS should be centos6 ready and at the mountain
    > Fabio will send updated instructions
    > people might be on holidays and we'll recschedule as needed

==== meeting ended here =====

> set_z0 release to operation strategy
  > GB will circulate a recap proposal
  > DLM need to check with Jenny and LBTI team

--------------------------------------

- JCG has a new form for end of day check, will circulate to AO-Support, looks neat!
- data handling, pending
- AO s/w multiple version convergence plan, update?
- anemometer test-rig, update?
- bcu2k in Tucson JTAG, update?
  Alessandro reports some issues (cables? bad connection for Mario) will get back to it on Monday
- AOB:
   - Argos
      - workstations at the mountain?
      - SOUL compat using “Alfio’s trick” plans?

MB, GB, BM, DLM, GT, XZ, JCG


Action Item
  MB+GB: setup the "template", using either the spreadsheet or doc.
    > carried over again

  GB/XZ: mmc/freeze patch tests and schedule
    > carried over

  DLM,AO: triage /local/aomeas/adsec_calib
    > 1 or 2 weeks

  GB maybe discuss some new AO / TCS / system integration items

> set_z0 release to operation strategy

  > caveat
    > daytime tests passed, did not show over-currents
    > may not be related to loop rate
    > similar policy as was done for LBTI at 1700Hz
      > e.g. stop on 1st overcurrent event (stop = no ncpa, or 900, or both)
      > LBTI doesn't re/de rotate, use constant ncpa
        > success at night with LBTI is not an indicator for LUCI
        > LBTI has 'soft' and 'hard' ncpa config
        > both were causing skip-frames, although small acceptable number
          > but the impact of hard, vs soft, vs soft-soft is noticeable (worse to better)
        > was tested night of Nov 3 (MST)
          > softer soft ncpa
          > did they use default values from SX ? doesn't make sense
          > even very small values (10nm) were causing issues - need to check with Jenny and LBTI team
    > possible future plans
      > ncpa updates based on re-rotator angle, not fixed time rate (10 sec)
      > ncpa values are logged? yes
      > skip frames:
        > caused by excessive forces
        > but, the NCPA were small enough it should not happen
          > to be investigated
      > check config SX vs DX for thershold, skip frames, etc (+/- 0.77), are they the same?
    > LBTO can support LBTI comm. what is the plan?
      > the values they tried last night seemed irrelevant (very small) so why apply them?
      > maybe it was a debug effort (as opposed to optics/physicsi/perfomance)

  > GB will circulate a recap proposal
  > DLM need to check with Jenny and LBTI team

- JCG has a new form for end of day check, will circulate to AO-Support, looks neat!
- data handling, pending
- AO s/w multiple version convergence plan, update?
- anemometer test-rig, update?
- bcu2k in Tucson JTAG, update?
  Alessandro reports some issues (cables? bad connection for Mario) will get back to it on Monday
- AOB:
   - Argos
      - workstations at the mountain?
      - SOUL compat using “Alfio’s trick” plans?

MB, GB, JCG, BR, BM, DLM, JP, XZ, GT


Action Item
  MB+GB: setup the "template", using either the spreadsheet or doc.
    > carried over again

  GB/XZ: mmc/freeze patch tests and schedule
    > carried over

  DLM,AO: triage /local/aomeas/adsec_calib
    > 1 or 2 weeks

  GB maybe discuss some new AO / TCS / system integration items


- MG support contract
    > MG support starts Nov 1st

- catchup on recent activities
   - z0
    > Reply from Enrico, use 1700 Hz
    > Schedule tests Tue/Wed next week
      > 1.7kHz + NCPA + new housekeeper on SX on TUESDAY
      > same test plan as earlier
      > DLM will coordinate and organize the work
      > parallel tests ocam-bad-frames on the other side

  - new new hk
    > ready to go

- data handling, pending
  > some files have been flushed to the NAS by XZ

- AO s/w multiple version convergence plan, update?
  * set_z0 failed on DX
    > IDL error with BANK keyword on DX adsec
    > tested with LUCI SX at night https://wiki.lbto.org/AdaptiveOptics/20220608_SX_Night
    > ASM DX mmc does not have the patch
    > LBTI needs ncpa @900 on DX for its commissioning plan
      > LBTI SX is using the patched that support the BANK keyword set_Z0, and it is working
      > JP will propose to Arcetri to use Monday daytime for that
      > MMC version needs to be patched for set_z0 on DX on MONDAY
        > if it works, we keep at night
        > if it fails, we revert both ASM and WFS
      > BM will take care of the merge https://github.com/LBTO/ao-supervisor/pull/126/files

  * LBTI-SOUL-comm meeting Monday at 8.00am
  * test start Monday at 9.00am, Fabio to be online figure the issue with config issue

  * LBTI run
  * modification for freeze keyword has been implemented by XZ for mmc
    * XZ contacted Enrico about deploying the change, no answer yet
  * needs to be tested. Prepare test plan, schedule, etc.
    * only needed for calibration (not "typical" regular night operation)
    * freeze vs flat

- anemometer test-rig, update?
  * MW2 worked on it, will produce a document to detail the hardware
  * XZ setup device IP, config, etc.
    > XZ working on the sw modifications
  * it's alive and available IP 192.168.18.162
  * looking at default baud rate, work in progress

- bcu2k in Tucson JTAG, update?
  * Alessandro cable should arrive Nov, may get some free sample. TBD
  * tbd, MB will ask Ale

- stats on AO failures related to hexapod and wfs stages, how big a contributor?
  * keep for later when DLM is here, could be a long discussion
  * other aspects (e.g. patrol field in OT, skip frames, resilience ... etc.
  > hexapod faults occur daytime, configuration is different from nighttime
    > nightime maybe once or twice a year
    > IT6326
  > wfs stages
    > info was collected, used to have sticky motor/timeeout, but addressed during SSD maintenance
    > probably not a big contributor

- AOB:
   - Argos
      - workstations at the mountain?
      desktop shortcuts, simplify operations
      IT recommends an OS update.
      Sebastian is going to visit LBT? part of formal handover, iterating at the moment
      - SOUL compat using “Alfio’s trick” plans?

- power bump -> hk should restore
  - someone checks GB or XZ
  - power bump more likely during stormy weather
  - asm UPS ? MW2 will discuss/review, not sure yet
    system to cover short power bumps in the plan

  MB auto-rev control presentation next week, round up with GT later this afternoon for Luci/sx

> We (Guido and Xianyu) have discussed an issue that was brought up today at the AOG meeting, the configuration control for the ASMs. Our basic structure to achieve this control is the following:
> 1 - identify the list of files required to run the ASM
> 2 - identify the actual files that correspond to this list, this is the task of the person who calibrates the system and creates a configuration tagged with "to be used"
> 3 - develop a mechanism that allows the following:
>       1 - a way backup the current configuration (with a tag)
>       2 - a way to check the current configuration against the "to be used" configuration
>       3 - a way to restore a saved configuration
> 
> These points can be further discussed with the goal of getting a 'configuration control' underway. 
> 
> After we decide on a final method, do you think that the SW group can implement this kind of configuration control?

DLM
  SX adsec, anything before Aug 2022 can be ignored ?
  DX adsec, anything before Mar 2022 can be ignored ?

> conclusion:
  AOscientist should look at 1. and 2. and provide "simple rule"

/local/aomeas/adsec_calib

MB, AC, BR, BM, DM, JCG, XZ

Action Item
  MB+GB: setup the "template", using either the spreadsheet or doc.
    > carried over again

  GB/XZ: mmc/freeze patch tests and schedule
    > carried over

  DLM,Ao: triage /local/aomeas/adsec_calib
    > 1 or 2 weeks

  GB maybe discuss some new AO / TCS / system integration items

- MG support contract
    > Roberto sent the document, in the hand of Joe and Gene

- catchup on recent activities
   - z0
    > Simone suggested to try 1700 + ncpa
    > Doug will coordinate daytime for tests (with BM, XZ, GR)
  - new new hk
    > XZ working on the changes today, could be ready for next week.
         will check with Arcetri, github review, etc.

- data handling, pending
  > some files have been flushed to the NAS by XZ
- AO s/w multiple version convergence plan, update?
  * LBTI run
  * modification for freeze keyword has been implemented by XZ for mmc
    * XZ contacted Enrico about deploying the change, no answer yet
  * needs to be tested. Prepare test plan, schedule, etc.
    * only needed for calibration (not "typical" regular night operation)
    * freeze vs flat

- anemometer test-rig, update?
  * MW2 worked on it, will produce a document to detail the hardware
  * XZ setup device IP, config, etc.
    > XZ working on the sw modifications
  * it's alive and available IP 192.168.18.162

- bcu2k in Tucson JTAG, update?
  * Alessandro cable should arrive Nov, may get some free sample. TBD

- stats on AO failures related to hexapod and wfs stages, how big a contributor?
  * keep for later when DLM is here, could be a long discussion
  * other aspects (e.g. patrol field in OT, skip frames, resilience ... etc.
  > hexapod faults occur daytime, configuration is different from nighttime
    > nightime maybe once or twice a year
    > IT6326
  > wfs stages
    > info was collected, used to have sticky motor/timeeout, but addressed during SSD maintenance
    > probably not a big contributor

- AOB:
   - Argos
      - workstations at the mountain?
      desktop shortcuts, simplify operations
      IT recommends an OS update.
      Sebastian is going to visit LBT? part of formal handover, iterating at the moment
      - SOUL compat using “Alfio’s trick” plans?

- power bump -> hk should restore
  - someone checks GB or XZ
  - power bump more likely during stormy weather
  - asm UPS ? MW2 will discuss/review, not sure yet
    system to cover short power bumps in the plan

  MB auto-rev control presentation next week, round up with GT later this afternoon for Luci/sx

> We (Guido and Xianyu) have discussed an issue that was brought up today at the AOG meeting, the configuration control for the ASMs. Our basic structure to achieve this control is the following:
> 1 - identify the list of files required to run the ASM
> 2 - identify the actual files that correspond to this list, this is the task of the person who calibrates the system and creates a configuration tagged with "to be used"
> 3 - develop a mechanism that allows the following:
>       1 - a way backup the current configuration (with a tag)
>       2 - a way to check the current configuration against the "to be used" configuration
>       3 - a way to restore a saved configuration
> 
> These points can be further discussed with the goal of getting a 'configuration control' underway. 
> 
> After we decide on a final method, do you think that the SW group can implement this kind of configuration control?

DLM
  SX adsec, anything before Aug 2022 can be ignored ?
  DX adsec, anything before Mar 2022 can be ignored ?

> conclusion:
  AOscientist should look at 1. and 2. and provide "simple rule"

/local/aomeas/adsec_calib

MB, AC, BR, BM, GT, GB, XZ

Action Item
  MB+GB: setup the "template", using either the spreadsheet or doc.
    > carried over for next week. MB create a template and submit to GB for review

  JCG: add "s/w (reverted to/set to) night-time version" in the daytime sign-off email
    > also discussed at AOG yesterday. It is better now, with point of contact for daytime tests.

  JCG: z0 test, available Wed and Thu, not Fri - create telescope time request
    > done

  GB/XZ: mmc/freeze path tests and schedule
    > carried over

  MB auto-rev control presentation next week, round up with GT later this afternoon for Luci/sx
  GB maybe discuss some new AO / TCS / system integration items

- MG support contract
    > MB will remind Roberto to send us a final version of the contract at Monday's ASM meeting

- catchup on recent activities
   - z0/hk test update
    > 4000 iterations at 900Hz, no issue
    > Simone asked 1.5kHz, SOUL acceptance "spec"
    > *only closing the loop* (i.e., no set_z0 tests yet)
      > found out 1700 Hz is ok
                  1500 Hz troublesome
                  1350 Hz troublesome
    > thus, next step ???
      > discuss with ATT (see GB email summary for more complete details)
      > 127Hz t/t resonnance, "fake pie-shape" (see IT 7697 and maybe others - JHill mention he could see it with OVMS)
      > IIR filter implemented by Arcetri (Guido Agapito)
      > we had configs for 1350 / 1700, not 1500
      > should we run 1700 + set_z0 ?
        - maybe discuss at ATT
        - how is the loop rate determined, AO guide star magnitude, lookup table = discrete or continuous ???
          - probably but __TBC__
            > filter(/s) is discrete ?
            > wfs rate is continuous ?
          - how does the IIR "tune" itself
    > do we keep the patches for ECD?
      > GB/XZ looking into another improvement for HK (timeout on telemetry data)
      > set_z0 is a good patch to have, but probably not the cause of all the issues
      > GB/XZ will make a proposal
      > next ECDs are Oct24-26, then Nov 5 onward


- ASM meeting Monday 16 at 12:30

- data handling, update?
- AO s/w multiple version convergence plan, update?
  * modification for freeze keyword has been implemented by XZ for mmc
  * needs to be tested. Prepare test plan, schedule, etc.
    * only needed for calibration (not "typical" regular night operation)
    * freeze vs flat

- anemometer test-rig, update?
  * MW2 working on it

- bcu2k in Tucson JTAG, update?
  * Alessandro getting some cables, schedule next week

- stats on AO failures related to hexapod and wfs stages, how big a contributor?
  * keep for later when DLM is here, could be a long discussion
  * other aspects (e.g. patrol field in OT, skip frames, resilience ... etc.

- AOB:
   - Argos
      - workstations at the mountain?
      desktop shortcuts, simplify operations
      IT recommends an OS update.
      Sebastian is going to visit LBT? part of formal handover, iterating at the moment
      - SOUL compat using “Alfio’s trick” plans?

- power bump -> hk should restore
  - someone checks GB or XZ
  - power bump more likely during stormy weather
  - asm UPS ? MW2 will discuss/review, not sure yet
    system to cover short power bumps in the plan

MB, GB, CV, XZ, JP, BR, JCG

Action Item
  MB+GB: setup the "template", using either the spreadsheet or doc.
  JCG: add "s/w (reverted to/set to) night-time version" in the daytime sign-off email
  JCG: z0 test, available Wed and Thu, not Fri - create telescope time request
  GB/XZ: mmc/freeze path tests and schedule

- MG support contract and its implications
  - email group e.g. microgate-support@lbto.org
    * LBTO staff only
    * first request to MG sent by GB or MB, but includes the microgate-support@lbto.org
    * cc'd on replies to it's available for anyone to follow.
  - form for requests ?
    - send questions / requests to LBT tech reps (GB/MB), filter what we may already know
    - ESO uses a form scheme
      - reply and follow up from MG, came in the same form?
    - keep track of activities:
      - GB like google doc, but use an easy to find location (e.g. AOG drive / microgate)
      - use a template (structured and easier to describe the request)
      
      e.g. use google form -> generates spreadsheet - allows attachments?

      sheet: pro, easy to get a summary. con, lose the document aspect
      google: search very fast, better than wiki

- improving communication with sciops, day time tests and notification about rolling things back
  - explicit better than implicit, send email to telescopework@lbto
  - maybe use a template (check out list, similar to JCG email)

- catchup on recent activities
   - z0/hk test update? plans for Fri. (or Sat / Mon) ECD?
    - hk test successful, but new features could be desirable
      - timeout set to 1 second, could be a config option? no
        when iterating over the bcus
      - could be released, but other developements considered.
        Best wait next iteration GB/XZ are working on.
      - better planning for "night" release
    - z0 no entirely completed (3000 vs 10000 updates)
      release for ECD? or continue with the test
      DLM is away
      SOUL+DX next week (Wed, Thu, Fri, cont Sat)
      JCG will coordinate the tests next week

  * Conclusion: we'll try schedule for next week ECD, next Saturday Oct 15 = split night with PETS.

- data handling, update?
- AO s/w multiple version convergence plan, update?
  * modification for freeze keyword has been implemented by XZ for mmc
  * needs to be tested. Prepare test plan, schedule, etc.
    * only needed for calibration (not "typical" regular night operation)
    * freeze vs flat

- anemometer test-rig, update?
  * MW2 will be back Mon and start on it
- bcu2k in Tucson JTAG, update?
  * Alessandro getting some cables

- stats on AO failures related to hexapod and wfs stages, how big a contributor?
  * keep for later when DLM is here, could be a long discussion
  * other aspects (e.g. patrol field in OT, skip frames, resilience ... etc.

- AOB:
   - Argos
      - workstations at the mountain?
      desktop shortcuts, simplify operations
      IT recommends an OS update.
      Sebastian is going to visit LBT? part of formal handover, iterating at the moment
      - SOUL compat using “Alfio’s trick” plans?

- power bump -> hk should restore
  - someone checks GB or XZ
  - power bump more likely during stormy weather
  - asm UPS ? MW2 will discuss/review, not sure yet
    system to cover short power bumps in the plan

MB, GB, BR, BM, DE, DLM, GT, JP, SH, XZ

- update on set_z0 / hk logging patch / loop rate /test and schedule (XZ, BM, GB)
  - hk patch has been reviewed and test plan in the work
  - daytime test (gains=0) scheduled for next Tue, Oct 4 + Wed, Oct 5
    already on telescope daytime calendar
    team: BM, XZ, GB, DLM
    nb: SOUL team working on DX, happens in parallel
  - if pass
    - set_z0 test will continue on (gains > 0)
    - if pass:
      - option 1) revert, but plan for release at next ECD night
      - option 2) keep
      - half ECD Fri, Oct 7 - more options Oct 8, Oct 10
        - we'll use option 1) and wait ECD

- update on moxa and anemometer test rig (XZ)
  - new unit should be at the mountain, Cindy tracking its location
  - MW2 will set it up, back from vacation on Oct 10
  - tcpdump capture script from BM
    - running currently, automatic email and pcap captured files
    - runs on both sides
  - XZ has a new new patch (ops s/w). Create a PR so we can review it.

- update on open AI wrt. AO confcalib, DLM code in github (DLM)
  - already discussed during AOG

- update on bcu2k / JTAG (if needed, MB)
  - device has been ordered and being expedited, should arrive next Mon Oct 3
  - Alessandro will update Mario about our way forward

- disk space, log and data management, uniform system services (IT, dns, background jobs, etc) - recycled from 20220902
   - document from XZ https://docs.google.com/document/d/1fWAWboZ2nZoJbF_1I5zK25fii2_Y6NNv/edit
    - only covers the ASM
    - expand to include all WFS, Argos, etc.
    - *** XZ will lead the implementation with help from IT/SW ***
      - e.g., SH suggest we could run the job on the synologies rather than the machines/VMs
    - review how the email alarms are generated (e.g. if level at 80, alarm if it exceeds)
    - guideline on using the data (currently done on the adsec machine, future = use VMs)
  - DE moving flao_data on a new share.
    - Initial rsync about to complete
    - DE will coordinate with XZ about swapping the mounts today Sep 30
    - the export will be tighten (replace * to the ones actually used = asm + soul-luci)
    - LBTI isn't using the NAS mount (they only mount the adsec)
      - data handling process is manually done on LBTI, should be automated
    - investigate 2-steps copy
      e.g. copy wfs to adsec, and later copy adsec to NAS
      follow up with Fabio et al. - could be historical
  - mount situation, available to world? needs to be discussed
  - how are we doing with space
    - 49 TB total
    - 26 TB free after rsync (flao-data = 12TB)
    - SX/DX asm each 5.5TB used, possibly 10TB more to offload on NAS
        - mostly AO calibration and AO loop data
    - do we have data that is already > 5 years? probably
      - could use a leveling strategy ("pruning") using disk space rather than age
    - are sx/dx following the same structure ?
    - wfs soul/lbti should also be aligned

- AOB

    - PB and ATT
      - GB working on presentation with XZ
      - meeting will be post ATT, GB will send a meeting invitation, week of Oct. 10
      - GB/MB will catch up (this afternoon 3:00-3:30)

MB, JP, AC, AV, BR, BM, GT, GB, JCG, 

ESM on DX for ECD Jul12-13 ?
  - TBD daytime work needed before can be tried at night.
    - DX asm for restart
    - some actuators disabled recently
    - ongoing discussion (check flat with 4D, etc)
    - probably need more time to calibrate, etc > 2 weeks
      - ESM is 12 modes
    - to be iterated during AOG

BM, MB, XZ review comments in PR120
  - pending, not critically urgent, waiting for Jacopo report
  - phase1 = allow the mode they use, and the amplitude < they used.

GB iterate TSS remote power switch with JR and ENG
  - om-going, in the hands of GB and others

JC, DM, BM review comment in github Issue115 and disable ncpa while optimize discussion
  - comments from Alfio in the wiki QA
  - will iterate

AOB
---
  - comments on test plan sent by BM?
    - GB overall good but busy this week
    - nothing critically urgent
    - for restart, 1-2 nights for Luci
      - BM send input to BR by mid-Aug to include in the plan
  - ao.sys.sw meeting will be suspended until Fri - Aug 12
    - other restart planning meeting usually planned mid-Aug
    - M3 out (sx) until Aug 24th
    - mmc work before shutdown w. new actuators removed (GB will follow with a meeting)
  - during SSD
    - SX removed by Jul12-13, put rigid (XZ will be here, he and others in SWG are familiar with the procedure)
      - update the PDF instructions to inc. the f/w change (GB)
    - DX powered off, complete shutdown, cover, etc (usual procedure)
  - MB off/remote Jun25-Jul31

MB, JP, CV, BR, JC, GT, DM, BG, BM, XZ, ??

++++++++++++
Action Items
++++++++++++
  - BM runtime, upgrades recap
  - BM, MB, XZ  review comments in PR120
  - GB iterate TSS remote power switch with JR and ENG
  - JC, DM, BM review comment in Issue115 and disable ncpa while optimize discussion

++++++++++
Discussion
++++++++++

- dx mmc freeze         https://github.com/LBTO/ao-supervisor/pull/140
    GB - modification ready in IDL. Waiting for Armando to review.
         writing the rationale pending (other power outage)
         only needed for AO, not SL
         no Luci-AO on DX, no LBTI until restart (early September)
         LBTI impact (to have the patch ready) is not certain, see Jun runs
    XZ - PR129 is mmc (that we are running now)
         PR140 patch is ready to go (deployed, but not used)

- dx mmc + "latest" (patches we have on SX)
    merge ASM
    merge WFS (LBTI handled by Fabio)
    daytime test, etc. Update Wiki AO Software Test. BM will make an update, interested parties can look at it.
    BM will take the lead for this activity (schedule, liaise with FR, AO support, etc)
    in fact, SX stable == SX latest

- housekeeper changes
    being reviewed in github, BM test on VM

- ncpa ICE              https://github.com/LBTO/ao-supervisor/pull/120
    review other comments BM, XZ, MB, APU in the PR
    range check missing. Logic to implement?
    allow IIF interface for SHARK-NIR. Note: they only use mode XX (xx TBC)
    phase 1: allow the mode they use and val < the max (35?) they used
    phase 2: more complete solution (lots of work)
      note: LUCI/LBTI currently have some checks for ncpa? probably not
            LBTI doesn't use ncpa that often, iLocater use slightly different ones but
            LUCI has a table + rotator dependent
            The GUI allows to type in numbers? no
    this feature is about IIF/ICE, existing instruments use lower level directly from WFS
    Al shared an update (email) from Jacopo, will be discussed Mon Jun 20 (although mostly about shipping)
      need more iteration how they intend to use NCPA/asm vs. Internal/DM
      GB thinks it is internal DM
    The IIF should be generic

- fault recovery        https://github.com/LBTO/ao-supervisor/pull/79
    Patch ready since end of 2021
    May need more work, keep on the back burner or phase it. GB and XZ will decide
    Disable button is good for now

- revision control      Recap table what *runs* where, realignments pending/needed?
    BM will create a recap / table
    disable button: on SX, missing on DX? To be checked
    also add what could be tested next (day) and future (latest)

- IDL license server    status?
    XZ: back to lbto license server
    GB: remote vs local server, dependencies
    XZ: local version is old, only one seat
        IDL checker actually spawn an instance to check if it's working ... so would defeat the single seat
          may work for 1 minute ... but dies when the ping kicks in
        Harris said our license is not "upgradable" within the current arrangement
        Things might get better as we upgrade OS, IDL version, etc.
        Steward license server could be used if we had a complete failure of our own one

- lbti wfs INDI         PG looking into it
- log analysis          wfs freq/mode, from where?
                        maybe in the fastdiagnostic.log > can be use if not available for history
                        JC will share info by email (update modulation ...)
                          WFS > Python command: WfsArbScripts.updateModulation(self, sensor = 'FLAOWFS', instr = 'LUCIFER', freq = 900)
                        IDL log may have some too, runAO possibly not used during the day (AOS)
                        Ocam2k may have limited discrete freq. (or exp time)
                        params are "tuned" only if you do AoCheck (check flag on the AO GUI)
                        Processed    50 cycles of  7702 vars in  0.996 s (  50.2 Hz). HW frame rate    50.2 [DiagnApp.cpp:832]

- TSS                   remote power switch, James Reidl email
                        GB will join the discussion
                        moxa/adam process? e.g. running from a VM - for now, no action

- mmc on SX             restart schedule?
                        to be discussed next week? will be more clear once we have BM recap.
                          SX will be in the lab!
                          idea to create a timeline for convergence

- AOB

    JC - disable ncpa during optimize gain.
       - is it implemented?
        DM: yes it is, done by APU long time ago
        BM, DM and JC will regroup and let us know - (BM has some info from APU)
          https://github.com/LBTO/ao-supervisor/issues/115

- z0 daytime test outcome and schedule for next ECD (5 minutes)
  complete and passed

- AOS/RTDB patch (5 minutes)
  complete

- ocam2k lost frame, network, LBTI INDI branch out discussion (5 minutes)
  MB will send an email, reboot dxunit as a catch all
  EPI confirmed they would reboot on next test, but meanwhile system was used at night and no issue (uptime is 135 days)
  MB talked to PGR to check it out

- AO ops logging OSA tool/APU scripts update (5 minutes)
  MB looking at APU script

- steps to go > 900Hz (40 minutes)
    - preliminary stats, to be refined, MB provided a snapshot, APU working on it separatly
    - different oc causes - some fake, some true. hard to extract from log
      - e.g. shall is rested, cannot have any real oc (physically)
      - e.g. seeing limited OC, some correlated to wind and facing into it
    - change close loop count vs. close loop duration
    - Steve's student collected and annotated LBTI stats, google spreadsheet
    - 1-wk close scrutinity data, XZ shared link to google drive:
      https://drive.google.com/drive/u/0/folders/19CGOUwFXbBcOIO9ToZGNj9-3ZuRbK1FL
    - APU's script to be extended to include OC events, but they come from housekpeer/fasdiag, he looks at arbitraor logs (I think), easy enough
    - currently, the conclusion is that it's a statistcak exercise, daytime may not reveal much because it's so sparse

- AOB
    - "bug" in StopAO with mmc.
    GBR and XZH will provide a doc, there is a flag "hold"
    normally will flat the shell. Hold would leave it where it is.
    existing code does not work well with mmc.
      implementation might have been problematic anyways
    patch submited, allow freeze and which modes to freeze, discussion with Armando and Marco, pending
      candidate to test next week.
    in parallel, purpose of the hold and deos it really work
      used as a diagnostic feature (e.g. before it skips to understand the issue)
      need more iteration with Enrico to understand his use case
      history: might have been a feature requested by LBTI, restart with less modes
      note the current pause does that (freeze in place)
      theory: it was meant to keep t/t, possibly because the offload is not working properly
    JC will be available for test
      XZ is taking some time off...
      We need to accept the change on soul-mixed-control, Brandon will check/fix the github and PR situation
      GB will coordinate the test/date/plan - 2 hours minimum
      if it works, next ENG is July 4
        day time test to check
        not needed at night, needs more iteration

    GB will open the discussion with Arcetri
    WFS eng gui has an option to freeze or not (checkbox)
    is it related to the pause+skip and we discussed to flatten during pause (possibly keeping t/t to offload itself)
    offload to hexapod is 1Hz, but applied porbably less.
      offload goes to PSF and the TCS decides what goes where (TBC)
    offload to M1 probably even slower (10-30sec?) - but probably not an issue (see below)
    "issue" with ncpa offset makes the job harder on the wfs
    LBTI has sot/hard ncpa, maybe we should do that with Luci also
      bin1 + ncpa is definteyl a bad combo
      ncpa with luci is to be defined, may cause the psf to be same if not worse
      need some more discussion and evaluation - extreme perf vs stability, usual discussion
        possibly an option in PIT/OT, like a check box - many use case perf is not essential
        OSCO always failing at bin1 (bright because twilight)
        does bin1 work at all (even without ncpa)? i.e. reliability
        who's allowed to edit the ACE/AOS tables
        need to confirm LBTI 400 modes, LUCI 560?

    - SOUL-comm
      RIXes ...
        SoW is in most part h/w spec
        there is some tables/requirements for FLAO, Soul is required to operate "as well"
        FLOA was accepted based on requirement (on sky) - where is the doc?

google link wiht SOUL documents from Enrico:  https://drive.google.com/drive/folders/1_YSZBX1MBahkttMcrwgV9b-PKvUqBI1a
memo of understanding: https://drive.google.com/file/d/1q2YNl4IwhUwrmuV2ZbTb4zOUZXxcxvdw/view

- z0 daytime test outcome and schedule for next ECD (5 minutes)
  date ? http://doc.lbto.org/web/LBT_2022A.pdf
    JC: test ok, BM sent email instructions
    Mon 6 June (Tue 7 + Jul 4 also available)
    ECD: regular observation using NCPA, second night probably. JP will be on AO
    JP: start with dev version from the start of the night - all agree
    BR: writing the sciops OT scripts and test plan, use "known to be stable AO" scenario

- AOS/RTDB patch (5 minutes)
    June 6 or 7, when Yang comes back
    try again (once) may work. will tell us more and if it still fails, more intrusive reap/re-establish link
    msgs/rtdb is what we have, has occasional issues (in general, not limited to AOS/AO), using established 3rd party middleware (e.g. custom RPC that became ICE) would bring a lot more refactoring, avoid if possible or longer term
    IT8397, IT8528 are different. but close enough, timeout -5001 on read or write/ack-handshake

- ocam2k lost frame, network, LBTI INDI branch out discussion (5 minutes)
    ocam2k going out live (display, but also gaps in TN) , every 10s or so
    reboot lbti-wfs-dx as pro-active measure, pass it by Fabio
    remove INDI ? LMircam might be using it (other than fill some fits keyword)
    used for other housekeeping tasks (e.g. power on/off equipment etc), should use 200% CPU or coredump in a loop
    ITs with ocam not live, 8250, 3957 - remind Enrico or others to enter ITs
      not same as badframe, not same at ocam2k not live from ITs, does not prevent operation
      no need to make ITs for now
      MB will send an email, reboot dxunit as a catch all

- AO ops logging OSA tool/APU scripts update (5 minutes)
    in progress

- steps to go > 900Hz (40 minutes)
    - preliminary stats, to be refined, MB provided a snapshot, APU working on it separatly
    - different oc causes - some fake, some true. hard to extract from log
      - e.g. shall is rested, cannot have any real oc (physically)
      - e.g. seeing limited OC, some correlated to wind and facing into it
    - change close loop count vs. close loop duration
    - Steve's student collected and annotated LBTI stats, google spreadsheet
    - 1-wk close scrutinity data, XZ shared link to google drive:
      https://drive.google.com/drive/u/0/folders/19CGOUwFXbBcOIO9ToZGNj9-3ZuRbK1FL
    - APU's script to be extended to include OC events, but they come from housekpeer/fasdiag, he looks at arbitraor logs (I think), easy enough
    - currently, the conclusion is that it's a statistcak exercise, daytime may not reveal much because it's so sparse
- AOB

Topic revision: r65 - 11 Aug 2023, MatthieuBec
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback