Arcetri AO Software Q&A

Week of 2022-10-30

• [JCG]. The WFS control Gui has the botton SET SAFE for the OCAM safety. Can we log that command in the in the wfsarbitrator log?
  The wfarbitrator log save the keyword: setsafe; Received cmd: setSafe(). No action required.

Week of 2022-08-22

• [JCG] It is observed the auxloop generates multiples logs files (auxloop.L.*)with a size of 250Mb. The day August,16th there is 27 logs files with a total memory of 6.3Gb of auxloop . almost the 100% of the info is related to not attached to the share memeory. Could we define the relevant information to be logged?, and also a better way to catch those repeated information.
  • [BM] was thinking maybe we could log when shared memory becomes available and when it becomes unavailable and then suppress the rest of the error messages.

• [JCG] There is an instruction of how to build a ARGOS rec for true sensing from FLAO rec, now we have SOUL, could you check the information that we have?, is it still valid?.. Link to the wiki: https://wiki.lbto.org/ARGOS/HowToBuildFlaoAndTrueSensRecsForArgos
• [BM/JCG] Can irtcccd be safely removed from the interfaces list in processList.conf on soul-sxwfs and soul-dxwfs?

Week of 2022-06-13

• [JCG] The Optimize gain seems not disabling the NCPA tag when it waa running with the ncapm tag on. Can we add a disable of the ncpa tag in the optimize gain process?
  • [AP] I am not sure that disabling NCPA during autogain is a good idea, especially if NCPAs are big, they will change the pyramid response and the optimized gains might be differen
    • [JCG]. So, this is an important point when the ncap is activated. As it is on operations now, the NCPA is applied after the GOPT and if I remember correctly it is not applied before the optimize gain process. I agree if the NCPA vector is large it could have an offset in the slopes and the Gopt will not catch it accordely. I think this is a point to clarify and also we need to have a better understaing or computation of the NCAP vectors.
  • [BM] Should setting wfsarb.<side>.NCPA_TRACK.REQ=0 at the beginning of autogain.doit do it? (#115)
    • [AP] Yes it should be enough. Actually my first comment in that issue was wrong, I updated it.
  • [BM] Unrelated, is wfsarb.<side>.AUTOGAIN_RUNNING a catch-all for "pause optModalGainTask/modalPlotTask?" I noticed that both optimizeGain and updateNcpa flag it while they're running. Maybe it was originally set only by optimizeGain and then later decided updateNcpa needed to pause them too?
    • [AP] Your reconstruction is correct, but I have a hard time remembering why we added it to updateNcpa. The update is very fast and AUTOGAIN_RUNNING will go back to zero in a fraction of a second. Coupled with the fact that the auxloops tasks only check this variable at the beginning of each iteration, they will probably never catch it. I would be in favor of removing the AUTOGAIN_RUNNING writes from updateNcpa and see if nothing breaks.

Week of 2022-05-30

• [JCG] How many scripts are essential for the Soul LUCI maintenance and check? The directory scripts has many python scripts, so it will be good to know how many of those are good to keep and maintain.

Week of 2022-05-23

• [BM] Does anything observe the command result SysCmdRetry (AbstractSystemCommandResult) / CMD_EXECUTION_RETRY (CommandStatus) and retry? I can't find it used anywhere, but I've noticed there are a lot of adsec and wfs arbitrator commands, error codes from IdlSystem, and some MastDiagnSystem and OptLoopDiagSystem commands that return SysCmdRetry.
  • [AP] The idea was that a command could fail because of reasons external to the AO system (i.e. bad seeing, etc) and thus one could just retry the command, as opposed to an error that points to some hardware or software problem. But in the end all retries are treated as ordinary errors, so this feature is not really used.
• [BM] We have an old slide that mentions a suggestion you had for disabling downloading of slopes from the AdSec: "the Switch BCU sends down UDP packets with the loop slopes, but they are not used for diagnostic purposes, and use up a lot of bandwidth/internal BCU load. / not so easy to do, requires coordinated SW and configuration update." What are your thoughts for this?
  
  • Looking at the data size (BcuSwitchFrameSizeDw = 2868), is this the data downloaded by MasterDiagnostic::_bcuSwitchFramesDownloader? So would it be a matter of just resizing this/changing the memory address range it requests to not download the slopes? (Or is that all it's grabbing? I guess I need to check what it's requesting ...)
  • And then _adSecBouncer sends this data over UDP to the WFS, which then shares it as ADSECBUF? What all data is sent? I see that both _bcuSwitchFramesDownloader and _bcuMirrorFramesDownloaders put data into _adSecDiagnBuffer, so maybe it's more complicated because we still want some of the data, and we'd need to reorganize that buffer and how it gets parsed?
    • And what processes read from ADSECBUF?
    • [MB] EnableAdSecBouncer seems to be set to 1 in our config, but the IP is a WFS private that cannot be reached... What was the intent of the bouncer? maybe the entire feature can be removed?
  • [AP] yes I still think that disabling the slope downloading from the AdSec would be an improvement. But I don't know if it can be done entirely on the AO software side, without having to touch something on the Switch BCU firmware. The data is sent by the BCUs autonomously, the MasterDiagnostic process is just listening for it, so it's not enough to change the BcuSwitchFrameSizeDw value since it has to match what the Switch BCU is doing. The buffer to be sent is determined by the Switch BCU firmware, based on the memory map configuration. The memory map is under our control and is configured by IDL while starting up the system (in the LoadProgram command I believe), but which memory buffers are involved are probably hard-coded in the firmware. I think that we have to confirm with Microgate.
  • [AP] The bouncer sends a fraction of the data to the WFS, so that it can be correlated with the WFS data (using frame counters) and saved all together. The data that is "bounced" is the MirrorOutputDiagn structure in lib/AdsecDiagnLibStructs.h and does not include the slopes. The IP to sent it to is set by the SelectFocalStation arbitrator command, which scans adsecarb.conf for a list of keywords starting with "FocalStation_". Each keyword has the IP or hostname of the wfs, the UDP port, and an integer that selects a fiber input port on the Switch BCU.

Week of 2022-05-09

• [MB] TCS/AOS RTDBVar create=true/false, see slack https://lbtobs.slack.com/archives/C02GMFJV4SD/p1651619240797739
• [MB] TCS/AOS old style communication with arbitrator (pre-ICE / NT), can we remove it for sanity's sake? There could be a worrisome Zero() method that resets the AOS[LR]flaoICEifEP (haven't investigated fully), less code seems better and because old/new compiled, it leaves some things dangling (e.g. the missing op:code for unused waffle). More things to clean-up on ao-supervisor if we decided it.
  • [AP] It should be possible to remove everything using the arblib/aoArb files, that is, all commands to the AO Arbitrator not using ICE. The MsgD connection should be kept, because there are several variables that are mirrored in both directions.
• [MB] remove TV_IMAGE (ccd47) from AOS RTvar ?
  • [AP] It used to be a variable to display the ccd47 image on the AOS GUI, but it has never worked properly and I see that it has been basically lost from the AO codebase, so yes it can be removed.
• [MB] email started by Fabio about /local/aomeas mount and rsync ... next steps?

Week of 2022-04-11

• Collect Adsec in PauseAO or Open loop using the optical loop data GUI.
  
  • [JCG] Can we have the option to collect Positions/commands of the Adsec data in the PauseAO scenario?.
  • [BM] While debugging the pause and resume commands, we've noticed that when the WFS is paused, the optical loop data reports 0's for positions. I think this is because the MasterDiagnostic pauses the adsec frame downloader when the loop is paused (MasterDiagnostic.cpp:L674-699). Would it be safe to just disable this check, and would it allow us to keep collecting position/command data when the loop is open?
    • [AP] Now that pauses keep the loop closed, it is probably safe. The adsec frame downloader was disabled during pauses because no frames were expected since no slopes were sent.
• Gain zero. disturbance running.
  • [JCG] In the RunAO -PauseAO the gain goes to zero, with this gain zero the disturbance is still running. Can/should the disturbance be stopped/disabled in gain zero scenario?
  • [MB] I see we discussed it earlier in https://github.com/LBTO/ao-supervisor/pull/44/files#r725199403, maybe it's a non issue. We'd be interested to understand better your comment in https://github.com/LBTO/ao-supervisor/pull/44/files#r725204177 . Nota Bene: we seem to understand the disarm, stop_fastlink, rearm when it will restart (in previous implementation). As I understand (TBC?) two types of signal can be injected: the sine waves generated by adsec used by gopt (terminology=disturbance?), and other 'daytime' (terminology=turbulences?) generated by the wfs slopes (e.g. simulate atmosphere) ? I'm missing how the 2 bits to enable/disable work - one goes to adsec, the other to wfs, both play their own CBs?
    • [AP] Both bits are set in the WFS memory and are transferred to the adsec together with the slopes. The first bit will trigger a disturbance frame whenever a slope frame arrives. The second bit triggers a disturbance frame on the so-called "oversampling" frames. These are frames with zero slopes that are auto-generated internally by the adsec when the slopes from WFS stop arriving - so that the whole computational chain keeps running. And also, the oversampling allows to generate internal frames at a rate higher than the WFS - say, the WFS is running at 200Hz, and the oversampling can do 4x at 800Hz, and this way, the disturbance can be run at a higher frequency than the WFS, for a more realistic behaviour. The details of how the oversampling works have always escaped me
  • [BM] In the WFS arbitrator, maybe we can:
    • doPauseLoop: store the disturb status and call sc.disable_disturb if its sc.get_disturb_status shows it's enabled
    • doCloseLoop: call sc.enable_disturbance if it was previously enabled
      • [AP] I guess this is OK, it's basically the same code that was deleted in https://github.com/LBTO/ao-supervisor/pull/44/files#r725199403 with the enabling moved to the close loop command. But maybe you mean doResumeLoop?
  • [MB] I know gopt, do we have a list of all the background optimization/auxloops tasks that might be running in the background, and are we sure they will suspend during our "pause"?
    • [AP] No centralized list AFAIK... gopt, pupil centering (on the WFS) and offloading (on the adsec) are the ones that must be suspended because they depend on feedback from the system. To this I would add the modal gain optimization in the auxloops, when it will start working (currently disabled AFAIK). Others like the NCPA tracking are open loop applications.
• skip frames prior to pause/offset interrupts the offset
  • [MB] we're wondering if the new pause (set-gains-0) because it simply leaves the shell where it is, and that could be commands that make the system skip, is the reason we're seeing this. Otherwise stated, we're wondering if the previous implementation (that interrupted the fast-link and transitioned to OSF), might have had some other effect, for example flattening the shell, and thus eliminate the skips? Could this explain the new behavior? could "flattening" the thin shell if it's skipping be a solution? This would be a flatten of the TS only, not a flatten as available from the GUI that (as I understood) also moves the hexapd.
    • [AP] The previous implementation also left the shell position where it was, since the only way of moving it is to send a new slope frame that will trigger a delta-command. But maybe skips will not be generated if a slope frame does not arrive, and that may be a difference. The "Apply flat" button only flattens the shell by (as I understand it) a complete reset of the internal integrator and a re-application of the original flat commands, the hexapod is not touched and, if it is tilted in some direction, it will remain that way.
• Camera lens tracking stops at 50% skip
  • [MB] is there a snowball effect? does it recover on its own?
    • [AP] It stops when there are too much skips because the pupil shape is probably distorted, and since the algorithm relies on the pupil being more or less round, it would probably move the camera lens away from the good position. It should restart on its own if the adsec manages to stop skipping... which does not happen very often at those percentages.
• [MB] can we control close-loop with/without gopt individually?
  • [AV] Ways to get a closed loop without the gopt disturbance:
  1. Close the loop entirely from the engineering (WFS and Adsec Control) GUIs
  2. Close the loop from FLAOGUI, then open and reclose from the engineering GUIs
  3. Set the amplitude of the disturbance to 0 in the ACE table, then close from FLAOGUI
  4. There's a parameter called "doOptLoopDisturb" in AOARB.conf: https://github.com/LBTO/ao-confcalib/blob/abede11544f634c1d3ef24471c9d9e1e8f8443c0/conf/adsec/672b/processConf/AOARB.conf#L37 (I haven't tried this one)

Week of 2022-04-04

• Hold Button.
  • [JCG] There is a "new" button in the Adsec Engineering gui called HOLD GAIN?. Is this option available and what is it does?
    • [AP] It was added following a request from LBTI: it takes the current gain filename and writes it into the gain vector editbox. The idea is that, if the loop opens or crashes or stops for any reason, one can reclose it and then click on the gain "Apply" button in order to quickly have the same loop gain as before.
    • [JCG]. So what happen with the Gopt?.The re-close option counts for the Gopt gain.

Week of 2022-03-21

• [BR] ncpa status in data dictionary:
  • Is the NCPA state tracked anywhere in a variable that can be sent to the Data Dictionary?
    • [AP] Not at the moment. It should be something published by the ADSEC process, but we have not implemented it.
  • What effort does it take to add a new variable (or multiple variables) to the AOS?
    • [MB] AOS implements a mirroring of RTDB, addind new variables is done here: https://github.com/LBTO/tcs/blob/1047ac7380535b8b9cc00053b82c99aed75f00f2/aos/FLAOifApp.cpp#L1185 new entries should be declared in the TCS reflective memory, The best approach is to use "spare" slots so the map remains backward compatible, but this is optional. We typically bump the TCS release and rebuild it entirely. This is done here https://github.com/LBTO/tcs/blob/1047ac7380535b8b9cc00053b82c99aed75f00f2/aos/refmem/refmem.list. Few more steps are desirable e.g., showing it on the AOS GUI. The mapping between DD and refmem is done here https://github.com/LBTO/tcs/blob/1047ac7380535b8b9cc00053b82c99aed75f00f2/iif/configuration/IIF/DDNames.conf In summary, it is quite simple for AOS. Creating new RTDB variable in ao-supervisor is also quite simple. Finding where and when to update the values might be the biggest challenge.

Week of 2022-03-14

• bayside stages:
  • [BM] what's the history of CopleyCtrl vs StagesCtrl?
    • [AP] StagesCtrl is the program developed at the beginning of the FLAO project, dedicated to control the three Bayside stages. Each stage has two controllers encoders (a rotary one on the motor, and a linear one on the stage itself), and the Bayside-provided controllers had some issues dealing with this configuration. We ended up replacing them with Xenus controllers developed by Copley Controls, which proved much more stable, but it required a new control program. I guess that the StagesCtrl code could be removed at this point, since we aren't going back for sure.
  • [BM] IT 8602 describes events where the bayside stages report an error that they've been requested to move out of their range, but simplemotor.py/bayside_stage.py aren't set up to detect this and instead timeout and retry twice, wasting a lot of time. I have an idea of how to fix this, but want to make sure I understand the intended use of some AOApp RTDB vars first. Comments much appreciated: https://github.com/LBTO/ao-supervisor/issues/112#issuecomment-1068543777
• Hold Button.
  • [JCG] There is a "new" button in the Adsec Engineering gui called HOLD GAIN?. Is this option available and what is it does?
    • [AP] It was added following a request from LBTI: it takes the current gain filename and writes it into the gain vector editbox. The idea is that, if the loop opens or crashes or stops for any reason, one can reclose it and then click on the gain "Apply" button in order to quickly have the same loop gain as before.

Week of 2022-03-07

• auxloops/skipFrameTask:
  • [MB] logs from (new auxloops developement?) do not go into standard location, we just realized that. Can we move them to standard ~/aologs instead

    [sxwunit@lbti-sxwfs source]$ ls -al /home/sxwunit/aoroot/bin/*.log -rw-rw-rw-. 1 sxwunit sxwunit 11879455 Mar 9 23:44 /home/sxwunit/aoroot/bin/auxloops.log -rw-rw-rw-. 1 sxwunit sxwunit 1960739032 Mar 9 23:44 /home/sxwunit/aoroot/bin/ModalplotTask.log -rw-rw-rw-. 1 sxwunit sxwunit 258605212 Mar 9 23:44 /home/sxwunit/aoroot/bin/OptModalGainTask.log -rw-rw-rw-. 1 sxwunit sxwunit 76093212 Mar 9 23:44 /home/sxwunit/aoroot/bin/SkipframeTask.log 

    • [AP] True, we have to move them
  • [MB] skipFrameTask "silently" reduce the gains
    • could this break our "new" pause-set-g0/resume-from-values-stored-in-gain-file (I am thinking it does not really matter after the resume, but there could be corner cases I do not understand)
  • [MB] skipFrameTask reduce the gains, how do they get resinstated to normal when skip is gone?
  • [MB] skipFrameTask seems to be a development feature, never been used according to [AV]?
  • [JCG] skip frame reduction. Do we know that the reduction percentage will work? I've seeing during daytime work that a reduction of the gain or even applying gain zero doen't reduce the amount of skip fames.
    • [AP] General answer for the skipFrameTask: unfortunately we haven't worked a lot on it, and now it seems that it could have unexpected interaction with the rest of the system. If it is enabled in the defalt configuration, I would recommend to disable it for the time being.
• noob/misc
  • [MB] housekeeper and fastdiagnostic seem both to probe status out of the asm. I have not looked at the details and they could be different, but there seem to be some overlap (skipframe for example). Was that done on purpose, or could be something to think about refactoring?
    • [AP] they read status at two very different speeds (1Hz for housekeeper and ~100hz for fast diagnostic), there is little overlap but some counters are duplicated. At the beginning, we just used the fastdiagnosic for the skipframe counter. But, after the SOUL upgrade we say that often the status vector read from the Switch BCU was all zero, and we reimplemented the skipframe detection using the data read by the housekeeper. I guess that it could be considered a firmware bug on the Switch BCU.