Telemetry Data Visualization Software Design

Initial Design Thoughts

Doug Summers, 23-July-2015:
  1. A "standard Dygraph" should be prototyped. You have already demonstrated a graph, but some needed alterations include:
    • a single parameter should have one axis presentation by default
    • two parameters should have two y axis presentation by default
    • three or more parameters should have a user selectable choice of single graph presentation (default), or synchronized individual graphs.
    • Connecting lines vs dots (data points) should be selectable (data points default).
    • trend lines (what Dygraphs calls regression lines) should be a selectable option
    • for multiple parameters, an "enable/disable" field should be displayed to allow for fast GUI updates without data reload.
    • We want to have the date and {condensed stream | parameter(s)} prominently displayed in the title (if it's more clear, let's use two separate lines)
    • the label "UT" should appear somewhere near the left or right X axis (so folks will know that time is presented in UT).
  2. The "entry" page as Kellee called it should be prototyped. Not all features can be implemented right away. I'd like to see us have the page created, with the following working:
    • default date of today always entered, with ability to select another date (the crosshatch is a calendar in case it wasn't obvious).
    • "end date" will not be worked now (grey out). Infrastructure to support multiple file parameter collapse and concatenation needs to be implemented before this can work.
    • Time filter should be able to be entered (start/end). I think Kellee will have this capability working for single files in a reasonable time, so we can include it on the GUI.
    • We won't have any "canned" graphs yet, but we should have a placeholder for a list. I will want to think about and expand on this concept later.
    • The filter section (containing subsys, side, and stream list) is desired to be worked now. For now, the selections will drive to a single stream selection for infrastrcture processing to CSV format and graphed presentation using date and time filtering as indicated.
    • The "custom" (create/browse) should be delayed until we make progress on the above items first.
  3. Once items 1 and 2 are done, you could begin to prototype the custom graph GUI. Some notes:
    • The date and time filter fields should be populated with whatever info was entered on the entry GUI. If none, default to "today". Fields can be changed by the user if desired.
    • If possible, searches should allow for partial match, displaying match info in the name/desc/stream fields as matches occur.
    • If a parameter is selected (highlighted), an "append" would capture and append the parameter into the "parameter list" field.
    • a "save" operation should allow the user to save the list to a named file in a path we should probably restrict (so we can easily find all the saved scripts and keep some order).
    • a "plot" or "draw" button would call Kellee's infrastructure, and then launch the graph from the resulting CSV. The original GUI should remain available for alteration if needed.
    • As in item 1 above, infrastructure for multi-file collapse and concatenation won't be available in the first prototype. If you can setup for exploration of a single file parameter collapse using this GUI, that should be helpful to test Kellee's primitive for single file parameter column and time compression.

Doug Summers, 17-July-2015:
I would like to pass along some ideas I have for consideration during prototype telemetry visualization. I've spoken to and demonstrated some concepts related to these ideas in the WPC/OC this week, and I can tell you that the management team is universally excited to be able to realize a generic capability like what we've been discussing. I drew up some thoughts for initial GUIs, and those are captured via hand-drawn cartoons (below).

  • Notional telemetry entry page:

  • Notional custom telemetry graph page:

Here are some thoughts that go along with the cartoons:

  • Two web GUIs seem to be sufficient for good filtering/graphing.
  • For canned or filtered subsystem/stream based output, selections of subsystem name, B/L/R, and filtered stream are needed (+draw button) unless a different date/time filter is needed from "today".
  • For custom visualizations, create or browse (for existing definitions) can be done. The concept of a "visualization definition file" will come in handy below.
  • Custom visualizations require the second GUI.
  • We will need a telemetry dictionary:
    • definition of parameters, columns, subsystem, and stream association in HDF5 files
    • definition of related metadata (units, descriptions - useful for display and search)
    • We will need to consider some auto conversions (radians are not user units)
  • The second GUI implements several features/ideas:
    • parameter entry and/or search capture with append
    • The visualization definition can be created/saved or found and modified via a browse feature
    • A plot/draw launch (leaving the original screen intact if possible for further use)
    • A default date (UT today), with optional entries for date ranges and time filters
      if no changes to date/time filters, just launch the graph. (very few button clicks required to view a graph)
    • A search capability on the database is desired
      wildcard search? with selection ability to append to the parameter list for identified options

  • based upon the ideas above, I see several important interfaces:
    • The telemetry dictionary is one.
    • The visualization definition file is another
    • The program parameter interface (both for creation of CSV file, but also for time/date filtering)
    • The CSV file (driven by Dygraphs requirements)
    • We may want to consider a better filesystem structure along the way (how about /telemetry/subsys/date, or equiv?).

A question we'll have to wrestle with is: how will we deal with changing stream definitions in the telemetry dictionary? A second question we might want to consider is: Should we have a (hidden) field in the definition file or a separate list to allow for display of the "canned" list?

For now, all this should be food for thought in the prototype effort. Take it all with a grain of salt. Some of the ideas are clearly for longer-term, but hopefully having an idea of the long term will keep us from painting ourselves into a corner too quickly in the prototype. I would like to see us tackle simple (today) graphs for stream based tememetry (with time filters) certainly this year. Can we do better? I'm hoping so. We all have lots to do with other tasks, but let's see how far we can get.

Stephen, one speed improvement I noticed from a perusal of the test cases for Dygraphs is that there is a field "enable/disable" flag that should probably be used (as opposed to redraw of all data each time). A good look at all the test cases should be a requirement so you'll have a good idea of implemented capabilities "out of the box".

Finally, I would like to reiterate again that we should reuse existing code wherever possible. This includes dygraphs code/test suite, the HDF5 tools/mods we've made, and TCS/astro library functions (time conversions, etc.). I know Kellee is getting close with one piece of the puzzle. We can meet again and discuss as soon as some of the other items on our plate allow.

p.s. I'll note that in the future, some consideration of similar concepts (but not HDF5) might be valuable for webbased viewing of syslog fragments and the alarm handler. I'm in the formative stages of thinking through the issues involved.
I Attachment Action Size Date Who Comment
IMG_20150717_084357243.jpgjpg IMG_20150717_084357243.jpg manage 2 MB 18 Aug 2015 - 16:43 UnknownUser Notional telemetry vis web pages
IMG_20150721_084649827.jpgjpg IMG_20150721_084649827.jpg manage 2 MB 18 Aug 2015 - 16:43 UnknownUser Notional telemetry vis web pages
Topic revision: r2 - 08 Apr 2016, KelleeSummers
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback