OVMS Software Troubleshooting

• OVMS Notes on Software wiki - links to docs, build procedures, etc.
• OVMS System on Engineering wiki
• UEI Vibration Sensor Interface Boards User Manual - includes the API the real-time software uses to communicate with the accelerometers

Machines / Users

Checking / Starting / Stopping

1. Check if real-time data is coming to the ovms telemetry machine
    

ssh ovms@ovms
ovmsSniffer -i 192.168.53.60 -f 1000
cntl-C to get out of the barrage of data 

1. Check if telemetry is being recorded

    ssh ovms@ovms
    ovmsTelemetryClient -s 
    Checking connection with LBT...
    Ctrl \ to abort the command.
    Telemetry is paused

2. If you see

   OvmsTelemetryI::telException::Telemetry service is not connected to the LBT

   then you are running -s the first time after telemetry restart. Just resume telemetry:

ovms@ovms ~]$ ovmsTelemetryClient -r
Checking connection with LBT...
Ctrl \ to abort the command.
Resuming the telemetry service
Telemetry is running at 1000 Hz
Max Freq =2600 Hz
Min Freq =1000 Hz

1. If you want to pause/resume telemetry collection, use the -p or -r options to ovmsTelemetryClient (more details below).

1. Log in to the RACKtangle (root@ovms-uei) and do a ps command (see more detail below)
2. If it's not running, or you kill it, restart it with
    

sudo service ovms-svc stop/start

Configuration

• The ovmsMonitor.cfg file (in /lbt/ovms/current/etc/) tells the applications running on the ovms Linux box which slot/channel is which accelerometer. This file is used by the ovms monitor GUI as well as telemetry. This file does not influence the multicast data.
  This file references the channels by slot and number.
  This file must be modified if hardware is modified.

• The config.txt file on the ovms-uei machine tells the OVMSplus prediction software which channels to use in the calculations.
  This file references the channels by incremental numbers, 1 through 44. Slot 1, channel 1 is 1; slot 2 channel 4 is 8; etc.

• The sensitivities.txt file on the ovms-uei machine tells the OVMSplus prediction software which gain factors to use in the calculations for particular focal station geometries (specified with the -P command line argument). Beware that focal station geometries are not applied in Mode 3 (see IT 7486).
  Copy in a modified version with scp sensitivities.txt root@ovms-uei: and then restart ovmsrtService.

Big OVMS Monitor Displays

Telemetry

SHELL=/bin/bash
OVMS_TEL_CLIENT_CONFIG=/lbt/ovms/current/etc/ovmsTelClientConfig.cfg
00 00 * * * /lbt/ovms/current/bin/ovmsTelemetryClient -r > /dev/null 2>&1
00 14 * * * /lbt/ovms/current/bin/ovmsTelemetryClient -p > /dev/null 2>&1


[ovms@ovms]$ ovmsTelemetryClient -h
Client for the OVMS Telemetry Interface Service (Ver. 0.6.1)
Usage :  ovmsTelemetryClient [-Option] [  ]
  -p (pause) : Pauses the archive of the vibration data into the Telemetry system.
  -r (resume) : Resumes the archive of the vibration data into the Telemetry system.
  -s (getStatus) : Shows the status of the OVMS Telemetry interface.
  -f Freq (setSamplingRateHz) : Sets the Sampling rate to archive the vibration data into Telemetry.
  -x (shutdown) : Shutdown the OVMS Telemetry interface. This may take some seconds...
  -h (help) : Shows the help page.(See also: man ovmsTelemetryClient)

  login to ovms as ovms
  > ovmsTelemetryClient -s         to check status
  > ovmsTelemetryClient -p         to pause collection if it is running
  > ovmsTelemetryClient -x         to stop the ovmsTelemetry daemon
  > cd /lbt/ovms/current/etc
  > edit ovmsTelConfig.cfg, setting TelService.Logger.Level to 2
  > cd
  > ovmsTelemetry &                    to run the ovmsTelemetry daemon so that it prints debug/error/trace messages
  > ovmsTelemetryClient -r             to resume telemetry collection and see what the problem is

Reduced rate Telemetry

  I ran a test of the OVMS telemetry collection at 500Hz.
This applies only to the HDF5 telemetry collection, not the real-time
broadcast data.

The OVMS telemetry collection frequencey defaults to the minimum rate
in the configuration file.  A config file change, and a bounce of the
telemetry daemon will make it run at the lower rate. I tested it
today, but put the configuration back to minimum 1KHz after my test.

To default to 500Hz:
   1. pause telemetry and stop the daemon - ovmsTelemetryClient -x
   2. change the file  /lbt/ovms/current/etc/ovmsTelConfig.cfg
       TelService.Sample.MinFreq=1000  to  500
   3. restart the daemon - sudo service ovms-svc start
   4. restart telemetry collection - ovmsTelemetryClient -r 

Real-Time Multicast

To stop/start the real-time service:

ssh root@ovms-uei.mountain.lbto.org
~ # ./ovmsrtService -f 1000 -d 12 -D 3.0 -H 0 -G 0 -P LBTI-noM1 -E 2 &



==== ovmsrtService version 1.0.3
#Board is set to 12
Estimation mode set to 2 (5 estimation channels)
#Board is set to 12
There are 4 channels specified:   0   1   2   3 
Multicast IP Address is set to 192.168.53.62
RT Service Listen Port is set to 4321
Operation Frequency is set to 1000.000000 Hz
Status Package Period is set to 2
High-Pass filter is set to 0
Gain is set to 0
Verbose level set to 0
ICP current set to 5 mA
Prediction Horizon is set to 0.000000 ms
Estimation mode set to 2 (5 estimation channels)
Configuration "LBTI" used for focal plane sensitivities

Mirror Data initialized
~ # Estimator initialized
Estimation successfully initialized: k=0.98  dt=0.00  T_delay=0.000000
Opening the datagram socket...OK.
Adding multicast group...OK.
Disabling the loopback...OK.
Setting the local interface...OK
Setting the socket priority...OK
Setting the socket multi-cast TTL...OK
One shot mode: Setting task period to 1000000 ns (1 ms)

 1098 root      9428 S    ./ovmsrtService -f 1000 -d 12 -D 3.0 -H 0 -G 0 -P LBTI-noM1 -E 2

usage: ./ovmsrtService [options]

        -h : display help
        -d n : selects the number of devices to use (default: 1)
        -f nnnn : set the rate of the UEI operation ([1-5000]Hz, default: 1000 Hz)
        -c "x,y,z,..." : select the number of channels to use (default: 4 channels)
        -s n : set the period to send the status package (default: 2 sec)
        -i "xxx.xxx.xxx.xxx" : set the multicast ip-address of the UEI device (default: 192.168.53.62)
        -p n : set the port of the UEI RT Service  (default: 4321)
        -a n: Set the ICP current value in mA (default: 5mA)
        -G n : set the Gain of the input cards (default: 0) where            
                 n = 0 -> Gain 1 (-12.5v/25v)           
                 n = 1 -> Gain 2 (-12v/12v)           
                 n = 2 -> Gain 5 (-5v/5v)            
                 n = 3 -> Gain 10 (-2.5v/2.5v)
        -H n : set the High-pass filter of the input cards (default: 0) where             
                n = 0 -> DC Coupling             
                n = 1 -> 0.1 Hz             
                n = 2 -> 1 Hz             
                n = 3 -> 10 Hz
        -E n : selects the OPD Estimation Mode (default: 0), where            
                 n = 0 -> no estimation           
                 n = 1 -> estimate differential opd, and focal plane motion (3 values)           
                 n = 2 -> estimate differential opd and SX/DX focal plane motion (5 values)           
                 n = 3 -> estimate displacement and rotation around x- and y-axis for each mirror separately (18 values)
        -v n : Set the verbose mode on for debugging purposes where             
                n = 0 -> Debug messages off             
                n = 1 -> Debug messages on
        -D n : Set the prediction horizon for the delay compensation (in ms, default: 0.0)
        -P n : Set the instrument for focal plane sensitivities configuration to use from "sensitivities.txt" (default: n=LBTI)

Error -10 Initializing Communication with IOM

To shutdown the racktangle:

ssh root@ovms-uei.mountain.lbto.org
~ # poweroff

Hardware Issues and History

Multicast Tools

 ovmsSniffer -i 192.168.53.60 -f 1000 

 ovmsSniffer -i 127.0.0.1 -f 1000

Troubleshooting Notes from User Manual

 > ssh root@ovms-uei

# ps

 1098 root      9428 S    ./ovmsrtService -f 1000 -d 12 -D 3.0 -H 0 -G 0 -P LBTI-noM1 -E 2

> ovmsTelemetryClient -s

> man ovmsTelemetryClient

[ERROR]: WorkerThread::Exception: [Error] DataBroker [::recv()] Error reading
from socket. No data. Too many timeouts.

[ERROR]: Telemetry connection with UEI has timeout.

 > ovmsTelemetryClient -r

[ERROR]: LBT Telemetry is not working. Once the problem is fixed restart the
OVMS Telemetry service.

[ERROR]: Telemetry service is in an error state. Try to resume it or restart
the OVMS Telemetry service.

[INFO]: Telemetry is paused.

 > ovmsTelemetryClient -r

[ERROR]: Connection refused, Telemetry service [ovmsTelemetry] is not accepting
connections or it is not running.