xWRLx432 Gesture Recognition Users Guide

Table of Contents

Device Compatibility

⚠️ This lab is currently only compatible with the xWRLx432. Please note, the machine learning model trained for this demo was trained using the xWRL6432. Performance on the xWRL1432 device is slightly degraded because of this and is not intended as a production-ready demo.

Overview

This lab demonstrates the use of TI mmWave sensors for gesture recognition applications. The range, velocity, and angle data from mmWave sensors can enable the detection and classification of several natural gestures. The example provided in this demo can recognize 6 distinct hand gestures: Left swipe, Right swipe, Up swipe, Down swipe, Push, and Pull at a distance of 2m.

Quickstart

Prerequisites

Prerequisite 1 - Run Motion and Presence Detection Lab

Before continuing with this lab, users should run the Motion and Presence Detection Demo for the xWRLx432 to gain familiarity with the sensor’s capabilities and the tools used in the Radar Toolbox.

Prerequisite 2 - XDS Firmware

If using an xWRLx432BOOST EVM, the latest XDS110 firmware is required. The XDS110 Firmware runs on the microcontroller onboard the xWRLx432BOOST, which provides the JTAG Emulation and serial port communication over the XDS110 USB port. Packet loss has been observed on the serial port (COM ports) with some versions of the XDS110 firmware which can cause the demo to fail.

The latest XDS110 firmware is installed with the latest version of Code Composer Studio. CCS version 12.2 or later required.

Prerequisite 3 - PC CPU and GPU Resources

The Industrial Visualizer requires sufficient GPU and CPU resources to process the incoming UART data every frame and run the demo smoothly. Users with insufficient PC resources may notice occasional visualizer lag or missed frames in the log file. The demo should still run on most PCs regardless of hardware resources, but performance might vary.

Requirements

Hardware Requirements

Item Details
Device xWRL6432BOOST or xWRL1432BOOST
Computer PC with Windows 10. If a laptop is used, please use the ‘High Performance’ power plan in Windows.
Micro USB Cable

Software Requirements

Tool Version Download Link
TI mmWave L SDK 5.5.4.2 TI mmWave SDK 5.x
Uniflash Latest Uniflash tool is used for flashing TI mmWave Radar devices. Download offline tool or use the Cloud version
Code Composer Studio Latest Code Composer Studio
TI Radar Toolbox Latest Radar toolbox should be downloaded to access binaries and source code. Download Instructions in the readme file.

1. Configure the EVM for Flashing Mode

Follow the instructions for Hardware Setup of Flashing Mode

2. Flash the EVM using Uniflash

Flash the .appimage fil in the prebuilt_binaries directory. Follow the instructions for using UniFlash

The flashable binary can be found at: <RADAR_TOOLBOX_INSTALL_DIR>\source\ti\examples\Industrial_and_Personal_Electronics\Gesture_Recognition\xWRLx432_gesture_recognition\prebuilt_binaries\

3. Configure the EVM for Functional Mode

Follow the instructions for Hardware Setup of of Functional Mode

4. Run the Lab

4.1. Open Visualizer

Navigate to <RADAR_TOOLBOX_INSTALL_DIR>\tools\visualizers\Applications_Visualizer\Industrial Visualizer. The visualizer software may be run from Python 3.7.3 or as a standalone executable. In general, running the visualizer from the Python source runs faster and results in fewer errors over long periods of time. If running Python directly from source, run the setUpEnvironment.bat script first to ensure all the correct packages and versions are installed. Then run the visualizer either directly from source with python gui_main.py, or as an executable by running the Industrial_Visualizer.exe

4.2. Reset the xWRLx432BOOST EVM by pressing the RESET_SW button

4.3. Select the arrowed options as shown in the below graphic.

  1. Set the Device to appropriate device (xWR6432 or xWR1432) as indicated by arrow 1

  2. Set CLI COM to the appropriate COM port as indicated by arrow 2. If the xWRLx432BOOST EVM is plugged in when the visualizer is opened, then the COM ports filled in automatically. If they do not appear, check the Device Manager on Windows for the XDS110 Class Application/User UART port, and use this port number in the visualizer.

  3. Select the xWRLx432 Gesture Recognition demo as indicated by arrow 3.

  4. Press the Connect button as indicated by arrow 4

  5. Press the Select Configuration button as indicated by arrow 5. Navigate to <RADAR_TOOLBOX_INSTALL_DIR>\radar_toolbox\source\ti\examples\Industrial_and_Personal_Electronics\Gesture_Recognition\xWRLx432_gesture_recognition\chirp_configs and select the appropriate cfg file based on your EVM.

  6. Press the Start and Send Configuration button as indicated by arrow 6.

4.4. Running the Demo

This lab demonstrates the following features.

Perform the above gestures while standing at a distance of 2m from the radar sensor. The detected gestures will be displayed in the visualizer.

This concludes the Quickstart Section

Developer’s Guide

Import Lab Project to CCS

To import the source code into your CCS workspace, a CCS project is provided in the lab at the path given below.

🛑 Error during Import to IDE
If an error occurs, check that the software dependencies listed above have been installed. Errors will occur if necessary files are not installed in the correct location for importing.

Build the Lab

Selecting Rebuild instead of Build ensures that the project is always re-compiled. This is especially important in case the previous build failed with errors.

🛑 Build Fails with Errors
If the build fails with errors, please ensure that all the software requirements are installed as listed above and in the mmWave SDK release notes.

Execute the Lab

There are two ways to execute the compiled code on the EVM:

Configuration Commands

The configuration file (.cfg) can be found at <RADAR_TOOLBOX_INSTALL_DIR>\source\ti\examples\Industrial_and_Personal_Electronics\Gesture_Recognition\xWRLx432_gesture\chirp_configs\gesture_recognition_2m_xwrl6432.cfg. The majority of the configuration commands for this demo are the same as that of the Motion and Presence Detect Demo. You can find more information on these configuration commands in the SDK documentation at <MMWAVE_SDK5_INSTALL_DIR>\docs\api_guide_xwrL64xx\MOTION_AND_PRESENCE_DETECTION_DEMO.html.

The additional configuration commands which are unique to this demo are described below.

UART Output Data Format

This demo outputs data using a TLV (type-length-value) encoding scheme with little endian byte order. For every frame, a packet is sent consisting of a fixed sized Frame Header and then a variable number of TLVs depending on what was detected in that scene. The TLVs for this demo include the extracted features used for the neural network inference as well as the detected gesture for that frame.

Frame Header

Length: 40 Bytes
A Frame Header is sent at the start of each packet. Use the Magic Word to find the start of each packet.

Value Type Bytes Comments
Magic Word uint64_t 8 syncPattern in hex is: ‘02 01 04 03 06 05 08 07’
Version uint32_t 4 Software Version
Total Packet Length uint32_t 4 In bytes, including header
Platform uint32_t 4 A6843
Frame Number uint32_t 4 Frame Number
Time [in CPU Cycles] uint32_t 4 Message create time in cycles
Num Detected Obj uint32_t 4 Number of detected points in this frame
Num TLVs uint32_t 4 Number of TLVs in this frame
Subframe Number uint32_t 4 Sub-Frame number

TLV Header

Length: 8 Bytes
A TLV Header is sent at the start of each TLV. Following the header is the the TLV-type specific payload.

Value Type Bytes Comments
Type uint32_t 4 TLV Type
Length [number of bytes] uint32_t 4 In bytes

TLVs

Gesture Features TLV

Type: MMWDEMO_OUTPUT_EXT_RANGE_DOPPLER_FEATURES = 350
Size: 64 Bytes
The Gesture Features TLV consists of an array of extracted features some of which are used as input to the neural network. Each extracted feature is defined as given below.

Value Type Bytes Comment
rangeAvg float 4 range average across Range-Doppler heatmap
dopplerAvg float 4 doppler average across Range-Doppler heatmap
dopplerAvgPos float 4 doppler positive average across Range-Doppler heatmap
dopplerAvgNeg float 4 doppler negative average across Range-Doppler heatmap
numPoints float 4 number of detected points above a threshold across Range-Doppler heatmap
azimWtMean float 4 azimuth weighted mean across Range-Doppler heatmap
elevWtMean float 4 elevation weighted mean across Range-Doppler heatmap
azimWtDisp float 4 azimuth weighted displacement across Range-Doppler heatmap
elevWtDisp float 4 elevation weighted displacement across Range-Doppler heatmap
dopAzimCorr float 4 Hybrid feature which is a correlation between Doppler and Azimuth for numFrames
dopElevCorr float 4 Hybrid feature which is a correlation between Doppler and Elevation for numFrames
dopPosNegCorr float 4 Hybrid feature which is a correlation between Doppler positive and Doppler negative correlation for numFrames
dopPosElevCorr float 4 Hybrid feature which is a correlation between Doppler positive and Elevation for numFrames
dopPosAzimCorr float 4 Hybrid feature which is a correlation between Doppler positive and Azimuth for numFrames
dopNegElevCorr float 4 Hybrid feature which is a correlation between Doppler negative and Elevation for numFrames
dopNegAzimCorr float 4 Hybrid feature which is a correlation between Doppler negative and Azimuth for numFrames
Gesture Classifier Output TLV

Type: MMWDEMO_OUTPUT_EXT_CLASSIFIER_OUTPUT = 351
Size: 1 Byte
The Gesture Classifier Output TLV consists of a single uint8_t indicating the detected gesture.

Gesture Identifier
No Gesture 0
Left to Right 1
Right to Left 2
Up to Down 3
Down to Up 4
Push 5
Pull 6
Presence Detection Output TLV

Type: MMWDEMO_OUTPUT_EXT_PRESENCE_OUTPUT = 352
Size: 1 Byte
The Presence Detection Output TLV consists of a single uint8_t indicating the detected presence. This TLV is not sent out in gesture-only mode.

Status Value
No Presence Detected 0
Presence Detected 1
Presence Detection Threshold TLV

Type: MMWDEMO_OUTPUT_EXT_PRESENCE_DETECT_THRESHOLD = 353
Size: 4 Bytes
This TLV contains the sum of magnitude across 2nd Doppler bin from range ‘x’ to range bin ‘y’ in the presence mode. This TLV is not sent out in gesture-only mode.

Stats TLV

Type: MMWDEMO_OUTPUT_EXT_MSG_STATS = 354
Size: 24 Bytes

Value Type Bytes Comment
interFrameProcessingTime uint32_t 4 Interframe processing time in usec
transmitOutputTime uint32_t 4 Transmission data transmit time in usec
powerMeasured[4] uint16_t[4] 8 Power at 1.8V, 3.3V, 1.2V and 1.2V RF rails (1LSB = 100 uW)
tempReading[4] int16_t[4] 8 Temperature: Rx, Tx, PM, DIG. (°C), 1LSB = 1°C

Neural Network

This demo utilizes a neural network model which runs on the ARM core making predictions based on extracted features. A sliding window buffer of features, extracted from the data across a number of frames, is used as input to the neural network. The model outputs a set of probabilities, each represents the probability that the input data belongs to a specific gesture.

Features

The model is trained on following extracted features.

Extracted Feature Description
Doppler Average Weighted average of the doppler across the range-doppler heatmap
Elevation Weighted Mean Select the N cells of the range-doppler heatmap with the highest magnitude and for each cell compute the elevation bin (via angle-FFT). The average elevation is the weighted average of all these elevation bin indices.
Azimuth Weighted Mean Select the N cells of the heat-map with the highest magnitude and for each cell compute the azimuth bin (via angle-FFT). The average azimuth is the weighted average of all these azimuth bin indices.
Number of detected points Number of cells in the heatmap that have a magnitude above a certain threshold
Range Average Weighted average of the range across the heatmap.

Retraining

Users may wish to retrain the neural network model for various reasons such as adding or removing certain gestures. A full procedure for retraining and deploying a model is not provided in this guide; however, the extracted features output over UART when running this application can be saved and used training data for a new model.

Need More Help?