Overview
===========
This lab demonstrates the use of TI mmWave sensors to count and track multiple people simultaneously.
Detection and tracking algorithms run onboard the IWR6843AOP, IWR6843ISK-ODS, or IWR6843ISK mmWave sensors and are used to localize people and track their movement with a high degree of accuracy.
mmWave sensors can reduce false detections from challenging environments such as direct sunlight, no-light, fog, or smoke, and are particularly suited for privacy-conscious applications.
In this demonstration, localization and tracking is performed upon any moving object in the scene; a new feature allows static objects to be ignored until a person is determined to have stopped moving.
The IWR6843 mmWave sensor outputs a data stream consisting of 3 dimensional point cloud information and a list of tracked objects which can be visualized using the software included in this lab.
<img src="images/pplcount_overview_block2.png" width="450"/>
[[r! IWR6843 ES2.0 Only
This lab is only compatible with ES2.0 version of IWR6843. Check the device version on your IWR6843 using the on-chip device markings as shown below>
1. If line 4 reads `678A`, you have an ES2 device. In this case, this lab is compatible with your EVM.
2. If line 4 reads `60 GHZi`, you have an older ES1 device. In this case, the lab is NOT compatible with your EVM. ES2 IWR6843ISK/IWR6843ISK-ODS boards are orderable from the EVM link above.
<img src="images/iwr6843_silicon_revision.png" width="500"/>
]]
[[r! AoP ES2.0 EVM only
The IWR6843 AoP version of this lab is only compatible with ES2.0 silicon and the corresponding EVM. Please ensure your EVM is the same as in the below image.
<img src="images/iwr6843aopevm1.png" width="300"/>
]]
Quickstart
===========
### Prerequisite 1
[[y! Run Out of Box Demo
Before continuing with this lab, users should first run the out of box demo for the EVM.
This will enable users to gain familiarity with the sensor's capabilities as well as the various tools used across all labs in the mmWave Industrial Toolbox.
]]
### Prerequisite 2
[[r! XDS110 firmware downgrade required
The XDS110 Firmware runs on the microcontroler onboard the MMWAVEICBOOST Carrier Card which provides the JTAG Emulation and serial port communication over the XDS110 USB port. We have observed packet loss on the serial port (COM ports) with some versions of the XDS110 firmware which can cause the demo to fail.
To avoid packet loss, users **must** downgrade the XDS110 firmware on their carrier card to version 2.3.0.18 as given below:
1. Got to the <a href="https://software-dl.ti.com/ccs/esd/documents/xdsdebugprobes/emu_xds_software_package_download.html" target="_blank">XDS Emulation Software (EMUPack) Download</a> page and download version `8.1.0.00005` (only 32-bit version is available for this release but it is compatible with 64-bit Windows).
2. Run the installer to install the EMU pack on your Windows PC.
3. Follow the instructions provided in `XDS110_firmware_downgrade.pdf` included in the docs directory to downgrade the firmware and power cycle the EVM.
This issue will be fixed in a future emulation pack update which will be available on the <a href="https://software-dl.ti.com/ccs/esd/documents/xdsdebugprobes/emu_xds_software_package_download.html" target="_blank">XDS Emulation Software (EMUPack) Download</a> page
]]
1. Hardware and Software Requirements
-----------
### Hardware
Item | Details
--------------------------|-----------------
Device | [IWR6843ISK](http://www.ti.com/tool/IWR6843ISK) or [IWR6843ISK-ODS](http://www.ti.com/tool/IWR6843ISK-ODS) or [IWR6843AOP](http://www.ti.com/tool/IWR6843AOPEVM) optionally with [Industrial mmWave Carrier Board](http://www.ti.com/tool/MMWAVEICBOOST).
Mounting Hardware | The EVM needs to be mounted at a height of ~1.5-2.5m with a slight downtilt. An [adjustable clamp style smartphone adapter mount for tripods](https://www.amazon.com/Vastar-Universal-Smartphone-Horizontal-Adjustable/dp/B01L3B5PBI/) and a [60-75" tripod](https://www.amazon.com/Neewer-Portable-centimeters-Camcorder-kilograms/dp/B01N6JCW8F/) can be used to clamp and elevate the EVM. This is only an example solution for mounting; other methods can be used so far as setup specifications are met.
Computer | PC with Windows 7 or 10. If a laptop is used, please use the 'High Performance' power plan in Windows.
Micro USB Cable | Due to the high mounting height of the EVM, an 8ft+ cable or USB extension cable is recommended.
Power Supply | 5V, >3.0A with 2.1-mm barrel jack (center positive). The power supply can be wall adapter style or a battery pack with a USB to barrel jack cable.
### Software
Tool | Version | Download Link
----------------------------|---------------------------|--------
TI mmWave SDK | 3.5.x.x | [Link to Latest mmWave SDK](http://software-dl.ti.com/ra-processors/esd/MMWAVE-SDK/latest/index_FDS.html). To access a previous version of the mmWave SDK scroll to the bottom of the table and click the link under "MMWAVE-SDK previous release". Repeat to continue stepping back to previous versions.
mmWave Industrial Toolbox | Latest | Download and install the toolbox. Go to [Using TI Resource Explorer & the mmWave Industrial Toolbox](../../../../docs/readme.html) for instructions.
Uniflash | Latest | Uniflash tool is used for flashing TI mmWave Radar devices. [Download offline tool](http://www.ti.com/tool/UNIFLASH) or use the [Cloud version](https://dev.ti.com/uniflash/#!/)
<a name="flash_the_evm"></a>
2. Flash the EVM
-----------
1. Set the device in flashing mode:
a. If using ISK standalone module, follow the instructions for [setting Modular EVM to flashing mode](../../../common/docs/hardware_setup/hw_setup_isk_ods_modular_mode_flashing.html)
b. If using ISK with ICBOOST, follow the instructions for [Hardware Setup of ICB for Flashing Mode](../../../common/docs/hardware_setup/hw_setup_mmwaveicboost_mode_flashing.html)
2. Follow the instruction to [Flash the mmWave Device](../../../common/docs/software_setup/using_uniflash_with_mmwave.html)
Image | Location
--------------------------|------------
Meta Image 1/RadarSS | `C:\ti\<mmwave_industrial_toolbox_install_dir>\labs\people_counting\68xx_3D_people_counting\prebuilt_binaries\3D_people_count_68xx_demo.bin`
3. Physical Setup
-----------
1. Setup the device for functional mode
a. If using ISK standalone, follow the instructions for [setting Modular EVM to functional mode](../../../common/docs/hardware_setup/hw_setup_isk_ods_modular_mode_functional.html)
b. If using the ICBOOST, follow the instructions for [Hardware Setup of ICB for Functional Mode](../../../common/docs/hardware_setup/hw_setup_mmwaveicboost_mode_functional.html)
2. For best results, the EVM should be positioned high enough to be above the top of tracked objects and with a slight down tilt.
The aim is to position the EVM so that the antenna beam can encompass the area of interest.
If the down tilt is too severe, noise from ground clutter would increase and the effective sensing area would decrease.
If threre is no down tilt, counting performance would be worse for cases in which one person is in line with and shielded by another person.
Given the antenna radiation pattern of the EVM, consideration should be taken to not mount the EVM too close or oriented with beam directed to the ceiling as this can increase the noise floor and result in less optimal performance.
<img src="images/downtilt.jpg" width="700"/>
**Setup Requirements:**
* Elevate EVM: 1.5-2.5m high
* Down tilt: ~10-15 degree
**Setup using suggested tripod and smartphone clamp mount:**
1. Screw on clamp mount to tripod
2. Clamp EVM across its width below power barrel jack to attach EVM
3. Adjust tripod head for ~10 degree down tilt (Tip: Bubble or level smartphone apps can be used to measure down tilt)
4. Plug in micro-usb and power supply to EVM
5. Extend tripod so that the EVM is elevated 1.5-2.5m from the ground
6. Position EVM and tripod assembly in desired location of room. The EVM should be positioned so that the 120 degree FOV of the EVM antenna encompasses the area of interest and points to the region in which people are expected to enter the space.
<img src="images/mounting_setup.png" width="600"/>
<a name="RuntheLab"></a>
4. Run the Lab
-----------
To run the lab, launch and configure the visualizer which displays the detection and tracked object data received via UART. See the [instructions in the visualizer folder](../../visualizer/docs/ba_visualizer_user_guide.html)
Please ensure you use the default chirp for your device:
* ISK: ISK_6m_default.cfg or ISK_14m_extended.cfg
* ISK_ODS: ODS_6m_default.cfg
* AOP: AOP_6m_default.cfg
[[y! Device must be restarted before sending a new configuration.
]]
Chirp Parameter (Units) | Value
-------------------------|------------
Start Frequency (GHz) | 60.6
Slope (MHz/us) | 54.725
Samples per chirp | 96
Chirps per frame | 288
Frame duration (ms) | 50
Sampling rate (Msps) | 2.950
Bandwidth (MHz) | 2249
Range resolution (m) | 0.084
Max Unambiguous Range (m)| 7.2
Max Radial Velocity (m/s)| 8.38
Velocity resolution (m/s)| 0.17
Azimuth resolution (deg) | 14.5
Elevation resolution (deg) | 58
Number of Rx | 4
Number of Tx | 3
Developer's Guide
===========
Build the Firmware from Source Code
-----------
<a name='reqs'></a>
### 1. Software Requirements
Tool | Version | Download Link
----------------------------|---------------------------|--------------
TI mmWave SDK | 3.5.x.x | [Link to Latest mmWave SDK](http://software-dl.ti.com/ra-processors/esd/MMWAVE-SDK/latest/index_FDS.html). To access a previous version of the mmWave SDK scroll to the bottom of the table and click the link under "MMWAVE-SDK previous release". Repeat to continue stepping back to previous versions.
mmWave Industrial Toolbox | Latest | Download and install the toolbox. Go to [Using TI Resource Explorer & the mmWave Industrial Toolbox](../../../../docs/readme.html) for instructions.
Code Composer Studio | 8.3.1 | [Code Composer Studio v8](http://processors.wiki.ti.com/index.php/Download_CCS#Code_Composer_Studio_Version_8_Downloads)
### 2. Import Lab Project
For the People Counting lab, there are two projects, the DSS for the C674x DSP core and the MSS project for the R4F core, that need to be imported to CCS and compiled to generate firmware for the xWR6843.
[[b! Project Workspace
When importing projects to a workspace, a copy is created in the workspace. All modifications will only be implemented for the workspace copy. The original project downloaded in mmWave Industrial Toolbox is not touched.
]]
1. Start CCS and setup workspace as desired.
2. Import the project(s) specified below to CCS from the src/ folder. See instructions for importing [here](../../../../docs/readme.html#import-ccs-projects-from-the-mmwave-industrial-toolbox-into-code-composer-studio).
* **3D_people_count_68xx_dss**
* **3D_people_count_68xx_mss**
3. Verify that the import occurred without error: in CCS Project Explorer, both **3D_pplcount_68xx_mss** and **3D_pplcount_68xx_dss** should appear.
### 3. Build the Lab
The DSS project must be built before the MSS project.
1. Select the **3D_people_count_68xx_dss** so it is highlighted. Right click on the project and select **Rebuild Project**. The DSS project will build.
2. Select the **3D_people_count_68xx_mss** so it is highlighted. Right click on the project and select **Rebuild Project**. The MSS project will build, the the lab binary will be constructed automatically.
2. On successful build, the following should appear:
* In 3D_people_count_68xx_dss → Debug, **3D_people_count_68xx_dss.xe674** (this is the C67x binary used for CCS debug mode)
* In 3D_people_count_68xx_mss → Debug, **3D_people_count_68xx_mss.xer4f** (this is the Cortex R4F binary used for CCS debug mode) and **3D_people_count_68xx_demo.bin** (this is the flashable binary used for deployment mode)
{{y Selecting Rebuild instead of Build ensures that the project is always re-compiled. This is especially important in case the previous build failed with errors.}}
[[r! Build Fails with Errors
If the build fails with errors, please ensure that all the software requirements are installed as listed above and in the mmWave SDK release notes.
]]
[[b! Note
As mentioned in the [Quickstart](#quickstart) section, pre-built binary files, both debug and deployment binaries are provided in the pre-compiled directory of the lab.
]]
### 4. Execute the Lab
There are two ways to execute the compiled code on the EVM:
* Deployment mode: the EVM boots autonomously from flash and starts running the bin image
* Using Uniflash, flash the **PC_lab_68xx.bin** found at `<PROJECT_WORKSPACE_DIR>\PC_mss_68xx\Debug\PC_lab_68xx.bin`
* The same procedure for flashing can be use as detailed in the Quickstart [Flash the EVM](#flash_the_evm) section.
* Debug mode: Follow the instructions for [Using CCS Debug for Development](../../../common/docs/software_setup/using_ccs_debug.html)
After executing the lab using either method, the lab can be visualized following the steps in [Run The Lab](#RuntheLab).
<a name="configuration commands"></a>
Modifying Configuration File
-----------
{{y The current configuration files for both ISK and ODS have been tuned for the best performance. Sometimes, it may be beneficial to tune, for example, if you are changing the max range or other chirp parameter. }}
The configuration files included with the lab include commands that are standard to the mmWave SDK and commands unique to this lab. Below are 4 commands which are related to the location of the device and must be configured. These can be controlled through the GUI without modifying the cfg; these are listed here for your convienence. Otherwise, the commands can be found in supporting documents:
1. Black - SDK User's Guide
2. Green and Red - 3DPeoplecountingDemo_ConfigurationDetails.pdf in the docs folder of this lab
3. Blue - pplcount_customization_guide.pdf in the docs folder of this lab
<img src='images/configExample.jpg' width="600"/>
* staticBoundaryBox [-X] [X] [NearY] [FarY] [-Z] [Z]
* This sets boundaries where static points can be used by the tracker and tracks are allowed to become static. Each value denotes an edge of the 3D cube. Currently, it is recommend to keep NearY greater than or equal to 2.
* boundaryBox [-X] [X] [NearY] [FarY] [-Z] [Z]
* This sets boundaries where tracks can exists. Only points inside the box will be used by the tracker. Each value denotes an edge of the 3D cube.
<img src='images/evm_orientation.png' width="600"/>
* sensorPosition [Z] [AzimuthTilt] [ElevationTilt]
* Z - height of sensor.
* AzimuthTilt - horizontal rotation of sensor with respect to the back wall, as represented by FarY.
* ElevationTilt - vertical rotation of the sensor with respect the the ground.
* trackingCfg [TrackerEnabled] [DefaultConfiguration] [MaxPoints] [MaxTracks] [MaxRadialVelocity] [RadialVelocityResolution] [FrameTime]
* TrackerEnabled - enables the tracker with value 1
* DefaultConfiguration - Do not change Default configuration
* MaxPoints - max points for tracker to consider (does not affect signal chain)
* MaxTracks - max tracks tracker can allocate (does not affect signal chain)
* MaxRadialVelocity - maximum doppler value from signal chain (does not affect signal chain)
* RadialVelocityResolution - doppler resolution of signal chain - does not affect signal chain)
* FrameTime - Ensure the frametime matches the frame period of the frameCfg command.
Data Formats
-----------
A TLV(type-length-value) encoding scheme is used with little endian byte order. For every frame, a packet is sent consisting of a fixed sized **Frame Header** and then a variable number of TLVs depending on what was detected in that scene. The TLVs can be of types representing the 2D point cloud, target list object, and associated points.
<img src="images/packet_structure.png" width="600"/>
### Frame Header
Size: 48 bytes
```Matlab
frameHeaderStructType = struct(...
'sync', {'uint64', 8}, ... % syncPattern in hex is: '02 01 04 03 06 05 08 07'
'version', {'uint32', 4}, ... % 0xA6843
'totalPacketLen', {'uint32', 4}, ... % See description below
'platform', {'uint32', 4}, ... % 600MHz free running clocks
'frameNumber', {'uint32', 4}, ... % In bytes, including header
'subFrameNumber', {'uint32', 4}, ... % Starting from 1
'chirpProcessingMargin', {'uint32', 4}, ... % Chirp Processing margin, in ms
'frameProcessingMargin', {'uint32', 4}, ... % Frame Processing margin, in ms
'trackProcessTime', {'uint32', 4}, ... % Tracking Processing time, in ms
'uartSentTime' , {'uint32', 4}, ... % Time spent to send data, in ms
'numTLVs' , {'uint16', 2}, ... % Number of TLVs in thins frame
'checksum', {'uint16', 2}); % Header checksum
```**Frame Header Structure: name, type, length**
```Matlab
% Input: frameheader is a 52x1 double array, each index represents a byte of the frame header
% Output: CS is checksum indicator. If CS is 0, checksum is valid.
function CS = validateChecksum(frameheader)
h = typecast(uint8(header),'uint16');
a = uint32(sum(h));
b = uint16(sum(typecast(a,'uint16')));
CS = uint16(bitcmp(b));
end
```**validateChecksum(frameheader) in MATLAB syntax**
### TLVs
The TLVs can be of type **POINT_CLOUD_2D**, **TARGET_LIST_2D**, or **TARGET_INDEX**.
#### **TLV Header**
Size: 8 bytes
```Matlab
% TLV Type: 06 = Point cloud, 07 = Target object list, 08 = Target index
tlvHeaderStruct = struct(...
'type', {'uint32', 4}, ... % TLV object
'length', {'uint32', 4}); % TLV object Length, in bytes, including TLV header
```**TLV header**
Following the header, is the the TLV-type specific payload
#### **Point Cloud TLV**
Type: POINT_CLOUD_2D
Size: sizeof (tlvHeaderStruct) + sizeof(pointUnit) + sizeof (pointStruct) x numberOfPoints
Each Point Cloud TLV consists of an array of points. Each point is defined in 8 bytes. The pointUnit struct is used to uncompress the each point to five floats (20 bytes).
```java
pointUnit = struct(...
'elevationUnit', {'float', 4}, ... % Multiply each point by this value - used for compression
'azimuthUnit', {'float', 4}, ... % Multiply each point by this value - used for compression
'dopplerUnit', {'float', 4}, ... % Multiply each point by this value - used for compression
'rangeUnit', {'float', 4}, ... % Multiply each point by this value - used for compression
'snrUnit', {'float', 4}); % Multiply each point by this value - used for compression
```**Point Structure**
```java
pointStruct = struct(...
'elevation', {'int8_t', 1}, ... % Elevation in radians
'azimuth', {'int8_t', 1}, ... % Azimuth, in radians
'doppler', {'int16_t', 2}, ... % Doppler, in m/s
'range', {'uint16_t', 2}, ... % Range, in meters
'snr', {'uint16_t', 2}); % SNR, ratio
```**Point Structure**
#### **Target Object TLV**
Type: TARGET_LIST_3D
Size: sizeof (tlvHeaderStruct) + sizeof (targetStruct2D) x numberOfTargets
Each Target List TLV consists of an array of targets. Each target is defined in 40 bytes.
```java
targetStruct2D = struct(...
'tid', {'uint32', 4}, ... % Track ID
'posX', {'float', 4}, ... % Target position in X dimension, m
'posY', {'float', 4}, ... % Target position in Y dimension, m
'velX', {'float', 4}, ... % Target velocity in X dimension, m/s
'velY', {'float', 4}, ... % Target velocity in Y dimension, m/s
'accX', {'float', 4}, ... % Target acceleration in X dimension, m/s2
'accY', {'float', 4}, ... % Target acceleration in Y dimension, m/s
'posZ', {'float', 4}, ... % Target position in Y dimension, m
'velZ', {'float', 4}, ... % Target velocity in Y dimension, m/s
'accZ', {'float', 4}, ... % Target acceleration in Y dimension, m/s
```**Target Structure**
#### **Target Index TLV**
Type: TARGET_INDEX
Size: sizeof (tlvHeaderStruct) + numberOfPoints (NOTE: here the number of points are for frame n-1)
Each Target List TLV consists of an array of target IDs. A targetID at index ***i*** is the target to which point ***i*** of the previous frame's point cloud was associated.
Valid IDs range from 0-249.
```java
targetIndex = struct(...
'targetID', {'uint8', 1}); % Track ID
```**Target ID Structure**
Other Target ID values:
Value | Meaning
------------|-----------
253 | Point not associated, SNR too weak
254 | Point not associated, located outside boundary of interest
255 | Point not associated, considered as noise
Customization
-----------
#### **Tuning Guide**
* Please refer to the **People Counting Demo Customization Guide** which can be found at `C:\ti\<mmwave_industrial_toolbox_install_dir>\labs\people_counting\68xx_people_counting\docs\pplcount_customization_guide.pdf`
Need More Help?
===========
* Find answers to common questions on <a href="https://e2e.ti.com/support/sensor/mmwave_sensors/w/wiki" target="_blank">mmWave E2E FAQ</a>
* Search for your issue or post a new question on the <a href="https://e2e.ti.com/support/sensor/mmwave_sensors/f/1023" target="_blank">mmWave E2E forum</a>
* See the SDK for more documentation on various algorithms used in this demo. Start at <MMWAVE_SDK_DIRECTORY>/docs/mmwave_sdk_module_documentation.html>