Audio-Visual System Demo

Demo Background

Audio-Visual systems enable applications like teleconferencing, presentation systems, and more. Techniques like keyword spotting and speech transcription help computers react to vocal commands, and computer vision enables computers to infer meaning from imagery. Combining these two types of sensor data enables a wide range of multimedia applications.

This demo shows a audio-visal system in which audio analyzed for specific keywords. These keywords control which areas the camera is focused on such that specific areas can be zoomed in on to better show a speaker - this keyword spotting uses a neural network running on the Arm CPUs. In tandem, a deep learning model runs on images from the camera to recognize where peoples' faces are using the C7xMMA deep learning accelerator. The faces are isolated and displayed on a section of the screen.

Source code is available on Texas Insturments github under the edgeai-demo-audio-visual repository.

How to get started

Please find the following resources for reproducing the demo. This requires:

1. Get an AM62A starter kit EVM, an IMX219 csi-2 camera module (raspberry Pi compatible), and aUSB microphone
2. Download the Edge AI Linux SDK
3. Load the Edge AI Linux SDK via an SD card using the quick start guide
4. Log into the EVM through a network connection using SSH
5. Clone the git repo for this demo onto the EVM (or copy all files to the SD card)
6. Follow instructions within the readme to setup audio packages, setup the IMX219 camera, and run the demo

These steps are validated on the 9.0 Edge AI Linux SDK.

General Edge AI and AM62A resources