This project uses innovative passive and active pervasive electromagnetic sensing with radio frequency (RF) and IR signals from cell phones and IR imagers and distributed active millimeter wave (mm-wave) radars to detect concealed threats and enhance the situational awareness of STCPs. This will be done both directly using the mm-wave radar to sense specific metallic threats – such as rifles – and indirectly using the RF signals from cell phones and IR imagers to flag unusual crowd-gathering conditions. Useful real-time situational awareness data must be both sensitive to a wide variety of anomalous signatures and flexible to adapt to the needs of a specific location. The project will provide situational awareness that will be integrated into the Virtual Sentry framework.
The two sub-projects are 1) Passive IR/EM (PIREM) sensing that can augment crowd-sourced data, and 2) Active EM/millimeter wave radar (AEM) sensing of physical anomalies with video-cueing. Initially, the sensors will be assumed to be mounted on stationary platforms, such as streetlight poles and buildings, but will eventually be considered for unmanned road and airborne vehicle mounting. These two sensing modalities can fuse their information by having the RF sensor direct the radar to concentrate on finding concealed threats among specific groups of individuals who are behaving unusually. Both can be integrated into the Virtual Sentry framework to enable real-time decision support to stakeholders charged with protecting STCPs. Passive IR/EM Sensing (PIREM) Human behavior is no longer confined to the physical environment, nor solely on an online virtual environment, but a hybrid of the physical and virtual. Situational awareness, therefore, can be achieved by augmenting physical sensors (e.g., video analytics) with specific social media content. For example, other SENTRY projects are fusing social media posts combined with video surveillance to identify potential risks. However, social networking is becoming increasingly fractured and, in some cases, in private apps. This project, therefore, asks, “how can we develop sensing technology that allows us to identify the hybrid physical-digital characteristics of crowds and individuals to provide additional depth to real-time surveillance and situational awareness, even if the content is ‘hidden’ from the public?” This is done with respect for data privacy since it does not “read” any data; it solely measures the presence of RF emissions, not the content.
To accomplish this, we are developing a hybrid physical-digital sensing network compromised of low-cost, portable (drone or fixed infrastructure) nodes that not only acquire visible and IR (thermal images), but also provide a map of the real-time “electromagnetic situational awareness.” The sensing network monitors electromagnetic radio frequency (RF) energy produced by nearly any piece of electronics such as cell phones, computers, or vehicles (including drones). Each device has an electromagnetic “fingerprint,” and the characteristics of that fingerprint change as device usage changes. Monitoring these fingerprints provides insight into both individual-level behavior and crowd-level behavior. For example, a crowd that suddenly shifts to an increase in RF energy transmission indicates a rapid increase in video streaming, and possibly an event that requires attention (e.g., an outbreak of violence), or something benign. To further provide depth of data, we include visible and IR imaging. Additionally, with the increase of 5G usage in the US, we can leverage the high directionality of 5G transmissions to potentially pinpoint the precise location within a crowd where an event is occurring, even before it can be aggregated via social media (and without requiring triangulation of videos).
These systems require less processing power and human interaction than video surveillance. For example, RF monitoring of many locations across a city can find an increased human activity outside of the normal range, which then will alert human operators (or the Virtual Sentry) to give increased attention to certain locations. Active EM (millimeter wave radar) sensing Active EM (millimeter-wave radar sensing) (AEM): This aspect of the RB.2 project aims to create pervasive, inexpensive millimeter wave (mm-wave) radar arrays that can detect large metal objects that could potentially be weapons, concealed on persons at distance and in crowds. This will be done in conjunction with tracking video, taking advantage of the 75 penetrating aspects of mm-wave radar to detect specific metallic threats under clothing. The radar will make use of a novel multibeam antenna configuration that avoids most of the expensive phased array beam-forming components to keep costs comparable to video cameras. The feasibility of detecting body-worn threats at distance is established with full-wave electromagnetic modeling. Various sensor configurations are being tested for effective target discrimination using the same synthetic aperture radar imaging method to be ultimately used with the actual hardware.
