
2021 – Present
This project aims to tackle the challenges associated with automatically analyzing videos from extensive camera networks in crowded environments. Existing computer vision algorithms struggle to understand typical human behaviors, identify specific people or objects, and recognize actions from various camera angles. The proposed solution is to create new computer vision algorithms that can learn typical patterns in videos based on time, spot specific people or objects across cameras, and recognize actions from different camera angles.
The project utilizes generative and synthetic data in several advanced modeling techniques to develop new computer vision algorithms. These techniques include memory-based models that recognize patterns in context, models that can identify objects from any camera angle, and models that can recognize actions based on movement patterns.
The project aims to apply its findings by integrating the developed algorithms into the camera networks to improve surveillance at crowded places like stadiums and malls. By adding advanced algorithms to existing camera networks, the system will be able to track people in real-time and spot unusual behavior. This technology will contribute to the Virtual Sentry Framework by providing important data to detect and respond to potential threats effectively and quickly.
The project improves upon current technologies by creating more reliable and precise models that can handle uncertainty, offer explanations for their decisions, and function effectively in crowded environments. Bias is reduced by concentrating on long-term data from wide-area camera networks and integrating non-visual data.
In 2024, the project improved action recognition; for example, a person picking up a bag on video can now be identified by the algorithm in video from different perspectives, including behind, side, and front camera views. This action recognition is easy for humans but challenging for computer software. The project improved the models’ capabilities to track human interactions and recognize actions in real-time. Additionally, the project seeks to develop a user-friendly interface for sharing results.
The project involves collaborations with stakeholders from SENTRY’s Stadium Security Case Study and Surface Transportation Case Study to test and refine algorithms in live environments. These partnerships provide access to real-world data, testing environments, and expert feedback to help transition the technology from research to practical application.