Image
Introduction
Modern developments in machine learning have led to increased emphasis on and use of advanced driver assistance systems (ADAS) and attempts to fully automate driving. In turn, pedestrian detection systems have become a common benchmark in computer vision, often used as a representative example of broader image detection, segmentation and classification approaches used throughout academia and industry alike.
In practical use, ADAS and autonomous driving systems must reason about pedestrian intent, not just location. The JAAD (Joint Attention for Autonomous Driving) dataset provides road-side dashcam videos with bounding boxes coupled with rich annotations describing pedestrian actions (crossing intention, gaze direction). This project utilizes the JAAD dataset, fine-tunes a pedestrian detector that emphasises near-field or occluded agents, and couples it with a spatio-temporal behaviour model so the final inference pipeline can draw tracks and intent overlays on arbitrary videos.
Running the case on an HPC cluster
First, the environment should be set up. In this example, python toolchain is managed by uv:
wget -qO- https://astral.sh/uv/install.sh | sh uv sync
Afterwards, the dataset (2.9 GB) should be downloaded and prepared (extracts frames with OpenCV):
uv run train.py prepare-data --data-root data --fps 10 --split default
The detector trains on JAAD frames using the Faster R-CNN model shipped with PyTorch as a base and finetuning it on the extracted frame images. The training process is customizable with various options:
uv run train.py train-detector --data-root data --fps 10 --split default --epochs 16 --batch-size 2 \ --workers 8 --pin-memory --cache-images --cache-format decoded --output-dir models/detector \ --freeze-epochs 2 --lr 0.005 --lr-scheduler cosine --warmup-epochs 1 --ema-decay 0.999 \ --second-stage-epochs 8 --second-stage-lr 0.001 --second-stage-freeze-backbone-epochs 1
The full tutorial is available in the Git repository
Results
Although the exact results can vary based on different training parameters chosen, an example case of the accuracies obtained with default parameters is provided below.
Detector performance (JAAD validation)
- ADAS mAP - 0.648
- COCO mAP - 0.480
Behaviour classifier performance
- Crossing accuracy - 0.794
- Looking accuracy - 0.866
More demo-cases, that similarly to this one demonstrate particular HPC solutions to different problems in a simplified yet practical way, are available in a Git repository created by NCC Latvia.