AI for pedestrian detection, cross-frame tracking and intent classification

Introduction

Modern developments in machine learning have led to increased emphasis on and use of advanced driver assistance systems (ADAS) and attempts to fully automate driving. In turn, pedestrian detection systems have become a common benchmark in computer vision, often used as a representative example of broader image detection, segmentation and classification approaches used throughout academia and industry alike.

In practical use, ADAS and autonomous driving systems must reason about pedestrian intent, not just location. The JAAD (Joint Attention for Autonomous Driving) dataset provides road-side dashcam videos with bounding boxes coupled with rich annotations describing pedestrian actions (crossing intention, gaze direction). This project utilizes the JAAD dataset, fine-tunes a pedestrian detector that emphasises near-field or occluded agents, and couples it with a spatio-temporal behaviour model so the final inference pipeline can draw tracks and intent overlays on arbitrary videos.

The project has been realised by NCC Latvia.

Design choices and challenges

One of the main challenges was to effectively use the available memory and set up image cache on disk. Additional customisations for training efficiency include denser small-object anchors, widened ROI heads and multi-scale photometric/geometric augmentation. Other architecture and design choices were:

• Backbone - torchvision fasterrcnn_resnet50_fpn_v2 seeded with COCO weights; optional backbone LR multiplier lets the head learn faster than the trunk.
• Augmentation - multi-scale resizing (short side 512-832 px), flips, blur, grayscale, jitter, channel dropout, additive Gaussian noise, and optional strong-augmentation toggle.
• Optimisation - SGD with cosine or (multi)step schedulers, configurable warmup, AMP, gradient clipping, EMA smoothing, and two-stage fine-tuning to re-focus the ROI heads after the backbone stabilises.
• Data plumbing - DataLoader caching (decoded or encoded), per-worker decoded LRUs, pinned memory, and persistent workers mitigate disk bottlenecks on large nodes.
• Visual encoder - ResNet-18 trunk (ImageNet weights when certificates allow, otherwise the code falls back to random init and warns).
• Temporal head - bidirectional LSTM + dropout feeding two independent 3-way classifiers (crossing, looking). Loss weighting treats "unclear" as a soft class to prevent overconfident predictions.
• Sequences - crops aligned with tracker boxes, padded context, downsampled and normalised using ImageNet statistics. sequence_length and stride balance temporal coverage and GPU usage.

Running the case on an HPC cluster

First, the environment should be set up. In this example, python toolchain is managed by uv:

wget -qO- https://astral.sh/uv/install.sh | sh 
uv sync

Afterwards, the dataset (2.9 GB) should be downloaded and prepared (extracts frames with OpenCV):

uv run train.py prepare-data --data-root data --fps 10 --split default

The detector trains on JAAD frames using the Faster R-CNN model shipped with PyTorch as a base and finetuning it on the extracted frame images. The training process is customizable with various options:

uv run train.py train-detector --data-root data --fps 10 --split default --epochs 16 --batch-size 2 \
 --workers 8 --pin-memory --cache-images --cache-format decoded --output-dir models/detector \
 --freeze-epochs 2 --lr 0.005 --lr-scheduler cosine --warmup-epochs 1 --ema-decay 0.999 \
 --second-stage-epochs 8 --second-stage-lr 0.001 --second-stage-freeze-backbone-epochs 1

The full tutorial is available in the Git repository 

Results

Although the exact results can vary based on different training parameters chosen, an example case of the accuracies obtained with default parameters is provided below.

Detector performance (JAAD validation)

• ADAS mAP - 0.648
• COCO mAP - 0.480

Behaviour classifier performance

• Crossing accuracy - 0.794
• Looking accuracy - 0.866

The algorithm works for different light conditions and pedestrian numbers. Three examples of the detected pedestrians and their intents (are they crossing the road or not, looking at the car or not) are shown below.

Benefits of this demo-case

The main benefit of the code is the demonstration of a typical, industry-style workflow that includes all required steps from image extraction to classifier training. The demo-case can serve as

• an illustrative example which attracts SMEs and other companies to HPC applications;
• a base for a similar project of image recognition, segmentation or classification.

More demo-cases, that similarly to this one demonstrate particular HPC solutions to different problems in a simplified yet practical way, are available in a Git repository created by NCC Latvia.