bikeped: Real-Time Pedestrian-Cyclist Collision Warning

Infrastructure-side collision warning system for urban intersections. Runs on an NVIDIA Jetson Orin with a 200-degree fisheye camera, producing audible and visual alerts at 30 fps with 30 ms processing latency.

Technical Report: A Real-Time Bike-Pedestrian Safety System with Wide-Angle Perception and Evaluation Testbed for Urban Intersections (arXiv:2604.17046)

Quick Start

Requirements

• Python 3.10+
• numpy, scipy, matplotlib, pyyaml, opencv-python

pip install numpy scipy matplotlib pyyaml opencv-python

For the live system (mqtt_bridge.py): NVIDIA Jetson AGX Orin, TensorRT, ultralytics, paho-mqtt, busylight, play_sounds.

Reproduce Paper Results

Run all experiments with a single command:

python run_experiments.py                  # full pipeline (~45 min)
python run_experiments.py --skip-optimizer  # skip optimizer (~10 min)

Or run individual experiments:

# Table 5 (pipeline comparison, includes fisheye localization error)
python decision_testbench.py --compare

# Ablation without localization error
python decision_testbench.py --no-bbox-noise --compare

# Figure 4 (latency sweep) — generates paper/fig_camera_latency.pdf
python decision_testbench.py --latency-sweep

# Monte Carlo with size-aware detection failures
python decision_testbench.py --monte-carlo --mc-trials 50

# Optimizer (differential evolution, ~30 min)
python decision_testbench.py --optimize

# Height-pitch camera placement sweep (Figure 8b, ~30 min)
python run_height_pitch_sweep.py

# Bounding box localization error analysis
python decision_testbench.py --error-map

# Generate all publication figures
python generate_figures.py

# Render scenario simulation videos
python sim_visualizer.py --no-display

Verify BEV Projection Consistency

The testbench and the deployed system (mqtt_bridge.py) must produce identical ground-plane projections. The following verifies zero-error agreement across the full pixel grid:

python -c "
import numpy as np
from numpy import pi, sin, cos, sqrt

def mqtt_pixel_to_ground(fw, fh, ch, cp, fov, cx, cy, radius):
    dc = 2*radius
    yg, xg = np.meshgrid(np.arange(fh), np.arange(fw), indexing='ij')
    dx = xg - cx; dy = yg - cy
    r = np.sqrt(dx**2+dy**2); phi = np.arctan2(dy,dx)
    ifoc = dc*180/(fov*pi); theta = r/ifoc
    st,ct = np.sin(theta),np.cos(theta); sp,cp2 = np.sin(phi),np.cos(phi)
    rays = np.stack([ct,st*cp2,-st*sp],axis=-1)
    rl = np.linalg.norm(rays,axis=-1,keepdims=True); rays = rays/(rl+1e-10)
    pr = cp*pi/180; cpr,spr = cos(pr),sin(pr)
    rp = np.stack([rays[...,0]*cpr-rays[...,2]*spr,rays[...,1],
                   rays[...,0]*spr+rays[...,2]*cpr],axis=-1)
    rz = rp[...,2]; valid = rz<-1e-6
    t = np.zeros_like(rz); t[valid] = -ch/rz[valid]
    gc = np.zeros((fh,fw,3))
    for i in range(3): gc[...,i] = np.array([0,0,ch])[i]+t*rp[...,i]
    gc[~valid] = np.nan
    return gc, valid

from decision_testbench import FisheyeCamera
import json, yaml

with open('config.yaml') as f: cfg = yaml.safe_load(f)
cc = cfg.get('camera', {})
calib = json.load(open('camera_calibration.json'))
H=cc.get('height',3.6576); FOV=calib['fov_deg']; R=int(calib['radius_px'])
P=cc.get('pitch_deg',0.0); CX=calib['cx']; CY=calib['cy']
W=int(calib['frame_width']); HH=int(calib['frame_height'])

mqtt_g, mqtt_v = mqtt_pixel_to_ground(W,HH,H,P,FOV,CX,CY,R)
cam = FisheyeCamera(height=H, fov_deg=FOV, radius_px=R, frame_width=W,
    frame_height=HH, cam_x=5.0, cam_y=-13.0, pitch_deg=P,
    cx_override=CX, cy_override=CY)

diffs = []
for pv in range(0, HH, 50):
    for pu in range(0, W, 50):
        if not mqtt_v[pv,pu]: continue
        mx,my = mqtt_g[pv,pu,0], mqtt_g[pv,pu,1]
        md = sqrt(mx**2+my**2)
        tb_gnd, tv = cam.pixel_to_ground(np.array([pu,pv],dtype=np.float64))
        if not tv: continue
        td = sqrt((tb_gnd[0]-cam.cam_x)**2+(tb_gnd[1]-cam.cam_y)**2)
        diffs.append(abs(md-td))

diffs = np.array(diffs)
print(f'Compared {len(diffs)} points: max_diff={diffs.max():.1e} m')
assert diffs.max() < 1e-6, 'BEV MISMATCH'
print('PASS: testbench and mqtt_bridge BEV projections match exactly')
"

Perception Pipeline

The testbench and simulator replicate the physical capture and processing pipeline of the deployed system:

graph LR
    subgraph Physical Model
        A[Fisheye lens<br/>3500x3500 circle] --> B[Sensor captures<br/>3840x2160<br/>top/bottom clipped]
    end

    subgraph mqtt_bridge
        B --> C[Downsample<br/>1280x720]
        C --> D[YOLO detection<br/>1280x1280]
        D --> E[Scale bbox<br/>to 3840x2160]
        E --> F[CoordinateMapper<br/>pad 3840x3840<br/>crop 3500x3500]
        F --> G[Ground coord LUT<br/>equidistant model]
        G --> H[BEV position<br/>camera-relative]
    end

    subgraph Testbench / Simulator
        I[3D world position] --> J[FisheyeCamera<br/>world_to_pixel<br/>3500x3500]
        J --> K{On sensor?<br/>rows 670-2829}
        K -->|yes| L[projected_bbox_area<br/>→ YOLO input scale]
        K -->|no| M[Not visible]
        L --> N[BBox error<br/>bottom-center offset]
        N --> O[pixel_to_ground<br/>→ camera-relative]
    end

    H -.->|verified zero-error| O

The ground-plane projection in decision_testbench.py has been numerically verified against mqtt_bridge.py across 2310 grid points with zero error (see BEV verification command above).

Repository Structure

mqtt_bridge.py            Live system (detection, tracking, decision, MQTT)
decision_pipeline.py      Shared three-stage decision module
decision_testbench.py     Offline evaluation, optimization, Monte Carlo, latency sweep
sim_visualizer.py         Scenario visualization (renders MP4 videos)
generate_figures.py       Publication figure generation (EPS/PDF)
run_experiments.py        Reproduce all paper experiments in one command
run_height_pitch_sweep.py Height-pitch camera placement Monte Carlo sweep
eval_models.py            YOLO model evaluation on fisheye-augmented COCO
calibrate_fisheye.py      Fisheye lens calibration (checkerboard + bundle adjustment)
latency_report.py         Per-frame latency profiling
config.yaml               All runtime parameters
camera_calibration.json   Calibrated intrinsics (loaded automatically)
paper/                    LaTeX source

Decision Pipeline

Three stages evaluated in sequence:

1. Pedestrian presence — no pedestrian = IDLE.
2. Cyclist memory — no cyclist in last N frames = SAFE.
3. Pairwise closing check — for each bike-ped pair within [d_min, d_max], compare pairwise distance at frame t vs t-k using both agents' historical positions. Closing + cyclist moving = ALERT.

Selected parameters (optimizer-derived): N=58 frames, d=[1.9, 24.8] m, lookback k=2, speed-adaptive d_max. Braking model uses 85th-percentile field measurements (PRT=0.84 s, decel=1.96 m/s^2).

Conformance Scenarios

24 scripted scenarios covering: safe crossings (3), standard approaches (5), high-speed/accelerating (3), accessibility (2), multi-agent (3), edge cases (3), non-linear trajectories (4). Of 24, 21 contain ground-truth danger intervals. E-Scooter and Accelerating E-Bike scenarios use e-bicycle braking profile (decel=6.0 m/s^2).

Configuration

Runtime parameters in config.yaml. Decision parameters under decision: match the optimizer-derived values deployed on the Jetson.