A modular Python project for autonomous driving research and prototyping, fully integrated with the BeamNG.tech simulator and Foxglove visualization. This system combines traditional computer vision algorithms and deep learning (CNN, UFLD) with real-time sensor fusion and autonomous vehicle control to tackle:
- Multi-Lane Detection: UFLD, Traditional CV
- Traffic Sign: Classification & Detection
- Traffic Lights: Classification & Detection
- Multi-Class Object Detection: Vehicles, pedestrians, cyclists and more
- Multi-Sensor Fusion: Camera, Lidar, Radar, GPS, IMU
- Real-Time Control: PID steering, cruise control (CC), automatic emergency braking (AEB)
- Visualization: Real-time monitoring with Foxglove WebSocket + multiple CV windows
- Configuration System: YAML-based modular settings
Looking Ahead: While VisionPilot currently runs on BeamNG.tech, integration with the CARLA Simulator is planned on our roadmap. If you are interested in helping build the CARLA bridge, PRs are welcome and much appreciated!
- VisionPilot: Autonomous Driving Simulation, Computer Vision & Real-Time Perception (BeamNG.tech)
- Overview
- Table of Contents
- Usage
- Demos
- Multi-Lane Detection Stress Testing
- Emergency Braking (AEB)
- Blind Spot Detection (BSD)
- Sign Detection & Classification
- Traffic Light Detection & Classification
- Lane Detection & Keeping (v2)
- Previous Lane Detection & Keeping (v1)
- Ultra-Fast Lane Detection (UFLD)
- Foxglove Visualization
- Multi Camera Scene Segmentation
- Sensor Suite
First, download and install BeamNG.tech from the official website. This is required to run the simulation environment.
Navigate to the config directory and update the path to your BeamNG.tech installation in the configuration files.
Configuration Files Overview:
| File | Purpose |
|---|---|
| beamng.yaml | Main configuration file containing the path to your BeamNG.tech installation. Required for basic setup |
| config.py | Python configuration module that loads and manages all YAML settings |
| control.yaml | Steering, throttle, and braking control parameters (PID tuning) |
| perception.yaml | Computer vision pipeline settings (lane detection, object detection thresholds) |
| scenarios.yaml | Simulation scenario definitions and test environments |
| sensors.yaml | Sensor configuration (camera, LiDAR, radar parameters) |
Update beamng.yaml with your BeamNG.tech installation path. For basic usage, this is the only essential configuration file you need to modify. Advanced users can also customize sensors.yaml to adjust sensor parameters.
Download the pretrained models for object detection, traffic light detection, traffic sign detection, and classification from the releases page. Place all models in the models root folder with the following structure:
Note: A first release will come soon with pretrained models. For now, you can train your own models or use publicly available weights.
models/
├── object_detection/
│ └── object_detection.pt
├── traffic_light/
│ └── traffic_light_detection.pt
├── traffic_sign/
│ ├── traffic_sign_detection.pt
│ └── traffic_sign_classification.h5
└── ufld/
└── culane_res18.pth
Verify that the model loading section in your main script matches your model directory structure:
print("[Main] Loading local models...")
local_models = {}
local_models['vehicle'] = YOLO('models/object_detection/object_detection.pt')
local_models['traffic_light'] = YOLO('models/traffic_light/traffic_light_detection.pt')
local_models['sign_detect'] = YOLO('models/traffic_sign/traffic_sign_detection.pt')
local_models['sign_classify'] = load_model('models/traffic_sign/traffic_sign_classification.h5')Update these paths if your model structure differs from the default.
The Ultra-Fast Lane Detection (UFLD) model requires additional configuration. Open src/perception/lane_detection/main.py and verify the model paths:
model_path = MODELS_DIR / "ufld" / "culane_res18.pth"
config_path = PROJECT_ROOT / "ufldv2" / "configs" / "culane_res18.py"Ensure both the model weights (culane_res18.pth) and configuration file (culane_res18.py) are in the correct locations. Download the UFLD model weights and place them in models/ufld/.
You can obtain the UFLD configuration files by:
Cloning the entire UFLD repository to root (Recommended):
git clone https://github.com/cfzd/Ultra-Fast-Lane-Detection-v2.git ufldv2This clones the repository into a ufldv2/ folder in your project root, making the config files automatically accessible.
Navigate to the scripts directory and run the startup script:
Linux/macOS:
cd scripts
./start_simulation.shWindows Powershell:
cd scripts
start_simulation.batThe simulation will initialize BeamNG.tech, load the perception models, and begin streaming sensor data and AI predictions in real-time.
Evaluation of the multi-lane perception pipeline across various environmental edge cases, including high-glare transitions, low-light tunnels, and heavy atmospheric fog:
Extended Demo: Watch the full video here
Watch the Emergency Braking System (AEB) in action with real-time radar filtering and collision avoidance:
Extended Demo: Watch the full video here
See the Blind Spot Detection (BSD) system in action using radar data to identify vehicles in the blind spot:
Extended Demo: Watch the full video here
This demo shows real-time traffic sign detection and classification:
Extended Demo: Watch the full video here
VisionPilot does not yet support multi-camera. This is for demonstration purposes only.
This demo shows real-time traffic light detection and classification:
No extended Demo avaliable yet.
Watch the improved autonomous lane keeping demo (v2) in BeamNG.tech, featuring smoother fused CV+SCNN lane detection, stable PID steering, and robust cruise control:
Extended Demo: Watch the full video here
Note: Very low-light (tunnel) scenarios are not yet supported.
The original demo is still available for reference:
Lane Keeping & Multi-Model Detection Demo (v1)
Watch the UFLD perform real-time lane detection with temporal spline smoothing on highway video.
Extended Demo: Watch the full video here
Note: Because UFLDv2 operates as an internal feature module in VisionPilot's multi-feature voting pipeline, this standalone demo highlights the underlying model's perception capabilities before its output is merged into the final pixel voting matrix.
See real-time LiDAR point cloud streaming and autonomous vehicle telemetry in Foxglove Studio:
Extended Demo: Watch the full video here
See real-time image segmentation using front and rear cameras:
Extended Demo: Watch the full video here
More demo videos and visualizations will be added as features are completed.
The vehicle is equipped with a comprehensive multi-sensor suite for autonomous perception and control:
| Sensor | Specification | Purpose |
|---|---|---|
| Front Camera | 1920x1080 @ 50Hz, 70° FOV, Depth enabled | Lane detection, traffic signs, traffic lights, object detection |
| LiDAR (Top) | 80 vertical lines, 360° horizontal, 120m range, 20Hz | Obstacle detection, 3D scene understanding |
| Front Radar | 200m range, 128×64 bins, 50Hz | Collision avoidance, adaptive cruise control |
| Rear Left & Right Radar | 30m range, 64×32 bins, 50Hz | Blindspot monitoring, rear object detection |
| Dual GPS | Front & rear positioning @ 50Hz | Localization |
| IMU | 100Hz update rate | Vehicle dynamics, pose estimation |
 |
 |
 |
| Sensor Array | Front Radar | Lidar Visualization |
Configuration files are located in the
/configdirectory:
-
2D Object & Scene Detection
- Sign classification & Detection (CNN / YOLO)
- Traffic light classification & Detection (CNN / YOLO)
- Multi-class object detection (Cars, Trucks, Buses, Pedestrians, Cyclists)
- Road Marking Detection (Arrows, Crosswalks, Stop Lines)
-
3D Perception & Spatial Estimation
- Speed Estimation using detection from camera and lidar
- Lidar Object Detection
- 💤 Multi Camera Setup (Will implement after all other camera-based features are finished)
- Multi-Object Tracking (MOT)
-
Lane & Drivable Area Segmentation
- Lane detection Fusion (UFLD / CV)
- 🔥 Ultra Fast Lane Detection (UFLD) integration
- Traditional CV Lane Detection (with Majority Voting & Lighting condition Detection)
- Improve voting system and add additional features
- Lighting Condition Detection
- Detect multiple lanes
- 🔥 Handle dashed lines better in lane detection
-
Sensor Hardware Integration
- Integrate Radar
- Integrate Lidar
- Integrate GPS
- Integrate IMU
- 💤 Ultrasonic Sensor Integration
-
State Estimation & Mapping
- Kalman Filtering (Standard & Extended)
- 💤 SLAM (simultaneous localization and mapping)
- Build HD Map of the BeamNG.tech map
- Localize Vehicle on HD Map
-
Low Level Motion Control
- Vehicle Control integration (Throttle, Steering, Braking)
- Integrate PIDF controller for steering and speed control
- Improve PIDF controller tuning
- 💤 Model Predictive Control (MPC) for more advanced control strategies
-
Safety & Driving Assist
- Adaptive Cruise Control (ACC)
- Cruise Control (CC)
- Automatic Emergency Braking (AEB)
- 🔥 Blind Spot Monitoring (BSD)
- Dynamic Target Speed
- Curve Speed Optimization
- Adaptive Cruise Control (ACC)
-
Tactical & Behavior Planning
- Behavior Tree Architecture (Stop, Yield, Lane Change, Overtake)
- Traffic Rule Enforcement (Stop at red lights, stop signs, yield signs)
- 🔥 Lane Change Logic (Check Blindspot, Signal, Execute)
- Obstacle Avoidance (Depends on Behavior Tree)
- Parking Logic (Parallel / Perpendicular Path Finding)
-
Trajectory & Path Planning
- Frenet Frame Transformation
- Global Path Planning
- Local Path Planning
- Trajectory Prediction (Surrounding Vehicle Intent)
- Integrate and test in BeamNG.tech simulation
- Modularize and clean up BeamNG.tech pipeline
- CARLA Simulator Integration (Planned / Help Wanted)
Note: CARLA support is planned for multi-simulator testing, but active development hasn't started yet. PRs and community contributions are very welcome!
- Environmental Conditions (Fog, Night, Dawn/Dusk, Tunnels/Low-Light)
- Traffic scenarios (Light, Moderate, Heavy)
- 💤 Physical RC Deployment
- ⭐ Full Foxglove visualization integration (Overhaul needed)
- Modular YAML configuration system
- Real-time drive logging and telemetry
Note: Considering moving away from Foxglove entirely to build a custom dashboard. Not a priority at this time.
- Spatial & Path Visualization
- 🔥 Birds-Eye View (BEV)
- Inverse Perspective Mapping (IPM)
- Map & Real Time Perception Overlay
- Trajectory & Path Plan Overlays in Foxglove
- 💤 Microservices Architecture
- Containerize models with Docker
- Aggregator service for concurrent inference orchestration
- Message Broker (Redis)
- Vibe-Code a website for the project
- Redo project structure for better modularity
- README Demo Media
- Performance Benchmarks
- Documentation
Driver Monitoring would've been pretty cool but human drivers are not implemented in BeamNG.tech or Carla
🔥 High Priority
⭐ Refining / In Progress (Working baseline, needs tuning and improvements)
💤 Backlog / Postponed (Nice to have, deferred)
Status: This project is currently in active development. A stable, production-ready release with pre-trained models and complete documentation will be available eventually.
- Simulator Support: Currently only validated in BeamNG.tech. CARLA simulator integration is planned, but not yet implemented.
- Tunnel/Low-Light Scenarios: Camera perception fails below certain lighting thresholds
- Multi-Camera Support: Single front-facing camera only (future roadmap)
- PID Controller Tuning: May oscillate on tight curves
- Real-World Testing: Only validated in simulation (BeamNG.tech), for now...
Datasets:
- CU Lane, LISA, GTSRB, Mapillary, BDD100K
Simulation & Tools:
- BeamNG.tech by BeamNG GmbH
- Foxglove Studio for visualization
- Docker & Docker Compose for containerization
Special Thanks:
- Kaggle for free GPU resources (Model Training)
- Mr. Pratt (Teacher/Supervisor) for guidance
Academic Papers & Research:
Ultra Fast Deep Lane Detection v2
@ARTICLE{qin2022ultrav2,
author={Qin, Zequn and Zhang, Pengyi and Li, Xi},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
title={Ultra Fast Deep Lane Detection With Hybrid Anchor Driven Ordinal Classification},
year={2022},
doi={10.1109/TPAMI.2022.3182097}
}If you use VisionPilot in your project, please cite:
@software{visionpilot2026,
title={VisionPilot: Autonomous Driving Simulation, Computer Vision & Real-Time Perception},
author={Julian Stamm},
year={2026},
url={https://github.com/visionpilot-project/VisionPilot}
}Title: BeamNG.tech
Author: BeamNG GmbH
Address: Bremen, Germany
Year: 2025
Version: 0.35.0.0
URL: https://www.beamng.tech/
This project is licensed under the MIT License - see LICENSE file for details.