Background
Three-dimensional reconstruction from ordinary video is challenging because object detection, depth estimation, camera geometry, and multi-view alignment must operate consistently despite changes in viewpoint, image quality, lighting, and scene depth.
As part of a three-person graduation project, I worked on a computer-vision pipeline for reconstructing vehicles from multiple video viewpoints in an Intelligent Transportation Systems (ITS) setting.
Rather than relying on a single reconstruction technique, the project combined pre-trained computer-vision models with geometric methods to investigate whether partial 3D representations of vehicles could be produced from relatively short video sequences.
Research Goals
- Detect vehicles across video frames.
- Estimate scene depth from monocular image data.
- Calibrate camera geometry and relate image coordinates to three-dimensional structure.
- Identify relevant visual features within detected vehicle regions.
- Construct point-cloud representations from video frames.
- Align reconstructions obtained from different viewpoints to reduce geometric inconsistencies.
- Identify practical limitations arising from the interaction between learned vision models and geometric reconstruction methods.
Dataset
Because a suitable public dataset containing the same scene from multiple viewpoints was not available for the project, the team collected its own visual data.
A relatively small dataset based on approximately 30 seconds of video was used because photogrammetry and depth-processing operations were computationally demanding.
The project also had to account for:
- camera movement
- varying image and lighting conditions
- limited video quality
- computational requirements
- operating-system and software compatibility
- privacy considerations associated with real-world video collection
Methodology
Vehicle Detection
The initial implementation explored YOLOv3, but its configuration and binary requirements created practical difficulties in the available development environment.
The pipeline was therefore migrated to YOLOv5, loaded through PyTorch Hub.
Predictions were filtered to retain relevant vehicle classes such as:
- cars
- trucks
- buses
This provided bounding boxes and object centres for later reconstruction stages.
Camera Calibration
Camera calibration was performed using a chessboard-based calibration procedure.
Intrinsic parameters described camera-specific properties such as focal length and lens distortion, while extrinsic parameters represented the camera's spatial relationship with the external scene.
Both OpenCV and MATLAB's Camera Calibration Toolbox were explored as part of the calibration workflow.
Depth Estimation
Several approaches to depth estimation were investigated.
Formula-based and disparity-based approaches were initially explored but did not provide sufficiently useful results for the project.
The pipeline therefore adopted MiDaS DPT_Large for monocular depth estimation.
Video sequences were decomposed into frames and processed individually to obtain depth maps. A reference object in the scene was subsequently used to scale and normalise depth values across frames.
MiDaS provided useful near-field estimates, but its reliability declined for more distant objects, exposing an important limitation of monocular depth estimation.
Feature Detection
The project initially considered SIFT for feature extraction but selected FAST because of its lower computational cost.
FAST keypoints were detected across successive frames and filtered using YOLOv5 bounding boxes so that reconstruction focused on features associated with detected vehicles rather than the entire scene.
3D Reconstruction
Depth information, detected vehicle regions, and visual features were combined to generate an intermediate point-cloud representation.
The initial point cloud was relatively low resolution but demonstrated that the different components of the pipeline could be integrated into a common 3D representation.
Multi-View Alignment
A major challenge emerged when point clouds generated from different camera viewpoints were combined directly.
Direct alignment produced substantial geometric inconsistencies.
The pipeline was therefore extended using explicit geometric transformations:
- rotation
- scaling
- translation
One point cloud was rotated by 180 degrees around the y-axis. Extreme coordinates were then identified along the x, y, and z dimensions, allowing three-dimensional bounding boxes to be constructed.
These bounds were used to scale and translate the point clouds into closer spatial alignment.
Results
The project produced an end-to-end experimental pipeline connecting:
vehicle detection → depth estimation → feature extraction → point-cloud reconstruction → multi-view geometric alignment
Several concrete results emerged during development.
Vehicle Detection
Moving from YOLOv3 to YOLOv5 simplified the implementation and produced usable vehicle detections across the collected video frames.
Depth Estimation
MiDaS generated depth maps that were useful for reconstructing nearby objects, although performance deteriorated as object distance increased.
This exposed depth estimation as one of the major reliability constraints in the overall reconstruction pipeline.
Initial 3D Reconstruction
Combining YOLOv5 detections, MiDaS depth estimates, and FAST keypoints produced an intermediate 3D point cloud of a vehicle.
The reconstruction remained low resolution, but it demonstrated that a partial three-dimensional representation could be generated from the available video data.
Multi-View Reconstruction
Directly combining point clouds from different viewpoints produced poor alignment.
Introducing rotation, scaling, and translation transformations reduced these geometric inconsistencies and produced more coherent combined point-cloud views.
The final results demonstrated a proof of concept for integrating learned visual models with classical geometric transformations for partial multi-view vehicle reconstruction, while also exposing the limitations of the approach.
Contributions
- Contributed to a year-long multi-camera computer-vision graduation project in an ITS setting.
- Integrated YOLOv5 vehicle detection into a broader reconstruction pipeline.
- Applied MiDaS DPT_Large for monocular depth estimation after alternative depth approaches proved inadequate.
- Used camera calibration to connect image-space information with three-dimensional geometry.
- Applied FAST feature detection and filtered keypoints using vehicle bounding boxes.
- Generated an intermediate 3D vehicle point cloud from video observations.
- Combined reconstructions from multiple viewpoints using rotation, scaling, translation, and 3D bounding-box information.
- Adapted the methodology when initial techniques produced inadequate results rather than retaining the original pipeline unchanged.
- Identified practical failure modes involving depth ambiguity, image quality, software compatibility, and point-cloud alignment.
Limitations
The final reconstruction remained a partial and relatively low-resolution representation rather than a production-ready 3D vehicle model.
Several limitations affected performance:
- MiDaS depth estimates became less reliable for distant objects.
- Video quality constrained the information available for reconstruction.
- Direct point-cloud correspondence across viewpoints was insufficient and required explicit geometric transformations.
- Processing requirements limited the amount of video that could be used.
- The system was not evaluated as a real-time traffic-management or vehicle-tracking deployment.
These limitations mean that the project should be interpreted as a proof-of-concept reconstruction pipeline, rather than evidence of a deployable ITS system.
Impact
The strongest outcome of the project was the integration of several computer-vision and geometric components into a single reconstruction pipeline, together with an analysis of how limitations in individual components propagated into the final output.
Reliable object detection alone was not sufficient to guarantee a reliable 3D reconstruction: the final result also depended on depth quality, camera geometry, feature extraction, and cross-view alignment.
The project therefore provided practical experience in system-level model integration, iterative methodology design, and failure analysis, where the quality of a final computational output depends on the interaction between several imperfect components.