Enhancing Synthetic Data Realism for Autonomous Vehicles Using Segmentation-Guided ControlNet

Deep learning models require high-quality and domain-specific datasets, but collecting real-world data is costly and challenging in many fields, including autonomous vehicles. Synthetic data generated via 3D modelling offers a scalable alternative, but applicability to real-world data may be hindered by limited realism. 3D-modelled datasets provide accurate ground-truth data, and we demonstrate these can be used for conditioning diffusion models as an effective alternative to approximate segmentations and depths from pre-trained models. To enhance synthetic data realism, we fine-tuned a Stable Diffusion model with Low-Rank Adaptation (LoRA) on the KITTI autonomous vehicle dataset and used ControlNet for realistic image generation. Stable Diffusion is commonly used for image generation, while LoRA fine-tunes pre-trained models efficiently. A key contribution is ensuring semantic consistency between generated and source images using two ControlNet annotators (edges and segmentation maps) along with accurate segmentation maps from the Virtual-KITTI dataset to preserve structural accuracy. Text-guided prompts were used to control the generation of targeted features and attributes. We evaluated our approach on two downstream tasks: object detection and depth estimation. Evaluation results indicate that YOLOv8 object detector trained on the refined data improves the Mean Average Precision (mAP) by over 20% points, relative to one trained on the original data. Furthermore, our approach achieves Frechet Inception Distance (FID) score of 41.95, indicating high visual quality and realism. This work highlights the effectiveness of diffusion-based techniques in bridging the domain gap between synthetic and real-world datasets. The KITTI fine-tuned Stable Diffusion model and our refined version of VKITTI are available at: (https://github.com/IqraNosheen786/Enhancing-Synthetic-Data-Realism-Using-Segmentation-Guided-ControlNet/).