Multi-Weather Segmentation

A research project investigating semantic segmentation robustness across diverse weather conditions for autonomous driving. Using the CARLA simulator, we collected a 3,600-image dataset spanning 9 weather-time combinations and systematically evaluated three training strategies — from clear-only baseline to weather-aware curriculum learning — on a DeepLabV3 backbone with 19 CityScapes-standard classes.

View on GitHub

Highlights

• Designed Weather-Aware Curriculum Learning (3-stage progressive training): clear → clear+soft rain → all 9 conditions with 10:20:70 hard-rain-biased sampling, boosting avg mIoU from 0.614 to 0.776+ over clear-only baseline
• Built custom CARLA data collection pipeline: ego vehicle + autopilot traffic, 512×512 camera at 1.6m height, 9 weather-time combos (Clear/SoftRain/HardRain × Noon/Sunset/Night), 3,600 annotated frames total
• Clear-only baseline degrades severely under domain shift — mIoU drops from 0.779 (ClearNoon) to 0.291 (HardRainNight); curriculum learning maintains ~0.7+ across all conditions
• Evaluated per-weather mIoU across all 9 conditions: curriculum training shows consistent robustness gains, especially on rare hard-rain nighttime scenarios where clear-only fails catastrophically
• Weighted sampling in Stage 3 (10:20:70 ratio) explicitly counteracts class imbalance between easy and safety-critical hard-rain samples

Training Strategies

Three strategies compared on DeepLabV3 with ResNet-50 backbone, pre-trained on COCO 20-class subset:

The curriculum approach avoids the instability of direct multi-weather training while preventing the clear-weather bias of setting 0.

Dataset

The 9 weather-time combinations in the CARLA dataset: Clear/SoftRain/HardRain × Noon/Sunset/Night — 400 images per condition, 3,600 total

Model Architecture

DeepLabV3: atrous convolution + ASPP (Atrous Spatial Pyramid Pooling) captures multi-scale context — critical for parsing diverse scene elements under weather degradation

Qualitative Results

Each row shows RGB input / Model Prediction / Ground Truth. Comparing Clear-Only (Setting 0) vs. Curriculum Learning (Setting 2) on adverse weather conditions:

Soft Rain — Noon

Setting 0 (Clear-Only) on SoftRainNoon — degraded prediction under rain-induced domain shift

Setting 2 (Curriculum) on SoftRainNoon — improved boundary clarity and class consistency

Hard Rain — Noon

Setting 0 (Clear-Only) on HardRainNoon — severe failure on drivable area and vehicle boundaries

Setting 2 (Curriculum) on HardRainNoon — substantially more coherent segmentation despite heavy rain occlusion

Quantitative Results

Per-weather mIoU across all 9 conditions — curriculum and mixed-weather training show consistent gains over clear-only baseline

Validation mIoU training curves for all three settings — curriculum learning converges more stably than direct mixed-weather training

Technical Summary