Latency–Sustainability Trade-Offs in On-Device vs. Near-Edge Offloaded Semantic Segmentation

Edge computing offers low-latency inference for computer vision and minimizes wide-area network usage; however, the sustainability implications of transferring deep learning workloads to proximate servers are not thoroughly examined. This study examines the latency-sustainability trade-off associated with conducting semantic segmentation on-device compared to offloading the task to a near-edge server, while adhering to a service-level objective (SLO) of 150 ms for the 95th-percentile latency. We employ a lightweight segmentation model on an NVIDIA Jetson AGX Orin for on-device processing, alongside a larger model on an edge server equipped with three RTX A6000 GPUs, utilizing a Triton inference server. This setup allows us to evaluate end-to-end latency, accuracy (measured by mIoU and pixel accuracy), energy consumption, and carbon intensity (assessed through CodeCarbon) under varying network conditions. We conduct experiments on the Cityscapes dataset. The findings indicate that on-device processing maintains consistent tail latency in suboptimal network conditions, while offloading enhances accuracy and throughput in optimal network environments. An oracle SLO- and carbon-aware policy can achieve a reduction in operational carbon intensity by 15–20% compared to static baselines.