Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Most recent semantic segmentation methods adopt a fully-convolutional network with an encoder-decoder architecture, where the encoder progressively reduces spatial resolution and learns more abstract semantic concepts with larger receptive fields. Because context modeling is critical for segmentation, recent efforts focused on enlarging the receptive field through dilated/atrous convolutions or inserting attention modules, yet the underlying encoder-decoder FCN architecture remained unchanged. This paper offers an alternative perspective by treating semantic segmentation as a sequence-to-sequence prediction task. It deploys a pure transformer, without convolution and without resolution reduction, to encode an image as a sequence of patches, so that global context is modeled in every layer of the encoder, which is then combined with a simple decoder to form the SEgmentation TRansformer (SETR).

Extensive experiments show SETR achieves new state of the art on ADE20K with 50.28% mIoU and Pascal Context with 55.83% mIoU, along with competitive results on Cityscapes. Notably, it achieved first position on the highly competitive ADE20K test server leaderboard on the day of submission. The work mattered because it demonstrated that a convolution-free transformer encoder could serve as a powerful backbone for dense prediction, helping shift semantic segmentation toward transformer-based designs.