Contour-relaxed Superpixels

Abstract

We have proposed and evaluated in several recent papers a versatile scheme for image segmentation/ pre-segmentation that generates a partition of the image into a selectable number of patches ('superpixels').

This segmentation is designed to yield maximum homogeneity of the 'texture' inside of each patch, and maximum accordance of the contours with both the image content as well as a Gibbs-Markov random field model.

In contrast to current state-of-the art approaches to superpixel segmentation, 'homogeneity' does not limit itself to smooth region-internal signals and high feature value similarity between neighboring pixels, but is applicable also to highly textured scenes. The 'energy' functional that is to be maximized for this purpose has only a very small number of design parameters, depending on the particular statistical model used for the images.
 
The capability of the resulting partitions to deform according to the image content can be controlled by a single parameter. We show by means of an extensive comparative experimental evaluation that the compactness-controlled Contour-relaxed superpixel method outperforms the state-of-the art superpixel algorithms with respect to boundary recall and undersegmentation error while being faster or on a par with respect to runtime.

References

Christian Conrad, Matthias Mertz, and Rudolf Mester
Contour-relaxed Superpixels, LNCS: 9th Intl. Conf. on Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR), 2013

Rudolf Mester, Christian Conrad, and Alvaro Guevara
Multichannel Segmentation Using Contour Relaxation: Fast Super-Pixels and Temporal Propagation, LNCS in Proc. 17th Scandinavian Conference on Image Analysis (SCIA), 2011

Alvaro Guevara, Christian Conrad, and Rudolf Mester
Boosting segmentation results by contour relaxation, IEEE 18th International Conference on Image Processing (ICIP), 2011

Code

Source code of the C++ implementation used to generate the results presented in the related papers is available for download. See detailed build instructions contained within the zip file on how to build and use the software.  If you use our software in your work, please be sure to cite the related papers.

Download Contour-relaxed Superpixels, Version 0.1-r2, 05.08.2013

Example Results (click to enlarge)

Compactness of superpixels can be varied from fluid to static. Figure shows boundary overlays and region mean images.

Our method has no intrinsic bias towards the initial boundary layout. The following images show typical results of our method when  initialized with square blocks and a diamond pattern.

Comparison to other methods (click to enlarge)

Due to the emphasis on statistically homogeneity instead of smoothness of a superpixel the proposed method does not spend 'energy' on random region contours. 

The following images show typical results where other superpixel methods based on a smoothness assumption fail. Green boundaries: result for the proposed method. Red boundaries: (top) method by Veksler et al. (ECCV 2010) and (bottom) SLIC Superpixels (EPFL Technical Report no. 149300, TPAMI 2012).

Large scale test of current superpixel methods

Daniel Stutz at RWTH Aachen performed a large scale test of many current superpixel approaches, including VSI's Contour Relaxed Superpixels. Have a look at their test results, and try out the CRS Superpixels yourself (code available on this web page).

David Stutz and Alexander Hermans and Bastian Leibe: Superpixel Segmentation using Depth Information, September, 2014, RWTH Aachen.

 

Work that uses Contour-relaxed Superpixels

Application of Countour-Relaxed Superpixels in Remote Sensing

(Stefanski, Mack, Waske 2013)

In [StefanskiMackWaske2013], the authors introduce a strategy for a semi-automatic optimization of object-based classification of multitemporal data by using Random Forest (RF), and the Superpixel Contour (SPc) algorithm. The SPc is used to generate a set of different levels of segmentation, using various combinations of parameters in a user-defined range. Finally, the best parameter combination is selected based on the cross-validation-like out-of-bag (OOB) error that is provided by RF. Therefore, the quality of the parameters and the corresponding segmentation level can be assessed in terms of the classification accuracy, without providing additional independent test data. To evaluate the potential of the proposed concept, Stefanski et al. focus on land cover classification of two study areas, using multitemporal RapidEye and SPOT 5 images. A classification that is based on eCognition?s widely used multiresolution segmentation algorithm (MRS) is used for comparison. Experimental results underline that the two segmentation algorithms SPc and MRS perform similar in terms of accuracy and visual interpretation. The proposed strategy that uses the OOB error for the selection of the ideal segmentation level provides similar classification accuracies, when compared to the results achieved by manual-based image segmentation. Overall, the proposed strategy is operational and easy to handle and thus economizes the findings of optimal segmentation parameters for the Superpixel Contour algorithm.

Example segmentation results with the Contour-relaxed superpixel algorithm, courtesy of J. Stefanski, B. Mack, B. Waske

Reference
Jan Stefanski, Benjamin Mack, and Björn Waske:
Optimization of Object-Based Image Analysis With Random Forests for Land Cover Mapping. IEEE Journal Of Selected Topics In Applied Earth Observations And Remote Sensing, in print (2013)

 

Application of Contour-Relaxed Superpixels in Image denoising

(Åström, Zografos, Felsberg 2013)

The Contour-relaxed superpixels can be used in order to find 'stationary areas' in images which effectively control denoising methods.

This figure (courtesy Vasileios Zografos, Freddie Åström, Linköping University) shows the process of a superpixel-controlled denoising.

Reference
Freddie Åström, Vasileios Zografos, and Michael Felsberg
Density Driven Diffusion, LNCS in Proc. 18th Scandinavian Conference on Image Analysis (SCIA), 2013