Visual Computing

University of Konstanz
Computer Graphics Forum

Using Saliency for Semantic Image Abstractions in Robotic Painting

M. Stroh, P. Paetzold, D. Berio, R. Kehlbeck, F. F. Leymarie, O. Deussen, N. Faraj
Teaser of Using Saliency for Semantic Image Abstractions in Robotic Painting

Semantically guided image abstraction: (a) Original image; (b) combined saliency map generated by our method; (c) resulting enhanced abstraction with varying degrees of abstraction (facades, ground, windows on tower); (d) color quantized abstraction mapped to customized palette (15 colors); (e) painting by robot using oil colors on canvas; (f) e-David robot and painting

Material

Paper (.pdf, 25.3MB)

Abstract

We present an adaptive, semantics-based abstraction approach that balances aesthetic quality and structural coherence within the practical constraints of robotic painting. We apply panoptic segmentation with color-based over-segmentation to partition images into meaningful regions aligned with semantic objects, while providing flexible abstraction levels. Automatic parameter selection for region merging is enabled by semantic saliency maps, derived from Out-of-Distribution segmentation techniques in combination with machine learning methods for feature detection. This preserves the boundaries of salient objects while simplifying less prominent regions. A graph-based community detection step further refines the abstraction by grouping regions according to local connectivity and semantic coherence. The runtime of our method outperforms optimization-based image vectorization methods, enabling the efficient generation of multiple abstraction levels that can serve as hierarchical layers for robotic painting. We demonstrate the quality of our method by showing abstraction results, robotic paintings with the e-David robot, and a comparison to other abstraction methods.

BibTeX

@article{Stroh2025UsingSaliencySemantic,
  abstract  = {Abstract We present an adaptive, semantics-based abstraction approach that balances aesthetic quality and structural coherence within the practical constraints of robotic painting. We apply panoptic segmentation with color-based over-segmentation to partition images into meaningful regions aligned with semantic objects, while providing flexible abstraction levels. Automatic parameter selection for region merging is enabled by semantic saliency maps, derived from Out-of-Distribution segmentation techniques in combination with machine learning methods for feature detection. This preserves the boundaries of salient objects while simplifying less prominent regions. A graph-based community detection step further refines the abstraction by grouping regions according to local connectivity and semantic coherence. The runtime of our method outperforms optimization-based image vectorization methods, enabling the efficient generation of multiple abstraction levels that can serve as hierarchical layers for robotic painting. We demonstrate the quality of our method by showing abstraction results, robotic paintings with the e-David robot, and a comparison to other abstraction methods.},
  author    = {M. Stroh, P. Paetzold, D. Berio, R. Kehlbeck, F. F. Leymarie, O. Deussen, N. Faraj},
  doi       = {https://doi.org/10.1111/cgf.70259},
  eprint    = {https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.70259},
  journal   = {Computer Graphics Forum},
  keywords  = {CCS Concepts, • Computing methodologies → Non-photorealistic rendering, Image processing, • Applied computing → Fine arts},
  number    = {7},
  pages     = {e70259},
  title     = {Using Saliency for Semantic Image Abstractions in Robotic Painting},
  url       = {https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.70259},
  volume    = {44},
  year      = {2025}
}