Our vision is to build a painting machine that mimics human painters and is able to distribute real paint on a real canvas. e-David (acronym for: Electronic Drawing Apparatus for Vivid Interactive Display) is an ordinary welding robot, which typically is used for manufacturing cars. We combine the machine with sensors, a camera and control computer. A computer program continuously produces drawing commands that are executed by the machine. Since we work with real materials and brushes, the output of the machine is never perfect. Therefore we establish a visual control loop using a camera that captures the output of the machine. New drawing commands are computed based on the difference between what is on the canvas and what we want to have on the canvas. This visual feedback also allows us to correct errors on the canvas induced by paints and imprecisions of the painting process. This way we gradually approach a given input image.
After building a first e-David that was nearly immoveable due to its size, in 2015 we created a second version that uses a small robot (Kuka Youbot) and a moveable canvas. This machine can be disassembled easily, transported and exhibited.
Our hypothesis is that painting – at least the technical part of the painting process – can be seen as an optimisation processes in which color is manually distributed on a canvas until one is able to recognize the content – regardless if it is a representational painting or not. (Human) optimization happens intuitively during painting and is highly dependent on the medium and its restrictions. So-called academic art used quite specific algorithms for scene layout as well as color and content composition for creating art works. Bob Ross is famous for his mechanistic and algorithmic approach to painting.
On the other side computer graphics and so-called non-photorealistic rendering methods had a lot of progress in imitating painting styles using the computer. By simulating media and stroke composition a lot of painting-like images were produced, typically by computing pixel information that later was printed on a conventional printer. E-David will substitute this by distributing real paint on real canvas and thus will enable us to implement the whole process of painting production using an optimization framework. This will allow us to investigate human “optimisation processes” and to find out to what extent such processes can be formulated using algorithms. Semantic information can be integrated to optimize representational paintings; a tree will be painted in a different way than a face even if the colors are similar. Such semantic information can be obtained from semantic image analysis of the given target image. A number of other questions arise immediately:
It is of general interest to find out to what extent a machine is able to learn the whole process of creating a painting. The results might even influence our perception of what art is – besides the imitation of existing drawing styles (imagine what Seurat would have been thinking about a robot that helped him with the millions of dots) the machine might enable new techniques since labor plays no role any more. This way very complex visual art works can be created. The software and hardware platform will be open to artists, which can realize art works on a Java-based/processing environment.
The setup can be used for the analysis of existing art works and painting styles. The robot enables us to execute brush strokes in a precisely timed sequence. This will allow us to set up experiments for color mixture and ordering of brush strokes. Accomplished by appropriate image analysis techniques this might allow us investigating the temporal development of artworks (and even the detection of forgeries).
Algorithms for learning the usage of drawing tools and their visual control are prototypical for a more general class of optimization methods that can be applied in different areas of CIM. Examples are welding and painting robots, especially in small businesses in which efficient reprogramming and adaptation of robot programs is necessary.
Computer models for color description, their mixture and interaction with the canvas are still very limited. Existing approaches allow this only to a limited extent, especially if complex materials are used. There are no methods to compare computer simulations and real artifacts, this also hinders a thorough evaluation. Within the e-David project we will continuously supervise the paint application. However, this supervision is not trivial since real paint has a number of 3D effects, specular reflection and other disturbing properties. A special setup with polarized light from small angles is used to overcome these problems, we will calibrate the images by using spectroscopic methods.