Image-driven trajectory reconstruction using (12). (a) Reconstructing an input trajectory (grey) starting from the motor plan (red) and trajectory (black) on the left. We start with values of ∆ti = 1 and δi = 0 and the motor plan and trajectory initially match. Here, we optimize also the shape parameters Ac resulting in the motor plan and trajectory on the right. In this instance, a significant reconstruction error (blue circle) occurs on the lower part of the “K”, likely due to the presence of an inflection. (b) A second reconstruction keeping Ac fixed. (c) the generated speed profile for (b) with the corresponding lognomals in blue and the original (Gaussian smoothed) speed profile in grey. The latter is not taken into consideration, but the minimum time cost produces a similar number and location of peaks. (c) Reproduction with a 7-axis Franka robot and a chalk marker.
Material
Abstract
Large image generation and vision models, combined with differentiable rendering technologies, have become powerful tools for generating paths that can be drawn or painted by a robot. However, these tools often overlook the intrinsic physicality of the human drawing/writing act, which is usually executed with skillful hand/arm gestures. Taking this into account is important for the visual aesthetics of the results and for the development of closer and more intuitive artist-robot collaboration scenarios. We present a method that bridges this gap by enabling gradient-based optimization of natural human-like motions guided by cost functions defined in image space. To this end, we use the sigma-lognormal model of human hand/arm movements, with an adaptation that enables its use in conjunction with a differentiable vector graphics (DiffVG) renderer. We demonstrate how this pipeline can be used to generate feasible trajectories for a robot by combining image-driven objectives with a minimum-time smoothing criterion. We demonstrate applications with generation and robotic reproduction of synthetic graffiti as well as image abstraction.