The five main tasks of interactive text generation as supported by generAItor (see Section 3.3). The beam search tree is the key element (see Section 4), facilitating visualization and interaction with the model’s decisions. Each task has a set of widgets associated (see Section 5), providing task-specific visualizations, controls, and interaction possibilities. Following our proposed tree-in-the-loop paradigm, the tasks are interwoven and can be combined in an iterative process, centered around the beam search tree.
Material
Abstract
Large language models (LLMs) are widely deployed in various downstream tasks, e.g., auto-completion, aided writing, or chat-based text generation. However, the considered output candidates of the underlying search algorithm are under-explored and under-explained. We tackle this shortcoming by proposing a tree-in-the-loop approach, where a visual representation of the beam search tree is the central component for analyzing, explaining, and adapting the generated outputs. To support these tasks, we present generAItor, a visual analytics technique, augmenting the central beam search tree with various task-specific widgets, providing targeted visualizations and interaction possibilities. Our approach allows interactions on multiple levels and offers an iterative pipeline that encompasses generating, exploring, and comparing output candidates, as well as fine-tuning the model based on adapted data. Our case study shows that our tool generates new insights in gender bias analysis beyond state-of-the-art template-based methods. Additionally, we demonstrate the applicability of our approach in a qualitative user study. Finally, we quantitatively evaluate the adaptability of the model to few samples, as occurring in text-generation use cases.
BibTeX
@article{Spinner2024🌳generAItorTree,
abstract = {Large language models (LLMs) are widely deployed in various downstream tasks, e.g., auto-completion, aided writing, or chat-based text generation. However, the considered output candidates of the underlying search algorithm are under-explored and under-explained. We tackle this shortcoming by proposing a tree-in-the-loop approach, where a visual representation of the beam search tree is the central component for analyzing, explaining, and adapting the generated outputs. To support these tasks, we present generAItor, a visual analytics technique, augmenting the central beam search tree with various task-specific widgets, providing targeted visualizations and interaction possibilities. Our approach allows interactions on multiple levels and offers an iterative pipeline that encompasses generating, exploring, and comparing output candidates, as well as fine-tuning the model based on adapted data. Our case study shows that our tool generates new insights in gender bias analysis beyond state-of-the-art template-based methods. Additionally, we demonstrate the applicability of our approach in a qualitative user study. Finally, we quantitatively evaluate the adaptability of the model to few samples, as occurring in text-generation use cases.},
address = {New York, NY, USA},
articleno = {14},
author = {T. Spinner, R. Kehlbeck, R. Sevastjanova, T. Stähle, D. A. Keim, O. Deussen, M. El-Assady},
doi = {10.1145/3652028},
issn = {2160-6455},
issue_date = {June 2024},
journal = {ACM Trans. Interact. Intell. Syst.},
keywords = {Large language models, beam search tree, natural language generation, explainability, language transformers, visual analytics},
month = {June},
number = {2},
numpages = {32},
publisher = {Association for Computing Machinery},
title = {🌳-generAItor: Tree-in-the-loop Text Generation for Language Model Explainability and Adaptation},
url = {https://doi.org/10.1145/3652028},
volume = {14},
year = {2024}
}
