🎨 ARtVista: Gateway To Empower Anyone Into Artist

ACM CHI 2024

Trong-Vu Hoang^1,2,† Quang-Binh Nguyen^1,2,† Duy-Nam Ly^1,2 Khanh-Duy Le^1,2 Tam V. Nguyen³ Minh-Triet Tran^1,2 Trung-Nghia Le^1,2,*

¹ University of Science, VNU-HCM, Ho Chi Minh City, Vietnam
² Vietnam National University, Ho Chi Minh City, Vietnam
³ University of Dayton, Ohio, United States
^† Equal contribution; ^* Corresponding author

Four participants moving from spoken ideas through AR guidance to completed paintings with ARtVista — ARtVista guides four pilot-study participants from spoken ideas to paintings using generative AI and augmented reality.

Abstract

Drawing enables people to express their imagination and emotions. However, translating abstract ideas into visually coherent representations and bridging the gap between mental visualization and practical execution can be challenging. We introduce ARtVista, a novel system integrating augmented reality and generative AI. ARtVista recommends reference images aligned with users’ ideas, generates sketches to draw, and creates vivid paintings in different styles. Its Paint-by-Number mode transforms reference images into color-grouped templates, empowering users to create compelling artwork without advanced drawing skills. A pilot study found positive responses to the system’s usability, its support for visualizing ideas, and its guidance throughout the painting process.

Method Overview

ARtVista is a Unity mobile application that turns spoken English prompts into reference images, sketch levels, and painting-style variants. Stable Diffusion with LCM-LoRA supports image generation, while the AR interface overlays the selected composition on tracked drawing paper. Its Paint-by-Number mode clusters colors with K-Means and outlines the resulting regions.

ARtVista workflow covering reference image generation, sketch and painting generation, and Paint-by-Number simulation — **Figure 3.** ARtVista’s three core capabilities: reference image generation, sketch and painting generation, and Paint-by-Number simulation.

Automatic Number Painting Regions

ARtVista creates its Paint-by-Number template by using K-Means to cluster the source image’s colors, then applying graph traversal to remove small isolated regions. Edge detection outlines each remaining region, and every region receives the index of its corresponding color cluster.

ARtVista Interface

The ARtVista interface lets users enter prompts by voice, choose a generated reference, and convert it into either a painting-style image or a composition sketch. In AR mode, the selected sketch is overlaid on the live camera feed and kept aligned with specially designed drawing paper, allowing users to focus on painting instead of repeatedly repositioning the guide.

Qualitative Results

Four participants created paintings with ARtVista and its Paint-by-Number mode. They valued the support for visualizing ideas and completing an artwork, while identifying opportunities to improve speech recognition, color selection, drawing through the screen, and alignment between paint and template.

Paintings created by four pilot-study participants using ARtVista — **Figure 6.** Paintings created by participants using ARtVista.

Citation

If you find this work useful, please consider citing:

@inproceedings{Hoang2024ARtVista,
  title = {ARtVista: Gateway To Empower Anyone Into Artist},
  author = {Hoang, Trong-Vu and Nguyen, Quang-Binh and Ly, Duy-Nam and Le, Khanh-Duy and Nguyen, Tam V. and Tran, Minh-Triet and Le, Trung-Nghia},
  booktitle = {ACM Conference on Human Factors in Computing Systems (CHI)},
  year = {2024},
}