AI UX PlaygroundNewsletterJoin 2K+ AI designers and PMs on Substack. New teardowns, patterns, and prompts as they drop.

Inputs

Gesture Input

Interpret drawn shapes, strokes, or hand gestures as commands or search queries. Show recognition feedback and undo when the guess is wrong. Built for touch and pen where sketching beats menus.

Interactive demo

What’s confusing on this signup?
Circle what feels unclear and I’ll explain it. Scribble to rewrite.
Circle here

Overview

How might we design gesture input so people can trust and act on AI output?

When to use

  • Ideal for touch interfaces, creative applications, and mobile tools where gesture-based input provides intuitive, natural interaction methods.

When to skip

  • Precision tasks better served by keyboard shortcuts.
  • Accessibility requirements mandate non-gesture alternatives.
  • Gesture recognition latency breaks drawing flow.

Rules

  • Gestures with no on-screen legend or tutorial.

  • Conflicts with OS-level system gestures.

  • No undo when gesture misfires.

  • Requiring perfect shapes with no tolerance.

Evidence

ProductImplementation
Apple PencilScribble and lasso selection in Notes and markup.
Samsung S PenAir actions and circle-to-search gestures.
Surface PenInk to shape and gesture shortcuts in Windows ink.
Creative appsCircle lasso for AI edit regions on canvas.

FAQ

What gestures work for AI actions?

Circle to search, lasso to inpaint, swipe to dismiss suggestions, pinch to zoom canvas regions.

Feedback during gesture?

Show live stroke, recognized intent label, and cancel zone.

Gesture vs voice commands?

Gestures are spatial on touch or pen. Voice commands are spoken action triggers.

Discoverability?

First-run overlay, settings cheatsheet, and hover hints on pen hover where supported.