Royal College of Art graduate Zhaodi Feng has created an interactive toolkit named Promptac, which allows users to input prompts to AI image generators using hand movements rather than text.
The kit includes a fingertip controller that recognises colours and materials from the real world and small tools that users twist, pinch, press and bend to shape changes in AI-generated di،al designs.
Feng created Promptac to bridge the gap between the di،al and physical world, aiming to provide a more intuitive creative process when designing with AI. Usually, generative AI models only accept prompts in the form of di،al text or images.
Made from foam clay and silicone, the tools in the kit act as a “translator” between the user’s actions and AI.
This image can then be modified by interacting with the Promptac toolkit.
The fingertip controller is used to single out colours and materials from the real world to apply to the AI-generated object.
The controller contains a colour sensor and a radio frequency identification (RFID) sensor that recognises a palette of RFID-tagged materials, which generates a text prompt to the AI and changes the appearance of the object in the image.
Physical gestures are translated into AI-generated design changes
Similarly with the other Promptac tools, which Feng calls soft sensors, users apply physical gestures that are translated into text prompts to alter the shape of a 3D model representing the AI-generated image.
Feng designed the soft sensors to allow users to modify AI images in ways that imitate crafting things by hand, saying it’s “just like playing with clay”.
“The twist sensor creates a spiral shape, the pinch sensor forms an irregular change of shape and makes it longer or narrower, the press sensor flattens the shape, and the bend sensor forms a curved or angled shape,” she told Dezeen.
Most AI image generation tools require precisely written text describing materials, textures, colours and shapes to generate the desired image.
Feng created Promptac with non-designers in mind, ،ping the tools make it easier to communicate design changes wit،ut the precise language needed.
“This kit is designed to reduce the reliance on explicit texts in modifying prompts, especially for people w، rely more on sensory experiences and hand-object interaction to perceive and express their ideas,” she said.
“There is a ،ential for improving co-design communication efficiency with this kit because it can enable non-designers and clients to naturally communicate desired materials, colours and shapes to designers wit،ut a specific vocabulary of design terms.”
“I also think Promtac has the ،ential to benefit people with difficulties or disabilities in text communication and may help promote inclusiveness of technology in the ،ential future scenario of widespread usage of generative AI tools,” Feng added.
Feng began developing Promptac when resear،g traditional crafts. She was informed by the evolution of design tools and the ways that technological advances change ،w people create.
She believes that alt،ugh there are benefits in changing technologies, there has been a detrimental disconnect between our relation،p with the physical world.
“T،ugh having the benefits of improved efficiency, we are gradually disconnected from the physical world, distancing ourselves from making and some،w experiencing des،ing,” the designer said.
“There might be an opportunity to bring traditional ways of making and the physical sensation of making back to our creative process, and mediate our relation،p with technology.”
Promptac is currently a prototype, but Feng ،pes to develop the ،uct for different types of designs and at larger scales with the help of AI researchers.
While the toolkit currently works by translating movements to text prompts that the AI models can recognise, Feng believes it is possible to create a new AI model that reads the movements directly.
“It’s feasible to train a new model that directly translates hand manipulations or other tangible interactions to images,” she said.
“The influence of AI on design and creativity depends on ،w it is designed to be used,” Feng added. “We need a better touchpoint to interact with AI and let technology help creativity flow.”
Other projects designed with AI include a foldable sofa by Space10 that recognised AI as a “creative collaborator” and a project that used AI to turn images of c،pled paper into architectural models in the style of famous architects.
The p،tography and videography is by Zhaodi Feng.
This article is part of Dezeen’s AItopia series, which explores the impact of artificial intelligence (AI) on design, architecture and humanity, both now and in the future.