A technology has emerged that can implement precise 3D motion with simple text input, without complex initial setup.

Professor Baek Seung-ryeol's research team at the UNIST (President Yong-Hoon Lee) Graduate School of Artificial Intelligence announced on the 19th that they have developed a technology (Text2HOI) that generates interaction movements between hands and objects by entering text in a prompt input window.

It is expected that the technology that precisely controls complex interactions between hands and objects with a single line of text will advance the commercialization of 3D virtual reality in the future.

This technology can implement actions such as grabbing and releasing objects and interacting with objects through text commands. It can be applied to various fields such as virtual reality (VR), robotics, and medicine, and is characterized by the fact that anyone can use it easily as it does not require a complex setup process.

Analyzes user-entered text to predict contact points on command targets related to hand gestures.

For example, if you input the command “pass the apple with both hands,” it probabilistically calculates the possible points of contact between your hands and the apple. Then, when making a motion to pick up an apple, the position and angle of the hand are adjusted considering the size and shape of the apple to implement fine hand movements.

The technology has applications in a wide range of industries, including simulating medical surgical procedures, controlling character movements in games and virtual reality, and virtually performing complex scientific experiments. In robotics, precise hand gesture control is also expected to enable natural interaction with robots.

Professor Baek Seung-ryeol said, “Text2HOI technology can be applied to various fields such as virtual reality (VR/AR), robotics, and medicine,” and added, “We will continue to pursue research that is helpful to society in the future.”

“We hope that this will serve as a cornerstone for the relationship between text prompts and the generation of hand-object interaction gestures, and that it will lead to more related research in the future,” said first author, researcher Jun-Wook Cha.

The results of this study were published online on June 17 at the Conference on Computer Vision and Pattern Recognition, a global artificial intelligence conference.

This research was supported by the Ministry of Science and ICT (IITP), the National Research Foundation of Korea (NRF), the Ministry of Science and ICT (MSIT), the Korea Institute of Ocean Science and Technology Promotion (KIMST), and CJ Enterprise AI Center.