2024
How do virtual objects nudge a group’s movements?
Overview
We present "Gravitational Field" (GravField) an experimental and participatory live performance set in a co-located mixed reality environment. Participants wearing AR headsets are invited to join and collectively create real-time music through their collaborative body movements, guided by the programmed settings from live-coders. This innovative experiment harnesses intercorporeal signals derived from participants' body movements, including factors like distance, area formation, relative height differences, and head motion synchronization. Live-coders dynamically map these signals into metaphorical audiovisual patterns in augmented reality, providing participants with cues about the relationships between players. The live-coded patterns shape and influence participants' improvisation of body movements, while adjustments in live coding dynamically alter how these movements generate sound, influencing the overall improvisational experience. In addition, spectators outside the live performance area can enjoy the mixed reality performance through projection screens and TVs, enhancing audience engagement. GravField aims to explore the vast potential of intercorporeal signals, creating a communicative, playful, and co-creative space where players' interconnected bodies become the instruments of expression.
From live coding to live-configured objects
The later Nudging the Somas study develops GravField around an “object jockey” who changes the properties of virtual objects while participants move together. Ropes, springs, and magnetic fields provide recognizable metaphors; their apparent weight, elasticity, and force can change during the encounter. Visual and sonic feedback makes these configurations perceptible through movement.
The objects invite coordination without prescribing a fixed choreography. Participants can interpret a pull, negotiate distance, or discover another way of moving together as the configuration changes. The CHI 2026 paper examines this process through performance videos, interviews, soma trajectories, and field notes. It reports how ambiguity and live adjustment supported open-ended participation, and develops design principles for guiding collective movement while preserving agency. Published study.
Publications
Botao Amber Hu, Yuemin Huang, Mingze Chai, Xiaobo Aaron Hu, Yilan Elan Tao, and Rem RunGu Lin. 2025. Improvising within “GravField”: A Participatory Live-coding Performance Exploring How Digital Objects Mediate Intercorporeal Movements in Collocated Mixed Reality. NIME 2025, Music track, pp. 158–163. Proceedings and performance video · DOI.
Botao Amber Hu, Yilan Elan Tao, Rem RunGu Lin, Mingze Chai, Yuemin Huang, and Rakesh Patibanda. 2026. Nudging the Somas: Exploring How Live-Configurable Mixed Reality Objects Shape Open-Ended Intercorporeal Movements. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. Paper.
Botao Amber Hu, Yuemin Huang, Mingze Chai, Xiaobo Aaron Hu, Yilan Elan Tao, and Rem RunGu Lin. 2024. "GravField: Live-Coding Bodies through Mixed Reality". In Adjunct Proceeding of SIGGRAPH Asia 2024 (SA '24)*. XR. Read more
Botao Amber Hu, Yuemin Huang, Mingze Chai, Xiaobo Aaron Hu, and Yilan Elan Tao. 2024. "GravField: Towards Designing an Inter-bodily Live-Coding Performance System within Collocated Mixed Reality Field". In Companion Proceedings of the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY Companion '24). Work-in-Progress. Read more
Botao Hu, Yuemin Huang, Mingze Chai, Yilan Tao and Xiaobo Hu. 2024. "GravField: A Participatory Performance Exploring Intercorporeality as Live-Coding Instruments within a Co-located Mixed Reality". In Proceedings of the 2024 International Conference on Live Coding (ICLC ‘24)*. Live Performance. Read more
Exhibitions
NIME 2025, Canberra, Australia — participatory live performance. Conference programme.
SIGGRAPH Asia 2024 XR. "GravField". Tokyo, Japan.
The International Conference on Live Coding 2024 Live Performance. "GravField". Shanghai, China.
Credits
Concept Designer: Botao Amber Hu
Concept Designer: Yuemin Huang
Interaction Designer: Botao Amber Hu
Sound Designer: Mingze Chai
Technical Artist: Xiaobo Aaron Hu
Project Manager: Yilan Tao
Research Assistant: Yilan Tao
Project contributors: Botao Amber Hu, Yuemin Huang, Mingze Chai, Yilan Tao, Xiaobo Aaron Hu