Electronic ARTrium

Design and build an interesting and engaging interactive sensor-driven dynamic art installation. The installation consists of a combination of three kinds of original art — sound, mechatronics, and video — that all respond to participant motion in a synchronized way. Key points in participants’ poses and facial expressions, extracted in real time from camera output, are processed and then streamed over Ethernet to control the images, sound, and mechatronic movement. The goal is to give the participants a sense of wonder and intrigue as they experiment with how their body movements change the art.  

The overall purpose of this project is to explore the most effective mechanisms for interactive multimedia art.  A secondary purpose is to attract young people to STEM fields (science, technology, engineering, and mathematics).

The first generation of this installation was entitled “Raise Your Hand” and ran in a two-week show in the lobby of the Ferst Center for the Arts, November 1-14 (see https://www.youtube.com/watch?v=gC2FGTFGS2c and

https://www.youtube.com/watch?v=qtsGjhYW__A).  We expect to have another show in the Ferst Center in Spring 2024 and we will also seek the opportunity for shows in other venues at Georgia Tech or in metro Atlanta. The student will be expected to help with set building, transportation, set up, and breakdown for shows, as well as serving as a guide for participants during shows.

Going forward, we will continue aiming to create an entertaining interactive multi-media experience for the participant, while building on the successes and using the feedback from the first show.  We expect to continue using cameras and MediaPipe for pose and facial key points information, Ethernet for UDP packet communication, Unity 3D for the video, Max or Max for Live for the music, mechatronics driven by microcontrollers, and an invisible LIDAR screen to protect participants from the mechatronics and vice versa.  Changes we anticipate are having more and separated sections, more mechatronics, use a larger set of gestures, expand to recognizing pose of multiple people per section, and improve the facial expression recognition.  We are also interested in interaction between a mechatronic figure and a similar figure in the video, in addition to interaction between participants and the exhibit. We also want to experiment with new ways to use music with mechatronics, such as incorporating motor sound as percussion and musical composition around or in concert with the motion of mechatronic figures.