Electronic ARTrium

2021 ~ Present | BMW


Design and build an interesting and engaging interactive sensor-driven dynamic art installation. The installation consists of a combination of three kinds of original art — sound, mechatronics, and video — that all respond to participant motion in a synchronized way. Key points in participants’ poses, extracted in real time from camera output, are processed and then streamed over Ethernet to control the images, sound, and mechatronic movement. The goal is to give the participants a sense of wonder and intrigue as they experiment with how their body movements change the art.  


Issues Involved or Addressed

• Fabrication and motion control of mechatronic elements, such as blooming flowers, talking Snowy Owl, honeycomb cells that individually light up and move forward and backward, and a Dancing Snake

• Creation of interactive videos of scenes, animals, and plants

• Lighting control for timed spot lighting and strobe (to simulate lightning)

• Implementation of interactive music and sound effects

• Write C# programs to control the music, video, and mechatronics in all sections.

• Integration of a bar-code printer at the entrance to the exhibit and four bar code scanners, one scanner at the entrance to each section, to track the player’s progress in the game

• Implementation of LIDAR to detect occupancy in each section

• Implementation of speech/text-to-gesture software to control movements of video and mechatronic characters

• Detecting the player’s pointing direction using a depth camera

• Set building and decoration

• Networking the elements of the exhibit

• Website development

The first generation of this installation was entitled “Raise Your Hand” and ran in a two-week public show in the lobby of the Ferst Center for the Arts, November 1-14, 2022. (see https://www.youtube.com/watch?v=5bcnNZb77v8 , https://www.youtube.com/watch?v=gC2FGTFGS2c , or https://www.youtube.com/watch?v=qtsGjhYW__A). 

Our second show, which will take the form of a life-sized four-section interactive video game with mechatronics, will be in the Ferst Center in Spring 2025.  The participant must find the clue in each of the first three sections through game play.  A different interactive mechatronic narrator in each section coaches the participant and converses with Mr. Bee, the main character, who has become lost because of a storm. The participant will help Mr. Bee find his way back to the hive by body tilting to avoid “attack clouds” in Section 1, pointing to the right sequence of mechatronic blooming flowers in Section 2, assuming a series of poses that unlock keyholes in Section 3, and using the clues to unlock the entry to the hive in Section 4.

The new students will be mentored by more experienced students.  The Sound Design sub-team will work with stems and sound effects created by a professional music and sound effects composer.

The overall project team includes this VIP team and may also include one or more interdisciplinary senior capstone design teams and undergraduate research students.

Methods and Technologies

  • Rhubarb and Unity for lip synch in both video and animatronic characters
  • Mediapipe, USB cameras, depth cameras and Python for 3D pose detection
  • LIDAR for section occupancy
  • Blender for animated object creation
  • Unity for video game creation, including most of the interactive audio
  • Circuit Design for lights and motors
  • Arduino for motion and lights control
  • Ethernet for networking
  • CAD for mechanical design
  • 3D Printing and other makerspace fabrication techniques for mechatronics fabrication
  • Woodworking for set fabrication
  • Wordpress for website development

Academic Majors of Interest

  • ComputingComputational Media
  • ComputingComputer Science
  • ComputingHuman-Computer Interaction
  • DesignIndustrial Design
  • DesignMusic Technology
  • EngineeringComputer Engineering
  • EngineeringElectrical Engineering
  • EngineeringMechanical Engineering
  • EngineeringRobotics
  • Ivan AllenLiterature, Media, and Communication

Preferred Interests and Preparation

The student should be interested in the intersection of art and technology and in building things. Object oriented programming experience will be helpful.




Meeting Schedule & Location

Meeting Location 
Klaus 1440
Meeting Day 

Team Advisors

Dr. Mary Ann Weitnauer
  • Electrical and Computer Engineering

Partner(s) and Sponsor(s)


Related Sites