Conductor Following Ilmonen T., Hiipakka J. Helsinki University of Technology Laboratory of Telecommunications Software and Multimedia Action in Data Dress Suit Aim The Conductors Gestures Methods Implementation Aim Aim of the conductor following is to synchronize electronic music with human movements. Our main concern has been with capturing the rhythm of the tune. Other nuances of the performance such as crescendo and diminuendo can also be recognized. The Conductors Gestures - the Source Usually tempo is expressed with the right hand. The bottom of the movement marks an accented part of bar. Quite often these are the same as where the quarters are. In figure one there are patterns that have become international standards for time signatures of 3/4 and 4/4. Figure 1. Standard patterns In practice the patterns can change quite a lot from these with the dynamics of the piece being played. Sometimes they are even quite completely ignored. More realistic patterns are in figure 2. Figure 2. Realistic hand movements Left hand controls other parts of the performance such as showing sings for groups of players in the orchestra and general nuances. Methods for Tempo Tracking There are many methods that can be used to extract the tempo from conductors movements. On the Beat Since the vertical extrema define the rhythm it is possible to use only these to track the tempo. This basically means looking for the moments when conductors baton or hand turns its direction. The problem with this kind of action is that it is impossible to react quickly to changes in tempo or to predict the time when next beat will occur. Also the continuous nature of the conductors movements is lost for discontinuous pulse output, resulting in a loss of information. Oscillator One may notice the conductors hand goes periodically up and down. One possibility is to try to lock an oscillator into this movement. In practice this can be (and indeed has been) implemented comparing the extrema of not only position, but also the speed and acceleration of conductors movements to those of an oscillator and then update the oscillator accordingly. Problems do however arise since the real movements do not follow any precise function but rather evolve all the time. Thus even simple highly periodic movements cannot be reliably tracked without a large number of heuristics. Artificial Neural Networks Our current system is based on back-propagating neural network. As inputs to this network serve the coordinates of the batons tip (or the right hand) and some past speed vectors. As output is the knowledge of the musical position within a single beat. This technique can also be understood as an elaborate way to filter out all but the fundamental frequency of the signal. This method has proved to be stable and quite robust. The output of the network is however not fit for directly controlling the tempo of the piece. Additional heuristics are used to smooth undesired artifact out of the tempo and to make the interpretation more human-like. Musical information describing the music is given to the program to make it react in a proper way to the various gestures. This resembles the information of the musical events as a musician would think of them. Ideally we would have model of a musician that would understand how to correctly react to the gestures. Ongoing research is targeted at improving the networks behaviour and making its output musically more relevant. Physical Implementation There are three parts to the operation of this system. Motion data is gathered using magnetic motion tracking device. As sample rate we usually stay in the range between 25 and 27 Hz. Higher sampling frequencies don't actually bring extra information to the operation, but going a lot below 25 Hz one starts lose accuracy. Also latency grows as the sampling rate drops. Sensors are mounted to a Data Dress Suit and gloves. There is one sensor that tracks the baton movements. This is mounted directly to the batons handle and the position of the batons tip can be calculated as both the position and the orientation of the sensor are known. Analysis software runs in real-time in a unix workstation (SGI or Linux-PC). This program controls the gathering of data and the playback of associated music. Playback of musical material is usually done with a syntheziser controlled by MIDI. We also do often have animation to match the music and this is naturally synchronized to the conductor also.

This page is maintained by Tommi Ilmonen, E-mail: Tommi.Ilmonen(at)hut.fi. The page has last been updated 30.9.1999. URL: http://www.tcm.hut.fi/Research/DIVA/following.html