Most of my research with Shimon has focused on improvisation and note generating techniques. Recently I have been developing a system that integrates the cognitive with the physical. As a result of the physical design, Shimon is unable to play certain licks or motifs. This improvisation system combines Shimon's physical path planning processes with the note-generating decision processes so that the melodies or motifs generated by the system are something that Shimon is able to play.

 

Here is an example of an improvisation generated by the system for the song "All the things you are" (using a midi piano)

 

 

 

 

Another research focus with Shimon is anticipation. Using computer vision techniques we track drum sticks or mallets and follow their trajectory in order to predict when and which drum the stick will strike. 

 

 

 

 

 

 

Here is a video from the Listening Machines performance in 2011 of a song I wrote for Shimon and drum set. Shimon listens for different features such as tempo, if I'm playing in or out of time, repeating patterns, and note density and generates a response. There is a written melody that Shimon references as well as improvised sections where Shimon generates notes autonomously.