In A State

The interface entitled 'In a State' is a collaboration between artist Adinda van ‘t klooster and composer/programmer Nick Collins. It captures live audio and uses machine learning to analyse the emotions expressed in improvised piano music, rendering live 3D animations based on drawings by the artist in response. A research tool on this website is used to help judge the effectiveness of certain graphics in terms of expressing particular emotions. Findings through this research tool will further influence the graphics in the ‘In a State’ interface.

During the performance a computer agent improvises alongside the pianist following the emotions detected in the music. As the detected emotions change so will the animated graphics. The system is both aesthetically interesting and informative of musicians’ expression of emotional state; it helps to highlight the problematic nature of emotion research and the ambiguities therein.

The first prototype of this live interactive performance was trialled at NIME2014 in London. Nick Collins improvised to the interface accompanied by a live electroacoustic agent. Both the pianists’ and the agents’ music were analysed for emotional expression and visualised with accompanying abstract animations. Computer agent and pianist tended to synchronise in terms of emotional expression though often not by mirroring the other’s emotion exactly, but by approximating it. To read about the technical background of the system, see here.