Skip to content

Interactive Sonification

Jump to bottom
stc edited this page May 3, 2018 · 8 revisions

Sonification

Sonification is the use of nonspeech audio to convey information. More specifically, sonification is the transformation of data relations into perceived relations in an acoustic signal for the purposes of facilitating communication or interpretation.Any technique that uses data as input, and generates (eventually only in response to additional excitation or triggering) sound signals may be called sonification, if and only if

  • (A) the sound reflects properties / relations in the input data.

  • (B) the transformation is completely systematic. This means that there is a precise definition of how interactions and data cause the sound to change.

  • (C) the sonification is reproducible: given the same data and identical interactions/triggers the resulting sound has to be structurally identical.

  • (D) the system can intentionally be used with different data, and also be used in repetition with the same data. (Thomas Hermann)

Sonification image source: https://iem.kug.ac.at/en/research/computer-music/sonification.html

Comments to (A): Real-world acoustics are typically not a sonification (although they deliver object property specific systematic sound) since there is no external input data. However, a bottle filling with rain, hit with a spoon once per second, can be seen as a sonification. The data here is the amount of rain, measured by the fill level, and the other conditions are also fulfilled.

Comments to (B): Randomness may be allowed here (e.g. as temporal jitter to increase perceptability), yet it is important to declare where and what random elements are used (e.g. the mean, variance, distribution of used noise)

Comments to (C): Sample-based identity is not necessary, yet all possible psychophysical tests should as a limit over many repetitions come to identical conclusions.

Comments to (D): In result, “playing a musical instrument” is not a sonification of the performer’s emotional state, since it can not be repeated with the identical data. However, the resulting sound is a sonification of the interactions with the instrument (regarded as data), and in fact, music can be heard with the focus to understand interaction patterns.

Interaction

Interaction's original meaning is a situation where two or more people or things communicate with each other or react to each another, such as in a meaningful conversation. See more on this in cybernetics, especially Conversation Theory (by Gordon Pask).

Feedback Loop image source: feedback loops

However, on the workshop, we are referring to interaction in a simpler way, where it means that we are manipulating parameters of the system during the process. This helps in the active exploration and navigation within the sonic data. This might be to optimize a sonification in order to make a specific structure in the sound more salient, thus to optimize the sonification according to some objective.

Code that we use during the session:

  • HelloPd
  • PureDataBasics
  • ProcessingBasics
  • ImageSonification
  • LiveCamSonification
  • DataSonification
  • WebAPISonification

Resources

Sonification:

sonification.de & Sonification HandBook(pdf), Auditory Displays: ICAD

Pure Data:

BangBook (pdf), Programming Electronic Music in Pd

Processing:

Books by Daniel Shiffman

Clone this wiki locally