About
Program
Participants
Proceedings
FAQ
Info@
Engineering
andMusic
Version:
1.3
31.
August 2001 |
|
Sound and Meaning in Auditory
Data Display
Thomas Hermann
Abstract
This paper focusses on the connection between human listening and
datamining. The goal in the research field of datamining is to
find
patterns, to detect hidden regularities in data. Often, high-dimensional
datasets are given which are not easily understood from pure inspection
of
the table of numbers representing the data. There are two ways
to solve
the datamining problem: one is to implement perceptional capabilities
in
artificial systems - this is the approach of machine learning. The
other
way is to make usage of the human brain which actually is the most
brilliant data mining system we know. In connection with our
sensory
system we are able to recognize and distinguish patterns, and this
capability is usually exploited when data is presented in form of a
visualization. However, we also have extremely high-developed pattern
recognition capabilites in the auditory domain, and the field of
sonification addresses this modality by rendering auditory representation
for data for the joint purposes of deepening insight into given data
and
facilitating the monitoring of complex processes.
An unanswered question is how high-dimensional data could or should
sound.
This paper looks at the relation between sound and meaning in our real
world and transfers some findings onto the sonification domain.
The result
is the technique of Model-Based Sonification, which allows the development
of sonifications that can easily be interpreted by the listener.
Auditory Perception and Environmental Listening
In evolution, the auditory senses developed because they provide us with
relevant information to survive. However, we use listening far beyond
this
primary purpose by developing language and music. These three
sorts of
auditory signals shall be distinguished by their different functions.
The elementary and oldest function of listening is the detection of
sound.
Sounds in our environment provide us with awareness and they are able
to
draw our attention to potentially dangerous events (e.g. approaching
enemies). Besides this, sound allows us to extract lots of information
about a 'sounding' object: its size, material, surface, tension and
so on.
We are able to abstract from sounds to properties of the sounding object
or
sound process because the connection between sound and an object is
fixed,
given by the laws of physics. Because the coupling mechanism (physical
laws) were constant over a large time scale, evolution was able to
develop
'hard-wired' mechanisms to interprete such signals and pull out relavant
information from the signal without the need for conscious processing.
Sound emerges as a consequence of excitation of physical objects, thus
objects/instruments in equilibrium are usually silent. Humans often
excite
objects consciously and thus get sound as a feedback to their actions.
So
they can relate the sound to their actions and learn about the world
from
this interaction loop. E.g. pressing a button, we know from the sound
(besides haptical feedback) if our action succeeded. While these
observations seem self-evident, they are often ignored when considering
sonification and techniques to access data by acoustic
representations. Using this relation between sound and meaning, an
alternative to Parameter Mapping, the prevailing sonification technique,
is
developed in the next section.
Let us now focus on the relation between sound and meaning within other
types of sound signals. In language, spoken words receive their meaning
within a cultural context and the association is learned by each child.
The relation between the sound of the word 'table' and the meaning
of this
word must be learned and is somehow arbitrary. Obviously, humans have
also
excellent capabilities in learning and accessing the meaning of learned
auditory patterns. While the information within environmental sounds
is
analog, language emphasizes the communication of symbolic information
or
abstract content. In sonification, verbal messages are suited
to label
categorial data or provide symbolic labels within an analog auditory
data
display.
Musical Information lies in between these two sound types. Controlling
sounding objects by human supervisors leads to sound that both gives
information about the instrument and the performer. Harmonic relations
find
an analog in physical laws (Fourier decomposition of periodic signals)
while melodic and rhythmic structures are related to prosodic patterns
in
language and narrative.
Data Sonification
High-dimensional data is given by vectors of numbers. The question is how
to generate meaningful sounds from given data. The traditional approach
is
to synthesize a sonification by superimposing sonic events whose attributes
(onset, frequency, duration, amplitude, etc) are mapped by numeric
values
of the given data vector. This technique is known as parameter mapping.
The
main disadvantage is, that there is no canonical way to specify the
mapping
and that the mapping must be known prior to interpreting the
sonification. For high-dimensional data, a mapping might be long and
complex. There are further problems with interfering perceptional
dimensions.
Model-Based Sonification
Model-Based Sonification solves some of the problems mentioned above by
providing a structured method to link data to a sounding object.
Different
from parameter mapping, data is taken to parameterize a sound model,
which
rests in a state of equilibrium without excitation. Dependent
upon the
model, dynamical laws are introduced which cover the temporal evolution
of
the model, giving rise to acoustic signals which represent the
sonification.
The main advantages of this model-based approach is, that the sound
and its
meaning with respect to the data are connected in the same way as in
the
real world. Thus, intuitive metaphors can be applied to interact
with the
model. Think of a model where the local tension of a membrane surface
is
parameterized by data. It can be struck, plucked, rubbed, etc. to make
it
produce sound. Information about the data is given in the control loop
between human excitation and system reaction. As a model definition
is not
bound to specific data, humans get the chance to become familiar with
the
sound space of a model. Furthermore, the model can be applied to arbitrary
data. If dynamical laws are applied which resemble physical laws
that
govern sounding objects, the sonifications are likely to be sounds
within
the soundspace we are familiar with from our listening experience.
Sonification models emphasize acoustic signals as a feedback to human
actions. This offers new perspectives for human supervision of complex
data
and control of data manipulations. The investigation of sonification
models and their utility for exploration of high-dimensional data is
the
topic of current ongoing research. |
