


The enclaves of post-industrial capitalism are surrounded by audiovisual panoramas that have emerged from European and American avant-garde experiments in composition and multimedia experience. Avant-garde approaches towards the technologization of music and democratic media choice were transformed within the consumer culture of the 1980s, and are now consolidated as commerce within modern algorithmic culture. More than any other medium, video games most fully reproduce the encompassing, participatory ideal of the postwar avant-garde, but transfused with a relentless commercialism. Video games surround the player with choice, but the impetus to design these enveloping audiovisual environments does not come from the attempt to democratize culture — it comes from an overarching consumerism and the rationalization of computer logic. Avant-garde experiments in participatory art, named ‘democratic surrounds’ by Fred Turner, and interactive music in games share a genesis in the politics of media participation that developed after World War 2, but also in the logic of computerization. The modularity and automation of new media that creates a panorama of uniquely individuated audience experiences is not new, but the meaning of media interaction has been radically transformed within a hyper consumerism that developed towards the end of the millennium.



application/pdf
2036-461X
16178
33
en
https://creativecommons.org/licenses/by/4.0
Cinéma & Cie. Film and Media Studies Journal
The Avant-Garde Roots of Video Game Music and Algorithmic Culture
Text.Serial.Journal
https://riviste.unimi.it/index.php/cinemaetcie
19
Thematic issue / Section thématique
2019-10-01
2021-09-02
2019-10-01
2024-02-14
Cinéma & Cie
Donal Fullam
2019/10/01
University College Dublin
2036-461X
33
en
The Avant-Garde Roots of Video Game Music and Algorithmic Culture
https://riviste.unimi.it/index.php/cinemaetcie/article/download/16178/16653
19
The Avant-Garde Roots of Video Game Music and
Algorithmic Culture
Donal Fullam, University College Dublin




Abstract

   The enclaves of post-industrial capitalism are surrounded by audiovisual
   panoramas that have emerged from European and American avant-garde
   experiments in composition and multimedia experience. Avant-garde
   approaches towards the technologization of music and democratic media
   choice were transformed within the consumer culture of the 1980s, and are
   now consolidated as commerce within modern algorithmic culture. More
   than any other medium, video games most fully reproduce the encompassing,
   participatory ideal of the postwar avant-garde, but transfused with a relentless
   commercialism. Video games surround the player with choice, but the impetus
   to design these enveloping audiovisual environments does not come from the
   attempt to democratize culture — it comes from an overarching consumerism and
   the rationalization of computer logic. Avant-garde experiments in participatory
   art, named ‘democratic surrounds’ by Fred Turner, and interactive music in
   games share a genesis in the politics of media participation that developed
   after World War 2, but also in the logic of computerization. The modularity
   and automation of new media that creates a panorama of uniquely individuated
   audience experiences is not new, but the meaning of media interaction has been
   radically transformed within a hyper consumerism that developed towards the
   end of the millennium.


Introduction

   Interactive multimedia has emerged as a critical means for cultural engagement
in the twenty-first century. Video games, websites, social media, advertising,
new media art, and all kinds of apps and software now feature integrated media
elements — sound, music, video, images — that users can engage with through
different kinds of interaction. By employing a degree of choice, users are each
granted different encounters within defined sets of parameters, structuring media
experiences as individual and unique. Video games present a convergence point
for all kinds of media, and video game music, as an aspect of the multimedia
form, can be analysed according to the potential to individuate player experience
                                                  Cinéma & Cie, vol. XIX, no. 33, Fall 2019
                                          Donal Fullam


through the historical conditions of its development. Using Fred Turner’s concept
of the ‘democratic surround’,1 and Ted Striphas’ ‘algorithmic culture’,2 it is
possible to evaluate video game music as an aspect of contemporary multimedia
that is deeply connected to twentieth century avant-garde composition. Video
games are the product of a complex history of converging technologies, but also
converging political and economic conditions. Game history is normally treated
chronologically and the interactive nature of games is often treated cognitively,
and sometimes critically, but almost never in terms of cultural history. What are
the historical conditions through which this industry and increasingly popular
hobby has emerged, and what can they tell us about the current conditions
of gaming? By describing the formal and cultural connections between these
different musical approaches in terms of technology, algorithmic culture and art
history, the machine logic at the heart of modern media and the relay from mid-
century experimentalism to modern commerce can be disclosed.
   Twentieth century European and American avant-garde composers
approached musical composition in different but ultimately convergent ways
— as a totally determined parametric system, but also as aleatoric experiences
of chance and interaction. Taking inspiration from Arnold Schönberg (1874-
1951), composers associated with serialism wanted to completely control all
musical parameters to create objective ‘integral’ systems,3 conflating composition
with research, in imitation of the new technical work carried out in postwar
laboratories.4 Following John Cage, another school of thought emerged, where
composition could be a work of interactive multimedia — immersive and
subjective, with elements of chance and choice presenting a sense of democratic
participation. These avant-garde approaches are often placed on opposing ends
of the compositional spectrum due to their extreme formal differences, but they
both arise within similar conceptual and cultural contexts. For both sides of
the twentieth century avant-garde, music was a technology that enabled the
composer to create automated pieces and processes for audiences to experience
subjectively.
   Video games represent a kind of modern gesamtkunstwerk, combining
video, animation, stagecraft, modelling, architecture, lighting, narrative, game
mechanics and, of course, sound effects and music. In these terms they can be
categorized as multimedia — the combination of disconnected media sources
into homogenous, all encompassing media experiences. Music can operate as a
facet of multimedia because it has been historically theorized within European
art music as an auditory object, comprised of parametric elements, and

1
  Fred Turner, The Democratic Surround: Multimedia and American Liberalism from World War II
to the Psychedelic Sixties (Chicago: The University of Chicago Press, 2015).
2
  Ted Striphas, ‘Algorithmic Culture’, European Journal of Cultural Studies, 18.4-5 (2015), 395-412.
3
  Markus Bandur, Aesthetics of Total Serialism: Contemporary Research From Music to Architecture,
(Basel: Birkhäuser, 2001), p. 11.
4
  Georgina Born, Rationalizing Culture: IRCAM, Boulez, and the Institutionalization of the Avant-
Garde (Berkeley and London: University of California Press, 1995), p. 1.

62
            The Avant-Garde Roots of Video Game Music and Algorithmic Culture


organized through a technical system. The abstraction of music as a system of
parameters allows it to be placed within incongruous formats but also allows
it to be automated — it is the ability to automate the system that forms the
nexus between twentieth century avant-garde experiments and contemporary
video game music. Automated, algorithmic compositional techniques emerge
at different times, from different contexts, and according to different political,
aesthetic, and technological conditions. They reflect ideological assumptions but
also refract these assumptions in new ways. What becomes clear from looking at
the emergence of these approaches during different times is that, while formal
aspects radically change, underlying technical structures remain the same. The
concept of music as a technical system that can be algorithmically organized
underpins these seemingly disparate approaches, and while the political aims
of twentieth century avant-garde composers were discarded, their technical
inventions were retained and rearranged to suit the new circumstances of our
panoptic, panaural algorithmic culture.


Music In Technology/Technology in Music

   The concept of music that emerges from what is generally called Western
classical, or European art music, is a technical system of modular parts that can
be organized through sets of parameters — in other words, a technology. The
word ‘technology’ entered the English language during the seventeenth century,
coming from the root techne (create) and logos (ordering).5 Logos, and the
sense of ordering that it implies, can be understood as logic, ‘in series of steps
in order, and reasoning also in steps’. When techne and logos are combined as
technology, the word is understood to ‘refer to a “creation of order” (as in skill
or art used to create order-yielding work), or that in which order is created’.6
Technology can be defined as ‘a system created by humans that uses knowledge
and organization to produce objects and techniques for the attainment of specific
goals’.7 Contemporary perceptions of technology tend to reduce the concept to
specific material objects, but historically the word did not necessarily signify the
objects themselves, rather, the skill of doing things and the abstraction of logical
skill into a system.
   Music is a complex human activity, and although it can be abstracted as
theory and reified as a recorded object, it is still normally conceptualized as art,
rather than product or technology. Still, music as a social practice and expressive
medium has always been entangled with techne and logos, with musical
instruments thought to be among the earliest technological objects, and evidence


5
  La Shun L. Carroll, ‘A Comprehensive Definition of Technology from an Ethological Perspective’,
Social Sciences, 6.4 (October, 2017), 1-20, (p. 6).
6
  Ivi, p. 6.
7
  Rudi Volti, Society and Technological Change (New York: Worth Publishers, 2018), p. 29.

                                                                                              63
                                       Donal Fullam


of abstract musical theorization since before 1500 BCE.8 Musical instruments
are a kind of technology but musical material is traditionally conceptualized as
an artistic, expressive medium, not usually thought to be technological. Along
with the introduction of the word technology, seventeenth and eighteenth
century European theorists and composers began to conceptualize music as a
form of technology,9 by abstracting and quantifying acoustic material as the
fundamental musical element and systematically organizing tonal, rhythmic and
formal elements. These projects were generally intended for pedagogical use10
but they also asserted the understanding of music as acoustic material that could
be abstracted according to fundamental laws.
    Musical-theoretical literature during the eighteenth century reflects profound
changes in European intellectual thought due to the upheavals in science and
philosophy11 occurring at the time. The beginnings of an overarching project by
theorists to ‘rationalize the system of harmonic tonality’12 can be understood as
a pedagogical necessity for practical guidance but also as a deeper ideological
analogue with movements towards philosophical rationalization. What emerges
with the detailed organization of music into an apparently universal system is the
conception of music as a kind of technology: rational, impartial and autonomous.
In many ways music was already thought of in this sense, but large scale projects
of theorization during the eighteenth century allowed this form of thinking
to cohere among the theoretically literate and proliferate amongst amateur
musicians and the public. Students of theory and analysis of tonality are normally
taught the foundations in the same way — how the four part harmonic rules
of counterpoint are used to write chorale music through voice leading, chord
progression, cadential treatment, and so on, and how these parametric aspects
are organized within larger forms. This is a systematized way of understanding
music as an objective classification of parts, that are easily replicated as an overall
totality. This kind of system presents many different kinds of benefits — works
are easily reproducible with the correct information; the different parts have been
schematized, making it user friendly; it is modular, so parts can be reorganized
and recycled; new, unique instances can be created with similar material, and
the methodology and stylistics of the system can essentially be reduced to an
algorithmic determinacy. When the complex social processes of music are
excised in favour of the abstraction of musical material into a technical system,
it is essentially conceptualizing music as a kind of technology. twentieth century
European composers associated with the avant-garde brought the rationalization


8
    Sam Mirelman, ‘A New Fragment of Music Theory from Ancient Iraq’, Archiv für
Musikwissenschaft, 6.1 (2010), 45-51 (p. 45).
9
   Robert W. Wason, ‘Musica Practica: Music Theory as Pedagogy’, in The Cambridge History of
Western Music Theory, ed. by Thomas Christensen (Cambridge: Cambridge University Press,
2002), 46–77 (p. 55).
10
   Ibidem.
11
   Ivi, p. 53.
12
   Ivi, p. 54.

64
           The Avant-Garde Roots of Video Game Music and Algorithmic Culture


of music to an unprecedented degree, ushering in a new age of mechanistic
musical control. One of the main benefits of this systematic concept of music for
twentieth century compositional practices and the development of video game
music is the level of parametric automation it allows. Interactive music in video
games has roots in the early configuration of music as a modular, technological
system, but the realization of immersive 3D sound worlds is possible because of
the configuration of music and sound as a more total parametric science during
the twentieth century.


European Avant-Garde - Serialism

   In 1921 Arnold Schönberg famously invented a new formal technique called
twelve-tone music, which seemed to present an original conception of the
fundamentals of music itself. In order to create a sense of atonal balance and to
disrupt the hierarchy of nineteenth century tonal theory, twelve-tone music uses
each of the twelve notes of the chromatic scale within a ‘row’, which becomes
the main musical material. The row is subjected to a number of transformations
— retrograde, inversion, and retrograde-inversion — in lay terms, backwards,
upside down and backwards-upside down. The formal aspect of the technique
is more complex than can be described within this essay, but essentially, it is
an algorithmic process that uses musical parameters as its input, and produces
atonal musical material as its output — the row transformations constitute
the algorithm, ‘a formal process or set of step-by-step procedures’.13 Further
developments in Europe led to total serialism: the attempt to micro-manage and
control every aspect of musical sound and to organize it algorithmically.
   Olivier Messiaen (1908-1992) set the prototype with Mode de valeurs et
d’intensités (1949) while Pierre Boulez (1925-2016) developed the technique
towards what became known as total, or integral serialism. For integral serialism,
timbre, duration, frequency, and amplitude, the dynamic characteristics of sound,
became organizational fragments rather than expressive attributes.14 Karlheinz
Stockhausen (1928-2007) attempted to create a parametric measure of all musical
dynamics, not only as a way to more tightly control these aspects, but to also
combine them all into one integral system.15 He applied serial methods to pitch,
rhythm, dynamics, timbre, density, and time to assert an overarching sense of unity,
as a musical unified field theory in line with Einstein’s groundbreaking theory of
relativity.16 A vast new terrain had been opened and composers saw themselves


13
   Striphas, p. 403.
14
   Paul Griffiths, Modern Music and After (New York: Oxford University Press, 1995), p. 29.
15
   Robert P. Morgan, ‘Stockhausen’s Writing on Music’, The Musical Quarterly, 61.1 (January
1975), 1-16 (p. 3).
16
   Thom Holmes, Electronic and Experimental Music: Technology, Music, and Culture (New York
and London: Routledge, 2008), p. 124.

                                                                                        65
                                          Donal Fullam


as a vanguard heralding a progressive new era, with the creation of a completely
new music unencumbered by antecedent relationships, representations, symbols
or structures – an objective, rigorously scientific approach that might expose
reality rather than emotion.17 Schönberg’s approach and ensuing theories of
everything were attempts to further technologize music, to remove it from social
context and treat it as an object of scientific scrutiny. The organization of music
purely as a set of acoustic parameters granted composers a new level of control,
allowing them to manipulate music and sound at the microscopic level.
   Composers had identified where power was centralizing and were busy
molding themselves and their craft into forms more hospitable to these
institutions and their characters; the analysts, the specialists and the technocrats,
around whom new loci of power were beginning to whirl.18 Composers followed
the technologizing impulse to its logical conclusion in the hope of discovering
universal forms, and to position music within the sphere of technical fields that
were providing new ways of understanding the world. The radical reconfiguration
of composition as a tonal science during the middle of the twentieth century was
characterized by its exponents as a progressive strategy to rethink musical form,
and while the approach was popular within intellectual circles, the formal and
structural aspects quickly became redundant.19 The extreme contortions that
traditional compositional forms endured through serialism put most listening
audiences off and the music never gained popularity outside academia. However,
the dissolution of traditional form and the freeing of dissonance led by Schönberg
and the serial composers is not as important as the academic literature implies.
Most modernist or avant-garde composers did not completely embrace total
serialism or completely deterministic models of composition, so it is not possible
to characterize the entire oeuvre of Boulez or Stockhausen in opposition to
previous or contemporaneous styles. What was embraced, and what remains
the most relevant characteristic of serialism, was the idea that music could be
completely quantified as sets of parameters. Purely rational integral serialism
pursued the idea of music as a parametrically organized material to a merciless
degree. The concept that musical sound could be organized as discrete sets of
parameters and automatically generated within algorithmic systems was taken
up and transformed by American avant-garde composers in the second half of
the 20th century, and the total organization of musical values continues. The
quantization of musical sound that began during the 18th and 19th centuries,
and reached an apex within the 20th century avant-garde, now saturates the
contemporary world of music.




17
   Reginald Smith Brindle, The New Music (London: Oxford University Press, 1975), p. 23.
18
   Born, p. 100.
19
   Joseph N. Straus, ‘The Myth of Serial “Tyranny” in the 1950s and 1960s’, The Musical Quarterly,
83.3 (Autumn, 1999), 301-343 (p. 302).

66
            The Avant-Garde Roots of Video Game Music and Algorithmic Culture


American Avant-Garde - Democratic Surround

   The American postwar avant-garde centered around John Cage (1912-
92), ‘possibly the most influential musician in the world’20 at the time. On the
surface Cage and his followers seemed to be doing the opposite of the European
serialists — where they were closed, Cageans was open, where serialists sought
determinacy and the systematic, Cageans seemed to counter with indeterminacy
and chance. Their similarities went deeper than their differences however, as
both groups “sought ‘automatism,’ the resolute elimination of the artist’s ego or
personality from the artistic product”.21 Cage described his chance methodology
as a way to remove the composer’s hand and thus the composer’s ego, so music
could simply reveal itself, but the outsourcing of composition to systems of chance
and computers is also symptomatic of a much broader cultural and technological
shift that occurred during the twentieth century. Cage wanted to create processes
that could be experienced, and to create platforms for other people to interact,
offering up ‘a view of artistic practice as a leveled collaboration among artist,
audience, and materials’.22
   Cage is popularly known as the composer of 4’33 (1952), in which a pianist
opens a piano and sits silently for four minutes and thirty three seconds. The
ambient sounds of the auditorium, creaks and squeaks, people moving, coughing,
and whispering are the musical material, meant to be heard subjectively by
each audience member. Cage advocated for the emancipation of noise and the
appreciation of all sound as musical and meaningful - a stance normally attributed
to his study of Zen Buddhism, but which can also be understood as a reaction to
media politics during the twentieth century. 4’33, as with much of Cage’s work, is
intended to immerse audience members in a visual and auditory experience that
extends beyond the stage, giving audience members a sense of participation. Cage
considered it to be his most important work of chance and indeterminacy, and it
set the format for his experiments in multimedia participation and happenings.
   Following Cage, American composers began experimenting with the use of
space and indeterminacy during the middle of the twentieth century — these
experiments were encouraged by a national drive to create participatory,
democratic forms of art, in opposition to perceived fascist and authoritarian
modes of top down communication.23 Fred Turner coined the term ‘democratic
surround’ to describe these new media models — multi-image, multi-sound
source environments created by artists associated with the 1960s counterculture,
designed to model and produce a more democratic society.24 The practice of

20
   Richard Taruskin, Music in the Late Twentieth Century: The Oxford History of Western Music
(Oxford: Oxford University Press, 2010), p. 55.
21
   Ibidem.
22
   Fred Turner, From Counterculture to Cyberculture: Stewart Brand, the Whole Earth Network, and
the Rise of Digital Utopianism (Chicago: The University of Chicago Press, 2006), p. 47.
23
   Turner, The Democratic Surround, p. 28.
24
   Ivi, p. 143.

                                                                                             67
                                          Donal Fullam


creating visual panoramas was not new, but American artists and composers
associated with the counterculture used emerging technologies to create more
fully panoptic and panaural multimedia experiences and happenings that gave
audience members a sense of participation by allowing degrees of choice.
   John Cage’s HPSCHD, a juxtaposition of chance procedures with computer
technology, was premiered to an audience of 6000 on May 16, 1969, at the University
of Illinois. The event featured seven harpsichord players arranged around the
auditorium, playing seven solo pieces created from randomly processed music
by Mozart, Beethoven, Chopin, Schumann, Schönberg and Cage. Slides and
films were projected on giant screens hanging from the ceilings and on the walls,
and computer generated sounds played from fifty-two PA speakers.25 Lasting for
about five hours, it was a huge event — a hypnotic environment of sounds and
images where audience members were encouraged to move in and out of the
building, around the hall, and through the performing area to experience unique
interactions of sound, music and visuals. Audiovisual aspects ‘were meticulously
and systematically randomized so that it was left to the spectators to fill in the
space between sound and image with their random noises and movements’.26
People were encouraged to think of all the sound and noise they heard as
important as the snatches of music that could be heard through the cacophony,
presenting them with an overwhelming audiovisual encounter that used elaborate
algorithmic planning and procedures to allow participation through choice.
   The kinds of multimedia experiences that Cage and many others were creating
became a fixture of American artistic and cultural life during the 1960s and
beyond.27 Artists and musicians staged elaborate events with huge, all surrounding
PA systems, projectors, lighting, and performances that became known as human
be-ins or happenings. Many other composers associated with the American and
European avant-garde embraced the turn towards chance systems and democratic
participation too. Serial composition systems share aspects of aleatoric design,
although they were not necessarily created in the pursuit of new forms of agency.
Indeterminacy and choice are characteristic of Cage’s work but composers who
are primarily associated with the strict ordering of serialism also adopted choice
based forms that feature player-indeterminacy. Stockhausen’s Klavierstücke XI
(1956) allows the pianist to choose tempo and dynamic levels, and which order
to play nineteen modular sections from a large single page score,28 and different
orderings of Boulez’s Third Sonata (1958) can be chosen by the pianist during
performance.29 Composers associated with the American avant-garde more fully


25
    Stephen Husarik, ‘John Cage and LeJaren Hiller: HPSCHD, 1969’, American Music, 1.2
(Summer, 1983), 1-21, (p. 15).
26
   Ivi, p. 1.
27
   Turner, The Democratic Surround, p. 147.
28
   Elliott Antokoletz, A History of Twentieth-Century Music in a Theoretic-Analytical Context, (New
York: Routledge, 2014), p. 397.
29
   William G. Harbinson, ‘Performer Indeterminacy and Boulez’s Third Sonata’, Tempo. New
Series, 169, (June 1989), 16-20 (p. 1).

68
              The Avant-Garde Roots of Video Game Music and Algorithmic Culture


embraced Cage’s reconfiguration of performer and audience hierarchies. Terry
Riley’s In C (1964) grants musicians control over different decisions as the piece is
performed — how long to play sections for or whether to play them at all — making
modules available as choices within a loose but ultimately stable framework. La
Monte Young’s Dream House (1993) produces droning environmental sounds
that adapt according to the listener’s position in a room by utilizing the complex
harmonic arrangements of sound reacting with itself in three dimensional space.
These kinds of interactive characteristics can be traced through European classical
musical developments from the middle of the twentieth century as a gradual
process of ceding control — opening an interactive space between composers
and performers, but also between composers and listeners.
   John Cage was always willing to go a step further and it is in his work that
the inherent contradictions become most apparent. He created musical and
multimedia platforms that gave performers a sense of their own autonomy and
allowed audiences to value their subjective experiences, but also to participate
as equals. Freedom of interpretation and movement was meant to emulate and
produce a broader sense of freedom within a democratic society, as, according
to Michael Nyman, ‘with all those parts and no conductor, you can see that even
this populous a society can function without a conductor’.30 The impulse to create
modular, indeterminate and aleatoric pieces realized by Boulez, Stockhausen,
Cage, Riley, and others during the period can be framed according to an emerging
media culture that values open, interactive forms within rule-based frameworks.
The use of these compositional techniques can be seen as reactions to media
politics but also as composers displaying a precocious grasp of what was on the
threshold — a culture of media creators providing platforms for others to interact.
In theory, avant-garde composers and artists were attempting to democratize
composition, but inherent contradictions radically altered the practical results.
Cage suggested that his approach was more participatory and thus more democratic
than previous methods but he was still dictating the compositional parameters,
the audience’s relationship with the composition and the audiences relationship
with the composer. Audience members were not really granted an equal status but
allowed to participate within the boundaries set by Cage. The top-down media
methodologies that composers and artists attempted to bypass were reproduced
as less strict but nonetheless stable hierarchies. Avant-garde techniques meant to
demonstrate democratic potentials through audience choice became structured
as managerial hierarchies, and were quickly used in the design of commercial
products. By failing to inoculate the new techniques from commercialization,
they were easily appropriated within the explosion of commercial culture during
the 1970s and 1980s. The algorithmic control of music as parameters and the
inclusion of audiences through chance and choice within multimedia experiences
circumscribes the logic of new media, most explicitly in video games.


30
     Richard Kostelanetz, Conversing With Cage (New York: Routledge, 2003), p. 263.

                                                                                      69
                                        Donal Fullam


Video Games

   Video game sound and music is dynamic, it adapts in response to the actions of
the player and is contextually generated according to the needs of particular events
and gameplay states. This fundamental characteristic differentiates game audio
from that of film and other traditionally passive forms of sonic representation in
entertainment. Game music is usually process driven, supporting the ‘temporal
structure of the game narrative and gameplay’31 and while these structures are
immensely variable, the systems that control them are logical and predictive.
Changes are managed according to physical processes of input by players
who are in turn responding to in game action, flexible interchanges that must
be absolutely tracked and predicted by the game engine to create a coherent
experience.
   Early games featured simple looped monophonic melodies and crude noise
effects which, despite the limitations of relatively primitive technology, displayed
the basic elements that have come to define audio aspects of the medium —
contextual generation of music and player evocation of sounds. Failure to reach
the end of a level in Super Mario Bros. (1985) before the countdown timer has
reached one hundred will trigger a tempo shift in the music that tells the player
to move faster. Entering certain pipes will trigger the pipe sound and then the
famous underground theme, communicating to the player that an environmental
boundary has been crossed according to their actions.32 These kinds of musical
changes in early games are often accomplished by simple horizontal switching
but they nevertheless respond to the player and progress in real time response
to their actions, or lack of action. Later games present more complex musical
interactions and soundscape cues that are carefully balanced, faded, transformed
through effects and generated by the game according to player input and
changing events. In Tomb Raider (1996) sound and music emerges differently for
every player, creating soundscapes and musical developments that are precisely
individuated for each instance. Moving from a cave system to the lost dinosaur
valley replaces ambient sounds of echoing wind and dripping water with the roar
of a waterfall and angry snarl of Velociraptors. A tense nondiegetic orchestration
quickly fades in as a Tyrannosaur enters the scene and fades out as soon as it is
defeated.33 No two players will hear the same combination of sounds or exact
appearance of music — audio is tailored to individual experience. Spore (2008)
features a complex generative system created by Brian Eno and Paul Chilvers
that creates an ‘ever changing, musically cohesive dynamic soundtrack”34 using


31
   Isabella van Elferen, ‘¡Un Forastero! Issues of Virtuality and Diegesis in Videogame Music’,
Music and the Moving Image, 4. 2 (2011), 30-39 (p. 34).
32
   Nintendo, Super Mario Bros (1986), N.E.S.
33
   Eidos, Tomb Raider, (1996) Microsoft Windows and other platforms.
34
   Tim Summers, Understanding Video Game Music (Cambridge: Cambridge University Press,
2016), p. 22.

70
            The Avant-Garde Roots of Video Game Music and Algorithmic Culture


a computer model called cellular automata.35 In Portal 236 (2011) music is
structured around the player’s activity — horizontally switched when moving to
new areas, or vertically re-orchestrated through modular systems when puzzle
pieces are assembled. At an abstract but more fundamental level, all audio
in Portal 2 is interactive — game systems that control sound and music are
continually modifying parameters according to player action, creating a coherent
acoustic space and a sense of enveloping verisimilitude.
    A video game is a machine with many interlocking and interdependent parts
— in complex, first person perspective games, audio aspects are determined
by technical systems relying on banks of parameters of myriad microscopic
gradations. If the player turns their avatar ninety degrees left, the systems
controlling sound will react so that the acoustic space reflects the new position.
Sonic events heard previously from the front of the player will now be heard on
the right with all degrees in between accounted for. Audio emanating from in
front of the player is not actually playing from a spatial position — parameters
of panning, volume, reverb and frequency are adjusted so that audio appears to
be coming from a position relative to the player. In Portal 2 (2011) puzzle pieces
that add additional layers of music will be heard by each player differently, as
they individually arrive at solutions but also as they move around the 3D space.
The modular organization of audio as parameters allows it to be controlled and
deployed according to specific conditions, individualizing listener experience.
Interactive, dynamic and procedural music and sound in video games is possible
because of these processes — the theoretical rationalization of music has allowed
it to be conceptualized as a modular system of interlocking and interdependent
acoustic parts. Sound is generated through complex algorithmic structures and
precomposed modular parts are reconfigured in accordance with each player’s
actions.37 These systems create highly individualized musical experiences, within
the parameters set by the game’s designers and composers. No two players of
the same game are ever likely to hear exactly the same audio combinations and
contiguities, even if the music and sound heard by each is broadly similar.
    Video games are part of a modern media culture that is characterized by
interactivity, audience participation and a blurring of boundaries between
consumers and producers, but they are produced in a more commercial
context than earlier avant-garde experiments. The interactive nature of game
sound and video games in general can be understood as the expression of an
impulse that characterizes modern media production and consumption — an
increasing necessity for input from audiences, mediated by algorithms. The idea

35
   Ian Steadman, ‘Brian Eno on music that thinks for itself’, https://www.wired.co.uk/article/
brian-eno-peter-chilvers-scape [accessed 21 December, 2018].
36
   Valve Corporation, Portal 2, (2011), Microsoft Windows and other platforms. Composed by
Mike Morasky.
37
   Elizabeth Medina-Gray, ‘Modularity and Dynamic Play: Video Game Music and its Avant-garde
Antecedents’, paper presented at Ludomusicology conference, Liverpool University, 12-13 April
2013.

                                                                                           71
                                       Donal Fullam


that producers and consumers share collaborative duty is common, so much so
that the term ‘prosumer’38 was coined to designate the new role consumers play.
Analysis often places players somewhere between the subject position of ‘author/
producer and audience/consumer’39 but the term only applies to one side, as
the producers subject position has not changed. Producers are keen to develop
an ‘all in it together’ mentality, aligning consumers with the task of production
but in terms of consumer culture, rather than twentieth century experiments in
subjective meaning making and democratic inclusion. The political intentions
of earlier avant-garde approaches have been mostly abandoned, but, within the
context of consumer culture, the concept of music as a technological component
and the individuating power of interactivity have been retained and radically
transformed. Modern approaches are directly connected to practices that
emerged around the middle of the twentieth century in Europe and America
but are also expressions of something different — a ‘commercial surround.’ We
are surrounded by media and music but not in the way that Schönberg or Cage
envisioned. The impetus to design contemporary multimedia environments does
not come from the confrontation with top down modes of media — it comes
from an overarching consumer and algorithmic culture.


Algorithmic Culture

   Algorithms are at the heart of modern media use and it is the potential to
individualize media interaction through parametric ordering that is the connective
tissue between musical composition, cultural interaction, consumer subjectivity
and the contemporary media culture in which we have become enmeshed. Video
games are an expression of this context that Alexander Galloway and Ted Striphas
term ‘algorithmic culture’, a societal movement towards the use of complex
automated systems to create and mediate cultural products. The overwhelming
power of computerization seems to place algorithms within the realms of the occult,
but a definition is not complex — algorithms are step-by-step processes used to
complete tasks, and can be carried out by people or automated using machines
— particularly computers. In contemporary use “an algorithm is any well-defined
computational procedure that takes some value, or set of values, as input and
produces some value, or set of values, as output. An algorithm is thus a sequence
of computational steps that transform the input into the output.”40 In video games
a simple algorithm can take the player action as input, and horizontally switch
one music track for another or add new vertical layers of harmony or melody, as


38
   Alvin Toffler, The Third Wave (New York: Bantam Books, 1980).
39
   Karen Collins, Playing with Sound: A Theory of Interacting with Sounds and Music in Video
Games (Cambridge, MA: The MIT Press, 2013), p. 11.
40
   Introduction to Algorithms, ed. By Thomas H. Cormen and Charles E. Leiserson (Cambridge,
MA and London: The MIT Press, 2009), p. 5.

72
            The Avant-Garde Roots of Video Game Music and Algorithmic Culture


output. Complex systems can output music using computational models and more
complicated structures of code, in response to the players input, individualizing the
experience of virtual worlds. This kind of composition is not new — the impulse
to treat music as an algorithmically determined system can be traced back through
the twentieth century avant-garde and further, to the foundations of functional
harmony. Western perceptions of musical tonality itself can be described as a
type of algorithmic schema and an expression of a broader impulse to systematize
aspects of cultural articulation. Algorithmic culture does not necessarily arise from
the twentieth century, but the increasing mechanization and computerization of
societies has spurred its development and pervasiveness.
    The necessity for input from users and the feedback that generates
individualized experiences might paint algorithmic culture as a triumph of
democratic public culture,41 but there are hidden dangers. Algorithms are used
to structure cultural life and artistic expression in increasingly labyrinthine
configurations, with all kinds of modern musical interaction tending to come
within the algorithmic domain. Spotify’s Discover Weekly uses machine learning
to individually structure user listening habits42 and YouTube’s recommendation
algorithm uses deep neural network architectures to order, rank and tailor each
users feed.43 According to Striphas, “part of what is at stake in algorithmic
culture is the privatization of process: that is, the forms of decision making
and contestation that comprise the ongoing struggle to determine the values,
practices and artifacts — the culture, as it were — of specific social groups.”44
In video games, the goals of the avant-garde have materialized in unexpected
ways — the individualization of experience has been expanded, but within the
context of consumerism, it is also transformed. Sound and vision is structured
around activity, and participation engenders new forms of subjective experience
in an infinite field of digital choice, but participation within algorithmic culture
is increasingly managed according to the dictates of commerce.


Conclusion

  Video game sound and music is a convergence of both European and American
twentieth century avant-garde compositional approaches — the parametric and
the interactive — transformed as an aspect of commercial culture. The invention
and refinement of serial composition by European composers during the twentieth
century framed music as a technological material that could be precisely measured


41
   Striphas, p. 407.
42
   Sophia Ciocca, ‘How Does Spotify Know You So Well?’, https://hackernoon.com/spotifys-discover-
weekly-how-machine-learning-finds-your-new-music-19a41ab76efe [accessed 22 December 2018].
43
   Alex Giamas, ‘How YouTube’s Recommendation Algorithm Works’, https://www.infoq.com/
news/2016/09/How-YouTube-Recommendation-Works [accessed 22 December 2018].
44
   Striphas, p. 406.

                                                                                              73
                                       Donal Fullam


and manipulated as sets of parameters. Beginning with Schönberg’s twelve-tone
music and culminating in integral serialism, composers created methods that could
generate music by treating notes as autonomous parameters within an algorithmic
system. To automate composition, music had to be expressed as information,
so composers began to more fully quantize musical values — from a system of
notes, to a system of every available acoustic attribute. The formal concepts of
the European avant-garde are now largely abandoned but serial composers were
harbingers of the technical and digital transformation of music into an automated
multimedia component. The hardware, theoretical concepts and organizational
procedures that engrossed composers and technicians during the first half of
the twentieth century were continually iterated on and developed through the
proceeding decades, and the conception and implementation of music as a system
of parameters is now more relevant than ever. Contemporary musical practice of
all forms is totally saturated by technology that works through the organization
of audio into parametric systems. Musicians, composers, sound designers and
technicians who use any digital instruments, effects, recording or reproduction
technology are engaging with the parametric organization of musical sound. The
quantization of music as a system of acoustic parameters allows it to be used as
a component within many forms of multimedia today, especially in video games.
In games, the parametric management of acoustic elements has reached a level of
sophistication that Schönberg or Stockhausen could barely have imagined.
   Composers associated with Cage and the American avant-garde took seriously
Marshall McLuhan’s theory that the medium, rather than the content, is the
message. They wanted to manufacture subjectivity by presenting randomized,
automated systems to individuate experience, creating the conditions for audience
members to see themselves as aesthetic and political actors within a spectacle.
Following Cage, American composers and media artists combined music, sound
and visuals within spatial multimedia assemblies to grant audience members a
range of choice, and a sense of participation. Offering choices creates a sense of
shared production, increasing the perceived value of the experience. By sharing
the task of production, alienation from artistic or entertainment experiences
can be diminished, as participation includes audiences as co-producers. This is
the theory, but in practice the distance between artists and audiences can also
increase, the connection occluded by interactive frameworks and managerial
hierarchies that are mediated through algorithmic processes. The impulse to treat
music according to algorithmic systems can be detected throughout the history
of European music — in medieval isorhythmic composition,45 twentieth century
musical dice games,46 and more recently in serial and avant-garde techniques,
but it is particularly apparent within twentieth century and contemporary


45
   Keith Muscutt, ‘Composing with Algorithms: An Interview with David Cope’, Computer Music
Journal, 31.3, (Fall, 2007), p. 10.
46
   Stephen A. Hedges, ‘Dice Music in the Eighteenth Century’, Music & Letters, 59. 2 (April
1980), 180-187.

74
               The Avant-Garde Roots of Video Game Music and Algorithmic Culture


media culture. The intertwined histories of European and American avant-
garde composition can reveal the underlying threads that constitute the warp
and weave of our current algorithmic audio culture. The serialists parametric
organization and algorithmic control of music converged with the interactive
multimedia of the American avant-garde, forming the logic of music and sound
in new media towards the end of the millennium.
   Rather than a direct line of descent between video game and avant-garde
composition, the similarities and differences between approaches can be
characterized according to related musical tools used to facilitate the application
of choice within an emerging algorithmic culture. In the democratic surrounds,
freedom was presented as choice within frameworks designed by specialists,
rather than a dialectic between participants exercising equal power. Participants
had no access to Cage’s algorithmic methods of musical organization, but could
instead choose from within a range set by the composer. Still, Cage was usually
physically present during performances and the public nature of the surrounds
did present levels of media participation that were not usually available. Video
games present infinite fields of possible choices and music that responds to player
behavior, but the interaction between players and composers, or consumers and
producers, is mediated through the black box of machine and code. Avant-garde
experiments pointed towards an increasingly public level of engagement but
the current model of algorithmic culture seems to be instigating the ‘gradual
abandonment of culture’s publicness and the emergence of a strange new breed
of elite culture purporting to be its opposite.’47 We are surrounded by sound
and vision but not in the way that twentieth century avant-garde composers and
artists envisioned. The automatism48 they sought is ubiquitous, but as an element
of consumer culture it becomes a method of social organization that structures
consumer choice, rather than an escape from hierarchy towards democratic
participation.




47
     Striphas, p. 1.
48
     Taruskin, p. 37.

                                                                                   75
