Modes of Media in Digital Performance

In the late 1980s and early 1990s, theatre and performance artists such as George Coates, the Wooster Group, Troika Ranch, Mark Reaney, and Laurie Anderson began to explore the use of emerging digital technologies to expand the possibilities of live performance. Those who were scenic designers, such as Reaney, who established i.e.VR at the University of Kansas, conceived of digital media primarily as a way to enhance theatrical illusion through virtual scenery. By contrast, performers and directors, like Anderson and the Wooster Group’s Elizabeth LeCompte, exploited technology’s ability to probe the nature and significance of live performance itself.

These days, digital media is widely accessible to theatre practitioners and has become commonplace, which has led to higher and more consistent standards for technical proficiency. At the same time, its proliferation runs the risk of dulling its experimental and subversive edge—rendering it just another tool to create work that conforms to pre-existing theatrical conventions and audience expectations. At this juncture in the history of performance, as the novelty of media-enhanced performance is fading and a narrowing set of conventions is consolidating around media design practices (much like what happened during the early development of cinema as a narrative artform), it is important to remind ourselves of the many radically different ways that artists can and have incorporated media into their performance work.

Over the past twenty-five years as a director, installation artist, and teacher of digital performance, I have developed and refined a taxonomy of roles that media plays in live performance. The current version of the taxonomy distinguishes six basic modes: scene, prop, actor, costume, audience, and mirror. While some performances consistently call upon media to serve just one of these functions, many use media in a multiplicity of ways, sometimes simultaneously. Distinguishing clearly among these modes can help artists to resist calcifying expectations and clichés, move out of their comfort zone, and identify new creative possibilities. The taxonomy has proven invaluable in my teaching, enabling my students to define the use of media with more clarity and precision both in their own work and that of others.

Media as scene

Media functions as scenery to the extent that it defines the environment within which the performer is acting. For example, the physical set for The Animals and Children Took to the Streets (2011), produced by 1927, consisted simply of three white screens with small cutout windows. Meticulous animations in an expressionistic, hand-drawn style filled the screens, depicting a series of constantly changing environments, including the exterior of a tenement house covered with crawling cockroaches, domestic interiors, street scenes, a police station, and a park.

When the media functions as virtual scenery, the performer interacts with it by navigating through it. The spectator will interpret a change in the image as a jump to a new location and will interpret a shift in the perspective within the image as an indication of movement within the scene. In this way, the performer’s virtual or fictional location is anchored to the media.

For example, when Peter Pan and the children fly to Neverland in Threesixty Entertainment’s Peter Pan 360 (2009), designed by William Dudley, the performers are suspended in front of an enormous, 360-degree 3D animation of London sweeping rapidly beneath and around them. This effect is reminiscent of the piece de resistance of Lincoln J. Carter’s 1898 production of Chattanooga, one of the first uses of filmed scenery in the United States, in which two actors fought aboard a stationary locomotive while the filmed scenery raced toward the audience behind them.


Media as prop

At several points in The Animals and Children Took to the Streets, a live performer made sweeping gestures, which led to animated puffs of dust appearing on the screen. At such moments, the media is not simply providing an environment within which the character is acting, but it constitutes a virtual prop with which the character interacts. Any object in a virtual scene will function as a prop as soon as a performer uses it. When the media functions as a prop, the performer/character is the source of agency, manipulating, controlling, or triggering changes. The media’s relationship to the performer is reactive and instrumental.

For example, in a vignette from Golan Levin and Zach Lieberman’s Messa Di Voce called “Pitchpaint” (2003), two performers draw on the screen with their voices; when they sing descending notes, the lines on the screen curl clockwise, and when they sing ascending notes, they curl counterclockwise. Electronic musicians have created a wide variety of digital instruments that allow them to trigger synthesized or sampled sounds and images interactively. An early, well-known example among electronic musicians is a custom-designed instrument called lady’s glove, first created in 1991, which Laetitia Sonami embedded with an array of sensors, allowing her to trigger various sonic events—tones, voice samples, and sometimes images—by moving her fingers. Such digital instruments blur the boundaries between composing and performing.

three actors "flying" onstage

Peter Pan 360 by Threesixty Entertainment. Designed by William Dudley.

Media as actor

Media functions as an actor when the live performer relates to it as an autonomous agent—that is, as a subject in its own right with sentience and volition. The relationship between the live performer and the virtual actor is responsive and interactive. In 1914, vaudeville performer and animation pioneer Winsor McCay performed live with a projected animation of Gertie the Dinosaur. At one point, McCay scolds Gertie, who responds with tears. To make amends, McCay tosses an apple at the screen—in reality a red cardboard prop that he had clandestinely stashed into his coat pocket—and into Gertie’s mouth. More recently, in The Animals and Children Took to the Street, one of the primary characters is a girl who appears exclusively as an animation on screen and who interacts with various live actors throughout the play.

In both of these cases, the animated sequences are pre-rendered and the live performers must adapt their performances to it, creating the illusion that the media is responsive. In other cases, however, a mediatized actor might respond dynamically to a live performer in real-time. In my own production of The Tempest at the University of Georgia in 2000, Ariel was a computer animation controlled in real-time using motion capture (to my knowledge the first use of real-time motion capture in live theatre); the mo-cap performer was visible on stage but never—until the very end of the play—acknowledged by any other actor.

Another way a truly interactive dynamic can be created is telematically, when a remote performer on screen interacts in real-time with a live one on stage. For example, during each performance of the Builders Association’s Continuous City (2008), the actor Rizwan Mirza, playing a character named J.V., engaged in improvised video chats with members of his real family in London and Virginia.

Media need not be humanoid to function as a virtual actor. In 1997, Flavia Sparacino, Glorianna Davenport, and Sandy Pentland, a team of researchers from MIT, presented a piece called “Improvisational TheaterSpace” in which a live performer interacted with text projected on the stage. According to Sparacino, Davenport, and Pentland, “the text is just like another actor—able to understand and synchronize its performance to its human partner’s gestures, postures, tone of voice, and words.” In 1988, jazz trombonist and digital composer George Lewis created a computer program called Voyager that improvises with live musicians by detecting melodic and rhythmic patterns in real-time performance and generates its own original music in response. Lewis describes Voyager as “a nonhierarchical, improvisational, subject-subject model of discourse, rather than a stimulus/response setup,” clearly articulating the distinction between media as actor and as prop or instrument.

Media need not be humanoid to function as a virtual actor.

Media as costume

Costumes in conventional theatre move in conjunction with performers, merging with them to define the character’s identity. In the same way, media that functions as a virtual costume is coextensive with the performer. For example, in D.A.V.E. (1999), Klaus Obermaier and Chris Haring projected video onto a man’s face and shirtless chest to give him a woman’s breasts and face and bulging frog-like eyes, and to create the illusion that he was squashing his face between his fists and peeling the skin off his arm, shoulder, and chest.

In some of the most sophisticated examples of media-as-costume, the media object and the live performer conjoin to create a cyborg identity. For example, at several points during the Wooster Group’s To You, The Birdie! (2002), a video monitor slides in front of a live performer’s head or legs to create an uncanny image of a whole body composed of half flesh and half video.

The costume function, as I am defining it, ranges from the use of media to enhance conventional costumes—such as by projecting images onto a fabric costume or embedding LEDs within it—to cyborg technology that conjoins multiple subjects, some live, some mediated, to produce new hybrid identities. In all of these cases, the media and performer(s) merge together in the imagination of the spectator to constitute a single fictional or virtual subject.

Media as mirror

David Rokeby, creator of the pioneering interactive installation Very Nervous System (1986–1990), proclaims that “an interactive technology is a medium through which we communicate with ourselves… a mirror. The medium not only reflects back, but also refracts what it is given; what is returned is ourselves, transformed and processed.”

Media functions as a mirror when it reflects the performer’s actions. The most banal example is the kind of live-feed video used to help spectators see close-ups of musical performances and athletic events in huge stadium venues—though, as the Wooster Group has demonstrated in a series of productions over the past fifty years, such technically straightforward live-feed video can be used to highly sophisticated aesthetic effect.

To function as a media mirror, however, the media need not visually resemble the performer at all. What matters is that the spectators perceive a continuous, reflective connection between the performer and the media object, and as a result can “read” the performer’s actions through the media. Synesthetic media, which translates a performance from one modality to another modality, exemplifies this mode: for example, a music visualization algorithm might automatically generate shapes that grow and shrink as the volume increases and decreases, that fill with warmer colors as the pitch rises and cooler colors as it descends, and/or that become rounder when the notes are sustained and more jagged when the notes are staccato.

A media mirror has no agency of its own, as a media actor does, but simply follows the performer’s actions. It does not merge with the performers, as costume media does, but runs in parallel with them. And it is not subject to the performer’s intentional control, as a media prop is. A media mirror, such as a live-feed video, reflects the performer regardless of the performer’s intentions, desires, or even, in the case of covert surveillance, knowledge. A performer’s relationship to his or her media reflection is fundamentally passive.

There is not and should not be a one-size-fits-all approach to incorporating media into performance.

Media as audience

Finally, the media can function as a virtual audience by responding to the live performance. While media-as-actor engages in a two-way interaction with the performers, media-as-audience engages in a one-way interaction. To adopt a term from film theory, it is non-diegetic, functioning out of the live performer’s frame. A basic example of media-as-audience is musical underscoring in the tradition of nineteenth century melodrama, which has become a ubiquitous convention in film. The same function can be fulfilled visually with washes of color and abstract images that convey mood. These examples of media-as-audience are purely affective, serving to model and manipulate the audience’s emotional responses to the performance.

Media-as-audience, however, can and often does assume a cognitive dimension as well, commenting on the live performance, as Erwin Piscator did in his productions in the 1920s that used analogue media such as documentary film footage. The Almeida Theatre’s 1997 production of The Cenci, with media created by Studio Azzurro, is one of many examples of a production that used digital media primarily as commentary on the live performance, with the projected images acting “as a visual counterpoint that does not attempt to explain the text but rather to open it up to different interpretations and comparisons.”

Digital Media in the Production Process

Distinguishing between these six very different modes of media helps to explain why media is often so difficult to integrate smoothly and effectively into the conventional theatrical production process. Most theatre companies clearly delineate responsibilities and timelines for each member of the production team, including the director and the scenic, costume, lighting, and prop designers. It is often unclear how media should fit into this process. In some cases, media is construed as an aspect of the lighting design. In others, it falls under the purview of the scenic designer, or of a separate projection designer who works in conjunction with the scenic designer. In other cases, media is introduced by the costume designer and has little or no impact on the scenic design.

In all of these instances, the performers typically do not interact with the media until technical rehearsals. Sometimes, however, directors conceive of media as an integral element of the production’s concept and content, and either create the media themselves or work intimately with a media designer throughout the process. In such cases, the director may try to integrate media as early as possible into rehearsals, fashioning and modifying it along with the work of the actors.

There is not and should not be a one-size-fits-all approach to incorporating media into performance. Digital media is protean and dynamic, and to exploit the prodigious and barely tapped potential of its use in performance demands a supple and creative approach—one that is adapted to the artists’ goals and the function of the media in each particular work.

Bookmark this page

Log in to add a bookmark




Add Comment
Newest First

Hi David, nice condensation of your article in the "Performance and Media" volume. It strikes me that in each of your categories, digital media functions as a surrogate for physical-world theater elements, even if it provides unique capacities. One can say the same of plays produced and performed in a wholly digital environment: physical-world theater remains a reference point, an ever-present allusion. What's not captured, then, is performance that doesn't allude to physical-world counterparts, even to the performers themselves -- what one might call, for want of a better phrase, "born digital" performance. I have in mind a performance genre which few of its participants call theater, in fact very likely they would object to the phrase and deny that they're doing theater at all (and likewise, virtually no theater scholars recognize it, and some find the notion utterly objectionable), but it is theater nonetheless: online role-playing. I'm referring specifically to an online activity in which players create fictional characters who interact in largely or wholly improvised scenes (an activity that does not inherently involve battling monsters and the like). By calling online role-playing "born digital," I don't mean it has no history or physical-world analogues: it has parallels in live action role-playing (with which it is roughly coeval), and tabletop RP games before that. But online RP doesn't retain those physical-world genres by being their surrogate. Many online role-players would even be horrified by the suggestion they take their activity offline. It lives and dies in a virtual world. Thus while from an analytical perspective one can identify material fitting your categories in online RP, their sociocultural register is quite different, possibly complicating the categories themselves. What this brings out is that your categories are essentially defined only in technological terms; but technologies are always created and utilized in society, and obtain their meaning there. Integrating that fact into your analysis would I think provide important qualitative nuances. (I've published an article about online RP as theater -- much constrained by publication requirements, sadly -- and it's a fulcrum for the book I'm currently writing.)

This is great, David. Thank you for posting.
There’s a lot to discuss here. Two things jump out in particular: 1) your suggestion that something important has changed with digital performance—an assessment that Steve Dixon made rather differently years ago in his book, and 2) the taxonomy itself.
I'll limit myself to briefly commenting on the second point. Let me say first that I imagine your taxonomy would be extremely useful to use in teaching young designers, giving them a language to think through different approaches.  
My reaction to the taxonomy was in some ways similar to Philip's. What does it do, I wondered, when we borrow familiar terms from theatre to describe approaches to media design for performance? 
Of course, categories and taxonomies are never innocent or transparent. They always produce and delimit the possible and the thinkable. They’re always ideological. That your taxonomy does this isn't a flaw. It's just the way categories work. 
So while these categories may be handy in the classroom, they make me curious about where they compel the designer to focus their attention, and what possibilities they foreclose. 
Your taxonomy remediates theatre, drawing terms and concepts from performance and translating them into a new medium. And that’s cool. So many digital media terms are remediations of other contexts—desktop, folders, files, cut and paste, scroll, zoom, etc. 
Early film functioned within a formal language imported from theatre, a language that limited it until folks like Eisenstein and others could develop/discover a language “native” to film. 
What’s interesting to me is thinking through functions of digital media that are not “theatrically native,” but rather specific (or more specific) to media. Is there a sense where media in the theatre functions simply as “screen”? What are the subcategories of that? What can media help us do that is unthinkable in a theatrical context?
In recent years, I’ve worked with planetarium software. It’s basically VR software where the virtual world is the known universe. But rarely did we use this in a locative sense, as a “place-where-the-characters-are.” More typically it functioned as a visual accompaniment, not a literal location for the action. I might compare it to musical underscoring as opposed to diagetic sound.  
Recently Apple held its World Wide Developers Conference. I think of how Tim Cook (like Steve Jobs before him) uses the screen onstage. It’s not scenery or prop or mirror or agent. What is it? Is a theatrical metaphor appropriate? 

Thanks for this comprehensive article! I found your concept of media-as-actor intriguing. As a puppeteer, I tend to think about this realm as a category of digital puppetry. For me, digital puppetry encapsulates any time that a performer (or puppeteer-technician, as some see folks working behind the scenes) manipulates an image or object as though it is alive. That would possibly include some variants of performing text (if the words and letters had intention and responsiveness to action rather than just aesthetically pleasing choreography) as well as motion-captured performance. You might find some theoretical writings about puppetry and technology interesting in this regard, as they introduce corollary concepts related to things like the distance and complexity of control between the performer/animator and performing object, and questions of authorship related to performance (is it the designer of the media? the maker of the system? the person creating the physical performance being mapped?). I definitely relate to what you're saying about where the projection designer fits into the process. On big shows, I have advocated for more time with actors earlier in the process so that they may explore and co-create with puppets and performing objects. More and more, I think other design elements need to be part of rehearsal explorations earlier in the process. 

Hi David!

I don't think you and I have any real disagreement on the first question, so I'll just add one little quibble with your saying that cinematic experimentation was driven underground. To take an example of interest to both of us, motion capture and CGI, experimentation and innovation in terms both of the technology itself and how it can be used has happened very much above ground. This is in part because experimentation and innovation in this area are very capital-intensive. One of the problems I've had for a long time with the academic discourse around digital media in performance is that it reflects the pro-avant-garde prejudice of, for example, performance studies. Correct me if I'm wrong, but I have not seen much academic writing on Cirque du Soleil's use of digital media. I have read some stuff in trade journals about it, and it's pretty impressive. But theater and performance studies retain a high-art bias that makes high-end entertainment like the Cirque suspect except as an object of critique. 

With all due respect (not just a cliché, I assure you!), I don't think you actually answered my second question. I get that there are benefits to using familiar language. But I still think that using conventional theatrical terms (plus a few others, admittedly) undermines your purpose of advocating for the innovative and edgy use of the technologies in question. For instance, saying that performing technologies can be understood as actors makes them seem comfortably familiar in a way that negates any challenge they may present to our understanding of what performance and performers are or can be, partly by implying that interacting with such entities takes no adjustment on the part of the human actors. I also note that your examples all focus on instances in which the technologies are "just like another actor" and don't discuss instances in which they are not, which is necessary if you want to use a discussion of digital tech as a means of expanding the category of performer. Maybe this is all stuff you pursue in the longer version; I'm just addressing what's here!

Thanks for this very clear and useful breakdown of uses of digital media in performance. 

I have two related comment/questions. My first is, do you have an empirical basis for arguing that the proliferation of digital media in performance which has rendered it conventional in many ways has actually had a deleterious impact on its more experimental and subversive uses? I don't see why the mainstreaming of digital technology would necessarily mean that non-mainstream artists can no longer use it in those ways. I agree that digital media has become "just another tool to create work that conforms to pre-existing theatrical conventions and audience expectations," but how does this prevent it from being used in other ways?

My second also relates to this quotation. In light of your concern about the uses of digital media conforming to existing conventions and expectations, why do you use highly traditional and conventional theatrical terms in your taxonomy? Isn't the effect of this precisely to assimilate the uses of digital technology you describe to the existing vocabulary of theater?

I'd love to hear your responses and I hope others will weigh in as well to create the dialogue you wish to stimulate.



Thank you for these useful and stimulating comments!

With regard to the first point about the mainstreaming of digital technologies: I agree that the growing use of digital media in mainstream theatre does not preclude experimental artists from continuing to push the aesthetic boundaries and interrogate the social and political implications of technology—just as the emergence of cinema as a mainstream medium for popular entertainment didn't put an end to avant-garde filmmaking. There's still a lot of subversive and exciting work being done with digital technology, and I'm confident that will continue. What I say in my essay is that the proliferation of digital media in performance "runs the risk of dulling its experimental and subversive edge." My worry, in part, stems from my own incredibly good fortune in having had, over the past twenty-five years, free reign as a director to takes risks with my use of digital technology and experiment with both my process and goals. My freedom was enhanced by the fact that my theatrical collaborators came to the process without any expectations about how media was "supposed" to work in theatre, and shared with me the sense that we were embarking on a new adventure. Now that major theatre programs are offering advanced degrees in projection design, I worry that the use of media will become compartmentalized, and experimentation with media will be driven underground (as was the case with film). But I hope I'm wrong!

With regard to your second point about my nomenclature: I must admit that I had exactly your worry when I settled on these terms. Though I hope this taxonomy comes off as straightforward and intuitive, the process of developing it was slow and painstaking, and I didn't hit on the theatrical terminology until very late in that process. In my initial attempt to distinguish among the ways that live performers interact with media objects, I was using the terms "Autonomous, Instrumental, Navigational, Coextensive, and Reflective." Only after I had formulated these categories did I have the epiphany that most of these concepts mapped neatly onto theatrical concepts: respectively, actor, prop, scene, and costume. (There is no clear theatrical analogue for the "reflective" mode, which I now call "mirror" — and I didn't add the sixth category, audience, until later.) The advantage of the theatrical terms is that they are much more accessible and mnemonic than the initial, much more abstract terms. But the theatrical terms are just metaphors for the more general interactive categories. For example, props are instrumental, but many things that are instrumental—such as, to take the most obvious example, musical instruments—are not literally "props." I do find it fascinating that the categories I developed almost all ended up having such clear theatrical analogues. I attribute that to the wisdom that philosophers like Wittgenstein and J.L. Austin attributed to ordinary language. As Austin said, "our ordinary words are much subtler in their uses, and mark many more distinctions, than philosophers have realized."

Let me take this opportunity to note that I first developed this taxonomy for the book Performance and Media: Taxonomies for a Changing Field, which I co-authored with Sarah Bay-Cheng and Jennifer Parker-Starbuck (University of Michigan Press, 2015). That version of the taxonomy is much more extensive than the one I discuss here, incorporating the variables of time, space, and perspective — but I added the sixth mode, audience, after the book was published.