A Research Overview of Mobile Projected User Interfaces
---
paper_title: Projector phone use: practices and social implications

paper_content:

Phones with integrated pico projectors are starting to be marketed as devices for business presentations and media viewing, and researchers are beginning to design projection-specific applications and interaction techniques to explore a broader array of possible uses. To begin to document how people use projector phones outside the laboratory, we present the results of a 4-week exploratory field study of naturalistic use of commodity projector phones. In our analysis, we consider how context, such as group size, relationships, and locale, influences projector phone use. A key observation is that users can readily exploit the new facilities of these devices to author interesting effects by employing representational techniques such as superimposition, scaling, translation, and motion. Thus, even the "basic" projector phone platform affords novel interaction modalities. Finally, we discuss the social implications of projector phone use for privacy and control, extrapolating from our observations to envision a future in which these devices are ubiquitous. With ubiquity, projector phone use may become problematic in public settings, motivating new rules of etiquette and perhaps laws, yet it may also engender new forms of creative expression.
---
paper_title: A TAXONOMY OF MIXED REALITY VISUAL DISPLAYS

paper_content:

Paul Milgram received the B.A.Sc. degree from the University of Toronto in 1970, the M.S.E.E. degree from the Technion (Israel) in 1973 and the Ph.D. degree from the University of Toronto in 1980. From 1980 to 1982 he was a ZWO Visiting Scientist and a NATO Postdoctoral in the Netherlands, researching automobile driving behaviour. From 1982 to 1984 he was a Senior Research Engineer in Human Engineering at the National Aerospace Laboratory (NLR) in Amsterdam, where his work involved the modelling of aircraft flight crew activity, advanced display concepts and control loops with human operators in space teleoperation. Since 1986 he has worked at the Industrial Engineering Department of the University of Toronto, where he is currently an Associate Professor and Coordinator of the Human Factors Engineering group. He is also cross appointed to the Department of Psychology. In 1993-94 he was an invited researcher at the ATR Communication Systems Research Laboratories, in Kyoto, Japan. His research interests include display and control issues in telerobotics and virtual environments, stereoscopic video and computer graphics, cognitive engineering, and human factors issues in medicine. He is also President of Translucent Technologies, a company which produces "Plato" liquid crystal visual occlusion spectacles (of which he is the inventor), for visual and psychomotor research.
---
paper_title: Theme issue on personal projection

paper_content:

Mobile projection interfaces are no longer fiction. Integrated pico projectors in mobile and wearable devices could make mobile projection ubiquitous within the next few years and create commonplace practices where walls, desks, floors, and t-shirts act as projection surfaces for these kinds of new devices. Several projector phones— mobile phones with built-in projectors—are commercially available, have been demonstrated or are announced [5]. It is expected that such projector phones will be integrated in many of the next generation mobile handsets. Furthermore, we have seen the first research toward the integration of pico projectors and cameras into various wearable systems, such as stand-alone personal projectors [4], watches [2], pendants [3], or headsets [6], leading to new form factors, interaction techniques, and applications. These mobile projection units, often equipped with a camera or other additional sensors for tracking input, have a great potential to overcome the limited input and output capabilities of mobile and wearable devices. They can be expected to have a significant impact on the cultural usage habits of content consumption and media sharing. This makes mobile projectors an interesting and important platform for research in the area of Human–Computer Interaction. Current research in this field mainly focuses on the design, development, and evaluation of novel interaction techniques and applications, on the study of how this technology is used in realistic contexts and how it affects the usage of mobile and wearable devices. In addition, new sensing and tracking methods are being investigated in order to support novel ways of interaction and emerging applications. Therefore, we are still at the beginning of understanding and addressing the multifaceted challenges of this emerging research area of Human–Computer Interaction. Of course, mobile projectors have been used in various other research disciplines for a couple of years. As the first paper of this special issue vividly shows, their history even goes back to the seventeenth century, where ‘‘magic lanterns’’ provided early forms of mobile projection. More recently, previous research has focused on technical aspects, such as hardware construction, optics, projection geometry [1], energy consumption, and interaction with projected multi-scale user interfaces [4]. Through continuous miniaturization, these devices have now reached form factors that allow us to focus on HCI research in the context of realistic pervasive environments, such as mobile settings. This theme issue on personal projection is a follow-up of the successful Ubiprojection 2010 workshop being held in conjunction with the Pervasive 2010 conference in E. Rukzio (&) University of Duisburg-Essen, Essen, Germany e-mail: Enrico.Rukzio@uni-due.de
---
paper_title: Personal Projectors for Pervasive Computing

paper_content:

Projectors are pervasive as infrastructure devices for large displays but are now also becoming available in small form factors that afford mobile personal use. This article surveys the interaction space of "projectors on the move" and reviews input and output concepts, underlying sensing challenges, and emerging applications.
---
paper_title: Toolkit support for interactive projected displays

paper_content:

This paper presents a software toolkit designed to enable the rapid development of multimedia-rich, multi-touch enabled, and interactive projection-based displays. For instance: door displays, floor displays, wall displays, and interactive tables. Despite recent technological advances and the commercialization of hardware required to achieve this at a relatively low cost, creating and deploying such displays remains a difficult task, even for those with the essential technical skills and experience. We assert that greater accessibility of toolkits like the one presented in this paper will reduce these barriers and allow people (not necessarily from the ubiquitous computing domain) to apply the technology to their own fields. To assess this toolkit's suitability for this role, we present a system and usability evaluation. We observed that participants were able to quickly create their own novel display deployments. Our findings offer insights for potential toolkit users and those considering how to write programs for future ubiquitous projected display environments.
---
paper_title: ProjectorKit: easing rapid prototyping of interactive applications for mobile projectors

paper_content:

Researchers have developed interaction concepts based on mobile projectors. Yet pursuing work in this area - particularly in building projector-based interactions techniques within an application - is cumbersome and time-consuming. To mitigate this problem, we contribute ProjectorKit, a flexible open-source toolkit that eases rapid prototyping mobile projector interaction techniques.
---
paper_title: Two Unusual Projection Spaces

paper_content:

Two immersive projection environments, both unconventional, both exploring different methods of 3D and panoramic imaging, and both produced as art installations, are described. Displacements (1980-1984) recreated an interior living space using a panoramic motion picture method and relief projection. Be Now Here (1995-1997) recreated outdoor public plazas using a panoramic motion picture method, stereopsis, and four channel sound. Both installations were unusual in that no intentions existed for anything more general or useful than the installations themselves as individual artworks.
---
paper_title: Using mobile projection to support guitar learning

paper_content:

The guitar is one of the most widespread instruments amongst autodidacts, but even though a huge amount of learning material exists, it is still hard to learn especially without a guitar teacher. In this paper we propose an Augmented Reality concept that assists guitar students mastering their instrument using a mobile projector. With the projector mounted onto the headstock of the guitar, it is possible to project instructions directly onto the strings of the guitar. With that the user is easily able to realize where the fingers have to be placed on the fretboard (fingering) to play a certain chord or a tone sequence correctly.
---
paper_title: LuminAR: portable robotic augmented reality interface design and prototype

paper_content:

In this paper we introduce LuminAR: a prototype for a new portable and compact projector-camera system designed to use the traditional incandescent bulb interface as a power source, and a robotic desk lamp that carries it, enabling it with dynamic motion capabilities. We are exploring how the LuminAR system embodied in a familiar form factor of a classic Angle Poise lamp may evolve into a new class of robotic, digital information devices.
---
paper_title: FACT: fine-grained cross-media interaction with documents via a portable hybrid paper-laptop interface

paper_content:

FACT is an interactive paper system for fine-grained interaction with documents across the boundary between paper and computers. It consists of a small camera-projector unit, a laptop, and ordinary paper documents. With the camera-projector unit pointing to a paper document, the system allows a user to issue pen gestures on the paper document for selecting fine-grained content and applying various digital functions. For example, the user can choose individual words, symbols, figures, and arbitrary regions for keyword search, copy and paste, web search, and remote sharing. FACT thus enables a computer-like user experience on paper. This paper interaction can be integrated with laptop interaction for cross-media manipulations on multiple documents and views. We present the infrastructure, supporting techniques and interaction design, and demonstrate the feasibility via a quantitative experiment. We also propose applications such as document manipulation, map navigation and remote collaboration.
---
paper_title: Personal Projectors for Pervasive Computing

paper_content:

Projectors are pervasive as infrastructure devices for large displays but are now also becoming available in small form factors that afford mobile personal use. This article surveys the interaction space of "projectors on the move" and reviews input and output concepts, underlying sensing challenges, and emerging applications.
---
paper_title: Penbook: bringing pen+paper interaction to a tablet device to facilitate paper-based workflows in the hospital domain

paper_content:

In many contexts, pen and paper are the ideal option for collecting information despite the pervasiveness of mobile devices. Reasons include the unconstrained nature of sketching or handwriting, as well as the tactility of moving a pen over a paper that supports very fine granular control of the pen. In particular in the context of hospitals, many writing and note taking tasks are still performed using pen and paper. However, often this requires time-consuming transcription into digital form for the sake of documentation. We present Penbook - a system providing a touch screen together with a built-in projector integrated with a wireless pen and a projection screen augmented with Anoto paper. This allows using the pen to write or sketch digital information with light on the projection surface while having the distinct tactility of a pen moving over paper. The touch screen can be used in parallel with the projected information turning the tablet into a dual-display device. In this paper, we present the Penbook concept, detail specific applications in a hospital context, and present a prototype implementation of Penbook.
---
paper_title: Bonfire: a nomadic system for hybrid laptop-tabletop interaction

paper_content:

We present Bonfire, a self-contained mobile computing system that uses two laptop-mounted laser micro-projectors to project an interactive display space to either side of a laptop keyboard. Coupled with each micro-projector is a camera to enable hand gesture tracking, object recognition, and information transfer within the projected space. Thus, Bonfire is neither a pure laptop system nor a pure tabletop system, but an integration of the two into one new nomadic computing platform. This integration (1) enables observing the periphery and responding appropriately, e.g., to the casual placement of objects within its field of view, (2) enables integration between physical and digital objects via computer vision, (3) provides a horizontal surface in tandem with the usual vertical laptop display, allowing direct pointing and gestures, and (4) enlarges the input/output space to enrich existing applications. We describe Bonfire's architecture, and offer scenarios that highlight Bonfire's advantages. We also include lessons learned and insights for further development and use.
---
paper_title: Using mobile projection to support guitar learning

paper_content:

The guitar is one of the most widespread instruments amongst autodidacts, but even though a huge amount of learning material exists, it is still hard to learn especially without a guitar teacher. In this paper we propose an Augmented Reality concept that assists guitar students mastering their instrument using a mobile projector. With the projector mounted onto the headstock of the guitar, it is possible to project instructions directly onto the strings of the guitar. With that the user is easily able to realize where the fingers have to be placed on the fretboard (fingering) to play a certain chord or a tone sequence correctly.
---
paper_title: LuminAR: portable robotic augmented reality interface design and prototype

paper_content:

In this paper we introduce LuminAR: a prototype for a new portable and compact projector-camera system designed to use the traditional incandescent bulb interface as a power source, and a robotic desk lamp that carries it, enabling it with dynamic motion capabilities. We are exploring how the LuminAR system embodied in a familiar form factor of a classic Angle Poise lamp may evolve into a new class of robotic, digital information devices.
---
paper_title: FACT: fine-grained cross-media interaction with documents via a portable hybrid paper-laptop interface

paper_content:

FACT is an interactive paper system for fine-grained interaction with documents across the boundary between paper and computers. It consists of a small camera-projector unit, a laptop, and ordinary paper documents. With the camera-projector unit pointing to a paper document, the system allows a user to issue pen gestures on the paper document for selecting fine-grained content and applying various digital functions. For example, the user can choose individual words, symbols, figures, and arbitrary regions for keyword search, copy and paste, web search, and remote sharing. FACT thus enables a computer-like user experience on paper. This paper interaction can be integrated with laptop interaction for cross-media manipulations on multiple documents and views. We present the infrastructure, supporting techniques and interaction design, and demonstrate the feasibility via a quantitative experiment. We also propose applications such as document manipulation, map navigation and remote collaboration.
---
paper_title: A pre-history of handheld projector-based interaction

paper_content:

I present a pre-history of contemporary handheld projector-based interaction to inform the design of future interactive systems. I begin by documenting the two main types of pre-cinema handheld projection from Europe and Japan, the handheld magic lantern and the utsushi-e performance. I then present a summary of projection techniques used by performers when interacting with these devices. I situate these techniques within contemporary research and illustrate how they are being used and built upon with contemporary technology. Finally, I discuss how knowledge of pre-cinema handheld projection devices, techniques, and performance can inform the design of future handheld projector systems.
---
paper_title: Pico-ing into the future of mobile projection and contexts

paper_content:

Ten years ago, we were on the verge of having cameras built into our mobile phones, but knew very little about what to expect or how they would be used. Now we are faced with the same unknowns with mobile projector phones. This research seeks to explore how people will want to use such technology, how they will feel when using it, and what social effects we can expect to see. This paper describes our two-phase field investigation that uses a combination of methods to investigate how, when, and why mobile projections may be used. The first study used an experience sampling method to investigate responses to a range of different media types, and, for example, the choice of surfaces used in each case. The second study asked users to create video diary entries showing when, where, and why they would have wanted to project information. Together these studies provide complementary insights into the future use of mobile projector phones. Our results cover detailed responses to a range of media types from the first study, while the second identified which of the known mobile information needs were commonly recorded by participants. Both studies provide insights that may help shape the hardware, software, and interaction design of mobile projector phones as they become increasingly available.
---
paper_title: ShadowPuppets: supporting collocated interaction with mobile projector phones using hand shadows

paper_content:

Pico projectors attached to mobile phones allow users to view phone content using a large display. However, to provide input to projector phones, users have to look at the device, diverting their attention from the projected image. Additionally, other collocated users have no way of interacting with the device. We present ShadowPuppets, a system that supports collocated interaction with mobile projector phones. ShadowPuppets allows users to cast hand shadows as input to mobile projector phones. Most people understand how to cast hand shadows, which provide an easy input modality. Additionally, they implicitly support collocated usage, as nearby users can cast shadows as input and one user can see and understand another user's hand shadows. We describe the results of three user studies. The first study examines what hand shadows users expect will cause various effects. The second study looks at how users perceive hand shadows, examining what effects they think various hand shadows will cause. Finally, we present qualitative results from a study with our functional prototype and discuss design implications for systems using shadows as input. Our findings suggest that shadow input can provide a natural and intuitive way of interacting with projected interfaces and can support collocated collaboration.
---
paper_title: Projector Phone: A Study of Using Mobile Phones with Integrated Projector for Interaction with Maps

paper_content:

First working prototypes of mobile phones with integrated pico projectors have already been demonstrated and it is expected that such projector phones will be sold within the next three years. Applications that require interaction with large amounts of information will benefit from the large projection and its high resolution. This paper analyses the advantages and disadvantages of an integrated projector when interacting with maps, and discusses findings useful for the development of mobile applications for projector phones. We report in particular the implementation of an application that uses either the screen of the mobile phone, the projection or a combination of both. These three options were compared in a user study in which the participants had to perform three different tasks with each option. The results provide clear evidence for the positive aspects of using a built-in projector, but also show some negative aspects related to text input.
---
paper_title: Interacting with dynamically defined information spaces using a handheld projector and a pen

paper_content:

The recent trend towards miniaturization of projection technology indicates that handheld devices will soon have the ability to project information onto any surface, thus enabling interfaces that are not possible with current handhelds. We explore the design space of dynamically defining and interacting with multiple virtual information spaces embedded in a physical environment using a handheld projector and a passive pen tracked in 3D. We develop techniques for defining and interacting with these spaces, and explore usage scenarios.
---
paper_title: PenLight: combining a mobile projector and a digital pen for dynamic visual overlay

paper_content:

Digital pen systems, originally designed to digitize annotations made on physical paper, are evolving to permit a wider variety of applications. Although the type and quality of pen feedback (e.g., haptic, audio, and visual) have a huge impact on advancing the digital pen technology, dynamic visual feedback has yet to be fully investigated. In parallel, miniature projectors are an emerging technology with the potential to enhance visual feedback for small mobile computing devices. In this paper we present the PenLight system, which is a testbed to explore the interaction design space and its accompanying interaction techniques in a digital pen embedded with a spatially-aware miniature projector. Using our prototype, that simulates a miniature projection (via a standard video projector), we visually augment paper documents, giving the user immediate access to additional information and computational tools. We also show how virtual ink can be managed in single and multi-user environments to aid collaboration and data management. User evaluation with professional architects indicated promise of our proposed techniques and their potential utility in the paper-intensive domain of architecture.
---
paper_title: MouseLight: bimanual interactions on digital paper using a pen and a spatially-aware mobile projector

paper_content:

MouseLight is a spatially-aware standalone mobile projector with the form factor of a mouse that can be used in combination with digital pens on paper. By interacting with the projector and the pen bimanually, users can visualize and modify the virtually augmented contents on top of the paper, and seamlessly transition between virtual and physical information. We present a high fidelity hardware prototype of the system and demonstrate a set of novel interactions specifically tailored to the unique properties of MouseLight. MouseLight differentiates itself from related systems such as PenLight in two aspects. First, MouseLight presents a rich set of bimanual interactions inspired by the ToolGlass interaction metaphor, but applied to physical paper. Secondly, our system explores novel displaced interactions, that take advantage of the independent input and output that is spatially aware of the underneath paper. These properties enable users to issue remote commands such as copy and paste or search. We also report on a preliminary evaluation of the system which produced encouraging observations and feedback.
---
paper_title: SideBySide: ad-hoc multi-user interaction with handheld projectors

paper_content:

We introduce SideBySide, a system designed for ad-hoc multi-user interaction with handheld projectors. SideBySide uses device-mounted cameras and hybrid visible/infrared light projectors to track multiple independent projected images in relation to one another. This is accomplished by projecting invisible fiducial markers in the near-infrared spectrum. Our system is completely self-contained and can be deployed as a handheld device without instrumentation of the environment. We present the design and implementation of our system including a hybrid handheld projector to project visible and infrared light, and techniques for tracking projected fiducial markers that move and overlap. We introduce a range of example applications that demonstrate the applicability of our system to real-world scenarios such as mobile content exchange, gaming, and education.
---
paper_title: Marauders light: replacing the wand with a mobile camera projector unit

paper_content:

Classic paper-based maps provide high-resolution, large-scale information and are ubiquitous in larger cities and even outdoors. In this paper we present a combination of their advantages with a mobile camera projector unit to create a new mobile and intuitive buddy finder interface. Current friend or buddy finder systems on mobile phones such as Google Latitude suffer from the small screen estate and display size of mobile devices. To overcome this problem we use a lightweight mobile camera projector unit to augment the paper map with a projected overlay of the buddies' positions. By using digital and geo-referenced representations of public maps no extra preperation for the tracking is needed. The idea is presented by enhancing Google Latitude, which allows users to browse the positions of their friends, to project the positions directly on the paper map without the cumbersome panning and zooming of a digital map on a small display.
---
paper_title: Map torchlight: a mobile augmented reality camera projector unit

paper_content:

The advantages of paper-based maps have been utilized in the field of mobile Augmented Reality (AR) in the last few years. Traditional paper-based maps provide high-resolution, large-scale information with zero power consumption. There are numerous implementations of magic lens interfaces that combine high-resolution paper maps with dynamic handheld displays. From an HCI perspective, the main challenge of magic lens interfaces is that users have to switch their attention between the magic lens and the information in the background. In this paper, we attempt to overcome this problem by using a lightweight mobile camera projector unit to augment the paper map directly with additional information. The "Map Torchlight" is tracked over a paper map and can precisely highlight points of interest, streets, and areas to give directions or other guidance for interacting with the map.
---
paper_title: Motionbeam: a metaphor for character interaction with handheld projectors

paper_content:

We present the MotionBeam metaphor for character interaction with handheld projectors. Our work draws from the tradition of pre-cinema handheld projectors that use direct physical manipulation to control projected imagery. With our prototype system, users interact and control projected characters by moving and gesturing with the handheld projector itself. This creates a unified interaction style where input and output are tied together within a single device. We introduce a set of interaction principles and present prototype applications that provide clear examples of the MotionBeam metaphor in use. Finally we describe observations and insights from a preliminary user study with our system.
---
paper_title: Projected interfaces: enabling serendipitous interaction with smart tangible objects

paper_content:

The Projected Interfaces architecture enables bi-directional user interaction with smart tangible objects. Smart objects function as both input and output devices simultaneously by cooperating with projector-camera systems to achieve a projected display on their surfaces. Tangible manipulation of the object and camera-based tracking allow interaction directly with the projected display. Such hybrid interfaces benefit both from the flexibility offered by the GUI and the intuitiveness of TUI. In this paper we present the theory behind how to consider interaction for projected interfaces with an architecture design and a proof of concept implementation using an augmented photograph album.
---
paper_title: Interactive Environment-Aware Handheld Projectors for Pervasive Computing Spaces

paper_content:

This paper presents two novel handheld projector systems for indoor pervasive computing spaces. These projection-based devices are "aware" of their environment in ways not demonstrated previously. They offer both spatial awareness, where the system infers location and orientation of the device in 3D space, and geometry awareness, where the system constructs the 3D structure of the world around it, which can encompass the user as well as other physical objects, such as furniture and walls. Previous work in this area has predominantly focused on infrastructure-based spatial-aware handheld projection and interaction. Our prototypes offer greater levels of environment awareness, but achieve this using two opposing approaches; the first infrastructure-based and the other infrastructure-less sensing. We highlight a series of interactions including direct touch, as well as in-air gestures, which leverage the shadow of the user for interaction. We describe the technical challenges in realizing these novel systems; and compare them directly by quantifying their location tracking and input sensing capabilities.
---
paper_title: RFIG lamps: interacting with a self-describing world via photosensing wireless tags and projectors

paper_content:

This paper describes how to instrument the physical world so that objects become self-describing, communicating their identity, geometry, and other information such as history or user annotation. The enabling technology is a wireless tag which acts as a radio frequency identity and geometry (RFIG) transponder. We show how addition of a photo-sensor to a wireless tag significantly extends its functionality to allow geometric operations - such as finding the 3D position of a tag, or detecting change in the shape of a tagged object. Tag data is presented to the user by direct projection using a handheld locale-aware mobile projector. We introduce a novel technique that we call interactive projection to allow a user to interact with projected information e.g. to navigate or update the projected information.The ideas are demonstrated using objects with active radio frequency (RF) tags. But the work was motivated by the advent of unpowered passive-RFID, a technology that promises to have significant impact in real-world applications. We discuss how our current prototypes could evolve to passive-RFID in the future.
---
paper_title: Cooperative augmentation of mobile smart objects with projected displays

paper_content:

Sensors, processors, and radios can be integrated invisibly into objects to make them smart and sensitive to user interaction, but feedback is often limited to beeps, blinks, or buzzes. We propose to redress this input-output imbalance by augmentation of smart objects with projected displays, that—unlike physical displays—allow seamless integration with the natural appearance of an object. In this article, we investigate how, in a ubiquitous computing world, smart objects can acquire and control a projection. We consider that projectors and cameras are ubiquitous in the environment, and we develop a novel conception and system that enables smart objects to spontaneously associate with projector-camera systems for cooperative augmentation. Projector-camera systems are conceived as generic, supporting standard computer vision methods for different appearance cues, and smart objects provide a model of their appearance for method selection at runtime, as well as sensor observations to constrain the visual detection process. Cooperative detection results in accurate location and pose of the object, which is then tracked for visual augmentation in response to display requests by the smart object. In this article, we define the conceptual framework underlying our approach; report on computer vision experiments that give original insight into natural appearance-based detection of everyday objects; show how object sensing can be used to increase speed and robustness of visual detection; describe and evaluate a fully implemented system; and describe two smart object applications to illustrate the system's cooperative augmentation process and the embodied interactions it enables with smart objects.
---
paper_title: Vision-Based Detection of Mobile Smart Objects

paper_content:

We evaluate an approach for mobile smart objects to cooperate with projector-camera systems to achieve interactive projected displays on their surfaces without changing their appearance or function. Smart objects describe their appearance directly to the projector-camera system, enabling vision-based detection based on their natural appearance. This detection is a significant challenge, as objects differ in appearance and appear at varying distances and orientations with respect to a tracking camera. We investigate four detection approaches representing different appearance cues and contribute three experimental studies analysing the impact on detection performance, firstly of scale and rotation, secondly the combination of multiple appearance cues and thirdly the use of context information from the smart object. We find that the training of appearance descriptions must coincide with the scale and orientations providing the best detection performance, that multiple cues provide a clear performance gain over a single cue and that context sensing masks distractions and clutter, further improving detection performance.
---
paper_title: From Focus to Context and Back: Combining Mobile Projectors and Stationary Displays

paper_content:

plus context displays combine high-resolution detail and lower-resolution overview using displays of different pixel densities. Historically, they employed two fixed-size displays of different resolutions, one embedded within the other. In this paper, we explore focus plus context displays using one or more mobile projectors in combination with a stationary display. The portability of mobile projectors as applied to focus plus context displays contributes in three ways. First, the projector's projection on the stationary display can transition dynamically from being the focus of one's interest (i.e. providing a high resolution view when close to the display) to providing context around it (i.e. providing a low resolution view beyond the display's borders when further away from it). Second, users can dynamically reposition and resize a focal area that matches their interest rather than repositioning all content into a fixed high-resolution area. Third, multiple users can manipulate multiple foci or context areas without interfering with one other. A proof-of-concept implementation illustrates these contributions.
---
paper_title: PicoTales: collaborative authoring of animated stories using handheld projectors

paper_content:

In this article we describe a novel approach to collaborative video authoring using handheld projectors. PicoTales are created by sketching story elements on a projector+phone prototype, and then animated by moving the projected image. Movements are captured using motion sensor data, rather than visual or other tracking methods, allowing interaction and story creation anywhere. We describe in detail the design and development of our prototype device, and also address issues in position estimation and element tracking. An experiment was conducted to evaluate the prototype, demonstrating its accuracy and usability for ad-hoc creation of story videos. The potential of the system for story authoring is shown via a further experiment looking at the quality of the animated story videos produced. We conclude by considering possible future developments of the concept, and highlighting the benefits of our design for collaborative story capture.
---
paper_title: Multi-user interaction using handheld projectors

paper_content:

Recent research on handheld projector interaction has expanded the display and interaction space of handheld devices by projecting information onto the physical environment around the user, but has mainly focused on single-user scenarios. We extend this prior single-user research to co-located multi-user interaction using multiple handheld projectors. We present a set of interaction techniques for supporting co-located collaboration with multiple handheld projectors, and discuss application scenarios enabled by them.
---
paper_title: Steerable projection: exploring alignment in interactive mobile displays

paper_content:

Emerging smartphones and other handheld devices are now being fitted with a set of new embedded technologies such as pico-projection. They are usually designed with the pico-projector embedded in the top of the device. Despite the potential of personal mobile projection to support new forms of interactivity such as augmented reality techniques, these devices have not yet made significant impact on the ways in which mobile data is experienced. We suggest that this `traditional' configuration of fixed pico-projectors within the device is unsuited to many projection tasks because it couples the orientation of the device to the management of the projection space, preventing users from easily and simultaneously using the mobile device and looking at the projection. We present a study which demonstrates this problem and the requirement for steerable projection behaviour and some initial users' preferences for different projection coupling angles according to context. Our study highlights the importance of flexible interactive projections which can support interaction techniques on the device and on the projection space according to task. This inspires a number of interaction techniques that create different personal and shared interactive display alignments to suit a range of different mobile projection situations.
---
paper_title: SideBySide: ad-hoc multi-user interaction with handheld projectors

paper_content:

We introduce SideBySide, a system designed for ad-hoc multi-user interaction with handheld projectors. SideBySide uses device-mounted cameras and hybrid visible/infrared light projectors to track multiple independent projected images in relation to one another. This is accomplished by projecting invisible fiducial markers in the near-infrared spectrum. Our system is completely self-contained and can be deployed as a handheld device without instrumentation of the environment. We present the design and implementation of our system including a hybrid handheld projector to project visible and infrared light, and techniques for tracking projected fiducial markers that move and overlap. We introduce a range of example applications that demonstrate the applicability of our system to real-world scenarios such as mobile content exchange, gaming, and education.
---
paper_title: Interactive Phone Call: Synchronous Remote Collaboration and Projected Interactive Surfaces

paper_content:

Smartphones provide large amounts of personal data, functionalities, and apps and make a substantial part of our daily communication. But during phone calls the phone cannot be used much beyond voice communication and does not offer support for synchronous collaboration. This is owed to the fact that first, despite the availability of alternatives, the phone is typically held at one's ear; and second that the small mobile screen is less suited to be used with existing collaboration software. This paper presents a novel in-call collaboration system that leverages projector phones as they provide a large display that can be used while holding the phone to the ear to project an interactive interface anytime and anywhere. The system uses a desktop metaphor user interface and provides a private and a shared space, live mirroring of the shared space and user defined access rights to shared content. We evaluated the system in a comparative user study. The results of the user study highlight the general benefits of synchronous in-call collaboration and in particular the advantages of the projected display and our developed concepts. Our findings inform future designers of synchronous remote collaboration software for interactive surfaces.
---
paper_title: ShadowPuppets: supporting collocated interaction with mobile projector phones using hand shadows

paper_content:

Pico projectors attached to mobile phones allow users to view phone content using a large display. However, to provide input to projector phones, users have to look at the device, diverting their attention from the projected image. Additionally, other collocated users have no way of interacting with the device. We present ShadowPuppets, a system that supports collocated interaction with mobile projector phones. ShadowPuppets allows users to cast hand shadows as input to mobile projector phones. Most people understand how to cast hand shadows, which provide an easy input modality. Additionally, they implicitly support collocated usage, as nearby users can cast shadows as input and one user can see and understand another user's hand shadows. We describe the results of three user studies. The first study examines what hand shadows users expect will cause various effects. The second study looks at how users perceive hand shadows, examining what effects they think various hand shadows will cause. Finally, we present qualitative results from a study with our functional prototype and discuss design implications for systems using shadows as input. Our findings suggest that shadow input can provide a natural and intuitive way of interacting with projected interfaces and can support collocated collaboration.
---
paper_title: Investigating mid-air pointing interaction for projector phones

paper_content:

Projector phones, mobile phones with built-in projectors, might significantly change the way we are going to use and interact with mobile phones. The potential of combining the mobile and the projected display and further the potential of the mid-air space between them have yet to be explored. In this paper we assess these potentials by reporting two user studies: First, an experimental comparison of four techniques for target selection on the projection, including interaction on the touchscreen of the projector phone as well as performing pointing gestures in mid-air around the phone. Our results indicate that interacting behind the phone yields the highest performance, albeit showing a twice as high error rate. Second, a follow-up experiment where we analyzed the performance of the two best tech-niques of the first study within realistic mobile application scenarios such as browsing and gaming. The results show that mobile applications benefit from the projection, e.g., by overcoming the fat-finger problem on touchscreens and increasing the visibility of small objects. Our findings speak for the integration of a tracking camera at the bottom of the projector phone to enable mid-air pointing interaction.
---
paper_title: OmniTouch: wearable multitouch interaction everywhere

paper_content:

OmniTouch is a wearable depth-sensing and projection system that enables interactive multitouch applications on everyday surfaces. Beyond the shoulder-worn system, there is no instrumentation of the user or environment. Foremost, the system allows the wearer to use their hands, arms and legs as graphical, interactive surfaces. Users can also transiently appropriate surfaces from the environment to expand the interactive area (e.g., books, walls, tables). On such surfaces - without any calibration - OmniTouch provides capabilities similar to that of a mouse or touchscreen: X and Y location in 2D interfaces and whether fingers are "clicked" or hovering, enabling a wide variety of interactions. Reliable operation on the hands, for example, requires buttons to be 2.3cm in diameter. Thus, it is now conceivable that anything one can do on today's mobile devices, they could do in the palm of their hand.
---
paper_title: LightBeam: Interacting with Augmented Real-World Objects in Pico Projections

paper_content:

Pico projectors have lately been investigated as mobile display and interaction devices. We propose to use them as 'light beams': Everyday objects sojourning in a beam are turned into dedicated projection surfaces and tangible interaction devices. This way, our daily surroundings get populated with interactive objects, each one temporarily chartered with a dedicated sub-issue of pervasive interaction. While interaction with objects has been studied in larger, immersive projection spaces, the affordances of pico projections are fundamentally different: they have a very small, strictly limited field of projection, and they are mobile. This paper contributes the results of an exploratory field study on how people interact with everyday objects in pico projections in nomadic settings. Based upon these results, we present novel interaction techniques that leverage the limited field of projection and trade-off between digitally augmented and traditional uses of everyday objects.
---
paper_title: On-body interaction: armed and dangerous

paper_content:

Recent technological advances in input sensing, as well as ultra-small projectors, have opened up new opportunities for interaction -- the use of the body itself as both an input and output platform. Such on-body interfaces offer new interactive possibilities, and the promise of access to computation, communication and information literally in the palm of our hands. The unique context of on-body interaction allows us to take advantage of extra dimensions of input our bodies naturally afford us. In this paper, we consider how the arms and hands can be used to enhance on-body interactions, which is typically finger input centric. To explore this opportunity, we developed Armura, a novel interactive on-body system, supporting both input and graphical output. Using this platform as a vehicle for exploration, we proto-typed many applications and interactions. This helped to confirm chief use modalities, identify fruitful interaction approaches, and in general, better understand how interfaces operate on the body. We highlight the most compelling techniques we uncovered. Further, this paper is the first to consider and prototype how conventional interaction issues, such as cursor control and clutching, apply to the on-body domain. Finally, we bring to light several new and unique interaction techniques.
---
paper_title: WUW - wear Ur world: a wearable gestural interface

paper_content:

Information is traditionally confined to paper or digitally to a screen. In this paper, we introduce WUW, a wearable gestural interface, which attempts to bring information out into the tangible world. By using a tiny projector and a camera mounted on a hat or coupled in a pendant like wearable device, WUW sees what the user sees and visually augments surfaces or physical objects the user is interacting with. WUW projects information onto surfaces, walls, and physical objects around us, and lets the user interact with the projected information through natural hand gestures, arm movements or interaction with the object itself.
---
paper_title: HideOut: mobile projector interaction with tangible objects and surfaces

paper_content:

HideOut is a mobile projector-based system that enables new applications and interaction techniques with tangible objects and surfaces. HideOut uses a device mounted camera to detect hidden markers applied with infrared-absorbing ink. The obtrusive appearance of fiducial markers is avoided and the hidden marker surface doubles as a functional projection surface. We present example applications that demonstrate a wide range of interaction scenarios, including media navigation tools, interactive storytelling applications, and mobile games. We explore the design space enabled by the HideOut system and describe the hidden marker prototyping process. HideOut brings tangible objects to life for interaction with the physical world around us.
---
paper_title: Skinput: appropriating the body as an input surface

paper_content:

We present Skinput, a technology that appropriates the human body for acoustic transmission, allowing the skin to be used as an input surface. In particular, we resolve the location of finger taps on the arm and hand by analyzing mechanical vibrations that propagate through the body. We collect these signals using a novel array of sensors worn as an armband. This approach provides an always available, naturally portable, and on-body finger input system. We assess the capabilities, accuracy and limitations of our technique through a two-part, twenty-participant user study. To further illustrate the utility of our approach, we conclude with several proof-of-concept applications we developed.
---
paper_title: Combining multiple depth cameras and projectors for interactions on, above and between surfaces

paper_content:

Instrumented with multiple depth cameras and projectors, LightSpace is a small room installation designed to explore a variety of interactions and computational strategies related to interactive displays and the space that they inhabit. LightSpace cameras and projectors are calibrated to 3D real world coordinates, allowing for projection of graphics correctly onto any surface visible by both camera and projector. Selective projection of the depth camera data enables emulation of interactive displays on un-instrumented surfaces (such as a standard table or office desk), as well as facilitates mid-air interactions between and around these displays. For example, after performing multi-touch interactions on a virtual object on the tabletop, the user may transfer the object to another display by simultaneously touching the object and the destination display. Or the user may "pick up" the object by sweeping it into their hand, see it sitting in their hand as they walk over to an interactive wall display, and "drop" the object onto the wall by touching it with their other hand. We detail the interactions and algorithms unique to LightSpace, discuss some initial observations of use and suggest future directions.
---
paper_title: Paper windows: interaction techniques for digital paper

paper_content:

In this paper, we present Paper Windows, a prototype windowing environment that simulates the use of digital paper displays. By projecting windows on physical paper, Paper Windows allows the capturing of physical affordances of paper in a digital world. The system uses paper as an input device by tracking its motion and shape with a Vicon Motion Capturing System. We discuss the design of a number of interaction techniques for manipulating information on paper displays.
---
paper_title: Permulin: collaboration on interactive surfaces with personal in- and output

paper_content:

Interactive tables are well suited for co-located collaboration. Most prior research assumed users to share the same overall display output; a key challenge was the appropriate partitioning of screen real estate, assembling the right information 'at the users' finger-tips through simultaneous input. A different approach is followed in recent multi-view display environments: they offer personal output for each team member, yet risk to dissolve the team due to the lack of a common visual focus. Our approach combines both lines of thought, guided by the question: "What if the visible output and simultaneous input was partly shared and partly private?" We present Permulin as a concrete corresponding implementation, based on a set of novel interaction concepts that support fluid transitions between individual and group activities, coordination of group activities, and concurrent, distraction-free in-place manipulation. Study results indicate that users are able to focus on individual work on the whole surface without notable mutual interference, while at the same time establishing a strong sense of collaboration.
---
paper_title: OmniTouch: wearable multitouch interaction everywhere

paper_content:

OmniTouch is a wearable depth-sensing and projection system that enables interactive multitouch applications on everyday surfaces. Beyond the shoulder-worn system, there is no instrumentation of the user or environment. Foremost, the system allows the wearer to use their hands, arms and legs as graphical, interactive surfaces. Users can also transiently appropriate surfaces from the environment to expand the interactive area (e.g., books, walls, tables). On such surfaces - without any calibration - OmniTouch provides capabilities similar to that of a mouse or touchscreen: X and Y location in 2D interfaces and whether fingers are "clicked" or hovering, enabling a wide variety of interactions. Reliable operation on the hands, for example, requires buttons to be 2.3cm in diameter. Thus, it is now conceivable that anything one can do on today's mobile devices, they could do in the palm of their hand.
---
paper_title: Projector phone use: practices and social implications

paper_content:

Phones with integrated pico projectors are starting to be marketed as devices for business presentations and media viewing, and researchers are beginning to design projection-specific applications and interaction techniques to explore a broader array of possible uses. To begin to document how people use projector phones outside the laboratory, we present the results of a 4-week exploratory field study of naturalistic use of commodity projector phones. In our analysis, we consider how context, such as group size, relationships, and locale, influences projector phone use. A key observation is that users can readily exploit the new facilities of these devices to author interesting effects by employing representational techniques such as superimposition, scaling, translation, and motion. Thus, even the "basic" projector phone platform affords novel interaction modalities. Finally, we discuss the social implications of projector phone use for privacy and control, extrapolating from our observations to envision a future in which these devices are ubiquitous. With ubiquity, projector phone use may become problematic in public settings, motivating new rules of etiquette and perhaps laws, yet it may also engender new forms of creative expression.
---
paper_title: Proxemic interaction: designing for a proximity and orientation-aware environment

paper_content:

In the everyday world, much of what we do is dictated by how we interpret spatial relationships, or proxemics. What is surprising is how little proxemics are used to mediate people's interactions with surrounding digital devices. We imagine proxemic interaction as devices with fine-grained knowledge of nearby people and other devices -- their position, identity, movement, and orientation -- and how such knowledge can be exploited to design interaction techniques. In particular, we show how proxemics can: regulate implicit and explicit interaction; trigger such interactions by continuous movement or by movement of people and devices in and out of discrete proxemic regions; mediate simultaneous interaction of multiple people; and interpret and exploit people's directed attention to other people and objects. We illustrate these concepts through an interactive media player running on a vertical surface that reacts to the approach, identity, movement and orientation of people and their personal devices.
---
paper_title: ThirdEye: a technique that enables multiple viewers to see different content on a single display screen

paper_content:

In this note, we present ThirdEye, a technique that enables multiple viewers to see different content on a single display screen as the same time.
---
paper_title: Lumitrack: low cost, high precision, high speed tracking with projected m-sequences

paper_content:

We present Lumitrack, a novel motion tracking technology that uses projected structured patterns and linear optical sensors. Each sensor unit is capable of recovering 2D location within the projection area, while multiple sensors can be combined for up to six degree of freedom (DOF) tracking. Our structured light approach is based on special patterns, called m-sequences, in which any consecutive sub-sequence of m bits is unique. Lumitrack can utilize both digital and static projectors, as well as scalable embedded sensing configurations. The resulting system enables high-speed, high precision, and low-cost motion tracking for a wide range of interactive applications. We detail the hardware, operation, and performance characteristics of our approach, as well as a series of example applications that highlight its immediate feasibility and utility.
---
paper_title: LightBeam: Interacting with Augmented Real-World Objects in Pico Projections

paper_content:

Pico projectors have lately been investigated as mobile display and interaction devices. We propose to use them as 'light beams': Everyday objects sojourning in a beam are turned into dedicated projection surfaces and tangible interaction devices. This way, our daily surroundings get populated with interactive objects, each one temporarily chartered with a dedicated sub-issue of pervasive interaction. While interaction with objects has been studied in larger, immersive projection spaces, the affordances of pico projections are fundamentally different: they have a very small, strictly limited field of projection, and they are mobile. This paper contributes the results of an exploratory field study on how people interact with everyday objects in pico projections in nomadic settings. Based upon these results, we present novel interaction techniques that leverage the limited field of projection and trade-off between digitally augmented and traditional uses of everyday objects.
---
paper_title: AIREAL: interactive tactile experiences in free air

paper_content:

AIREAL is a novel haptic technology that delivers effective and expressive tactile sensations in free air, without requiring the user to wear a physical device. Combined with interactive computers graphics, AIREAL enables users to feel virtual 3D objects, experience free air textures and receive haptic feedback on gestures performed in free space. AIREAL relies on air vortex generation directed by an actuated flexible nozzle to provide effective tactile feedback with a 75 degrees field of view, and within an 8.5cm resolution at 1 meter. AIREAL is a scalable, inexpensive and practical free air haptic technology that can be used in a broad range of applications, including gaming, mobile applications, and gesture interaction among many others. This paper reports the details of the AIREAL design and control, experimental evaluations of the device's performance, as well as an exploration of the application space of free air haptic displays. Although we used vortices, we believe that the results reported are generalizable and will inform the design of haptic displays based on alternative principles of free air tactile actuation.
---
paper_title: HideOut: mobile projector interaction with tangible objects and surfaces

paper_content:

HideOut is a mobile projector-based system that enables new applications and interaction techniques with tangible objects and surfaces. HideOut uses a device mounted camera to detect hidden markers applied with infrared-absorbing ink. The obtrusive appearance of fiducial markers is avoided and the hidden marker surface doubles as a functional projection surface. We present example applications that demonstrate a wide range of interaction scenarios, including media navigation tools, interactive storytelling applications, and mobile games. We explore the design space enabled by the HideOut system and describe the hidden marker prototyping process. HideOut brings tangible objects to life for interaction with the physical world around us.
---
paper_title: Flexpad: highly flexible bending interactions for projected handheld displays

paper_content:

Flexpad is an interactive system that combines a depth camera and a projector to transform sheets of plain paper or foam into flexible, highly deformable, and spatially aware handheld displays. We present a novel approach for tracking deformed surfaces from depth images in real time. It captures deformations in high detail, is very robust to occlusions created by the user's hands and fingers, and does not require any kind of markers or visible texture. As a result, the display is considerably more deformable than in previous work on flexible handheld displays, enabling novel applications that leverage the high expressiveness of detailed deformation. We illustrate these unique capabilities through three application examples: curved cross-cuts in volumetric images, deforming virtual paper characters, and slicing through time in videos. Results from two user studies show that our system is capable of detecting complex deformations and that users are able to perform them quickly and precisely.
---

Title: A Research Overview of Mobile Projected User Interfaces

Section 1: Introduction
Description 1: Introduce the miniaturization of projectors and their integration into smartphones, highlighting the potential for novel user interfaces and the main challenges in developing mobile projected user interfaces.

Section 2: Abstract
Description 2: Provide a summary of the research on mobile projected user interfaces, outlining the key research directions and future challenges.

Section 3: Research Overview
Description 3: Discuss the broader landscape of mobile projected user interface research, categorizing it by the relationship between the projector and the projection surface.

Section 4: Nomadic Projection Interfaces
Description 4: Describe interfaces where the pico projector is fixed in the vicinity and projects onto a fixed surface, often supplementing static working environments with additional interactive displays.

Section 5: Handheld Projection Interfaces
Description 5: Explore interfaces where the user holds the projector, leveraging its portability for augmenting large information spaces, creating environment-aware interfaces, and enabling handheld multiuser interfaces for collaboration.

Section 6: Tangible Projection Interfaces
Description 6: Emphasize interfaces that incorporate the projector into wearable or movable physical objects, enhancing interaction through tangible surfaces such as the human body or everyday items.

Section 7: Research Challenges and Conclusion
Description 7: Summarize the technological and interaction design challenges for mobile projected user interfaces, discussing potentials for future research and the outlook for pico projector technology in consumer products.