3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review
---
paper_title: A Survey of Augmented Reality

paper_content:

This paper surveys the field of augmented reality AR, in which 3D virtual objects are integrated into a 3D real environment in real time. It describes the medical, manufacturing, visualization, path planning, entertainment, and military applications that have been explored. This paper describes the characteristics of augmented reality systems, including a detailed discussion of the tradeoffs between optical and video blending approaches. Registration and sensing errors are two of the biggest problems in building effective augmented reality systems, so this paper summarizes current efforts to overcome these problems. Future directions and areas requiring further research are discussed. This survey provides a starting point for anyone interested in researching or using augmented reality.
---
paper_title: Recent Advances in Augmented Reality

paper_content:

In 1997, Azuma published a survey on augmented reality (AR). Our goal is to complement, rather than replace, the original survey by presenting representative examples of the new advances. We refer one to the original survey for descriptions of potential applications (such as medical visualization, maintenance and repair of complex equipment, annotation, and path planning); summaries of AR system characteristics (such as the advantages and disadvantages of optical and video approaches to blending virtual and real, problems in display focus and contrast, and system portability); and an introduction to the crucial problem of registration, including sources of registration error and error-reduction strategies.
---
paper_title: A Survey of Augmented Reality Technologies, Applications and Limitations

paper_content:

We are on the verge of ubiquitously adopting Augmented Reality (AR) technologies to enhance our percep- tion and help us see, hear, and feel our environments in new and enriched ways. AR will support us in fields such as education, maintenance, design and reconnaissance, to name but a few. This paper describes the field of AR, including a brief definition and development history, the enabling technologies and their characteristics. It surveys the state of the art by reviewing some recent applications of AR technology as well as some known limitations regarding human factors in the use of AR systems that developers will need to overcome.
---
paper_title: A TAXONOMY OF MIXED REALITY VISUAL DISPLAYS

paper_content:

Paul Milgram received the B.A.Sc. degree from the University of Toronto in 1970, the M.S.E.E. degree from the Technion (Israel) in 1973 and the Ph.D. degree from the University of Toronto in 1980. From 1980 to 1982 he was a ZWO Visiting Scientist and a NATO Postdoctoral in the Netherlands, researching automobile driving behaviour. From 1982 to 1984 he was a Senior Research Engineer in Human Engineering at the National Aerospace Laboratory (NLR) in Amsterdam, where his work involved the modelling of aircraft flight crew activity, advanced display concepts and control loops with human operators in space teleoperation. Since 1986 he has worked at the Industrial Engineering Department of the University of Toronto, where he is currently an Associate Professor and Coordinator of the Human Factors Engineering group. He is also cross appointed to the Department of Psychology. In 1993-94 he was an invited researcher at the ATR Communication Systems Research Laboratories, in Kyoto, Japan. His research interests include display and control issues in telerobotics and virtual environments, stereoscopic video and computer graphics, cognitive engineering, and human factors issues in medicine. He is also President of Translucent Technologies, a company which produces "Plato" liquid crystal visual occlusion spectacles (of which he is the inventor), for visual and psychomotor research.
---
paper_title: A Survey of Augmented Reality

paper_content:

This paper surveys the field of augmented reality AR, in which 3D virtual objects are integrated into a 3D real environment in real time. It describes the medical, manufacturing, visualization, path planning, entertainment, and military applications that have been explored. This paper describes the characteristics of augmented reality systems, including a detailed discussion of the tradeoffs between optical and video blending approaches. Registration and sensing errors are two of the biggest problems in building effective augmented reality systems, so this paper summarizes current efforts to overcome these problems. Future directions and areas requiring further research are discussed. This survey provides a starting point for anyone interested in researching or using augmented reality.
---
paper_title: A REVIEW OF 3D GESTURE INTERACTION FOR HANDHELD AUGMENTED REALITY

paper_content:

Interaction for Handheld Augmented Reality (HAR) is a challenging research topic because of the small screen display and limited input options. Although 2D touch screen input is widely used, 3D gesture interaction is a suggested alternative input method. Recent 3D gesture interaction research mainly focuses on using RGB-Depth cameras to detect the spatial position and pose of fingers, using this data for virtual object manipulations in the AR scene. In this paper we review previous 3D gesture research on handheld interaction metaphors for HAR. We present their novelties as well as limitations, and discuss future research directions of 3D gesture interaction for HAR. Our results indicate that 3D gesture input on HAR is a potential interaction method for assisting a user in many tasks such as in education, urban simulation and 3D games.
---
paper_title: Mobile Augmented Reality Survey: From Where We Are to Where We Go

paper_content:

The boom in the capabilities and features of mobile devices, like smartphones, tablets, and wearables, combined with the ubiquitous and affordable Internet access and the advances in the areas of cooperative networking, computer vision, and mobile cloud computing transformed mobile augmented reality (MAR) from science fiction to a reality. Although mobile devices are more constrained computationalwise from traditional computers, they have a multitude of sensors that can be used to the development of more sophisticated MAR applications and can be assisted from remote servers for the execution of their intensive parts. In this paper, after introducing the reader to the basics of MAR, we present a categorization of the application fields together with some representative examples. Next, we introduce the reader to the user interface and experience in MAR applications and continue with the core system components of the MAR systems. After that, we discuss advances in tracking and registration, since their functionality is crucial to any MAR application and the network connectivity of the devices that run MAR applications together with its importance to the performance of the application. We continue with the importance of data management in MAR systems and the systems performance and sustainability, and before we conclude this survey, we present existing challenging problems.
---
paper_title: Perceptual issues in augmented reality revisited

paper_content:

This paper provides a classification of perceptual issues in augmented reality, created with a visual processing and interpretation pipeline in mind. We organize issues into ones related to the environment, capturing, augmentation, display, and individual user differences. We also illuminate issues associated with more recent platforms such as handhelds or projector-camera systems. Throughout, we describe current approaches to addressing these problems, and suggest directions for future research.
---
paper_title: Augmented Reality in Surgery

paper_content:

Objective To evaluate the history and current knowledge of computer-augmented reality in the field of surgery and its potential goals in education, surgeon training, and patient treatment. Data Sources National Library of Medicine's database and additional library searches. Study Selection Only articles suited to surgical sciences with a well-defined aim of study, methodology, and precise description of outcome were included. Data Synthesis Augmented reality is an effective tool in executing surgical procedures requiring low-performance surgical dexterity; it remains a science determined mainly by stereotactic registration and ergonomics. Strong evidence was found that it is an effective teaching tool for training residents. Weaker evidence was found to suggest a significant influence on surgical outcome, both morbidity and mortality. No evidence of cost-effectiveness was found. Conclusions Augmented reality is a new approach in executing detailed surgical operations. Although its application is in a preliminary stage, further research is needed to evaluate its long-term clinical impact on patients, surgeons, and hospital administrators. Its widespread use and the universal transfer of such technology remains limited until there is a better understanding of registration and ergonomics.
---
paper_title: 3D User Interfaces: Theory and Practice

paper_content:

Foreword. Preface. I. FOUNDATIONS OF 3D USER INTERFACES . 1. Introduction to 3D User Interfaces. What Are 3D User Interfaces? Why 3D User Interfaces? Terminology. Application Areas. Conclusion. 2. 3D User Interfaces: History and Roadmap. History of 3D UIs. Roadmap to 3D UIs. Scope of This Book. Conclusion. II. Hardware Technologies for 3D User Interfaces. 3. 3D User Interface Output Hardware. Introduction. Visual Displays. Auditory Displays. Haptic Displays. Design Guidelines: Choosing Output Devices for 3D User Interfaces. Conclusion. 4. 3D User Interface Input Hardware. Introduction. Desktop Input Devices. Tracking Devices. 3D Mice. Special-Purpose Input Devices. Direct Human Input. Home-Brewed Input Devices. Choosing Input Devices for 3D Interfaces. III. 3D INTERACTION TECHNIQUES. 5. Selection and Manipulation. Introduction. 3D Manipulation Tasks. Manipulation Techniques and Input Devices. Interaction Techniques for 3D Manipulation. Design Guidelines. 6. Travel. Introduction. 3D Travel Tasks. Travel Techniques. Design Guidelines. 7. Wayfinding. Introduction. Theoretical Foundations. User-Centered Wayfinding Support. Environment-Centered Wayfinding Support. Evaluating Wayfinding Aids. Design Guidelines. Conclusion. 8. System Control. Introduction. Classification. Graphical Menus. Voice Commands. Gestural Commands. Tools. Multimodal System Control Techniques. Design Guidelines. Case Study: Mixing System Control Methods. 8.10. Conclusion. 9. Symbolic Input. Introduction. Symbolic Input Tasks. Symbolic Input Techniques. Design Guidelines. Beyond Text and Number Entry. IV. DESIGNING AND DEVELOPING 3D USER INTERFACES. 10. Strategies for Designing and Developing 3D User Interfaces. Introduction. Designing for Humans. Inventing 3D User Interfaces. Design Guidelines. 11. Evaluation of 3D User Interfaces. Introduction. Background. Evaluation Metrics for 3D Interfaces. Distinctive Characteristics of 3D Interface Evaluation. Classification of 3D Evaluation Methods. Two Multimethod Approaches. Guidelines for 3D Interface Evaluation. V. THE FUTURE OF 3D USER INTERFACES. 12. Beyond Virtual: 3D User Interfaces for the Real World. Introduction. AR Interfaces as 3D Data Browsers. 3D Augmented Reality Interfaces. Augmented Surfaces and Tangible Interfaces. Tangible AR Interfaces. Agents in AR. Transitional AR-VR Interfaces. Conclusion. 13. The Future of 3D User Interfaces. Questions about 3D UI Technology. Questions about 3D Interaction Techniques. Questions about 3D UI Design and Development. Questions about 3D UI Evaluation. Million-Dollar Questions. Appendix A: Quick Reference Guide to 3D User Interface Mathematics. Scalars. Vectors. Points. Matrices. Quaternions. Bibliography. Index.
---
paper_title: Recent Advances in Augmented Reality

paper_content:

In 1997, Azuma published a survey on augmented reality (AR). Our goal is to complement, rather than replace, the original survey by presenting representative examples of the new advances. We refer one to the original survey for descriptions of potential applications (such as medical visualization, maintenance and repair of complex equipment, annotation, and path planning); summaries of AR system characteristics (such as the advantages and disadvantages of optical and video approaches to blending virtual and real, problems in display focus and contrast, and system portability); and an introduction to the crucial problem of registration, including sources of registration error and error-reduction strategies.
---
paper_title: Historical Oslo on a handheld device – a mobile augmented reality application

paper_content:

Abstract Mobile augmented reality (AR) applications can provide just-in-time information based on the user's preferences and context and thus improve the tourist experience. Due to various problems, the potential of this technology has yet to be fully exploited. In this paper, we present the design, implementation and evaluation of a mobile AR application for historical Oslo that aims to bring history to life by providing historical pictures of a location, depending on the direction in which the camera is pointing. This application can run offline and is designed as a generic framework where a similar application for a new city can be created by simply replacing the city-specific database.
---
paper_title: Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR

paper_content:

Although Augmented Reality technology was first developed over forty years ago, there has been little survey work giving an overview of recent research in the field. This paper reviews the ten-year development of the work presented at the ISMAR conference and its predecessors with a particular focus on tracking, interaction and display research. It provides a roadmap for future augmented reality research which will be of great value to this relatively young field, and also for helping researchers decide which topics should be explored when they are beginning their own studies in the area.
---
paper_title: The MagicBook: a transitional AR interface

paper_content:

Abstract The MagicBook is a Mixed Reality interface that uses a real book to seamlessly transport users between Reality and Virtuality. A vision-based tracking method is used to overlay virtual models on real book pages, creating an Augmented Reality (AR) scene. When users see an AR scene they are interested in they can fly inside it and experience it as an immersive Virtual Reality (VR). The interface also supports multi-scale collaboration, allowing multiple users to experience the same virtual environment either from an egocentric or an exocentric perspective. In this paper we describe the MagicBook prototype, potential applications and user feedback.
---
paper_title: SlidAR: A 3D positioning method for SLAM-based handheld augmented reality

paper_content:

Abstract Handheld Augmented Reality (HAR) has the potential to introduce Augmented Reality (AR) to large audiences due to the widespread use of suitable handheld devices. However, many of the current HAR systems are not considered very practical and they do not fully answer to the needs of the users. One of the challenging areas in HAR is the in-situ AR content creation where the correct and accurate positioning of virtual objects to the real world is fundamental. Due to the hardware limitations of handheld devices and possible restrictions in the environment, the correct 3D positioning of objects can be difficult to achieve we are unable to use AR markers or correctly map the 3D structure of the environment. We present SlidAR, a 3D positioning for Simultaneous Localization And Mapping (SLAM) based HAR systems. SlidAR utilizes 3D ray-casting and epipolar geometry for virtual object positioning. It does not require a perfect 3D reconstruction of the environment nor any virtual depth cues. We have conducted a user experiment to evaluate the efficiency of SlidAR method against an existing device-centric positioning method that we call HoldAR. Results showed that SlidAR was significantly faster, required significantly less device movement, and also got significantly better subjective evaluation from the test participants. SlidAR also had higher positioning accuracy, although not significantly.
---
paper_title: Professional Augmented Reality Browsers for Smartphones: Programming for junaio, Layar and Wikitude

paper_content:

Create amazing mobile augmented reality apps with junaio, Layar, and Wikitude!Professional Augmented Reality Browsers for Smartphones guides you through creating your own augmented reality apps for the iPhone, Android, Symbian, and bada platforms, featuring fully workable and downloadable source code. You will learn important techniques through hands-on applications, and you will build on those skills as the book progresses.Professional Augmented Reality Browsers for Smartphones:Describes how to use the latitude/longitude coordinate system to build location-aware solutions and tells where to get POIs for your own augmented reality applicationsDetails the leading augmented reality platforms and highlights the best applicationsCovers development for the leading augmented reality browser platforms: Wikitude, Layar, and junaioShows how to build cross-platform location-aware content (Android, iPhone, Symbian, and bada) to display POIs directly in camera viewIncludes tutorials for building 2D and 3D content, storing content in databases, and triggering actions when users reach specific locationswrox.comProgrammer ForumsJoin our Programmer to Programmer forums to ask and answer programming questions about this book, join discussions on the hottest topics in the industry, and connect with fellow programmers from around the world.Code DownloadsTake advantage of free code samples from this book, as well as code samples from hundreds of other books, all ready to use.Read MoreFind articles, ebooks, sample chapters, and tables of contents for hundreds of books, and more reference resources on programming topics that matter to you.Wrox Professional guides are planned and written by working programmers to meet the real-world needs of programmers, developers, and IT professionals. Focused and relevant, they address the issues technology professionals face every day. They provide examples, practical solutions, and expert education in new technologies, all designed to help programmers do a better job.
---
paper_title: A study on improving close and distant device movement pose manipulation for hand-held augmented reality

paper_content:

Hand-held smart devices are equipped with powerful processing units, high resolution screens and cameras, that in combination makes them suitable for video see-through Augmented Reality. Many Augmented Reality applications require interaction, such as selection and 3D pose manipulation. One way to perform intuitive, high precision 3D pose manipulation is by direct or indirect mapping of device movement. There are two approaches to device movement interaction; one fixes the virtual object to the device, which therefore becomes the pivot point for the object, thus makes it difficult to rotate without translate. The second approach avoids latter issue by considering rotation and translation separately, relative to the object's center point. The result of this is that the object instead moves out of view for yaw and pitch rotations. In this paper we study these two techniques and compare them with a modification where user perspective rendering is used to solve the rotation issues. The study showed that the modification improves speed as well as both perceived control and intuitiveness among the subjects.
---
paper_title: Handheld Guides in Inspection Tasks: Augmented Reality versus Picture

paper_content:

Inspection tasks focus on observation of the environment and are required in many industrial domains. Inspectors usually execute these tasks by using a guide such as a paper manual, and directly observing the environment. The effort required to match the information in a guide with the information in an environment and the constant gaze shifts required between the two can severely lower the work efficiency of inspector in performing his/her tasks. Augmented reality (AR) allows the information in a guide to be overlaid directly on an environment. This can decrease the amount of effort required for information matching, thus increasing work efficiency. AR guides on head-mounted displays (HMDs) have been shown to increase efficiency. Handheld AR (HAR) is not as efficient as HMD-AR in terms of manipulability, but is more practical and features better information input and sharing capabilities. In this study, we compared two handheld guides: an AR interface that shows 3D registered annotations, that is, annotations having a fixed 3D position in the AR environment, and a non-AR picture interface that displays non-registered annotations on static images. We focused on inspection tasks that involve high information density and require the user to move, as well as to perform several viewpoint alignments. The results of our comparative evaluation showed that use of the AR interface resulted in lower task completion times, fewer errors, fewer gaze shifts, and a lower subjective workload. We are the first to present findings of a comparative study of an HAR and a picture interface when used in tasks that require the user to move and execute viewpoint alignments, focusing only on direct observation. Our findings can be useful for AR practitioners and psychology researchers.
---
paper_title: A New You: From Augmented Reality to Augmented Human

paper_content:

Traditionally, the field of Human Computer Interaction (HCI) was primarily concerned with designing and investigating interfaces between humans and machines. The primary concern of Surface Computing is also still about designing better interfaces to information. However, with recent technological advances, the concept of "enhancing", "augmenting" or even "re-designing" humans themselves is becoming a very feasible and serious topic of scientific research as well as engineering development. "Augmented Human" is a term that I use to refer to this overall research direction. Augmented Human introduces a fundamental paradigm shift in HCI: from human-computer-interaction to human-computer-integration. In this talk, I will discuss rich possibilities and distinct challenges in enhancing human abilities. I will introduce recent projects conducted by our group including design and applications of wearable eye sensing for augmenting our perception and memory abilities, design of flying cameras as our external eyes, a home appliance that can increase your happiness, an organic physical wall/window that dynamically mediates the environment, and an immersive human-human communication called "JackIn".
---
paper_title: Designing an augmented reality multimodal interface for 6dof manipulation techniques

paper_content:

Augmented Reality (AR) supports natural interaction in physical and virtual worlds, so it has recently given rise to a number of novel interaction modalities. This paper presents a method for using hand-gestures with speech input for multimodal interaction in AR. It focuses on providing an intuitive AR environment which supports natural interaction with virtual objects while sustaining accessible real tasks and interaction mechanisms. The paper reviews previous multimodal interfaces and describes recent studies in AR that employ gesture and speech inputs for multimodal input. It describes an implementation of gesture interaction with speech input in AR for virtual object manipulation. Finally, the paper presents a user evaluation of the technique, showing that it can be used to improve the interaction between virtual and physical elements in an AR environment.
---
paper_title: Expected user experience of mobile augmented reality services: a user study in the context of shopping centres

paper_content:

The technical enablers for mobile augmented reality (MAR) are becoming robust enough to allow the development of MAR services that are truly valuable for consumers. Such services would provide a novel interface to the ubiquitous digital information in the physical world, hence serving in great variety of contexts and everyday human activities. To ensure the acceptance and success of future MAR services, their development should be based on knowledge about potential end users' expectations and requirements. We conducted 16 semi-structured interview sessions with 28 participants in shopping centres, which can be considered as a fruitful context for MAR services. We aimed to elicit new knowledge about (1) the characteristics of the expected user experience and (2) central user requirements related to MAR in such a context. From a pragmatic viewpoint, the participants expected MAR services to catalyse their sense of efficiency, empower them with novel context-sensitive and proactive functionalities and raise their awareness of the information related to their surroundings with an intuitive interface. Emotionally, MAR services were expected to offer stimulating and pleasant experiences, such as playfulness, inspiration, liveliness, collectivity and surprise. The user experience categories and user requirements that were identified can serve as targets for the design of user experience of future MAR services.
---
paper_title: Application of Augmented Reality Techniques in Through-life Engineering Services

paper_content:

Abstract Augmented Reality (AR) is an innovative human-machine interaction that overlays virtual components on a real world environment with many potential applications in different fields, ranging from training activities to everyday life (entertainment, head-up display in car windscreens, etc.). The capability to provide the user of the needed information about a process or a procedure directly on the work environment, is the key factor for considering AR as an effective tool to be also used in Through-life Engineering Services (TES). Many experimental implementations have been made by industries and academic institutions in this research area: applications in remote maintenance, diagnostics, non-destructive testing, repairing and setup activities represent the most meaningful examples carried out in the last few years. These applications have concerned different working environments such as aerospace, railway, industrial plants, machine tools, military equipment, underground pipes, civil constructions, etc. The keynote paper will provide a comprehensive survey by reviewing some recent applications in these areas, emphasizing potential advantages, limits and drawbacks, as well as open issues which could represent new challenges for the future.
---
paper_title: Rotation and translation mechanisms for tabletop interaction

paper_content:

A digital tabletop offers several advantages over other groupware form factors for collaborative applications. However, users of a tabletop system do not share a common perspective for the display of information: what is presented right side up to one participant is upside down for another. In this paper, we survey five different rotation and translation techniques for objects displayed on a direct touch digital tabletop display. We analyze their suitability for interactive tabletops in light of their respective input and output degrees of freedom, as well as the precision and completeness provided by each. We describe various tradeoffs that arise when considering which, when and where each of these techniques might be most useful.
---
paper_title: Virtual reality and augmented reality as a training tool for assembly tasks

paper_content:

In this paper we investigate whether virtual reality (VR) and augmented reality (AR) offer potential for the training of manual skills, such as for assembly tasks, in comparison to conventional media. We present results from experiments that compare assembly completion times for a number of different conditions. We firstly investigate completion times for a task where participants can study an engineering drawing and an assembly plan and then conduct the task. We then investigate the task under various VR conditions and context-free AR. We discuss the relative advantages and limitations of using VR and AR as training media for investigating assembly operations, and we present the results of our experimental work.
---
paper_title: Integrated view-input ar interaction for virtual object manipulation using tablets and smartphones

paper_content:

Lately, mobile augmented reality (AR) has become very popular and is used for many commercial and product promotional activities. However, in almost all mobile AR applications, the user only views annotated information or the preset motion of the virtual object in an AR environment and is unable to interact with the virtual objects as if he/she were interacting with real objects in the real environment. In this paper, in an attempt to realize enhanced intuitive and realistic object manipulation in the mobile AR environment, we propose an integrated view-input AR interaction method, which integrates user device manipulation and virtual object manipulation. The method enables the user to hold a 3D virtual object by touching the displayed object on the 2D touch screen of a mobile device, and to move and rotate the object by moving and rotating the mobile device while viewing the held object by way of the 2D screen of the mobile device. Based on this concept, we implemented three types of integrated methods, namely the Rod, Center, and Touch methods, and conducted a user study to investigate the baseline performance metrics of the proposed method on an AR object manipulation task. The Rod method achieved the highest success rate (91%). Participants' feedback indicated that this is because the Rod method is the most natural, and evoked a fixed mental model that is conceivable in the real environment. These results indicated that visualizing the manipulation point on the screen and restricting the user's interactivity with virtual objects from the user's position of view based on a conceivable mental model would be able to aid the user to achieve precise manipulation.
---
paper_title: Advanced Interaction Techniques for Augmented Reality Applications

paper_content:

Augmented Reality (AR) research has been conducted for several decades, although until recently most AR applications had simple interaction methods using traditional input devices. AR tracking, display technology and software has progressed to the point where commercial applications can be developed. However there are opportunities to provide new advanced interaction techniques for AR applications. In this paper we describe several interaction methods that can be used to provide a better user experience, including tangible user interaction, multimodal input and mobile interaction.
---
paper_title: Mobile markerless augmented reality and its application in forensic medicine

paper_content:

Purpose ::: During autopsy, forensic pathologists today mostly rely on visible indication, tactile perception and experience to determine the cause of death. Although computed tomography (CT) data is often available for the bodies under examination, these data are rarely used due to the lack of radiological workstations in the pathological suite. The data may prevent the forensic pathologist from damaging evidence by allowing him to associate, for example, external wounds to internal injuries. To facilitate this, we propose a new multimodal approach for intuitive visualization of forensic data and evaluate its feasibility.
---
paper_title: A Survey of Augmented Reality

paper_content:

This paper surveys the field of augmented reality AR, in which 3D virtual objects are integrated into a 3D real environment in real time. It describes the medical, manufacturing, visualization, path planning, entertainment, and military applications that have been explored. This paper describes the characteristics of augmented reality systems, including a detailed discussion of the tradeoffs between optical and video blending approaches. Registration and sensing errors are two of the biggest problems in building effective augmented reality systems, so this paper summarizes current efforts to overcome these problems. Future directions and areas requiring further research are discussed. This survey provides a starting point for anyone interested in researching or using augmented reality.
---
paper_title: A Novel Multi-touch Approach for 3D Object Free Manipulation

paper_content:

In the field of scientific visualization, 3D manipulation is a fundamental task for many different scientific datasets, such as particle data in physics and astronomy, fluid data in aerography, and structured data in medical science. Current researches show that large multi-touch interactive displays serve as a promising device providing numerous significant advantages for displaying and manipulating scientific data. Those benefits of direct-touch devices motivate us to use touch-based interaction techniques to explore scientific 3D data. However, manipulating object in 3D space via 2D touch input devices is challenging for precise control. Therefore, we present a novel multi-touch approach for manipulating structured objects in 3D visualization space, based on multi-touch gestures and an extra axis for the assistance. Our method supports 7-DOF manipulations. Moreover, with the help from the extra axis and depth hints, users can have better control of the interactions. We report on a user study to make comparisons between our method and standard mouse-based 2D interface. We show in this work that touch-based interactive displays can be more effective when applied to complex problems if the interactive visualizations and interactions are designed appropriately.
---
paper_title: SmartSkin: an infrastructure for freehand manipulation on interactive surfaces

paper_content:

This paper introduces a new sensor architecture for making interactive surfaces that are sensitive to human hand and finger gestures. This sensor recognizes multiple hand positions and shapes and calculates the distance between the hand and the surface by using capacitive sensing and a mesh-shaped antenna. In contrast to camera-based gesture recognition systems, all sensing elements can be integrated within the surface, and this method does not suffer from lighting and occlusion problems. This paper describes the sensor architecture, as well as two working prototype systems: a table-size system and a tablet-size system. It also describes several interaction techniques that would be difficult to perform without using this architecture
---
paper_title: tBox: a 3d transformation widget designed for touch-screens

paper_content:

3D transformation widgets are commonly used in many 3D applications operated from mice and keyboards. These user interfaces allow independent control of translations, rotations, and scaling for manipulation of 3D objects. In this paper, we study how these widgets can be adapted to the tactile paradigm. We have explored an approach where users apply rotations by means of physically plausible gestures, and we have extended successful 2D tactile principles to the context of 3D interaction. These investigations led to the design of a new 3D transformation widget, tBox, that can been operated easily and efficiently from gestures on touch-screens.
---
paper_title: Rotation and translation mechanisms for tabletop interaction

paper_content:

A digital tabletop offers several advantages over other groupware form factors for collaborative applications. However, users of a tabletop system do not share a common perspective for the display of information: what is presented right side up to one participant is upside down for another. In this paper, we survey five different rotation and translation techniques for objects displayed on a direct touch digital tabletop display. We analyze their suitability for interactive tabletops in light of their respective input and output degrees of freedom, as well as the precision and completeness provided by each. We describe various tradeoffs that arise when considering which, when and where each of these techniques might be most useful.
---
paper_title: The design and evaluation of 3D positioning techniques for multi-touch displays

paper_content:

Multi-touch displays represent a promising technology for the display and manipulation of 3D data. To fully exploit their capabilities, appropriate interaction techniques must be designed. In this paper, we explore the design of free 3D positioning techniques for multi-touch displays to exploit the additional degrees of freedom provided by this technology. Our contribution is two-fold: first we present an interaction technique to extend the standard four view-ports technique found in commercial CAD applications, and second we introduce a technique designed to allow free 3D positioning with a single view of the scene. The two techniques were evaluated in a preliminary experiment. The first results incline us to conclude that the two techniques are equivalent in term of performance showing that the Z-technique provides a real alternative to the statu quo viewport technique.
---
paper_title: A screen-space formulation for 2D and 3D direct manipulation

paper_content:

Rotate-Scale-Translate (RST) interactions have become the de facto standard when interacting with two-dimensional (2D) contexts in single-touch and multi-touch environments. Because the use of RST has thus far focused almost entirely on 2D, there are not yet standard techniques for extending these principles into three dimensions. In this paper we describe a screen-space method which fully captures the semantics of the traditional 2D RST multi-touch interaction, but also allows us to extend these same principles into three-dimensional (3D) interaction. Just like RST allows users to directly manipulate 2D contexts with two or more points, our method allows the user to directly manipulate 3D objects with three or more points. We show some novel interactions, which take perspective into account and are thus not available in orthographic environments. Furthermore, we identify key ambiguities and unexpected behaviors that arise when performing direct manipulation in 3D and offer solutions to mitigate the difficulties each presents. Finally, we show how to extend our method to meet application-specific control objectives, as well as show our method working in some example environments.
---
paper_title: Two-Finger Gestures for 6DOF Manipulation of 3D Objects

paper_content:

Multitouch input devices afford effective solutions for 6DOF (six Degrees of Freedom) manipulation of 3D objects. Mainly focusing on large-size multitouch screens, existing solutions typically require at least three fingers and bimanual interaction for full 6DOF manipulation. However, single-hand, two-finger operations are preferred especially for portable multitouch devices (e.g., popular smartphones) to cause less hand occlusion and relieve the other hand for necessary tasks like holding the devices. Our key idea for full 6DOF control using only two contact fingers is to introduce two manipulation modes and two corresponding gestures by examining the moving characteristics of the two fingers, instead of the number of fingers or the directness of individual fingers as done in previous works. We solve the resulting binary classification problem using a learning-based approach. Our pilot experiment shows that with only two contact fingers and typically unimanual interaction, our technique is comparable to or even better than the state-of-the-art techniques. © 2012 Wiley Periodicals, Inc.
---
paper_title: The structure of object transportation and orientation in human-computer interaction

paper_content:

An experiment was conducted to investigate the relationship between object transportation and object orientation by the human hand in the context of humancomputer interaction (HCI). This work merges two streams of research: the structure of interactive manipulation in HCI and the natural hand prehension in human motor control. It was found that object transportation and object orientation have a parallel, interdependent structure which is generally persistent over different visual feedback conditions. The notion of concurrency and interdependence of multidimensional visuomotor control structure can provide a new framework for human-computer interface evaluation and design.
---
paper_title: Integrality and Separability of Multitouch Interaction Techniques in 3D Manipulation Tasks

paper_content:

Multitouch displays represent a promising technology for the display and manipulation of data. While the manipulation of 2D data has been widely explored, 3D manipulation with multitouch displays remains largely unexplored. Based on an analysis of the integration and separation of degrees of freedom, we propose a taxonomy for 3D manipulation techniques with multitouch displays. Using that taxonomy, we introduce Depth-Separated Screen-Space (DS3), a new 3D manipulation technique based on the separation of translation and rotation. In a controlled experiment, we compared DS3 with Sticky Tools and Screen-Space. Results show that separating the control of translation and rotation significantly affects performance for 3D manipulation, with DS3 performing faster than the two other techniques.
---
paper_title: Sticky tools: full 6DOF force-based interaction for multi-touch tables

paper_content:

Tabletop computing techniques are using physically familiar force-based interactions to enable compelling interfaces that provide a feeling of being embodied with a virtual object. We introduce an interaction paradigm that has the benefits of force-based interaction complete with full 6DOF manipulation. Only multi-touch input, such as that provided by the Microsoft Surface and the SMART Table, is necessary to achieve this interaction freedom. This paradigm is realized through sticky tools: a combination of sticky fingers, a physically familiar technique for moving, spinning, and lifting virtual objects; opposable thumbs, a method for flipping objects over; and virtual tools, a method for propagating behaviour to other virtual objects in the scene. We show how sticky tools can introduce richer meaning to tabletop computing by drawing a parallel between sticky tools and the discussion in Urp [20] around the meaning of tangible devices in terms of nouns, verbs, reconfigurable tools, attributes, and pure objects. We then relate this discussion to other force-based interaction techniques by describing how a designer can introduce complexity in how people can control both physical and virtual objects, how physical objects can control both physical and virtual objects, and how virtual objects can control virtual objects.
---
paper_title: Two-finger 3D rotations for novice users: surjective and integral interactions

paper_content:

Now that 3D interaction is available on tablets and smart phones, it becomes critical to provide efficient 3D interaction techniques for novice users. This paper investigates interaction techniques for 3D rotation with two fingers of a single hand, on multitouch mobile devices. We introduce two new rotation techniques that allow integral control of the 3 axes of rotation. These techniques also satisfy a new criterion that we introduce: surjection. We ran a study to compare the new techniques with two widely used rotation techniques from the literature. Results indicate that surjection and integration lead to a performance improvement of a group of participants who had no prior experience in 3D interaction. Qualitative results also indicate participants' preference for the new interaction techniques.
---
paper_title: A one-handed multi-touch mating method for 3d rotations

paper_content:

Rotating 3D objects is a difficult task. We present a new rotation technique based on collision-free "mating" to expedite 3D rotations. It is specifically designed for one-handed interaction on tablets or touchscreens. A user study found that our new technique decreased the time to rotate objects in 3D by more than 60% in situations where objects align. We found similar results when users translated and rotated objects in a 3D scene. Also, angle errors were 35% less with mating. In essence, our new rotation technique improves both the speed and accuracy of common 3D rotation tasks.
---
paper_title: Markerless visual fingertip detection for natural mobile device interaction

paper_content:

The vision-based detection of hand gestures is one technological enabler for Natural User Interfaces which try to provide a natural and intuitive interaction with computers. In particular, mobile devices might benefit from such a less device-centric but more natural input possibility. In this paper, we introduce our ongoing work on the visual markerless detection of fingertips on mobile devices. Further, we shed light on the potential of mobile hand gesture detection and present several promising use cases and respective demo applications based on the presented engine.
---
paper_title: PalmSpace: continuous around-device gestures vs. multitouch for 3D rotation tasks on mobile devices

paper_content:

Rotating 3D objects is a difficult task on mobile devices, because the task requires 3 degrees of freedom and (multi-)touch input only allows for an indirect mapping. We propose a novel style of mobile interaction based on mid-air gestures in proximity of the device to increase the number of DOFs and alleviate the limitations of touch interaction with mobile devices. While one hand holds the device, the other hand performs mid-air gestures in proximity of the device to control 3D objects on the mobile device's screen. A flat hand pose defines a virtual surface which we refer to as the PalmSpace for precise and intuitive 3D rotations. We constructed several hardware prototypes to test our interface and to simulate possible future mobile devices equipped with depth cameras. We conducted a user study to compare 3D rotation tasks using the most promising two designs for the hand location during interaction -- behind and beside the device -- with the virtual trackball, which is the current state-of-art technique for orientation manipulation on touch-screens. Our results show that both variants of PalmSpace have significantly lower task completion times in comparison to the virtual trackball.
---
paper_title: Evaluating RGB+D hand posture detection methods for mobile 3D interaction

paper_content:

In mobile applications it is crucial to provide intuitive means for 2D and 3D interaction. A large number of techniques exist to support a natural user interface (NUI) by detecting the user's hand posture in RGB+D (depth) data. Depending on a given interaction scenario, each technique has its advantages and disadvantages. To evaluate the performance of the various techniques on a mobile device, we conducted a systematic study by comparing the accuracy of five common posture recognition approaches with varying illumination and background. To be able to perform this study, we developed a powerful software framework that is capable of processing and fusing RGB and depth data directly on a handheld device. Overall results reveal best recognition rate of posture detection for combined RGB+D data at the expense of update rate. Finally, to support users in choosing the appropriate technique for their specific mobile interaction task, we derived guidelines based on our study.
---
paper_title: Visual gesture interfaces for virtual environments

paper_content:

Virtual environments provide a whole new way of viewing and manipulating 3D data. Current technology moves the images out of desktop monitors and into the space immediately surrounding the user. Users can literally put their hands on the virtual objects. Unfortunately techniques for interacting with such environments have yet to mature. Gloves and sensor based trackers are unwieldy, constraining and uncomfortable to use. A natural, more intuitive method of interaction would be to allow the user to grasp objects with their hands and manipulate them as if they were real objects. We are investigating the use of computer vision in implementing a natural interface based on hand gestures. A framework for a gesture recognition system is introduced along with results of experiments in colour segmentation, feature extraction and template matching for finger and hand tracking and hand pose recognition. Progress in the implementation of a gesture interface for navigation and object manipulation in virtual environments is discussed.
---
paper_title: Image plane interaction techniques in 3D immersive environments

paper_content:

This paper presents a set of interaction techniques for use in headtracked immersive virtual environments. With these techniques, the user interacts with the 2D projections that 3D objects in the scene make on his image plane. The desktop analog is the use of a mouse to interact with objects in a 3D scene based on their projections on the monitor screen. Participants in an immersive environment can use the techniques we discuss for object selection, object manipulation, and user navigation in virtual environments. CR Categories and Subject Descriptors: 1.3.6 [Computer Graphics]: Methodology and Techniques - InteractionTechniques; 1.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism - VirtualReality. Additional Keywords: virtual worlds, virtual environments, navigation, selection, manipulation.
---
paper_title: Two-Finger Gestures for 6DOF Manipulation of 3D Objects

paper_content:

Multitouch input devices afford effective solutions for 6DOF (six Degrees of Freedom) manipulation of 3D objects. Mainly focusing on large-size multitouch screens, existing solutions typically require at least three fingers and bimanual interaction for full 6DOF manipulation. However, single-hand, two-finger operations are preferred especially for portable multitouch devices (e.g., popular smartphones) to cause less hand occlusion and relieve the other hand for necessary tasks like holding the devices. Our key idea for full 6DOF control using only two contact fingers is to introduce two manipulation modes and two corresponding gestures by examining the moving characteristics of the two fingers, instead of the number of fingers or the directness of individual fingers as done in previous works. We solve the resulting binary classification problem using a learning-based approach. Our pilot experiment shows that with only two contact fingers and typically unimanual interaction, our technique is comparable to or even better than the state-of-the-art techniques. © 2012 Wiley Periodicals, Inc.
---
paper_title: Integrality and Separability of Multitouch Interaction Techniques in 3D Manipulation Tasks

paper_content:

Multitouch displays represent a promising technology for the display and manipulation of data. While the manipulation of 2D data has been widely explored, 3D manipulation with multitouch displays remains largely unexplored. Based on an analysis of the integration and separation of degrees of freedom, we propose a taxonomy for 3D manipulation techniques with multitouch displays. Using that taxonomy, we introduce Depth-Separated Screen-Space (DS3), a new 3D manipulation technique based on the separation of translation and rotation. In a controlled experiment, we compared DS3 with Sticky Tools and Screen-Space. Results show that separating the control of translation and rotation significantly affects performance for 3D manipulation, with DS3 performing faster than the two other techniques.
---
paper_title: Dual-Finger 3D Interaction Techniques for mobile devices

paper_content:

Three-dimensional capabilities on mobile devices are increasing, and the interactivity is becoming a key feature of these tools. It is expected that users will actively engage with the 3D content, instead of being passive consumers. Because touch-screens provide a direct means of interaction with 3D content by directly touching and manipulating 3D graphical elements, touch-based interaction is a natural and appealing style of input for 3D applications. However, developing 3D interaction techniques for handheld devices using touch-screens is not a straightforward task. One issue is that when interacting with 3D objects, users occlude the object with their fingers. Furthermore, because the user's finger covers a large area of the screen, the smallest size of the object users can touch is limited. In this paper, we first inspect existing 3D interaction techniques based on their performance with handheld devices. Then, we present a set of precise Dual-Finger 3D Interaction Techniques for a small display. Finally, we present the results of an experimental study, where we evaluate the usability, performance, and error rate of the proposed and existing 3D interaction techniques.
---
paper_title: Sticky tools: full 6DOF force-based interaction for multi-touch tables

paper_content:

Tabletop computing techniques are using physically familiar force-based interactions to enable compelling interfaces that provide a feeling of being embodied with a virtual object. We introduce an interaction paradigm that has the benefits of force-based interaction complete with full 6DOF manipulation. Only multi-touch input, such as that provided by the Microsoft Surface and the SMART Table, is necessary to achieve this interaction freedom. This paradigm is realized through sticky tools: a combination of sticky fingers, a physically familiar technique for moving, spinning, and lifting virtual objects; opposable thumbs, a method for flipping objects over; and virtual tools, a method for propagating behaviour to other virtual objects in the scene. We show how sticky tools can introduce richer meaning to tabletop computing by drawing a parallel between sticky tools and the discussion in Urp [20] around the meaning of tangible devices in terms of nouns, verbs, reconfigurable tools, attributes, and pure objects. We then relate this discussion to other force-based interaction techniques by describing how a designer can introduce complexity in how people can control both physical and virtual objects, how physical objects can control both physical and virtual objects, and how virtual objects can control virtual objects.
---
paper_title: Using custom transformation axes for mid-air manipulation of 3D virtual objects

paper_content:

Virtual Reality environments are able to offer natural interaction metaphors. However, it is difficult to accurately place virtual objects in the desired position and orientation using gestures in mid-air. Previous research concluded that the separation of degrees-of-freedom (DOF) can lead to better results, but these benefits come with an increase in time when performing complex tasks, due to the additional number of transformations required. In this work, we assess whether custom transformation axes can be used to achieve the accuracy of DOF separation without sacrificing completion time. For this, we developed a new manipulation technique, MAiOR, which offers translation and rotation separation, supporting both 3-DOF and 1-DOF manipulations, using personalized axes for the latter. Additionally, it also has direct 6-DOF manipulation for coarse transformations, and scaled object translation for increased placement. We compared MAiOR against an exclusively 6-DOF approach and a widget-based approach with explicit DOF separation. Results show that, contrary to previous research suggestions, single DOF manipulations are not appealing to users. Instead, users favored 3-DOF manipulations above all, while keeping translation and rotation independent.
---
paper_title: A Survey of Augmented Reality

paper_content:

This paper surveys the field of augmented reality AR, in which 3D virtual objects are integrated into a 3D real environment in real time. It describes the medical, manufacturing, visualization, path planning, entertainment, and military applications that have been explored. This paper describes the characteristics of augmented reality systems, including a detailed discussion of the tradeoffs between optical and video blending approaches. Registration and sensing errors are two of the biggest problems in building effective augmented reality systems, so this paper summarizes current efforts to overcome these problems. Future directions and areas requiring further research are discussed. This survey provides a starting point for anyone interested in researching or using augmented reality.
---
paper_title: One-handed interaction with augmented virtual objects on mobile devices

paper_content:

We present a one-handed approach for augmented reality and interaction on mobile devices. The proposed application considers common situations with mobile devices such as when a user's hand holds a mobile device while the other hand is free. It also supports natural augmented reality environment such as when a user interacts with augmented reality contents anytime and anywhere without special equipment such as visual markers or tags. In our approach, a virtual object is augmented on the palm of a user's free hand, as if the virtual object is just sitting on the palm, using a palm pose estimation method. The augmented virtual object reacts (e.g. moving or animation) to motions of the hand such as opening or closing the hand based on fingertip tracking. Moreover, it provides tactile interactions with the virtual object by wearing a tactile glove with vibration sensors. This paper describes how to implement the augmented reality application, and preliminary results show its potential as a new approach to mobile augmented reality interaction.
---
paper_title: Combining multi-touch input and device movement for 3D manipulations in mobile augmented reality environments

paper_content:

Nowadays, handheld devices are capable of displaying augmented environments in which virtual content overlaps reality. To interact with these environments it is necessary to use a manipulation technique. The objective of a manipulation technique is to define how the input data modify the properties of the virtual objects. Current devices have multi-touch screens that can serve as input. Additionally, the position and rotation of the device can also be used as input creating both an opportunity and a design challenge. In this paper we compared three manipulation techniques which namely employ multi-touch, device position and a combination of both. A user evaluation on a docking task revealed that combining multi-touch and device movement yields the best task completion time and efficiency. Nevertheless, using only the device movement and orientation is more intuitive and performs worse only in large rotations.
---
paper_title: Shift: a technique for operating pen-based interfaces using touch

paper_content:

Retrieving the stylus of a pen-based device takes time and requires a second hand. Especially for short intermittent interactions many users therefore choose to use their bare fingers. Although convenient, this increases targeting times and error rates. We argue that the main reasons are the occlusion of the target by the user's finger and ambiguity about which part of the finger defines the selection point. We propose a pointing technique we call Shift that is designed to address these issues. When the user touches the screen, Shift creates a callout showing a copy of the occluded screen area and places it in a non-occluded location. The callout also shows a pointer representing the selection point of the finger. Using this visual feedback, users guide the pointer into the target by moving their finger on the screen surface and commit the target acquisition by lifting the finger. Unlike existing techniques, Shift is only invoked when necessary--over large targets no callout is created and users enjoy the full performance of an unaltered touch screen. We report the results of a user study showing that with Shift participants can select small targets with much lower error rates than an unaided touch screen and that Shift is faster than Offset Cursor for larger targets.
---
paper_title: Mobile augmented reality interaction techniques for authoring situated media on-site

paper_content:

We present a set of mobile augmented reality interaction techniques for authoring situated media: multimedia and hypermedia that are embedded within the physical environment. Our techniques are designed for use with a tracked hand-held tablet display with an attached camera, and rely on "freezing" the frame for later editing.
---
paper_title: Napkin sketch: handheld mixed reality 3D sketching

paper_content:

This paper describes, Napkin Sketch, a 3D sketching interface which attempts to support sketch-based artistic expression in 3D, mimicking some of the qualities of conventional sketching media and tools both in terms of physical properties and interaction experience. A portable tablet PC is used as the sketching platform, and handheld mixed reality techniques are employed to allow 3D sketches to be created on top of a physical napkin. Intuitive manipulation and navigation within the 3D design space is achieved by visually tracking the tablet PC with a camera and mixed reality markers. For artistic expression using sketch input, we improve upon the projective 3D sketching approach with a one stroke sketch plane definition technique. This coupled with the hardware setup produces a natural and fluid sketching experience.
---
paper_title: Experiments in 3D interaction for mobile phone AR

paper_content:

In this paper we present an evaluation of several different techniques for virtual object positioning and rotation on a mobile phone. We compare gesture input captured by the phone's front camera, to tangible input, keypad interaction and phone tilting in increasingly complex positioning and rotation tasks in an AR context. Usability experiments found that tangible input techniques are best for translation tasks, while keypad input is best for rotation tasks. Implications for the design of mobile phone 3D interfaces are presented as well as directions for future research.
---
paper_title: Back-of-device interaction allows creating very small touch devices

paper_content:

In this paper, we explore how to add pointing input capabilities to very small screen devices. On first sight, touchscreens seem to allow for particular compactness, because they integrate input and screen into the same physical space. The opposite is true, however, because the user's fingers occlude contents and prevent precision. We argue that the key to touch-enabling very small devices is to use touch on the device backside. In order to study this, we have created a 2.4" prototype device; we simulate screens smaller than that by masking the screen. We present a user study in which participants completed a pointing task successfully across display sizes when using a back-of device interface. The touchscreen-based control condition (enhanced with the shift technique), in contrast, failed for screen diagonals below 1 inch. We present four form factor concepts based on back-of-device interaction and provide design guidelines extracted from a second user study.
---
paper_title: FingARtips: gesture based direct manipulation in Augmented Reality

paper_content:

This paper presents a technique for natural, fingertip-based interaction with virtual objects in Augmented Reality (AR) environments. We use image processing software and finger- and hand-based fiducial markers to track gestures from the user, stencil buffering to enable the user to see their fingers at all times, and fingertip-based haptic feedback devices to enable the user to feel virtual objects. Unlike previous AR interfaces, this approach allows users to interact with virtual content using natural hand gestures. The paper describes how these techniques were applied in an urban planning interface, and also presents preliminary informal usability results.
---
paper_title: Smartphone as an augmented reality authoring tool via multi-touch based 3D interaction method

paper_content:

In this paper we present an Augmented Reality (AR) authoring tool for smartphones which facilitates intuitive interactions that manipulate the augmented virtual objects in real-time. A novel 3D interaction method using multi-touch interface and camera pose is proposed for intuitive authoring. With the gestures of two fingers on the touch screen, the user can adjust 3 DOF translation and 3 DOF rotation to a selected virtual object. The capabilities of the authoring tool are demonstrated on a smartphone.
---
paper_title: Freeze-Set-Go interaction method for handheld mobile augmented reality environments

paper_content:

Mobile computing devices are getting popular as a platform for augmented reality (AR) application, and efficient interaction methods for mobile AR environments are considered necessary. Recently, touch interfaces are getting popular and drawing attention as a future standard interface on mobile computing devices. However, accurate touch interactions are not that easy in mobile AR environments where users tend to move and viewpoints easily get shaky. In this paper, the authors suggest a new interaction method for handheld mobile AR environments, named 'Freeze-Set-Go'. The proposed interaction method lets users to 'freeze' the real world view tentatively, and continue to manipulate virtual entities within the AR scene. According to the user experiment, the proposed method turns out to be helping users to interact with mobile AR environments using touch interfaces in a more accurate and comfortable way.
---
paper_title: Finger tracking for interaction in augmented environments

paper_content:

Optical tracking systems allow three-dimensional input for virtual environment applications with high precision and without annoying cables. Spontaneous and intuitive interaction is possible through gestures. The authors present a finger tracker that allows gestural interaction and is simple, cheap, fast, robust against occlusion and accurate. It is based on a marked glove, a stereoscopic tracking system and a kinematic 3D model of the human finger. Within our augmented reality application scenario, the user is able to grab, translate, rotate, and release objects in an intuitive way. We demonstrate our tracking system in an augmented reality chess game, allowing a user to interact with virtual objects.
---
paper_title: High Precision Touchscreens: Design Strategies and Comparisons with a Mouse

paper_content:

Abstract Three studies were conducted comparing speed of performance, error rates and user preference ratings for three selection devices. The devices tested were a touchscreen, a touchscreen with stabilization (stabilization software filters and smooths raw data from hardware), and a mouse. The task was the selection of rectangular targets 1, 4, 16 and 32 pixels per side (0·4 × 0·6, 1·7 × 2·2, 6·9 × 9·0, 13·8 × 17·9 mm respectively). Touchscreen users were able to point at single pixel targets, thereby countering widespread expectations of poor touchscreen resolution. The results show no difference in performance between the mouse and touchscreen for targets ranging from 32 to 4 pixels per side. In addition, stabilization significantly reduced the error rates for the touchscreen when selecting small targets. These results imply that touchscreens, when properly used, have attractive advantages in selecting targets as small as 4 pixels per size (approximately one-quarter of the size of a single character). A variant of Fitts' Law is proposed to predict touchscreen pointing times. Ideas for future research are also presented.
---
paper_title: Freeze view touch and finger gesture based interaction methods for handheld augmented reality interfaces

paper_content:

Interaction techniques for handheld mobile Augmented Reality (AR) often focus on device-centric methods based around touch input. However, users may not be able to easily interact with virtual objects in mobile AR scenes if they are holding the handheld device with one hand and touching the screen with the other, while at the same time trying to maintain visual tracking of an AR marker. In this paper we explore novel interaction methods for handheld mobile AR that overcomes this problem. We investigate two different approaches; (1) freeze view touch and (2) finger gesture based interaction. We describe how each method is implemented and present findings from a user experiment comparing virtual object manipulation with these techniques to more traditional touch methods.
---
paper_title: Poster: Real-time markerless kinect based finger tracking and hand gesture recognition for HCI

paper_content:

Hand gestures are intuitive ways to interact with a variety of user interfaces. We developed a real-time finger tracking technique using the Microsoft Kinect as an input device and compared its results with an existing technique that uses the K-curvature algorithm. Our technique calculates feature vectors based on Fourier descriptors of equidistant points chosen on the silhouette of the detected hand and uses template matching to find the best match. Our preliminary results show that our technique performed as well as an existing k-curvature algorithm based finger detection technique.
---
paper_title: Interactions in the air: adding further depth to interactive tabletops

paper_content:

Although interactive surfaces have many unique and compelling qualities, the interactions they support are by their very nature bound to the display surface. In this paper we present a technique for users to seamlessly switch between interacting on the tabletop surface to above it. Our aim is to leverage the space above the surface in combination with the regular tabletop display to allow more intuitive manipulation of digital content in three-dimensions. Our goal is to design a technique that closely resembles the ways we manipulate physical objects in the real-world; conceptually, allowing virtual objects to be 'picked up' off the tabletop surface in order to manipulate their three dimensional position or orientation. We chart the evolution of this technique, implemented on two rear projection-vision tabletops. Both use special projection screen materials to allow sensing at significant depths beyond the display. Existing and new computer vision techniques are used to sense hand gestures and postures above the tabletop, which can be used alongside more familiar multi-touch interactions. Interacting above the surface in this way opens up many interesting challenges. In particular it breaks the direct interaction metaphor that most tabletops afford. We present a novel shadow-based technique to help alleviate this issue. We discuss the strengths and limitations of our technique based on our own observations and initial user feedback, and provide various insights from comparing, and contrasting, our tabletop implementations
---
paper_title: DigitEyes: vision-based hand tracking for human-computer interaction

paper_content:

Computer sensing of hand and limb motion is an important problem for applications in human-computer interaction (HCI), virtual reality, and athletic performance measurement. Commercially available sensors are invasive, and require the user to wear gloves or targets. We have developed a noninvasive vision-based hand tracking system, called DigitEyes. Employing a kinematic hand model, the DigitEyes system has demonstrated tracking performance at speeds of up to 10 Hz, using line and point features extracted from gray scale images of unadorned, unmarked hands. We describe an application of our sensor to a 3D mouse user-interface problem. >
---
paper_title: A handle bar metaphor for virtual object manipulation with mid-air interaction

paper_content:

Commercial 3D scene acquisition systems such as the Microsoft Kinect sensor can reduce the cost barrier of realizing mid-air interaction. However, since it can only sense hand position but not hand orientation robustly, current mid-air interaction methods for 3D virtual object manipulation often require contextual and mode switching to perform translation, rotation, and scaling, thus preventing natural continuous gestural interactions. A novel handle bar metaphor is proposed as an effective visual control metaphor between the user's hand gestures and the corresponding virtual object manipulation operations. It mimics a familiar situation of handling objects that are skewered with a bimanual handle bar. The use of relative 3D motion of the two hands to design the mid-air interaction allows us to provide precise controllability despite the Kinect sensor's low image resolution. A comprehensive repertoire of 3D manipulation operations is proposed to manipulate single objects, perform fast constrained rotation, and pack/align multiple objects along a line. Three user studies were devised to demonstrate the efficacy and intuitiveness of the proposed interaction techniques on different virtual manipulation scenarios.
---
paper_title: Real-time hand-tracking with a color glove

paper_content:

Articulated hand-tracking systems have been widely used in virtual reality but are rarely deployed in consumer applications due to their price and complexity. In this paper, we propose an easy-to-use and inexpensive system that facilitates 3-D articulated user-input using the hands. Our approach uses a single camera to track a hand wearing an ordinary cloth glove that is imprinted with a custom pattern. The pattern is designed to simplify the pose estimation problem, allowing us to employ a nearest-neighbor approach to track hands at interactive rates. We describe several proof-of-concept applications enabled by our system that we hope will provide a foundation for new interactions in modeling, animation control and augmented reality.
---
paper_title: Balloon Selection: A Multi-Finger Technique for Accurate Low-Fatigue 3D Selection

paper_content:

Balloon selection is a 3D interaction technique that is modeled after the real world metaphor of manipulating a helium balloon attached to a string. Balloon selection allows for precise 3D selection in the volume above a tabletop surface by using multiple fingers on a multi-touch-sensitive surface. The 3DOF selection tasks is decomposed in part into a 2DOF positioning task performed by one finger on the tabletop in an absolute 2D Cartesian coordinate system and a 1DOF positioning task performed by another finger on the tabletop in a relative 2D polar coordinate system. We have evaluated balloon selection in a formal user study that compared it to two well-known interaction techniques for selecting a static 3D target: a 3DOF tracked wand and keyboard cursor keys. We found that balloon selection was significantly faster than using cursor keys and had a significantly lower error rate than the wand. The lower error rate appeared to result from the user's hands being supported by the tabletop surface, resulting in significantly reduced hand tremor and arm fatigue.
---
paper_title: The Continuous Interaction Space: Interaction Techniques Unifying Touch and Gesture On and Above a Digital Surface

paper_content:

The rising popularity of digital table surfaces has spawned considerable interest in new interaction techniques. Most interactions fall into one of two modalities: 1) direct touch and multi-touch (by hand and by tangibles) directly on the surface, and 2) hand gestures above the surface. The limitation is that these two modalities ignore the rich interaction space between them. To move beyond this limitation, we first contribute a unification of these discrete interaction modalities called the continuous interaction space. The idea is that many interaction techniques can be developed that go beyond these two modalities, where they can leverage the space between them. That is, we believe that the underlying system should treat the space on and above the surface as a continuum, where a person can use touch, gestures, and tangibles anywhere in the space and naturally move between them. Our second contribution illustrates this, where we introduce a variety of interaction categories that exploit the space between these modalities. For example, with our Extended Continuous Gestures category, a person can start an interaction with a direct touch and drag, then naturally lift off the surface and continue their drag with a hand gesture over the surface. For each interaction category, we implement an example (or use prior work) that illustrates how that technique can be applied. In summary, our primary contribution is to broaden the design space of interaction techniques for digital surfaces, where we populate the continuous interaction space both with concepts and examples that emerge from considering this space as a continuum.
---
paper_title: Robust finger tracking with multiple cameras

paper_content:

This paper gives an overview of a system for robustly tracking the 3D position and orientation of a finger using a few closely spaced cameras. Accurate results are obtained by combining features of stereo range images and color images. This work also provides a design framework for combining multiple sources of information, including stereo range images, color segmentations, shape information and various constraints. This information is used in robust model fitting techniques to track highly over-constrained models of deformable objects: fingers.
---
paper_title: Triangle cursor: interactions with objects above the tabletop

paper_content:

Extending the tabletop display to the third dimension using a stereoscopic projection offers the possibility to improve applications by using the volume above the table surface. The combination of multi-touch input and stereoscopic projection usually requires an indirect technique to interact with objects above the tabletop, as touches can only be detected on the surface. Triangle Cursor is a 3D interaction technique that allows specification of a 3D position and yaw rotation above the interactive tabletop. It was designed to avoid occlusions that disturb the stereoscopic perception. While Triangle Cursor uses an indirect approach, the position, the height above the surface and the yaw rotation can be controlled simultaneously, resulting in a 4 DOF manipulation technique. We have evaluated Triangle Cursor in an initial user study and compared it to a related existing technique in a formal user study. Our experiments show that users were able to perform all tasks significantly faster with our technique without loosing any precision. Most of the subjects considered the technique easy to use and satisfying.
---
paper_title: Using Kinect for hand tracking and rendering in wearable haptics

paper_content:

Wearable haptic devices with poor position sensing are combined with the Kinect depth sensor by Microsoft. A heuristic hand tracker has been developed. It allows for the animation of the hand avatar in the virtual reality and the implementation of the force rendering algorithm: the position of the fingertips is measured by the hand tracker designed and optimized for Kinect, and the rendering algorithm computes the contact forces for wearable haptic display. Preliminary experiments with qualitative results show the effectiveness of the idea of combining Kinect and wearable haptics.
---
paper_title: Integrality and Separability of Multitouch Interaction Techniques in 3D Manipulation Tasks

paper_content:

Multitouch displays represent a promising technology for the display and manipulation of data. While the manipulation of 2D data has been widely explored, 3D manipulation with multitouch displays remains largely unexplored. Based on an analysis of the integration and separation of degrees of freedom, we propose a taxonomy for 3D manipulation techniques with multitouch displays. Using that taxonomy, we introduce Depth-Separated Screen-Space (DS3), a new 3D manipulation technique based on the separation of translation and rotation. In a controlled experiment, we compared DS3 with Sticky Tools and Screen-Space. Results show that separating the control of translation and rotation significantly affects performance for 3D manipulation, with DS3 performing faster than the two other techniques.
---
paper_title: Mid-air interactions above stereoscopic interactive tables

paper_content:

Stereoscopic tabletops offer unique visualization capabilities, enabling users to perceive virtual objects as if they were lying above the surface. While allowing virtual objects to coexist with user actions in the physical world, interaction with these virtual objects above the surface presents interesting challenges. In this paper, we aim to understand which approaches to 3D virtual object manipulations are suited to this scenario. To this end, we implemented five different techniques based on the literature. Four are mid-air techniques, while the remainder relies on multi-touch gestures, which act as a baseline. Our setup combines affordable non-intrusive tracking technologies with a multi-touch stereo tabletop, providing head and hands tracking, to improve both depth perception and seamless interactions above the table. We conducted a user evaluation to find out which technique appealed most to participants. Results suggest that mid-air interactions, combining direct manipulation with six degrees of freedom for the dominant hand, are both more satisfying and efficient than the alternatives tested.
---
paper_title: Visual tracking of bare fingers for interactive surfaces

paper_content:

Visual tracking of bare fingers allows more direct manipulation of digital objects, multiple simultaneous users interacting with their two hands, and permits the interaction on large surfaces, using only commodity hardware. After presenting related work, we detail our implementation. Its design is based on our modeling of two classes of algorithms that are key to the tracker: Image Differencing Segmentation (IDS) and Fast Rejection Filters (FRF). We introduce a new chromatic distance for IDS and a FRF that is independent to finger rotation. The system runs at full frame rate (25 Hz) with an average total system latency of 80 ms, independently of the number of tracked fingers. When used in a controlled environment such as a meeting room, its robustness is satisfying for everyday use.
---
paper_title: Using custom transformation axes for mid-air manipulation of 3D virtual objects

paper_content:

Virtual Reality environments are able to offer natural interaction metaphors. However, it is difficult to accurately place virtual objects in the desired position and orientation using gestures in mid-air. Previous research concluded that the separation of degrees-of-freedom (DOF) can lead to better results, but these benefits come with an increase in time when performing complex tasks, due to the additional number of transformations required. In this work, we assess whether custom transformation axes can be used to achieve the accuracy of DOF separation without sacrificing completion time. For this, we developed a new manipulation technique, MAiOR, which offers translation and rotation separation, supporting both 3-DOF and 1-DOF manipulations, using personalized axes for the latter. Additionally, it also has direct 6-DOF manipulation for coarse transformations, and scaled object translation for increased placement. We compared MAiOR against an exclusively 6-DOF approach and a widget-based approach with explicit DOF separation. Results show that, contrary to previous research suggestions, single DOF manipulations are not appealing to users. Instead, users favored 3-DOF manipulations above all, while keeping translation and rotation independent.
---
paper_title: Mockup Builder: Direct 3D Modeling On and Above the Surface in a Continuous Interaction Space

paper_content:

Our work introduces a semi-immersive environment for conceptual design where virtual mockups are obtained from gestures we aim to get closer to the way people conceive, create and manipulate three-dimensional shapes. We present on-and-above-the-surface interaction techniques following Guiard's asymmetric bimanual model to take advantage of the continuous interaction space for creating and editing 3D models in a stereoscopic environment. To allow for more expressive interactions, our approach continuously combines hand and finger tracking in the space above the table with multi-touch on its surface. This combination brings forth an alternative design environment where users can seamlessly switch between interacting on the surface or in the space above it depending on the task. Our approach integrates continuous space usage with bimanual interaction to provide an expressive set of 3D modeling operations. Preliminary trials with our experimental setup show this as a very promising avenue for further work.
---
paper_title: Gesture-based interaction via finger tracking for mobile augmented reality

paper_content:

The goal of this research is to explore new interaction metaphors for augmented reality on mobile phones, i.e. applications where users look at the live image of the device’s video camera and 3D virtual objects enrich the scene that they see. Common interaction concepts for such applications are often limited to pure 2D pointing and clicking on the device’s touch screen. Such an interaction with virtual objects is not only restrictive but also difficult, for example, due to the small form factor. In this article, we investigate the potential of finger tracking for gesture-based interaction. We present two experiments evaluating canonical operations such as translation, rotation, and scaling of virtual objects with respect to performance (time and accuracy) and engagement (subjective user feedback). Our results indicate a high entertainment value, but low accuracy if objects are manipulated in midair, suggesting great possibilities for leisure applications but limited usage for serious tasks.
---
paper_title: Real-time hand interaction for augmented reality on mobile phones

paper_content:

Over the past few years, Augmented Reality has become widely popular in the form of smart phone applications, however most smart phone-based AR applications are limited in user interaction and do not support gesture-based direct manipulation of the augmented scene. In this paper, we introduce a new AR interaction methodology, employing users' hands and fingers to interact with the virtual (and possibly physical) objects that appear on the mobile phone screen. The goal of this project was to support different types of interaction (selection, transformation, and fine-grain control of an input value) while keeping the methodology for hand detection as simple as possible to maintain good performance on smart phones. We evaluated our methods in user studies, collecting task performance data and user impressions about this direct way of interacting with augmented scenes through mobile phones.
---
paper_title: Free-hand interaction for handheld augmented reality using an RGB-depth camera

paper_content:

In this paper, we present a novel gesture-based interaction method for handheld Augmented Reality (AR) implemented on a tablet with an RGB-Depth camera attached. Compared with conventional device-centric interaction methods like keypad, stylus, or touchscreen input, natural gesture-based interfaces offer a more intuitive experience for AR applications. Combining with depth information, gesture interfaces can extend handheld AR interaction into full 3D space. In our system we retrieve the 3D hand skeleton from color and depth frames, mapping the results to corresponding manipulations of virtual objects in the AR scene. Our method allows users to control virtual objects in 3D space using their bare hands and perform operations such as translation, rotation, and zooming.
---
paper_title: Poster: Markerless fingertip-based 3D interaction for handheld augmented reality in a small workspace

paper_content:

Compared with traditional screen-touch input, natural gesture-based interaction approaches could offer a more intuitive user experience in handheld Augmented Reality (AR) applications. However, most gesture interaction techniques for handheld AR only use two degrees of freedom without the third depth dimension, while AR virtual objects are overlaid on a view of a three dimensional space. In this paper, we investigate a markerless fingertip-based 3D interaction method within a client-server framework in a small workspace. Our solution includes seven major components: (1) fingertip detection (2) fingertip depth acquisition (3) marker tracking (4) coordinate transformation (5) data communication (6) gesture interaction (7) graphic rendering. We describe the process of each step in details and present performance results of our prototype.
---
paper_title: A REVIEW OF 3D GESTURE INTERACTION FOR HANDHELD AUGMENTED REALITY

paper_content:

Interaction for Handheld Augmented Reality (HAR) is a challenging research topic because of the small screen display and limited input options. Although 2D touch screen input is widely used, 3D gesture interaction is a suggested alternative input method. Recent 3D gesture interaction research mainly focuses on using RGB-Depth cameras to detect the spatial position and pose of fingers, using this data for virtual object manipulations in the AR scene. In this paper we review previous 3D gesture research on handheld interaction metaphors for HAR. We present their novelties as well as limitations, and discuss future research directions of 3D gesture interaction for HAR. Our results indicate that 3D gesture input on HAR is a potential interaction method for assisting a user in many tasks such as in education, urban simulation and 3D games.
---
paper_title: 3D gesture interaction for handheld augmented reality

paper_content:

In this paper, we present a prototype for exploring natural gesture interaction with Handheld Augmented Reality (HAR) applications, using visual tracking based AR and freehand gesture based interaction detected by a depth camera. We evaluated this prototype in a user study comparing 3D gesture input methods with traditional touch-based techniques, using canonical manipulation tasks that are common in AR scenarios. We collected task performance data and user feedback via a usability questionnaire. The 3D gesture input methods were found to be slower, but the majority of the participants preferred them and gave them higher usability ratings. Being intuitive and natural was the most common feedback about the 3D freehand interface. We discuss implications of this research and directions for further work.
---
paper_title: Balloon Selection: A Multi-Finger Technique for Accurate Low-Fatigue 3D Selection

paper_content:

Balloon selection is a 3D interaction technique that is modeled after the real world metaphor of manipulating a helium balloon attached to a string. Balloon selection allows for precise 3D selection in the volume above a tabletop surface by using multiple fingers on a multi-touch-sensitive surface. The 3DOF selection tasks is decomposed in part into a 2DOF positioning task performed by one finger on the tabletop in an absolute 2D Cartesian coordinate system and a 1DOF positioning task performed by another finger on the tabletop in a relative 2D polar coordinate system. We have evaluated balloon selection in a formal user study that compared it to two well-known interaction techniques for selecting a static 3D target: a 3DOF tracked wand and keyboard cursor keys. We found that balloon selection was significantly faster than using cursor keys and had a significantly lower error rate than the wand. The lower error rate appeared to result from the user's hands being supported by the tabletop surface, resulting in significantly reduced hand tremor and arm fatigue.
---
paper_title: Real-time hand interaction for augmented reality on mobile phones

paper_content:

Over the past few years, Augmented Reality has become widely popular in the form of smart phone applications, however most smart phone-based AR applications are limited in user interaction and do not support gesture-based direct manipulation of the augmented scene. In this paper, we introduce a new AR interaction methodology, employing users' hands and fingers to interact with the virtual (and possibly physical) objects that appear on the mobile phone screen. The goal of this project was to support different types of interaction (selection, transformation, and fine-grain control of an input value) while keeping the methodology for hand detection as simple as possible to maintain good performance on smart phones. We evaluated our methods in user studies, collecting task performance data and user impressions about this direct way of interacting with augmented scenes through mobile phones.
---
paper_title: Touch and hand gesture-based interactions for directly manipulating 3D virtual objects in mobile augmented reality

paper_content:

Mobile augmented reality (AR) has been widely used in smart and mobile device-based applications such as entertainment, games, visual experience, and information visualization. However, most of the mobile AR applications have limitations in natural user interaction and do not fully support the direct manipulation of 3D AR objects. This paper proposes a new method for naturally and directly manipulating 3D AR objects through touch and hand gesture-based interactions in handheld devices. The touch gesture is used for the AR object selection and the natural hand gesture is used for the direct and interactive manipulation of the selected objects. Thus, the hybrid interaction makes the user more accurately interact with and manipulate AR objects in the real 3D space, not in the 2D space. In particular, natural hand gestures are detected by the Leap Motion sensor attached to the front or back of mobile devices. Thus the user can easily interacts with 3D AR objects for 3D transformation to enhance usability and usefulness. In this research, comprehensive comparative analyses were performed among the proposed approach and the widely used screen touch-based approach and vision-based approach in terms of quantitative and qualitative aspects. Quantitative analysis was conducted by measuring task completion time and failure rate to perform given tasks such as 3D object matching and grasp-hang-release operation. Both tasks require simultaneous 3D translation and 3D rotation. In addition, we have compared the gesture performance depending on whether the gesture sensor is located in the front or the back of the mobile device. Furthermore, to support other complex operations, an assembly task has also been evaluated. The assembly task consists of a sequence of combining parts into a sub-assembly. Qualitative analysis was performed through enquiring questionnaire after the experiment that examines factors such as ease-of-use, ease-of-natural interaction, etc. Both analyses showed that the proposed approach can provide more natural and intuitive interaction and manipulation of mobile AR objects. Several implementation results will also be given to show the advantage and effectiveness of the proposed approach.
---
paper_title: Free-hand interaction for handheld augmented reality using an RGB-depth camera

paper_content:

In this paper, we present a novel gesture-based interaction method for handheld Augmented Reality (AR) implemented on a tablet with an RGB-Depth camera attached. Compared with conventional device-centric interaction methods like keypad, stylus, or touchscreen input, natural gesture-based interfaces offer a more intuitive experience for AR applications. Combining with depth information, gesture interfaces can extend handheld AR interaction into full 3D space. In our system we retrieve the 3D hand skeleton from color and depth frames, mapping the results to corresponding manipulations of virtual objects in the AR scene. Our method allows users to control virtual objects in 3D space using their bare hands and perform operations such as translation, rotation, and zooming.
---
paper_title: Mid-air interactions above stereoscopic interactive tables

paper_content:

Stereoscopic tabletops offer unique visualization capabilities, enabling users to perceive virtual objects as if they were lying above the surface. While allowing virtual objects to coexist with user actions in the physical world, interaction with these virtual objects above the surface presents interesting challenges. In this paper, we aim to understand which approaches to 3D virtual object manipulations are suited to this scenario. To this end, we implemented five different techniques based on the literature. Four are mid-air techniques, while the remainder relies on multi-touch gestures, which act as a baseline. Our setup combines affordable non-intrusive tracking technologies with a multi-touch stereo tabletop, providing head and hands tracking, to improve both depth perception and seamless interactions above the table. We conducted a user evaluation to find out which technique appealed most to participants. Results suggest that mid-air interactions, combining direct manipulation with six degrees of freedom for the dominant hand, are both more satisfying and efficient than the alternatives tested.
---
paper_title: 3D Finger CAPE: Clicking Action and Position Estimation under Self-Occlusions in Egocentric Viewpoint

paper_content:

In this paper we present a novel framework for simultaneous detection of click action and estimation of occluded fingertip positions from egocentric viewed single-depth image sequences. For the detection and estimation, a novel probabilistic inference based on knowledge priors of clicking motion and clicked position is presented. Based on the detection and estimation results, we were able to achieve a fine resolution level of a bare hand-based interaction with virtual objects in egocentric viewpoint. Our contributions include: (i) a rotation and translation invariant finger clicking action and position estimation using the combination of 2D image-based fingertip detection with 3D hand posture estimation in egocentric viewpoint. (ii) a novel spatio-temporal random forest, which performs the detection and estimation efficiently in a single framework. We also present (iii) a selection process utilizing the proposed clicking action detection and position estimation in an arm reachable AR/VR space, which does not require any additional device. Experimental results show that the proposed method delivers promising performance under frequent self-occlusions in the process of selecting objects in AR/VR space whilst wearing an egocentric-depth camera-attached HMD.
---
paper_title: Dynamics of tilt-based browsing on mobile devices

paper_content:

A tilt-controlled photo browsing method for small mobile devices is presented. The implementation uses continuous inputs from an accelerometer, and a multimodal (visual, audio and vibrotactile) display coupled with the states of this model. The model is based on a simple physical model, with its characteristics shaped to enhance usability. We show how the dynamics of the physical model can be shaped to make the handling qualities of the mobile device fit the browsing task. We implemented the proposed algorithm on Samsung MITs PDA with tri-axis accelerometer and a vibrotactile motor. The experiment used seven novice users browsing from 100 photos. We compare a tilt-based interaction method with a button-based browser and an iPod wheel. We discuss the usability performance and contrast this with subjective experience from the users. The iPod wheel has significantly poorer performance than button pushing or tilt interaction, despite its commercial popularity.
---
paper_title: Combining multi-touch input and device movement for 3D manipulations in mobile augmented reality environments

paper_content:

Nowadays, handheld devices are capable of displaying augmented environments in which virtual content overlaps reality. To interact with these environments it is necessary to use a manipulation technique. The objective of a manipulation technique is to define how the input data modify the properties of the virtual objects. Current devices have multi-touch screens that can serve as input. Additionally, the position and rotation of the device can also be used as input creating both an opportunity and a design challenge. In this paper we compared three manipulation techniques which namely employ multi-touch, device position and a combination of both. A user evaluation on a docking task revealed that combining multi-touch and device movement yields the best task completion time and efficiency. Nevertheless, using only the device movement and orientation is more intuitive and performs worse only in large rotations.
---
paper_title: Face to face collaborative AR on mobile phones

paper_content:

Mobile phones are an ideal platform for augmented reality. In this paper we describe how they also can be used to support face to face collaborative AR applications. We have created a custom port of the ARToolKit library to the Symbian mobile phone operating system and then developed a sample collaborative AR game based on this. We describe the game in detail and user feedback from people who have played it. We also provide general design guidelines that could be useful for others who are developing mobile phone collaborative AR applications.
---
paper_title: SlidAR: A 3D positioning method for SLAM-based handheld augmented reality

paper_content:

Abstract Handheld Augmented Reality (HAR) has the potential to introduce Augmented Reality (AR) to large audiences due to the widespread use of suitable handheld devices. However, many of the current HAR systems are not considered very practical and they do not fully answer to the needs of the users. One of the challenging areas in HAR is the in-situ AR content creation where the correct and accurate positioning of virtual objects to the real world is fundamental. Due to the hardware limitations of handheld devices and possible restrictions in the environment, the correct 3D positioning of objects can be difficult to achieve we are unable to use AR markers or correctly map the 3D structure of the environment. We present SlidAR, a 3D positioning for Simultaneous Localization And Mapping (SLAM) based HAR systems. SlidAR utilizes 3D ray-casting and epipolar geometry for virtual object positioning. It does not require a perfect 3D reconstruction of the environment nor any virtual depth cues. We have conducted a user experiment to evaluate the efficiency of SlidAR method against an existing device-centric positioning method that we call HoldAR. Results showed that SlidAR was significantly faster, required significantly less device movement, and also got significantly better subjective evaluation from the test participants. SlidAR also had higher positioning accuracy, although not significantly.
---
paper_title: Virtual object manipulation using a mobile phone

paper_content:

A heat insulating box, such as a refrigerator box, comprises a heat insulator and a box member that is in contact with the heat insulator. The heat insulator is formed of a urethane foam using either HCFC-123 (CHCl2CF3) or HCFC-141b (CH3CCl2F) or both as a forming agent, and the box member is formed of an acrylonitrile/ethylene- PROPORTIONAL -olefinic rubbery polymer/styrene resin (A/epdm/S resin), an acrylonitrile/alkyl acrylate ester rubbery polymer/styrene resin (ASA resin), a mixture of an A/epdm/s resin and an ASA resin, or a mixture of an ASA resin with an acrylonitrile/butadiene/styrene resin.
---
paper_title: A study on improving close and distant device movement pose manipulation for hand-held augmented reality

paper_content:

Hand-held smart devices are equipped with powerful processing units, high resolution screens and cameras, that in combination makes them suitable for video see-through Augmented Reality. Many Augmented Reality applications require interaction, such as selection and 3D pose manipulation. One way to perform intuitive, high precision 3D pose manipulation is by direct or indirect mapping of device movement. There are two approaches to device movement interaction; one fixes the virtual object to the device, which therefore becomes the pivot point for the object, thus makes it difficult to rotate without translate. The second approach avoids latter issue by considering rotation and translation separately, relative to the object's center point. The result of this is that the object instead moves out of view for yaw and pitch rotations. In this paper we study these two techniques and compare them with a modification where user perspective rendering is used to solve the rotation issues. The study showed that the modification improves speed as well as both perceived control and intuitiveness among the subjects.
---
paper_title: Integrated view-input ar interaction for virtual object manipulation using tablets and smartphones

paper_content:

Lately, mobile augmented reality (AR) has become very popular and is used for many commercial and product promotional activities. However, in almost all mobile AR applications, the user only views annotated information or the preset motion of the virtual object in an AR environment and is unable to interact with the virtual objects as if he/she were interacting with real objects in the real environment. In this paper, in an attempt to realize enhanced intuitive and realistic object manipulation in the mobile AR environment, we propose an integrated view-input AR interaction method, which integrates user device manipulation and virtual object manipulation. The method enables the user to hold a 3D virtual object by touching the displayed object on the 2D touch screen of a mobile device, and to move and rotate the object by moving and rotating the mobile device while viewing the held object by way of the 2D screen of the mobile device. Based on this concept, we implemented three types of integrated methods, namely the Rod, Center, and Touch methods, and conducted a user study to investigate the baseline performance metrics of the proposed method on an AR object manipulation task. The Rod method achieved the highest success rate (91%). Participants' feedback indicated that this is because the Rod method is the most natural, and evoked a fixed mental model that is conceivable in the real environment. These results indicated that visualizing the manipulation point on the screen and restricting the user's interactivity with virtual objects from the user's position of view based on a conceivable mental model would be able to aid the user to achieve precise manipulation.
---
paper_title: Mobile phone based AR scene assembly

paper_content:

In this paper we describe a mobile phone based Augmented Reality application for 3D scene assembly. Augmented Reality on mobile phones extends the interaction capabilities on such handheld devices. It adds a 6 DOF isomorphic interaction technique for manipulating 3D content. We give details of an application that we believe to be the first where 3D content can be manipulated using both the movement of a camera tracked mobile phone and a traditional button interface as input for transformations. By centering the scene in a tangible marker space in front of the phone we provide a mean for bimanual interaction. We describe the implementation, the interaction techniques we have developed and initial user response to trying the application.
---
paper_title: An empirical evaluation of virtual hand techniques for 3D object manipulation in a tangible augmented reality environment

paper_content:

In this paper, we present a Fitts' law-based formal evaluation process and the corresponding results for 3D object manipulation techniques based on a virtual hand metaphor in a tangible augmented reality (TAR) environment. Specifically, we extend the design parameters of the 1D scale Fitts' law to 3D scale and then refine an evaluation model in order to bring generality and ease of adaptation to various TAR applications. Next, we implement and compare standard TAR manipulation techniques using a cup, a paddle, a cube, and a proposed extended paddle prop. Most manipulation techniques were well-modeled in terms of linear regression according to Fitts' law, with a correlation coefficient value of over 0.9. Notably, the throughput by ISO 9241-9 of the extended paddle technique peaked at around 1.39 to 2 times higher than in the other techniques, due to the instant 3D positioning of the 3D objects. In the discussion, we subsequently examine the characteristics of the TAR manipulation techniques in terms of stability, speed, comfort, and understanding. As a result, our evaluation process, results, and analysis can be useful in guiding the design and implementation of future TAR interfaces.
---
paper_title: Gesture-based interaction via finger tracking for mobile augmented reality

paper_content:

The goal of this research is to explore new interaction metaphors for augmented reality on mobile phones, i.e. applications where users look at the live image of the device’s video camera and 3D virtual objects enrich the scene that they see. Common interaction concepts for such applications are often limited to pure 2D pointing and clicking on the device’s touch screen. Such an interaction with virtual objects is not only restrictive but also difficult, for example, due to the small form factor. In this article, we investigate the potential of finger tracking for gesture-based interaction. We present two experiments evaluating canonical operations such as translation, rotation, and scaling of virtual objects with respect to performance (time and accuracy) and engagement (subjective user feedback). Our results indicate a high entertainment value, but low accuracy if objects are manipulated in midair, suggesting great possibilities for leisure applications but limited usage for serious tasks.
---
paper_title: Combining multi-touch input and device movement for 3D manipulations in mobile augmented reality environments

paper_content:

Nowadays, handheld devices are capable of displaying augmented environments in which virtual content overlaps reality. To interact with these environments it is necessary to use a manipulation technique. The objective of a manipulation technique is to define how the input data modify the properties of the virtual objects. Current devices have multi-touch screens that can serve as input. Additionally, the position and rotation of the device can also be used as input creating both an opportunity and a design challenge. In this paper we compared three manipulation techniques which namely employ multi-touch, device position and a combination of both. A user evaluation on a docking task revealed that combining multi-touch and device movement yields the best task completion time and efficiency. Nevertheless, using only the device movement and orientation is more intuitive and performs worse only in large rotations.
---
paper_title: A REVIEW OF 3D GESTURE INTERACTION FOR HANDHELD AUGMENTED REALITY

paper_content:

Interaction for Handheld Augmented Reality (HAR) is a challenging research topic because of the small screen display and limited input options. Although 2D touch screen input is widely used, 3D gesture interaction is a suggested alternative input method. Recent 3D gesture interaction research mainly focuses on using RGB-Depth cameras to detect the spatial position and pose of fingers, using this data for virtual object manipulations in the AR scene. In this paper we review previous 3D gesture research on handheld interaction metaphors for HAR. We present their novelties as well as limitations, and discuss future research directions of 3D gesture interaction for HAR. Our results indicate that 3D gesture input on HAR is a potential interaction method for assisting a user in many tasks such as in education, urban simulation and 3D games.
---
paper_title: Shift: a technique for operating pen-based interfaces using touch

paper_content:

Retrieving the stylus of a pen-based device takes time and requires a second hand. Especially for short intermittent interactions many users therefore choose to use their bare fingers. Although convenient, this increases targeting times and error rates. We argue that the main reasons are the occlusion of the target by the user's finger and ambiguity about which part of the finger defines the selection point. We propose a pointing technique we call Shift that is designed to address these issues. When the user touches the screen, Shift creates a callout showing a copy of the occluded screen area and places it in a non-occluded location. The callout also shows a pointer representing the selection point of the finger. Using this visual feedback, users guide the pointer into the target by moving their finger on the screen surface and commit the target acquisition by lifting the finger. Unlike existing techniques, Shift is only invoked when necessary--over large targets no callout is created and users enjoy the full performance of an unaltered touch screen. We report the results of a user study showing that with Shift participants can select small targets with much lower error rates than an unaided touch screen and that Shift is faster than Offset Cursor for larger targets.
---
paper_title: The fat thumb: using the thumb's contact size for single-handed mobile interaction

paper_content:

Modern mobile devices allow a rich set of multi-finger interactions that combine modes into a single fluid act, for example, one finger for panning blending into a two-finger pinch gesture for zooming. Such gestures require the use of both hands: one holding the device while the other is interacting. While on the go, however, only one hand may be available to both hold the device and interact with it. This mostly limits interaction to a single-touch (i.e., the thumb), forcing users to switch between input modes explicitly. In this paper, we contribute the Fat Thumb interaction technique, which uses the thumb's contact size as a form of simulated pressure. This adds a degree of freedom, which can be used, for example, to integrate panning and zooming into a single interaction. Contact size determines the mode (i.e., panning with a small size, zooming with a large one), while thumb movement performs the selected mode. We discuss nuances of the Fat Thumb based on the thumb's limited operational range and motor skills when that hand holds the device. We compared Fat Thumb to three alternative techniques, where people had to precisely pan and zoom to a predefined region on a map and found that the Fat Thumb technique compared well to existing techniques.
---
paper_title: Art of defense: a collaborative handheld augmented reality board game

paper_content:

In this paper, we present Art of Defense (AoD), a cooperative handheld augmented reality (AR) game. AoD is an example of what we call an AR Board Game, a class of tabletop games that combine handheld computers (such as camera phones) with physical game pieces to create a merged physical/virtual game on the table-top. This paper discusses the technical aspects of the game, the design rationale and process we followed, and the resulting player experience. The goal of this research is to explore the affordances and constraints of handheld AR interfaces for collaborative social games, and to create a game that leverages them as fully as possible. The results from the user study show that the game is fun to play, and that by tightly registering the virtual content with the tangible game pieces, tabletop AR games enable a kind of social play experience unlike non-AR computer games. We hope this research will inspire the creation of other handheld augmented reality games in the future, both on and off the tabletop.
---
paper_title: 3D gesture interaction for handheld augmented reality

paper_content:

In this paper, we present a prototype for exploring natural gesture interaction with Handheld Augmented Reality (HAR) applications, using visual tracking based AR and freehand gesture based interaction detected by a depth camera. We evaluated this prototype in a user study comparing 3D gesture input methods with traditional touch-based techniques, using canonical manipulation tasks that are common in AR scenarios. We collected task performance data and user feedback via a usability questionnaire. The 3D gesture input methods were found to be slower, but the majority of the participants preferred them and gave them higher usability ratings. Being intuitive and natural was the most common feedback about the 3D freehand interface. We discuss implications of this research and directions for further work.
---
paper_title: SlidAR: A 3D positioning method for SLAM-based handheld augmented reality

paper_content:

Abstract Handheld Augmented Reality (HAR) has the potential to introduce Augmented Reality (AR) to large audiences due to the widespread use of suitable handheld devices. However, many of the current HAR systems are not considered very practical and they do not fully answer to the needs of the users. One of the challenging areas in HAR is the in-situ AR content creation where the correct and accurate positioning of virtual objects to the real world is fundamental. Due to the hardware limitations of handheld devices and possible restrictions in the environment, the correct 3D positioning of objects can be difficult to achieve we are unable to use AR markers or correctly map the 3D structure of the environment. We present SlidAR, a 3D positioning for Simultaneous Localization And Mapping (SLAM) based HAR systems. SlidAR utilizes 3D ray-casting and epipolar geometry for virtual object positioning. It does not require a perfect 3D reconstruction of the environment nor any virtual depth cues. We have conducted a user experiment to evaluate the efficiency of SlidAR method against an existing device-centric positioning method that we call HoldAR. Results showed that SlidAR was significantly faster, required significantly less device movement, and also got significantly better subjective evaluation from the test participants. SlidAR also had higher positioning accuracy, although not significantly.
---
paper_title: A study on improving close and distant device movement pose manipulation for hand-held augmented reality

paper_content:

Hand-held smart devices are equipped with powerful processing units, high resolution screens and cameras, that in combination makes them suitable for video see-through Augmented Reality. Many Augmented Reality applications require interaction, such as selection and 3D pose manipulation. One way to perform intuitive, high precision 3D pose manipulation is by direct or indirect mapping of device movement. There are two approaches to device movement interaction; one fixes the virtual object to the device, which therefore becomes the pivot point for the object, thus makes it difficult to rotate without translate. The second approach avoids latter issue by considering rotation and translation separately, relative to the object's center point. The result of this is that the object instead moves out of view for yaw and pitch rotations. In this paper we study these two techniques and compare them with a modification where user perspective rendering is used to solve the rotation issues. The study showed that the modification improves speed as well as both perceived control and intuitiveness among the subjects.
---
paper_title: Balloon Selection: A Multi-Finger Technique for Accurate Low-Fatigue 3D Selection

paper_content:

Balloon selection is a 3D interaction technique that is modeled after the real world metaphor of manipulating a helium balloon attached to a string. Balloon selection allows for precise 3D selection in the volume above a tabletop surface by using multiple fingers on a multi-touch-sensitive surface. The 3DOF selection tasks is decomposed in part into a 2DOF positioning task performed by one finger on the tabletop in an absolute 2D Cartesian coordinate system and a 1DOF positioning task performed by another finger on the tabletop in a relative 2D polar coordinate system. We have evaluated balloon selection in a formal user study that compared it to two well-known interaction techniques for selecting a static 3D target: a 3DOF tracked wand and keyboard cursor keys. We found that balloon selection was significantly faster than using cursor keys and had a significantly lower error rate than the wand. The lower error rate appeared to result from the user's hands being supported by the tabletop surface, resulting in significantly reduced hand tremor and arm fatigue.
---
paper_title: The Continuous Interaction Space: Interaction Techniques Unifying Touch and Gesture On and Above a Digital Surface

paper_content:

The rising popularity of digital table surfaces has spawned considerable interest in new interaction techniques. Most interactions fall into one of two modalities: 1) direct touch and multi-touch (by hand and by tangibles) directly on the surface, and 2) hand gestures above the surface. The limitation is that these two modalities ignore the rich interaction space between them. To move beyond this limitation, we first contribute a unification of these discrete interaction modalities called the continuous interaction space. The idea is that many interaction techniques can be developed that go beyond these two modalities, where they can leverage the space between them. That is, we believe that the underlying system should treat the space on and above the surface as a continuum, where a person can use touch, gestures, and tangibles anywhere in the space and naturally move between them. Our second contribution illustrates this, where we introduce a variety of interaction categories that exploit the space between these modalities. For example, with our Extended Continuous Gestures category, a person can start an interaction with a direct touch and drag, then naturally lift off the surface and continue their drag with a hand gesture over the surface. For each interaction category, we implement an example (or use prior work) that illustrates how that technique can be applied. In summary, our primary contribution is to broaden the design space of interaction techniques for digital surfaces, where we populate the continuous interaction space both with concepts and examples that emerge from considering this space as a continuum.
---
paper_title: Real-time hand interaction for augmented reality on mobile phones

paper_content:

Over the past few years, Augmented Reality has become widely popular in the form of smart phone applications, however most smart phone-based AR applications are limited in user interaction and do not support gesture-based direct manipulation of the augmented scene. In this paper, we introduce a new AR interaction methodology, employing users' hands and fingers to interact with the virtual (and possibly physical) objects that appear on the mobile phone screen. The goal of this project was to support different types of interaction (selection, transformation, and fine-grain control of an input value) while keeping the methodology for hand detection as simple as possible to maintain good performance on smart phones. We evaluated our methods in user studies, collecting task performance data and user impressions about this direct way of interacting with augmented scenes through mobile phones.
---
paper_title: Touch and hand gesture-based interactions for directly manipulating 3D virtual objects in mobile augmented reality

paper_content:

Mobile augmented reality (AR) has been widely used in smart and mobile device-based applications such as entertainment, games, visual experience, and information visualization. However, most of the mobile AR applications have limitations in natural user interaction and do not fully support the direct manipulation of 3D AR objects. This paper proposes a new method for naturally and directly manipulating 3D AR objects through touch and hand gesture-based interactions in handheld devices. The touch gesture is used for the AR object selection and the natural hand gesture is used for the direct and interactive manipulation of the selected objects. Thus, the hybrid interaction makes the user more accurately interact with and manipulate AR objects in the real 3D space, not in the 2D space. In particular, natural hand gestures are detected by the Leap Motion sensor attached to the front or back of mobile devices. Thus the user can easily interacts with 3D AR objects for 3D transformation to enhance usability and usefulness. In this research, comprehensive comparative analyses were performed among the proposed approach and the widely used screen touch-based approach and vision-based approach in terms of quantitative and qualitative aspects. Quantitative analysis was conducted by measuring task completion time and failure rate to perform given tasks such as 3D object matching and grasp-hang-release operation. Both tasks require simultaneous 3D translation and 3D rotation. In addition, we have compared the gesture performance depending on whether the gesture sensor is located in the front or the back of the mobile device. Furthermore, to support other complex operations, an assembly task has also been evaluated. The assembly task consists of a sequence of combining parts into a sub-assembly. Qualitative analysis was performed through enquiring questionnaire after the experiment that examines factors such as ease-of-use, ease-of-natural interaction, etc. Both analyses showed that the proposed approach can provide more natural and intuitive interaction and manipulation of mobile AR objects. Several implementation results will also be given to show the advantage and effectiveness of the proposed approach.
---
paper_title: Back-of-device interaction allows creating very small touch devices

paper_content:

In this paper, we explore how to add pointing input capabilities to very small screen devices. On first sight, touchscreens seem to allow for particular compactness, because they integrate input and screen into the same physical space. The opposite is true, however, because the user's fingers occlude contents and prevent precision. We argue that the key to touch-enabling very small devices is to use touch on the device backside. In order to study this, we have created a 2.4" prototype device; we simulate screens smaller than that by masking the screen. We present a user study in which participants completed a pointing task successfully across display sizes when using a back-of device interface. The touchscreen-based control condition (enhanced with the shift technique), in contrast, failed for screen diagonals below 1 inch. We present four form factor concepts based on back-of-device interaction and provide design guidelines extracted from a second user study.
---
paper_title: Free-hand interaction for handheld augmented reality using an RGB-depth camera

paper_content:

In this paper, we present a novel gesture-based interaction method for handheld Augmented Reality (AR) implemented on a tablet with an RGB-Depth camera attached. Compared with conventional device-centric interaction methods like keypad, stylus, or touchscreen input, natural gesture-based interfaces offer a more intuitive experience for AR applications. Combining with depth information, gesture interfaces can extend handheld AR interaction into full 3D space. In our system we retrieve the 3D hand skeleton from color and depth frames, mapping the results to corresponding manipulations of virtual objects in the AR scene. Our method allows users to control virtual objects in 3D space using their bare hands and perform operations such as translation, rotation, and zooming.
---
paper_title: Smartphone as an augmented reality authoring tool via multi-touch based 3D interaction method

paper_content:

In this paper we present an Augmented Reality (AR) authoring tool for smartphones which facilitates intuitive interactions that manipulate the augmented virtual objects in real-time. A novel 3D interaction method using multi-touch interface and camera pose is proposed for intuitive authoring. With the gestures of two fingers on the touch screen, the user can adjust 3 DOF translation and 3 DOF rotation to a selected virtual object. The capabilities of the authoring tool are demonstrated on a smartphone.
---
paper_title: Integrated view-input ar interaction for virtual object manipulation using tablets and smartphones

paper_content:

Lately, mobile augmented reality (AR) has become very popular and is used for many commercial and product promotional activities. However, in almost all mobile AR applications, the user only views annotated information or the preset motion of the virtual object in an AR environment and is unable to interact with the virtual objects as if he/she were interacting with real objects in the real environment. In this paper, in an attempt to realize enhanced intuitive and realistic object manipulation in the mobile AR environment, we propose an integrated view-input AR interaction method, which integrates user device manipulation and virtual object manipulation. The method enables the user to hold a 3D virtual object by touching the displayed object on the 2D touch screen of a mobile device, and to move and rotate the object by moving and rotating the mobile device while viewing the held object by way of the 2D screen of the mobile device. Based on this concept, we implemented three types of integrated methods, namely the Rod, Center, and Touch methods, and conducted a user study to investigate the baseline performance metrics of the proposed method on an AR object manipulation task. The Rod method achieved the highest success rate (91%). Participants' feedback indicated that this is because the Rod method is the most natural, and evoked a fixed mental model that is conceivable in the real environment. These results indicated that visualizing the manipulation point on the screen and restricting the user's interactivity with virtual objects from the user's position of view based on a conceivable mental model would be able to aid the user to achieve precise manipulation.
---
paper_title: The design and evaluation of 3D positioning techniques for multi-touch displays

paper_content:

Multi-touch displays represent a promising technology for the display and manipulation of 3D data. To fully exploit their capabilities, appropriate interaction techniques must be designed. In this paper, we explore the design of free 3D positioning techniques for multi-touch displays to exploit the additional degrees of freedom provided by this technology. Our contribution is two-fold: first we present an interaction technique to extend the standard four view-ports technique found in commercial CAD applications, and second we introduce a technique designed to allow free 3D positioning with a single view of the scene. The two techniques were evaluated in a preliminary experiment. The first results incline us to conclude that the two techniques are equivalent in term of performance showing that the Z-technique provides a real alternative to the statu quo viewport technique.
---
paper_title: A screen-space formulation for 2D and 3D direct manipulation

paper_content:

Rotate-Scale-Translate (RST) interactions have become the de facto standard when interacting with two-dimensional (2D) contexts in single-touch and multi-touch environments. Because the use of RST has thus far focused almost entirely on 2D, there are not yet standard techniques for extending these principles into three dimensions. In this paper we describe a screen-space method which fully captures the semantics of the traditional 2D RST multi-touch interaction, but also allows us to extend these same principles into three-dimensional (3D) interaction. Just like RST allows users to directly manipulate 2D contexts with two or more points, our method allows the user to directly manipulate 3D objects with three or more points. We show some novel interactions, which take perspective into account and are thus not available in orthographic environments. Furthermore, we identify key ambiguities and unexpected behaviors that arise when performing direct manipulation in 3D and offer solutions to mitigate the difficulties each presents. Finally, we show how to extend our method to meet application-specific control objectives, as well as show our method working in some example environments.
---
paper_title: Consumed endurance: a metric to quantify arm fatigue of mid-air interactions

paper_content:

Mid-air interactions are prone to fatigue and lead to a feeling of heaviness in the upper limbs, a condition casually termed as the gorilla-arm effect. Designers have often associated limitations of their mid-air interactions with arm fatigue, but do not possess a quantitative method to assess and therefore mitigate it. In this paper we propose a novel metric, Consumed Endurance (CE), derived from the biomechanical structure of the upper arm and aimed at characterizing the gorilla-arm effect. We present a method to capture CE in a non-intrusive manner using an off-the-shelf camera-based skeleton tracking system, and demonstrate that CE correlates strongly with the Borg CR10 scale of perceived exertion. We show how designers can use CE as a complementary metric for evaluating existing and designing novel mid-air interactions, including tasks with repetitive input such as mid-air text-entry. Finally, we propose a series of guidelines for the design of fatigue-efficient mid-air interfaces.
---
paper_title: Freeze view touch and finger gesture based interaction methods for handheld augmented reality interfaces

paper_content:

Interaction techniques for handheld mobile Augmented Reality (AR) often focus on device-centric methods based around touch input. However, users may not be able to easily interact with virtual objects in mobile AR scenes if they are holding the handheld device with one hand and touching the screen with the other, while at the same time trying to maintain visual tracking of an AR marker. In this paper we explore novel interaction methods for handheld mobile AR that overcomes this problem. We investigate two different approaches; (1) freeze view touch and (2) finger gesture based interaction. We describe how each method is implemented and present findings from a user experiment comparing virtual object manipulation with these techniques to more traditional touch methods.
---
paper_title: 3D Finger CAPE: Clicking Action and Position Estimation under Self-Occlusions in Egocentric Viewpoint

paper_content:

In this paper we present a novel framework for simultaneous detection of click action and estimation of occluded fingertip positions from egocentric viewed single-depth image sequences. For the detection and estimation, a novel probabilistic inference based on knowledge priors of clicking motion and clicked position is presented. Based on the detection and estimation results, we were able to achieve a fine resolution level of a bare hand-based interaction with virtual objects in egocentric viewpoint. Our contributions include: (i) a rotation and translation invariant finger clicking action and position estimation using the combination of 2D image-based fingertip detection with 3D hand posture estimation in egocentric viewpoint. (ii) a novel spatio-temporal random forest, which performs the detection and estimation efficiently in a single framework. We also present (iii) a selection process utilizing the proposed clicking action detection and position estimation in an arm reachable AR/VR space, which does not require any additional device. Experimental results show that the proposed method delivers promising performance under frequent self-occlusions in the process of selecting objects in AR/VR space whilst wearing an egocentric-depth camera-attached HMD.
---
paper_title: Mobile Augmented Reality Survey: From Where We Are to Where We Go

paper_content:

The boom in the capabilities and features of mobile devices, like smartphones, tablets, and wearables, combined with the ubiquitous and affordable Internet access and the advances in the areas of cooperative networking, computer vision, and mobile cloud computing transformed mobile augmented reality (MAR) from science fiction to a reality. Although mobile devices are more constrained computationalwise from traditional computers, they have a multitude of sensors that can be used to the development of more sophisticated MAR applications and can be assisted from remote servers for the execution of their intensive parts. In this paper, after introducing the reader to the basics of MAR, we present a categorization of the application fields together with some representative examples. Next, we introduce the reader to the user interface and experience in MAR applications and continue with the core system components of the MAR systems. After that, we discuss advances in tracking and registration, since their functionality is crucial to any MAR application and the network connectivity of the devices that run MAR applications together with its importance to the performance of the application. We continue with the importance of data management in MAR systems and the systems performance and sustainability, and before we conclude this survey, we present existing challenging problems.
---
paper_title: Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR

paper_content:

Although Augmented Reality technology was first developed over forty years ago, there has been little survey work giving an overview of recent research in the field. This paper reviews the ten-year development of the work presented at the ISMAR conference and its predecessors with a particular focus on tracking, interaction and display research. It provides a roadmap for future augmented reality research which will be of great value to this relatively young field, and also for helping researchers decide which topics should be explored when they are beginning their own studies in the area.
---
paper_title: A Review on Industrial Augmented Reality Systems for the Industry 4.0 Shipyard

paper_content:

Shipbuilding companies are upgrading their inner workings in order to create Shipyards 4.0, where the principles of Industry 4.0 are paving the way to further digitalized and optimized processes in an integrated network. Among the different Industry 4.0 technologies, this paper focuses on augmented reality, whose application in the industrial field has led to the concept of industrial augmented reality (IAR). This paper first describes the basics of IAR and then carries out a thorough analysis of the latest IAR systems for industrial and shipbuilding applications. Then, in order to build a practical IAR system for shipyard workers, the main hardware and software solutions are compared. Finally, as a conclusion after reviewing all the aspects related to IAR for shipbuilding, it proposed an IAR system architecture that combines cloudlets and fog computing, which reduce latency response and accelerate rendering tasks while offloading compute intensive tasks from the cloud.
---
paper_title: Haptic augmented reality interface using the real force response of an object

paper_content:

This paper presents the haptic interface system that consists of a base object and a haptic device. The desired force response is achieved by the combination of the real force response of the base object and the virtual force exerted by the haptic device. The proposed haptic augmented reality (AR) system can easily generate the force response of a visco-elastic object with a cheap haptic device and a base object that has the similar visco-elastic property to the target object. In the demonstration, the force response of the target object was generated by using a haptic device only (VR) and using both a haptic device and a base object (AR), respectively. The evaluation experiments by participants show that the AR method has better performance than the VR method. This result indicates the potential of the proposed haptic AR interface.
---
paper_title: Expected user experience of mobile augmented reality services: a user study in the context of shopping centres

paper_content:

The technical enablers for mobile augmented reality (MAR) are becoming robust enough to allow the development of MAR services that are truly valuable for consumers. Such services would provide a novel interface to the ubiquitous digital information in the physical world, hence serving in great variety of contexts and everyday human activities. To ensure the acceptance and success of future MAR services, their development should be based on knowledge about potential end users' expectations and requirements. We conducted 16 semi-structured interview sessions with 28 participants in shopping centres, which can be considered as a fruitful context for MAR services. We aimed to elicit new knowledge about (1) the characteristics of the expected user experience and (2) central user requirements related to MAR in such a context. From a pragmatic viewpoint, the participants expected MAR services to catalyse their sense of efficiency, empower them with novel context-sensitive and proactive functionalities and raise their awareness of the information related to their surroundings with an intuitive interface. Emotionally, MAR services were expected to offer stimulating and pleasant experiences, such as playfulness, inspiration, liveliness, collectivity and surprise. The user experience categories and user requirements that were identified can serve as targets for the design of user experience of future MAR services.
---
paper_title: Simulating Haptic Feedback Using Vision: A Survey of Research and Applications of Pseudo-Haptic Feedback

paper_content:

This paper presents a survey of the main results obtained in the field of “pseudo-haptic feedback”: a technique meant to simulate haptic sensations in virtual environments using visual feedback and properties of human visuo-haptic perception. Pseudo-haptic feedback uses vision to distort haptic perception and verges on haptic illusions. Pseudo-haptic feedback has been used to simulate various haptic properties such as the stiffness of a virtual spring, the texture of an image, or the mass of a virtual object. This paper describes the several experiments in which these haptic properties were simulated. It assesses the definition and the properties of pseudo-haptic feedback. It also describes several virtual reality applications in which pseudo-haptic feedback has been successfully implemented, such as a virtual environment for vocational training of milling machine operations, or a medical simulator for training in regional anesthesia procedures.
---
paper_title: Current issues in handheld augmented reality

paper_content:

Equipped with powerful processors, cameras for capturing still images and video, and a range of sensors capable of tracking location, orientation and motion of the user, modern smartphones offer a sophisticated platform for implementing handheld augmented reality (AR) applications. Despite the advances in research and development, implementing AR applications for the smartphone platform remains a challenge due to many open problems related to navigation, contextawareness, visualization, usability and interaction design, as well as content creation and sharing. This paper surveys a number of open challenges and issues, trade-offs and possible solutions in implementing handheld AR applications.
---
paper_title: Integrated view-input ar interaction for virtual object manipulation using tablets and smartphones

paper_content:

Lately, mobile augmented reality (AR) has become very popular and is used for many commercial and product promotional activities. However, in almost all mobile AR applications, the user only views annotated information or the preset motion of the virtual object in an AR environment and is unable to interact with the virtual objects as if he/she were interacting with real objects in the real environment. In this paper, in an attempt to realize enhanced intuitive and realistic object manipulation in the mobile AR environment, we propose an integrated view-input AR interaction method, which integrates user device manipulation and virtual object manipulation. The method enables the user to hold a 3D virtual object by touching the displayed object on the 2D touch screen of a mobile device, and to move and rotate the object by moving and rotating the mobile device while viewing the held object by way of the 2D screen of the mobile device. Based on this concept, we implemented three types of integrated methods, namely the Rod, Center, and Touch methods, and conducted a user study to investigate the baseline performance metrics of the proposed method on an AR object manipulation task. The Rod method achieved the highest success rate (91%). Participants' feedback indicated that this is because the Rod method is the most natural, and evoked a fixed mental model that is conceivable in the real environment. These results indicated that visualizing the manipulation point on the screen and restricting the user's interactivity with virtual objects from the user's position of view based on a conceivable mental model would be able to aid the user to achieve precise manipulation.
---

Title: 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review

Section 1: INTRODUCTION
Description 1: Write an introduction that defines augmented reality (AR) using various references, explains its importance, and highlights the focus of the survey.

Section 2: MOTIVATION
Description 2: Discuss the motivation behind studying 3D object manipulation techniques in handheld mobile AR, highlighting the lack of comprehensive overviews on the topic.

Section 3: HANDHELD MOBILE AR INTERFACE
Description 3: Describe the significance of handheld mobile AR interfaces, differences from traditional desktop/tabletop AR interfaces, and focus on three major interaction parts: tangible user interaction, multimodal input, and mobile interaction.

Section 4: TOUCH-BASED INTERACTION
Description 4: Present touch-based interaction techniques for 3D object manipulation in handheld mobile AR, including the challenges and solutions proposed by various studies.

Section 5: 3D OBJECT MANIPULATION FOR HANDHELD MOBILE DISPLAYS
Description 5: Explain how touch-based interaction techniques have been adapted for small touch-based displays such as smartphones and discuss their effectiveness in 3D object manipulation.

Section 6: MID-AIR GESTURES-BASED INTERACTION
Description 6: Detail mid-air gestures-based interaction techniques for 3D object manipulation in handheld mobile AR, including the historical development and recent advancements in the field.

Section 7: 3D OBJECT MANIPULATION FOR HANDHELD MOBILE AR
Description 7: Discuss how mid-air gestures-based interaction techniques have been applied and evaluated in handheld mobile AR environments, and compare them with other interaction techniques.

Section 8: DEVICE-BASED INTERACTION
Description 8: Describe device-based interaction techniques where the device itself is used for 3D object manipulation, and present examples of early implementations and recent advancements.

Section 9: REMAINING ISSUES IN 3D OBJECT MANIPULATION WITHIN HANDHELD MOBILE AR
Description 9: Highlight the primary remaining issues in 3D object manipulation within handheld mobile AR, including occlusion, fatigue, position mismatch, and the need for prior knowledge.

Section 10: CONCLUSION AND FUTURE DIRECTIONS
Description 10: Conclude the survey by summarizing key findings and discussing potential future research directions in 3D object manipulation techniques for handheld mobile AR.