Virtual and Mixed Reality in Telerobotics: A Survey
---
paper_title: The phantom robot: predictive displays for teleoperation with time delay

paper_content:

An enhanced teleoperation technique for time-delayed bilateral teleoperator control is discussed. The control technique selected for time delay is based on the use of a high-fidelity graphics phantom robot that is being controlled in real time (without time delay) against the static task image. Thus, the motion of the phantom robot image on the monitor predicts the motion of the real robot. The real robot's motion will follow the phantom robot's motion on the monitor with the communication time delay implied in the task. Real-time high-fidelity graphics simulation of a PUMA arm is generated and overlaid on the actual camera view of the arm. A simple camera calibration technique is used for calibrated graphics overlay. A preliminary experiment is performed with the predictive display by using a very simple tapping task. The results with this simple task indicate that predictive display enhanced the human operator's telemanipulation task performance significantly during free motion when there is a long time delay. It appears, however, that either two-view or stereoscopic predictive displays are necessary for general three-dimensional tasks. >
---
paper_title: Stable Adaptive Teleoperation

paper_content:

Telerobotics, the body of science and technology which bridges human control and purely autonomous machines, is expected to be a merging point of modern developments in robotics, control theory, cognitive science, machine design, and computer science. Besides traditional applications in space, subsea, and handling of hazardous material, many new potential uses of advanced telerobotic systems have recently been suggested or explored, such as safety applications or microsurgery. This paper is a preliminary study of what role recent results in adaptive robot control and the understanding of time-delays may play in the development of effective telerobotics systems, which would best exploit the presence of the human operator while making full use of available robot control technology and computing power. The key paradigm it explores is that of simplifying, transforming, or enhancing the remote dynamics perceived by the operator by proper use of adaptive robot control techniques and tools from passivity theory.
---
paper_title: Sensor-based space robotics-ROTEX and its telerobotic features

paper_content:

In early 1993 the space robot technology experiment ROTEX was flown with space-shuttle Columbia. A multisensory robot onboard the spacecraft successfully worked in autonomous modes, teleoperated by astronauts, as well as in different telerobotic ground control modes. These included online teleoperational and telesensor-programming: a task-level oriented programming technique involving learning-by-showing concepts in a virtual environment. The robot's key features were its multisensory gripper and the local sensory feedback schemes that are the basis for shared autonomy. The corresponding man-machine interface concepts, which use a six-degree-of-freedom non-force-reflecting control ball and visual feedback to the human operator, are explained. Stereographic simulation on the ground was used to predict not only the robot's free motion but even the sensor-based path refinement onboard. Prototype tasks performed by this space robot were the assembly of a truss structure, connecting/disconnecting an electrical plug (orbit replaceable unit exchange), and grasping free-floating objects. >
---
paper_title: Imposing motion constraints to a force reflecting telerobot through real-time simulation of a virtual mechanism

paper_content:

In a teleoperation system, assistance can be given to the operator by constraining the telerobot position to remain within a restricted subspace of its workspace. A new approach to motion constraint is presented in this paper. The control law is established simulating a virtual ideal mechanism acting as a jig, and connected to the master and slave arms via springs and dampers. Using this approach, it is possible to impose any (sufficiently smooth) motion constraint to the system, including non-linear constraints (complex surfaces) involving coupling between translations and rotations. Physical equivalence ensures that the controller is passive. Experimental results obtained with a 6-DOF teleoperation system are given. Other applications of the virtual mechanism concept include hybrid position-force control and haptic interfaces.
---
paper_title: The use of Virtual Fixtures to enhance telemanipulation with time delay

paper_content:

An improved process is described for the preparation of 5-lower alkyl-1-phenyl-1,3,4,6-tetrahydro-5H-benz[f]-2,5-oxazocines in which 2-(N-hydroxyethyl=N=lower alkylaminomethyl)-benzhydrols are cyclized in the presence of aqueous hydrogen halide.
---
paper_title: Telerobotics, Automation, and Human Supervisory Control

paper_content:

Theory and models of supervisory control - frameworks and fragments supervisory control of anthropomorphic teleoperators for space, undersea, and other applications supervisory control in transportation, process, and other automated systems social implications of telerobotics, automation, and supervisory control.
---
paper_title: Projective virtual reality: bridging the gap between virtual reality and robotics

paper_content:

The new paradigm of projective virtual reality is presented as a sophisticated new approach to the control and supervision of robotic and automation systems. The idea behind projective virtual reality is to "project" actions that are carried out by users in the virtual world into the real world, primarily by means of robots but also by other means of automation. Robot control technology thus provides the user in the virtual reality with a "prolonged arm" into the physical environment, paving the way for a new quality of a user friendly man machine interface for automation applications. The paper provides the background of projective virtual reality which is based on latest results in the fields of task planning, world modeling, cooperative robot control and sensor-based robot control.
---
paper_title: Multi-robot teleoperation using direct human hand actions

paper_content:

This paper presents a novel concept for teleoperation using direct human hand(s) actions which we called 'the hidden robot' concept. The proposed teleoperation scheme is composed of three main components: the operator/ computer loop, the execution loop and between them the bilateral transformation modules linked by the communication channel. Within the operator/computer master loop, the operator performs what we call a 'virtual task', without being constrained by the slave robot. At this stage, the bilateral transformation layer is in charge of extracting, at the low level, pertinent parameters from the virtual task and transforming them onto robot control signals. The execution loop performs control of the slave robot(s) to achieve the desired task, described by the virtual one. At this stage, the transformation layer extracts pertinent data to provide feedback when possible and needed. It also makes sure that the task is being performed correctly in the real site; otherwise, it takes necessary recovery ...
---
paper_title: Web Interfaces for Mobile Robots in Public Places

paper_content:

The growth of the World Wide Web provides unique opportunities to bring robots closer to people. The vision behind such endeavors ranges from relatively simple web-based inspections and surveillance applications to highly versatile applications that use robots connected to the web to establish a remote telepresence in dynamic and populated environments. In the latter scenario, robots play the role of a physical mediator, enabling remote people to acquire information, explore, manipulate, communicate, and interact physically with people far away. The article describes a series of web interfaces designed to remotely operate mobile robots in public places through the web. The design of these interfaces specifically addresses issues such as low bandwidth of interconnections, control brokering, and shared control as well as interaction with people in the robot's environment, which arise naturally in applications with web-based robot control. The interfaces have been tested extensively using two deployed service robots, which were installed as interactive tour guides in two museums. The article also discusses trade-offs and limitations of web-based robots that interact with people in populated public places.
---
paper_title: Internet robots: a new robotics niche

paper_content:

Web robots are a concept that has captured the interest of researchers and developers. Their potential is being investigated at the University of Western Australia, where a six-axis robot has been made teleoperable over the World Wide Web. The site has been active since September 1994 and the robot has also been exhibited at the Australian National Science and Technology Centre in Canberra. Someone considering offering services with this technology will be interested in which groups of people find it attractive, the level of commitment that operators show, and how the robot users behave. These questions are addressed in the article with some demographic data on operators, an analysis of how long they remain interested, and an examination of how long they will wait to access the service. The user interface affects operator behavior, so some interface variations and their effect on operator actions are considered. The opportunities presented, likely application areas, and the potential for generating an income are examined.
---
paper_title: MINERVA: a second-generation museum tour-guide robot

paper_content:

This paper describes an interactive tour-guide robot, which was successfully exhibited in a Smithsonian museum. During its two weeks of operation, the robot interacted with thousands of people, traversing more than 44 km at speeds of up to 163 cm/sec. Our approach specifically addresses issues such as safe navigation in unmodified and dynamic environments, and short-term human-robot interaction. It uses learning pervasively at all levels of the software architecture.
---
paper_title: Introduction: the unique phenomenon of a distance

paper_content:

D esig ning In/ormation Technology, Richard Coyne, 1995 Technoromanticism: Digital Narrative, Holism, and the Romanc e 0/ the Real, Richard Coyne, 1999 The Visual Mind, edited by Michele Emmer, 1994 The Robot in the Garden: Telerobotic s and Telepist emology in the Ag e 0/ the Internet, edited by Ken Goldberg, 2000 Leonard o Almanac , edited by Craig Harris, 1994 In Search o/ Innovation: The Xerox PARe PAIR Project, edited by Craig Harris, 1999 The Digital Dialectic: New Essays on New M edia, edited by Peter lunenfeld, 1999 Immersed in Technology: Art and Virtual Environments, edited by Mary Anne Moser with Douglas Macleod, 1996 The Robot in the Garden
---
paper_title: VR-based Teleoperation of a Mobile Robotic Assistant: Progress Report

paper_content:

This report presents work in progress within the framework of a research project aiming at the development of a mobile robotic system to perform assistive tasks in a hospital environment. This robotic assistant will consist of a mobile robot platform equipped with a variety of on-board sensing and processing equipment as well as a small manipulator for performing simple fetch-and-carry operations. In this report, we focus on the design of the teleoperation system integrating virtual reality techniques and Web-based capabilities in the human operator interface. Relative work found in the literature in the fleld of intervention and service telerobotics is reviewed, and an overview of the methodologies that will be followed is presented. Some speciflc issues requiring particular attention for the design of a teleoperation system for the mobile robotic assistant are investigated and include: (a) the speciflcation of the teleoperation modes supported by the system, integrating various automatic computer assistance and shared-autonomy behaviour-based control modes, (b) the design of the user interface, built on Java technology to enable web-operation and support various multimodal VR-based functionalities, and (c) the integration with the other subsystems and control modules of the mobile robotic assistant, in the framework of a general teleoperation/telemanipulation control architecture.
---
paper_title: Teleplanning by human demonstration for VR-based teleoperation of a mobile robotic assistant

paper_content:

Focuses on the integration of local path planning techniques in a multimodal teleoperation interface, for the efficient remote control of a mobile robotic assistant. The main principle underlying this scheme is related to finding new ways to establish an efficient human-robot cooperation framework, where humans and robots take charge of the parts of the tasks that they can perform more efficiently. For the teleoperation of a mobile robotic platform, a simple application of this general principle could be to commit the human operator in performing the necessary global planning operations, which are more demanding in terms of complex reasoning and required "intelligence", while other more local tasks such as collision avoidance and trajectory optimization are dedicated to the telerobotic system. We propose an implementation of this principle within a mobile robot teleoperation interface integrating virtual reality techniques and Web standards. The paper describes the multimodal interface and the design principles followed, as well as the integration of a local path planning method. This scheme, called "computer-assisted teleplanning by human demonstration", aims at providing active assistance to the human operator, enabling him to indicate in a natural way the desired global motion plan, for a more efficient teleoperation of a mobile robotic assistant.
---
paper_title: Visuo-Haptic Interface for Teleoperation of Mobile Robot Exploration Tasks

paper_content:

With the spread of low-cost haptic devices, haptic interfaces appear in many areas in the field of robotics. Recently, haptic devices have been used in the field of mobile robot teleoperation, where mobile robots operate in unknown and dangerous environments performing particular tasks. Haptic feedback is shown to improve operator perception of the environment without, however, improving exploration time. In this paper, we present a haptic interface that is used to teleoperate a mobile robot in exploring polygonal environments. The proposed visuo-haptic interface is found to improve navigation time and operator perception of the remote environment. The human-operator can simultaneously select two different commands, the first one being set as "active" motion command, while the second one is set as a "guarded" motion type of navigation command. The user can feel a haptic equivalent for both types of teleguidance motion commands, and can also observe in real-time the sequential creation of the remote environment map. Comparative evaluation experiments show that the proposed system makes the task of remote navigation of unknown environments easier.
---
paper_title: Telerobotic manipulator for hazardous environments

paper_content:

The Center for Engineering Design at the University of Utah, in collaboration with Sarcos Research Corporation and others, recently developed the Dextrous Teleoperation System (DTS). The DTS is a master/slave 10 degree of freedom (dof) arm that includes a 3 dof dextrous end effector. The system has demonstrated a number of significant remote manipulation capabilities. This article discusses design considerations, features, and capabilities of the slave Dextrous Arm. An earlier article discussed potential applications, general features, and system capabilities of the DTS. Subsequent articles will discuss the master and control systems.
---
paper_title: Bilateral Teleoperation of Mobile Robot over Delayed Communication Network: Implementation.

paper_content:

In a previous paper we proposed a bilateral teleoperation framework of a wheeled mobile robot over communication channel with constant time delay. In this paper we present experimental results Our goal is to illustrate and validate the properties of the proposed scheme as well as to present practical implementation issues and the adopted solutions. In particular, the bilaterally teleoperated system is passive and the system is stable in the presence of time delay. Internet has been used as the communication channel and a buffer has been implemented to maintain a constant time delay and to handle packet order.
---

Title: Virtual and Mixed Reality in Telerobotics: A Survey

Section 1: Introduction
Description 1: This section introduces the concepts of Virtual Reality (VR) and Augmented Reality (AR), along with their significance and multidisciplinary nature. It outlines the scope of VR and AR in enhancing human-computer interaction and their potential applications.

Section 2: Augmented Reality: Basics
Description 2: This section explains the fundamentals of Augmented Reality systems, their principles, and the diverse application domains including medical, industrial, and entertainment fields.

Section 3: Overview of VR Applications in Robotics
Description 3: This section provides an overview of how VR and AR technologies are applied in robotics, particularly in enhancing manufacturing processes, programming, control, and assessment of robotic systems.

Section 4: Telerobotics: Historical Evolution
Description 4: This section traces the historical development of telerobotics, highlighting key milestones, the fusion of telemanipulation and robotic technologies, and the creation of telerobotics as an advanced field.

Section 5: Teleoperation and Virtual Reality: Synergy
Description 5: This section discusses the integration of virtual reality techniques in teleoperation systems, detailing how VR can serve as predictive models and aid in off-line teleprogramming schemes to improve teleoperation performance.

Section 6: Web-Based Telerobots
Description 6: This section surveys the development and current state of web-based telerobotic systems, detailing how these systems operate over the Internet and the challenges they face, particularly time delay and user interface limitations.

Section 7: VR in Telerobotics: Application Scenarios
Description 7: This section provides examples of VR-based teleoperation systems for both robot manipulators and mobile robotic platforms, demonstrating how VR can enhance the transparency and stability of these systems.

Section 8: Robot Telemanipulators
Description 8: This section describes specific applications of VR in the telemanipulation of robots, including long-distance multirobot teleoperation and virtual laboratories for remote training.

Section 9: Mobile Robotics
Description 9: This section covers the application of VR and AR in the teleoperation of mobile robots, addressing the challenges and illustrating successful implementations in scenarios like underwater robots and space exploration robots.

Section 10: Teleoperation of a Mobile Robotic Assistant
Description 10: This section details the teleoperation system for a mobile robotic assistant, including the teleoperation interface design and the modes of operation ranging from direct control to shared autonomy.

Section 11: Haptic Teleoperation for Remote Exploration
Description 11: This section examines the integration of haptic feedback in the teleoperation of mobile robots, showcasing how haptics can improve the operator's perception and control of the remote environment.

Section 12: Conclusion - Future Research Directions
Description 12: This section summarizes the survey and discusses future research directions in VR and telerobotics, emphasizing the potential for VR to revolutionize human-robot interaction and the development of advanced telerobotic systems.