Advanced Medical Displays: A Literature Review of Augmented Reality
---
paper_title: A Survey of Augmented Reality

paper_content:

This paper surveys the field of augmented reality AR, in which 3D virtual objects are integrated into a 3D real environment in real time. It describes the medical, manufacturing, visualization, path planning, entertainment, and military applications that have been explored. This paper describes the characteristics of augmented reality systems, including a detailed discussion of the tradeoffs between optical and video blending approaches. Registration and sensing errors are two of the biggest problems in building effective augmented reality systems, so this paper summarizes current efforts to overcome these problems. Future directions and areas requiring further research are discussed. This survey provides a starting point for anyone interested in researching or using augmented reality.
---
paper_title: A frameless stereotaxic integration of computerized tomographic imaging and the operating microscope.

paper_content:

A computer-based system has been developed for the integration and display of computerized tomography (CT) image data in the operating microscope in the correct perspective without requiring a stereotaxic frame. Spatial registration of the CT image data is accomplished by determination of the position of the operating microscope as its focal point is brought to each of three CT-imaged fiducial markers on the scalp. Monitoring of subsequent microscope positions allows appropriate reformatting of CT data into a common coordinate system. The position of the freely moveable microscope is determined by a non-imaging ultrasonic range-finder consisting of three spark gaps attached to the microscope and three microphones on a rigid support in the operating room. Measurement of the acoustic impulse transit times from the spark gaps to the microphones enables calculation of those distances and unique determination of the microscope position. The CT data are reformatted into a plane and orientation corresponding to the microscope's focal plane or to a deeper parallel plane if required. This reformatted information is then projected into the optics of the operating microscope using a miniature cathode ray tube and a beam splitter. The operating surgeon sees the CT information (such as a tumor boundary) superimposed upon the operating field in proper position, orientation, and scale.
---
paper_title: Recent Advances in Augmented Reality

paper_content:

In 1997, Azuma published a survey on augmented reality (AR). Our goal is to complement, rather than replace, the original survey by presenting representative examples of the new advances. We refer one to the original survey for descriptions of potential applications (such as medical visualization, maintenance and repair of complex equipment, annotation, and path planning); summaries of AR system characteristics (such as the advantages and disadvantages of optical and video approaches to blending virtual and real, problems in display focus and contrast, and system portability); and an introduction to the crucial problem of registration, including sources of registration error and error-reduction strategies.
---
paper_title: The computer scientist as toolsmith II

paper_content:

A process for producing a photographic film having thereon a magnetic recording layer by applying a dispersion of a magnetic substance to an antihalation layer of a light-sensitive film for cinema, where the dispersion contains a compound or compounds, having at least two or more isocyanato groups or thioisocyanato groups or the anti-halation layer is previously processed with a solution containing the compound. A photographic film where either the magnetic layer contains, or the antihalation layer has been processed with, such a compound or compounds.
---
paper_title: A TAXONOMY OF MIXED REALITY VISUAL DISPLAYS

paper_content:

Paul Milgram received the B.A.Sc. degree from the University of Toronto in 1970, the M.S.E.E. degree from the Technion (Israel) in 1973 and the Ph.D. degree from the University of Toronto in 1980. From 1980 to 1982 he was a ZWO Visiting Scientist and a NATO Postdoctoral in the Netherlands, researching automobile driving behaviour. From 1982 to 1984 he was a Senior Research Engineer in Human Engineering at the National Aerospace Laboratory (NLR) in Amsterdam, where his work involved the modelling of aircraft flight crew activity, advanced display concepts and control loops with human operators in space teleoperation. Since 1986 he has worked at the Industrial Engineering Department of the University of Toronto, where he is currently an Associate Professor and Coordinator of the Human Factors Engineering group. He is also cross appointed to the Department of Psychology. In 1993-94 he was an invited researcher at the ATR Communication Systems Research Laboratories, in Kyoto, Japan. His research interests include display and control issues in telerobotics and virtual environments, stereoscopic video and computer graphics, cognitive engineering, and human factors issues in medicine. He is also President of Translucent Technologies, a company which produces "Plato" liquid crystal visual occlusion spectacles (of which he is the inventor), for visual and psychomotor research.
---
paper_title: Analysis of head pose accuracy in augmented reality

paper_content:

A method is developed to analyze the accuracy of the relative head-to-object position and orientation (pose) in augmented reality systems with head-mounted displays. From probabilistic estimates of the errors in optical tracking sensors, the uncertainty in head-to-object pose can be computed in the form of a covariance matrix. The positional uncertainty can be visualized as a 3D ellipsoid. One useful benefit of having an explicit representation of uncertainty is that we can fuse sensor data from a combination of fixed and head-mounted sensors in order to improve the overall registration accuracy. The method was applied to the analysis of an experimental augmented reality system, incorporating an optical see-through head-mounted display, a head-mounted CCD camera, and a fixed optical tracking sensor. The uncertainty of the pose of a movable object with respect to the head-mounted display was analyzed. By using both fixed and head mounted sensors, we produced a pose estimate that is significantly more accurate than that produced by either sensor acting alone.
---
paper_title: Optical Versus Video See-Through Head-Mounted Displays in Medical Visualization

paper_content:

We compare two technological approaches to augmented reality for 3-D medical visualization: optical and video see-through devices. We provide a context to discuss the technology by reviewing several medical applications of augmented-reality re search efforts driven by real needs in the medical field, both in the United States and in Europe. We then discuss the issues for each approach, optical versus video, from both a technology and human-factor point of view. Finally, we point to potentially promising future developments of such devices including eye tracking and multifocus planes capabilities, as well as hybrid optical/video technology.
---
paper_title: Head-worn displays: a review

paper_content:

Head-worn display design is inherently an interdisciplinary subject fusing optical engineering, optical materials, optical coatings, electronics, manufacturing techniques, user interface design, computer science, human perception, and physiology for assessing these displays. This paper summarizes the state-of-the-art in head-worn display design (HWD) and development. This review is focused on the optical engineering aspects, divided into different sections to explore principles and applications. Building on the guiding fundamentals of optical design and engineering, the principles section includes a summary of microdisplay or laser sources, the Lagrange invariant for understanding the trade-offs in optical design of HWDs, modes of image presentation (i.e., monocular, biocular, and stereo) and operational modes such as optical and video see-through. A brief summary of the human visual system pertinent to the design of HWDs is provided. Two optical design forms, namely, pupil forming and non-pupil forming are discussed. We summarize the results from previous design work using aspheric, diffractive, or holographic elements to achieve compact and lightweight systems. The applications section is organized in terms of field of view requirements and presents a reasonable collection of past designs
---
paper_title: Technologies for augmented reality systems: realizing ultrasound-guided needle biopsies

paper_content:

We present a real-time stereoscopic video-see-through augmented reality (AR) system applied to the medical procedure known as ultrasound-guided needle biopsy of the breast. The AR system was used by a physician during procedures on breast models and during non-invasive examinations of human subjects. The system merges rendered live ultrasound data and geometric elements with stereo images of the patient acquired through head-mounted video cameras and presents these merged images to the physician in a head-mounted display. The physician sees a volume visualization of the ultrasound data directly under the ultrasound probe, properly registered within the patient and with the biopsy needle. Using this system, a physician successfully guided a needle into an artificial tumor within a training phantom of a human breast. We discuss the construction of the AR system and the issues and decisions which led to the system architecture and the design of the video see-through head-mounted display. We designed methods to properly resolve occlusion of the real and synthetic image elements. We developed techniques for realtime volume visualization of timeand position-varying ultrasound data. We devised a hybrid tracking system which achieves improved registration of synthetic and real imagery and we improved on previous techniques for calibration of a magnetic tracker. CR
---
paper_title: Use of an augmented-vision device for visual search by patients with tunnel vision

paper_content:

PURPOSE ::: To study the effect of an augmented-vision device that superimposes minified contour images over natural vision on visual search performance of patients with tunnel vision. ::: ::: ::: METHODS ::: Twelve subjects with tunnel vision searched for targets presented outside their visual fields (VFs) on a blank background under three cue conditions (with contour cues provided by the device, with auditory cues, and without cues). Three subjects (VF, 8 degrees -11 degrees wide) carried out the search over a 90 degrees x 74 degrees area, and nine subjects (VF, 7 degrees -16 degrees wide) carried out the search over a 66 degrees x 52 degrees area. Eye and head movements were recorded for performance analyses that included directness of search path, search time, and gaze speed. ::: ::: ::: RESULTS ::: Directness of the search path was greatly and significantly improved when the contour or auditory cues were provided in the larger and the smaller area searches. When using the device, a significant reduction in search time (28% approximately 74%) was demonstrated by all three subjects in the larger area search and by subjects with VFs wider than 10 degrees in the smaller area search (average, 22%). Directness and gaze speed accounted for 90% of the variability of search time. ::: ::: ::: CONCLUSIONS ::: Although performance improvement with the device for the larger search area was obvious, whether it was helpful for the smaller search area depended on VF and gaze speed. Because improvement in directness was demonstrated, increased gaze speed, which could result from further training and adaptation to the device, might enable patients with small VFs to benefit from the device for visual search tasks.
---
paper_title: Synchronizing 3D movements for quantitative comparison and simultaneous visualization of actions

paper_content:

In our poster presentation at ISMAR '04, we proposed the idea of an AR training solution including capture and 3D replays of subtle movements. The crucial part missing for realizing such a training system was an appropriate way of synchronizing trajectories of similar movements with varying speed in order to simultaneously visualize the motion of experts and trainees, and to study trainees' performances quantitatively. In this paper we review the research from different communities on synchronization problems of similar complexity. We give a detailed description of the two most applicable algorithms. We then present results using our AR based forceps delivery training system and therefore evaluate both methods for synchronization of experts' and trainees' 3D movements. We also introduce the first concepts of an online synchronization system allowing the trainee to follow movements of an expert and the experts to annotate 3D trajectories for initiation of actions such as display of timely information. A video demonstration provides an overview of the work and a visual idea of what users of the proposed system could observe through their video-see through HMD.
---
paper_title: Dynamic superimposition of synthetic objects on rigid and simple-deformable real objects

paper_content:

A current challenge in augmented reality applications is the accurate superimposition of synthetic objects on real objects within the environment. This challenge is heightened when the real objects are in motion and/or are nonrigid. In this article, we present a robust method for realtime, optical superimposition of synthetic objects on dynamic rigid and simple-deformable real objects. Moreover, we illustrate this general method with the VRDA Tool, a medical education application related to the visualization of internal human knee joint anatomy on a real human knee.
---
paper_title: Augmented workspace: designing an AR testbed

paper_content:

We have implemented a tabletop setup to explore augmented reality (AR) visualization. We call this setup an "augmented workspace". The user sits at the table and performs a manual task, guided by computer graphics overlaid on to his view. The setup serves as a testbed for developing the technology and for studying visual perception issues. The user wears a custom video see-through head mounted display (HMD). Two color video cameras attached to the HMD provide a stereo view of the scene, and a third video camera is added for tracking. The system runs at the full 30-Hz video frame rate with a latency of about 0.1 s, generating a stable augmentation with no apparent jitter visible in the composite images. Two SGI Visual Workstations provide the computing power for the system. In this paper, we describe the augmented workspace system in more detail and discuss several design issues.
---
paper_title: Dynamic Registration Correction in Video-Based Augmented Reality Systems

paper_content:

Augmented reality systems allow users to interact with real and computer-generated objects by displaying 3D virtual objects registered in a user's natural environment. Applications of this powerful visualization tool include previewing proposed buildings in their natural settings, interacting with complex machinery for purposes of construction or maintenance training, and visualizing in-patient medical data such as ultrasound. In all these applications, computer-generated objects must be visually registered with respect to real-world objects in every image the user sees. If the application does not maintain accurate registration, the computer-generated objects appear to float around in the user's natural environment without having a specific 3D spatial position. Registration error is the observed displacement in the image between the actual and intended positions of virtual objects. >
---
paper_title: Using virtual reality to teach radiographic positioning.

paper_content:

Using virtual reality to teach radiographic positioning overcomes many of the limitations of traditional teaching methods and offers several unique advantages. This article describes a virtual reality prototype that could be used to teach radiographic positioning of the elbow joint. By using virtual reality, students are able to see the movement of bones as the arm is manipulated. The article also describes the development and challenges of using virtual reality in medical education.
---
paper_title: Neurosurgical Guidance Using the Stereo Microscope

paper_content:

Many neuro- and ENT surgical procedures are performed using the operating microscope. Conventionally, the surgeon cannot accurately relate information from preoperative radiological images to the appearance of the surgical field. We propose that the best way do this is to superimpose image derived data upon the operative scene. We create a model of relevant structures (e.g. tumor volume, blood vessels and nerves) from multimodality preoperative images. By calibrating microscope optics, registering the patient in-theatre to image coordinates, and tracking the microscope intra-operatively, we can generate stereo projections of the 3D model and project them into the microscope eyepieces, allowing critical structures to be overlayed on the operative scene in the correct position. We have completed initial evaluation with a head phantom, and are about to start clinical evaluation on patients. With the head phantom a theoretical accuracy of 4.6mm was calculated and the observed accuracy ranged from 2mm to 5mm.
---
paper_title: Computer-vision-enabled augmented reality fundus biomicroscopy.

paper_content:

PURPOSE ::: To guide treatment for macular diseases and to facilitate real-time image measurement and comparison, investigations were initiated to permit overlay of previously stored photographic and angiographic images directly onto the real-time slit-lamp biomicroscopic fundus image. ::: ::: ::: DESIGN ::: Experimental study in model eyes, and preliminary observations in human subjects. ::: ::: ::: METHODS ::: A modified, binocular video slit lamp interfaced to a personal computer and framegrabber allows for image acquisition and rendering of stored images overlaid onto the real-time slit-lamp biomicroscopic fundus image. Development proceeds with rendering on a computer monitor, while construction is completed on a miniature display interfaced directly with one of the slit-lamp oculars. Registration and tracking are performed with in-house-developed software. ::: ::: ::: MAIN OUTCOME MEASURES ::: Tracking speed and accuracy, ergonomic acceptability. ::: ::: ::: RESULTS ::: Computer-vision algorithms permit robust montaging, tracking, registration, and rendering of previously stored photographic and angiographic images onto the real-time slit-lamp fundus biomicroscopic image. In model eyes and in preliminary studies in a human eye, optimized registration permits near-video-rate image overlay with updates at 3 to 10 Hz and misregistration errors on the order of 1 to 5 pixels. ::: ::: ::: CONCLUSIONS ::: A prototype for ophthalmic augmented reality (image overlay) is presented. The current hardware/software implementation allows for robust performance.
---
paper_title: A frameless stereotaxic integration of computerized tomographic imaging and the operating microscope.

paper_content:

A computer-based system has been developed for the integration and display of computerized tomography (CT) image data in the operating microscope in the correct perspective without requiring a stereotaxic frame. Spatial registration of the CT image data is accomplished by determination of the position of the operating microscope as its focal point is brought to each of three CT-imaged fiducial markers on the scalp. Monitoring of subsequent microscope positions allows appropriate reformatting of CT data into a common coordinate system. The position of the freely moveable microscope is determined by a non-imaging ultrasonic range-finder consisting of three spark gaps attached to the microscope and three microphones on a rigid support in the operating room. Measurement of the acoustic impulse transit times from the spark gaps to the microphones enables calculation of those distances and unique determination of the microscope position. The CT data are reformatted into a plane and orientation corresponding to the microscope's focal plane or to a deeper parallel plane if required. This reformatted information is then projected into the optics of the operating microscope using a miniature cathode ray tube and a beam splitter. The operating surgeon sees the CT information (such as a tumor boundary) superimposed upon the operating field in proper position, orientation, and scale.
---
paper_title: A fully automated calibration method for an optical see-through head-mounted operating microscope with variable zoom and focus

paper_content:

Ever since the development of the first applications in image-guided therapy (IGT), the use of head-mounted displays (HMDs) was considered an important extension of existing IGT technologies. Several approaches to utilizing HMDs and modified medical devices for augmented reality (AR) visualization were implemented. These approaches include video-see through systems, semitransparent mirrors, modified endoscopes, and modified operating microscopes. Common to all these devices is the fact that a precise calibration between the display and three-dimensional coordinates in the patient's frame of reference is compulsory. In optical see-through devices based on complex optical systems such as operating microscopes or operating binoculars-as in the case of the system presented in this paper-this procedure can become increasingly difficult since precise camera calibration for every focus and zoom position is required. We present a method for fully automatic calibration of the operating binocular Varioscope/spl trade/ M5 AR for the full range of zoom and focus settings available. Our method uses a special calibration pattern, a linear guide driven by a stepping motor, and special calibration software. The overlay error in the calibration plane was found to be 0.14-0.91 mm, which is less than 1% of the field of view. Using the motorized calibration rig as presented in the paper, we were also able to assess the dynamic latency when viewing augmentation graphics on a mobile target; spatial displacement due to latency was found to be in the range of 1.1-2.8 mm maximum, the disparity between the true object and its computed overlay represented latency of 0.1 s. We conclude that the automatic calibration method presented in this paper is sufficient in terms of accuracy and time requirements for standard uses of optical see-through systems in a clinical environment.
---
paper_title: Development of the Varioscope AR. A see-through HMD for computer-aided surgery

paper_content:

In computer-aided surgery (CAS), an undesired side-effect of the necessity of handling sophisticated equipment in the operating room is the fact that the surgeon's attention is drawn from the operating field, since surgical progress is partially monitored on the computer's screen. Augmented reality (AR), the overlay of computer-generated graphics over a real-world scene, provides a possibility to solve this problem. The technical problems associated with this approach, such as viewing of the scenery within a common focal range on the head-mounted display (HMD) or latency in the display on the HMD, have, however, kept AR from widespread usage in CAS. The concept of the Varioscope AR, a lightweight head-mounted operating microscope used as a HMD, is introduced. The registration of the patient to the pre-operative image data, as well as pre-operative planning, take place on VISIT, a surgical navigation system developed at our hospital. Tracking of the HMD and stereoscopic visualisation take place on a separate POSIX.4-compliant real-time operating system running on PC hardware. We were able to overcome the technical problems described above; our work resulted in an AR visualisation system with an update rate of 6 Hz and a latency below 130 ms. It integrates seamlessly into a surgical navigation system and provides a common focus for both virtual and real-world objects. First evaluations of the photogrammetric 2D/3D registration have resulted in a match of 1.7 pixels on the HMD display. The Varioscope AR with its real-time visualisation unit is a major step towards the introduction of AR into clinical routine.
---
paper_title: A head-mounted operating binocular for augmented reality visualization in medicine - design and initial evaluation

paper_content:

Computer-aided surgery (CAS), the intraoperative application of biomedical visualization techniques, appears to be one of the most promising fields of application for augmented reality (AR), the display of additional computer-generated graphics over a real-world scene. Typically a device such as a head-mounted display (HMD) is used for AR. However, considerable technical problems connected with AR have limited the intraoperative application of HMDs up to now. One of the difficulties in using HMDs is the requirement for a common optical focal plane for both the realworld scene and the computer-generated image, and acceptance of the HMD by the user in a surgical environment. In order to increase the clinical acceptance of AR, we have adapted the Varioscope (Life Optics, Vienna), a miniature, cost-effective head-mounted operating binocular, for AR. In this paper, we present the basic design of the modified HMD, and the method and results of an extensive laboratory study for photogrammetric calibration of the Varioscope's computer displays to a real-world scene. In a series of 16 calibrations with varying zoom factors and object distances, mean calibration error was found to be 1.24 /spl plusmn/ 0.38 pixels or 0.12 /spl plusmn/ 0.05 mm for a 640 /spl times/ 480 display. Maximum error accounted for 3.33 /spl plusmn/ 1.04 pixels or 0.33 /spl plusmn/ 0.12 mm. The location of a position measurement probe of an optical tracking system was transformed to the display with an error of less than 1 mm in the real world in 56% of all cases. For the remaining cases, error was below 2 mm. We conclude that the accuracy achieved in our experiments is sufficient for a wide range of CAS applications.
---
paper_title: A frameless stereotaxic operating microscope for neurosurgery

paper_content:

The purpose of the frameless stereotaxic operating microscope is to display computed tomography (CT) or other image data in the operating microscope in the correct scale, orientation, and position without the use of a stereotaxic frame. A nonimaging ultrasonic rangefinder allows the position of the operating microscope and the position of the patient to be determined. Discrete fiducial points on the patient's external anatomy are located in both image space and operating room space, linking the image data and the operating room. Physician-selected image information, e.g. tumor contours or guidance to predetermined targets, is projected through the optics of the operating microscope using a miniature cathode ray tube and a beam splitter. Projected images superpose the surgical field, reconstructed from image data to match the focal plane of the operating microscope. The algorithms on which the system is based are described, and the sources and effects of errors are discussed. The system's performance is simulated, providing an estimate of accuracy. Two phantoms are used to measure accuracy experimentally. Clinical results and observations are given. >
---
paper_title: Registration Error Analysis for Augmented Reality

paper_content:

Augmented reality AR systems typically use see-through head-mounted displays STHMDs to superimpose images of computer-generated objects onto the user's view of the real environment in order to augment it with additional information. The main failing of current AR systems is that the virtual objects displayed in the STHMD appear in the wrong position relative to the real environment. This registration error has many causes: system delay, tracker error, calibration error, optical distortion, and misalignment of the model, to name only a few. Although some work has been done in the area of system calibration and error correction, very little work has been done on characterizing the nature and sensitivity of the errors that cause misregistration in AR systems. This paper presents the main results of an end-to-end error analysis of an optical STHMD-based tool for surgery planning. The analysis was done with a mathematical model of the system and the main results were checked by taking measurements on a real system under controlled circumstances. The model makes it possible to analyze the sensitivity of the system-registration error to errors in each part of the system. The major results of the analysis are: 1 Even for moderate head velocities, system delay causes more registration error than all other sources combined; 2 eye tracking is probably not necessary; 3 tracker error is a significant problem both in head tracking and in system calibration; 4 the World or reference coordinate system adds error and should be omitted when possible; 5 computational correction of optical distortion may introduce more delay-induced registration error than the distortion error it corrects, and 6 there are many small error sources that will make submillimeter registration almost impossible in an optical STHMD system without feedback. Although this model was developed for optical STHMDs for surgical planning, many of the results apply to other HMDs as well.
---
paper_title: Computer-assisted stereotactic microsurgery for the treatment of intracranial neoplasms.

paper_content:

This paper describes a stereotactic CO2 laser system for the removal of intra-axial, intracranial neoplasms. The volume of the neoplasm is transferred into stereotactic space by computer reconstruction of data derived by computed tomography (CT) performed under stereotactic conditions. The tumor volume is sliced in a plane orthogonal to the surgical approach, and slices at specific distances from the focal point of the stereotactic frame are displayed on a graphics monitor in the operating suite along with a cursor representing the position of the surgical laser. Laser vaporization of sequential slices of the tumor results in a cavity, the formation of which is monitored by anteroposterior and lateral roentgenograms. Fifteen stereotactic laser procedures have been performed on 13 patients, and the results are discussed. By this method, it is theoretically possible to remove all of an intracranial neoplasm detected by CT scanning.
---
paper_title: Intra-operative Real-Time 3-D Information Display System based on Integral Videography

paper_content:

A real-time 3-D surgical navigation system that superimposes the real, intuitive 3-D image for medical diagnosis and operation was developed in this paper. This system creates 3-D image based on the principle of integral photography (IP), named "Integral Videography (IV)", which can display geometrically accurate 3-D image and reproduce motion parallax without any need of special devices. 3-D image was supenmposed on the surgical fields in the patient via a half-silvered mirror as if they could be seen through the body. In addition, a real-time IV algorithm for calculating the 3-D image of surgical instruments was used for registration between the location of surgical instruments and the organ during the operation. The experimental results of puncturing a point location and avoiding critical area showed the errors of this navigation system were in the range of 2-3mm. By introducing a display device with higher pixel density, accuracy of the system can be improved.
---
paper_title: Accuracy of Needle Implantation in Brachytherapy Using a Medical AR System - A phantom study

paper_content:

Brachytherapy is the treatment method of choice for patients with a tumor relapse after a radiation therapy ::: with external beams or tumors in regions with sensitive surrounding organs-at-risk, e. g. prostate tumors. The ::: standard needle implantation procedure in brachytherapy uses pre-operatively acquired image data displayed as ::: slices on a monitor beneath the operation table. Since this information allows only a rough orientation for the ::: surgeon, the position of the needles has to be verified repeatedly during the intervention. ::: Within the project Medarpa a transparent display being the core component of a medical Augmented ::: Reality (AR) system has been developed. There, pre-operatively acquired image data is displayed together with ::: the position of the tracked instrument allowing a navigated implantation of the brachytherapy needles. The ::: surgeon is enabled to see the anatomical information as well as the virtual instrument in front of the operation ::: area. Thus, the Medarpa system serves as "window into the patient". ::: This paper deals with the results of first clinical trials of the system. Phantoms have been used for evaluating ::: the achieved accuracy of the needle implantation. This has been done by comparing the output of the system ::: (instrument positions relative to the phantom) with the real positions of the needles measured by means of a ::: verification CT scan.
---
paper_title: Surgical navigation by autostereoscopic image overlay of integral videography

paper_content:

This paper describes an autostereoscopic image overlay technique that is integrated into a surgical navigation system to superimpose a real three-dimensional (3-D) image onto the patient via a half-silvered mirror. The images are created by employing a modified version of integral videography (IV), which is an animated extension of integral photography. IV records and reproduces 3-D images using a microconvex lens array and flat display; it can display geometrically accurate 3-D autostereoscopic images and reproduce motion parallax without the need for special devices. The use of semitransparent display devices makes it appear that the 3-D image is inside the patient's body. This is the first report of applying an autostereoscopic display with an image overlay system in surgical navigation. Experiments demonstrated that the fast IV rendering technique and patient-image registration method produce an average registration accuracy of 1.13 mm. Experiments using a target in phantom agar showed that the system can guide a needle toward a target with an average error of 2.6 mm. Improvement in the quality of the IV display will make this system practical and its use will increase surgical accuracy and reduce invasiveness.
---
paper_title: An image overlay system for medical data visualization.

paper_content:

Image Overlay is a computer display technique which superimposes computer images over the user's direct view of the real world. The images are transformed in real-time so they appear to the user to be an integral part of the surrounding environment. By using Image Overlay with three-dimensional medical images such as CT reconstructions, a surgeon can visualize the data 'in-vivo', exactly positioned within the patient's anatomy, and potentially enhance the surgeon's ability to perform a complex procedure. This paper describes prototype Image Overlay systems and initial experimental results from those systems.
---
paper_title: An Accuracy Certified Augmented Reality System for Therapy Guidance

paper_content:

Our purpose is to provide an augmented reality system for Radio-Frequency guidance that could superimpose a 3D model of the liver, its vessels and tumors (reconstructed from CT images) on external video images of the patient. In this paper, we point out that clinical usability not only need the best affordable registration accuracy, but also a certification that the required accuracy is met, since clinical conditions change from one intervention to the other. Beginning by addressing accuracy performances, we show that a 3D/2D registration based on radio-opaque fiducials is more adapted to our application constraints than other methods. Then, we outline a lack in their statistical assumptions which leads us to the derivation of a new extended 3D/2D criterion. Careful validation experiments on real data show that an accuracy of 2 mm can be achieved in clinically relevant conditions, and that our new criterion is up to 9% more accurate, while keeping a computation time compatible with real-time at 20 to 40 Hz.
---
paper_title: An automatic registration method for frameless stereotaxy, image guided surgery, and enhanced reality visualization

paper_content:

There is a need for frameless guidance systems to help neurosurgeons to plan the exact location of a craniotomy, to define the margins of tumors and to precisely identify locations of neighboring critical structures. We have developed an automatic technique for registering clinical data, such as segmented MRI or CT reconstructions, with the patient's head on the operating table. A second method calibrates the position of a video camera relative to the patient. The combination allows a visual mix of live video of the patient with the segmented 3D MRI or CT model, enabling enhanced reality techniques for planning and guiding neurosurgical procedures, and to interactively view extracranial or intracranial structures non-intrusively. Extensions of the method include image guided biopsies, focused therapeutic procedures and clinical studies involving change detection over time sequences of images. >
---
paper_title: Image guidance of breast cancer surgery using 3-D ultrasound images and augmented reality visualization

paper_content:

This paper describes augmented reality visualization for the guidance of breast-conservative cancer surgery using ultrasonic images acquired in the operating room just before surgical resection. By combining an optical three-dimensional (3-D) position sensor, the position and orientation of each ultrasonic cross section are precisely measured to reconstruct geometrically accurate 3-D tumor models from the acquired ultrasonic images. Similarly, the 3-D position and orientation of a video camera are obtained to integrate video and ultrasonic images in a geometrically accurate manner. Superimposing the 3-D tumor models onto live video images of the patient's breast enables the surgeon to perceive the exact 3-D position of the tumor, including irregular cancer invasions which cannot be perceived by touch, as if it were visible through the breast skin. Using the resultant visualization, the surgeon can determine the region for surgical resection in a more objective and accurate manner, thereby minimizing the risk of a relapse and maximizing breast conservation. The system was shown to be effective in experiments using phantom and clinical data.
---
paper_title: Enhancing reality in the operating room

paper_content:

Three dimensional computer models of the anatomy generated from volume acquisitions of computed tomography and magnetic resonance imaging are useful adjuncts to 2D images. This paper describes a system that merges the computer generated 3D models with live video to enhance the surgeon's understanding of the anatomy beneath the surface. The system can be used as a planning aid before the operation and provide additional information during an operation. The application of the system to a brain operation is described. >
---
paper_title: Robust Hand-Eye Calibration of an Endoscopic Surgery Robot Using Dual Quaternions

paper_content:

This paper presents an approach for applying a dual quaternion hand-eye calibration algorithm on an endoscopic surgery robot. Special focus is on robustness, since the error of position and orientation data provided by the robot can be large depending on the movement actually executed. Another inherent problem to all hand-eye calibration methods is that non-parallel rotation axes must be used; otherwise, the calibration will fail. Thus we propose a method for increasing the numerical stability by selecting an optimal set of relative movements from the recorded sequence. Experimental evaluation shows the error in the estimated transformation when using well-suited and ill-suited data. Additionally, we show how a RANSAC approach can be used for eliminating the erroneous robot data from the selected movements.
---
paper_title: Hybrid method for both calibration and registration of an endoscope with an active optical tracker

paper_content:

Abstract In this paper, we present a hybrid method for calibration of an endoscope and its registration with an active optical tracker. Practically, both operations are done simultaneously by moving an active optical marker in the field of view of the two devices. By segmenting image data, the LEDs composing the marker are extracted and the transformation matrix between the two referentials (homography) is calculated. By reformulating the calibration problem, registration and calibration parameters are extracted from the homography. As camera calibration and registration is an indispensable step for augmented reality or image-guided applications, this technique can easily be used in the operating field because it is fast, accurate and reliable. We currently are using this technique with an augmented reality system for laparoscopic procedures.
---
paper_title: Development of an endoscopic navigation system based on digital image processing

paper_content:

We developed a new system to couple the endoscope to an optical position measurement system (OPMS) so that the image frames from the endoscope camera can be labeled with the accurate endoscopic position. This OPMS is part of the EasyGuide Neuro navigation system, which is used for microsurgery and neuroendoscopy. Using standard camera calibration techniques and a newly developed system calibration, any 3-dimensional (3-D) world point can be mapped on to the view from the endoscope. in particular, we can display the coordinates of any anatomical landmark of the patient as it is viewed from the current position of the camera. This and other image-processing techniques are applied to the labeled frame sequence in order to offer the neurosurgeon a variety of control modules that increase the safety and flexibility of neuroendoscopic operations. Several modules, including a new motion alarm system and the “tracking” and “virtual map” modules, were tested in a human cadaveric model using the frontal and occipit...
---
paper_title: Volumetric Image Guidance via a Stereotactic Endoscope

paper_content:

We have developed a surgical setup based on modern frameless stereotactic techniques that enables surgeons to visualize the field of view of the surgical endoscope, overlaid with the real-time and volumetrically reconstructed medical images, of a localized area of the patient’s anatomy. Using this navigation system, the surgeon visualizes the surgical site via the surgical endoscope, while exploring the inner layers of the patient’s anatomy by utilizing the three-dimensionally reconstructed image updates obtained by pre-operative images, such as Magnetic Resonance and/or Computed Tomography Imaging. This system also allows the surgeon to virtually “fly through and around” the site of the surgery to visualize several alternatives and qualitatively determine the best surgical approach. Moving endoscopes are tracked with infra-red stereovision cameras and diodes, allowing the determination of their spatial relation to the target lesion and the fiducial based patient/image registration.
---
paper_title: Camera-marker alignment framework and comparison with hand-eye calibration for augmented reality applications

paper_content:

An integral part of every augmented reality system is the calibration between camera and camera-mounted tracking markers. Accuracy and robustness of the AR overlay process is greatly influenced by the quality of this step. In order to meet the very high precision requirements of medical skill training applications, we have set up a calibration environment based on direct sensing of LED markers. A simulation framework has been developed to predict and study the achievable accuracy of the backprojection needed for the scene augmentation process. We demonstrate that the simulation is in good agreement with experimental results. Even if a slight improvement of the precision has been observed compared to well-known hand-eye calibration methods, the subpixel accuracy required by our application cannot be achieved even when using commercial tracking systems providing marker positions within very low error limits.
---
paper_title: Development of a camera model and calibration procedure for oblique-viewing endoscopes

paper_content:

Oblique-viewing endoscopes (oblique scopes) are widely used in medical practice. They are essential for certain procedures such as laparoscopy, arthroscopy and sinus endoscopy. In an oblique scope the viewing directions are changeable by rotating the scope cylinder. Although a camera calibration method is necessary to apply augmented reality technologies to oblique endoscopic procedures, no method for oblique scope calibration has yet been developed. In the present paper, we formulate a camera model and a calibration procedure for oblique scopes. In the calibration procedure, Tsai's calibration is performed at zero rotation of the scope cylinder, then the variation of the external camera parameters corresponding to the rotation of the scope cylinder is modeled and estimated as a function of the rotation angle. Accurate estimation of the rotational axis is included in the procedure. The accuracy of this estimation was demonstrated to have a significant effect on overall calibration accuracy in the experimental evaluation, especially with large rotation angles. The projection error in the image plane was approximately two pixels. The proposed method was shown to be clinically applicable.
---
paper_title: Endoscope Calibration and Accuracy Testing for 3D/2D Image Registration

paper_content:

New surgical navigation techniques incorporate the use of live surgical endoscope video with 3D reconstructed MRI or CT images of a patient's anatomy. This image-enhanced endoscopy requires calibration of the endoscope to accurately the register the real endoscope video to the virtual image. The calibration and accuracy testing of such a system and a simple yet effective linear method for lens-distortion compensation are described.
---
paper_title: Fiducial-free registration procedure for navigated bronchoscopy

paper_content:

Navigated bronchoscopy has been developed by various groups within the last decades. Systems based on CT data and electromagnetic tracking enable the visualization of the position and orientation of the bronchoscope, forceps, and biopsy tools within CT data. Therefore registration between the tracking space and the CT volume is required. Standard procedures are based on point-based registration methods that require selecting corresponding natural landmarks in both coordinate systems by the examiner. We developed a novel algorithm for a fully automatic registration procedure in navigated bronchoscopy based on the trajectory recorded during routine examination of the airways at the beginning of an intervention. The proposed system provides advantages in terms of an unchanged medical workflow and high accuracy. We compared the novel method with point-based and ICP-based registration. Experiments demonstrate that the novel method transforms up to 97% of tracking points inside the segmented airways, which was the best performance compared to the other methods.
---
paper_title: Implementation, calibration and accuracy testing of an image-enhanced endoscopy system

paper_content:

This paper presents a new method for image-guided surgery called image-enhanced endoscopy. Registered real and virtual endoscopic images (perspective volume renderings generated from the same view as the endoscope camera using a preoperative image) are displayed simultaneously; when combined with the ability to vary tissue transparency in the virtual images, this provides surgeons with the ability to see beyond visible surfaces and, thus, provides additional exposure during surgery. A mount with four photoreflective spheres is rigidly attached to the endoscope and its position and orientation is tracked using an optical position sensor. Generation of virtual images that are accurately registered to the real endoscopic images requires calibration of the tracked endoscope. The calibration process determines intrinsic parameters (that represent the projection of three-dimensional points onto the two-dimensional endoscope camera imaging plane) and extrinsic parameters (that represent the transformation from the coordinate system of the tracker mount attached to the endoscope to the coordinate system of the endoscope camera), and determines radial lens distortion. The calibration routine is fast, automatic, accurate and reliable, and is insensitive to rotational orientation of the endoscope. The routine automatically detects, localizes, and identifies dots in a video image snapshot of the calibration target grid and determines the calibration parameters from the sets of known physical coordinates and localized image coordinates of the target grid dots. Using nonlinear lens-distortion correction, which can be performed at real-time rates (30 frames per second), the mean projection error is less than 0.5 mm at distances up to 25 mm from the endoscope tip, and less than 1.0 mm up to 45 mm. Experimental measurements and point-based registration error theory show that the tracking error is about 0.5-0.7 mm at the tip of the endoscope and less than 0.9 mm for all points in the field of view of the endoscope camera at a distance of up to 65 mm from the tip. It is probable that much of the projection error is due to endoscope tracking error rather than calibration error. Two examples of clinical applications are presented to illustrate the usefulness of image-enhanced endoscopy. This method is a useful addition to conventional image-guidance systems, which generally show only the position of the tip (and sometimes the orientation) of a surgical instrument or probe on reformatted image slices.
---
paper_title: Correction of distortion in endoscope images

paper_content:

Images formed with endoscopes suffer from a spatial distortion due to the wide-angle nature of the endoscope's objective lens. This change in the size of objects with position precludes quantitative measurement of the area of the objects, which is important in endoscopy for accurately measuring ulcer and lesion sizes over time. A method for correcting the distortion characteristic of endoscope images is presented. A polynomial correction formula was developed for the endoscope lens and validated by comparing quantitative test areas before and after the distortion correction. The distortion correction has been incorporated into a computer program that could readily be applied to electronic images obtained at endoscopy using a desk-top computer. The research presented here is a key step towards the quantitative determination of the area of regions of interest in endoscopy. >
---
paper_title: Endoscopic Surgery: The History, the Pioneers

paper_content:

The introduction of endoscopy into surgical practice is one of the biggest success stories in the history of medicine. Endoscopy has its roots in the nineteenth century and was initially developed by urologists and internists. During the 1960s and 1970s gynecologists took the lead in the development of endoscopic surgery while most of the surgical community continued to ignore the possibilities of the new technique. This was due in part to the introduction of ever more sophisticated drugs, the impressive results of intensive care medicine, and advances in anesthesia, which led to the development of more radical and extensive operations, or "major surgery." The idea that large problems require large incisions so deeply dominated surgical thinking that there was little room to appreciate the advances of "key-hole" surgery. Working against this current, some general surgeons took up the challenge. In 1976 the Surgical Study Group on Endoscopy and Ultrasound (CAES) was formed in Hamburg. Five years later, on the other side of the Atlantic, the Society of American Gastrointestinal Endoscopic Surgeons (SAGES) was called into being. In 1987 the first issue of the journal Surgical Endoscopy was published, and the following year the First World Congress on Surgical Endoscopy took place in Berlin. The sweeping success of the "laparoscopic revolution" (1989-1990) marked the end of traditional open surgery and encouraged surgeons to consider new perspectives. By the 1990s the breakthrough had been accomplished: endoscopy was incorporated into surgical thinking.
---
paper_title: Evaluation of a novel calibration technique for optically tracked oblique laparoscopes

paper_content:

This paper proposes an evaluation of a novel calibration method for an optically tracked oblique laparoscope. We present the necessary tools to track an oblique scope and a camera model which includes changes to the intrinsic camera parameters thereby extending previously proposed methods. Because oblique scopes offer a wide 'virtual' view on the surgical field, the method is of great interest for augmented reality guidance of laparoscopic interventions using an oblique scope. ::: ::: The model and an approximated version are evaluated in an extensive validation study. Using 5 sets of 40 calibration images, we compare both camera models (i.e. model and approximation) and 2 interpolation schemes. The selected model and interpolation scheme reaches an average accuracy of 2.60 pixel and an equivalent 3D error of 0.60 mm. ::: ::: Finally, we present initial experience of the presented approach with an oblique scope and optical tracking in a clinical setup. During a laparoscopic rectum resection surgery the setup was used to augment the scene with a model of the pelvis. The method worked properly and the attached probes did not interfere with normal procedure.
---
paper_title: A Method for Tracking the Camera Motion of Real Endoscope by Epipolar Geometry Analysis and Virtual Endoscopy System

paper_content:

This paper describes a method for tracking the camera motion of a real endoscope by epipolar geometry analysis and image-based registration. In an endoscope navigation system, which provides navigation information to a medical doctor during an endoscopic examination, tracking the camera motion of the endoscopic camera is one of the fundamental functions. With a flexible endoscope, it is hard to directly sense the position of the camera, since we cannot attach a positional sensor at the tip of the endoscope. The proposed method consists of three parts: (1) calculation of corresponding point-pairs of two time-adjacent frames, (2) coarse estimation of the camera motion by solving the epipolar equation, and (3) fine estimation by executing image-based registration between real and virtual endoscopic views. In the method, virtual endoscopic views are obtained from X-ray CT images of real endoscopic images of the same patient. To evaluate the method, we applied it a real endoscopic video camera and X-ray CT images. The experimental results showed that the method could track the motion of the camera satisfactorily.
---
paper_title: A four-step camera calibration procedure with implicit image correction

paper_content:

In geometrical camera calibration the objective is to determine a set of camera parameters that describe the mapping between 3-D reference coordinates and 2-D image coordinates. Various methods for camera calibration can be found from the literature. However surprisingly little attention has been paid to the whole calibration procedure, i.e., control point extraction from images, model fitting, image correction, and errors originating in these stages. The main interest has been in model fitting, although the other stages are also important. In this paper we present a four-step calibration procedure that is an extension to the two-step method. There is an additional step to compensate for distortion caused by circular features, and a step for correcting the distorted image coordinates. The image correction is performed with an empirical inverse model that accurately compensates for radial and tangential distortions. Finally, a linear method for solving the parameters of the inverse model is presented.
---
paper_title: Hybrid bronchoscope tracking using a magnetic tracking sensor and image registration

paper_content:

In this paper, we propose a hybrid method for tracking a bronchoscope that uses a combination of magnetic sensor tracking and image registration. The position of a magnetic sensor placed in the working channel of the bronchoscope is provided by a magnetic tracking system. Because of respiratory motion, the magnetic sensor provides only the approximate position and orientation of the bronchoscope in the coordinate system of a CT image acquired before the examination. The sensor position and orientation is used as the starting point for an intensity-based registration between real bronchoscopic video images and virtual bronchoscopic images generated from the CT image. The output transformation of the image registration process is the position and orientation of the bronchoscope in the CT image. We tested the proposed method using a bronchial phantom model. Virtual breathing motion was generated to simulate respiratory motion. The proposed hybrid method successfully tracked the bronchoscope at a rate of approximately 1 Hz.
---
paper_title: Modeling and calibration of automated zoom lenses

paper_content:

Camera systems with automated zoom lenses are inherently more useful than those with fixed-parameter lenses. Variable-parameter lenses enable us to produce better images by matching the camera's sensing characteristics to the conditions in a scene. They also allow us to make measurements by noting how the scene's image changes as the parameters are varied. The reason variable-parameter lenses are not more commonly used in machine vision is that they are difficult to model for continuous ranges of lens settings. ::: We show in this thesis that traditional modeling approaches cannot capture the complex relationships between control parameters and imaging processes. Furthermore, we demonstrate that the assumption of idealized behavior in traditional models can lead to significant performance problems in color imaging and focus ranging. By using more complex models and control strategies we were able to reduce or eliminate these performance problems. ::: The principal contribution of our research is a methodology for empirically producing accurate camera models for systems with variable-parameter lenses. We also developed a comprehensive taxonomy for the property of "image center." To demonstrate the effectiveness of our methodology we applied it to produce an "adjustable," perspective-projection camera model based on Tsai's fixed camera model. We calibrated and tested our model on two different automated camera systems. In both cases the calibrated model operated across continuous ranges of focus and zoom with an average error of less than 0.14 pixels between the predicted and the measured positions of features in the image plane. We also calibrated and tested our model on one automated camera system across a continuous range of aperture and achieved similar results.
---
paper_title: Flexible Calibration of Actuated Stereoscopic Endoscope for Overlay in Robot Assisted Surgery

paper_content:

Robotic assistance have greatly benefited the operative gesture in mini-invasive surgery. Nevertheless, the surgeon is still suffering from the restricted vision of the operating field through the endoscope. We thus propose to augment the endoscopic images with preoperative data.This paper focuses on the use of a priori information to initialise the overlay by a precise calibration of the actuated stereoscopic endoscope. The flexibility of the proposed method avoids any additional tracking system in the operating room and can be applied to other augmented reality systems. We present quantitative experimental calibration results with the da Vinci? surgical system, as well as the use of these results to initialise the overlay of endoscopic images of a plastic heart with a coronary artery model.
---
paper_title: A system to support laparoscopic surgery by augmented reality visualization

paper_content:

This paper describes the development of an augmented reality system for intra-operative laparoscopic surgery support.The goal of this system is to reveal structures, otherwise hidden within the laparoscope view. To allow flexible movement of the laparoscope we use optical tracking to track both patient and laparoscope.The necessary calibration and registration procedures were developed and bundled where possible in order to facilitate integration in a current laparoscopic procedure. Care was taken to achieve high accuracy by including radial distortion components without compromising real time speed.Finally a visual error assessment is performed, the usefulness is demonstrated within a test setup and some preliminary quantitative evaluation is done.
---
paper_title: A flexible new technique for camera calibration

paper_content:

We propose a flexible technique to easily calibrate a camera. It only requires the camera to observe a planar pattern shown at a few (at least two) different orientations. Either the camera or the planar pattern can be freely moved. The motion need not be known. Radial lens distortion is modeled. The proposed procedure consists of a closed-form solution, followed by a nonlinear refinement based on the maximum likelihood criterion. Both computer simulation and real data have been used to test the proposed technique and very good results have been obtained. Compared with classical techniques which use expensive equipment such as two or three orthogonal planes, the proposed technique is easy to use and flexible. It advances 3D computer vision one more step from laboratory environments to real world use.
---
paper_title: Endoscopic Augmented Reality Navigation System for Endonasal Transsphenoidal Surgery to Treat Pituitary Tumors: Technical Note

paper_content:

OBJECTIVE ::: Endoscopes have been commonly used in transsphenoidal surgery to treat pituitary tumors, to compensate for the narrow surgical field. Although many navigation systems have been introduced for neurosurgical procedures, there have been few reports of navigation systems for endoscopic operations. This report presents our recently developed, endoscopic, augmented reality (AR) navigation system. ::: ::: ::: METHODS ::: The technology is based on the principles of AR environment technology. The system consisted of a rigid endoscope with light-emitting diodes, an optical tracking system, and a controller. The operation of the optical tracking system was based on two sets of infrared light-emitting diodes, which measured the position and orientation of the endoscope relative to the patient's head. We used the system during endonasal transsphenoidal operations to treat pituitary tumors in 12 recent cases. ::: ::: ::: RESULTS ::: Anatomic, "real," three-dimensional, virtual images of the tumor and nearby anatomic structures (including the internal carotid arteries, sphenoid sinuses, and optic nerves) were superimposed on real- time endoscopic live images. The system also indicated the positions and directions of the endoscope and the endoscopic beam in three-dimensional magnetic resonance imaging or computed tomographic planes. Furthermore, the colors of the wire-frame images of the tumor changed according to the distance between the tip of the endoscope and the tumor. These features were superior to those of conventional navigation systems, which are available only for operating microscopes. ::: ::: ::: CONCLUSION ::: The endoscopic AR navigation system allows surgeons to perform accurate, safe, endoscope-assisted operations to treat pituitary tumors; it is particularly useful for reoperations, in which midline landmarks may be absent. We consider the AR navigation system to be a promising tool for safe, minimally invasive, endonasal, transsphenoidal surgery to treat pituitary tumors.
---
paper_title: Development of a navigation system for endoluminal brachytherapy in human lungs

paper_content:

The endoluminal brachytherapy of peripherally located bronchial carcinoma is difficult because of the complexity to position an irradiation catheter led by a bronchoscope to a desired spot inside a human lung. Furthermore the size of the bronchoscope permits only rarely the insertion of a catheter into the fine segment bronchi. We are developing an image-guided navigation system which indicates a path for guidance to the desired bronchus. Thereby a thin catheter with an enclosed navigation probe can be led up directly to the target bronchus, either by the use of the video of the bronchoscope or by the use of virtual bronchoscopy. Because of the thin bronchi and their moving soft tissue, the navigation system has to be very precise. This accuracy is reached by a gradually registering navigation component which improves the accuracy in the course of the intervention through mapping the already covered path to the preoperatively generated graph based bronchial tree description. The system includes components for navigation, segmentation, preoperative planning, and intraoperative guidance. Furthermore the visualization of the path can be adapted to the lung specialist's habits (video of bronchoscope, 2D, 3D, virtual bronchoscopy etc.).
---
paper_title: A Navigation System for Augmenting Laparoscopic Ultrasound

paper_content:

Establishing image context is the major difficulty of performing laparoscopic ultrasound. The standard techniques used by transabdominal ultrasonographers to understand image orientation are difficult to apply with laparoscopic instruments. In this paper, we describe a navigation system that displays the position and orientation of laparoscopic ultrasound images to the operating surgeon in real time. The display technique we developed for showing the orientation information uses a 3D model of the aorta as the main visual reference. This technique is helpful because it provides surgeons with important spatial cues, which we show improves their ability to interpret the laparoscopic ultrasound.
---
paper_title: Laparoscope Self-calibration for Robotic Assisted Minimally Invasive Surgery

paper_content:

For robotic assisted minimal access surgery, recovering 3D soft tissue deformation is important for intra-operative surgical guidance, motion compensation, and prescribing active constraints. We propose in this paper a method for determining varying focal lengths of stereo laparoscope cameras during robotic surgery. Laparoscopic images typically feature dynamic scenes of soft-tissue deformation and self-calibration is difficult with existing approaches due to the lack of rigid temporal constraints. The proposed method is based on the direct derivation of the focal lengths from the fundamental matrix of the stereo cameras with known extrinsic parameters. This solves a restricted self-calibration problem, and the introduction of the additional constraints improves the inherent accuracy of the algorithm. The practical value of the method is demonstrated with analysis of results from both synthetic and in vivo data sets.
---
paper_title: Augmented Reality Visualization for Laparoscopic Surgery

paper_content:

We present the design and a prototype implementation of a three-dimensional visualization system to assist with laparoscopic surgical procedures. The system uses 3D visualization, depth extraction from laparoscopic images, and six degree-of-freedom head and laparoscope tracking to display a merged real and synthetic image in the surgeon’s video-see-through head-mounted display. We also introduce a custom design for this display. A digital light projector, a camera, and a conventional laparoscope create a prototype 3D laparoscope that can extract depth and video imagery.
---
paper_title: Registration of real and CT-derived virtual bronchoscopic images to assist transbronchial biopsy

paper_content:

This paper describes research work motivated by an innovative medical application: computer-assisted transbronchial biopsy. This project involves the registration, with no external localization device, of a preoperative three-dimensional (3-D) computed tomography (CT) scan of the thoracic cavity (showing a tumor that requires a needle biopsy), and an intraoperative endoscopic two-dimensional (2-D) image sequence, in order to provide assistance in transbronchial puncture of the tumor. Because of the specific difficulties resulting from the data being processed, a multilevel strategy was introduced. For each analysis level, the relevant information to process and the corresponding algorithms were defined. This multilevel strategy, thus, provides the best possible accuracy. Original image processing methods were elaborated, dealing with segmentation, registration and 3-D reconstruction of the bronchoscopic images. In particular, these methods involve adapted mathematical morphology tools, a "daemon-based" registration algorithm, and a model-based shape-from-shading algorithm. This pilot study presents the application of these algorithms to recorded bronchoscopic video sequences for five patients. The preliminary results presented here demonstrate that it is possible to precisely localize the endoscopic camera within the CT data coordinate system. The computer can thus synthesize in near real-time the CT-derived virtual view that corresponds to the actual endoscopic view.
---
paper_title: Image overlay guidance for needle insertion in CT scanner

paper_content:

We present an image overlay system to aid needle insertion procedures in computed tomography (CT) scanners. The device consists of a display and a semitransparent mirror that is mounted on the gantry. Looking at the patient through the mirror, the CT image appears to be floating inside the patient with correct size and position, thereby providing the physician with two-dimensional (2-D) "X-ray vision" to guide needle insertions. The physician inserts the needle following the optimal path identified in the CT image rendered on the display and, thus, reflected in the mirror. The system promises to reduce X-ray dose, patient discomfort, and procedure time by significantly reducing faulty insertion attempts. It may also increase needle placement accuracy. We report the design and implementation of the image overlay system followed by the results of phantom and cadaver experiments in several clinical applications.
---
paper_title: Merging visible and invisible: two Camera-Augmented Mobile C-arm (CAMC) applications

paper_content:

This paper presents the basic concept of CAMC and some of its applications. A CCD camera is attached to a mobile C-arm fluoroscopy X-ray system. Both optical and X-ray imaging systems are calibrated in the same coordinate system in an off-line process. The new system is able to provide X-ray and optical images simultaneously. The CAMC framework has great potential for medical augmented reality. We briefly introduce two new CAMC applications to the augmented reality research community. The first application aims at merging video images with a pre-computed tomographic reconstruction of the 3D volume of interest. This is a logical continuation of our work on 3D reconstruction using a CAMC (1999). The second approach is a totally new CAMC design where using a double mirror system and an appropriate calibration procedure the X-ray and optical images are merged in real-time. This new system enables the user to see an optical image, an X-ray image, or an augmented image where both visible and invisible are combined in real-time. The paper is organized in two independent sections describing each of the above. Experimental results are provided at the same time as the methods and apparatus are described for each section.
---
paper_title: Three-Dimensional Slice Image Overlay System with Accurate Depth Perception for Surgery

paper_content:

In this paper, we describe a three-dimensional (3-D) display, containing a flat two-dimensional (2-D) display, an actuator and a half-silvered mirror. This system creates a superimposed slice view on the patient and gives accurate depth perception. The clinical significance of this system is that it displays raw image data at an accurate location on the patient’s body. Moreover, it shows previously acquired image information, giving the capacity for accurate direction to the surgeon who is thus able to perform less-invasive therapy. Compared with conventional 3-D displays, such as stereoscopy, this system only requires raw 3-D data that are acquired in advance. Simpler data processing is required, and the system has the potential for rapid development. We describe a novel algorithm, registering positional data between the image and the patient. The accuracy of the system is evaluated and confirmed by an experiment in which an image is superimposed on a test object. The results indicate that the system could be readily applied in clinical situations, considering the resolution of the pre-acquired images.
---
paper_title: Tomographic reflection to merge ultrasound images with direct vision

paper_content:

Tomographic reflection is a method that may be used to merge the visual outer surface of a patient with a simultaneous ultrasound scan of the patient's interior. The technique combines a flat-panel monitor with a half-silvered mirror such that the image on the monitor is reflected precisely at the proper location within the patient. In this way, the ultrasound image is superimposed in real time on the view of the patient along with the operator's hands and any invasive tools in the field of view. Instead of looking away at an ultrasound monitor, the operator can manipulate needles and scalpels with direct hand-eye coordination. Invasive tools are visible up to where they enter the skin, permitting natural visual extrapolation to targets in the ultrasound slice. Tomographic reflection is independent of viewer location, requires no special apparatus to be worn by the operator, nor any registration of the patient.
---
paper_title: Laser projection augmented reality system for computer-assisted surgery

paper_content:

A new augmented reality apparatus was evaluated. The device uses scanned infrared and visible lasers to project computer generated information such as surgical plans, entry pints for probes etc, directly onto the patient. In addition to projecting the plan, the device can be integrated with a 3D camera and is capable of measuring the location of projected infrared laser spots. This can be used to ensure that the display is accurate, apply corrections to the projection path and to assist in registration. The projection system has its own Application Programmer’s Interface (API) and is a stand-alone add-on unit to any host computer system. Tests were conducted to evaluate the accuracy and repeatability of the system. We compared the locations of points projected on a flat surface with the measurements obtained from a tracked probe. The surface was rotated through 60 degrees in 5 degree increments and locations measured from the two devices agreed to within 2mm. An initial host application was also developed to demonstrate the new unit. Fiducials representing vertices along a proposed craniotomy were embedded into a plastic skull and a projection path defining the craniotomy was calculated. A feedback-based optimization of the plan was performed by comparing the measurement taken by the camera of these coordinates. The optimized plan was projected onto the skull. On average, the projection deviated by approximately 1mm from the plan. Applications include identification of critical anatomical structures, visualization of preplanned paths and targets, and telesurgery or teleconsultation.
---
paper_title: A Novel Laser Guidance System for Alignment of Linear Surgical Tools: Its Principles and Performance Evaluation as a Man-Machine System

paper_content:

A novel laser guidance system that uses dual laser beam shooters for the alignment of linear surgical tools is presented. In the proposed system, the intersection of two laser planes generated by dual laser shooters placed at two fixed locations defines the straight insertion path of a surgical tool. The guidance information is directly projected onto the patient and the surgical tool. Our assumption is that a linear surgical tool has cylindrical shape or that a cylindrical sleeve is attached to the tool so that the sleeve and tool axes are aligned. The guidance procedure is formulated mainly using the property that the two laser planes are projected as two parallel straight lines onto the cylindrical tool surface if and only if the cylinder axis direction is the same as the direction of the intersection of the two laser planes. Unlike conventional augmented reality systems, the proposed system does not require the wearing of glasses or mirrors to be placed between the surgeon and patient. In our experiments, a surgeon used the system to align wires according to the alignment procedure, and the overall accuracy and alignment time were evaluated. The evaluations were considered not to be simply of a mechanical system but of a man-machine system, since the performance depends on both the system accuracy and the surgeon's perceptual ability. The evaluations showed the system to be highly effective in providing linear alignment assistance.
---
paper_title: A frameless stereotaxic integration of computerized tomographic imaging and the operating microscope.

paper_content:

A computer-based system has been developed for the integration and display of computerized tomography (CT) image data in the operating microscope in the correct perspective without requiring a stereotaxic frame. Spatial registration of the CT image data is accomplished by determination of the position of the operating microscope as its focal point is brought to each of three CT-imaged fiducial markers on the scalp. Monitoring of subsequent microscope positions allows appropriate reformatting of CT data into a common coordinate system. The position of the freely moveable microscope is determined by a non-imaging ultrasonic range-finder consisting of three spark gaps attached to the microscope and three microphones on a rigid support in the operating room. Measurement of the acoustic impulse transit times from the spark gaps to the microphones enables calculation of those distances and unique determination of the microscope position. The CT data are reformatted into a plane and orientation corresponding to the microscope's focal plane or to a deeper parallel plane if required. This reformatted information is then projected into the optics of the operating microscope using a miniature cathode ray tube and a beam splitter. The operating surgeon sees the CT information (such as a tumor boundary) superimposed upon the operating field in proper position, orientation, and scale.
---
paper_title: Navigated three dimensional beta probe for optimal cancer resection

paper_content:

In minimally invasive tumor resection, the desirable goal is to perform a minimal but complete removal of cancerous cells. In the last decades interventional nuclear medicine probes supported the detection of remaining tumor cells. However, scanning the patient with an intraoperative probe and applying the treatment are not done simultaneously. The main contribution of this work is to extend the one dimensional signal of a beta-probe to a four dimensional signal including the spatial information of the distal end of the probe. We generate a color encoded surface map of the scanned activity and guide any tracked surgical instrument back to the regions with measured activity. For navigation, we implement an augmented reality visualization that superimposes the acquired surface on a visual image of the real anatomy. Alternatively, a simulated beta-probe count rate in the tip of a tracked therapeutic instrument is simulated showing the count number and coding it as an acoustic signal. Preliminary tests were performed showing the feasibility of the new designed system and the viability of such a three dimensional intraoperative molecular imaging modality.
---
paper_title: Hybrid navigation interface for orthopedic and trauma surgery

paper_content:

Several visualization methods for intraoperative navigation systems were proposed in the past. In standard slice based navigation, three dimensional imaging data is visualized on a two dimensional user interface in the surgery room. Another technology is the in-situ visualization i.e. the superimposition of imaging data directly into the view of the surgeon, spatially registered with the patient. Thus, the three dimensional information is represented on a three dimensional interface. We created a hybrid navigation interface combining an augmented reality visualization system, which is based on a stereoscopic head mounted display, with a standard two dimensional navigation interface. Using an experimental setup, trauma surgeons performed a drilling task using the standard slice based navigation system, different visualization modes of an augmented reality system, and the combination of both. The integration of a standard slice based navigation interface into an augmented reality visualization overcomes the shortcomings of both systems.
---
paper_title: Integration of stereoscopic DSA and 3D MRI for image-guided neurosurgery

paper_content:

Abstract We demonstrate the feasibility and utility of using anatomical/vascular correlation in image-guided surgery, by interfacing a PC-based stereoscopic Digital Subtraction Angiography (DSA) analysis system to a three-dimensional (3D) image based surgical workstation that has been modified to allow presentation of stereoscopic images. Numerical values representing the position and angulation of a hand-held probe are transmitted to both systems simultaneously, enabling the probe to be visualized stereoscopically in both anatomical and vascular images during the surgical procedure. The integration of the patient's vascular and anatomical data in this way provides the surgeon with a complete overview of brain structures through which he is passing the electrode-guiding cannulas, enabling him to avoid critical vessels en route to the targets.
---
paper_title: New Visualization Techniques for in Utero Surgery: Amnioscopy with a Three-Dimensional Head-Mounted Display and a Computer-Controlled Endoscope

paper_content:

ABSTRACT Endoscopic fetal surgery may reduce preterm labor associated with open hysterotomy but is partially limited by current visualization technology. We investigated a three-dimensional (3D) imaging system coupled to a head-mounted display (3D-HMD) and also employed a computer-controlled zoom endoscope for noninsufflated amnioscopy. Pregnant sheep were prepared in aseptic fashion for general anesthesia. Uterine access was obtained following maternal laparoscopy. A 10-mm zoom endoscope (Vista Medical Technologies, Carlsbad, CA) was used to examine the fetus and uterine contents. Fetal limbs were exteriorized for microsurgery. A new system (Vista Medical Technologies) was attached to an operative microscope, permitting projection of a 3D image via an HMD. The fetus and umbilical cord were inspected using the zoom endoscope, which changes the depth of focus under computer control. Basic manipulations of the fetus and cord were easily completed. Real-time 3D fetal imaging was accomplished. The added depth...
---
paper_title: Task performance in endoscopic surgery is influenced by location of the image display.

paper_content:

ObjectiveTo investigate the influence of image display location on endoscopic task performance in endoscopic surgery.Summary Background DataThe image display system is the only visual interface between the surgeon or interventionist and the operative field. Several factors influence the correct
---
paper_title: Soft tissue navigation for laparoscopic prostatectomy: evaluation of camera pose estimation for enhanced visualization

paper_content:

We introduce a novel navigation system to support minimally invasive prostate surgery. The system utilizes ::: transrectal ultrasonography (TRUS) and needle-shaped navigation aids to visualize hidden structures via Augmented ::: Reality. During the intervention, the navigation aids are segmented once from a 3D TRUS dataset and ::: subsequently tracked by the endoscope camera. Camera Pose Estimation methods directly determine position ::: and orientation of the camera in relation to the navigation aids. Accordingly, our system does not require any ::: external tracking device for registration of endoscope camera and ultrasonography probe. In addition to a preoperative ::: planning step in which the navigation targets are defined, the procedure consists of two main steps which ::: are carried out during the intervention: First, the preoperatively prepared planning data is registered with an ::: intraoperatively acquired 3D TRUS dataset and the segmented navigation aids. Second, the navigation aids are ::: continuously tracked by the endoscope camera. The camera's pose can thereby be derived and relevant medical ::: structures can be superimposed on the video image. ::: This paper focuses on the latter step. We have implemented several promising real-time algorithms and ::: incorporated them into the Open Source Toolkit MITK (www.mitk.org). Furthermore, we have evaluated them ::: for minimally invasive surgery (MIS) navigation scenarios. For this purpose, a virtual evaluation environment ::: has been developed, which allows for the simulation of navigation targets and navigation aids, including their ::: measurement errors. Besides evaluating the accuracy of the computed pose, we have analyzed the impact of an ::: inaccurate pose and the resulting displacement of navigation targets in Augmented Reality.
---
paper_title: Recent Advances in Augmented Reality

paper_content:

In 1997, Azuma published a survey on augmented reality (AR). Our goal is to complement, rather than replace, the original survey by presenting representative examples of the new advances. We refer one to the original survey for descriptions of potential applications (such as medical visualization, maintenance and repair of complex equipment, annotation, and path planning); summaries of AR system characteristics (such as the advantages and disadvantages of optical and video approaches to blending virtual and real, problems in display focus and contrast, and system portability); and an introduction to the crucial problem of registration, including sources of registration error and error-reduction strategies.
---
paper_title: Determination of ventilatory liver movement via radiographic evaluation of diaphragm position.

paper_content:

PURPOSE ::: To determine the accuracy of estimation of liver movement inferred by observing diaphragm excursion on radiographic images. ::: ::: ::: METHODS AND MATERIALS ::: Eight patients with focal liver cancer had platinum embolization microcoils implanted in their livers during catheterization of the hepatic artery for delivery of regional chemotherapy. These patients underwent fluoroscopy, during which normal breathing movement was recorded on videotape. Movies of breathing movement were digitized, and the relative projected positions of the diaphragm and coils were recorded. For 6 patients, daily radiographs were also acquired during treatment. Retrospective measurements of coil position were taken after the diaphragm was aligned with the superior portion of the liver on digitally reconstructed radiographs. ::: ::: ::: RESULTS ::: Coil movement of 4.9 to 30.4 mm was observed during normal breathing. Diaphragm position tracked inferior-superior coil displacement accurately (population sigma 1.04 mm) throughout the breathing cycle. The range of coil movement was predicted by the range of diaphragm movement with an accuracy of 2.09 mm (sigma). The maximum error observed measuring coil movement using diaphragm position was 3.8 mm for a coil 9.8 cm inferior to the diaphragm. However, the distance of the coil from the top of the diaphragm did not correlate significantly with the error in predicting liver excursion. Analysis of daily radiographs showed that the error in predicting coil position using the diaphragm as an alignment landmark was 1.8 mm (sigma) in the inferior-superior direction and 2.2 mm in the left-right direction, similar in magnitude to the inherent uncertainty in alignment. ::: ::: ::: CONCLUSIONS ::: This study demonstrated that the range of ventilatory movement of different locations within the liver is predicted by diaphragm position to an accuracy that matches or exceeds existing systems for ventilatory tracking. This suggests that the diaphragm is an acceptable anatomic landmark for radiographic estimation of liver movement in anterior-posterior projections for most patients.
---
paper_title: Scale-Invariant Registration of Monocular Endoscopic Images to CT-Scans for Sinus Surgery

paper_content:

We present a scale-invariant registration method for 3D structures reconstructed from a monocular endoscopic camera to pre-operative CT-scans. The presented approach is based on a previously presented method [2] for reconstruction of a scaled 3D model of the environment from unknown camera motion. We use this scaleless reconstruction as input to a PCA-based algorithm that recovers the scale and pose parameters of the camera in the coordinate frame of the CT scan. The result is used in an ICP registration method to refine the registration estimates.
---
paper_title: Automatic Patient Registration for Port Placement in Minimally Invasive Endoscopic Surgery

paper_content:

Optimal port placement is a delicate issue in minimally invasive endoscopic surgery, particularly in robotically assisted surgery. A good choice of the instruments' and endoscope's ports can avoid timeconsuming consecutive new port placement. We present a novel method to intuitively and precisely plan the port placement. The patient is registered to its pre-operative CT by just moving the endoscope around fiducials, which are attached to the patient's thorax and are visible in its CT. Their 3D positions are automatically reconstructed. Without prior time-consuming segmentation, the pre-operative CT volume is directly rendered with respect to the endoscope or instruments. This enables the simulation of a camera flight through the patient's interior along the instruments' axes to easily validate possible ports.
---
paper_title: Calibration Requirements and Procedures for a Monitor-Based Augmented Reality System

paper_content:

Augmented reality entails the use of models and their associated renderings to supplement information in a real scene. In order for this information to be relevant or meaningful, the models must be positioned and displayed in such a way that they blend into the real world in terms of alignments, perspectives, illuminations, etc. For practical reasons the information necessary to obtain this realistic blending cannot be known a priori, and cannot be hard wired into a system. Instead a number of calibration procedures are necessary so that the location and parameters of each of the system components are known. We identify the calibration steps necessary to build a computer model of the real world and then, using the monitor based augmented reality system developed at ECRC (GRASP) as an example, we describe each of the calibration processes. These processes determine the internal parameters of our imaging devices (scan converter, frame grabber, and video camera), as well as the geometric transformations that relate all of the physical objects of the system to a known world coordinate system. >
---
paper_title: Markerless endoscopic registration and referencing

paper_content:

Accurate patient registration and referencing is a key element in navigated surgery. Unfortunately all existing methods are either invasive or very time consuming. We propose a fully non-invasive optical approach using a tracked monocular endoscope to reconstruct the surgical scene in 3D using photogrammetric methods. The 3D reconstruction can then be used for matching the pre-operative data to the intra-operative scene. In order to cope with the near real-time requirements for referencing, we use a novel, efficient 3D point management method during 3D model reconstruction. ::: ::: The presented prototype system provides a reconstruction accuracy of 0.1 mm and a tracking accuracy of 0.5 mm on phantom data. The ability to cope with real data is demonstrated by cadaver experiments.
---
paper_title: Magneto-Optic Tracking of a Flexible Laparoscopic Ultrasound Transducer for Laparoscope Augmentation

paper_content:

In abdominal surgery, a laparoscopic ultrasound transducer is commonly used to detect lesions such as metastases. The determination and visualization of position and orientation of its flexible tip in relation to the patient or other surgical instruments can be of much help to (novice) surgeons utilizing the transducer intraoperatively. This difficult subject has recently been paid attention to by the scientific community [1, 2, 3, 4, 5, 6]. Electromagnetic tracking systems can be applied to track the flexible tip. However, the magnetic field can be distorted by ferromagnetic material. This paper presents a new method based on optical tracking of the laparoscope and magneto-optic tracking of the transducer, which is able to automatically detect field distortions. This is used for a smooth augmentation of the B-scan images of the transducer directly on the camera images in real time.
---
paper_title: Methods for Modeling and Predicting Mechanical Deformations of the Breast Under External Perturbations

paper_content:

Currently, High Field (1.5T) Superconducting MR imaging does not allow live guidance during needle breast procedures, which allow only to calculate approximately the location of a cancerous tumor in the patient breast before inserting the needle. It can then become relatively uncertain that the tissue specimen removed during the biopsy actually belongs to the lesion of interest. A new method for guiding clinical breast biopsy is presented, based on a deformable finite element model of the breast. The geometry of the model is constructed from MR data, and its mechanical properties are modeled using a non-linear material model. This method allows imaging the breast without or with mild compression before the procedure, then compressing the breast and using the finite element model to predict the tumor's position during the procedure in a total time of less than a half-hour.
---
paper_title: The use of active breathing control (ABC) to reduce margin for breathing motion

paper_content:

Abstract Purpose: For tumors in the thorax and abdomen, reducing the treatment margin for organ motion due to breathing reduces the volume of normal tissues that will be irradiated. A higher dose can be delivered to the target, provided that the risk of marginal misses is not increased. To ensure safe margin reduction, we investigated the feasibility of using active breathing control (ABC) to temporarily immobilize the patient's breathing. Treatment planning and delivery can then be performed at identical ABC conditions with minimal margin for breathing motion. Methods and Materials: An ABC apparatus is constructed consisting of 2 pairs of flow monitor and scissor valve, 1 each to control the inspiration and expiration paths to the patient. The patient breathes through a mouth-piece connected to the ABC apparatus. The respiratory signal is processed continuously, using a personal computer that displays the changing lung volume in real-time. After the patient's breathing pattern becomes stable, the operator activates ABC at a preselected phase in the breathing cycle. Both valves are then closed to immobilize breathing motion. Breathing motion of 12 patients were held with ABC to examine their acceptance of the procedure. The feasibility of applying ABC for treatment was tested in 5 patients by acquiring volumetric scans with a spiral computed tomography (CT) scanner during active breath-hold. Two patients had Hodgkin's disease, 2 had metastatic liver cancer, and 1 had lung cancer. Two intrafraction ABC scans were acquired at the same respiratory phase near the end of normal or deep inspiration. An additional ABC scan near the end of normal expiration was acquired for 2 patients. The ABC scans were also repeated 1 week later for a Hodgkin's patient. In 1 liver patient, ABC scans were acquired at 7 different phases of the breathing cycle to facilitate examination of the liver motion associated with ventilation. Contours of the lungs and livers were outlined when applicable. The variation of the organ positions and volumes for the different scans were quantified and compared. Results: The ABC procedure was well tolerated in the 12 patients. When ABC was applied near the end of normal expiration, the minimal duration of active breath-hold was 15 s for 1 patient with lung cancer, and 20 s or more for all other patients. The duration was greater than 40 s for 2 patients with Hodgkin's disease when ABC was applied during deep inspiration. Scan artifacts associated with normal breathing motion were not observed in the ABC scans. The analysis of the small set of intrafraction scan data indicated that with ABC, the liver volumes were reproducible at about 1%, and lung volumes to within 6%. The excursions of a "center of target" parameter for the livers were less than 1 mm at the same respiratory phase, but were larger than 4 mm at the extremes of the breathing cycle. The inter-fraction scan study indicated that daily setup variation contributed to the uncertainty in assessing the reproducibility of organ immobilization with ABC between treatment fractions. Conclusion: The results were encouraging; ABC provides a simple means to minimize breathing motion. When applied for CT scanning and treatment, the ABC procedure requires no more than standard operation of the CT scanner or the medical accelerator. The ABC scans are void of motion artifacts commonly seen on fast spiral CT scans. When acquired at different points in the breathing cycle, these ABC scans show organ motion in three-dimension (3D) that can be used to enhance treatment planning. Reproducibility of organ immobilization with ABC throughout the course of treatment must be quantified before the procedure can be applied to reduce margin for conformal treatment.
---
paper_title: A Survey of Medical Image Registration

paper_content:

The purpose of this paper is to present a survey of recent (published in 1993 or later) publications concerning medical image registration techniques. These publications will be classified according to a model based on nine salient criteria, the main dichotomy of which is extrinsic versus intrinsic methods. The statistics of the classification show definite trends in the evolving registration techniques, which will be discussed. At this moment, the bulk of interesting intrinsic methods is based on either segmented points or surfaces, or on techniques endeavouring to use the full information content of the images involved.
---
paper_title: Registration of head volume images using implantable fiducial markers

paper_content:

In this paper, we describe an extrinsic point-based, interactive image-guided neurosurgical system designed at Vanderbilt University as part of a collaborative effort among the departments of neurological surgery, computer science, and biomedical engineering. Multimodal image-to- image and image-to-physical registration is accomplished using implantable markers. Physical space tracking is accomplished with optical triangulation. We investigate the theoretical accuracy of point-based registration using numerical simulations, the experimental accuracy of our system using data obtained with a phantom, and the clinical accuracy of our system using data acquired in a prospective clinical trial by six neurosurgeons at four medical centers from 158 patients undergoing craniotomies to resect cerebral lesions. We can determine the position of our markers with an error of approximately 0.4 mm in x-ray computed tomography (CT) and magnetic resonance (MR) images and 0.3 mm in physical space. The theoretical registration error using four such markers distributed around the head in a configuration that is clinically practical is approximately 0.5 - 0.6 mm. The mean CT-physical registration error for the phantom experiments is 0.5 mm and for the clinical data obtained with rigid head fixation during scanning is 0.7 mm. The mean CT-MR registration error for the clinical data obtained without rigid head fixation during scanning is 1.4 mm, which is the highest mean error that we observed. These theoretical and experimental findings indicate that this system is an accurate navigational aid that can provide real-time feedback to the surgeon about anatomical structures encountered in the surgical field.
---
paper_title: Registration-free laparoscope augmentation for intra-operative liver resection planning

paper_content:

In recent years, an increasing number of liver tumor indications were treated by minimally invasive laparoscopic ::: resection. Besides the restricted view, a major issue in laparoscopic liver resection is the enhanced visualization ::: of (hidden) vessels, which supply the tumorous liver segment and thus need to be divided prior to the resection. ::: To navigate the surgeon to these vessels, pre-operative abdominal imaging data can hardly be used due to intraoperative ::: organ deformations mainly caused by appliance of carbon dioxide pneumoperitoneum and respiratory ::: motion. While regular respiratory motion can be gated and synchronized intra-operatively, motion caused by ::: pneumoperitoneum is individual for every patient and difficult to estimate. ::: Therefore, we propose to use an optically tracked mobile C-arm providing cone-beam CT imaging capability intraoperatively. ::: The C-arm is able to visualize soft tissue by means of its new flat panel detector and is calibrated ::: offline to relate its current position and orientation to the coordinate system of a reconstructed volume. Also ::: the laparoscope is optically tracked and calibrated offline, so both laparoscope and C-arm are registered in the ::: same tracking coordinate system. ::: Intra-operatively, after patient positioning, port placement, and carbon dioxide insufflation, the liver vessels are ::: contrasted and scanned during patient exhalation. Immediately, a three-dimensional volume is reconstructed. ::: Without any further need for patient registration, the volume can be directly augmented on the live laparoscope ::: video, visualizing the contrasted vessels. This augmentation provides the surgeon with advanced visual aid for ::: the localization of veins, arteries, and bile ducts to be divided or sealed.
---
paper_title: An automatic registration method for frameless stereotaxy, image guided surgery, and enhanced reality visualization

paper_content:

There is a need for frameless guidance systems to help neurosurgeons to plan the exact location of a craniotomy, to define the margins of tumors and to precisely identify locations of neighboring critical structures. We have developed an automatic technique for registering clinical data, such as segmented MRI or CT reconstructions, with the patient's head on the operating table. A second method calibrates the position of a video camera relative to the patient. The combination allows a visual mix of live video of the patient with the segmented 3D MRI or CT model, enabling enhanced reality techniques for planning and guiding neurosurgical procedures, and to interactively view extracranial or intracranial structures non-intrusively. Extensions of the method include image guided biopsies, focused therapeutic procedures and clinical studies involving change detection over time sequences of images. >
---
paper_title: Endoscope-based hybrid navigation system for minimally invasive ventral spine surgeries.

paper_content:

The availability of high-resolution, magnified, and relatively noise-free endoscopic images in a small workspace, 4-10 cm from the endoscope tip, opens up the possibility of using the endoscope as a tracking tool. We are developing a hybrid navigation system in which image-analysis-based 2D-3D tracking is combined with optoelectronic tracking (Optotrak) for computer-assisted navigation in laparoscopic ventral spine surgeries. Initial results are encouraging and confirm the ability of the endoscope to serve as a tracking tool in surgical navigation where sub-millimetric accuracy is mandatory.
---
paper_title: Intraoperative Laparoscope Augmentation for Port Placement and Resection Planning in Minimally Invasive Liver Resection

paper_content:

In recent years, an increasing number of liver tumor indications were treated by minimally invasive laparoscopic resection. Besides the restricted view, two major intraoperative issues in laparoscopic liver resection are the optimal planning of ports as well as the enhanced visualization of (hidden) vessels, which supply the tumorous liver segment and thus need to be divided (e.g., clipped) prior to the resection. We propose an intuitive and precise method to plan the placement of ports. Preoperatively, self-adhesive fiducials are affixed to the patient's skin and a computed tomography (CT) data set is acquired while contrasting the liver vessels. Immediately prior to the intervention, the laparoscope is moved around these fiducials, which are automatically reconstructed to register the patient to its preoperative imaging data set. This enables the simulation of a camera flight through the patient's interior along the laparoscope's or instruments' axes to easily validate potential ports. Intraoperatively, surgeons need to update their surgical planning based on actual patient data after organ deformations mainly caused by application of carbon dioxide pneumoperitoneum. Therefore, preoperative imaging data can hardly be used. Instead, we propose to use an optically tracked mobile C-arm providing cone-beam CT imaging capability intraoperatively. After patient positioning, port placement, and carbon dioxide insufflation, the liver vessels are contrasted and a 3-D volume is reconstructed during patient exhalation. Without any further need for patient registration, the reconstructed volume can be directly augmented on the live laparoscope video, since prior calibration enables both the volume and the laparoscope to be positioned and oriented in the tracking coordinate frame. The augmentation provides the surgeon with advanced visual aid for the localization of veins, arteries, and bile ducts to be divided or sealed.
---
paper_title: 3D Ultrasound System Using a Magneto-optic Hybrid Tracker for Augmented Reality Visualization in Laparoscopic Liver Surgery

paper_content:

A three-dimensional ultrasound (3D-US) system suitable for laparoscopic surgery that uses a novel magneto-optic hybrid tracker configuration. Our aim is to integrate 3D-US into a laparoscopic AR system. A 5D miniature magnetic tracker is combined with a 6D optical tracker outside the body to perform 6D tracking of a flexible US probe tip in the abdominal cavity. 6D tracking parameters at the tip are obtained by combining the 5D parameters at the tip inside the body, the 6D parameters at the US probe handle outside the body, and the restriction of the tip motion relative to the handle. The system was evaluated in comparison with a conventional 3D ultrasound system. Although the accuracy of the proposed system was somewhat inferior to that of the conventional one, both the accuracy and sweet spot area were found to be acceptable for clinical use.
---
paper_title: 1 Navigation in Endoscopic Soft Tissue Surgery- Perspectives and Limitations

paper_content:

Despite rapid developments in the research areas of medical imaging, medical image processing, and robotics, the use of computer assistance in surgical routine is still limited to diagnostics, surgical planning, and interventions on mostly rigid structures. In order to establish a computer-aided workflow from diagnosis to surgical treatment and follow-up, several proposals for computer-assisted soft tissue interventions have been made in recent years. By means of different pre- and intraoperative information sources, such as surgical planning, intraoperative imaging, and tracking devices, surgical navigation systems aim to support surgeons in localizing anatomical targets, observing critical structures, and sparing healthy tissue. Current research in particular addresses the problem of organ shift and tissue deformation, and obstacles in communication between navigation system and surgeon. In this paper, we review computer-assisted navigation systems for soft tissue surgery. We concentrate on approaches that can be applied in endoscopic thoracic and abdominal surgery, because endoscopic surgery has special needs for image guidance due to limitations in perception. Furthermore, this paper informs the reader about new trends and technologies in the area of computer-assisted surgery. Finally, a balancing of the key challenges and possible benefits of endoscopic navigation refines the perspectives of this increasingly important discipline of computer-aided medical procedures.
---
paper_title: Registration of physical space to laparoscopic image space for use in minimally invasive hepatic surgery

paper_content:

While laparoscopes are used for numerous minimally invasive (MI) procedures, MI liver resection and ablative surgery is infrequently performed. The paucity of cases is due to the restriction of the field of view by the laparoscope and the difficulty in determining tumor location and margins under video guidance. By merging MI surgery with interactive, image-guided surgery (IIGS), the authors hope to overcome localization difficulties present in laparoscopic liver procedures. One key component of any IIGS system is the development of accurate registration techniques to map image space to physical or patient space. This manuscript focuses on the accuracy and analysis of the direct linear transformation (DLT) method to register physical space with laparoscopic image space on both distorted and distortion-corrected video images. Experiments were conducted on a liver-sized plastic phantom affixed with 20 markers at various depths, After localizing the points in both physical and laparoscopic image space, registration accuracy was assessed for different combinations and numbers of control points (n) to determine the quantity necessary to develop a robust registration matrix. For n=11, average target registration error (TRE) was 0.70/spl plusmn/0.20 mm. The authors also studied the effects of distortion correction on registration accuracy For the particular distortion correction method and laparoscope used in the authors' experiments, there was no statistical significance between physical to image registration error for distorted and corrected images. In cases where a minimum number of control points (n=6) are acquired, the DLT is often not stable and the mathematical process can lead to high TRE values. Mathematical filters developed through the analysis of the DLT were used to prospectively eliminate outlier cases where the TRE was high. For n=6, prefilter average TRE was 17.4/spl plusmn/153 mm for all trials; when the filters were applied, average TRE decreased to 1.64/spl plusmn/1.10 mm for the remaining trials.
---
paper_title: Predicting error in rigid-body point-based registration

paper_content:

Guidance systems designed for neurosurgery, hip surgery, and spine surgery, and for approaches to other anatomy that is relatively rigid can use rigid-body transformations to accomplish image registration. These systems often rely on point-based registration to determine the transformation, and many such systems use attached fiducial markers to establish accurate fiducial points for the registration, the points being established by some fiducial localization process. Accuracy is important to these systems, as is knowledge of the level of that accuracy. An advantage of marker-based systems, particularly those in which the markers are bone-implanted, is that registration error depends only on the fiducial localization error (FLE) and is thus to a large extent independent of the particular object being registered. Thus, it should be possible to predict the clinical accuracy of marker-based systems on the basis of experimental measurements made with phantoms or previous patients. This paper presents two new expressions for estimating registration accuracy of such systems and points out a danger in using a traditional measure of registration accuracy. The new expressions represent fundamental theoretical results with regard to the relationship between localization error and registration error in rigid-body, point-based registration. Rigid-body, point-based registration is achieved by finding the rigid transformation that minimizes "fiducial registration error" (FRE), which is the root mean square distance between homologous fiducials after registration. Closed form solutions have been known since 1966. The expected value (FRE/sup 2/) depends on the number N of fiducials and expected squared value of FLE, (FLE/sup 2/), but in 1979 it was shown that (FRE/sup 2/) is approximately independent of the fiducial configuration C. The importance of this surprising result seems not yet to have been appreciated by the registration community: Poor registrations caused by poor fiducial configurations may appear to be good due to a small FRE value. A more critical and direct measure of registration error is the "target registration error" (TRE), which is the distance between homologous points other than the centroids of fiducials. Efforts to characterize its behavior have been made since 1989. Published numerical simulations have shown that (TRE/sup 2/) is roughly proportional to (FLE/sup 2/)/N and, unlike (FRE/sup 2/), does depend in some way on C. Thus, FRE, which is often used as feedback to the surgeon using a point-based guidance system, is in fact an unreliable indicator of registration-accuracy. In this work the authors derive approximate expressions for (TRE/sup 2/), and for the expected squared alignment error of an individual fiducial. They validate both approximations through numerical simulations. The former expression can be used to provide reliable feedback to the surgeon during surgery and to guide the placement of markers before surgery, or at least to warn the surgeon of potentially dangerous fiducial placements; the latter expression leads to a surprising conclusion: Expected registration accuracy (TRE) is worst near the fiducials that are most closely aligned! This revelation should be of particular concern to surgeons who may at present be relying on fiducial alignment as an indicator of the accuracy of their point-based guidance systems.
---
paper_title: An automatic six-degree-of-freedom image registration algorithm for image-guided frameless stereotaxic radiosurgery.

paper_content:

A frameless radiosurgical treatment system has been developed by coupling an orthogonal pair of real-time x-ray cameras to a robotically manipulated linear accelerator to guide the therapy beam to treatment sites within a patient's cranium. The two cameras observe the position and orientation of the patient's head in the treatment system coordinate frame. An image registration algorithm compares the two real-time radiographs to a corresponding pair of digitally synthesized radiographs derived from a CT study of the patient. The algorithm determines all six degrees of translational and rotational difference between the position of the head in the CT coordinate frame and its position in the treatment room coordinate frame. This allows translation of treatment planning coordinates into treatment room coordinates without rigidly fixing the patient's head position during either the CT scan or treatment. In this paper the image registration algorithm is described and measurements of the precision and speed with which the process can determine the patient's position are reported. The tests have demonstrated translational uncertainty of 0.5-1.0 mm per axis and rotational uncertainty of 0.6-1.3 degrees per axis, accomplished in approximately 2 s elapsed time.
---
paper_title: Depth perception: a major issue in medical AR: evaluation study by twenty surgeons

paper_content:

The idea of in-situ visualization for surgical procedures has been widely discussed in the community [1,2,3,4]. While the tracking technology offers nowadays a sufficient accuracy and visualization devices have been developed that fit seamlessly into the operational workflow [1,3], one crucial problem remains, which has been discussed already in the first paper on medical augmented reality [4]. Even though the data is presented at the correct place, the physician often perceives the spatial position of the visualization to be closer or further because of virtual/real overlay. ::: ::: This paper describes and evaluates novel visualization techniques that are designed to overcome misleading depth perception of trivially superimposed virtual images on the real view. We have invited 20 surgeons to evaluate seven different visualization techniques using a head mounted display (HMD). The evaluation has been divided into two parts. In the first part, the depth perception of each kind of visualization is evaluated quantitatively. In the second part, the visualizations are evaluated qualitatively in regard to user friendliness and intuitiveness. This evaluation with a relevant number of surgeons using a state-of-the-art system is meant to guide future research and development on medical augmented reality.
---
paper_title: Measurement of absolute latency for video see through augmented reality

paper_content:

Latency is a key property of video see through AR systems since users' performance is strongly related to it. However, there is no standard way of latency measurement of an AR system in the literature. We have created a stable and comparable way of estimating the latency in a video see through AR system. The latency is estimated by encoding the time in the image and decoding the time after camera feedback. We have encoded the time as a translation of a circle in the image. The cross ratio has been used as an image feature that is preserved in a projective transformation. The encoding allows for a simple but accurate way of decoding. We show that this way of encoding has an adequate accuracy for latency measurements. As the method allows for a series of automatic measurements we propose to visualize the measurements in a histogram. This histogram reveals meaningful information about the system other than the mean value and standard deviation of the latency. The method has been tested on four different AR systems that use different camera technology, resolution and frame rates.
---
paper_title: Registration Error Analysis for Augmented Reality

paper_content:

Augmented reality AR systems typically use see-through head-mounted displays STHMDs to superimpose images of computer-generated objects onto the user's view of the real environment in order to augment it with additional information. The main failing of current AR systems is that the virtual objects displayed in the STHMD appear in the wrong position relative to the real environment. This registration error has many causes: system delay, tracker error, calibration error, optical distortion, and misalignment of the model, to name only a few. Although some work has been done in the area of system calibration and error correction, very little work has been done on characterizing the nature and sensitivity of the errors that cause misregistration in AR systems. This paper presents the main results of an end-to-end error analysis of an optical STHMD-based tool for surgery planning. The analysis was done with a mathematical model of the system and the main results were checked by taking measurements on a real system under controlled circumstances. The model makes it possible to analyze the sensitivity of the system-registration error to errors in each part of the system. The major results of the analysis are: 1 Even for moderate head velocities, system delay causes more registration error than all other sources combined; 2 eye tracking is probably not necessary; 3 tracker error is a significant problem both in head tracking and in system calibration; 4 the World or reference coordinate system adds error and should be omitted when possible; 5 computational correction of optical distortion may introduce more delay-induced registration error than the distortion error it corrects, and 6 there are many small error sources that will make submillimeter registration almost impossible in an optical STHMD system without feedback. Although this model was developed for optical STHMDs for surgical planning, many of the results apply to other HMDs as well.
---
paper_title: Augmented workspace: designing an AR testbed

paper_content:

We have implemented a tabletop setup to explore augmented reality (AR) visualization. We call this setup an "augmented workspace". The user sits at the table and performs a manual task, guided by computer graphics overlaid on to his view. The setup serves as a testbed for developing the technology and for studying visual perception issues. The user wears a custom video see-through head mounted display (HMD). Two color video cameras attached to the HMD provide a stereo view of the scene, and a third video camera is added for tracking. The system runs at the full 30-Hz video frame rate with a latency of about 0.1 s, generating a stable augmentation with no apparent jitter visible in the composite images. Two SGI Visual Workstations provide the computing power for the system. In this paper, we describe the augmented workspace system in more detail and discuss several design issues.
---
paper_title: Reaching for objects in VR displays: lag and frame rate

paper_content:

This article reports the results from three experimental studies of reaching behavior in a head-coupled stereo display system with a hand-tracking subsystem for object selection. It is found that lag in the head-tracking system is relatively unimportant in predicting performance, whereas lag in the hand-tracking system is critical. The effect of hand lag can be modeled by means of a variation on Fitts' Law with the measured system lag introduced as a multiplicative variable to the Fitts' Law index of difficulty. This means that relatively small lags can cause considerable degradation in performance if the targets are small. Another finding is that errors are higher for movement in and out of the screen, as compared to movements in the plane of the screen, and there is a small (10%) time penalty for movement in the Z direction in all three experiments. Low frame rates cause a degradation in performance; however, this can be attributed to the lag which is caused by low frame rates, particularly if double buffering is used combined with early sampling of the hand-tracking device.
---
paper_title: Analysis of head pose accuracy in augmented reality

paper_content:

A method is developed to analyze the accuracy of the relative head-to-object position and orientation (pose) in augmented reality systems with head-mounted displays. From probabilistic estimates of the errors in optical tracking sensors, the uncertainty in head-to-object pose can be computed in the form of a covariance matrix. The positional uncertainty can be visualized as a 3D ellipsoid. One useful benefit of having an explicit representation of uncertainty is that we can fuse sensor data from a combination of fixed and head-mounted sensors in order to improve the overall registration accuracy. The method was applied to the analysis of an experimental augmented reality system, incorporating an optical see-through head-mounted display, a head-mounted CCD camera, and a fixed optical tracking sensor. The uncertainty of the pose of a movable object with respect to the head-mounted display was analyzed. By using both fixed and head mounted sensors, we produced a pose estimate that is significantly more accurate than that produced by either sensor acting alone.
---
paper_title: An Accuracy Certified Augmented Reality System for Therapy Guidance

paper_content:

Our purpose is to provide an augmented reality system for Radio-Frequency guidance that could superimpose a 3D model of the liver, its vessels and tumors (reconstructed from CT images) on external video images of the patient. In this paper, we point out that clinical usability not only need the best affordable registration accuracy, but also a certification that the required accuracy is met, since clinical conditions change from one intervention to the other. Beginning by addressing accuracy performances, we show that a 3D/2D registration based on radio-opaque fiducials is more adapted to our application constraints than other methods. Then, we outline a lack in their statistical assumptions which leads us to the derivation of a new extended 3D/2D criterion. Careful validation experiments on real data show that an accuracy of 2 mm can be achieved in clinically relevant conditions, and that our new criterion is up to 9% more accurate, while keeping a computation time compatible with real-time at 20 to 40 Hz.
---
paper_title: Method for estimating dynamic EM tracking accuracy of surgical navigation tools

paper_content:

Optical tracking systems have been used for several years in image guided medical procedures. Vendors often state static accuracies of a single retro-reflective sphere or LED. Expensive coordinate measurement machines (CMM) are used to validate the positional accuracy over the specified working volume. Users are interested in the dynamic accuracy of their tools. The configuration of individual sensors into a unique tool, the calibration of the tool tip, and the motion of the tool contribute additional errors. Electromagnetic (EM) tracking systems are considered an enabling technology for many image guided procedures because they are not limited by line-of-sight restrictions, take minimum space in the operating room, and the sensors can be very small. It is often difficult to quantify the accuracy of EM trackers because they can be affected by field distortion from certain metal objects. Many high-accuracy measurement devices can affect the EM measurements being validated. EM Tracker accuracy tends to vary over the working volume and orientation of the sensors. We present several simple methods for estimating the dynamic accuracy of EM tracked tools. We discuss the characteristics of the EM Tracker used in the GE Healthcare family of surgical navigation systems. Results for other tracking systems are included.
---
paper_title: Estimating and adapting to registration errors in augmented reality systems

paper_content:

All augmented reality (AR) systems must deal with registration errors. While most AR systems attempt to minimize registration errors through careful calibration, registration errors can never be completely eliminated in any realistic system. In this paper, we describe a robust and efficient statistical method for estimating registration errors. Our method generates probabilistic error estimates for points in the world, in either 3D world coordinates or 2D screen coordinates. We present a number of examples illustrating how registration error estimates can be used in AR interfaces, and describe a method for estimating registration errors of objects based on the expansion and contraction of their 2D convex hulls.
---
paper_title: Online estimation of the target registration error for n-ocular optical tracking systems

paper_content:

For current surgical navigation systems optical tracking is state of the art. The accuracy of these tracking systems is currently determined statically for the case of full visibility of all tracking targets. We propose a dynamic determination of the accuracy based on the visibility and geometry of the tracking setup. This real time estimation of accuracy has a multitude of applications. For multiple camera systems it allows reducing line of sight problems and guaranteeing a certain accuracy. The visualization of these accuracies allows surgeons to perform the procedures taking to the tracking accuracy into account. It also allows engineers to design tracking setups interactively guaranteeing a certain accuracy. ::: ::: Our model is an extension to the state of the art models of Fitzpatrick et al. [1] and Hoff et al. [2]. We model the error in the camera sensor plane. The error is propagated using the internal camera parameter, camera poses, tracking target poses, target geometry and marker visibility, in order to estimate the final accuracy of the tracked instrument.
---
paper_title: Predicting error in rigid-body point-based registration

paper_content:

Guidance systems designed for neurosurgery, hip surgery, and spine surgery, and for approaches to other anatomy that is relatively rigid can use rigid-body transformations to accomplish image registration. These systems often rely on point-based registration to determine the transformation, and many such systems use attached fiducial markers to establish accurate fiducial points for the registration, the points being established by some fiducial localization process. Accuracy is important to these systems, as is knowledge of the level of that accuracy. An advantage of marker-based systems, particularly those in which the markers are bone-implanted, is that registration error depends only on the fiducial localization error (FLE) and is thus to a large extent independent of the particular object being registered. Thus, it should be possible to predict the clinical accuracy of marker-based systems on the basis of experimental measurements made with phantoms or previous patients. This paper presents two new expressions for estimating registration accuracy of such systems and points out a danger in using a traditional measure of registration accuracy. The new expressions represent fundamental theoretical results with regard to the relationship between localization error and registration error in rigid-body, point-based registration. Rigid-body, point-based registration is achieved by finding the rigid transformation that minimizes "fiducial registration error" (FRE), which is the root mean square distance between homologous fiducials after registration. Closed form solutions have been known since 1966. The expected value (FRE/sup 2/) depends on the number N of fiducials and expected squared value of FLE, (FLE/sup 2/), but in 1979 it was shown that (FRE/sup 2/) is approximately independent of the fiducial configuration C. The importance of this surprising result seems not yet to have been appreciated by the registration community: Poor registrations caused by poor fiducial configurations may appear to be good due to a small FRE value. A more critical and direct measure of registration error is the "target registration error" (TRE), which is the distance between homologous points other than the centroids of fiducials. Efforts to characterize its behavior have been made since 1989. Published numerical simulations have shown that (TRE/sup 2/) is roughly proportional to (FLE/sup 2/)/N and, unlike (FRE/sup 2/), does depend in some way on C. Thus, FRE, which is often used as feedback to the surgeon using a point-based guidance system, is in fact an unreliable indicator of registration-accuracy. In this work the authors derive approximate expressions for (TRE/sup 2/), and for the expected squared alignment error of an individual fiducial. They validate both approximations through numerical simulations. The former expression can be used to provide reliable feedback to the surgeon during surgery and to guide the placement of markers before surgery, or at least to warn the surgeon of potentially dangerous fiducial placements; the latter expression leads to a surprising conclusion: Expected registration accuracy (TRE) is worst near the fiducials that are most closely aligned! This revelation should be of particular concern to surgeons who may at present be relying on fiducial alignment as an indicator of the accuracy of their point-based guidance systems.
---
paper_title: Neurosurgical Guidance Using the Stereo Microscope

paper_content:

Many neuro- and ENT surgical procedures are performed using the operating microscope. Conventionally, the surgeon cannot accurately relate information from preoperative radiological images to the appearance of the surgical field. We propose that the best way do this is to superimpose image derived data upon the operative scene. We create a model of relevant structures (e.g. tumor volume, blood vessels and nerves) from multimodality preoperative images. By calibrating microscope optics, registering the patient in-theatre to image coordinates, and tracking the microscope intra-operatively, we can generate stereo projections of the 3D model and project them into the microscope eyepieces, allowing critical structures to be overlayed on the operative scene in the correct position. We have completed initial evaluation with a head phantom, and are about to start clinical evaluation on patients. With the head phantom a theoretical accuracy of 4.6mm was calculated and the observed accuracy ranged from 2mm to 5mm.
---
paper_title: Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth

paper_content:

Publisher Summary ::: This chapter discusses three questions: Why are there so many sources of information about layout? How is it that one perceives layout with near-metric accuracy when none of these sources yields metric information about it? Can one not do better, theoretically, in understanding the perception of layout than simply make a list? The answer to the first question begins with Todd's answer. Perceiving layout is extremely important to human beings, so important that it must be redundantly specified so that the redundancy can guard against the failure of any given source or the failure of any of the assumptions on which a given source is based. However, information redundancy is only part of the answer. Different sources of information about layout metrically reinforce and contrast with each other, providing a powerful network of constraints. The answer to the second proceeds from this idea. Through the analysis of depth-threshold functions for nine different sources of information about layout, one can begin to understand how those sources of information sharing the same-shaped functions across distances can help ramify judgments of layout by serving to correct measurement errors in each. Third, on the basis of the analyses and the pattern of functions, it suggests that list making has misled about space and layout. Psychologists and other vision scientists have generally considered layout, space, and distance as a uniform commodity in which observers carry out their day-to-day activities.
---
paper_title: Perceptual issues in augmented reality

paper_content:

Between the extremes of real life and Virtual Reality lies the spectrum of Mixed Reality, in which views of the real world are combined in some proportion with views of a virtual environment. Combining direct view, stereoscopic video, and stereoscopic graphics, Augmented Reality describes that class of displays that consists primarily of a real environment, with graphic enhancements or augmentations. Augmented Virtuality describes that class of displays that enhance the virtual experience by adding elements of the real environment. All Mixed Reality systems are limited in their capability of accurately displaying and controlled all relevant depth cues, and as a result, perceptual biases can interfere with task performance. In this paper we identify and discuss eighteen issues that pertain to Mixed Reality in general, and Augmented Reality in particular.
---
paper_title: Surface transparency makes stereo overlays unpredictable: the implications for augmented reality.

paper_content:

The principle of using stereoscopic displays is to present the viewer with an accurate perception of 3D space. Stereopsis is a powerful binocular cue that will supplement any monocular information in the scene. Our work with an optical augmented reality system has highlighted one scenario where an accurate sense of depth cannot be easily achieved from stereoscopic images. In our augmented reality system we use stereo images of anatomical structures overlaid on the patient for surgical guidance. It is essential that the surgeon can accurately localize the images during surgery. When the stereo images are presented behind the transparent physical surface the perception of the depth of the images can become unstable and ambiguous, despite good system calibration, registration and tracking. This paper reviews possible reasons for the failure in accurate depth perception and presents some ideas on how this might be corrected for in an optical augmented reality
---
paper_title: Depth perception: a major issue in medical AR: evaluation study by twenty surgeons

paper_content:

The idea of in-situ visualization for surgical procedures has been widely discussed in the community [1,2,3,4]. While the tracking technology offers nowadays a sufficient accuracy and visualization devices have been developed that fit seamlessly into the operational workflow [1,3], one crucial problem remains, which has been discussed already in the first paper on medical augmented reality [4]. Even though the data is presented at the correct place, the physician often perceives the spatial position of the visualization to be closer or further because of virtual/real overlay. ::: ::: This paper describes and evaluates novel visualization techniques that are designed to overcome misleading depth perception of trivially superimposed virtual images on the real view. We have invited 20 surgeons to evaluate seven different visualization techniques using a head mounted display (HMD). The evaluation has been divided into two parts. In the first part, the depth perception of each kind of visualization is evaluated quantitatively. In the second part, the visualizations are evaluated qualitatively in regard to user friendliness and intuitiveness. This evaluation with a relevant number of surgeons using a state-of-the-art system is meant to guide future research and development on medical augmented reality.
---
paper_title: pq-space Based Non-Photorealistic Rendering for Augmented Reality

paper_content:

The increasing use of robotic assisted minimally invasive surgery (MIS) provides an ideal environment for using Augmented Reality (AR) for performing image guided surgery. Seamless synthesis of AR depends on a number of factors relating to the way in which virtual objects appear and visually interact with a real environment. Traditional overlaid AR approaches generally suffer from a loss of depth perception. This paper presents a new AR method for robotic assisted MIS, which uses a novel pq-space based non-photorealistic rendering technique for providing see-through vision of the embedded virtual object whilst maintaining salient anatomical details of the exposed anatomical surface. Experimental results with both phantom and in vivo lung lobectomy data demonstrate the visual realism achieved for the proposed method and its accuracy in providing high fidelity AR depth perception.
---
paper_title: Virtual Eyes Can Rearrange Your Body: Adaptation to Visual Displacement in See-Through, Head-Mounted Displays

paper_content:

Among the most critical issues in the design of immersive virtual environments are those that deal with the problem of technologically induced intersensory conflict and one of the results, sensorimotor adaptation. An experiment was conducted to support the design of a prototype see-through, head-mounted display (HMD). When wearing video see-through HMDs in augmented reality systems, subjects see the world around them through a pair of head-mounted video cameras. The study looked at the effects of sensory rearrangement caused by a HMD design that displaced the user's “virtual” eye position forward (165 mm) and above (62 mm) toward the spatial position of the cameras. The position of the cameras creates images of the world that are slightly downward and inward from normal. Measures of hand-eye coordination and speed on a manual pegboard task revealed substantial perceptual costs of the eye displacement initially, but also evidence of adaptation. Upon first wearing the video see-through HMD, subjects' pointing errors increased significantly along the spatial dimensions displaced (the y dimension, above-below the target, and z dimension, in front-behind the target). Speed of performance on the pegboard task decreased by 43% compared to baseline performance. Pointing accuracy improved by approximately 33% as subjects adapted to the sensory rearrangement, but it did not reach baseline performance. When subjects removed the see-through HMD, there was evidence that their hand-eye coordination had been altered. Negative aftereffects were observed in the form of greater errors in pointing accuracy compared to baseline. Although these aftereffects are temporary, the results may have serious practical implications for the use of video see-through HMDs by users (e.g., surgeons) who depend on very accurate hand-eye coordination.
---
paper_title: Augmented-reality visualizations guided by cognition: perceptual heuristics for combining visible and obscured information

paper_content:

One unique feature of mixed and augmented reality (MR/AR) systems is that hidden and occluded objects an be readily visualized. We call this specialized use of MR/AR, obscured information visualization (OIV). In this paper, we describe the beginning of a research program designed to develop such visualizations through the use of principles derived from perceptual psychology and cognitive science. In this paper we surveyed the cognitive science literature as it applies to such visualization tasks, described experimental questions derived from these cognitive principles, and generated general guidelines that can be used in designing future OIV systems (as well improving AR displays more generally). We also report the results from an experiment that utilized a functioning AR-OIV system: we found that in relative depth judgment, subjects reported rendered objects as being in front of real-world objects, except when additional occlusion and motion cues were presented together.
---
paper_title: Resolving multiple occluded layers in augmented reality

paper_content:

A useful function of augmented reality (AR) systems is their ability to visualize occluded infrastructure directly in a user's view of the environment. This is especially important for our application context, which utilizes mobile AR for navigation and other operations in an urban environment. A key problem in the AR field is how to best depict occluded objects in such a way that the viewer can correctly infer the depth relationships between different physical and virtual objects. Showing a single occluded object with no depth context presents an ambiguous picture to the user. But showing all occluded objects in the environments leads to the "Superman's X-ray vision" problem, in which the user sees too much information to make sense of the depth relationships of objects. Our efforts differ qualitatively from previous work in AR occlusion, because our application domain involves far-field occluded objects, which are tens of meters distant from the user. Previous work has focused on near-field occluded objects, which are within or just beyond arm's reach, and which use different perceptual cues. We designed and evaluated a number of sets of display attributes. We then conducted a user study to determine which representations best express occlusion relationships among far-field objects. We identify a drawing style and opacity settings that enable the user to accurately interpret three layers of occluded objects, even in the absence of perspective constraints.
---
paper_title: Direct volume rendering with shading via three-dimensional textures

paper_content:

A new and easy-to-implement method for direct volume rendering that uses 3D texture maps for acceleration, and incorporates directional lighting, is described. The implementation, called Voltx, produces high-quality images at nearly interactive speeds on workstations with hardware support for three-dimensional texture maps. Previously reported methods did not incorporate a light model, and did not address issues of multiple texture maps for large volumes. Our research shows that these extensions impact performance by about a factor of ten. Voltx supports orthographic, perspective, and stereo views. This paper describes the theory and implementation of this technique, and compares it to the shear-warp factorization approach. A rectilinear data set is converted into a three-dimensional texture map containing color and opacity information. Quantized normal vectors and a lookup table provide efficiency. A new tesselation of the sphere is described, which serves as the basis for normal-vector quantization. A new gradient-based shading criterion is described, in which the gradient magnitude is interpreted in the context of the field-data value and the material classification parameters, and not in isolation. In the rendering phase, the texture map is applied to a stack of parallel planes, which effectively cut the texture into many slabs. The slabs are composited to form an image.
---
paper_title: Hybrid navigation interface for orthopedic and trauma surgery

paper_content:

Several visualization methods for intraoperative navigation systems were proposed in the past. In standard slice based navigation, three dimensional imaging data is visualized on a two dimensional user interface in the surgery room. Another technology is the in-situ visualization i.e. the superimposition of imaging data directly into the view of the surgeon, spatially registered with the patient. Thus, the three dimensional information is represented on a three dimensional interface. We created a hybrid navigation interface combining an augmented reality visualization system, which is based on a stereoscopic head mounted display, with a standard two dimensional navigation interface. Using an experimental setup, trauma surgeons performed a drilling task using the standard slice based navigation system, different visualization modes of an augmented reality system, and the combination of both. The integration of a standard slice based navigation interface into an augmented reality visualization overcomes the shortcomings of both systems.
---
paper_title: Marching cubes: A high resolution 3D surface construction algorithm

paper_content:

We present a new algorithm, called marching cubes, that creates triangle models of constant density surfaces from 3D medical data. Using a divide-and-conquer approach to generate inter-slice connectivity, we create a case table that defines triangle topology. The algorithm processes the 3D medical data in scan-line order and calculates triangle vertices using linear interpolation. We find the gradient of the original data, normalize it, and use it as a basis for shading the models. The detail in images produced from the generated surface models is the result of maintaining the inter-slice connectivity, surface data, and gradient information present in the original 3D data. Results from computed tomography (CT), magnetic resonance (MR), and single-photon emission computed tomography (SPECT) illustrate the quality and functionality of marching cubes. We also discuss improvements that decrease processing time and add solid modeling capabilities.
---
paper_title: Acceleration techniques for GPU-based volume rendering

paper_content:

Nowadays, direct volume rendering via 3D textures has positioned itself as an efficient tool for the display and visual analysis of volumetric scalar fields. It is commonly accepted, that for reasonably sized data sets appropriate quality at interactive rates can be achieved by means of this technique. However, despite these benefits one important issue has received little attention throughout the ongoing discussion of texture based volume rendering: the integration of acceleration techniques to reduce per-fragment operations. In this paper, we address the integration of early ray termination and empty-space skipping into texture based volume rendering on graphical processing units (GPU). Therefore, we describe volume ray-casting on programmable graphics hardware as an alternative to object-order approaches. We exploit the early z-test to terminate fragment processing once sufficient opacity has been accumulated, and to skip empty space along the rays of sight. We demonstrate performance gains up to a factor of 3 for typical renditions of volumetric data sets on the ATI 9700 graphics card.
---
paper_title: Recovery of surgical workflow without explicit models

paper_content:

Workflow recovery is crucial for designing context-sensitive service systems in future operating rooms. Abstract knowledge about actions which are being performed is particularly valuable in the OR. This knowledge can be used for many applications such as optimizing the workflow, recovering average workflows for guiding and evaluating training surgeons, automatic report generation and ultimately for monitoring in a context aware operating room. ::: ::: This paper describes a novel way for automatic recovery of the surgical workflow. Our algorithms perform this task without an implicit or explicit model of the surgery. This is achieved by the synchronization of multidimensional state vectors of signals recorded in different operations of the same type. We use an enhanced version of the dynamic time warp algorithm to calculate the temporal registration. The algorithms have been tested on 17 signals of six different surgeries of the same type. The results on this dataset are very promising because the algorithms register the steps in the surgery correctly up to seconds, which is our sampling rate. Our software visualizes the temporal registration by displaying the videos of different surgeries of the same type with varying duration precisely synchronized to each other. The synchronized videos of one surgery are either slowed down or speeded up in order to show the same steps as the ones presented in the videos of the other surgery.
---
paper_title: Liver Surgery Planning Using Virtual Reality

paper_content:

We have developed LiverPlanner, a virtual liver surgery planning system that uses high-level image analysis algorithms and virtual reality technology to help physicians find the best resection plan for each individual patient. Preliminary user studies of LiverPlanner show that the proposed tools are well accepted by doctors and lead to much shorter planning times
---
paper_title: Laparoscopic Virtual Mirror New Interaction Paradigm for Monitor Based Augmented Reality

paper_content:

A major roadblock for using augmented reality in many medical and industrial applications is the fact that the user cannot take full advantage of the 3D virtual data. This usually requires the user to move the virtual object, which disturbs the real/virtual alignment, or to move his head around the real objects, which is not always possible and/or practical. This problem becomes more dramatic when a single camera is used for monitor based augmentation, such as in augmented laparoscopic surgery. In this paper we introduce an interaction and 3D visualization paradigm, which presents a new solution to this old problem. The interaction paradigm uses an interactive virtual mirror positioned into the augmented scene, which allows easy and complete interactive visualization of 3D virtual data. This paper focuses on the exemplary application of such visualization techniques to laparoscopic interventions. A large number of such interventions aims at regions inside a specific organ, e.g. blood vessels to be clipped for tumor resection. We use high-resolution intra-operative imaging data generated by a mobile C-arm with cone-beam CT imaging capability. Both the C-arm and the laparoscope are optically tracked and registered in a common world coordinate frame. After patient positioning, port placement, and carbon dioxide insufflation, a C-arm volume is reconstructed during patient exhalation and superimposed in real time on the laparoscopic live video without any need for an additional patient registration procedure. To overcome the missing perception of 3D depth and shape when rendering virtual volume data directly on top of the organ's surface view, we introduce the concept of a laparoscopic virtual mirror: A virtual reflection plane within the live laparoscopic video, which is able to visualize a reflected side view of the organ and its interior. This enables the surgeon to observe the 3D structure of, for example, blood vessels by moving the virtual mirror within the augmented monocular view of the laparoscope.
---
paper_title: pq-space Based Non-Photorealistic Rendering for Augmented Reality

paper_content:

The increasing use of robotic assisted minimally invasive surgery (MIS) provides an ideal environment for using Augmented Reality (AR) for performing image guided surgery. Seamless synthesis of AR depends on a number of factors relating to the way in which virtual objects appear and visually interact with a real environment. Traditional overlaid AR approaches generally suffer from a loss of depth perception. This paper presents a new AR method for robotic assisted MIS, which uses a novel pq-space based non-photorealistic rendering technique for providing see-through vision of the embedded virtual object whilst maintaining salient anatomical details of the exposed anatomical surface. Experimental results with both phantom and in vivo lung lobectomy data demonstrate the visual realism achieved for the proposed method and its accuracy in providing high fidelity AR depth perception.
---
paper_title: Interactive Focus and Context Visualization for Augmented Reality

paper_content:

In this article we present interactive focus and context (F+C) visualizations for augmented reality (AR) applications. We demonstrate how F+C visualizations are used to affect the user's perception of hidden objects by presenting contextual information in the area of augmentation. We carefully overlay synthetic data on top of the real world imagery by taking into account the information that is about to be occluded. Furthermore, we present operations to control the amount of augmented information. Additionally, we developed an interaction tool, based on the magic lens technique, which allows for interactive separation of focus from context. We integrated our work into a rendering framework developed on top of the Studierstube augmented reality system. We finally show examples to demonstrate how our work benefits AR.
---
paper_title: Laparoscopic Virtual Mirror New Interaction Paradigm for Monitor Based Augmented Reality

paper_content:

A major roadblock for using augmented reality in many medical and industrial applications is the fact that the user cannot take full advantage of the 3D virtual data. This usually requires the user to move the virtual object, which disturbs the real/virtual alignment, or to move his head around the real objects, which is not always possible and/or practical. This problem becomes more dramatic when a single camera is used for monitor based augmentation, such as in augmented laparoscopic surgery. In this paper we introduce an interaction and 3D visualization paradigm, which presents a new solution to this old problem. The interaction paradigm uses an interactive virtual mirror positioned into the augmented scene, which allows easy and complete interactive visualization of 3D virtual data. This paper focuses on the exemplary application of such visualization techniques to laparoscopic interventions. A large number of such interventions aims at regions inside a specific organ, e.g. blood vessels to be clipped for tumor resection. We use high-resolution intra-operative imaging data generated by a mobile C-arm with cone-beam CT imaging capability. Both the C-arm and the laparoscope are optically tracked and registered in a common world coordinate frame. After patient positioning, port placement, and carbon dioxide insufflation, a C-arm volume is reconstructed during patient exhalation and superimposed in real time on the laparoscopic live video without any need for an additional patient registration procedure. To overcome the missing perception of 3D depth and shape when rendering virtual volume data directly on top of the organ's surface view, we introduce the concept of a laparoscopic virtual mirror: A virtual reflection plane within the live laparoscopic video, which is able to visualize a reflected side view of the organ and its interior. This enables the surgeon to observe the 3D structure of, for example, blood vessels by moving the virtual mirror within the augmented monocular view of the laparoscope.
---

Title: Advanced Medical Displays: A Literature Review of Augmented Reality

Section 1: Introduction
Description 1: This section provides an overview of the evolution and development of augmented reality in the medical field, including historical milestones and key technological advancements.

Section 2: Overview of Medical AR Systems and Technologies
Description 2: This section introduces different categories of medical augmented reality systems and technologies, highlighting their limitations and advantages.

Section 3: HMD Based AR System
Description 3: This section discusses head-mounted display (HMD)-based AR systems, including both optical and video see-through designs, their tracking technologies, and applications in medicine.

Section 4: Augmented Optics
Description 4: This section covers augmented optics including operating microscopes and binoculars, focusing on their use in brain surgery, neurosurgical interventions, and other medical procedures.

Section 5: AR Windows
Description 5: This section describes the concept of AR windows, involving semi-transparent mirrors and autostereoscopic screens for augmenting medical images without the need for head-worn displays.

Section 6: Augmented Monitors
Description 6: This section explains how video images can be augmented on ordinary monitors, defining the point of view with tracked video cameras and applying this system in medical settings.

Section 7: Augmented Endoscopes
Description 7: This section explores the integration of augmented reality techniques in endoscopic procedures, addressing calibration, tracking, and visualization challenges.

Section 8: Augmented Medical Imaging Devices
Description 8: This section outlines devices that augment images without a tracking system, including fluoroscopic overlays and tomographic reflection systems, for enhancing visualization in medical imaging.

Section 9: Projections on the Patient
Description 9: This section discusses systems that project augmented data directly onto the patient, detailing the benefits and limitations of such approaches for medical applications.

Section 10: Potential Benefits of AR Visualization
Description 10: This section evaluates the advantages of augmented reality in medical visualization, including image fusion, 3D interaction, stereoscopic visualization, and improved hand-eye coordination.

Section 11: Current Issues
Description 11: This section highlights the current limitations and challenges faced by medical AR systems, such as registration, tracking, calibration, time synchronization, error estimation, and depth perception.

Section 12: Perspective
Description 12: This section provides insights into the future prospects of medical augmented reality, emphasizing the need for seamless integration into clinical workflows and advancements in user interaction and depth perception.