Integrazione di building information modeling e realtà aumentata per migliorare l’indagine di edifici storici
10.6092/issn.1973-9494/6174
Questa pubblicazione descrive un sistema sperimentale per supportare l’indagine di edifici storici utilizzando Building Information Modeling (BIM) e Realtà Aumentata (AR). Il sistema richiede l’utilizzo di software off-line per realizzare la rappresentazione BIM e definisce un metodo per integrare dati diagnostici in BIM. Il sistema offre accesso a tali dati durante l’indagine del sito usando occhiali AR supportati dalle tecnologie marker e marker-less. L’innovazione principale del sistema è la possibilità di contestualizzare attraverso AR non solo le proprietà BIM esistenti ma anche dati da strumenti non-invasivi. Valutazioni utente mostrano come l’utilizzo del sistema possa migliorare la percezione degli ingegneri durante il processo di indagine.
application/pdf
1973-9494
6174
en
Lucio Colizzi
Conservation Science in Cultural Heritage
computer vision
Integrating Building Information Modeling and Augmented Reality to Improve Investigation of Historical Buildings
Text.Serial.Journal
https://conservation-science.unibo.it/
15
Articles
2015-12-30
2016-05-30
2015-12-30
2017-09-04
133-163
CSCH
Lucio Colizzi
2015/12/30
10.6092/issn.1973-9494/6174
133
Centro di progettazione, design e tecnologie dei materiali (CETMA), Brindisi
1973-9494
computer vision
en
163
Henry, P., Krainin, M., Herbst, E., Ren, X., & Fox, D. (2012). RGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments. The International Journal of Robotics Research, 31(5), 647-663.
Integrating Building Information Modeling and Augmented Reality to Improve Investigation of Historical Buildings
https://conservation-science.unibo.it/article/download/6174/5943
15
I
 NTEGRATING BUILDING INFORMATION MODELING AND
 AUGMENTED REALITY TO IMPROVE INVESTIGATION OF
HISTORICAL BUILDINGS

    Francesco Chionna, Francesco Argese,
    Vito Palmieri, Italo Spada, Lucio Colizzi*
    Centro di progettazione, design e tecnologie dei materiali (CETMA), Brindisi, Italy


   Keywords: �������������������������������������������������������������������
             information modeling, augmented reality, software engineering, com-
puter vision


    1. Introduction

Nowadays conservation of historical buildings requires structural engineering studies
during the evaluation and preservation process [1].This process, moreover, needs a
preliminary investigation of the structural condition, involving the use of non-invasi-
ve and non-destructive technologies, such as thermography, stratigraphy and tomo-
graphy. During the first phase of the conservation process (known as the Knowledge
phase), a structural engineer acquires data on the site and then analyzes them in his
office without having a contextualized view of the entire problem.
In recent years, new technologies have emerged, allowing the development of new
tools, which help to simplify access to information about the historical building. In parti-
cular, developments in the field of Building Information Modeling (BIM) and Augmented
Reality (AR), have opened new perspectives that have improved the typical investi-




                                                                                                      CONSERVATION SCIENCE IN CULTURAL HERITAGE
gation process [2]: BIM offers a digital representation of the physical and functional
characteristics of a facility [3], while AR offers tools to view a physical, real-world envi-
ronment augmented by computer-generated information [4].
In this scenario, this paper presents an experimental study which describes the pro-
cess developed to build a tool aimed at improving the investigation of historical buil-
dings. The study includes:
      1. BIM information based on IFC (Industry Foundation Classes) file format;
      2. AR visualization of BIM information using tablet and 3D glasses, contextuali-
           zed to the scenario that the user is watching.
      Most of the work in this experimental research addressed:
      ● retrieving and developing BIM information related to the historical building by
           integrating data from non-invasive tools;
      ● evaluating AR as a visualization tool to display BIM information, contextuali-
           zed to the building scenario the user is examining.
The results were tested on the case study “Monastero Fortezza di Santo Spirito”, a
fortified monastery, located in Ocre in the province of L’Aquila, Italy.




* Corresponding author: lucio.colizzi@cetma.it


                                                                                                133
                                                                                                           2. Related Studies

                                                                                                           Other works have been proposed on the use of BIM and AR to improve investigation
                                                                                                        of buildings, each of them dealing with a specific aspect of the problem. In the following
                                                                                                        subsections, there is a brief analysis of previous work on each aspect.

                                                                                                           a. BIM

                                                                                                           Building Information Modeling (BIM) is an emerging technology in the Architecture/
                                                                                                        Engineering/Construction (AEC) industry. BIM is a digital representation of the physi-
F. Chionna, F. Argese, V. Palmieri, I. Spada, L. Colizzi - Integrating building information modeling




                                                                                                        cal and functional characteristics of a building, which can serve as a shared knowledge
                                                                                                        resource for building information. Based on semantics and object oriented techniques,
                                                                                                        the BIM 3D modeling approach facilitates access to data about the building, including the
                                                                                                        objects with their features, used in the design, construction, and operation phases [3].
                                                                                                        In addition, BIM’s parametric design capability enables quick, interactive, and real-time
                                                                                                        design changes. BIM was not designed for applications in historical buildings, but a num-
                                                                                                        ber of researchers have investigated how BIM itself, or its extensions, could be useful to
                                                                                                        this field. In particular, research conducted by Oreni et al. [5] illustrates the usefulness of
                                                                                                        switching from 3D content modelling to Historic Building Information Modelling (HBIM),
                                                                                                        to support the conservation and management of building heritage by proposing a library
                                                                                                        for the analysis of vaults and wooden beam flooring. Another study by Pauwels et al. [6]
                                                                                                        investigates the advantages of an alternative approach to BIM, called AIM (Architectural
                                                                                                        Information Modelling), for applications in building documentation for virtual heritage,
                                                                                                        combined with historical analysis tools.

                                                                                                           b. AR visualization

                                                                                                           Augmented Reality is a field of research that aims to create an environment where
                                                                                                        computer generated data are inserted into the user’s view of a real-world scene. It
                                                                                                        plays a key role in improving and enriching the visualization of objects contextualized
                                                                                                        in a specific place. In literature, there are many different technologies based on differ-
                                                                                                        ent principles to implement tracking of Augmented Reality, but in this paper we will cite
                                                                                                        only Computer Vision, owing to its use of non-invasive tools.
                                                                                                           Steps in visualizing 3D objects in Augmented Reality using Computer Vision in-
                                                                                                        clude:
                                                                                                           1. recognition of a marker (a.k.a. marker-based approach) or of an object in the
                                                                                                                scene (a.k.a. marker-less approach);
                                                                                                           2. context aware methods for visualization in real-time of video and augmented
                                                                                                                contents that enable them to be used in a mobile setting.
                                                                                                           Many studies conducted on the two topics have been well summarized in a recent
                                                                                                        survey conducted by Shoaib [7]. Both marker-based and marker-less approaches have
                                                                                                        been proposed in many commercial ready-to-use solutions [8]; however, a marker-less
                                                                                                        approach is not always possible because it requires specific lighting conditions and the
                                                                                                        availability of identifiable features in the scenario.
                                                                                                           AR technology has developed in line with mobile technologies and often takes the
                                                                                                        form of application software that can be displayed on smartphones, tablets or even
                                                                                                        innovative AR glasses. Despite this, visualization in real-time can be a problem when
                                                                                                        using devices with low performance if it involves complex 3D models.


                                                                                                  134
  The most popular topics regarding AR visualization are interaction and Mobile Aug-
mented Reality.
  Feiner and Hollerer [4] have identified six basic technological AR components:
   – Computational platform for processing and computing all relevant informa-
        tion needed to visualize objects in AR on the display.
   – Display to present virtual objects to the user.
   – Registration of environment. Registration of camera input and head orien-
        tation helps to align AR objects correctly with the real world.
   – Wearable input and interaction technologies to enable user mobility and
        collaboration with other users.
   – Wireless networking for instant communication with other people and cen-
        tral databases.
   – Data storage and access technology to provide the user with all context
        relevant data in the environment intended for augmentation.

   c. BIM and AR integration

     Both BIM and AR integration have been investigated in many research studies related
to facility management and tools for the AEC (Architecture Engineering Construction) in-
dustry. Koch et al. [10] have proposed a conceptual framework to support facility mainte-
nance operators in performing daily on-site jobs combining Building Information Models
and natural markers for AR-based maintenance support. Irizarry et al. [11] have proposed
InfoSPOT, a mobile Augmented Reality (AR) tool for accessing information about the fa-
cilities where they manage maintenance. Woodman et al. [12] have proposed a software
system for mobile mixed reality interaction with complex BIM models. Gheisari et al. [13]
have explored the advantages of an approach using BIM and Mobile Augmented Reality
in Facilities Management by conducting a survey among facility managers to acquire
feedback. Hakkarainent et al [14] have proposed software architecture to provide mobile




                                                                                                      CONSERVATION SCIENCE IN CULTURAL HERITAGE
users with real-time access to CAD and BIM information by using augmented reality at
the construction site. El Ammari has proposed a framework for collaborative BIM-based
marker-less Mixed Reality (BIM3R), integrating CMMS (Computerized Maintenance
Management Systems), BIM and video-based tracking in a BIM3R setting, to retrieve
information based on user time and location, visualize maintenance operations, and sup-
port collaboration between the field and the office to enhance decision-making.


   d. Our contribution

   Many studies have been conducted on BIM applied to historical buildings, on AR and
on the integration of BIM and AR technologies for facilities management and there is no
doubt about their usefulness. To our knowledge, there is no previously published work
applying these technologies to improve historical building investigation. It is therefore an
area revealing a field that presents new challenges, due to the limited accessibility and
use of non-invasive tools for analysis. This situation could greatly benefit from the avail-
ability of new tools, which would integrate and offer a contextualized view of analytical re-
sults using non-invasive tools directly on-site. The system addresses the following issues:
   1. explore the problems of this approach when applied to historical buildings, and
         propose solutions;
   2. compare this approach with the traditional one and evaluate advantages.

                                                                                                135
                                                                                                           3. System overview

                                                                                                           The main aim of the system is to create a tool to improve the investigation of histori-
                                                                                                        cal buildings. The system has been developed in three main steps:
                                                                                                           1. Integration of diagnostic data into BIM representation of the historical building;
                                                                                                           2. Development of a software module to read BIM data;
                                                                                                           3. Development of an Augmented Reality module to detect features of specific
                                                                                                                building parts or markers inside the historical building and augment visualiza-
                                                                                                                tion using BIM information.
                                                                                                           The project includes a device to display real-time augmented viewing during the
F. Chionna, F. Argese, V. Palmieri, I. Spada, L. Colizzi - Integrating building information modeling




                                                                                                        investigation, which should be used by the engineer or the architect during the indoor
                                                                                                        on-site analysis of the historical building.
                                                                                                           Figure 1 represents the logical architecture of the complete system. This paper
                                                                                                        describes only the modules relating to the BIM building representation, diagnostic data
                                                                                                        integration into BIM and the adopted Augmented Reality technology. A description of
                                                                                                        each module developed for this research work is given in the following sub-sections.




                                                                                                        Figure 1. Logic architecture of the system

                                                                                                           a. Diagnostic data integration into BIM

                                                                                                            Integration of diagnostic data into BIM is crucial for the described system because
                                                                                                        it allows information to be added to each section, preserving the semantic value of
                                                                                                        the BIM representation. During the development phase of this module, our focus was
                                                                                                        mainly on merging different kinds of information by simplifying access for the end-user.
                                                                                                        To achieve this goal, a customized workflow was adopted with three main phases: BIM
                                                                                                        building representation, diagnostic data integration into BIM and the implementation of
                                                                                                        a parser to read BIM data. In the following sub-sections, there is a description of each
                                                                                                        phase.

                                                                                                           BIM building representation

                                                                                                           In this step, a 3D representation of the historical building was built using the com-
                                                                                                        mercial software Archicad. The result was a BIM representation of the building includ-
                                                                                                        ing many parts with semantic references. The software offers many exporting formats,
                                                                                                        among which there are proprietary and non-proprietary ones. One of the most adopted


                                                                                                  136
non-proprietary formats is the IFC (Industry Foundation Classes [3], an open standard
data model developed by the International Alliance for Interoperability (IAI).
    The choice of IFC data model allows the building geometry and materials property
information to be exported using a BIM authoring tool; being an open standard, it can
also be extended with new features to add new information if required. More specifically,
the IFC-SPF was adopted for the BIM representation: this is a text format defined by ISO
10303-21 (“STEP-File”) having file extension “.ifc”. The basic SPF structure divides each
file into a header and a data section. The header section has information about:
    •     the IFC version used;
    •     the application that exported the file;
    •     the date and time the file was exported;
    •     the name, company and authorizing person that created the file (optional).
    •     In the following text there is an example of a header of an IFC file.

   HEADER;
   FILE_DESCRIPTION((‘IFC 2x platform’),’2;1’);
   FILE_NAME(
   ‘Example.ifc’,
   ‘2005-09-02T14:48:42’,
   (‘The User’),
   (‘The Company’),
   ‘The name and version of the IFC preprocessor’,
   ‘The name of the originating software system’,
   ‘The name of the authorizing person’);
   FILE_SCHEMA((‘IFC2X2_FINAL’));
   ENDSEC;

  The data section contains all instances for the entities of the IFC specification.




                                                                                                    CONSERVATION SCIENCE IN CULTURAL HERITAGE
These instances have a unique (within the scope of a file) STEP Id, the entity type
name and a list of explicit attributes. In the following text there is an example of a data
section of an IFC file.

   DATA;
   #4= IFCPERSON($,’Undefined’,$,$,$,$,$,$);
   #6= IFCORGANIZATION($,’Undefined’,$,$,$);
   #10= IFCPERSONANDORGANIZATION(#4,#6,$);
   #13= IFCORGANIZATION(‘GS’,’Graphisoft’,’Graphisoft’,$,$);
   #14= IFCAPPLICATION(#13,’18.0.0’,’ArchiCAD-64’,
   ‘IFC2x3 add-on version: 3006 ITA FULL’);
   #15= IFCOWNERHISTORY(#10,#14,$,.ADDED.,$,$,$,1425638738);
   #16= IFCSIUNIT(*,.LENGTHUNIT.,.MILLI.,.METRE.);
   #17= IFCSIUNIT(*,.AREAUNIT.,$,.SQUARE_METRE.);
   #18= IFCSIUNIT(*,.VOLUMEUNIT.,$,.CUBIC_METRE.);

   Diagnostic data integration into BIM

   The integration of diagnostic data into BIM requires the use of a customized ap-
proach because, at present, the adopted software does not support this feature na-


                                                                                              137
                                                                                                        tively. As previously anticipated, the use of IFC offers the possibility of adding new
                                                                                                        fields to extend the file format with other features. For each part of the representation,
                                                                                                        studies carried out with non-invasive tools were added using a customized field of the
                                                                                                        BIM. Figure 2 shows an image of thermal camera analysis (left) and the layout of the
                                                                                                        building with the indication (right) of the location related to the acquired data.
F. Chionna, F. Argese, V. Palmieri, I. Spada, L. Colizzi - Integrating building information modeling




                                                                                                        Figure 2. Thermal image and building layout

                                                                                                           To extend the primary data structure with new information, IfcProperty and IfcProp-
                                                                                                        ertySet custom properties were used. Table 1 shows an example related to the defini-
                                                                                                        tion of a set of properties for Infrared thermal imaging of the historical building.

                                                                                                        Table 1. Set of properties for thermography
                                                                                                                  Investigation                              Thermography
                                                                                                             Name of property set                        Thermography building
                                                                                                                                                            Property name
                                                                                                                                                                filename
                                                                                                                                                                  note
                                                                                                                                                                 author
                                                                                                                                                                  date
                                                                                                                                                                  scale
                                                                                                                                                             camera model
                                                                                                                                                            spatial resolution
                                                                                                                                                           Thermal resolution
                                                                                                                                                                accuracy
                                                                                                                                                             Spectral range
                                                                                                                                                                   FPA
                                                                                                                                                           Sensor resolution
                                                                                                                                                            Measuring range




                                                                                                  138
   IFC parser implementation

   In order to implement integration of the resulting ifc file with the rest of the system,
a C++ ifc library was adopted. Among the various commercial and open source tools
available for working with IFC data [17] our choice was IfcPlusPlus, an open source
C++ library that offers classes to simplify reading and writing of IFC files in STEP
format. The main features of the library are, availability of easy and efficient memory
management tools; the availability of a parallel reader for rapid parsing on multi-core
GPUs (Graphics Processing Unit) and integration with the rendering engine Open-
SceneGraph [15] for visualization purposes.

   b. Software architecture

   As shown in Figure 3, the system is modular to allow reuse of components and
includes four main modules:
     – The BIM module software, which is responsible for handling and accessing
         BIM data including the geometric information of the model and diagnostic data.
     – The ArModule software, which is used for object recognition and tracking.
         It is based on marker and marker-less algorithms depending on the features
         of the reference scenario. The tracking algorithms were implemented using
         the OpenCV library [16] while visualization was implemented using the Open-
         SceneGraph library [15];
     – The GUI Module, which provides a user interface to browse and visualize
         data, handling user interaction through different types of widgets;
     – The DB Module, which is responsible for handling and accessing user profile
         data and was developed using the Soci library [18].




                                                                                                    CONSERVATION SCIENCE IN CULTURAL HERITAGE




Figure 3. Software architecture

   c. AR visualization

   Hardware

   In order to develop the ArModule, a pair of 3D glasses and a pc tablet were used,
as shown in Figure 4. The following sub-sections provide an overview of each device,
and explain why they were chosen.
   Vuzix Wrap™ 920AR: The Vuzix Wrap 920AR eyewear is composed of two parts:


                                                                                              139
                                                                                                            –    a stereo camera system to capture the real world;
                                                                                                            –    a display that shows the captured scene to the user with the visual equivalent
                                                                                                                 of a 67-inch display seen from a distance of 10 feet (3m).
                                                                                                          The twin camera system of the Wrap 920AR provides two video sources, each con-
                                                                                                        nected to a PC as a standard USB video camera device. Each camera captures 640 x
                                                                                                        480 VGA video at a speed of 30 frames per second.
                                                                                                          The display capabilities of the Wrap 920AR can be used on virtually any composite
                                                                                                        compatible device. In order to use it in an augmented reality system, it can be connect-
                                                                                                        ed to a Windows-based personal computer through the included Wrap VGA Adapter.
                                                                                                        The image produced by the eyewear, which can be displayed in standard 2D or 3D,
F. Chionna, F. Argese, V. Palmieri, I. Spada, L. Colizzi - Integrating building information modeling




                                                                                                        can be augmented with any form of computer-generated data in the form of text, still
                                                                                                        images, or video.

                                                                                                            Asus Eee Slate EP121: The Asus Eee Slate is a device with high performance
                                                                                                        and portability, equipped with an Intel ® Core i5 ™, a large ultrasensitive touchscreen
                                                                                                        Wide Screen (12.1”, 1280 x 800 pixel), which can be used freehand or with an input-
                                                                                                        precision stylus. It runs Windows 7 Home Premium 64 bit, weighs 2.5 lbs and its bat-
                                                                                                        tery life is up to 3.5 hours.
                                                                                                            The device uses SSD memory for storage instead of a traditional mechanical hard
                                                                                                        drive, which is useful to speed up performance of developed software, thus ensuring
                                                                                                        its use in real time. The device can be connected to the chosen eyewear by means of
                                                                                                        two USB ports, Atheros WiFi 802.11b/g/n and a mini-HDMI port.
                                                                                                            These features make this device suitable for supporting video analysis during explo-
                                                                                                        ration campaigns. Coupled with Vuzix Wrap 920AR eyewear via usb, it can guarantee
                                                                                                        the direct acquisition and real-time processing of the scene.
                                                                                                            As an alternative to using the stylus to control the PC tablet, we experimented the
                                                                                                        use of a mobile phone as a mouse replacement. In particular, our experimental system
                                                                                                        uses Bluetooth communication to control the PC tablet through the smartphone touch
                                                                                                        screen, allowing the user to select a particular region of interest (ROI) on the screen in
                                                                                                        order to process only a specific area of the scene.




                                                                                                        Figure 4. Devices used for experimental system

                                                                                                           Software methods and technologies for tracking

                                                                                                           In visual tracking, the system determines the pose estimation based on direct ob-
                                                                                                        servations of the scene. In an environment without robust features it is impossible
                                                                                                        to recognize a predefined pattern model based solely on visual observations of the
                                                                                                        scene. One solution to overcome these challenges is to add an easily detectable pre-


                                                                                                  140
defined image inside the environment and use computer vision techniques to identify
it. This ‘marker’ is a sign or image a computer system can detect in a video using im-
age processing, pattern recognition and computer vision techniques. Once detected, it
defines the pose estimation: this approach is called marker-based tracking.
    Another approach for visual tracking is to use feature-based methods, in which there
is a model of the scene or part of the scene (e.g. an image pattern). It compares visual
observations with the model and finds the best match, which defines the pose estimation.
In feature-based tracking, the system detects optical features exclusive to the images and
acquires information about the environment by observing movements between frames.
    In our ArModule, feature-tracking and marker-based tracking are mutually non-ex-
clusive. In fact, marker-based methods outperform feature-based methods in certain
cases (e.g. environments with large white walls, have almost no features and there-
fore, feature tracking is impossible or at the least, very unreliable), while the marker-
less approach is used for visual tracking in recognizable environments.
    The Augmented Reality module was developed following a processing pipeline,
consisting of the following stages:


       Capture   Preprocessing    Detection     Recognition    Tracking      Rendering


Figura 5. Stages of AR module development

   •     Capture: in the first step the capture device acquires the scene in real-time; in
         this experimental system the Vuzix Wrap 920AR was used;
    •    preprocessing: all operations in this stage aim at improving image data, remov-
         ing undesired distortions and/or enhancing certain image features, needed for
         subsequent processing steps;
    •    detection: in this stage, the pipeline applies both a feature detection and an




                                                                                                    CONSERVATION SCIENCE IN CULTURAL HERITAGE
         extraction algorithm;
    •    recognition: once the features are generated by the detection stage, a matching
         algorithm is used to match the features against a database of known patterns;
    •    tracking: this stage uses tracking-by-detection algorithms, which allows track-
         ing of known patterns through a detection analysis on sequential frames;
    •    rendering: the data collected in the previous steps are used to augment the
         scene by adding information about the object of analysis identified in the scene.
         For example, BIM property information could be added to the augmented scene
         when a fresco is identified.
    The recognition and tracking steps also include a pose estimation, which allows re-
trieval of the spatial relationship between the user’s point of view and the virtual object
by means of identification, in the scene, of common features (points , lines, spots) and,
through knowledge of these features (appearance, size, position in 3D space). The
result is a qualitative measure of 6DoF (Degree of Freedom) of the object in the actual
scene, which is crucial during the rendering step, to display the position and orientation
of the augmented content in a consistent way.
    The structure of the software module, which handles the pose estimation, can be
conceptually divided into two stages: learning phase and detection phase. Prior to the
first phase, a camera calibration step is required to avoid issues related to image dis-
tortions resulting from the optics used. The first phase extracts and identifies the scene


                                                                                              141
                                                                                                        storing the descriptors of its structural elements. In the second phase, all the stored
                                                                                                        elements are used to perform a matching with the elements identified in the scene; at
                                                                                                        the end, the pose of 3D object reference is estimated. Figure 6 illustrates a flowchart
                                                                                                        representing these two phases.
F. Chionna, F. Argese, V. Palmieri, I. Spada, L. Colizzi - Integrating building information modeling




                                                                                                        Figure 6. Flowchart of learning and detection phases

                                                                                                           In order to address research and development towards algorithms for key-point de-
                                                                                                        tection, features description and key-point matching, different vision-based techniques
                                                                                                        were tested.
                                                                                                           In particular, the following algorithms were taken into consideration:
                                                                                                           Speeded-up Robust Features (SURF): SURF was presented by Bay et al [19]. It
                                                                                                           is a local feature detector and descriptor that uses an integer approximation of the
                                                                                                           determinant of Hessian blob detector to detect interest points. Its feature descriptor
                                                                                                           is based on the sum of the Haar wavelet response around the point of interest. The
                                                                                                           descriptor can also be obtained in the form of a vector, based on the distribution of
                                                                                                           the intensity content within the local image regions of the points. The SURF detector
                                                                                                           and descriptor utilize the integral images to reduce computation time, which makes
                                                                                                           them almost three times faster than the SIFT detector and descriptor. This technique
                                                                                                           ensures that the points of interest are robust against image rotation and scale.
                                                                                                           Scale-Invariant Feature Transform (SIFT): SIFT was published by David Lowe
                                                                                                           [20]. In the first step of the algorithm, the local maxima or minima of the Difference
                                                                                                           of Gaussians (DoG) are used to identify potential interest points, which are invariant
                                                                                                           to scale and orientation. Subsequently, points that have low contrast or are poorly
                                                                                                           localized on an edge are deleted from the list of key-points. Using the remaining


                                                                                                  142
   key-points, a consistent orientation is assigned. When the key-points are located
   and assigned with dominant orientations, the feature vectors are calculated. This
   technique can robustly identify patterns even under partial occlusion, since the SIFT
   feature descriptor is invariant to uniform scaling, orientation, and partially invariant
   to related distortions and changes in illumination.
   Oriented FAST and Rotated BRIEFF (ORB): ORB is based on the visual descriptor
   BRIEF [21] (Binary Robust Independent Elementary Features) and the FAST [22]
   key-point detector with several improvements. Both these techniques are attractive
   due to their good performance and low cost. In the first phase, it uses FAST to find
   key-points, and then it applies the Harris corner measure to calculate and remove
   potential edge points. At this stage, the orientation of each corner point is estimated
   based on the centroid intensity in regions localized in points recognized as corners.
   In order to improve rotation invariance, the central moments are computed making
   the ORB descriptor rotation-invariant. In addition, another benefit of using the ORB
   detector and descriptor is their computational efficiency when compared with the
   SIFT and SURF feature detectors and descriptors.
   Features from Accelerated Segment Test (FAST): FAST was proposed by Ros-
   ten and Drummond [22]. It is a corner detection method that can be used to extract
   feature points and to track and map objects in many computer vision tasks. The
   FAST detection procedure includes two main steps: in the first step, the potential
   corner points in an image are classified with a segment test; in the second step, a
   score value is calculated at each potential corner point. The score values are used
   to remove the false corners classified previously. In general, the most promising
   advantage of the FAST corner detector is its computational efficiency. As its name
   states, it is undoubtedly fast, faster than many other well-known feature extraction
   methods, such as the difference of Gaussians (DoG) used by SIFT and Harris. This
   technique is consequently well-suited for real-time video processing application be-
   cause of its high-speed performance.




                                                                                                               CONSERVATION SCIENCE IN CULTURAL HERITAGE
   In order to evaluate the performance of detectors and descriptors, we investigated
the speed and quantity of features extracted by each method, in conjunction with the
accuracy and speed of extraction and matching of the descriptors. All the features
were obtained with default parameters given by their implementation. In order to evalu-
ate the performance of various algorithms in quantitative terms we used some selected
image patterns and scene images. In particular, we used a pattern size of 347 x 246
pixels, which was identified in a scene image measuring 640 x 480 pixels.
   For each algorithm we extracted the number of features, the number of correct
matches and the processing time; then a mean was calculated related to all the sets of
patterns and scene images. The obtained results are summarized in Table 2.

Table 2. Algorithms performance
                                                         Features    Descriptors   Descriptors   Total
                          Pattern    Scene     Correct
 Detector   Descriptor                                   Detection    extraction    matching     Time
                         Features   Features   Matches
                                                           (ms)         (ms)          (ms)       (ms)
  SURF        SURF         169       1014       158         53           52            3         112
  SIFT        SIFT         408       2312       322        167          241            14        425
  ORB         ORB          408        500        24         8            6             3          20
  FAST        ORB         1102       5226       960         3            13            58         78




                                                                                                         143
                                                                                                           Each detector-descriptor set showed different results. In the following items, there
                                                                                                        are some considerations and comparisons on performance obtained by the algorithms
                                                                                                        in each step:
                                                                                                           •     SIFT: the most time-consuming step in SIFT is description, followed by detec-
                                                                                                                 tion. Because of its high computing complexity, SIFT spends most of the time
                                                                                                                 extracting and describing features. The matching process does not take long
                                                                                                                 and the matching algorithms for vector description are efficient.
                                                                                                           •     SURF completes detection and description in almost the same amount of time.
                                                                                                                 Compared to SIFT, the processing time is significantly reduced. The matching
F. Chionna, F. Argese, V. Palmieri, I. Spada, L. Colizzi - Integrating building information modeling




                                                                                                                 phase also takes only a short time.
                                                                                                           •     FAST is one of the most efficient feature detectors. However, the many de-
                                                                                                                 tected feature points lead to huge time consumption for the matching process,
                                                                                                                 despite the fact ORB is a binary descriptor that makes matching of the binary
                                                                                                                 description easier and faster than vector description.
                                                                                                           •     ORB: all three steps in ORB require almost the same amount of time, and com-
                                                                                                                 plete processing in a fairly short time. Compared to other descriptors tested in
                                                                                                                 this experiment, ORB performs much better than others.
                                                                                                           The advantage in terms of processing speed due to its use of binary description, is
                                                                                                        clearly demonstrated in this case.
                                                                                                           The application processes the real-time scene frame by frame, and only when the
                                                                                                        whole image matching process is finished for the current frame is the next frame then
                                                                                                        processed. This mechanism causes a delayed display when the entire matching phase
                                                                                                        takes longer than the frame rate of the video sequence acquisition. Display lag is par-
                                                                                                        ticularly evident in SIFT. Although the FAST ORB optimizes some steps, it is still unable
                                                                                                        to achieve the requirements of real-time applications.
                                                                                                           The ORB-ORB processing speed is quite fast, with no display lag observed and
                                                                                                        video sequences played back smoothly. In matching results, SIFT and SURF show
                                                                                                        outstanding performance, a considerable number of detected feature key-points, and
                                                                                                        the matching result is accurate and robust. Although the ORB-ORB detects less fea-
                                                                                                        ture key-points compared to SIFT, SURF and FAST, it nevertheless obtains a sufficient
                                                                                                        number of correct matches.
                                                                                                           In conclusion, despite their high accuracy and robustness, SIFT and SURF are
                                                                                                        unsuitable for use in a real-time application because of the long processing time. ORB-
                                                                                                        ORB satisfies the stringent real-time requirements, shows good performance and is
                                                                                                        the better trade-off in terms of accuracy and real time performance.

                                                                                                           d. AR and BIM integration

                                                                                                           In order to integrate AR and BIM technologies, each pattern identifiable through AR
                                                                                                        was connected to a specific identifier available in the BIM model. In this manner, by
                                                                                                        using the BIM reader, AR can retrieve additional information related to the identified
                                                                                                        pattern and can show it to the user superimposed on the real-time visualization in the
                                                                                                        glasses.
                                                                                                           Figure 7 shows the experimental system in which the user wears the eyewear and
                                                                                                        is able to view all the information related to the historical building through the glasses
                                                                                                        and can interact with the application using a smartphone to change the type of data to
                                                                                                        be visualized.



                                                                                                  144
Figure 7. Experimental system showing data related to building and infrared camera data



   4. Conclusions and future work

   An experimental system improving the investigation of historical buildings was de-
veloped focusing on the following aspects:
   – BIM representation;




                                                                                                    CONSERVATION SCIENCE IN CULTURAL HERITAGE
   – integration of output from non-invasive tools into BIM;
   – Augmented Reality visualization.
   Extensive tests performed by engineers and architects have revealed that such a
system may enhance the investigation of historical buildings. The system was found to
be helpful to contextualize information for indoor analysis of the building and to simplify
access to BIM information.
   Further research is required to eliminate the use of markers for Augmented Reality
completely and standardize a new BIM format (or extensions) including output from non-
invasive tools. In order to improve recognition in particular conditions, more research
must be done into 3D features recognition which, in the coming months, will be sup-
ported by the release of new mobile consumer devices integrating an RGB-D camera
(e.g. Intel RealSense technology, Microsoft Hololens, and Google Tango Project). This
type of camera enables modeling in 3D indoor environments [23], opening up new sce-
narios for Augmented Reality features detection through mobile devices. In addition,
standardization and integration of new data in BIM or the creation of a new standard for
cultural heritage could simplify reusability of acquired data and reconstructed buildings.

  Acknowledgments
Research activities in this paper are partly funded by the research program PONREC
PROVACI (Technologies for seismic protection and valorization of cultural heritage).
Grant Cod. PON01_02324.

                                                                                              145
                                                                                                        References

                                                                                                        [1]    Fischetti, D. C. (2009). Structural Investigation of Historic Buildings: A Case Study
                                                                                                               Guide to Preservation Technology for Buildings, Bridges, Towers and Mills. John
                                                                                                               Wiley & Sons.
                                                                                                        [2]    Wang, X. (2009). Augmented reality in architecture and design: potentials and
                                                                                                               challenges for application. International Journal of Architectural Computing, 7(2),
                                                                                                               309-326
                                                                                                        [3]    Eastman, C., Eastman, C. M., Teicholz, P., & Sacks, R. (2011). BIM handbook:
F. Chionna, F. Argese, V. Palmieri, I. Spada, L. Colizzi - Integrating building information modeling




                                                                                                               A guide to building information modeling for owners, managers, designers, engi-
                                                                                                               neers and contractors. John Wiley & Sons.
                                                                                                        [4]    Höllerer, T., & Feiner, S. (2004). Mobile augmented reality. Telegeoinformatics:
                                                                                                               Location-Based Computing and Services. Taylor and Francis Books Ltd., Lon-
                                                                                                               don, UK, 21.
                                                                                                        [5]    Oreni, D., Brumana, R., Cuca, B., & Georgopoulos, A. (2013, July). HBIM for
                                                                                                               conservation and management of built heritage: Towards a library of vaults and
                                                                                                               wooden bean floors. In CIPA 2013XXV International Symposium, ISPRS Annals
                                                                                                               (Vol. 164, pp. 1-6).
                                                                                                        [6]    Pauwels, P., Verstraeten, R., De Meyer, R., & Van Campenhout, J. (2008). Ar-
                                                                                                               chitectural Information Modelling for Virtual Heritage Application. In Digital Herit-
                                                                                                               age–Proceedings of the 14th International Conference on Virtual Systems and
                                                                                                               Multimedia (pp. 18-23).
                                                                                                        [7]    Shoaib, H. A Survey of Augmented Reality. interactions, 24, 34.
                                                                                                        [8]    Amin, D., & Govilkar, S. COMPARATIVE STUDY OF AUGMENTED REALITY
                                                                                                               SDK’S.
                                                                                                        [9]    Abboud, R. (2014). Architecture in an Age of Augmented Reality.
                                                                                                        [10]   Koch, C., Neges, M., König, M., & Abramovici, M. (2012). BIM-based augmented
                                                                                                               reality for facility maintenance using natural markers. In Proceedings of the EG-
                                                                                                               ICE International Workshop on Intelligent Computing in Engineering.
                                                                                                        [11]   Irizarry, J., Gheisari, M., Williams, G., & Walker, B. N. (2013). InfoSPOT: A mobile
                                                                                                               Augmented Reality method for accessing building information through a situation
                                                                                                               awareness approach. Automation in Construction, 33, 11-23.
                                                                                                        [12]   Woodward, C., & Hakkarainen, M. (2011). Mobile mixed reality system for archi-
                                                                                                               tectural and construction site visualization. INTECH Open Access Publisher.
                                                                                                        [13]   Gheisari, M., Goodman, S., Schmidt, J., Williams, G., & Irizarry, J. (2014). Explor-
                                                                                                               ing BIM and mobile augmented reality use in facilities management. In Proceed-
                                                                                                               ings of the Construction Research Congress (pp. 1941-1950).
                                                                                                        [14]   Hakkarainen, M., Woodward, C., & Rainio, K. (2009, October). Software archi-
                                                                                                               tecture for mobile mixed reality and 4D BIM interaction. In Proc. 25th CIB W78
                                                                                                               Conference (pp. 1-8).
                                                                                                        [15]   Burns, D., & Osfield, R. (2004). Open scene graph a: Introduction, b: Examples
                                                                                                               and applications.
                                                                                                        [16]   Bradski, G. (2000). The opencv library. Doctor Dobbs Journal, 25(11), 120-126.
                                                                                                        [17]   Dhillon, R. K., Jethwa, M., & Rai, H. S. (2014). Extracting Building Data from BIM
                                                                                                               with IFC. Int. J. on Recent Trends in Engineering and Technology, 11(1).
                                                                                                        [18]   Sobczak, M. (2004). A simple oracle call interface. Dr. Dobb\’s Journal, 29(12),
                                                                                                               52-55.



                                                                                                  146
[19] Bay, H., Tuytelaars, T., & Van Gool, L. (2006). Surf: Speeded up robust features.
     In Computer vision–ECCV 2006 (pp. 404-417). Springer Berlin Heidelberg.
[20] Lowe, D. G. (1999). Object recognition from local scale-invariant features. In-
     Computer vision, 1999. The proceedings of the seventh IEEE international con-
     ference on (Vol. 2, pp. 1150-1157). Ieee.
[21] Calonder, M., Lepetit, V., Strecha, C., & Fua, P. (2010). Brief: Binary robust inde-
     pendent elementary features. Computer Vision–ECCV 2010, 778-792.
[22] Rosten, E., & Drummond, T. (2006). Machine learning for high-speed corner de-
     tection. In Computer Vision–ECCV 2006 (pp. 430-443). Springer Berlin Heidel-
     berg.
[23] Henry, P., Krainin, M., Herbst, E., Ren, X., & Fox, D. (2012). RGB-D mapping:
     Using Kinect-style depth cameras for dense 3D modeling of indoor environ-
     ments. The International Journal of Robotics Research, 31(5), 647-663.

  Biographical notes

   Francesco Chionna is a computer science engineer. He has many years’ experi-
ence of working as a researcher, and four years as a manager of other engineers
and scientists. He has managed industrial and military research projects focused on
developing software and systems for virtual and augmented reality applications used
by several Universities and private companies. He has hands on experience in all
stages of system development, including requirements definition, design, architecture,
coding, testing and delivery to customer. His research focuses on ICT innovations for
construction applications including visualization technologies, image processing and
machine learning.

    Francesco Argese is a computer science engineer. After graduating with a master
degree at the University of Salento, he began working as a researcher at the Division




                                                                                                  CONSERVATION SCIENCE IN CULTURAL HERITAGE
of Computer Engineering of the CETMA Consortium, Engineering, Design & Materials
Technologies Centre. Over the years he has gained important experiences in the ICT
fields of Software Engineering, Virtual Reality, Augmented Reality and collaborative
systems by conducting research and achieving results of interest in several research
projects of national importance.

    Vito Palmieri is a Computer Science engineer. After graduating from the Polytech-
nic University of Bari, he started working as a researcher at the Division of Computer
Engineering of the CETMA Consortium, Engineering, Design & Materials Technologies
Centre. His research interests are mainly focused on Virtual Reality, Augmented Real-
ity, Human Machine Interaction and Computer Vision.

   Italo Spada is senior researcher at the CETMA Consortium in the field of new me-
dia and interaction applied to cultural heritage, industrial design, scientific research
and training. He is currently coordinating national and international projects dedicated
to new technologies, edutainment, entertainment, virtual archeology and museum in-
teraction. He works mainly on the following areas: technologies for 3D buildings, meth-
odologies and tools for the optimization of different complex geometric patterns, im-
mersive and interactive space design, dynamic real-time tracking of the human body,



                                                                                            147
                                                                                                        3D rigging, augmented reality mobile applications and interaction with dynamic 3D
                                                                                                        digital contents.

                                                                                                           Lucio Colizzi is an Information Technology engineer. After many years of experi-
                                                                                                        ence in ICT, robotics, Virtual Reality and data processing and modeling, in 2001 he be-
                                                                                                        came the ICT department Director of CETMA. He has co-ordinated important national
                                                                                                        research programs and in the last ten years has also been coordinator on several pro-
                                                                                                        jects in the field of development of technologies for cultural heritage. He has obtained
                                                                                                        many post-graduate qualifications: BPR, Concurrent Engineering and Quality Function
                                                                                                        Deployment, STEP –ISO 10303, Simple ++, eM-Plant, eM-Planner, Informix Dynamic
F. Chionna, F. Argese, V. Palmieri, I. Spada, L. Colizzi - Integrating building information modeling




                                                                                                        Server Administration and Performance Tuning, Design for Manufacturing & Assembly,
                                                                                                        Object Oriented design with UML, Microsoft certifications: C#, ADO.NET, ASP.NET;
                                                                                                        XML Web Services.NET; PROJECT MANAGEMENT (Bocconi University). As a result
                                                                                                        of his many interests he has taught “Project Management and Group Dynamics” as
                                                                                                        Adjunct Professor at the universities of Bologna and Lecce.




                                                                                                  148
