---
abstract: |
    **Background:** consolidation is an effective technique to improve resource utilization and reduce energy footprint in cloud data centers. It can be implemented in a centralized or a distributed fashion. Distributed consolidation approaches are currently gaining popularity because they are often more scalable than their centralized counterparts and they avoid a single point of failure.

    **Objective:** To present a comprehensive, unbiased overview of the state-of-the-art on distributed consolidation approaches.

    **Method:** A of the existing distributed consolidation approaches.

    **Results:** 19 papers on distributed consolidation categorized in a variety of ways. The results show that the existing distributed consolidation approaches use four types of algorithms, optimize a number of different objectives, and are often evaluated with experiments involving simulations.

    **Conclusion:** There is currently an increasing amount of interest on developing and evaluating novel distributed consolidation approaches. A number of research gaps exist where the focus of future research may be directed.

    **Keywords:** Cloud computing, Data center, Virtual machine, Consolidation, Placement, Energy-efficiency
author:
- |
    Adnan Ashraf[^1] , Benjamin Byholm, Ivan Porres\
    \
bibliography:
- 'references.bib'
title: |
    Distributed Virtual Machine Consolidation: A Systematic\
    Mapping Study
---

Introduction {#sec:background}
============

Energy footprint of cloud data centers is a matter of great concern for cloud providers [@Mastelic:2014]. According to a recent report [@Shehabi:2016], data centers in the United States consumed an estimated 70 billion kilowatt-hours of electricity in 2014, which corresponds to 1.8% of total United States electricity consumption. High energy consumption not only translates into a high operating cost, but also leads to huge carbon emissions. The ever increasing demand for computing resources to provide highly scalable and reliable services has caused an energy crisis [@Kaur:2015:EET]. The high energy consumption of data centers can partly be attributed to the large-scale installations of computing and cooling infrastructures, but more importantly it is due to the inefficient use of the computing resources [@Beloglazov:2012]. Production servers seldom operate near their full capacity [@Barroso:2007]. However, even at the completely idle state, they consume a substantial proportion of their peak power [@Fan:2007:PPW]. Therefore, under-utilized servers are highly inefficient.

Hardware virtualization technologies allow to share a among multiple, performance-isolated platforms called to improve resource utilization. Further improvement in resource utilization and reduction in energy consumption can be achieved by consolidating on under-utilized . The basic idea of consolidation is to migrate and place the on as few as possible and then release the remaining, unused for termination or for switching to a low-power mode to conserve energy. A consolidation approach uses live migration to consolidate on a reduced set of . consolidation has emerged as one of the most effective and promising techniques to reduce energy footprint of cloud data centers [@Beloglazov:2012; @Farahnakian:2015].

Figure \[fig:MotivatingExampleConsolidation\] presents a simple hypothetical scenario to illustrate the consolidation process. The first half of Figure \[fig:MotivatingExampleConsolidation\] shows three where each hosts multiple and every uses a certain amount of the resources. It is assumed that due to some significant load variations, 2 and 3 have become under-utilized. The under-utilized in such a scenario may continue to remain under-utilized for hours, days, or even weeks unless the existing require more resources or some new are placed on the under-utilized . Therefore, it is difficult to provide a resource and energy efficient allocation of without consolidation of on the under-utilized . The second half of Figure \[fig:MotivatingExampleConsolidation\] shows that after migrating all from 2 to 3, 2 can be turned-off or switched to a low-power mode.

![A simple example to illustrate the consolidation of on under-utilized to conserve energy[]{data-label="fig:MotivatingExampleConsolidation"}](consolidationExample.pdf){width="95.00000%"}

There is currently an increasing amount of interest on developing and evaluating efficient consolidation approaches for cloud data centers. Over the past few years, researchers have used a multitude of ways to develop novel consolidation approaches. Some of these approaches have been recently reported in the form of nonsystematic literature reviews such as [@Ahmad:2015] and [@Pires:2015]. However, the drawback of these existing nonsystematic studies is that they provide a partial and possibly biased overview of the state-of-the-art on consolidation. For a comprehensive and unbiased coverage of the existing literature on consolidation, there is a need to study the existing consolidation approaches in a systematic way.

consolidation can be implemented in a centralized or a distributed fashion. Traditional consolidation approaches, such as [@Beloglazov:2012; @Farahnakian2013UCC; @Ferreto:2011; @Hwang:2013; @Ferdaus2014; @Gao:2013; @Farahnakian2014PDP], tend to be centralized. A centralized consolidation approach uses a centralized algorithm on a centralized architecture and does not provide support for multiple, geographically distributed data centers. The main drawbacks of centralized consolidation approaches include limited scalability and lack of robustness due to a single point of failure. On the other hand, a distributed or decentralized consolidation approach uses a distributed algorithm or a distributed architecture for  [@Farahnakian:2015; @Feller2012CloudCom] or provides support for multiple, geographically distributed data centers [@Lucanin:2015; @Xu:2015:ICPADS]. Distributed consolidation is a recurring theme in recent consolidation approaches such as [@Farahnakian:2015; @Marzolla:2011; @Feller2012CloudCom]. Distributed approaches are gaining popularity because they have benefits over centralized approaches. They are often more scalable than their centralized counterparts and they avoid a single point of failure [@Marzolla:2011; @Masoumzadeh:2015]. Feller et al. [@Feller2012CloudCom] showed that their proposed consolidation algorithm does not compute a solution (in a reasonable amount of time) on a centralized architecture, but finds a good solution on a distributed architecture. Lucanin and Brandic [@Lucanin:2015] reported that their consolidation algorithm for geographically distributed data centers finds a good solution for a large-scale problem comprising ten thousand . Sedaghat et al. [@Sedaghat:2016] showed that their proposed distributed consolidation algorithm scales to tens of thousands of and without compromising on the quality of the solution. Sedaghat et al. [@Sedaghat:2014] reported that their proposed distributed consolidation algorithm finds a near-optimal solution for 100,000 in a reasonable amount of time. Marzolla et al. [@Marzolla:2011] showed that their proposed distributed consolidation algorithm is resilient to major failures and outages involving a thousand . Therefore, distributed consolidation approaches are more suitable for large-scale data centers involving thousands of and .

We present a systematic study of the existing distributed consolidation approaches. The objective is to present a comprehensive, unbiased overview of the state-of-the-art on distributed consolidation approaches. Considering the broad nature of the research objective, it was not appropriate to launch a . Therefore, we launched a  [@Petersen:2008; @Budgen:2008; @Petersen:2015]. A follows the same principled process as a , but: (1) it has a broader scope, (2) it uses different criteria for inclusions/exclusions and quality assessments, and (3) its data collection and synthesis tend to be more qualitative than for a  [@Wohlin:2012]. It is intended to map out the research that has been undertaken rather than to answer a detailed research question [@Budgen:2008].

We proceed as follows. Section \[sec:design\] presents the design and schedule of our study. The results of the are presented in Section \[sec:results\]. In Section \[sec:validity\], we discuss major threats to the validity of the results presented in this paper. Finally, we present our conclusions in Section \[sec:conclusions\].

Study Design {#sec:design}
============

One of the most important differences between a nonsystematic literature review and a is that a follows an unbiased and repeatable process. Moreover, the process is documented as a review protocol. Therefore, we defined a review protocol for our on distributed consolidation approaches. In this section, we present the design of our study and the review process.

Research Questions {#sec:rqs}
------------------

The are as follows:

-   1: What approaches have been developed for distributed consolidation?

-   2: What kinds of algorithms are being used in the existing distributed consolidation approaches?

-   3: What objectives are being optimized in the existing distributed consolidation approaches?

-   4: How are the existing distributed consolidation approaches being evaluated?

-   5: What are the most popular publication forums for distributed consolidation papers and how have they changed over time?

1 is the basic question for obtaining an overview of the state-of-the-art on distributed consolidation. 2 is aimed at obtaining the types of algorithms which are being used in the existing distributed consolidation approaches. Possible types include heuristics, metaheuristics, and machine learning algorithms. Moreover, the algorithms may also be categorized into offline and online optimization algorithms.

3 concerns the objectives which are being optimized. Possible objectives include minimizing energy consumption, minimizing the number of active , minimizing violations, minimizing the number of migrations, minimizing cost, minimizing network traffic, maximizing performance, maximizing reliability, and minimizing resource utilization. 3 also deals with the number of objectives which are being optimized and how the optimization problem is formulated. Possible problem formulations include single-objective, multi-objective (two or three objectives) with an , pure multi-objective, and many-objective (four or more objectives) [@Lucken:2014].

4 concerns the evaluation method. The most common evaluation method in the consolidation literature is experiment. An experiment may involve the use of prototype implementations or simulations. Moreover, the experiment design may involve realistic, synthetic, or hybrid load patterns. Similarly, an experimental evaluation may or may not include a comparison of the results with other existing consolidation approaches. Finally, a comparison of the results may or may not include statistical tests to assess the statistical significance of the results.

5 is a typical question for in software engineering [@Petersen:2008]. The objective is to identify the most popular, peer-reviewed publication forums with respect to distributed consolidation papers. The publication forums may include journals, conferences, and workshops. In addition, the second part of 5 concerns the frequencies of published papers in popular forums over time to see the trends.

Based on the , the  [@Kitchenham:07] is presented in Table \[tab:picoc\].

  **Aspect**         **Value**
  ------------------ -------------------------------------------------------------------------------
  Population (P)     Energy-aware and cost-effective data center management techniques
  Intervention (I)   Distributed consolidation/placement approaches/ algorithms/methods/heuristics
  Comparison (C)     No comparison intervention
  Outcomes (O)       An overview of the state-of-the-art on distributed consolidation approaches
  Context (C)        Cloud data centers

  : []{data-label="tab:picoc"}

Search Strategy for Primary Studies {#sec:search_strategy}
-----------------------------------

This section presents our search strategy. It is based on the and guidelines described by Kitchenham and Charters [@Kitchenham:07] and Wohlin et al. [@Wohlin:2012].

### Search Terms

Table \[tab:terms\] presents the most important search terms along with their alternate spellings. The search terms are primarily based on the and in Section \[sec:rqs\]. Moreover, they are also in line with recent and prominent works on consolidation such as [@Beloglazov:2012; @Farahnakian:2015; @Farahnakian2013UCC; @Ferreto:2011; @Hwang:2013; @Marzolla:2011; @Ferdaus2014; @Gao:2013; @Feller2012CloudCom; @Farahnakian2014PDP].

  **\#**   **Search term**     **Alternate spellings**
  -------- ------------------- --------------------------------------------------
  1        Consolidat\*        Consolidate, consolidating, consolidation
  2        Plac\*              Place, placing, placement
  3        Virtual machine\*   Virtual machine, virtual machines
  4        \*                  , s
  5        Server\*            Server, servers
  6        Algorithm\*         Algorithm, algorithms
  7        Approach\*          Approach, approaches
  8        Method\*            Method, methods
  9        Heuristic\*         Heuristic, heuristics
  10       Cloud               None
  11       Data center         Data center, datacenter, data centre, datacentre
  12       Distributed         None
  13       Decentralized       None

  : Search terms[]{data-label="tab:terms"}

### Search Strings

The search strings are presented in Table \[tab:strings\]. They are formed by making appropriate combinations of the search terms presented in Table \[tab:terms\]. We used Boolean *AND* and Boolean *OR* operators to make the search strings. The two search strings in Table \[tab:strings\] can also be combined into one search string by using the Boolean *OR* operator. Therefore, the papers which contain any of the two search strings were retrieved. The search strings were validated against a set of known papers [@Farahnakian:2015; @Marzolla:2011; @Feller2012CloudCom].

  **\#**   **Search string**
  -------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  1        (Distributed OR Decentralized) AND Consolidat\* AND (“virtual machine\*” OR \* OR server\*) AND (algorithm\* OR approach\* OR method\* OR heuristic\*) AND (cloud OR “data center” OR datacenter OR “data centre” OR datacentre)
  2        (Distributed OR Decentralized) AND Plac\* AND (“virtual machine\*” OR \*) AND (algorithm\* OR approach\* OR method\* OR heuristic\*) AND (cloud OR “data center” OR datacenter OR “data centre” OR datacentre)

  : Search strings[]{data-label="tab:strings"}

### Databases

The search strings in Table \[tab:strings\] were searched in the publication title, abstract, and keywords. The following digital libraries were searched: (1) Xplore, (2) Digital Library, (3) ScienceDirect, and (4) SpringerLink. The search strings were customized for each digital library. Moreover, since using multiple digital libraries creates duplicates, the search results were analyzed to identify and remove the duplicates.

Study Selection Criteria {#sec:selection_criteria}
------------------------

This section presents our inclusion and exclusion criteria for primary studies.

### Inclusion Criteria

The inclusion criteria for primary studies are as follows:

-   Distributed or decentralized consolidation approach or algorithm or method or heuristic *AND*

-   For cloud data center(s) *AND*

-   Written in English *AND*

-   Published in a peer-reviewed journal, conference, or workshop of computer science, computer engineering, or software engineering

In addition, if several papers presented the same consolidation approach, only the most recent was included.

### Exclusion Criteria

The exclusion criteria are the inverse of the inclusion criteria. If several papers presented the same consolidation approach, all except the most recent were excluded.

Study Selection Procedure {#sec:selection_procedure}
-------------------------

The study selection procedure was applied on the search results to remove false positives. It comprises two phases, namely (1) title and abstract level screening and (2) full-text level screening.

### Title and Abstract Level Screening

In this phase, the inclusion/exclusion criteria in Section \[sec:selection\_criteria\] were applied to publication title and abstract. To minimize researcher bias, two researchers (first and second author) independently analyzed the search results. Afterwards, the results were compared and disagreements were resolved through discussions. Moreover, for any unresolved disagreements consensus meetings [@Dyba:2008] involving all three researchers (first, second, and third author) were arranged. The short-listed studies from this phase were used as input for the second phase.

### Full-text Level Screening

In this phase, the selected studies from the first phase were further analyzed on the basis of full-text. Two researchers (first and second author) independently applied the inclusion/exclusion criteria in Section \[sec:selection\_criteria\] on the full-text. In this phase, the researchers also documented a reason for each excluded study [@Usman:2014]. The results were compared in a similar way as in the first phase and disagreements between the researchers were resolved through discussions and consensus meetings. The output of this phase was the final short-listed set of primary studies.

Study Quality Assessment Checklist and Procedure {#sec:quality}
------------------------------------------------

The final selected studies based on the selection procedure in Section \[sec:selection\_procedure\] were assessed for their quality merit. To minimize researcher bias, two researchers (first and second author) independently assessed the quality merit of the final selected studies. Any studies not meeting the minimum quality requirements were excluded from the final set of primary studies.

Table \[tab:checklist\] presents the checklist for study quality assessment. For each question in the checklist, a three-level, numeric scale was used [@Usman:2014]. The levels are: yes (2 points), no (0 point), and partial (1 point). Based on the checklist and the numeric scale, a study could score a maximum of $34$ and a minimum of $0$ points. Moreover, for each study, the two independent scores from the two researchers were aggregated by computing arithmetic mean. We used the first quartile ($34/4=8.5$) as the cutoff point for the inclusion of studies. Therefore, if a study scored less than $8.5$ points, it was excluded due to its lack of quality merit. The researchers documented the obtained score of each included/excluded study.

[|l|p[14.2cm]{}|]{} **\#** & **Question**\
\
1 & Is the consolidation approach clearly described?\
2 & Is the pseudocode of the proposed algorithm included and clearly described?\
3 & Is a concise mathematical notation used?\
4 & Is the underlying theory clearly described?\
5 & Are the assumptions clearly stated?\
6 & Are the optimization objectives clearly stated?\
\
7 & Is the evaluation method clearly described?\
8 & Is a prototype implementation presented?\
9 & Is a simulation presented?\
10 & Is the experimental design clearly described?\
11 & Is the experimental setup clearly stated?\
12 & Are realistic load patterns used in the experimental design?\
13 & Are results from multiple different experiments included?\
14 & Are results from multiple runs of each experiment included?\
15 & Are the experimental results compared with other state-of-the-art consolidation approaches?\
16 & Is a statistical test used to assess the statistical significance of the results?\
17 & Are the limitations or threats to validity clearly stated?\

Data Extraction Strategy {#sec:data_extraction}
------------------------

Table \[tab:data\_extraction\] presents the data extraction form which was used to record the data that the researchers extracted from the primary studies. Two researchers (first and second author) independently extracted data from all of the primary studies. The extracted data were used to classify the primary studies in a number of different ways. If a paper contained more than one consolidation approaches, it was classified as coming under more than one heading [@Budgen:2008].

Budgen et al. [@Budgen:2008] reported that differences in the classification of papers is a recurring theme in , even for experienced researchers. The two researchers differed considerably when classifying the papers according to the data extraction form in Table \[tab:data\_extraction\]. The extracted data were compared and the differences were resolved in consensus meetings and by referring back to the original papers [@Dyba:2008].

[|p[12.5cm]{}|p[0.8cm]{}|p[1.6cm]{}|]{} **Data item** & **Value** & **Additional notes**\
\
Data extractor name & &\
Data extraction date & &\
Study (S1, S2, S3, ...) & &\
Bibliographic reference (title, authors, year, journal/conference/workshop name) & &\
Author affiliations and countries & &\
Publication type (journal, conference, or workshop) & &\
\
Type of algorithm used (e.g., heuristic, metaheuristic) & &\
Specific algorithm used (e.g., first-fit decreasing, genetic algorithm) & &\
Online or offline optimization & &\
Number of optimization objectives & &\
Name(s) of optimization objective(s) & &\
Problem formulation (e.g., single-objective, multi-objective with ) & &\
Name of evaluation method (e.g., analytical, experiment) & &\
Evaluation tool (e.g., simulation, prototype implementation) & &\
Type of load pattern(s) used (realistic, synthetic, or hybrid) & &\
Name(s) of other consolidation approach(es) used for the comparison of the results & &\
Study (s) of other consolidation approach(es) used for the comparison of the results & &\
Name of the statistical test used (e.g., Wilcoxon Signed-Rank Test) & &\
Statistically significant results (yes, no) & &\

Synthesis of the Extracted Data {#sec:synthesis}
-------------------------------

The extracted data based on the data extraction strategy in Section \[sec:data\_extraction\] were synthesized separately for each . The papers were categorized in a variety of dimensions and counts of the number of papers in different categories were recorded [@Wohlin:2012].

Perhaps the most important result of a is a systematic map, which allows to identify evidence clusters and evidence deserts to direct the focus of future and to highlight areas where more primary studies should be performed [@Kitchenham:07]. It is important to present the systematic map in an appropriate visual format that provides a quick overview of the field and supports better analyses [@Petersen:2008]. Therefore, a visual representation of the systematic map was created by using appropriate chart types including a pie chart and several bubble charts [@Petersen:2015]. The visual representation of the systematic map allowed thematic analysis [@Wohlin:2012] to see which categories were well investigated and to identify research gaps [@Petersen:2008].

Schedule of the Study {#sec:timetable}
---------------------

The initial draft of the review protocol was written on 09.3.2016. From 10.3.2016 to 18.3.2016, the protocol went through internal and external reviews resulting in several major and minor revisions. The internal reviewers were the authors themselves, while the external reviewer was Muhammad Usman[^2] from the Department of Software Engineering at Blekinge Institute of Technology.

Table \[tab:timetable\] presents the schedule of the . The duration for title and abstract level screening was based on approximately 40 abstracts per day. Similarly, the duration for full-text level screening was based on approximately 5 full-text papers a day. The durations in Table \[tab:timetable\] implicitly also include approximately 10$\%$ time for consensus meetings to resolve disagreements among the researchers. It should be noted that the search for primary studies was completed on 24.3.2016. Therefore, any papers published after this date are not covered in the mapping study.

  **\#**   **Activity**                         **Duration**       **Due date**
  -------- ------------------------------------ ------------------ --------------
  1        Protocol review and revision         07 business days   18.3.2016
  2        Search for primary studies           04 business days   24.3.2016
  3        Title and abstract level screening   07 business days   06.4.2016
  4        Full-text level screening            07 business days   15.4.2016
  5        Quality assessment                   02 business days   19.4.2016
  6        Data extraction                      05 business days   26.4.2016
  7        Data synthesis                       05 business days   03.5.2016
  8        Initial draft of paper               10 business days   18.5.2016

  : Schedule of the study[]{data-label="tab:timetable"}

Results {#sec:results}
=======

In this section, we present the results of the on distributed consolidation approaches. Table \[tab:paper\_count\_stages\] presents the number of papers in different stages of the . The results show that the initial search retrieved 202 results. However, out of 202, 86 were found duplicate and were subsequently removed. The main reason for such a large number of duplicate results is that the results from the Digital Library were based on the Guide to Computing Literature, which provides an expanded search that includes papers from the full-text collection as well as from a number of other digital libraries including Xplore, ScienceDirect, and SpringerLink. The advantage of using the Guide to Computing Literature is that it often finds more papers, but the disadvantage is that some of those papers are also found in the Xplore, ScienceDirect, and SpringerLink digital libraries. There were also a few cases where a consolidation approach was published in several papers. In such cases, only the most recent paper was included.

  **SMS Stage**                               **Number of papers**
  ------------------------------------------ ----------------------
  Initial search results                              202
  After removing duplicates                           116
  After title and abstract level screening             41
  After full-text level screening                      21
  After quality assessment                             19

  : Number of papers in different stages[]{data-label="tab:paper_count_stages"}

From the remaining 116 papers, 75 papers were removed in the title and abstract level screening, resulting in 41 papers. Another 20 papers were removed in the full-text level screening. Out of these, 15 did not present a consolidation approach [@Simonin:2013; @Wang:2009; @Elmroth:2011; @Jing:2015; @Kabir:2014; @Khosravi:2013; @Pham:2016:HVM; @Teyeb:2014; @Jung:2010; @Ravi:2014; @Wen:2011; @Abdul:2014; @Chen:2013:ICC; @Gu:2015:ICNC; @Islam:2015], 2 presented a centralized consolidation approach [@Feller:2011; @Hermenier:2011], 2 were published in several papers [@Gutierrez:2013; @Masoumzadeh:2013], and 1 had a duplicate record [@Sedaghat:2016]. Finally, from the remaining 21 papers, 2 papers were removed in the quality assessment stage because their aggregate scores were below the cutoff point [@Larsson:2011:SMI; @Donadio:2014]. It resulted in a total of 19 papers for data extraction and synthesis. Table \[tab:final\_papers\] presents study and references of the final selected primary studies. Each primary study in Table \[tab:final\_papers\] presents a distributed consolidation approach.

  ****   **Authors**                              **Title**                                                                                                                                                              **Year**   **Ref.**
  ------ ---------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------- -----------------------
  S1     Sedaghat et al.                          Decentralized cloud datacenter reconsolidation through emergent and topology-aware behavior                                                                            2016       [@Sedaghat:2016]
  S2     Zhang et al.                             Dynamic service placement in shared service hosting infrastructures                                                                                                    2010       [@Zhang:2010:DSP]
  S3     Farahnakian et al.                       Using ant colony system to consolidate VMs for green cloud computing                                                                                                   2015       [@Farahnakian:2015]
  S4     Feller et al.                            A case for fully decentralized dynamic VM consolidation in clouds                                                                                                      2012       [@Feller2012CloudCom]
  S5     Gutierrez-Garcia and Ramirez-Nafarrate   Collaborative agents for distributed load management in cloud data centers using live migration of virtual machines                                                    2015       [@Gutierrez:2015]
  S6     Lucanin and Brandic                      Pervasive cloud controller for geotemporal inputs                                                                                                                      2015       [@Lucanin:2015]
  S7     Marzolla et al.                          Server consolidation in clouds through gossiping                                                                                                                       2011       [@Marzolla:2011]
  S8     Masoumzadeh and Hlavacs                  A cooperative multi agent learning approach to manage physical host nodes for dynamic consolidation of virtual machines                                                2015       [@Masoumzadeh:2015]
  S9     Mastroianni et al.                       Analysis of a self-organizing algorithm for energy saving in data centers                                                                                              2013       [@Mastroianni:2013]
  S10    Pham et al.                              Joint consolidation and service-aware load balancing for datacenters                                                                                                   2016       [@Pham:2016:CL]
  S11    Sedaghat et al.                          Divide the task, multiply the outcome: cooperative VM consolidation                                                                                                    2014       [@Sedaghat:2014]
  S12    Wang et al.                              Multiagent-based resource allocation for energy minimization in cloud computing systems                                                                                2016       [@Wang:2016]
  S13    Xu et al.                                Electricity cost minimization in distributed clouds by exploring heterogeneity of cloud resources and user demands                                                     2015       [@Xu:2015:ICPADS]
  S14    Adhikary et al.                          Energy-efficient scheduling algorithms for data center resources in cloud computing                                                                                    2013       [@Adhikary:2013]
  S15    Mehta et al.                             Energy cost management for geographically distributed data centres under time-variable demands and energy prices                                                       2013       [@Mehta:2013]
  S16    Sakumoto et al.                          Autonomous decentralized mechanisms for generating global order in large-scale system: using Metropolis-Hastings algorithm and applying to virtual machine placement   2012       [@Sakumoto:2012]
  S17    Sonnek et al.                            Starling: minimizing communication overhead in virtualized computing platforms using decentralized affinity-aware migration                                            2010       [@Sonnek:2010]
  S18    Velasco et al.                           Elastic operations in federated datacenters for performance and cost optimization                                                                                      2014       [@Velasco:2014]
  S19    Kavvadia et al.                          Elastic virtual machine placement in cloud computing network environments                                                                                              2015       [@Kavvadia:2015]

  : Study s and references of the final selected primary studies[]{data-label="tab:final_papers"}

Before presenting detailed results, we start with a simple, graphical overview of the topic. Figure \[fig:wordcloud\] presents a word cloud of distributed consolidation approaches generated from the titles and abstracts of the 19 primary studies. Common English words and those appearing only once were removed from the word list. Moreover, different forms and alternate spellings of a word were aggregated. The word cloud shows that some of the most frequent words include energy, cloud, data, and cost, which appear 56, 55, 46, and 45 times, respectively.

![Word cloud of the titles and abstracts of the 19 primary studies[]{data-label="fig:wordcloud"}](wordcloud.pdf){width="79.00000%"}

RQ1: Distributed VM Consolidation Approaches
--------------------------------------------

The 19 distributed consolidation approaches can be classified into three distinct categories: (1) pure distributed consolidation algorithms, (2) centralized algorithms with a distributed architecture for consolidation, and (3) consolidation algorithms for geographically distributed data centers. Table \[tab:three\_categories\] presents the three categories of distributed consolidation approaches. A pure distributed consolidation approach uses a distributed rather than a centralized algorithm to find a migration plan that optimizes the placement of on . In contrast, a distributed architecture approach provides a decentralized schema or multi-agent architecture for  [@Farahnakian:2015; @Feller2012CloudCom], but implements a centralized consolidation algorithm. The approaches in the third category extend centralized algorithms to support consolidation in multiple, geographically distributed data centers [@Lucanin:2015; @Xu:2015:ICPADS]. Table \[tab:three\_categories\] shows that 14 primary studies present pure distributed consolidation algorithms, 2 present centralized algorithms with a distributed architecture, and 3 present consolidation algorithms for distributed data centers.

  **Category**                   **Study s and references**                                        **Count**
  ------------------------------ ---------------------------------------------------------------- -----------
  Pure distributed               S1, S2, S5, S7, S8, S9, S10, S11, S12, S14, S16, S17, S18, S19       14
  Distributed architecture       S3, S4                                                                2
  For distributed data centers   S6, S13, S15                                                          3

  : Categories of distributed consolidation approaches[]{data-label="tab:three_categories"}

RQ2: Algorithm Types and Names
------------------------------

Table \[tab:synthesis\_1\] presents algorithm types and algorithm names along with the study and counts. The results show that the most frequently used algorithm in distributed consolidation approaches is distributed or coordinated local search heuristic, which was used in six primary studies. The second most used algorithms are and greedy, which were both used in two primary studies. All other algorithms were used in only one primary study. The results also show that S6 appears under two headings, namely greedy and genetic algorithm. This is because S6 presents a hybrid, two-stage optimization approach that combines a genetic algorithm with a greedy, best-fit approach for local improvement.

  **Type**           **Algorithm name**                                                   **Study s**                  **Count**
  ------------------ -------------------------------------------------------------------- --------------------------- -----------
                     Distributed or coordinated local search                              S1, S5, S7, S11, S12, S19        6
                     Greedy                                                               S6, S18                          2
                     Weighted maximum bipartite matching                                  S13                              1
                     Static threshold-based                                               S14                              1
                     Distributed bartering algorithm                                      S17                              1
                     Local search                                                         S2                               1
                                                                                          S3, S4                           2
                     Genetic algorithm                                                    S6                               1
                     Constraint programming-based large neighborhood search               S15                              1
  Machine learning   Fuzzy Q-learning algorithm                                           S8                               1
                     Probabilistic search                                                 S9                               1
                     Gibbs sampling and the alternating direction method of multipliers   S10                              1
                     Metropolis-Hastings algorithm                                        S16                              1

  : Types of algorithms and specific algorithms[]{data-label="tab:synthesis_1"}

Figure \[fig:algo\_types\] presents a summary of the four algorithm types from Table \[tab:synthesis\_1\]. It shows that 11 primary studies presented a heuristic approach, 5 presented a metaheuristic, 3 presented a statistical approach, and 1 presented a machine learning approach. Therefore, the results indicate that heuristics and metaheuristics are currently the most popular algorithm types for distributed consolidation, which collectively account for 80% of all algorithm types.

![Algorithm types[]{data-label="fig:algo_types"}](AlgoTypes.pdf){width="69.00000%"}

Table \[tab:synthesis\_2\] presents a classification of the primary studies with respect to online and offline optimization techniques. The results show that only 2 primary studies presented an online optimization approach, while 14 primary studies are based on offline optimization. Moreover, 3 primary studies were classified as inconclusive because the papers do not contain sufficient information in this regard. Therefore, the results indicate that 89% of the primary studies either use offline optimization or are classified as inconclusive. This is an important result and is in line with past research in this area. Other researchers have also found that most of the optimization problems in cloud computing are currently being addressed with offline optimization techniques [@Harman2013]. However, a drawback of the offline optimization techniques is that they require complete knowledge of the problem including possible future events, which is often difficult in a real world setting. In contrast, an online optimization algorithm receives input and produces output in an online manner [@Beloglazov:2012]. Therefore, to be able to better cope with workload variability of different types of modern applications, consolidation should be performed continuously in an online manner [@Beloglazov:2012; @Farahnakian:2015]. Hence, there is a clear need to develop online optimization approaches for distributed consolidation.

  **Online or offline**   **Study s**                                                        **Count**
  ----------------------- ----------------------------------------------------------------- -----------
  Online                  S3, S9                                                                 2
  Offline                 S2, S4, S5, S6, S7, S10, S12, S13, S14, S15, S16, S17, S18, S19       14
  Inconclusive            S1, S8, S11                                                            3

  : Online or offline optimization[]{data-label="tab:synthesis_2"}

RQ3: Objectives
---------------

Table \[tab:synthesis\_4\] categorizes the primary studies with respect to the different problem formulations and the number of optimization objectives. The results show that 11 studies proposed a multi-objective (two or three objectives) problem formulation with an . Similarly, 2 studies presented a many-objective (four or more objectives) [@Lucken:2014] problem formulation with an . The remaining 6 studies proposed a single-objective problem formulation. The results also indicate that 74% of the primary studies optimize either one or two objectives. Hence, there is currently more published work on approaches that optimize fewer objectives. It should be noted that, although 13 approaches optimize two or more objectives, none of the studies proposed a pure multi-objective or many-objective problem formulation. The benefit of the approach is that it reduces complexity and often improves runtime of the algorithm by limiting the search to a subspace of the feasible solutions. However, the drawback is that a correct combination of the objectives requires certain weights to be assigned to each objective, which often requires an in-depth knowledge of the problem domain [@Pires:2015]. Therefore, the assignment of the weights is essentially subjective [@Hu:2013:Pareto]. In contrast, pure multi-objective and many-objective approaches do not require weights and often provide a more elaborate search of the search space. Hence, there exists a research gap. There is a need to develop novel distributed consolidation approaches that formulate the problem as pure multi-objective or many-objective.

  **Problem formulation**   **Objectives**   **Study s**                          **Count**
  ------------------------- ---------------- ----------------------------------- -----------
  Single-objective          1 objective      S7, S12, S13, S14, S17, S19              6
                            2 objectives     S2, S4, S6, S8, S9, S10, S11, S16        8
                            3 objectives     S3, S5, S18                              3
                            4 objectives     S15                                      1
                            5 objectives     S1                                       1

  : Problem formulations and number of optimization objectives[]{data-label="tab:synthesis_4"}

Figure \[fig:algo\_types\_objectives\] presents the number of primary studies (as bubble size) with respect to the four different algorithm types and the five different number of objectives. Again, S6 appears twice. The results show that in 6 out of 11 studies involving heuristics, a simpler problem formulation was proposed because it optimizes a single objective. Whereas, none of the other algorithm types was used for single-objective optimization. Therefore, although heuristics are currently the most published algorithm type for distributed consolidation, they are mostly used for simpler, single-objective problem formulations.

![Number of primary studies (as bubble size) with respect to algorithm types and number of optimization objectives[]{data-label="fig:algo_types_objectives"}](AlgoTypeObjs.pdf){width="79.00000%"}

Table \[tab:synthesis\_5\] categorizes the primary studies on the basis of different optimization objectives. The results show that minimizing energy consumption is the most commonly found optimization objective in the currently published distributed consolidation approaches as it appears in 11 out of 19 studies. Other popular optimization objectives include minimizing violations and minimizing the number of migrations, which were found in 6 and 4 studies, respectively. According to Table \[tab:synthesis\_5\], the existing distributed consolidation approaches optimize a total of twelve different objectives, which can be classified into four higher-level categorizes, namely network, performance, energy, and cost. However, there is a clear emphasis on minimizing energy consumption.

  **Category**   **Name of objective**                        **Study s**                                          **Count**
  -------------- -------------------------------------------- --------------------------------------------------- -----------
                 Minimizing migration cost                    S1, S15, S16                                             3
                 Minimizing network communication cost        S1, S17, S18                                             3
                 Minimizing the number of migrations          S3, S4, S5, S6                                           4
                 Minimizing the risk of resource contention   S1                                                       1
                 Minimizing violations                        S2, S3, S8, S9, S15, S18                                 6
                 Minimizing delay cost                        S10                                                      1
                 Maximizing load balancing                    S1, S11, S16                                             3
                 Minimizing energy consumption                S3, S6, S8, S9, S10, S11, S12, S13, S14, S15, S18       11
                 Maximizing the number of released            S4, S7                                                   2
                 Maximizing resource efficiency               S1, S5                                                   2
                 Minimizing the overall operational cost      S15, S19                                                 2
                 Minimizing rental cost                       S2                                                       1

  : Names of optimization objectives[]{data-label="tab:synthesis_5"}

Figure \[fig:algo\_types\_objective\_names\] presents the number of primary studies (as bubble size) with respect to the four different algorithm types and the twelve different optimization objectives. Once again, S6 appears twice under minimizing the number of migrations and under minimizing energy consumption. The results in Figure \[fig:algo\_types\_objective\_names\] show that minimizing energy consumption and minimizing violations are the two common objectives in all four algorithm types. The bubble plot in Figure \[fig:algo\_types\_objective\_names\] also highlights a number of research gaps for potential future primary studies. It shows that 9 out of 12 optimization objectives are currently being addressed with only one or two algorithm types. For example, minimizing network communication cost appears only in heuristic approaches. So far, the other three algorithm types have not been investigated for this objective. Hence, there is an opportunity to develop new distributed consolidation approaches to bridge these research gaps.

![Number of primary studies (as bubble size) with respect to algorithm types and optimization objectives[]{data-label="fig:algo_types_objective_names"}](AlgoTypeObjNames.pdf){width="89.00000%"}

RQ4: Evaluation Methods and Tools
---------------------------------

We also extracted data to categorize the primary studies on the basis of the evaluation methods used. We found that experiment is the most common evaluation method for distributed consolidation approaches as it was used in all 19 primary studies. Moreover, analytical models were used in 4 studies to additionally provide a proof of a theoretical approximation or limit. Therefore, experiment is the main evaluation method.

Table \[tab:synthesis\_8\] categorizes the primary studies on the basis of the different types of evaluation tools and load patterns used in the experiments. The results show that the most common evaluation tool in distributed consolidation approaches is simulation as it was used in 16 studies. Moreover, 3 studies presented a prototype implementation. Therefore, simulation is currently the most popular evaluation tool for distributed consolidation approaches. According to Table \[tab:synthesis\_8\], synthetic, realistic, and hybrid load patterns were used in 11, 7 and, 3 studies, respectively. It should be noted that the experiments in S2 and S3 were based on both synthetic and realistic load patterns. The results illustrate that synthetic load patterns are the most common type of load patterns used in the existing distributed consolidation approaches. Moreover, the experiments in 10 studies used simulations involving synthetic load patterns. On the other hand, there is currently only one study under each load pattern type that uses a prototype implementation. Therefore, there is clearly a need for more primary studies involving prototype implementations under all three types of load patterns.

  **Evaluation tool**   **Load pattern**   **Study s**                                    **Count**
  --------------------- ------------------ --------------------------------------------- -----------
                        Synthetic          S1, S2, S3, S5, S7, S12, S14, S15, S16, S19       10
                        Realistic          S2, S3, S8, S9, S10, S13                           6
                        Hybrid             S6, S11                                            2
                        Synthetic          S4                                                 1
                        Realistic          S18                                                1
                        Hybrid             S17                                                1

  : Types of evaluation tools and load patterns used[]{data-label="tab:synthesis_8"}

Table \[tab:synthesis\_10\] categorizes the primary studies based on the names of the other consolidation approaches that were used in the primary studies for a comparison of the results. The results show that 9 studies did not contain a comparison of the results. Moreover, each of first-fit decreasing, static threshold approach, median absolute deviation, interquartile range, and best-fit decreasing was used in 2 studies. Each of the other 15 approaches was used in only one study. Therefore, the results show that a total of 20 different approaches were used for a comparison of the results in 10 primary studies. Moreover, most of the approaches were used in only one study. Hence, there is currently no general agreement in the cloud computing community concerning good approaches for a comparison of the results.

  **Name of other approach**                           **Study s**                                 **Count**
  ---------------------------------------------------- ------------------------------------------ -----------
  None                                                 S5, S7, S9, S10, S13, S15, S16, S18, S19        9
  First-fit decreasing                                 S1, S4                                          2
  Centralized greedy algorithm                         S2                                              1
  Ant colony optimization with vector algebra          S3                                              1
  Static threshold approach                            S3, S8                                          2
  Median absolute deviation                            S3, S8                                          2
  Interquartile range                                  S3, S8                                          2
  Local regression                                     S3                                              1
  Sercon                                               S4                                              1
  V-MAN                                                S4                                              1
  Best-fit decreasing                                  S6, S12                                         2
  Random                                               S11                                             1
  Incremental consolidation using thresholds           S11                                             1
  One-shot merge                                       S11                                             1
  Genetic algorithm                                    S12                                             1
  Energy and migration-cost-aware approach (pMapper)   S12                                             1
  Probability-based approach                           S12                                             1
  Energy-aware hierarchical scheduling                 S14                                             1
  Energy-aware scheduling for infrastructure clouds    S14                                             1
  No migration                                         S17                                             1
  Optimal                                              S17                                             1

  : Names of other consolidation approaches used for a comparison of the results[]{data-label="tab:synthesis_10"}

Table \[tab:synthesis\_11\] presents study of the other distributed consolidation approaches that were used for the comparison of the results in the primary studies. The results show that S4 is the only primary study in which the results of the proposed approach were compared with another distributed consolidation approach (S7). It is an important result. It shows that all approaches used for the comparison of the results except V-MAN (S7) are centralized approaches. Consequently, in 9 out of 10 studies that contain a comparison of the results, the results of the proposed distributed consolidation approach are compared with centralized consolidation approaches. Therefore, there exists little evidence on how the different distributed consolidation approaches compare to one another. For more meaningful comparisons of the results in future primary studies on distributed consolidation, we recommend that one or more of the 19 approaches studied in this should be considered.

  **Study of other approach**   **Study s**                                                                         **Count**
  ----------------------------- ---------------------------------------------------------------------------------- -----------
  None                          S1, S2, S3, S5, S6, S7, S8, S9, S10, S11, S12, S13, S14, S15, S16, S17, S18, S19       18
  S7                            S4                                                                                      1

  : Study s of other consolidation approaches used for the comparison of the results[]{data-label="tab:synthesis_11"}

Another important question concerning the evaluation of a distributed consolidation approach is that whether or not a statistical test was used to assess the statistical significance of the results. Therefore, we extracted this information form each primary study. The results show that none of the 19 studies includes a statistical test. Therefore, it is not a common practice to use statistical testing.

RQ5: Publication Forums
-----------------------

Table \[tab:synthesis\_14\] presents publication forum names for the 19 distributed consolidation papers over time. The results show that the papers are published in 17 different publication forums. There are only two publication forums in which more than one papers have been published. These include Transactions on Services Computing and International Conference on Cloud Computing Technology and Science. Therefore, it is difficult to conclude whether or not there are any popular publication forums for distributed consolidation approaches.

  **Publication forum name**                                                                                                                                 **Year**   **s**
  ---------------------------------------------------------------------------------------------------------------------------------------------------------- ---------- --------
  Future Generation Computer Systems                                                                                                                         2016       S1
  Proceedings of the 9th IFIP TC 6 International Conference on Networking                                                                                    2010       S2
  IEEE Transactions on Services Computing                                                                                                                    2015       S3, S5
  IEEE 4th International Conference on Cloud Computing Technology and Science                                                                                2012       S4
  IEEE Transactions on Cloud Computing                                                                                                                       2015       S6
  IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks                                                                        2011       S7
  IEEE 4th Symposium on Network Cloud Computing and Applications                                                                                             2015       S8
  IEEE 27th International Parallel and Distributed Processing Symposium Workshops PhD Forum                                                                  2013       S9
  IEEE Communications Letters                                                                                                                                2016       S10
  IEEE 6th International Conference on Cloud Computing Technology and Science                                                                                2014       S11
  IEEE Transactions on Systems, Man, and Cybernetics: Systems                                                                                                2016       S12
  IEEE 21st International Conference on Parallel and Distributed Systems                                                                                     2015       S13
  IEEE 10th International Conference on High Performance Computing and Communications & IEEE International Conference on Embedded and Ubiquitous Computing   2013       S14
  IEEE/ACM 6th International Conference on Utility and Cloud Computing                                                                                       2013       S15
  9th Asia-Pacific Symposium on Information and Telecommunication Technologies                                                                               2012       S16
  39th International Conference on Parallel Processing                                                                                                       2010       S17
  Computer Communications                                                                                                                                    2014       S18
  Computer Networks                                                                                                                                          2015       S19

  : Publication forums for distributed consolidation papers over time[]{data-label="tab:synthesis_14"}

Figure \[fig:pub\_types\_years\] presents the number of primary studies with respect to the three different publication types over time. The results show that 8 primary studies were published in journals, 10 were published in conferences, and 1 was published in a workshop. Moreover, from 2010 to 2014 (five years), a total of 10 primary studies were published. Whereas, 6 studies were published in 2015 alone. Similarly, 3 studies are published already in 2016. Furthermore, all 8 journal papers were published between 2014 and 2016. Hence, there is currently an increasing amount of interest on developing and evaluating novel distributed consolidation approaches and publishing them in more rigorous publication forums.

![Number of primary studies (as bubble size) with respect to publication types over time[]{data-label="fig:pub_types_years"}](PubTypeYears.pdf){width="79.00000%"}

Validity Evaluation {#sec:validity}
===================

In this section, we discuss major threats to the validity of the results presented in this paper. The first main threat is related to the coverage of the relevant literature. To mitigate this threat, we designed a comprehensive search strategy based on the and guidelines in [@Kitchenham:07; @Wohlin:2012]. The search terms were extracted from the and were validated against a set of recent and prominent works on consolidation including [@Beloglazov:2012; @Farahnakian:2015; @Farahnakian2013UCC; @Ferreto:2011; @Hwang:2013; @Marzolla:2011; @Ferdaus2014; @Gao:2013; @Feller2012CloudCom; @Farahnakian2014PDP]. Similarly, the search strings were validated against a set of known studies on distributed consolidation including [@Farahnakian:2015; @Marzolla:2011; @Feller2012CloudCom]. The search was performed in four major computer science digital libraries. Finally, the search in the Digital Library was performed by using the Guide to Computing Literature, which provides an expanded search.

The second threat is related to the selection of the primary studies. The results show that 75 out of 116 papers were excluded in the title and abstract level screening. It is possible that some relevant papers were erroneously excluded during the initial screening phase. To mitigate this threat, two researchers (first and second author) independently screened the titles and abstracts of each paper. The results were compared and disagreements were resolved through discussions. Moreover, for any unresolved disagreements, consensus meetings [@Dyba:2008] were arranged. A similar approach was used in the full-text level screening phase.

The third major threat is related to data extraction and classification of studies. As reported by Budgen et al. [@Budgen:2008], differences in the classification of papers is a recurring theme in , even for experienced researchers. To mitigate this threat, two researchers (first and second author) independently extracted data from all 19 primary studies. The extracted data were compared and the differences were resolved in consensus meetings and by referring back to the original papers.

Conclusions {#sec:conclusions}
===========

In this paper, we presented a of distributed consolidation approaches. We used and guidelines in the literature to design a comprehensive search strategy. The initial search returned 202 results from four major computer science digital libraries. After the removal of duplicate results and the application of the inclusion/exclusion criteria at two levels, 21 primary studies were selected. Finally, 2 studies were excluded in the quality assessment stage, which left 19 primary studies for data extraction and synthesis.

The objective of the was to provide a comprehensive, unbiased overview of the state-of-the-art on distributed consolidation approaches. The comprises five concerning: (1) existing approaches, (2) types of algorithms being used, (3) objectives being optimized, (4) evaluation methods and tools being used, and (5) popular publication forums over time. The results of the first showed that 14 out of 19 studies presented pure distributed consolidation algorithms, while 2 studies presented centralized algorithms with a distributed architecture for consolidation and 3 studies presented consolidation algorithms for geographically distributed data centers.

The answer to the second showed that the existing distributed consolidation approaches use four different types of algorithms, namely heuristics, metaheuristics, machine learning algorithms, and statistical approaches. Moreover, heuristics and metaheuristics are currently the most popular algorithm types. The most frequently used algorithm is distributed or coordinated local search heuristic, while and greedy are the second most used algorithms. Only a small fraction of the existing distributed consolidation approaches can be categorized as using an online optimization technique. Hence, online optimization techniques are currently not sufficiently investigated for distributed consolidation.

For the third concerning optimization objectives, we categorized the primary studies with respect to number and name of objectives and problem formulation. The results showed that nearly $\frac{3}{4}$ of the primary studies optimize either one or two objectives and only a few approaches optimize more than two objectives. The existing distributed consolidation approaches optimize a total of 12 different objectives. The most popular optimization objective is minimizing energy consumption. Other popular objectives include minimizing violations and minimizing the number of migrations. 9 out of 12 objectives are currently being addressed with only one or two algorithm types. Hence, the focus of future primary studies may be directed to investigate the remaining algorithm types for optimizing these objectives. About $\frac{2}{3}$ of the studies presented a multi-objective or many-objective problem formulation with an , while the rest of the studies presented a single-objective problem formulation. None of the studies presented a pure multi-objective or many-objective problem formulation. Hence, future research may be directed to develop pure multi-objective and many-objective distributed consolidation approaches.

The results of the fourth showed that experiment is the most common evaluation method for distributed consolidation approaches. The most common evaluation tool is simulation. Moreover, synthetic load patterns are the most common type of load patterns. Therefore, simulations involving synthetic load patterns are currently the most common evaluation tool. We also extracted data with respect to the other consolidation approaches that were used for a comparison of the results. The results showed that 9 studies did not contain a comparison of the results. Moreover, a total of 20 different approaches were used for a comparison of the results in the remaining 10 studies. In 9 out of 10 studies that contained a comparison of the results, the results of the proposed distributed consolidation approach were compared with centralized consolidation approaches. Therefore, there exists little evidence on how the different distributed consolidation approaches compare to one another. Hence, there is a need for more comparative studies involving multiple distributed consolidation approaches. We recommend that one or more of the 19 approaches studied in this should be considered for more meaningful comparisons of the results in future studies.

The answer to the fifth showed that the 19 studies were published in 17 different publication forums. Therefore, it is difficult to conclude whether or not there are any popular publication forums for distributed consolidation approaches. There is currently an increasing amount of interest on developing and evaluating novel distributed consolidation approaches for cloud data centers and publishing them in more rigorous publication forums.

Acknowledgements {#acknowledgements .unnumbered}
================

The authors would like to thank Muhammad Usman at Blekinge Institute of Technology for his valuable comments on the review protocol. The work was supported by the Need for Speed (N4S) Program (<http://www.n4s.fi>). Adnan Ashraf also received a research grant from the Ulla Tuominen Foundation.

[^1]: Corresponding author e-mail: `adnan.ashraf@abo.fi`

[^2]: <https://www.bth.se/eng/staff/muhammad-usman-muu/>
