Six reasons for rejecting an industrial survey paper
---
paper_title: Preliminary Findings from a Survey on the MD State of the Practice

paper_content:

In the context of an Italian research project, this paper reports on an on-line survey, performed with 155 software professionals, with the aim of investigating about their opinions and experiences in modeling during software development and Model-driven engineering usage. The survey focused also on used modeling languages, processes and tools. A preliminary analysis of the results confirmed that Model-driven engineering, and more in general software modeling, are very relevant phenomena. Approximately 68% of the sample use models during software development. Among then, 44% generate code starting from models and 16% execute them directly. The preferred language for modeling is UML but DSLs are used as well.
---
paper_title: Principles of survey research: part 1: turning lemons into lemonade

paper_content:

Surveys are probably the most commonly-used research method world-wide. Survey work is visible not only because we see many examples of it in software engineering research, but also because we are often asked to participate in surveys in our private capacity, as electors, consumers, or service users. This widespread use of surveys may give us the impression that surveybased research is straightforward, an easy option for researchers to gather important information about products, context, processes, workers and more. In our personal experience with applying and evaluating research methods and their results, we certainly did not expect to encounter major problems with a survey that we planned, to investigate issues associated with technology adoption. This article and subsequent ones in this series describe how wrong we were. We do not want to give the impression that there is any way of turning a bad survey into a good one; if a survey is a lemon, it stays a lemon. However, we believe that learning from our mistakes is the way to make lemonade from lemons. So this series of articles shares with you our lessons learned, in the hope of improving survey research in software engineering.
---
paper_title: Stakeholders' Perception of Success: An Empirical Investigation

paper_content:

Different stakeholders involved in the software development may attribute success to different indicators. Analogously they may support different factors considered the root of successful projects. The study presented in this paper explores how different stakeholders perceive project success and what they deem the effect of specific factors on the project outcome. The study highlighted both commonalities and differences among three main stakeholder classes. A substantial agreement was observed concerning the characteristics that make a project or product successful. As far as the factors that could lead to success are concerned, more bias emerged.
---
paper_title: SOA adoption in the Italian industry

paper_content:

We conducted a personal opinion survey in two rounds – years 2008 and 2011 – with the aim of investigating the level of knowledge and adoption of SOA in the Italian industry. We are also interested in understanding what is the trend of SOA (positive or negative?) and what are the methods, technologies and tools really used in the industry. The main findings of this survey are the following: (1) SOA is a relevant phenomenon in Italy, (2) Web services and RESTFul services are well-known/used and (3) orchestration languages and UDDI are little known and used. These results suggest that in Italy SOA is interpreted in a more simplistic way with respect to the current/real definition (i.e., without the concepts of orchestration/choreography and registry). Currently, the adoption of SOA is medium/low with a stable/positive trend of pervasiveness.
---
paper_title: Practical Experiences in the Design and Conduct of Surveys in Empirical Software Engineering

paper_content:

A survey is an empirical research strategy for the collection of information from heterogeneous sources. In this way, survey results often exhibit a high degree of external validity. It is complementary to other empirical research strategies such as controlled experiments, which usually have their strengths in the high internal validity of the findings. While there is a growing number of (quasi-)controlled experiments reported in the software engineering literature, few results of large scale surveys have been reported there. Hence, there is still a lack of knowledge on how to use surveys in a systematic manner for software engineering empirical research.
---
paper_title: Maturity of software modelling and model driven engineering: a survey in the Italian industry

paper_content:

Background: The main claimed advantage of model driven engineering is the improvement of productivity. However, little information is available about its actual usage during software development and maintenance in the industry. Objective: The main aim of this work is investigating the level of maturity in the usage of software models and model driven engineering in the Italian industry. The perspective is that of software engineering researchers. Method: First, we conducted an exploratory personal opinion survey with 155 Italian software professionals. The data were collected with the help of a web-based on-line questionnaire. Then, we conducted focused interviews with three software professionals to interpret doubtful results. Results: Software modelling is a very relevant phenomenon in the Italian industry. Model driven techniques are used in the industry, even if (i) only for a limited extent, (ii) despite a quite generalized dissatisfaction about available tools and (iii) despite a generally low experience of the IT personnel in such techniques. Limitations: Generalization of results is limited due to the sample size. Moreover, possible self-exclusion from participants not interested in modelling could have biased the results. Conclusion: Results reinforce existing evidence regarding the usage of software modelling and (partially of) model driven engineering in the industry but highlight several aspects of immaturity of the Italian industry.
---
paper_title: Software migration projects in Italian industry: Preliminary results from a state of the practice survey

paper_content:

Software migration is a fundamental and complex task in software maintenance, particularly relevant in recent years given the pervasiveness of Web and of mobile technologies. In the context of an Italian Research Project devoted to the empirical assessment of migration techniques and tools, this paper reports on a survey, performed among 59 Italian Information Technology (IT) companies, with the aim of investigating about their experiences in software migration, their main migration goals, and the pieces of technology adopted. The research project and the survey focused, in particular, on in-house migration projects towards the Web, service oriented architectures and wireless environments. A preliminary analysis of results confirmed that software migration is a very relevant phenomenon. Most migration activities performed in recent years targeted the Web, with a few number of migrations towards mobile and towards service-oriented architectures. Among other things, the survey highlights a limited, and insufficient, usage and availability of tools for supporting migration tasks.
---
paper_title: Actual vs. perceived effect of software engineering practices in the Italian industry

paper_content:

A commonly cited limitation of software engineering research consists in its detachment from the industrial practice. Several studies have analyzed a number of practices and identified their benefits and drawbacks but little is known about their dissemination in the industry. For a set of 18 practices commonly studied in the literature, this paper investigated diffusion, effect on the success, and perceived usefulness in 62 actual industrial projects from 28 Italian IT companies. In particular we proposed a classification of these perceptions and we were able to classify 14 practices. We found statistical evidence that 7 factors have an actual effect (positive for 6 of them, negative for one). Moreover 77% (10 out of 13) of the known good practices (e.g., importance of good project schedule or complete requirements' list) are perceived consistently by the industry. For a few other practices (having a champion's support, using metrics, reducing quality) we noticed a lack of awareness in the industry. Starting from these observations we propose guidelines for industrial practice and suggestions for academic research.
---
paper_title: Migration of information systems in the Italian industry: A state of the practice survey

paper_content:

Context: Software migration-and in particular migration towards the Web and towards distributed architectures-is a challenging and complex activity, and has been particularly relevant in recent years, due to the large number of migration projects the industry had to face off because of the increasing pervasiveness of the Web and of mobile devices. Objective: This paper reports a survey aimed at identifying the state-of-the-practice of the Italian industry for what concerns the previous experiences in software migration projects-specifically concerning information systems-the adopted tools and the emerging needs and problems. Method: The study has been carried out among 59 Italian Information Technology companies, and for each company a representative person had to answer an on-line questionnaire concerning migration experiences, pieces of technology involved in migration projects, adopted tools, and problems occurred during the project. Results: Indicate that migration-especially towards the Web-is highly relevant for Italian IT companies, and that companies tend to increasingly adopt free and open source solutions rather than commercial ones. Results also indicate that the adoption of specific tools for migration is still very limited, either because of the lack of skills and knowledge, or due to the lack of mature and adequate options. Conclusions: Findings from this survey suggest the need for further technology transfer between academia and industry for the purpose of favoring the adoption of software migration techniques and tools.
---
paper_title: An exploratory survey on SOA knowledge, adoption and trend in the Italian industry

paper_content:

The main aim of this work is investigating the level of knowledge and diffusion of SOA (Service Oriented Architecture) in the Italian industry. We are also interested to understand what is the trend of SOA (positive or negative?) and what are the methods, technologies and tools really used in the industry.
---
paper_title: Stakeholders' Perception of Success: An Empirical Investigation

paper_content:

Different stakeholders involved in the software development may attribute success to different indicators. Analogously they may support different factors considered the root of successful projects. The study presented in this paper explores how different stakeholders perceive project success and what they deem the effect of specific factors on the project outcome. The study highlighted both commonalities and differences among three main stakeholder classes. A substantial agreement was observed concerning the characteristics that make a project or product successful. As far as the factors that could lead to success are concerned, more bias emerged.
---
paper_title: Software migration projects in Italian industry: Preliminary results from a state of the practice survey

paper_content:

Software migration is a fundamental and complex task in software maintenance, particularly relevant in recent years given the pervasiveness of Web and of mobile technologies. In the context of an Italian Research Project devoted to the empirical assessment of migration techniques and tools, this paper reports on a survey, performed among 59 Italian Information Technology (IT) companies, with the aim of investigating about their experiences in software migration, their main migration goals, and the pieces of technology adopted. The research project and the survey focused, in particular, on in-house migration projects towards the Web, service oriented architectures and wireless environments. A preliminary analysis of results confirmed that software migration is a very relevant phenomenon. Most migration activities performed in recent years targeted the Web, with a few number of migrations towards mobile and towards service-oriented architectures. Among other things, the survey highlights a limited, and insufficient, usage and availability of tools for supporting migration tasks.
---
paper_title: Migration of information systems in the Italian industry: A state of the practice survey

paper_content:

Context: Software migration-and in particular migration towards the Web and towards distributed architectures-is a challenging and complex activity, and has been particularly relevant in recent years, due to the large number of migration projects the industry had to face off because of the increasing pervasiveness of the Web and of mobile devices. Objective: This paper reports a survey aimed at identifying the state-of-the-practice of the Italian industry for what concerns the previous experiences in software migration projects-specifically concerning information systems-the adopted tools and the emerging needs and problems. Method: The study has been carried out among 59 Italian Information Technology companies, and for each company a representative person had to answer an on-line questionnaire concerning migration experiences, pieces of technology involved in migration projects, adopted tools, and problems occurred during the project. Results: Indicate that migration-especially towards the Web-is highly relevant for Italian IT companies, and that companies tend to increasingly adopt free and open source solutions rather than commercial ones. Results also indicate that the adoption of specific tools for migration is still very limited, either because of the lack of skills and knowledge, or due to the lack of mature and adequate options. Conclusions: Findings from this survey suggest the need for further technology transfer between academia and industry for the purpose of favoring the adoption of software migration techniques and tools.
---
paper_title: Preliminary Findings from a Survey on the MD State of the Practice

paper_content:

In the context of an Italian research project, this paper reports on an on-line survey, performed with 155 software professionals, with the aim of investigating about their opinions and experiences in modeling during software development and Model-driven engineering usage. The survey focused also on used modeling languages, processes and tools. A preliminary analysis of the results confirmed that Model-driven engineering, and more in general software modeling, are very relevant phenomena. Approximately 68% of the sample use models during software development. Among then, 44% generate code starting from models and 16% execute them directly. The preferred language for modeling is UML but DSLs are used as well.
---
paper_title: SOA adoption in the Italian industry

paper_content:

We conducted a personal opinion survey in two rounds – years 2008 and 2011 – with the aim of investigating the level of knowledge and adoption of SOA in the Italian industry. We are also interested in understanding what is the trend of SOA (positive or negative?) and what are the methods, technologies and tools really used in the industry. The main findings of this survey are the following: (1) SOA is a relevant phenomenon in Italy, (2) Web services and RESTFul services are well-known/used and (3) orchestration languages and UDDI are little known and used. These results suggest that in Italy SOA is interpreted in a more simplistic way with respect to the current/real definition (i.e., without the concepts of orchestration/choreography and registry). Currently, the adoption of SOA is medium/low with a stable/positive trend of pervasiveness.
---
paper_title: Maturity of software modelling and model driven engineering: a survey in the Italian industry

paper_content:

Background: The main claimed advantage of model driven engineering is the improvement of productivity. However, little information is available about its actual usage during software development and maintenance in the industry. Objective: The main aim of this work is investigating the level of maturity in the usage of software models and model driven engineering in the Italian industry. The perspective is that of software engineering researchers. Method: First, we conducted an exploratory personal opinion survey with 155 Italian software professionals. The data were collected with the help of a web-based on-line questionnaire. Then, we conducted focused interviews with three software professionals to interpret doubtful results. Results: Software modelling is a very relevant phenomenon in the Italian industry. Model driven techniques are used in the industry, even if (i) only for a limited extent, (ii) despite a quite generalized dissatisfaction about available tools and (iii) despite a generally low experience of the IT personnel in such techniques. Limitations: Generalization of results is limited due to the sample size. Moreover, possible self-exclusion from participants not interested in modelling could have biased the results. Conclusion: Results reinforce existing evidence regarding the usage of software modelling and (partially of) model driven engineering in the industry but highlight several aspects of immaturity of the Italian industry.
---
paper_title: Actual vs. perceived effect of software engineering practices in the Italian industry

paper_content:

A commonly cited limitation of software engineering research consists in its detachment from the industrial practice. Several studies have analyzed a number of practices and identified their benefits and drawbacks but little is known about their dissemination in the industry. For a set of 18 practices commonly studied in the literature, this paper investigated diffusion, effect on the success, and perceived usefulness in 62 actual industrial projects from 28 Italian IT companies. In particular we proposed a classification of these perceptions and we were able to classify 14 practices. We found statistical evidence that 7 factors have an actual effect (positive for 6 of them, negative for one). Moreover 77% (10 out of 13) of the known good practices (e.g., importance of good project schedule or complete requirements' list) are perceived consistently by the industry. For a few other practices (having a champion's support, using metrics, reducing quality) we noticed a lack of awareness in the industry. Starting from these observations we propose guidelines for industrial practice and suggestions for academic research.
---
paper_title: An exploratory survey on SOA knowledge, adoption and trend in the Italian industry

paper_content:

The main aim of this work is investigating the level of knowledge and diffusion of SOA (Service Oriented Architecture) in the Italian industry. We are also interested to understand what is the trend of SOA (positive or negative?) and what are the methods, technologies and tools really used in the industry.
---
paper_title: Stakeholders' Perception of Success: An Empirical Investigation

paper_content:

Different stakeholders involved in the software development may attribute success to different indicators. Analogously they may support different factors considered the root of successful projects. The study presented in this paper explores how different stakeholders perceive project success and what they deem the effect of specific factors on the project outcome. The study highlighted both commonalities and differences among three main stakeholder classes. A substantial agreement was observed concerning the characteristics that make a project or product successful. As far as the factors that could lead to success are concerned, more bias emerged.
---
paper_title: Principles of survey research: part 1: turning lemons into lemonade

paper_content:

Surveys are probably the most commonly-used research method world-wide. Survey work is visible not only because we see many examples of it in software engineering research, but also because we are often asked to participate in surveys in our private capacity, as electors, consumers, or service users. This widespread use of surveys may give us the impression that surveybased research is straightforward, an easy option for researchers to gather important information about products, context, processes, workers and more. In our personal experience with applying and evaluating research methods and their results, we certainly did not expect to encounter major problems with a survey that we planned, to investigate issues associated with technology adoption. This article and subsequent ones in this series describe how wrong we were. We do not want to give the impression that there is any way of turning a bad survey into a good one; if a survey is a lemon, it stays a lemon. However, we believe that learning from our mistakes is the way to make lemonade from lemons. So this series of articles shares with you our lessons learned, in the hope of improving survey research in software engineering.
---

Title: Six Reasons for Rejecting an Industrial Survey Paper

Section 1: INTRODUCTION
Description 1: Introduce the motivation and background for industrial surveys in software engineering and explain their importance and challenges.

Section 2: EXPERIENCE GAINED
Description 2: Summarize the authors' experiences and the details of several industrial surveys they have conducted over the past ten years.

Section 3: METHOD
Description 3: Describe the methodology used to analyze the reviewer comments about the industrial survey papers, including categorizing the comments and analyzing their frequency.

Section 4: CRITICISMS
Description 4: Present the categorized criticisms received for the survey papers, including their frequency and groundedness, and discuss each criticism in detail.

Section 5: No Practical Usefulness
Description 5: Discuss the common criticism regarding the lack of practical usefulness of the survey results and provide a rebuttal.

Section 6: Sampling Bias
Description 6: Describe the issues related to sampling bias, including self-selection bias, sampling frame, and the representativeness of the sample, and suggest possible mitigation strategies.

Section 7: Obvious Conclusions
Description 7: Address the criticism that survey results are often deemed obvious or non-controversial and provide a counter-argument.

Section 8: People's Perceptions
Description 8: Explain the limitation of surveys collecting self-reported data and argue for the importance of human perceptions in software engineering.

Section 9: Non Respondents
Description 9: Discuss the challenges of analyzing non-respondents and the computation of response rates.

Section 10: Limited Geographical Scope
Description 10: Address the criticism of limited geographical scope in survey studies and discuss its implications on the generalizability of the findings.

Section 11: CONCLUSIONS
Description 11: Summarize the main findings and argue for a more lenient and understanding approach to evaluating industrial survey papers, emphasizing the importance of such studies despite their limitations.