1 Introduction

When companies and universities work in tandem to push the frontiers of knowledge, they become a powerful engine for innovation and economic growth” (Edmondson et al. 2012).

The global software industry and the academic world of Software Engineering (SE) are both large communities. However, unfortunately, a small ratio of SE practitioners and researchers collaborate with members of the other community, and the reality is that these two communities are largely disjoint (Glass 2006; Garousi et al. 2016a; Briand et al. 2017). For example, at an academic (industrial) SE conference, only a handful of practitioners (researchers) are usually present (if any), and vice versa.

This is not a new problem. Since the inception of SE in the late 1960’s, both communities have generally done little to bridge the “chasm” between them (Glass 2006), and the ratio of collaborative projects is thus relatively small compared to the number of research projects in the research community and SE activities in the industry. Various reasons have been suggested to explain the low number of industry-academia collaborations (IAC), e.g., difference of objectives between the two communities, industrial problems lacking scientific novelty or challenges, and low applicability or lacking scalability of the solutions developed in academia (Garousi et al. 2016a; Briand 2012). Yet, for the SE research community to have a meaningful future, there is a critical need to better connect industry and academia.

As we, members of the SE community, pass and celebrate the “50 years of SE” (as of this writing in 2018) (Ebert 2018; Broy 2018), many members of the SE community highlight the need for (more) industry–academia collaborations in SE (Carver and Prikladnicki 2018; Basili et al. 2018).

This need comes as no surprise to the SE community, because, being an applied discipline, it has long seen industrial relevance and impact of research activities to be of outmost importance. An indicator for this importance to the SE research community is the ACM SIGSOFT Impact project (n.d.; Osterweil et al. 2008), which was conducted in the years from 2002 to 2008. This project measured and analyzed the impact of SE research on practice. To stress the importance of IAC and to discuss success stories on how to “bridge the gap”, various workshops and panels are regularly organized within international research conferences. An example is the panel “What industry wants from research” conducted at the International Conference on Software Engineering (ICSE) 2011, in which interesting ideas from companies such as Toshiba, Google and IBM, were presented. Another international workshop on the topic of long-term industrial collaborations on SE (called WISE) was organized in 2014, which hosted several noteworthy talks. In 2016, a conference panel was held on “the state of software engineering research” (Storey et al. 2016), in which several panelists discussed the need for more IAC in SE. Similar activities have been continuing up to the present day.

While the disconnect between research and practice perhaps hurts practitioners less than researchers, they too have recognized this missed opportunity. The classic book “Software Creativity 2.0” (Glass 2006) dedicated two chapters to “theory versus practice” and “industry versus academe” and presented several examples (which the author believes are “disturbing”) on the mismatch of theory and practice (referring to academia and industry, respectively). An interesting blog called “It will never work in theory” (www.neverworkintheory.org) summarized the status-quo on the issue of the IAC as follows: “Sadly, most people in industry still don’t know what researchers have found out, or even what kinds of questions they could answer. One reason is their belief that software engineering research is so divorced from real-world problems that it has nothing of value to offer them”. The blog further stated that: “Instead of just inventing new tools or processes, describing their application to toy problems in academic journals, and then wondering why practitioners ignored them, a growing number of software development researchers have been looking to real life for both questions and answers”.

Another recent trend among practitioners, perhaps indicating their willingness to leverage high-quality research, is the creation of reading groups specifically designed to read, discuss, and present academic papers that could impact their work. This movement, broadly known as “Papers we love” (www.paperswelove.org), has groups in over forty major cities. However, after reviewing the papers read and presented in the above community, at least as of this writing, we found that almost all papers are on theoretical computer sciences topics (such as databases and algorithms) and we did not find any papers on SE being the subject of presentation/discussions among that community.

In summary, we observe that, while perhaps our communities’ history of collaboration has been weak, the enthusiasm on both sides makes this an ideal time to systematize and increase our efforts. Towards this end, the challenges, patterns (i.e., the best practices that promise success), and anti-patterns (what not to do) in IAC projects were recently synthesized in a Systematic Literature Review (SLR) (Garousi et al. 2016a). Taking those results as an input, the goal of the study reported in this article is to characterize IAC projects with respect to the challenges, patterns, and anti-patterns identified by the SLR. To address this goal, we conducted a worldwide opinion survey to gather the data from researchers and practitioners. In summary, this article makes the following contributions:

  • A comprehensive IAC-focused empirical study based on evidence and quantitative assessments of challenges, patterns, and anti-patterns (Garousi et al. 2016a)

  • A quantitative ranking of the challenges, patterns, and anti-patterns in a large set of IAC projects internationally (across 101 projects and in 21 countries)

  • A set of evidence-based recommendations to ensure success and to prevent problems in IAC projects

The rest of this article is structured as follows. In Section 2, we present a review of the related work. In Section 3, we describe the context of our study and review existing process models for IACs in SE. In Section 4, we introduce the study goal, research questions and research methodology. In Section 5, we discuss demographics of our study’s dataset. In Section 6, we present the answers to our study’s RQs. Finally, in Section 7, we draw conclusions and suggest areas for further research.

2 Background and Related Work

In this section, we first provide an overview of the related work. Afterwards, to establish a theoretical foundation for our work, we review the existing theories and models of IACs.

2.1 Related work

A recent SLR (Garousi et al. 2016a) synthesized the body of literature on the subject of IAC projects in SE by reviewing a set of 33 papers in this area, e.g., (Petersen et al. 2014a; Sandberg et al. 2011; Lamprecht and Van Rooyen 2012). The SLR derived a list of 64 challenges, 128 patterns, and 36 anti-patterns for IAC projects. Among the 33 papers reviewed in (Garousi et al. 2016a), 17 studies reported the number of projects that the observations were based on. There were on average 4.8 projects reported in each paper (the range was from 1 to 22 projects). While the SLR shared insightful experience and evidence on the topic, we believe that the SE community still lacks the following two types of empirical evidence: (1) most of the experience is reported by focused (single) teams of researchers and practitioners and there is a need for evidence based on a larger, more distributed set of IAC projects to reduce the sampling bias; (2) challenges, success patterns, and anti-patterns in IAC projects have been reported rather sparsely and sporadically and there is a need for more systematic synthesis.

Aside from the SLR, while many other studies, e.g., (Petersen et al. 2014a; Sandberg et al. 2011; Lamprecht and Van Rooyen 2012), discuss challenges and success patterns in IAC projects, they report results from one or a few projects in local contexts. The current article aims to provide a larger-scale snapshot on the state of IAC projects, sampled from several countries.

In another survey, Wohlin et al. (2012) investigated the success factors for IAC in SE. Overall, 48 researchers and 41 practitioners from Sweden and Australia participated in the survey. The most important lessons from the study are that (1) buy-in and support from company management is crucial, (2) there must be a champion on the industrial side (company), i.e., someone who is passionate about the IAC and is driving it, and not only a person who merely has been “assigned” the responsibility to coordinate with the research partner, (3) different categories of people have different views on the purpose and goals of the IAC, and (4) social skills are important, particularly if a long-term collaboration shall be established. Different from Wohlin et al.’s survey (Wohlin et al. 2012), the units of analysis in our dataset are research projects, and not individuals. Furthermore, our study is not limited to success factors but, in addition, investigates challenges, success patterns, and anti-patterns.

Other empirical studies on IAC have been reported in other fields such as management (Barnes et al. 2002; Barbolla and Corredera 2009). For example, the study presented in (Barbolla and Corredera 2009) assesses the most influential factors for success or failure in research projects between university and industry. The study is based on interviews with 30 university researchers. It concludes that the company’s real interest and involvement during an IAC project, its capacity to assimilate new knowledge, and a confident attitude towards the participating university researchers are the crucial factors for assuring a successful collaboration.

Another study published in 2017 (Ivanov et al. 2017) is a paper entitled: “What do software engineers care about? Gaps between research and practice”. The authors surveyed software engineers with regard to what they care about when developing software. They then compared their survey results with the research topics of the papers recently published in the ICSE/FSE conference series. The authors found several discrepancies. For example, while software engineers care more about software development productivity than software quality, papers on research areas closely related to software productivity – such as software development process management and software development techniques – have been significantly less often published than papers on software verification and validation, which account for more than half of publications. The study also found that software engineers are in great need for techniques for accurate effort estimation, and they are not necessarily knowledgeable about techniques they can use to meet their needs.

One of the research questions (RQs) in this article (see Section 4.1) assesses the industrial impacts of the surveyed IAC projects. Previous efforts to this issue have been reported, e.g., the ACM SIGSOFT Impact project (n.d.; Osterweil et al. 2008), which, according to its website (ACM SIGSOFT, "SIGSOFT Impact project n.d.), was active in the period of 2002–2008. Several papers were authored in the context of the Impact project which synthesized and reported the impact of research on practice, e.g., one in the area of software inspections, reviews and walkthroughs (Rombach et al. 2008), and another about the impact of research on middleware technology (Emmerich et al. 2007).

This article is a follow-up to a recent conference paper (Garousi et al. 2017a) and extends it substantially in the following ways: (1) our previous study was based on data from only 47 projects while, based on a follow-up survey, this article is based on a larger dataset (101 projects); and (2) only a few aspects of data and demographics were previously reported, while more detail is reported in this article.

The current work also builds upon another paper co-authored by the first author and his colleagues (Garousi et al. 2016b) in which a pool of ten IAC projects conducted on software testing in two countries (Canada and Turkey) were analyzed with respect to challenges, patterns, and anti-patterns. A set of empirical findings and evidence-based recommendations have been presented in (Garousi et al. 2016b). For example, the paper reports that even if an IAC project may seem to possess all the major conditions to be successful, one single challenge (e.g., confidentiality disagreements) can lead to its failure. As a result, the study recommended that both academics and practitioners should consider all the challenges early on and proactively work together to eliminate the risk of encountering a challenge in an IAC project. While there are slight similarities between (Garousi et al. 2016b) and the current article, the set of RQs and the foci of the two publications differ. Paper (Garousi et al. 2016b) was based on ten IAC projects in software testing in two countries, while this paper is based on 101 projects in all areas of SE in 21 countries.

2.2 Theories and Models of Industry-Academia Collaborations

There exists a large body of literature about IAC in fields like management science and research policy, e.g., (Vedavyas 2016; Al-Tabbaa and Ankrah 2016; Lin 2017; Huang and Chen 2017), and also in SE (see the survey paper in (Garousi et al. 2016a)). A search for the phrase “(industry AND academia) OR (university AND industry)” in paper titles in the Scopus academic database (www.scopus.com), on March 15, 2018, returned 3371 papers, denoting the scale of attention to this important issue in the scientific community in general. Papers on IAC could be classified into two categories: (1) papers that propose heuristics and evidence to facilitate IAC, e.g., (Al-Tabbaa and Ankrah 2016; Huang and Chen 2017); and (2) papers that propose theories and models for IAC, e.g., (Nagaoka et al. 2014; Shimer and Smith 2000; Carayol 2003; Mindruta 2013; Banal-Estañol et al. 2017; Banal-Estañol et al. 2013; Ankrah and Al-Tabbaa 2015; Simon 2008).

To establish and conduct an effective IAC, the collaboration partners (researchers and practitioners) need to understand the underlying important concepts and theory (how, why, and when) behind the motivations, needs, and factors involved in a typical IAC. In their paper entitled “Where’s the theory for software engineering,”, Johnson et al. write that “To build something good, you must understand the how, why, and when of building materials and structures” (Johnson et al. 2012). Also, understanding and utilizing theories of IAC provides us with a solid foundation for designing our own research method and opinion survey used in this study (see Section 4). Johnson et al. state that most theories (explanations) in SE have three characteristics (Johnson et al. 2012): (1) they attempt to generalize local observations and data into more abstract and universal knowledge; (2) they typically represent causality (cause and effect); and (3) they typically aim to explain or predict a phenomenon. On a similar note, a highly-cited study in the Information Systems (IS) domain, which assessed the nature of theory in information systems, distinguished several types of theories (Gregor 2006): (1) theory for analyzing, (2) theory for explaining, (3) theory for predicting, and (4) theory for design and action. Thus, having an initial theoretical basis for IAC in SE can help us explain and characterize IAC as a phenomenon, and facilitate analysis of causality (cause and effect), e.g., helping us decide what practices have the potential of yielding more success in IAC.

We provide in the following a review of the existing studies that proposed theories and models for IAC (Nagaoka et al. 2014; Shimer and Smith 2000; Carayol 2003; Mindruta 2013; Banal-Estañol et al. 2017; Banal-Estañol et al. 2013; Ankrah and Al-Tabbaa 2015; Simon 2008). The study reported in (Nagaoka et al. 2014) focused on sources of “seeds” and “needs” in IAC and their matching process. Seeds were defined as “the technology which served as the base for cooperative research” and needs were defined as “specific use envisaged for the output of the joint research” (Nagaoka et al. 2014). The study focused on several research questions including: (1) how important are the seeds and needs for initiating IACs?; and (2) does matching based on efficiency criteria (the research capability of a partner and the good fit between industry and academic) result in a successful IAC? It then argued that there often exist specific seeds and needs motivating a given IAC project and presented a simple analytic model to quantify the output from collaboration between industry and academic partners. The study also used the “assortative matching” theory (Shimer and Smith 2000) to characterize the matching process between partners. Assortative matching is a matching pattern and a form of selection in which partners with similar objectives match with one another more frequently than would be expected under a random matching pattern (Shimer and Smith 2000).

In 2003, a paper published in the Journal on Research Policy proposed a typology of IAC and argued that firms involved in high (low) risk projects are matched with academic teams of high (low) excellence (Carayol 2003). The authors collected a list of 46 IAC projects in Europe and the United States. An outcome of the study was a typology of IAC built on a formal procedure: a multi-correspondence analysis followed by an ascendant hierarchical classification. The typology exhibited five types of collaborations, positioned inside circles on a 2D plane, in which the x-axis is the risk, novelty and basicness of research, and the y-axis corresponds to the research platform (number of partners), which goes from bilateral research to networked research.

A study published in 2012, entitled “Value creation in university-firm research collaborations: a matching approach”, explored the partner attributes that drive the matching of academic scientists and firms involved in IAC (Mindruta 2013). The study modeled the formation of IAC as an endogenous selection process driven by synergy between partners’ knowledge-creation capabilities and identified ability-based characteristics as a source of complementarity in IAC.

Banal-Estañol et al. developed a theoretical matching model to analyze IACs (Banal-Estañol et al. 2017). The model predicts a positive assortative matching (Shimer and Smith 2000) in terms of both scientific ability and affinity for type of research. The study suggests that “the most able and most applied academics and the most able and most basic firms shall collaborate rather than stay independent”. Before deciding whether to collaborate, academics and firms weigh the benefits in terms of complementarities and the costs in terms of divergent interests. Recent evidence stresses the importance of the characteristics of the matched partners in assessing collaboration outcomes. Banal-Estañol et al. showed in (Banal-Estañol et al. 2013) that the research projects in collaboration with firms produce more scientific output than those without them, if and only if the firms in the project are research-intensive.

The theoretical model developed in (Banal-Estañol et al. 2017) considers and analytically models all the important factors in IAC, e.g., investment (time and money) levels and outcome of projects, which were modeled as follows. When an academic or firm runs a project on their own, the number of positive results (or the probability of obtaining a positive result) depends on its own ability and investment. It was modeled by TAIA and TFIF, where TA (resp. TF) represents the academic’s (resp. firm’s) technical ability, or efficiency, and IA (resp. IF) represents the academic’s (resp. firm’s) investment level. The parameter TA measures the technical and scientific level of a given academic, her publications, the patents and know-how she owns, the quality of the research group (lab) she works in, etc., whereas the parameter TF measures the scientific level of a given firm, its absorptive capacity, the level of its human capital, etc. The theoretical model was then applied to a set of 5855 projects in a project database of the UK’s Engineering and Physical Sciences Research Council (EPSRC) and the predictions provided by the model received “strong support” by the teams of involved academics and firms (Banal-Estañol et al. 2017).

In management science, a SLR on the topic of IAC was published in 2015 (Ankrah and Al-Tabbaa 2015). The SLR reviewed a pool of 109 primary studies and investigated the following RQs: (1) What are the organizational forms of IACs?; (2) What are the motivations for IACs?; (3) How are IACs formed and operationalized?; (4) What are the factors that facilitate or inhibit the operation of IACs?; and (5) What are the outcomes of IACs? The SLR identified five key aspects that underpin the theory of IAC: necessity, reciprocity, efficiency, stability, and legitimacy. The SLR showed that, in the IAC literature, researchers emphasize the role of interdependency and interaction theories in the genesis, development and maintenance of IAC. Interdependency theories stress the impact of the external environment on the formation of IAC, while interaction theories explore the internal development and maintenance of IAC. Furthermore, the SLR (Ankrah and Al-Tabbaa 2015) argued that various perspectives and theories have been widely used in IAC, including transaction costs economics, resource dependency, strategic choice, stakeholder theory, organizational learning, and institutional theory. Transaction Cost Economics (TCE) assumes that transaction (or economic exchange) is the basic unit of analysis for an organization’s economic relationships, where these relationships are sought to reduce production cost and increase efficiency. Therefore, it may provide an explanation why universities and companies are inclined to engage in IAC, i.e., minimize the sum of their technology development costs. Finally, by synthesizing the pool of 109 primary studies, the SLR (Ankrah and Al-Tabbaa 2015) presented a conceptual process framework for IAC, as shown in Fig. 1.

Fig. 1
figure 1

A conceptual process framework for IAC (source: (Ankrah and Al-Tabbaa 2015))

Another study, published in the European Journal of Innovation Management, proposed a process model for IAC which “can be utilized by practitioners from both academia and industry in order to improve the process of research collaboration and facilitate more effective transfer of knowledge” (Simon 2008). This study highlighted the social interactions assuming that “social capital can be regarded as an important factor when developing collaborations”. The process model, as proposed in (Simon 2008), is shown in Fig. 2. This process model resembles the process framework presented in (Ankrah and Al-Tabbaa 2015) (Fig. 1) in terms of the process flow, with the exception that the former has an extra phase called “Terrain mapping” in the beginning. As discussed in (Simon 2008), mapping of IAC terrain is the initial process stage where industry and market analysis is undertaken in order to develop a detailed understanding of the “collaboration opportunity landscape”. This analysis should initially be broad-based, but as requirements are understood in more detail this should lead to more focused activities. If possible, this information gathering exercise should be extended to include industry’s current needs that could be gained through person-to-person interactions and networking. Some of the authors of the current paper have experience with terrain mapping activities, e.g., in a set of 15+ software-testing projects in Canada and Turkey (Garousi et al. 2016b), and experience with selecting the “right” topics for an IAC (Garousi and Herkiloğlu 2016).

Fig. 2
figure 2

A process model for IAC (source: (Simon 2008))

Furthermore, compared to the model in Fig. 1, the model in Fig. 2 has four additional components: (1) social capital, (2) technical mission, (3) business mission, and (4) collaboration agent. In this context, social capital corresponds to the networks of relationships among participants in an IAC who collaborate and enable the IAC to execute effectively. It includes factors such as familiarity, trust, a common understanding, and a long-term commitment to collaboration. Technical mission and business mission are quite self-explanatory, i.e., an IAC should create “value” both in terms of technical and business missions. Collaboration agent is a role or individual who personally drives forward the collaboration and is responsible for achieving the required objectives in order to initiate and deliver the collaboration. In the recent SLR on IAC in SE (Garousi et al. 2016a), the term “champion” was used as a synonym for the term “collaboration agent”.

Albeit the slight difference in the terminology used, there are other semantic similarities between the two models in Figs. 1 and 2, e.g., the process flow are almost the same as an IAC usually starts from “proposition “or “formation” phase. In this stage, parties aligned the university’s research offering to the company’s strategy and needs and specifically to the technology development plans for the relevant products and services that are delivered by the company (Simon 2008). The IAC then continues to the next phases and finished in the “evaluation” or “outcomes” phase, in which benefits of IAC are actually implemented and, usually, a formal or informal post-project evaluation is conducted by both sides. “Motivations” are the need drivers of an IAC in Fig. 1, while in Fig. 2, the terms “technical” and “business missions” are used to refer to the same concept.

Another interesting model to assess research “closeness” of industry and academia was proposed by Wohlin in (Wohlin 2013a). In a talk entitled “Software engineering research under the lamppost “(Wohlin 2013a), Wohlin presented, as shown in Fig. 3, five levels of closeness between industry and academia, which could be seen as a maturity model. IAC projects usually take place in Level 5 (One team) and sometimes in Level 4 (Offline).

Fig. 3
figure 3

Five (maturity) levels of closeness between industry and academia (source: (Wohlin 2013a))

In Level 5, the IAC is indeed done in “one team”, a specific industrial challenge is identified, draft solutions are evaluated and validated in iterations and final solutions are usually implemented (adopted) in practice. In Level 4, the IAC is offline and often remote”. As in Level 5, a specific industrial problem is identified but the solution is done offline, or rather remotely, in academia. Once ready, a “pre-packaged” solution is offered that is challenging to implement (adopt) in industry due to its generality.

In Level 1 (Not in touch), Level 2 (Hearsay), and Level 3 (Sales pitch), the linkage between industry and academia is non-existent or weak, and thus one cannot refer to the linkage as a proper IAC.

3 Initial Context and Process Models for Industry-Academia Collaborations in SE

To put our study in context, we present a domain model (context diagram) and provide definitions for the terms used in this context.

Figure 4 depicts a UML class diagram representing a typical domain model (context diagram) for IAC projects. Note that, for brevity, this diagram does not include the cardinality details. Researchers and practitioners participate in a given IAC project. Either of them or both could act as the initiator. There is usually a need that drives the project offering one or more solutions with impact. Solutions are, in fact, the contributions of an IAC project to its industrial partner(s). Solutions are expected to have (positive) impact in the industrial context, e.g., an example solution could be a new software refactoring method for a company providing positive impact by saving software maintenance costs. To keep our study focused, we only consider industrial impact and do not consider “academic” impact (Poulding et al. 2015) of an IAC.

Fig. 4
figure 4

A domain model for IAC projects

There is at least one object of study in the form of a (software) process or a software system. For example, an IAC project may target improving software testing processes of a given company. Papers are usually written as a result of the project. Funding sources may support the project. Partners involved in an IAC project naturally expect the project to be successful. The level of success is assessed by a set of success criteria, which are defined (at least implicitly) by the partners, e.g., publication of papers, training of PhD students and young researchers, getting insights, lessons learned, or new research ideas, and solving the need that triggered the project in the first place.

In terms of conceptual terminology, the scope of a typical IAC project might not be immediately clear. We use in this study the “project” notion in the same way as typically used by funding agencies, e.g., national agencies, such as the NSERCFootnote 1 in Canada or TÜBİTAKFootnote 2 in Turkey, or international agencies, such as the European Commission’s Horizon-2020 program.Footnote 3 An IAC project can take various forms, e.g., technology transfer and consultancy, but there should be some sort of research involved in it, to make it within the scope of our definition in this paper. A SE IAC project is a project in which at least one academic partner and at least one industrial partner formally define a SE-related research topic.

The trigger for an IAC project is usually a real industrial “need” (or challenge), e.g., improving test automation practices in a company (Garousi and Herkiloğlu 2016), or is based on academic research, e.g., assessing the benefits of software documentation using UML. As a concrete example may serve one of the authors’ action-research IAC (Santos and Travassos 2009; Petersen et al. 2014b; Stringer 2013) conducted with a software company in Turkey. Early in the collaboration process, the partners systematically scoped and defined a set of topics to work on (Garousi and Herkiloğlu 2016), e.g., (1) increase test automation, (2) assess and improve an in-house test automation framework, (3) establish a systematic, effective and efficient GQM-based measurement program for the testing group, and (4) assess and improve test process maturity using TMMi (Garousi et al. 2018a).

The presented overview of existing work about IAC theory (Nagaoka et al. 2014; Shimer and Smith 2000; Carayol 2003; Mindruta 2013; Banal-Estañol et al. 2017; Banal-Estañol et al. 2013; Ankrah and Al-Tabbaa 2015; Simon 2008) enables us to lay a solid foundation for designing our research method (see Section 4). Various models have been presented (e.g., see Figs. 1 and 2) and while there are many similarities across different models, there does not exist one single unified model. We should mention that each model usually takes a certain view (perspective) on the nature of IAC or highlights certain aspects. For example, both (Ankrah and Al-Tabbaa 2015; Simon 2008) took a process view but while the former highlighted the issues of motivations and facilitating/impeding factors, the latter highlighted social capital and collaboration agent.

In our study, we focus on the process aspect and on cause/effect-relationships in IAC within the SE domain. In addition, we incorporate the set of challenges and patterns provided by the IAC SLR published in (Garousi et al. 2016a).

Thus, we consider the models presented in (Ankrah and Al-Tabbaa 2015; Simon 2008) as our baseline and extend/adapt them to fit our purpose, as illustrated in Fig. 5. We synthesized our process model from three sources: (1) the models presented in (Ankrah and Al-Tabbaa 2015; Simon 2008); (2) our experience in IAC, e.g., (Garousi et al. 2016b); and (3) the SLR study published in (Garousi et al. 2016a). In our study, we use this process model to understand and characterize IAC in a way inspired by the authors of (Kemper 2013) who stated that “… a way to evaluate a theory is by its derivations, that is, what does the theory help us to understand?”. Note that our model is not a collaboration model (like those discussed in (Petersen et al. 2014a; Sandberg et al. 2011; Gorschek et al. 2006)) but a process model for IAC projects, including important factors of interest to our study (e.g., collaboration need, challenges and patterns). We do not claim this model to be a unified complete model for IAC within the SE domain. We rather see it as an initial step towards such a model corresponding to our needs in this study.

Fig. 5
figure 5

A typical (simplified) process for IAC projects (inspired by (Ankrah and Al-Tabbaa 2015))

According to the grounded theory technique (Corbin and Strauss 2014), if the dynamic and changing nature of events is to be reflected in a process model, then both structure and process aspects must be considered. Therefore, the model in Fig. 5 is centered in a linear process for collaboration but also supported by the structural elements, i.e., the cross-cutting concerns such as challenges and patterns, need for collaboration, outputs, results and contributions to the literature, and impact on the software project or product under study. The process model has four phases: (1) Inception: team building and topic selection; (2) Planning: defining the goal, scope, etc.; (3) Operational: running, controlling and monitoring; and (4) Transition: technology/ knowledge transfer and impact.

Three fundamental concepts related to IAC projects are depicted in Fig. 5 (marked with gray backgrounds): industrial needs, developed solutions, and impacts. IAC projects mostly are started and executed based on industrial needs (Garousi et al. 2016a; Garousi and Herkiloğlu 2016). Throughout the project, partial or full solutions are developed which are expected to address that need (represented by the link between “solution” and “need” in Fig. 5). The developed solution(s) is (are) expected to have positive impacts on the studied context (a project or a case under study).

4 Research Goal and Method

We discuss in this section the study goal, research questions, study context, and research method.

4.1 Goal and Research Questions

Formulated using the Goal, Question, Metric (GQM) approach (Solingen and Berghout 1999), the overall goal of this study is to characterize a set of IAC projects in SE, with respect to the challenges, patterns, and anti-patterns identified by the SLR study (Garousi et al. 2016a). Our study contributes to the body of evidence in the area of IAC, for the benefit of SE researchers and practitioners in conducting successful projects in the future. Based on the overall goal, we raised the following research questions (RQs):

  • RQ 1 (focusing on technical SE aspects of projects)- What types of industrial needs initiated the IAC projects under study, what solutions were developed, and what industrial impacts the projects provided?

  • RQ 2 (focusing on operational aspects of projects)- To what extent did each challenge, reported in the SLR study (Garousi et al. 2016a), impact the IAC projects?

  • RQ 3 (focusing on operational aspects of projects)- To what extent did each pattern and anti-pattern impact the IAC projects?

Note that, compared to our previous paper (Garousi et al. 2017a), RQ 1 has been added. Both RQ 2 and 3 were partially addressed in (Garousi et al. 2017a) but without in-depth analysis. Also, the analyses in (Garousi et al. 2017a) were based on a smaller dataset compared to the current article. Furthermore, we conduct and report additional analyses in this paper, e.g., an in-depth analysis of the demographics of the dataset, and an in-depth analysis of how the challenges affected the projects. Thus, this article is a substantial extension of (Garousi et al. 2017a).

Furthermore, we believe this work makes a novel contribution to the community by studying both the technical SE aspects of a large set of IAC projects via RQ 1 (needs, solutions and impacts), as well as their operational aspects and characteristics via RQs 2 and 3 (challenges, patterns and anti-patterns).

4.2 Research Method (Survey Design)

To answer the study’s RQs, we designed and conducted an opinion survey. Our goal was to gather as many data points (opinions) from researchers and practitioners world-wide. Table 1 shows the structure of the questionnaire used to conduct the survey. To provide traceability between the questionnaire and the RQs, we also show in Table 1 the RQs addressed by each part of the questionnaire. Furthermore, we designed the survey in a way to fully match the IAC process model in Fig. 5. Due to space constraints, we do not provide the full survey questionnaire, as presented to participants, in this paper, but it can be found as a PDF file in an online source (Garousi et al. 2016c).

Table 1 Structure of the questionnaire used for the survey

In designing the survey, we benefitted from the survey guidelines in SE (Molleri et al. 2016). Some example survey guidelines that we utilized from (Molleri et al. 2016) are as follows: (1) identifying the research objectives, (2) identifying and characterize target audience, (3) designing sampling plan, (4) designing the questionnaire, (5) piloting test questionnaire, (6) distributing questionnaire, and (7) analyzing results and writing the paper. We also used the recommendations from w.r.t characterizing units of observation, units of analysis, establishing the sampling frame and recruitment strategies (Mello et al. 2014a, b).

We were also aware of validity and reliability issues in survey design and execution (Molleri et al. 2016). One aspect of the survey’s validity, in this context, is how well the survey instrument (i.e. the questions) measures what it is supposed to be measured (construct validity). External validity of a survey relates to the representativeness of the results for the population from which respondents are sampled. The reliability of a survey refers to the question whether a repeated administration of the questionnaire at different points in time to the same group of people would result in roughly the same distribution of results each time. We dealt with those validity issues in both the survey design and execution phases.

It was intended that respondents would respond to the questionnaire with respect to each single IAC project they had participated in. The unit of analysis in our survey is a single IAC project. Therefore, a participant could provide multiple answers; each one for a single project he or she was involved in. The considered IAC projects could be completed, (prematurely) aborted or ongoing (near completion). We included aborted projects in the survey and its dataset so that we could characterize the factors leading to abnormal termination of IAC projects.

Part 1 of the questionnaire has 11 questions about demographics (profile) of the respondent and the IAC project. Part 2 of the questionnaire asked about the needs, developed solutions, and impact of the IAC project. Part 3 asked about the challenges, patterns, and anti-patterns in the project, as adopted from the SLR study published in (Garousi et al. 2016a). For example Q3.1 asked the participants to “rate the extent of the negative effect each of the following challenges had on the industry-academia collaboration in the project” and listed ten categories of challenges (again, adopted from the SLR (Garousi et al. 2016a)):

  1. 1.

    Lack of research relevance (LRR)

  2. 2.

    Problems associated with the research method (RM)

  3. 3.

    Lack of training, experience, and skills (LTES)

  4. 4.

    Lack or drop of interest / commitment (LDRC)

  5. 5.

    Mismatch between industry and academia (MIA)

  6. 6.

    Communication-related challenges (CRC)

  7. 7.

    Human and organizational challenges (HOC)

  8. 8.

    Management-related challenges (MRC)

  9. 9.

    Resource-related challenges (RRC)

  10. 10.

    Contractual, privacy and IP (intellectual property) concerns (CPC)

We asked participants about the negative impact of each challenge in their projects using a five-point Likert scale: (0): no impact, (1): high negative impact, (2): moderate negative impact, (3): high negative impact, and (4): very high negative impact. We asked similar questions to gather scale data for 15 categories of patterns and four categories of anti-patterns, as adopted from the SLR (Garousi et al. 2016a) and listed below:

  1. 1.

    Proper and active knowledge management (PAKM)

  2. 2.

    Ensuring engagement and managing commitment (ENMC)

  3. 3.

    Considering and understanding industry’s needs, and giving explicit industry benefits (CUIN)

  4. 4.

    Having mutual respect, understanding and appreciation (HMRU)

  5. 5.

    Being Agile (BA)

  6. 6.

    Working in (as) a team and involving the “right” practitioners (WTI)

  7. 7.

    Considering and manage risks and limitations (CMRL)

  8. 8.

    Researcher’s on-site presence and access (ROSP)

  9. 9.

    Following a proper research/data collection method (FPRM)

  10. 10.

    Managing funding/recruiting/partnerships and contracting privacy (MFRP)

  11. 11.

    Understanding the context, constraints and language (UCCL)

  12. 12.

    Efficient research project management (ERPM)

  13. 13.

    Conducting measurement/ assessment (CMA)

  14. 14.

    Testing pilot solutions before using them in industry (TPS)

  15. 15.

    Providing tool support for solutions (PTS)

  16. 16.

    (Anti-pattern): Following self-centric approach (FSCA)

  17. 17.

    (Anti-pattern): Unstructured decision structures (UDS)

  18. 18.

    (Anti-pattern): Poor change management (PCM)

  19. 19.

    (Anti-pattern): Ignoring project, organizational, or product characteristics (IPOP)

For more details about each of the above patterns, the reader may refer to the SLR (Garousi et al. 2016a). Part 4 of the survey included five questions about outcome, success criteria and success levels of the projects. To keep the current paper focused, we are not including any data nor raise any RQs about those aspects, and plan to analyze those parts of our dataset in future papers.

4.3 Validation of the Survey Design

As mentioned above, construct validity was an important issue and we ensured that the survey instrument (i.e., the questions) would measure what our study intended to measure. Parts 2 and 3 of the survey were explicitly designed to ensure a direct mapping with the study goal and its associated RQs (Section 4.1). We designed questions in each survey section (part) to gather data about the following aspects of a typical IAC project: Part 2 focused on need, developed solutions, and impact of the project. Part 3 focused on challenges, patterns and anti-patterns in the projects.

To ensure construct validity, we conducted two rounds of pilot applications of the questionnaire used in the survey, first among the authors and then, in addition, with five practicing software engineers selected from our industry network. The main issue we considered in the pilot phase was to ensure that the survey questions would be understood by all participants in the same manner.

We were also aware about the importance of reliability/repeatability of the survey instrument. We applied the test-retest reliability check (Kline 2013) for this purpose. We asked two participants (who had provided their emails addresses in the main data collection phase), one practitioner and one researcher, to re-fill the survey. The second time of filling the survey by those two participants was about 1 year after the first time (Fall 2018 versus Fall 2017, see the next section). For measuring reliability for two tests, we calculated the Pearson correlation coefficient of the numerical data fields in the survey (e.g., challenge Likert scales), as suggested in the statistics sources (Kline 2013). The correlations for the two participants were 0.85, and 0.72; and the average value was 0.78, which is interpreted as an “acceptable” reliability measure for survey instruments (Kline 2013).

4.4 Survey Execution and Data Collection

For data collection, we sent invitations by email to SE researchers and practitioners who were known in the community to be active in IAC projects and to the authors of the primary studies reviewed in the SLR (Garousi et al. 2016a). The survey was anonymous, but the participants could provide their names and emails if they wanted to receive the results of our study. The total number of invitations and the resulting response rates are discussed further below.

Our sampling method was convenience sampling which is the dominant approach chosen in survey and experimental research in SE (Sjoeberg et al. 2005). Albeit its drawbacks and potential risk of bias in the data, this does not mean that convenience sampling is generally inappropriate (Ferber 1977). Convenience sampling is also common in other disciplines such as clinical medicine and social sciences (e.g. (Kemper and Stringfield 2003)).

We are aware of the importance of external validity and reliability of the survey results and instruments, i.e., representativeness of the dataset and appropriateness of the data sampling (Gobo 2004). There have been many discussions about advantages and disadvantages of convenience sampling, e.g., (Gobo 2004; Sackett and Larson 1990). Regarding its limitations, it has been said that “because the participants and/or settings are not drawn at random from the intended target population and universe, respectively, the true representativeness of a convenience sample is always unknown” (Sackett and Larson 1990). At the same time, researchers have recommended two alternative criteria to explore the external validity of convenience samples: sample relevance and sample prototypicality (representativeness) (Sackett and Larson 1990). Sample relevance refers to the degree to which membership in the sample is defined similarly to membership in the population. For instance, an example of sample irrelevance taken from a field outside SE, would be a study of executive decision making conducted with a sample of university student (Sackett and Larson 1990). There exist also studies about this issue in SE, e.g., (Höst et al. 2000). Sample prototypicality refers to the degree to which a particular research case is common within a larger research paradigm. An example of prototypicality would be a study exploring the benefits of software design patterns; although a sample of senior executives completing such a survey could be collected, a sample of “technical staff”, i.e., software developers, would be more prototypical of when such benefits would actually be observed. With sample representativeness, sample relevance is assumed (Sackett and Larson 1990). In summary, while using convenience sampling in our work (similar to many other survey studies in SE) the representativeness (and thus external validity) of our study results could be limited, we still ensured meeting the other two external validity aspects, i.e., relevance and sample representativeness, since we sent the survey to researchers and practitioners who have been active in IAC projects and have first-hand experience of initiating and conducting IAC projects.

Data collection via the online survey was conducted in two phases. The first phase was conducted in Fall 2016. The second phase was conducted in Fall 2017. In the first phase, we sent invitations to a large number of SE researchers and practitioners (about 100) who were known in the community to be active in IAC projects and the authors of the primary studies reviewed in the SLR (Garousi et al. 2016a). About two-thirds (2/3) of the (100) invitations were sent to researchers, while the rest (1/3) were sent to the practitioners in our network. Unfortunately, we received a response rate from the SE community (only 11 data points). The response rate was 9.1%. Since we (the authors of this study) have also been active in IACs, we also provided data points related to our past/current projects. In total, during the first phase, the authors of this study contributed 36 data points, creating a dataset with a total of 47 data points. We reported an initial analysis based on those 47 data points in a conference paper (Garousi et al. 2017a).

The second phase of data collection was conducted in Fall 2017, in which we sent 150 invitations. Similar to the phase #1, the recipients were again the researchers and practitioners who were known in the community to be active in IAC projects. About 100 of the invitations were sent to the same pool of the recipients, as we had sent in phase #1. We developed an additional set of 50 researchers and practitioners in the phase #2. Similar to the first phase, about two-thirds (2/3) of the 150 invitations in the second phase were sent to researchers, while the rest (1/3) were sent to the practitioners in our network. In the second phase of data collection, we were more proactive in our survey invitation strategy (e.g., we personally emailed and reminded our collaborators to fill out the survey) and the response rate (32.7%) increased compared to the first phase (9.1%). In the expanded dataset, 60 data points were from our invited participants in addition to the 47 data points provided by the author team. Since the study’s authors all have been active in IACs throughout their careers, it was natural for us to also include the data points from the author team, since we could get a more enriched dataset. When we entered the data into the questionnaire, we ensured treating it with outmost care and seeing ourselves as independent participants to prevent any bias in data collection. Each co-author contributed between 3 and 19 data points. Details about the composition and evolution of the dataset are shown in Table 2.

Table 2 Details and statistics on the composition of the dataset

To ensure the quality of data, we screened the 107 raw data points. One data point was excluded since one respondent had entered one single data point as a proxy (aggregate) for her/his IAC projects in all her/his entire career and thus the provided measures were not valid for our survey. We excluded five more data points since the only reported research method was a practitioner-targeted survey, which cannot be considered an actual IAC. Thus, the final dataset contained 101 projects after screening.

As shown in Table 2, in the final dataset, 46 data points were from the study authors, and 55 data points from the community at large (i.e., not from the authors of this study). In the rest of this article, we refer to the projects using the labeling of Pi, with i ranging from 1 to 101. These IDs are indicated in the dataset file.

We also wondered about how many respondents provided the information on the 101 projects. We had some identifying information of the respondents (e.g., emails) and used them to gather this information. In total, 64 respondents provided the information on the 101 projects. Each respondent provided between 1 and 19 data points. A majority of the respondents (57 people) provided only one data point, thus we can say that a large number of data points came from different people.

For transparency and to benefit other researchers, after removing identification and sensitive information about the projects, we have shared the raw dataset of the survey publicly in online sources (phase #1 in (Garousi et al. 2018b), and phase #2 in (Garousi et al. 2018c)). The full survey questionnaire can be found as a PDF file in an online source (Garousi et al. 2016c).

4.5 Data Analysis Method

We used both quantitative and qualitative analysis techniques. Many questions in our survey instrument are closed questions. Thus, we could apply simple descriptive statistics and visualization techniques (e.g., bar charts, histograms, boxplots, and individual value plots) to analyze and visualize the data received from the survey participants.

Answers to open-ended questions were analyzed using the qualitative coding (synthesis) technique (also called, “open/axial” coding) (Miles et al. 2014). For one of the questions (“needs” addressed in the projects), we also using the word clouds technique to visualized the responses (results in Section 6.1.1).

Qualitative coding of each open-ended question was done by one co-author, and was peer reviewed by one other author at least, to ensure highest quality and to prevent bias. In the case of conflicting opinions by different authors, we had planned to conduct group discussions to reach consensus (but this never happened).

We provide below an example of how we conducted the qualitative analysis by showing how the analysis was done on one of the open-ended about industrial impacts of the projects (Section 6.1.3). Free-text responses for 92 of the 101 data points were provided for that question by the respondents. We used qualitative coding (Miles et al. 2014) to classify industrial impacts of the projects into three categories:

  • (1) Positive impacts on the industry partner, backed by quantitative evidence (measures) in the provided response;

  • (2) Positive impacts, backed by qualitative statements; and

  • (3) No impacts on industry (yet), work in the lab only, or “not sure”.

Qualitative coding (synthesis) (Miles et al. 2014) is a useful method for data synthesis and has been recommended in several SE research synthesis guidelines, e.g., (Cruzes and Dybå 2010; Cruzes and Dybå 2011; Cruzesa and Dybåb 2011). Table 3 shows examples of how we conducted qualitative analysis of data received for one survey question on several projects. For example, for project P18, the respondent wrote: “The industry partner did not adopt the approach, to the best of our knowledge” and thus it was easy to classify it under the “No impacts on industry” group.

Table 3 Examples of how we conducted qualitative analysis on data of one of the survey questions

As we were also interested in the SE topics of the IAC projects in the dataset, another task in our data analysis was to extract the SE topic(s) of each IAC project. We did not have a specific question about this aspect in the survey (Section 4.2) but we were able to derive the SE topics of each IAC project by looking at its need (tackled challenges). To classify the SE topics, we used the latest version (3.0) of the Software Engineering Body of Knowledge (SWEBOK) (Bourque and Fairley 2014). The SWEBOK describes 15 Knowledge Areas (KAs), out of which 12 KAs are focused on technical aspects of SE, e.g., requirements, design, construction, and configuration management. Three other SWEBOK KAs cover mathematical, computer and engineering foundations. For example, two respondents mentioned the following needs for their projects: “supporting change impact analysis” for project P3 in the pool, and “to enable/improve quality assurance and engineering process improvement” for project P4. Based on the data, project P3 was classified under the KA “configuration management” and P4 was classified under the KAs “process” and “quality”. Note that each project could be classified under more than one SWEBOK KA.

5 Demographics of the Dataset

We present the demographics of the dataset in this section.

5.1 Breakdown by Affiliation Types and Countries

One of the questions asks about the respondent’s affiliation types (researcher or practitioner). 83 respondents said they are working at universities or research centers, 21 respondents said they had industry affiliations, and three respondents said they had both affiliation types (such as graduate students who also work part-time in companies).

Another question asked about the country (or countries) in which the IAC project was conducted. Figure 6 shows the data. 21 different countries are represented in the dataset. Although the authors’ countries of residence influenced the country distribution, the survey sample is diverse in terms of geographical distribution, covering Europe, Asia, North and South America.

Fig. 6
figure 6

Breakdown of the dataset by countries

5.2 Research Methods Used in IAC Projects

To understand the demographics of the IAC projects under study, we asked about the research methods used in each project. We used the classification of empirical SE research methods provided by Easterbrook et al. (Easterbrook et al. 2008) for this purpose, as shown in Fig. 7.

Fig. 7
figure 7

Breakdown of research methods used in the projects

Figure 7 shows that the most used research methods are industrial case studies (Runeson and Höst 2009) and “action-research” (Santos and Travassos 2009; Petersen et al. 2014b; Stringer 2013). This observation is as expected since these methods are, perhaps, better suited than other methods for IAC projects. Action research is “research initiated to solve an immediate problem” (Stringer 2013) and thus many industry-focused researchers are increasingly adopting it as the research method of choice, e.g., (Petersen et al. 2014b; Doležel and Buchalcevová 2015; Christensen et al. 2008; Viikki and Palviainen 2011). By being rooted in industrial practice, these research methods typically generate rich contextual information and many qualitative insights (Santos and Travassos 2009). As per our experience, e.g., (Garousi et al. 2016b), industrial case studies usually apply either the “exploratory” or the “improving” type, or both, rather than other case study types (descriptive, explanatory) (Runeson and Höst 2009). This helps, for example, to understand how and why certain practices, techniques, tools and methods actually work in a specific context.

In projects using as their research method controlled experiments with practitioners, initiation of the IAC usually starts from the researchers’ side (refer to the process model in Fig. 5). In these cases, typically, researchers have a research method or tool that they wish to apply in practice to assess its benefit and usefulness (Runeson and Höst 2009).

  • Observation 1: Industrial case studies and action-research are the most popular research methods used in IAC projects.

5.3 Number of Resulting Papers from each Project

We asked in another question about papers resulting from each project. Figure 8 shows that more than 90% of the projects have at least one publication. This means that there is a fair chance that others can learn from the project experience and that knowledge about successful – or unsuccessful – IAC projects is being conveyed to a larger community. However, since most publication venues are of academic nature, there is a high risk that the lessons learnt from the reported projects are mainly shared within academia. This is due to the fact that most industrial practitioners do not read scientific journals or conference proceedings on a regular basis (Chakrabarti and Lindemann 2015).

Fig. 8
figure 8

Histogram of number of resulting papers in each project

  • Observation 2: Over 90% of IAC projects, in our dataset, resulted in at least one publication.

5.4 Who initiated the Projects

IAC projects can be initiated by academic partners, industrial partners, or jointly by both sides (Fig. 5). In our dataset, the situation was quite balanced, i.e., the academic partners initiated 24 projects (22.4%) and the industry partners also initiated 24 projects. However, 52 projects (48.6%) were initiated by both sides jointly. No response was provided for the remaining one project.

We justified the above breakdown as follows. There can be different reasons for having an IAC project initiated by just one partner. One possibility is that the motivation for starting a collaboration is heavily academic-driven. For example, an academic might approach a company in order to validate a new SE method. On the other hand, the trigger for starting a collaboration might be heavily practice-driven. For example, a company might approach a university or research institute in order to get support for improving its SE practices. Thus, in spite of the myth that industry has in general lower motivation, compared to academia, to start collaborations (Garousi et al. 2016a), we see that many projects were initiated by industry.

Finally, there might exist established industry-academia relationships among a set of collaborators. In this situation, partners who have already a track record of successful collaborations might simply initiate the next IAC project jointly, building upon the trustful established relationship and capitalizing upon results of past IAC projects.

One could do further in-depth (correlation) analysis on collaborator type who initiated the project versus other variables in the dataset, e.g., country, and looking into a question such as: Are there differences between countries in terms of who initiates the IAC projects? However, for space constraint and also since our dataset is not large enough, we do not conduct and report such an analysis in this paper. But as discussed in Section 4.4, the entire raw dataset file (Garousi et al. 2018c) can be used for such extended analyses.

  • Observation 3: Industry is as motivated as academia to start IAC projects.

5.5 Project Duration

The “individual-value” plot in Fig. 9 shows the durations of IAC projects in our dataset (in months). The mean and median values of the IAC project durations are about around 22 and 15 months, respectively. About 4% of the projects took at most 13 months, whereas the durations of only about 5% of the projects were larger than 48 months. Overall, the durations of the IAC projects, in most cases, range between 4 and 48 months (i.e., 4–6 months: 18%, 7–12 months: 28%, 13–24 months: 20%, 25–36 months: 19%, and 37–48 months: 7%).

Fig. 9
figure 9

Individual-value plot of duration of project (in months)

  • Observation 4: More than 90% of the IAC projects in our dataset lasted from 4 months to 4 years.

6 Analysis and Results

We analyze the data and answer each of the RQs one by one in the following sub-sections.

6.1 RQ 1-Industrial Needs, Solutions Developed by, and Impacts of the IAC Projects

Since RQ 1 spans over three aspects, we review the results related to each of the three aspects separately in the next three sub-sections.

6.1.1 Needs (Problems) Addressed in the IAC Projects

Table 4 shows the needs and problems to be solved, the solutions developed, and the achieved impact of a representative subset of projects. Respondents provided qualitative narratives of the needs (problems) addressed in their projects.

Table 4 A representative subset of needs of, solutions developed by, and impacts of the projects

A number of projects were initiated to improve various aspects, e.g., P1’s goal was to improve test models used for model-based testing, P5 aimed at improving requirements specifications, and P13 aimed at improving the tool-support traceability analysis in embedded software. Other projects were set up based on needs to develop approaches or tools for other purposes, e.g., need for P3 was to provide and deploy an approach to support change-impact analysis, and P81 focused on the need to provide a practical way to perform regression test selection.

Of course, we would expect that there would be direct relationships among the needs, solutions, and impacts of a given IAC project. After all, a project is initiated with the goal of providing a solution (or improvement) to a given need and it is expected that the solution will have (positive) impact on the context (e.g., software process or product under analysis). We discuss examples of such expected linkage among the three aspects in the following section.

Given the diversity and wealth of data capturing IAC project needs, we also extracted a high-level picture of the needs by using word-cloud visualization, as shown in Fig. 10. As we can see, “testing”, “improvement”, “quality”, “process”, “automation”, and “systems” are among the most popular needs addressed in the projects. Let us note that we are not using word-cloud as a “formal” data analysis tool in this paper, but rather as a simple visualization to show a high-level trend of the topics.

Fig. 10
figure 10

Word-cloud of “needs” (problems) addressed in the projects (created using www.wordle.net)

As stated in Section 4.4, the entire raw dataset is available online in a file (Garousi et al. 2018c). The IAC project needs/solutions data can be reviewed in that online file and may be used for further analysis. For brevity, we only present summaries of each analyzed aspect in this article.

  • Observation 5: The IAC projects under study have covered a variety of needs / problems and have developed different types of solutions (contributions) with a variety of industrial impacts.

Figure 11 shows the distribution of our data points across SE topics. Each data point corresponds to one IAC project. With the exception of the KA “professional practice”, all SWEBOK KAs have been modestly covered. Testing and software quality are the most frequently mentioned topics in the dataset. We initially felt that the authors and their research areas had some impact on the dataset in this regard, but then we noticed that the survey participants outside the author team also frequently mentioned projects on testing.

Fig. 11
figure 11

Breakdown of data points by the SE topics based on SWEBOK

Several data points were classified under more than one KA. Figure 12 shows the number of KAs per IAC project in a histogram. More than half of the projects (57 out of 101) focused on a single KA, while the rest focused on more than one KA. This indicates that some IAC projects addressed multiple SE subject areas. For example, the goal of project P64 was to “better understand how people deal with code quality when using Continuous Integration”, which made us classify it under three KAs: Configuration management, Construction, and Quality.

Fig. 12
figure 12

Histogram of number of SWEBOK KAs per project

One project (P20) focused on seven KAs, since its goal was to develop “intelligent software systems to reduce the pumping cost of oil pipelines”, engineering “scientific software” (Segal and Morris 2008; Farhoodi et al. 2013). Project P20 was an IAC project among SE researchers and mechanical/electrical engineers addressing the entire process of developing an optimization software for oil pipelines with focus on requirements, design, development, testing and other KAs of the SWEBOK. Four other projects (P33, P35, P36 and P37) focused on six KAs, each. Thus, we see that a mix of projects with respect to coverage of SE topics is included in our study pool.

  • Observation 6: Most of the IAC projects seem to focus on exactly one topic of SE, i.e., one SWEBOK KA. Fewer projects considered multiple SE topics, e.g., an IAC project was to develop a large software system applying best practices in seven KAs.

6.1.2 Contributions of (Solutions Developed by) the IAC Projects

Respondents were asked to provide qualitative narratives of the solutions developed and offered in their projects (see a glimpse of the corresponding data in Table 4). Solutions are in fact the contributions of an IAC project to its industrial partner(s). Analysis of solutions showed that they usually are about development or customization of (new) tools or approaches and are related to analysis, testing, measurement and processes.

For example, in P14, “an automated environment configuration testing was developed and released in the industrial context”. In project P15, to address the problem that “cost of manual testing was too high and there were too many regression faults”, “a tool for automated test code generation for black-box unit testing was developed and released in the industrial context”.

To get a better understanding of the solution/contribution types developed in the IAC projects, we classified the solutions according to the following popular taxonomy of contributions used in SLR studies, e.g., (Zhi et al. 2015; Garousi et al. 2015; Petersen et al. 2015): approach (method, technique), tool, empirical study only, process, model, metric, any other contribution (solution type). Figure 13 shows the resulting classifications. As we can see, approaches and tools are the two most common project contributions (solutions), which appeared in 35 (32.7%), and 32 (29.9%) projects, respectively.

Fig. 13
figure 13

Classifications of project contributions based on the taxonomy of contribution types used in SLR studies (Zhi et al. 2015; Garousi et al. 2015; Petersen et al. 2015)

Contributions of 15 projects (14.0% of the dataset) were empirical studies only. In these IAC projects new approaches or tools were not developed but existing ones taken from the literature were applied in industrial settings. For example, in project P12, existing defect taxonomies were adopted from the literature to link requirements to test artifacts and to use those links to improve effectiveness and efficiency of tests as well as the requirements review process. In P24, researchers and practitioners did not develop a novel approach or a tool, but instead conducted an empirical assessment and improvement of the software test processes in the company using existing maturity models, like Test Maturity Model integration (TMMi) (TMMI Foundation 2017) and Test Process Improvement (TPI) (Koomen and Pol 1999).

In 14 IAC projects (13.1%), solutions were process-related. For example, in P4, efforts were spent on software process improvement (SPI) in multi-disciplinary engineering contexts for improving defect detection and engineering processes. P8 introduced risk-based testing into existing test processes. In P29, a company-wide measurement process based on the “Goal, Question, Metric” (GQM) paradigm was developed.

In ten IAC projects (9.3%), solutions were model driven. For example, in P5, a Bayesian-network model for identifying common causes of requirements engineering problems was proposed. In P46, a meta-architecture model was developed to improve architectural design practices in a company. Participants of P95 developed a knowledge map model.

In two IAC projects (1.9%), solutions were metric-focused. For example, motivated by the need to identify agile metrics for assessing the performance of IT development teams, project P55 developed a set of metrics to distinguish leading and lagging aspects in agile teams (Cohn 2009).

  • Observation 7: Most IAC projects in the pool tend to focus on approach or tool contributions.

6.1.3 Industrial Impacts of the IAC Projects

Similar to project needs/contributions, respondents provided qualitative narratives of the industrial impacts of their projects. To give a glimpse into the related qualitative data, Fig. 14 shows a word-cloud of the textual data provided by participants. We can see in the word-cloud that terms like “improved”, “effectiveness”, “change”, and “effort” were mentioned several times.

Fig. 14
figure 14

Word-cloud of industrial impacts in the projects (created using www.wordle.net)

A closer look at their occurrence in the raw data reveals that many projects improved aspects such as effectiveness and effort. Some participants provided concrete quantitative impact (improvement) measures in their feedback. For example, the participant reporting on P81 mentioned that “Based on measurements, our technique reduced end-to-end testing time for about 65% in industrial setting”, and for P14, it was mentioned that “The development environment instability issues were automatically detected by the tool and corrected in minutes. [Thanks to the project solution], the service downtime reduced to 0-10 minutes per week.

Furthermore, as discussed in Section 4.5, we used a qualitative coding approach (Miles et al. 2014) to classify industrial impacts of the projects into three categories: (1) Positive impacts on the industry partner, backed by quantitative evidence (measures) in the data point; (2) Positive impacts, backed by qualitative statements; and (3) No impacts on industry (yet), work in the lab only, or “not sure”.

Figure 15 shows the results according to this classification. For a given project, a respondent could provide quantitative and/or qualitative impacts (none, either or both). Since not all respondents provided this information, the sum of the classified data points only added up to 97, and not to 101 (the total number of data points).

Fig. 15
figure 15

Classifying the industrial impacts of the projects

73 IAC projects (~68%) reported at least one type of positive impact (quantitatively or qualitatively measured). 47 IAC projects reported quantitative positive impact. For example, a tool-supported approach for extending/refining test models based on execution traces collected during exploratory test activities was developed in P1 and it was reported that “The number of faults that can be detected by model-based test was increased by at least 30% for the conducted case studies”. For P50, the developed approach “improved test [activities] by automating it and reduced it by 10% for a year”.

30 IAC projects reported positive qualitative impact. For example, it was reported for P7 that “Based on three case studies from large enterprises, four success factors for introducing risk-based testing, i.e., (1) risk drivers like an inhomogeneous distribution of risks associated to the various parts of the tested software system, (2) an improvement goal with regards to effectiveness or efficiency as well as, (3) a proper risk identification, and (4) risk analysis approach were identified”.

For a large percentage of the IAC projects in our dataset respondents reported positive impact and outcome. We attributed this to the widely-discussed “reporting” bias, which refers to researchers (survey participants) “under-reporting unexpected or undesirable results [IAC projects in our case], while being more trusting of expected or desirable results” (McGauran et al. 2010).

It was surprising to observe that 20 projects (19.8%) had “no impacts on industry”. Respondents gave various reasons for such a lack of (measured) impact, stating, e.g., that industrial impact was not formally assessed (e.g., in P58), respondents were “Not sure [about impact]” (P77, P78, P80), “The approach was evaluated on a case-study industrial-like system in the ‘lab’. The industry partner did not adopt the approach, to the best of our knowledge” (P18), “Project is still in progress (towards the end)” (P46), “The UML tool prototype showed feasibility, but was not used in the end (the tool was discontinued)” (P67), and “So far, it [the project] has not led to a change in the use of safety analysis techniques” (P84).

  • Observation 8: A large share of IAC projects (76.2%) reported positive impacts on their industrial contexts. In about 20% of the projects, industrial impacts were not formally assessed or the project outcomes did not have noticeable impacts.

6.2 RQ 2- Challenges and their Negative Impacts in the IAC Projects

As discussed in the section on survey design (Section 0), one survey question asked about the impacts of ten types of challenges (adopted from the SLR (Garousi et al. 2016a)) in the projects, e.g., “Lack of research relevance” and “Lack of training, experience, and skills (of researchers or practitioners)”. Responses were provided using the following five-point Likert scale: (0): no negative impact, (1): minor negative impact, (2): moderate negative impact, (3): high negative impact, and (4): very high negative impact.

6.2.1 Ranking the Negative Impacts of Challenges

Figure 16 shows the histogram of all the reported impacts of the challenges, and both median and average values. The median values of the distributions are all equal to 1. Thus, we see that, in our set of IAC projects, the challenges had, in general, minor or moderate negative impacts. The three challenges with the highest and the three challenges with the lowest negative impacts are:

  • Mismatch between industry and academia (MIA), average impact = 1.47 (out of 4)

  • Human and organizational challenges (HOC), average impact =1.32

  • Lack or drop of interest/commitment (LDRC), average impact =1.28

  • Communication-related challenges (CRC), average impact = 1.08

  • Problems associated with the research method (RM), average impact = 1.08

  • Lack of research relevance (LRR), average impact = 0.90

Fig. 16
figure 16

Histogram of impact of the challenges

As shown in Fig. 16, all histograms are skewed towards the left indicating that the reported impacts of the challenges are low in general. In all histograms, the number of “0 - No negative impact” responses is higher than or equal to each of the other responses. This may denote the experience and expertise of the participants, in general, in carrying out IAC projects.

As the above ranking shows, mismatch between industry and academia (MIA) is the challenge with the highest observed negative impact. It has been reported in previous studies, e.g., (Sandberg et al. 2011; Wohlin 2013b; Rombach and Achatz 2007), that academia and industry have indeed different cultures, goals and objectives, e.g., academics’ main goal is to conduct research and publish papers, while industry’s main goal is revenue generation and to excel in their (software) business. Thus, it is very likely that this challenge is to some degree always present in IAC projects.

An aspect of mismatch between industry and academia is choosing the “right” topics for collaboration (what problem to focus on in an IAC project?). This issue has been extensively discussed in the literature. For some researchers, industrial problems lack scientific novelty or challenges, while some practitioners believe that most of the solutions developed in the academia have low applicability and scalability (Garousi et al. 2016a; Briand 2012). The first author and some of his industry colleagues had success in addressing this challenge in (Garousi and Herkiloğlu 2016), in which they reported a set of guidelines and a grounded-theory-based approach for selecting the right topics in IAC projects with focus on software testing. This approach could be easily transferred to other areas of SE.

  • Recommendation 1: Choose the “right” topic for your IAC, since one aspect of mismatch between industry and academia is failing to choose the right topics for collaboration. Readers are encouraged to review guidelines such as (Garousi and Herkiloğlu 2016) to tackle this problem.

Furthermore, the SLR study (Garousi et al. 2016a) divides the challenge of industry-academia mismatch into those sub-challenges: Different terminology and ways of communicating; and Different communication channels and directions of information flow. To address those challenges, the SLR study suggested the following practices: (1) Having prior projects between researchers and practitioners and positive experience (which facilitates communication in current projects); (2) Personal interaction among researchers and practitioners during data collection; and (3) Running workshops and seminars (increases visibility across organizations, allows to show relevance, strength and ability of both sides).

  • Recommendation 2: Tackle industry-academia mismatch in your projects using the variety of remedies proposed in the SLR study (Garousi et al. 2016a).

To address the second most important types of challenges, i.e., “human and organizational challenge”s, we observed that in IAC projects it is considered a good idea to involve the key staff within the company to clarify the common goals, to support the knowledge transfer, and to validate the ideas.

The third most important challenge, i.e., “lack or drop of interest/commitment”, should also be properly addressed. A crucial aspect in many human endeavors is commitment. Whenever organizations are involved, it is very important to guarantee the commitment from top managers. For example, ensuring the direct participation of CEOs of small companies (such as software startups) was indicated by one of our participants (P8) as being relevant for boosting the interest/commitment. In many contexts, having a champion is all that is needed to have success. It is also important to ensure engagement and to manage commitment.

The challenge with the lowest negative impacts is “lack of research relevance” (LRR, average = 0.90 out of 4), which means that the IAC projects included in our study, in their vast majority, are indeed based in industrial problems that require a research-oriented approach. In fact, in almost half of the projects (48 projects, 47.5%), the LLR challenge had no adverse impact at all (see the LRR histogram in Fig. 16). If we consider the cases where the LLR challenge had no negative or just a minor negative impact, the percentage of projects goes up to almost 75%. However, we should note that industrial relevance of academic research in general, has been criticized in many papers in SE, e.g., (Briand 2011; Jain et al. 2013; Lo et al. 2015), and in other disciplines (Mason 2001; Panda and Gupta 2014; Fox and Groesser 2016; Tan and Tang 2016; Olson et al. 2008). Since our dataset is in fact made up of IAC projects in which research had to be aligned with industrial needs, it is not surprising that this challenge has been rated very low.

  • Observation 9: Mismatch between industry and academia (MIA) and Human and organizational challenges (HOC) have the highest negative impacts on projects.

When assessing challenges of the projects, we were aware of the ratio of the projects (~68%) for which positive impacts (outcomes) were reported (as discussed in Section 6.2). One would expect that the projects with positive outcomes are less likely to be impacted by the challenges, compared to projects with no or negative impacts in the sample. For this purpose, we divided the two groups of projects and calculated for each group the sum of the reported challenge levels (as gathered in the 5-point Likert scale introduced in Section 4.2). For example, for project P1, the levels of the ten considered challenges were: 0, 0, 2, 1, 1, 3, 3, 3, 3, and 0, which sums up to 16. Figure 17 shows two boxplots for the sums over challenge levels of the projects having positive impact (on the right) versus those without (on the left). We can see that there is no significant difference between the two cases, thus not confirming our hypothesis that the projects with positive outcomes were less likely to be impacted by the challenges, compared to projects with no or negative impacts in the sample.

Fig. 17
figure 17

Boxplot group sum of challenges for the projects having positive impact versus those without

6.2.2 How the Challenges Affected the IAC Projects

In one survey question we asked how the challenges impacted the projects. 70 of the 101 data points provided answers to this question. The full list of verbatim responses to this question is provided in Appendix 1.

To make sense of the textual data, we conducted qualitative coding and mapped the explanations provided by respondents to the list of challenges. Each explanation could refer to one or more challenges. We provide a few examples below:

  • Explanation provided in P1: “Lack of resources (man hours) that can be allocated by the employees who have to continuously follow up development activities and meet deadlines”: This was mapped to the challenge category: Resource-related challenges (RRC)

  • Explanation provided in P2: “Relocation of key people within the company”: This was mapped to the challenge category: Human and organizational challenges (HOC)

  • Explanation provided in P3: “Long negotiations before being able to deploy tool”: This was mapped to the challenge category: Mismatch between industry and academia (MIA)

  • Explanation provided in P4: “The availability of industry contact persons made it challenging, because there are unavailable for a certain time interval or it took quite long to get answers.“: This was mapped to the challenge category: Resource-related challenges (RRC)

  • Explanation provided in P5: “After causal analysis sessions, we came up with a set of Improvement actions, but not all of them could be adopted, since this would change organizational standards beyond the scope of the project. Therefore, most of the reported benefits actually came from institutionalizing inspections and categorizing defects (and not from the causal analysis itself).”: This was mapped to the challenge category: Human and organizational challenges (HOC)

  • Explanation provided in P6: “Organizational commitment and resources to implement [the] approach was difficult to get”: This was mapped to two challenge categories: Lack or drop of interest / commitment (LDRC), and Resource-related challenges (RRC)

We compared the number of times each of the challenges appeared in the provided explanations of how the challenges affected the projects versus the average negative impact of challenges (based on rubric data) as discussed in Section 6.2.1. The related visualization in the form of a scatterplot is shown in Fig. 18. The Pearson correlation of the two data series in this figure is 0.606, denoting a moderate correlation. We were actually expecting a higher correlation. The reason for the relatively low correlation is that some respondents mentioned negative impact for certain challenges (based on rubric data) but provided explanations related to other challenge types.

Fig. 18
figure 18

Scatterplot of number of times each challenge appeared in the provided explanations of how the challenges affected the projects versus the average negative impact of challenges

We then looked at the coded data (the groups under each challenge type),and synthesized the responses. We report a summary of that synthesis below, grouped by the type of challenges and ordered according to the frequencies with which challenge types occurred in the provided explanations. To support our discussion, we provide verbatim quotes of the received responses.

Resource-Related Challenges (RRC)

22 of the 70 (31%) provided explanations focused on RRC. Most of those explanations mentioned lack of time to invest in the IAC project as one main challenge that had negative impact.

For the case of P4, “the availability of industry contact persons made it challenging, because there are unavailable for a certain time interval or it took quite long to get answers”. In many cases, unfortunately, the IAC project gets low priority when compared to other business-critical tasks. A project employee was involved in other important commercial projects during the research collaboration and therefore had to quit.

In addition to time and human resources, lack or shortage of financial resources was also a recurring challenge. For example, the data point P12 reported that “the project was not formally funded”. We actually had a question in the demographics section asking about source(s) of funding (if any) for each project. As per our counting, 14 of the 101 projects were not funded (yet). A non-funded IAC is often executed out of curiosity and passion from both sides, or when there is a Master or PhD student who already works in the company and conducts her/his thesis on the topic that becomes the IAC. The first author had several such students in Turkey which resulted in successful theses and IACs.

Often, researchers work for free when there is no research funding for an IAC, sometimes only using the internal funding and resources of their universities. An example is P26, a project that was an industrial case study in automated visual GUI testing (Garousi et al. 2017b). In such IAC projects, due to lack of funding, there are inherent challenges to run the project, e.g., hard to hire additional PhD students, and inability to attend conferences to present papers, etc.

Mismatch between Industry and Academia (MIA)

Mismatch between industry and academia was mentioned in 20 responses as a major challenge. A mismatch is possible w.r.t. different aspects, e.g., mismatch in expectations (P34, P67, P79, P87), mismatch in viewpoints of what is a technical challenge (P50, P87), mismatch in time horizon (P51, P53, P63, P74), mismatch in willingness to change (P57), mismatch in assumptions (P76), and mismatch in objectives and backgrounds (P86).

For P34, it was mentioned that “The project ended up being more a development/engineering one, with a minimal necessity to include research efforts. The management of the project was interested in creating research evidences, like patents and papers. This, the team decided to write scientific papers to address the more research-oriented perspectives of the project”. For 67, it was mentioned that “industry wanted solutions that would easily integrate in their current processes. academics wanted to propose their own methods”.

For P50, it was mentioned that “It was very hard to find previous research done on this topic. And whenever we talked to other researchers, they didn’t think it was a problem. Whereas when talking with practitioners, we often got the question if/when they would be able to give it a try. I guess you could say both sides perceive different challenges for (or their impact on) the industry”.

For P51, it was mentioned that “The long-term research scope of researchers at universities does not link with short-term goals of many companies”. For P34, it was mentioned that “Academics wanted to use UML, but no industrial partners were using it”. For P86, it was mentioned that “the organization and the researchers had different objectives and backgrounds. so, each side tried to make the resulting products be like what they like. finally, the results did not make either of them happy”.

Contractual, Privacy and IP Concerns (CPC)

Challenges related to CPC were the third most frequently discussed challenge and reported in eleven cases. For example, for P16, it was mentioned that “Security and privacy concerns were not discussed early on in the project. The only cancelation reason of the project was just that which was a non-technical issue, i.e., inability to get security clearance for two graduate students planned to be involved in the project”. Thus, while contractual and privacy concerns may look trivial for people new to IAC projects, they sometimes get very critical and lead to cancelation of a project in which a lot of effort has already been spent.

A similar, but less critical, situation was reported for P49. The respondent mentioned his/her disappointment with the fact that, although “rich findings and results from the industry” were generated, their research team could not “exhaustively discuss it and share it with the community. All the discussion and knowledge sharing [instead had] to be [done] through open-source, and rather smaller subjects”. The same issue was reported for P68: “Privacy concerns made the process for publication harder and caused the paper to not have specific measurements that would have strengthened the arguments”. Similarly, for P80, it was mentioned that “Not surprisingly, enterprises are paranoiac when it comes to security. This poses several challenges, e.g., from accessing to the company software for experiments to even name the company in published papers”. In other projects (such as P69), due to security concerns, “all the development had to be done on-site [in industry]”.

Issues w.r.t. IP were mentioned for several projects. For example, for P75, it was mentioned that “Multiple small to medium sized enterprises [were] involved in the project, each wanting to retain their own IP”.

In some cases, team members had to go through long negotiations, e.g., in P3, “long negotiations [were held] before being able to deploy [the] tool”.

Lack of Training, Experience, and Skills (LTES)

Ten of the 70 provided explanations discussed negative impacts of challenges related to LTES. In P15, “lack of SE training in industry side caused the industry folks to undervalue the novelty of test techniques”. In P35, “the project required knowledge of the software test process. A lack of experience, by the team, on the software development lifecycle made the team’s velocity lower and a slight delay in the development of the application”. Also, in P36, “development approaches required a multidisciplinary team with a significant learning curve on business context and tools to be adopted”.

Several respondents mentioned technical and training challenges w.r.t. knowledge of UML in their projects, e.g., in P77, “industrial practitioners often need training to understand them [UML models], as they do not [often] use UML during their regular work”.

Another project (P91) whose topic was on software process improvement, mentioned a similar challenge: “(…) their [the practitioners’] lack of process modeling knowledge and difficulty in explaining the processes while communicating the required information, their difficulty in reading the models”.

Another similar issue was raised in P99: “The resistance of managers who do not have the sufficient background to understand the research ideas but still have a huge impact on the decision-making process”.

Lack or Drop of Interest / Commitment (LDRC)

Eight of the 70 provided explanations discussed negative impacts in projects related to LDRC. For P22, it was reported that “interest / commitment dropped early on”. Such a situation could happen due to many reasons, e.g., when expectations are not met or when they are not clearly communicated.

In P32, the following observation was reported: “Much of the research published by academia was not of interest to the companies”. That project’s goal was to “understand problems in mobile application testing and addressing those problems by transferring solutions proposed in the literature”. When the researchers provided a synthesized summary of the existing mobile testing literature to their industry partners, it seemed that the industrialists did not show much interest in reviewing them.

In general, many projects reported that organizational commitment and resources to implement research approaches were “difficult to get”, e.g., P6. For P85 also, it was reported that “it was hard to keep the motivation of the key personnel in the organization motivated, since they did not have much incentive to contribute to an innovative project as it required more effort than their regular job descriptions”.

Human and Organizational Challenges (HOC)

Human and organizational challenges were reported in six cases. In some cases, “relocation of key people within the company” (P2) caused issues in the IAC project. It often becomes an issue when a given IAC project’s industry contact point (also called “champion” (Wohlin et al. 2012)) changes the position within the company or moves to another company. In the former case, since the champion is still in the company, the same person may be able to continue her/his role in the project, or someone else may “take over” the role of contact point. If no proper arrangements are made, the project will most likely be paused or terminated (e.g., P13).

There were also organizational challenges, e.g., in P5, the team members “came up with a set of improvement actions, but not all of them could be adopted, since this would impact changing organizational standards beyond the scope of the project”.

Resistance to change, by staff and organizations, was another mentioned challenge, e.g., in P88 (“additional effort was needed to be spent to overcome the resistance of employees”) and P91.

Management-Related Challenges (MRC)

Six of the 70 provided explanations discussed negative impacts due to management-related challenges. Changes in company management or priorities often produce challenges in IAC projects, e.g., P39 and P44.

IAC projects may also struggle to get management to “believe in the business value of the collaboration” (P53). It is also usually not easy to engage management “because academia is basically slow and [often] unwilling to adapt” (P57).

Lack of Research Relevance (LRR)

Five responses discussed negative impacts due to lack of research relevance. In P13, “the connection link between industry and academia got weak in middle of project due to turn-over in the company, and thus gradually led to lack of research relevance”.

For P38, “the research relevance of the project lost its significance during the project. Although the project had focused on relevant research issues related to test processes, those research issues were not exploited since the defined test process had not been sufficiently studied”.

In P41, “the idea was to build a measurement system in [an] EU project. Parts of the measurement where not novel and those got implemented. All the other goals were unrealistic and did not get implemented. [There were] lots of bureaucracy involved”.

Unsuitable Research Methods (RM)

Four responses discussed negative impacts due to unsuitable research methods. For P67, it was reported that “Industry wanted solutions that would easily integrate in their current processes. Academics wanted to propose their own methods, tools/techniques that they are fond of, without considering the actual needs of the industrial partners”. Such a research approach is often called “advocacy research” (also called the “research-then-transfer”) (Potts 1993; Glass 1994) which is often not received well by industry. Researchers have commented on how to address such an issue. For example, (Tichy et al. 1993) called for “a paradigm shift ... from purely theoretical and building-oriented to experimental”. (Griswold and Opdyke 2015)suggested to researchers to “see the world [SE practice] as it is”. (Glass 1994) suggested using the “industry-as-laboratory” approach, in which “researchers identify problems through close involvement with industrial projects, and create and evaluate solutions in an almost indivisible research activity” (Potts 1993).

For P47, it was reported that “We [the team members] missed a major flaw to validity through combined lack of understanding related to test flakiness”, denoting issues in the research method. A challenge in P53 was that “[the] tension between scientific quality and practical applicability cannot be avoided but can be balanced with experience”.

For P57, it was reported that “they [researchers] come in with a hammer and look for nails”, again denoting fundamental issues in research methods. Similarly, for P67, it was reported that: “academics wanted to propose their own methods, tools/techniques that they are fond of, without considering the actual needs of the industrial partners”, again highlighting the negative impact of the “advocacy research” (also called “research-then-transfer”) (Potts 1993; Glass 1994).

Communication-Related Challenges (CRC)

Three responses discussed negative impacts due to communication-related challenges. Communication habits/patterns of the involved parties might be different, e.g., speed in responding to emails or technical tasks (such as data collection). If both sides (researchers and practitioners) do not show “understanding” for such different styles, there could be challenges.

For P53, it was reported that “Communication habits/patterns are challenges and need champions on both sides to be overcome”. For P38, it was reported that” Due to some connection problems (Skype mainly), there was some challenges related to communication”. Thus, we can that, for the case of remote collaborations, it is important to ensure smooth and reliable teleconferencing facilities.

P20 was a project on engineering (development) of scientific software (Segal and Morris 2008; Farhoodi et al. 2013). For that project, it was reported that “Since software engineers worked with other types of engineers in the oil industry (e.g., chemical and mechanical engineers), as expected, there was considerable gap of knowledge in the two sides w.r.t. the other side. The two sides had to struggle to communicate and understand each other in many occasions”.

One Challenge could Lead to One or More Additional Challenges

By analyzing the data, we also came across a few cases in which a challenge could lead to one or more additional challenges. For example, for P10, there was a “demanding industry partner” and expectations of both sides were not clearly communicated and thus one side put more demand on the other side. Such a situation often negatively impacts the relationship (the so called “social capital” (Al-Tabbaa and Ankrah 2016)) which in turn degrades the performance of the IAC project, and could result in the Drop of interest/commitment” challenge.

As another example, for P13 “the connection link between industry and academia got weak in middle of project due to turn-over in the company”. Such a challenge “gradually led to lack of research relevance”.

  • Recommendation 3: Review how the challenges affected the projects in the dataset of this study (summarized above), and the suggested best practices provided in the SLR study (Garousi et al. 2016a) to be able to cope with similar challenges in your IAC projects.

6.2.3 Other Challenges Observed

Another survey question (see Q3.3 in (Garousi et al. 2016c)) asked participants about any other challenge(s) they might have observed in their projects, in addition to the ten categories of challenges that we had asked in a previous question (as discussed above in Section 6.2.1). This question was answered for 24 of the 101 projects.

After analyzing the provided responses, we found that 20 of those 24 “other challenges” could actually be classified under our own ten challenge types (see the above sections). We show a few of those responses that we classified under our own ten challenge types below:

  • Human and organizational challenges

    • “Difficulty to introduce new tool in a strict process “(P3)

    • “Restructuring of company leading to moved role for championing individual “(P82)

  • Mismatch between industry and academia

    • “Gap between research prototypes and industry projects (different key goals of project stakeholders) “(P4)

    • “Keeping a long-term research focus while delivering (interesting enough) short-term results for the service and developing unit “(P43)

    • “Publications are great, but for industry it is not the end-goal. The end-goal is to use the tool... which requires more work than in a typical IAC set-up “(P48)

    • “Shifting business priorities “(P73)

  • Contractual, and privacy concerns

    • “The workers council did not allow us to collect all the data that we wanted to have (e.g., how long does a person work for the company, ...) “(P45)

    • “Publishing the results were difficult due to privacy / sensitive information concerns. We succeeded, but it was somewhat difficult “(P46)

  • Management-related challenge

    • “Changes in top-management of software companies; often leading to new and adjusted goals. Research should be more agile in following these changes “(P51)

  • Lack of training, experience, and skills

    • “The lack of know-how on the system domain by the academics “(P57)

The remaining four of those 24 “other challenges” were related to getting funding in support of the IAC project. We itemize those responses below:

  • “Time and effort spent for bureaucratic issues due to the involvement (communication with) multiple funding partners” (P1)

  • “Lack of understanding for empirical approach by sponsors. Terminology was also a problem as people often tend to make (negative) associations with certain terms. We simply talked about technology transfer when actually meaning technical action research (to not be associated with ivory tower research)” (P66). The first author also recently experienced these challenges in the context of a recent grant proposal in which he partnered with three industrial companies and two other research institutes on the topic of “testing scientific software”. After submitting the proposal to the Dutch national funding agency (called: NWO), it was rejected and the main comment was that most of the work was empirical and there was little “scientific” merit in the proposal.

  • “To get funding, in the grant applications you need to promise the Moon (a grant application in Software Engineering that does not also cure AIDS and cancer as byproduct is not worthy to fund). But what you get in the end (i.e. in terms of PhD students and post-docs to hire) is not enough to even cover 1/10th of what promised... the industrial partners might not be happy about it” (P67)

  • “It was not easy to get the project funded, given the bureaucracy at a Brazilian federal university. I just moved to a privately held university [name removed to keep anonymity] that has a huge tradition in promoting university-industry collaborations. Promoting and facilitating industry collaborations was the main motivation for my move”. (P71)

6.2.4 RQ 3- Practices (Patterns and Anti-Patterns) and their Impacts in the IAC Projects

For RQ 3, we measured the impacts of applying practices in the IAC projects. We intentionally did not classify practices into patterns (what to do to ensure success) or anti-patterns (what not to do to avoid failure) as it was done in the survey (Garousi et al. 2016a), since we wanted to see whether extracted data for impacts related to each practice would denote it as a practice resulting in positive or negative outcomes (i.e., patterns and anti-patterns). As discussed in Section 4.2 (survey design), responses were provided using the following five-point Likert scale: (−2) - Very negative impact; (−1): Negative impact; (0): Neutral (the practice did not have any impact); (+1): Positive impact; and (+2): Very positive impact. An “empty” answer would denote that “The practice was not applied / NA”.

Figure 19 shows the reported impacts of practices on projects, classified using the five-point Likert scale. For example, the “Ensuring engagement and managing commitment (ENMC)” practice had largely positive (48 votes) or very positive impact (29 votes).

Fig. 19
figure 19

Reported impacts of patterns and anti-patterns on projects (all values)

To be able to compare the impacts of practices, we calculated the average values for all the 101 data points. For example, if there were equal number of −2 and + 2 values for a given practice (and not other values), the average would be 0, denoting that considering all data points, that practice had overall a neutral impact. Figure 20 shows the average values.

Fig. 20
figure 20

Reported impacts of patterns and anti-patterns on projects (average values)

6.2.5 Patterns (What to do to Ensure Success)

Based on the values in Fig. 20, the three practices with the highest positive impacts are:

  1. (1)

    Working in (as) a team and involving the “right” practitioners (WTI), average impact = 1.39 (in range of [−2, 2])

  2. (2)

    Having mutual respect, understanding and appreciation (HMRU), average impact =1.37

  3. (3)

    Considering and understanding industry’s needs, and giving explicit industry benefits (CUIN), average impact =1.28

  • Observation 10: The three most important practices in IAC projects to increase chances of success are: (1) Working in (as) a team and involving the “right” practitioners, (2) Having mutual respect, understanding and appreciation, and (3) Considering and understanding industry’s needs, and giving explicit industry benefits.

We noticed interesting “contrasting” situations for some of the practices. For example, the HMRU practice had an overall positive impact (i.e., it ranked the second most helpful practice above), however, in the case of one data point (P22), HMRU had a “high” negative impact. The following supporting comment was entered by the corresponding participant for P22: “the industry side did not appreciate the need for software testing research”. For another project (P21), “moderate” negative impact was reported for the HMRU practice, with the following comment: “appreciation from the industry side decayed by time”. Thus, we can see that while most of the helpful practices were supporting IAC projects, there could be exceptional opposing cases due to differences in the project contexts. Note that, in such cases, it is not appropriate to say that the practice itself is an anti-pattern or had a negative impact. One should rather say that not being able to satisfy (implement) the practice yields a negative impact (like the examples of projects P21 and P22).

We had two follow-up questions asking for a narrative explanation of the effect of the practice with the most positive impact. We reviewed those explanations about manifestations of the patterns and matched them with the corresponding patterns. We show a subset of the matching data in Table 5, which presents qualitative data and lessons learned.

  • Recommendation 4: Review and consider using the patterns (as listed in Table 5) in your IAC projects.

Table 5 Patterns and their manifestations in the sample data points (quotes from the data)

Based on the qualitative data provided by the participants, we give the following recommendations with respect to the top-3 reported patterns to increases chances of success in IAC projects:

  • Having mutual respect, understanding and appreciation:

    Recommendation 5: Develop a relationship of trust with the involved practitioners. Trustful relationship is key in the successful execution of many IAC projects. Often, the physical presence of the researchers at the industrial partners’ facilities, during some periods of time, proves helpful in developing this relationship of trust.

  • Working in (as) a team and involving the right practitioners:

    Recommendation 6: Develop a team spirit in your IAC projects. Like in collective sports, team spirit is crucial for a successful IAC project. A good practice, indicated by many of our respondents, to boost this team spirit is to consider the presence of the academics at the facilities of the industrial partners. This practice seems to strengthen the bond between the two sides.

    Recommendation 7: Organize frequent and regular meetings during the IAC project. One recommendation is to meet twice a week, to ensure that the knowledge is exchanged among the team members. Another guideline is to involve academics with domain knowledge (i.e., knowledge on the subjects being addressed by the IAC project). In fact, researchers should have solid technical and practical background and be able to speak the language of the practitioners.

  • Considering and understanding industry’s needs, and giving explicit industry benefits:

    Recommendation 8: Make sure that each side adapts, as much as possible, to the other one (e.g., in terms of terminology, needs, etc.). Academia and industry have different mindsets and distinct time frames. A good recommendation is for academics to give explicit industry benefits and solve the right problem. In some cases, this entails simple things such as using real and relevant data, in which the industry partner has a direct interest.

    Recommendation 9: Follow iterative approaches in IAC projects: Adopting iterative approaches, such as the ones suggested by agile methods, seem useful as they allow industrial partners to iteratively adapt their needs, a practice that is followed in some contexts (Hicks and Foster 2010).

6.2.6 Anti-Patterns (What not to do to Avoid Failure)

In terms of anti-patterns (the last four practices in Fig. 20), the participants reported negative impacts for them, as was expected. The anti-pattern with the highest negative impact is “Following self-centric approach” (FSCA) in conducting IAC projects, i.e., each side (industry and academia) focuses only on its needs and objectives in the project. We have had personal experience in how this particular anti-pattern can ruin an IAC, for example, we have been involved in an IAC in which a researcher just focused on the academic objectives (publishing papers) after the project started and did not consider the industrial partner’s objective of decreasing test costs in practices (although it was mentioned in the project description for the funding agency). This anti-pattern led to a decrease of mutual trust and loss of “social capital” (Al-Tabbaa and Ankrah 2016) on the side of the industrial partner which unfortunately led to a poor execution of the IAC project at the end.

As Fig. 20 shows, the other three anti-patterns were also empirically observed in the dataset: Poor change management; Ignoring project, organizational, or product characteristics; Unstructured decision structures. We reviewed the quotes from the dataset that provided narrative explanation on manifestations of the anti-patterns and matched them with the four anti-patterns types. We show a subset of the matching data in Table 6. As we can see, there is a huge amount of qualitative data and lessons learned in these quotes, which we present and hope that readers can benefit from.

  • Recommendation 10: Review and take the anti-patterns (as listed in Table 6) as cautionary lessons in your IAC projects.

Table 6 Anti-patterns and examples of their manifestations in the sample data points (quotes from the data)

Based on the qualitative data provided by the participants, we provide the following recommendations with respect to the top-three reported anti-patterns to increase chances of success in IAC:

  • Poor change management:

    Recommendation 11: Be more flexible to change in the IAC project. We have observed in a few projects of our own that many academics are usually work in big up-front design (planned) fashion and are not flexible to changes. However, industry in general is becoming more flexible to change and we have seen that this can sometimes cause issues when things do not go as planned, e.g., when the project’s main contact point in the company leaves the company without notice.

  • Ignoring project, organizational, or product characteristics:

    Recommendation 12: Adopt and pay attention to needs of both sides of the IAC project. In some cases, researchers are interested in just proving the concept, while the industry partners are often looking for tools with full functionality and relevant quality attributes (usability, portability, maintainability).

    Recommendation 13: Pay attention to organizational characteristics (norms). Researchers are encouraged to have the “soft” skills to follow the specific written and non-written organizational norms.

  • Following a self-centric approach:

    Recommendation 14: Avoid self-centric approach. We have observed in a few projects of our own that many researchers follow self-centric approach in that they consider their ideas and method unquestionable and thus act in a self-centric fashion. This usually turns off industry partners and should be avoided.

6.3 Discussion

In this section, we first discuss the implications of our findings and recommendations and then the limitations and potential threats to validity of this study.

6.3.1 Implication of Findings

Based on the synthesis of all the data provided by the participants in the previous sections, we offer four take-away messages in this article that should ensure successful IAC projects. They are the following: (1) establishing a common goal; (2) recognizing and understanding the fundamental difference; (3) understanding and team work; and (4) managerial topics.

A common goal is a cornerstone of any IAC project. In the histogram showing the impact of challenges (Fig. 16), we see that lack or drop of interest is the challenge with the highest negative impact. Mismatch between industry and academia is the challenge with the second highest impact. Both of these challenges reflect that a common goal for the research project is missing. Without a common goal that is both interesting and beneficial for both parties, IAC projects have a little chance of succeeding. This is highlighted by the quote stating that “Finding common ground is hard but important” (P52).

The fundamental difference denotes the fact that, ultimately, academics want to get top journal publications and material for Ph.D. degrees from the projects. Industry wants a solution that is useful and comes with low maintenance cost. For academics, making improvements to a tool or process or writing a publication is a common research goal. Industry, on the other hand, wants long-term solutions, as they have to live with the new tool or process for the years to come, while the academics move to the next challenge. For example, a new tool that is correct 90% of the time might be a great improvement in a specific industry setting and thus the industry partner would in a next step focus on making sure that the tool is usable and maintenance free (able to handle all special cases, for example). In contrast, academics would focus on developing a new prototypical tool that is 95% correct. Thus, the academic interest would be to continue to improve the tool’s correctness no matter whether the new prototype renders the tool less usable and increases maintenance cost or learnability (for example requiring longer running time, more manual steps, more information from different data sources). Our suggestion for this fundamental issue is that both sides recognize that such a point exists where interest would differ and define what to do at this point. In an ideal case, industry would start to ensure tool usability and low maintenance, while academics would be allowed to work on the tool until prior work is beaten.

Understanding and team-work ensure that IAC projects move smoothly and the common goal is not lost in the process. Industry and academia have different cultures, backgrounds, and objectives. Thus, it is natural that all of the top-3 ranked success criteria deal with the topic of gaining understanding and forming a team:

  1. 1.

    Having mutual respect, understanding and appreciation (HMRU)

  2. 2.

    Working in (as) a team and involving the “right” practitioners (WTI)

  3. 3.

    Considering and understanding industry’s needs, and giving explicit industry benefits (CUIN)

Even when a common goal has been found and a team with a mutual understanding and respecting has been formed, an IAC project can fail due to management related issues. For example, contractual and privacy concerns need to be taken into account. Getting and keeping higher levels of management commitment is important, as otherwise top-level mangers can abruptly abort IAC projects if they think the company employees are wasting their time. Internal company policies need to be understood; for example, some units or sites of a large company may not have permission (or have limitations) to be involved in research activities. Some of these managerial topics could be impossible to bypass and may lead to ending a research project before it has even begun.

We should also mention in this context that challenges of an IAC project should not be analyzed in isolation, as challenges are often inter-related with patterns and anti-patterns. Figure 21 shows a cause-effect diagram of inter-relations of challenges, patterns, and anti-patterns (adopted from (Garousi et al. 2016b)). Challenges and anti-patterns are expected to negatively impact success, while applying patterns is expected to positively impact success. Challenges necessitate the need for applying patterns that address those challenges. Patterns and anti-patterns could neutralize each other. For example, the anti-pattern “Following self-centric approach (FSCA)” could damage (neutralize) the benefits gained by applying the pattern “Working as a team and involving practitioners (WTI)”.

Fig. 21
figure 21

Inter-relationship of challenges, patterns and anti-patterns in IACs (taken from (Garousi et al. 2016b))

Furthermore, anti-patterns usually bring more challenges. Thus, participants of an IAC usually apply patterns (which we discuss later in the context of RQ 2) to address challenges. For example, to decrease the cultural and objective mismatch between industry and academia, various practices, such as those grouped under the “Understanding the context, constraints and language (UCCL)” pattern (Garousi et al. 2016a), are usually applied.

6.3.2 A Checklist for IAC Projects

In order to help researchers and practitioners who plan to conduct an IAC, we suggest a checklist as shown in Table 7. The checklist is based on the recommendations extracted in Sections 6.1 and 6.2. The first ten items in the checklist apply to both researchers and practitioners. The 11th item is particularly important to be checked by researchers. After each checklist item, we indicate the recommendation(s) from which it is derived.

Table 7 A checklist for IAC projects, derived from earlier recommendations

6.3.3 Limitations and Potential Threats to Validity

In this section, we discuss potential threats to the validity of our study and steps we have taken to minimize or to mitigate them. The threats are discussed in the context of the four types of threats to validity based on a standard checklist for validity threats presented in (Wohlin et al. 2000): internal validity, construct validity, conclusion validity and external validity.

Internal Validity

Internal validity is a property of scientific studies which reflects the extent to which a causal conclusion based on a study and the extracted data is warranted (Wohlin et al. 2000). A threat to internal validity in this study lies in the selection bias (i.e., lack randomness of the projects participating in our survey).

Construct Validity

Construct validity is concerned with the extent to which the objects of study truly represent theory behind the study (Wohlin et al. 2000). In other words, it relates to whether we actually measured industry academy collaboration project in our study. One form of industry academy collaboration project we might have covered only partially are Master’s thesis student projects, which in many countries are made while the students work on the industry payroll. A typical goal of such thesis is to improve a tool or a process used by a company. These projects might have been missed because they do not often result in academic papers, the university does not formally manage them, and the academic contributions are limited. At the same time, based on discussion with industry, it seems that these projects deliver timely and real benefits to industry and industry is also continuously willing to invest their money for thesis student projects. Let us recall from Section 4.4 that, in total, 64 respondents provided the information on the 101 projects. Each respondent provided between 1 and 19 data points. A majority of the respondents (57 people) provided only one data point, thus we can say that a large number of data points came from different people. The data points from the same respondent correspond to different projects and thus there is some level of independence between them.

It is also common for people to deflect their answers when they feel being evaluated and based on what they think is the intended result of a study. To mitigate this, we informed participants prior to the survey that our motive in this study was to take a snapshot of IAC projects and that we did not intend to publish any identifying information so that participants will remain anonymous.

Conclusion Validity

Conclusion validity of a study deals with whether correct conclusions are reached through rigorous and repeatable treatment (Wohlin et al. 2000). We analyzed, qualitatively, challenges and success criteria of IAC projects. For each RQ, we attempted to reduce the bias by seeking support from the data gathered in the survey and subsequent statistical analysis. Thus, all the conclusions that we present in this article are strictly traceable to data.

External Validity

External validity is concerned with the extent to which the results of this study can be generalized (Wohlin et al. 2000). Given the moderate number of 101 projects included in the analysis (after screening), the external validity is somewhat limited, but still the largest one reported so far in the literature. Also, the results might be more or less representative, depending on SE area in the scope of an IAC project. In the set of projects we used for our analysis the fields “testing”, “quality” and “process” are the most frequent ones, while “professional practice”, “configuration management” and “maintenance” are the least frequent ones. This could indicate that some SE areas are more appropriate and relevant for conducting successful IAC projects than others.

As discussed in Section 4.4, we were vigilant about survey reliability, i.e., representativeness of the dataset and sampling (Gobo 2004). Although we used “convenience” sampling, we took various measures to ensure survey reliability. In summary, by using convenience sampling in our work (similar to many other survey studies in SE), although representativeness could be limited, but we increased the external validity of the survey by ensuring sample relevance and sample prototypicality (representativeness) (refer to Section 4.4 for details).

Furthermore, since the unit of analysis in this study is an IAC project, the true population is all IACs in SE. In our survey execution, to ensure getting as many data points as possible, we have asked each respondent to provide as many data points (IAC projects) as possible, thus we have taken the convenient sampling approach for sampling respondents conveniently, and also sampling projects conveniently, which is away from true random sampling. Therefore, the external validity is impacted with this convenient sampling approach.

Also, as discussed in Section 6.1.3, a large percentage of the sample included projects reporting positive impact and outcome. We attributed this to the widely-discussed “reporting” bias, which refer to researchers (survey participants) “under-reporting unexpected or undesirable results [IAC projects in our case], while being more trusting of expected or desirable results” (McGauran et al. 2010). Approaches to address this potential validity threat could be to increase ratio of less-successful projects by removing from the dataset a randomly-selected subset of projects with positive impact, or to encourage survey participants to also report projects with no impact. We had actually made that encouragement in our invitation emails. As for the former approach to increase ratio of less-successful projects, we decided not to remove from the dataset any of the projects with positive impacts as to not decrease our dataset size.

6.4 Conclusions and Future Work

This article makes four contributions. Firstly, we report the largest survey, ever in the Software Engineering (SE) community, on industry-academia collaborations (IAC) projects. Our results are based on 101 different projects from 21 different countries, and covering 11 (out of 12) SWEBOK KAs. Secondly, we show that lack or drop of interest/commitment (LDRC) is the most highly observed challenge in the projects. On the other hand, the challenge with the highest perceived negative impact is resource-related challenges (RRC) followed closely by LDRC. Thirdly, our statistical analysis unexpectedly shows that perceived challenges were not correlated with project success or failure. Fourthly, in order to ensure success in IAC projects, we suggest focusing on three issues: (1) Finding and maintaining a common goal; (2) focusing on mutual understanding (between academia and industry) and teamwork; and (3) making sure that managerial topics do not prevent project success. In fact, good management in research projects cannot guarantee success, but poor management can prevent or shutdown a project, therefore effectively preventing any type of success to occur.

Our future work directions include the following possibilities: (1) to use findings from this study with respect to challenges, patterns, and anti-patterns in our upcoming IAC projects; (2) to adapt the paper to have industry practitioners as the targeting audience; and (3) to analyze which issues are most likely to be relevant in order to guarantee that a given IAC project is successful, taking into account its characteristics (SE topic, culture of the country where it is being developed, project initiators, etc.). This sort of result can provide important indicators for SE researchers and practitioners, so that they can analyze them before progressing with IAC projects.

Furthermore, we plan to conduct methodological and empirical studies for improving IAC in SE by adopting novel ideas from other fields, e.g., the concept of social capital and its role in IAC (Al-Tabbaa and Ankrah 2016), the contingent role of collaboration-specific attributes in IAC (Lin 2017), and focusing on “innovation performance” in IAC (Huang and Chen 2017).