Introduction

Environmental problems, animal diseases, problems with animal well-being and food scandals have marked the agricultural sector in the Netherlands and elsewhere. In the 1990s, these problems came high on the political and societal agenda, and in the Netherlands plans were developed to redesign the agricultural sector into a more sustainable direction. Government, agricultural sectors, and other involved groups started to take actions to make agricultural practice more sustainable. Performance indicators were developed to see to what extent these actions were effective. Social (e.g., employment rates), economic (e.g., production rates), and ecological (e.g., emissions and waste rates) performance indicators were used to monitor agricultural practice (see Boone et al. 2007; Brouwer et al. 2004). Comparing the quantitative results of the monitoring process from year to year gave insight into the development of sustainability performance of the agricultural sector.

Although very valuable for policy makers, the numbers that result from this type of evaluation do not give insight into the normative dimension of sustainability. They do not reflect how the sector feels about the relative importance of the social, ecological, and economic dimensions of sustainability, about the ambitions of the sector with regard to sustainability and whether the agricultural sector is “on the right track” towards realizing these ambitions. To better grasp the normative dimension, the Ministry of Agriculture, Nature, and Food Quality in the Netherlands requested for a participatory, qualitative evaluation of sustainability performance of Dutch agriculture. The evaluation was to be carried out as a dialogue approach with actors from the agricultural sector.

We believe that, due to the growing attention to sustainability and due to the international trend towards governance and stakeholder involvement (see also Bruges and Smith 2008; Pretty and Vodouhê 1997), there is a need for well-documented evaluation approaches in the field of sustainability. With this article, we hope to meet this need by sharing our experience with a participatory evaluation approach to Dutch agriculture and by reflecting on the usefulness of the evaluation outcomes for Dutch agricultural policy and for people outside the Netherlands. We start with positioning our approach in evaluation literature, most notably referring to the Fourth Generation Evaluation approach by Guba and Lincoln (1989). In the next section we present our approach to evaluate sustainability performance of Dutch agriculture, characterize this approach as a Fourth Generation Evaluation, and present its main outcomes. We then introduce a set of quality criteria—credibility, transferability, fairness, and authenticity—and use these criteria to evaluate our approach. In the last section we draw conclusions on the usefulness of our approach for policy.

Different approaches to evaluation

The literature on evaluation is rich and diverse. Several categorizations of evaluation approaches are presented and discussed. A general distinction that is made is between “classic” evaluation forms on the one hand, and “democratic” and “constructivist” evaluation forms on the other (Hanberger 2001; Edelenbos and van Buuren 2005). Whereas the methodology of classic evaluations is positivistic in the sense that it assumes to be objective, neutral, and presenting the facts, democratic and constructivist forms of evaluation are characterized by the inclusion of stakeholders and emphasis on values and worldviews rather than on facts (Garaway 1995; Abma 2004).Footnote 1

To illustrate developments in evaluation procedures over time, we refer to the work of Guba and Lincoln (1989) who distinguished four forms of evaluation, ranging from First Generation to Fourth Generation Evaluation. First Generation Evaluation represents a technical approach and can best be compared with school tests (Guba and Lincoln 1989). In these tests children demonstrate mastery of the “facts” and the evaluator (teacher) has a technical role, measuring variables and using appropriate instruments for this. Second Generation Evaluation is characterized by a descriptive and objective-oriented approach in which students are not the only object of evaluation anymore, like in First Generation Evaluation. In this approach, patterns of strengths and weaknesses with respect to certain stated objectives (desired learning outcomes) are described. The evaluator has the role of describer (although the technical aspects of that role are also still important). Measurement is no longer the equivalent of evaluation, but one of several tools that may be used in its service. Third Generation Evaluation is characterized by efforts to include judgments in the act of evaluation. Due to its descriptive nature, this is generally neglected in Second Generation Evaluation approaches. In Third Generation Evaluation, not only performance but also the objectives themselves become subjected to evaluation. Standards are needed against which judgments can be made and the evaluator fulfills the role of judge (retaining the earlier technical and descriptive functions as well).

Guba and Lincoln (1989) see a number of pervasive problems with the first three generations of evaluation approaches. The first problem is that the approaches have a tendency towards managerialism, the manager often being the client or sponsor who commissions or funds the evaluation. The second problem is the failure of the approaches to accommodate value pluralism as they assume objectivity and value-freedom in evaluation procedures. The third problem is overcommitment to the scientific paradigm of inquiry, meaning that the evaluation approaches ignore the context in which they takes place, too strongly rely on “hard” quantitative data (“numbers”), truth finding, and scientific rigor. Furthermore, Guba and Lincoln (1989) state, adherence to the scientific paradigm relieves the evaluator of any moral responsibility for his or her actions.

In response to these problems, Guba and Lincoln (1989) propose an alternative approach which they characterize as a Fourth Generation Evaluation approach. This is a participatory, constructivist approach to evaluation, whose key dynamic is negotiation. Guba and Lincoln (1989, pp. 8–9) assign six properties to the Fourth Generation Evaluation approach. First, it takes the position that evaluation outcomes are not a description of “how things really are” but of how individual actors or groups of actors “make sense” of their situations. Second, it recognizes the plurality of values that shape the constructions through which people make sense of the situations in which they find themselves. Third, it suggests that the constructions that people make are linked to the (social, cultural, political) context in which they are formed and to which they refer. Fourth, it recognizes that this form of evaluation can empower or disempower particular stakeholder groups in a variety of ways. Fifth, it suggests that evaluation must have an action orientation, in order to relegate follow up and prevent the non use of evaluation outcomes (see also Greene 1987). Sixth, it insists on full participatory involvement, in which the participants are equal partners in every aspect of the evaluation process.

A fourth-generation evaluation of Dutch agriculture

In the Netherlands, the Ministry of Agriculture, Nature and Food Quality wanted to get insight into ambitions, perceptions, and interests that agricultural stakeholders have with regard to sustainability. The Ministry requested an evaluation approach that involves agricultural stakeholders and encourages learning about ways to make the agricultural sector more sustainable. To be able to monitor if and how perceptions and ambitions of stakeholders may change over time, the Ministry considered repeating the evaluation approach every few years.

Two Dutch research groups, who are specialized in stakeholder dialogues and process facilitation, were asked to design the evaluation approach. This resulted in a design for a series of stakeholder workshops with actors from various agricultural sectors. The evaluation design has a number of “Fourth Generation” properties (see Table 1), which we explain below.

Table 1 Fourth Generation Evaluation properties in the stakeholder dialogues on Dutch agriculture
Full size table
Table 2 Sustainability matrix that was used in the ‘silent wall’ method
Full size table

Aim of the stakeholder workshops

The evaluation design included seven stakeholder workshops that were to take place between April 2006 and December 2006.Footnote 2 The first six workshops were about the main agricultural sectors in the Netherlands: pig farming, arable farming, greenhouse horticulture, dairy farming, poultry farming, and open-air cultivation. The seventh workshop was a synthesis meeting in which three crosscutting themes were discussed that had been derived from the sectoral workshops.

Main aim of the sectoral workshops was to get insight into participants’ views, expectations, and ambitions with regard to sustainability performance of their specific sector. The time horizon for the workshops was the year 2020, which was expected to be sufficiently far away to encourage participants to take a distance from their present situation, interests, and concerns, and to be creative and open-minded. At the same time, the year 2020 was expected to be sufficiently near so as to prevent the discussion from becoming fantasizing, utopian, and unrealistic. The people who took part in the workshop were asked the following questions:

  1. 1.

    What sustainability ambitions should your sector have for the year 2020?

  2. 2.

    In the light of these ambitions, how does your sector currently perform?

  3. 3.

    Which trends and factors influence—positively or negatively—realization of the sustainability ambitions?

  4. 4.

    What kind of actions are required to realize the ambitions and by whom?

The synthesis workshop had a broader focus. Main aim of this workshop was to get insight into participants’ views, ideas, and expectations with respect to three crosscutting themes that are relevant to most of the agricultural sectors: (1) innovation, (2) the retail as a linchpin between producers and consumers, and (3) tensions between agriculture and spatial planning.

Design of the stakeholder workshops

The workshops were designed to give room to diversity of viewpoints in at least two ways, relating to the methods that were used and the mode of moderation. Each of the sectoral workshops followed more or less the same procedure.Footnote 3 In this procedure, the “silent wall” method was used to identify and discuss participants’ ambitions with regard to sustainability performance of their specific sector. The project team covered a wall of the meeting room with paper and participants were invited to write down their ambitions. The silent wall was structured by means of a sustainability matrix (see Table 2). The silent wall method is based on the assumption that it leads to a fair process in which all participants have equal opportunity to contribute to the discussion. During the silent wall exercise, participants are not allowed to speak with each other, communication takes place by writing notes, questions, and remarks on the wall.

After the silent wall the group took a collective decision on which three ambitions they wanted to discuss further. Usually, the group chose three ambitions in line with the three dimensions of sustainability: social, ecological, and economic. The subgroups discussed sustainability performance of the sector in the present situation and they identified factors, trends, and actions that are needed to realize the specific ambition. At the end of the workshop, the subgroups discussed their results with each other and they received a reflection from a sector-specific scientific expert.

For the synthesis workshop, the “fish-bowl” method was used. Three rounds of discussion took place, one for each of the themes. Participants were divided into two circles, an inner circle that hosted six to eight participants who were experts on the specific theme, and an outer circle that hosted the other participants. The participants in the outer circle were allowed temporarily to step into the inner circle to contribute to the discussion.

In addition to these methods, also the mode of moderation aimed to articulate a plurality of viewpoints and opinions. At the start of each sectoral workshop, the moderators emphasized the importance of a variety of viewpoints and opinions to be expressed, also (maybe even particularly) the viewpoints that do not fit the mainstream view. The moderators also stressed that their aim was to facilitate an open dialogue and that they did not have a predetermined position with regard to the sector. There was a team of three moderators who jointly organized and facilitated the six sectoral dialogues. For the synthesis workshop, the project team involved an external moderator who was well known and well established in the agricultural sector and who was also asked to encourage an open dialogue.

Participants in the stakeholder workshops

Since it is difficult to involve everybody in the inquiry process, a relevant question was whose viewpoints and values are going to be taken into account, whose will be excluded, and on the basis of which selection criteria? The stakeholder workshops needed to be able to do justice to the complex character of the issue of sustainability. This required the plurality of positions, interests, and values that characterize the issue of sustainable agriculture needed to be included in the dialogue process. To identify and select participants, the project team together with a sectoral expert composed a first list of about sixty potential participants. Two criteria were used for this. The first criterion was the position and interest of an actor in the chain of goods and services for the specific sector. We aimed for a diverse group of representatives including farmers, agribusiness, retailers, NGOs, policy makers, etc. The second criterion was the kind of information and insights that an actor can contribute to the discussion. Also here, we strived for diversity, including scientific knowledge, practical experience, creative ideas, strategic insights, etc. This list was complemented with additional potential participants whom we found through Internet search and whom were suggested to us by other experts. Also, we asked a number of key actors from the specific sectors to comment upon our list of potential participants and to provide additions. This resulted in a final list of about eighty to ninety potential participants.

For the synthesis workshop we did not so much invite participants with a sector-specific interest but we aimed for generalists with a broad scope on agriculture and sustainability, and for people with knowledge and experience on one of the three themes that were on the agenda. The identification and selection of participants took place on the basis of an Internet search, suggestions of external experts, and key actors (snowball method). The sector-specific experts were not consulted this time.

Role of the results of the stakeholder workshop for policy

The Ministry of Agriculture, Nature, and Food Quality had the obligation to report on an annual basis to parliament about the progress that was made in the agricultural sector towards sustainability. The results of the stakeholder workshops were seen as an input to this report, together with the results from the quantitative monitoring activities. Both these inputs would enable the Ministry to put the quantitative monitoring results in the perspective of long-term ambitions for the agricultural sectors.

Focus on stakeholders’ own roles and responsibilities

The discussion during the stakeholder workshops on how to realize the sustainability ambitions in the various agricultural sectors focused on both the role of government and the role of other stakeholders in the sectors. It was seen as a desirable side effect of the workshops if participants felt prompted and empowered to act, but the workshops were not particularly designed for this.

Full participation of the stakeholders

The stakeholder workshops needed to be fully participatory. This means that participants needed to be involved not only in the actual workshops but also in the design and preparation of the workshops. Full participation is likely to enhance the use of evaluation outcomes (Greene 1987). We followed this logic in the preparation of the workshops. For each of the sectoral workshops, a sector-specific expert was asked to develop a fact sheet with data on production, import and export, emissions, use of fertilizers and pesticides, and employment rates that characterize the sector. The expert was also asked to make a list of relevant actors in the sector and to suggest three to four key actors with whom the project team should conduct interviews. The results of these interviews would then help the project team prepare the workshop.

The synthesis workshop was prepared in a slightly different way. The project team used the outcomes of the sectoral workshops to extract three important cross-sectoral themes to be discussed. Also, the project team conducted six interviews with key actors, and several external experts were consulted about the three themes and the participants to be invited.

The “goodness” of a Fourth Generation Evaluation

Based on the work by Guba and Lincoln (1989) and others (e.g., Pretty 1994; Webler 1995; Pretty and Vodouhê 1997; Groot 2002; Abma 2004) we selected the following four criteria to judge the quality of our evaluation approach: credibility, transferability, fairness, and authenticity.

Credibility

Credibility refers to the match between the constructed realities of respondents (stakeholders) and those realities as presented by the evaluator and attributed to various stakeholders (Guba and Lincoln 1989, p. 237). In the evaluation of the Dutch agricultural sector, credibility is understood as the degree of correspondence between the interpretation of outcomes by the project team and the perception that the participants in the workshops had about the dialogue results and their own contributions to it. Credibility can be assessed by means of “participant checks,” i.e., testing hypotheses, data, and interpretations with members of the stakeholders groups from whom the original constructs were collected, “peer debriefing,” i.e., involving scientific peers who have not been involved in the evaluation to reflect on the outcome, and “triangulation,” i.e., the use of a variety of sources, methods, perspectives, and researchers.

Transferability

Transferability refers to the extent to which the receiver (the user) is able to apply the evaluation outcomes to his/her own context (Guba and Lincoln 1989). To allow receivers to make transferability judgments about the evaluation outcomes of the Dutch agricultural sector, all the hypotheses that were set out for the evaluation need to be described, as well as the time, place, context, and culture in which those hypotheses were found to be salient.

Fairness

Fairness refers to the extent to which different constructions of people, and their underlying value structures are solicited and honored in the evaluation process. These different constructions must be presented, clarified, checked and taken into account in a balanced way. Following Webler (1995, p. 51), we relate fairness of the stakeholder workshops to four requirements. First of all, anyone who considers him- or herself to be potentially affected by the results of a workshop must have an equal opportunity to attend that workshop. Secondly, every participant in a workshop must have an equal opportunity to make validity claims. Thirdly, every participant must have an equal opportunity to challenge the claims made by others. Fourthly, every participant in a workshop must have an equal opportunity to influence the agenda (in terms of topics to be discussed) and the outcomes of that workshop.

Authenticity

Authenticity refers to the extent to which the people involved have improved their own constructions and have increased their understanding of the constructions of others.

Evaluating the Dutch stakeholder workshops on sustainable agriculture

To evaluate the stakeholder workshops on the basis of the four criteria mentioned above, we use data from questionnaires that the participants filled out at the end of each workshop, observation reports that the project team prepared after each workshop, and feedback from the project’s steering committee and other experts. Before we do this, we first present the main outcomes of the stakeholder workshops, to give insight into the sustainability ambitions that the participants formulated and the activities that feel need to be undertaken.

Main outcomes of stakeholder workshops

In each sectoral workshop, the groups formulated ambitions with regard to the people, planet and profit dimensions of sustainability. It turned out that the ambitions in the various sectors were rather similar: (1) to improve the image of the agricultural sector in favor of a “society driven” and “society based” sector (people); (2) to organize closed cycles and closed systems for animal production sectors (in terms of nutrient, energy use, water); for plant production sectors the ambition was to diminish the pressure on the environment (planet); and (3) to establish continuation in economic terms and to improve the position of the sector in the market (profit). To realize the ambitions, the stakeholder groups identified particularly four activities that need to be carried out:

Improve education and knowledge transfer

One of the participants in the greenhouse horticulture workshop said: “Education can be the bridge between knowledge and practice.” Another participant mentioned that: “Generic knowledge needs to be translated to more specific knowledge in an interactive process between users and suppliers of knowledge.” Better education, from primary school onwards, can improve children’s knowledge about and interest in the agricultural sector. Agricultural training courses should be improved to promote the development of farming profession in a more sustainable direction. Educational institutions can also help to develop sustainable agriculture as an educational theme.

Create opportunities for entrepreneurship

“Entrepreneurs should be able to be real entrepreneurs and they should not become victimized by their surroundings,” according to a participant. Much of present-day legislation is based on means rather than ends, which does not stimulate farmers to apply their entrepreneurial skills to meet sustainability targets. Therefore, government needs to create the right conditions for sustainable agriculture and tailor-made solutions. Agricultural stakeholders also asked for a consistent long-term agricultural policy and for government to think along with them.

Encourage knowledge development and innovation

A strong knowledge base is a strength of Dutch agriculture—even a potentially successful export product—but also a matter of concern for agricultural stakeholders. Particularly smaller agricultural sectors wondered whether sufficient knowledge would continue to be available. By funding research, the Ministry of Agriculture, Nature, and Food Quality contributes to knowledge development and innovation in agriculture. As one participant put it: “Government should intensify its efforts, stimulating entrepreneurial skills and encouraging innovations, both technical and social innovations.”

Create a level playing field

Internationalization of the agricultural market offers both opportunities and threats. Participants expected the Ministry to use its European contacts to stimulate a level playing field in which Dutch farmers can compete under fair and equal conditions with other European farmers. One of the participants in the pig farming workshop said: “For reasons of competiveness, the Netherlands should not take a front position in the implementation of EU regulations.”

These four activities mainly concern government actions but the stakeholders also acknowledged their own responsibilities to realize the ambitions. They agreed to seek more collaboration, not only within the agricultural sectors but also with experts at universities and research institutes and with government. In addition, stakeholders realized they can make further efforts to improve their public image, for instance by adapting their production processes and developing certification schemes.

Credibility of the stakeholder workshops

How credible has the participatory evaluation of Dutch agriculture been? As mentioned, credibility refers to the correspondence between the outcomes as interpreted by the project team and the perception of stakeholders about project results and their own contributions to it. The degree of credibility depends on the presence of sufficient participants’ checks, peer debriefing, and triangulation.

The project team conducted participants’ checks at several occasions, before, during, and after the workshops. Before each workshop, the project team conducted interviews with key actors and the interview reports were sent to them for comments. Their additions and revisions (often just a few) were integrated into the final interview reports. During each workshop, subgroups presented their results to each other and there was opportunity for discussion. At the end of each workshop, a draft report was sent out to the participants with a request to provide comments. In general, the participants only suggested small changes, no major revisions, which may suggest that the project team had interpreted the discussions during the workshop well.

To some extent the project team also used “peer debriefing” as a way to enhance credibility, although not in a structured way. Sectoral scientific experts were asked to comment upon the draft report of each workshop and, in a more informal way, the workshop report has been discussed with peers. During these discussions, the peers emphasized that the outcomes of the stakeholder workshops should be seen “snapshots,” taken at a certain point in time and produced by a certain group of stakeholders, and should therefore be considered with care.

In addition to peer debriefing on the contents, at the end of evaluation process the project team organized peer debriefing on the methodological design of the workshops with a group of experts in the field of process facilitation. During this discussion, doubts arose about the extent to which the workshops were “true dialogues” in which actors actually question each other’s assumptions, values, and goals. In the stakeholder workshops there was not enough time for such an intensive interaction to happen.

The last criterion to evaluate credibility is triangulation, which refers to the use of a variety of sources, methods, evaluators and perspectives. The project team used several sources and methods during the preparation of the workshops, such as reports (literature review), stakeholders from the sector (interviews) and the Internet (digital search). For each workshop, the project team consulted one or two scientific sectoral experts to provide information about the situation in the sector, relevant stakeholders, etc. These experts all worked at the same institute and all had their own network established in “their” sector. Although these experts were very knowledgeable, from the perspective of triangulation, it would have been better not to rely entirely on their input but to also involve other scientific experts, who came from other institutions and who had other perspectives and networks. This was only done to a limited extent and in an ad hoc way.

During the workshops, triangulation of methods was limited. The project team only applied deliberative methods, notably the silent wall (during the sectoral workshops) and fish-bowl method (during the synthesis workshop), to stimulate the participants to discuss and to exchange viewpoints and ideas.

Triangulation of evaluators was also limited. A team of three moderators (who were part of the project team) facilitated the subgroup discussions. Usually, the same moderator facilitated respectively the “social,” “ecological,” and “economic” subgroup, as this allowed the moderators to compare between the different sectors. From the perspective of triangulation, however, it might have been better for the moderators to rotate so as to prevent the moderator’s perspective to dominate the discussion.

Triangulation of perspectives was pursued during the workshops by means of the group of participants who represented various parts of the chain of goods and services (see also under “fairness”).

Transferability

This criterion refers to the extent to which a user is able to apply the evaluation outcomes to his/her own context. To allow receivers of the workshop results to make transferability judgments, the project team wrote two detailed reports on, respectively, the results (Borgstein et al. 2007) and methodology of the dialogues (Groot et al. 2007). These reports describe the goals, procedure, and outcomes of the dialogue workshops, as well as the assumptions on which certain choices were made (e.g., with regard to method selection and stakeholder selection). What is lacking in the reports is a description of the political context in which the dialogues took place. At the time the project was initiated—about four years ago—the then Minister of Agriculture had established a “think tank” for the transition towards sustainable agriculture. The idea of a stakeholder dialogue with actors from various agricultural sectors corresponded well with the ideas of the think tank to induce a transition process within the agricultural sector towards a more sustainable performance. However, in 2001, the term “transition” was phased out, the term “innovation” was introduced, and the attention for sustainability slowly declined. A new Minister came, who focused much more on the economics of the agricultural sector rather than on the environmental aspects. In other words, at the time the project was developed the political context was more oriented towards sustainability in the agricultural sector than at the time the project was carried out. This change in the political context could be of relevance for those who try to judge the transferability of the dialogue results.

In the questionnaires, the participants expressed a rather skeptical view of the degree of transferability of workshop results. Participants doubted whether the dialogue discussions were an appropriate reflection of the discussions that generally take place in the sector. They referred to group composition and argued that some relevant actors were missing. One of the participants in the workshop on open-air cultivation used the term “usual suspects” to characterize those who attended. In the workshop on pig farming the international market was hardly discussed, most probably because a large Dutch meat producing organization that is an important player in the international market was not present at the workshop. The participants also referred to workshop planning and argued that there was not enough time to discuss all relevant themes.

Fairness

Fairness refers to the extent to which different constructions, values, and opinions of people are solicited and honored in the evaluation process. For the dialogue workshops to be fair, they needed to meet four conditions: the ability to attend, to contribute, to discuss, and to decide. Regarding the ability to attend, the project team actively recruited in different parts of the chain of goods and services so as to involve a diverse group of people, but if somebody who was not invited wanted to participate on his/her own initiative, the project team welcomed this person to attend the workshop. This however hardly happened. How did the group composition work out? The average response rate was about 40% and each stakeholder workshop was attended by 30–40 people. About one-third of the participants were working in the primary sector as farmer, grower or nurseryman. Most of them were men. In each workshop about three researchers, working for different research institutes, participated. The same number counts for NGOs. Depending on the agricultural sector at stake, each workshop was attended by two to three representatives of different NGOs such as the Animal Protection Society, Staatsbosbeheer,Footnote 4 and the Netherlands Society for Nature and Environment. In most of the workshops, the provincial government was represented by one person only. Regarding the ministries, each workshop was joined by at least one staff member of the Ministry of Agriculture, Nature, and Food Quality. In a few cases, a representative of the Ministry of Housing, Spatial Planning, and the Environment was present as well. It appeared to be more difficult to motivate the large-size agribusiness companies, and the retail industry, and banks to participate in each of the sectoral workshops. These people were more interested in participating in the synthesis workshop, which was of a more general (i.e., not sector specific) character.

The second and third conditions for fairness are taken together as they are very much related. The ability to contribute is about the ability of the participants to make validity claims. The ability to discuss concerns their ability to challenge the validity claims made by others. The design of the stakeholder dialogues was geared to meet these conditions and to allow all participants to say what they considered to be important. As a participant in the workshop on greenhouse horticulture described it: “An informal atmosphere was created during the workshop due to which a large variety of issues could be discussed.”

During the introductory part of the workshop, all the participants received the opportunity to express their personal ambitions for the year 2020.Footnote 5 The silent wall exercise particularly encouraged participants who were a bit shy or who were easily intimidated by other (more dominant) participants, to contribute to the discussion. And the discussion in fairly small subgroups (up to eight people) also encouraged more silent and/or shy people to speak up. In the synthesis workshop, particularly participants in the inner circle were motivated to contribute and to discuss. It turned out that most people in the outer circle did not use the opportunity (and were not encouraged sufficiently by the moderator) to step into the inner circle to join the discussion, they mainly listened.

That workshop design enabled the participants to contribute and discuss during the dialogue process shows from the participants’ questionnaires (see Fig. 1). On a scale of 1 (low) to 10 (high) the participants evaluated their ability to express their opinions during the seven workshops with an average score of 7.8. The workshop on pig farming was evaluated least positive (7.5), closely followed by the synthesis workshop (7.6), whereas the workshop on open-air cultivation had the highest score (8.2). The relative low score of the pig farming workshop may the result of the alternative design that was used for this workshop (see footnote 5).

Fig. 1
figure 1

Participants’ evaluation of their ability to express their opinions (scale 1–10)

Full size image

The last condition for fairness, the ability to decide, refers to the ability of participants to influence the agenda and the outcomes of the dialogue workshops. The workshops (particularly due to the use of the silent wall method) were characterized by a bottom up approach in which participants were able to decide upon issues they wanted to discuss, within the general aim and design of the project and the particular workshop. Participants were also able to contribute to collective decision making about the outcomes of the workshops, for instance during the plenary feedback session in which the subgroups were given the opportunity to comment upon each other’s results. Although the participants appreciated the bottom-up approach that was followed, the results of the questionnaires reveal a critical view on the degree to which this approach had led to a discussion about the “right issues” (i.e., issues that are relevant for the sector) (see Fig. 2).

Fig. 2
figure 2

Participants’ evaluation of degree to which discussion was about the “right” issues (scale 1–10)

Full size image

On a scale of 1 (low) to 10 (high) the average score for this was 6.9, which is low compared to the other fairness results (in Fig. 1). Several participants stated that many relevant issues were touched upon but not discussed thoroughly enough. Others said the discussion remained at too abstract a level and should have been more concrete. A few participants claimed that the workshops did make clear the ambitions for the sectors but insufficiently addressed the instruments and policies that are needed to realize these ambitions.

Authenticity

This criterion is about learning and refers to the extent to which participants have improved their own constructions (ontological authenticity) and their understanding of the constructions of others (educative authenticity). In general, the dialogue workshops were characterized by an open and fair atmosphere in which people listened to each other and were actively engaged in discussions. The questionnaires did not include specific questions that relate to authenticity, but we have some indications and observations that enable us to apply this criterion.

In the questionnaires, several participants argued that the dialogue workshops had not yielded many surprising or new results for them. According to them, the issues and problems that were found important during the workshops (notably, a bad image with the public and too strict rules and regulation) are the same issues and problems that the sector has faced for years. Observations during the workshops, however, made clear that participants were often surprised about the results of their discussions. They were for instance surprised about the large amount and variety of ambitions that resulted from the silent wall discussion. During the workshop on arable farming the participants stated that they had expected the ambitions to focus on the economic dimension of sustainability but according to the silent wall results the sector pursued many other ambitions as well such as nature conservation, transparency, environmental quality, and a consumer orientation. Another surprising result was that, despite the international character of the agricultural sector, for many participants it turned out to be easier to formulate ambitions at the national level.

Another example comes from the workshop on pig farming. From the results of the silent wall discussion it became clear that participants found it difficult to think in terms of sustainability ambitions. Some of them wrote down ambitions such as “animal health,” “attention for the environment,” and “innovation” but many of them tended to think in terms of obstacles (e.g., rules and regulation) that needed to be removed to make their lives as pig farmers easier. It took quite some effort from the moderator to make these participants realize that removing these obstacles should not be seen as an ambition but as a means to realize a certain ambition. After the participants realized this, the discussion about ambitions became much more fruitful.

Learning at the collective level and better understanding the constructions of others (educative authenticity) was not always easy to achieve. A notable example comes from the pig farming workshop in which many farmers felt that rules and regulations by government put too many restrictions on them. They also mistrusted animal protection NGOs, particularly the more radical ones (one farmer strongly suggested to not invite these anymore). The high level of mistrust and skepticism made farmers reluctant to listen to the alternative viewpoints of these participants, and one participant said that: “The results of this workshop do not reflect an approach by which the sector will succeed in the future.”

But fortunately there were also successes. In several workshops, the participants realized there were tensions between the different dimensions of sustainability. For instance, the tension in diary farming between cost price reduction by up scaling on the one hand (“profit”) and the image of the agricultural sector and animal well-being on the other hand (“people”). Or the tension between the open systems in which animals can walk outside to improve image and animal well-being (“people”), and the closed systems in which animals are kept inside to reduce environmental pollution (“planet”). Although these tensions are not new and they were definitely not solved, the stakeholder dialogues did contribute to a further understanding of the various viewpoints and positions.

Conclusions

To wrap up, we will address three questions: (1) Did the approach to evaluate sustainability in Dutch agriculture reach its goals? (2) Did the evaluation provide useful insights for Dutch policy makers? and (3) Did the evaluation provide useful insights for people outside the Netherlands?

Did the evaluation approach reach its goals?

The dialogue workshops pursued four goals: (1) to get insight into perceptions and visions of involved actors with regard to sustainability in the agricultural sector, (2) to get insight into sustainability ambitions of involved actors and into the differences and similarities between these ambitions, (3) to develop a monitoring approach that can be used every few years to see how perceptions and ambitions of involved actors with regard to sustainability develop, and (4) to encourage learning about ways to make the agricultural sector more sustainable.

These four goals have been realized to a large extent, but improvements can be made. Regarding the first two goals, we conclude that the results of the workshops give insight into the sectors at a given moment in time, based on the perceptions of a particular group of actors. If the dialogue workshops had been more of a “true dialogue,” the insights that were generated would have had a more profound character and be based on a confrontation rather than on a registration of viewpoints and knowledge. Regarding the third goal, we conclude that the dialogue workshops have made clear which elements of the workshop design worked well and which elements need to be improved for the dialogue workshops to be an appropriate monitoring methodology. The silent wall approach proved to be an efficient method to articulate people’s ambitions and to ensure that everyone (also the silent people) actually contributed to the discussion. Another conclusion we draw is that the workshop program was too ambitious, given the time available. The groups only had a few hours available to formulate ambitions, describe the current situation, and discuss actions that are needed to realize the ambitions. On a future occasion, more time needs to be spent on preparing the workshops (e.g., a more extensive interview round) and/or on the dialogue process itself (e.g., two workshops instead of one). A third conclusion is that the current dialogue workshops were designed as an evaluation process rather than a monitoring process. This was also intended, as it was the first time that such a dialogue process took place. Now the dialogue workshops have provided a “t = 0 measurement,” the next series of workshops will need to be redesigned to be able to provide a “t = 1 measurement.” One way to do this could be to confront the participants involved in the next round with the results of the current evaluation to see if/how their opinions and perceptions have changed and why.

It is hard to draw conclusions about the fourth goal as the learning effect of the dialogue workshops has not been evaluated in a structured way. Fortunately, there are several indications that the participants have learned about both their own constructions and the constructions of others (see section on authenticity). However, the learning effect could have been higher if the workshops had allowed for more deliberation and debate, as this would have given insight into the arguments and assumptions that are behind the expressed opinions and statements.

Does the approach provide useful insights for Dutch policy makers?

To answer this question, we refer to Van Zeijts et al. (2007) who compared the results of the stakeholder workshops with the results of the quantitative evaluation that is common in Dutch agriculture. The comparison suggests that the two approaches are complementary to each another. The quantitative evaluation gives insight into the progress of the past years with regard to social, economic, and ecological indicators, such as food safety and certification (“people”), family income and level of innovation (“profit”), and pesticides and water use (“planet”). The stakeholder workshops make clear what are the goals and ambitions that agricultural stakeholders have with regard to sustainability and how the current situation needs to be changed to realize these ambitions. These “soft” insights provide the quantitative evaluation approach with a reference to better understand the “numbers,” to prioritize the used indicators and identify new indicators on the basis of societal preferences and values.

Both evaluations show that Dutch agriculture gradually becomes more sustainable and more responsive to societal needs and preferences. The environmental pressure has gone down, more and more farmers are involved in agricultural nature conservation and the sector has become more transparent about its activities (e.g., through labeling). The quantitative evaluation results also show that in the past five years no major changes with regard to environmental quality have taken place in Dutch agriculture. The stakeholder workshops provide an explanation for this tendency, which is that the “low-hanging fruit” has already been picked and the agricultural stakeholders have not increased their environmental ambitions during the last few years, except for renewable energy (particularly in greenhouse horticulture). This is mainly because environmental policy for the agricultural sector, which is the main driver for the agricultural sector to invest in environmental measures, has not developed much in the last few years. Also, there were other points of concern that needed the agricultural stakeholders’ attention, such the low level of innovation in some of the sectors and the lack of interest with young people to work in the agricultural sector.

An important contribution of the stakeholder workshops to the evaluation of Dutch agriculture is the identification of new indicators and the re-prioritization of existing indicators. Stakeholders felt that the “number of students at agricultural schools” is an important indicator for assessing Dutch agriculture from a people’s perspective. This indicator has so far not been used in quantitative evaluation but as a result of the stakeholder workshops can now be incorporated in future evaluations. Also other indicators can be considered to become included in quantitative evaluation procedures, such as “image” and “transparency.” Economic indicators about innovation and the level playing field may also be interesting to include.

The comparative study (Van Zeijts et al. 2007) is rather positive about the usefulness of the results of the stakeholder workshops. One indication that the results are actually used by policy makers is a recent letter on “sustainable dairy farming” of the present minister of Agriculture, Nature and Food Quality that was sent to Dutch parliament (Verburg 2008), containing information derived from the stakeholder workshops.

The participants were skeptical about the proper use of the workshop results by responsible policy makers (see Fig. 3). On a scale of 1 (low) to 10 (high), the average degree of trust that the results will be considered in the development of new policies for the agricultural sector is 6.8. Lowest score comes from the workshop on pig farming (5.8). This is not surprising as this sector appeared to have the strongest mistrust towards government in general. Other sectors, such as dairy farming, are more optimistic (7.4).

Fig. 3
figure 3

Degree of participants’ trust that the results of the workshops will be properly used by the responsible policy makers (scale 1–10)

Full size image

Does the evaluation approach provide useful insights for people outside the Netherlands?

The present-day trend in agriculture in the Netherlands is towards more sustainability with more openness to the wishes of society. Agricultural policy objectives are particularly focused on issues that strongly influence the public image of agriculture such as food safety, infectious animal diseases, and animal welfare. The Dutch Ministry of Agriculture, Nature, and Food Quality, strives for safe, responsibly produced food and a lively countryside. Rural areas must remain economically viable while preserving landscape and heritage. The ministry also seeks to strengthen the international competitive position of the agriculture sector based on socially responsible enterprises. In line with Dutch policy in general, the working philosophy of the ministry is to use a decentralized or regional approach wherever it is possible and a central national approach whenever it is considered needed.

These trends are not specific for the Netherlands. There is an international trend towards decentralization, multi-actor governance, and sustainability. This trend has led to a need for approaches and methodologies that support multi-stakeholder assessments in ongoing processes, and for evaluations of such approaches and methodologies. The approach that is documented in this article is an example of a Fourth Generation Evaluation approach of sustainable agricultural practice and policies that allows relevant stakeholders to articulate and negotiate their viewpoints. The Fourth Generation Evaluation approach as critically reflected upon in this article provides useful insight in process architecture, in terms of a justification of methods used, sequence of methods, and process dynamics. It also addresses the role of evaluators acting as change agents possessing specific process facilitation skills. It has also raised at least three points of attention for evaluation practitioners. The first relates to group composition and the need to collect and commit a diverse group of stakeholders with different (conflicting) interests and values. The second relates to learning and the importance of establishing a “true dialogue” (see also Van de Kerkhof 2006) with plenty of room for debate and confrontation of opinions, values, and perceptions. The third relates to the need for an action orientation in evaluation design (Guba and Lincoln 1989).