EES 2022: It’s not (only) you, it’s me (too)

Version en français ici

At the beginning of June, almost half of Quadrant Counseil’s team was in Copenhagen for this latest version of the European Evaluation Conference: three days, more than 200 sessions and more than 600 participants from Europe and elsewhere.

The previous face-to-face edition dates from before COVID and the war in Europe, it was in Thessaloniki in 2018. This edition had marked a return to issues of values and social justice, after the methodological frenzy of the years 2000 and 2010. Thanks to the online edition held in September 2021, we've avoided, at least in part, the profusion of opportunistic communications on evaluation in times of COVID assessment which has plagued many conferences. COVID was still there, but we were a little bit more in the world of tomorrow.

The title of this edition was Evaluation at a watershed, a phrase used in English to convey the idea of being “at a turning point”. Perhaps the image of the waterfall would better reflect the urgency of the situation, and the need to paddle against the current.

Beware, it’s Denmark after all, the water is fresh.

Two observations first:

• First, the papers' high quality this year. A professional or scientific conference is like a box of chocolates, but one in which some would be stuffed with sleeping pills, outdated or frankly toxic. Certainly, we were all happy to be there; we did not attend all sessions and are therefore not immune to evaluator bias or selection bias. We can still rejoice at a very good overall level of contributions, and if it were possible to go see the celebs like Michael Q. Patton, who had made the trip from Minnesota where he resides, you could also push the doors of many rooms with your eyes closed and be pleasantly surprised.
• Second observation, a different audience, with the presence of French and French-speaking people in number, after years of being a handful of us. Spanish speakers were here too! In addition to the officials of international organisations, consultants, researchers and associations had made the trip. Anyway, we didn‘t hear only English (or Danish obviously) and that wasn’t a bad thing!

Evaluators need to change

It is always difficult to say whether an international conference has focused on a particular topic, despite the themes supported by the organisers (gender-neutral English doesn't convey the fact that they were mostly women, thanks to them!). But in the end, there is a colour, a tone that emerges. In Maastricht, it was methodological diversity; in Thessaloniki, the return to values and social justice.

For this year, the Society wanted to focus on four transformations: evaluation systems, evaluators and evaluators, content — in and by evaluation, and in methodologies. These subjects were of course present in Copenhagen. They include in-depth discussions on causal mechanisms and impact assessments that are alternative or complementary to the counterfactual (including our own communication on the principles of the contribution analysis); or the always strong presence of questions relating to values and ethics. The different transformational approaches were also very present.

The central theme, however, was the awareness of the climate, social and societal emergency that we collectively face, and the complexity of the challenges that we have to deal with — something that the Director of the European Environment Agency, Hans Bruyninckx, reminded perfectly well in his presentation. By introducing the concept of “super-wicked” problems, he rejects once and for all the idea that there would be simple answers to the issues we face. Better: he invites evaluators, not so much to report on the past but to be actors in the current transitions (transition being a concept that is familiar to us in France, but not present so far in the evaluation debates).

However, this transition is not so easy. Evaluators may themselves be very conservative in their practices, or taken over by routines, or risk-adverse (as can be seen in these thousands of project evaluations using the DAC criteria, even if in private the same people who produced them would deem them inadequate). However, given the urgency, the complexity of the transitions to be undertaken, the evaluators together seemed to understand that they too had to change; that they are part of the systems that need to evolve. In his introduction, Tom Ling put it in select terms: “Let’s entertain the possibility that we might be wrong”. This could be translated more brutally: there is no more neutrality in this world. You're either part of the solution, or you're part of the problem”.

This change has been discussed at multiple levels. Let’s start perhaps with the most difficult and original in this conference, that is, changes that are intimate in nature. The world of evaluation, like others, has for many years indulged in talking about complexity and transformation — words that border on being buzzwords or fuzzwords and which would often reveal an abyssal void if one were to dig a little deeper. This year, the acronym VUCA (for Volatility, Uncertainty, Complexity and Ambiguity) was on all lips, to reflect the state of permanent crisis in which we live.

VUCA, a pill that's hard to swallow... unknown source

However, to be interested in the complex is to embark on a personal challenge. For example, learning to navigate uncertainty where evaluators like – through their monitoring systems, models and other methods – to create certainty; it means accepting being oneself part of the systems being studied, which calls for enlarging the scope of evaluations, but also abandoning false positions of independence or objectivity; questioning one's own lenses and how they affect what evaluators look at and discuss (and what they don't look at); and asking to what extent evaluation is a key ingredient of the current status quo and the difficulty of actually changing public policies.

A very good example of these developments in the intimate domain, the session of Barbara Schmidt-Abbey, Joan O'Donnell and Kirsten Bording Collins (who also won the prize for the largest number of w in a title): Being Evaluation Practitioners In A Volatile And Uncertain World: What Do We Do When We Do What We Do? They emphasise the need for reflexivity in and practice to evolve.

Barbara invites us to think systematically AND systemically

Second level of change, those that fall within the relationships between evaluators and evaluators with their environment. During these three days, a large number of metaphors were used to characterise the roles they should be able to take: The evaluator as craft artisan, bricoleur, codesigner, partner, critical friend, knowledge broker, trustful companion... In any case, in a context of maximum uncertainty, the evaluator must first try to do the right thing — and doing the right thing can be a multitude of practices, far beyond the traditional roles of evaluation. It is how, argues Martin Reynolds, evaluation can become a real public service (would we say, in France, a common good? in any case we clearly agree).

The third level, which is essential, is the systems and initiatives in which evaluators are involved: how can we ensure that we are at the heart of the dynamics of change, whether institutional, or fueled by coalitions of organisations, communities or citizens? While the evaluation community complains a little too often that its work is not being used sufficiently, something new was going in in Copenhagen: how, rather, can we be where changes are happening now, to help them as we know how to do it? How can we identify and assist those who make changes happen — even when they do not ask for an evaluation? How to meet the needs that evaluators can actually meet and meet well, even if they are not qualified as evaluative needs?

All these changes require changes in knowledge (where are the concepts, theories and frameworks that will allow us to think about transitions?); competences (including the skills needed to stimulate dynamics or make oneself heard — mostly missing from today's initial and continuous training, which are more method-oriented); but also of stance: learning to look at the future rather than the past, for example (or in the words of Stefano D’Errico and his colleagues: “From what works? to what will work?”); or assuming one's values and a moral commitment to change; being ready to use one's expert intuition in support of the interventions carried out, and how to design and implement them (where our usual credibility is in the data we collect and analyse); or finally learning to convey, in depth, what we derive from evaluations and from our accumulated experience to those actors of change — who spoke about stories to be built?

Lessons learned: cutting kittens is wrong (Joan O'Donnell, Maynooth University)

Conclusion

With this 2022 session, the most optimistic will say that the European evaluation community has begun a “skeptical turn” about evaluation, as Peter Dahler-Larser and Estelle Raimondo have stated. It is (perhaps!) beginning to recognise that evaluation is not always a panacea, that evaluators do not always do the right thing, and that they must start their transition — be the agents of change they call for. It’s not easy... We think of the late Eleanor Chelimsky: it takes courage to be an evaluator...

We want to believe that this is a longlasting change. A good evidence of this would be to see, at the next conference, those who were missed the most, i.e. those actors of ecological, social or societal change, who have everything to gain from evaluation but remain strangely absent from these kinds of conferences. Let's do it!

Thomas Delahais

P.S.: EES was perhaps for the first time this year sensitive to its carbon footprint. It was possible to come by train (and we‘re not the only ones), but it wasn’t that easy! Why not next time choose a destination accessible to the greatest number and encourage to come by train, and leave the plane to those who cross the oceans?

Thanks to Sébastien Galéa, Benoît Simon and Quadrant Conseil colleagues for their comments and suggestions on the initial text; and thanks to May Pettigrew and Barbara Schmidt-Abbey for their suggestions on the English text!

Rare image of a change maker – an elected official, actually – lost among evaluators