EES 2022: It’s not (only) you, it’s me (too)

Version en français ici

At the beginning of June, almost half of Quadrant Counseil’s team was in Copenhagen for this latest version of the European Evaluation Conference: three days, more than 200 sessions and more than 600 participants from Europe and elsewhere.

The previous face-to-face edition dates from before COVID and the war in Europe, it was in Thessaloniki in 2018. This edition had marked a return to issues of values and social justice, after the methodological frenzy of the years 2000 and 2010. Thanks to the online edition held in September 2021, we've avoided, at least in part, the profusion of opportunistic communications on evaluation in times of COVID assessment which has plagued many conferences. COVID was still there, but we were a little bit more in the world of tomorrow.

The title of this edition was Evaluation at a watershed, a phrase used in English to convey the idea of being “at a turning point”. Perhaps the image of the waterfall would better reflect the urgency of the situation, and the need to paddle against the current.

Copenhagen Beware, it’s Denmark after all, the water is fresh.

Two observations first:

Evaluators need to change

It is always difficult to say whether an international conference has focused on a particular topic, despite the themes supported by the organisers (gender-neutral English doesn't convey the fact that they were mostly women, thanks to them!). But in the end, there is a colour, a tone that emerges. In Maastricht, it was methodological diversity; in Thessaloniki, the return to values and social justice.

For this year, the Society wanted to focus on four transformations: evaluation systems, evaluators and evaluators, content — in and by evaluation, and in methodologies. These subjects were of course present in Copenhagen. They include in-depth discussions on causal mechanisms and impact assessments that are alternative or complementary to the counterfactual (including our own communication on the principles of the contribution analysis); or the always strong presence of questions relating to values and ethics. The different transformational approaches were also very present.

The central theme, however, was the awareness of the climate, social and societal emergency that we collectively face, and the complexity of the challenges that we have to deal with — something that the Director of the European Environment Agency, Hans Bruyninckx, reminded perfectly well in his presentation. By introducing the concept of “super-wicked” problems, he rejects once and for all the idea that there would be simple answers to the issues we face. Better: he invites evaluators, not so much to report on the past but to be actors in the current transitions (transition being a concept that is familiar to us in France, but not present so far in the evaluation debates).

However, this transition is not so easy. Evaluators may themselves be very conservative in their practices, or taken over by routines, or risk-adverse (as can be seen in these thousands of project evaluations using the DAC criteria, even if in private the same people who produced them would deem them inadequate). However, given the urgency, the complexity of the transitions to be undertaken, the evaluators together seemed to understand that they too had to change; that they are part of the systems that need to evolve. In his introduction, Tom Ling put it in select terms: “Let’s entertain the possibility that we might be wrong”. This could be translated more brutally: there is no more neutrality in this world. You're either part of the solution, or you're part of the problem”.

This change has been discussed at multiple levels. Let’s start perhaps with the most difficult and original in this conference, that is, changes that are intimate in nature. The world of evaluation, like others, has for many years indulged in talking about complexity and transformation — words that border on being buzzwords or fuzzwords and which would often reveal an abyssal void if one were to dig a little deeper. This year, the acronym VUCA (for Volatility, Uncertainty, Complexity and Ambiguity) was on all lips, to reflect the state of permanent crisis in which we live.

VUCA VUCA, a pill that's hard to swallow... unknown source

However, to be interested in the complex is to embark on a personal challenge. For example, learning to navigate uncertainty where evaluators like – through their monitoring systems, models and other methods – to create certainty; it means accepting being oneself part of the systems being studied, which calls for enlarging the scope of evaluations, but also abandoning false positions of independence or objectivity; questioning one's own lenses and how they affect what evaluators look at and discuss (and what they don't look at); and asking to what extent evaluation is a key ingredient of the current status quo and the difficulty of actually changing public policies.

A very good example of these developments in the intimate domain, the session of Barbara Schmidt-Abbey, Joan O'Donnell and Kirsten Bording Collins (who also won the prize for the largest number of w in a title): Being Evaluation Practitioners In A Volatile And Uncertain World: What Do We Do When We Do What We Do? They emphasise the need for reflexivity in and practice to evolve.

Schmidt Barbara invites us to think systematically AND systemically

Second level of change, those that fall within the relationships between evaluators and evaluators with their environment. During these three days, a large number of metaphors were used to characterise the roles they should be able to take: The evaluator as craft artisan, bricoleur, codesigner, partner, critical friend, knowledge broker, trustful companion... In any case, in a context of maximum uncertainty, the evaluator must first try to do the right thing — and doing the right thing can be a multitude of practices, far beyond the traditional roles of evaluation. It is how, argues Martin Reynolds, evaluation can become a real public service (would we say, in France, a common good? in any case we clearly agree).

The third level, which is essential, is the systems and initiatives in which evaluators are involved: how can we ensure that we are at the heart of the dynamics of change, whether institutional, or fueled by coalitions of organisations, communities or citizens? While the evaluation community complains a little too often that its work is not being used sufficiently, something new was going in in Copenhagen: how, rather, can we be where changes are happening now, to help them as we know how to do it? How can we identify and assist those who make changes happen — even when they do not ask for an evaluation? How to meet the needs that evaluators can actually meet and meet well, even if they are not qualified as evaluative needs?

All these changes require changes in knowledge (where are the concepts, theories and frameworks that will allow us to think about transitions?); competences (including the skills needed to stimulate dynamics or make oneself heard — mostly missing from today's initial and continuous training, which are more method-oriented); but also of stance: learning to look at the future rather than the past, for example (or in the words of Stefano D’Errico and his colleagues: “From what works? to what will work?”); or assuming one's values and a moral commitment to change; being ready to use one's expert intuition in support of the interventions carried out, and how to design and implement them (where our usual credibility is in the data we collect and analyse); or finally learning to convey, in depth, what we derive from evaluations and from our accumulated experience to those actors of change — who spoke about stories to be built?

Miaou Lessons learned: cutting kittens is wrong (Joan O'Donnell, Maynooth University)

Conclusion

With this 2022 session, the most optimistic will say that the European evaluation community has begun a “skeptical turn” about evaluation, as Peter Dahler-Larser and Estelle Raimondo have stated. It is (perhaps!) beginning to recognise that evaluation is not always a panacea, that evaluators do not always do the right thing, and that they must start their transition — be the agents of change they call for. It’s not easy... We think of the late Eleanor Chelimsky: it takes courage to be an evaluator...

We want to believe that this is a longlasting change. A good evidence of this would be to see, at the next conference, those who were missed the most, i.e. those actors of ecological, social or societal change, who have everything to gain from evaluation but remain strangely absent from these kinds of conferences. Let's do it!

Thomas Delahais

P.S.: EES was perhaps for the first time this year sensitive to its carbon footprint. It was possible to come by train (and we‘re not the only ones), but it wasn’t that easy! Why not next time choose a destination accessible to the greatest number and encourage to come by train, and leave the plane to those who cross the oceans?

Thanks to Sébastien Galéa, Benoît Simon and Quadrant Conseil colleagues for their comments and suggestions on the initial text; and thanks to May Pettigrew and Barbara Schmidt-Abbey for their suggestions on the English text!

changer Rare image of a change maker – an elected official, actually – lost among evaluators

EES 2022 — It’s not (only) you, it’s me (too)

dimanche 17 juillet 2022

Feedback on the European Evaluation Society Conference in Copenhagen

🚂 EES 2022 - C’est pas (que) toi, c’est moi (aussi)

mardi 12 juillet 2022

Retour sur la conférence de la société européenne d'évaluation à Copenhague

📚 Trois articles d'évaluation (nº7, 15 juin 2022)

mercredi 15 juin 2022

Dans ce numéro, cinéma à tous les étages : La Revanche des Sith, Octobre rouge et 120 battements par minute... ou presque

🎙 L'évaluation des contrats de ville

mercredi 01 juin 2022

Publication

📚 L'évaluation en contexte de développement

mardi 31 mai 2022

Publication

💡 Recherche et action publique : il faut être deux pour danser le tango

vendredi 25 mars 2022

Publication

📚 Trois articles d'évaluation (nº6, 15 mars 2022)

mardi 15 mars 2022

Dans ce numéro, la cartographie des controverses rencontre la science comportementale, et la recherche l'action publique.

💡 Qu'attendre de la recherche pour éclairer l'action publique

jeudi 20 janvier 2022

Publication

📚 Trois articles d'évaluation (nº5, 15 décembre 2021)

mercredi 15 décembre 2021

Numéro spécial Anthologie

Citations relatives à l'évaluation des politiques publiques

lundi 20 septembre 2021

Des citations inspirantes pour qui évalue.

📚 Trois articles d'évaluation (nº4, 15 septembre 2021)

mercredi 15 septembre 2021

Dans ce numéro, évaluation et bureaucratie, l'ultime combat, enquêter avec d'autres êtres, et oldiesbutgoodies, on sauve le monde avec Bob Stake !

💡 Glossaires de l'évaluation

dimanche 12 septembre 2021

Liste de glossaires

💡 7 bonnes raisons de prendre en compte l'égalité entre les femmes et les hommes dans ma pratique d'évaluation

vendredi 09 juillet 2021

Oldies but goodies (Karine Sage)

📚 Trois articles d'évaluation (nº3, 15 mai 2021)

samedi 15 mai 2021

Dans ce numéro, des échecs, des échecs, des échecs, l'évaluation pleinement décrite et pleinement jugée et la réception des politiques du handicap. Pas de oldiesbutgoodies, mais ça reviendra pour le numéro 4 !

💡 Les évaluations participatives, plus facile à dire qu'à faire

jeudi 29 avril 2021

Nouvel article publié (Thomas Delahais)

Quatre jeux de cartes d'évaluation francophones

mardi 27 avril 2021

À l'occasion de la sortie de Strateval, nous revenons sur 3 autres jeux de cartes autour de l'évaluation

💡 Qu'est-ce que l'analyse militante apporte à l'évaluation des politiques publiques ?

jeudi 08 avril 2021

Nouvel article publié (Marc Tevini)

📚 Trois articles d'évaluation (nº2, 15 février 2021)

lundi 15 février 2021

Dans ce numéro, évaluation féministe quésaco, apprentissage et redevabilité même combat ? et oldiesbutgoodies, des éléments pour une sociologie de l'évaluation... C'est le sommaire de ce numéro 2.

📽 L'évaluation des politiques sociales, de quoi parle-t-on ?

vendredi 15 janvier 2021

Introduction au séminaire de l'IRTS HDF du 26/01.

📽 L'analyse de contribution en pratique

mardi 15 décembre 2020

Introduction à la formation à l'analyse de contribution

📚 Trois articles d'évaluation (nº1, 15 novembre 2020)

dimanche 15 novembre 2020

Dans ce numéro, plongée en pleine guerre froide avec la Realpolitik de l'évaluation, des idées pour professionnaliser l'évaluation, et oldiesbutgoodies, de quoi se demander ce que les évaluateurs et les évaluatrices défendent dans leur métier... C'est le sommaire de ce numéro 1.

📽 L'évaluation participative, plus facile à dire qu'à faire

lundi 09 novembre 2020

Intervention de T Delahais au Congrès de la SEVAL organisé par le GREVAL à Fribourg, le 4 septembre 2020.

💡 En situation anormale, l'évaluation doit devenir la norme

jeudi 29 octobre 2020

Contribution de T Delahais et M Tevini en réponse à l'appel de la SFE, "Ce que la crise sanitaire nous apprend sur l'utilité et les pratiques d'évaluation".

💡 Comment évaluer la transition

vendredi 23 octobre 2020

Nouvel article publié (Thomas Delahais, Karine Sage, Vincent Honoré)

💡 Quels mécanismes à l'œuvre dans les innovations ?

mardi 06 octobre 2020

Nouvel article publié (Agathe Devaux-Spatarakis)

📚 Trois articles d'évaluation (v0, 30 juillet 2020)

jeudi 30 juillet 2020

À l'honneur pour cette édition, la sagesse pratique des évaluateurs et des évaluatrices, soit « la capacité de faire les bons choix, au bon moment, pour les bonnes raisons » ; ce que les mots et concepts de l'évaluation perdent et gagnent à leur traduction d'une langue à l'autre et oldies but goodies, une piqûre de rappel quant à la vocation démocratique de l'évaluation en France... C'est le sommaire de ce numéro 0.