lundi 03 juillet 2023
The challenges of external validity, towards an interdisciplinary discussion?
Version en français ici
At the beginning of June, almost half of Quadrant Counseil’s team was in Copenhagen for this latest version of the European Evaluation Conference: three days, more than 200 sessions and more than 600 participants from Europe and elsewhere.
The previous face-to-face edition dates from before COVID and the war in Europe, it was in Thessaloniki in 2018. This edition had marked a return to issues of values and social justice, after the methodological frenzy of the years 2000 and 2010. Thanks to the online edition held in September 2021, we've avoided, at least in part, the profusion of opportunistic communications on evaluation in times of COVID assessment which has plagued many conferences. COVID was still there, but we were a little bit more in the world of tomorrow.
The title of this edition was Evaluation at a watershed, a phrase used in English to convey the idea of being “at a turning point”. Perhaps the image of the waterfall would better reflect the urgency of the situation, and the need to paddle against the current.
Beware, it’s Denmark after all, the water is fresh.
Two observations first:
It is always difficult to say whether an international conference has focused on a particular topic, despite the themes supported by the organisers (gender-neutral English doesn't convey the fact that they were mostly women, thanks to them!). But in the end, there is a colour, a tone that emerges. In Maastricht, it was methodological diversity; in Thessaloniki, the return to values and social justice.
For this year, the Society wanted to focus on four transformations: evaluation systems, evaluators and evaluators, content — in and by evaluation, and in methodologies. These subjects were of course present in Copenhagen. They include in-depth discussions on causal mechanisms and impact assessments that are alternative or complementary to the counterfactual (including our own communication on the principles of the contribution analysis); or the always strong presence of questions relating to values and ethics. The different transformational approaches were also very present.
The central theme, however, was the awareness of the climate, social and societal emergency that we collectively face, and the complexity of the challenges that we have to deal with — something that the Director of the European Environment Agency, Hans Bruyninckx, reminded perfectly well in his presentation. By introducing the concept of “super-wicked” problems, he rejects once and for all the idea that there would be simple answers to the issues we face. Better: he invites evaluators, not so much to report on the past but to be actors in the current transitions (transition being a concept that is familiar to us in France, but not present so far in the evaluation debates).
However, this transition is not so easy. Evaluators may themselves be very conservative in their practices, or taken over by routines, or risk-adverse (as can be seen in these thousands of project evaluations using the DAC criteria, even if in private the same people who produced them would deem them inadequate). However, given the urgency, the complexity of the transitions to be undertaken, the evaluators together seemed to understand that they too had to change; that they are part of the systems that need to evolve. In his introduction, Tom Ling put it in select terms: “Let’s entertain the possibility that we might be wrong”. This could be translated more brutally: there is no more neutrality in this world. You're either part of the solution, or you're part of the problem”.
This change has been discussed at multiple levels. Let’s start perhaps with the most difficult and original in this conference, that is, changes that are intimate in nature. The world of evaluation, like others, has for many years indulged in talking about complexity and transformation — words that border on being buzzwords or fuzzwords and which would often reveal an abyssal void if one were to dig a little deeper. This year, the acronym VUCA (for Volatility, Uncertainty, Complexity and Ambiguity) was on all lips, to reflect the state of permanent crisis in which we live.
VUCA, a pill that's hard to swallow... unknown source
However, to be interested in the complex is to embark on a personal challenge. For example, learning to navigate uncertainty where evaluators like – through their monitoring systems, models and other methods – to create certainty; it means accepting being oneself part of the systems being studied, which calls for enlarging the scope of evaluations, but also abandoning false positions of independence or objectivity; questioning one's own lenses and how they affect what evaluators look at and discuss (and what they don't look at); and asking to what extent evaluation is a key ingredient of the current status quo and the difficulty of actually changing public policies.
A very good example of these developments in the intimate domain, the session of Barbara Schmidt-Abbey, Joan O'Donnell and Kirsten Bording Collins (who also won the prize for the largest number of w in a title): Being Evaluation Practitioners In A Volatile And Uncertain World: What Do We Do When We Do What We Do? They emphasise the need for reflexivity in and practice to evolve.
Barbara invites us to think systematically AND systemically
Second level of change, those that fall within the relationships between evaluators and evaluators with their environment. During these three days, a large number of metaphors were used to characterise the roles they should be able to take: The evaluator as craft artisan, bricoleur, codesigner, partner, critical friend, knowledge broker, trustful companion... In any case, in a context of maximum uncertainty, the evaluator must first try to do the right thing — and doing the right thing can be a multitude of practices, far beyond the traditional roles of evaluation. It is how, argues Martin Reynolds, evaluation can become a real public service (would we say, in France, a common good? in any case we clearly agree).
The third level, which is essential, is the systems and initiatives in which evaluators are involved: how can we ensure that we are at the heart of the dynamics of change, whether institutional, or fueled by coalitions of organisations, communities or citizens? While the evaluation community complains a little too often that its work is not being used sufficiently, something new was going in in Copenhagen: how, rather, can we be where changes are happening now, to help them as we know how to do it? How can we identify and assist those who make changes happen — even when they do not ask for an evaluation? How to meet the needs that evaluators can actually meet and meet well, even if they are not qualified as evaluative needs?
All these changes require changes in knowledge (where are the concepts, theories and frameworks that will allow us to think about transitions?); competences (including the skills needed to stimulate dynamics or make oneself heard — mostly missing from today's initial and continuous training, which are more method-oriented); but also of stance: learning to look at the future rather than the past, for example (or in the words of Stefano D’Errico and his colleagues: “From what works? to what will work?”); or assuming one's values and a moral commitment to change; being ready to use one's expert intuition in support of the interventions carried out, and how to design and implement them (where our usual credibility is in the data we collect and analyse); or finally learning to convey, in depth, what we derive from evaluations and from our accumulated experience to those actors of change — who spoke about stories to be built?
Lessons learned: cutting kittens is wrong (Joan O'Donnell, Maynooth University)
With this 2022 session, the most optimistic will say that the European evaluation community has begun a “skeptical turn” about evaluation, as Peter Dahler-Larser and Estelle Raimondo have stated. It is (perhaps!) beginning to recognise that evaluation is not always a panacea, that evaluators do not always do the right thing, and that they must start their transition — be the agents of change they call for. It’s not easy... We think of the late Eleanor Chelimsky: it takes courage to be an evaluator...
We want to believe that this is a longlasting change. A good evidence of this would be to see, at the next conference, those who were missed the most, i.e. those actors of ecological, social or societal change, who have everything to gain from evaluation but remain strangely absent from these kinds of conferences. Let's do it!
P.S.: EES was perhaps for the first time this year sensitive to its carbon footprint. It was possible to come by train (and we‘re not the only ones), but it wasn’t that easy! Why not next time choose a destination accessible to the greatest number and encourage to come by train, and leave the plane to those who cross the oceans?
Thanks to Sébastien Galéa, Benoît Simon and Quadrant Conseil colleagues for their comments and suggestions on the initial text; and thanks to May Pettigrew and Barbara Schmidt-Abbey for their suggestions on the English text!
Rare image of a change maker – an elected official, actually – lost among evaluators
lundi 03 juillet 2023
The challenges of external validity, towards an interdisciplinary discussion?
mardi 27 juin 2023
Les défis de la validité externe, un sujet d'échange interdisciplinaire ?
vendredi 16 juin 2023
Le name and shame est-il une politique publique efficace?
jeudi 02 mars 2023
Du outcome harvesting dans une évaluation de projet
jeudi 19 janvier 2023
Quelles distinctions entre l'évaluation et les pratiques voisines ?
mardi 01 novembre 2022
Quelle démarche pour une cartographie des usages de l'évaluation d'impact ?
samedi 15 octobre 2022
Un numéro très très lutte – pour l'équité raciale, entre objectivistes et subjectivistes, et oldies but goodies pour faire entendre l'évaluation dans un contexte politisé.
mardi 04 octobre 2022
dimanche 17 juillet 2022
Feedback on the European Evaluation Society Conference in Copenhagen
mardi 12 juillet 2022
Retour sur la conférence de la société européenne d'évaluation à Copenhague
mercredi 15 juin 2022
Dans ce numéro, cinéma à tous les étages : La Revanche des Sith, Octobre rouge et 120 battements par minute... ou presque
mercredi 01 juin 2022
mardi 31 mai 2022
lundi 02 mai 2022
vendredi 25 mars 2022
mardi 15 mars 2022
Dans ce numéro, la cartographie des controverses rencontre la science comportementale, et la recherche l'action publique.
jeudi 20 janvier 2022
mercredi 15 décembre 2021
Numéro spécial Anthologie
lundi 20 septembre 2021
Des citations inspirantes pour qui évalue.
mercredi 15 septembre 2021
Dans ce numéro, évaluation et bureaucratie, l'ultime combat, enquêter avec d'autres êtres, et oldiesbutgoodies, on sauve le monde avec Bob Stake !
dimanche 12 septembre 2021
Liste de glossaires
vendredi 09 juillet 2021
Oldies but goodies (Karine Sage)
samedi 15 mai 2021
Dans ce numéro, des échecs, des échecs, des échecs, l'évaluation pleinement décrite et pleinement jugée et la réception des politiques du handicap. Pas de oldiesbutgoodies, mais ça reviendra pour le numéro 4 !
jeudi 29 avril 2021
Nouvel article publié (Thomas Delahais)
mardi 27 avril 2021
À l'occasion de la sortie de Strateval, nous revenons sur 3 autres jeux de cartes autour de l'évaluation
jeudi 08 avril 2021
Nouvel article publié (Marc Tevini)
lundi 15 février 2021
Dans ce numéro, évaluation féministe quésaco, apprentissage et redevabilité même combat ? et oldiesbutgoodies, des éléments pour une sociologie de l'évaluation... C'est le sommaire de ce numéro 2.
vendredi 15 janvier 2021
Introduction au séminaire de l'IRTS HDF du 26/01.
mardi 15 décembre 2020
Introduction à la formation à l'analyse de contribution
dimanche 15 novembre 2020
Dans ce numéro, plongée en pleine guerre froide avec la Realpolitik de l'évaluation, des idées pour professionnaliser l'évaluation, et oldiesbutgoodies, de quoi se demander ce que les évaluateurs et les évaluatrices défendent dans leur métier... C'est le sommaire de ce numéro 1.
lundi 09 novembre 2020
Intervention de T Delahais au Congrès de la SEVAL organisé par le GREVAL à Fribourg, le 4 septembre 2020.
jeudi 29 octobre 2020
Contribution de T Delahais et M Tevini en réponse à l'appel de la SFE, "Ce que la crise sanitaire nous apprend sur l'utilité et les pratiques d'évaluation".
vendredi 23 octobre 2020
Nouvel article publié (Thomas Delahais, Karine Sage, Vincent Honoré)
mardi 06 octobre 2020
Nouvel article publié (Agathe Devaux-Spatarakis)
jeudi 30 juillet 2020
À l'honneur pour cette édition, la sagesse pratique des évaluateurs et des évaluatrices, soit « la capacité de faire les bons choix, au bon moment, pour les bonnes raisons » ; ce que les mots et concepts de l'évaluation perdent et gagnent à leur traduction d'une langue à l'autre et oldies but goodies, une piqûre de rappel quant à la vocation démocratique de l'évaluation en France... C'est le sommaire de ce numéro 0.