Evaluation for transitions

How transitions change evaluations, and how evaluations can help transitions

T. Delahais - 15.01.2025

This is the transcript of an intervention by Thomas Delahais at the Rencontres internationales francophones en évaluation environnementale on January, 15th 2025. The French version is available here.

The first time I was confronted with a transition initiative was a dozen years ago, and it was the 'Transformation sociale et écologique régionale en Nord-Pas-de-Calais'. The idea was to experiment with a new approach to public policy, based on a dozen or so key issues such as food systems, energy-efficient home renovation and certain local ecosystems. The aim was to bring together all the actors concerned, start from the multiple problems that policies were trying to solve, sketch out a desirable, viable future, and propose new solutions to get there. And it was so different from what was usually done on these subjects, where European or national policies were rolled out, or where people did what they always did, or what their neighbours did, that it was very suspicious. In fact, in the first inception interview I conducted, I was told “we hope this evaluation will show that the whole thing is a scam”!

On the contrary, we showed that the processes involved were enormously valuable; but that they were also complex, fragile, uncertain, involved little-known mechanisms and so on.

Conventional evaluation tools had trouble accounting for the effects of a transition initiative of this type. If we started from the objectives to account for the effects, it was a failure, but if we started from reality, from what was actually happening, we could say that there were failures, which is normal for a public innovation, but also enormous successes. Over the long term, above all, thinking in terms of transition has led us to review our evaluation practices, and it's on this subject that I'd like to speak today.

Transitions :

uncertain
total
multi-level
innovative
contested

The word 'transition' is used by many actors, to talk about many different things. Here I'd like to define it as follows:

• It's a term that describes the transition from today's carbon-intensive, environmentally destructive society to a sustainable one.
• It refers both to the process of change, and to the policies designed to implement and manage this process.

Transitions are uncertain. We don't know what a sustainable society looks like, or how to get there. The consequences of today's actions are difficult to predict.

They affect all aspects of society at the same time. In fact, there are several transitions. They cannot include only the environment, or only the climate, or only buildings, etc.

They are multi-actor, multi-level, multi-stage. Their success depends on the congruence of change at all these levels.

They require technical, social, organizational and human innovations, whose implementation and effects are themselves uncertain. There is competition between these innovations to move towards a sustainable society.

Finally, they are fundamentally contested, not only because of the widespread uncertainty about what needs to be done and how, but also because they fundamentally affect the values and beliefs of individuals and societies.

Not all transition policies take the same approach. Some have a fundamentally technical orientation and seek solutions to the problems posed by pollution, climate change, etc.; others have a fundamentally political orientation and seek first to invent a sustainable world before finding the paths to get there. Many integrate both orientations, but they cannot be evaluated in the same way.

What about evaluation?

I'm talking here about a broad family of practices that are concerned with the consequences that interventions have had or could have. The diagram above from the Griffith Centre for System Innovation proposes a representation of this extended family.

Well, evaluation is strongly affected by these characteristics of transitions; but it also has many resources and potential to contribute to transitions.

Evaluation affected by transitions :

scope
uncertainty about the way to proceed
conflicting values
cultures and practices

How is evaluation affected? Is it sometimes unsuited to these transition initiatives?

The first concern is the boundaries that are set for us. Evaluation is often a marketplace activity, in which we respond to terms of reference to provide an evaluation service focused on a given project, program or territory, over a given period of time. Yet transitions often require the integration of several dimensions, different types of actors, points of view and so on. The problem is that we often don't know, at the outset, what the right scope of evaluation should be. How can we make this scope evolve along the way?

A second issue is that, with Transitions, we no longer know 'what works' and what is the right thing to do. An infrastructure may follow the best rules, but turn out to be unsuitable a few years later; because, on the one hand, the climate and the living world are evolving at a frantic pace; because transitions affect every aspect of our lives, with largely unexpected consequences; and because we're fumbling in the dark to adapt. In our evaluations of European programs, we're starting to ask ourselves: 'Is this infrastructure future-proof?' rather than 'is it relevant to yesterday's challenges?' This leads to a lot of uncertainty, because we're no longer judging them on known criteria, but on tentative criteria that are more or less plausible. So how can we make judgements and propose improvements if we don't know what needs to be done?

A third concern is the level of contestation and conflict associated with transitions. Because transitions affect all subjects, they are judged according to very different sets of values and beliefs. Evaluation is inclined to seek a “scientific truth”, but the criteria and indicators used are also imbued with values. We sometimes arrive at a judgement of either success or failure for the same intervention, depending on the criteria used - all of which are valid. What is a “good” transition?

The final issue I'd like to raise here: The tendency to focus on public decisions, whereas transitions involve all public and private stakeholders, both in formal decisions, but also in implementation practices, administrative and technical acts, habits... which fly under the radar even though they are just as important as the decision. Do we want to support decisions when we know that they will be undermined or not applied in practice? How can we contribute to a change in culture and practice, and not just to one-off decisions?

Evaluation in support of transitions :

critical friend
pluralist and participatory
focused on the culture of evaluation

The problems I'm talking about are complex, and we don't have a silver bullet to fix them. But I would like to highlight some of the things that evaluation can do here.

I won't go into the methodological dimensions. What evaluators know well is that “being proved right” is certainly not a good enough reason to prevail in a decision. In fact, taking care of the decision-making process is just as important as producing knowledge.

First of all, we must recognize that evaluation is a political activity, i.e. it is not external to the transitions, it is not a neutral tool that would provide an external truth to which to refer.

After our evaluation of TESR, which I mentioned earlier, we had to acknowledge that we were in favour of transitions, i.e. that we want them to happen, and that they should be an opportunity to improve the world. And so, in a way, this has led us to position ourselves as “critical friends” of transition initiatives, and more generally of those who are ready to get involved in transitions. I believe this is fundamental, and the touchstone for everything else. It goes without saying that friendship doesn't prevent us from being demanding, empirically rigorous and courageous enough to get difficult messages across.

This posture of critical friend leads us to rethink quite fundamentally the skills needed to be a good evaluator. It's a question of being benevolent to transition makers, but also of knowing how to communicate and collaborate, and of having the courage to get difficult messages across. It also changes the responsibility that evaluators have - not to carry the transition on their shoulders, but to play their part in it.

It also allows us to work with people and stakeholders for whom evaluation is part of the problem, rather than the solution. We have to contend with a great deal of mistrust on the part of many actors, who nonetheless have important values and perspectives to bring to bear on evaluation.

We can start from the premise that, in the case of transitions, there are always several criteria for judging a project or intervention, and the achievement of objectives - often poorly formulated and uncertain - or compliance with legal rules are not always the most relevant. In fact, defining criteria and agreeing on a “good transition” are often part of political work. Sometimes - and often - the criteria that emerge are not consistent with each other. This is difficult for us evaluators, who like consistency. But transitions often mean making the least bad choice rather than the best, managing paradoxical demands between different planetary limits to be preserved, between defending the environment and protecting populations, managing technical and political issues, and so on. So we need nuanced judgements, based on different sources and different types of knowledge. But we also need to vary our perspectives, to move away from a confrontation between the environment and the economy, and think in terms of sustainable territorial well-being, for example.

The evaluation process is an important opportunity for these values, perspectives and knowledge to emerge. It is also an opportunity to involve the various stakeholders in an in-depth discussion that goes beyond consultation or information. An essential challenge, for example, is to get all the stakeholders to ask evaluation questions that really reflect what matters to them. This requires long-term work, and the role of evaluation is to ensure that the points of view of the various stakeholders are considered, translated and that there is negotiation about what matters. Isn't a good evaluation first and foremost one that has changed the way stakeholders look at the important things, the problems to be solved, that has enabled them to avoid off-the-shelf solutions and to propose acceptable compromises?

But if we want to go beyond specific projects or decisions to support transitions, which are necessarily ongoing, we also need to move beyond this one-off evaluation process, which is very limiting. As we have said, transitions progress by trial and error, by combinations of interventions, by seizing opportunities... and all this does not always correspond to a well-defined project! This is what also leads us, as evaluators, to reconsider our role in greater depth: to become long-term companions on the road to a sustainable future; to be present throughout the implementation process to anticipate difficulties, to help make the interventions carried out more favourable to transitions; and to give priority to building a culture of evaluation in transitions.

This is also important because ambitious transition projects are often subject to crossfire. They are always necessarily unrealistic, but also insufficient; there are always solutions that seem easier and better, and so on. But a report, even with an irreproachable methodology, will not save a transition initiative. In reality, evaluators have an essential role to play in nurturing transition tives that are desirable, but also supported by robust empirical evidence: and this is done largely outside the framework of delivering a contract, over the long term. And evaluations are called on to transfer and disseminate lessons between different stakeholders, different levels of government and different sectors.

All this once again raises questions about evaluation as a service, and in particular the one-off dimension of public commissioning. Doesn't the future of evaluation in transitions lie first and foremost in a better dissemination of an evaluative posture, shared by all the stakeholders, that enables us to reflect on the consequences of the actions taken whenever necessary; to be attentive to values and different perspectives; or to favour empirical verification over prejudices about ‘what works’ or not? Thank you very much.