Simply put: Hausfather et al. (2019)

This post is based on the paper "Evaluating the Performance of Past Climate Model Projections" by Hausfather et al. (2019) published in Geophysical Research Letters. It explores an important question: How well did early climate models capture climate change?

 

Image source 

Note that this study focused on a previous generation of climate models. Today, these models are operationally obsolete, i.e. they are no longer used for climate projections as they have been superseded by improved and more advanced models. But results from these modelling studies provide outputs for long timeframes. For examples, the oldest studies considered (Manabe, 1970; Mitchell, 1970 and Benson, 1970) provide data for 30 years from 1970 to 2000. In contrast, at the time this paper was being written, the newest study included (from the IPCC Assessment Report 4) provided only 10 years of data from 2007 to 2017. So even if this study does not tell us, How well do current climate models capture climate change, in a way it provides us a minimum estimate for climate model ability.

 

Comparison of temperature vs time

This analysis includes results from 17 models. Hausfather et al. compared these 17 model projections i.e. what the climate models said would happen, with observations i.e. what actually happened. 10 out of 17 model projections were consistent with the observations. What of the remaining 7? 4 showed more warming than observed, 3 showed less. Thus, there was no systematic bias.

But there is an important catch here, discussed well in the paper too. Climate model projections do not depend solely on climate physics. Since anthropogenic carbon emissions are the main cause of climate change, it is necessary to first project future emissions to be able to project future climate. And this solidly falls outside the expertise of climate scientists. Climate scientists take care of this anthropogenic uncertainty by including several future emissions scenarios in their studies. However, to reiterate, the development of these future scenarios is outside the purview of climate science.

Regarding this, Hausfather et al. write:

"Model projections rely on two things to accurately match observations:

1. accurate modeling of climate physics, and
2. accurate assumptions around future emissions of CO2 and other factors affecting the climate.

...

While climate models should be evaluated based on the accuracy of model physics formulations, climate modelers cannot be expected to accurately project future emissions and associated changes in external forcings, which depend on human behavior, technological change, and economic and population growth."

So we come to an important conclusion: that climate models should not be evaluated based simply on comparisons of climate parameters (such as temperature), since part of their shortcomings may actually lie in inaccurate assumptions regarding emissions.

To overcome this difficulty, Hausfather et al. compare another metric between model projections and observations. They call this the implied TCR and it captures the change in temperature versus change in radiative forcing. In simple terms, radiative forcing may be termed as the "cause" behind climate change. And thus, this metric captures the models' response to the cause, reflecting model ability while bypassing shortcomings in the projections regarding the cause in the first place.

 

Comparison of temperature vs implied TCR

The results are more encouraging than before. Using this metric, 14 out of 17 projections were consistent with observations. What of the other 3? Two showed higher implied TCR than observations, one showed lower. Once again, no systematic bias showed up. In the words of the authors, "We find no evidence that the climate models evaluated in this paper have systematically overestimated or underestimated warming over their projection period."

 

Projections vs predictions

One of my pet peeves of late has been the use of the phrase "climate predictions". To me, it puts climate science in the same ranks as many sources of dubious and unsubstantiated warnings about the future; a grave injustice to decades of work of developing rigorous models which are based on physical principles that work universally (and not just in climate science). To this end, a line in this paper caught my attention. "Climate model projections are usefully thought about as predictions conditional upon a specific forcing scenario." To me, that captures the distinction between a "projection" and a "prediction". A prediction says, "This is what will happen." A projection says, "This is what will happen if ... ", an attitude which is much more forthright, transparent and accepting of the underlying assumptions which must be imported from elsewhere. Climate scientists can not develop socio-economic projections or forecast how much CO2 will be released by humankind, but only attempt to forecast the climate's response to these emissions. Along the same lines, climate scientists can also not forecast what the effects of the climate's response would be on humankind. (If we do discuss all this, it must be based on what we learn from other authoritative sources.)

An example: Hansen et al. (1988)

One of the 17 studies considered in this paper is the study by Hansen et al. (1988) which used 3 future scenarios to project the future. Hausfather et al. compared these 3 projections with observations in Fig. 3 in their paper.

Source: Hausfather et al. (2019) [Geophysical Research Letters]

As you may expect looking at the projected temperatures, the 3 scenarios reflect high (A), medium (B) and low (C) emissions scenarios. The grey dashed vertical line marks 1987, the start of the projection period. Everything before that is a hindcast, everything after, a forecast. As Hausfather et al. note, "While models are commonly evaluated by comparing "hindcasts" of prior climate variables to historical observations, the development of hindcast simulations is not always independent from the tuning of parameters that govern  unresolved physics." Thus, a truer test of model ability lies in evaluating forecasts. From the figure above we can see that scenario C seemed to be closest to the observations till about 2015. After that, it seems that scenario B could be closer to reality.

But why bother with an evaluation of obsolete models?

As I write earlier, this evaluation does not tell us about currently operational models. But since those are more advanced that the ones studied here, this study may be seen as providing a lower bound for climate ability. How have climate models advanced? Two aspects can be easily visualized. Firstly, climate models now include more and more components of the Earth's climate system.

 

Image Source

 

Secondly,  they operate at greater spatial resolution.

Image Source: IPCC AR4

This should give us an idea that current climate models also perform satisfactorily. (There are independent assessments focusing on that, but this blog post has already got out of hand.) Hopefully, this would encourage more readers to "believe" in climate models, and more importantly, in the dire crisis that they project for our future.

---

To subscribe to such posts, go here.

---

Related reading:

1. Analysis: How well have climate models projected global warming? - Carbon Brief
2. Study confirms climate models are getting future warming projections right - NASA JPL