Mervyn King

How did I like this book? Well, I'm not certain.

HA HA HA. It was OK, but it sure seemed to be saying the same thing over and over again for 433 pages and a lengthy appendix: Most situations in life involve a radical uncertainty, rendering them unsuitable for statistical modeling or probabilistic forecasting.

Of all the chapters basically saying the same thing from different perspectives, my favorite was "Evolution & Decision-Making," which leads with a quote attributed to SF author Bruce Sterling: "Computation is not thinking... You are much more like hour house cat than you are ever going to be like Siri."

To quote further the beginning of the chapter, "Behavioural economics has identiifed a raft of ways in which humans depart from axiomatic rationality. These behaviours are described as 'biases,' signs of human failure... It is as though God had given us two legs so that we could run or walk, but made on leg shorter than the other so that we could not run or walk very well... We are not defective versions of computers trained to optimise in small-world problems, but human beings with individual and collective intelligence evolved over millennia."

That's basically what it's about, although this is the only chapter with an evolutionary bent; it's all about how 'real-world' problems are nothing like 'small-world' problems that researchers come up with in the lab.

So take heart! You aren't a broken machine. You're an exceedingly smart house cat!

Also memorable, in the chapter "The Use & Misuse of Models," was a brief historical overview of the collapse of the cod industry in Newfoundland. "The [Dominion] Fisheries Office developed complex models on which its recommendations [for total allowable catch] were based. But cod stocks continued to decline. For the year 1992, the total allowable catch was set at 145,000 tonnes. That proved to be the last year of commercial cod fishing on the Grand Banks." The authors do not lay responsibility for this solely with the modelers, of course, but maintain that their 'evidence' ended up justifying the policy of "greedy fishermen and mendacious politicians" rather than actually protecting fish stocks. This example of mismanagement by model struck me enough to read up on the subject in Wikipedia, where the sad story can be read in more detail: "Over 35,000 fishermen and plant workers from over 400 coastal communities became unemployed... Newfoundland has since experienced a dramatic environmental, industrial, economic, and social restructuring, including considerable outward migration..."

Something of a detour from the main idea of the book; but one example of how the illustrations and anecdotes chosen by the authors are very powerful and well conveyed. Sorry to detour on the detour, but listen to a "No More Fish, No Fishermen," a song on this topic I heard long ago on public radio and never forgot:
https://www.youtube.com/watch?v=uPw74oTuliM… (altro)

[a:John Kay|10137|John Kay|https://s.gr-assets.com/assets/nophoto/user/u_50x66-632230dc9882b4352d753eedf9396530.png] and [a:Mervyn King|20813783|Mervyn King|https://s.gr-assets.com/assets/nophoto/user/u_50x66-632230dc9882b4352d753eedf9396530.png] have provided a fascinating tour of how probabilistic reasoning has been used in decision-making. They offer insights across history and disciplines that cast light on the many abuses of probabilistic reasoning. However, ultimately you have to think that the main abuse that spurs the authors is the financial models that failed spectacularly in the Global Financial Crisis. This is a work that should be read by every economist, statistician, financial analyst, policy adviser and management consultant.

Much as I found the endless sting of accounts fascinating, it left a question. If assumptions about future probability is so suspect, can you use models at all? The authors touch on this in Chapter 20, when they list four lessons for the use of models in business and government (lesson one is "keep it simple stupid", just not in that language). This seems to be insufficient.

Financial markets, insurance, cost-benefit analysis and risk management all involve making assumptions about the future. If the assumption is not in the form of a probability or a distribution of probabilities, then it will be a point forecast (which is not necessarily better). The physicist Neils Bohr observed that "Prediction is very difficult, especially if it's about the future". As a public servant (for a decade in the Australian Department of Finance) and as a management accountant, much of my time was spent assessing investment proposals - and the art was to find the assumptions. Judgement can then be applied to see whether the assumptions are credible. The challenge provided models is that the assumptions may not be immediately obvious. I would add to the author's list:

5. the assumptions in a model should be explicitly identified, and it should be possible to change the assumptions to see alternative outcomes.

6. Understand how the assumptions were arrived at - was it on historical data or someone's professional judgement? How could it be made more robust?

7. Have the model reviewed independently to see if indeed all the assumptions are clear and plausible.

The authors have found fault with WebTAG - the United Kingdom's transport analysis guidance - for the many assumptions that are built into the costing models. But the authors have missed the point of these assumptions. They have not been provided because they are an accurate forecast of the future, but because the Department of Transport will be a need to choose between many proposals and that is easier if (as far as practicable) they are using the same assumptions.

And the book ends by quoting the Prussian general and military theorist, Carl von Clausewitz, about the inherent uncertainty of war. For me, this is particularly unsatisfactory. My current contract is advising the Australian Department of Defence. Around the world, defence departments spend billions of dollars on addressing future conflict which is inherently uncertain. But as taxpayers we want to ensure that waste is minimised. Just including that it is complex does not help.… (altro)

This was probably the most thought-provoking book I've read this year.

When I say that, I don't mean it was the kind of book that made me reconsider everything I believed, though it did some of that. Rather, nearly every page had me pausing to consider and mentally debate the book's arguments, which were always interesting even when I ultimately came down against the authors. (Of course, I have a larger-than-normal interest in epistemology and uncertainty, so your mileage may vary...)

The main point of the authors if to respond to an intellectual movement that claims to be foregrounding uncertainty in their analysis. This movement, inspired by Bayesian statistics, tries to get away from narrative-driven beliefs by instead quantifying the likelihood of those beliefs, and constantly updating those likelihoods in response to new information. The authors argue this movement is misguided, and actually does the opposite of what its backers claim it does — by quantifying risk, they say, people are applying false precision to things that are actually radically uncertain — impossible to quantify.

"Reasonable uncertainty is uncertainty which can be removed by looking something up (I am uncertain which city is the capital of Pennsylvania) or which can be represented by a known probability distribution of outcomes (the spin of a roulette wheel). With radical uncertainty, however, there is no similar means of resolving the uncertainty — we simply do not know."

Put another way, it is the difference between "risk" and "uncertainty" where "risk" means "unknowns which could be describe with probabilities" and "uncertainty" which can't. Today, the authors argue, we tend to treat uncertain things as if they are actually risks that can be precisely quantified. They give the example of national security advisers meeting with President Barack Obama in 2011, giving their assessments of whether Osama bin Laden was actually in a compound in Abbottabad, Pakistan — one advisor said there was a 95 percent chance bin Laden was there, while another said 80 percent and another 40 percent. Obviously these percentages are completely different from, say, the 50 percent chance that a fair coin will come up heads.

But this example also brings up one of the problems with the book: it's rather over-focused on issues inside the field of economics (and adjacent areas), and the author's arguments against various forms of probabilistic reasoning run into more issues when they move past critiquing over-quantified economic models and move to day-to-day decision-making.

To return to the prior example, in a very literal sense, the statement that there was a 95 percent chance Bin Laden was in Abbottabad is meaningless. Either he was there or he wasn't; it wasn't like you could raid the compound 20 times and expect to find Bin Laden 19 times. But this wasn't a case where the only options for belief were "he's there," "he's not there," and "we don't know." One can believe it is "more likely than not" that something is true, that evidence suggests something but doesn't prove it. Saying "95 percent" may not have any solid statistical basis, but isn't it a perfectly fine synonym for "almost certain"? To be sure, we need to make sure not to take that 95 percent estimate too seriously, as a real, empirical probability. But at a certain point, applied to real life and not economic models, this argument becomes a straw man.

Another favorite straw man argument the authors use is to mock the idea that actual people making real decisions have a "Bayesian dial floating over their heads" — a reference to the model of Bayesian statistics, which starts with a "prior probability" that something is true and then updates that probability based on evidence. Real decisions, they say, are based on narratives, not statistical models. Again, this is an argument that is obviously true in a very narrow sense — as they demonstrate, even professional economists and statisticians usually don't use their probabilistic methods for making life decisions — but falls down a bit when taken a little more loosely. It's perfectly possible to approach life in a pseudo-Bayesian sense, starting with your belief about what is the case, and updating it as you learn more, even if you're not actually constantly performing Bayesian math in your head like an imaginary person in an economic model.

But even if many of their arguments fall apart a bit when applied to real life and not to economics, this is still a helpful book for lay readers. Their emphasis on knowing when to say "I do not know" and the value of asking "What is going on here?" are well-taken. And their targets aren't just straw men — over-quantified economic models are real, and used as the basis for all sorts of hugely consequential decisions. (Among other things, they cite the bank models before the housing bubble burst in 2007-8, for which the collapse of the housing market allegedly involved "25-standard deviation moves several days in a row." As they note, "our universe has not existed long enough for there to have been days on which 25 standard deviation events could occur"; the problem was the models' assumptions, inputs and algorithms were wrong, and considered an event that actually did happen as basically impossible.) I think their points are made too strongly for laypeople's purposes, and are too focused on economics rather than daily life, but they're still well-taken.… (altro)