1. A couple of years ago, during my postgraduate studies, I was doing some research into the theoretical properties of public housing allocation systems in the UK. British councils tend to use a type of programme known as a ‘choice based letting’ system, and I was interested in how close these would be to the algorithms we used in matching theory problem sets.

The answer turned out to be ‘not at all’, and while there were probably a number of reasons for that one of the starkest was the difference between the theoretical agents in an economic model — who knows what they want, and will choose their ‘strategy’ accordingly — and the people who actually went out to find public housing to live in.

One of the most interesting discoveries from that brief sojourn out of the ivory tower was that people often held totally erroneous beliefs about the way the matching process worked. These beliefs were not confined to more general myths (‘they all go to refugees’), or a reasonable lack of understanding of how an algorithm might work, but took in some truly bizarre misunderstandings of the relatively simple part the user played in operating the system.

While the allocation algorithm is simple — applicants nominate their most preferred houses in order, then the applicant highest on the list picks a house, and so on down — the interactions between bidders can make the end result hard to predict, and people often fail to follow instructions. In turn, this leads to a sort of mythology about the best way to get a council house. People would stay up to bid in the middle of the night, convinced that being ‘first in line’ would finally get them into a permanent house, or would bid on houses they didn’t want out of concern that failing to use all their bids would render them ineligible. Neither of these things were true.

2 Fast-forward to May 30th, 2020. Deputy Chief Medical Officer Jonathan Van Tam is being asked whether the UK’s new Coronavirus contract tracing system could lead to fraudsters calling people up to get access to their personal data. Apparently not: “I don’t think people are going to fall for this”.

Well, maybe we’ll be lucky. But anyone with a landline knows that the moment your number gets out there, you will be inundated with calls from ‘Microsoft technical support’, ‘HMRC’, ‘Apple’ and someone who knows you were in an accident that wasn’t your fault. Generally speaking, the people making these calls aren’t doing for the sheer joy of having people hang up on them. They’re making them because they work. And they work because the people who pick up the phone are generally less switched on than the people Van Tam usually associates with.

What we have, again, is a system that makes perfect sense from the point of view of the intelligent people that designed it, but that might not be quite so intuitive for those who will end up interacting with it.

The common factor between the cases as I see them is the failure of the designers to respect a hidden constraint on their plans: the ability of the end user to understand the system and their place in it.

3  I would guess broadly that this type of design failure becomes more frequent the greater the distance between the designer and the end user; an awful lot of information is filtered out as we move up the levels of an organisational hierarchy, and while the broad strokes of a problem may be presented the details are very unlikely to be.

Compounding this, I suspect that that the people towards the top of an organisation tend to be rather brighter than those at the bottom. Certainly, they are likely to be better educated and have more experience with the sort of task involved in building out a product, and for many purposes these things are pretty much equivalent. The tendency of talent to rise to the top — where it can be better put to use — leads to two further problems. The first is that the people who engage with customers day-to-day as people rather than points of data are unlikely to have the skills to build systems for themselves, or to identify how to meet user needs. The second is that the people at the top who build the system are surrounded by people of similar ability who are not at all representative of the general population.

At this point, I want to take a brief detour. It is all but certain that you, the person reading this, are also not representative of the population at large. It is also very likely that you don’t know how unrepresentative you are. Robin Hanson made this point well in a 2009 post; in a random sample of US adults, only 52% of respondents could look at a table in an almanac and correctly answer a question about whether US oil exports went up or down between 1976 and 1978.

Take a different example. Around the world, 15 year old students are asked questions to compare performance across countries. One item begins by telling students that when it is 1:00 AM in Berlin it is 10:00 AM in Sydney, then asks what the time in Berlin will be when it is 7:00 PM in Sydney. What proportion of American teenagers do you think got that question right? 90%? 80%?

Just under 46% of American students answered it correctly.

The thing is, these people aren’t stupid. That’s the point. They’re average, and you don’t really know what that looks like. And that’s a problem, because it’s going to be someone like you who designs the systems they rely on and work within in everyday life.

4 From this point of view, the biggest challenge in designing a system isn’t building in the capability to hand every possible case fed into it; it’s designing one that can be used and run by the less capable without totally falling apart.

What is particularly difficult about this is that the construction of the problem (the less capable operators serving the less capable users) rules out the obvious solution of staffing your structure with benevolent social planners who can carefully work out the optimal solution for their client (¹). Instead, whatever you come up with has to be resilient to the normal on both ends.

I suspect this understanding is what results in some of the more frustrating rigidities in bureaucratic systems. A lot of time and work goes into defining the plausible use cases, and minimising the difficulty involved in handling them; call centre workers are given scripts, websites provide neat prompts, and the entire edifice is set up to funnel people into the best path to a solution with minimal fuss. The problem with this structure appears when something slightly out of the ordinary arises, because it really doesn’t offer much in the way of flexibility or independent decision making to the people operating it. It doesn’t do this because it can’t; the rules need to be simple and predictable, because discretion is likely to result in chaos.

From this angle, the primary problem of civilisation isn’t how to stop ourselves destroying one another. Humans might be selfish, grasping, greedy, venal, petty, tainted by original sin, and so on, but we are also members of a social species that generally manages to avoid the state of all against all. Instead, it’s how to harness people towards a common scheme; sure, preventing violence and theft is a part of this, but so are safety regulations and making sure we don’t build cities on top of active faults (looking at you, America). Less Hobbes’ Leviathan, and more Duns’s Leviathan (²).

5 So, back to Van Tam. In a plan’s transmission from the ivory tower to the concrete office block, quite a lot can go awry. Van Tam’s failure to understand the people who will be using his service — the elderly, the vulnerable, the scared, and the sick — is going to leave people open to scammers, who I expect will take full advantage of this golden opportunity.

And this really isn’t good enough. Highly capable people like Van Tam owe the less capable people who rely on them better than this. We can argue about how intelligence works until the cows come home, but I think we can probably broadly agree that (a) it exists and (b) people possess it in differing degrees. Why it does so is a different matter entirely, and perhaps with good schooling and tuition we could all operate on the level of Euler or Goethe. But for the most part people don’t have access to the time or resources to undertake that sort of schooling, and we have to take them as we find them.

This is, in some senses, the strongest case for redistribution and paternalism. We owe these people. We were lucky enough to be born with good genetics, or into a family that valued education, or in the catchment area for a good school, or with whatever your preferred explanation for differing ability is. As Scott Alexander has remarked, this debt is especially marked in a society which has carefully removed every outside source of support that they could once have relied on; they can’t hunt deer or gather berries or grow wheat because they have no land and have no right to it. And short of the long-overdue unwinding of the Norman conquest and the enclosure movement, they are’t going to get it. We are responsible for making the world we build one that they can navigate.

And when the government sets up a system that gives scammers a golden opportunity to fleece these people, then flings up its hands and says ‘well, I don’t think they’ll fall for it’, that is a total abnegation of that responsibility.

1. This does happen with council housing allocation for some areas operating a programme of ‘assisted choice’; staff work to present people with a carefully selected choice of available houses. From a standard economic theory viewpoint, this system is unappealing. In practice, I am unaware of any extant evidence on its relative merits.
2. This is a little unfair to poor John Duns Scotus, a philosopher who was given the title ‘Doctor Subtilis’ for the quality of his insights. Unfortunately, later propaganda gave rise to the modern word ‘Dunce’ and my desire to make a cheap point means that I am also going to besmirch the good man’s name