Grandmaster of None: The Engineer’s Gambit
Chess engines price short-term pain against positional value, which is why engineering teams lose multi-factor decisions a chess engine would win.
I’ve been part of plenty of design reviews where the decision was made long before the meeting started. It could be material selection, a build-or-buy decision, or an architecture choice. In these scenarios, a faithful scoring matrix is used to help make the decision; criteria down the side, weights in a column, a total at the bottom.
The matrix is an illusion though, as it still relies on judgement to allocate weighting to it. The decision came first, and the matrix came second – produced to make the decision look like it had been arrived at, rather than announced.
The scoring matrix is doing its real job, which is giving the decision a paper trail. But everyone who’ll live with the consequences of the decision knows exactly what happened. The option that won was the one somebody wanted to win, and the weights got nudged until the arithmetic agreed.
This approach isn’t actually always a bad thing either; in the context that the decision is passed by someone of authority or expertise, the matrix scoring and the outcome are one of the same – as it’s all based in the context of a learned approach. It does, however, fall on it’s face ion the context when the approach is adopted by junior or lesser-experienced team members. The matrix isn’t always faithful in practice as it is on paper – especially in context of project decisions where the decision is for something of a individualistic, vernacular or specific nature.
Anyone who’s read anything of my work will know I’m a sucker for stealing the homework of more successful parallel universes. So to try and address this problem, the workflow shouldn’t really aim to be borne from decision science, or from systems engineering.
My novel take is to steal the process from a chess engine – the program that plays chess on computers. For clarity, chess programming isn’t my area at all, and I’m happy to be corrected on the finer details; but it’s been interesting learning about it. And I’ve found it ludicrously ironic that engineers have spent fifty years teaching machines to make multi-factor trade-off decisions transparently and consistently, and we’ve barely borrowed a thing from the very thing we systemized for them to do in the first place.
The weighting game
To be clear, I’m not about to tell you to use a variant of a weighted decision matrix. You probably already know about them; Stuart Pugh formalised the tool in the 1980s, it’s been a fixture of systems engineering ever since, and the decision-science research is on its side: when you’ve got several conflicting criteria and a few people who care about different ones, the structured scoring is the best approach really – and it stops everybody from different departments arguing until the loudest voice in the meeting wins. There’s even been contexts where I’ve even argued for this kind of approach myself – for instance, there’s huge merit in applying an algorithmic approach to data centre site valuation with a weighted index.
The weighted-matrix tool isn’t the problem, but it’s the two ways it reliably gets abused:
Retrospective weighting As mentioned in our introductory example. Set your weights after you’ve seen how the options score and then tune them (consciously or not), until your preferred answer comes out on top. The matrix stops being a decision tool and becomes a justification tool.
False precision. A more nuanced factor that where an answer buries the mathematics underpinning it. For instance, visually a total of 4.10 in one category compared to 3.95 in another looks like a decisive win. However, if you shift one weight by five points, then the ranking flips. It enables a judgement call to be disguised a measurement. And when it’s presented in figures, the methodology isn’t typically scrutinized.
Both criticisms are fair and common – and a chess engine solves both.
Board-level decisions
Strip a chess engine down and it’s running your scoring matrix, except it’s been forced to do it properly. A position comes in, and the engine scores it as a weighted sum of factors:
Material first (queen worth nine, rook five, bishop and knight three, pawn one);
Then king safety, pawn structure and mobility, which is simply how many squares your pieces command. Dozens of weighted terms, summed into one number measured in centipawns, hundredths of a pawn.
That number is the evaluation function, and for decades the entire craft of engine building was tuning those weights against each other. Not adding more factors. Getting the relative weights right.
There’s a misconception with this though, and it should be clarified that the evaluation function doesn’t choose the move. A completely separate system, the search, generates and explores the candidate moves, millions of positions a second. Evaluation just scores whatever static position the search hands it. They’re two separate jobs, deliberately kept apart; the thing proposing options has no say in how they’re scored, and the thing scoring has no idea who proposed them.
To translate that into our context, the person advocating an option is usually also the person scoring it, and quite often the person who set the weights too. An engine would never allow that.
Generate the options as a group, with nobody scoring. Then set and lock the weights before anyone sees how the options land against them. Once the weights are fixed in ignorance of the answer, retrospective weighting becomes impossible, because there’s nothing left to refine or amend. I suppose the approach could feel bureaucratic for about ten minutes, but I’d suspect that eventually it’ll just feel honest.
Playing the long game
The second concept worth stealing is the distinction an engine draws between tactical and positional value, because it’s the one a scoring matrix almost certainly collapses. Immediate material gain is tactical; it pays off now. Pawn structure and king safety are positional; they might pay off in forty moves, or never pay off at all. The engine holds both on the board at once and scores them side by side. A simple scoring matrix would flatten everything onto one number and one unspoken time horizon; and typically, that horizon is nearly always the near term, because the near-term factors are the ones everyone in the room can feel more tangibly.
An illustrative example: Material choice for a component.
If we take a structural material choice for a long, exposed service life: carbon steel against a corrosion-resistant alloy. If we split the factors the way an engine would:
If we sum each column flat, the carbon steel wins comfortably - because three of the six factors reward it. They’re also the ‘loudest’ three factors, that most people are going to care about in the immediate short term. But the three criteria it effectively ‘loses’ on are as valid, but just only become apparent long after the project that chose it has been concluded.
An engine model wouldn’t let you ignore the three losing factors (Corrosion, Maintenance, Downtime) just because the payoff sits in the future after; positional health is scored as real value now, even when nobody can say exactly when it will carry any relevancy or take effect. [Caveat: Treat the numbers as rough, by the way. The point of the split isn’t the totals, it’s seeing that you’re trading two different kinds of value on two different clocks and pretending they’re one.]
Knowing when to resign
The technique isn’t hugely transferrable though, and there are a few flaws. Chess engines have a famous flaw called the horizon effect. Because the search can only look so far ahead, a bad outcome that’s certain but delayable can be pushed just past the point where the engine stops looking, and the engine will play something actively worse to postpone it, because the lesser damage sits inside the visible horizon and the real disaster sits beyond it. Searching deeper doesn’t fix it; as there’s always a new horizon. In our context, it’s rqually as common and applicable - consider how often an engineering decision does the same thing on purpose. Deferring a maintenance liability into a phase the business case never models. Optimising hard for the milestone that can be seen, and shoving the cost just past the edge of the analysis. Your decision has a horizon too; but the difference is the engine’s flaw is accidental, and ours is usually a choice.
There’s a couple of others too; chess hands its engine a single currency: every factor converts to centipawns, because centipawns map cleanly to the one thing that matters, the probability of winning. But often, nearly all engineering decisions has no centipawn equivalent. Pounds, kilograms, risk, downtime and reputational exposure don’t share a unit, and the moment you force them into one combined score you’ve copmmited the same crime – smuggling in judgement in, but calling it a measurement. So in reality, this could be a useful discipline; but could also result in a terrible verdict. It should also be considered that the engine gets to tune its weights against millions of self-play games with a known result. There could be manu scenarios where you don’t get to replay the same decisions, as every project is unique. You’d only find out whether the allocated weights were right in about thirty years when the project has been successful or a persistent nightmare.
This is exactly why the transparent, hand-tuned engine is the right model for us, and the modern black boxes aren’t. Stockfish’s NNUE network and AlphaZero’s self-taught evaluation are stronger models than anything a human ever hand-tuned, but you similarly can’t ask either why it likes a position and get an answer you could put in front of a review board, let alone a regulator. Being able to explain why the weights are what they are is part of the decision, not paperwork after it. The legible model beats the accurate one, because a decision nobody can interrogate is less of a decision, and arguably just more of a guess with good PR.
So the lesson here would suggest to take the architecture and the humility:
Keep the people generating options away from the people scoring them.
Fix your weights before you know the answer.
Score the factors that pay off over the asset’s whole life as real value on their own clock, not as an afterthought to the ones that pay off this quarter.
And know where your own horizon sits, the line past which the model goes blind and a human has to take over.
The best engine ever built still gets the position wrong when the decisive move sits one square beyond where it can see – so there’s still a place for us after all. On a macro-level in the context of the current news climate where we’re apparently all soon meant to be obsolete in the wake of the new AI, this finding is rather promising.
TH