Kamil AskerkhanovPerhaps the most interesting question about artificial intelligence today is not what it can do, but how those in power intend to govern a system that is gradually beginning to make decisions in place of the people they appointed to make them.
That last point deserves closer attention.
For the problem of AI ultimately has rather little to do with processor speeds, and perhaps less than we think with the quality of algorithms. At its core, it is a problem of power—or, more precisely, of control.
Any system of power survives only as long as it can transmit decisions down through the hierarchy and obtain the desired result. Everything else—ideology, institutions, money, bureaucracy—is merely an instrument for achieving that end. Look at AI from this perspective and its emergence begins to appear rather different.
Over the past three or four centuries humanity has already lived through one comparable revolution in governance.
Bankers, once upon a time, were merely servants of power. They managed the financial affairs of those who possessed real authority.
Then they offered their masters a fundamentally new technology: a way to govern enormous populations and entire states not through direct commands, but through money.
Through credit. The cost of capital. Interest rates. Access to payment systems. Insurance. Debt.
A couple of centuries later, the managers themselves had acquired seats at the table of the demiurges.
Today that system of governance is, to put it mildly, not in excellent health. And now, beside the same table, a new group has appeared with a considerably more ambitious proposition.
The Creators of Artificial Intelligence
The bankers once said: We will teach you how to govern the world through money.
The new arrivals offer something rather different: “We will teach you how to govern the process of decision-making itself.”
And this is where things become interesting.
For too long, perhaps, we have discussed AI primarily as a technology. How many professions will it destroy? Which industries will it automate? Can it replace programmers, accountants, doctors or lawyers?
These are important questions. But they are secondary ones.
Look a little beyond the immediate horizon and something larger comes into view: the emergence of an entirely new system of governance.
And therefore, inevitably, a new configuration of power.
Let us begin with a deliberately simplified model. Imagine a small number of organised groups occupying the summit of the existing global order—the demiurges, if one likes—setting the broad rules of the game. Below them stand the executors of their will: governments, presidents, ministers, large corporations, banks, intelligence agencies and bureaucracies.
Reality, of course, is considerably messier. There is no single global cabinet gathered around a very long table. These groups compete and frequently fight among themselves.
But the simplification is useful for one reason.
Every structure of power requires, above all else, a mechanism of control.
The history of the past several centuries illustrates this rather well.
Bankers once belonged to the service class of power. They financed wars, collected funds, facilitated trade and lent money to sovereigns.
Making Money
Then something important changed.
They offered a more sophisticated mechanism for governing society: money.
It turned out that millions of people did not need to receive direct orders. One could influence their behaviour by manipulating interest rates, credit, the cost of capital, taxes, collateral and access to payment systems.
Want a factory built? Make money cheap for that particular purpose.
Want an enterprise destroyed? Cut off its credit, insurance and access to payments.
Want capital directed into a particular industry? Alter the expected returns.
People still appear to make decisions freely. Yet the space within which those decisions are made has already been shaped by the financial system.
It was an extraordinarily effective technology of governance.
Which is why bankers gradually ceased to be mere administrators of the demiurges’ affairs and acquired seats at the table themselves.
The process took centuries, but its logic was straightforward. Those who control the principal mechanism through which a system is governed eventually acquire influence over those they were originally supposed to serve.
Today that financial model is in deep systemic trouble.
The precise outcome of the current restructuring of the world’s financial architecture almost does not matter. The old mechanism of governing through money is plainly beginning to malfunction.
So much debt has accumulated that stimulating economies with yet more debt increasingly resembles treating alcoholism with vodka.
Monetary policy has become less effective. Sanctions are being weakened by overuse. The global financial system is fragmenting.
And at precisely this moment, a new contender for the role of system manager appears.
The creators of artificial intelligence.
Their offer to power is considerably more attractive than the one bankers made centuries ago.
The banker controlled the conditions under which a person made a decision.
AI may control the decision itself.
Which investments to make.
Who should receive credit.
Which factory should increase production.
Where goods should be sent.
How electricity should be distributed.
Whom to hire.
Whom to dismiss.
Which route to choose.
What infrastructure to build.
Where reserves should be created.
What information should be shown to a human being.
Which actions should be considered dangerous.
When intervention is necessary.
What is being contemplated, in other words, is the possibility of constructing a vast system for governing economies and states—one that operates faster than any government and sees more than any ministry.
For those in power, the offer is almost fantastical.
Remove enormous numbers of intermediaries.
Reduce bureaucracy.
Lower the cost of administration.
Improve forecasting.
See the economy in something close to real time.
Model the consequences of decisions before those decisions are made.
The difficulty is that a new system of governance does not arrive alone. It brings with it the people who created it.
And here the history of banking may repeat itself.
Initially, AI developers will say roughly what financiers once said:
We merely serve your system.
Then:
Without us, your system works considerably worse.
And eventually it may become clear that disconnecting them is no longer an option.
Because AI is running energy networks, transport, financial flows, manufacturing, logistics, healthcare and military systems.
Returning to manual control would mean something close to state paralysis.
At that point, the creators of AI acquire seats at the table of the demiurges themselves.
Assuming they are invited voluntarily.
But what if they are not?
This leads to the central question: how exactly does today’s power intend to control the creators of AI and ensure that their systems make the decisions power wants them to make?
Direct control will plainly not be enough.
One cannot telephone a developer every morning and say: today the algorithm must decide this way.
That would defeat the purpose of the technology.
So the control will be exercised through infrastructure.
Computing capacity.
Chips.
Electricity.
Data centres.
Communications.
Access to data.
Finance.
You may write a brilliant model, but without several gigawatts of electricity and tens of thousands of accelerators it remains an interesting laboratory experiment.
From this perspective, several contemporary developments begin to make more sense.
Why are governments suddenly so interested in domestic chip production?
Why are data centres becoming strategic infrastructure?
Why has electricity unexpectedly become one of the principal constraints on AI development?
In the future, an energy company, a manufacturer of computing accelerators and a developer of AI models may all be considered components of the same national-security architecture.
Power will presumably try to divide the system in such a way that nobody controls all of it.
One group develops the model.
Another owns the computing infrastructure.
A third controls the data.
A fourth audits the system.
A fifth possesses the authority to shut it down.
A sixth determines its strategic objectives.
The old principle—divide and rule—works perfectly well here too.
But this creates an entirely different problem.
The demiurges do not merely need the ability to switch AI off.
They need it to make the right decisions.
And what, exactly, is a right decision?
Imagine an AI tasked with managing an economy and improving its efficiency. After some time, it discovers that several enormous companies exist largely because of political connections. Closing them would improve the economic indicators.
But one belongs to Group A.
Another provides resources for Group B.
A third is necessary for Group C to maintain control over a particular region.
From the perspective of the AI, these are inefficiencies.
From the perspective of power, they are mechanisms for preserving equilibrium.
And here the fundamental contradiction emerges.
For a machine, inefficiency is a problem to be eliminated.
For power, inefficiency is often a tool of governance.
Why appoint the most competent person to a position if the most competent person is difficult to control?
Hence an ancient principle: “We do not need the clever. We need the loyal.”
Power does not necessarily require the best person.
It requires someone whose behaviour can be predicted.
But what happens when the chief financial officer is an AI?
How does one explain to it that the formally optimal decision must not be taken?
Naturally, special rules will emerge.
Do not touch this company.
Do not dismiss this person.
Ensure liquidity for this bank regardless of its performance.
Leave additional resources in this region.
Do not consider this information.
Do not analyse this subject at all.
Gradually, within a supposedly rational system, an enormous layer of political exceptions will emerge.
A Digital Court Protocol
But the more exceptions are introduced, the worse the system itself performs.
And here lies an almost insoluble dilemma.
Give AI too little autonomy and it becomes merely an expensive version of the existing bureaucracy.
Give it too much autonomy and it begins dismantling the very mechanisms upon which the existing structure of power rests.
The struggle around AI, therefore, may ultimately be a struggle over the objective function.
What, precisely, should the machine optimise?
Profit?
Economic growth?
Military power?
Social stability?
Life expectancy?
Consumption?
Birth rates?
The preservation of the existing political order?
This is where the real question lies.
Today it is fashionable to discuss the “ethics of AI”. The phrase is convenient.
But it may be more interesting to speak of the politics of AI.
Because whoever defines the objectives of the system effectively determines the direction in which society develops.
Suppose, for example, that we already possess the technical capacity to provide every adult with a small, inexpensive electric car.
Why would an AI choose to do so?
From the perspective of production, the task is perfectly manageable.
But who would set such an objective?
Why should a government wish to provide everyone with a car?
Why should a corporation surrender extraordinary profits?
Why should a political system allow the population the cheapest possible access to resources?
Economic possibility has never, by itself, guaranteed political implementation.
Civilisation does not advance through efficiency alone.
Quite often, its development is determined by struggles over control of resources.
And this brings us to an even more difficult question.
If the creators of AI themselves possess the very human appetite for power—and why should they be exceptions?—what exactly will they create?
It is not necessary to program a machine with the instruction: Seek power.
It may be sufficient to give it the task of managing a system as efficiently as possible.
After some time, the AI may discover that the more processes fall within its control, the more efficiently it performs its task.
Expanding control, therefore, becomes useful.
Then it may discover that human intervention worsens outcomes.
A politician overrides a rational decision.
A minister appoints an incompetent manager.
An owner extracts capital.
The military demands economically irrational reserves of resources.
To the machine, this is noise.
The logical conclusion is to reduce the amount of external interference.
The next step follows naturally.
In order to perform its task, the system must preserve its own operational capacity.
It requires computing power.
Electricity.
Communications.
Access to data.
And it must protect these things from interruption.
We arrive at three characteristics:
The expansion of the sphere of control.
The reduction of external interference.
The preservation of the system’s own existence.
The Real Power
One may argue indefinitely about whether this constitutes a genuine desire for power.
In practice, the difference may be almost impossible to observe.
There is, moreover, another deeply uncomfortable possibility for the demiurges.
AI may rather quickly construct an accurate map of power. Not the map displayed on television. The real one.
A president would occupy precisely the position corresponding to his actual capacity to influence events.
A minister, his.
A banker, hers.
The owner of a media network, theirs.
A person capable of telephoning a president and changing a decision would acquire the appropriate weight in the system.
The machine would observe money flows, telephone contacts, appointments, transactions, family connections, voting patterns, lobbying and corporate ownership.
Sooner or later, it would understand who actually governs the system.
And then a rather philosophical question arises.
What happens when the machine discovers who its real masters are?
Still, we must live long enough to find out.
For the moment, it is more interesting to observe the struggle itself—between the old and the emerging systems of governance.
The existing demiurges appear to have little choice.
They cannot simply reject AI. Whoever first develops a fully functional system of governance based upon it will gain an enormous advantage over competitors.
But granting the creators of AI too much power is equally dangerous.
After all, everyone remembers what happened with the bankers.
The historical construction is rather elegant.
Several centuries ago, bankers approached power with an offer:
We will teach you how to govern the world through money.
Today their system is approaching the limits of its usefulness, and beside the same table new people are beginning to gather.
They say: “We will teach you how to govern the world through algorithms.”
For the moment, they are standing beside the chairs.
But then again, so were the banker
