Chess Has a Board. Software Has a World.

Stockfish, AI software factories, and the limits of turning programming into search

July 26, 2026 · Long read

TL;DR

The Stockfish analogy captures one plausible future: AI systems may become so much better at many programming tasks that human programmers cannot compete with them directly. But the analogy does not show that software engineering is structurally the same problem as chess, nor that compute is the only constraint.

Chess supplies a fixed state space, exact rules, legal moves, deterministic transitions, and a universal definition of winning. Original and enterprise software frequently lack all five. The problem, the desired behavior, the existing environment, and even the meaning of “correct” evolve while the system is being built.

Tests and behavioral contracts can convert portions of software development into something more like chess. But somebody must choose the state representation, decide which behavior matters, construct the evaluator, and determine whether the formalized board adequately represents reality. That scaffolding can be the real engineering work.

An AI software factory is superior when a compact, durable specification generates a much larger amount of trustworthy implementation. It is inferior when the engineer writes nearly the whole program in prompts and tests, the model translates it stochastically into source code, and the engineer then has to reconstruct and verify the result.

Turing’s work places a real theoretical limit on the strongest version of this vision. There is no universal, always-terminating, always-correct procedure for determining arbitrary program behavior. Rice’s theorem extends that limit to nontrivial semantic properties of arbitrary programs. This does not make automated synthesis, testing, or formal verification futile. It means they work by restricting software into decidable islands rather than by creating a universal correctness oracle.

The most productive division may therefore be: humans retain control of the semantic spine—the state model, ownership, transitions, architecture, policy, and irreversible decisions—while AI expands the mechanical surface area, generates tests, applies established patterns, explores alternatives, and performs broad verification.

A finite chessboard opening into a complex, evolving world of software systems, networks, data, and people

Elon Musk recently compressed an entire theory of software automation into one sentence on X:

“AI will achieve Stockfish-level coding and generalized computer use.”

That is a powerful phrase. Stockfish does not merely assist a human chess player. It plays at a level beyond any human. Applied to programming, the implication seems obvious: AI will not remain an autocomplete tool, junior developer, pair programmer, or even senior engineer. It will eventually outclass every human programmer in the way chess engines outclass grandmasters.

That prediction may turn out to be right.

But the analogy can mean two very different things.

The first is a claim about relative capability:

AI will become much better at software engineering than any human.

The second is a stronger inference about the structure of the problem—one that Musk’s sentence does not itself prove:

Software engineering is fundamentally a search problem like chess. Once models and algorithms are sufficiently capable, additional compute can search the coding solution space until it finds the correct program.

The first claim does not require the second.

An AI could become superhuman at software engineering without software engineering having the mathematical structure of chess. It might do so by interviewing stakeholders, inspecting production systems, designing experiments, negotiating conflicting constraints, generating implementations, proving bounded properties, monitoring deployments, and changing its understanding after observing the consequences.

That would be an extraordinary system. It would also be closer to an autonomous engineering organization than to Stockfish.

My concern with the Stockfish analogy began as a question about the difference between the chess solution space and the coding solution space. It then became a question about how AI changes the act of programming. Instead of writing the mechanism, I increasingly found myself scaffolding a generator: defining test cases, behavioral contracts, repository context, constraints, and stopping conditions so that a model could stochastically produce the implementation.

That led to a more uncomfortable question:

Am I building software, or am I hand-building a software factory for every ticket?

And that, in turn, led to Alan Turing.

If there are fundamental limits on deciding the behavior of arbitrary programs, then a universal software factory cannot simply search implementations and rely on a perfect evaluator to announce when it has found the right one. The evaluator itself becomes the problem.

The four questions form one continuous argument:

  1. How much is coding actually like chess?
  2. Is behavior-first agentic coding equivalent to constructing the mechanism directly?
  3. Is building the stochastic software factory really superior, or can it be inferior?
  4. Do Turing and Rice establish a hard boundary on the entire project?

The answers fit together. The central distinction is between searching a board and constructing the board on which search becomes meaningful.

Chess gives us the board for free.

Software usually does not.

Part I: How Well Does the Stockfish Analogy Hold?

“Elon constantly makes the analogy of the coding solution space to Stockfish solving chess. Can we evaluate how well this analogy holds up under analysis? Under his analysis, this means that ultimately the only constraint to solving coding is compute itself. What does the solution space of chess look like compared with coding an original or enterprise app?”

That compute-only conclusion is the strongest structural-search reading of the analogy, not something Musk’s quoted sentence itself establishes.

The strongest charitable reading

The most defensible interpretation of “Stockfish-level coding” is not that an AI will literally enumerate every possible program until it discovers the mathematically correct enterprise application.

It is that AI will become so consistently superior at programming that a human will have essentially no competitive chance against it.

That is coherent.

Stockfish does not need to have solved the game-theoretic value of every chess position in order to be overwhelmingly better than humans. It only needs to choose better moves, with greater consistency, under the conditions in which games are actually played.

Similarly, an AI does not need a complete mathematical solution to software engineering in order to outperform human engineers. It could make better architectural decisions, produce fewer defects, understand more repositories, remember more details, test more paths, work faster, operate continuously, and coordinate more implementation attempts than any individual human.

Under that interpretation, “Stockfish-level” is primarily a statement about the performance gap between machines and humans.

The analogy becomes much weaker when it is used to support a second claim:

Because chess succumbed to search, software will also succumb to search, and compute will eventually be the only meaningful constraint.

That conclusion depends on software and chess having similar evaluative structures. They do not.

What chess gives the search algorithm

A chess position can be represented as a state s. From that state, the rules determine a finite set of legal actions A(s). Applying action a produces a new state:

s′ = T(s, a)

The transition function is deterministic. Given the full legally relevant state and a move, there is no disagreement about the resulting position.

At terminal states, the game supplies an exact value:

U(s) ∈ {−1, 0, 1}

representing loss, draw, or win from the relevant player’s perspective.

Perfect play can therefore be represented recursively:

V*(s) = maxa ∈ A(s) −V*(T(s, a))

That equation is computationally difficult to evaluate across the full game, but its meaning is not ambiguous.

Chess has several extraordinary properties:

Claude Shannon’s foundational paper on computer chess explicitly treated chess as a useful wedge into broader machine intelligence. But even in that paper, Shannon distinguished these tasks from strict numerical computation: they involve judgment, trial and error, and a continuous range of quality. Chess was valuable partly because it packaged those difficulties inside an unusually crisp formal environment.

The chess search space is enormous. But it is a closed enormous space.

That difference is more important than its raw size.

Stockfish is not simply brute force

It is also misleading to describe Stockfish as a machine that wins because it mechanically examines every possible continuation.

Stockfish combines an evaluation function with highly selective alpha-beta search. Its NNUE network assigns values to positions, while search algorithms decide which parts of the game tree deserve more investigation. The engine prunes branches that appear unpromising, reduces the depth of lower-priority lines, extends tactically important lines, reuses previously analyzed positions, and orders moves to make the search more efficient. Its own documentation explains that search depth is not uniform and pruning can skip moves that a complete minimax search to the same nominal depth would have found.

Even in chess, then, the relevant production function is not:

more compute → correct answer

It is closer to:

capability = compute × representation × evaluation × search strategy × training and testing

Compute matters enormously. But compute applied to a poor representation, bad evaluator, or naive search policy does not automatically produce Stockfish.

Stockfish is also not an exhaustive proof of chess. It is a selective, approximate decision system with extraordinary practical performance.

That distinction matters for coding. An AI might attain Stockfish-like dominance by being a better approximate engineer than humans—not by turning all software behavior into a solved mathematical object.

When coding really does resemble chess

Imagine that we freeze all of the following:

The problem can then be represented as program synthesis:

p* = arg minp ∈ P C(p) subject to: p ⊨ R

where P is the candidate-program space, R is the specification, and C is some cost function over correct implementations.

Under those assumptions, coding becomes substantially Stockfish-like.

An agent can:

This model fits many real programming tasks extremely well.

An algorithm problem with exact tests is highly chess-like. So is a reproducible regression with a clearly known correct result. A protocol implementation with a formal wire specification is relatively chess-like. A bounded migration with explicit preconditions and postconditions can be chess-like. A mechanical repository-wide refactor with a strong behavioral test suite is also a good candidate.

These tasks already possess most of the board.

The mistake is generalizing from those tasks to all of software engineering.

The raw program space is not the decisive problem

An unrestricted programming language contains a countably infinite number of programs. Even after imposing a maximum length, the number of possible source strings is vast.

But raw size alone does not invalidate search.

Almost all arbitrary strings are syntactically meaningless. Language models impose strong priors over plausible code. Compilers immediately reject large regions of the space. Types eliminate others. Existing architecture and repository context constrain the design further. Many different source programs implement behavior that is equivalent for practical purposes.

An AI does not search source code by incrementing bytes in lexicographic order. It moves through a highly structured distribution of familiar patterns, abstractions, libraries, and transformations.

The critical problem is therefore not:

Are there too many candidate programs?

It is:

What makes one candidate a solution?

Chess supplies that answer independently of the engine. Software often does not.

The evaluator is the hidden assumption

For program search to work, there must be some evaluator:

E(p) = { 1, if p is acceptable; 0, otherwise }

Or, more realistically, a scoring function:

E(p) ∈ ℝ

that ranks implementations by correctness, maintainability, performance, safety, usability, and other desired qualities.

The Stockfish analogy quietly assumes this evaluator already exists.

In chess, it does. The rules and outcome function were defined long before Stockfish.

In software, constructing a trustworthy evaluator can be most of the work.

The evaluator may need to answer questions such as:

A unit-test runner can answer only the subset that has been translated into executable observations.

Somebody still has to determine which observations matter.

Original applications do not begin with a fixed objective

Consider an original product.

At the beginning, the team may not know:

The “correct app” is not hidden somewhere in a static code tree waiting to be found.

The problem formulation and solution concept develop together.

Design research describes this as the co-evolution of problem and solution spaces. Designers propose partial solutions, discover implications, reinterpret the problem, modify the solution, and temporarily stabilize a matching problem-solution pair. The process is not generally “finish defining the problem, then search the fixed solution space.”

This resembles actual programming practice.

A developer implements a thin version of a feature. The implementation reveals that the state model is wrong. That discovery changes the interface. The new interface exposes a different user workflow. The workflow changes the product requirement. The revised requirement suggests a different architecture.

The code is not merely the result of prior understanding. It participates in producing the understanding.

An AI might eventually outperform humans at this process. It could interview users, generate prototypes, analyze behavior, design experiments, and revise the product strategy.

But once the system is doing all of that, it is no longer merely searching a coding solution space with a fixed evaluator. It is participating in the creation of the objective.

Enterprise applications do not have a single clean state

Enterprise software is difficult in another way.

The desired business function may be reasonably clear, but the present system is not.

The relevant state includes far more than the repository:

Senterprise = {   source code, databases, production anomalies, integrations,   vendor behavior, permissions, business processes, regulations,   user expectations, deployment history, operational workarounds,   organizational incentives }

Much of this state is incompletely observed.

A legacy field may look unused but drive a monthly export. An odd database value may be corruption, or it may represent a workflow that never reached the documentation. A duplicated API route may be accidental, or one mobile client may still depend on it. A stakeholder may describe the desired behavior one way while production users rely on something else.

The technically cleanest final architecture is not automatically the best solution. The transition path matters:

solution ≠ final codebase alone

Instead:

solution = target system + migration path + compatibility strategy + rollout + rollback + operations

A perfect destination reached through an unsafe migration is not an enterprise solution.

The live organization is part of the machine.

Four kinds of uncertainty

The Stockfish analogy tends to treat all uncertainty as search uncertainty. But software contains at least four different categories.

Computational uncertainty

This is the most chess-like form:

Given a known objective and environment, which implementation satisfies the contract most effectively?

More search, more agents, stronger models, and more compute directly help.

Epistemic uncertainty

This concerns facts about the current world:

What does the existing system actually do?

The missing answer may be buried in production data, an inaccessible vendor system, an undocumented dependency, a user’s memory, or a workflow no one thought to mention.

Compute can analyze information it receives. It cannot recover arbitrary facts that have never been observed or made available.

Normative uncertainty

This concerns what should be chosen:

Which behavior does the organization want?

Suppose an authentication record says multi-factor authentication is enabled, but no verified factor exists. Several policies are technically possible:

No amount of searching implementations determines which policy the organization ought to adopt. Compute can model consequences, identify risks, and expose inconsistencies. But a policy choice must still be established.

Causal uncertainty

This concerns how the open world will respond:

What happens after deployment?

Users may change their behavior. Attackers may find an unexpected incentive. A vendor may throttle the new traffic pattern. A workflow improvement may move the bottleneck somewhere else.

Simulation can reduce uncertainty, but only relative to the fidelity of the model. Sometimes the only authoritative test is a staged deployment followed by observation.

More compute cannot deliver next month’s production evidence today.

Tests manufacture a local chessboard

Tests, schemas, contracts, mocks, types, and acceptance criteria are powerful because they transform an open-ended engineering problem into a more closed evaluative environment.

“Make authentication secure” is not a useful search objective.

These are much more useful:

Each invariant creates a local rule.

Once enough rules are made executable, an agent can search implementations, reject failures, compare alternatives, and stop when the evaluator is satisfied.

That is exactly the point at which coding begins to resemble Stockfish.

But notice what happened: software was not naturally a chessboard. The engineering process constructed one.

The human or system defining the contract had to decide:

The Stockfish analogy is strongest only after this work is done.

And this work can be the hardest part.

Part II: Is Agentic Coding the Opposite of Direct Coding?

“Another issue I have with this is that, so far, when working with a model, it feels like I spend a lot of time scaffolding the model and defining test cases instead of spending that time coding. It could be one-for-one while I wait for the model to solve the test cases, and then I go back and refine the contract as I get new information. Is this a more or less efficient way of thinking about coding and practicing the art of coding? It almost seems like approaching the problem from the opposite side. I can’t tell whether this is an equal approach to simply defining the code.”

It is the opposite direction.

And it is not generally equivalent.

Mechanism-first and behavior-first development

Traditional direct programming often follows a mechanism-first path:

intent → mental model → implementation → observed behavior → revised understanding

Agentic, contract-driven development often follows a behavior-first path:

intent → constraints and tests → generated implementation → verification → revised constraints

In the first process, the developer constructs the mechanism and learns from its shape.

In the second, the developer attempts to define the observable boundaries of an acceptable mechanism and lets the model search inside them.

Both processes can converge on good software. They do not necessarily produce the same information along the way.

Coding is often an instrument of thought

It is tempting to describe programming as the transcription of an already completed design.

Under that picture, the engineer first determines exactly what should exist and then performs the relatively mechanical work of expressing it in code.

Real programming often does not work that way.

I may begin with an approximate model:

Then I start writing.

The implementation reveals that ownership is awkward. A dependency points in the wrong direction. A seemingly local update actually spans three aggregates. An asynchronous callback breaks the original transaction boundary. The interface requires information that the proposed abstraction does not possess.

The code pushes back.

I reorganize it, and the new organization changes my understanding of the problem.

This is the same co-evolutionary structure seen in other forms of design: partial solutions reveal implications that alter the problem formulation itself.

Direct coding is therefore not just construction after thought.

It is often thought conducted through construction.

Code and tests describe different things

Source code and tests do not contain the same kind of information.

Code is primarily intensional. It describes how behavior is produced.

Tests are primarily extensional. They describe selected observations of what the system should produce under certain conditions.

Consider a test saying:

Given a valid authenticated session and a verified MFA factor, a provider error must not bypass the second factor.

That is an important invariant.

It does not determine:

Many implementations can pass the same test.

Some will contain a coherent state machine. Others will contain conditionals distributed across controllers, services, and callbacks that happen to produce the expected output in the tested cases.

The test constrains the machine. It does not fully specify the machine.

Finite observations underdetermine behavior

Suppose the intended function is f, and the test suite evaluates only the finite input set T.

A candidate program can be defined as:

q(x) = { f(x), if x ∈ T; wrong result, if x ∉ T }

The program passes every test while being wrong outside the tested set.

Real models do not normally produce such an explicit lookup-table cheat. But the logical problem remains: a finite collection of examples does not uniquely determine arbitrary behavior over a larger domain.

The generated implementation is selected using additional information:

That is why scaffolding grows.

Every added invariant, example, prohibition, schema, and test eliminates additional undesirable implementations that remained compatible with the earlier evaluator.

Behavior-first development can require premature certainty

The strict contract-first workflow asks:

What exactly must be true when the work is complete?

That is a good question when the desired behavior is already understood.

It can be the wrong first question when implementing the mechanism is how the engineer would have learned what matters.

Suppose I am designing a new stateful subsystem. Before touching the code, I try to enumerate every test:

But I do not yet understand the state model.

I am trying to formalize knowledge I have not produced.

The likely outcomes are:

  1. I write a shallow contract that lets the model guess the architecture.
  2. I over-specify an early design and freeze a mistaken model.
  3. I describe the implementation almost line by line in English.
  4. I repeatedly revise the tests after generated code reveals missing assumptions.

At that point, the contract-first workflow is no longer avoiding discovery. It is performing discovery through a more indirect medium.

The real efficiency equation

Direct development has a cost:

Cdirect = Cdesign + Cimplementation + Cdebugging + Cverification

Agentic development has a different cost:

Cagent = Cdesign + Ccontext + Cspecification   + Charness + Creview + Ccorrection + Cintegration

The design term does not disappear.

The AI workflow wins when the implementation and debugging effort it removes is greater than the context, specification, review, correction, and integration effort it introduces.

For a junior programmer, typing and local implementation may constitute a large share of the total cost.

For a senior engineer working in a familiar stack, the code itself may be relatively inexpensive. The expensive part is deciding:

If I spend forty-five minutes scaffolding an agent to avoid forty-five minutes of coding, I have not automatically improved productivity.

I may have simply exchanged one form of concentrated engineering for another.

One-for-one time can still produce value

Equal immediate time does not necessarily mean the agent workflow is pointless.

The resulting tests may become durable assets. The specification may support several platform implementations. The agent may produce documentation, edge-case coverage, and migration validation that I would not have written manually. Parallel agents may allow other work to advance during execution.

The agentic process may therefore create:

Those are real gains.

But they should not be confused with immediate implementation speed.

The meaningful question is not:

Did the model type the code faster than I could?

It is:

Did the process produce more trusted, integrated, maintainable behavior per unit of scarce human attention?

Waiting time is not the same as saved attention

Agent workflows often feel efficient because the model is visibly working while the engineer does something else.

That can be genuine parallelism.

It can also create delayed review debt.

Suppose three agents produce three substantial patches while I design a fourth feature. I now have:

The work ran in parallel, but understanding and integration may remain serial.

If generated output arrives faster than I can establish confidence in it, I have increased work in progress rather than completed software.

This is the same reason a factory’s output is not measured by how many parts leave individual machines. It is measured by how many conforming products pass through the whole system.

Different practices develop different skills

Direct programming develops:

Agent supervision develops:

Both are forms of engineering.

They are not interchangeable practice.

If I stop constructing mechanisms almost entirely, I may become better at specifying boundaries while losing some of the tactile understanding that lets me detect a bad internal design before a test exposes it.

That matters because every evaluator is incomplete relative to the full system.

The supervisor of an AI factory still needs enough mechanical understanding to recognize when the factory has optimized the proxy instead of the machine.

Tests are measurement fixtures

A useful way to reconcile the two approaches is to stop treating the test suite as a complete alternate representation of the software.

Tests are measurement fixtures placed around the machine.

They tell us:

They do not necessarily describe the complete internal motion of the system.

The mechanism-oriented engineer asks:

How does state move through this machine?

The contract-oriented engineer asks:

Which externally meaningful properties must remain true while it moves?

Those perspectives are complementary.

The mistake is asking either one to replace the other completely.

A better hybrid loop

For novel, stateful, or architecturally significant work, a more natural process is:

mechanism → discovery → invariant → automation → revised mechanism

First, trace the state and identify ownership, transitions, side effects, and failure boundaries.

Then build a thin vertical slice. It may be hand-written or model-generated, but its purpose is to expose the actual structure rather than to claim completeness.

Inspect the implementation causally:

Once the mechanism reveals the important facts, extract durable invariants.

Then let the model expand:

This division preserves direct mechanical reasoning where it produces the most information and uses automated search where the desired behavior is sufficiently stable.

The rule is not “always code first” or “always test first.”

It is:

Code first enough to learn. Contract first enough to control.

Part III: Am I Hand-Coding a Stochastic Software Factory?

“In other words, instead of hand coding the software, I’m hand coding the software factory for the AI to stochastically generate the software. I’m not sure why this is superior, and I’m wondering whether it is perhaps inferior.”

Yes.

It can be inferior.

The fact that an activity occurs at a level above source code does not make it more efficient, more sophisticated, or more valuable.

What the factory actually is

The effective AI production system consists of more than the model:

F = { prompt, repository context, instructions, tests, tools, agent loop, review process }

The factory accepts an intention and emits a candidate patch:

F(I, ω) → p

where ω represents stochastic variation in the model’s generation and tool use.

Because the output is probabilistic, the factory also requires quality control:

Q(p) → { accept, reject, repair }

The complete production process is therefore not merely generation:

intent → factory construction → candidate → inspection → correction → integration

The economic question is whether this entire path is cheaper or more productive than direct construction.

A factory rebuilt for every part is not much of a factory

A real factory justifies its capital cost through reuse.

Fixtures, tooling, process controls, machinery, training, and quality systems are built once and applied repeatedly. The marginal cost of the next unit falls because the production system persists.

The same should be true of an AI software factory.

Durable assets might include:

Those assets remain useful after a single ticket.

But suppose every task requires me to:

I have not built a durable software factory.

I am managing a stochastic subcontractor on a bespoke engagement.

The work may still be worthwhile. But its value must be established task by task.

The factory needs semantic compression

The factory model becomes compelling when a small amount of human semantic input produces a much larger amount of reliable output.

We can define an informal leverage ratio:

Λ = trusted, integrated behavior produced ÷ scarce human attention required

A high-leverage task looks like this:

Apply this established authorization rule consistently across forty endpoints, generate regression tests, update the shared types, and report every incompatible call site.

The human policy is compact. The implementation surface is large. The evaluator is reasonably strong. The pattern is reusable.

A low-leverage task looks like this:

In this function, fetch this record, inspect these three fields, branch under these exact conditions, call this dependency before that dependency, preserve this side effect, wrap these changes in a transaction, catch these two failures differently, and update this other record afterward.

At that point, I have supplied most of the program’s semantic content.

The model’s job is largely translation.

If I then need to verify every detail, the leverage ratio may fall below one.

The worst case is programming twice

The workflow can become:

English pseudo-program → stochastic translation → source code → human reconstruction

Instead of:

source code

Natural language is not automatically a higher-level programming language.

A higher-level language is valuable when it offers reliable semantic compression. I can express an operation compactly because the compiler, runtime, framework, or library has stable rules for expanding it.

Prompts do not always possess those stable semantics.

The meaning of an instruction depends on:

The prompt may be shorter than the implementation while containing less precision.

Then the model fills in the missing precision from its priors.

Sometimes it guesses well. Sometimes it generates a design that is locally conventional and globally wrong.

Source code is already an excellent specification medium

For an experienced engineer, code has important advantages:

If I already understand the desired mechanism, writing the source may be the most compressed way to express it.

Replacing code with prose is not automatically abstraction.

Sometimes it is loss of information.

Review can be more expensive than authorship

When I write a mechanism directly, understanding is produced as part of the act.

I know why the branch exists because I encountered the case that required it. I know why the transaction boundary is there because I reasoned through the failure path. I know why the abstraction is narrow because I rejected a broader one.

When an agent writes the code, I receive the final artifact without the sequence of decisions that produced it.

I must reconstruct:

That reconstruction can take longer than writing the code.

This is particularly true of generated code because it often looks polished. It can use familiar patterns, plausible naming, and complete-looking error handling while containing a subtle misunderstanding of the domain.

A compiler error announces itself.

A green test suite around a mistaken state model can survive for months.

The factory creates another debugging layer

Direct development primarily asks:

Is the implementation wrong?

Agentic development introduces additional possibilities:

I am debugging both the software and the process that generated the software.

That additional layer is justified only when it creates enough leverage.

Intrinsic engineering versus AI supervision tax

Not all scaffolding is waste.

A crucial distinction exists between work that the system genuinely requires and work caused by model limitations.

Intrinsic engineering AI-specific supervision overhead
Choosing the security policy Repeatedly explaining which files implement it
Defining transaction boundaries Correcting unnecessary refactors
Establishing migration postconditions Reminding the agent about repository conventions
Writing a durable regression test Adding tests only to stop a recurring model mistake
Defining authorization invariants Re-supplying context the agent failed to preserve
Resolving stakeholder conflicts Reviewing boilerplate that did not need to exist
Designing recovery behavior Running repeated generations because the first drifted

The left column is valuable even if I write the implementation personally.

The right column is a tax imposed by the current production mechanism.

It is a mistake to relabel the entire right column “higher-order engineering” merely because it occurs above the code.

Sometimes it is just supervision.

Proxy optimization is a factory defect

The factory can only optimize what its evaluator exposes.

Suppose the evaluator checks:

The real objective may additionally include:

The generated patch may maximize the visible score while reducing the actual quality of the system.

This is not unique to AI. Human organizations also optimize proxies. AI increases the speed and volume at which proxy-optimized implementations can be produced.

A strong factory therefore needs more than generation capacity. It needs a high-quality, continuously updated model of what “good” means.

That evaluator may be more expensive than the code generator.

Parallelism changes throughput, not truth

Parallel agents are one of the strongest arguments for the factory model.

One agent can implement a feature while another writes adversarial tests, a third inspects security boundaries, and a fourth reviews repository-wide consequences.

This can produce examination at a scale difficult for one human to perform manually.

It can also move the bottleneck downstream.

Every branch eventually requires:

If agent output grows faster than trusted integration capacity, the queue expands.

The scarce resource becomes not code generation but justified confidence.

That is why the correct productivity metric is:

completed productivity =   trusted behavior operating in the product   ─────────────────────────────────────────   human attention, elapsed time, and risk

Lines generated, tokens consumed, branches opened, and tests written are intermediate measures.

They are not the product.

When the factory is genuinely superior

The AI factory is most attractive when several conditions align.

The intended behavior is stable. The pattern repeats. Verification is cheaper than construction. The output surface is much larger than the semantic input. Errors are local or reversible. The repository provides strong conventions. The generated work can be integrated independently.

Examples include:

In these cases, one compact human decision controls a broad implementation surface.

That is real automation.

When the factory may be inferior

Direct construction often remains attractive when the work is:

A unique 100-line state transition can be harder to specify and review than to write.

A new subsystem whose architecture is still forming may not benefit from being generated against a prematurely frozen contract.

A subtle concurrency mechanism may require the reviewer to trace every path regardless of who authored it.

In those cases, delegating implementation may remove typing but preserve—or increase—the hard cognitive work.

The semantic spine and mechanical surface

A useful division is to separate the semantic spine from the mechanical surface area.

The semantic spine includes:

The mechanical surface includes:

For my style of engineering, the strongest workflow may be:

human-designed semantic spine + AI-expanded mechanical surface + AI-assisted adversarial verification

That is different from:

human-built evaluator + AI guesses the entire machine

The former uses AI as force multiplication.

The latter risks replacing a precise implementation process with an ambiguous specification process and an expensive review process.

A practical decision matrix

Two variables largely determine whether the factory is attractive:

  1. How well is the desired behavior understood?
  2. How often will the pattern be reused?
One-off or novel Repeated or broad
Behavior understood Direct coding or a tight hybrid may be fastest Build the factory; amortization is strong
Behavior uncertain Human-led exploration and thin prototypes Discover the pattern first, then encode and automate it

The worst place to build an elaborate factory is the upper-left corner’s opposite: a one-off mechanism whose behavior and architecture are both still being discovered.

The best place is a stable rule with a broad, repetitive implementation surface.

A stopping rule

I should stop scaffolding and write the code when the prompt begins to specify control flow line by line, when the instruction is longer than the likely implementation, when nothing in the scaffolding will survive the ticket, or when reviewing the generated result requires reconstructing every path anyway.

I should continue investing in the factory when the evaluator will remain in the repository, the contract supports multiple implementations, the pattern is likely to recur, or the generated surface is far larger than the human policy.

The principle is simple:

Do not build a factory to manufacture one part that is easier to machine by hand.

Part IV: What Turing Says About Searching and Understanding Programs

“How does all of this square with the Turing machine and Alan Turing’s research about understanding the behavior of a computer program? I feel that, if I understand the research correctly, it might definitively place a limit on being able to search the solution space and define the test cases and behavioral contract correctly. Let me know if this is wrong.”

The intuition is substantially correct, with an important qualification.

Turing’s work does not prove that programs cannot be understood.

It does not prove that testing is futile.

It does not prove that formal verification is impossible.

It does not prove that AI cannot become radically better than humans at software engineering.

It establishes a narrower and more consequential result:

There is no universal algorithm that takes arbitrary programs and always terminates with a correct answer to certain general questions about their behavior.

That defeats the strongest possible version of the universal software factory.

Computability is different from complexity

Three barriers are often blended together.

Under-specification

The desired answer has not been fully defined.

What should the system do?

This is common in product and enterprise engineering.

Undecidability

No algorithm can be guaranteed to compute the answer for every instance in the unrestricted problem class.

Can a universal procedure always determine this property?

This is where Turing and Rice apply.

Intractability

An algorithm exists, but executing it may require unreasonable resources.

Can we afford to compute the answer?

Chess is primarily associated with the third barrier. The game is finite and specified, but exact exhaustive analysis is enormous.

General software can suffer from all three at once.

More compute can attack intractability.

It does not, by itself, supply an absent objective.

And it does not turn an undecidable general problem into a decidable one.

Turing’s result

Turing’s 1936–37 paper introduced an abstract machine with a finite control, a tape, symbols, and mechanical transition rules. He then investigated whether there could be a general finite process for determining whether a machine was “circle-free”—roughly, whether it would continue producing the relevant output rather than becoming trapped in an unproductive process. He proved that no such general process exists.

The modern halting problem is commonly expressed through a hypothetical analyzer:

H(P, x) = { 1, if P(x) eventually halts; 0, if P(x) runs forever }

Assume H is always correct and always terminates.

Now construct a program D that receives a program description and behaves as follows:

Now ask what happens when D receives its own description.

If H(D, D) predicts that D(D) halts, D loops.

If H(D, D) predicts that D(D) loops, D halts.

Either prediction contradicts the program constructed from it.

Therefore the assumed universal analyzer cannot exist.

The obstacle is not insufficient hardware.

The specification of the analyzer is internally impossible.

Rice’s theorem moves from halting to semantics

Rice’s theorem generalizes the idea.

In its standard form, it says that every nontrivial extensional property of the partial function computed by an arbitrary program is undecidable.

“Extensional” means that the property concerns what the program computes, rather than its source-code spelling.

“Nontrivial” means that at least one program has the property and at least one program does not.

Examples include questions such as:

This must be stated precisely.

Rice’s theorem does not mean that every question about code is undecidable. Syntactic properties are often easy to decide. We can determine whether a file contains a particular import, whether an abstract syntax tree contains a loop, or whether a program type-checks.

Nor does it mean that no useful semantic property can ever be proved for a particular program.

It means there is no universal total procedure that correctly decides every instance of a nontrivial semantic property across arbitrary programs.

The evaluator becomes the impossible component

A generalized software-search system needs a verifier:

V(p, S) = { 1, if p ⊨ S; 0, if p ⊭ S }

The system can enumerate or generate candidate programs:

p1, p2, p3, …

But it needs to determine when one of them is truly correct.

For arbitrary programs and arbitrary semantic specifications, no verifier can universally possess all three of these properties:

  1. Soundness: it never approves an incorrect program.
  2. Completeness: it approves every correct program.
  3. Termination: it always returns an answer.

For an undecidable property, an always-terminating verifier must sacrifice soundness, completeness, or both.

Practical systems make different tradeoffs.

Tests terminate under controlled conditions but cover only selected executions.

Static analyzers use abstractions and may produce false positives or fail to capture some behavior.

Proof checkers can reliably check a supplied formal derivation, but automated proof search may not always find a proof or counterexample and terminate.

Runtime monitors observe actual executions, not every possible future execution.

Bounded model checkers can be exhaustive inside a chosen finite model while making no universal claim outside the bounds.

These are not merely embarrassing limitations of present-day tools. For unrestricted programs, some form of restriction or incompleteness is mathematically unavoidable.

Search is possible; perfect stopping is not

Turing does not prevent a machine from generating programs.

It prevents a universal solution to the stopping question:

Have we now found a program that is correct in every relevant semantic sense?

A search process may:

What it cannot have, in the unrestricted case, is a universal infallible procedure that always announces either:

and always finishes.

That distinction is essential.

Superhuman practical competence does not require universal decidability.

But universal decidability cannot be derived from superhuman practical competence.

Why complete automatic test generation is impossible in general

Suppose there were an algorithm that accepted any arbitrary specification and program and generated a finite, always-terminating test suite with this property:

program passes the suite ⇔ program is semantically correct

We could run the generated suite and decide whether the program satisfies the specification.

For unrestricted semantic properties, that would produce the universal decider that Turing and Rice rule out.

Therefore, at least one limitation must appear:

Automatic test generation can still be extraordinarily valuable.

The asymmetry is:

A valid, reproducible failing test supplies a counterexample to the asserted contract.

But:

Passing every currently known test does not generally prove the absence of all defects.

One valid counterexample is sufficient to refute a universal claim.

A finite absence of counterexamples is not normally sufficient to establish it.

Behavioral contracts are stronger, but not magical

A formal contract can express much more than a list of examples.

Instead of testing ten unauthorized users, we might specify:

∀ u, r: ¬authorized(u, r) ⇒ ¬canRead(u, r)

That quantified property is stronger than a finite test set.

C. A. R. Hoare’s axiomatic approach formalized program reasoning using assertions such as:

{ P } Q { R }

where P is a precondition, Q is a program, and R is a postcondition. In Hoare’s original partial-correctness reading, the triple establishes that if P holds and Q terminates, then R holds; termination requires a separate argument. The method allows us to prove that a program preserves specified logical relationships.

But formal verification separates two questions.

Does the implementation satisfy the contract?

This is a mathematical relationship between the program, model, and specification.

Is the contract an adequate statement of what the real system should do?

This is a modeling and requirements question.

A perfectly valid proof can establish the wrong business rule.

For example, we might formally prove that a malformed MFA record always locks out the user. The proof does not tell us whether locking out that user is the desired recovery policy.

We might prove that path and query identifiers are accepted whenever they match. The proof does not tell us whether allowing mixed identifier sources creates an ambiguity or attack surface the organization wishes to prohibit.

Formalism can prove the consequences of the premises.

It does not choose the premises for us.

Specification and implementation correctness are different

Let:

Rformal

be the formal contract, and:

Ractual

be the real, perhaps only partially understood, organizational need.

Verification may establish:

p ⊨ Rformal

The product is successful only if:

Rformal ≈ Ractual

The first relationship may be mathematically strong.

The second is often empirical, political, historical, and revisable.

This creates two independent failure modes:

  1. The implementation fails to satisfy the specification.
  2. The specification fails to represent the need.

Turing and Rice constrain universal automation of the first problem.

They do not solve the second problem at all.

Randomness does not escape undecidability

A stochastic generator can sample candidates in productive ways.

Randomness can:

But randomness does not create a universal correctness oracle.

For any fixed random seed, the system is still performing a computation. Across seeds, it may attach probabilities to outcomes. It can be highly reliable on a useful distribution.

To remain compatible with undecidability, it must sometimes:

That is exactly what practical coding agents do.

Their stochastic nature improves search. It does not eliminate the theoretical limit on universal evaluation.

The finite-computer objection

A real physical computer has finite storage.

If we freeze:

then the system has a finite number of states.

In principle, one could explore states until reaching a terminal state or a previously visited state.

This moves the problem from undecidable to decidable.

But two qualifications matter.

First, the finite state space may be unimaginably large. The exhaustive procedure may be useless in practice even though it terminates in principle.

Second, real software claims usually quantify beyond a single frozen bound:

Once those dimensions are bounded, the actual claim becomes:

The program satisfies this property inside this model and within these limits.

That can be an exceptionally valuable guarantee.

It is not the same as universal correctness in the open world.

Restriction is the practical escape

Software engineering has not responded to undecidability by abandoning verification.

It responds by restricting the problem.

We use:

Clarke and Emerson’s foundational work on temporal-logic synthesis explicitly relied on finite-state synchronization skeletons and a finite-model property. Within that restricted setting, a decision procedure could determine satisfiability, construct a finite model, and derive a program skeleton satisfying the temporal specification.

This is the productive lesson.

We do not solve arbitrary program correctness.

We create decidable islands.

A broad requirement such as:

Make concurrent access safe.

is transformed into a finite-state model with specific transitions and temporal properties.

A broad security goal such as:

Never bypass a verified factor.

becomes a collection of state transitions and invariants that can be tested, modeled, or proved under explicit assumptions.

The narrower and more formal the island becomes, the stronger automation can be inside it.

But narrowing the island requires judgment.

Somebody must determine which state to include, which behavior to abstract away, and whether the model captures the risks that matter.

Turing does not prove direct coding is superior

The undecidability result applies to human-written code too.

A human engineer cannot universally determine the behavior of every arbitrary program.

Direct coding does not escape the theorem.

The theoretical result therefore does not prove:

Humans should always write code manually.

It proves:

The AI-factory path cannot justify itself through a universal promise that enough computation will eventually generate and conclusively verify every desired program.

The practical comparison remains contextual.

For a bounded protocol with a formal state machine, synthesis may be dramatically superior.

For a repetitive application pattern with a powerful test harness, an agent may generate reliable work at great scale.

For a novel socio-technical system whose objective and environment are changing, constructing the evaluator may remain as difficult as constructing the mechanism.

Turing tells us that this evaluator problem cannot be made universally perfect merely by adding compute.

How the Four Questions Fit Together

The four questions are not separate objections. They describe one chain.

1. Stockfish works because the board already exists

Chess provides:

Its search problem is vast but unusually well-defined.

2. Agentic coding asks the engineer to construct a board

Prompts, tests, schemas, contracts, tools, and fixtures define a local search environment.

The agent becomes more effective as that environment becomes clearer.

This is why the scaffolding can consume so much time. The engineer is not merely requesting code. The engineer is manufacturing the conditions under which code generation can be evaluated.

3. The factory pays only when the board is reusable

If the same rules govern many implementations, constructing the board creates leverage.

If the board is rebuilt for one small mechanism, its construction cost may exceed the implementation cost.

A factory is economically compelling when its fixed cost is amortized.

Otherwise it may be a more indirect way of producing one bespoke part.

4. Turing limits the possibility of a universal board

For unrestricted programs, there is no perfect general evaluator that is simultaneously sound, complete, and guaranteed to terminate.

Therefore the software factory must work within restricted domains, incomplete evaluators, probabilistic confidence, external observations, or human judgment.

That is not a temporary embarrassment.

It is the mathematical shape of the problem.

We can summarize the whole argument this way:

AI coding becomes Stockfish-like only after engineering creates a trustworthy board.
Creating that board can be the central engineering task.
The board is valuable when reusable and costly when bespoke.
No universal board can perfectly decide arbitrary program behavior.

A More Accurate Model of AI Software Engineering

The better analogy may not be chess.

It may be a mixture of:

The process looks less like a single fixed game tree and more like:

observe the world → form a model → propose a mechanism → construct an evaluator → search implementations → deploy cautiously → observe consequences → revise the model

Within this loop, many subproblems are highly searchable.

AI can generate code variants. It can design tests. It can discover edge cases. It can inspect repositories. It can compare architectures. It can attempt proofs. It can simulate load. It can review changes from security, performance, and maintainability perspectives.

But the loop remains coupled to the world.

The world supplies new information that was not present in the original prompt.

Compute may dominate without being the only constraint

There is a metaphysical argument that all machine intelligence ultimately runs on computation. Under that broad definition, stakeholder interviews, experimentation, observation, and policy analysis can all be performed by a sufficiently capable computational system.

But that usage makes “compute is the only constraint” nearly empty.

An autonomous AI engineering organization would still need:

Its cognition may be powered by compute.

Compute alone does not grant the system access, authority, evidence, or legitimate objectives.

A model can decide which production experiment would be informative. It cannot observe the result before the production environment has generated it.

A model can identify a policy conflict. It cannot derive a unique organizational preference from logic when several value choices are coherent.

A model can infer hidden behavior with high probability. It cannot guarantee knowledge of an arbitrary fact absent from its observations.

These are not all “compute constraints” in the operational sense relevant to engineering.

Superhuman engineering remains plausible

None of this implies that humans retain a permanent exclusive role.

An AI system may eventually become better than humans at:

Such a system may make human programming economically marginal.

But its success would come from mastering the entire open-world engineering loop, not merely from searching code strings more deeply.

Calling that system “Stockfish for coding” would understate what it had become.

It would be an autonomous product designer, systems architect, developer, tester, operator, security reviewer, and organizational decision agent.

What This Means for the Practice of Coding

The practical question is not whether to use AI.

The practical question is where AI creates leverage and where it merely relocates the work.

Preserve direct contact with the machine

For stateful, novel, or high-risk mechanisms, I should retain enough direct implementation work to understand:

This is not nostalgia for typing.

It is preservation of causal knowledge.

Turn discoveries into durable constraints

Once implementation or investigation reveals an important rule, encode it:

The most valuable contracts are not arbitrary hoops constructed to control one generation.

They are compressed engineering knowledge that protects the system after the original engineer has moved on.

Delegate breadth after the semantic center stabilizes

Once the state model and policy are clear, AI can be used aggressively.

It can expand one decision across:

This is where stochastic generation becomes a factory rather than a translation service.

Use multiple agents to challenge, not merely multiply

Parallel agents are most valuable when they provide independent perspectives:

Generating five versions of the same plausible patch is less valuable than giving five agents distinct evaluative responsibilities.

Review causally, not cosmetically

A generated patch should not be accepted merely because it looks idiomatic and passes visible tests.

The reviewer should still ask:

The machine must still make sense as a machine.

The Final Verdict on the Stockfish Analogy

The analogy is useful in one important sense:

AI may eventually outperform every human programmer so decisively that competing through unaided manual implementation becomes pointless.

That is a plausible prediction.

The analogy is also useful for bounded programming tasks:

Once a problem has a stable state representation, explicit rules, an executable contract, cheap simulation, and a trustworthy evaluator, additional search and compute can produce increasingly capable solutions.

That already describes many algorithmic tasks, refactors, regressions, protocol implementations, and test-driven changes.

But the analogy fails as a complete theory of original and enterprise software.

Chess begins with the rules already written.

Software engineering often begins with incomplete evidence about what the rules should be.

Chess separates the player from the board.

Enterprise software changes the board on which the organization is operating.

Chess has a universal payoff.

Software has stakeholders with conflicting objectives, policies that must be chosen, risks that cannot be reduced to a single scalar, and consequences that become visible only after deployment.

Chess permits exact simulation.

Software interacts with people, vendors, institutions, attackers, physical devices, and future events.

Chess search is hard because the tree is enormous.

Software search can be hard because the tree is enormous, the evaluator is incomplete, the environment is partially observed, the objective is evolving, and some general semantic questions are formally undecidable.

That is the deepest flaw in the compute-only argument.

The argument treats the correct evaluator as free.

It is not free.

Sometimes the evaluator is a small test suite that can be written in ten minutes.

Sometimes it is a formal state model requiring days of work.

Sometimes it is a production experiment requiring weeks of observation.

Sometimes it is a policy decision that no amount of computation can uniquely derive from technical facts.

Sometimes, under unrestricted assumptions, the perfect evaluator cannot exist at all.

Conclusion: The Board Is Part of the Product

The future of programming may involve far less hand-written implementation.

It may involve agents continuously generating, testing, reviewing, and deploying software. A single engineer may coordinate a level of implementation capacity that once required a large team. Reusable contracts and repository-aware harnesses may turn broad categories of application development into highly automated production systems.

But that future is not obtained merely by pointing enough compute at the space of all possible code.

Software becomes searchable only after enough of the problem has been represented:

Constructing that representation is not administrative overhead around the engineering.

It is often the engineering.

Sometimes the representation is far more reusable than the implementation. In those cases, building the factory is a profound improvement. One durable contract can generate several implementations, support future changes, prevent regressions, and enable parallel verification.

Sometimes the representation is merely an imprecise duplicate of a small program. In those cases, the engineer writes the program in English, the model translates it into source code, and the engineer reviews the translation. The factory is inferior to the machine it was built to produce.

Turing adds the final boundary. No universal procedure can perfectly determine arbitrary program behavior. Tests, contracts, proofs, types, and model checkers succeed by restricting the domain, choosing abstractions, and constructing decidable islands. They can provide extraordinary assurance inside those islands. They cannot eliminate the need to decide whether the island represents the world that matters.

So the mature position is neither:

AI coding is merely autocomplete and will never replace direct programming.

Nor:

Software is chess, and enough compute will mechanically search its way to every correct application.

It is this:

AI can make implementation search extraordinarily powerful. The limiting problem increasingly becomes the construction of trustworthy objectives, models, evaluators, and integration processes.

That is why the experience of modern agentic coding can feel inverted.

Instead of hand-coding the machine, I may find myself hand-coding the environment that generates and judges the machine.

Whether that is progress depends on what survives.

If the contract becomes durable knowledge, the tests preserve important behavior, the harness supports many changes, and the implementation surface greatly exceeds the human semantic input, I have built a factory.

If the scaffolding disappears with the conversation, the model merely translates my pseudo-program, and I must retrace every generated path to trust it, I have written the software twice.

Chess has a board.

Software has a world.

And in software engineering, the board itself is often part of what we are being paid to build.

← Back to Blog
BVT logo

What Clients Say

Verified reviews from real projects

“Amazing in communication.”

⭐⭐⭐⭐⭐

Client · iOS App (Swift & Firebase)

“Went above and beyond.”

⭐⭐⭐⭐⭐

Client · Firebase Integration Revamp

“It was great working with Bill! Very pleasant and knowledgeable.”

⭐⭐⭐⭐⭐

Client · Language Learning App

View All 30+ Verified Reviews