How can we build an AI without knowing how it works?

There's a strange fact sitting underneath the entire AI industry, and almost nobody selling AI says it out loud: the people who build these systems don't fully understand how they work. That's not a scandal, and it's not a secret. It's the single most important fact about modern AI — and once you actually get it, both the hype and the fear become much easier to judge.

What is one of these AI models, actually?

Worth pinning down before we go further. An AI model is, physically, one very large file — billions of numbers, nothing else — that takes text in and puts text out, a small piece at a time. It was made by having a program read an enormous slice of the internet: books, sites, code, conversations. But it doesn't keep the pages — it keeps the patterns. So when you ask it something, it isn't looking anything up, and it doesn't see everything at once; it's producing the next word, then the next, from what training pressed into its numbers. Whether that adds up to real understanding is a live question — one this series keeps coming back to. What matters for this post: nobody wrote those numbers by hand.

Didn't somebody build it? How can they not know how it works?

Almost everything humans have ever made was understood before it was built. A steam engine is designed: a person placed every valve and piston for a reason they could state. Understanding came first, the machine second. AI broke that pattern — and the break is precise. What engineers actually designed is the training process: the structure of the network, the goal ("predict the next word well"), and the loop that adjusts it. That part is as well understood as any engine. What nobody designed is the model itself. Training starts with billions of random numbers and nudges them trillions of times, each nudge making the next prediction slightly better. The final arrangement of those billions of numbers — the thing that actually writes your email or answers your question — was found by that process, not authored by any person. We built the machine that builds the machine. We understand the kiln completely. The crystal that comes out of it is the mystery.

How can you make something without understanding it?

We've done it before — for most of history, actually. Humans ran breweries for roughly five thousand years before anyone knew yeast existed: we understood the procedure (do these steps, get beer) with zero understanding of the mechanism (microorganisms doing chemistry). We bred wolves into border collies over millennia by selecting for behaviour, without knowing genes existed. The pattern is the same every time: you can control a process that produces a thing without understanding the thing it produces. Training an AI is selective pressure — "get better at predicting text" — applied trillions of times. That's why the honest way to describe a modern AI model is not built, like a bridge. It's grown, like a crop.

So is AI a complete black box?

No — and this is where it gets genuinely interesting. The mystery lives at a very specific level. At the micro level, understanding is perfect: every one of those billions of numbers can be read, every operation inspected. Nothing is physically hidden — it's not a black box in the sense of "we can't see in." At the level of meaning, understanding is poor: which combination of numbers implements "this sentence is sarcastic"? Which part does the carrying when it adds 47 and 38? That's the open question. It's like having a perfect recording of every neuron firing in a brain and still not knowing what a thought is. We see everything. We understand little of what we see — so far.

Is anyone actually figuring it out?

Yes — there's an entire scientific field working on exactly this, called interpretability, and it works the way biology does: treat the grown system like an organism and study it with instruments. Researchers have learned to find "features" — directions inside the network that light up for one real concept. In a famous experiment, Anthropic (the company behind Claude) isolated a feature for the Golden Gate Bridge and turned it up; the model started steering every conversation toward the bridge. Finding a feature and steering the model with it is real mechanical understanding — a wire traced through the crystal. The field is young and its own researchers are the first to say most of the map is still blank. But you don't have to take anyone's word for it: on a free site called Neuronpedia you can look inside real models yourself, no coding required. The excavation is happening in public, and you can watch.

Why should a business owner care?

• •It calibrates your vendor radar. Anyone who tells you their AI is fully understood and perfectly predictable either doesn't know the field or hopes you don't. The honest claim is: enormously capable, genuinely useful, not fully explainable end-to-end.
• •It explains the weirdness. An AI that's brilliant at a hard task and confidently wrong at an easy one isn't malfunctioning — that's the expected shape of a grown system, with strengths and gaps nobody designed.
• •It should change how AI gets installed in your business. You don't hand a grown system the keys on trust. You give it small scopes, keep a human sign-off on anything that reaches a customer, and check it against your own numbers. That's not fear — that's how professionals handle powerful things that aren't fully mapped. Medicine deploys drugs that passed trials before the mechanism is fully understood; it does so with monitoring and controls, not blind faith.

The one thing to take away

AI was grown, not built — and the industry's own builders are still mapping what they grew. You don't need to fully understand a thing to get real value from it; brewers didn't, for five thousand years. But the people you hire to install it should respect the line between what's known and what isn't, and be straight with you about which side of the line a claim sits on. That's how we work: plain terms, small proofs, you stay in control of anything that touches your customers. If you want to talk through where AI actually fits your business — including the parts where the honest answer is "not yet" — book a 30-minute fit call and we'll find your best first move together.

The questions this series is chasing

This is the first post in Grown, Not Built — a series on how AI actually works, written for business owners deciding what to trust. Each one picks up roughly where this one ends: at something the people who built these systems still can't fully explain.

• •What is an AI actually "thinking" when it answers you?
• •Would an AI lie — or blackmail — to survive?
• •If it's grown, does that mean it's alive?

None of them have tidy answers yet. Not ours, and not the builders'. That's the thread this series pulls on.