Chapter Content
Okay, so, um, in the last chapter, we talked about the gloves, the snowman, and the Rubik's Cube, right? And how they're all, like, problems with different levels of difficulty. But, like, at the end of the day, they kind of, sort of, need the same general approach to solve 'em. You need a high-level strategy, paired with, you know, little tricks and heuristics to, like, navigate all the possibilities efficiently. The harder the problem, well, the more you gotta, you know, search around to get through the complexity of the solution space.
So, remember, a solution is, like, the physical thing, the actual thing, that makes the internal process work. Gloves are the solution to cold hands because they’re physical, right? They're made of stuff that's connected in a way that keeps your hands warm. Human Rubik's Cube solutions are basically just sets of rules written down. You know, turn this, then that.
But the deep learning solution to a Rubik's Cube, that's, like, a whole different ballgame. It's not something you can just write down. The human way is understandable, right? You can see the steps. But the deep learning way? It's basically a black box. You can peel back layers and look at the model, but you're not really gonna figure out exactly what it's *doing* in a way that makes sense. It's opaque.
It's weird, right? Our brains are super complex, the most complex thing we know, and we come up with a solution that you can understand. Deep learning's complex too, but its solution is, like, totally unreadable. You'd think that we'd come up with similar solutions, but, you know, not really.
Actually, deep learning *does* create solutions that are kinda like ours, but only if the problem is, like, *really* hard. And the Rubik's Cube isn't *that* hard, you know? It's a puzzle with a fixed set of possibilities, a manageable number of moves. It’s only big compared to simpler problems.
Because it's not, like, a fundamentally hard problem, it can have either a simple solution or a complex one. Simple is the human, rules-based version. Complex is the deep learning, unreadable version. So why does deep learning choose complex over simple?
Well, deep learning uses this, like, trial-and-error kind of thing to build its models. And that process, that, you know, *variation, iteration, and selection*, that's what makes things complex. It's gonna give you a solution that's about as complex as the problem itself. The Rubik's Cube has some built-in complexity, so deep learning's solution taps into that.
That's why the deep learning solution is, like, the best you can get in terms of performance, even if you don't understand it. Deep learning can solve the cube in under a second! Faster than the best human speedcubers. It's not about having, like, robot hands or being a computer that never gets distracted. It's about how the trial-and-error approach is just a better way of computing than trying to analyze everything. And, like, even human solvers do this! The faster they are, the more they use intuition and pattern recognition, you know?
Humans want a nice set of rules, like wanting to remember exactly what you did when you accidentally baked the best apple pie. But we only do this because, like, we're expected to give explanations and teach people how to do things step-by-step. There's no reason to think that an explainable solution is the best one. Actually, we should expect the opposite, especially when we're dealing with the real world, not just games.
Deep learning takes advantage of all the complexity that's already in the Rubik's Cube problem. It finds complex patterns and relationships that you can't express as simple rules. Deep learning finds strategies that use complexity in ways that you can't understand after the fact. That's what our brains do too when we're facing real-world problems. You can throw a spear, hit someone perfectly, and not be able to explain exactly how you did it. If you try to break spear-throwing down into steps, you're gonna end up with a much worse way of throwing spears.
Deep learning's, like, encoding knowledge in its model. Implicit knowledge is always gonna be better in complex situations because it’s, you know, hidden. Things that are hidden kind of come out on their own. They're not put together. That's so fundamental that explainability should really be a sign that you're looking at something too simplistically.
Solving hard problems needs complex solutions. What we create in this age of complexity can't be a simple, complicated set of, like, well-thought-out pieces. Whatever internal process our creations use, we should expect that it'll be unreadable. Explanations of how complex things work are, like, fairy tales, and they'll only limit our innovation.
So, you know, people think that these ways of solving hard problems are just approximations. Like, you're settling for something that's not as good or exact as a more direct approach. Deep learning uses heuristics and optimization to find its solutions. And people say that these methods are, like, "softer" than the rules-based approaches of traditional computing. So, they assume that these "softer" approaches are less good, and that what you find this way is less than the best.
With humans, heuristics rely on our personal opinions and intuitions, which means, like, bias and "error." Different people use heuristics differently, so you'll get different results even with the same situation. Heuristics are about speed, not accuracy. They want a solution that's good enough, not the *best* solution. And, with machines, it's basically the same thing. Using heuristics feels like skipping over the details to get to the answer faster.
Seeing "softer" approaches as approximations comes from a problem with how we think about science and engineering. You'd only see heuristics as approximations if you thought the *best* way to solve a problem was to do it deterministically. Like, you have to understand all the little pieces and how they work together. That's the only way to be, like, rigorous.
This is about how you *find* a solution and how you *use* it. Finding a solution with rules-based reasoning seems better than jumping around and making decisions based on high-level patterns. And using a rules-based system feels more sensible than some model whose inner workings are unreadable.
But complexity doesn't work that way. It's not just a souped-up version of what we see in simple systems. Complexity is unreadable because the way complex systems process matter and information is totally different. The statistical noise you see in complex systems isn't just a blurry version of things. It's what makes them work! Trying to zoom in and find little pieces that work in a deterministic way is looking for something that's not there. Yeah, there are small pieces. And yeah, they interact. But not in a simple, cause-and-effect way. The interactions in complex systems create things that don't exist at smaller scales.
Nature's solutions aren't approximations. They're exactly what solves the problem. Nature isn't just the little pieces we use to define knowledge. You can look at a molecule and define its atoms, but you can't explain the molecule's behavior by only looking at the atoms. You know? You can look at the cell and see its organelles, but you can't explain how the cell works by only looking at the organelles.
Nature's ways of computing emerge on their own, across all the scales of the solution. Humans struggle to see processing as anything other than a series of mechanical steps that change inputs into outputs. But processing just means that matter and/or information are being changed, not that it has to happen through mechanical means.
So, earlier we talked about the difference between deterministic and non-deterministic processes. Deterministic processes are predictable. The same inputs always give you the same outputs. Non-deterministic processes are not predictable. The same inputs can give you different results. Deterministic processes don't allow randomness, while non-deterministic processes use it.
But things aren't that simple with nature's solutions. They have both determinism and non-determinism. Things are both reproducible and non-reproducible at the same time. Think about a beaver. Obviously, there's a lot of reproducibility, or determinism, to it. We always know a beaver when we see one. It has a specific look and set of behaviors. A beaver doesn't make completely different outputs with the same inputs.
But the beaver is also very non-deterministic. It has to be, since it has to deal with challenges that you can't solve with strict determinism. How could you convert inputs like predators, food, territory, temperature, drought, and parasites using some set of rules that always does the same thing? Predators can come from anywhere, temperature changes are unpredictable, and parasites can infect you any time. A deterministic system could never handle all the possibilities.
But the beaver survives. Even though there are all these random and unpredictable inputs, the beaver reliably converts them into the same few outputs every time. It can only do that with abstraction. The structures in the mountains, the riverbeds, and the beaver's own body work because they're physical abstractions that condense matter, energy, and information into a few reliable outputs. Physical abstractions are key to how nature computes because it's the only way you can get this, like, flexible determinism.
Humans do facial recognition, not by analyzing every detail, but by spotting high-level patterns that stay the same even when things change. If someone moves their head, if there are shadows on their face, if they're soaked in rain, we can still recognize them. That's abstraction at work. You can't describe the essence of a face with analysis because there are no exact rules for identifying faces. You can only solve facial recognition if you focus on something much more abstract.
To notice high-level patterns, you need a lot of information compression. Things like the distance between their eyes, the width of their nose, have to be compressed into a few signals that the mind can use to identify someone. That's exactly what abstraction is. Compressing information into a few things you can act on.
All problems in nature are hard. They're not solved by slow analysis. They're solved quickly because they rely on abstractions in the form of heuristics and pattern recognition. A problem that takes a long time to solve is actually a simple one. That sounds wrong, but it's true in computational terms. Only simple problems can be broken down and analyzed. Truly hard problems can't be solved that way. They require high-level ways of solving. You can only manage hard problems by abstracting away the details into higher-order things. And we're not talking about finding a solution for the first time here. We're talking about how you implement the solution you've already found, like how the human brain identifies faces. Finding the solution to a hard problem takes a long time, but implementing the found solution does not.
We also have human-made examples of flexible determinism. Deep learning models produce solutions that are reliable enough to be considered deterministic, but also flexible enough to solve hard problems. The kind of determinism you see in deep learning has nothing in common with the determinism of traditional engineering. Traditional engineering makes solutions with predictable outputs. And while errors and environmental factors can throw things off, the inner workings of simple systems are built to be fully deterministic.
But deep learning is flexible even in its static form. The settings that convert inputs to outputs are already set, but they provide extreme flexibility by condensing a lot of information into a lot less information. This is the kind of determinism you see in nature. It's not achieved by the behavior of rules.
You can still have rules at a lower level. Ants and termites have rules about how they interact. But there's no direct line from those interactions to the behavior that emerges. The local interaction rules just set the stage for the interdependence that complex systems need. They're not what leads to the outputs that solve the higher-level problem. It's not the rules that are converting inputs to outputs. That's happening at the higher level of physical abstraction.
That's why trying to explain how deep learning models work is, like, totally irrational. The way deep learning computes can't be explained in the causal sense. But a lot of scientists and engineers don't get this. There are tons of people trying to give a "scientific" explanation of what deep learning models are doing. They're stuck in our old way of thinking. If someone ever gives you a description of how deep learning works internally, it's either false or deep learning isn't actually complex. And all signs point to it being false.
Deep learning systems will never be built in the deterministic sense because whatever level of determinism they have wasn't put there on purpose. There's no path from input to output in deep learning, because what a deep learning model creates is an emergent way of transforming information that rules can't.
Of course, it's not just deep learning. Deep learning is the human-made version of what we see in nature. It's not as good as nature, but it shows us hints of the complexity you need to achieve the flexible determinism of nature's solutions. The human brain is, of course, the best example. It's the most complex thing we know, with all its neurons and connections and emergent properties. It has extreme flexible determinism, which lets us navigate our complex environments really well.
A lot of scientists and engineers act like the uncertainty in complex systems comes from things like external issues, noise, or randomness. That's true for deterministic systems, but not for complex ones. Complexity isn't just a messy version of determinism. It works in a fundamentally different way. There's a big change when a system goes from simple to complex.
We can't build solutions the way we always have. The outputs we need have to come from emergence, and only complex objects can do emergence. We can't make specific choices about pieces and connections that produce known outcomes. Instead, we have to step outside the system, like nature does.
Building things in the age of complexity means using processes that are similar to natural selection. Variation, iteration, and selection are the main things. The internal setups of our solutions have to emerge on their own.
To build technology is to solve problems with physical things. It has always depended on design, but the number of pieces and interactions we need now means we can't really design anymore. We can't set the settings within our creations directly. We don't have access to the information we'd need to set those settings, and even if we did, that's not how nature works. Approaches based on design are based on reductionism, and you can't use reductionist thinking to make things that work in complex situations.
But building things still needs knowledge about the principles, materials, and processes involved in construction. Things behave differently under different conditions, and you need that knowledge to make sensible decisions. Like with traditional engineering, building complex things that work still needs an understanding of how structural integrity, safety, functionality, and efficiency play out in the environments we put things in.
The difference now is that that knowledge can't be based on the internal details of our creations. That's totally different from how science and engineering operate today. To build an office tower with structural integrity means using knowledge of how loads are distributed within the building. The interactions between beams, columns, and foundations are used to design and build a working structure.
Making objects that meet safety standards means understanding how various things interact and how they might fail. To foresee potential hazards and reduce risks, you have to make conscious decisions about which pieces to use and how to connect them. Think about fire-rated walls, sprinkler systems, and evacuation routes. We've made our creations efficient by optimizing their internal workings. To find areas for improvement, we've had to think about the connections between pieces and use that knowledge to streamline processes, minimize waste, and reduce costs. Functionality has always been possible by understanding the internal mechanisms and processes by which outputs are produced.
All of these examples are based on internal knowledge of how things work. As we move into the age of complexity, we can't just throw away all that knowledge about the principles, materials, and processes. The difference is that we have to reframe that knowledge in terms of external properties. Just as natural selection achieves true sophistication without any internal knowledge, we have to step outside the systems we create and set our standards there.
To do this, we have to understand what emergence is. Not with some reductionist story that tries to connect small and large scales via causal connections, but by using knowledge of the universal properties of information, computation, and evolution.
We can only understand how nature computes by stepping back and looking at what nature is doing to achieve its emergent structures and behaviors. You won't find this in the annals of science. Not because people haven't tried, but because their attempts still smack of the old reductionism. We need an objective look at what nature is doing with information, computing, and evolution to create its solutions. Only by stepping outside our current way of thinking and seeing nature for what it is can we shed our outdated reductionist lens and bring forth truths about how nature works.