Chapter Content
Okay, so like, we've seen how humans actually make progress, right? It's not about just being smarter, or, you know, working harder, or even because of like, so-called geniuses. It's about designing, like, higher and higher levels of abstraction, which is kind of a mouthful, I guess. But basically, each generation solves the same problems as before, only way faster because they're starting from a more advanced point.
But, here's the thing, design depends on determinism, and that, like, totally falls apart when you're dealing with *really* hard problems. You know, the kind we *need* to solve now. The number of factors involved, all the interactions... it's just way too much for a rules-based system to handle.
So, we're left with this *huge* question, which is like, how do we keep making progress if we can't even design our solutions anymore? How do we even think about these things when there's no clear cause and effect, no set of steps to follow, and we don't even fully understand the systems we're creating? How do we even give the next generation a leg up if we can't deliberately build interfaces to handle all this detail?
Basically, we've, like, hit a wall with progress by abstraction. And, you know, I've said before that abstraction is like, a universal thing – all systems need it to progress. So, the only way forward is to find a new way to abstract, a way that *doesn't* depend on design. It *has* to exist because nature does it all the time.
Nature is all about complexity, right? So, that means complexity is essential, and it's what we need to be building. It's not just about making things more complicated, adding more pieces, it's about achieving *genuine* complexity.
This means moving past simple machines built with simplistic reasoning, and, like, embracing a process that might seem, well, kinda crazy. The newness we need is actually ancient. It’s the way nature works, without thinking or reasoning, that binds things together and creates these amazing abstractions that solve hard problems. And giving up on this whole, supposed enlightened thinking, this reductionist past of ours, it feels like the only way to move forward. It's kind of a hard pill to swallow, though, right?
But look, it can't just be, like, randomly mashing things together and hoping for the best. It's gotta be within the realm of possible solutions, and that's a tricky balance. It seems like, well, it goes against how we're used to solving problems.
But, not everything humans have built is totally deterministic. There are things out there like societies, markets, certain technologies, that work because of things that weren't deliberately designed. So, we *do* have the capacity to create things the way nature does. Which kinda hints at how we might keep progressing without design.
Like, think about things like cities, financial systems, electrical grids, the internet, even AI systems. What puts these things into that realm of true complexity is that their outputs aren't *just* from deliberate engineering. They show these early signs of emergence, which is when structures and behaviors pop up that don't even exist in the individual parts.
Take electrical grids. They have voltage fluctuations, power oscillations, all sorts of weird stuff that wasn't actually engineered in. But that's not just gremlins in the machine, it's actually how they work! Their stability comes from being self-organizing. They can even adapt to failures, disasters, even cyberattacks, by rerouting power. Sure, having backup pathways is deliberate, but the self-healing, the way they isolate problems, that comes from the *interaction* of all those parts, not direct human control.
Cities aren't just planned and designed, you know? They're also self-organizing. Where businesses set up shop depends on tons of things, like accessibility, who the customers are, who their competitors are. Commercial districts kind of pop up in unexpected ways. Street layouts, building density, they come from a mix of geography and individual decisions, not just city planners. Cities are these hubs of activity, where everything's connected, giving them their own unique feel. And as people interact, you get communities, public transport that grows out of unplanned traffic patterns and unforeseen needs.
Financial systems are the same. It's all these different companies, tools, people in the market, all interacting. What investors, traders, and organizations do together creates market prices, trading, and volatility that no one sees coming. Supply, demand, how people feel, the way information spreads... it all mixes together. What prices end up being isn't so determined; it comes from all sorts of opinions, beliefs, and strategies. Rules that keep things stable and fair come from different needs and goals coming together. Financial systems can usually survive shocks, disasters, and pressure, and no one ever put those abilities in them when they built them!
The internet also gives people ways to share ideas and make decisions together. Because it connects so many different devices, it's scalable, resilient, and adaptable. Online groups and social networks just pop up, as people chat, work together, and take action. Letting people create and share what they want has led to things going viral, memes, trends, and even movements!
Now, this can backfire, of course. Electrical grids can turn a small problem into a major blackout. Markets can crash and create feedback loops that destabilize things. It’s just a cost of doing business! What lets complex systems work can also make things that aren’t helpful to us. The thing is, no one plans for these, and none of them would work if they didn't have emergence.
But, like, probably the most complex thing humans have made are AI systems, especially the models that drive them. They use machine learning, like deep learning, to turn data into smart outputs, like natural language or facial recognition. But *how* that happens is kind of a mystery, at least, not in the way scientists and engineers usually explain things. No one really understands how AI turns inputs into outputs. That’s because AI isn’t programmed like regular software. The scaffolding is rules-based, but the inner workings are more like those grids, cities, and markets.
To get how deep learning works, we have to change how we think about “how things work.” We can’t just look at a set of steps that show information transforming. Instead, engineers can only program the outer framework and trust that it will eventually get to what's needed.
Think about fitting a line to data. It's a common way to find trends. If you plot temperature against ice cream sales, you'll probably get a line that shows higher temperatures mean more sales.
Now you have something that can predict sales. You can look at tomorrow's temperature and guess how much ice cream you'll sell. You can even put the line in a math equation, make a function. A function takes inputs and spits out outputs. Plug in the temperature and boom, there’s your predicted ice cream sales.
It's kind of like what our minds do when we learn about the world. We use our senses to collect data, and we use that to build models about how things work. When kids learn to talk, they build models about grammar, vocab, sentence structure. The "function" they learn takes inputs (sounds) and spits out outputs (meaning).
Learning, even in the abstract, is like learning functions. When we meet people, we pick up on social dynamics. We use our expectations of behavior and social cues to navigate situations. Our mental models let us predict how others will react. When we drive, we use models of the road, traffic, and other drivers. We anticipate hazards and decide on speed, direction, timing. None of that would be possible without internal functions that turn inputs into outputs. To learn is to "fit lines" to sensory data to create models of the world around us.
Building a model needs parameters, because that's what gives it its form. Parameters are like knobs you turn to adjust the line to fit the data. A common method is to use linear regression to fit a linear equation to observed data. Whenever you see a line striking through a bunch of data points on a plot there is a good chance it is linear regression at play. You can adjust the slope of the line to make it more or less steep, or move it up and down. Finding the right slope and intercept is how you get the best fit to the data. In other words, finding the best positions of our knobs is how one attempts to model the relationship between input (independent variable) and output (dependent variable) using data.
Linear regression is essentially a "plug and play" approach, whereby the data collected in an experiment are plugged into a closed form expression to produce the slope and intercept needed to fit the line. This means that parameters are not so much found as they are calculated directly.
But here’s the thing, linear regression isn't really learning. It's just calculating parameters based on a predefined mathematical structure. If we *really* want to learn the knobs that define how a function converts inputs to outputs, we wouldn't make assumptions about what a function is supposed to look like. We would do what true learning requires: trial-and-error. We'd start by guessing the parameter values, see how wrong we are, adjust the values, and try again. We'd keep guessing, assessing, and adjusting until we find a good fit.
This is what machine learning, the computing approach that powers today’s AI systems, does. Instead of forcing data into a defined structure and calculating parameters directly, it uses tons of data and keeps guessing until it converges. In deep learning, a model is fit to data, but instead of a straight line, it's something high-dimensional that we can’t even picture. The function learned by AI isn’t a simple mapping between inputs and outputs, it’s a monstrous, complex expression with billions of parameters.
There are no easily explainable expressions in AI. We can’t peel back its layers and see a causal chain that turns inputs into outputs. It’s not plug-and-play math, it’s iterative optimization and decision-making, used to learn like people learn.
Deep learning is a whole different way to build software. Because AI has no visible function, we can’t know what’s inside. We can’t design the guts of an AI to produce the outputs we need. All we can do is set up a high-level process that tries a billion knobs until it gives the answers we expect.
Today's AI is created by programming artificial neurons, interconnected units of code that mimic neurons in the human brain. Our nervous system processes and transmits information using these neurons and electrical and chemical signals. Each neuron gets signals from other neurons through structures called dendrites. The neurons integrate those signals and, if the input goes over a certain point, it generates an action potential. Action potentials move along axons, which are long projections of the neuron that carry electrical impulses away from it. Axons are the main transmission lines of the nervous system.
In other words, the human brain seems to process information by using a massive network of interconnected units that each transmit electrical signals depending on their interaction with neighbors. Deep learning mimics that, using artificial neurons and information sent between them as numerical values; values that represent the activation levels and strengths of connections.
The parameters in deep learning, called weights and biases, number in the billions and relate to how strong the connection is between neurons and how flexible each neuron is. While the parameters in linear regression mean something specific because of its mathematical form, the parameters in a deep learning model only make sense when considering the network as a whole.
So how are these parameters set without a mathematical expression? How can we find their values when there are billions of them? Deep learning does it by passing huge amounts of data through the system, adjusting the parameter values (which start out as random values) until the whole system produces the outputs we need. This is done using optimization algorithms where the model tries to minimize the difference between the output it predicts and the actual output needed. For instance, in facial recognition, the actual output is the identity of the face (e.g. Bob), and the predicted output is the best guess at any time (e.g. Bill? Susan?). This means the way deep learning approaches problems is not through deliberate calculation but rather by closing the gap between predicted and actual labels until they mostly agree. While not all forms of deep learning use labels, the fundamental approach is the same; closing some gap between what the model initially thinks and what is.
It’s a game of guessing and iterating, not calculation. This more holistic way to build machines is totally different from how software and statistics have worked. In deep learning we admit uncertainty by stepping outside the system of interest and only employing a higher-level process that converges on its own. This is a critical distinction, as it shows us an important truth about how complex systems are created. We can’t fashion complexity by piecing together individual things and connecting it up like an internal combustion engine. While neural network architectures are designed in terms of the number and arrangement of artificial neurons, the process by which the parameters inside the network are found and set are out of our hands.
Deep learning distinguishes between internal and external parameters. The weights and biases are the internal parameters that directly affect the model's use and transformation of information. There are also hyperparameters, which are external to the model architecture itself. These can be thought of as settings or configurations that govern the training process. They include things like the learning rate, batch size, number of epochs, and various network architecture choices such as the number of layers and neurons per layer.
It is the hyperparameters that are configured by the engineer, not the internal model parameters, and yet it is the internal model parameters that must be set into some specific configuration for the model to work. This shows just how different AI engineering is from anything humans have built prior. In almost all other cases, humans look to reach into the guts of systems and make deliberate choices about how energy and information will move through the system.
People have a hard time understanding the difference between deterministic and nondeterministic processes. The individual artificial neurons are deterministic pieces of code, since the same inputs and weights will always produce the same output. This makes them inline with traditional, rules-based computer programming. However, when these neurons work together in aggregate, the behavior of the neural network is hardly deterministic, especially during the training phase. This is the critical transition that happens in truly complex objects. There is a profound disconnect between the pieces something is made of, and the properties that define its structure and behavior.
As information passes through multiple layers of artificial neurons, each applying its own piece of nonlinearity, the network learns highly complex and nonlinear mappings from inputs to outputs. Whatever deterministic (rules-based) code is used on the small scale brings about something very different at the larger scale. Such complexity unleashes the signature capacity of complex objects to mirror the natural world in a far deeper and more realistic fashion than anything humans could design on their own.
This is how AI systems show signs of true complexity, and why the approach used to build AI is inline with how nature builds. The buildup of many small deterministic nonlinearities into some aggregate nondeterminism is what we see in nature’s wet, slimy and dynamic objects. This is not to say today’s AI is on par with nature’s solutions, only that it mirrors the structure and behavior of natural complexity, and speaks to what happens when you embrace a from-the-outside approach to building things. This is the only way categorically hard problems can be solved. One must step outside the internal workings of a system and allow it to converge on its own, permitting the internals to settle naturally.
This is why deep learning is thought of as a kind of alchemy. This makes many uncomfortable, especially more traditionally minded engineers. There is an ongoing effort to try to explain AI mathematically, which smacks of the reductionism and determinism that works directly against complexity. I will elaborate on this issue later.
Deep learning models are objects that verge on true complexity, because their functionality relies on properties that were never put into the system by us. In some sense, things like cities, financial markets and AI were not created at all, rather they precipitated out from the countless interactions of smaller things that look nothing like the bigger things they led to.
More important than whether AI systems are truly complex is the fact that AI engineering is our best example of humans building under a completely different paradigm. Deep learning represents the first deliberate attempt by humans to not build the way humans always have. While the emergence of electrical grids, cities, markets and the internet were realized later, deep learning embraced a fundamentally different paradigm from the onset. Deep learning is possible because it does not attempt to instruct a computer how to carry out its tasks.
Whether or not today’s approach to AI achieves true artificial general intelligence (AGI) is not the point. Deep learning is a sign of things to come. Deep learning represents the kind of engineering that redefines how humans build things. We know that we cannot solve our most outstanding challenges using the deterministic engineering of our ancestors.
This is not just a shift in how we build things. It gets to the very heart of how our world currently defines knowledge, skill and the ability of individuals to contribute to the economy. Our world runs on the notion of design, because design is all about reaching into the guts of systems and using what we see to take the next step. Design maps the efforts of builders onto preexisting structures. This is not how nature builds.
So, we need to, like, totally rethink how we approach things. To embrace the nondeterminism needed to create in the age of complexity we need an entirely different approach. This new approach must be rooted in a position that is diametrically opposed to how we currently expect to build things.
There is a fundamental directionality to how complexity works, and it is opposite to what we are told by the academic narrative. It is not foundations that lead to good things, it is intermixing, chaos and uncertainty that bring about the structures we end up codifying in our textbooks. In simpler terms, the groundwork, the structuring, the design we are told to put in place from the onset of a project directly works against what is needed for the right structures to emerge.
To do that we need to, like, demystify emergence. We need a clear description of what it is, that fits with what we know about information, computation, and evolution.
And, honestly, I haven't really found a good explanation of it out there. They all try to explain it within the current scientific framework, which, like, inevitably leads to the kind of reductionism that goes against complexity.
Complexity needs a different kind of explanation. Not an explanation of causality but of properties. The kind of explanation that doesn’t reach into the guts of systems, wrapping outdated notions of averages and renormalization onto its pieces. Such explanations erroneously suggest that what we observe at the high-level is merely some smeared-out fuzzy version of what we see at the lower level. Such attempts are steeped in outdated notions of root cause and pathways, which unsuccessfully try to coerce complexity into the linearity and determinism of simple systems.
Instead, we can see complexity as the inevitable outgrowth of physical systems that solve their hard problems, by settling into organized configurations on their own. We need not appeal to theories that call together matter by strict cause-and-effect. Nor must we conjure up some ill-defined argument that attempts to stuff our gaps in knowledge with meaningless abstract descriptions. Not having a causal explanation for something does not mean it cannot be explained. Mechanisms can be revealed that adhere to the properties and constraints of natural systems, without concocting fairy tale stories about how pieces bump into pieces.
The first step to getting over this hurdle is pretty obvious. Humans have always looked to nature for inspiration. Most things we build have some counterpart found in the natural systems that surround us. It is nature that lets us know what is possible. But using nature as our muse is no longer enough. Nature tells us far more than just what is possible. If we step back, shed our reliance on reductionism and determinism, and accept complexity for what it is, we can also learn how nature produces her complex solutions. We can learn to build as nature builds.