Chapter Content
Okay, so, uh, this chapter is all about, like, going meta. And, you know, at first, it sounds kind of abstract, right? But it's actually a pretty important idea.
So, basically, something is meta when it talks about itself. The easiest example is, um, metadata. It's data about data. It's, like, describing the characteristics of the data itself, you know, the format, the content, all that stuff. Or, like, meta-analysis, where you're combining results from *multiple* studies, instead of just one, to get, like, a stronger conclusion.
The whole point of going meta is to, like, learn something that's deeper and more universally true than just any single instance. You know, each study in a meta-analysis has its own biases, right? So it only gives you, like, a limited view. Going meta is about finding a deeper truth, like, a latent truth, at the heart of the system. Something that's, you know, hidden, unlabeled. It's definitely there, but any single part is just one tiny piece of the truth. Only by, like, bringing a bunch of pieces together can you get close to the real thing.
And, like, this is actually how abstractions are created, right? Abstractions are made by, you know, spotting similarities between things that might seem different at first. Then you make a category out of those similarities. And you can't do that from inside the system. Like, you gotta step *outside* to see those similarities, you know, to put the system into a larger category. That's why you have to go meta to create abstractions. It's a self-referential process where you look at something from above and, like, glue different things together.
It's not just for mental abstractions, either. Physical abstractions need that meta-level view, too. Like, think about a stick shift in a car. It's, like, a higher-level interface that coordinates all those inner parts. You couldn't have created that unless you could see how the different internal pieces work together towards the same goal. You can't see that holistic structure from *inside* the engine. Gotta look from the outside.
Now, meta isn't just another level of abstraction, okay? Abstractions are high-level constructs that, you know, include all the little pieces. But, like, *creating* an abstraction is a *process* that happens outside the system. Abstraction is *inside* the system, but going meta, that's outside. If you bundle pieces together and, you know, put an interface on it, that new abstraction could only have been formed by stepping outside, totally.
Okay, so, all this makes sense for, like, human-made things, but what about the abstractions that nature creates? The argument here is that progress by abstraction is absolute, meaning, any system that evolves to solve hard problems, it *has* to make higher levels of physical abstraction. And that nature does this through emergence. Emergent structures are levels of physical abstraction, which condense information to solve really hard problems. But, how's nature going meta, you know?
Think about deep learning models, right? They work because something just sort of *happens* inside these models that converts inputs to outputs. It's a black box, so we can't totally understand it. But we do know there are different levels of abstraction that, like, help with that conversion.
Like, in facial recognition, the AI uses increasing levels of abstraction to pull out the important features from the image. First, it gets the low-level stuff, like edges, corners, textures. Then, it combines those to find more complex patterns, like eyes, nose, mouth. It keeps going, combining the previous level into, like, expressions and poses, and eventually it captures the essence of a face.
And it all works thanks to the self-referencing nature of deep learning. Self-referencing in AI, it comes from how the system updates its own internal settings based on previous tries. So, the model makes a guess, and, like, it's got some error, right? It uses that error to make the *next* guess better. It's, like, a feedback loop. And to engineer a good model, it's not about knowing exactly how it works inside, it's about putting in a good framework for self-referencing.
That's why deep learning is a version of going meta. It uses feedback to look at itself and see if things are going well. It's different than setting parameters or writing code. The rules that end up in the solution, they came about automatically, not by planning. This is why a self-referencing approach is fundamentally different. It's about admitting you don't know everything and letting the variation, iteration, and selection help you find the right way. It's the opposite of design, really.
Because the internal values in the model are found automatically, the human engineering only happens *outside* the model. Like, we choose the hyperparameters to set up the training phase. But that's another problem altogether. We can't know how to deliberately set the *best* hyperparameters. To find them, we also gotta go meta, trying different values until something good shows up. We experiment with different learning rates, batch sizes, epochs, network architectures, until we find the right mix. When we go meta, we're learning *how to learn*.
The AI version of going meta uses stuff like hyperparameter optimization, meta-learning, and neural architecture search. Don't worry about the differences, the important part is that they're all about stepping outside the problem space and working externally to find what works, ultimately.
Just like meta-analysis finds a deeper truth, no single study could show, meta-learning in AI tries to find something no single model can have: a latent structure that converts input to outputs really well. Again, it's more about learning *how* to learn, not learning just one thing.
So, for deep learning systems to work well, they gotta have abstractions in the model, and those abstractions need a way to step outside the system. And it's not just for deep learning. It's true for any complex solution. Going meta leads to the abstractions that turn inputs to outputs in a, you know, flexibly deterministic way. You can't solve hard problems without it.
Alright, so, problems have structure, right? And this structure comes from the levels of abstraction that define it. The biggest problem in nature is survival, of course. But that's just the top level. Below that is food, water, shelter. Then below *that* are the activities that get you those things. Hunting, foraging, digging, etc. Then below *that* are the challenges that those activities create. It's all levels of abstraction, with lower levels fitting into the higher-level category of the challenge.
Because solutions in nature are matter doing the work of solving problems, nature's solutions are reflections of problem structure. It's not just an intuitive way to see things, but in the specific patterns that form. Fractals, spirals, waves, hexagonal packing, dendritic branching, you name it. These are all reflections of problem structure.
Take the nested fractal of a Romanesco broccoli, or the spiral seeds in a sunflower head, right? You're seeing how that organism's set of challenges are arranged by abstraction. The environment doesn't just ask the sunflower to survive, it asks that deeper levels of challenge get resolved so the ultimate survival goal can happen.
The things we see in nature, they come from the overlap between matter and problem structure. It's true for life and for non-living things. Salts, metals, rocks, rivers, mountains, they're all configurations that are built to address the structures of the problems they solve. So nature's solutions aren't that different from the environment, they're part of the same holistic overlap between problem and solution.
And this way of defining problems isn't just something we project onto the world. Under complexity, that's not how it works. We're not talking about a causal set of steps that turn little pieces of matter into big patterns. We're talking about the inevitable problem-solving that nature's doing, in line with what we know about information, computation, and evolution. It's totally reasonable to see those patterns as reflections of the problem.
It all points to the connection between informational and physical abstraction. Problem structures give us an informational version of abstraction, the kind we use to define and navigate the world. The physical structures we see in nature give us the physical version of those abstractions. For nature to make rocks, rivers, mountains, and all those critters, it has to be manifesting physical levels of abstraction that overlap with the informational abstractions that make up their environments. It's pretty cool.
So, nature is a way more powerful computer than anything we can build. Our AI systems have some self-referencing and abstraction, but nature goes, like, *way* further. Even when we *try* to go meta with AI, we still can't set the internal parameters, so we just work from the outside by setting the values related to training. The learning rate, batch size, layers, neurons, and so on. We hope those values help get the best possible abstractions.
But, like, what should those values be? It's its own meta-problem. We're not as interested in the best parameters for a single model, we're interested in how to learn the best parameters *for any* model. Meta-learning, hyperparameter optimization, neural architecture search all try to solve that problem of finding the best external values.
One way to do it is to combine many different models into one better "meta-learner." The idea is that a single model is too narrow, but many models might learn something more general. But there are only so many models we can combine before it becomes too much. Each model needs training and validation. Then the meta-problem of combining those models also needs experiments and validation. That problem has its own set of parameters, too. And why stop there? We could combine different meta-learning frameworks, and on and on.
The possibilities, like, explode. It makes you wonder if going meta is even *feasible*. Even if we accept we can't design everything, is it really the only way to make building possible? Trial-and-error takes so much computing power. It seems impossible.
But design can't cut through that mess, either. Design has a faulty premise when it comes to complex things. Design can't be the answer. But it seems like going meta doesn't really solve the computational problem. Just because we step outside doesn't mean those lower-level problems just vanish. So, how can going meta be a way to *build* things?
Again, nature has the answer. Nature reaches *super high* levels of meta, to create whole ecosystems. Just look at our classification system. Organisms make species, species make genera, genera make families, and so on, all the way up to domains. These are all levels of physical abstraction that work together to solve problems. They're all made automatically, by nature's self-referencing and abstraction creation.
Nature can go meta *way* more effectively than humans can. The amount of computing power is unbelievable. Nature keeps stepping outside a level and finding the parameters that work to make the next level.
Of course, nature has basically infinite resources compared to us. Extreme distributed parallel processing, with billions of processors working at once. Each piece, like a cell or a neuron, only does simple computations, but together they solve crazy complex functions. And nature operates on much longer timescales. Evolution takes millions of years to do things through natural selection. Nature explores a truly enormous space, with massive variation, iteration, and selection, all at the same time.
There's also a huge difference in energy efficiency. Biological systems are way more efficient than anything we make. The human brain only uses, like, 20 watts, while supercomputers and deep learning systems need way more. And yet the brain does vision, motor control, and reasoning really well on just 20 watts. It's in its own category.
Biology is also integrated with the physical world. All the levels of abstraction communicate directly with their environments. The computations that turn inputs into outputs are happening as physical processes in the system, not as models of behavior. It's not surprising nature goes meta more effectively. It has parallelism, time, energy efficiency, and physical integration to step outside systems and create those nested abstractions.
So, how can we act like nature, without all the computational power? How can we make really complex objects that solve problems like nature does, without those resources?
One way is to, like, hack nature where she is. Synthetic biology, for example, engineers the genetic material of organisms to have the traits we want. It's been used for things like cleaning up pollution, making drugs, biofuels, and even changing bacteria to do simple logic, or act as actuators inside machines. Synthetic biology skips the need to engineer emergence, because we're using stuff that already evolved.
The real problem is using it the wrong way. Hacking existing solutions might have some narrow benefit, but it comes with all the design problems under complexity. It's guaranteed to have unexpected side effects that will probably reduce the solution's long-term effectiveness.
The real thing is that we have to engineer emergence ourselves, not just repurpose nature. Our creations have to make their *own* complexities, and arrive at *their own* emergent structures and behaviors, based on the environments we put them in. There are deep internal things that can't be seen, which let natural solutions work effectively. Those have to turn into physical abstractions built by automatic self-referencing.
To engineer emergence, we have to maximize the parallelism, energy efficiency, and physical integration we see in nature. We can't do that under design. Design forces us to make specific, detailed decisions that turn into unnatural versions of the stuff we need.
Think about how an AI researcher wants to design systems by deciding exactly how they should work and interact with the environment. They want to pick the data sources for training and evaluation. They want to pick the algorithms based on computational needs. They want to design the neural network architectures, deciding on layers, activation functions, connections, and optimization techniques, based on design principles and math.
But the truth is, the biggest improvements in deep learning haven't come from design. They've come from throwing more data and computing power at the problem. The specific architecture is way less important than people think. It's not that the structures don't matter, it's that they don't come from design, like research articles claim. The structures that end up working are mostly a side effect of more data and more computing power.
That's what you should expect when you're building complex things. AI research works for reasons that most AI researchers don't seem to realize. It's easy to think your design choices made the progress, but progress under complexity has way more to do with randomness and chance than design. It's a proven statement, in line with how complex systems evolve.
Design under complexity gets in the way of creating good solutions. Design choices rob complex systems of their natural details and dependencies. They get in the way of the internal communication between pieces that has to happen for things to work. Design also makes it harder to explore possibilities because it makes our solutions less flexible.
Of course, AI design isn't like traditional engineering. AI is our best example of stepping outside systems and letting things converge. But we're still too close to the solutions. Nature goes all the way to the domain under taxonomy, but we're still operating just outside the individual organism. If we just want a basic model that predicts narrowly defined things, that's okay. But to achieve something like general intelligence, we have to do better. The more powerful our solution, the more complexity we have to create.
To get the parallelism, efficiency, and integration we need, we have to operate at a level much more meta than the solution we're trying to make. If we want to build a brain, we won't get it by trying to architect a brain. We can't just think of human intelligence as something that happens inside a head. We're deeply connected social creatures. There's no intelligence without levels of aggregation that go beyond a single brain. Meta has to reach much higher than the organism to make the best organism. Design keeps us too close.
When we stay too close to what we're building, trial-and-error becomes impossible. It's because the things we're trying to mix and match are too defined. But when we focus on the surface, using the highest-level problem statement, the internals figure themselves out. That's how complexity works.
The harder the problem, the bigger and more complex the possibility space. We have to search that space. As the difficulty of the problem increases, the softer our problem-solving has to get. The less analytical we can be about searching.
We can think of heuristics and pattern recognition as tools that help us find things inside a massive space without searching too much. But that only works if we apply them to the highest-level signals a complex situation sends. Design forces us to define searching too much. It doesn't use what nature gives us. Design makes us try trial-and-error on low-level things that don't have the meaning we think they do. It's not for us to know how things interact, only that interactions will happen if more of the system is forced to survive stressors. Reaching higher in our meta efforts is letting nature work itself out.
So, how can we operate at the highest level of meta without running into limitations? 1) Keep problem statements as general as possible, work on the surface. 2) Create flexible internals, rather than designing specifics.
We keep problem-solving on the surface by only sticking to the highest-level target. An example in AI is building an entity that has realistic conversations. That's it. Today, we decompose that problem into purpose, target audience, scope, intent, context, ambiguity, tone, personality. But that's reductionist. We don't know those specifics, only that the solution has to solve the topmost problem definition. Decomposing the problem guarantees we'll engineer it poorly under complexity.
Second, we create flexible internals by not designing specifics. Think about activation functions in neural networks. They make the model non-linear. There are a lot of choices, each with advantages and drawbacks. Sigmoid, hyperbolic tangent, ReLU, leaky ReLU, and so on. Trying to mix and match those with all the other variables in a neural network is like moving one piece in a Rubik's cube, only to mess up everything else. There are too many choices to make trial-and-error possible.
But imagine we don't try to design the best activation function as a standalone thing. Instead, we create something less defined and more flexible. Like learnable activation functions, that change dynamically. The key is not to design the flexibility, but to make something that changes as part of a whole. The less defined the thing is, the better. Just like mitochondria lose their meaning outside a cell, activation functions have a role that can't be defined outside the network. It's part of a holistic solution, and its presence in the group is what matters. It's not for us to know what the function should look like. Just that its structure and function emerge naturally while we focus on the high-level targets.
To get the best activation functions, network architectures, data pipelines, loss functions, and optimization algorithms, we can't design them. We have to let them materialize by only focusing on the topmost level of the problem. We have to reach meta as high as possible by keeping the internals flexible.
It makes some people uncomfortable that progress is defined under a lack of precision. But that's how complexity works. More data and computing power thrown at flexible constructs is how the best possible structures will emerge. Nature does that all the time. That's what it'll mean to build in the age of complexity, if we do it right.
Okay, so, what's the thing we're actually after here, the meta structures?
No two snowflakes look the same. Actually, that's only true up close. Zoom in on a snowflake, and you'll see a unique pattern. Just like a fingerprint. But zoom out enough to see the whole flake, and it's familiar. All snowflakes can be called "snowflakes" because they share a pattern. But we can't define that pattern precisely. If we asked seven artists to draw the essence of a snowflake, they'd look similar, but not the same. We all understand what a snowflake looks like, but it's not a precise definition. It's a latent structure that we know intuitively.
The essence of something, like a snowflake, is what we're after when we learn. To know a snowflake isn't just to know one instance, but to see many instances and create an unlabeled abstraction in our mind that defines it. To learn is to understand the hidden template that many instances of something adhere to, but never show explicitly.
That's why there are no words to describe what the essence of something is. Learning piano, swinging a golf club, or developing theories in quantum mechanics is tapping into a hidden shared pattern among experiences. It's not something that can be broken into pieces and taught to others. That's why true skill feels more like a feeling than something you can describe. If you can explain it precisely, it's either not a real skill or you're not good at it.
It's also why experience is the only way to get true skill. Not just seeing the same thing, but experiencing many different instances of the same latent thing. Then the truths will become recognizable, because they're what persist when everything else changes. A piano player has to play different songs to understand what it means to play piano. A golfer has to take countless swings to get what makes a good swing.
True learning doesn't happen by going deep into one version of something, but by exposing yourself to different aspects. It's the unspoken, intuitive, and latent structure we're after. It's what lives in the mind, but has no name.
You can predict what a snowflake looks like, but you can't predict what the next one will look like. That's why details don't matter in nature, at least not the way we're usually taught. Details are just there to be in flux, in service to the next level of abstraction. There are always many ways to achieve the same thing. To look at the details is to see one instance of a collection that serves the next level up.
So, the only way to really describe a snowflake is to talk about it in the meta sense. Only the shared properties of all snowflakes are true knowledge of its structure and behavior. Any detailed account will be mired in reductionism and causal reasoning, and it won't have much to do with why a snowflake looks the way it does.
The real structure that defines the snowflake is its meta structure. A meta structure is the latent pattern that doesn't directly appear in nature. Meta structures are the hidden templates nature's solutions try to become, but never do. Anything we build in the age of complexity will have a meta structure, because the emergent structures we see in complex objects move towards latent patterns.
And knowing meta structures is true validation when building complex things. Meta structures don't care about the internal details. They don't mess with the natural convergence of matter and information that makes complex things work. That's why they're a good way to validate things under complexity.
Okay, so, all the way up, all the way down, group selection.
Like, AI engineering is steeped in design, right? Engineers want to design systems by deciding exactly how they should function. I argued against those efforts. The best structures are a side effect of working on the surface, without regard for the internals. We avoid unworkable trial-and-error, and we integrate into the physical realities we're trying to model.
The activation function has a role that can't be defined in reductionist terms. It's part of the holistic solution, so we're after the best group possible. Now, how does nature always create the best group, and how can we do the same?
Let's look at how humans form groups now. This isn't like the emergence of societies, but groups like sports teams, students, and employees. Groups that are picked for performance. We want skill diversity, so we hire for expertise and knowledge. We define roles and responsibilities, like leadership. We want certain personalities and work styles. We want cultural fit and shared values. We also think about experience and location.
It all seems to make sense, until you realize it's just design. We're assuming we know the roles that people should play in the group, and we're picking people based on that. Now, you should know that's not a good way to make groups. All we can know is that the best group is needed, not what specific roles will make the best pieces. Nature doesn't make those explicit choices about roles. Nature can select the best group by doing group selection.
A species exists because organisms work together to solve the species problem, which is to make groups of organisms that are reproductively isolated from each other. Maybe that helps with genetic compatibility and less reproduction error. Maybe it ensures different species occupy distinct ecological niches to avoid competing for resources. Maybe it improves diversification.
The next level, the genus, should bring more ecosystem functioning. Resource partitioning, pollination, seed dispersal, and new levels of biodiversity, stability, and habitat structuring all work at that level.
All problems have structure, and nature manifests what's needed to solve that structure. The levels of abstraction that sit beneath survival are attended to by groups. A group is a collection of pieces that solve a level inside the problem's structure.
So, the selection in natural selection has to be group selection. That's been controversial, because the idea goes against things like selfish-gene theory. Selfish-gene theory says that individual genes are the units of selection, and they act in their own self-interest to survive and reproduce. It's a gene-centered view of evolution, saying that genes want to replicate, and organisms are just the means to do it.
Group selection is different. It focuses on the evolution of traits that benefit the group, even if they're bad for an individual's fitness. Selection acts on groups, favoring traits that benefit the whole group.
The problem with Selfish Gene theory is that it doesn't match how nature computes. Saying that there's a single type of construct that's uniquely selected doesn't adhere to what we know about information, computation, and evolution. Nature goes meta at all levels, to create physical abstractions to compress information and compute answers to hard problems. If evolution only selected genes, the meta process wouldn't work.
Selfish Gene theory is just reductionism in biology. It's tempting to see genes as having the primary role in evolution, but really, a gene is just one piece of a larger group that nature selects.
At all levels of selection, the group matters. Atoms have their properties because the nucleus and electrons work together. A molecule's behavior in a reaction depends on how all the electrons work together. We can keep climbing up the ladder of complexity and see a larger dependence on group-level properties.
It's not about the individual, it can't be. Nature's problem-solving happens at all levels, and only the group can solve the next level up. Abstraction is about combining pieces into a unit, and that unit solves a new problem. It compresses information to map inputs down to what's needed. The unit of selection in natural selection has to be the group.
Think about a snowbank and a snowflake. Organisms solve different problems than species, and a snowflake solves a different problem than a snowbank. A snowflake provides surfaces for water molecules to adhere to and crystallize on. The six-sided structure maximizes the surface area, to capture water vapor. The structure makes them float through the air as they fall.
A snowbank is another level of complexity, formed by snowflakes. In a group, snowflakes solve the problem of insulation, by trapping air between crystals. That keeps the temperature stable, reducing heat loss. The white color (which individual snowflakes don't have) helps regulate the climate by reflecting sunlight.
Individual polar bear hairs solve a different problem than the group of hairs. Single hairs are hollow, which makes them transparent, but as a group, they offer insulation and create a white color. An individual hair connects to nerve endings, giving the bear touch sensitivity. Collectively, the hair solves camouflage, water repellency, and buoyancy.
Always selecting for the group is how nature goes meta. To be meta is to look at all parts of a group and find what they have in common. Nature is always hovering above the details and "noticing" how pieces work together to survive. Nature selects the best group because it selects for groups, not individuals.
By selecting for the group, nature is effectively making "analogies." It's finding connections between things, automatically, because only the shared pieces are the material configurations that survive. By seeing emergence as the side effect of nature selecting the group, we can understand how nature creates levels of physical abstraction, without design. It hints at how we can do it too.
The things we consider when designing a group aren't necessarily bad. Diversity, expertise, knowledge, leadership, personalities, and skills all make sense. But they have to emerge on their own, just like mitochondria emerge in cells. We can't select for those attributes, because we don't know how they'll relate to each other in the wild. The way we can select for the group is to just add variety. Not a type of variety, just variety. That's a meta property of systems that evolve effectively. They have a lot of variation, like nature. We can also aim for iteration. Not a type of iteration, just iteration. And as for selection, it's the group that has to be selected. We have to select from the top, letting the bottom-up flux of matter and information manifest what works.
Okay, so, we've taken the journey to demystify emergence. It's not an edge case phenomenon that's hard to explain, but the inevitable and automatic process that matches the truths of information, computation, and evolution.
Nature's patterns are levels of physical abstraction created by nature's meta mechanism. It bundles details into categories that compute answers to hard problems.
Our reductionist paradigm can't explain emergence, because it relies on pieces and causality. The inner knowledge that our current paradigm relies on is the opposite of what makes nature work. It's not a story of defined pieces working together in a deterministic way, it's a story of entropy, information compression, and group selection.
Humans have implemented a low-dimensional version of progress by abstraction. But as we solve harder problems, our historical approach to progress can't work anymore. We have to look to nature and her mechanisms to see how true sophistication is achieved.
That has big consequences. It gets to the heart of how we define knowledge, skill, and quality of life. It strikes at our notions of meritocracy and people's ability to contribute. We have to bring forth a new era of innovation that avoids the notion of design. We can't rely on structures or inner knowledge, because that's not how nature works.