The neocortex stands out as a stunning achievement of biological evolution. All mammals have this swath of tissue covering their brain, and the six layers of densely packed neurons within it handle the sophisticated computations and associations that produce cognitive prowess. Since no animals other than mammals have a neocortex, scientists have wondered how such a complex brain region evolved.
The brains of reptiles seemed to offer a clue. Not only are reptiles the closest living relatives of mammals, but their brains have a three-layered structure called a dorsal ventricular ridge, or DVR, with functional similarities to the neocortex. For more than 50 years, some evolutionary neuroscientists have argued that the neocortex and the DVR were both derived from a more primitive feature in an ancestor shared by mammals and reptiles.
Now, however, by analyzing molecular details invisible to the human eye, scientists have refuted that view. By looking at patterns of gene expression in individual brain cells, researchers at Columbia University showed that despite the anatomical similarities, the neocortex in mammals and the DVR in reptiles are unrelated. Instead, mammals seem to have evolved the neocortex as an entirely new brain region, one built without a trace of what came before it. The neocortex is composed of new types of neurons that seem to have no precedent in ancestral animals.
The paper describing this work, which was led by the evolutionary and developmental biologist Maria Antonietta Tosches, was published last September in Science.
This process of evolutionary innovation in the brain isn’t limited to the creation of new parts. Other work by Tosches and her colleagues in the same issue of Science showed that even seemingly ancient brain regions are continuing to evolve by getting rewired with new types of cells. The discovery that gene expression can reveal these kinds of important distinctions between neurons is also prompting researchers to rethink how they define some brain regions and to reassess whether some animals might have more complex brains than they thought.
Back in the 1960s, the influential neuroscientist Paul MacLean proposed an idea about brain evolution that was wrong but still had a lasting impact on the field. He suggested that the basal ganglia, a grouping of structures near the base of the brain, were a holdover from a “lizard brain” that evolved in reptiles and was responsible for survival instincts and behaviors. When early mammals evolved, they added a limbic system for the regulation of emotions above the basal ganglia. And when humans and other advanced mammals arose, according to MacLean, they added a neocortex. Like a “thinking cap,” it sat at the top of the stack and imparted higher cognition.
This “triune brain” model captivated the public imagination after Carl Sagan wrote about it in his 1977 Pulitzer Prize-winning book The Dragons of Eden. Evolutionary neuroscientists were less impressed. Studies soon debunked the model by showing conclusively that brain regions do not evolve neatly one on top of another. Instead, the brain evolves as a whole, with older parts undergoing modifications to adapt to the addition of new parts, explained Paul Cisek, a cognitive neuroscientist at the University of Montreal. “It’s not like upgrading your iPhone, where you load up a new app,” he said.
The best-supported explanation for the origin of new brain regions was that they evolved mostly by duplicating and modifying preexisting structures and neural circuits. To many evolutionary biologists, such as Harvey Karten of the University of California, San Diego, the similarities between the mammalian neocortex and the reptilian DVR suggested that they are, in evolutionary terms, homologous—that they both evolved from a structure passed down from an ancestor shared by mammals and reptiles.
But other researchers, including Luis Puelles of the University of Murcia in Spain, disagreed. In the development of mammals and reptiles, they saw signs that the neocortex and the DVR took shape through completely different processes. This hinted that the neocortex and DVR evolved independently. If so, their similarities had nothing to do with homology: They were probably coincidences dictated by the functions and constraints on the structures.
The debate over the origins of the neocortex and DVR stretched out over decades. Now, however, a recently developed technique is helping to break the stalemate. Single-cell RNA sequencing enables scientists to read out which genes are being transcribed in a single cell. From these gene expression profiles, evolutionary neuroscientists can identify a wealth of detailed differences between individual neurons. They can use those differences to determine how evolutionarily similar the neurons are.
“The advantage of looking at gene expression is that you’re profiling something that’s comparing apples to apples,” said Trygve Bakken, a molecular neuroscientist at the Allen Institute for Brain Science. “When you compare gene A in a lizard to gene A in a mammal, we know … that those are really the same thing because they have a shared evolutionary origin.”
The technique is ushering in a new era for evolutionary neuroscience. “It’s shown [us] new cell populations that we just didn’t know existed,” said Courtney Babbitt, an expert in evolutionary genomics at the University of Massachusetts, Amherst. “It’s hard to research something you don’t know exists.”
In 2015, breakthroughs in single-cell RNA sequencing increased the number of cells it could be used for in a sample by an order of magnitude. Tosches, who was then just beginning her postdoc in the lab of Gilles Laurent of the Max Planck Institute for Brain Research in Germany, was excited to use the technique to study the origins of the neocortex. “We said, ‘OK, let’s give it a try,’” she recalled.
Three years later, Tosches and her colleagues published their first results comparing the neuron cell types in turtles and lizards to those in mice and humans. The differences in gene expression suggested that the reptilian DVR and the mammalian neocortex evolved independently from different regions of the brain.
“The 2018 paper was really a landmark paper in that it was the first really comprehensive molecular characterization of neural types between mammals and reptiles,” said Bradley Colquitt, a molecular neuroscientist at the University of California, Santa Cruz.
But to truly confirm that the two brain areas didn’t evolve from the same ancestral source, Tosches and her team realized they needed to know more about how the neural cell types in mammals and reptiles might compare to the neurons in an ancient common ancestor.
They decided to look for clues in the brain of a salamander called the sharp-ribbed newt. (It takes its name from its ability to push its ribs out through its skin to poison and impale predators.) Salamanders are amphibians, which split away from the lineage they shared with mammals and reptiles about 30 million years after the first four-legged animals wandered onto land and millions of years before the mammals and reptiles split from each other. Like all vertebrates, salamanders have a structure called a pallium that sits near the front of the brain. If salamanders had neurons in their pallium that were similar to neurons in the mammalian neocortex or the reptilian DVR, then those neurons must have existed in an ancient ancestor that all three groups of animals shared.
In their 2022 paper, Tosches’ lab performed single-cell RNA sequencing on thousands of salamander brain cells and compared the results to data collected previously from reptiles and mammals. Tiny salamander brains, each about one-fiftieth the volume of a mouse brain, were painstakingly prepared and labeled by the researchers. The brains were then put into a machine about the size of a shoebox that prepared all the samples for sequencing in about 20 minutes. (Tosches noted that before the recent technological improvements, it would have taken a year.)
After the researchers analyzed the sequencing data, the answer to the debate became clear. Some of the neurons in the salamander matched neurons in the reptilian DVR, but some did not. This suggested that at least parts of the DVR evolved from the pallium of an ancestor shared with amphibians. The unmatched cells in the DVR seemed to be innovations that appeared after the amphibian and reptile lineages diverged. The reptilian DVR was therefore a mix of inherited and novel types of neurons.
Mammals, however, were a different story. Salamander neurons didn’t match anything in the mammalian neocortex, although they did resemble cells in parts of the mammalian brain outside the neocortex.
Moreover, several kinds of cells in the neocortex—specifically, the types of pyramidal neurons that make up the majority of neurons in the structure—didn’t match with cells in the reptiles either. Tosches and her colleagues therefore suggested that these neurons evolved solely in mammals. They aren’t the first researchers to propose that origin for the cells, but they are the first to produce evidence for it using the powerful resolution of single-cell RNA sequencing.
Tosches and her team propose that essentially all of the mammalian neocortex is an evolutionary innovation. So while at least part of the reptilian DVR was adapted from the brain region of an ancestral creature, the mammalian neocortex evolved as a new brain region burgeoning with novel cell types. Their answer to the decades of debate is that the mammalian neocortex and the reptile DVR are not homologous because they don’t have a common origin.
Georg Striedter, a neuroscience researcher at the University of California, Irvine, who studies comparative neurobiology and animal behavior, hailed these findings as exciting and surprising. “I felt like it was providing really good evidence for something that I had only speculated about,” he said.
The new answer from Tosches’ team doesn’t mean that the neocortex in mammals evolved to sit neatly atop older brain regions, as the triune brain theory proposed. Instead, as the neocortex expanded and new types of pyramidal neurons were born within it, other brain regions kept evolving in concert with it. They didn’t just hang on as an ancient “lizard brain” underneath. It’s even possible that the complexity emerging in the neocortex pushed other brain regions to evolve—or vice versa.
Tosches and her colleagues recently uncovered proof that seemingly ancient brain regions are still evolving in a second paper that appeared in the September 2022 issue of Science. She teamed up with Laurent, her postdoc mentor, to see what single-cell RNA sequencing could reveal about new and old cell types in a comparison of a lizard brain to a mouse brain. First they compared the full array of neural cell types in each species to find the ones that they shared, which must have been passed down from a common ancestor. Then they looked for neural cell types that differed between the species.
Their results showed that both conserved and novel neural cell types are found all over the brain—not just in the brain regions that appeared more recently. The entire brain is a “mosaic” of old and new cell types, said Justus Kebschull, an evolutionary neuroscientist at Johns Hopkins University.
Some scientists, however, say it’s not that easy to declare the debate over. Barbara Finlay, an evolutionary neuroscientist at Cornell University, thinks it is still necessary to look at how neurons are generated and how they migrate and connect up during development, rather than only comparing where they end up in adult amphibian, reptilian and mammalian brains. Finlay thinks it would be “terrific” if those findings could all be pulled together. “I think we will in time,” she said.
Tosches noted that amphibian brains could have lost some complexity that was present in an earlier common ancestor. To know for sure, Tosches said that researchers will need to use single-cell RNA sequencing on primitive bony fish species or other amphibians that are still alive today. That experiment could reveal whether any of the types of neurons seen in mammals had predecessors in animals before amphibians.
The work from Tosches and her colleagues has also prompted new discussions about whether the field should reconsider what a cerebral cortex is and which animals have one. The current definition says that a cerebral cortex must have visible neural layers like the neocortex or DVR, but Tosches regards that as “baggage” left over from traditional neuroanatomy. When her team used the new sequencing tools, they found evidence of layers in the salamander’s brain as well.
“There is no reason, to me, to say that salamanders or amphibians don’t have a cortex,” Tosches said. “At this point, if we call the reptilian cortex a cortex, we should be calling also the salamander pallium a cortex.”
Babbitt thinks that Tosches has a point. “How these things were defined with classical morphology is probably just not going to hold up just based on the tools that we have now,” Babbitt said.
The question bears on how neuroscientists should think about birds. Experts agree that birds have impressive cognitive abilities that can match or surpass those of many mammals. Because birds descended from reptiles, they too have a DVR—but for some reason, neither their DVR nor their other “cortex-like” brain regions are organized into obvious layers. The absence of visible layers doesn’t seem to have stopped these regions from supporting complex behaviors and skills. Nevertheless, birds still aren’t recognized as having a cortex.
Such a strong focus on looks might be leading scientists astray. As the new single-cell data from Tosches’ team shows, “looks can be deceiving when it comes to homology,” Striedter said.
Original story reprinted with permission from Quanta Magazine, an editorially independent publication of the Simons Foundation whose mission is to enhance public understanding of science by covering research developments and trends in mathematics and the physical and life sciences.