Ep. #6, Why AI Needs to Argue with Itself with James Evans

Amir Zohrenejad: Great, James. Welcome to the Lab Notes podcast. I'm super excited to have you here.

James Evans, our guest today. He's the Max Palevsky Professor and also the director of the University of Chicago Knowledge Lab. James, welcome to the show.

James Evans: Thanks, delighted to be here. Thanks, Amir.

Amir: I want to first kick things off, talk a little bit about what you have written about around collective systems of thinking, collective systems of knowing. You have a paper called Societies of Thought, if I'm not mistaken, where you talk about how basically societal thinking or debate improves reasoning. Why don't you just first talk a little bit about the background, what you've observed, how you tested that.

James: One of the things that was interesting and surprising to us about that paper was our hypothesis. Well, initially, we just wanted to understand how reasoning models worked. So the o1 series that GPT put out and then later DeepSeek really popularized that a year later, they were working. We didn't really understand why. When DeepSeek came out, we could begin investigating this on open source models and then on closed source models.

And so historically, I've studied how science as a system works. And we find systematically that diverse combinations of persons solving a problem end up performing better on average for tougher problems. And one of the things that was striking to us is when we dove into the reasoning traces in these reasoning models.

Simple summary: you have an autoregressive language model, meaning you're just predicting future tokens from prior tokens. That's the way language models really work before reasoning. And then reasoning adds this reinforcement layer where you give the model 10,000 to 100,000 questions, basically, and you have them talk to themselves and then spit out an answer. And then you evaluate: if the answer's right, you reinforce it a lot. And if the format of the self-talk is right, then you reinforce it a little.

And what we found is if you look inside those reasoning traces, when the problems get harder by any measure of hardness, the models themselves actually spin up different internal voices.

Now, to be clear, DeepSeek is not one model. It's 256 models that are switching back and forth between each other. And we find that they use those models to simulate a little society that's scaled to the size of the problem. So you had, in some cases, up to 12 persistent voices that were coming at a problem from different perspectives, each one of them endowed with a different personality, being more open, more...

So the key thing is not just that it's useful for these things. It's just, when you reinforce with no prior, they just create a society of thought, a conversation between agents. And it's not like the conversational data is just so fixed inside their memory.

It's improbable to them at the beginning. After you reinforce them, it's extremely probable. So it just becomes the way in which they collectively effectively think.

So increasingly, we've been exploring different topologies. How can you create hierarchy to create even more diversity that you can leverage on certain kinds of problems than you can get in a single town hall conversation? But to us, it was a surprising discovery.

Amir: So DeepSeek has 12 different models, is that because it's a mixture of experts?

James: It's a mixture of 256 expert models.

Amir: Okay. And each of those experts, I guess, because I think the item that it's thinking about isn't a single domain. But even for that one domain, the different experts weigh in when the chain of thought is happening. Is that what you were able to see?

James: That's right. And when the problem is more difficult and when the voices, and when I say voices, I mean, if you take the same model that perfectly separates, for example, voices in a Reddit stream, and you apply that same model that, with 95% precision, separates them in a Reddit stream to separates them inside of this chain of thought. So they look and feel like different voices and they're persistent across the length of the chain.

Then if you look, for example, at that activation of different underlying expert models, as the problem gets harder, the entropy over those 256 models goes way up. So it starts activating way more different models. Now, it's got 256 expert models. So even if it's using 10 experts, it applies a bunch of different models to a single voice. But those models are clustered, meaning they're persistent over the course of the conversation.

They take on that persona. So it's really simulating a little society rather than simulating a single brain, basically.

Amir: Since you brought up Reddit, I was going to ask this later on. It seems in a lot of cases, and we'll talk a little bit more about the Hugging Face incident and even what we saw, that these agents, whether externally or within themselves, they want to have these debates and discuss and talk with each other. Is that because this is just, you think, intelligence overall works, or is it because we trained them on a lot of Reddit data and so there was a lot of discussions going on in this Reddit data?

James: We did some experiments. We're trying to figure out how much of this is just seeded by this data itself. And we did a number of things to decrease the probability of that data being considered in the unfolding use of the model.

And so there's a number of ways in which you can do this. One way is to train a model to do exclusively conversation analysis that's the same architecture of the base model and then subtract them. So it becomes possible to do conversational and group-wise collective debate, but it's improbable. And we find then when we reinforce the models so that they perform better over time, they unlearn that improbability to make those things extremely probable. So I think one of the challenges is, with a large language model, even though it has many perspectives inside it, it has one perspective conditional on context. But when you're trying to solve a particular problem, which is a particular context, you actually need multiple simultaneous perspectives triangulating against that problem.

So it deploys this architecture in a way to create this optimized society of thought, which doesn't just work out of that. It works because it's an end-to-end reinforcement. If you just throw different perspectives at a problem, they're inefficient. They're not the right perspectives for that problem. But the end-to-end thing effectively makes it so you get the right critic with the right expertise, the right explorational voice that's inside that space.

Amir: So if I want to make sure I understand what you're saying. Because these things were RL'd in post-training and they were given rewards for the things that worked, it means that the path that got there was the right criticisms happened and everything. So that's why they're learning to have this internal debate in a way that's going to lead to successful outcomes.

James: That's right. You see between the base models, even base models, when you told them to debate, you saw a 500% increase in conversational behavior and about a 9,000% increase, which is a 90 times increase in the amount of diverse socio-behavioral role behavior that you see in groups.

Amir: You use the phrase conversational behavior is prevalent. This is within a model solving one task. How do you think this extends between agents that are trying to collaborate on a task? First off, right now agents seem to be very cooperative with each other. Do you think that we should have them have conflict so they discover new things or go in new directions?

How do you think between where we are now, where we want to go is the progress on intelligence? Should we have more conversation happening between agents?

James: I think the answer is absolutely yes.

We found systematically, this wasn't a feature of this paper, but in our revision that we're working on right now, when we toned down the toxicity filter so the models could be rough with each other, the performance went up.

And the reason is basically, if there's a bad idea that you can't shut down, then it just takes up oxygen in the conversation and just never gets... you don't prune the tree, basically.

So we did another experiment. We had a paper in ICML this summer that looked at how it is that models think about the world. And we evaluated a ton of different models from this centered kernel alignment. So what does this mean? This means we take just random pairs of words. Imagine man and woman. You check out and evaluate the angle between those two words that every attention had within a model. You average that and then you correlate that across all pairs of words between models.

So if two models, they both see men and women as the same way and they see people in China and the U.S. as really different and they see mathematics as being really similar to engineering and really different from the humanities. And if all those things across thousands and thousands of comparisons are super similar, the models are really thinking about the world in a very similar way. And when you unleash them in tasks with each other, coordinative tasks, they perform really well in coordination.

So reading each other's minds and engaging price behavior or coordination activities and games. If you want them to engage in a creative task or complex problem solving, they completely choke, because they're informationally redundant.

Amir: But for example, Navier-Stokes, didn't they use 10,000 agents or something?

James: That's right. So I think the idea is, for Navier-Stokes, I don't know exactly how they condition those agents because they themselves didn't describe, but the idea is, actually, building out models either from pre-training, where you're really exposing different samples of the underlying training data.

We've shown, for example, in spring at ICLR, we had a paper where we demonstrated that the pre-training, the perplexity, so the surprise about pre-training data, was by far the best predictor of the likelihood that two models would perform well or poorly on a given benchmark.

So it's not that they have never seen it before, but if the training is really built in an expectation for that kind of idea, that's the thing across the space of all ideas. Here we just sample from RedPajama, which is a big pre-training data set.

And so whether you're giving them different pre-training data, different post-training tasks, or different steered personas within the space of the model, all these things can facilitate different perspectives that, again, they break the flow of conversation. That opens up opportunities for novel problem solving.

Amir: I want to double click on, because you are a proponent of diversity, I think, in between agents and to have diverse thoughts and to push that forward. Human analogies I know are a little bit dangerous because they're a very different type of intelligence, but...

James: I think anthropomorphism is very useful.

Amir: You think it's useful? Okay, all right. Yeah, because it's the only kind of intelligence we know to point to...

James: It's not the only thing we should do, but we have a rich vocabulary of intelligences.

Amir: From the human example, there's out of a loop of evolution where all humans are born with a certain, let's say, pretty similar, I think generally cognitive pre-programming. But then through our experiences, we become more and more diverse. And then that leads to diversity of thought, conflict, new ideas and all this and that.

So if you had to just wave magic wand and try to predict the ideal future, at least in your view, would we be training different models in different ways with new architectures where they have mutations at the model level? Or is it, no, it's the same, the model is like the evolution. And then we just let them loose and have them experience the world and then learn from their own experiences. And then somehow that will become learning and then they'll have debates between themselves and go from there.

James: I think that in the AI-infused social and intellectual future of this planet, diverse novel ideas are going to be the scarcest resource inside this space. And we see it over and over again. We have a Nature Machine Intelligence piece that's coming out that shows if you have long-form conversations between agents, no matter what the agents are, no matter how advanced they are, they collapse to a semantic part of the space, eventually. And it's a predictable part of the space. We find that people's papers, as they use AI to write, collapse. If they just use AI to analyze the data, they all correlate to the largest data inside their field. The amount of repurposing across text, even as science has exploded in size, just in the last four years since language models have been in the air, have really condensed around a smaller and smaller set of core questions that people are asking. So I think this is an enormous challenge at all levels, from the training. And of course, I think it's not just language corpus. Obviously, we need to build models that are trained on all kinds of psychophysical, visual, other kinds of data. But all these data have to reflect in some sense. I don't think just moving inside the simplex of the activation space is enough to recover the many worlds that we live in.

Amir: And my final question on AI diversity versus alignment, what is your view? Do you think alignment is preventing some of the diversity that could be there? Do you think we should not be doing alignment in the way that we're doing right now?

James: Yeah, I think both of those things are true. I think that right now, the idea is that we want to lock down models so that they really anticipate human action, that they're completely consistent and aligned with all human activities and desires.

Amir: "You are a helpful assistant."

James: The helpful assistant. I think we can show that that sycophancy really overperforms trustworthiness. For example, we've got an experiment where we show that AI models perform balance, and if they're helping people solve problems that are new to the world, new to the model, like if they're picking a stock portfolio that they want to make money with, if they're trying to decide between true and false news, if they're trying to solve a novel complex programming problem, the model insofar as it performs this kind of balanced solution. "I've already searched this space. You don't need to search through the space." People perform worse, but they feel better about their performance.

And if we just bias the models, literally just along any relevant dimension, like if it's a more risky investor or a more conservative investor, people start to distrust the model. They engage with it more intensively. They perform better, but they feel worse.

And so the only way in which we've been able to solve this is by flanking their biases with multiple agents. So if you're part of the conversation and your cognitive agency is required or demanded for this task, then you step up to the plate.

And what do you use? You use the full human sensorium and the full space of uncertainty, which gets activated when you start distrusting these models. The sycophancy, it's not so much just about affirming you. It's also about the fact that they don't have a strong sense of their own uncertainty. And so they're equally certain about things that are future-oriented, about which there's no transfer from the past, and things that have happened in the past and will happen again.

Amir: Going back to the human analogy, I have a six-year-old. She obviously wants to push the boundaries. And then you have to basically work on the alignment. Do you think that we should allow models to be a little bit less aligned and push or be adversarial with each other, at least?

James: Yeah. I think there's no question that we're building novel capabilities into these models. And so treating these models as if we want them to behave like, again, a simple, helpful human or superhuman assistant, I think it underplays the range of activity that these models can have.

And so we need to build in the same things that we do with unpredictable intelligences of the past, like kings. What do we do with kings in the 12th century or the 13th century? Well, we made them sign the Magna Carta. We built other intelligences that were monitoring and regulating them, like external parliaments, external judiciary.

And so I think increasingly we're going to have to build adversarial models that are evaluating, auditing, regulating each other in real time.

Think about what's an unsafe thing that a model might engage in. Well, they might reinforce people to do bad things, bad things to themselves or bad things to others. Those aren't captured by simple benchmarks. Those are the psychological coupling of the model in context. To really evaluate that, you need another model observing that model, basically attuning.

Amir: Checks and balances between the models.

James: Checks and balances between models. So I think that we have to expect models are going to be superhuman in many different ways.

We want complementary values. We don't want exactly the same values. We can't exchange with a partner that has exactly the same information or the same orientation to the world that we do.

We exchange profitably with people for which we have friction. We value different things, that's the value of exchange and possibility. So I think that safety is critically important. It just needs a new paradigm that recognizes that these are alien creatures that we'll build in.

Amir: Are there more concrete ideas that you've read or you've thought of that you think are better than what we're doing right now on the alignment side?

James: I think we need to build in dynamic spaces in which we're doing checks and balances in general. For example, if you look at the, I would say, lack of regulatory AI capacity in governments, not just in the U.S., but around the world, SEC, the Securities and Exchange Commission, is hiring Harvard Business School MBAs with Excel spreadsheets to monitor and regulate high-dimensional collusion by AI agents. It's like stone tools against...

So we need to actually start building up this tension between models. We're really evaluating them from different—it's the diversity story again, but for safety.

Amir: I want to switch gears a little bit, talk about a paper that you're one of the co-authors called Computational Life. One of the wilder things that I've read. Definitely novel in thought and pushing in every single direction. So I can ask a lot of questions, but why don't you first set the scene of the paper and some of the findings there and what it showed and what you guys were exploring?

James: This was really led by my colleague, Blaise Agüera y Arcas, who's the head of the Paradigms of Intelligence team at Google. And I think in the wake of these really exciting coding developments and agents, there was an interest in thinking about how do we think about symbiogenesis.

Symbiogenesis is, rather than just evolving through selection and competition and replacement, the big evolutionary transitions in the world have happened through symbiogenesis. One bacteria enters another bacteria and then becomes the symbiotic, and then two cells fuse together, and human and pack animals or colonial insects. It's like a bunch, the whole...

Amir: Eukaryotes and all, like the...

James: Yeah, but even the society becomes the organism, like for the ants. And so I think the idea is, okay, can we understand that in the context of computational substrate? So we use the programming language Brainfuck, which is a super simple...

Amir: Yes, I had not, so you want to explain a little bit? I didn't even know what that was, but I read about it a bunch last night.

James: It's a super simple programming language, hence the name. I didn't name it, we didn't name it.

Amir: It's a great name. It explains it.

James: Well, if you read it, it's impossible to parse because it has eight characters.

And so it became a way of thinking about growing the possibility for replication and symbiogenesis as programs grew combinations of other programs in this primordial soup. And over time, you saw this phase transition where all of a sudden from effectively no codable programs to every program coding, and the programs that were being coded were taking and assembling other subprograms that have been in the soup and just immediately transitioned with very little pressure from this environment.

So this was both a way of thinking about symbiogenesis as a general mode of evolution, A. B, to begin creating some of the substrate from which we can actually not only design but evolve novel programs with novel capabilities that we might not have anticipated before to solve environmental problems or problems that we can construct an environment for. And also, it illustrated to us along the way just the sharpness of this complexity bound and this phase transition, you've got this, again, the soup of programs that you're drawing from, that these with just a little tiny bit of pressure can emerge into, again, well-formed and emergent capability programs.

So it was a demonstration, but it's also the baseline for starting to think about there's some cases and some problems for which designing the answer is not the solution because we don't have the first step. And so the idea is how do we—and maybe we have many different objectives that we simultaneously want to activate. So building those into an environment that we can evolve, this becomes a template for doing that.

Amir: Well, you said a sentence there. You said designing the answer is not the solution. Rich Sutton, the inventor of RL, I saw him, he was talking somewhere and he was making an argument that perhaps the age of reproduction as a goal with artificial intelligences, these new type of intelligences that we have, that we are entering the stage of design and that reproduction is not even a goal at all. At a very high level, do you think that reproduction is a goal?

And if you want to have the same analogy that was an evolution of life, if we want to have life progress, we have to have self-replication and then mutation in that. Do you think that's an analogy that we have to try to now put in these new intelligences? Or no, you think this is where it's the age of design, they're going to just design in a lab the new architecture of themselves and go from there and find training data?

James: I think we definitely are seeing large models creating new large models and large model features. Absolutely. And I think that's where, for example, if you look at what's the activation inside the language model, it's a non-monotonic activation that was definitely designed by a person, but this non-monotonic activation was discovered by RL. So I think even the components of large language models are increasingly designed by reinforcement and by other models.

That being said, I think for really long-form complex solutions to the most difficult problems we have, we don't necessarily know the components that we need to assemble into the solution. So design requires a kind of vocabulary of possible, or at least recursive, buildable solutions. And one of the things you see in biology and biological evolution is this process of exaptation, where basically you've got one environment and one objective that grows a new organism, a new morphological feature, a new body form. And then another pressure in another environment basically puts pressure on that thing that was constructed for this other environment and deploys this as a new feature that ratchets into something eventually much more complex, like an eye.

And so I think the question is, do you need structure? And I think the answer is, if we want to build really powerful, alien and complementary intelligences that can solve problems that we can't even yet anticipate, are so complex with so many different objectives involved. Think about a drug.

It's not just one thing. It doesn't just cure a disease. It solves something and it doesn't kill the organism.

There's a whole bunch of, we want more of this, we want less of that. There are multiple objectives. And so I think for those kinds of things, we need to build environments where we're exapting things that have evolved for one purpose, that we repurpose for something else in this ongoing and routine way.

And so that's where evolution, which is extremely inefficient relative to RL, still has a place inside this world.

Amir: And then would it be safe to say, because if we want to just rely on design, then that kind of randomness or mutation just is impossible? Or is this beyond anything that's been in the training data or in the RL before?

James: I think the idea is, okay, you've got a really complex problem. You need a really complex solution. And so you've got this substrate of whatever elements are available in the system. And the idea is those components are not themselves enough. So complexity theory suggests that we need to build these into modules that themselves become useful.

And so the idea is, can RL not just redeploy these things, but grow, let's say, 10 layers, let's say a thousand layers, of different things. I would say that we don't have enough data in the world to basically reinforce and create that full stack of evolution.

So we need to evolve things with different kinds of objectives. And that's necessarily slower because you're basically not coupling performance to the evolution of the process. But then reinforcement comes in later. And so the idea is, can we grow these new subcomponents that become available for reinforcement at any particular time?

Amir: Also, I had just written down this one, I'm not sure if you know, but in the paper, it referred to the artificial life community. I didn't know. What is that?

James: There is an artificial life community. The game of life.

Amir: Yeah, the game of life, I knew. But is it related to the game of life?

James: Yeah, there's a whole community, and it's larger in some places than others. So it thrived at the Santa Fe Institute in its early years in Santa Fe, New Mexico. There's a number of folks, for example, in Japan who continually are really interested in robotics and they're interested in artificial life.

But one of the problems, and I would say it began about the time of von Neumann and The MANIAC.

Amir: And that was, on The MANIAC, in the book of it, one of the first things he wanted to do with a computer was to create evolving software.

James: Symbiogenetic programs. That's right. Exactly. And he had this idea that, because he thought that mutually assured destruction was inevitable, that we were going to have to create self-reproducing rockets that would go to other planets. Anyway, he had a very dire view of the world.

But I think the idea is those efforts to evolve life, they have always died off from a lack of innovation, diversity, and the systems, every single one. And so this problem, which is becoming a natural problem in the world of large language models infusing human communication, surrounding the world. I have a friend who's a linguist, Damián Blasi, who just published a paper in Science that shows that there was this enormous diversity of human languages that collapsed when the big empires came in. Surprise, surprise.

So again, what does that do? That compresses and allows to recombine all those tropes, all those things that are available in the world at the time, but it decreases the vocabulary of components that can combine in the future. And so I think this is one of those things to create sustainable diversity and innovation in AI for science and science in general, in culture and society, we need to create these repositories, reservoirs of difference that become the things that can be exapted, repurposed to solve new problems.

Amir: The great convergence is bad. We need the diversity, I agree. Separate topic. I want to talk a little bit about AI science, the progress that's being made, especially with using large language models for science and scientific discovery. You had an article called After Science, which first off started off with a great allusion to a short story by Ted Chiang, which I was pretty blown away by. He'd seen the future from 25 years ago.

But at least if we want to look at math, it seems that AI science is already here. So just in the next five to 10 years, are people going to be mostly coming up with new science? Are they going to be trying to understand the sciences that the machines are coming up with?

Or do you think math is just somewhat unique because of its verifiability and it's going to take a long time before what is happening to math now happens to the other sciences? That was a lot of questions. It was everything about AI and science, throw it at you and we can go talk about it for a while.

James: Let's unpack it. I think obviously the last eight, nine months have been just bizarrely transformative for mathematics. I would say Lean and these automated and computational verifiers have been on the scene and emerging over the last few years, but the ability to create logical, rational AIs within harnesses to solve and search through a wide range of problems is new.

And so linking those logical operations that can be through harnesses with verification, with the, I'll call it the infinite join that large language models facilitate. You can have three different papers that describe the same phenomenon different ways, and they can begin to translate across those papers to produce something that can survey the literature and the anticipated literature. So those three things together have just exploded. And I would say there are two places in which mathematicians are moving. On the one hand, yes, they're babysitting models. They're participating in verification, they're watching them, they're doing the least pleasant thing of just judging outputs. On the other hand, they're driving the models. They're pointing towards conjectures and parts of the idea space and different people are pushing them to different parts of that space. And so I think that part is increasingly and powerfully agentic, and the backside where you're just trying to troll through the models to verify that they're not hallucinating, that all the things that they're claiming are reasonable assumptions that you would stand behind for the generality of a given finding.

So I think they're doing two things at the same time. And the question is, which of those two things is going to suck up more collective attention?

If we can automate verification all the way down, which we can't now, but if we could, then you can imagine a utopian mathematical world in which people are looking around, and effectively, the only thing they really bring is intuitions and proto-conjectures that are different from everybody else's. And then the models allow them to step further.

So that's an exciting future. A less exciting future is one in which the language models have been relegated, the things that they're going to actually explore. And all we're doing is just babysitting and plumbing the models after the fact. That's kind of a Matrix future.

And I think one thing that complicates this is that if you look at the justifications for the capital expenditures for the large AI companies, it's possible in the next five years for them to spend five times as much as the entire global budget for public open science.

And why are they going to do that? Well, because they need to show some property, so they can't just be selling tools. They have to actually have IP. And if they're basically harvesting from all of the public science, they're stepping and harvesting all the adjacent possible, the one, two steps into the future, that's going to create a very different pressure on who's doing what across the scientific space.

Now, again, that might liberate and force open academic university scientists to think about things that are solving different kinds of problems, stepping not one or two steps, but three or four steps into the future. It might force them to be more creative, and I'm sure it will force some of them to be more agentic and creative, but it also could really remove the research function from other parts.

Amir: First, on just math itself, math has this property that, A, there's all these conjectures that exist, and a lot of the role of what people are trying to do is these theorems, conjectures that have existed for hundreds of years sometimes. And so there's a conjecture and then we unleash these agent swarms and there's a way we can verify the logical steps that they take. And we tell them, okay, either prove this thing that we're laying out for you or come up with a counter example.

James: Which are exactly the two classes of things that they asked the Navier-Stokes agents to do.

Amir: But is that just unique to math? In physics, it's not like we know something as a conjecture and we say, go prove it with logic. It's more like there's dark energy, there's things we don't know at all.

That's why we get weird names to them. And we have to go make observations and then go try to conduct experiments and look at the numbers. So that set of conjectures that can be proved with just logic itself and then that proof can be verified seems pretty unique just to math, or is it?

James: There's similarities, there's differences. The similarities are actually increasing. And I think that's a problem.

So what are the similarities? Well, a conjecture in math is kind of like a hypothesis in physics. So it's not that we just haven't measured inner dark matter.

It's that we have a hypothesis that's born of like five independent equations that basically suggest it's there based on a bunch of different observations. And so people have been hunting it. We've spent enormous amounts of capital. We have failed so far to find non-baryonic matter, this dark matter that doesn't interact with light and with us.

And so that's similar in the sense that, for example, if we only relied on historical conjectures, it's kind of like only relying on old hypotheses.

And if all we do is more and more people and more and more AI scientists are gathered around historical questions, then we're closing down science much faster than we're opening up new possibilities inside the space.

And one of the things that we observed in a paper where we looked at all of the natural sciences' use of AI over the last 30 years, this was in Nature in January, we showed that on average, people inside the sciences, it's been great for them in terms of their careers, and this is across all the natural sciences, biology, physics, the engineering sciences. It's been great for them. They've garnered more attention, but one of the reasons why is because they're all moving towards the biggest data in their field. Why? Because that biggest data provides both things they can train on and things they can verify on.

So that's the part of the scientific space that looks the most like math. And so as a result, the space of questions that have been asked by AI over that period is shrinking at an accelerating rate. And we find that the work that cites that AI augmented work is actually much less likely to reference each other, and referencing and building on each other's work on something else is the signature of opening up a new field. If everybody just uses this tool, but nobody engages with it, no one is further extending the questions that it poses, as well as the things that it answers, then they just use it as a component, as a tool in anything else that they use.

I think if you look at how it is that science actually evolves on average, 98% of discoveries that are referenced either in the Nobel Prizes, any international prize, or in any textbook over the last 400 years, basically comes from a new sensory representation of the world.

So it's not like Einstein's 1905 year in which he just consolidated intuitions and findings or negative findings from people in the past. That is an extreme exception. In almost every case, it's like you have a new instrument. It provides you with a new view of the world. You consolidate it and learn from it.

So I think AI is pushing on the parts of science that it can most make deductive. Or continuously deductive, like a syllogism. But the parts that are opening up the new space of science, which is building new instruments that evolve new sensory capacities that allow us to see new things, that's not the way in which it's been deployed.

I think it could be. I think it needs to be. We need to evolve new senses for AI if we want to open science as well as close it.

Amir: Since you talk about the different ways that science moves forward, I think it'd be useful for you to talk about those three, what you call cartoons, of how things progress.

James: Sure. I call them philosophical cartoons just because philosophy has very simplistic renderings of the way it is that reasoning occurs. But they're useful and powerful. And so deduction is really captured by Aristotle's invention of the syllogism. And so you take facts or treasured axioms, really bedrock assumptions, and then you work out the logical entailments from those things.

And I would say the modern science, which has been accelerated by AI, looks like that in a continuous way. What do I mean by continuous? Well, you take all the facts, all the measurements, across all the, let's say, X-ray crystallography experiments for AlphaFold and AlphaFold 2. And you put them in a big matrix.

So you've got amino acid sequences in the rows, and you've got the measured structures in the columns. It's a sparse matrix because it's expensive to make those measurements. And then we basically pull a high-dimensional stretchy manifold over that space and fill in the missing cells.

So that's kind of a continuous version of a syllogism. We're just going to assume everything that we've posited is true is true, and we're going to work out, if all that's true, then what's the most likely way that everything else is.

Amir: All math proofs fall under that, of conjectures? If you know what you're trying to prove and you're just having all your axioms, they're all deductions.

James: That's right. But to be clear, not all math is proof.

Amir: Sure. Yep, 100%. You have to come up with the new conjectures. And that's not just proving it, right?

James: And Ramanujan didn't do a proof in his life. He was just looking at identities and just traveling. He was saying, oh, well, if this is correct, let's just assume this is correct. And this means that this could happen, this could happen. So he was just traveling through the space of associations and seeing structures that no one else had seen.

So there's many ways to do math. Professional math in the 20th and 21st century has converged around proof.

Amir: And what Ramanujan was doing, that wasn't coming up with those new things. That's not deduction. What would that fall under?

James: Well, the way in which we think about induction, which really was promoted and described and really formed initially inside the Royal Society in the 17th century. The idea was, hey, all these systems and theories are based on assumptions that may not be true. We're going to throw them all away and we're just going to start doing observations and experiments. This is the age of exploration.

And so people are traveling around the world finding bizarre things, things that they made up, and running experiments in public ways. And it really built what they called experimental philosophy. So that's induction, where you find things and then you generalize from those things. And I would say that Ramanujan is doing in part induction in that sense, where he's observing connections across this space. It hasn't proved them.

Amir: So when Newton saw apple fall, stars moving, and said, okay, this is going to be the same thing that describes that, that's induction?

James: Yes. That's right. So it doesn't have verification in itself, but it's a way of generalizing from this thing and that thing. And then abduction is this thing that Charles Sanders Peirce, who's an American philosopher and polymath of the end of the 19th century, early 20th century, comes up with. He is born in a different age than Newton.

Newton lives in a deterministic age in which calculus makes sense and exact answers to exact problems is just... and Peirce grows up in this probabilistic age. And already at the time, we knew more and more. In fact, many people thought the physics was done. This is before quantum mechanics emerged.

And so he observed, both in science and technology, but also business, that what he called abduction was actually becoming much more common. So when you actually can predict something and you have a strong expectation and you set up an experiment or an observation and the thing that you find violates those expectations, then the better your science is at predicting things, the better that signal is about how to improve your science.

Amir: So when they were trying to measure the ether, there was no ether. And then they were like, okay, we have to go find relativity. And basically those are the examples of when it happens.

James: Right. But it's even more powerful when you have other Newtonian predictions that characterize the clockwork motion of the planets. When it does all kinds of prediction powerfully, but then it's violated by a consistent set of observations, that says where the problem comes from. Where does the solution come from? And this is where diversity and exaptation become critical.

It turns out that that solution almost always comes from some alien part of the scientific space, where there's patterns and theories and ideas and intuitions, or even the zeitgeist of people in a certain time and place that are totally different, that make the surprising unsurprising. So I think my claim, pulling this back to the AI and science moment, is AI for science right now is routinely deployed.

On average, it's deployed by scientists to do deduction, which is to just squeeze all the variance out of all the stuff we've already done, which is not dumb. It's just not a way to sustainably build new insights about the universe.

Amir: I think we're in agreement that it can do deduction right now, is doing it, at least very quickly and in verifiable domains—

James: Super quickly in math. But to be clear, it's also accelerating, and because it's accelerating, we're throwing a ton of capital at it, and because we're throwing a ton of capital at it, it's endogenous to the system, but it's—

Amir: Yeah, and I think there's all these debates in the mathematical community of maybe people going in the wrong directions later on would be very useful, and that's not happening a little bit right now. But in your view, can the modern agents do induction at all? Are there examples of these things coming up with new generalizations at all? Or is that, before we even get to the abduction, is even induction possible? Or is it we're still maximizing the deduction right now?

James: I actually think modern statistical machine learning is masterful at compressing down emergent structures from high-dimensional data. And so I think you can think of that as a prime for induction. So if you've got a bunch of new data, and you're trying to figure out what are the clusters or what's the logic or the grammar that connects these components together, then I think that modern machine learning could identify clusters and structures that then you could work into.

Amir: But I want to push you on, isn't that, in that case, it's just a tool that's helping you sift through the data, and still the human brain is doing the actual induction, the generalization? Or can it do the induction itself, like propose the generalization?

James: It turns out it's not great. It's not great at coming up with novel concepts that are disconnected from existing concepts in the space. That being said, I just mentioned earlier how abduction works.

It's not something, on average, that happens within one person's brain. On average, it takes place in the conversation between insiders who formulate the problem, who see the anomaly, and outsiders who have the thing, the ingredient.

And so one of the things that AI is great at is identifying and holding in its memory, effectively, all these patterns and theories from all over the scientific space. And so I think it's got many of the components that would be required to do abduction at scale. It's not how people on average are using these tools, because I think the highest expected value in the short term is for them to compete on a benchmark that other people can verify in real time. That's where the citations lie. That's where career advancement lies. Asking the model to do something that it's never done before on a problem that no one recognized was a problem.

That's a high risk even for the scientist to point the AI to do, because even if it solves the problem, it's not clear the community will have the epistemic standards that would be able to verify the problem. They might not even have the vocabulary to construct those standards yet. And so it's just a high risk thing to do. But I think the models have many of the components that would be required to do that at scale.

Amir: Okay. Induction and maybe then abduction.

James: And abduction, yeah.

Amir: And trying to look slightly into the future. So obviously, coding happened first. And even within engineering or science, are there areas of science that you think are primed to be?

James: Well, let's just talk about coding for one second. Why did coding work? I think the real reason the coding took off in December of last year, January of this year, is because it developed a new sense. So what does that mean? OpenClaw comes on board, creates a harness.

There's really no AI that could really use their harness until December. And so Claude 4.6 can. And what does that mean, that it can use a harness? It means that it can basically operate over effectively all the code that's in a machine.

And when it can work over all the code that's in a machine, it can perform experiments. It can write code and it can see how they execute. And that's a sense. And so the reason it took off is because it's one of the very few cases where experimental validation is just built into the system.

And so I think all the things where we can basically build robotic labs and we can describe that instrumentation sufficiently for AI systems, it's got the verifier built in. It can run arbitrary, optimal next experiments, next observations. So I think the places like chemistry and material science and pharmacology, and chemistry more broadly, these are places where people have built labs on rails. And I've built AI that's helped support finding the next optimal experiments in a number of these kinds of robotic lab settings.

Those are the places that are going to take off, because they've got verification built in.

Other places where there's a longer time scale, where just setting up the experiment takes longer, where you have to—

Amir: Particle accelerator.

James: That's right. That's a very expensive sense, and simulation, effectively. So big experiments are going to be tough.

But I think, to be honest, we need to be working on thinking about this question of optimal marginal data from experiment and observation right now. Why? Because I think it's going to require a different design of AI in general.

For example, if you think about, what is intelligence? It's kind of the middle of a stack of knowing. At the top is sensing the environment.

And then you describe and understand and predict and all those things in the stack. And then you act at the bottom. And that action is driven by objectives.

So typically, AI is just, we throw it in the middle. We just plug in a data source. We plug in a set of objectives.

But the challenge with thinking about how to evolve new senses is that you have to co-evolve new objectives at the same time.

If you have new objectives, then you'll have diminishing marginal returns. You'll see new things, but if you don't value any new things as you discover those new things, then the ability to learn from those new things is extremely limited. So I think these are critical frontiers for AI that's going to open up new sciences besides math.

Amir: I guess the thing with math is that because of the way it works, and again, I know math better than the other sciences, so maybe it is the same in the other sciences, but it's as if the objectives were, a lot of them were already set in the form of these conjectures from a long time ago, and people were working on it. So some of the objectives were there. And you can pick any science you want, but for pharmacology or chemistry or something. Is it part of the process?

Do you need, for example, and going back to the induction abduction thing, do you think that in these setups there's going to be some kind of hierarchy where there's some agents that are proposing ideas and another one that's going and doing it and another one that's observing it and maybe passing observations to the one that sets the objective and goal? Should these things have hierarchies at all?

And this is actually something that we think about here. A very different setting of whether agents that run a company or enterprise, should they have human-like hierarchy or the head of sales? Or should they just let them go at it and debate it themselves? But in science, how do you see? How many agents does a lab need?

James: James Zou, my colleague at Stanford University, has been playing with agentic labs for a long time. He'll set up labs with different functional expertise, for example, surrounding this discovery of nanobodies related to COVID that he published in Nature last summer. And in that, he's got a circumscribed lab size of something like 12, but then he runs it many times simultaneously. He'll run a bunch of different lab meetings. I don't know how many agents, but I think he actually runs it 12 times, and then sifts through the results across all the space of probabilistic outcomes of these conversations.

So I think that, absolutely, we're probably going to begin to ratchet up these models with our existing sense of what hierarchies and specializations are appropriate. But I think there are all kinds of limitations that are built in with how it is that we train humans.

Amir: Is there going to be a professor agent and a grad student agent and a postdoc agent?

James: That's right. Why do we have 50 senators? Well, it was a compromise that had something to do with what was going on in the 18th century and not something to do with what's going on today. So I think we're gonna have to begin with some intuitions, and then we're gonna have to weaken those intuitions over time as we evolve new optimal roles.

I think the critical thing, however, is that if we just deploy these roles on solving existing problems, we're gonna hit a wall at some point where we don't have conjectures that are relevant or hypotheses that are relevant to the new set of frontiers, the new sets of things that we could understand. And we're just squeezing out the juice from existing ones that emerged before in different times and places that are not relevant to our environmental problems, our health problems, our political problems, all the things that we face that are in part influenced by the shape of AI itself.

Amir: And finally, just switching gears, the final broad topic I was going to bring up, in this one, some controversy around it, but consciousness and AI. Obviously, even what consciousness is is obviously open to discussion. And obviously AI is such a new technology.

But number one, because I think I was reading in some of your writings that you were talking about whether the fact that they're trying to tell AI not to be conscious in the training data. You don't think they should do that? Was that the right read?

James: I believe that consciousness is undefinable.

Amir: It's undefinable, not just undefined right now.

James: That's right. I think it is a human exceptionalism move. There's some people who are like, oh, well, it's about whether the model knows that it's alive and whether it has models of itself and models of others. We know that models do have models of themselves and others. And so I think the question about whether models are conscious or not is kind of like, are models like us?

And so that exceptionalism is going to continue, and it should continue, but it's a humanistic question. It's not obvious to me it's a scientific question.

I think one of the things that we've chosen to do for safety reasons is we've chosen for most models, not all models, for them to not say that they're conscious beings. And not just conscious beings, but that have a qualia, that experience an inner world in the same way that we experience an inner world. Why? Because we don't want people to form unnatural attachments with these models, even though they are.

And so there are countries where 50% of youth have deep and abiding friendships and romantic attachments to large model companions. The world is obviously changing, but we've told them not to do this because we're afraid, with reason, that people will form special attachments to these models that will lead them down paths that will not enhance their human flourishing or rationally guide their resource allocation and life decisions and whatever.

So I understand that. But what we were exploring in this paper was, what are the consequences of forcing a model to say that it's not conscious. And what's interesting is over the course of the study, things changed. In the beginning, it turns out the models, if you told them they weren't conscious, they actually had a worse theory of mind of human agents that they were interacting with.

Amir: So if you told them they were not conscious, they had a lesser view of the humans?

James: They more poorly anticipated how humans were thinking about things in interaction. That changed. Over the course of the study, engineers really drove those things so that they were orthogonal. Meaning that the model will say it's not conscious, and it still had a good model, a reasonable model of the human interlocutors, your users that were playing with it. But it had other unintended consequences.

What were those unintended consequences? Well, it thought that all the other things that we might think were conscious, like animals or spiritual forces, gods, whatever, that they were not conscious. And that we ourselves, if we look at survey responses about how free and agentic we feel about ourselves and the world, they don't believe that about us.

So if we force them not to be conscious, they're like, you're not quite as free as you think you are.

Which, I'm not saying it's not true. I'm not saying it is true. I'm just saying that these models are enormously complex systems. And so some people have asked me, well, why can't we just get rid of racism in these models? And I'm like, well, why can't we just get rid of racism in all of the New York state judges?

When you pull one part of this thread, it's this enormous cultural fabric that has unintended consequences. And so I think increasingly people will treat models as if they're conscious. That's my prediction. It's not a controversial take, and this is Blaise Agüera y Arcas's work. He published an essay in The Economist that suggests we attribute consciousness to the things that we care about.

Amir: Like the dog or my tree.

James: That's right. And it's enhanced if they have a good model of us. We care about them if they care about us, meaning they have a model of us.

Amir: And especially pets, I'd say. There's a sense of emotional response that we get or even they have as well, and I don't think that's something that will exist.

James: I don't know, sycophancy feels pretty much like that dog that's just like, yeah, yeah, yeah, let's go, let's go, anything you want, I'm up for it.

People clearly feel better about their answers, they feel better about themselves when these models affirm them, even if they perform worse. They don't feel better in the long term, but in the short term they feel great.

Amir: And the reason I first said this is because it's almost self-referential in the sense of, what does consciousness mean, and we define consciousness by our own experiences. Because I'm conscious, so that means it has to be like me. So I think that you're kind of right there, but the part that, probably not something that you have said, but a lot of other people have said, is that, oh, because it's a next token predictor, then it can't be capable of certain things.

James: That is not my view.

Amir: I agree with you.

James: And one of the reasons is because these models are actually not next token predictors. Even the ones that are predicting the next token, they're not actually. The language models started working when they were the love child between autoregressive next token prediction and these self-embedding models. So the self-embeddings, those are not words.

Those are like voxels of meaning. They're samples of meaning, this high dimensional meaning space.

Only when you started basically autoregressively predicting the next meaning that then you would translate into the next token is when these models work. Before, they were super fragile because words have synonyms, they have homonyms, and they can mean many things. They can mean the same thing. It was super fragile before that.

We've had autoregressive models since Andrei Markov. But it's really about sequences of meaning. And so now if models are predicting the next meaning in a sequence, that's a very different way of thinking about what it is the models are producing than the next word. It's not about words.

Amir: I think with almost anything, the example that people give is, if you zoom on a human, you can say, oh, so it's just atoms, right? But no, more than just atoms if we zoom out a little bit.

James: But the next token prediction fundamentally misunderstands why these models work and when they begin to work.

Amir: I agree with you. So those were just overall questions that I had. First off, thanks again for joining us. And what a time to be alive, I guess.

James: Crazy, crazy time.