Ep. #3, Exploring Recursive Intelligence with Qizheng Zhang

Amir Zohrenejad: Well, Qizheng, welcome to the show. Excited to have you here. First and foremost, why don't you introduce yourself and tell everyone what you're working on?

Qizheng Zhang: Yeah, thank you so much for having me today. My name is Qizheng. I'm currently a fourth-year PhD student at Stanford. My research focuses on recursive self-improvement for AI systems.

So, for example, I have been developing novel methods for self-improving AI, like agentic context engineering and, most recently, Meta-Harness. I'm also very interested in directions like understanding why we have these gains from self-improving AI and why they would fail in some other cases. That is my most recent focus.

Amir: Got it. So just at a very high level, though, self-improving AI, recursive AI is really top of mind in the overall industry and in the research community. And so as you're thinking about either continual learning, self-improving AI, at a high level, how much of this stuff do you think belongs in the harness? And how much of this stuff do you think belongs in updating the model weights and biases automatically?

Qizheng: Yeah, I think that's a great question, because there have been a lot of debates recently, especially online, about whether we should just stay in a context or the harness space or we should go into the weight space. And I think the short answer here is that these two methods are not directly competing with each other, or I wouldn't say one is an alternative for the other.

It's not that we're doing something in the context space — it's not that it's preventing us from doing something in a weight space. And I personally believe that they have different relative advantages. For example, for doing things in the context and harness space, the obvious advantage is that it's enabling faster development iterations. It's also giving you much more interpretable learning outcomes, as you can see directly what has been learned from, for example, the context space and so on.

For things that go into weights, or things you want to internalize — for example, if you have something that will be persisted for a long time, some very repeated job — what happened many times in the future — or some more fundamental capability you want the model to have persistently, internally — that should go into a weight space. So briefly speaking, I don't think these two methods are "I pick A, so I would not be going for B" completely.

It's more like, I believe ultimately it would be a combination of both.

Amir: And so I'll take an example. So I was reading — Cursor, for example, with the last model that they built themselves. With my understanding, obviously Cursor has their own harness that they built and they improved, but they took a lot of output stuff from that and they did a lot of post-training on Kimi K2.

So in your view, let's say if you're building a harness and you have access to the model to update it, do you think, for now, do they have to be hand in hand? That's my question. Do you have to be improving your own harness and then taking some of that and feeding it back down into the learning and updating?

Or no — for now, I mean, it's the state of the art today. Do you think these two are still separate from each other? Basically, the improvements of the model are a different team, different people working on it, versus people improving the harness automatically.

Qizheng: Yeah, I wouldn't say these two are two separate jobs in some sense, especially given now that model development — a lot of the effort are spent on the post-training phase. I would say a lot of the times, many of the efforts we spend on post-training are overlapping with the efforts we're spending on developing harness or even test-time approaches.

So when we first, for example, talk about test time-related methods a few years ago, we were mostly talking about chain of thought, or how we enable longer thoughts in reasoning models. So that was described as something that you spend more compute on a single question during test time. That was the original test-time compute. And then this is something that we internalize into the model via the post-training process by, for example, doing RLVR-style post-training.

So in some sense, I wouldn't say these two efforts are disjoint. So if we have a very specific capability we observe that would be helpful during the inference stage — for example, making agents better, like the agent loop — then this is usually something worth putting into the model weights via the post-training process.

I haven't been working on the post-training process myself. But my observation is that a lot of the labs have post-trained models so it knows how to iteratively improve a solution, even without having explicit harness instructing the model, for example, to try different solutions, observe the output, and move on.

Amir: So let's kind of move on — and I think eventually this is an important question I would like to come back to — but I first want to talk a little bit about jumping into Meta-Harness. That was one of the more interesting and most exciting papers I've actually read in the past few months.

It seemed to me, obviously, it built a lot of the work that you had done previously with ACE, or basically GEPA and DSPy. Why don't you first talk a little bit about Meta-Harness, the problem you guys were solving and the approach that you took?

Qizheng: Yeah, first of all, I want to dedicate all the credits to the first author of the paper, Yoonho. So I learned a lot while working on this project with him. And also, we have some very surprising takeaways from our observation as well.

So originally, I met Yoonho, the first author of the project, at NeurIPS last year. So we were talking about things we can do in a context space, because we both believe this is a huge opportunity. And no matter in terms of infrastructure or in terms of algorithm methodology, I think this is still very elementary in its current stage.

So we were talking about some of the limitations we observe in the ACE paper. So the ACE paper — the high-level idea is that we optimize the context continuously over a long agent session. For example, it could be more than 100 turns.

Amir: And there's a playbook that keeps getting updated, right?

Qizheng: Yeah, exactly.

So we observed that many of the failures we observe in ACE are not fundamentally addressable just by updating the context itself. And one example is the agent loop we have been using.

So for the ACE project, we've mostly been using the ReAct agent. And in many cases, we just have the while loop, which goes and just runs the ReAct agent in this iterative fashion.

And one thing we tried — very simple change to some of the ACE experiments — was that we just changed some of the way the loop was structured. For example, we can add additional verification inside the loop.

It doesn't have to be just the one-time accuracy check. We could check it, for example, if you have multiple different answers, and so on. And then based on that, you can update the loop more than just the context.

So harness, on a higher level — we did not originally have that term coined. So originally, what we have in mind is the code around the language model that caused the language model to finish a task. So that's the high-level concept we originally have in mind.

We did not call it harness originally. We have this code calling an LM in mind. It could be a loop, it could be a giant file. So it could take various different forms.

Amir: And so I think with the context, obviously, in, let's say, ACE, it was mostly just outputs, right, of the model, the traces, and now it's obviously the code that's around the model, the data that's going in. And so when you talk about the harness, does it also include all the tools that the agent has? And do you update the tools themselves? Or are those tools static as you're in Meta-Harness, updating the harness?

Qizheng: Yeah, I'll give a very concrete example here of what we're updating and what we're referring to as harness. So when I say the code around the language model that calls the language model to finish a task, you can think of a Python file. Let's say runTask.py.

So the simplest harness we can come up with is just going to be a one-line language model call. Let's say in that Python file, the only thing you have is call language model. So that's it. And you can just run this Python file to attempt to finish the task.

And some more fancy things you can do include, for example, doing some context curation by having some few-shot examples or previous experience or rules into the context. So before you call the language model, just add a line, say, curate context. So that's what we do for ACE.

Another thing which is very popular in things like Cursor, Claude Code, is we have this agent loop. So you put a while loop around calling language model. Let's say now you go from the one-line harness to an agent loop as a harness.

So in the loop, you can do a lot of things like verification. You can initialize sub-agents to do a lot of things. So yeah, that's the simplest example.

You can imagine a file which contains everything you have. It could be context. There could also be tools supplied explicitly in this file. And in the Meta-Harness work, we explicitly update this Python file as the harness.

Amir: Yeah, and you update the tools. You can create new agents. All of that are things that might happen as Meta-Harness runs.

Qizheng: Yes, the search space is large as long as you can fit your logic inside this Python file.

Amir: So going a little bit deeper, when I was looking through the Meta-Harness paper and the architecture — so A, there is a file system where you are recording the harnesses and each iteration of the harness.

And there's a code proposer that proposes the next version of the harness. And this is a part that I couldn't really grasp. So in each of these harnesses, you do store the traces of the test results as it ran against, let's say, the benchmark that you're trying to put in. All of these were basically stored in the file system after each iteration.

The one part I still didn't really understand — I know you guys did it, basically took three benchmarks. One was LawBench, the other one was Terminal-Bench or coding bench.

The verification or the test results — you talk about the fact that the actual test results are not stored in the harness and instead you guys just basically save the traces or basically the output.

That part, if you could basically talk a little bit about: let's say the algorithm proposes a harness. You run this against the benchmark, right?

The results — is it kind of RL-style verification, true, false? Or what are the results and how do you propose the next one? What do you pass into the code proposer to get the next version?

Qizheng: Yeah, sure. So I think one of the core spirits of Meta-Harness is that there is a huge file system containing everything we've seen in the past in the previous search iterations.

So this includes, for example, traces we've used to solve a specific problem, for example, in Terminal-Bench, environment feedback from the environment or the agent evaluation environment. It also contains, for example, the previous harnesses we've searched and how good these harnesses are.

So we store, for example, quality metrics associated with each harness. And then we store everything that I mentioned here in the file system. And we make that file system available to the proposer every iteration. So other than agent loop, we have this file system which stores everything we have and make it available to the coding agent.

Amir: Got it. And so walk me through — that coding agent that wants to go search through this and propose a new set of code.

Right now, you talk about the fact that it searches, right? And the searches here is not semantic search or anything like complicated search. It's just grepping, cats, I thought, right?

So do you just tell it, this is the score you got on the benchmark, this was the previous trace, propose something new? Or how do you steer it in the right direction?

Qizheng: Yeah, I think another very exciting point we observed from the results or the experiments is that, intuitively, it seems like you give the coding agent something, you ask it to do a search. It seems that the search will be good only if you manually specify what is going to be a good direction or not.

So for example, if you look at the more conventional training literature, no matter if it's pre-training or post-training, then what happens is that everything is guided by, for example, the reward model, or you just look at the cross-entropy loss and you look at what the gradients are. The gradients are going to tell you, oh, you should go in this direction.

So something we observe in Meta-Harness is that even if we do not have explicit gradients in weight space, we still have a lot of signals we use for guiding the search, the proposer in the right direction.

So one example is that we see that the proposer coding agent is able to look at the failures we have in previous searched harnesses. Like if a harness causes a specific problem in a previous iteration, and given that we give it the evaluation results, it knows what failure is associated with each harness. It's able to form its own hypothesis based on, for example, what part of the harness might be contributing to the failure.

So in this case, it formed its own hypothesis and used that hypothesis-driven approach to search for the next target harness in the new iteration. So even though there is no explicit gradient, we still take whatever reward signal or evaluation metric we could have and make it available to the proposer agent. And that enables the coding agent, the proposer, to form new hypotheses about what we should do better about.

So different from, for example, things like parameter sweeping or even grid search, where we do search just in a very brute force approach in a very fixed parametric space, what happens here is that because we have such a large possibility of what the harnesses could be for the next situation, we must have some way to guide the search agent here using all the signals we have.

But yeah, we are very happy to see that too. It's able to summarize what's going wrong and say, oh, probably I should do something better in a different way for the next iteration.

Amir: Yeah, and so what you pass to it, I guess, is, A, you do pass the test results in each iteration to the proposer, plus you search through all the previous harnesses and find the different failure and search for those things — or maybe is that even hard-coded?

Because I know you basically say the harness that is improving the harness is the harness itself. Or that Meta-Harness itself, it is just deciding what to search for and how to take these test results plus how many different harnesses to search. Is that guided or is that hard-coded in any of that? Or no, it's all up to the Meta-Harness itself?

Qizheng: So one thing we are exactly right about is that Meta-Harness itself is also a harness. So given that it is a harness, there are always things that the human would control first or the human would, for example, initialize as the first draft of design and we can probably optimize later.

So briefly speaking, you're right. There are components that we hard-coded. But I have to say that if you want some optimizer for some specific harness, then there must be design choices that you make first before you can, for example, try it and observe results.

Amir: Okay. And so there's one part of the paper that I also wasn't sure about, because I know in the law bench parts you talk about and the coding parts that basically the search is happening with grep. But I think you do look at retrieval augmented systems in the context of Meta-Harness. And specifically, I think it's in the part where you guys are doing the math evals. In at least the examples that you guys did, how many runs would it take?

A, let's say, on the coding side, what were you guys starting off with? And then how many iterations would this take to hill climb to? Because you guys basically beat some of the other best results on these benchmarks. Was it hundreds of iterations, tens of thousands of iterations? How many times did Meta-Harness run and propose the harnesses to get to those results?

Qizheng: Yeah, so we have one of the full search figures. And if you look at one of the appendix, we have that search figure completed. And we also listed out, for example, the result for each iteration.

So, for example, if you have a better harness as the previous best, then we list it as a best. Or if you have something that's not as good, we mark it as something that we would discard. We would still put it into the history, but it wouldn't be used as the best harness eventually.

So for the Terminal-Bench case in particular, we used one of the top ranking harnesses from the Terminal-Bench leaderboard, which has already been a battlefield right now. So we use a Terminus 2 harness there, as a starting point.

And then what happens is that we run through the Meta-Harness loop. And what happens is that it's not that we get improvement every single iteration, which is somehow counterintuitive, because in the case of, let's say, context optimization, what happens is that if you refine a context every iteration, you usually get non-decreasing performance every iteration, because you get more information over time. The case for harness evaluation is pretty different, which is you could have something worse for the next iteration.

Amir: But it's also similar to gradient. Isn't gradient the same — you might get something worse, which means you have to go the other way, off the vector space or off the graph?

Qizheng: Yeah, there is no guarantee that you get something better. But what happens is that we see that things are improving if you look at the entire search process from a much higher level, let's say every five iteration, every 10 iteration, we can make steady progress.

Because in many of the cases, progress comes in very unexpected forms. For example, probably in the second iteration, we have something that's on par with the first one. But on a fifth iteration, there could be some aha moment, which improves the harness by a lot. So that's something we see in a search process.

It's not something that's gracefully improving. It's more like we have discrete points in the search process where we see bumps in performance.

Amir: Got it. So then, based on what you're saying, because you guys, let's say, in the Terminal-Bench case, were starting with what was already a leader on the bench — but did you remember roughly, order of magnitude, how many runs you ran this on to improve it or get your better results?

Qizheng: Yeah, so the end-to-end experiments were between 50 to 100.

Amir: Yeah, that's fine. I just like the order of magnitude. So in theory, can I just zero-shot this and say, I don't have anything — Meta-Harness, start from scratch and try to come up with the first version to attack Terminal-Bench, and then just keep hill climbing until you do it? Or no, it wouldn't even start in the right direction?

Qizheng: Yeah, that's a great idea. We don't have results on that. Well, probably Yoonho has some results on that, but I think that's something I've been looking to recently as well, which is, what if you have different starting points? How would the search process eventually end up?

Which is, we might be curious if a better starting point will lead to better results in the search process. So I don't have a scientifically verified answer, but I will say my hypothesis is yes. Which is that if you have a better starting point, it gives you better search results after, for example, a fixed number of iterations.

Amir: Yeah, sure. But my question would be, if you didn't have a good starting point, could you still get there?

So if you run it a thousand times, but start off with nothing, will you get the best? Or do you think it won't even go in the right direction from the beginning? Because the starting point had some humans who had done certain things that could get it.

Qizheng: My personal take is yes. But I also believe the variance is huge.

Amir: Your personal take is yes, it could start from nothing.

And so, obviously, then what are the implications? So do you think we're basically a few steps away from getting to the point where humans will have less of this and you can start from scratch and zero-shot?

Basically, are these things truly optimizers or no? Are we really at the point where AI can build the best agents itself from scratch?

Qizheng: So I think that's a great point. And I've been really into this question recently. There's a name for this kind of system recently, or I usually call it auto-research style systems, but there could be different names. But a high-level auto-research style system refers to any of these agent loops, which iteratively optimizes what you have. It could be a harness, could also be a solution to a specific problem, could be some agent, POSC, etc.

The one thing I was curious about recently was, if we can just have this optimizer and it does things really well, does that mean we don't need any human effort in the process? We can just have this optimizer running forever and just keep improving things and you don't have any external signals. So my personal take is no, that is not the case. So one thing I like mentioning when I talk to others is, I was reading a post on X the other day from some AI lab. So they ran some auto-research style optimization for the NanoGPT speed run.

So the NanoGPT speed run is a benchmark which tests, for example, agent or a human's ability to optimize a pre-training pipeline for a 100M scale model within a limited compute budget. So as we know, there are a lot of choices you can make during pre-training, there could be optimizer-related choices, which optimizer are you using, and you could use different hyperparameters as well. It could also be the model parameter or model architecture.

So you're allowed to change everything there, and you're given some GPU compute to try that. The goal is that you want the trained model to perform the best, for example, in terms of validation loss and so on.

So what that AI lab observed is that with auto-research agent, it's only able to replicate 9.3% of best human performance on that specific benchmark.

Amir: 9.3%.

Qizheng: Let's say just let the auto-research agent run forever and just say, I give it this benchmark task, everything you can access there. It's not even on par with what the best human can perform.

And so the reason is, they did some ablation studies. The reason there is because humans are mostly doing innovative research on what the algorithm should look like. For example, the optimizer and so on. So there are a lot of large structure changes or even not eventually implemented a lot of these proposals.

So the agents, on the other hand, are doing very small changes like hyperparameter tuning or very small engineering tricks. So I believe, my memory may be wrong, but they observed that the Codex agents spent 121 H100 hours just tuning one or two hyperparameters in the code. It seems like they're just stuck there. You can't drag them out of these very small changes.

So even if they sometimes thought of doing innovative changes or large changes in their reasoning chain, they mostly would not implement that because they feel like it's too risky, the changes are too large. And if you enforce these changes, say you have to make these large changes just like what the humans did, usually they did very badly. They couldn't properly handle

Amir: Large structural changes.

So at a high level, would the takeaway, at least with today's technology, be that these things are, in a certain way, doing gradient descent — what's an incremental improvement in going in that direction — but fundamentally coming up with a new architecture or reimagining things, they still don't seem to be able to do that. Is that fair — they're good at optimizing things and making small changes, but a re-architecture is not something that you can still rely on an auto-research agent to go do.

Qizheng: Yeah, I think fundamentally, if it does not have external signals or guidance from humans, especially on larger ideas or innovative approaches to things, it is very difficult for these research agents, at least at this point in time, to come up with their own solution.

Amir: I did want to spend a little bit of time thinking, because if we assume that a lot of software code is going to be written by optimizers and auto-research type solutions, how you verify something in correctness. Because these things — the extreme version of it is reward hacking. But even a middle version — if this were really to be software that would be used in production, then you do care about things like security and efficiency of the code and these things.

As you just think ahead in terms of these optimizers and these self-improving agents, are those things baked into it as it's hill climbing and proposing new code? Or do you think that will just be a completely different agent?

This one will just try to improve accuracy, for the lack of a better word — whatever it's optimized to hill climb on. And then another agent will come and check and run tests to see if it's secure, is it efficient? Because when humans write code, while you do have checks on security and efficiency and tests, still it is the developer's job to think about these things as they're writing code.

Qizheng: Yeah, I think software engineering is a slightly different case because it's a very, very well-developed discipline with very mature principles in security testing and so on.

My personal feeling is that even if you have a very capable agent of doing everything, I would still add some additional guardrails in the form of harnesses and probably multi-agent checks to make sure things are reliable. Just like for human-shaped code, we would always have peer reviews and testing in production and so on.

So fundamentally, we still need these capabilities like writing secure code and writing correct code in a model. These are also things that are pretty doable in terms of reward modeling if you're training the model. But ultimately, it wouldn't be the entirety of the large ecosystem of software engineering.

Amir: As I was reading the ACE paper, an item that I think was one of the problems that you alluded to in that paper was brevity bias. Now, I have actually handwritten some of these things, but usually there was this concept or idea that if you put too much in the context window, especially if it's similar at a semantic level, it can confuse the model.

So it's usually good to have things be summarized and very clear and very direct instructions. That goes exactly counter to that brevity bias that you were alluding to. Obviously, the models have improved a lot since I've handwritten this stuff.

So what is your view? Do you think it's actually good to give it as much examples, everything as possible, or should one try to keep it shorter in the prompts to the model?

Qizheng: Yeah, I always believe that you should give whatever you have to the model of the agent. And the core reason here is because I believe there's a fundamental shift in paradigm from individual models to larger harnesses or systems in the past few months.

For example, when we're thinking of the problem of context compression or context slot, we're thinking of it from a model perspective. So the model takes in a finite context window. And in this case, you definitely want the context window to be of low noise and of high information density. In this case, having summarization or even manual clean up your context window is definitely going to be helpful for the model.

But then what we see, for example, in Meta-Harness and many of the recent agent harnesses work is that you can give it everything you have in a form of file system or a database. So the agent could decide itself whatever it should look at in that specific file system or database.

So you're not giving it information in a form of a context window. You're giving it as a file system. And you can just tell it that you wouldn't necessarily need everything here.

When we're working on ACE, one challenge we see is that it is very difficult for humans to decide ahead of time, before an agent starts running, what information is important for the agent to finish a specific task. And I think that is the core reason why I believe you should not manually compress or discard information before you give it to the agent.

Because sometimes humans compress things that humans deem not important. But from an agent perspective, maybe in a long-running task, it is somehow valuable at a specific occasion. So that's why I believe this is the case.

Amir: Do you think existing file systems aren't enough? Or do you think, if I play this out, what you're describing — all agent runs, all agent traces in the near future are going to be dumped into file systems that then future agents are always going to be searching from?

I know for Meta-Harness and the work you guys did there, both the retrieval and the storing was off of existing technologies, but I would assume that amount of that data is going to explode very, very quickly, and you have to do really good searches on it.

I guess let me just make this a bit broader question. The work that you did with ACE, the work that you were doing on self-improving agents and these harnesses that have to get stored and retrieved — if you want to play this out, what do you think are the next steps in this research that are areas of discovery as the research goes forward? And what are the things that you think can be productized today?

Qizheng: Yeah, I would first go to the search question here because I believe agent search is getting really big right now. So we're going beyond, for example, just doing graph or doing embedding-based search right now. So this challenges us to restructure how we store information and how we look up information.

So the naive approach is you put everything in a repo and things are going to be organized in terms of markdown files, HTML files. And it seems like it's working well in many scenarios. But just like you said, as we have a scaled up amount of information, this is definitely not the ultimate solution. And also how the agents should do search.

There have been quite a few recent work and also startup that does this. So for research, there is a work called DCI, the direct corpse interaction, that reimagines how the agents should look of information. And I believe agent search is fundamentally something we should redesign in this new infrastructure layer for agents and so on.

And in terms of where's next, I believe one huge missing part that especially the academia community is not touching on now is we're not treating things like context optimization or harness-related optimization as a rigorous field of study yet.

For example, a lot of the recent work, they propose some new method for self-improvement or for harness engineering by making some very smart changes to how we do things right now, and we have some improvement on benchmarks. So that's what most of the papers are doing. So if we want to make this a rigorous field of study, there are a lot more that are missing right now. For example, we want to fundamentally understand where the performance gain is from.

Why do we get so much better performance if we change the harness? Is it because the harness itself is good? Or is it because the model already has that capability, but the harness is smartly eliciting that capability from the model?

And we also need better evaluation. For example, a lot of the evaluation, the benchmarks are developed for models. They're not designed for harnesses or for test time approaches. So, for example, the benchmarks we use in ACE and Meta-Harness, all these benchmarks are benchmarks developed for models.

So if we have some test time approach on these benchmarks and get better performance, sometimes I just wonder whether this is real apple-to-apple comparison with base models, because we're doing something additionally to the base model during inference.

Amir: So basically, your view or vision is perhaps the benchmarks that we have for models, they should be zero-shotting it, and then we should have separate benchmarks just for the harnesses that we build around them?

Qizheng: Not necessarily benchmarks. But we definitely need better evaluation principles. For example, if we're evaluating this benchmark using a harness that uses a lot of additional test-time compute, that should be taken into consideration.

So probably we should use token cost or maybe some other metrics. So complexity of the system should be part of the evaluation pipeline.

Amir: That's actually a good segue, I guess. And one of your earlier works, Caravan. So in Caravan, there's a model, obviously, as far as a very specific use case, which I'll let you describe, but where, due to latency reasons, you have to run certain tasks with a very small model.

And then there's a much, much larger model that distills. I'll let you talk a little bit more about the right terminology of the smaller model.

And I want to talk about both that idea, and also, I know, because you've done a bunch of work at Microsoft in terms of model routing. Right now, you can let sometimes some of these apps themselves decide, but I do oftentimes feel I'm using a far too complex model to do something that's very, very simple.

How do you think smaller models are going to be trained, used? Do you think these workloads are going to shift to the smaller models? And talk a little bit about that or what your views are on the topic.

Qizheng: I feel like small models are definitely underappreciated at this point, especially when I talk to friends and people I know working on products. The small models are really lifesavers in many of the cases, especially when you have very strict budget, latency constraints, and so on. These models are really lifesavers.

So I think in the case of Caravan, we're treating large models as models that can provide additional signals, information.

Amir: Maybe you can talk about what Caravan was doing, because I think it's a very interesting use case.

Qizheng: Essentially, what we're doing for Caravan is we're doing distillation. So the small model is running online, a very high throughput inference, even with network line rate. We can process a large amount of network volume in real time with that small model.

Amir: You were checking every single packet on network to see malicious bad actors, right? Bad packets that were coming in.

Qizheng: Yeah, exactly. That's the vision we had for per-packet inference. So we want inference to happen in real time.

So you don't have to save that data and run batch inference, causing additional overhead and storage and DRAM, et cetera. So that was the vision.

And then to make sure that small model is running properly — for example, if you have distribution shift, you want to make sure you detect that timely. So the main challenge in the network environment is that you don't have a lot of smart ways to do that. So you could use the more conventional tools, but many of these are not designed for real-time machine learning.

So what we did was we used the large models as additional providers for signals. So we used large models to, for example, provide labels for sample data periodically, and using these labels, we can check whether the small model has derailed from what we wanted to perform. There could be new attacks. There could be new kinds of traffic.

So the large model is used to provide signals so we can detect these changes. And if necessary, we retrain the small models.

Amir: And so are you aware of places where this overall architecture is being used right now?

Qizheng: So I wouldn't say the exact same system is used, because we were using it coupled with some ASIC we developed. So there were a lot of FPGA prototyping. It would be dead for the small model.

But the idea of having the small model running online, and having a large model monitoring it, and if necessary, distilling the small model, is a pretty well-used idea, especially in recent years and recent months. It's simply not realistic to have the large model always on for large volumes of queries and traffic.

Amir: So just to wrap things up, from your own research areas and things you see, are there areas that you think right now are under-researched? Are there areas that you think are exciting that more people should be doing research on that's not being researched right now?

Qizheng: Yeah, of course. I would focus on the research direction I've been working on, which is more like agents and methods for recursive self-improvement.

I feel like the most underappreciated direction right now is evaluation. So we've been developing a lot of benchmarks for models, for example, for individual models: rankings, leaderboards, arena, so on. But there have been so few work on evaluation methodology for test-time approaches, for agents, for harnesses, and so on.

Because we've been doing a lot of extra things as compared to the atomic model. We could use, for example, multiple models, multi-agent systems to solve a tasking and benchmark. But there is no principled way to account for that additional resource usage. And it's very difficult to do an apple-to-apple comparison if you develop some new approach in this space.

And within academia, it seems like if you work on benchmark-related research, people would feel like you're not doing a lot.

Amir: So this has been a theme that's come up in the past as we've spoken with other researchers, which is working on benchmarks. Even though if I go look at most researchers' number of citations, their benchmark work is cited the most, because other people have to go test against that benchmark. But it's generally maybe seen as not being innovative work.

Why is that? Because at a high level, it would seem like if these optimizers and auto-research work, then the humans actually are going to have to be building evals. It's going to be a big part of our jobs.

Would it be fair to say it's looked down upon? Or it's just not seen as — you have to do other work. You can't just be focusing on building evals and benchmarks.

Qizheng: I think the fundamental problem here is that if you build a benchmark, you are not solving an immediate problem.

For example, let's say I build a benchmark for software engineering. Usually, if I want this to be a good paper, let's say from an academia perspective, I would try to prove that this is a challenging benchmark. Existing models probably score 20% on it. So it's a hard benchmark.

But it's almost impossible for me to propose a good solution for this benchmark, because that just eliminates the purpose of proposing the benchmark. So it seems to me that we have been spending a lot of time developing new solutions, new methods, or new models. But we haven't been really thinking about what are some real settings we should develop so we can better understand what models are not good at, and why models are good at some specific aspects they are showing.

So benchmarks are not just for showing that models are doing badly. They're also a very good tool for understanding, for example, why models are doing well on this aspect and not on some other aspect.

I do feel like that is one of the core areas that we should definitely put more effort into.

Amir: Well, Qizheng, thank you so much for coming on. This has been really interesting, and best of luck. We'll have to stay in touch and see your research as it comes along.

Qizheng: Yeah, thank you so much for having me today.

Amir: All right, thanks.