Ep. #2, The Infrastructure of Intelligence with Hanchen Li

Amir Zohrenejad: Well Hanchen, thank you for coming on. I'm excited to have you. Why don't you first introduce yourself and tell us a little bit about what you're working on.

Hanchen Li: Hey, everyone. Hi, Amir. Nice to see you again. My name is Hanchen. I'm a first year PhD at UC Berkeley, advised by Ion Stoica and Joseph Gonzalez. I work mostly on AI systems. So previously I worked more on the serving side of LLMs.

And right now I'm actually actively exploring new things. So my most recent paper was on prompt learning. Basically you're trying to learn a system prompt that improves agent performance based on a series of agent traces.

Amir: Okay. Yeah, so I was kind of looking at some of the research you've done before. So whether it was on the KV Cache side or on some kind of continual learning. It seems like it's an overall thread where you are thinking about or working a lot about things that have to do with context.

Now whether that's context of getting smarter with context or how to basically serve context very, very quickly at the KV Cache layer, save it and reuse it. Overall, do you think context lengths are going to get longer and longer in the models? Do you think this is going to be as you're going to have background agents that are going to go for a long time, do you think-- Or right now with compaction, a lot of things get lost in the context. But do you think we're going to have infinite context windows?

Hanchen: For specific models?

Amir: Yeah, on the models themselves, yeah.

Hanchen: Okay. I'll just begin with what I think. I do not think we'll have infinite context model and I actually don't think it's rational to have like a context window of more than let's say 10 million context. But take it with the grain of salt because I think in 2023 I thought it would not go beyond 100k but now we are at 1 million so probably it will change.

But I just simply at this moment I don't see a reason to extend beyond 10 million. The reason is because I was actually-- I don't know if you guys read the company that was coming out, they released this super long like 14 million context-- It was Subquadratic, I think.

Amir: Yeah, they're based in Florida too or something.

Hanchen: Yeah. In their example they were saying if they feed the whole React repository into the context it's like around that size, like 5 million tokens or something. So I think it's like there are so many ways they can avoid this from happening.

And from an efficiency point of view there's no reason for you to put everything all there at once. It does save a lot of time I agree but I think it's like you can do some smart changes to the workflow so that you don't have to put all of them together in the same time. So yeah, but the answer is no, I don't think it'll continually increase.

Amir: You don't think it could continually increase. So let's go talk a little bit about the KV Cache which is I know a place that you kind of have done a lot of research on. S o first let's talk a little bit at a high level about what the KV Cache is, what it does and then within, go into the pre-fill and decode phases and why we need a KV Cache and why it's such a key primitive when it comes to inference systems.

Hanchen: So a very textbook explanation for KV Cache is that most of the models, or at least the original GPT models, they were in transformer architecture. So in these transformer architectures you needed to perform attention and what attention basically is doing is for each new token you need to read over the previous tokens states, you need to get some information from previous tokens.

And the form which these information are stored is in something called KV Cache. Basically it's like two tensors that are calculated based on past tokens and it's supposedly capturing the relationship between the text and it's very important because it's basically what the LLM actually captures internally as information from the text.

So for example if you give a book to LLM and you ask a question, the reason why LLM knows like this specific line is at page something, it's because they've internalized everything in the form of KV Cache. So that's why it's important.

Amir: So if there's like different attention heads or if there's like a mixture of experts like in those architectures, would each of those experts have their own KV Cache that are separate or is it just one general output of the model?

Hanchen: Okay, so the different experts are actually in the later layers right? Like they're not in the attention layers. So I think it's still like per token, like different architectures, in the end it's still like--

Well there are hybrid models which for some attention heads there's not KV Cache, it's more like a linear attention layer which the size will be much smaller. But for almost all the models nowadays they still kept, like for each token there's still a bit of the KV Cache attached to it so that they can actually do like a really long context inference or like long context understanding.

Amir: Okay, so obviously as the context grows, obviously on the decode phase, once the first token is generated and once you get the next one, obviously the whole thing can be reused. Right? Because it's basically going the same way. It's easy to just append it.

But so I know you've done a lot of work even for the pre-fill phase to try to reuse the KV Cache or chunks of the KV Cache. Can you like walk through like a little bit of like. Okay, so if I, let's say, if I'm working in a code base so the files for example won't change all the time.

So can we basically do like the state of the art inference systems right now? Basically take each of the files and store it and just reuse it or do we use compute and calculate the cache every single time for like the entire code base and the files that are being passed?

Hanchen: Okay, that's a very interesting question. I would like to answer it in two ways. One is like what you would see, let's say, if you're using Claude Code you do not have access to the underlying inference system but you can observe some patterns.

And the other thing is let's say if you have access to all these agents for example there's three agents that you can actually run yourself and you can see how they reuse KV Cache end to end, because you have access to everything. So for the closed auth inference models which you don't have access to, what happens is that if you submit it, actually not Claude Code, let's say just OpenAI API, like Claude API. What you do is if you submit a request with the shared prefix it actually gets a much faster response because it's stored somewhere in GPU or the HBM or CPU memory.

Amir: When you say share prefix, does it have to be all exactly the same or can long trucks of it be the same and it will still hit some of the cache?

Hanchen: According to public information, I mean that's what people know. I think the current technology mostly support only prefix. We can talk about if you have access to customized inference solution, because we used to write a paper back at UChicago and now it's in LMCache, we can also talk about that.

You can reuse a part of the prefix that's not like exact match but for most of the systems nowadays it has to be exactly the same. You cannot have any change. One interesting story that I learned is that Anthropic, when they notice people are using-- Actually I'm not sure. Okay, so I think it's Claude Code. When you're using another model or something.

I don't know the exact story about it but basically it's like there is a provider that says, "okay, if you're using an alternative provider, let's say you're using Kimi as a provider and myself is Claude Code, I don't want you to use Kimi." So what they would do is they would actually insert little spaces in the middle so that what happens let's say if a million--

Amir: It would never be a cache hit.

Hanchen: Yeah, if you have a million context and then if I just inserted a small space in the very beginning, then the whole part is gone. But it's very easy to fix. But if you don't fix it you actually have zero cache hit rate, like almost zero cache hit rate. So it's basically the story you have to have exact prefix but then if you have access to the inference system you can do a lot of more interesting things.

Amir: Okay, but so like if you had to go build an inference system using like open source tools and the best, state of the art kind of LMCache technologies, like even as it loops, it's not like the entire basically prompt that gets passed in with the entire context will be exactly the same because obviously it will change every single time. Right?

Hanchen: Are you talking about like within one session?

Amir: Yeah, within each session, each like a single inference call, as they call into the API, there'll be some change usually. Right? Like most of the data in the context will be the same.

Hanchen: Yes.

Amir: Like it'll just go do one tool and then report back that this is what I did.

Hanchen: Or you'll do another tool call.

Amir: Yeah, exactly. It'll come back and report what happened. But then like the code base didn't change probably that much. But there'll be some new information. So right now when you do that, do you reuse the KV Cache? Does it get used again? Will that be a KV Cache hit? Or will that be--

Hanchen: Yeah, like if the prefix is the same then like you will be KV Cache hit.

Amir: Is the prefix not the entire prompt that's getting passed?

Hanchen: Oh yeah, it is. It's basically like you have some starting point like, from position number zero, all the way up to like some point. If it's a match then like you can just basically use KV Cache, if it's stored. That's normally how it works. And if you can reuse it, it can save a lot of the compute. And yeah, it's a great thing to have KV Cache hits.

Amir: Yeah. And so like just very naively, is there anything like imagine-- Like we're a large company and we have a code base that a lot of developers are working on it. So in each session most of the code in the code base will be the same.

Most of the tools and the skills and all the things that are going to be in our Claude Code will be the same. It'll just be the output of the tools and the code changes that will be different. Can we not just like pre-calculate everything and just have it-- Like can we just compile everything into the KV Cache and just like every time there's a new code commit just like be like, hey, update this, compile it once and then be super fast and save a lot of money?

Hanchen: So in my understanding, what you're proposing is like we can have like a big state which is like basically stores everything in there?

Amir: Yeah.

Hanchen: Oh yeah, that's actually a great idea. That's actually, what my old advisor was at Chicago, Junchen Jiang -- Like, he actually like imagined this all the time. When we were like, like at UChicago, he was like imagining like, if we can just store this.

So the problem of this is this, like the current way of storing things is like doing this KV Cache format. And the problem here is that like if everyone in the company is using the same code base, I could just change the prompt later on, right? Like, okay, that's actually a good idea.

I think that's probably like, okay, it depends on how long the context is. Like, it doesn't get infinite at all. Also, like, my understanding is that if it gets super loud, let's say 1 million context, the performance still deteriorates. So it's worse than just manually fetching new things. But I think you're right. Suppose we have a model that just doesn't care how long it is.

Amir: Yeah, because then again, coding is the example. It does seem like, again, to give analogies from previous times. If you think of the KV Cache as the binary, you can do it beforehand, compile it and then just store it somewhere and just only update it once there's a code change or a tool change or something and just pass the new stuff that goes down. But instead of all of this happening at what effectively is at runtime inference time--

Hanchen: Okay, I was actually thinking about what you were proposing. I think there's a problem that we're missing here. So for example, if I'm changing a part of the code, if we too are in Hackathon, we are on a single code base and we are just changing part of the tool. Then what will happen is if you change a bit in the middle, should we recompile?

Amir: I think we'd have to recompile.

Hanchen: But then every change that I make, I have to recompile the whole thing then--

Amir: Yes. But you like, you know, with compilers through the technology, not to compile the whole thing just like the one part. Haha.

Hanchen: Oh okay. That's a great idea.

Amir: Because right now when I have like, you know, with compilers, the way I work, it's not like I recompile everything. That's why it compiles so quickly. Haha.

Hanchen: It's definitely an amazing idea because like it brings me back to one of the papers I was going to say, like to share non-prefix Cache. It's one of the papers that we wrote back at UChicago. I think it's now LMCache. It was called CacheBlend. So what it does, it's basically saying if you have multiple chunks of the text, what you can do is precompile each part separately and then combine them at runtime with, let's say, one fourth of the compute that you would need to preview the whole thing.

The preview is calculating the KV Cache for the whole text. So with that idea, you probably just need to recompile a small chunk and then selectively recompute with one fourth of the whole compute. But still you would need to do one fourth of the recompute of a million context, which is still a lot of cost. So I think that's why people are still not in it.

Amir: Okay, so I mean, as we're talking about KV Cache, like talk about Continuum, which is like one of the papers that, that you worked on this year. So obviously caches in like traditional computing, there's a TTL and basically they get kicked out of the cache.

You proposed a way of basically bringing the idea of TTL to the kind of inference for the KV Cache. Talk a little bit about like why that is useful. Talk a little bit about why it's not as, as I was reading this, it's not as simple as just setting a number and okay, I passed this number. I'm going to push that-- Like counting that number, the benefit and the cost is quite complex. So we can kind of talk a little bit about it.

Hanchen: Okay, this is the background of the paper is basically we are doing agent inference. So like, we are not doing like the traditional chatbot inference. We're having agents. So basically what the agent does as you were talking about, I think we got to like, bit of it. But like, just like going backwards is about basically like you, you have an agent, you give it a problem, it goes in the code base.

At the moment we don't have the fancy technology that you just envisioned. So what you do is actually you make a tool and you call a tool, for example, let me list out the files in there. Let me cache out a file in there. And then you kind of like append the tool output to the agent and you kind of like continue this step for like 50 steps and then finally you get an output.

So that's a rough load right there. So what the problem was was that during this tool step, when the LLM says, okay, I'm gonna give you this tool, and you have executed it, so the agent goes on. Okay, let me take this I need to do this. Let me just go ahead and execute this.

So during this time the inference engine needs to decide what's the fate of the KV Cache during that time. And there's a lot of complex logic here. So basically you can put the KV Caching in GPU memory. It actually was not what the current open source inference engine was designed to do.

But this thing means like you cannot use the space for other requests at the moment. And so it potentially has to be like a really short amount of period. But you can also store them in CPU or store them in disks. Like there are different places you can offload them to. But basically the idea like in the paper was that you need to pick the optimal time based on a variety of factors.

For example one problem it brings on, it might get a little bit too technical, but I'll just give a brief overview of this. So basically what happens is if you kick it out, even though you might not lose it, if you have an offloading position in the underlying memory hierarchy you will still need some time before it gets back because other request has to finish.

You cannot just randomly kick other people out during inference. So it introduces more delay at each tool step. And if you have 50 tool steps it's actually a lot. So we did the calculation. So basically if you have 50 tool steps, if you have plenty of offloading locations you will not have much throughput decrease, but the delay would actually increase.

It's kind of like we are trying to go to market and just eat plenty of food, but there's only one chair. So what happens is instead of you finishing the meal and now I take it, it's kind of we take turns to each have a scoop and then in the end we both finish at the end. So that's what's happening with the--

Amir: The overall GPU was the same. It's just like the latency flaws are--

Hanchen: Yeah, the latency for us all increases. It's like a scheduling change. I think it's one of the more recent works on agent serving because I don't think there were agents back in the day. So I think we were actually the first ones to actually do experiments on SWE-bench and a real agent benchmark. So it's a kind of cool thing.

Amir: And so the benefit is clear. So what's the cost that they basically the memory is going to stay longer, full-- Like how do you calculate that on the fly? Like in a very generic way that will work for any like, let's just stick to coding, for example.

So let's say like you're an inference system that only cares about coding and let's say you have to serve all these sessions and you generally know that there's going to be reused code bases, there's going to be reused tools. The outputs of those are going to be different, those are going to be reused skills.

So the costs are clear, the benefits. How can you calculate that on the fly in a way to like-- Because then you have-- Is the TTL a single number or change dynamically for the give and the module?

Hanchen: Oh that's a very good question. It changes based on the workload and based on the tool call that it is spitting out.

Amir: Yeah. And so basically as the entire call gets packed in or based off-- How do you calculate that on the fly? Like what are the numbers that go in to determine the cost and the benefit to decide like what the TTL should be?

Hanchen: Okay, so before I start, this is like disclaimer that I don't think the method that we're proposing is a de facto solution. It's just like this is like one of the solutions that's out there. But like I think the input and output space is defined by this solution. It's probably the de facto solution.

So for example like the inputs that we taking in are include first the function call type or like-- Basically it signifies how long the function will probably be. And second is the dynamic of the workload. So basically if you have a really long horizon task which every agency can take multiple steps, the bloat problem is going to only become more severe. So you need to take that into consideration.

And I think the third thing is also like the model and the GPU type that you're running on. Basically it calculates how fast is reloaded back, how many memory is going to take. And yeah the output was basically like how. So we try to keep it simple because by the time we were working on it we wanted to be something that an engine could easily implement. We don't want it to be a super complex thing.

So in order to keep that minimal interface, what we designed was basically a time which you can hold that KV Cache inside GPU without clearing it. And after that time is up you reduce the reference counter by one and then the inference end and continues what it is originally it's going to do. If it's want to clear it out. You can clear it out, it's fine.

But that output is only to maintain a very minimal interface with the existing serving engine. Because let's say Vyond developers, they don't actually want to maintain a large corpse of scheduler code because you know that's what makes everything messy and that's why they designed the Vyond V1. So you know we try to keep them interfaces as minimized as possible.

Amir: Okay, my final question just overall on the KV Cache before we kind of move to like more continual learning and things that are happening in context, are there ideas, approaches, innovations in the KV Cache, again, outside the labs that are kind of coming that you're kind of excited about or kind of new direction things are going?

Hanchen: Yeah, I'm not personally like leading a project in this but I'm like involving several projects, agent inference and stuff. So I personally think it's still very exciting problem for agent infrastructure. So basically I think we should think about it in this way. I think there are two ways that people can contribute to agent inference.

One way is to write very architectural level code which basically CUDA kernels that actually improves kernel performance that would speed up everything by 20%. ideally that's one way to do it. or you can do what Deepseek did was basically our new architecture style. As long as it works fine, it's very good.

So that's the first way if you're going to contribute. Second way is like by improving the scheduling layer. So basically it's the orchestration layer which was what I was mainly working on before. Also in Continuum it was like a scheduling level change.

So I think the problem with scheduling level change is it's very workload dependent.

So basically what happens is for the workload that we were doing, the only reason that we are still able to write a paper in 2025 it was archived in 2025s, was that agent was not here in 2023. If it were here people would have already written the paper and there's no innovation to be done.

But the good part is that in this last mile delivery of agent performance you can actually get like a 2x, a very easy 2x or 3x gain just by reducing the bubbles. Because if you do not have the scheduling you're just cleaning out caching every time. I don't know if you heard of the term MFU. Have you heard of it?

Amir: No.

Hanchen: Okay, so basically MFU was basically like how much compute the GPU is using. So it's basically like GPU has like memory bandwidth constraint and compute constraint. So a lot of the time when you're doing pre training, you're probably trying to optimize the clusters MFU. It's like trying to use as much compute as possible so you don't waste your money on the GPUs.

So what happens for scheduling level is if you do it really wrong, you could get a very high MFU but having very bad performance because you're doing repetitive computing over and over again. I think this is a shift of mind. You are not optimizing for MFU but instead you optimize for the real output delivered.

And with agent workload people are talking about CPUs. CPUs are running low. I don't think CPUs are running low but they could be a bottleneck if you write the software wrong. So there are a lot to be done on the software side for these last mile delivery of agent performance. I think it's exciting. So people should still work on this area.

Amir: Any specific kind of approaches that you are excited about that others are doing?

Hanchen:

I think the approach is like people should just try to run real workloads and they will see because a lot of times the problem in academia is someone did a way of evaluation in some workload and everyone just tries to optimize for that workload.

So I don't think that's true. So for example for the new papers that we've been writing, we actually collect you know like real agent trace from Claude Code and stuff.

Amir: Yeah, so we can jump in. You guys took like 400,000 traces of, some very high number of traces--

Hanchen: Oh no, no, no. That was not collected by us.

Amir: But you guys went and analyzed them?

Hanchen: Yeah, we went and analyzed them. Well actually Claude analyzed them. That blog was actually experiment.

Amir: Okay.

Hanchen: I was trying to test if Claude could write a blog.

Amir: Okay. Did it do it?

Hanchen: I think I changed probably like half an hour like myself manually. And the rest was basically prompting. I was trying to see if I prompted enough, will I be able to get things. It's actually pretty good. Okay. I think the way to put it, someone said it like on the Internet, it's like it's a very smart intern.

Amir: Okay.

Hanchen: So it would do whatever you tell it to do. But it doesn't have the sense of what should be put in where.

Amir: Okay. Kind of like switching gears a little bit to continual learning just again, big picture, three years from now, five years from now, whether it's agents or physical AI, we want-- Because as you talked about, the difference between an intern sometimes and like a very smart AI which can do crazy math an intern can't, is that then an intern can learn organically on the job and you can talk to it and it knows when to ask you for feedback and that learns from it.

Do you think most of learning will be happening in context and in memory, or is it going to, before continual learning to really work, these things have to get pushed down to the weights of the model. And we're going to have models that once a day are going to get updated or once a week get updated? Or do you think it's going to be mostly just like these are things that we're going to solve at the context layer?

Hanchen: Oh, that's a very good question.

So, first of all, I think continual learning is actually an abused term.

Amir: Okay.

Hanchen: Like, I think continual learing is a really abuse term. I've been.

Amir: Dario agrees with you. Haha.

Hanchen: Dario agrees with me.

Amir: I saw he was saying continual learning is used in a way, like, t he solution might not be what people say it is. It might be in context.

Hanchen: Okay, so I actually wrote a blog about this. Follow my Twitter if you're interested in the blog. Okay, so basically the idea is that I think I take it more from an engineering approach. Like, basically I have a problem I'm trying to solve in some way.

Amir: Okay.

Hanchen: So I think when people are talking about weight update and context, they are talking about two type of problems. Let's say I have two scenarios. One is like, I'm Cursor, I'm trying to make my model better. After like three months of inference, I paid so much money for the GPU to like run, like Composer. Or like, I paid. I don't know if they pay premium for Claude API, but like, they do pay money.

So if I run all these, can I get something out of it? So, I think that's the first use case that people are talking about. That's where wait, update is probably more useful. And the second use case is probably just like, I'm talking with Claude every day. Can it just learn my preference? Right? Like, I do not spend like a billion dollars every quarter on compute. But I want it to be smarter.

For example, I told it like, when I write a blog, I do not like you to use this style. Right? Like it's, it's like two different scenarios. I, I think these are completely two different scenarios which people would, would call both continual learning. So I think I would first like to make the distinction here.

I think there are two parts and also the problem of whether you need to choose weight update and reinforcement learning or context engineering. It becomes actually very clear after you separate the prompts.

Amir: Okay.

Hanchen: So I think it highly depends on what kind of signals you're taking and what kind of things you're trying, what kind of ability it's trying to capture.

Amir: Let's imagine we have agents. Let's stick to the same coding example. If we have a large code base and then like we have a bunch of people working on it. We want it as it goes. And when it tries to do something and we give it feedback or we tell it, it has to do things differently, you can either store those somewhere in the context, let's say, or basically like there's items that can just get kind of injected at the right time.

And we can talk a little bit, I guess. Actually, since we're on the top, let's talk about ACE, actually the agentic context engineering. You guys were proposing from I say what I was reading there, that you want to develop like a playbook, let's say that kind of basically gets updated over time and that will be the context. And as the agent goes and does things, it updates that playbook itself. Right?

And you have a few different models, I think as a curator and whatnot that basically goes and basically sets it up. How is that better than just taking everything that's a log, storing it in memory through in a graph database or something and just grabbing the right pieces and just injecting it back? And like how is that different from just what memory is right now?

Hanchen: I think ACE is like, the playbook is actually a form of the memory. And the argument we are trying to make in the paper is that first of all, format is really important. Like, because like a lot of what people were doing before this is basically they were doing like some kind of evolve pattern. This is like you take a system prompt you just generate the new system prompt based on older system prompt and new feedback.

But the problem is you're generating a whole new system prompt and that behavior is actually super uncontrollable. So basically you're taking an intern, you're asking, "hey, this is my code base or this is my draft. Please write a new draft."And they're probably going to just mess it up.

But instead would you just tell the intern, "this is a draft. Can you just update one section, add a bullet point at a time," and then do it each time. So the ink chain is actually much more controllable in a sense. So this is what essentially ACE is trying to do. But if you have a well maintained memory, I'm pretty sure if you tune it enough it probably will get a very good result as well. So yeah, I think format is important. That's part of what--

Amir: And the playbook, what is the format? Is it like--

Hanchen: Oh, okay.

Amir: Yeah.

Hanchen: So still keep in mind it was in August 2025. It was before Claude Memory actually. Before Claude Memory actually. Probably before CLAUDE.MD actually. So basically it's similar to what Markdown is doing. Basically saying let's have different sections. For example, like the code snippets you need to record all the important code snippet that you could potentially use in the future in there.

And you have common mistakes, like mistake you've made throughout the rounds that you had. And there is like API specifications. It's kind of like these sections and for each section you can add an entry that says, "okay, this is what I need to take note. This is really helpful. Like this is very helpful. This is like medium helpful." So it's basically like preparing the form of a memory so that it knows better how to use them at one time.

Amir: Okay, so let's say then it's a markdown file and I'll take a coding example, something that actually happened recently to me. So we had bought a data enrichment API provider and I basically we had our internal CRM. I was like go and hook this up and go enrich this data from this API. And I didn't know the API was behind a paywall that you had to go log in somewhere to get it and then it couldn't do it.

And then it starts spinning and spinning and spinning. I was trying to find other open source code samples that somehow used it and was inferring like Claude Code was trying to infer. This is how this API is called somewhere else. So I'm going to go try to do that and then I'll like stop doing that. Like here is where you can access it without logging in to anything.

How would that get stored somewhere so that it doesn't make the same mistake again? Do you guys have another thing where as these logs happen, something else goes and processes these and updates the MD file. Like, how would it like recall that over time? Like how would that get fixed?

Hanchen: This is a great question. Yeah, we were talking with an SRE firm. Basically they have the similar process. Basically, SRE is you're debugging every day. So what it does is like you're spending a lot of effort and then you finally find out the root cause. So what do you do is put that root cause, like summarize it. You have a refiner.

Basically it takes this trace. It says, okay, let me look at the trace, what I've done, what happened, and let me see, what's the takeaway here. The takeaway is like, okay, when I have this problem, I should not do blah, blah, blah. It should actually call an API because all the effort previously was like wrong. It's messy. It didn't lead any anywhere.

But you know, like my, my owner Amir told me that like, this is what I should do. And I suddenly work. And then like the curator sees this text and says, okay, let me create a bullet point that says when I want to do like login, I go to this website and do such things. Like if there's a code snippet which is like say, do the example code snippet that work and just plugging it in there. And then like it would, it would just like next time he reads this and he sees the problem, he just knows.

Amir: Got it. And like generally, how often do you think this would run? It would be always in real time looking at what's coming out. Or would be like once a day or like how, how often would you go and say like, "Hey, these are my learnings from the day. I should update my playbook."

Hanchen: Okay, this is a great question as well. I see two pattern here. So like one way to do it is like you can do it per day. Like it's all like a procession. You can do a procession basically. Like that gives you the fastest learning. But like also in the same time, it probably consumes more token than before.

And the problem with running them, okay, that actually like, so you can actually run it let's say once per week or something. We can collect like 2000 traces, runs on that and just run some prompt learning method on this on top of this. But the problem with this is it actually runs much slower because you now need to process 2000 at the same time. So that leads to the--

We were thinking about this problem as what you were saying. And then it naturally leads to the Combee paper, which it's basically trying to say that's the paper I was mentioning. It's basically saying like, if I have 2,000 traces, I'm trying to learn a system prompt, can I spin up parallel curators and how do I aggregate them in a way such that the information loss is not too severe?

Amir: Yep. So actually I think this is good to kind of switch to Combee. So again my understanding with Combee was like similar approach to Trace, but just like you have many, many agents that are doing this stuff and are continually creating these logs. So do you think that over time like this contact, you're going to have all these agents running and you're going to have streams of this stuff coming in--

Hanchen: Yeah.

Amir: And then something there is going to be continually updating the context of all these things and it's like injecting-- like with Combee right now, like is it just a lot of logs all in the same place and then like. Or is there some structure in terms of how you were going to store it and retrieve the right ones and give it back?

Hanchen: Right now it's very primitive. It's the same thing. You take all the traces, but like it's like from 2000 traces, How do I, you know, do them fast? Like I think--

Amir: What do you mean by "do them fast?"

Hanchen: Okay, so how do I create a system prompt that is really fast? Like I do not organize the traces. It's focusing on the step where like I have like traces store, which I format. And I'm just trying to produce a system problem. Like I don't care about the internal differences.

So the purpose of the paper is mainly in the aggregation method. So basically it's trying to say if you group them and then you duplicate some information in there, it actually solves a lot of the problems that you would face. So yeah, that's kind of the point. But like I agree with you that if you have complex, more complex methods that actually fits your application scenario, you may get better, but that's highly application dependent.

Amir: I believe when I was reading ACE, like you guys specifically say brevity and summarization is bad, right? You don't, you don't want to have brevity, summarization. You want like the full parts of the relevant traces all to go into the prompt, right?

Hanchen: Exactly.

Amir: So as you think about like multiple agents and multiple, like creating a lot of traces. And as you said, context window is not going to be infinite. It shouldn't be infinite. You're very quickly going to run out. You only have to like, you have to identify what the relevant pieces--

Like right now, like you can't pass everything that comes from all these logs. When the curator works, isn't it in effect summarizing things before it stores it? Or how does the curator store things where it's not summarizing but it's keeping the actual logs? Because that playbook, if it's going to be just like an MD file, it can't grow forever.

Hanchen: Yes, that's actually one of the main problems that we were having. Because I was iClear and people-- This is actually one of the most asked questions there. So I think the concern here is that when you actually have too many traces, it's probably actually natural to separate them into groups. Because let's say if you have 5,000, probably some of them are redundant and probably, some of them are related to one type of query.

It's kind of like some of them are like coding based on when you're using Claude Code. Some of them are coding questions. Some of them just like some chores like you just need to do. And yeah, like you should separate them. And we are not. Like it's it's because like it's just a paper. It's not a product. But I agree with you, you should do this.

Amir: Okay. And just kind of like overall kind of big picture, and again, the stuff is moving pretty quickly but like are there specific areas within again, because I think a lot of what you're doing is like helping kind of this growing context to help learn from it, to serve this faster, to store it any in terms of your own research or research that folks are doing that you think is, are pretty exciting and you want folks to kind of go check out or things that are exciting you right now?

Hanchen: One thing I think people should really check out is like I think benchmark is underrated.

Amir: Benchmarks are underrated?

Hanchen: Yeah. I think good benchmarks are highly underrated.

Amir: Okay.

Hanchen: The reason is because benchmarks are actually a very good way for academics to keep being at the frontier.

Amir: So that's interesting because like there is also you hear of like, oh, everyone's benchmark maxing and everything does well on the Benchmarks.

Hanchen:

So I think there are two type of people in academia now. One is like, let's just do benchmarks. Let's make benchmarks. The other one is like, okay, if you do benchmark, you're actually giving up . You're giving up improving the models.

Amir: Okay.

Hanchen: So I've actually had like very close friends telling me about this, both at Snapper and Berkeley. They were like, okay, I don't want to do benchmarks. Benchmark is like when you're giving up. Like, I think it's right--

Amir: When you're giving up on like innovating?

Hanchen: Yeah. So I'll give the reasons why I think benchmark is still important because, you know, you can actually show that the frontier models are not good at something and you need improvement on something without 10,000 GPUs.

So this is why I think benchmark is good. The reason why they say you're giving up is because you're saying, I cannot beat Claude. I cannot beat GPT5. And that's why they say it's a bad thing. But you know what, I probably agree that we cannot beat Claude. I don't have 10,000 GPUs.

Amir: So wait. I think even if you in an open source setting come like, I think when Postgres was written, IBM's databases were much better and faster, but it was just this was pretty close and it was open source and it kicked off something where like now everyone wants to use this. So do you think there's a perception of like, oh, if we do something in the open source inference realm or model realm, if it's not as good as, you know, the state of the art close model--

So if you get close, that's still pretty good, right? Or is that not the perception within academia right now?

Hanchen:

I think the problem with AI is that it's too much of a capital game now. GPU is way too expensive for us to have.

For almost every-- I'm actually interning at a startup.

Amir: Which one?

Hanchen: Anuttacon. Do you know Genshin Impact?

Amir: Yes.

Hanchen: It's created by the same founder. So basically they own, I heard they are pretraining models. So basically, you know, a random lab would have like much more GPUs than all the top 20 CS grad schools combined together. They would have more GPUs than that.

It's just like there's no way you're going to pre train your model without a lot of resources. Like, but like you can try. Like I think it's something that's worth trying. But you know like there's no reason for me to start pitching other people because no one will give me 2,000 GPUs.

You know if there is a chance I think viewers should still do it. But like if like in most of the time I think users can just work on benchmarking and stuff because it's, it is actually what you can show that the models fail in a specific way. For example I co-authored a paper, I'm actively running the community for a thing called Frontier-CS. It's basically a long horizon task.

It's for open ended problems. You can actually show Claude actually they fail so bad on these type of benchmarks and they actually doesn't perform that good even compared with Kimi. So there are tasks that, there's interesting findings you can have just like by spending 10k API credit and you can just find a lot of interesting things. It's one of the cheap ways to do research.

Amir: Yeah, so I agree with you. So I do want to talk a little bit more about benchmarks because it seems that there's obviously a lot of benchmarks, a lot of benchmarks coming out. You're basically saying that there's a maybe stigma and academia that if you're working on benchmarks you're not like innovating let's say.

Hanchen: I don't agree with this by the way.

Amir: Yeah, I know you don't. So what does make a good benchmark? Does it have to be difficult? Does it have to be representative? Does it have to have a lot of different questions, does it have to be diverse or what, what is a good benchmark? What are the places where you feel like there's a big need to make good benchmarks that don't exist right now?

Hanchen: I think my mentor Joey had a very good point on this. He said you want to have some insights on the benchmark. You don't want to just say benchmark and then all models fail. It doesn't teach us anything. So I think what he actually wants to teach us is you want to show that the model fails in certain capabilities. And you want to hint.

So for example if it's Anthropic, pre training, post training lead sees the benchmark, sees the results, sees the insight, you can say oh we actually did not train for this or like oh we should have done this. It's just we haven't and then you want to have this.

After they see this they're like okay let me actually start going back to my training process and do some changes to process. And then when they normally do it they just buy the data. Right? This is what happened to Deterministic Engines and stuff. They just started buying a lot of data.

Amir: It is a big business selling environments and data.

Hanchen: Yeah, I know. Yeah. So I think it's like you want to make them realize they actually lacking a fundamental capability and you want to show them why and you want to have like a complete reasoning chain of like why they fail.

Amir: Okay. And like is it trial and error? How do people find how? Like let's say like if you're trying to build a benchmark for code or some other task and the person who's sitting there trying to come up with these questions is do you just try different things, see which one is fails at? How do you get intuition of like oh, these are the things I should try for it to fail?

Hanchen: I guess I'll just describe the process--

Amir: Yeah. Why don't you talk about Frontier-CS, talk about how and who is involved with it and like the good benchmark question creators. What's the process of finding the intuition of like oh, I should go do that?

Hanchen: So, so, so basically okay, so Cho Yong is another first year PhD. He was the main driver of the product. Basically he was sitting right behind me. So we were talking about it. We were probably the one-- Okay we not the only two but like we both had a little bit-- Like I had a little bit of competitive programming experience. He had a lot.

He's a world finalist in ICPC. I don't know if you like it's like the biggest event. So basically he was talking a lot with other competitive programmers in US and China and stuff. And one thing he realized is they are already benchmaxed on the code forces problems which is standard. You have a problem, you try to solve it and you have a correct solution.

If you get a solution, you're good. So this is the one problem. But like for there's another type of problem which is more novel which basically means you can submit multiple programs. There's no optimal solution. The question is designed for you to not being able to have the best solution in mind when it's at the very beginning which is more like a real world. For example.

Amir: Yeah. Someone in the hallway tells you hey can you do so and so? Haha.

Hanchen: Okay, it's always like Heavybit. You raise this much and how much can you return? Give me the optimum amount I can return. You can't actually return infinity. Right? And then what Claude, actually. Okay, I got to go there. But basically we were trying to say, okay, Claude was actually never mean to. It was never trained on this. It was not meant to do this. No one--

They train on a lot on street bench, terminal bench. And if we make something that it has not seen before, will it just fail? And in which way will it fail? And it actually fails. So the thing what Claude does with Claude Code is it keeps thinking forever.

Amir: Yeah.

Hanchen: So if you show it up a problem, it says, okay, I'm learning to get 100 or zero, because that's what rewards us. Let me think about what should I do with this? Oh, I don't know. I don't have a solution. Let me think more and let me think more. Let me think more. In the end, it's just like, keep thinking. It's just like we said, at 30 minutes time window for it to think and just keep thinking for like 30 minutes. It doesn't give you anything.

And we'll be like, okay, you should at least give us something. Right? We actually explicitly tell it in the prompt. It's not like a systematic evaluation. It's like we explicitly tell it in the prompt that you give us some solution first and then you can keep improving. It just does it.

It's kind of like the super sensitive good grade kid. Just like, oh, if I don't get an A, I'm going to fail. It's like this kind. Claude is just being like, I think it's like, it shows like a-- It's like personality flaw, like what Claude is doing.

So, yeah, like I think it's quite interesting to see these models fail at these problems.

Amir: Other than CS, is there any other benchmarking that you're involved in or know about that people are trying to do? Other than straight coding, obviously?

Hanchen: I think they are. There are. I want to talk about, like the very new benchmark. It's like in computer science, which I think. Okay, I think it's really popular. I don't think people should take care of it. It's called Program Bench. It's just coming out. I think the scenario is actually--

I don't know what's the meaning of the scenario because I don't know if you've heard of it, but the brief intro will be just like you're giving the specification of a package and you are asking ARM to just rewrite the whole package, whatever. Then you test if they pass all the tests. But I mean software engineering doesn't work like that.

It could be a good capability test, but software engineering simply doesn't work like that. So it doesn't show productivity or anything. So I think it depends on what you want to do. But it's research so I guess people just want to show if it has capability to rewrite the whole program. It's cool, but I don't see a practical meaning, but I think it's cool. I have mixed opinion on this.

Amir: Yeah. So again Program Bench, the way the benchmark is, it's basically entire libraries, packages and it's like go rewrite it?

Hanchen: Yeah, it's like it doesn't give you anything. It's like, "here are the specifications, here are the tests you need to pass. Write me the whole thing." I think it's quite interesting idea. If we can write it, it might mean something special but I don't see a practical use case--

Amir: Well, I think it will make libraries useless. Right. Everyone will just-- These things will just spin up their own libraries because we do sometimes talk every year about right now just think of NPM or all these Python packages and libraries that are out there. There's some discussion on the developer community of like basically whether and historically these have been open source reused and then like businesses are building on it, but also like the fact that they reused makes it much easier to go build some software.

But at the same time they do introduce some forms of bloat because you're basically bringing in all these code bases that you didn't effectively write. So there's some talk of like this instead of like basically importing libraries everything every time.

Hanchen: Oh so you just rewrite everything?

Amir: Everything, every time.

Hanchen: Okay. So that, that could be the use case. Like yeah, yeah.

Amir: Yeah. So like basically you don't, you don't-- Like imagine without any imports outside your own code repo, everything is just like from your own code repo, but only the pieces that you want. Right?

Hanchen: It would be a really interesting thing if you actually do that in the future.

Amir: Yeah. I mean there's going to be a lot of similar code in different places to say like manage time strings --

Hanchen: At this point of life, I still think it's a waste of energy and efficiency.

Amir: It's definitely a world of energy for sure. Someone should do the carbon footprint calculation. Haha.

Hanchen: I think the people in Nvidia released the video. They rewrite the whole Pytorch. I don't know what it means, but sure, you can rewrite it. We are still in the phase where we are discovering whether it has real meanings. So I think it's good that people are trying our experiments.

Amir: Cool.

Hanchen: I don't understand what it means at the moment, but we probably will discover what it means in one or two years. It's exciting.

Amir: Great. Manchen, this has been great. Thank you for joining us. I really enjoyed the discussion. And we're keeping an eye out. Let's see what you guys at working on Sky Lab.

Hanchen: We're actually making new benchmarks and stuff. So yeah, I'm excited. I'll keep you updated.

Amir: Yeah, send them to us. We also have a bunch of newsletters and whatnot too. We'd love to have those showcased.

Hanchen: Yep. Thanks a lot.

Amir: Thank you for coming by.