Ep. #4, The New Big Data of Inference with Junchen Jiang

Amir Zohrenejad: Junchen, welcome to the show. Excited to have you here. But before we get started, why don't you introduce yourself, tell us what you're working on.

Junchen Jiang: I'm Junchen. I'm the CEO and co-founder of Tensormesh. Previously, I'm a faculty at the University of Chicago. My team was the team that originally created the open source library called LMCache, which is today the de facto open source layer for KVCache storage and management.

Amir: I think for folks who are computer scientists, but are not experts on AI inference internals, why don't we start off where you give an overview of the different components of inference systems, and specifically, how the KVCache fits in there, and why it's such an important piece of the inference ecosystem in your view.

Junchen: Well, we didn't invent KVCache. KVCache was a concept that was built in transformer architecture. So first of all, we didn't invent this term itself.

And KVCache is sort of a unique concept today because of many factors. First of all, it's key to reducing inference cost to improve throughput of an inference system. And it's also a key to reducing inference delays.

Now, more than that, KVCache is also a very important type of data that encodes model's attention information. So KVCache can serve many different purposes.

Now, what's unique about KVCache today is — I did this analysis the other day. If you look at a Google trend, of Google search interest trend. And I compare KVCache's trend from last year to today with the trend of many other influence optimization techniques like MoE, LoRA, sparse attention, and things like that, even page attention. You can see the interest about KVCache just grow much faster than any other techniques in this area.

So KVCache is definitely getting a lot of interest, but I actually think people have a lot of misinformation about KVCache as well. For example, I would say KVCache is purely an inference time concept. And a lot of people ask me, can KVCache improve training? It cannot. But KVCache is very useful too in inference time.

And people also have this misconception about KVCache being KV store, key-value store. It is not. KVCache is a combination of multiple big tensors, 3D tensors. They are very big tensors called key tensors and V-tensors, but they have fundamentally nothing to do with key value store that database people used before.

And also, people misunderstand KVCache as just a cache. It's beyond cache. It's very useful data that you can store persistently, permanently, and you can analyze it in the future, and you can change it in the future in order to improve efficiency and also improve accuracy.

Amir: Got it. I have a lot of questions about why it's not a transient piece of data and why it is truly data that can be reused. But just to get a better understanding, so the KVCache — imagine, from the application developer's view, if they're running inference, so they have a certain kind of prompt, let's take a RAG scenario, certain kind of text that gets injected.

And as inference is happening, when you go through the tokenizer and the different attention heads, do they each have their own KVCache or there's one KVCache for each kind of prompt and the next token that's calculated?

Junchen: I think it's useful to explain the basics of KVCache first, okay? Just imagine in all these AI applications today, it's very common for the model to see the same prefix multiple times.

For example, you give the model a document to analyze and ask a question. It gives you the answer and then later you ask another question about the same document, or maybe you ask a follow-up question, the same document will be read again by the model as a prefix, as the same prefix.

And KVCache is the representation of model's internal understanding of that document. So if you keep it in GPU memory or in CPU memory and then load it back into GPU memory when it's reused again, that KVCache can help the model to skip all the processing of the document.

Now, you mentioned these heads and layers, and that's all part of the dimensions, the 3D dimensions of the KVCache. So KVCache records the models embedding at each layer and each head within a layer. And together, that's why it's three-dimensional. It's token by layer, by head. But it's just a big blob of floating point numbers that represent models and that's standing off a big prefix.

Amir: Got it. And just to understand, the size of this data, you can run us through that. It could give us sense, just order of magnitude.

So pick whatever model size you want or context size you want. Are we talking megabytes? Are we talking gigabytes? How big does the KVCache get?

Junchen: It used to be a simple answer. When the model was a full attention model, which means KVCache is the shape of number of tokens by number of layers by number of heads 3D, right? Back in the days, that was a year ago, the KVCache size is easy to compute, to calculate, to estimate. For example, for a Llama model, 70B model, thousands of tokens will have a KVCache the size of tens of gigabytes. It's easy to estimate because it grows linearly with the number of tokens.

These days, with the proliferation of linear states, linear models or hybrid models, the KVCache size is not really a function of the number of tokens, but the function of the model architecture itself. So some models have big linear states that represent whatever number of tokens in the context. But that size is really model architecture dependent.

For example, Kimi K3 is a much bigger linear state compared to other models, but it has more linear layers than other models. So today, it's much smaller and it's more or less independent to the context lens. But again, that size is still a function of the model architecture. I think we're talking about hundreds of megabytes for long context.

Amir: I know you recently put out the paper, what do you call it? Is it the Internet of KVCache? Or Internet for Data or Internet for AI data?

Junchen: Wow, that was almost two and a half years ago.

Amir: Okay, all right.

So one of the central premises of what you do is that the KVCache is not some transient internal calculation that the GPU is doing. That this is this computation, this data. It's big data. Let's say it's similar to other forms of big data.

So in a way, as someone from a data background, I think what makes data is it has to outlive the system that produced it. You have to be able to inspect it. You have to be able to port it from one place to another place. That's the definition of non-caching.

So a cache is usually something that's very ephemeral. So you just don't do another calculation again.

So tell us more about why your belief that this is not a cache and it is truly data, it should be ported around, it should be shared — I mean, you think even between different calls, different tenants, like basically that it's like a CDN, like these can be cached and sorted.

Junchen: Right. I mean, I used to use the analogy with CDN, but I think people these days don't remember what CDN is. I was actually asked by one student, is CDN a neural network or something? But I got what you meant.

I think that it's very important to differentiate caching versus data because caching and data require very different infrastructure.

Now, everyone in inference infra sort of went through a journey that all began with people thinking the inference engine is at the center of the world and you get the best inference engine, you get the best inference performance. And then they realized that the improvement of performance really relies on whether you can cache the important KVCache, the reusable KVCache in the system.

Now, at the beginning, people are saying, well, why don't you just keep in the GPU memory? Because that was where KVCache was created in the first place. You keep it there.

But then you realize, this guy and this user will come back maybe in 10 minutes. And by then, the GPU memory is very precious, so the KVCache may be evicted by that time. So you want to keep it in CPU memory.

And then you realize, a lot of agents may not come back in 10 minutes. They may come back in a day or two days. And they may want to reuse KVCache after the inference cluster is turned off and turned back on again.

So there you have this gradual realization that the KVCache is not just something you put in GPU memory, CPU memory, but also persistently. So there is this side of things, right? So this aspect of gradual realization that this data really need to be persistently stored. Now, that's one aspect.

But the other aspect to it is mostly from a research perspective, that from academia community, that KVCache can be analyzed, can be optimized. And that is really, I think, the real driving force for people to think this as real data, as opposed to caching.

Because cache is not just ephemeral. It's also black box data. You reuse it. But once you start to analyze it, once you start to optimize it, once you start to change its content, it starts to go beyond caching to real data.

Now, I want to say people have figured out what's the best way to organize this type of data. Even LMCache, which is now being used more and more as a data management system, it is still fundamentally organizing data into chunks and lookup systems. It's still built around this concept of reusing the data as much as possible.

But we're getting there. I think there are more and more research looking at KVCache as a data concept. And they're talking about how to organize the data in a way that's easy to analyze and optimize.

Amir: Got it. And so I guess this is a good segue to talk about your work on LMCache, on CacheBlend, these new technologies that you've brought to the KVCache. First and foremost, as I understand it, you want to try to reuse these KVCache calculations, not just for the prefill phase, which is trivial, but on the decode phase as well. So you want to share and store this across different nodes.

So for example, let's take a use case where there's a coding agent and we have a large code repository. Why can we not just kind of pre-compile the different files that is in the code once for that target architecture and then just grab it when it's in KVCache representation instead of calculating the files over and over again?

Kind of like the way you compile code before you run it, why can't you take all these files and just pre-compile them and not do it in runtime?

Junchen: That makes total sense. I think that's also the original reason we started in the CacheBlend project.

So first of all, let me rephrase what you said, right? So every AI agent, especially a coding agent, it has a sort of knowledge base from which it can get context, right? Code repository will be a knowledge base. A bunch of documents in enterprise policies will be, right? Or legal code will be, all right?

These are knowledge base from which the agent will try to grab a piece of knowledge as context for every query, right? So it is very natural to believe that, well, can the model just read all the text in the knowledge base once, or the code in the knowledge base once, and then cache its knowledge in a KVCache format. So in the future, if people want to use this to assist the AI model to answer certain questions, you can just grab a KVCache, a model can skip all the processing of this knowledge.

That is a very straightforward idea. I would say everyone wants to do that, especially enterprise use cases. However, when you put the context in the middle of the prompt, it is not straightforward to reuse this KVCache anymore because the KVCache only embeds model's understanding of a particular context or particular text when the text is read in an isolated way, not with the context of anything else.

So the KVCache of a text, if you let the model read it without any user questions around it, it will only have the understanding of this text itself. So when you sort of blend that knowledge with the particular user's context or agent's context in this application, you have to merge their KVCache.

And that merging is not trivial because the model have to read the whole context together. And not just this document, but also the pre-computed document, but also this particular uses chat history, for example, right? That blending is what this paper CacheBlend does. And that blending will still allow the model to reuse the KVCache, largely reuse the KVCache of this knowledge base, but also dynamically create the cross-attention between this document and the user's context.

Amir: So the cross-chunk attention — basically what I was describing does not work because you need that cross-chunk attention and where the file will get, which chunk of the file, the KVCache calculation might be different, right?

Junchen: Exactly.

Amir: Based on where it is.

Junchen: Exactly.

Amir: And so, let's imagine a long context, is it just the location that affects it? Or is it more to it than that? Can you just reorder them?

Junchen: It's more than location. Well, you mentioned location. There's this thing called positional encoding or rotary encoding. The positional encoding can be updated very quickly. That's not the real problem.

The real problem is semantically, the user's context and this particular documents need to combine. Their KVCache need to be combined, got blended. Now, I would say you mentioned LMCache and CacheBlend.

I think the bigger picture here, the deeper sort of trend here is when you look at every text data or human-readable data, there is a KVCache format of it. The model will translate that into KVCache format. And the KVCache format is what we call model-native data or AI-native data.

Now, on the face of it, you can say, well, anything you do with the human-native text format data can be applied.

There should be an operation operator, corresponding operator to the AI-native data, the KVCache format. But it's not that straightforward, right? Like you can concatenate any two strings together very easily. But when you try to concatenate two KVCache together, it becomes not that trivial. You need to create new algorithms like CacheBlend to enable the concatenation of KVCache. You can, of course, think about any other things you do with human-readable text. You need to do some innovation to enable that in the AI-native format.

Amir: Yeah, so if you want to mathematically think about it, like here there's a bunch of text, there's some sets, I can reorder them and it will change the meaning. Is there another basically arithmetic similarly in the KVCache side?

Junchen: Yeah, it can re-org and chop them up and concatenate and do things. How to do that in the KVCache world.

Amir: And that's what CacheBlend does, basically figures out how to rearrange these things. So is the accuracy then perfect, or is there any kind of degradation as you're trying to do this?

Junchen: It's very interesting. Obviously if you do this very carelessly it will hurt quality. For example, in CacheBlend, there's a ratio, or we call alpha ratio, that dictates how much recompute, how much blending you do, right?

Now, of course, if you do 100% blending, that means you just recompute the whole KVCache. If you do 0% replending, you only update the positional encoding, but you don't reconstruct across the tension at all. Usually, we set that to be 10% to 15% to 20%. So that range, you get a lot of benefit from reusing the KVCache, but you reconstruct certain amount of cross-attention.

Now, what's interesting is that 10, 20% on average will give you almost the same. Actually, I would say in most cases, it's the exact same quality on average. But if you're in certain cases, 10% or 15% recompute give you higher accuracy than 100% recompute.

There is this hypothesis we have about this. We don't have the real proof because neural network is just mathematically too messy to prove anything.

But our hypothesis is sometimes the cross-attention is not needed for all the tokens, right? It's like if you read two documents together, you don't need the cross-attention across all of them, right? Sometimes you turn off the cross-attention for many of the tokens will help the model focus on the right cross-attention. Probably those 10, 20% cross-attention that CacheBlend actually recompute.

So I wouldn't say all these KVCache operations will necessarily hurt quality. They can even improve quality.

Amir: Got it. In CacheBlend, the kind of changes that you make to the KVCache, The GPU does it or the CPU?

Junchen: It needs to be done in GPU. So because it basically does a selective prefill.

Amir: Do you dynamically kind of figure out where the line is, where it's better to recompute things than to use the CacheBlend update?

Junchen: Yeah, well, I'd love to have the first offer of that paper to sit here to give you probably two hours of analysis about this. But yeah, so it's dynamic. And in certain models, it improves quality with a dynamically selected ratio.

For example, in, I think, GPT-OSS, that model or that kind of sliding window model, CacheBlend improves quality when you select the threshold based on the deviation. If you fully recomputed the first layer versus reusing all the KVCache in the first layer, you calculate a deviation.

That deviation will give you a distribution: which token have more in need to be recomputed or less in need to be recomputed. And the KVCache, you select based on that distribution. And there's an algorithm to do that. If you do that very well with sliding window models, you can even improve quality.

Amir: Got it. And kind of starting to zoom out a little bit, I think in some of your papers you've talked about and I've read, and I think you're kind of alluding somewhat to it right now, that when thinking about the KVCache, it's not just the cache hit ratio that you need to think about, because some cache misses might be far more costly than others.

So when you think about using the KVCache efficiently, what are the right metrics to think about? Or the systems engineers who are thinking about KVCache optimization or using systems similar to LMCache, what are the metrics that you think are the right metrics to keep in mind?

Junchen: First of all, I think people shouldn't be asking what's the KVCache hit rate or how much you improve the hit rate. They should be asking how much you reduce the miss rate.

This sounds mathematically the same, but look, we all take OS classes, right? CPU hit rate is not the right metric. CPU cache miss rate is the right metric.

Because if you have a system with 5% miss rate versus 1% miss rate, that's a whole world of difference between them, right? But if you look at a hit rate, oh, it's 95% versus 99%. It's the same thing, right?

So because the miss rate really talks about the cost, how much you really need to recompute in a delay. One is 5x the other one. Exactly. So that I think is a misconception in the community right now.

Amir: So you think the cache miss rate is what you need to care about? And those cache miss rates, obviously depending how long it is to recompute or how big it is, they might be quite different, right? Are those metrics that people track in inference systems or no?

Junchen: Yes, they are. But the reason we should focus on miss rate is this.

These days, a lot of product can claim they hit all the previous cache, right? The storage vendors, they usually use LMCache, and they claim they can improve the cache from 0% to 60% to 70%.

Who cares? Essentially, you still need to recompute 30%, 40% of the context. It's where you can reduce the hit rate from 20%, 30%, 40% down to 2%, 3%, 4%. That really counts, right?

And in order to do that, you need to do a lot of optimizations, not just storing more KVCache, but to enable KVCache reuse, KVCache hit when it is not prefixed, for example, or you need to enable KVCache hit when it's a semantic matching, right? So it's not exactly the prefix, but they are talking about the same thing. You can still reuse this KVCache.

So I would say reducing it from 30, 40, 20% down to just less than 5% or even 2%. That's the key step.

Amir: That's the key step. So now, especially since we're talking about prefix and decode.

As I understand it, increasingly inference providers are completely disaggregating these two, and they have different fleets for prefix and different fleets for decode. I think KVCache on the prefix side as easy or trivial or not very complicated.

But how does that affect the way, obviously, you're thinking about the KVCache? First off, is this a trend that you see, and do you think this is the future? And if it's not, why not? If it is, how does that affect sharing of the KVCache, thinking of KVCache as big data that can be reused?

Junchen: I think it is definitely a trend. And I will say industry is doing even crazier things than just disaggregating prefill phase and decoding phase.

They're even using different hardware. Maybe using one GPU vendor for prefill and another GPU vendor for decoding. Maybe using one inference engine for prefill and another inference engine for decoding.

And I've even heard people trying to do disaggregating prefill and decoding across clusters and send KVCache between them with high-speed internet.

Amir: But then the KVCache, because if it's different chipsets, then the KVCache would be different, right?

Junchen: It's a great question. So I should clarify that KVCache is only a function of the model. It doesn't depend on the hardware. It doesn't depend on the inference engine.

So theoretically, you should be able to disaggregate and use different hardware and inference engine on one side and different on the other. KVCache would be the sort of communication data between them.

It's like, if you have two people trying to collaborate, we use human language to communicate. KVCache as a language for different inference nodes in the cluster to communicate.

And definitely, industry, especially these hardware vendors, are taking note on this. They are releasing storage and network devices specifically optimized for KVCache transfer and storage.

Amir: Yeah, so it seems then you also believe that the trend is pre-fill and decode to be increasingly separated and specialized. How does that affect KVCache reuse?

Junchen: It's a great news for us, because KVCache is not just a storage problem for one node. Now it becomes a distributed storage problem, because the KVCache is created not just by the prefiller but also the decoder. And these KVCaches need to be shared at some point.

For example, the KVCache created by the decoder need to be reused by the prefiller in the next phase, in the next round of conversation, because the decoder output will be the context of the next query, right? Say part of the prefill. So KVCache will flow back to a prefiller.

But again, that's just one example. If you think about one prefiller and prefilled KVCache is going to be used by five different decoders. These five different decoders need to share the KVCache, right? All of a sudden, you create a topology of prefiller nodes and decoder nodes, and they all share KVCache across each other.

At a higher level, if you zoom out from that, you can think about agent, different models, they share KVCache across themselves. So I think disaggregating anything in the inference world likely will create more and more demand for KVCache storage and distribution.

Amir: I think one item that I came across and you alluded to in the beginning is how on newer architectures there is a goal of trying to reduce the size of the KVCache. I think DeepSeek with MLA, I read the claim at 93% reduction. I saw Professor Han at MIT had a paper called DuoAttention, where he's trying to reduce the size of the KVCache. Do you think some of the work they're doing maybe goes against the bitter lesson, which is just throw more memory and compute, don't make the architecture more complicated. Do you think this is a direction that increasingly things will go, or do you think the solution is better reuse of the KVCache? Or do you think, no, these are both going to happen in parallel and will be more efficient, but still context are going to grow and we need both of them.

Junchen: Fundamentally, there will be more demand for AI workload. People are still figuring out that, but look, we're probably in the early 90s if you make an analogy with the internet. So we're in the early stage of figuring out all the use cases, and look how much the internet traffic grow from early 90s to 2010s. So I would say despite all the KVCache reduction techniques, there will be more and more traffic and there will be more and more KVCache.

You mentioned DeepSeek, right? Even with DeepSeek, we did a calculation, a B200 GPU for only 24 hours. It creates almost 100 terabytes of KVCache. That's all a compressed KVCache form.

One single GPU, not for one node, one single GPU for 24 hours. There's a lot of KVCache created by these GPUs anyway.

And everyone knows Jevons' dilemma, right? That you will have more traffic. If you make it more efficient, you only have more traffic and compensate the improvement. So that's going to be the future.

So my understanding is these model architecture companies, the model companies, they make KVCache smaller not for the purpose of eliminating the need for KVCache storage or KVCache organization. Their main driving force is to enable longer decoding or longer context within the GPU memory. They're willing to cram in more tokens in one GPU's capacity.

So fundamentally, we're not looking at the same problem. I mean, we're not trying to solve the same problem in different ways. We're trying to solve different problems. And as a result, with more traffic, there's just going to be more need for KVCache infrastructure.

Amir: Got it. And I want to talk about now, again, going further up in the stack, there's two different trends or use cases, technologies, and how those affect the KVCache or optimizations that have to be made.

And one is going to be just like longer horizon agents, which is an area we are actually been thinking quite a bit about here at Heavybit. And the other one is other modalities like voice and video.

So let's start with longer horizon agents.

So assuming, and this is something that I believe, that we're going to go from this kind of prompt response, go do this, it'll come back, go do this, it'll come back, to a world where these agents are going to be running by themselves for very long periods of time, and they'll only come to us when they're going to own the outcomes and then come to us when they actually need additional direction. In such a world where I believe, there's going to be a lot of things that are going to be stored in memory, and memory, I don't mean GPU memory, I mean agent memory. That is also a form of storage right now in text. What's the interplay there between that and the KVCache as these agents become longer horizon?

Junchen: It's a great way to put it. Now, normally when people ask me long horizon or agent workload, I would say that's a great fit for KV caching because the more interaction the model does with the word, the more history it accumulates and the more KVCache hit you can have. But I think you made a great point when you say this interplay between memory in the text form versus memory in a KVCache form, right? Because I think I don't have a great answer to that. If your key point here is how to balance these two type of memories, I don't have a great answer to that. I think at this point, all I can say is the infrastructure engineers are storing KVCache as much as possible, and application developers, independent to the infrastructure engineers, are storing as much text format data as possible. And each of them are trying to organize the knowledge or the memory in their own ways. And the application developers, they're creating indexing or hierarchical structures or searchable ways to organize text. And infrastructure engineers are trying to store more KVCache, and a company like us are creating ways to identify which context is more relevant to a particular query.

So at this point, I don't think these two communities are talking to each other in the first place, let alone creating an organic way to jointly optimize memory.

But I think we will get there. And I think once these two communities start to collaborate, they will create very useful and interesting tools. I would say that will be the next generation of AI long-horizon agent memory.

Amir: As of today, do you think agent application developers, people building agents, should be aware of the KVCache at all? Or is it something that...

Junchen: Oh, they definitely should for a very practical reason. If your infrastructure system, the inference system, only can reuse prefix KVCache, you better think the application developers are better off reusing context in the prefix part as much as possible. If you chop it up and recompile, concatenate different memory together every time, you're going to just miss all the KVCache. Reorder the tools every time. Exactly, exactly.

I think people learn their lessons. And even these days, agent harnesses, they by default just reuse prefix as much as possible. However, if you talk to any agent memory companies or researchers, reusing prefix is not necessarily always the best way. So there's going to be this tension.

I think right now this tension is not surfaced because these two communities are not talking to each other or they don't think there's a way for them to talk to each other. But I think as token costs become higher, non-sustainable, and applications try to get better accuracy and infrastructure people creating more and more crazy optimizations, they're going to, at some point, try to talk to each other.

Amir: Now coming to the second item, so voice, video. Obviously, with video, there's so much more. The size of the data, I assume, even for the KVCache, is much, much larger, I don't know.

For those modalities, anything specific about KVCache technology that you guys are looking at or differences that exist, at least today or going forward?

Junchen: Multimodality is definitely a so far overlooked corner of inference efficiency. I mean, there are a lot of models and inference engines built for multimodal data, but I think efficiency is an overlooked corner of that.

Now, the direct impact on KVCache is once people have more multimodal input, especially multimedia input, it'll be just 10X or 100X more tokens easily.

Think about how many tokens you're going to have in one 10-second video. It's going to be probably the same number of tokens as in a thousand-page book. Just one 10-second video.

So caching will definitely help a lot, but caching itself is not enough because the model itself also needs to realize that there's so much redundancy across the frames. Now today, a lot of these models just do frame sampling, like one frame every eight frames. But that's so stupid. You may lose a lot of information or you may oversample at the same time. So the model needs to be smarter.

But again, in the meantime, KVCache is going to be a big part of the solution because it's just more tokens and caching skips a lot of compute for you.

Amir: Yeah, so like if I'm editing a video with some kind of AI model, if I'm like cropping some portion of it or something, like it's been pre-calculated before, right? I don't want to recalculate everything because I cropped like a border out of it, right?

Junchen: Exactly. And a very interesting thing is, look, if you have a video input or an image input, just say, and you want a model to reanalyze just a portion of that image, today, there's no way to do this other than reprocessing that image.

But with KVCache, you may be able to identify which tokens correspond to that part of the image. And just reuse KVCache right there. And it can save a lot of compute for you.

Amir: Got it. Kind of like zooming out on just the overall inference ecosystem, the open-weight ecosystem. So obviously right now there are open source inference systems like vLLM, SGLang. There's companies around those open source inference engines like Inferact and RadixArk. NVIDIA has Dynamo, obviously companies like Baseten, Fireworks.

Junchen: SRT as well, yeah.

Amir: Yeah, so do you think we're going to converge to fewer open source ones? Do you think there's going to continue to be a whole lot more overall inference systems?

I mean, you're both — have been a researcher for a long time, you're in an industry, we're leading a company in this space. What are your predictions of the overall inference ecosystem given the different players in the different spaces that they are in right now?

Junchen: There's industry aspect and there's a human aspect to it. I'll start with the human aspect because that's more fun.

In every wave of new technology, the ego of the creators of a technique or a piece of open source software or either way, that plays a real big factor in the development or the evolution of the ecosystem for the first phase.

Amir: When you say ego, do you mean taste? Because I'll do an analogy of database systems or programming languages.

Junchen: By ego, I mean the urge or the self-belief that they can build full-stack systems. They can build everything by themselves. And this generation of researchers or programmers is no different from the big data era, from the internet era, from the operating system era.

Everyone wants to build a full stack software with everything. They want to control every layer. And this has been how the first phase of every technology played out over decades.

So that, I would say, is a human aspect. But at some point, the industry factor will take over, will play a bigger and bigger role. The industry mentality is consolidation and specialization.

Unless you're a big company like Google and you consolidate every aspect about search, then sure, you are the ecosystem. There's only one company, a couple of companies there. But most of technology, in industry's trend, will eventually take over and try to emphasize specialization and consolidation. We're seeing that playing out right now.

So that's why, from the very beginning, we emphasize that the KVCache layer needs to be in a layer, independent layer by itself. We work with all these inference engines. We work with all the inference frameworks. We work with all the GPU and storage vendors. So when they eventually need a KVCache layer that can support different users, different engines, a common choice will be what we do.

Same thing with storage vendors. Last week we were at FMS. Almost every vendor we talk to at FMS, the storage vendors, we're using LMCache to show KVCache performance, because that's the library that they can use no matter the customer uses VRN or SGLAN or TensorRT or LND or Benamo, because it's a common denominator for everyone.

But going back to the human factor, we're still in the first phase of this industry, the inference system industry. And it's hard to say there's a very good way to convince everyone to just sort of sit together and say, well, let's make sure we have the best software for each layer.

And for example, a very famous professor who I work with very closely was trying to solve this problem, and it's hard because at the first phase everyone has very high ego that they can build everything by themselves and they can outpace everyone and they become the dominating player and they dictate all the interface of everyone. But let's see.

Amir: Alright, the LMCache project to the PyTorch Foundation. And obviously you're building on a lot of technologies that you did with LMCache. And you guys are an open source company.

Walk me through as a founder how you made that decision, why to be open source, your thoughts on open source, especially now when there's code gen tools — increasingly we hear it's very hard to maintain the quality because people create a lot of PRs that are sloppily written. Tell us about both the decision to donate LMCache to the PyTorch Foundation and building an open source business.

Junchen: First of all, let me clarify. We joined PyTorch Foundation as a landscape project, which is just a step before really donating the trademark to PyTorch Foundation.

Amir: Is that the plan?

Junchen: So it's certainly our plan to donate the trademark to somewhere. Whether PyTorch Foundation is the place or not depends on whether PyTorch or some foundation has the vision, the same as us, because donating a project to a foundation is partly for marketing purpose, right? So they need to believe in this vision as well.

Now, why we believe the LMCache project needs to be open source and needs to be governed by an open community, that goes back to why we have this project in the first place. At the beginning, we were trying to do research about KVCache optimization and we wrote papers. That was early 2024.

And we spent a year doing that and realized industry didn't have a way to deploy these research ideas. And we also project there will be more and more papers in this space. And there's a common bottleneck, which is simply the industry didn't have a platform to run these ideas.

So we open-source LMCache 2024. And the open-source project was and has always been our bridge to bring research ideas to industry. So the open-source project itself — it's always sort of a pipe for the research ideas to be used by industry.

And the pipe itself is not where the value is. And I think, in fact, every good engineer will eventually build a good pipe of that.

So we are basically serving the community by creating an open source, very reliable, I mean, you can say plumbing work project, right? But that project in itself is not our end goal. Our end goal is to create an ecosystem around KVCache, but not the library and the plumbing project itself.

Amir: Now, going a little bit to the work you've done in academia, AI researchers who are currently in academia — let's say you have a great PhD student — imagine they came to you, but imagine they're in some AI research program getting started. Well, let's say other than KVCache, which we've spoken a lot about — it could be adjacent to KVCache — but what are some interesting problems within AI systems that you would push them to go and try to work on and think about and try to go solve?

What are things that maybe you would try to talk them out of them and say you think are the bad ideas or not? Or what are areas you think people are not spending enough time on or areas you think people are spending too much time on?

Junchen: First of all, I don't push students to do this. I already learned my lesson. You know what?

KVCache was not something I pushed my students to do. KVCache is what my students pushed me to do at the very beginning. So sometimes these students have a better taste than me.

Now, the advice to this generation of graduate students, especially system students, the best advice I can give is take these advanced OS or advanced networking graduate level class. These classes are your menu for the next few years. Why do I say that?

These classes are usually looked over by these students. In fact, a lot of faculty, when they teach these graduate-level classes, they just ask students to present papers. But these classes are actually very, very important. They teach you the history of systems literature.

They teach you, at the beginning of operating system research, what were the original ideas and what were the novel ideas back then and why some of these ideas became industry standards, some of these ideas became just test-of-time papers, or why certain papers remain research ideas, right? That was the real goal for graduation study.

Actually, to me, the history of your area is what guides you to find your next topic. Because it's a cycle. Every time it's a cycle. We're at the beginning of the AI cycle.

But 15 years ago, those graduate students were at the beginning of the cloud or big data cycle. 20 years ago or 25 years ago, it was the internet, right? And before that, it was a distribution system and so forth.

So you can look at what ideas people had back then. It's a great guidance to say what might be the good idea for the next five years.

For example, just for the sake of argument, in the beginning of big data, right? When people built data centers, there were supercomputers. There were big machines from those big vendors. And back then, people were thinking, you know what, when we build big clusters, compute clusters, you should use those big machines.

But eventually, what became the industry standard was just putting together a lot of cheap machines with Ethernet. You disaggregate everything — storage, memories, computer network — they all are cheap devices, but when you organize them in a smart way, you have a much more reliable performance system than those more expensive big machines.

We're certainly seeing something similar happening right now in industry. You mentioned disaggregation, right? There's a trend, but I don't see anyone putting a principle to it. And there can be a best ACM dissertation award for that.

Amir: Some of the previous guests on the show who were PhD students talked about pull between, obviously, the need of working on creating new benchmarks, which is a big need and obviously gets a lot of visibility, but also this view sometimes in academia that working on benchmarks is not true research, maybe. Do you have thoughts there? Do you think PhD students should be spending time on benchmarks?

Or if one assumes that AI will be doing a lot of the research itself and can do a lot more research and self-improvement, does that make creating the benchmarks more of the human role? What are your views on the topic?

Junchen: My view about benchmarking is, first of all, why do people feel skeptical about benchmarking today? It's because they feel like benchmarking doesn't capture the uncertainty of the models and doesn't capture the corner case of the models.

But fundamentally, it's the human nature that people don't really feel comfortable with the way we benchmark these systems. But I would say that uncomfort will remain there, but will eventually become less and less important as people get more and more used to this technology. And eventually, benchmarking is needed anyway.

And obviously, the reluctance to this benchmark effort is mostly because of people's uncomfort to this technology itself. It's not towards the benchmarking itself.

Now, if you go back, like I said, in history, right? 20 years ago, when database became the new thing in the world, there were benchmark efforts like YCNB, right? And the Yahoo benchmark, you probably heard about that.

It's a very famous paper about this. People were skeptical about that back then too. People thought, well, if you benchmark a system performance without first principle, you just look at the numbers, you're probably just benchmarking how this particular company built its hardware. There's no real insight from it.

But eventually people were convinced that this benchmark is useful because eventually you are buying the databases from those vendors anyway. The things that you don't really feel comfortable, which was vendor-specific numbers, became part of your life, become part of the consumer's life anyway.

I think benchmarking today certainly have the same issue that people feel, well, you shouldn't be benchmarking a particular workload because it's not representative enough. People will get over that.

Amir: In terms of parting thoughts, you are in academia, you are in industry. One question, especially whenever I talk to folks who are getting their PhDs, is, should I finish my PhD? Should I go join a startup? Should I go to a major lab?

And I always say, I can't answer this question for you, but it's a question I get often. I'm sure you get it a lot. So any thoughts you have there?

Junchen: I think it's more and more clear these days that for a PhD to work, you aspire to be a faculty eventually. Now, of course, not everyone will become faculty. Not everyone will eventually be interested in academia.

But at least you think when you join a PhD program or you decide to apply for a PhD program, you have this ideal in mind, you have this dream in mind. Of course, you probably will feel like your dream is shattered after you just see the real academia.

But for PhD to be a real worth, if you don't have faculty life as your dream in mind, it becomes less and less sustainable for people to join PhD programs if they just want to go to industry. Because industry these days are moving faster than academia in many ways. Not every aspect, but in many ways.

I would say this is sort of a radical view I have, which is we can have two parallel PhD programs, right? One is you do some research and get some ideas for three years, like just a shorter PhD program, but eventually you get to industry, right? So industry is your real task ground of your ideas. You go to industry faster.

The other one is a longer cycle. You go five years, six years, you do more groundbreaking research. You spend more time to get one idea, like a really out there idea work and become a faculty maybe or become your startup at some point.

But I think at this point, if you want to stay in PhD program for five years, first of all, I don't encourage people to quit in the middle, okay? But if you really want to spend five years there, you need to have this dream of becoming a faculty. Otherwise, it's not sustainable.

Amir: Well, Junchen, thank you so much for coming on and being on the show. I think I learned a lot.

Junchen: Thank you.

Amir: Thank you.