Ep. #60, The Killer App for AI with Vasanth Mohan

Rachel Chalmers: Today, I am so pleased to have Vasanth Mohan on the show. Vasanth is Director of Technical Product Marketing and Developer Relations at SambaNova. Earlier in his career, he worked with thousands of developers across emerging technologies, including VR, AR, and edge computing, building a leading community around FusedVR.

At SambaNova, he now focuses on helping developers and enterprises unlock agentic AI with fast inference on SambaCloud. Thank you so much for coming on the show.

Vasanth Mohan: Excited to be here. Thanks for having me, Rachel.

Rachel: You have moved from VR render streaming to fast inference chips. What is the actual through line between those two cosmological constants?

Vasanth: It's super interesting. My career has been really at the bleeding edge of technology. That's one of the things that I've always been really, really excited about.

If you go back about a decade to when VR and AR were super hot, and I guess at the time, Facebook acquired Oculus.

Rachel: Metaverse!

Vasanth: And then the metaverse.

And it's been super interesting to see how Meta has then started to totally pivot also into the AI space as well. I actually do think there's some still really interesting hot stuff happening in the VR/AR side that will eventually make its way into the mainstream as AI continues to become and grow more and more into production.

But it's really been being at that forefront and intersection where I started with my own YouTube channel, again over a decade ago, really teaching VR, which got me into the broader developer relations ecosystem. Then moving from there to telcos, 5G, and the networking side of that, where VR was a use case, AI was also a use case. And then from there, making my way into the AI space, and now where I'm at at SambaNova, is really where that through line came in. But I would say what kept that moving is really just a passion for the technology and where that's going and all of the use cases that can really come to life at the forefront of all of this innovation.

Rachel: That totally makes sense. And it jives with something I've been thinking a lot about recently, which is that in an age where generating code is incredibly cheap, the information advantage moves to people who have a voice and a particular taste. The voice amplifies your taste and the taste is why people follow you. So you've got a somewhat agent-proof career already, I hope.

Vasanth: Well, you might want to call it agent-proof, though, but the number of agents that I have running and helping me out is also just a part of it. I do really believe in the fact that with where AI is going, it's very much how you embrace the cutting edge of all this technology that allows you to accelerate everything that you do.

And I think that's just generally good career advice also. What a company is really looking for is how you can better help them by helping yourself improve business outcomes. And that's really where, as long as you're trying to solve that, that's going to be a skill set and a drive that's irreplaceable and keeps you competitive in the space. Whether or not it's with AI or without AI, as long as you're hungry and building those relationships, that's the most important thing you can do. And I think that's true career-wise, also probably very true for founders and innovators that are building startups in the space.

Rachel: Absolutely. Founders have to zero in on the thing that they have that they can do better than anyone else in the world. Candidly, VCs have to do the same thing when we're raising a fund.

And I've always wanted augmented reality more than I wanted virtual reality because I like Earth. It's where I keep my stuff. I want augmented intelligence more than I want artificial intelligence as well because I like my own thoughts because they're mine.

Vasanth: I see a world where those devices probably merge together. But I think even a decade ago, it was always true that augmented reality would be the much bigger camp that everyone wants to be involved in, and AI will just inevitably help make that better.

I will say, I haven't done this recently, but in my building days with AR and VR, the biggest pain point was the time it takes to actually build the world and the simulation. And with AI, it's just absolutely crazy. Some of this stuff you'll even see with Astra that came out just a week or two ago.

Everyone's building these 3D simulations and 3D models and trying to build them as complicatedly as they can. And I think this is just generally true. It only gets better from here.

So just imagine what that looks like where AI can build these worlds in real time and react to everything that's happening around you. I think that is one of the things Meta is trying to do with the Meta glasses. It's still a very early prototype, but those worlds will start to intersect in, I couldn't tell you a timeframe, but I'm sure in the not-so-distant future.

Rachel: You gave a talk titled "Beyond the GPU," which I love, by the way. What is the first clear signal that an engineering team has outgrown a standard single architecture setup?

Vasanth: I always like to start at the use case. So what we see a lot when people are just getting started with AI is GPUs just make a ton of sense. They're very flexible, general purpose. But when you're starting and you're experimenting with different models, it doesn't really matter what the performance really looks like.

But as you get more sophisticated and the tools are getting into such a better shape that it's becoming very fast to get sophisticated, you start to reach this natural plateau that you can get in terms of productivity. And what I mean by that is, if I talk to some of our analysts that we work with or customers, it is: how much can I get done in a day by working with all these agents. Whether that's Claude Code or Codex, or I love Hermes and use that a ton.

But you start to get bottlenecked by how fast this agent can run. And it impacts also your natural chain of thought when you're working with these models. The number of what I'd like to call super users of the technology, they'll run many different agents in parallel, but then they're going to be sitting there waiting 30 minutes, an hour, maybe multiple hours. And you lose your chain of thought when you're actually doing that.

And I think that's what has been creating this demand for faster and faster inference, because that is what makes these agents get much more closer to that real-time chain of thought that you have and overall increase your productivity as the agent is able to do a lot more for you in less time. And that's really, I think, the key area where folks have started to naturally look for solutions beyond the GPU.

Rachel: That's a great segue. You're talking about inference speed as a business layer, not just a technical one. Where is that speed actually going to show up on a P&L statement?

Vasanth: I think that's part of what people are still trying to figure out: how do I measure it, and how do I measure it across different use cases. Coding is by far the biggest one. And what's happening today is in the P&L, people are just measuring it in terms of tokens consumed. But that doesn't actually tell you anything about the productivity that those tokens have actually produced.

Rachel: I can burn tokens so fast and you will get nothing out of it.

Vasanth: Exactly. It's funny because I was looking at the analytics out of some of the agents I have deployed. It's so fast.

Within minutes, you can burn through millions and millions of tokens because these agents are just constantly spinning. But is it actually producing something meaningful on the other end of it?

That piece of it is, I think, part of what's really trying to be figured out in the industry today. I don't know if we necessarily have a clear-cut answer that not only depends from organization to organization, but department to department.

For example, yes, coding is the predominant use case. And I think at the super user side of it, it's very obvious that they are generating way more lines of code than they would have in any other scenario, and that is what's allowing at least the super user to be 10x productive. But that might be not trickling across the whole department.

And I think it's really interesting also. Well, how do you measure productivity in terms of, for example, for me, let's say I'm creating this data analysis spreadsheet or a slide.

You could probably measure productivity. And how long would it take me to do this without AI? Days versus one hour?

It's very loose and very subjective on how you measure it. So I think we have all of these anecdotal stories, but we're still trying to figure out what is that narrative that really concisely measures productivity.

Rachel: Well, I don't think there's one. I think we want to collapse it into one silver bullet, but we've had both Nathen Harvey and Nicole Forsgren on the show talking about DORA metrics.

And this is why I'm thinking a lot about taste, because those super users, lines of code is a real brute force metric for them. Some of the best engineers will sit there for an entire day and then they'll remove 10 lines of code. And that's why they're paid 10 times as much as the junior engineers, because they have taste.

I've also been thinking a lot about what Charity Majors has been saying about how we're really eating the seed corn. We're dropping junior engineers in it, because that process of really pushing your brain against the GitHub repo and trying to make yourself understand the larger scale architecture of the system disappears with coding agents. So we're not necessarily training as many people with that systems intuition, which is another version of what I'm calling taste.

Vasanth: It was interesting. I was just watching Jensen on the All In podcast, and what he said was his favorite character on the keyboard is the backspace button. It is how many lines of code you can delete to simplify it. And I have the same problem with all of my agents, as they over-generate and over-engineer a bunch of different things as opposed to simplifying it.

Rachel: I'll really age myself here, but there's a wonderful scene in Douglas Coupland's novel, Microserfs, where the genius programmer wanders from office to office and everybody's got their code up on the whiteboards, and he'll just erase it and optimize it.

Vasanth: That's exactly it. It's less is more. Less is really more. That's what makes measuring productivity so much harder.

And then on top of that, you layer, okay, how much more can I do with faster tokens versus basic tokens? Hard to say.

This is what's literally happening at the forefront. Everyone is trying to solve this in silos. And there's honestly very few discussions of what this actually means in the broader context of what's happening in AI.

Rachel: So let's dig into that a little bit. What is the failure mode when inference is too slow for an agent loop?

Vasanth: So it always depends on what's the outcome. It could be something as simple as I have my deadline tomorrow. If I don't solve this for the end customer by tomorrow, well, you're out of luck. You lost the deal.

That could be probably somewhere in the middle of the reality of a lot of situations. It's how much time do you have to complete said task. And it might be an oversimplification of a very specific deal and code that's needed for a customer. But I think that happens across the board in a lot of organizations.

You have a delivery deadline. It's how much time does it take to solve it. And if I can solve that in less time, then great.

I'm able to complete the task. I'm able to generate more ROI for the business. If I can't, you lose deals. And I think that that's happening across the board.

That said, I think there's plenty of agents that also are batch processes that you just run them overnight. For example, I'll have news agents or competitive intel agents that are running behind the scenes, and they just generate a report for me in the morning.

And I can read that, summarize it. If it took one hour versus two hours over at 2 a.m. or 3 a.m., do I really care? No, not really.

And so it's a question of which type of agent, because I think there's really this bifurcation of these batch mode agents, and there's a lot of those use cases, as well as these interactive ones where I'm in a coding session or I'm building slides or I'm building a spreadsheet and I need to be really fast and iterative. And in doing so, it allows me to compress days of work into maybe less than a day, maybe an hour.

Rachel: You have spent years building developer communities in VR and AR before the market cooled. What did that teach you about hype cycles that applies to agentic AI now?

Vasanth: I think there's always these multiple sets of users that are there. There's the super users who are really in the weeds of things. And then there's people who just are investing, get excited about it, don't really necessarily know all the bells and whistles of what makes something happen.

And these hype cycles, whether it's VR, whether it's crypto, there's the mass adoption of it, of people who are learning, there's the super users of it. And you'll get a sense from the super users on how much business value there is.

And I think of the case of VR and AR, you had these super users, but there's a lot of issues when it came to commercial deployments, whether it's sim sickness, whether it is accessibility and space requirements, etc. A lot of issues that you would see at the forefront that naturally made you realize, okay, the business use case and impact is limited. And crypto goes through these waves and hype cycles as well.

I think AI is very, very different in that we are actually seeing a ton of usage and deployment, unlike those other hype cycles. I think there's valid concerns around overbuilding, and when do we know that there's too much or there's a lot of tokens that, even with OpenAI, my subscription there, I'm sure I'm using way more tokens than I'm actually paying for.

So I think there's a lot of concerns around how much of the demand is real versus how much supply is there. I think there's a question of how you measure the productivity, but I don't think anyone would dispute that people are actually moving faster using AI. And I think it's just a matter of trying to figure out what is the right product market fit for how much people are willing to pay, and what are the different sub-segments of this inference and model size that are out there that are best suited for different use cases.

But the use cases are at least there, which is, I think, something in the other hype cycles. Everyone always asks, what is the killer use case?

For AI, I think it's actually very clear. It is coding. I don't think there's any dispute on coding being the killer use case.

I think like VR, people are still asking that question. Even in the crypto community, I think people are asking that question. But I occasionally hear that question in the AI space, and it's immediately always answered with coding, which is, I think, very, very different than what we've seen in other hype cycles.

Rachel: Yeah, it is. SambaCloud is built on custom silicon, not commodity GPUs. What is the trade-off enterprises don't see coming when they commit to that bet?

Vasanth: So maybe to take it one step back. What is SambaNova and what do we do? So we built our own ASIC and hardware called an RDU, which is a very fundamental different architecture that allows you to do inference at much faster speeds and also with a lot less energy consumption.

And that's what we have deployed in our cloud today, running some of the latest and greatest open source models that are out there. It's really here more as a showcase of what is possible with our hardware that developers and enterprises can leverage to use in whatever agents they're actually building.

Now, as it relates to the GPU ecosystem, and when do you choose an ASIC versus a GPU, I think there's really two key factors that you're looking at. One is the speed that we talked about when it comes to how productive I want to be with the agents and what I'm running. And the second one, and I think this is very true in sovereign deployments, is how much data center space do I have to actually deploy hardware in silicon. And especially for the performance profile that I'm looking for.

Because the challenge that you'll have, especially in the GPU deployments, is a lot of the key performance numbers and benchmarks you'll see are all running off of liquid-cooled infrastructure. And that's only 20% of the data centers out there in the world today. And of course, there's a big ramp to deploy more and more of them, but we have 80% of data centers in the world that are air-cooled, that are built using existing IT infrastructure that we could take advantage of, but we don't necessarily have the right systems to be able to actually deploy in there.

And I think that's becoming more and more and more of a concern as we're becoming compute constraint is, okay, well, data centers are taking a long time to ramp up. Energy is taking a long time to ramp up. So how can we do more with the existing infrastructure? And I think that's the big area that I think is still pretty untapped in the market today that becomes a real driving decision factor for developers and enterprises on how they pick and choose where they deploy their infrastructure.

Rachel: Do you worry that that's a transitional technology that might turn out to be cul-de-sac, like screen scrapers for mainframes?

Vasanth: The use of existing data centers?

Rachel: The air-cooled data centers, yeah.

Vasanth: Yes, I think the short answer to that is it's a point-in-time thing. There are definitely benefits with liquid cooling, but the capex that's required to build that out is very intensive and takes time.

And also the engineering complexity to get that deployed in a way that is like recycling water and using power very efficiently just takes time. And it's going to be different from region to region. I think here in the US, there's always going to be this rampant push to build more of it out in smaller areas like in Asia, like Singapore or Japan, where you don't have as much space and infrastructure. There will still inevitably be these pockets, especially when you're energy constrained as well.

So I do see liquid cool data centers starting to push more and more, but it is one of those things that's going to be a point-in-time thing as we're figuring out the maturity of the overall data center ecosystem.

Rachel: Totally makes sense. Completely switching tracks, if agents are the new interface, what does developer relations even mean when the developer might be another AI?

Vasanth: Well, we're exploring a lot of that, especially in the context of, I'll just take the example of documentation. It was super interesting just a few weeks ago when OpenAI showed at the Hot Chips conference their silicon and having agents bring up models against their custom silicon. And the way they did that is they were building out documentation in a way that's much more agent-friendly than it is human-friendly.

And I don't think that's actually a bad thing. I think as developers as a whole are embracing agents, you want it to be consumed in a way that is friendly to your agents to help you bring things up. I think there will always be opportunities to further tweak and refine code, especially at a systems architecture level. But the vast majority of, whether it's model bring up, whether it is application bring up, etc., is going to be done by an agent.

And what that means in the context of DevRel is, yes, you're optimizing a lot of your tooling and your documentation to be done by agents, but there's still a ton of value in the network connections that you can bring together, whether that's in actual events and bringing people together to talk through how do they improve their agents to work with these different tools.

And I really do believe in the human connection that DevRel can bring to the table in terms of building that ecosystem and community while the agents go off and do some of the development.

And I don't know when that world materializes, but I think it's within the next five to 10 years. Agents are going to be starting to do a lot more and more of that work, which I think is exciting. And then developers really are the orchestrators of all of that. That's already starting to emerge today.

Rachel: Yeah, it is. You mentioned your agent that monitors news sources overnight. Whose voices resonate with you? What are your favorite sources for learning?

Vasanth: Being in the semiconductor space, I'm really looking at a lot of our analysts and how they're thinking about hardware and technology. And SemiAnalysis to me is one of the best sources that's out there, especially when it comes to the articles and deep dives that they're doing across the whole stack when it comes to data centers and chips and performance benchmarks, etc. As well as many other analysts in the firm like Artificial Analysis or even some of the work that independent creators do. It's all incredibly exciting.

And then across Twitter, there are of course so many different creators that are testing models and showing the limits of it. And I really do believe, yes, there's information sources you can gather from looking at so many other peoples in the ecosystem.

But my personal belief, just coming also from a dev world experience, and I think it's very true in the AI space, is you have to try it. You have to get your hands on the technology and get a feel for it. And that resonates, and I see it all the time, in all of what people are calling benchmarking against these AI models. It might look great on the benchmarks, doesn't work well in practice, and the only way to figure that out is actually trying and testing things yourself.

Rachel: And the barrier to entry is so much lower than it's ever been before. It's so accessible.

Vasanth: Exactly.

Rachel: I'm coding again, which is wild and probably dangerous.

Vasanth: But it's fun. I just don't have the time. Again, going back to the agent, I had the agent code up a website that I can use to actually go and view all of the metrics on that agent. Can you imagine a world where you have to do that yourself? It would take days, weeks, or months.

Rachel: I grew up in that world, Vasanth.

Vasanth: Yeah, exactly. That's how crazy it is.

Rachel: I learned to code typing HTML.

Vasanth: Exactly. That's the same. Exactly the same. I went to a hackathon in the morning and I was like, you know what? I just need a visual for a talk I'm giving later this week.

I literally pull up a Telegram where my Hermes agent is connected. I just record an audio message of, can you build me this dashboard and have these metrics? And within 30 minutes, done. While I'm driving over to the hackathon.

How crazy is that? It's crazy.

Rachel: Okay, you're Prime Minister of the Solar System. Everything goes the way you want it to for the next five years. What does our future look like?

Vasanth: There's so much that's happening in the LLM space, and that's what I'm really glued to. But I think that the next wave of this is, well, one, in the video generation side, I think that's starting to mature, and it's really interesting.

And going back to some of the AR stuff we talked about, how cool would it be if we actually have these augmented AR glasses that we can use, interface with AI agents on the go? Voice becomes a lot more easier as opposed to typing and texting to agents. You're just recording real-time interacting with agents.

And then from there, even if I could control everything, I don't think I would get this in five years, but I would really like it in five years, is a world where physical AI is actually starting to emerge and is a lot more accessible. Whether it's Optimus or any of the other stuff that's happening in robotics, I don't see a world where that happens in five years. But if I could orchestrate everything to a T, I would love for that to happen in five years. I definitely think it's within the 10 to 15 year horizon.

Rachel: How about quantum? Are you a bull or a bear?

Vasanth: I'm neutral. I do think it's going to happen. I think there's a couple of breakthroughs that need to emerge for that to become more mainstream.

I am nervous about what that means for the larger internet as a whole. But quite frankly, I'm probably also nervous about that in the context of these AI models and their cyber capabilities as well.

Rachel: Little Harry Houdini agents.

Vasanth: Yeah, exactly. So that part to me is so hard to predict what's going to happen. Is the AI going to solve all our security problems first or is quantum going to come? Who knows?

And then is AI then going to solve all of the issues that quantum might create? Probably. So it's this whole cat race of so many intersecting technologies, but it's incredibly exciting to watch.

Rachel: Good news. Along with your prime ministership, you get an imperial spacecraft, an interstellar starship, a generation ship. What are you going to name it?

Vasanth: I might call it FusedAI, honestly. It's this brand extension of everything that I got to do with FusedVR, but now moving into the AI space. And it's this ability to merge all of these technologies together and create something generationally awesome. It's super exciting to me.

I love this concept of fusing different things and bringing things all together into really much more positive outcomes is always super exciting. And that's what this fusion concept in my head always enables. And I think that'd be a cool way to go off into the solar system and ecosystem.

Rachel: I love it. Vasanth, it's been amazing to have you on the show. Thank you so much and good luck with everything.

Vasanth: Appreciate it. Thanks for having me.