Ep. #30, Inside Unikraft and Unikernels with Felipe Huici

Brian Douglas: Welcome to another installment of Open Source Ready. John, you are back with me. How you doing?

John McBride: I'm back, I'm doing pretty well. It's kind of a wild time right now, which is a little surprising in December but it's fun. What about you?

Brian: Yeah, wild time. Actually it's pretty manageable, honestly, for the end of the year. I'm sure like in SaaS based world startups, everyone's sort of like either winding down or trying to figure out what they're doing for their side projects at the end of the year.

We're actually working on something really cool. So follow me on Twitter. I'll be tweeting out some holiday side project stuff which I'm looking forward to. But we're not here to talk about my side projects, we're here to talk about a bigger project which is Unikraft and we've got Felipe here to introduce that.

Felipe Huici: Thanks a lot. It's an honor to be here. I was planning on having an easy December and then I ended up going to re:Invent so it didn't end up being that easy in the end. But it's a pleasure to be here.

Brian: Oh, well, welcome. I was there last week as well. Actually I saw your colleague at Unikraft who's just moved to Bay Area. I forgot his first name.

John: Alex.

Brian: Alex, yes. I ran into Alex at re:Invent and I run into him m actually quite often in San Francisco. So he made the connection and I'm happy to learn more about what you guys are doing. So there's an open source project, Unikraft, and then there's like this unikernel but I'll let you explain what you guys are working on.

Felipe: Sure. So the open source project called Unikraft is a Linux Foundation project that tries to make it easy to build something called unikernels.

Unikernels are nothing more than just a specialized virtual machine. The idea being that a standard virtual machine maybe has something like Linux or Windows running and a whole distribution in applications and it can get fairly bulky when ultimately perhaps all you want to run is an application or two.

The idea with the inner kernel is okay, if the first class citizen if is-- that application could be Postgres or web server nginx, can we build a sort of purpose built operating system distribution only to run that and in so doing reduce the size of the image significantly, reduce how fast it can start, think milliseconds instead of seconds. And so that's what a unikernel tries to do. And Unikraft is sort of an SDK to try to make it as easy as possible to build such unikernels on a per application basis.

John: Yeah, I find this such a fascinating topic having built a bunch in the Linux ecosystem and containers, which is its own whole paradigm. To ground this in my own understanding, how does something like Unikraft kernel compare to like a Firecracker VM which is also like a quote unquote MicroVM?

Felipe: Yeah, it's a good question. So that's an adjacent technology and complementary. So basically starting from the bottom you have an x86 server, an ARM server, that's the hardware. Then on top of that you're going to install Linux. And Linux comes with a kernel module called KVM that allows you to simplify and run virtual machines. Right?

But then in user space you're going to put something called a virtual machine monitor (VMM) that allows you to talk to KVM and say please start a VM, stop a VM, what VMs are running? Right? There's multiple flavors of that. There's something called QEMU from a while back and then Amazon came along and created Firecracker.

Because if you were creating Virtual machines via QEMU, the virtual machines were slow to start and bulky, in addition to themselves being bulky too. So when Amazon was developing Lambda, they said we can improve on this virtual machine monitor. QEUMU was too chunky. Let's build a purpose built one, almost like a unikernel principle.

Like if I wanted to start Linux VMs, how can I start them as quickly as possible? Let's do a targeted, lean virtual machine monitor. That's Firecracker. And then with Firecracker now you can start virtual machines faster, any kind of virtual machine. In our case we may start unikernels, but you can start any sort of VM and you may have heard the term MicroVM. MicroVM by definition is any VM that gets started with Firecracker.

John: Okay. Yeah. I can personally attest to how chunky QEMU is. Lots of late nights with QEMU and KVN. It's very fascinating. Yeah, it feels like the next evolution from a Linux dev perspective, maybe too in the technical weeds, but the next evolution of being able to kind of align being with KVM and then some of the user space stuff around the application where containers almost felt like too far that way.

Would my assumption be correct there, where containers were very heavy on the application side, building a bunch of that stuff to support the application coming up?

Felipe: Yeah, so containers is almost, it's a higher layer isolation primitive. So the difference in a virtual machine is that each virtual machine has its own operating system, its own kernel. So if they're all running Linux, you have a copy of Linux on virtual machine A and you have a copy of Linux in virtual machine B and C and D and they're totally independent.

So whatever issue virtual machine A has in the operating system or anywhere in the distribution, it's not going to affect virtual machine B. Containers-- If you're running container A and then container B, you're sharing the same kernel underneath you. Right? So you have a much bigger common base and if that common base has any issues, you're all compromised together.

So for multi tenancy and when you go to the public cloud, oftentimes what happens when you send a container, deploy a container on the cloud, it gets put inside a virtual machine for isolation purposes. So now you have a container inside a virtual machine, which maybe doesn't make a whole lot of sense because ultimately you have application container, virtual machine underneath. Maybe you should just have application running directly on the virtual machine.

Brian: Yeah and I was going to ask a question because you mentioned Firecracker organically. But when I was trying to do research for Unikraft, I've been aware of it for a bit. John and I actually spent some time at the Linux Foundation earlier this year as employees and got to see a lot of how the sausage is made.

So Unikraft came up quite a bit. But I was curious, is this for replacement of Firecracker or what's the use case of where Unikraft is truly special?

Felipe: Yeah, sure. So Unikraft, the unikernel project. The best and fastest way to launch those small VMs is with a fast virtual machine monitor, namely Firecracker. You can also use Intel Cloud Hypervisor, you can use KVM, but that's going to go slower. Right?

So you can think of under the category of VMM you can choose either Firecracker or Intel Cloud Hypervisor or Kyma. They're doing basically the Same job. Launch VMs via KVM. Right? In our case Firecracker is the leanest, meanest, fastest one. So it's like a good compliment for a lean, mean virtual machine.

You put the sort of two together and that's Unikraft, the unikernel. One of the things to sort of expand. If you were to take that unikernel, and we did this naively, and turn it into an AMI and deploy it on Amazon and you hit go, it's not going to start in 10 milliseconds, it's going to start in 30 seconds or 40 seconds because obviously there's a lot of components that get in the way between you as starting something up and the virtual machine starting up.

So one of the things that we had to do as part of when we were building a product was okay, we're going to have to apply those same lean principles to all of the other components that appear end to end, when you send a request to a server and it actually sort of hits it and comes back. Namely the controller proxies the virtualization stack and then of course eventually the virtual machine images.

Brian: Yeah, so can we actually go back. You had mentioned it was a Linux Foundation project. So I'm actually pretty intrigued by that. So like what's the history? Was this started at the Linux foundation project first and then you decided to create a company around it?

Felipe: Yes. Basically it came out of research. I was doing a PhD at UCL in London and UCL has very close ties to Cambridge. In fact a lot of the Cambridge professors go to the pubs in London after hours. And so because of that connection we often also went to Cambridge and Cambridge was the source of the Xen Hypervisor project.

And back then KVM wasn't so well known or popular. So if you were doing research you were basically hacking the Xen Hypervisor. And sort of through that we started getting into that community, they have Xen Summit. And then somewhere along the way they said, hey, we have an incubation project. You can incubate your tiny unikernel project in that. And that kind of worked out and because of that we became the Linux Foundation project, which Xen is. And then from there we spun out.

Brian: Cool. And then do you still support the project through contribution? I guess you're still maintaining the project in general?

Felipe: Yeah, we're still maintaining the project. The open source project is the ability to build unikernels and then there's multiple targets you can run them on. I've been talking KVM a lot, but it'll also run on the Xen hypervisor. I believe there's some people getting it to run on Hyper V and other hypervisors. So that's more like the Microsoft route. But yeah, it's very much actively maintained.

Brian: Yeah, because I mentioned our time at the Linux Foundation, we also had Matt, who is the Commercial Open Source Software Foundation, also at the Linux Foundation, was one of our guests about a year ago actually. So we've had a lot of conversation around the business of open source.

So I'm intrigued by this because you've got this tool that works in multiple different ways. So Hypervisor and Firecracker, so folks are moving the needle forward for getting this to work other places. But how does this become a business per se? Like what is the value that Unikraft the company is providing?

Felipe: Sure, I guess the platform, as I hinted, is made up of multiple important components, one of which is the images. Right? But as I was saying--

If you have an image that boots in 10 milliseconds but you have a controller that starts things in 10 seconds, the end to end is going to be 10 seconds plus 10 milliseconds. Basically nobody's going to care that you optimize the heck out of these images on the back end.

Right? So the platform itself actually has a custom made controller, a custom made proxy, a custom made virtualization stack, and by now a pretty serious fork of Firecracker and those components are closed. So then the images are open and all of the tooling around the platform also is open source.

Brian: Okay, that makes a ton of sense. The value is like, basically you just have a better experience.

Felipe: Yes, basically the idea is if you can start any workload on the cloud in say, 10 milliseconds, cold, anything you can define in the Docker file, then you can put anything to sleep, scaled to zero the moment it's not active anymore. And then what the platform will do is when the next request comes, it'll buffer the request. The controller will say, hey, do I know what this is?

So the proxy will sniff it and buffer the request. The controller will say, hey, what is this? It will just say, hey, I know what this is. Will say to Firecracker, resume that VM, please. Then it'll tell the proxy unbuffer those packets. The packets get sent to the virtual machine, the virtual machine responds, and all of that happens within a cycle of about 10 milliseconds.

So the end user feels like it's always on, Christmas lights, all my services on all the time. But on the back end, anything that's idle, you just put to sleep. Right? This lets you bin pack a ton of workloads. So if you have 1 million users as your customer base, but you only have 1,000 active now, you can dimension your system for your active users and not for all of them.

John: Yeah. This is so interesting me, because years ago I spent time at VMware working on some of the Kubernetes platform stuff, and scale to zero was very hard on top of ESXi and vSphere because it took a very long time for one of those VSphere VMs to come back up.

And as soon as you scale to zero or needed to start to horizontally scale your services or something on top of Tanzu, which is all on top of the VMware stack, it was very painful for customers. So 10 milliseconds is wild to me. That's pretty incredible.

Felipe: Yeah. And also one of the things that we also needed to optimize was application start time, which we solved by snapshots. Because what you don't want to do is obviously be in the business of optimizing applications over and over and over and over again. So probably a lot of your listeners, or most have heard the term snapshot, where you basically, a virtual machine, you can take a picture of all of its memory, Right?

And once you've done that, you can put it to sleep and then wake it up from that snapshot file and it'll wake up exactly where it left off. In fact, it will never know that it was even put to sleep. And so what you can use that for is imagine you have a CI/CD pipeline. You're doing a release of your application, you built the image, you deploy it, and the first time you don't immediately make it available to your users, you let it run once, right? And you let the application fully initialize. Maybe it takes a second, maybe it takes two seconds.

Once it's fully done, you put it to sleep and you snapshot it, right? And once that's done, you make it available as a service and then it'll cold start in 9 milliseconds. So while the application maybe takes 30 seconds to get going the first time, the users will have access to it pre-initialized from the snapshot in 9 milliseconds, right?

Obviously the end-to-end delay will be as large as the crappiest link in the chain. And so we've had to optimize every single part of it all the way down to the application.

John: I'm so curious if you've found it difficult for entrenched companies who've really bought into like SRE practices and containers across microservices and adopting Kubernetes platforms and stuff-- Because, you know, I'm sold. Like I love the idea of being able to like snapshot something.

And I mean this really harkens back again to early in my career building stuff at VMware. We were doing this on the order of days, not milliseconds, unfortunately. But it was an elegant workflow that aligned really well with those enterprise use cases for vSphere and such. Have you found it hard for people to have that mind shift if they're really entrenched in the container ecosystem?

Felipe: Well, if it's containers, they're very used to the Docker file being the king of the primitives. And so we tell them, look, our platform is also Docker files. Give us your Docker file, we'll just run it. Right, okay. So that part's relatively smooth. And that also goes back to the principle we have around tooling.

The tooling team is as big as the platform team. And that's on purpose because we want the platform to feel very, very familiar, even though underneath it's not.

The Kubernetes question is a good one though, because Kubernetes is by definition slow to do a bunch of things, but it is super important as well. So how do you provide Kubernetes API-like stuff without sacrificing those millisecond semantics that we struggle for so long to build? And so what we ended up doing is--

I was telling you earlier about this idea of letting the application run once, as long as it needs and then snapshotting it. So we have a Kubernetes integration. And basically what it does, you can use Helm charts, you can use kubectl, you get the platform going once, imagine it's one server with our software installed on it and it'll pop up as a node.

So as far as Kubernetes is concerned, it looks just like a node. It just happens to be a bit special, it reacts a bit faster and it scales much better. Right?

John: Yeah, like a solo node type of cluster where like--

Felipe: Like a solo node that, you know, unbeknownst to Kubernetes sort of vertically expands and horizontally expands, but it's just a node, right?

John: Yeah, yeah.

Felipe: It's implemented as a virtual Kubelet. And then all of the scale to zero semantics, these millisecond semantics things coming on and on, that happens out of bounds to anything Kubernetes.

So to Kubernetes we just say it's all green all the time, always on, don't worry about it. But under the hood it's doing the ups and downs in millisecond semantics. So that's the sort of compromise to get the familiar API that everybody loves and is used to and Kubectl commands and so forth, but still keep the milliseconds semantics.

John: I love it. Wow. I'm speechless.

Brian: Yeah. John I was actually curious because based on your background I assumed you'd probably have more context from your background working on Bottlerocket, if you have use cases that you'd be using like a Unikernel for.

John: Bottlerocket was interesting because it built on a lot of core Linux foundations to provide about as secure as you possibly could environment for highly secure type of workloads. So that would then serve as a node in a Kubernetes cluster and then you could either schedule highly secured workloads onto that or just have an entire cluster full of Bottlerocket nodes.

It's a huge pain in the neck to work with though and develop on top of, because everything is container and there's this thing called SELinux which is this whole like Fedora Red Hat, like super secure lockdown Linux type of thing. So if you can work around those sort of restraints it's really powerful.

But yeah, I mean, Brian, you hit the nail on the head. I was going to ask next about like the security implications of building on top of a unikernel, because we on the Bottlerocket team went so far into containers to like lock as much down as possible. Like there's not going to be a shell on the system. Like you really only can run containers. There's not going to be Sudo or anything on there.

For a MicroVM with Unikraft, what are the security implications?

Felipe: Yeah, so basically everything on the platform is a VM or MicroVM. So that's security number one because you get harder level isolation on anything that runs on the platform, but then the unikernel principles default off. Unikraft itself is a modular operating system.

So that was by design to try to make it as easy as possible to configure it to the sort of minimal possible bits. And in fact some people took it to the automotive world because you can remove schedulers and really, really toy around with it down to the minimum.

John: Oh, sure.

Felipe: But more on the cloud side, yeah, you can build unikernels that are targeted to a Python runtime or just for running nginx or whatever it is. And yeah, by default none of these things have any SSH or any sort of the usual tooling that you have.

So you have a spectrum of how much of the Linux API do you support? Which is also why it took many years to mature the project and we've gone as far as even being able to support multiprocess and running postgres and things like that.

Of course you don't want to go 100% because at that point you're essentially implementing Linux to some extent. So for some really advanced workloads like headless browsers, things like that, we have a plug and play replacement where you can just change the configuration line. And now you're running a minimalistic Linux VM this time.

And because you have snapshots and the controller and all the other bits of the platform, you can still boot those very quickly.

John: Okay, interesting. Yeah. I think part of the pain that we were facing on Bottlerocket was that containers really aren't a security paradigm. And because you can rely on the, I guess, vertical isolation of a virtual machine, it makes a much better picture from a security perspective is kind of what I'm hearing you say.

Felipe: Yeah, and it's funny because many years ago when we were doing some of this virtual machine research, the container wave hit. Right? And anybody in the security realm knew that containers aren't more secure.

But the hyperbole of it was they're secure, they're light. This is the silver bullet for anything cloud and you don't need to look anywhere else for-- this will save us all type thing. Of course if you're a technical security researcher, you know that's not the case.

So we kind of were writing papers to say my VM is more secure and lighter weight than containers. But it took many years to get over that. I think containers have today they're at the right place. They're an amazing primitive for build time stuff for sharing dev environments. They're really an incredible tool. Not for nothing is our platform based on Docker files, but when you go to deploy and you need multi-tenancy and good isolation, they're no longer the right permitted in that environment.

John: Yeah interesting. There was something I read on Hacker News a week or two ago about how somebody hacked together a chat application between two containers that was talking on some like, I don't know what it was like slash etc slash proc or something. Something crazy that it was like basically unbound undefined space in Linux in container space that it was just able to communicate that information back and forth.

And obviously, you know, you would have to have both sides of that doing that. But it was just a little wild, because it definitely spoke to the misconception of containers being a security boundary, which they're not, unfortunately. Yeah.

Felipe: And it's funny because that's gotten-- These days you have things like isolates that are also thrown out there. It's funny how all of these things are just security isolation primitives put all at the same level. You have VMs or you have containers or you have isolates, but in fact they're not horizontal, they're vertical. Right?

VMs at the lowest level, then containers at the higher level and then very high isolates. Right? And the higher you go in the stack, the larger the stack under you which you're sharing is and the higher the possibility of vulnerabilities.

So it is kind of funny how-- I mean hyperscalers and the people doing multi tenancy know this, not for nothing, they're all running VMS underneath. Right?

John: Yeah.

Felipe: And when you deploy a container, it gets fit into a VM to make sure it's contained. That's the irony of it. You contain a container by putting it in a VM.

Brian: Yeah, I wanted to shift gears a little bit to go back to the Linux Foundation thing. I'm curious about your participation in the Linux Foundation. This is specifically a Linux Foundation project and not like a CNCF or Pytorch project, correct?

Felipe: Yes, that's right. We also have CNCF membership but when we went to them we said hey can we have one of those nice open source booths? We would like to also be CNCF. They told us literally there's a GitHub repo where they track issues like this and they had an open unresolved PR because they wouldn't know who would be governing a project that has both affiliations.

So they told us, you know, you have to give up your Linux Foundation nationality and come over to us. And we said let's just stay as we are.

Brian: Yeah, yeah, that's intriguing. It's funny, I actually do know the other half of that conversation. I was not in the room for that. But it was one of those situations where it did set a precedent moving forward of like there's no dual citizenship for foundation participation.

But I'm curious, like do you see the benefits of having a product that's very much ingrained inside the Linux Foundation? Do you get like marketing attention? I guess with CNCF, you don't have a sort of "in" in that way but like I guess I'm just looking at other companies that either are participating in the ecosystem or looking to participate. Can you speak on the benefits, if there are any?

Felipe: Yeah, I mean for sure the seal of approval of being a Linux Foundation project. You do have some marketing resources at your disposal. Early on I think we even had access to lawyers of the Linux Foundation that could sort of advise us. That was also there.

So our experience has been very positive. I cannot say, maybe the CNCF is the same or better, but we're not a CNCF projects so I couldn't possibly say, but we've been happy with the Linux Foundation.

Brian: Excellent. Yeah. Well Felipe, thanks so much for sharing about Unikraft. I did want to shift gears to Reads. But folks this is an open source project so if you're at all interested in what was mentioned or getting involved, check out the website Unikraft.com, but also check out the open source project as well.

Felipe: Which is Unikraft.org for the open Source project.

John: Oh, nice.

Brian: Excellent. Like a true open source, they got the org.

Felipe: We got the org.

Brian: So I got a question for you Felipe, which is, are you ready to read?

Felipe: I'm ready to read. I've never done that before. Let's do it.

Brian: John, do you want to go first with your reads?

John: Yeah, sure. So I had two reads this week. I think some pretty big ones and topical ones for us. But the first was this rather wild vulnerability in React that was disclosed by Vercel and then they backpatched it across React and Cloudflare and all this kind of stuff and it was being called React2Shell and this was a way to run remote code. So remote code execution vulnerability within server side rendered React.

I'm not really a React person anyways, but as far as I understand it, you know you can have components and things that get built on the server side. So when you're going to a website that can then send that rendered HTML divs, you know, all that stuff down to you that's been basically populated instead of doing that client side. People find a way to actually execute code, just any remote code on those servers.

This one is pretty wild. I've been reading through the like what it actually was and it, I still don't think I quite wrap my head around it. It's like some very out there bespoke things in React that they had to take advantage of to go and do this but very crazy. I'm curious Brian or Felipe if you saw this.

Brian: We did see this at at Continue because we are a Next app that's using server side rendering. So it was a a quick note from Vercel and then us just basically bumping the version to make sure that this is no longer a backdoor. Personally actually I don't spend a lot of time doing server side rendering apps.

For context folks, I worked at Netlify, we did the Jamstack back in the day. It was all about pre rendering and using client side bells and whistles. So I just never got super involved in the server side rendering world for any sort of serious projects. Even for OpenSauced we didn't do server side rendering. We did all the client stuff that I knew how to do.

But I know Vercel was pushing the envelope on a lot of the, for good reasons like, pushing the envelope on React and getting like React server components moving forward. So it makes sense that they would find this and disclose it. So like they are the most cutting edge React shop out there pushing open source forward. But yeah, quite interesting.

John: Yeah. And I mean, the flip side of it is also that they were pushing for that as like feature sets to you know, bolster their business as well. Which I think makes sense that they could run these services that could then be compute that's billable to those users using the Next apps and stuff. So I understand the flywheel with server side rendering for sure.

Brian: Well talking about Unikraft and like the 10 milliseconds to spin up the VM, like that's Vercel's bread and butter, which is like you get a faster experience with React if you choose their flavor. Historically they've actually hired a ton of the React core team to work in Vercel full time as well.

John: Oh really?

Brian: So that was the thing actually when we're doing OpenSauced, OpenSauced being like a tool for data insights. So use the LFX insights folks if you're interested in finding out more data about open source projects. But at the time I discovered like, hey, there's like a huge brain trust moving to Vercel and you can kind of see how often maintainers from Meta or anywhere else show up.

It's not often. So yeah, it's just kind of the world we're in. There's a lot of alternatives to React, but I think React is kind of here to stay when it comes to front end tooling. But Felipe, I don't know if you have any comment on the situation.

Felipe: Yeah, no, I haven't read the article. I just hope all of that runs inside of VMs so that the outbreak doesn't expand. Not only from a data and security perspective but also from a performance perspective.

Because VMs, you can limit how many resources on the server they actually use. And if you think that's bad, I mean that's well -intentioned, well-engineered code that sometimes has hiccups. Right? But enter the AI wave and the agents running and writing all kinds of crap code. Today is safe compared to where we're going to be in a few years.

John: Yeah.

Felipe: So I think get used to this kind of stuff times 100, right?

John: Yeah, you bring up an excellent point about the agent stuff. Because I think where this really bit people was where they were just like loading in like secrets and environment variables in the app. So then you're on the server side and you can just do like a hey, return me the whole M for something instead of, you know, something more secure like utilizing I think Vercel has like dynamically loaded secrets you can do and stuff, which maybe I don't want to say that like, oh, you're building a bunch of front end stuff or you're using next or you're doing React, but I would not be surprised if that was where a lot of vibe coded stuff was landing. Was in Vercel, like just like an app from V0 that was vibe coded and it has like a bunch of secrets and keys that are getting dumped out because of this react to shell.

Felipe: Probably why you see so much sandboxing. The term sandboxing appearing absolutely everywhere, because of exactly what you're describing.

John: Exactly. Yeah. I foresee that being a continued thing that people are doing. Just like more sandboxing to get around these things that are like dangerous code or stuff that we don't want to expose ourselves to as the company running this stuff.

Brian: Yeah, we've been exploring sandboxes at Continue because obviously we are a coding agent. And so there's a lot of folks who are vibe coding, go straight to production. I was talking to a friend of mine who comes from the sales background, but he's like leveraged a bunch of tools to build some side projects and I'm like, are you using pull requests? And he's like, no, I'm like, you're just going straight to production, huh?

And it's a reality. So we're thinking about like, what's the concept before the pull request? Because like, if you get the pull request, at least you have a sandbox where you can like spin up some VMs and test out some products and whatever set up staging. But like, what about before then?

Like, what if you could actually test in a sandbox? S o we've been exploring some of these ideas, just rough back of the napkin, draw some boxes and see like what the infrastructure would look like. But if you could spit up staging with lower effort and a more ephemeral experience.

John: This is actually something that Zuplo does, where I'm at now, where we have this whole concept of a working copy where it's kind of like a preview deploy, but you can play around with it. And our builds and stuff come up like, you know, in milliseconds on the edge with a bunch of these like super fast run times, like Deno and stuff now. So you could like just deploy a working copy really quick with a bunch of code changes and then there's your API gateway and test a bunch of stuff out that way you can then take that to PR, see the more in depth preview with like environments and stuff you set up.

But even that like we've, we've explored ideas and stuff. I'm definitely going to look at this, Felipe, with you know, fast kernels and the fast VMs to like get a good local dev experience where like all those dependencies could come up really quickly because that continues to be painful from like an end to end experience that even containers don't really satisfy.

Honestly Felipe, you're probably the best answer about sandboxes. Like, where do you see your customers using these kind of sandboxes and stuff?

Felipe: Yeah, for sure, ephemeral dev environments are a good fit, but also agents and agent generated code, headless browsers and then more traditional things like postgres databases and so forth. But going back to Brian, you know--

What's funny is that infra engineers still, to this day, say the phrase YOLO. "Let's just YOLO it," as a joke. But you know what, that's what vibe coding is. It's YOLO a thousand times a day, all the time. And we're going to get to a place where 90-some percent of the code out there is going to be YOLOed.

And so I wonder if at some point the vibe coding companies are going to have to have in their CI/CD pipeline, John what you're saying, a required step of having an ephemeral test environment before you YOLO it down to production. Because this thing where an agent, an LLM agent, randomly produces code and immediately deploys it and puts it out there, it's total chaos.

I mean of course it's the wild west right now and let's just deploy first and see what happens. But I think that wave is going to have to give room for a bit more sanity at some point.

John: Yeah, I agree. Well, I'll move on to my next read, which I think is very topical for this, where Anthropic and OpenAI and Block and a bunch of these other companies are joining a new Linux foundation foundation under the family of foundations.

This is the Agentic AI Foundation. They're donating MCP, which is huge. The model context protocol frequently comes up in the podcast as well as this thing called Goose, which we've talked to people on the Goose team before, it's kind of the agent for your computer. And then AGENTS.md, which is smaller project just around like consolidating how these agents can get context from projects and stuff.

But the MCP one was rather surprising to me. I'm also very curious who will end up being the executive director of this foundation. But all will be told with time I guess.

Felipe: Yeah, it's super interesting. I think it's a good initiative, almost like a no brainer type initiative. I'm happy the MCP stuff is open source. I think it needs to be a little bit like WWW for the new age.

Brian: Yeah I've been meaning to reach out to Den, who's at Microsoft who's been super involved in the MCP foundation type work and there's just like a ton of moving parts and with us only being a year in the MCP I think as of like a last week, it's interesting to see the sort of trajectory of now-- Okay, it's going to a foundation, we have proper governance. Well I wouldn't say we have proper governance yet. Like we have the hopes and dreams of proper governance in the future.

John: Right.

Brian: So like everything's TBD. We just had an announcement today but from my understanding and just again we were at the Linux Foundation so I was privy to a lot of these conversations and a lot of this work that was in progress. It seems like the ball's already been rolling at least for the past six months around this.

So I wouldn't be surprised if we ended up announcing beginning of the next year like here's the executive director, here's like the structure, here's plans moving forward and what we can expect.

Felipe: Yeah, you can reveal it now Brian, just tell us.

Brian: I think the only conversations that I know that were happening were more of like okay, where does it land? Is it its own foundation? Does it go to Pytorch, does it go to Linux? Does it go to CNCF? Which it quite makes too much sense but it seems like that's been figured out. But fortunately that was a little bit beyond my pay grade on the final decisions and the sort of crossing T's and dotting I's.

John: Yeah I am a little surprised it didn't wrap up into the existing AI foundation which I think from a meta perspective I'm surprised became its own foundation with the backbone being MCP because the whole flywheel is that like yeah companies, like yours, Felipe, join as members, donate money, they maintain projects, then also they get the marketing boost at events but those events and ticket sales to those, you know, go back into the flywheel and then there's courses and certificates and stuff.

And I just wonder how sustainable it is for such an early space to have like, I don't know what it would be like certified MCP developer or something or whatever these certificates might be, when the thing is changing so much, week over week. Who knows? I think it's a good call. I'm glad it's still in the open. That's my biggest takeaway from this.

Brian: Yeah, and I think it comes down to, I think the CNCF is like a behemoth of a foundation. Like there's a lot of moving parts. Linux Foundation is another big foundation as well, which Felipe, you could speak to that. But there's up and coming foundations that we're trying to build more ecosystems or-- "we are?" I'm not there anymore. The ecosystem needs to build where there's a lot of participation from a lot of these companies that are driving the future of MCPs.

So like WorkOS, as of tomorrow, will have their MCP night out here in San Francisco. Huge event has everyone who's ever been around MCP will join and I'm sure this will be the topic of conversation tomorrow. So I'll let you know next episode the details in the closed smoking room.

But I think at least if we could bet on open source and we could bet on an open ecosystem and we could bet on multiple stakeholders involved in the room. I have a very bullish take on that. More bullish about like I know ACP is part of the foundation as well. A lot of folks had question if MCP will still be around because Claude Code like ships skills and skills are like network turned off type MCP interactions.

That might be a horrible pitch on what it actually is but there's a lot of new evolving ecosystems happening faster and faster and I think if we spend less time on trying to figure out like what are all the things we're going to put on the stone tablets and like what is actually we can put in our actual lines of code I think we'll be better for that.

So I'm hoping the MCP Foundation pushes us towards actually practitioners shipping stuff as opposed to like getting a bunch of suits in a room to talk about the future.

John: Yeah, that's super fair. Well you heard it here first. Brian's going to spill the beans on the next episode, everybody. Brian, what about you? Do you have a read?

Brian: Yeah, I had one read, which is I saw a friend of the show, Thorsten Ball, tweeted on The X, about 200 tokens is plenty. Felipe, I didn't ask-- Are you doing any vibe coding, using any agentic programming?

Felipe: I'm not doing any vibe coding. I'm so disappointing. I'm really, really sorry. I apologize.

Brian: Okay. It's okay. We'll try to convert you next year. You just keep doing your old school abacus.

Felipe: I mean, I've been on Lovable. I've done some things, but I've done that some months ago. So I would consider myself a vibe coder if I was regularly doing it. I'm not.

Brian: Yeah, well, I think vibe coding has changed a ton in the last six months. So where you use like an Amp or you use like a Claude Code. I think the most important part is like, how large the session gets. And I think I was actually surprised with 200k being the number, because they had talked to some folks before that said the sweet spot was 100k.

But at the end of the day, it's like when you get to too many tokens, you start getting some hallucinations. It's like, hey, Claude, go home, you're drunk. Let's just start a new thing. Because that's where you start getting into some trouble.

So, being able to scope your work within 200k sessions, but also having that be your benchmark and what Amp calls, the article was written by Amp, but they call these threads. So like, you'll have a 200k thread and that's a good sweet spot to basically say, hey, let's pause there and either kick off a new session or maybe scope to work in a different way.

Felipe: Interesting. So you have like a token counter that lets you know when you're ready to reset?

Brian: Yeah, honestly I think Claude might not put this out in front, but I know, like, Gemini will put the token counter right there. Codex will put the token counter right there for you. And I think Amp as well.

John: Amp does it as well where it'll kind of prompt you to reset the thread or collapse it into a summary, so then feed it into a new one.

But I mean, I was going to tell you, Felipe, I mean, you're probably honestly coding circles around us, like writing hypervisors and kernels and all these things. I wonder how hard it would be to like go and contribute to a really deep stack like that, you know, versus something that has like all this training data available to it, like TypeScript and JavaScript and all these things. I'm guessing unikernel is in C.

Felipe: Yes, it's a good question because it's also a very common frequent internal debate that we have in our company because obviously--

John: Oh really?

Felipe: Yeah, yeah. Because we always, even at our off sites, we try to have a session where somebody tells everybody what the state of AI and AI tooling is to try to make sure that we're not missing some boat. Right? And we also do internal surveys. How much are you using AI? And there's a very clear correlation, as you go down the funnel down to the lower ends of the stack, people use it less and less and less. Right?

John: Yeah.

Felipe: So the people that work on tooling and coding Go and whatever more and more. Obviously there's the CI/CD tests and pipelines, that stuff for sure, maybe documentation or sort of the basis of documentation that you then curate and build on top of, yes.

But when you start to get down to Rust for Firecracker or C, low level stuff, those engineers are very reticent because they're like, I've either tried it or I know this guy next to me tried it and it introduced some really weird race condition that only gets triggered under load when running a thousand VMs on a box. And it took a week for me to debug it, which means that my productivity was a fifth of what it would be without AI.

This is a hypothesis that we test over and over, over time, because things are evolving so fast. Fundamentally, the skepticism maybe comes from the fact that to some extent LLMs are as good as the training data. And as you go down that funnel, there's less and less code available that does low-level stuff at scale, at high performance, under load, et cetera.

You have the Linux kernel and all the things, a few projects like that, but maybe you have hundreds, maybe a thousand, but you don't have tens of thousands or millions like you would for the upper stack type languages. Right?

John: Yeah. This was actually a huge problem for the Bottlerocket team, or at least in my eyes it was. I was there when ChatGPT first became a thing and obviously GPT 3.5, what they went out with for ChatGPT was not good with code. It was hallucinating and with stuff like Rust, like we were writing a lot of the pieces of Bottle rocket in, it just would not compile things and stuff that it could compile were just so dangerous.

Felipe: That's worse. When it doesn't compile, it's better, right?

John: Yeah, exactly, exactly. But we sort of came to the conclusion where it's like we're like the only thing that's doing this in the open source. We knew there were other-- like Flatcar's kind of doing something similar to Bottlerocket, but that's all like some Python stuff.

So it was very hard, we realized, for these models or at least very early ChatGPT now to adopt, you know, similar semantics and patterns because it was just like, it's just not in the training data. I'd be very curious to understand from that team how they're using it now because I know Amazon and AWS has Bedrock and a bunch of these things that t hey've built now.

It's a crazy world. It also makes me think of the Mitchell Hashimoto episode where he was describing to us how he was building Ghostty, at least at the time. This was months ago. I'm sure his workflows have changed now. But he was getting it to write C because it didn't have a lot of Zig in its training data.

Like, Zig is very new programming language. So he would get it to write C for the features he wanted and then like basically line by line go and take the C and put it into Zig. And I thought that was very interesting because it's like, yeah, you're like reviewing the code and you're understanding how it works and then being able to kind of do the mental mapping over to Zig is pretty impressive actually.

But still like relying on former semantics that would essentially be like C code somewhere that it's using. Right?

Felipe: So what you're saying, if I'm extrapolating, is that I should have our LLMs build Node.js code and then extrapolate it down to our C, low-level stuff and this is how we should build going forward. We should give it a try. I wonder if that would create race conditions or not.

John: Yikes. Haha.

Brian: Well, but before we start doing any bad ideas, I'm gonna have to wind down the show. Felipe, thanks so much for the conversation and sharing about Unikraft and unikernel, and generally just educating us. Wishing you all the best of luck in the future endeavors. And listeners, stay ready.