Ep. #54, The Age of Personalized Software with David Crawshaw

Getting Into Tech

Marc Campbell: Today, Benjie and I are here with David Crawshaw, who's currently building exe.dev. Welcome, David. So we're excited to have you here. You have a pretty amazing background and story through different tech companies.

Just to help get us started, would you tell us a little bit about how you initially got into tech? What was your first job?

David Crawshaw: Yeah, I got into programming before I got a job. I was programming as a kid. It was fun. I lived in a small town in North Queensland, Australia, where there's not a lot going on. I had a computer, learned to program it very early.

Benjie De Groot: What kind of computer, David?

David: They were mostly PCs because my parents owned a little medical practice there and wired up a network and my father moonlighted writing some medical record software for it. So these were MS-DOS PCs that we glued together with a Novell NetWare file server, so file share based, something that looked a bit like dBase III Plus for those who've seen it. It's a small programming language named Clipper, which was one of those sort of interpreted P-code languages.

That was like the first thing I ever learned to use. And I tried the QBasic thing because it was a Gorillas game, Weeks for Bananas. I don't know if you remember that on MSL. I played with that.

That's the world I started in. And then PCs turned into Windows and then JavaScript appeared at some point, but there were a bunch of other weird things in the middle there. Server-side rendering of web pages, all that sort of stuff. I just worked my way through the stack slowly over the years.

I eventually ended up with a job. What did I do? Someone paid me money for something. It was Google. They paid me money.

There was jobs before that, but they were sort of research work. But I ended up at Google mostly because I wanted to work with clusters of computers, like larger HPC style stuff, because that was a thing that my experience up to that point hadn't prepared me for.

And university for me was math and history. It wasn't computing. So I wanted to do something you can't do on your own.

And they provided that, as you could imagine. There was no end of opportunity to program large clusters at Google.

What year was that, that you joined Google?

That was somewhere around 2009, I think. Yeah, it was about then.

Benjie: So you were in Australia, and then you relocated for the Google gig?

David: I relocated for non-technical reasons. Family, basically, led me to the Bay Area. And then Google just happened by accident. Some recruiter contacted me, and I went down there to see them.

Actually, it's a pretty fun story, but it's sort of long and complicated. I'd have to get the ordering right to make it make sense. But from a professional standpoint, Google was like my first job in the US.

Borg and the Road to Kubernetes

Marc: You wanted to play around with large distributed kind of complex systems and stuff. And so Google clearly a good place to do that.

2009 though, much before distributed systems kind of became main, everybody's running Kubernetes today, right? Which came out of Google. So when you joined, was it kind of the Borg style and you adopted some of that tech or was it even before that was super popular there?

David: Borg was there. And I think about the time I joined is about the time a team started the Omega project, which is what became Kubernetes many years later. And so, yes, it was all their classic Borg system. And it was that the whole time I was at Google.

I was there some seven years and Kubernetes launched in the middle, but there was never actually an opportunity to use it at Google. It wasn't really used internally. I don't know what the current state of things are there, but all systems were built on Borg, including GKE at the time.

Because Borg provided features Kubernetes certainly didn't at that point, which were critical to running clusters. So when operations needed to drain racks of machines to change a power coupling or something like that, Borg had all of the machinery in it for doing that and meeting job SLAs and work like that. And I don't know if Kubernetes has grown all of that sort of pod live migration machinery yet, but Borg had it all and was designed entirely for that.

Marc: Yeah, Kubernetes in the early days was far from Google scale, too. Clusters of 10 nodes was a big deal in the early releases of Kubernetes. It's come a long way, for sure, right? Well, depending on it, for much larger workloads now.

David: Yeah, Borg was very much just solving problems. It was so interesting because when I joined, it was abandoned, basically. No one was working on it. And that's why when we started and then we turned into Kubernetes, it was going to replace Borg.

One of the interesting things that happened while I was there is some people picked up working on Borg as a side project and actually added a lot of extra features to it, which I think probably delayed the rollout of any new system like that at Google for many years. The original design doc for Omega had a whole list of limitations of Borg, and by the time any of that machinery was ready, most of those limitations had been removed by people just working on the original software.

Benjie: Borg seems like it's the B-52 of orchestration. For those that don't know, this is a bomber that the American Air Force has had for like 60 years. And we just keep extending the life of it.

Marc: There was a great white paper, I think from Brian Grant, Borg, Omega, and Kubernetes. They kind of talked through that journey. And we're not really here to talk a ton about Kubernetes, but the podcast is The Kubelist Podcast. It's kind of how we started. So it's always just interesting.

Projects at Google

So you joined to work on distributed systems, but what was the projects that you were working on while you were at Google?

David: Yeah, I started in a logs team supporting search quality, which involves lots and lots of data, as you can imagine, from all things search quality, which was really extremely educational because I had extremely limited exposure to the entire IR side of computer science, and so I wasn't really sure how any of it worked, and so I got to learn how it all worked, which was an enormous ton of fun.

But it also required petabytes of data here, petabytes of data there. And back in 2009, a petabyte was like a serious number. This wasn't like a fit it in your phone kind of thing.

Benjie: It's still a serious number. Come on.

Marc: It's only a serious number if you pay AWS to egress it. That's where it's really serious today. But no, it's large.

David: Yeah, so let's not pay egress bills. This was before a petabyte fit in half a rack. It felt like a lot of the time, like moving five petabytes across the country, which I think was my starter project.

There was a project that required moving some data from one data center to another and modifying it in process. That was a thing you would spend a couple of weeks doing. Now, it's not, because that's the nature of transistor density.

So, yes, I learned a lot about search quality in that process. I spent a couple of years doing that. Then I worked on Google Plus for a while there. That was fun.

I supported a lot of infrastructure there, which is also very educational. It's always interesting to work on a project that doesn't work. I met a lot of great people there. It was fun.

And then I ended up in a complicated way on the Go programming language team. And I ended up doing that basically because there were a lot of batch distributed system jobs in Google Plus that had been written using frameworks and machinery that didn't really solve the problems it had. And so I went and implemented some in Go and moved a bunch of internal machinery to Go before anyone else was using it.

The language I think came out in 2009, but a lot like Kubernetes, it wasn't actually used internally at Google for a couple of years after it came out. Because it was early days. Google had a very well-established system. It's very hard to introduce something to the enterprise, all the usual things.

And so I was one of the first internal customers of the language. And then I ended up on the programming language team out of that. And so I spent several years on the Go team.

And then I moved to New York in the process there and then worked from the New York office, which was fun. The Go team was split between Mountain View, New York, and Australia at that time.

Oh, that's like perfect for you.

Yes, there were people everywhere. It was fun.

Go was extremely educational. I'd done some programming language work before and some compiler work before, but I got to get really knee-deep into a compiler and runtime that was actually being used in production, which was extremely educational.

Again, I very much view the way I survive in an enterprise is finding something to learn in the process of doing the work. If I'm learning, then I don't mind digging into ancient code bases, the classic enterprise problems you run into.

And then in my last year at Google, I was actually on the Fuchsia operating system team working on their networking stack for a variety of reasons. But it was, again, a very interesting problem. I got to do a lot of interesting stuff.

Fuchsia had a microkernel design, and so it was very much a user-based program implemented the networking stack, which is really fun. And so you're trying to design modern system calls for that that worked on modern hardware because your theoretical approach to microkernels is very inefficient, and so you have to find efficient ways to do it.

On the networking side, that's actually pretty easy, but you have to think it through. You have to say, well, each system call has to be amortized over the cost of a set of packets rather than just a per-packet style system call. And so it was, again, very educational.

Starting Tailscale

Then at that point, I had a kid, and I left Google, and I had met my co-founder Avery in the process because he was Google New York as well. And he left for different reasons some six months later, and we said, let's start a startup.

And that turned into Tailscale, which is a networking company that does overlay networks and it was doing really well. It was very fun. We've been building Exe on it, so now I'm a big customer of it, which is really good.

Benjie: Well, give us a little bit more juice on Tailscale because we all love and use Tailscale all the time.

David: Yeah, of course, I have many stories because of many years of it. There were three of us who started it, me, Avery, and David Carney.

We were distributed from the beginning in 2019 when we started, so this was a pre-COVID thing, and we were distributed just because back to Montreal. I was in New York and David Carney was in Toronto. And so it's like, well, I guess we're distributed. We actually had a plan to start an engineering office in Seattle and hire out engineers there. And I would fly there every month was the plan. And I did that once in January 2020. And then that was the last time I did that.

Marc: In COVID. So that ended that.

David: That's right. So we got pushed in that direction to build a company. But there's lots of fun stories. We actually started with the notion of, well, there's a whole lot of things that are far too difficult in computing for no good reason for many people. Let's go and fix one of them.

We knew the general sense of the sorts of problems we wanted to work on, and we decided on the people, the three of us, to do it. We didn't actually know what to build when we started. We built it by going and finding potential customers, working with them on problems. And we spent a long time staring at Kubernetes in the process. We saw how small companies were using Kubernetes and they were spending far too much time operationally managing it.

And we said, this doesn't make sense. Your systems, the computers you use, shouldn't be this hard. And in particular, there was one startup we went into that had a whole series of Kubernetes clusters that they used — a production one, and they had all these test ones. And we just added up all the jobs and everything they were running on the system and we realized you could buy one moderate-sized computer and you could run all of those programs on it.

They had several full-time staff managing the operations of their clusters, and the fact that all that operational management fit on a single computer told us there was some fundamental issue here that we needed to solve, because managing a computer shouldn't take three people. It's one machine.

Benjie: And for context, this is 2019.

David: This is 2019.

Benjie: GKE is working. I guess EKS is just out. So yes, the complexity of running Kubernetes itself was infinite. And also overkill is kind of the big thing. Pre-K3s, pre-k0s, all that stuff.

David: Yes. And in a modern setting, you'd say, go use GKE or EKS or something like that. It would relieve a lot of their pain.

The fascinating thing is we couldn't figure out how to fix it. Honestly, we were staring at that problem. It was too hard for us. We were like, there has to be some vastly simpler way to solve what is effectively an organizational problem.

And we couldn't find a way to it. But there was another company we were working with simultaneously. There was a small Canadian bank.

And they had this fun problem: they had written a whole lot of .NET software built on Windows file sharing, which worked and had been working for 15 years. And then they had a phishing test done, which they failed because everyone fails a phishing test at a company when you do it.

And they got the standard advice from the auditor to put two-factor on all their systems, which is, again, very reasonable standard advice. And it's very hard to layer two-factor onto Windows file sharing. In fact, there was no mechanism at the time.

And when we did the research, we said, how do other banks solve this? This is a standard sort of problem.

It turns out the standard thing to do is to buy virtual desktop style software, so VDI machinery, which is like Citrix and all that sort of stuff, which takes a Windows setup and puts it in a box and then you put two-factor on that, was the idea.

And no one likes using that, it turns out, unsurprisingly. Why no one likes using it, no one can quite explain to me, but I feel the same way when I use it.

I'm like, this feels like there's one too many layers here. Why is my computer in a box? What's going on?

And so we went looking for other solutions. At one point, we said, wow, it's too bad we can't just two-factor all of the network and disconnect the file sharing connection that way.

We're like, oh, that must be possible. Let's just use a VPN. We'll go find a VPN that has good two-factor support and use that.

And we couldn't find a VPN that had good two-factor support at the time. And this was just sort of a really interesting moment in that industry, in computing.

The one piece of software we found that supported any kind of one-time password code had it such that when the login box appeared on your screen for the username and password, you type in your username, you type in your password, and then you would append the code to your password in the password box.

This was their idea. That was how they hacked it into their software because it was all very much surviving on shoestring budgets sort of thing.

So it was like a new hash of a password? Yeah, I know. It's kind of astonishing.

Marc: I feel like some enterprise software I've used, I maybe use that software or something because I remember being like, wait, this doesn't feel right at all.

David: No, exactly. And you can see the layers of software they're not allowed to touch and where they hacked it in in the middle.

And I've written a lot of code like that. So I sympathize that the poor engineer had to solve the problem. But I also couldn't inflict this on our customer.

And so this was one of those problems. We're staring at it going, oh, this is silly. I could write this in a weekend sort of thing.

From Weekend Hack to Product

So we hacked up one in a weekend and it ran on a machine in my closet in a Manhattan apartment. And Avery said, well, there's this new WireGuard thing. We should try it.

And I went and looked at it. And I'm like, oh yeah, this is great. It has a config file. We'll write software that writes WireGuard config files. And that literally was a weekend project.

And it provided the basis for a network that kind of worked for them. And they liked it. And we did the standard thing of rotating.

We had it expire every N hours when they had to log back in. And then we threw in two-factor into the UI. All very simple stuff, nothing clever at all.

And then we said, okay, let's productionize this and turn it into a product. And in the process of turning it into a product, we started writing all these design docs that looked like very traditional VPN software of, here's a box you deploy in each office and they interconnect with each other. The local machines connect to those boxes and you build a networking system out of it.

In the middle of doing that, it all got very complicated. And it's all taking a long time to build. And so I said, well, I'm just going to simplify all of this because I want to start using it locally.

Let's just take the control plane and just have one of them, we'll put it in the cloud, we'll have everyone connect to it in a multi-tenant way. And then we'll make the machines directly connect their data planes together.

And there's an email I sent Avery and Carney at the time, I said, I'm just going to do this because it's faster and we'll fix it later.

And I think Avery wrote back and said, yeah, mesh networks never work, but that sounds fine for now. And like you said, we'll fix it later. We had a backup plan for how we would work around this.

And it turns out you can actually go really far with direct connections. We started layering WebRTC techniques and getting machines to connect.

And so what was actually a short-term hack just to work on the software ourselves turned into the product and ended up creating a new way to use networks.

And now I think of it as normal. Of course, my machines directly connect over an overlay network. And so it was really fun.

And it was one of those classic threading-the-needle problems of, yes, mesh networks are infinitely complex and never work. But if your mesh is just a set of direct endpoints connecting, so it's only ever pairwise communication through the network rather than you actually form a complex multi-step graph, the problem greatly simplifies and you can actually push that concept really far.

Benjie: So wait, what year — obviously you guys had gotten together, worked on other stuff. When did that form? When was the weekend project, approximately? What weekend was that?

David: Oh, it was early 2019.

Benjie: Okay, early 2019. And so then you simplified this again — mesh networks never where it gets. I want a shirt for myself because I can't tell you how many ideas I've come up with over the years, especially — and David, I'm in New York as well — I'm sure every train ride you ever took, you had the same thought as I did: oh, why can't I get reception in the tunnel? I should just do a mesh network on the trains.

We've all had the exact same idea. Right?

David: A hundred percent. I didn't have an office at the time, right? I was in a New York apartment. It was a bit small, had a young kid.

And one or two days a week when we had some babysitting support, I'd catch the D down to the New York Public Library and work out of there.

And I'd be hacking away on the train because the D was really quiet that time of day because I wasn't going to commute hours. And then I'd be sitting in the library hacking on it.

And we came up with the whole relay system model because the New York Public Library blocked UDP at the time, and so I couldn't get direct WireGuard connections out of it. And it was just, okay, well, I have to have a TCP relay system then. It was all just implied from the work.

We really stumbled our way to that product, but stumbled with the direction of, we had a paying customer who was using it.

And it gave me a lot of insight into building products generally, that you can guide a product with a customer, but you also have to explore and stumble around a little bit to really figure it out.

It doesn't quite fit with the whole startup should focus and just do one thing and all of that sort of stuff, because there were kind of two things going on simultaneously. There was this bank's VPN problem. There was the 'how do I connect my computers at home' problem. I was trying to merge the two and then find a simple architecture to do it.

Benjie: I mean, I can make the argument that necessity is the mother of invention is kind of the startup way. Once you figure out how you focus, that was really good.

Wait, quick question, a little technical. When did you guys start thinking about STUN and TURN and getting through?

Because you started this at a bank. So I assume that getting to that control plane was already a pain in the backside. So was that out of the gate? Did you do TURN and STUN and all that stuff?

David: The control plane is super simple. It's always TLS connections to a public machine. And so it doesn't need any relaying system at all. On the data plane side, whenever you can't get a direct connection, you work through a crypto routing relay system, which looks a lot like TURN.

I didn't actually know about TURN at the time. I learned about WebRTC after sketching out a design for this. And crypto routing was just like the fastest way to get there.

So every WireGuard peer has a public-private key pair, and you configure each peer with the public keys of the other side. And so the reason it's a crypto key routing system is just you take the public key of the peer you're trying to send to and you box up the packet you want to send, encrypt it with that key, and you send it to the relay server. It takes the public key and just figures out where that peer is based on the key and sends it to them. And so we use keys as IP addresses.

And that showed up pretty early. It was the New York Public Library that drove that. It was the realization that UDP is blocked there. It's like, okay, then we need a TCP fallback.

Okay, we should run public servers on TCP and we should figure that out. And so that's where that came from.

The bank didn't have that problem because they had laptop users connecting into their offices. And so their office machines had public IPs that were easily reachable. There were some 60 or 70 people doing that. And none of them were on networks that blocked UDP. So it didn't come up in the customer's case.

Benjie: Okay, so the Lions were responsible for you guys figuring this all out. Okay, that makes sense.

David: Yeah, the Rose Reading Room.

Marc: I want to go back though. I think you said something super interesting that I think it's nuanced and often lost.

It's like trying to figure out this product to build. You can't just go heads down and build a product and have a solution in search of a problem and then go like, oh, who wants to buy this? Okay, that rarely works out these days.

But also, just build what the customer wants is also not it. I think the way you described that was really good.

And if you have the opportunity to work with a customer who identifies a problem, but you have the space to— You didn't just deliver two-factor into a Windows file share. That was the problem that they presented to you.

And you ended up building Tailscale based off of this, which is in nobody's mind, they could squint and see that as a solution. And then, to your own story, you couldn't even squint and see that as a solution. That's just the solution that came out of it.

David: Absolutely, yeah. I mean, it was months of exploratory work and solving multiple problems, both for me and them.

There's a thousand ways to build a company. And there's a thousand ways to build a startup, which is a subset of companies. And sometimes you can build a startup by just going and solving a problem the obvious way.

The advantage I bring to startups is thinking about new technology and architecture. And that is a process that I really describe as stumbling through. Because I'm trying to discover requirements and then discover potential solutions to it, and I need to find time to do that.

And so that's how I approached the problem. And yes, there's no way I could have designed this up front. I couldn't have written a doc and said, here's how we're going to do it, iterate it on the doc, and none of that waterfall-style process would have worked.

Benjie: Well, you did write a doc and you threw it out because you stuck with the mesh.

David: That's a good point. We wrote several docs, yes.

Stumbling Into exe.dev

Benjie: Okay, so then you're at Tailscale. Obviously, I think every one of our listeners knows the story, or not the whole story, but they know how successful Tailscale has become, how ubiquitous it is.

You are there for a few years. I would love to ask you a million other questions, but we're actually here to talk about exe.dev, and I'm very excited to talk about exe.dev. So speaking of stumbling, how did you stumble into exe.dev?

David: Yeah, I mean, I was at Tailscale for a while. I was looking for ways to use LLMs because they're a lot of fun.

Tailscale was in very good shape as a company. I had effectively hired myself out of a job. I think one of the critical roles of a founder is to try and replace yourself with people at a company.

And I couldn't find a good way to do anything with LLMs at Tailscale. We got asked at several board meetings, what's your AI strategy? And we were like, well, LLMs are awesome, but how do you fit a heuristic machine into networks which you want to be as deterministic as possible? We already struggle with that desperately in networks.

And I couldn't find anything there, so I said, let's step back, do it as a separate entity. So I stepped back from the role and then finally completely at the Series C and then started a new company with a friend of mine.

And we were very quiet very early because we played with lots of things. And we went into it saying, look, we want to improve developer tools using LLMs.

And this was nearly two years ago at this point. So this was pre-Cursor or any of those sorts of products. But we were sure that LLMs were good at generating code in various forms and helping with writing code. And there must be a way.

And so we built several products in the process of doing this. We started with Copilot's code completion system, which was the only product in the space at the time when we started. And we said, let's make it good for one programming language. And so we built a bunch of machinery to make it extra good for the Go programming language.

By the time we got it working, it did work. It was actually really nice.

We used a bunch of techniques to bring data in very quickly from the LSP into the inference process of generating the code. And that greatly improved the results you saw without slowing down the completion, which was a really interesting problem because you have to work really quickly, which in every role in LLMs, no one does. Everyone's very slow.

We got that working. And as we got it working, we could see agents over the hill. They didn't exist yet. There's no Claude Code or anything, but we knew.

Benjie: Sorry, David, just real quick. Exactly when is this? Obviously, Cursor and tab completion was Cursor's thing at the beginning there. And it sounds like this is before that. So it's like 2024 early, maybe?

David: Yeah. So let's see. Claude Code, like the beginning of agents, was early last year. I was like, yeah, February.

Benjie: Well, January, February of 2025.

David: Yeah, so this was later in 2024. So it was somewhere around August, September, October 2024. We had this Go code completion system working. Cursor existed by the time we were doing this, but it wasn't ubiquitous.

We built a VS Code plugin, which I don't recommend. We spent months fighting VS Code's plugin architecture. It turns out that's a huge mistake. In retrospect, you should fork VS Code.

That's what Cursor did. Yeah, well, we considered it.

We discussed it and we're like, no, the technical debt of doing that will be too high. We'll use their plugin architecture. It turns out we were wrong. You really should fork it.

The plugins give you just enough API surface area you think you can solve problems and then you can't. And you don't discover it until far too late into the process. Which, in retrospect, isn't surprising at all. That's almost every plugin architecture I've ever used. So it just wasn't obvious at the time.

Benjie: Well, the mesh network thing works. So it's fine to figure that you're one for two.

David: Well, if we're actually counting, it's more like two or three for 10,000. Almost everything I do fails. And that's the nature of exploration, right? It's just a constant series of failures and every now and again, something works.

I used to actually keep a list of my failures when I was at Google. We used to have a NetApp back at the beginning of Google. I had a doc of everything— every little project I tried at Google didn't work. And it was dozens of projects long.

Of course, one of my many failures is failing to retain that document when I left. So it's gone.

Benjie: Well, it was petabytes, and that was a lot back then.

David: Yeah, it wasn't quite that bad. It was fun. Anyway, let's see.

So we decided that tab completion wasn't the future. It wasn't worth pushing on, even though we had a thing that worked. We went about trying to figure out just pure code gen.

We built a pseudo agent that would generate Go packages for you and launched it as a website called Sketch. Again, this was pre-Claude Code.

Then we realized we should build something that runs on your local machine. And we spent a lot of time solving a lot of Git issues around that.

So we built a system for running in a container using Docker on your local machine. And then we would push the Git repo in there. We'd do the tool-calling LLM in there. And then we would extract the Git commit out of the container and put it into a branch on your Git repo.

And while we were building all this, Claude Code launched with a much simpler MVP that actually just was easy to get into. We launched our product not long after, but obviously, in retrospect, we used Git for the user interface. User interface was you issue a command and then a Git branch appears and you have to change branches and go look at the commit.

And there's no easier way, it turns out, to make your product inaccessible than to require people be Git experts to use it.

And so it wasn't the best user interface, I would say. People don't generally say nice things about Git's UI. I wouldn't recommend building a product around it. And so Sketch was just too inaccessible for people.

But in the process of building Sketch, we went and worked with some companies. And the first thing we discovered is when they tried to use the agent, it would try to run their test suite, which is the secret to making agents good, of course. Still is to this day. You've got to let agents actually do stuff.

And their test suites wouldn't run in a Docker container. And they wouldn't run for super obvious reasons. Like the first thing their test suite would try and do is start a Docker container inside a Docker container, and that wouldn't work.

And so we're like, oh, well, we'll ship it to the server and we'll use a cloud system we run and we'll use gVisor to back Docker, which lets you do Docker in Docker. So we solved that problem, went to the next company.

They tried to use Docker Compose in their test suite, which can't run in gVisor because it goes and intercepts DNS packets at the network interface layer, which is not a syscall gVisor exposes.

And so each time we would go to a company and try and use it, we would hit some new limit of the container architecture we were in.

Just Use Virtual Machines

And it was somewhere along there we said, look, what are we doing? These VMs just need a computer. We just have to use virtual machines. We have to give up on all of this attempting to make containers work business.

And so we started building that. And this was in about June, July last year.

But the product Sketch, everyone was bouncing off. People would come in, try it, and they'd stop using it. And so we had a product problem at the front while we were doing all this infrastructure work.

And it was in the middle of doing this, we said, well, actually, the most interesting part of our product is the infrastructure. It's the fact that we've made VMs easy to use. Let's make that the product. And that's where exe.dev came from.

So sort of in October, we built out exe.dev and launched that in December. End of December last year.

And the key insight of exe.dev, which was new at the time— the first thing I did was look for products that solve the problem. I don't build something if I don't have to.

The problem is, with agents, I need a lot of VMs. I need those VMs to be long-lived. I don't need the sandbox products as they're called, which are mostly short-lived VMs. Because sandbox has come out of the whole RL training world where I need a Python interpreter for a second.

These are like, I want to start a server. I want it to run for a few weeks while I work on a piece of code. Or I want to start a server and just keep it running.

I just want to run my blog things. And I want to build apps, basically, and I want the apps to stick around. But what's different about VMs that I could get elsewhere, for example, EC2, which is my classic source of VMs, or DigitalOcean, one of these products, droplets. The problem is I now need a lot of them because I can build an app by typing a sentence into my phone and letting an agent go build an app.

And I have hundreds of these things. And the problem is an EC2 VM costs me like 20 bucks a month, and a droplet—there used to be droplets that cost five bucks a month. I think the cost of compute's gone up, and so they can't sell them that cheaply anymore. But every VM I could get my hands on cost me money for each VM. But all of my apps are idle almost all the time. I forget about them for weeks at a time. They don't cost me five dollars to make, because I literally do it while I'm walking down the street. So I need lots of VMs. I need VMs the way I used to need Docker containers.

I need to just have hundreds of the things hanging about.

And what we realized is, from a technical perspective, that's actually very straightforward. What you do is you allocate compute for all your VMs to a cgroup parent in Linux. And you have this cgroup parent, and then each VM you start is a Linux process.

That's what a VM is. There's nothing else to it, really. I mean, that's a statement that's a bit like saying it's just a simple matter of programming. That's not really true.

But at the heart of it, a virtual machine is a Linux process. There's nothing else in there. And I can have as many Linux processes in a cgroup as I want that share resources, just like Docker containers do.

And it was odd to me that no one sold that as a product. Like, let me buy five CPUs from you and run 500 containers on it. Or 500 VMs, I mean.

And the only way to achieve that with existing systems was to go and purchase a VM from someone like Amazon, EC2, and then use nested virtualization on it, and then you pay the nested KVM cost of trying to do that. If you wanted to avoid that, you had to go buy a bare metal machine on Amazon to write all the software.

And then you don't just need to write the VM, you need to write the surrounding machinery. You need to go and write a web reverse proxy with an IAM proxy that connects to each of your VMs to let you get them URLs on the internet.

At some point we said, well, this is a hard enough machine to build, we should just sell that. And that's what exe.dev is. It's as many VMs as you want on a certain amount of compute you're paying for, with all of the access to the VM taken care of for you. So that was the core product.

We built another agent to put on it just because it turns out building agents is really easy. Everyone should build agents. And so we have an agent called Shelly.

The key thing about Shelly that's a little unusual is it's a web-based agent. Almost all agents are TUI-based things. We built it as a web-based agent specifically so you could use exe.dev on your phone. We just wanted people to show up and be able to build an app on their phone.

And it's surprisingly popular. People really like it. And if anything, we should make it a more prominent part of the product.

We don't even say on our website that you can just build apps directly with an agent in your VMs. We should make that a thing. So that's a thing I'm thinking about a lot.

The Cost of Nested Virtualization

Benjie: Quick question for those that are listening. You said there's a nested KVM cost. What do you mean by that? Just real quick.

David: So there's a trap system in your CPU for the physical hardware support for virtual machines, which helps a lot with managing several elements of the virtual machines. They all have their own TLBs and whatnot in the CPU.

And when you have one layer of VMs, so you have a hypervisor and virtual machines, the CPU switching between the virtual machines is extremely cheap. This is how VMs work. Otherwise, they would be untenable as an object if we had to fully virtualize them and do it in software.

So the CPU reconfiguring itself to run VM2 after running VM1 is extremely cheap at the first level. When you go down a second level and you try to run VMs inside your VM, you lose some of the hardware support. That would be the very high-level description of this. And so there's a higher cost associated with this.

And effectively, every VM switch, where it once cost you the equivalent of one system call-like object, now costs you several at one layer down. And if you go down another layer, you get this non-linearity actually in the cost of virtual machines.

So a VM inside a VM inside a VM is an extremely expensive object. I don't recommend it.

Benjie: Right, because you're leveraging the CPU architecture where there's the switching costs. And so is it 10%, 30%?

David: It's changed a lot over time, actually. I think it got down to nearly a 15% to 20% cost for nesting one level down, so VMs inside VMs.

Then all of Spectre happened and all of the side channel attacks on CPUs that slowed down system calls. And now I don't know what the cost is. I assume virtual machines paid a cost in that process. I would have to go and do the work.

When we first got virtualization hardware, nested VMs were extremely expensive. And they drove the cost down slowly over time by slightly improving CPUs for that.

Again, if you're dealing with two vCPUs and you don't really care about the performance of your nested VMs, you can totally do it. I do it for testing, right? I bring up exe.dev inside an exe VM and then nested VM.

All of the exe VMs I create are nested and it's fine. It's just the test suite runs quite slowly if I do that. We actually have bare metal hardware for running the test suites. It's much faster.

Marc: How do you actually think about that, though? So I think you're right. I think that the term sandboxes doesn't really define anything.

The remote agents, it's the use case, right? I'm going to do some RL workload, and I might need 100,000, 500,000 machines. At that point, speed to boot those is really fast. Performance to execute that's really fast, because that job that was going to take a day, I don't want it to take two weeks. That'd be silly.

But when I'm walking down the street and I pull out my iPhone and I type a prompt in because I want to have exe.dev build an app and Shelly's going to execute on it and it's going to run on bare metal and let's say it's going to run for two hours to build that app. That's how long the agent needs. If it was like a 15-20% overhead and it ran for two and a half hours, do I notice? Do I even care at that point?

David: Yeah, it's a good question. I would say the way I might make someone care about it: charge 20 bucks a month for two vCPUs for exe.dev. And so if I add 20 to that, do you care? We get a lot of complaints that 20 bucks a month is too much, and I'd love to make it cheaper.

I don't think making it more expensive would make anyone happy. And so it's fair. But absolutely, in terms of my laptop, right? If I run some nested VMs on my laptop, it's mostly idle most of the time. I don't care about the overhead there. I care about it in the test suite because it increases the wall time of the test suite, which sucks. I like all my tests to run in two minutes.

But yeah, you can totally do it.

Under the Hood: Snapshots and Live Migration

It turns out, though, there's lots of surrounding machinery to build around the VMs we want to run agents in. And that's been most of our effort at exe.dev — that and the VM management layer of live migrating between hosts.

We actually have a lot of really quite clever stuff under the hood at exe.dev that we haven't written about or advertised, and we really should. We do a lot of engineering without telling anyone about it.

Like, for example, exe VMs run directly on NVMe on the hosts they run in. So they're all local. And then we have background processes that stream diffs to the block device, the virtual block device, off that machine to another. And so there's a one-minute, at most one-minute-old replica of your VM on another machine.

Benjie: No, wait, wait. I want to make sure I understand this. So each physical host that is running the 50 VMs that I get for paying 20 bucks, it gives me two vCPUs, all of the file system, and actually everything is stored on local disk NVMe.

And then you have like a cron job or some type of job that is constantly doing diffs of the blocks to the virtual file system. So you have, max, a minute. I know I'm just repeating what you just said. I just want to make sure I understand it. Is that right?

Yep, that's right.

And then, so from a snapshotting perspective, that's very obvious why that's awesome.

How often are these VMs turned off and on? By the way, you're going to tell us more of the secret engineering stuff in a second. I'm sorry for it. It was very interesting. So if I turn on a VM and then I turn it off, obviously it flushes the disks to the virtual block device. How long does it take

David: to turn it back on? If you turn it off, as in you power off and then you have to bring it back online, it'll take a few seconds. But almost all of that time is inside your VM getting systemd working again, doing a whole bunch of Ubuntu stuff. It takes a fraction of a second to bring a Linux VM back online.

Benjie: But what if I have 100 gigs of something running in memory? What if I'm running a Postgres database that's got 20 gigs of memory being used? Which does not really fit in the size of what you're saying, but let's say I'm using a more expensive

David: plan on exe.dev. Yeah. The actual underlying hardware can do that in a fraction of a second, but given the fraction of the hardware you get, it might take several seconds.

I would suggest not turning it off. That's our general answer. Just keep it running.

Under the hood, we're doing a lot of things, right? On idle VMs, we do classic sort of ballooning techniques, and then we swap out a lot of your RAM.

And so if the machine's idle, you're effectively bringing it back from local NVMe swap when you do things. And we have tiered swap. It's not all just simple swap.

And all of that is about sort of reducing the cost it takes to run these machines. Because again, the vast majority of the VMs we run are idle. The thesis that we started with that people need a lot of VMs, but they don't need them very often, turns out to be true in general across Athlete. We have some users who keep everything hot all the time, but it's not that common.

Bare Metal Economics

Benjie: So underneath the hood, what kind of machine? Is this AWS bare metal? Where are you guys doing that?

David: We do several things. Again, we should have a page telling people about this. We've got a lot of talking to do.

So we do AWS bare metal for customers who want it, and we have some legacy users who are still on some AWS bare metal. Most of our machines we buy are physical bare metal devices, machines we buy on one or two-year terms from various providers.

We do that in several physical locations. And you can pick your location in our site. So you can have machines in LA or in New York, in London or in Sydney or Frankfurt, I think we have. And that's through sort of bare metal host providers. And then we are purchasing and racking machines in San Jose to build a region there because the nature of the cost structure of machines makes that work.

The AWS stuff is extraordinarily expensive, of course. It's several times the cost of doing it ourselves, which is why we're building our own machines out, unfortunately.

Benjie: Right, so you're leveraging like Latitude, maybe Hetzner and...

David: Latitude do a lot of our stuff, actually. We have a lot of machines with them. They're great. And yeah, we use others, too.

We don't use Hetzner, but there's others in the next. They used to be cheap. They just got super expensive, so...

Marc: Well, they don't offer really bare metal in the US, though, either, so there's some limitations around the platform.

David: Yeah, Hetzner are very clever in what they offer. They offer older hardware. They have an extremely heterogeneous mix of machines, which is really fascinating, but their machines are mostly very small. We use relatively large machines for our serving, because again, when you have a long tail of idle VMs, you get a lot of advantages by putting many on one machine, which is you get sort of a law of large numbers effect around a lot of problems. So we face a couple of challenges with saying we'll run your machines, and then most of them are idle. And again, this is where some of our clever software comes in. We effectively have to live migrate VMs off hosts that get too warm if a lot of machines need to come back.

And so we have to have a system for draining them over. And that's actually where the constant snapshotting comes in handy.

If we need to get a VM off a machine and we have an almost perfect replica of its disk somewhere else, then the cost of getting a VM off the machine is taking the block device diff and taking the RAM, zipping it up and sending it over the network. And so on most VMs, that actually ends up being quite small, which is nice.

So if you have a 100 gig disk, we don't have to copy the whole thing. So that lets us drain machines really quickly, which lets us balance out load.

But the truth is, with very large machines, and because most VMs are quite small, it's very infrequent that we have to balance out load, which is really nice. The draining machinery is mostly in case we need to swap out a machine and replace it with something else. That's what it's there for.

Benjie: And do you have networking stuff there to cache requests going in while it does the swap, or it's not even necessary?

David: Yeah, with the TLS proxy, we basically put a pause on the HTTP requests that are incoming. Right now — I'm not a huge fan of this — we actually cut the SSH connections when we live migrate a machine, which is why I don't like migrating them very often. There are techniques for preserving generic TCP connections to VMs as they migrate. They're very tricky to get right.

Benjie: Are you sure that's a thing? I mean, you're the Tailscale guy, so you know better than I do, but that sounds impossible.

David: Yeah, I sat next to the Google Cloud team that built it in New York and watch them struggle for years to make it work there. The plan was for it to go on under the hood under the compute products there. So they could move machines between physical compute without you noticing, which helps with uptime.

You can really hit your SLAs if, when you have to change that power coupling, you don't actually affect people's VMs. And there are other products that do it. VMware sell products that migrate network connections.

The easy way to do it, honestly, if you're not removing the physical machine: forward on the sockets from the underlying Linux kernel. You just move the packets around. If you can't do that, then effectively the best way to do it is to put a machine in front that handles all the packets and have it route

Benjie: the packets differently. But the proxy always needs to be there from the get-go is the only catch. It does.

David: You can do it in a HA kind of way. So effectively, if you can get the configuration for where packets route spread across multiple machines, you can have multiple entry points. If the packets can hit, then they route correctly. And that's actually where you see that more commonly is in highly available global load balancers.

The Linux kernel has a bunch of great machinery for supporting that. You'd be amazed how much of this is actually already built into the Linux kernel.

Fair Sharing With cgroups

Benjie: Sorry, my brain is like 15 minutes behind all the things that you're saying.

But just to make sure I understand this, so as a exe.dev user, I give you 20 bucks and you give me two vCPUs and eight gigs of RAM that I can use, right? But I can just fire up 50 different VMs and it kind of dynamically uses the resources of that too. What happens when my Bitcoin mining rig goes over the — okay, that sounds like from 15 years ago, obviously not doing that — but what happens when I overuse those resources? Like if I've got three VMs and one of them is using two vCPUs and 8 gigs RAM consistently.

But I've got these other two that are also running. How does that work from a customer perspective?

David: Yeah, it gets slow. I mean, at a very high level, the way we work with the smaller users, like individuals with just two vCPUs, is we keep all of your VMs on one physical machine and then we use cgroup controls for access to resources.

So effectively, all of your VMs are in a cgroup parent that gets two vCPUs associated to it. And so if you've got two VMs and they both want two CPUs with the resources, they get one each, and that gets fairly divided. And if you've got 50 VMs that all want two CPUs because you're running a Bitcoin mining rig, each one gets 1/25th of a CPU, which isn't a lot, but they're pretty powerful CPUs.

I mean, actually, because we're so RAM heavy, we use relatively small CPUs, which means the frequency is very high. So you actually get pretty good single thread performance out of our machines. But yes, you only get the resources allocated.

Under the hood, there's a bit more RAM than is advertised, which again, we should probably make a bigger deal out of, especially given how expensive RAM is these days. And that's really just to help with the swap layering of it.

We then do some page cache management. It's just ballooning, the classic ballooning style techniques. So keep that under control.

SSH as the CLI

Marc: So another kind of less in the weeds on cgroup and scheduling, but we played a lot with different remote execution environments for agents, sandboxes, whatever, different things.

exe.dev is unique in that you didn't really come out with your own CLI. You use SSH as a CLI, right? I'm curious, A, how that's working out, and B, where that idea — that's clever. How's that going?

David: It's working better than I thought it would work. Honestly, I didn't think we'd get very far because the classic thing that goes wrong with trying to use SSH as a CLI is it has no access to your local file system.

And usually that comes along a time when you want to add like a deploy command which goes and grabs a bunch of files from the directory you're in or something like that. And that should be a big deal, and weirdly it just hasn't really come up yet, which I'm really surprised by, honestly.

Benjie: I mean, it's probably the agents just saying use SCP or like XRs or whatever. Yeah, that's basically it.

David: Like SCP gets you into the VMs and we're not trying to do anything really high level with managing the images.

So I think if we tried to do Docker build for you under the hood, we would run into trouble pretty quickly. But also, I think if we tried to do that, we would run into trouble anyway, because it's just really hard to run a good Docker build server for people to build images.

So our current system is you can use custom images, but you have to go and figure out how to build them yourself somewhere and put them somewhere. And then we'll just run them for you.

Benjie: And so any OCI image I can upload, but then it runs in my VM.

David: Yeah, we turn an OCI image into a virtual block device and then start it as a machine. That's right. We inject sort of an init into it, but our init gets out of the way and lets systemd take over after a little bit.

Benjie: So you said Ubuntu. So this is the host OS or guest OS, whatever the right term is. That's all Ubuntu?

David: Guest OS, yeah, that's right. So it's Ubuntu in our default image. But if you bring your own Alpine image, then it's not Ubuntu. That's up to you.

Benjie: Right, but that's my guest. But I'm saying the underlying host that's doing the cgroup magic, that's Ubuntu.

David: Oh, our host? Actually, we do use Ubuntu. It's modified like you would expect.

There's a bunch of weird stuff going on in there. But I use Ubuntu just because I know it and it's there. We could use something else, but it's predictable.

Rolling Your Own Kernel

Benjie: I mean, I'm sure in a year or two, you guys will be releasing your own kernel, it sounds like.

David: Yeah, we keep the kernel very simple. It's really interesting, actually — here's a fun sort of historical aside. I think when Firecracker came out, a big deal got made about micro VMs and about them being fast, and you could start VMs quickly.

Interestingly, it wasn't really Firecracker or the VMMs that made VMs fast. What made VMs fast is a bunch of machinery in the Linux kernel for the guest that lets you turn off all of its hardware machinery.

It's fundamentally correct that makes VMs start really quickly. And then you can start them really quickly on QEMU or any old VM system. Linux is really doing the heavy lifting here.

And so, yes, we build a Linux kernel. It's stock, though. It's very standard stuff.

The big thing we do with our Linux kernel is we turn off module support and compile a lot of machinery into it automatically. Occasionally we get support requests to add features, and we do. But we just build in as much as we can to make it work for people.

Benjie: So also with machine type, because you're at the very low level there, obviously different CPUs have different functionality, or not functionality, but extra instructions. Do you have a minimum thing, or is it super duper flexible, or how does that work?

David: We do have problems. This is a tough one because you're making trade-offs and what do you actually want the product to look like? So to make live migration work, you need the same CPU micro-architecture.

And the way to work around this is to limit the amount of the CPU you expose inside the VMM to the guest. So you can turn features off and then you make more micro-architectures look like each other and then you make it easier to move VMs between machines. On the downside, then you have less access to interesting elements of the machines.

So our main workaround for this right now is we're sticking to a very narrow range of AMD EPYCs for everything outside of AWS. And right now, we don't migrate stuff out of AWS onto those machines, which is unfortunate.

When we do, we have to cold reboot them. So we send people emails warning them we're doing that. And that sucks, basically. That's very unfortunate.

Benjie: And how much is a cold reboot? It's still, I mean, obviously, there's processes running and stuff like that, but how long does that take? Oh, it's fast.

David: It's just a couple seconds to do the cold reboot. The problem with the cold reboot is if you go and get an agent to write your little fun program, yes, you can ask it to write a systemd service script so that it starts on boot. And we actually have a line somewhere in one of our agents asking it to do that just to help people.

But the other thing you can do is you can just start tmux or screen and run your program in it and you're done. And the thing keeps running. And we have a bunch of users who do that.

And every time we reboot them, we break their programs. And so I'm really not keen on rebooting people's VMs. It's a thing we only do if we have to.

Benjie: Maybe you just have like a service that's like a tmux restarter or screen. I mean, honestly, just bake that into your... Well, I guess if you're providing your own image, that doesn't work, obviously.

David: That's exactly the sort of thing I would solve these days with an agent. Just ask an agent to go around a machine, figure out what's running and how to make it work on reboot is absolutely something you could do.

You mean Shelly?

Benjie: Let's be specific here.

David: Yeah, that's right. Or another agent. I mean, Shelly gets more use than we expected. And it kind of makes sense to me. I prefer being in a browser to being in my terminal.

But we actually thought when we shipped it, people would use Claude Code and Codex on the thing. We installed them by default, and we expected that to be the primary mechanism of people using agents. They'd SSH in and then run Claude. Or these days, Pi. We put Pi on there by default as well.

Benjie: What about the OpenClaw Hermes stuff? Are you seeing people using...

David: We got a lot of people show up to run OpenClaw in the early days, because everyone was trying to buy a Mac Mini, and then Apple ran out of Mac Minis.

And then this was a good example of the model. People wanted to spin up five or six OpenClaws, and that would cost you a lot somewhere else. It was cheap with us.

And I don't actually know if people are running OpenClaw. We'd have to look at people's VMs, but we have a whole lot of VMs named OpenClaw1, OpenClaw2, like that across all sorts of users. And so just in the database, you can see a lot of machines are doing that.

Benjie: That's helpful for my mental model: why do I need 50 VMs? But that makes sense. You want to have different isolation layer and then also, hey, wife or partner, here's an OpenClaw for you and they don't have to think about it.

Interesting. And all these machines are addressable to the internets, obviously.

David: Yeah, and we've put a bunch of work into that. They're actually all sitting behind a global load balancer. And so the TLS termination is really quite close to you. And it's all behind an anycast network that we built out.

Well, a provider provides the anycast network; we built out the global load balancer. And yeah, we put URLs on them by default. There's a CNAME thing so you can bring your own domain, and you can run real things on it, right?

Like if you go to blog.exe.dev, that's an exe VM running our blog. And the blog software that we're using for editing blogs and publishing them is something we wrote using Shelly, a few prompts, because why go and learn a piece of blog software when you can just bespoke build the features you want?

And again, I think I built the first version mostly from my phone. It's in our team, and so people hack on it occasionally. You can just

Benjie: build software now. Turtles all the way down, my friend.

David: Yeah, the team wouldn't let me run things like status.exe.dev on exe.dev for obvious reasons. So we don't get to use as many exe VMs in production as I would like. But that's just good engineering.

The Business of exe.dev

Benjie: So from a business perspective, again, you've done Tailscale. Now you're doing exe.dev. And it sounds like from our research, you raised a series A, a pretty large $35 million series A. So obviously you're getting some traction.

When did you know, oh, the people are using it? How did you get it out there? What's the little bit of the business or the go-to-market out of that work? How did this all come together?

David: Yeah, we go back to Tailscale. We used a PLG motion for the business, which is fancy VC words for: we gave it away to individual developers who then had to go to work and use some VPN they didn't like. And so again, convince work to buy Tailscale. That was the business model.

Like make it appeal to people and they take it to work. We want to do something like that with exe.dev. And that's the model we're pushing right now.

So right now, the sum of our marketing is blog posts and tweets and such. There's nothing else there. We should do more. That's silly. Marketing actually works and we should go and get some.

Benjie: I mean, you're on the foremost podcast for Kubernetes,

David: So that's very helpful. Thank you for solving our marketing problem. That's great.

But yes, a whole stack of developers showed up really quickly early on, more than we thought, actually. It was really busy right out of the gate.

And that has translated into a stack of businesses showing up and using us. And the businesses have complex enterprise needs. We have customers who insist on running particular AWS regions, and that's a thing we'll support for larger companies.

We can run stuff in unusual reach, and that's because they have, for example, huge S3 buckets for data they want to work with on their VMs, and they want good network—

Benjie: —access to them. And not paying—

David: —egress. Exactly. And so the region matters. Sometimes even the availability zone matters, which is really interesting.

And so we provide that for companies who care about it. Most people are just looking for VMs and prefer the higher frequency single-threader performance we can get outside of AWS and want the NVMe, which is actually very tricky inside AWS.

The nature of the machines they provide makes it hard for us to build that platform in there. Just like an interesting technical limit of their cloud.

Benjie: There's data usage limits or data transfer limits. If I have two VMs talking to each other, does that count as part of my data transfer limits?

David: No, we don't charge for that. The only thing that matters to us is egress. Everyone basically charges by egress. Charges us by egress and so that's what we pass on.

Again, a nice feature about being outside AWS is our costs are a tiny fraction of what they are. We haven't dropped our actual egress data prices as far as we could because we're pushing everything through a global load balancer and the price varies around the world and we would have to make our pricing more complex to drop it.

But within the US and the EU, we can get the price very, very low, which is nice. It's kind of astonishing how different the data prices are inside a hyperscaler's cloud.

Marc: It's so crazy. AWS lists prices like $90 a terabyte and then you go to Latitude and it's like 10. You're like, why is this just such a huge difference?

I mean, I spent a lot of time recently optimizing for egress pricing. So there's clearly opportunities to do really good things and help out your customers and help keep that price down to them, though.

David: Yeah, I don't know of any technical reason why the hyperscalers need to charge those prices for networking. I think it is purely lack of competition.

Benjie: Why does FIFA have a hydration break, guys?

David: Yes, that's right.

Benjie: We know the answer. But yeah, why are Uber rides so expensive once all the local taxis are gone? I think we all know the answer.

Yeah, so I'm very excited to support exe.dev and kind of disintermediate that a little bit. And a bunch of the other folks out there are doing the same thing. How many people are at exe.dev right now?

David: We're pretty small right now. The full-time team is eight people. We're growing. We're almost entirely in the Bay Area, and so we're hiring in the Bay Area.

But we'd like a couple more engineers on the team. And we're ready to hire in marketing and such and figure that out. We want to keep the engineering team pretty small right now.

Marc: It's amazing what you can do with a small engineering team today, especially with really good folks who really understand how to use agents.

David: Yeah, exactly.

And also, it's amazing what you can't do with a large engineering team now. Because a lot of the processes you use to make a large engineering team work make it hard to use agents well.

And that's a real problem. I have friends who are still in enterprises and I go and talk to them and they can barely get anything done with an agent. It's just amazing.

Benjie: Most of the people I talk to in enterprise are like, oh yeah, my Cursor account finally turned on. I'm like, really? What are you talking about? Like, oh, it took two years to get procurement. I'm like, all right, well.

I think that's actually a really interesting point you make, David, because I'm seeing it so much where there was always, oh, you're a startup, you're nimble. And I'm like, that's true. But now it's exacerbated, like magnitudes of how much quicker you can do when you're small. It's unbelievable.

Picking an Agent and a Model

So we are recording this July 20th, 2026. So this is likely to change tomorrow. But what is your coding agent of choice these days?

David: Yeah, so I really want to push on this with people. The agent itself, the command line tool, the web interface, it's not that important. It's really easy to write one of those.

And one day part of a CS course is going to be make an agent semester, just like building a compiler. It's really easy stuff.

It's all in the model, all the heavy lifting. The only thing your agent has to do is not misuse a model. Like it has to follow the norms the model expects and the agent will get great results.

Benjie: Yeah, sorry, to clarify, I meant what model?

David: Yes, totally understand. But it's a really good question because my agent of choice right now is Shelly because just for dogfooding purposes, I got into it and now I use it for everything.

And it turns out I actually prefer it. We built an iOS app so I can use it on my phone. It's great.

The model of choice is really complicated right now. So it was really simple a few weeks ago because it was just Opus. Like I use Opus 4.8 for everything and it was straightforward.

What really broke everything was Fable coming out because Fable is better than Opus significantly, but also it's really slow and it doesn't do some things. Some very reasonable things.

It'll refuse to work on any of our networking machinery. The moment I show any of it, it freaks out and assumes I'm somewhat nasty. And so it won't touch any of our global load balancer or anything.

And so I can't use it for some things. And even when I can use it, it's so slow that I find myself, before doing anything, saying to myself, well, is this really worth waiting around for a high-end model to do its work?

And so I use Fable maybe a third of the time, and the other big change is GPT 5.6 is actually really good and sits probably somewhere between Opus and Fable. Just intuitively, that seems to be the results I get out of it.

And so I find myself right now bouncing between 5.6's high-end Sol model and Fable. I'd say it's about two-thirds 5.6 and one-third Fable maybe is about where I land right now.

Marc: Does your agent actually make those decisions or are you making those decisions when you're giving it a task?

David: No, this is what's so annoying: I'm making those decisions. So I don't think auto-routing works yet.

I know a lot of people tried to build it, but I've not used one that I'm convinced actually works. I don't think I can build one that works either. I think routing is a harder problem than agents are currently capable of. Fable can't route correctly.

Benjie: Right, intrinsically, you're burning tokens to route. So who does that? It's an interesting part. That's going to get solved soon. That's really cool.

David: The interesting thing for me is it's not price, right? Because even using Fable as much as I want, I don't reach $100 a day. And that's a price I'm willing to pay. So my engineering time, because just the huge advantage I get out of it.

But it's not that. It's the wall time of Fable that gets me. It's too slow to run, not the price.

And 5.6 is really good. Also, I really enjoy having more than one supplier of models. It's great. I like that there's two, but I hate having to make a choice.

Marc: It's true. And then 5.6 has fast mode, which gives you a 1.5x speed boost, and you know when you care. Fable's great, and I do use it sometimes, but 5.6 is my general model lately, too.

Benjie: Well, I thought Opus 4.5 was amazing, and then I thought 4.6 and 4.7 were basically trash. So I went over to 5.3 Codex and then 5.5. And so I've been loving that. I literally never even touched ChatGPT or any OpenAI product until 5.3 Codex.

4.8 Opus was great. And obviously this weekend, just because K3 came out, which apparently is overloading Moonshot servers and no one can use it now. So it's just a crazy time.

And I'm curious to see as exe.dev grows, if you're going to be able to get any telemetry around what models are being used underneath the hood. Obviously, you probably don't do too much of that, but I don't know how to do that without it being inappropriate for your users.

David: Yeah, we don't record people's tokens, but actually part of the exe.dev plan is you get some tokens as part of your monthly plan. And we have an LLM gateway that people can use all of those models through. So we actually had some data on how people use models.

I don't have up-to-date data over the last few weeks. As of a month ago, basically everyone used Opus, or they used more interesting freer models like Qwen or something like that. But that was pre-5.6 coming out, and it was pre-Fable and all the rest. So I need to go and look again.

Benjie: I'm telling you guys, 5.3 was spectacular.

No one believes me about this.

David: I believe you. I just didn't try it. It's still good. I had the same experience you did with Opus 4.6 and 4.7 — they were duds. And 4.8 is good again, but also it's 50% more expensive than 4.5.

Benjie: I started playing with Sonnet 5 again

because I was like, well, I love Sonnet historically. I don't know what any of this means.

The Age of Personalized Software

All right, we'll pull back from this rabbit hole for a second here. But hey, this is the stuff you run on exe.dev. Look, we're way over on time. This is super interesting conversation.

Before we jump, David, is there anything else that we haven't gotten to talk about with exe.dev that you're excited about that's coming up or that you've recently released or any other trends that you're seeing related to exe.dev that you want to share with us?

David: There's a broader agent trend, and this is a point I really want to make that I've convinced myself of, which is we're in the age of personalized software.

It's so easy to take a program you have the source code to and that you're using personally and make a change to it just for you. And you can just do it with a prompt.

And we're developers and we live in our dev tools. And I'm convinced that the future of dev tools has to be open source because everyone needs to personalize their tools.

And in the past, we personalized our tools: Vim had a plugin architecture and your 10,000 plugins and a config language, and you could configure it all. Same with VS Code and all the rest. We don't need to do that anymore.

You don't need a plugin architecture. You don't need a config file. You just need the source code to whatever tool you're using. And then you just need to ask an agent to change it.

And that, I believe, is the future of dev tools. It's the future of dev tools, not just because an agent can change it, but because an agent can keep it up to date for you.

You can set up a systemd timer to every night, go and grab the latest copy of upstream and tell it to rebase on top of it. And now the models can do that for you too. And that I'm certain is how my software works right now. I'm using hacked up copies of Shelly that keep themselves up to date with upstream. And I think everyone should be doing that.

And I'm reasonably sure that's the future that we should be building to. And I think this has implications for what software can be closed and what can be open. Services that are run through APIs can be closed source, but otherwise, I see open source as the future here.

Marc: It's also a great use case for exe.dev if I have a suite of open source software I'm customizing every night. The systemd agents are spinning up VMs and doing this all, shipping and giving it back to me.

David: It's good to have a VM to put your agent in because then you can give it root. And when it's trying to debug a networking problem, it can run tcpdump and figure it out. Give your agents tools and agency and they can work.

Benjie: This is all about David's trauma from trying to use the VS Code plugin system.

Marc: Yeah, plugins. I was thinking, you avoid this whole challenge of plugins aren't expensive enough.

Benjie: Yeah, this is the PTSD from that. That's really what this is about. It's not about exe.dev, it's not about you, it's about that frustration.

Marc: It's honestly cool too, because you extend it. But you've seen this kind of concept, some between Vercel and Cloudflare.

I think there was a term thrown around for about a week called slop forks too, where they would actually create it, but you get access to the test suite too, so your agent not only can take that open source software and rebase it and play your extensions in, but it can run the upstream test suite and say, it does everything that software does, plus this thing that we actually added in.

David: Yeah, absolutely. And it can write tests for the change you made too and preserve those.

I think slop forks are a bit trickier because you're trying to produce code that a lot of people are going to use and now you actually have to meet engineering standards.

But for the programs you use yourself, it's like, does it work? Okay, great, I'm done. My standards have always been lower for me. Now I have someone helping me write code for myself. It's great.

Marc: You're dogfooding your own product. You're kind of like, you know what? That's okay. I can go through that weird — I got to start it twice because it crashes once. It's all good.

Benjie: But the real underlying thing there, the insight is the point that open source is ironically the future, whereas everyone else is kind of like, oh, I got to close it because it's so easy to copy. But it's not going to work.

David: It's not going to work. No one will use it because they can't personalize it. This is my working theory right now.

Shelly is an open source agent that runs in your VM. And that's because we thought everything inside your VM should be yours. And so we open sourced it. And we really want to push on that and be like, you should use this.

And it doesn't have to be Shelly. You can use Pi. It's a great minimal open source agent. You can personalize it, but you don't need an extension system. You just need to hack on it.

Benjie: Amazing.

Wrapping Up

David, I think we can probably talk for like 17 more hours if you'd like, but we're kind of coming up on time. Well, we were way past time, if I'm being honest.

Really amazing to hear about the Tailscale stuff. And exe.dev is super exciting. And genuinely speaking, I was like, eh, what is this?

But now I'm kind of pumped about it. I'm going to go play with it personally now that I understand what it actually means because I didn't get it. I was like, well, 50 VMs and a VM. Now I get it and it's kind of amazing.

I'm super excited about seeing where exe.dev goes, and it is exe.dev, not exe.dev, for the listeners out there that stayed this long.

David: That's what everyone tells me. So that's what we're going with.

Benjie: All right. Well, I'm glad we could tell you what it is. David, thank you so much for coming on. We really appreciate the time.

David: Thanks for having me.

Marc: This was great fun.