Ep. #38, Reproducible Infrastructure with Graham Christensen

John McBride: Welcome back to another episode of Open Source Ready. I'm back again with Brian. How are you doing?

Brian Douglas: Fantastic. Right after this I'm going to jump on a plane to go to Seattle. So looking forward to that, to hang out with the Confluent team.

John: Very cool, very cool. Yeah, I think our guest was just on the west coast actually and has also recently gotten off a plane. But we're here to talk to Graham Christensen, the CEO of Determinate Systems. Graham, how are you doing?

Graham Christensen: Hey, yeah, I'm doing really well. Thanks for having me on. So I just got back from Planet Nix about a week ago and getting recovered and getting caught up frankly with all the cool stuff happening in the Nix world.

John: Yeah. How was that conference?

Graham: It was good. You know, I've been doing Nix consulting for a long time and one of the things that really stuck out to me about Planet Nix this year is sort of the age and maturity and the position in their careers the people there are in.

I'm seeing way more people that are way like more senior, more experienced, like principal engineers looking to bring Nix and less of like the young spark trying to get their enterprise to adopt Nix. Right? Which is always hard, but there's way more people in authority ready to do it. It's really exciting.

John: That is very exciting. Yeah. For our Audience, and you're maybe the best person to explain this, how would you describe Nix to the general audience here? Or I guess Nix and NixOS and the package system?

Graham: Totally. So Nix is funny because it really depends on who you are and the way you want to describe it.

The fundamental thing about Nix is I think it solves an original sin of computing, which is the failure to track and really keep track of the relationship between bits of software and configuration.

John: Yeah.

Graham: And we see that crop up all over the place, like building a Nix OS system or building another operating system, using it for development environments, which is always a catastrophe. Like without Nix, I mean, it's like-- Without Nix you've got to readme with Nix you know, "use during" or just run Nix shell or something like that. Or rather Nix develop, and that gets you in.

The way I see Nix today is teams around the world are really seeing it for the full breadth of what it can bring. And really what that is, it's the ability to manage the complexity of the world and sort of face it head on instead of sweeping it under the rug.

John: Yeah, totally. And you know, it's become one of the biggest open source projects ever on GitHub. I think it's at almost a million commits, which is kind of wild, you know, where all those packages end up landing.

And you and Determinate Systems have existed inside of that ecosystem for a little while. And I think it's very interesting. What are some of the open source things that y' all have been doing recently?

Graham: Yeah, so actually we started a long time ago, maybe like five or six years ago, as a consulting firm and at the time there were a ton of consulting firms out there and there was what I think was valid critique that Nix was sort of consulting ware. And I didn't like that. The way I saw it, Nix was the future and doing that as a consultant wasn't going to get it done. So anyway, that's why we pivoted over to doing product stuff. But anyway, sorry, what was the question? Haha.

John: What are Determinate Systems doing in the open source ecosystem right now?

Graham: Yeah, so we maintain Determinate Nix, which is a downstream distribution of Nix. It does have a closed source daemon attached to it called Determinate Nixd and we rightfully catch a lot of flak for that. But the really interesting part and the important part happens in fully in the open source distribution of Nix.

The things we've Done there are tons. Like we could spend a whole many podcasts on it, but we have stable Flakes. We've shipped something called Lazy Trees, which makes it much more possible to use Nix and monorepos. We just shipped support for WASM and at evaluation time, which is kind of crazy. And Providence. There's loads of stuff in there that really comes from our need to actually ship working features and working software to support our customers that are adopting Nix in bigger and bigger ways.

John: Yeah. This is something I've always wanted to pick a brain about. Why do you think now is a time that a lot of enterprises, obviously small teams, the indie hackers, but also the governments and huge cloud projects and products, where are they now suddenly coming online to Nix? Is there like a movement happening or like what's behind that?

Graham: Yeah, so a few things. Probably the most unpopular reason for this, frankly is Anduril. What has happened with Anduril is, and we can talk more about that, but what has happened with Anduril is like a ton of people, you know, they're a big company, a ton of people work for Anduril and have left Anduril and then go work in other government spaces and bring Nix with them. Right?

So I think that's a pretty significant factor. But that's not just it. I think the other part is that the technology of Nix has matured to the point where it's acceptable. You know, people have been hearing it long enough on the Internet to say, okay, well, this isn't just a flash in the pan, this is a real deal. This is a 20-year project that got over that hump five years ago, but it's totally working now.

And the other thing is, I think in many ways the world has caught up to Nix. Like it's ready for Nix in a way that it has never been before.

I remember like 2014 or so, Python packages and Python projects didn't really even write down the version of their dependencies. And that's just not acceptable anymore. You don't do that anymore. You always write down exactly what you want. And not just that, you write down the hash of what you want.

John: Yeah.

Graham: You know, back 10 years ago, the fights we were fighting as a Nix community were like, "hey, you should write down the hashes of your dependencies. Not just so we can fetch them, but just because it's a really good idea."

Brian: Yeah.

Graham: And finally, you know, the world's getting there.

John: Yeah, totally. There's this great project I Like NixOS Infect.

Graham: Absolutely.

John: It's like the people from Anduril went and infected all these other companies. Haha. It's hard to put down, once you pick it up, it's just such a good tool, right?

Graham: That's right. And if I may, there's a bit more. The whole issue of supply chain security, it's a huge topic across the board and I think a big wake up call for us because an industry was around SolarWinds. Right? That was a catastrophe. And I'm not going to stand here and say Nix will 100% solve that. I'm not that naive. But Nix makes a big difference.

When you build everything in a sandbox and you have ephemeral builders because everything is in a sandbox, it goes a long way. And I think this need to really understand how our software is built and again, how our software fits together, which is like the crux of Nix, I think it really makes for a ripe moment for Nix.

John: Yeah. We recently, you know, shipped something that in the open source, the NixOS based system for AI agents called Stereos.

Graham: Yeah.

John: And one of the things my friends asked me was like, how are you going to keep up to date with security updates? And you know, NixOS only really ships stable releases every six months. So there's like a kernel bug or something. You're either like overlaying that yourself or finding some other channel or something to grab that off of.

What would you say to that? Because that does seem to be one of those things that people are like, oh, Nix Packages is too slow or I'm either on unstable or even that is too slow. You know, I need to NPM global install all my things. Haha.

Graham: Right, sure, yeah. I mean there's two different perspectives there. One is it's too slow and the other one is it's too fast. So one of our big goals as a company is to make it possible and make it a safe choice for big companies to pick Nix and to roll it out.

And that's a tall order for a lot of places. But the angle of software security and getting security patches out the door, it's a big one. Companies and especially auditors really want to know that like somebody's responsible for it and there's like contractual obligations to get it done and the side effect of that is consequences if you don't.

So this is something that we actually just launched a couple weeks ago, Determinate Secure Packages. And what that is is a subset of Nix packages that we provide CVE guarantees around. So we've built our own SBOM tooling. We scan our subset constantly for vulnerabilities and then we apply patches and get them shipped to our customers on a contractual SLA, based on the severity.

John: Wow. My hat's off to you because that is no small order.

Graham: Nix Packages is huge. It has well over 100,000 packages across a bunch of architectures. And there's a really solid 80, 20 rule here. 20% of the packages offer about 80% of the value.

John: Sure.

Graham: So our approach here is that we picked the subset that we needed for our stuff, and we said, all right, we're going to cover these. And then every time we bring on a new customer, we do a gap analysis and say, what additional packages do you need us to cover? And we'll add it to the secured subset. And that's still, you know, we have a whole bunch of customers on it. And it's still not 20,000 packages.

John: Yeah, it does always tickle me, what's in mix packages. Like Rainbow Fart is an emacs package that just happens to be in Nix Packages as well. Haha.

Graham: Yeah. Haha.

John: So I'm glad you're not providing CVEs to Rainbow Fart. Haha.

Graham: For sure. Haha. So an interesting conversation I had was with a security team at a financial services company, and they were really worried because they found in their compliance scan references to crypto miners on their development machines.

John: Oh, wow.

Graham: And this was a problem.

John: Yeah.

Graham: And, you know, it turns out it was just Nix Packages. Like, Nix Packages has crypto miners packaged. It's not completely unreasonable. You know, they basically, all the software in the world that matters is in Nix Packages, plus a lot more. And so we work pretty hard to cover all the software that our customers use and make that guarantee.

John: Yeah.

Graham: There's a couple angles to look at secure packages from though. One of those is from the security patching, but another one is around: Is Nix Packages trustworthy and is the public binary cache trustworthy? And I've been talking about this problem a lot, and I catch a lot of heat for it. But, I'm not trying to be a jerk. Right? It's a real problem.

The community does a fantastic job maintaining Nix Packages, and the NixOS infrastructure team and the security team do a fantastic job. Right? In no way am I trying to say they don't do a good job or a good enough job doing this work. They do a great job, but it's just not quite enough for a lot of institutions. And that's a gap we're trying to fill.

Like the binary cache is maintained by volunteers. And when you're talking about the sort of the linchpin of the security of your entire infrastructure as a financial services institution or somebody in national security, rooting your chain of trust on a volunteer led group of individuals is a pretty scary proposition.

John: Yeah.

Graham: So we do a couple things. One, as part of secure packages, we rebuild our subset and we push it up to our own binary cache and we don't depend on the upstream cache at all for that. and the other thing we do is we actually vet the changes to Nix Packages, to do things like double check the packages and the source hashes are correct. And just to make sure that there's nothing shiesty in there.

John: Yeah.

Graham: Which again, I don't really even suspect, I don't think there will be, but it's possible. So we try to offer a sort of a firewall there.

John: Yeah, I actually have a ton of empathy for that point because when I was at AWS working on BottleRocket, we were trying to bring in a bunch of stuff onto the system that was basically Go packages and you can build those off of, you know, the Go proxy that Google maintains. It's really not too dissimilar from a a binary cache proxy, but I guess it's Go code.

Even that was sort of unacceptable, you know, when you get into like one of those like war room scenario things where it's like, okay, like if this scenario then this breaks down and then, you know, all of our secure customers are essentially pwned and therefore that's unacceptable.

So even like at one point, you know, I don't know if they ever did this, I left before saw this to fruition. But at one point they were talking about like building a mirror to like the Go sum proxy at Google for like the whole Go language just to be able to have some like, governance and isolation of like that blast radius potentially.

Graham: Absolutely, absolutely.

John: It's crazy when you start to think of like, yeah, just the many different kinds of problems. Even, you know, they weren't suspecting Google was going to do anything or even like the Go community to be, you know, anything funky with all that Go code. But when you're building secure systems, it's a totally different calculus. Right?

Graham: Yeah, I sort of see it as, you know, that same philosophy of owning your own availability.

It's not that I don't trust the Nix community. I have great respect for the entire Nix community. We want to make sure that our customers that are putting a lot of trust into this software are able to trust us to do it. And for us to be able to stand by it, that's just what it takes.

John: Yeah, totally.

Brian: I wanted to jump in because I'm learning a lot about Determinate Nix. I literally just wiped the machine a couple weeks ago and started fresh with Determinate Nix. And I was like, okay, I see the value, I see where I'm going. And then we had talked a couple weeks ago before the podcast and I got to learn a bit more of what you guys did because I always thought you guys were still consulting and then you kind of explained today you guys have a product.

I actually had a read that I just want to pull it in the conversation from Sequoia. Sequoia is a VC firm. It's been around for years. They were talking about services being the new software and immediately it clicked where you guys, you did the consulting thing and then you found out the product from those experiences.

And if I could like tldr the article itself, it's basically like the next wave of software, the trillion dollar business is going to be products that are masquerading as services. So when I think about terminate Nix, it feels like, oh yeah, I'm going to set up my reproducer of builds and my NixOS. You got to do secure packages.

It's going to be like a security thing, but it's a product itself. But yeah, I was curious like you don't have a lot of context on the outside of what I just shared about the article. But like how do you see the learnings from doing consulting to getting to this like now new world where you have a product? Any sort of hindsight there?

Graham: Yeah, totally. And this sort of dovetails to a lot of questions we get around LLMs and how AI is changing the value of software and all of that. As a consultant of Nix, on Nix and in that space, along with my co-founder Eelco, you know, we would hear basically the same few problems over and over and over.

And if you've seen what we've built and the services we offer, it might sound like just a sales track, but it really did come from years of hard learned lessons. Like a binary cache that is trivial to integrate. Right? We flake up cache. It uses GitHub's OIDC. It's like you drop it in, you sign up and you never have to touch it again. It's connected with SSO so you don't have to share secrets. You just log in with your standard like Microsoft Entra or whatever and you're in.

The ability to share private flakes and be able to exchange private NIX expressions. These are both very frustrating problems that I've seen companies have to solve over and over and over. And usually the solution was, well, you just gotta copy that token around, right? Or you just gotta copy that SSH private key around so that you can, you can clone your private repos. Totally unsatisfying.

And then the flip side of that is the security of it. And what about the CVEs and how do I make my security team and our auditors comfortable with what this is? And my answer has always been the community is great and just trust in the community. There's like a thousand eyes reviewing every change and there's a lot of truth to that. But at the end of the day when you're talking about things like CMMC compliance or FedRAMP or like financial auditors, like it doesn't cut it, right? So introduction of secure packages.

And then the rest of that is like challenges of using NIX on a big team and a big piece of software. A lot of the Nix ecosystem, there's a lot of academia in there and a lot of students and no shade at all. Like I love it. It's what makes Nix one of the most vibrant and incredible software projects in the world.

But that does limit the exposure to those 2,000 person companies with a Git repository that is, you know, 30 gigabytes on disk, right? Like you can't copy that to the store on every keystroke, right?

Brian: Yeah. And so I was at Planet Nix. So I got to hang out with a bunch of companies and a lot of devs who are just working on hardware and doing cool things. And then a week later in San Francisco, I hung out with Eelco and the team at a VC happy hour and the room was interesting.

There's a lot of folks that I knew in the space that were using Nix a lot of this single platform engineer that introduced Nix and is still maintaining it. I'm curious, how do you get from one person excited about Nix to then we went the whole, maybe not the whole company, but maybe that's the goal, you want to get the whole company getting the value out of this.

How do you get from that gap today? And how are you seeing that?

Graham: Yeah, so I mean, my favorite way is always around development environments, right? You throw together a flake.Nix and you very easily get to delete hundreds of lines of code from your GitHub Actions workflow or from the readme. You just install Direnv or run Nix develop and you're all set. Right? That's a pretty easy sort of slam dunk way to get started.

At this point, you know, five, 10 years ago, it was different. It was like, this is super scary, untested technology. Now it's much more common to see some Nix users already somewhere and so growing it from there, using it in CI, using it to like build production artifacts--

At this point, I see it sort of coming naturally. And really a lot more of the concern comes from how do we actually roll it out to all our developers? How do we do access control? Like, we rolled out Nix to a bunch of computers and now some people are seeing some funky issue with Nix. Like, what do we do now? Or even like training.

Like, who do we go to? We've got five engineers who really love Nix and 10 that don't really know Nix. Like, how do we bring them forward, everybody together? I don't know.

Probably the most exciting thing to me overall is that the ecosystem around Nix is growing. Right? It's not just us. There's tons of companies out there working to make this better.

John: Including a certain stealth company. Haha.

Brian: Yeah.

John: Whose name shall go unknown.

Brian: To be announced soon. Haha.

Graham: How mysterious.

Brian: But yeah. So John had built a thing and he alluded to it, like, we have a open source project that's leveraging Nix. And I was actually surprised because I was always like, not a skeptic, just more of like, I don't think I have time for Nix, until very recently.

But to see the actual engagement through Twitter and other social channels and folks just like actually gravitating to this stuff got me intrigued enough to-- When I need to wipe my machine to start fresh, I say, "Cool. Let's just do Nix. Let's just go in like hard mode and figure this out." Which ironically it wasn't actually that hard. So that was also the good part about it.

Graham: I love it.

John: I think the first thing I told you is I was like, I was like, "stop, go install the Determinate Nix. Because normal Nix on MacBook could destroy everything." Who knows? I don't know. It's always scary. Haha.

Graham: Haha. We've definitely worked really hard to make MacOS nice. So I appreciate that.

John: Yep.

Graham: It's really easy for somebody that's new to Nix to be like, "oh, this is my favorite new thing. I'm going to do everything in Nix." Right? And I do want to offer a little bit of caution of, like, use Nix up until the point where it's annoying, and then stop. Get out of Nix. Do it the fast way. Do it the easy way. Right? Like, get the utility out of Nix. You don't have to go all the way. Haha.

John: Yeah, I think. I think that's a great unlock because you can do literally everything. Like, you can have it build and run and manage services and start-- You know you're in the next shell or the develop shell, and here's everything.

And it's like, just running and booting everything up and building everything. And I was at that point where I was like, "No, this is maybe too much. I should just use a make file." Haha.

Graham: Yeah.

John: Which makes a lot more sense at this point. Right?

Graham:

There's definitely the temptation to just do everything in Nix, and then eventually you wake up and you're like, damn, my CI workflow takes 10 hours.

And another great thing to do sometimes is to just cut and be like, "all right, that was one Nix thing, and here starts another Nix thing," and then merge them together later. It doesn't have to be a giant project.

John: Yeah, totally. Well, we already brought up one read, so I would love to transition us to Reads, but my question to Graham is, are you ready to read?

Graham: Every day.

John: Okay. So, Graham, you actually had a few reads, which I think would be awesome to get into. This one from Graydon. I was perusing before the call. What is this about?

Graham: So this was from Graydon Hoare, who invented Rust.

And this is something that I think a lot of people are struggling with: What is my identity as an engineer in this world where AI is able to do more and more?

And one of the things that really struck me on that trip to California a couple weeks ago is the number of people, the number of engineers I met who I know as good engineers, saying, "I just don't write code anymore because the LLM does it for me," is really shocking. Right?

I mean, I don't write code every day because I also run a business. But the fact is that LLMs really are quite good at writing code. And it's not perfect, but it's pretty good. And that feels stressful, right? I feel stressed and a little worried. What does it look like in a world where all the code is written by an LLM? What are we gaining and what are we losing?

And like, how we already feel like software sucks. How much worse is it going to get? But also like, how much better could it get? I don't know. I think it's a double edged sword. And I find it exciting. But what I saw in that article from Graydon was like grappling with the fact that it's here, it's actually real and it's time to reckon with it. That's what I saw.

John: Yeah. Brian knows, and so do our listeners on this podcast, that I've been grappling with this for the better part of probably a year and a half, probably since like GPT4. Like, I feel like I've seen this coming for a long while and Brian and I have been early adopters of this tooling, you know, for probably the beginning, since the OpenSauced days.

Even back when I was like, "okay, I can probably get it to, in ChatGPT, just generate a bunch of boilerplate and then I can copy paste that over and then change it as I need and it, you know, saves me a little bit of time writing boilerplate code. But yeah, now I just like, I also feel worried, like I don't know what the future is in this.

And I feel like had an okay thumb on it, you know, what Kubernetes and cloud native was going to look like and services and SaaS and like the future of development workflows. And now I just have no idea. Like the developer life cycle, I feel like is going to change even more dramatically.

One of the things he says at the end of the article is like, "maybe it all implodes. Maybe it's, you know, very different." One of the things I worry about in that is that there's like a huge rug pull coming where-- And it is so hard to know, it's so hard to know how much AI inference is subsidized. I guess maybe until like Anthropic or OpenAI goes public and then it's like, "oops, sorry, your $200 a month Claude Code Max plan is actually $5,000. Please pay us."

Graham: Yeah, totally. Yeah, absolutely.

John: And like, I wonder and worry like when and if that calculus changes and what that is going to do for like people's atrophying skills and, or their like atrophying mental health states as we're all just like so knee deep in-- And I don't know, I feel like I'm too knee deep in some of this stuff sometimes. But Brian, what are your thoughts?

Brian: Yeah, actually related-- So I commented on someone's Twitter rant. I'll drop it in the chat in a second. But they're making the correlation to like Uber. And this is something, John, we've talked about already.

John: Yeah.

Brian: To ad nauseam, like on our live stream, which shout out, we do a live stream every week on at 11:00am Pacific up on John's YouTube. But what I'm getting at is like I'm going to take a ride to the airport after this call. It's going to cost me $80. There was a moment before they IPO'ed that it cost me like 25 to get from Oakland to SFO and it was like, it was a no brainer, like $5 Bart ride, $25 Uber. Yeah, perfect.

Now I don't do that anymore. well, I'm doing it today because I got a short window and TSA is having a moment right now. But what I'm getting at is like when it comes to like subsidized tokens and I think there's a lot of pushback on the Anthropic side about like subsidizing and like are they really subsidizing? I would say that the prompt caching is doing very well.

So I think it'll be less of a rug pull but maybe still will be. But I think what we're getting at is like if you're now addicted to the sort of IV drip of code is getting, not even just tab completion, it's like generated now you're like okay, well this is how we interact with code. I can just know what's going on sometimes, but going to be confident that when it breaks I'll just ship an agent on it and it'll fix it.

So we're getting atrophy around this. I'd say it's very similar to when I was in college. I changed my own brakes, I changed my own oil, I knew how the car worked. I was always into the maintenance. And now I'm this calendar invite on my Google calendar, let's take it in, and nowadays I actually have someone come to the house to fix it.

So now it's even worse. There's no pain, there's no friction. And I think the thing that came up in the replies of this is like back in the day you'd like, you'd have the pain and have the friction of like, oh, I gotta get through this bug. And now you're like that pain and friction is like maybe a prompt session for an hour that you get unblocked on that you could remember, hopefully maybe write a blog post on, but probably not.

Yeah, we're in interesting times. I imagine by Q4 of this year we're all have sort of come to an LLM moment.

John: Yeah, it kind of reminds me of years and years ago when I wrote Kotlin and basically the only way to compile Kotlin was using IntelliJ. And a seat at the time for an IDE, you know, and it's great, it's a great tool. Kotlin's amazing. You get to take advantage of the JVM without any of the Java. But it just was bonkers to me that a seat for that was like $1,500 a month or something, you know, depending on the enterprise plan.

And in the back of my mind, you know, my very Vim minded self was like, there's got to be some plugin in the open source ecosystem to like unleash people from this developer-- I don't even know what you call it, like and it, it harkens back to me like Stallman way back in the day, you know, being so fed up with having to, you know, basically pay for licenses at MIT on these like massive, you know, "you get an hour or two here or there and like you're not going to be able to actually use this on your own time."

So he goes and makes GNU. Right? I don't know what that unlock is for the freedom that is software, you know. And I get very wax poetic here. But I don't know. Graham, does any of that resonate with you?

Graham: Yeah, absolutely. I mean, I think one thing that I keep coming back to is if they turn it and it's 135 times more expensive, which that tweet suggested, like they don't have a product, that just won't happen. Right?

I also feel like there's a lot of tension in everybody's heads right now. So one of my reads was a bout Amazon terminating Chime, their Zoom alternative. And if the value of code is going down and the value of like the amount of effort needed to write code and have good code is going down, why would they shut this down if they could just maintain it almost for free? And have a product almost for free?

I mean one of the things that is in tension with the fact that well, we're just going to have more code. Right? And you still need to be able to architect your software well and really I think what this is going to end up doing is having sort of like software packages that are smaller, more focused that you can have an LLM focus on and like work on a specific bug. Right? Instead of a giant repository where it has to understand all of the context and fix huge wide scaling problems.

I don't know. I sort of feel like once the chips fall we're going to be basically where we are, we'll just have a lot more code.

John: Yeah, the Chime thing is hilarious to me because being a former Amazonian or whatever we call ourselves, that team is cursed. Haha. People hated being on that team. And yet it was one of the most critical teams to be on like EC2 or S3 which you know, they were getting paged constantly. Because everybody used Chime internally, if you were going to jump on an escalation bridge or something for an incident, you were jumping on a chime. There was no other option.

Graham: Right.

John: So I wonder if they just, and this is pure speculation, like I wonder if they just had a hard time keeping engineering headcount on that thing. Even back when I was there, you know, years and years ago they were having a hard time keeping headcount on that thing, you know, and it was gross. Nobody liked it. Haha.

Graham: That's tough spot to be. Haha.

John: Yeah. My hat's off to anybody who worked on that thing because it worked, it was great. just I don't think, I don't think it was appreciated for its time. Haha.

Well, I had I had a read and this was one-- It's a little cheeky but I wanted Graham's opinion on this. This is agelesslinux.org which is kind of a tongue in cheek website about some new laws, regulations in California, specifically on operating systems having to verify the age of the user.

And I know there's nuance in there and I'm probably not capturing it very well but a lot of the response from the free and open source software community, the Linux community, is that this is kind of silly. And you know, how can an OS-- I even thought about this with Stereos, it's like it's a completely headless operating system. The user is going to be an agent.

Like, are we going to have to do a verification for that? Kind of wild. I don't know. Did you see this, Graham?

Graham: Yeah, yeah, totally. To be honest, I don't really worry about it. There's a lot of hand wringing. Like there was that post about a calculator operating system saying we can't support California users anymore.

John: I remember that.

Graham: the bill says general purpose operating system and a calculator is not general purpose. An agent operating system is not general purpose. We've been working on a very small embedded operating system that is not general purpose. It's not connecting to any sort of app stores and having underage users.

You know, I do think that Ubuntu is going to have to. And the desktop operating systems are going to have to. But I just can't get myself too worked up and worried about it. Whatever's going to happen, happens and then it just is. And then you just patch it out. Right, because it's open source.

John: Haha. I mean we could be back to the days of ricing your distros so much that you just patch it out if you don't want it. What would you name that package? Like Lib Age Verify or something? Haha

Brian: Haha. Ageless Lib.

John: Ageless Lib. Lib Ageless.

Brian: Lib Agelessly. Haha

John: Haha. That's good.

Brian: That would be fun. That's a great conference talk.

John: I agree. I can't wait to hear it at-- What conference would that be at? It'd probably be at a hacker conference, honestly, like DEF CON.

Graham: It reminds me of a library called Lib Eat My Data. Have you heard of that one? Haha.

John: No.

Graham: It's a LIBC patch basically that makes FSYNC do nothing. Haha.

John: Oh no. Yikes.

Graham: And it's great for CI workflows or whatever where you really don't need to guard against hardware failure very much. But yeah, you know, Lib Eat My Age or something like that.

John: I actually do feel like I heard about something like this that some of the EC2 team kept around for like super ephemeral, like-- Yeah, we just don't need to like put it on the file system for these like little, little workloads. We get like incremental little better times on some of these things in the hypervisor. But anyways, Brian, did you have a read today?

Brian: Yeah, I did have a read, but Graham, before we jump, did you have one from Eelco? I think you dropped it in at the end.

Graham: Yeah, I actually dropped three down there. Haha.

John: Let's do that one.

Graham: I came prepared. So the first one is one of my favorite books. I actually do read it at night sometimes and it is the Purely Functional Software Deployment Model from Eelco.

Brian: You said this is bedtime reading?

Graham: Of course, everything is bedtime reading if you're committed. So this is a bound copy of Eelco's thesis about Nix. And I had a chance to come back to it and read it again after we shipped support for WASM in the Nix evaluator. And in particular the part that I was coming back to read was about the intentional model.

And I don't need to get too much into the weeds there, but one of the things about Nix is that it's historically been very, very broad. Right? You compile at the package level, you build at the package level. If you change anything in there, everything changes inside. Right? And it recompiles everything.

Something that people and, well, Eelco wanted to do back in the early 2000s was be able to target a single compilation unit or a single component file for compilation, which practically speaking, you really can't do with regular Nix. It's too complicated.

But Eelco put together an example using our recently launched WASM work, which actually uses WASM to parse the header files of Nix itself and create a precise build graph of every file and the relationship to everything else. And so you can change a single character in a single file and only recompile that one and then re link everything back together at the end.

It's sort of like Basel and how Basel is sort of file based but applied automatically to the entire Nix project. It's really neat. I don't know exactly where that's going to go. Right now it's a tech demo, but coming back to, what I would call a classic and revisiting what he was thinking about 20 years ago is always pretty rewarding.

John: Wow, that's amazing. I'm going to buy that. Is it on Amazon?

Graham: It's not, but I could get you a copy probably.

John: Okay, I'd love that.

Brian: I was just looking at it. I think Google Books is the only place I could find a link to it, but I'm not sure if I could actually read it.

Graham: I can get you a link for the show notes.

Brian: Yeah, that'd be great.

John: Nice.

Brian: Cool. I've got two reads. First read is actually Karpathy's AutoResearch. So this is a thing that actually was building something very similar and not realizing it. So Karpathy is like very known for being in Tesla and also OpenAI twice. Kind of just throws a bunch of stuff out there. There's a lot of talks and then reviews papers and gives his sort of like thumbprint on it.

So the AutoResearch is basically the agentic loop and really meant for like doing some small wins and then basically that-- Basically loop until it's done. Prior to me actually discovering this, I think he actually tweeted this out after I spent a weekend building, which is my next read, which is I built an agent to play Pokemon.

So essentially Pyboy is a Python Game Boy emulator. And I wanted to like manufacture a reason to use Tapes and Stereos, which are the two things that we've been sort of working on with our new thing which is to be announced. But essentially I just wanted to go from zero to pick your Pokemon and see if we can like get to agent to figure that out.

And inspired heavily by Claude Plays Pokemon which is on Twitch. The approach is very different on the Claude Plays Pokemon version. But I wanted to build a self healing software loop. And this experiment it ended up working pretty quickly. So Sunday night I got to basically get out of the room when you first start the game. And then at that point I was like wandering, or the character was wandering around Pallet Town.

And then by the next evening I got another crack at it and was able to get to the Pokemon just by building this sort of like reinforcement learning loop. And they're getting like super in the weeds. I essentially built like an observational memory through like this Tapes SQLite data dump that the agent would just like create a log which I called Pokedex and it would just like summarize.

I take like a thousand turns and it would summarize those thousand turns and then you could then start another epic and then that would essentially just reinforce based on the previous log from the Pokedex. And then I got like super in the Weeds where I attach Kafka to it and Flink so I can identify anomalies.

So if anything like broke, which is basically like there is a manufacturer places in the game where you like you can't cross a bridge because it's broken all this other stuff. And the agent would get stuck prior to anomaly detection, it would get super stuck on like, oh, can't get past this. Let's go try something else. And what I found pretty early is like, you've got to talk to people in the game to get context clues.

So like, I've got these special loops. So when you get blocked go find someone to talk to. Or there's another situation where I would like speedrun. So I was like, trying to avoid battles by running away. But then I eventually it got to the point fourth time you got to battle, you got to dig in.

So then I had to create another whole knowledge of like, learning Pokemon weaknesses which I didn't want to go to the Internet for. I just wanted to learn in game and speedrun but it's always constantly adding to the Pokedex, as a true Pokemon master.

So anyway, it was this weird rabbit hole throughout the week. I just kept going back to the, well, learning this thing. And then Karpathy's AutoResearcher repo and tweak came out and I was like, oh, let me like, add some of these concepts, which I already had a bunch of them. So the difference is I just had this reinforcement learning part on top of that.

Graham: Wow.

John: That's very cool.

Graham: That's fantastic.

Brian: Yeah, it's an intense week, but it was a lot of fun. Super rewarding. I got to Viridian City, which is the first map that has a gym. But you don't actually battle a gym. Anyway, that's too far in the weeds. Pokemon. Check it out.

I've now written three blog posts on it, which I have one coming out about the Kafka, Flink thing. Just waiting for that to get approved.

John: Nice. Very cool. Exciting stuff.

Graham: So that reminds me of something we're doing with our operating system, which is-- It's for embedded systems. It's Linux. It's a small Linux operating system built with Nix. And one of the challenges is having it support the hardware that our customers want to run it on.

And something we have in the works and that we've been experimenting with is having-- We've basically built out what we call a HITL, which hardware in the loop lab which can turn on and off the machines and observe all their inputs and outputs and serial consoles and whatnot over an HTTP API.

And I've been experimenting with setting up a skills file and saying, "here is the hardware target. Here's what we know about the board. Go figure out hardware support and write the Nix flake to support it."

So things like kernel configuration, baseline software, serial console interfaces, those sorts of things. Hoping and seeing if I can coax one of these LLMs to basically automatically do board breakout for me.

John: Wow, that's amazing. I mean I feel like that-- You know again going back to Karpathy, he had this talk about self healing infrastructure and I can imagine that being really powerful on the edge where if you needed to make tweaks or certain adjustments and then be able to bring that up from some like manager thing, bring up that hardware-- The future is here and it's going to be crazy. Haha.

Graham: Haha. For sure. Absolutely.

John: Yeah.

Graham: It's pretty neat though.

John: Yeah. Very cool. Well Graham, where can people find you? I'm guessing on X these days?

Graham: For sure. Yeah. Grhmc on X. Still working on getting the GrahamC version there. Haha. And then really a lot of my time is spent on the Determinate Systems Discord which is determinate.systems/discord.

John: Very cool. Well people, go check that out and everybody stay ready.