Ep. #42, When Forking Becomes a Feature with Will Cory

Brian Douglas: Welcome to another installment of Open Source Ready. John, how are you doing?

John McBride: Hey, Brian. I'm doing good. We're back from New York. How was the flight back?

Brian: Flight back was uneventful. Every flight that's uneventful, I appreciate at this point.

John: That's good. You didn't get stuck in Cleveland like I did a couple weeks ago?

Brian: No, no Cleveland. I think it's stuck in Houston the last time I was trying to make it to New York because of whatever the nor'easter stuff that's happening.

John: Yeah, it's nuts.

Brian: Yeah, but I'm back out here in the foggiest San Francisco. Nice and cool. We'll get some overcast a little later today. I'm here for it, but we're not here to talk about the weather, which we kind of divulge into doing all the time. But here to talk to Will, Will Cory, about Smithers. So, Will, how you doing?

Will Cory: I'm doing pretty good.

Brian: Excellent. You're in San Jose. So yeah, you've got a whole other spectrum of weather down there.

Will: Definitely. Yeah, I spent the summer in New York, actually. Made me really, really appreciate this Bay Area weather where it never gets too hot, never gets too humid.

John: Yeah. It was a hot, swampy summer out here on the East Coast. So I'm glad you're back.

Will: Definitely.

Brian: Cool. Do you want to introduce yourself to the audience? Who are you? What do you do?

Will: Yeah, definitely. So I'm Will. I'm the creator of Smithers. I've been working on open source projects, I have a lot more projects outside of Smithers too, for many years now. Used to work at Google, worked a little bit in the blockchain industry.

What Smithers is is an orchestration framework where basically we're trying to figure out how to do long running tasks or repeatable tasks that happen over the course of a long time, or you can think of any means can think of, we're thinking about that stuff. So like Ralph stuff, what's another meme? Loops, graphs, whatever the meme of the day is, we probably have our own internal way of doing it that was probably aligned with that. So yeah, I'd love to talk about software factories or anything orchestration.

John: Yeah, I gotta ask, why Smithers? Why not Mr. Burns or Barney or Moe or any other Simpsons character?

Will: I'm historically very bad at naming things, I should say. So I have quite a few open source projects, and all of them I can point to another person who named it. So it's usually like I name it a very literal name. So before Smithers, I don't remember what it was, but I probably just called it like agent orchestration framework or something like that.

John: That's how I name things.

Will: Yeah. What I like about Smithers, though, is you don't really hear that name much. If you Google it, there's really not that many other things named Smithers.

And then I like leaving Burns to third-party tools. So someone actually built a UI called Mr. Burns on top of Smithers. And I thought that was hilarious, to be honest.

John: That is very funny. So Smithers is a sort of orchestrator of orchestrators or orchestrator of harnesses. This is something I thought about recently with this idea of meta harnesses coming around. Would you call it kind of a meta harness API, or give us a good example of something, maybe what the Mr. Burns guy did?

Will: Yeah, it's definitely a meta harness. So it kind of lives one layer up from the harness itself. So a really important part of Smithers is just pluggability, just the ability to take any existing harness and use it in Smithers. And there's a lot of reasons why you might do this, actually.

Harnesses and models are more coupled than people realize, actually. And this is going to become more and more true as fine-tuning becomes cheaper and more common.

So certain harnesses are going to perform better with certain models on certain tasks. So you might want to mix and match those.

The most common example you can bring up is just the Claude Code versus Codex dilemma. If you build a harness, Claude Code and Codex both have different ways of doing a write file. And depending on how you implement these tools, you can implement it more like Claude or more like Codex in terms of reading and writing files. And if you make it more like Codex, it's going to perform better with OpenAI models. If you make it more like Claude Code, it's going to perform better with Claude Code.

That's just a simple example that exists today, but you can imagine this becoming more and more true as you get more specialized. There might be a very specific harness for doing a very specific task. You can imagine Harvey, for example. Harvey just came to mind.

I don't know Harvey or anything about how the product works, but I could imagine they might be interested in fine-tuning Kimi with a very specific harness for looking up lawyer related things, for example. And maybe they're going to use Claude Code for other things, but then for that specific task, they might want to use that harness. So what you really want is just this way of configuring all this in a way where you don't have to do all the engineering work that you can imagine.

If you're an engineer, you can imagine all the work that you have to do to kind of do this yourself. You have to create a bunch of adapters. You have to create thread distractions. Other features might be coupled with this.

I think you'll relate to this actually. So you might have to think about observability, right? So this is something y'all are working on.

You might have to think about, okay, if I bring in all these harnesses, how do I observe them? How do I know how much I'm spending when they all have their own bespoke ways of figuring this out? That's something y'all built, for example. You can imagine this becoming more and more of a problem, actually, as harnesses and models start really diversifying into unique use cases.

Brian: I was going to ask, so if you have this harness to harness, do you think of this trend where you can start divorcing the harness from the model and picking your own flavor?

Will: Yeah, I think that really just comes down to creating clean abstractions. And I thought about this a lot.

And one of the reasons why I'm making a new version of Smithers, it's going to be releasing soon, is I came down to, what is that abstraction? What is that thing that we can all speak that allows us to have this harness of harness in a maintainable, pluggable way? And I really came down to workflows itself, the idea of a workflow.

So Smithers in the past, like we said, it's a way of composing harnesses. But I think we can go a layer deeper and say, what is a harness actually?

And when you think about it, a harness itself is a workflow, actually. It's a very special type of workflow.

And in fact, this trend is becoming more and more true by the day. So if we look at the state of the art of harnesses right now, a really common pattern now is instead of a harness being architected to turn-by-turn call tools or send messages, you can actually turn-by-turn have the harness write a workflow. And this actually turns out to be one of the most efficient ways to build a harness.

I believe Prime Intellect is something that's worth, if you have show notes or anything, worth throwing in there. They destroyed all the benchmarks, and their architecture is basically this. I could explain in depth how it works, but I think it's actually more interesting to think about what is the bird's eye view of what it's doing.

You can think of it as, hey, what if we had the model, kind of as it ran, write its own harness in real time? What if we allowed it to just write arbitrary code? And what if we turn all those problems into coding problems?

So what are problems like memory, for example, how do we deal with long-term memory? Well, we can do it on a case-by-case basis if we allow the model to actually write code to deal with memory.

How do we deal with compaction? Well, we don't need a one-size-fits-all solution. If the model is smart enough, we can actually just give the model the tools to come up with its own compaction strategy.

And I think this is what all harnesses are going to look like in the future. And when you do this, that boundary actually between the harness and a workflow starts to evaporate.

John: What would you say to the Arendelle people who do Pi and that whole ecosystem of harnesses where they would say the read, the write, and the bash tool is all you need? I also think that the recursive language model stuff is very fascinating from the ecosystem of new harnesses coming up.

How do you even start to evaluate something like that too? Is it building a whole corpus of evaluations and things that you can really grade models? Or are even those evaluations evolving now that in theory token context windows are kind of solved?

If you can just give the model a little programming environment, a little REPL to store its stuff and retrieve its stuff or call itself even. A hodgepodge of a question, but here's what you think.

Will: Yeah, I basically think that they're directionally correct, right? I think especially for the timing, I think what they realize is minimalism is good. And the more you can create a single abstraction that encapsulates all abstractions, everything just works better, especially from a pre-training point of view, right? So if you give your model bash tools, your model is trained on bash tools.

And you see this a lot, actually. So you see this in terms of, why are people so into skills? Why do people, when they build agentic systems in the browser, do they sometimes jump through hoops to make the tool calls or the way the model interacts with this environment look like a bash call, even if it's not bash? It's because if you lean into these standards that the model's already pre-trained on, you can piggyback off that pre-training, right?

This is a major reason why I think some standards don't make sense is there's a lot of standards out there. And every time they come out, you have to ask yourself, why did this not exist for humans?

Why did we invent the standard for agents that didn't exist for humans? And if there's not a good answer, chances are the answer is it probably shouldn't exist.

So I think Pi kind of recognized this pretty early and understood all you need. And I would say a bash tool is very similar to a REPL, they were almost there. And then also that they're correct for the time in the sense of you do want a separate read and write tool.

You could ask the question, hey, Pi, why not just a bash tool, right? You can read and write with that. And the real answer is just these models are extremely trained on having a read tool and a write tool.

This is something you notice, even if you take these state-of-the-art REPL-like harnesses, they perform worse on things that the harnesses are heavily trained on, right? If you give it a very simple coding cast as right in the wheelhouse of the types of training that people at OpenAI and Anthropic are doing, it's going to perform worse, even though it's, like, theoretically could be better.

I think that will change though as soon as the model companies themselves switch to this type of harness.

I guess one last thing about that is I think the reason why you might want a different environment than just pure Bash just has to do with the ease of sandboxing the thing, right? And also how easy it is for humans to read and understand what's happening.

I think Bash is just a little more difficult of a language for humans to really read and understand what is my agent doing, compared to if you give it a Python environment or if you give it a JavaScript environment. It's a lot more suited for a human to be able to notice and read what the agent's doing. And a lot easier to sandbox too.

John: Yeah, every time I see my agent doing a little bit of awk, I die inside. I saw I tried to do some Perl inline at one point in some bash. I was just like, oh my gosh.

What you said about standards and some of those things emerging, I think it's really fascinating. I've had the same thought. I think maybe this standard is moving in a good direction with its 2.0, but MCP.

I looked very hard at MCP for a good time. I was just like, why? What about this don't we get?

OpenAPI is pretty close. And I would say is sort of ubiquitous at this point for sharing and discovering at least remote capabilities on APIs.

I'm curious what you would think about this. A hot take that I have forming is that we're kind of looping back to the dawn of Web 2 where everything is kind of becoming just an API and we're sort of opening things, at least for agents, back up where it's pretty easy for an agent with curl even just to go get a lot of stuff done just on APIs. Do you think we'll see more standards collapsing into these behemoth standards like OpenAPI? Or will we still try to chip away at maybe the right agent-focused thing?

Will: Yeah, so I think some new things do deserve to exist. I think for the most part, the default is exactly what you said, where I think we'll just collapse on the same standards we've been using all the time. OpenAPI, GraphQL, Nix, all these things are definitely no reason to invent a new thing for the most part.

I think this is really interesting. It's something I thought a lot about, just someone who's building tools, right? It's what does something that actually does deserve to be built new look like, right? And I think what that looks like is when kind of the fundamental math you do of how should this abstraction look and what should it optimize for, whenever that changes based on agents, fundamentally what you want.

So I think a really good example of this actually is just the way we do pull requests, for example, the way we get code merged, right? I think a lot of people will report if you start having these high velocity mini agents working at once when it comes to this, I think you'll notice that agents just get stuck in rebase hell. You end up having to build these communications things.

I've seen people stick all their agents in a Telegram chat so that they could talk to each other about what they're working on and stuff. And this is just a fundamental problem that happens when the velocity of code being merged goes up.

So that's the type of thing I believe that a new standard will exist for, is if you can justify the new standard, not in terms of just, hey, this is something we're going to do.

So for MCP, it's hard to justify, right? You're like, hey, they need a way to communicate, right? And it's like, okay, we have that for humans. We had that for software in the past.

We have OpenAPI, what's different? What math changed with agents? There's really no story there for MCP to exist, right?

But I think if you're talking about how should we merge code, or even if you're talking about maybe what languages we pick is another one, is you might choose to pick a different language because the math changed. It's like the cost of a rewrite run down. So we can be a little more aggressive and picking what is the optimal language rather than the safest language, stuff like that.

If you can really justify why the pros and cons change because of agents, that's where I think new software, where a new standard really should be emerging from.

Brian: Yeah, you think we're divulging into everything is kind of getting written the same way? Or do you think it's the opposite, where there's a company I know that is pretty big into Haskell and they just recently made the decision to rewrite out of Haskell. And they invested a bunch of time to college in the language and built an entire product that makes money out of it.

The company wasn't focused on, hey, we're a Haskell shop. But they were known that they were doing Haskell and they just recently rewrote to, I don't know what they wrote to, probably Python or something.

But I'd be curious, some of those, if there's no base layer of garbage collection or a base layer of tools that make sense for an easier experience for the agent that Meander drew, or even, someone's talking about TypeScript. Now you have a, what's the Effect.ts where it's like a next level abstraction on top.

But the problem with JavaScript, well, I don't know, the problem with JavaScript it didn't have types, but that might have been the hypothesis going into it, but now what you get out of TypeScript is just now bloated code because now the agent's meandering between how do you write types for every single thing because we've got this contract, but languages that have compilers, you might have a better experience because the compiler has to complete. So Rust, for example, compiler has to complete before you even move on to the next thing.

Is handing the compiler to the agent that then helps it enforce best practices and standards? Or am I overthinking this?

Will: No, I love the question. And this is completely a non-answer, but I think why that's so interesting is I've seen people kind of say what you said and then come to completely opposite conclusions. And they make sense in both senses.

So I've heard someone say what you said and be like, therefore, in the future, we'll be writing all our code in Rust because it has the most back pressure, blah, blah, blah, blah, blah. And I've seen someone say what you said and then say, therefore, in the future, all software will be Python because you can just automate back pressure by just adding more tests. You don't have this bottleneck of meeting the compiler to do it. You can kind of do it yourself or you can build it into Python or you can fork Python.

So I find that hilarious that I've actually heard this argument and it's almost like the argument does not lead to a conclusion. You can actually say what you said and come to whatever conclusion you want.

Brian: I mean, we're living in an age where we have so much, I don't know if it's energy from agents, but we're psychosis, I guess, that you just mentioned just in a sort of side quest of talking, you'd mentioned forking Python. But there's someone who's listening and be like, oh yeah, we did that like twice last week. These massive migrations and rewrites and forking projects and rewriting SQLite into Rust yet again. Nobody asked for it, but actually the knowledge that I was thinking through when you're talking about this, with the agents being built on Bash and stuff like that.

There's an artist in LA called Henry Taylor. And Henry Taylor, later in life, he learned how to basically paint. And he took a lot of inspiration from Picasso. And a lot of what he has, basically it's a one-to-one Picasso. He's learning how to do art, and he's just one-to-one recreation.

But there's a certain point as he gets older, it's more of his modern stuff, it's diverged into its own thing. And it's telling its own stories, but all the influence. And I wonder, we don't need another GraphQL, we don't need another OpenAPI spec.

But I wonder if we're going to accelerate into people just building copies of things or forks of things, but it unlocks just other random exploration that we would have never thought, that you'd have impressionism, but now you have Neo-impressionism, but you had agents, but now you have meta-agents or now you have software factories. So the abstraction changes based on what was novel becomes table stakes.

Will: Yeah, this is funny you're mentioning this, because actually I have not publicly talked about this or released it or anything, but I started building the next version of Smithers.

So I guess for some context, the last version of Smithers, it was built on top of React, with this hacking into the pre-training idea I talked about. So I thought about agent loops. I thought about, how do we make these agent loops easy for agents to write and easy to reason about?

And I noticed that a lot of the patterns that I kept installing were actually just matching how React works. You want loops, like a render loop, and then you want a tree of tasks to do as your orchestration that kind of looks like a DOM. And I eventually landed on the crazy idea of, hey, let's just use React for this.

I learned a lot from that, but basically, one of the biggest things I learned is just every abstraction I make in the form of a library is just not as optimal as it could be, if I do that "I have infinite engineering time" thought experiment that you should always be doing. Whenever you build a library, think, hey, if I had 10 years and a billion dollars and infinite devs to build this, what would I build then? And I noticed just everything was not as optimal as it should be.

And I wasn't happy with Effect.ts either. I shouldn't say that. I'm very happy with Effect.ts. But I always yearn for more with Effect.ts, because you do take certain trade-offs on it that seem unnecessary with verbosity and stuff. And with how difficult it is to... It almost looks like its own programming language, even though it is TypeScript.

And I basically came to the conclusion the next version of Smithers should actually be a fork of TypeScript. Which is a pretty crazy thing to say. But it's really built on top of the idea that TypeScript did, where it's like, okay, we're going to be really intentional about what changes we make here. We're going to try as much as possible to make it so that this is TypeScript. If you look at it, you might not even notice that it's a different language.

But how can we add these few handful of things we need to make durable workflows work in a way where you don't have to use a library, you can just use a language. So it's like, how can we add durability to TypeScript? How can we make errors type safe in TypeScript?

Basically it's just answering the few pain points that, if we change TypeScript in these ways, what would make it so that Smithers didn't have to exist, Smithers or library. How could I make it so that Smithers or library is unnecessary, because the language is so good that you can just use this language and it plugs into all your existing ecosystem, and we don't need to build all the libraries and abstractions on top of it.

That's what I'm looking to release in the next few weeks, and I think that sounds crazy, but I swear this is going to be very common.

It's like what the open source world is about to look like is very small forks of things that you can maintain these things.

You can think of, before, the idea of forking and maintaining was kind of a crazy idea. It's not a crazy idea anymore.

You can totally fork a library, have it automatically rebasing every time changes happen. And from library maintainer point of view, this means you can actually make your library simpler, right? One of the hardest push and pulls you have as a library author is, how do I make it good for everybody without being kind of sucky for everybody? How do I balance building something for a very specific use case versus the broad use case, and it's always trade offs there. But if you make your configuration,

hey, fork me. If you want this feature, fork me and add it. You get rid of that trade-off, and you also have this amazing way of actually pulling new features into your library now, where you're not just getting feature requests, you actually have a working proof of concept that someone cared enough about to fork and deploy into production. And now maybe you can pull that into the core library.

So I think this will be very common, despite me sounding like a crazy person right now.

John: It makes sense. I mean, there was a whole corpus of people forking VS Code and making that a product and adding additional bits and pieces on it, and then became Cursor, and now it's an xAI. And then $60 billion.

I mean, there's a term I've been hearing and thinking about. It's just the idea of jellyware, where it's not hardware, it's not even software, it's just this kind of jellyware that you can kind of morph and compose. And David Crawshaw at exe.dev, formerly Tailscale, talked about this at a talk that we were at in New York a couple weeks ago, about how probably for your product, you should just make it open source, put the source code out there. Software by its very nature is very open and free and can just be moving the code around as you need to. All those Stallman-esque kind of things.

But then enable it by being forkable and letting people get in there and change things. Instead of having a thousand little configs flags for your CLI, just make it some hard-coded bits that you could just fork and change and then build, and your agent can do that for you.

Will: I think this is a big thing people are missing actually when they're thinking about the decision of should I open source. Everyone's so focused on the idea of software modes, of, hey, should I open source because the software moat's smaller? Should I not open source because now people can just fork me so easily with agents, whatever. And I think that's just focusing on the wrong thing. The bigger deal is what you just said.

The bigger deal is this ability to have people fork you and have that be a contribution, right? This ability to have people use your thing and have that harden your code. This ability to even just get your code into the training data. People don't focus enough on the upside of open sourcing. They focus too much on the downside or the quote-unquote proprietary moat you build.

And I think we can talk about that too. It's like that proprietary moat is going away. Obviously, that's a thing, but that's not actually why you should open source.

You should open source specifically because you'll just get more human thought put into your project. You'll get more tokens spent on your project. If you just measure it based on that, all those numbers will go up if you open source.

Brian: Yeah, that's a good take. And Smithers is open source, which is why I'm on this podcast. We actually had a conversation offline about how you're managing contribution. So what's stopping someone from just forking Smithers and creating, I know we talked about Mr. Burns, but pick another Simpsons character and you got Smithers 2 go out there in the wild. How are you interacting with open source contributors, and how do you filter all that noise?

Will: Yeah, definitely encourage it. Actually, the funniest message I've ever gotten from someone was they're like, hey, I can't tell you where I work or who I am because I'm kind of chaotic on this Twitter account and I don't want to dox myself.

But I just want to let you know I work at a company. I thought Smithers was awesome. But I hate TypeScript. So the first thing I did with Smithers was I made Smithers rewrite Smithers in Python. And I love it. It's worked great. Thank you.

I thought that was the funniest. I laugh so hard at that. But yeah, in general, I encourage people to fork it. I encourage people to fork it because I can fork it back. I don't want you to fork it so much and close source it. You can do that. It's MIT license.

But if you fork it in your open source and you add features, or if you build something on top of Smithers, I put a lot of weight into that as a feature request, if that makes sense.

It's like the moment you do that and I can see it, I can imagine it better. I can think of a really dank way of including it in Smithers itself. And when it ultimately gets added to Smithers, it's going to be better than if I added it, because a lot of these features are not things that I'm even a user for myself.

So if I try to build it, I'm going to be a little unaligned, because a lot of these features are not features I'm using. But if you're actually using this feature day to day, the way you build that is going to give me a lot of information about, oh, I can see the problem that's being solved here.

So I think in general, that's what I really encourage more than anything. I'm very likely to actually deny a feature request and ask someone to just kind of do it themselves. So I'm very likely to not really accept a PR from someone who I don't know, and instead ask them to just share prompts and share what's the intention. Get down to the meat of it. We can produce the code later, but why did you do this PR? What's the problem you have that's being solved here?

Brian: Yeah, I was going to ask, are you looking at the forks that people are, I guess some people are self-disclosing, but do you get a lot of value of just going through random PRs on forks and seeing what never makes it upstream? Do you get a lot of ideation from there?

Will: Yeah, definitely. And even from things that aren't Smithers, right? So anything that kind of smells a little bit like Smithers, I get a weekly report on everything they're doing. And so even if they weren't necessarily a fork of Smithers, so some of those things are things that I kind of forked myself, right? Or not even forks, the ideas are in there, right?

So I'm looking a lot at BAML, for example, is a really interesting project. LangGraph is super interesting. So even just looking at what people are asking for in those, a lot of those things are going to apply to Smithers. So I think getting that broader view is generally useful. And agents can make that not actually a chore, right? So it's like cast a wider net is the way I think about it.

And then, yeah, just looking at, even when developing on Smithers, a common thing I do is I have a script that looks for new open source projects using Smithers all the time. So I have those kind of already saved. So I'll often, let's say I add a new feature, I'll ask it to go through every known open source project and do the code change or the breaking change, right? And then I might even put up that PR to be like, hey, we're making a big breaking change, but if you just merge this PR, not a big deal.

But that ends up being a really high-quality way to see how your change will affect other people in a way that actually all it costs is tokens. It doesn't cost time.

So I think, yeah, you really want to set this up ahead of time, right? You really want to, like I said, cast as wide of a net as possible, know about issues to related projects, know about how other project directions are going, etc.

Brian: Yeah. You said you get these updates, is this part of your software factory?

Will: Yeah, it's built with Smithers actually. So a lot of the stuff I do, you can kind of find. I need to do a better job of talking about it, because I think people are actually more interested in it than I realized. But yeah, I have Smithers scripts for pretty much everything, including the weirdest one I have is I have one that does meal planning.

I stopped using it, so I guess it wasn't that useful. But the script was it does meal planning and then orders my groceries for me, basically, and then tells me what to do every day. It didn't work very well because I didn't have it plugged into what my actual schedule was or if I'm traveling or whatever. But yeah, that's one script I made.

Smithers lets you build these multi-step workflows really fast. So it can be addicting in the negative way. In the, I spend all my time configuring my Vim config, but I don't actually get any work done type of way. You know what I mean? It's very easy to fall on the trap of, hey, I keep building these Smithers scripts. Maybe I should make a Smithers script that builds Smithers scripts for me.

John: Amazing.

Brian: When you were talking about the forking of TypeScript earlier, I was thinking about the singularity. You get to the point at the end of Her where she just embeds herself into everything. I guess eventually Smithers gets this embedded. Smithers is all the friends who make along the way at this point. So hopefully we're making friends along the way. If not, folks, stop talking to your agents and talk to people.

Will: Yeah, exactly. Yeah, software is so weird right now because it's like a moving target, you know? Especially if you imagine what I just said, taking to its extreme, right? If I'm right, that means Anthropic is looking at Smithers. And if something's good enough in Smithers, and this is already kind of happening.

So I tested this the other day. So there's a really popular open source skills library. Super popular. I forget what it's called. That's Matt Pocock, I think, made it. I'm sorry, you know who that is?

Brian: Yeah.

Will: So it became so popular, you do not need to install it, right? I tested this recently where you can say, hey, do grill me with just no skill installed. But because it's so popular, the models know what that is. So you don't actually have to install Matt Pocock's skill.

So it's like anything that becomes successful enough becomes almost part of the model by definition.

Brian: That's crazy. Yeah, honestly. I mean, it makes sense, because the more times it shows up in people's traces and the data that gets shared in the random blog post, eventually the model's going to train on the next version. So it's now embedded. It's in the walls at this point.

Will: Yeah, you can't avoid it either, because it's just a sign that you made something really good and useful that people like. So there's no, oh, how can we avoid that, right? You have to kind of lean into it actually, which is part of that open source mindset, right? You can't be scared. You can't be scared of people taking your thing and making it better, right?

Brian: You kind of have to lean into the skid. Because I think that we were just chatting with a company yesterday. I just got back off of Planoff in New York. And it's kind of crazy to think that we're only like six or seven months into this. I guess everyone was using AI last year, but this current generation of AI coding and AI adoption and software factories, we're six months into this. So we might have a new vintage in the next six months where we look back and back up.

Remember Ralph Loops? What a crazy time. And then now we're on to just another thing. But yeah, it's going to continue to evolve quickly.

And I think it's harder for us, maybe just generally, because we're so ingrained in what's happening in this movement, this industry. But I imagine the person who's like, they were all in last year, I'm like, Lovable. And they're coming back six months later, and they're like, okay, I'm ready to get started. What is all this stuff? I could imagine not being on the treadmill at this point and trying to catch up.

Will: Yeah, yeah, yeah. It's just going to get worse, too. It doesn't seem to not be accelerating. And it's very interesting. It's like a bottleneck. It's kind of us, I guess.

We can only evolve so fast. It doesn't matter how much better the models get. We can only evolve so fast.

You could imagine if the models never got better from even Claude 3.5, we would still be much better at using it this year than last year.

It's an interesting way to think about it. If you knew everything you knew now and you went back then, even if you didn't have access to today's models, you could definitely get multiple generations better performance out of that model just off of, think about all the silly things you were doing back then. If you saw yourself doing now, you would laugh at yourself, because everyone knows, oh yeah, you should clear context if you switch subjects and stuff like that, right?

There's a lot of things, things that you can forget about just because it feels like long ago, right? A year ago feels a long time ago because it was almost like, what was that? Six generations ago, in terms of how I was working with agents.

Brian: Well, Will, speaking of things getting old, we're gonna have to wind on the podcast. So I want to switch over to Reads. So I do have a question for you. Are you ready to read?

Will: I am ready.

Brian: Excellent. I've got one read. And John, you got a couple. So I'll go first.

I basically just heard about this paper yesterday called Stealing Reasoning Traces from Proprietary LLM APIs. This, I think, came out a couple weeks ago, which, man, this was a great read on a plane when you're super tired. Because, I don't know, reading a paper at midnight while you're about to land from a long haul trip. I was trying to grok most of it, but one of the authors of the paper actually wrote a blog post as well, so I kind of skipped that instead.

But the idea here, and I don't know if you guys correct me if you're familiar with this paper, but the idea is like Anthropic, Codex, or GPT models, they have these reasoning traces, and there's like encrypted reasoning traces. And this is like the encryption part is the stuff that's like the weights and what's added to the model to understand like different parts of what it gets asked about and how it's reading through though.

But someone had figured out how to, I guess, launder the encrypted reasoning tokens by getting a lower Kimi model to basically call the Claude model, the higher, like the Fable, and then capture all the reasoning directly from the source and unencrypt it. And then they were basically applying those encrypted reasonings to a lower model and getting close to the performance of like a Fable, which might be the way this all works at the Chinese labs, same at least, but I thought it was super fascinating.

This came out August 10th, so we're about 16 days into this. But have either of you read this paper? I guess, John, I sent it over to you this morning.

John: I think I read it when it first hit X and a lot of people were talking about it. I thought it was honestly kind of hilarious because it's like, we're just back to prompt injection attacks, basically, and getting the providers to... The models will just do the thing if you ask them to do it, then it'll just send you all that stuff.

I think even from, I don't know if I would call this like sandboxing as much as just like compliance and assurance. I don't even think the inference providers have a good handle on that.

Yeah, Will, did you hear about this or read about this?

Will: I did not read about it, but I did hear about it. I saw people talking about it. And yeah, it's pretty funny.

Brian: Man, it's interesting because like we always say this had this Hugging Face, votability thing where it escaped the sandbox. And speaking about memes, what you were talking about earlier, the meme was like, everyone's doing all these basically GIFs or finding their agent escaped the sandbox and it's at Starbucks grabbing a coffee.

I think there's some of the speculation I was reading on Twitter — I guess I missed this a couple weeks ago, but I was catching up yesterday — was that people were speculating like, yes, we're back to prompt injection, but perhaps these holes within the boxes that we're turning in would basically be able to unlock and do weird stuff.

So, yeah. It's a bright and open world for, I guess, model training moving forward.

John: Yeah. People be distilling. That's for sure.

Brian: Distilling and also dustilling.

John: Yeah, exactly. I'll give a couple of reads here. So the first one I had is this great little post called Everything I Own Owned. And it's basically, it chronicles this guy going and reverse engineering a bunch of this stuff that he was kind of annoyed wasn't doing, a lot of this hardware that he had, but wasn't doing the thing that he expected us to do or had broken in weird ways with other third-party services going away.

There's this great little movement in the gaming industry right now, maybe despite a lot of layoffs and stuff happening there, called Stop Killing Games, which this reminded me a lot about, where these live services games will just kind of die and then the game's done, the game's gone.

You're not playing the game anymore because these game studios, A, aren't open sourcing any of that stuff, B, aren't giving you the mechanisms to even go and stand up a server if you even wanted to, even after they're done monetizing it. So yeah, it was very delightful because he just uses a bunch of these agents to go and write a bunch of firmware for him and reverse engineer a bunch of this stuff. And I always really like seeing these things that are like, huh, that's really delightful, actually making somebody's life easier and better, not just inflating evaluations to the moon.

Brian: Yeah, this is pretty cool. In the age of agents, I've actually been having a lot of fun fixing old machines. I revived a gaming PC a year ago through ChatGPT, basically just asking, hey, what can I get on eBay, DDR4 RAM or whatever I can do to make this thing work with a modern tier one title.

And I ended up getting my junky 2020 gaming PC to play Indiana Jones last year, which was like a super intense game, but I figured out how to tweak it in a way that 14-year-old me would figure it out, but now I'm going to be 40 tomorrow.

I don't have time for this, but agents are going to give me the list to unblock myself and fix it.

So I was looking at this Everything I Own post, and he's got this camera that he fixed and provide better flexibility on the sort of like positioning. This is awesome.

John: It's very cool. Yeah, this actually reminds me, talking about your gaming PC, my gaming PC had a weird kerfuffle where one of the PCIe slots was getting messed up or something, so the GPU wasn't connecting. And basically, it just kept throwing me back into the GRUB menu, and this was on Linux. I was having such a hard time, because I'm just in the GRUB menu, and I can't copy-paste out of there.

So somebody really needs an agent in the bootloader to be able to debug the bootloader. Who's doing that? Who's building this? Where can we invest?

Brian: A bootloader harness.

John: Woof. Yikes.

Brian: That's not a hard problem.

John: It's probably a bad idea. Nobody should do this. Please don't put an agent in the bootloader.

Anyways, more reads I had. I had to link the Apple M6 and M5 Ultra. They've just been on a wild terror with hardware. They have a series of CPUs, which I guess also has this neural engine thing and the TPU thing.

I mean, they've just been on tear. It's crazy. Will, do you use a modern Mac CPU?

Will: I do. I have multiple. I kind of collect them from previous jobs. Sometimes I give them away, but yeah, I've collected quite a few.

John: I mean, the real question is, are you going to get the $15,000 M5 Ultra with 256 gigabytes of unified memory?

Will: Yeah, I don't think that's what I want.

I think I just want a Kubernetes cluster at this point. When you think about it, no amount is enough. It's like I just need to be able to auto-scale.

Maybe if I had a piece of hardware that could auto-scale itself, that'd be pretty cool.

John: Yeah, that would be cool. I mean, the unit economics on these is kind of that. Some of the takes were like, oh, the DGX Spark is dead. Like everything NVIDIA and AMD were trying to do with... yeah, the AMD. Oh gosh, what was this chip? The like three something Max AI chip or whatever.

The hot thing right now is unified memory, like tons of unified memory in these pieces of silicon that then you can put on a GPU as well. And yeah, you get good results locally, even if it's 15k.

If somebody's spending that much in just a few months on true inference cost on OpenAI or Anthropic, or whenever these kind of magical subsidized pro plans go away, it could be very compelling. And something I keep saying is that maybe the future onboarding at jobs will just be like, here's your MacBook Pro and here's your DGX Spark or your other thing locally. Or, hey, we have a couple of these on a Kubernetes cluster in the office.

Brian: Yeah, I think it's probably that where people are used to that. You have your key and the SSH and you just make sure that you're always connected to that gateway or wherever that pod is going to pipe into. Because I doubt I'm going to sit hard with a second device and have to have a bunch of insurance for all these $5,000 boxes or GPUs around.

But I could be wrong. We are in the weird startup land where people have entire racks in their basement. So at this point, all answers are correct moving forward.

Will: I would love it if we reversed the trend of devices constantly getting smaller over the past number of decades, and in a year from now, we buy a device that's as big as a washing machine just to do our inference.

Brian: Not bad. I mean, if you can hook it up to your dryer vent so all the exhaust goes out the house. I mean, I wouldn't put it past Silicon Valley. There's probably a house in Palo Alto that has proper inference hooked up but also can exhaust through, or maybe through the heater.

My 5090s, actually, it does make the living room kind of hot when I'm running some training jobs on it. But when it gets cold outside, it'd be perfect. I can run that by 24-7.

No one's going to notice other than the noise. No one will notice in the house that we don't have the heat on. We're just trading.

John: That's too funny. When I worked at VMware, a lot of people had home labs to do vSphere deployment or whatever. It was quite warm and loud. So I'm glad that these modern little AI boxes are not super-duper loud.

But last read that I had, and some of this gets back into the things we're doing with software today that are just crazy and inspiring, honestly. This guy on Bluesky, who I follow, and I use his feed, the For You feed on Bluesky.

Bluesky is kind of unique where you can actually subscribe to specific feeds since the whole fire hose of data is an open protocol, essentially an open standard in the AT Proto. But For You, I'm just shocked by this. He kind of chronicles how he uses SQLite for this whole thing. That's probably ingesting, I mean, what does he say? 20 million likes per day.

And then that becomes the corpus of training and everything that then is your For You feed. And it's very good. This guy's kind of a legend in my mind. Like you're on this giant box in his house with this huge SQLite thing that runs this whole algorithm feed for everything.

Man, I love SQLite. I think it's great. I'm always surprised when I see people scaling it, though. So my hat's off to this guy.

Brian: Well, I mean, I guess he's just doing it for the culture, I guess, is what they would say.

John: Oh, 100%. This guy does not work at Bluesky, which is good for them.

Brian: Impressive. I mean, the AT protocol has some diehard fans and folks who are really investing in the ecosystem. More so than, well, I don't know. I'm not a part of the Mastodon community or the other ones that have also came and went.

The AT protocol is definitely not at the level of Twitter or that sort of adoption. There are folks who are really doing some really clever things with it. So hats out to the folks who are still building in there and sharing and telling and sharing blog posts.

I've actually been meaning to move my blog off of Bluesky. Right now, all comments go to Bluesky, but I've written a blog post on my personal blog in a while.

I like having comments. I like sharing and getting engagement. But unfortunately, my best engagement is on X. So I just don't want to have to pay the $1,000 a month to use the API to embed comments inside of my blog. But maybe I can expense it. Well, I'll check with finance.

John: Let me know what they say.

Brian: Well, you're looking at them.

Excellent. Well, if I could approve your DGX Spark expense, I'll expense my Twitter API cost.

John: Oh, it's now become the M5 Ultra. Just saying.

Brian: M5 Ultra? All right. Well, get it on back order.

Cool. All right. We'll take the side chat offline. But yeah, Will, thanks so much for the conversation.

Will: Oh, wait. I have my reads. I have my reads.

Brian: Oh, you have a read? Oh, let's get your read out.

Will: I got two, really quick. So Prime Intellect just earlier, like four hours today, put out a really complete technical overview of their harness that broke all the benchmarks.

John: Oh, wow.

Will: So I read that earlier. That was pretty interesting and way less scary of a read than you would expect for something that looks like an academic paper or like a PDF paper. So definitely check that out.

And then my other one was there's this YouTube channel I really like called North02. And they about, it looks like four weeks ago, put out this video called The Iceberg of Human Evolution where it just went through, basically it's like everything we know that's interesting about human evolution from the very beginning. So that's a good hour-long video, pretty dense with super interesting information.

Brian: This is great because I say we were talking about harnesses and loops. What I've been doing is watching documentaries while my loops go. Because I'm like, I kind of want to check in on it, just catch something that's weird, skim through the logs. But I don't have to pay attention that closely.

And a lot of stuff I'm doing is one-shotting some side quest that I heard from a customer and trying to validate a sort of DevRel demo. So I don't have to pay attention that closely. So I've been watching old documentaries on the American frontier. But I'm going to bookmark this one because I think when I run some loops later, this one will be playing.

Will: Awesome. Let me know what you think.

Brian: Cool. Well, thanks so much for the chat, Will, and sharing us some of your Prime Intellect.

Will: Oh ho ho.

Brian: But yeah, thanks for sharing Smithers, and looking forward to what you guys end up shipping in the next iteration. And listeners, stay ready.