Ep. #12, The Limits of Lights-Out Coding with Dexter Horthy

Joe Ruscio: Welcome, everybody, again. Another episode of High Leverage. I'm super excited today to be joined by Dexter Horthy. Dex, welcome to the show.

Dexter Horthy: It's great to be here. I'm excited to yap for a bit.

Joe: Yeah, we're definitely going to yap. It's always helpful to give listeners just a little bit of background, like your origin story. I was like, what's your origin story? How did you come to acquire these superpowers? But how did you come to found HumanLayer?

Dexter: I wrote a little bit of code during an internship in high school where I was working at NASA. They had gotten this high-res data set of the South Pole, the moon, and they needed basically navigation code, code that would help you be, I'm at point A, I got to go to point B. How do I get there without exceeding some max incline, basically?

Joe: Like a rover or something?

Dexter: Yeah, how do I get the rover from point A? It can only go up max, a 40-degree angle cliff, and there's rocks and stuff. Basically wrote a really naive, non-math version of Dijkstra's algorithm for pathfinding, basically. So that was fun.

And I didn't write code for like four years. And then I was doing a bunch of CS stuff in college. And I was actually a physics major, but I ended up doing some CS.

And then I took my first job in Chicago at a company called Sprout Social and basically realized immediately that the most valuable people on the team were the people building the software factory.

We didn't call it that at the time, but it was like the people doing the developer productivity, the sandbox environments, the previews — they just built all the infrastructure that enabled everybody else to ship faster, to test their changes faster, to deploy well, to have confidence, to react to alerts, all this stuff.

Joe: This is kind of in the DevOps pipeline heyday.

Dexter: Yeah, it was like 2013. DevOps was a word and it was like the cult in Belgium in 2009 or whatever. So it was starting to become more mainstream.

And we had a little bit of Chef and then we went to Ansible and then we went back to Chef. And it was those days of still pets more than cattle.

There was no Kubernetes. There was no service catalog. Everything runs everywhere. There was no Docker, but...

Joe: Aspirationally cattle, I guess. Which I guess probably speaks to why those people were so valuable in terms of moving the needle closer and closer to that outcome.

Dexter: There was a lot of things that were not really documented in code, which is like, okay, this server runs these six services and this one over there runs these two and two copies of those run on that server over there.

They all had names. They called it the barnyard. There was pig one, pig two, pig three. There was cow one, cow two. And it was like, oh yeah, the ingest jobs run on pig, but the other things run — it was wild.

Joe: So you had this kind of, I guess, epiphany that, oh, building the thing that builds the thing, that you get a lot of leverage.

Dexter: Is higher leverage. That's the name of the show.

Joe: Higher leverage. There you go.

That's exactly why we're here, why the show, and why Heavybit were really focused — always felt like that's a great spot to create value.

Dexter: So anyway, I drifted there on every team I was on since then. And then eventually I kind of woke up and was like, it'd be way cooler than building internal tools for engineering teams at random startups or even big, big, thing looking companies.

What if we were at a company where that was the product? It was like we were building things that helped other people ship their software factory.

And so that was how I landed it. Replicated, spent a couple of years there writing code, contributed a lot of core parts to our container scheduler and things like that.

Joe: Shout out to Replicated, Heavybit portfolio company, Marc and Grant, we love you.

Dexter: Absolutely wonderful guys.

Was there for a while and then ended up doing basically go-to-market stuff for a couple of years of just, hey, look, we just parted ways with our head of sales. Do you want to just spend all your time working with our customers?

And our customers were awesome. It's like HashiCorp and Puppet and DataStax and all these incredible developer brands. And you're sitting down with the principal engineers and all those companies. The smartest people, those companies. And also, solving the on-prem problem is the nastiest, ugliest problem.

I know you've heard the story about GitHub where anyone can work on anything they want. And so they would hire people to work on GitHub Enterprise. And then within three months, they'd be like, this is not fun. I'm going to go about something else. This is actually hard.

I heard at some point that they actually had to change that rule. I don't know, it was before or after the Microsoft — it was like, okay, if you get hired for Enterprise, you are not subject to the work-on-whatever-you-want rule.

You kind of have to stick around and make this happen because nobody likes it and we need to get people who can do it. Yeah, somebody has to chop wood and carry water.

But basically, I've been working on software factories my whole career. Either helping other people do theirs and writing YAML in the trenches with them on the go-to-market side or actually building it internally ourselves.

Joe: And Replicated specifically, they were on the vanguard of, to the GitHub Enterprise analogy, of wrapping up software and helping people deliver it, quote unquote, on-prem.

Dexter: Yeah, on-prem will be AWS VPC, but it was like, you send the app to where the data is instead of forcing people to send their data up to some multi-tenant SaaS.

Joe: But which requires just, I guess, an incredible degree of not just automation, but actually correctness and a lot of different scenarios you have to adapt to.

Dexter: If you think managing a distributed database in your own cloud is hard, wait until you have to go debug it in someone else's cloud.

Joe: I was going to say, when Cow 1 and Cow 3 are in someone else's barnyard, it's a whole other story.

Dexter: And then our job was to help other people trying to do that and make their jobs easier and then take the lessons we learned and then pack it into the product and make it easier for anybody to debug or run or launch or install or upgrade software running in a data center they couldn't control.

Joe: I guess you have a more customer-facing role. So you're helping these people solve this problem at all of these industry-leading companies. And their customer list was incredible back then.

Dexter: I remember Mitchell Hashimoto came to our office and did a fireside chat with Marc, and I was like, holy, this is legit.

Joe: This is cool, yeah.

So I guess at some point, you finished up your journey there. What led you to take a step out and start your own company?

Dexter: It was something I kind of always wanted to do. This is advice I give to founders. A lot of people are like, well, I want to work at a startup for a couple years so I can see what it's like and then I'll go start my own thing, and I'm like, yeah, or you could just go do it. That's what I did.

I was like, okay, I did engineering, and then I did go to market, and then I was a PM for a year, Replicated, because I was like, oh, I just spent four years in the trenches with our customers. I have a lot of ideas about how we can make the product easier to use and stuff. I was like, okay, cool. I got all the Infinity Stones. Now I can go be a founder.

And, being an SF, you run into a bunch of founders who are 24, and they are learning all those things, and it hurts more when they learn. I had the blessing of, okay, I got to test all this stuff out, but they did learn it faster, because they had to.

And so if you're thinking of, should I be a founder or should I go learn?

You will learn all the same things. It may be a little sloppier, a little more chaotic, but you will probably learn most of the same things eventually. And you will learn it faster if you just go do it.

Yeah, I agree with that. So you jumped in.

It was just like, I want to do something. A friend tried to recruit me as CTO for his startup. And I bootstrapped a bunch of infrastructure for them. And I was like, look, this is cool.

And I think actually, I'm going to go do my — I love those guys, but I'm like, I'm not going to wake up every morning, jumping out of bed to solve this problem.

And so I was like, I do want to go do something. So I started shopping around. I was doing co-founder dating stuff and a buddy of mine basically was like, I have this problem I've had my whole career. I'm working on an open source thing to solve it.

And I was like, oh, that's cool. Let's go spend some time hacking on that. And so we basically jumped in and started building a product around this data engineering tooling, which — bad timing, because I think that was around 2023, 2024, when the data engineering party was kind of over. There was a lot of money that went in on the tail of dbt and Fivetran.

And then everyone kind of woke up a couple of years later and was like, this TAM is smaller than everybody thought. RIP, the modern data stack.

I mean, it's great. The people who use it — it's just obvious now. It's kind of like, oh yeah, Kubernetes — yeah, of course.

Joe: It is kind of fascinating, tangent, but the whole data stack, I think, is arguably one of the only pure ZIRP category phenomenons.

Dexter: It's crazy. And I actually don't think it's a bad product. I just think it's a really hard problem to solve. Actually, similar to shipping on-prem.

I talked to other founders who thought about on-prem and tried it. And they're like, you can solve it with code. But it's one of the few problems that we encounter that squarely has to be partially a people problem. Even if you solve 60% of the problem with code, you're still dealing with the random IT staff at every single company.

Everybody does it differently. And it's just really hard to generalize. And that's why they just try to avoid it. And I think data is the same way: if you're freaking GitLab, and you have a perfect product engineering culture, where every feature is tied to a customer metric, and everything rolls up into this perfect metric graph, and everything you do in every part of the business can be linked to some top-of-line outcome, like, great — but most companies don't have that. Most companies don't have the skill or the will to create that, and no amount of software can solve that.

Joe: Well, the skill or the will to create is kind of interesting, because it's one of those things where a partial commitment or partial budget — you have to be willing to go all the way, and also the financial commitment that requires, and then the outcome on the other side has to justify that, right? And so you're gonna be out with a lot of places where it's like, oh, you put in not quite enough. But I guess it's a very bimodal outcome.

You either fully commit to it and get a good outcome, or you spend a bunch of time and money and get pretty much nothing.

Dexter: Dump 20% of your burn into Snowflake for five years. And your company still works the same way it did five years ago.

Joe: Exactly. It's one of those classic, you have to ask the question, well, what changed? We spent all this money. What changed? So you started there, but I think you pretty quickly kind of...

Dexter: Nine months and I'm just like, okay, this is not going well. We had a bunch of really exciting pilots with really exciting logos and none of them converted. And my co-founder kind of burned out.

So I was like, cool, I'll take it solo. We've been playing around in the AI space. I'm like, I'm just going to go all in on AI. This was 2024. So it was like GPT-4o was out right around then. And so it was like, okay, cool, this agent thing. And so we built this human loop thing. I know this is supposed to be the short version, but it was like human loop for AI agents.

What if I'm running an AI agent on my laptop and I go get a coffee and it wants to send an email on my behalf. What if I just got a text message and I can be like, yep, go for it. The thing that kind of got commoditized with the OpenClaw explosion.

But we did that business for about nine months and got some revenue, but it felt like we were too early or, I don't know.

Do you know Michael Grinich from WorkOS? He has this bit. He's just like, if you want to build a really good horizontal dev tools startup, you kind of have to plan it around the emergence of a standard. And for him, I think, for WorkOS, it's kind of SAML 2.

Enterprise is like, okay, SAML 1 was impossible, right? Or it was just very hard, so not as many. SAML 2 was like, okay, it's built into Active Directory. It just works.

If we can just plug in and make this integration layer really easy, then everybody will just be like, as long as you integrate with this protocol, we have the best implementation of it kind of thing.

Joe: I think one of the things that's been interesting and want to get your take on. We invest exclusively in infrastructure, developer tools. And one of the things that's been really kind of, both leads to, I think, a generational opportunity, but also maybe a generational challenge in that the substrate that you're trying to build on continues to change so quickly.

The conversation we would have had in this room at that point in time versus six months later, versus six months ago or six months from now, probably, what matters is changing so quickly.

How as a founder are you navigating that? And you've been navigating it now for two years. And so first of all, how has that been? As you look back.

Dexter: There's some hard-learned lessons. We ditched the human loop thing because I think it was too early. There was no standard, right?

So anyone who wanted to implement our stuff, what we learned was, it looked like there was an ecosystem. At the time you had CrewAI and LangChain and Griptape and all these agent frameworks that seemed like, okay, if you build against this kind of emerging standard, then you'll be able to plug in anybody's app and it'll just work. And they'll probably figure it out and come together on something.

I went and did hundreds of customer interviews with founders and founding engineers and everybody I could find who was building an AI company, mostly vertical AI kind of stuff, who was selling six-figure deals, who was doing seven, eight figures in revenue — every single one of them was using none of that.

And I hadn't realized that. That was a hypothesis. I was like, oh, everyone was building it from scratch. Nobody was building in the same way.

There's some core things, but the core thesis was like, just use the API directly and build your own pipelines and workflows because it's changing too fast.

And so there was no standard for us to plug into. And so basically, when we wanted to get people to try HumanLayer, use it, integrate it, they were all really excited.

Yeah, I'd much rather buy this than build it. Oh, I have to change my architecture that much?

And so it was either, for bigger companies, hey, can you come in and do it for us, which we decided we didn't like doing. And then there was the, okay, cool. I'm just going to build this part myself.

Joe: I think there's all this talk in the last six or nine months, this point about forward-deployed engineers. And there is something to, hey, the adoption of AI or agentic workflows in an enterprise is significantly more nuanced than historically adopting horizontal SaaS. Right?

And maybe you need humans to help with that. Or if it's something where it's actually not really generalizable at all, can you hide that with humans?

And I think knowing as a business, wait, if we're doing forward-deployed engineers, are they truly connecting that last, not even mile, quarter mile? Or are they actually re-implementing the product at each customer site from scratch?

Dexter: Exactly. Are you beaming anything back up to the mothership or are you just a consulting company?

Joe: And I think there's a lot of consulting companies in Palantir-shaped clothing.

So what was this, late '24, early '25?

Dexter: This was early '25, basically.

And so from all these conversations, I kind of distilled out some principles and it was actually, 12-Factor Agents was basically this guide that we wrote that went super viral that was principles for building AI agents.

Actually, most of it is still, I think probably 80% of it, I would still be like, yes, that's good advice. A little bit of it has aged out, but I'm actually surprised how little of it has aged out at this point.

And it was almost like the manifesto of, here's why HumanLayer is actually not a good-shaped business: because the right way to build agents is to solve your own problems, optimize, use workflows and not just call tools in a loop and stuff.

Joe: Well, it's funny. 'Cause that goes all the way back to the 12-factor app. From Heroku.

I actually, in my prior life, I had a talk, which was 12-Factor Observability. Of course. People like the number 12. It just works.

But I think a funny thing with the 12, and I'll have to ask James, one of the founding partners here is the co-founder Heroku, although Adam Wiggins wrote the original, if I remember correctly.

If the 12 was actually a number they were targeting or if they just ended up with 12, if he had a bad morning, would it be the 11-factor app?

Dexter: So 12-Factor Agents is funny. It's actually 11. And then I just added the 12th one.

Joe: You got it. Well, now you have to, right?

Dexter: The 12th one's kind of a meme. It's like, oh, your agent is a stateless reducer and, hey, FP nerds, have fun.

Joe: I think given enough time, there's probably some truism where at least four of the factors end up being, like, not. If you're hitting eight out of 12, I think you're in good shape. So I don't know if that's where you're clocking yourself right now, but...

Dexter: We kind of wrote that and it was like, okay, cool. This is clearly resonating with people, but it was also like, I don't know if this is the company I want to build.

And we had been doing a lot of stuff with using coding agents and brownfield codebases and had discovered a lot of things. And I spent a lot of time with some of the people that I would consider, I want to say, 99.99th percentile in terms of skill at using AI for producing software. And basically it just, once again, distilled a bunch of really good ideas, put them into practice, actually refined it to work.

I don't know if you know, all these really good AI people end up being kind of like lone wolves because they're so much better than everybody else. We're like, okay, how do we make this work on a team? How do we make it collaborative? How do we make it easier for teams to standardize around a particular AI workflow? And that was kind of the seed of HumanLayer or CodeLayer or whatever.

We had an open-source IDE, basically, that was like, you launch this thing and grab it off of GitHub. And it lets you manage a bunch of Claude sessions in parallel. Many, many such products.

Joe: But I think the framing as a novel IDE is probably smart.

One of the things that has kind of, and especially before Claude Code was launched, and I think there's places for IDEs, but there was always part of me, it feels strange to assume there's this alien technology that we've conjured up out of the sand. And the best way to interact with it just happened to be the paradigm we've been using since the 1970s, right?

And I think for me, the first time I started using Claude Code, I was like, oh, not that it's the end-all, be-all, but this directionally now is something different, right? Where actually sitting in the text window or looking at the code is secondary, right? It's not the primary interface.

Dexter: And we've been on a crazy journey of, you should just read the plans and don't read the code, to, actually, you know what? I don't think models will be able to write good code unattended. And I'll get in all of that.

But the feeling I have right now is, we basically need a new GitHub. We need a GitHub for the things that happen before the code is actually produced.

So I don't know if you want to call them specs or plans or design docs, but there's all these things that good software engineering teams have been doing for a really long time.

Joe: Well, by the way, I do want to dig in on that. But a thing that has fascinated me for the last few years — I've been investing for almost 10 years. But prior to that, I spent 20 years running engineering teams and really focusing on this.

And there's a whole bunch of activities and practices and processes that top-decile, high-performing teams, right? If you look at the practices they engage in, that historically, most software engineering teams that were more, I'll charitably say, median teams, right? We're like, eh, we don't have time for that. That sounds like a lot of work.

They're all things that I think are table stakes, are absolutely required, to really use agents and agentic coding.

Dexter: You think you need to have good DORA metrics?

Joe: No, no, no. Be careful. I don't know that that's the best, but more in terms of what's a high-performing team.

But I mean, there are some aspects of that. The DORA people are actually pretty hard at trying to figure out what the new version is, but just things like having a spec, right?

Dexter: But that wasn't in DORA.

Joe: Yes.

Dexter: The whole point of DORA was, we're not going to tell you a tool to use or a process to follow or a role to hire for. What I think is magical about DORA — and I know a lot of people now think DORA is maybe kind of BS or, I don't know, a little hand-waving marketing stuff.

But it was the very first time. Up until that point, DevOps was, you knew it when you saw it. And no one could describe it. It was like, well, it's not using Jenkins and it's not hiring an SRE and it's not using Chef.

Joe: Well, actually, the problem was way back then, a lot of people were like, what's DevOps? And it was using Jenkins. Or using Chef, right? It was like, oh, it's a tool. And the classic kind of DevOps refrain was like, well, it's people and culture, not tools.

Dexter: It's a culture thing. It's like there's a team that uses Jenkins that's not doing well, and there's teams that use Jenkins that's doing well. So it can't be the Jenkins part. So what is it?

And DORA was the first time we were like, okay, if you can ship from your laptop to production fast, if you can do lots of small changes instead of a couple big ones, if you can ship without breaking prod most of the time, and when you break prod, you can fix it fast, then you're a high performer.

And it was finally a thing a CTO could be like, okay, no more being gaslit by engineers on what is and isn't DevOps. I can measure this now.

Joe: It's fair. What it did was it measured a bunch of lagging indicators and said, if these lagging indicators are present, there's a pretty decent chance you're doing good things. But doesn't tell you what to do. And like any metric, if you subscribe to Andy Grove and High Output Management, if you don't have the complementary kind of thing to track whether or not the metric's being gamed, it can absolutely be gamed.

Dexter: Right, what is that? Goodhart's law is, as soon as a metric becomes a goal, it's no longer a good metric.

Joe: Yeah, unless — I think Andy Grove's kind of insight was, well, as long as you have some other metric that's like intention,

Dexter: You have like a guardrail metric or something...

Joe: The classic — his had to do with fabs because he was at Intel. But if the manager at the factory red lines this thing in this way hit the actual target metric, I'll be able to measure this other thing that tells me quality's going down. Someone's cheating.

Dexter: I have one counter argument to that that I think is a metric that's very hard to game. Simple one. So we help people measure engineering productivity. We kind of end up having to because our product's thesis is, hey, we're gonna help your engineers ship twice as fast.

And the question I always ask when we're talking to customers is, cool, if you were to pick the right methodology or the right tool, whether it's Cursor or Claude Code or HumanLayer or whatever it is, two months later, you're sitting there, you're like, this is great. This is working. We made the right choice. I'm a great leader. My team is awesome.

How would you know? Other than vibes, is there any way you would know?

And most people don't have measurement. Some people have measurement and it's actually pretty good. Some people have measurement and it sucks.

But the thing I always recommend to people is, measure the number of pull requests. And managers say, well, but if we measure people on their number of pull requests, then they'll just make lots of little pull requests instead of one big one. And I'm like, yeah, I think pretty sure that's what you want, dude.

Joe: Another thing, I tweeted this like a year or two ago, but at the time, because it's funny, this I do think was still in question, but I was like, even hypothetically, if the use of models and coding is net neutral, all the things that you'll be required to do to use models will actually generally make you better as a software engineering team.

In terms of putting out plans, because in the early days, it's like, oh, if you want the model to work, well, you got to have tests. It was like a stone soup thing. It's like, oh, well, if you want the model to work good, the way to not have to write code is to have good tests and have good specs and have good...

Dexter: You're sitting on your 20-year-old Java monolith and you're like, all right, I guess I'll add a linter. Fine.

Joe: Fine, yeah, I have a linter. It's like all these things that historically you'd have to beg and control engineers to do when they didn't want to.

But bringing it back to specs, my last team, I always believe in, okay, you're an engineer. You've pulled down a ticket off of a Jira or Asana or whatever the hot — Linear — today. Unless it's like a quick bug fix or something. If it's like, hey, go build a new feature or whatever.

Dexter: The agent's not one-shotting that in a sandbox. That's not how that works.

Joe: Well, write like a one-page — and we had a little template — write the one-page spec, and that's because, generally speaking, this is all pre-AI, but having a human review a one-page English spec before you've written code and say, hey, I like this, but not that or that or that, is way cheaper than fighting through code to get to the same.

Dexter: Writing the code takes hours or days. Reviewing the code for anything substantial takes hours or days. And so it's like, okay, cool. Can we spend 30 minutes — sit down and read a spec and basically save hours at the review time by just kind of doing high leverage work, which is, let's align ahead of time.

The process that we've found works best — and we do this internally. It's available in HumanLayer. We don't force it on people. And actually my goal is, every team has a different kind of way of applying leverage based on their code base and all this stuff.

But it's like, okay, get the product requirements right. Which is like looking at mock-ups. We don't do Figma anymore. We just use HTML mock-ups.

Look at mock-ups, understanding what's the problem to solve. Understanding, if possible — we try as many of our specs as possible have, what is the business metric? How will we know that this worked?

And sometimes it's something simple like adoption — people are using it, and if they're not using it, then we come back to, we either make it better or we throw it out. Sometimes it's like someone can do this workflow in 20 minutes instead of 10 minutes. But something measurable, because then you can feed that back in your agent and have it help you iterate. Just like you would feed it back into your product team — now you can do it for the agent.

Joe: Well, hypothetically, you would have fed it back into your product team, right? But historically, most places wouldn't. Because honestly, it's one of those fascinating — any startup founder or team, you're like, oh, it's fun to build things, and we know we want to build this. And you'll appreciate — you were in customer-facing roles — it's a lot less fun to say, what will the business value be, and how do I actually validate that that will be the business? Oh, I gotta go talk to a bunch of people...

Dexter: Well, the number of calls, even as a PM, I was like, look at all these Figma mock-ups, we're gonna build this feature. We're like, oh, that's really sick. And then you're like, would you pay an extra 200 bucks a month if we built this?

And it's very uncomfortable. They're just like... And I was like, oh, you willing to put that in writing?

So it's like, okay, getting to that uncomfortable part of the conversation is the trick. But if you can do that first — this is Marty Cagan, right? It's like, figure out if people are going to pay for the thing before you spend six months building it.

Figure out the lowest fidelity prototype that you can do, whether it's internally on your team deciding what to build or even putting in customer hands that lets you decide if this is even worth spending another day on.

Joe: There's an old saying in academia: never let an hour in the library save you three weeks in the lab. And there's a software engineering version of that, which I think, again, going back — I think the agents now taking on more of the work of actually generating code kind of brings to the forefront.

Dexter: Well, some of the mechanics have changed, right? Because you've replaced person takes a ticket off the queue and builds the thing, and that takes hours or days, with agent takes a ticket off the queue and builds the thing. And now that takes 10 minutes, 20 minutes, depending on the thing. But usually it's done in under an hour. And then the review still takes hours or days, if it's a substantial thing.

And there's two answers there. You read the code and you suffer through it, or you turn the lights off.

You do the lights-off software factory, right? You know about this?

Joe: Yeah. I'm curious to take — the dark factory.

Dexter: The dark factory, right? This idea of, hey, there's car manufacturing plants in places where they literally have the lights off. Because there's no people there. It's all automated.

Joe: And they don't have eyes or optic nerves that require light to do their job.

Dexter: And so the core thesis is kind of built around this idea. For a year, it was just vibes that I had and that lots of good engineers I talked to had. And I actually did a bunch of research and dug into — obviously what's going on inside the labs is a little obfuscated. But you can look at the benchmarks as a signal of, where is the state of the art at evaluating the quality of these models at coding tasks.

And every single benchmark is, basically, the model knows the whole problem up front. It doesn't have to make decisions over time. It's not discovering the problem as it goes.

And it's even — most of the past generation of SWE-bench, anything before 2026 is like 15-minute problems. Here's an issue off some Ruby repo from 10 years ago. Can you fix the bug in 15 minutes?

Joe: And here's some probably clear tests that you can tell whether or not it worked.

Dexter: Exactly. And so it's like a very — to do reinforcement learning...

Joe: It's a very tight loop.

Dexter: Exactly. I was talking to Addy Osmani about this from DeepMind.

He's just like, yeah, maintainability has no fast oracle. You need an oracle to do both benchmarking and reinforcement learning.

And what we don't have right now is — because the penalty function of bad architecture is measured in months and years, it's very hard to train a model on this. Or very expensive.

Joe: I think there's just so much architecture, too, that is really situation dependent, right? There are great architectures for one set of problems that you put them in another set.

And then even, at least historically, in the history of software engineering, there's this book, Design Patterns, from the early 90s.

Dexter: Oh man, I read that. I read that when they only had a C++ version available. Now you can do it in JavaScript, but I had never read a line of C++ and I slogged through that entire thing, man.

Joe: Well, there's a point in time, and that book and Martin Fowler's refactoring book. But one of the fascinating things, there's a couple of those design patterns in there. Because I think there's like 20 patterns or something. Some of them are still amazing and are used in pretty much every piece of software you use.

Dexter: Well, most of them are built in language too, right? Like flyweights and singletons, you just get that in the syntax. Now you don't have to build an object around it.

Joe: And there are other ones where, or this goes back even to inheritance and C++. Everyone's collectively like, that was a terrible idea.

Dexter: We should not have done that.

Joe: And I mean, I did date myself. I was an undergrad in the late 90s in pure science. There's so much inheritance being stuffed down your throat.

The tests would be like, oh, you have a dog and a cow and a cat. And how would you lay out the classes? And now it's all interfaces, not inheritance, whatever. Composition.

Dexter: Rob Pike got that one right.

Joe: Yes, absolutely. I think he nailed that one.

Dexter: But going back to the dark factory thing, right?

Joe: But even your benchmarks, if it's not like, oh, the code's correct or not, or the code passes a speed test or not, it's like, oh, how maintainable is the code?

Dexter: The question is, if I stop reading this code and let the agent just cook for three months, am I going to hit a problem that the agent can't solve? And am I then going to have to go slug through this three months of code and spend weeks fixing a thing? And the whole time that's happening is like, my site is down, my users are pissed, and I am miserable.

Joe: I'm increasingly starting to feel like November 2022 when ChatGPT hit to roughly November of last year was this sort of three-year preamble, sort of warm-up.

Clearly something interesting is happening, and it's kind of like sticks and stuff you can kind of do stuff, like you don't have to go Stack Overflow anymore, or you can say, hey, I need a piece of code that does this and can generate a reasonably good thing. But it was November last year, when Opus, and then OpenAI's corresponding GPT, after the five dot whatever.

Dexter: Yeah, when the Codex model came, because it was Codex, and then it was GPT-5, and then 5.1 was a Codex, and then eventually they're fused back into one model, and Codex became the harness and the product around it.

Joe: But it was the models from late last year, so about three years after ChatGPT, where suddenly it was like a step change, and that, oh, the fundamental atomic primitive of the kind of coding problem you can give it and get a working result is now something where you could imagine that as a building block, that, oh, I could see at least certain pieces of software being assembled out of this. And then that's sort of the next immediate question, because I know people were trying to do software factories and loops prior to those models, but I think it was — and which is really important, like I say, there's always important work to have someone just kind of bonkers at the frontier.

Dexter: Doing like ticket-to-PR workflows, it's just like...

Joe: Doing something just that, like, hey, this barely works. It has no business in any professional setting. But it does work. To some degree.

And so that sort of just shows, okay, that's the limit. Someone always has to be understanding what's the limit? But then if you're a professional engineer, your task is, okay, that's the limit. Where short of the limit are we supposed to stop?

Dexter: And how can I move as fast as is safe without mortgaging my code base or getting paged at three in the morning because now everything's breaking all the time or getting fined millions of dollars because of some compliance thing.

At a certain point, there were classes of people and classes of code bases where I would recommend reading lots and lots of the code. Turning the lights off is just not an option and it won't be for a while.

Joe: Well, I think software exists on this spectrum where you have everything from life and mission critical software, right at one extreme side, and then typical enterprise software, a little short of that, all the way down to toy app side project.

Dexter: My vibe coded like Minecraft for slop fork or whatever, right?

Joe: Yeah, where, oh, 60% of the time it works all the time and that's fine. But a lot of people, particularly listeners here, kind of operate in that professional enterprise zone.

And so I guess, for starters, what do you think is feasible with models today in terms of, historically, let's just keep numbers simple, like 80% of an engineer's time. So if you were able to spend 20% of your time on this high cognitive, thinking, figuring out what should the trade-offs of the architecture be?

That's the other hard thing about testing good architectures. Architecture is trade-offs all the way down.

Dexter: And you don't know what's coming, right? If you know your whole roadmap for the year, yes, you should hire an architect and have them plan the whole thing out.

But if you are a business, obviously if you're a startup, but even if you're any kind of scaling business that needs to maintain flexible and agile and lean and be able to turn on a dime to catch new trends and catch new waves of AI and new things are possible.

You need to optimize not just for shipping, not just position of, where is your product? What can it do today? But your velocity of, how fast can you change and how even accelerated, how fast can you change direction?

Joe: Like I said, now we have this alien technology that in theory can take the 80% of your time that was spent laboriously, it's kind of funny to think, manipulating a physical device using your muscles to communicate, you know, from one kind of ether into another, to the digital ether thoughts that you...

Dexter: Was it 80% of your time?

Joe: Maybe more? I don't know. It just depends on, I think, your role.

Dexter: I don't know. We work with some enterprise customers and the vibe I get and what we kind of had starting to happen at Sprout when we were getting above 200 employees was like, okay, it's a two-day feature, but we actually are being disciplined about estimating it at two days.

The coding is going to take three or four hours, but we're going to sit in a meeting and talk about it. We're going to plan it. We're going to have unknowns. We're going to then go write the code.

Then I'm going to spend a couple hours getting a review and taking the feedback and I'm going to spend some time testing and verifying. So the actual end-to-end time is two days, even though the coding took three or four hours. If I use Claude to write my code or AI to write my code, okay, that part's 20 minutes now, but I haven't sped up any of those other things.

Joe: Well, and let alone, we're talking about DevOps pipelines, but how do you get the code out to production? I mean, that was a big part of what — well, I don't know how much of the planning I would loop into the code, or testing certainly. I think you could argue at least with Claude, whether or not it's good, you get the coding and the testing kind of in a swoop. Where are you guys focusing HumanLayer today? What do you think is the right approach with the model capabilities today for a typical engineering team at a Sprout Social?

Dexter: Yeah. Couple hundred employees, that's mostly our bread and butter is a staff engineer at a 300 engineer team will find us, use it for a couple weeks and then decide to start rolling out to their team.

And the basic thing that I think works really, really well is, one, exactly what we've always been doing. I mentioned there's that, okay, you're going to spend a quarter of your time doing planning and reviewing and aligning on what we're going to do. You're going to spend a quarter of your time coding, you're going to spend a quarter of your time getting code review and fixing it and fixing all the bugs, whatever it is. And then you're going to test, verify, deploy, whatever it is.

I think the number one tiny advice, the one word advice is seek leverage. Find ways to take that, you know, do 30 minutes of planning upfront, use AI to help, right?

Because writing an architecture doc could take you three days. You talk to a principal engineer at a pre-AI enterprise, they're spending a week writing an architecture proposal and it's like a 10, 20 page Google doc.

And then that goes to the engineers. And then they just talk about, okay, how are we going to split this down? And how's it going to be milestones? And what does the PM care about?

All of this stuff, you can accelerate a lot with AI, but there's a lot of leverage from having humans in a loop before you go write the code. Because once the code is written and then you're just yelling at the model and the pull request comments, it's going to be a lot slower and a lot harder. And the chance that you'll have to throw it out goes up, basically.

Joe: Well, the interesting thing I found models, and this isn't just for coding, but even to your point, I don't know, making a presentation or an architecture doc, or just saying at a super high level, oh, just go do X, Y, Z. It will generate an awful lot of content, most of which is, at extreme surface level, from a distance, you're like, oh, that looks good. And then as soon as you try to engage with the material, you're like, oh, I have to cut 80% of this and tell it to go...

Dexter: So I do this in writing. I'm writing a very long post right now.

And I was like, cool. Here's the transcript of my conference talk. Go write the headers. And just wrote all the headers.

And I tweaked the headers. I'm like, now go write the first beat for the first header. And I tweaked it and played with it. And it was like, now write the next beat. And now it's starting to get a pickup for my tone and how I structure stuff.

And so the same thing is true of code. Cool. Let's plan the high level. Give me the headers, right?

Almost like, I don't know. If you were in C code, it's like, give me the C header file. And then we're going to implement one function at a time.

But there is a way to do this where you're going two to three times faster compared to just YOLOing prompts into Claude and compared to the, yeah, I'm moving 100x faster but I'm going to throw all this out in six months because it's going to go to slop.

Joe: I was just talking with some founders this morning, very early stage, very senior technical experience people, but they were setting up, basically scaffolding up an initial cloud infrastructure for a distributed system. It's a little more sophisticated than just like a CRUD app. So, a couple different services.

And they just had Claude generate all of the Terraform or whatever as code. But then I think pretty intelligently we're like, okay, we spent like a day and it generated this whole thing. And you hit a button and it kind of goes. But now we're going to sit and actually work our way through all of it.

I think initially their thinking was like, oh, we're just going to spend a day or two reviewing it. And what actually happened was they spent three weeks pulling pieces out. And where they ended up was they were like, well, what we ended up doing was just pulling piece by piece. And as you pulled each piece out, you're like, oh, this isn't quite right. I got to change it, whatever.

And three weeks later, they have a great working thing that's about 20% of the original footprint Claude — I don't want to cast a sprint — the model pumped out. In hindsight, that maybe wasn't a win.

Dexter: Exactly. And you can overdo this, right? Some people just try to get their plan perfect.

So in our process, you have the product doc and then we have the system architecture, which is high level: what are the contracts? What are the data models? What's the flow? And maybe what's the shape of the new endpoints we're making, whatever it is.

And then we get into actually a thing that I think most people skip, which I call program design, which is, I think a lot of people say, okay, if we get the architecture right, the model could just cook and it'll do my thing.

But actually, you'll still have weird try-catches around everything. You'll have weird hacks of this object over here is calling methods inside from an internal class from this object over here.

The model has no intuition around, hey, that's going to make things harder in the future, because we couldn't teach it that during RL. We could teach it to solve programming problems, and we couldn't teach it to care about the maintainability.

And so we do the program design of, okay, here's the thing we're building. And then we do an outline, basically. And at any point, it's like, okay, we're not trying to get it 100% right. We're not trying to one-shot the architecture, because at that point, just write the code.

But it's like, how do I spend 10 minutes to give me an 80% chance that it's correct, so that on the next phase, I can spend 10 minutes and then spend 10 more minutes as we're going from the 50,000-foot view to the 25,000-foot view to the 10,000-foot view to then we go write the code.

Joe: Have you found, is there a scale or sophistication of a piece of software at which point it's like, oh, well, the bones of this are like good enough and strong enough that I can kind of just actually let the reins out a little bit on the model, or is this just a constant kind of challenge where you're trying to get it to the exact right throttle or whatever you want to call it?

Dexter: I think it's not a black and white thing. I think every code base is evolving and adopting and incorporating new patterns as you new functionality. So there's parts of our code base where I'm like, I know I can one-shot this, not only because I've seen Claude one-shot this or Codex one-shot this many times in the past, but also, I know that part of the code base. I know the examples of what to follow are very close to where it's going to have to write the code. So I know it's going to find them, and I know it's going to follow the patterns, and I have pretty high confidence that it's only going to have five to 10% rework needed max. And there's a low chance of that happening.

Joe: I've been kind of the mindset, or it feels intuitive, that in any sufficiently sophisticated software project — what you just said — there's going to be this gradient where, oh, there's certain places in the code where we can kind of let these things be almost unattended, or like a dark factory sort of thing might almost work, but then there's other parts in the code, and whether that's the hot path or, as Claude would say, the load-bearing part of the code, where you're like, we probably always want some eyes on both the plan and the output.

Dexter: You have some instincts about what's probably gonna not go well, and it's like, cool. Because again, it's like, what is the thing I can do that is 20% of the effort of reading through the code and fixing it, that gives me a, you know, 80% chance that I won't have to go read the — I think about it in terms of expected pain of, how likely is the model to get this wrong? And how hard will it be if it is wrong?

I don't know if it's going to get the color of the button right, but I'm perfectly happy to vibe polish that afterwards.

Joe: What's the cost of being wrong?

Dexter: Exactly. And what's the chance? It's a standard risk modeling.

But in November, December, we basically took our open source, chucked it, and basically rebuilt it from scratch. Basically, this was the end of my "don't read the code" cycle. We had gotten to the point where it was easier to rebuild from scratch than it would be to repair everything that was wrong.

Joe: Yeah, I've seen you speaking about this before. One of the things that is curious to me is, I mean, you guys were kind of at vanguard, or at least kind of at the vanguard.

Given the shift in models — not that the current models are some magic solve. But I think in hindsight, you could be like, oh, of course, in March of 2025, Sonnet or whatever couldn't do that. Do you think the models that exist today are closer? Or how much of that is just fundamentally won't ever work, relatively speaking, with transformer architecture versus, oh, we're just not there yet.

Dexter: I don't think it's about the architecture. It's about, what can we teach the model during RL? I say GPT-5.5 or 5.6, or even Fable, I will tell you right now, we'll not write code that gets better over time.

If you give it five features a day for the next six weeks, you will wake up to a ball of spaghetti.

Now, these models can create incredible refactors. You turn on UltraCode, you can go change 100,000 lines of code in an afternoon. But they will not do that unless you tell them.

And so the thesis of the dark factory is, no human steering about the code and how it looks.

If you are not going to steer it to architecture, if you're not playing a role in how should this be designed and taking your 20 years of hard-won intuition about what makes software suck to debug versus what makes it easy to maintain, then you will not by default get good code.

That's kind of where I'm going. That's my take.

Joe: Yeah, and this has kind of been my experience, even just with side projects, since I've been building a lot of stuff, especially since last fall, when actually I think the things get good enough that with the amount of time I can dedicate to coding —

Dexter: No, the scope of and complexity of a thing you can do without caring about the code grows with every new model improvement.

Joe: Well, but the thing that has been interesting to me is I definitely have repeatedly now had this experience. Because I've got some stuff that I've been running in my personal production or whatever you want to call it for nine months.

You'll say, oh, go bang out this new feature. And it's like, great, I've got it. It's kind of pretty sophisticated, messed with a bunch of stuff. It one-shots, it works great.

And then you go to roll and just use another part of the piece of software, and just completely host. Just totally broken.

Dexter: And the more you go, it gets harder and harder to fix something over here without breaking something over there.

Joe: I think part of it is just, like you said, by default, it just sort of puts these blinders on that just, I don't know anything. I've been curious about it.

Is that just because I haven't loaded or written some skill that's like, hey, you are a senior software engineer? Always consider what you might be breaking when you do something.

Dexter: I think if that skill was possible, we would have it already.

Joe: Right. But I guess the question, it sounds like this is endemic. This isn't just my complete lack of effort in my harness or something.

Dexter: Well, again, this is what it is. It's like if you put in zero effort, it will do what you want.

And I actually think that Opus 4.5 was not as smart as Opus 4.1. The reason why Opus 4.5 caused the blow up is because even though it wasn't as deeply intelligent in terms of problem solving and focusing along, paying attention along large — I think Opus 4.1 was actually a larger model.

But Opus 4.5 was two and a half times faster, and it was squishier. It was better at, you put a low-effort prompt in, you get something great out. And that was what made the change.

And that's actually, I think, how all of the go-to-market for the model harnesses from labs is, how do we make this more accessible to more people, sometimes at the cost of expert users?

Joe: Well, I think part of what they did, too, was, I think with some of the RL techniques, right, improve how the harness could use the model. The harnesses have, I think, gotten better too, or at least the foundation lab —

Dexter: I mean, they've gotten very good at — up until Claude Code, the way you built a good agent was, you got a model and then you gave it a set of tools, and you tweaked your tools and you did a bunch of evals, and you did GEPA or the DSPy thing, and you change your tools and optimized all the prompts to work for whatever the model wanted to call, however the model wanted to behave. You were like, how do I make the harness fit the model? And with the labs, they did that.

And then they took the model and they trained it in the harness. Now, how do we make the model better at calling the tools that are already in the harness instead of just doing it?

I gave a talk about this in November that was basically like, if you are building a coding agent harness and you do not also own the model, you are going to be a disadvantage of a team that can tweak the weights of the model to fit their harness however their harness works.

I mean, this is what OpenCode does. The reason why OpenCode is tied with Claude Code on the leaderboard is because when OpenCode calls the Claude Code inference, it looks exactly like — all the tool definitions are exactly the same.

All the responses are exact. When you read a file in Claude Code, it comes with all the line numbers. Everything is mirrored. And if you mirror all of that and you mirror the exact harness that the model wants to be living in because it was trained for it, that's how you get the best performance.

Joe: Interesting.

Going back to the dark factories. Couldn't every fifth feature, you put almost like a tracer round, right, where you put a thing in the factory which is like, hey, refactor everything or clean it up, right, or make it more maintained? I don't know.

Dexter: I mean, people do this, right? And the same thing with code review agents: review this code and make it more maintainable, whatever it is. I just think that what I have seen is the models don't know what well-factored code is. They can read books and apply what's in the weights. Most of what's in the weights is what they learned in RL, and we teach maintainability in RL.

So when Fable came out, I had to do a 5,000 line refactor of our ugliest React code. And I was like, okay, go do this. And it did kind of what I wanted, but not quite the shape.

And I gave it some pretty good steering. I was just like, okay, remove all the prop drilling and all these patterns that make the performance bad. And it brought in a bunch of patterns that I don't like that make the code base worse.

So you just tell it, hey, refactor and make it better. You may or may not get good results. But the thing that actually makes it better is you're like, okay, we have this anti-pattern over here and this thing over here.

There is a ceiling where it's like human judgment is required above that ceiling. To get passive. Yeah, you can say, hey, refactor this. And you might get good code. You might not.

Whenever I tell a model to YOLO refactor something, it's almost never worth it. I almost always end up throwing that commit out.

Joe: Well, I guess the ceiling is an interesting point because I think one of the challenges for engineering management is trade-off, right? How much should be invested? I mean, YAGNI, right? This is the classic, you aren't going to need it, right?

Which varies, like for this particular piece of software, is this an internal tool? Is this something that customers super dependent on? Is this something critical? So what ceiling am I okay with? Which I guess would kind of dictate how much review or not.

Because my kind of perspective has been that, I mean, future is not evenly distributed, but if you've fully adopted these things or you're trying to adopt as much AI as you can in your code generation, human review of code and human taste is the bottleneck.

Dexter: So how do you apply it in your workflow as efficiently as possible?

Joe: Efficiently as possible. How do you do what you can, to the extent you can remove that bottleneck or reduce its kind of constraint? What do you do? How do you do that?

Dexter: I mean, this is why we're building this space because we think this is super important. And you think about what GitHub did to software development is, I can now go to a CTO and say, hey, there's a — I mean, CVS is kind of a thing — but there's a single distributed place, a single centralized place where I can put my code together and release it from this place. And it's always up to date and it's always the source of truth.

And before that, what were we doing? We were emailing patches back and forth on the Linux mailing list.

Joe: Yeah, or you were FTP a tarball up to a server to deploy, which hopefully was the right one. Hopefully you tarred up the right directory on your laptop before you did that.

Dexter: And nowadays we are in Slack sending prompts back and forth. And it kind of feels like the same thing.

And it's like, how do we build a place for people that is optimized for collaboration on things that are not code, but are like code and are attached to code and live in the context of code and create workflows that are super human friendly that allow us to extract that — I don't like to say extract — but how do we make it as easy as possible for humans to leverage their judgment and intuition in the construction of software?

And I'm telling you right now, the answer is not stop reading the code and the answer is definitely not burn out your senior engineers by sending them a mountain of PRs that are 50% slop code every day.

Joe: Because you have only two choices. One is stop reading the code. Which to your point, you're clear about how successful you think that can be.

And then the other one is, the code has to be read. Which, by the way, I think you make a good point, which not a lot of people are, is there's real value if the code needs to be read or reviewed, or at least some chunk of it does, then everything you can do to, before that code gets in front of the human, improve the likelihood that it's good and it's terse, and it's not like the, hey, good news, your jaw, here's your machete, you have to figure out which 20 of this is actually — that's review.

Dexter: Yeah, exactly. Go spend three weeks picking apart the Terraform

Joe: and find the right thing.

I'm curious, a kind of segue I did want to talk about today. Just the last week, Kimi, Qwen both dropped what appear to be very capable open-weight models, which by the way is a continuing pattern that's now been going on for a few years.

Dexter: They said it would be July that the Chinese labs caught up to the US ones. And it's like, someone looks like that guy. We got that right.

Joe: See if he knows anything about horse racing.

What's your current thinking on where, and I guess particularly we could narrow it to, coding agents, right? What do you think the future portends for coding agents?

How much code is going to be written by state-of-the-art foundation models? How much is going to be written by open-weight models? What is the impact going to be of having these open-weight models that have these capabilities?

Dexter: I mean, I think we're seeing the impacts of both the frontiers competing and the open models hitting the scene is like, hey, we got more Fable 5. They kept saying they were going to cut it off. 5.6 drops. Okay, extent.

I mean, you can have whatever theory you want. The Anthropic story is, of course, oh, we just wanted to make sure we could really service the demand. And now we're ready to...

Oh, yeah, they figured it out. They made a heroic effort to figure out. We did it. We got you guys more Fable.

But no, I think competition is good for any market. And I think especially the competition between open and closed source is good.

We've done a couple evals on Kimi K3 and seems good. Has some of the weird Kimi ticks. We've tested every one of the Kimi models as it comes out.

We're working on being able to deliver GLM 5.2 inference to our customers as part of the product because we think it's a really good... If you can run it fast, it's very good for certain things where it's just put up a really fast quant of it. And the FP4 quants are actually pretty good for things like code-based research and things like that.

So I love the open ecosystem. I love competition because it just means we all get better stuff even from the closed. It makes the closed models better having the open models in the arena.

Joe: Yeah, I agree with that.

I just think particularly for enterprises, I've just always felt that, I mean, this is where the history of open source software comes from, is the benefits of, oh, I can own this thing. Or even if I'm paying a third party company to support the open source piece, but the avoidance of permanent lock-in, the ability to — for me, I always said with open source, it's for the vast majority of enterprise users, it's not like, oh, the free is in speech.

I've heard that one before. It's the right to inspect and right to repair, right? It's most people care about.

Dexter: Yeah, the tinkerers, all the people hoarding GPUs in their basement.

Joe: Well, it's just, if in the middle of the night I need your piece of software to do something, and I have good engineers on hand, I can crack open the open source software, the database, if I really need to fix it.

With open-weight models, I think just the ability, like if I'm a large enough enterprise and I have my own GPUs, quote unquote on-prem, right, I can trade off, oh, what subset of my tasks do I want to run on premium tokens, what tasks can I take a hit on cost...

Dexter: For your knowledge workers that are running summarizing emails and writing calendar events, you don't even need Sonnet.

I just talked to a guy from AnythingLLM, this guy Tim. With a bunch of ThinkPads, and all these ThinkPads have an NPU that no one knows really how to use. And they're just like, yeah, we'll run local models on your laptop. And for most knowledge work, you can use a tiny little Qwen model to do almost everything.

Joe: Well, I agree with that.

So then coming back to, the coding harnesses are best when their argument that they're super tied to the model, that kind of future argues for ideally a coding harness that I guess is not tied to a particular model, or am I going to need a different coding harness for each of my coding tasks that use different models?

Dexter: If the open models catch up, and they have, like, fully caught up, I can't tell you for sure that K3 is a drop-in for Fable. The vibe check online seems to say no.

But when they catch up or when they get close enough, you will see people building custom harnesses who think they can build a harness better than the labs will then do a fine-tune of one of these frontier open models in their own harness and they'll get the same benefit.

And so it's that benefit that only the frontier labs have of that ability to tune the model for their harness, for their product, one step up from the platform level into the product space. Now anyone can have that benefit.

Joe: Okay, cool.

Dexter: It's a good question.

Joe: No, it's something that keeps me up. I'm trying to figure out. I mean, it's just fine. I'm thinking some of the harnesses. I like exe.dev as one of our portfolio companies.

Dexter: Oh, yeah. I have the HumanLayer daemon running on exe.dev box right now and doing some stuff.

Joe: Yeah, it's a great agentic workload hosting platform, vibe coding platform. They have their own harness.

It's the Shelly thing. It's kind of cool. It has a pull-down menu, and you can just be like, oh, of the 15 models, I want to try this one, I want to try this one. And it just feels intuitively to me, if inference is a commodity at some level, which I do strongly believe it's going to be, then going back in networking, there's this thing called the end-to-end argument. It was originally for when people are arguing about what layer, application layer or network layer, transport layer, different features should be in. And the argument was, for most things, you want stuff close to the application layer, because that's the only place that all the context exists to really make the most fine-grained decision on an app-by-app basis.

Dexter: You don't want to put your database charting in your network table or something.

Joe: Right. And so there's things right now where, I don't know, for instance, there's these routers, and they're like, oh, okay, well, historically we've just been kind of this dumb switch where you just say, hey, I want some tokens from this one or some tokens from this one, but maybe now what we'll do is we'll actually look at the request coming in.

Oh, I don't think that works. It just seems like I can get why, if you're there, you're like, oh, it would be a good thing for me to automatically maybe route things to one model or the other, but you're missing some. The application should be the one for the most part, I think, deciding, oh, this should go to Fable. This should go to Qwen, sub chunks.

Dexter: Basically, yeah. Bring it. I think doing it at the inference time decision is not — you need to do that in the harness. And a harness is basically just a giant DAG that manages a bunch of different workflows, right?

Or a small DAG, I don't know. It's not acyclic. It's just a directed graph.

But it's like, okay, cool. We're going to do this. Okay, this tool came back. Okay, we're going to use a Haiku model to see if it's safe.

Okay, yes, then we run the bash command. There's all this deterministic stuff happening under the hood.

It feels like just calling tools in a loop, but there's tons under the hood for any harness.

And actually, I realized the thing I said was, oh, people are going to do a fine-tunement of an open model and ship it in their harness. This is Cursor Composer. This is exactly what Cursor did, is they took a Kimi-based model, fine-tuned it on their data in their harness, and then now they have their own model that's fast and cheap and good.

Joe: If you fast forward a few years, which is like infinity compared to what's happened in the last few years, what do you think the prototypical kind of software engineering team role looks like?

Because one of the things, even whether or not code's being read, I just remain convinced that humans have a very important role to play in software engineering for a very long time, at least serious enterprise software. But what do you think the typical day of a senior software engineer looks like a few years from now? What do you think their kind of key tasks? And Jensen had this thing where he was like, the job is not the tasks, or it's like the job is the outcome. Software engineers have had outcomes they've driven at a high level. Those outcomes still need to be driven, but the tasks are probably going to change dramatically.

Dexter: I think in the short term, it's still aligning and understanding. And the level of detail that you need to align with on the model, and the amount you can trust it to figure out what's under the hood, goes up over time.

But I actually don't like answering this question. Okay, you may have in your mind there's a 5% chance that we have the infinite abundance AI world where nobody has to work anymore, full AGI.

Maybe you think there's a 5% chance that we end up in the Terminator, right? We do a bad job at safety, and then we all get blown up at a nuclear explosion. The bad AGI.

And then you think in the middle there's a 90% chance that things just kind of keep getting better, within this range of, okay, it might get better at 2x a year, it might get better at 1.1x a year, but somewhere in there it's going to keep improving as technology just improves over time for everybody. And I think it's easy to say, cool, I think we're kind of still going to be on the same path, and we're going to be caring about the code, and we're going to be having to play a role in this for a little bit.

But I may believe that there's a 30% chance we get to that abundance world. I may believe that there's a 30% chance we're living in the Terminator.

But I still am probably going to live my life as if we're somewhere in that middle range, because nothing you do today matters if either of those first two things come true.

So the only thing you can really do is, unless you want to go try to be head of safety at a frontier lab, which I'm not personally really interested in doing, then you kind of plan for the middle path.

So whether or not I'm saying this is definitely how it's going to go, I'm like, this is what is worth investing time in.

Understanding software engineering and software architecture, and what makes systems good and what makes systems fall apart, is very much plus EV to keep investing there, keep caring about it, and keep building your factories around this idea that, hey, we are going to need to be the stewards of this system at some level of abstraction.

Joe: Well, I think that's a great note to end on. Thanks so much.

Dexter: Yeah, this was fun, dude.

Joe: We'll have to do it again sometime.

Dexter: Yeah, I'll give you the update in three months, when I decide I was wrong about everything.

Joe: Yeah, let us know.