Ep. #11, Durable Systems, Disposable Code with Chad Fowler

Adam Zimman: So I guess the first thing we should do before we forget is, Chad, we would love to have you introduce yourself.

Chad Fowler: Oh, okay. I'm Chad. I am now CTO and general partner at BlueYard Capital, which means I'm a VC, but I still identify as a developer. I've been developing software for 35-ish years, written some books and spoken at some conferences and stuff like that.

I live in Arkansas. I play jazz saxophone. I don't do any of these things in Arkansas, though, because no one does them here. So I sort of am professionally based in New York for both music and tech. And I'm currently writing a book for O'Reilly called Regenerative Software, which is the name that makes more sense than the Phoenix Architecture, which is this blog post series I've been writing that is sort of the topic of conversation today.

Heidi Waterhouse: So we all went and read that. And then, of course, we're going to make you explain it anyway for our audience. But it was super interesting.

And one of the things that I thought was like, many of these ideas have been around for a long time, but applying them to AI makes a lot of sense. So would you like to unpack a little bit what you mean by Phoenix Architecture, regenerative architecture?

Chad: Yes. And you're absolutely right. These ideas have been around for a long time.

And at risk of not answering your question quite yet, it all came from a bunch of crazy architectural stuff that I've been doing for almost 20 years. And I say almost 20 years because I actually remember the origin of this weird line of thinking, this path I've been on. But you're right, it's all architectural ideas around change.

So I'll just tell the story. I was in a car with Corey Haines, who was well-known in the software craftsmanship world at that time. And I was going to be speaking at a conference in Sweden like a month later, and we were talking about legacy software, and the fact that the word legacy is maligned in our industry. But in other industries, it is not. It's like positive, you know, or other walks of life or whatever, you know.

And I'm a musician, as I mentioned, leaving a legacy in music, that's what you want to do. The opposite of that is no one knows what you did and no one cares. So I decided I was going to do a talk called Legacy.

And this is the way that my brain works. I pick some concept and then I just like grind through my brain for days and hours and months and years in this case, unpacking what it means and like developing a point of view around it. And with legacy, the idea was like, how can I create systems that can survive longer than like five years or so? And my background at that point, I had been doing a bunch of these like rewrite projects.

And what I was realizing is I was actually a professional software euthanizer. I'd go around from company to company and take the hard work of people who put like sometimes 80-hour weeks into a project and just throw it away.

And sometimes it would be a big throwing away, and sometimes it would be a gradual throwing away. And I've seen all the versions of how to do this wrong and right.

Adam: You were like George Clooney coming in and just like cleaning house.

Chad: That's right, yeah. And I have the air miles to prove it too, unfortunately.

But I started thinking like, how can we make things that aren't destined to die so quickly? Why is software so brittle and so difficult to change, and therefore hard to keep alive for decades?

And so I turned to like a biological metaphor. I am now in my 50s. I don't really deserve to be here. If you look at the maintenance I've been doing on this particular biological system, it's not very well cared for, relatively speaking, but yet I persist.

And so I started thinking like, well, Kent Beck used to say, if it's hard, do it all the time. That's how you get good at certain practices. So I thought, well, what would happen if software was made so that it should just constantly be changing?

And pre-AI, that meant coming up with structures and architectures and culture on a team that would allow that to happen. And so, I'm not going to keep talking about this, but the extreme version of this is someone gets paged in the middle of the night. They wake up, the Haskell service is crashing, and they've never tried to learn Haskell. And if you've never tried to learn Haskell, then you don't know yet that it's hard to even do Hello World on the first day in Haskell.

So what do they do? Do they call me because I wrote the Haskell service? No. They rewrite the service in Go, and they deploy it, and the problem is solved.

So imagine you could do this, but at like a thousand times the speed. So this is sort of the direction that I was going all this time, and that was the sort of core thing.

How can you make it so this is possible, so that instead of refactoring or fixing code, you replace the code? What are the architectural constraints that make that possible, and the cultural constraints? And then as generative AI has taken shape, I have a hammer and everything looks like a nail, of course, but I think in this case, the hammer is way more relevant than it ever was.

Because now we truly can create systems where you look at the system, and from a usage perspective, it's the same thing. But today, 90% of the code is different than it was yesterday. That might be a bad idea, but it's possible now, and it hasn't been possible in the past.

So that's why I called it Phoenix Architecture, the idea that the system is rising from the ashes or the components are rising from the ashes after death. Probably it should have been called the Theseus architecture for the ship of Theseus, which turns out is a better metaphor for this, but it also doesn't roll off the tongue.

Adam: I mean, you know, I guess it depends on the circles you're in, right? If you'd timed the release with this summer's blockbuster, then maybe it would have...

Chad: Good point.

Adam: But yeah, no, this is fascinating. I think that, you know, the way that you talk about it is really interesting. And to your point, I think that early in my career, I spent over a decade at VMware, where the time that I spent there was a lot of explaining and working with people similar to yourself, in the sense of people that had legacy systems that were trying to figure out, like, how do I continue to move these forward.

And it came into two categories. How do I do a lift and shift so that I can continue to run these old systems on newer hardware without touching them? Because I have no idea how to touch them. Or what does this look like for me to start picking at the edges and maybe breaking monoliths down into smaller services or microservices and being able to redeploy into new architectures?

So there was a lot of kind of things that I noticed that were very similar to the problems that we're dealing with, you know, 20 plus years ago. But it was fascinating the way that you were kind of positioning it and the way that you had tied it back to this is what you can do when you start thinking about the advantage of rate of change that AI is capable of introducing. So that was something that I really appreciated you kind of connecting those dots and bringing that full circle.

The interesting thing that kind of caught our attention, and I think this came up when we were having a conversation with Charity, one of the things we've been talking a lot about for not only the Third Loop podcast, but also progressive delivery in general, is how do we as humans start to kind of accommodate this increased rate of change that we're starting to experience.

And AI has been this next step function acceleration that we've seen, right? Really kind of increasing the technological jerk that people have felt of being kind of pulled forward. But I think that the way that you position it is interesting and fascinating from the perspective of, like, the post that really kind of resonated with me was also Pace Layers, and talking about just because you can doesn't mean you should. And thinking much more conscientiously about who the consumers are of the software, whether it's other systems or services or humans. And that should factor into how readily and aggressively you choose to update or continue to change how things are done.

Heidi: The Pace Layers post was the one that I copied and pasted to, you know, three different people. I'm like, go read this right now, including my 23-year-old son who is just starting his software career. And he's like, oh, that's really interesting.

And James sent us a thing that he had written in 2008 about Pace Layers. And I'm like, it's extremely well articulated, but let's explain it for the audience. So do you want to do that, Chad, or do you want to see how I'm interpreting it?

Chad: Either way, mine is an interpretation as well. So why don't you explain it?

Heidi: All right.

So the way I understand it, pace layer is understanding how much trust is going into a layer so that you know how much risk you take on when you change it.

So if something is foundational and everything relies on it, you change it more slowly and more carefully and with more caution. And if something is high level, close to the customer, or otherwise consumed ephemerally, you can change it pretty much as fast as you can change it. Because that's the moment where things don't need to be as trustworthy because they are shorter lived.

So I don't want to say that top-level, you know, fast-paced things are fast fashion, but it's a little bit the same. It's like the styles are like this this month, and that's how we're going to do it. And we're going to retool and do something different next month because it will then be boots and leggings season instead of skirts season.

But that idea that what we are really thinking about is not where we are on the infrastructure to software continuum, but rather on the stability to ephemerality continuum.

Chad: Yeah, you did a much better job explaining it than I would have, and gave me new ways to think about it too. I love the fast fashion thing. I like that a lot better than the cattle idea too. I never liked that metaphor. It's just not nice, you know.

Heidi: Well, the one I extended the cattle metaphor to is, like, after cattle we get to biomes. So, like, when you have a petri dish full of little cultures, you're okay killing it off and re-inoculating and starting again. It isn't any one, like, cell that we care about, it is the culture that we care about.

Chad: Absolutely. The pace layer thing, it's interesting that he had written about that in 2008, because to me, it wasn't relevant until now. Because everything goes so fast now, it matters, right? And back then, you had to have a really good organization to be worried about changing things too quickly.

Adam: The interesting part about that article, and we'll share it in the show notes, was that James at the time was writing it in regards to some announcements from SAP.

Chad: Okay, very interesting. I need to read that.

Adam: So thinking about it in the context of, at the time, SAP — they were doing some of their push into the cloud, and they were starting to talk about, hey, look, you can make changes faster at this higher level with this abstraction layer. But under the covers, we'll still keep things consistent and capable of managing all of your physical manufacturing equipment. So it was the rates of change in terms of, like, cycles per month were significantly different than what we're dealing with now.

But it's funny, because this was one of the things that we encountered when we were writing Progressive Delivery. One of the books that we found to be kind of the greatest echo of what we were encountering in writing was actually a book called Future Shock that was written back in the 60s. And so this was literally — Alvin Toffler was talking about the fact that the pace of change was moving so fast at that point in time that people were going to develop a medical condition where they were actually shocked into being debilitated, because the pace of change was moving faster than their bodies could handle.

Chad: Their bodies. Wow.

Adam: Yes.

Heidi: It was sort of like the proto jet lag. Like in 68, you could definitely get jet lag, but not the same way you can now, because it took most of the day to cross the country.

Adam: But this was one of the things that we learned from our publisher at the time, or one of our editors. Early on, when we were starting the process with them, she's like, I just want to kind of let you all know that as you continue through this process of editing and doing the research that you need to for your book, there's a strong likelihood that you're going to get to a point where you throw your hands up and you say, this has all been written before.

And she said, I just want you to know that that's going to happen, and don't be discouraged, because just because it's all been written before doesn't mean that a retelling isn't worthwhile.

And I think that this was something that we kind of took to heart, because truly, the more we kind of dug into the ideas we were trying to express and the ways that we were trying to bring them forward, we found an increasing number of examples that we were able to draw upon and build upon that had been written before. But that doesn't mean it's not valuable.

And to that point, look, I have been appreciating your posts with the kind of Phoenix Architecture, because I think you're making the connections and you're verbalizing the connections in a way that is very relatable. And so I do appreciate that.

Chad: I hope so. Yeah, I appreciate you saying that. You know, I'm definitely just extending stuff that we've all been talking about forever and pattern matching.

And, you know, maybe this is me, just my hopium or whatever, but I think I'm discovering that there is a place for us old people too, who've done a bunch of stuff, even in this world where the models can do everything. And so I started thinking about this in terms of just like, well, a few things will happen.

One, a few years ago, I was thinking, I bet at some point I'll be able to use this stuff to create actual code that I'm going to deploy to production. That was a weird-sounding thing to say four years ago. And then it was like, I bet someday I'll be able to generate code and not look at it and deploy it to production.

And now this year, I pretty much would just bet every software developer in the future will be generating code and not reviewing it and pushing it to production. And even if you hate that idea, it doesn't matter, it's going to happen, because the incentives are lined up.

So how do we make that okay? Because it is going to happen. Right now, it's not okay. And the model providers aren't going to make it okay for us either.

You're not going to one-shot something from Mythos and just have that be something you can deploy — a massive, complex system — and maintain it over time.

Adam: So, would love to dig into this a little bit with you, because especially now, knowing a little bit more about your background than kind of this traveling euthanizer of software.

I think that I have worked with a lot of large — we call them enterprise when we want to be positive, right — companies, but that have legacy systems. And I look at the trend of how long it's taken some of these organizations to adopt newer frameworks or technologies, like even just kind of moving any workloads to the cloud or a cloud-like environment that they would run themselves. When you look at this change, this next trend, are you having conversations with people that you are surprised to see them moving faster than you would have expected?

Because I think that this is where I'm trying to read the tea leaves a little bit of, like: Is this going to be a storm that folks can weather like they did with the move to cloud computing? Or is this going to be something that, no, this is a trend that you're going to just cut left behind and start to experience customer loss or serious problems?

Chad: Career loss, yeah. I think it is that. I think you're screwed if you don't jump on this as a software developer. And I think you're irresponsible if you don't jump on it, because you're getting paid.

You know, imagine you could work five times more effectively today — forget about AI — than you do. And everyone around you is working at that level of effectiveness. But you're like, I don't like to do it like that. So I'm going to do one fifth of what you guys can do.

And that's your choice. And maybe you have, like, a moral objection to how they work, you know, based on some random thing. I don't know.

Adam: I mean, I think that there's also — the counter example would be you look at things like supercars, in the sense that you look at a Bugatti or a Ferrari, and they will intentionally produce a limited number of vehicles on an annual basis, charging a premium for their brand and exquisite craftsmanship. Do you think that there's a place for that in the world of software?

Chad: No, I don't. Well, there's a bunch of automation that actually happens in the creation of supercars.

Adam: Oh, completely. I'm not talking about their software.

Chad: You know, I don't even mean software. I just mean the consumer of the car doesn't care that it was handcrafted. They might care that some pieces are.

You know, you buy a Ferrari, there's some handcrafted stuff inside. These consumers, they care about the actual performance and usefulness of the car for what it's supposed to be, which I presume is —

Heidi: No, it's a Veblen good.

Chad: What's that?

Heidi: It's a Veblen good. It's valuable because it's expensive.

Chad: Yes, yeah. But I mean, these cars also — people who are race car drivers use them because they're good at that too. I don't have these cars and I don't care about them.

But I don't think software works this way, unless what we're assuming is that software is just going to be auto-generated junk where no one even cares what the user interface is as long as it works — then yes, there would be a place for an artisan.

But I don't think the separation of artisanal software versus mass-produced software is whether or not it was AI-generated. It's still about the human and how they direct the process, apply taste, apply architecture, etc.

And then the one thing I don't want to forget to say is you're talking about people moving to the cloud and whether or not they got left behind, and a bunch of people didn't, because organization moves so slowly.

Now I think it's very different, because on one end, the incentives to move to the cloud, there were competing forces, especially in large enterprises. A massive competing force is just the gravity of data and critical systems and moving things. So the default is no.

When you move, you save money, you have a more modern environment, you can go faster, blah, blah, blah. We know this, right? The incentives now are just — there isn't one, actually, other than fear, to avoid this new wave. It's just fear.

Because in terms of what we are supposed to be doing when we create software, if we can do it a thousand times faster, then we must do it a thousand times faster. Otherwise, it just doesn't make any sense. Like, there is not an excuse.

And I see all your faces like, surely that's not necessarily true. And I might be wrong, but, you know.

Adam: I don't think that it's wrong. I think that where some of our hesitation or some of our pushback would be — I think is a topic that we've talked a lot about — is the fact that there's a certain aspect of knowing what good looks like that a senior engineer brings to the experience of working with AI that is difficult to clearly articulate and clearly teach.

And I think that that's where things start to kind of get complicated, right? Where this idea of moving a thousand times faster, if you have no concept of taste, it's like bad automation, right? In the sense that, like, yes, I can make mistakes a thousand times faster.

And I think that one of the things you talk about is the user experience or the user interface, right? I think this is something that we talked about recently. Kim was talking — you know, living down in Mexico — that the explosion of AI has meant that a lot of the advertisements or signage, all of a sudden, like little bodegas, all look the same.

Because they're all asking AI to do it. And AI is choosing the same fonts, choosing the same kind of visualizations. And suddenly all of the personality has been homogenized.

And sure, maybe once it looks good, but then after that, you start to say, okay, now this just looks half-assed or crappy, right?

Chad: Yeah, that's what I was saying about artisanal software, too, that when you have tastes that you're applying, the difference can be very big. And as you say, senior developers, when you have architectural know-how, the difference can be really big.

And the Phoenix stuff, a big piece of what I'm trying to do — pre-AI, my move was to create these architectures where you have one system that has a very rigid, opinionated view of how systems should be put together. And for me, I usually do backend stuff and usually some things like web and internet-based. I won't be talking about, like, embedded systems in this context, but it's usually, like, a bunch of small REST services with some front end, probably with something like an HTTP3 or WebSocket connection so you can do something real time. And there's a path for doing CRUD that's just synchronous.

And then there's a path for mutations to flow through the system and allow their effects to ripple. And each of the services in this layered and broad architecture can and should be written in any language anyone wants, because when the services are heterogeneous, it makes them even harder to couple than if they were in the same language. You resist the temptation to do things like share libraries, and then you have these big gross monoliths that are just split into different runtimes.

And so I created this sort of an architectural environment where, you know, for the previous example, you could have a Haskell service and someone could throw it away and do it and go, and therefore you could have a Haskell service in an environment where normally you wouldn't. And you could try out — okay, we had, like, Elixir and Haskell and Rust when it was very early, and Go, JavaScript, just services of all types.

No one had to not try something because the system was too rigid or our culture was too rigid.

The reason that was okay is because everything was decoupled in a way that if someone screwed something up, it was small enough and contained enough that it could be replaced, meaning we could reason about it, we could see all the edges, we knew what the edges were, even the boundaries needed to be. And it was so small that you could look at it and understand it and just, you know, replace it at any point.

So, yeah, we were, like, very experimental and wild — like, seemingly irresponsible in some ways. But we gained the authority and the trust to do that by having a system that could withstand it.

Adam: Right. No, I mean, I think that the resilience aspect of it is unquestionably compelling.

I think where the kind of human aspect comes into play — in some ways, it's actually understanding this notion that we talk about, or the third loop, right? This idea of adoption, right? And who is adopting the output of whatever component or layer you're talking about? And what is the rate of change that they are capable, or that that system is capable, of adopting?

You know, if it's a human system, that rate of change may be lower than if it's a agentic system. And so how do you create the appropriate, you know, notion of feedback loops of that output to make sure that your pace for that layer is appropriate?

Chad: There are multiple reasons to want to capture the feedback, and one of them is to know that your pace is appropriate. And then there's just, like, does the thing do what you want it to do? I guess that one's first, right?

Adam: I mean, ideally, right?

Chad: Yeah. It's sort of funny, though, that we've had observability tools since long before that was a term that people even used — monitoring tools or whatever, logging. We've always had this.

And we've always incorporated the feedback from the systems back into the process of creating software and creating deployment environments. But we didn't do it necessarily explicitly. Maybe some of us didn't do it at all, right?

Heidi: That was my skeptical face. I'm like, have we, though?

Chad: Well, here's how I would say we have. It's pretty unglamorous. It keeps crashing. So the developers know it keeps crashing.

At least that's going to change something about how they're thinking about it, you know? But it's like that crude. And it has been that crude most of the time.

Now we have the ability to just hook. Like, if you say, okay, you've got a business requirement. With a business requirement comes a bunch of metrics that you also care about. You know, it could be like the cost to run the thing — speed or performance on requests that you're making, or whatever.

There's all sorts of, like, either technical or maybe broken-down business metrics that matter. These things are measurable, right?

So if you could just always measure all the things that are captured as requirements, and then make sure that they're being logged in a way that's usable so that it can be fed back into the beginning of the process, in this AI world, the generator could then say, well, it looks like this system is almost too expensive to run based on what we budgeted.

And so we probably need to rewrite it in Rust from Python because we can speed it up now. So let's go ahead and do that and deploy it. Oh, good, it's cheaper. We can remove some server instances now.

This sort of thing could be possible, and it could happen overnight. No one even knows that it's happening because the feedback flows through.

The other thing about feedback, though, is, like, some of the consumers are humans and they're looking at user interfaces. That's a little bit harder to flow automatically through. It's also something that shouldn't happen as frequently, and probably worth kind of looping back to the pace layer thing.

It is not so intuitive when you just think about pace layers as being architectural construct that the user interface is also a very, very slow-moving layer. And it has to be, right?

Heidi: Yeah. You cannot move user interface at AI speed and retain your customers, because they never use it that often — that much.

Chad: And then they can't find the record button in the podcast software when they get back to it again, right?

Heidi: Exactly. One of the things that I was thinking about when I was reading about this feedback and reliable output thing was promise theory. And the way that we keep coming back to: I promise that this tool will output this way.

I don't care what happens under the hood. It's a microservice. It's a — you know, I don't care what it even is. I just needed to have this kind of output.

And is that something that we can trust AI to do, or is it going to get ideas?

Chad: I think we cannot trust AI to do anything, but we can create constructs in which AI must deploy so that it will be checked.

So, you know, I mean, you can trust it too. I sort of started this journey into some of the Phoenix stuff because I was thinking about — here I go with Haskell again. If I were to just go three years ago in a ChatGPT and give it a definition of a Haskell function that I want, that is pure with no side effects by default, and it has a strong typing system — there's very little you can screw up.

And so if it compiles, I bet it works. That was my thought back then. And it was true, right? Because Haskell itself created these constraints, so it would be almost impossible to get it wrong unless it just completely hallucinated.

So what we need to do now with architectures is try to create those sorts of boundaries for our systems and for what the AI generates. So that, you know, ideally, we'd be generating literally just pure functions in Haskell. And that way, if it compiled at work, because that's how Haskell works.

But life isn't like that. And so what is the equivalent? Well, with Haskell, it's like very clear interfaces, which means input and output. Very clear, because the typing system is just really deep and powerful.

And then the side effect thing. Now, we very rarely write any software that doesn't have side effects, because you do things like write to databases.

But you can at least be very explicit in an architecture you create about what the side effects can be for a given component. And that can be part of the technical specification and part of the feedback that loops back through.

Like, well, this system's trying to do something it's not supposed to do. You know, it's not supposed to have this side effect, and it's now trying to have the side effect. Or even deploy it in such a way that that's impossible, you know, with, like, container technology and sandboxing or something, if you really want to be extreme.

But I think what you said is, like, really hitting the point hard of no one cares what language it's in except for the person that wrote it. The purpose of a service or a system or a component is what its inputs and outputs are and what its side effects are. That's it, right?

Heidi: But the purpose of a system is what it does.

Chad: Yes. But it is defined by the interfaces.

Adam: Well, we've talked about this before, too, that I love the way that you've articulated it in terms of the belief that just a few years from now, no engineer is going to be explicitly writing software, right?

Chad: Right.

Adam: The reason that we used to care was because humans were the ones who had to do the debugging. And so you chose a language that you weren't the only person who knew, ideally. So you weren't getting paged or called at 2 a.m. as the only person that knew how to read Haskell code.

And, you know, you made it so that it was something where different semantics of different languages were transparent as far as what they were doing for the engineers who were having to fix them. More often than not, that was a concern than the engineers who were doing the initial code development.

And now it's like, OK, well, you know, there's been a debate of, like, what is coding language? If we're asking agents to do this, does it matter? If you're good enough at defining what the constraints are and what the output is that's needed, and being able to kind of scope the input that you're going to provide, does it matter what language it's in?

Chad: Yeah, it does matter. But here's why. It matters because some languages are cheaper and faster to generate code in.

I was actually surprised to see — and it was done by a Ruby programmer, but a Ruby programmer who I really respect — did a study on this and determined that Ruby is actually the fastest, cheapest thing to generate. Because it is so succinct, it uses less tokens, there's all these conventions. So it's actually quite good.

The problem with Ruby, though — compare its performance to Rust or Haskell or Go — not good, right? So if you deploy stuff to production in Ruby, you're going to spend more running it.

And you also don't have, I don't know, type systems and things like this. Apparently, that didn't help, though, surprisingly. It was more expensive to even do Rust, which my experience is, Rust has been great with LLMs. But you care about what's fast and easy to generate, what's cheap to run, what's secure, what performs well in production.

And a story from this Wunderlist place where I was doing crazy stuff with the heterogeneous architecture. We launched a major rewrite — and sort of, like, it was actually another euthanization thing — but we launched this and we had something like 800 EC2 instances running on the day of the launch because it was very popular and the previous launch had been a disaster.

Over the course of six months, we replaced something like 60% of the system that had been in Ruby with Clojure, Scala, Go, and Rust. And we reduced the operating cost of the thing by something like, I don't know, 80%. And the system was the same.

The components we were replacing were like cells that were being regenerated by us very, very slowly at human speed. But we also had an architecture that made it not that hard to do this.

The system was the same. The users didn't care, didn't notice. We saved hundreds of thousands of dollars doing this.

So, like, this is why programming languages matter today, but not in the ways that they used to.

So, yeah, I was at RubyConf last week or a week before on stage talking to all these Ruby people in which I said, you should probably not use Ruby anymore. It's irresponsible because it's too expensive to run in production. And I still sort of believe that unless they prove me wrong.

Adam: Do you ever think about it in the context of, you know, this iterative development and being able to have architecture that is kind of understandable, tested and validated, that you would then start to swap out components? Would that be something like an approach that you would recommend or that you would condone? Or is it you saying, oh no, you should start with the most efficient languages first?

Chad: Oh, yeah. I don't think it matters anymore. I think you should start with what you're comfortable with. And if you feel like you're going to be reading code, you should choose a language that you love that's good for humans to read.

Because it's so easy to change these decisions now. So as long as you're flexible to take advantage of how easy it is to make this change and the architecture is set up to do it, the only thing I would say is, if I was going for something like this, I wouldn't deploy a system all in the same language anymore.

Heidi: Because each language has strengths and sometimes you need, like, actual mathematical handling.

Chad: Yes, and I also want to make sure my system doesn't depend on everything being in the same language. You know, if you've got one thing in a JVM and one thing like Go or Rust or something, they won't talk to each other through, like, a common serialization format of the VM, for example. So you won't have that coupling. You have to build how they talk to each other.

Heidi: Is that almost a security factor? Like, okay, I really want you to be separated so you cannot speak the same language?

Chad: Yes.

Heidi: Okay, that's interesting.

Chad: Well, it's security, and then it's everything about decoupling. But security is a big part of it, right?

Adam: Well, it's like, all of a sudden, if you have them all in the same language, even if it's inadvertent, you end up in the situation where you could have dependencies that span across services or domains that you didn't even know about or plan for.

Chad: Exactly, yeah. So it's always about dependencies and knowing what they are and removing them when they don't need to be there so that everything is as decoupled as makes sense for the given environment.

Also, from a security perspective, this idea of disposable software is very similar to immutable infrastructure, where there was a time we liked to have servers running for years. If you could do that with all the patches up to date, that's not good.

A Linux server sitting on the internet for years is almost definitely hacked somehow. It's got something going on.

So how long is it okay to have a Linux server running on the internet without taking it down or updating or whatever? Maybe an hour is a good answer to that. I don't know. There was a time that sounded crazy.

But that was another thing we did back 15 years ago or whatever, before this was normal, is to do this immutable infrastructure thing where the services or the servers were like cells that were being regenerated as well. And we started to get this feeling like if the servers are too old, then this is a brittle part of the system and it might be unhealthy. So the healthy thing to do is to always have them cycling. Because it means you always have a fresh image, you know no one's tampered with it unless they got into the image or whatever.

And I think software can be the same way, that the longer it sits around, the scarier it is, the more risky it is to touch it. So if we can make it so that we can just have it always be fresh within reason, we have a better sort of steady state of constant change that's easier to progress forward with.

Adam: Yeah, I think that the crucial, you know, kind of caveat to that is exactly what you've stated in some of your posts, is that you need to make sure that you have an underlying architecture that is capable of supporting that from a resilience perspective, right? So I guess on that point, though, how do you think about that notion of architectural design and scoping in the context of, I'm asking AI to build a system for me?

When you're starting on some of these systems or in your, you know, kind of exploration, have you ever tried to work with an AI to build the architecture? Or is it something where you're doing that and then, you know, you're kind of just saying, oh, I'm going to then prompt the LLM with the component or the kind of boundary scope that I have determined for this piece?

Chad: Yeah, I work with the AI to build the architecture, but I am very opinionated. So, you know, I could almost say I do the architecture and leave AI out of it, to answer the question. But the truth is, I don't leave AI out of much that I work on these days. And I like it to fill in the gaps because it's going to do a better job than me by myself, if it's me and the AI, usually.

Adam: So you use it for interrogation?

Chad: And even just coming up with ideas. I like to put the broad sort of thing forward and then have it give me ideas because it knows more about the state of the art than I do, obviously. Whatever last scraping happened, I can find the latest thing. Now I can search the web.

And so I think it would be silly for me to think that I know how to make the perfect architecture. What I know is the constraints and the sort of direction that I want, and it's all about decoupling. And then I let the AIs help me figure that out.

Now, one thing I'm doing is I have a reference implementation of the Phoenix Architecture that's in my GitHub, Chad slash Phoenix, pretty easy to remember. And there I have built a really intricate system that is literally a codification of everything I'm writing in this blog series and the book, testing out some ideas. It doesn't work very well yet. Parts of it do.

But the thing that I've noticed is, like, you know, one of the common objections when I start talking about throwing away — like, the code doesn't matter, the system is the asset, the code is just the implementation, the liability, you can replace it all the time — is, like, the usual question about, well, if you have fully specified the system such that it can be safely regenerated, you're therefore creating a programming language for the specs, you know, or you're using a language that's not as good as code. So WTF, you know.

My answer to that is, I don't have to write all the specs either. In the same way that I don't have to create the architecture down to the detail, I can work with AIs to figure out how to make the best specs.

In a legacy system, part of it can be from hooking in the observability stuff and getting the logs and seeing what's actually happening. It could then say, I think this is the specs for the system. Are these good? No, this actually isn't good. It's not behaving the way I want, you know.

I think of every layer of code generation as being this potentially additive set of processes that can run that are both human and computer, and the computer stuff might be AI. Or imagine you have, like, a thousand experts in different fields of study that can scrutinize every specification you write, and also understands how systems work and how user behavior works given performance on the internet. You know, like the old e-commerce stuff. Like we have all these sort of conventional ideas about how fast pages need to load so people don't abandon their carts.

But systems know all this stuff. We don't have to know it anymore. So we can create specifications that are kind of dumb and allow them to be evolved over time with the systems helping us, with the feedback from production, editing the specs, and then just regenerating the code to meet the specs over time.

And it's a living thing. So it always will be anyway, even if we didn't have AI, you know.

Kim Harrison: So I have a question. We've talked a lot about why we need people with experience and, you know, tastemakers, people who understand what good looks like.

What would you say to young people who are — either they're young in years or young in experience — but becoming new baby software developers who want to enter this field? How should they be thinking about this? What should they learn and focus on?

Chad: Yeah, it's such a good question and such a hard one to answer, especially as someone who is not young now. You know, if I go back to when I started in the '90s, I had a mentor that gave me advice that was not specifically about how I would write my code at the time. It was system-level stuff.

It was like, well, what this specific person told me was learn Perl, as one thing, learn Novell NetWare, because specifically it had this really amazing directory service, which was closely related to LDAP and shaped a whole bunch of things that happened afterward, and then learn another programming language that's not like Perl.

And he was trying to prepare me to be a system administrator, but the advice really drove me into my software career perfectly.

And none of it was about how to actually write code. It was about the sort of major conceptual areas that he thought I should be thinking about. I always thought about it as, like, the, you know, World of Warcraft-like skill tree kind of idea, where you've got three different areas and you're advancing up, and you get to decide how fast you go up on each and, you know, diversify your build or whatever. I think it's still exactly the same, actually. Like, I would tell someone, I don't know the exact things, but I would say, like, you have to just fully embrace AI, because that's how you write software now, just like when I got excited about dynamic languages in the year 2000 and everyone thought that was crazy. It was dangerous at the time and it was risky and seemed irresponsible, but we did stuff around it. But I think it's like that. Like, before that, maybe you would have been writing C and you would have been thinking about pointers and memory.

And then if you started writing code in 2000, you didn't have to do all that. Still the same job, right? So, you know, it's like, you use software. Probably there's something about functional programming — like, actually spend some time with a great functional programming language as an area of study, because I think that shapes how people think about systems. And then, like, I don't know, I would want people to do things like run massive load tests and crash the software they're writing as part of how they learn, or put it in production and be on call, you know.

That's the best way. Like, you will go from junior programmer to senior programmer by being on call when things have crashed and you're up for three nights in a row and no one has slept and the system has been down the whole time and there's one person who really, really knows how to do it, and then there's also you. I don't think the story is any different than it ever has been, except you can go faster. I don't know if any of you are on call now. I haven't been in years.

I kind of am desperate to get in a situation where I am, because I would feel like a superhero now, because I still have the things that I know from experience in the past, but I don't need to know all the details, and I don't need to remember, like, what are the switches for tcpdump on this version of Linux, you know, or BSD versus Linux or whatever. But yeah, you know, I think it's no different. I guess that's the thing. It's no different, but you just spend less time in minutia and probably almost zero time talking about how to do, like, object-oriented design well, but more talking about maybe systems the way you normally would have been talking about objects, systems the way you normally would have been talking about code. We now have this level of malleability and productivity that maybe that's what programming looks like, is actually doing in real time and seeing what the ramifications are of the different choices you make.

Heidi: So we just keep climbing up the abstraction stack.

Chad: Yeah, yeah, as we always have been, you know. And it is a little bit of a cop-out that I have made to say that we're just raised level abstraction again, like we've got a JVM now, object orientation. The only reason it's a bit too far is because this is not deterministic, whereas all these other things in the past have been.

But the truth is, like, massively outsourced, low-cost software development is also non-deterministic. Slow and expensive.

Adam: Well, I mean, I think that, you know, this is the thing that, you know, we were talking with Charity recently. And, you know, she pointed about that a lot of software developers think that their code is deterministic until it goes into production.

Chad: And then production tells you what your code actually does.

Adam: Yeah, exactly.

Well, I think that, you know, this has been a phenomenal conversation. Really enjoyed it.

Chad: Me too.

Adam: Chad, thank you so much for joining us. Kim, I think you have one final question.

Kim: Yes. We like to ask everybody, and we're especially curious to hear what you might have to say. Who else should we be talking to?

Chad: Well, you know, my default would be Charity, but that's how I got in touch with you. There is a person named Bryan Helmkamp who's building something called Fabro. It's fabro.sh.

He was the creator of Code Climate, the code quality service. He was the founder of that. Now he's working on Fabro.

And Fabro is a remarkable system for orchestrating work in this new world with huge ramifications. I just talked to him yesterday, and as he's building it out, I think the surface area that he's uncovering that needs to change and will change with agentic development all the way from developed to managed deployed services is really fascinating. So I would say Bryan Helmkamp's the person.

Adam: Awesome.

Heidi: Thank you so much. We really appreciated this talk.

And when is your book coming? Do we know yet? Are you still in the throes of writing?

Chad: I just got an email right before this started saying, "Let's prepare for early access release." So I think probably within the next few weeks, you can work at the first few chapters.

Heidi: Ooh, exciting. Well, we'd love a copy.

Chad: You will absolutely get copies, yes.

Heidi: Excellent. Well, thank you so much, Chad.

Chad: Thank you. This was fun.