Ep. #33, Retiring Ingress NGINX with James Strong & Marco Ebert

John McBride: Welcome back, everybody, to the Open Source Ready podcast. I'm here with Brian. How are you doing?

Brian Douglas: I'm doing fantastic. It's almost like this is our third call today, John. So here we go.

John: Many, many things happening at your new company. Right? The Stealth. Is that what it was?

Brian: Yeah, I'm, I'm co-founder at Stealth, which I also saw you're also co-founder at Stealth, so very excited to hear more about that.

John: What a coincidence.

Brian: Yeah.

John: Oh, my goodness. I know. But we didn't show up to talk about Stealth today. We're here to talk today with James and Marco. Welcome to the podcast, y'all.

James Strong: Hey, John and Brian. Excited to be here. Excited to talk about Ingress NGINX and our upcoming KubeCon.

Marco Ebert: Yeah, same, same. Very nice to be here and I'm happy to share some insights and experiences.

John: 100%. It's a pleasure. Well, why don't you give our audience a quick introduction to who you are, your claim to fame, and maybe tell us a little bit about what Ingress NGINX is and, real spicy, the impact that it had on the ecosystem.

James: Those are all fun questions. It's always fun talking about yourself. I guess I'll go first since I'm talking. I'm a principal solutions architect at Isovalent, now With Cisco, so I help customers manage and install Cilium, Tetragon, all of those fun security products.

I've been maintaining Ingress NGINX for about five years now. Been doing mostly the project management, the release management, getting things out. A claim to fame, I do have an O'Reilly book out on Kubernetes and networking.

John: Nice.

James: So that's pretty nice.

John: Yeah. What about you, Marco?

Marco: Yeah, my name is Marco. I'm working at Giant Swarm, we are offering, basically it's managed Kubernetes but you know that's not paying nowadays so it's a lot more. I'm a site reliability engineer there and with Ingress NGINX since I think November 2023.

Back then I was in a more network related team at my company and just a few months later changed to a whole different topic. So since then I'm doing this whole project Ingress NGINX just in my free time and because I love it and because I want to do it and not because it's somehow work related.

James: Well Marco, you're being really coy there. I want to give a shout out to Giant Swarm for offering his time while he was on that project and helping us support it. That's doing what you should do.

If your company or your products rely on an open source product, you should cut up some of your time and give developers' time to help those open source projects.

So I wanted to say thank you to Giant Swarm and to Marco for his time.

Marco: Thank you.

John: Yeah, 100%. That is something that we've been really ringing the gong on for the Open Source Ready podcast. But to kind of level set with the audience, what is Ingress NGINX even just giving some numbers, like probably the number of clusters this thing is installed on would be helpful.

James: So with Kubernetes, the most basic networking like Pod-to-Pod networking and cluster networking is all internal to the cluster. So we needed a way to expose the application traffic running inside the cluster to external clients outside the cluster. And one of the original ways to do that is an Ingress controller.

So the Ingress controller is essentially another component of software in Kubernetes because we know Kubernetes is a collection of different pieces of software, different pieces of controllers. And this allows us to expose the applications running inside of our cluster through load balancers and other networking means to get clients into the cluster, client traffic into the cluster.

Again, Ingress is a spec inside of Kubernetes. The Ingress NGINX is a controller that implements that specification.

So there are a lot of them out there. There's lots that are supported. I said I work for Isovalent. They also have an Ingress controller through Cilium. There's several other open source ones out there. There's a whole page. So that's essentially what an Ingress controller does for Kubernetes and for networking.

And as far as the numbers are concerned, this is one of the things that's difficult with open source software is you don't know who's running your software or what they're using it for, what they're doing with it.

We found last year, working with some of the folks on the Security Response Council, that 50% of clusters reported using Ingress NGINX. And that was I think from Datadog, from a data set from them. So that's a lot of clusters.

Marco: Yeah, you usually only get these insights and this data when actually shit hits the fan. So normally when everything is just working, you don't get any details. But often who is using it and where are people using it and how important is it to them? That only comes to the surface when they are actually having issues. And I mean that's already part of the sad story. Right?

James: Or when we break our release, right?

Marco: Right. Exactly.

John: When people come out of the woodwork to be like, "what is going on? Why did this break?"

Marco: Yeah, exactly.

John: Well, we're kind of burying the lead here. Recently there was an announcement, I guess it's kind of been a long time coming, but this March, so more or less in like a month, Ingress NGINX is being retired and in replace of newer, better technologies.

But I think the vibe kind of around the community is mixed. Can you speak to why this is being retired and where that leaves people? What options do they have these days now?

James: So as you can probably imagine, knowing those numbers and having us talk about this now that, we've had this conversation for a long time. When I started taking this over about five years ago, there was maybe four or five maintainers, no steady stream of contributors. And we really tried to hit the community meetings hard, every KubeCon talk that we gave.

So I think I've given about 10 KubeCon talks because I do two every year for the maintainer talk for Ingress NGINX. And we talked about needing contributors and needing folks to help stabilize the project. There was a point in time, I think, two years into my tenure, that we stopped accepting feature requests and PRs because we needed to stabilize the release process because it was all manual.

So it was a project that got really popular very quickly. It was included in documentation, it was the "go to," it was really easy to install and run. And so it exploded in popularity very quickly. Unfortunately, the amount of contributors, sustaining contributors, didn't follow with that level of usage. And so it became very difficult to continue to maintain the project.

Like I know before Marco and others came on to help, I was spending a lot of time just trying to maintain things. Like we had one maintainer who's not with us anymore because he burnt out. He was working on Saturdays and Sundays with a full time job and he just couldn't do it anymore. Like it was very difficult for him and it was a difficult decision. Like we talked about it for six months about him stepping away from the project.

So what I really think was the tipping point, and Marco can talk a little bit about this too because he did most of the work on this one, was when we had the security researchers from Wiz who find the IngressNightmare CVE that they called. And so it was really a wake up call that we really needed to do something more proactive than what we were doing. Like, you know, I was sending out tweets, dev emails we had the talks, like everything I could think of from a community perspective we were doing the outreach for. And it just really wasn't getting the traction that it needed.

And the thing about this project being inside the Kubernetes project is that we get a lot of support from the Kubernetes project, which is great. But it started to become a burden. Like we had CVEs that were open for longer than they should have because we just didn't have folks who had the time to fix them. And it became very apparent after the Ingress Nightmare. And I know we just released a couple more CVEs, some fixes for those.

So it became very apparent to the Kubernetes Org, to SIG Network, to the security team that it was time to shutter the project in favor of, like you said, the newer ways of doing things. That's Gateway API. So before we start talking more about Gateway API and this stuff, I'll let Marco talk a little bit more about his experience with the Security Research Council, working with Wiz and all of that, and just understanding the kind of toll that that takes.

Marco: I think I personally learned a lot about how you can actually run a project or which kind of also architectural decisions you shouldn't make. And maybe one should note, and that also adds on your initial question about what's Ingress NGINX, I mean there's NGINX in it. And NGINX basically I'd say runs like everywhere.

So a lot of people know it as a good web server. You can do a lot of stuff and that's all we get. Because you can do a lot of stuff. You will have people that approach you with their pull requests and feature requests, like there's this and that, configuration directors and NGINX and we would like to make use of that. Could you maybe implement that in your Ingress controller?

John: Right, right.

Marco: And since most developers, and I'm not going to blame anyone, do not get me wrong, but I think most developers, when they approach you with their feature request, they're really happy to implement it. But most of the time, or often I'd say they are not really thinking about how could someone exploit that if you're an evil person. And with that comes a lot of technical depth I think. And that's currently hitting us.

And that in the end also led to a lot of these security issues and also a lack of resources because at some point just gets really hard to maintain this monster with a lot of features, with a lot of "I'd like to have this and that and this and that."

Yeah, we can do this, we can do this, just open a PR, do that and then people leave, they contribute and then they leave and then you're kind of left alone with their feature and then people approach you with, "oh yeah, this and that feature, I'm using it, but it's not working," and there's a security flaw there.

James: That's why I use the word consistent maintainer. Like that's the key thing right there. Like we would have folks come in and implement a feature or two and then bounce once we release it and it's on the maintainers to maintain after that.

Marco: So because you were interested in numbers, I think that's really just an estimation in the end. Ingress API as a written down definition, maybe that's like 5% or even less of the feature set of Ingress NGINX. All the rest, all the other features are mostly implemented in a more or less vague way in annotations, not type safe, not really validated. And so you can do a lot, but you can also break a lot.

And I'd say in the end this just makes it really hard to maintain a project which is both loved from the user side, a lot of people use it because it's flexible and under the hood, still using NGINX, which is a high performance web server. But on the other hand, it might expose a lot of security risk if used in more or less a wrong way.

James: Yeah.

That was one of the biggest issues and biggest wake up calls for us. If you allow users to run arbitrary code, you're going to have a bad day. There's a whole adage: if you can solve a problem with Regex, you have two problems. We had compounding issues.

John: Yeah. I remember when I saw the IngressNightmare CVE, which I think at one point I saw something that was like 50% of all cloud environments with this installed have, you know, it was like crazy numbers, but not to double click on the numbers or anything.

What really impacted me about it was this is the kind of thing that would keep me up at night thinking about. I maintain a library called COBRA in Go, which gets the benefit of just being a library, not really a service you actually have to run. But even the like, little things would keep me up. I'm like, oh my gosh, I'm gonna like accidentally pwn the GO ecosystem or something and I have to be so careful.

So I can only imagine what that impact from like the human perspective had on you both, you know, and again, this like, I mean, correct me if I'm wrong, but this wasn't like a main part of your job or anything. This was like on your nights and weekends, right?

James: No, I did not get paid for those five years for doing it. It was complete freelance open source work.

John: Yeah. So like from a, if you're willing to answer this, from a personal perspective, because I deeply believe that open source is a human thing. It's obviously about technology, but there's so much about the communication and building the organizations and these relationships and talking to the people that you know and you can trust. Like, from a human perspective, what was that impact on you?

James: I'll be honest. It was, I think it was in the middle of December, Marco, when we were working through that issue. I had to step away. It was understanding the impact of the issue and then having to work with it and I just couldn't do it.

It was a turning point for me at that point because unfortunately one of the issues with this is that I just, I had no momentum to move into the InGate in our gateway API implementation. Like I just felt like we're just shifting the issue from one to the next and it was going to be the same and I just couldn't do it.

After this March deprecation, I think I'm gonna give myself a couple months before I go back into doing any more open source, especially with the Kubernetes environment. I just need the break. It's unfortunate, but, your mental health is more important than anything.

Between that CVE last year and the couple that we just released, it just weighs on you. I know that we only gave folks four months. We tried to push for a year, but that was just not in the cards. So we had to give folks four months. And I know that it made a lot of folks' day jobs very difficult.

But on the flip side, we didn't have the support that we needed. So you've got to be able to balance both of those things. And yeah, it was a difficult call. If you go and watch the announcement in, I think November, when we did that at KubeCon I almost had to walk off the stage. It was very difficult to do that. I don't know. Marco, how'd you feel?

Marco: Yeah, so about the IngressNightmare, I think it taught me a lot because in the very beginning we were like, frustrated or we didn't believe it and we were like, "ah, ah, no, that cannot be true. And we cannot reproduce it."

So sometimes you got that, like someone is telling you your solution, the piece of code you're working on, it's not working. And you're like, "ah no, that's your fault. That's your environment," you know.

John: Right.

Marco: And then it started to kick in and to actually, okay, it seems to be an issue and we need to fix that now. And we didn't have a lot of support for that, neither on a people level, communication level. We slowly got there. But overall it was to me, the first time battling such an issue in this project. And so there was a lot of chaos with communication.

But in the end, I can tell we learned a lot. And nowadays I would, and we did in the past, just I think last month we did a lot different and it was a lot easier because there were a lot of lessons we learned from IngressNightmare and comparing what we just fixed and what we published last month to IngressNightmare, I think there's really, really a lot of stuff we learned.

At least I personally, and I hope James, you too, can somehow be happy and proud of this because even though the project is being retired, just from a communication and also like incident handling point of view, we definitely improved and hopefully can use that in the future somewhere else. Even if it's just for a day to day job.

James: Yeah, no, I think comparing the two, and even when we were doing the announcement in November, like SIG Network leadership was there to support us. SRC was there supporters, like I think that whole KubeCon everybody was like, "are you okay? How are you doing?"

John: I think we talked at that KubeCon.

James: Yeah, yeah we did, yeah, yeah. And the maintainers were very supportive. Like everyone at the Maintainer Summit, like they knew and they understood. And nobody came up to us and was like upset or mad or anything like that. So never really any direct interactions, negative interactions.

Like of course people were upset and concerned but nobody really was like, you know why did you do this? Like you know, pointing the finger at us. Like it was a real eye opener where it was just like guys, we're doing this on our free time and we're doing it as best as we can.

John: Right.

Marco: Yeah.

James: So you know, it was a mixed bag. A lot of emotions at that KubeCon and the prior one when IngressNightmare came out.

Marco: I think looking back to that it feels like there's just a great failure culture overall.

People are not upset, they are understanding. They might be sad about it but still they try to understand you. And that's really something one has to appreciate because I wouldn't say it's given. It's nothing you can rely on and you can assume to be there.

John: Right. It's one of the best parts of the cloud native Kubernetes ecosystem to me is the great people in it. Is there a world like, is there like a weird multiverse reality where this got more support, or companies jumped in with engineering resources and Kubernetes Ingress NGINX lives on? Or was the path always to adopt these better APIs or these more, you know, cloud provider knit APIs together.

James: So that's a really good question John, because that kind of leads us into what we're attempting to do. So SIG Network has come out with a new set of APIs, quote unquote Ingress v2. It's Gateway API. So a lot of the things that we talked about, the extra functionalities like in Ingress NGINX that we provided via annotations, we've actually started baking into the Gateway API spec so that they can be tested, they can be, you know, better handled. They're not, you know, just quoting strings. It's not arbitrary code, it's actually in the spec.

So it's a much better version of Ingress and it does a much better job of breaking out responsibilities because if you took over the Ingress controller, you had everything, you had the keys of the kingdom. So Gateways helped break that up from a role's responsibility.

And we had the idea as maintainers that we would work on a Gateway API implementation based on NGINX to help migrate folks over from Ingress NGINX to Gateway API. And re: the previous conversation, it became apparent to me that I couldn't do it. Like it was difficult, you know, it's hard to admit failure to yourself.

And it was difficult when I made that realization that I just, I couldn't maintain another project. And there were already several other projects. I think Kgateway came out at one or two cons ago. Envoy Gateway is taking over. Yeah, it seems like a lot of folks are using Envoy Gateway. I'm hearing a lot of chatter about that. And you know, Cilium has their Gateway API implementation.

So there was a lot of headwinds already for us to do another Gateway API implementation and NGINX, for whatever reason was getting difficulties getting traction as the backend proxy. So we had a lot of things going against us and then we had, like I said, all of the emotional baggage, the security baggage of the issues with Ingress NGINX. So we weren't able to get that off the ground with InGate.

So that was part of the idea. So the idea was to always deprecate Ingress NGINX and migrate folks, but we were going to do it on a much longer timeline and get folks used to and up to speed and help the Gateway API spec. It didn't come to fruition. It was difficult. I think I had three or four conversations with the Gateway maintainers working on it myself.

And again, like I said at the top of the conversation, I'm not a day to day developer. It was difficult for me to just get started on using it with controller runtime. So maybe if we'd had more support, if there were more folks that were able to help on either instance, maybe. But there's a lot of "what ifs" in the world, so can't really dwell on that. That's where we are.

Marco: Yeah. So I think overall migrating from Ingress NGINX to something that implements Gateway API is less of, okay, we are migrating from this one product with this feature set to something else. And now we need to compare features and check what's in there and what's not. And because I think Gateway API from the ground is released hopefully in the future, also a lot better than Ingress API.

And so ideally with Gateway API there is kind of a feature or some path, some migration path, some replacement for what you did with annotations in Ingress NGINX before. So it's more like we're migrating from Ingress NGINX to the API to the point where it maybe doesn't make a difference which product you use under the hood for implementing this API.

And that's what I'm really happy about, that with Gateway API there is a lot of discussion about new API versions, about new features, and this has not happened in the past with Ingress API, I think we never really had a chance and it was just finalized with what it is and what it is compared to Ingress NGINX, I mean that's really just the minimum feature set done.

James: Yeah, I think I had a KubeCon talk with Rob Scott, one of the Gateway API maintainers, where we were comparing and looking at, I think this was at the time, Gateway API 1.3 and comparing the feature set that's in Ingress NGINX to what's in the Gateway API spec. It's out there somewhere on the KubeCon talk. I forget which one it was. Like I said, I did 10 of them.

But this is the time, like, I mean, November was the time that folks need to start seriously looking at an alternative because come March, the SRC will not be doing bug bounties.

We will not be triaging issues. There'll be no new releases. Folks, after March, will be running a risk of having a vulnerable cluster. And that's something that you need to, as an SRE, a platform engineer, a DevOps engineer, you run the risk of being compromised. It's as simple as that. There's no more plain speech to say: after March you shouldn't be running Ingress NGINX.

Brian: It's probably fair to say that at the release of this recording, it will be March. So if you're listening to this, please go update or go switch rather. There's no updates.

James: Yeah. And we all know the speed of enterprise. So folks should be making this and putting in this planning. They should be doing the testing, looking at the feature set that they run right now with Ingress NGINX and see what's comparable in any of the Gateway API specs that are out there or even the supported Ingresses that are out there currently.

It's one of those things where it's going to take time and it's going to be painful, but it has to be done. And if you find that you're using a lot of features that aren't in Gateway API or in an implementation, I think that's a good indication that you should probably come to the Gateway API community meetings, come to the community meetings of an implementation and help offer support, especially if you're a large corporation that has the resources to be able to do that.

John: Yeah, we had Davanum "Dims" Srinivas on a couple months ago, but yeah, he said the same thing. You know, if you want to get involved and start contributing, showing up to the SIG meetings even if it's something you're remotely interested in it'll be useful.

So I want to wind down this part of the conversation and move us to Reads. But before I do that, I just want to say a huge thank you to both you to your time and dedication into the project. I know personally it was impactful at VMware. I know lots and lots of people that were using it and was pretty incredible. So thank you both so so much.

James: And we gave it five more years, so I'm happy about that.

John: And sometimes that's all you can do. You did your best, that's for sure.

James: A tour of duty.

Marco: Yep. Yep.

John: Well, with that being said, James and Marco, are you ready to read?

Marco: Sure.

James: Yeah.

John: Well, I'll start us off with a couple of Reads. This is a really good one, especially for this conversation. It was, "I love the work of the ArchWiki Maintainers." For those who don't know, if you're like deep into Linux, whatever, there's probably an ArchWiki article for it. If it's some crazy boot thing, if it's anything that has to do with the system or the way that things work, not even just Arch, but it's a great, great Linux resource.

I've been knee deep in a bunch of this Linux stuff recently, so serendipitous for this conversation because those maintainers are also putting in a lot of time and energy for a free and open source project. Brian, when was the last time you were on the ArchWiki?

Brian: I'll be honest, not very often. If I'm there I probably took the wrong turn because, yeah, definitely not running Arch.

John: That's fair. James, Marco, what about you?

James: I showed you my networking stack beforehand. I keep it simple, stupid. I try not to do too many fancy things outside of work. I just run Ubuntu on my network stack, on my server. Yeah, if I'm on the ArchWiki, I know I'm deep into something that I've made a mistake.

John: So true.

Marco: Yeah, I don't know. No, this time, it's already gone for me. So computers work and it's not Arch. Sorry. Let's say I've run the company James is now working for in my private network in the past. And damn, it was good, but also cost me a lot of time.

So, yeah, I'm all down to just simple network and whatnot. And I think running Arch, no, I can definitely spend that time a lot better.

John: Yeah.

James: I had a coworker a while back who just couldn't keep doing work because he had to keep recompiling something for, like, his wi-fi drivers because he was running Arch on his work laptop. And we had to have the conversation where it's like, it's interfering with work. You need to not use Arch at work.

John: Classic.

Marco: Yeah, yeah.

John: Should have been using Nix.

Marco: Oh, I was actually, I wanted, before we started the podcast, the recording, I was about to warn you because I updated macOS some days ago and it all of a sudden started to panic. And I never had a kernel panic in macOS ever before. And then it did three times and two of them were in a meeting and I was like, oh, will the computer stand the recording?

And at this moment I was really happy because for years I never got to the point where the computer just crashes because I'm not using Windows and Linux for work for years now. I'm happy with that. I totally understand people doing that. But at some point I just got tired of what James told. I think it's a bit like with maintaining an open source project, but the project is your computer.

John: Yeah.

Marco: And you're the only maintainer.

John: it's very true. It's very true.

Brian: Nowhere to send all your angry issues, except to yourself.

John: You need like a whole linear backlog just for your own things.

Marco: Yeah. Yeah. I mean, some people love it and I can totally understand it. At some point in my life, I was that guy. But I think life changes you. And then you just focus on other stuff and you might be interested in spending that time on, for me, for example, climbing. I don't know maybe there's a time where I'm a lot more into computers in my private time again.

James: Yeah, yeah.

I don't want to debug kernels. I'd rather go play rugby.

John: There you go. I clearly need to reprioritize my life then. But speaking of Apple's excellent quality of software, my other pick today was iOSCountdown.Win, a hilarious little website that is this like 114 day countdown for somebody, you know, pleading for Apple to fix the iOS keyboard.

I think this is probably like the fourth or fifth time I've brought this up on the podcast, how annoying the iOS keyboard has been to me. But I love a good countdown website. I don't know what it is about it, it just feels so, I don't know, exciting.

Ingress NGINX should have had a big scary countdown on the top of Kubernetes, you know, and could have been very fun. Very, very scary.

Marco: Yeah, we had some detection and if it runs in some development environment it shows you this countdown when you open the landing page of your product.

John: Oh wow.

James: Oh that'd be fun. Yeah, put a little telemetry in there.

John: Yeah.

Marco: Like if you call local host and the first thing you get is the countdown.

John: Yeah, just, "you need to get off of this." Yeah, exactly. It goes through the Ingress. That's funny.

James: So can you not run a different keyboard on iOS? I'm sorry, I'm an Android user.

Marco: Limited. I mean you can do it. But the base implementation probably is the same because Apple doesn't allow you to do stuff you can do with Android. But my first thought when I opened the link was oh damn, I'm not going to read that because currently I'm happy with iOS 26 and the keyboard.

Maybe it's because I'm using the German keyboard. I don't know how swift with English in the end. But I'm not going to read the website because maybe I start noticing what this guy is writing there and then I'm like, oh damn, he's right, he's right. Oh damn.

John: It's true. It is a bit of the curse of knowledge. Once you read the website--

Marco: It cannot be unseen.

John: Yeah, exactly. Once you start seeing the unseen, you never go back. But Brian, you also had a few picks. What have you got this week?

Brian: Yeah, I just had one pick and it was a tweet, post, whatever you call those social media posts on X. And it's funny because we went this entire conversation without talking about AI. So I apologize.

John: That's a win for us, man. So long.

Brian: Yeah I was scrolling through the Twitter and they mentioned YC. I've applied to YC once. And they used to have a question where it's like, " add a GitHub repo that you're most proud of," to kind of show if you're technical, which is like a fun thing you can put there.

They changed it to now, "add a coding agent session that you're most proud of." And I thought that was very unique because I was like, I'm not sure if I have, like, proud coding agent sessions. Like, honestly, I haven't even thought about my sessions in that frame of mind.

But I'm like, okay, well, YC is now asking you to basically show your, I guess, Amp threads. Or I guess you'd have to use tapes.dev to sort of get your sessions out there and upload them.

James: I've got one. I mean, I don't know if it's great, but on certain platforms on Android, there's about 12 buttons to turn off AI. So I asked, I think it was Claude at the time, I asked Claude to write an Android app that would turn off all the buttons for me, so it's just one button. It wrote it. I never tried to get it to run on my phone, but I think that would be fun, using AI to turn off AI.

Marco: Yeah, this sounds like AI cannibalism somehow. I don't know.

John: Right.

James: Oh, you know what? I should have looked at this before. There was one thing that came out, I think, in the past week about AI, and they set up a bunch of AIs to compete against each other with vending machines.

John: Oh, yeah.

James: And they eventually just started being cartels and monopolies. Like, the one AI would have a product that others didn't. So it spiked the costs and dropped the cost of the ones that the other ones were to undercut the others. So it was just wild.

Marco: Wow.

John: Yeah, I saw something like that. It was crazy.

Brian: Yeah. I just found the Reddit post for it. I'm dropping it in the links.

James: I just like how it just devolved into the worst of things that we have to deal with, like AI late-stage capitalism.

Brian: Yeah, that was like, years ago. Like back when we had just machine learning, but like the Tay bot and how that sort of devolved into the awkwardness. Yeah, we don't have to go to a whole rabbit hole, but obviously the Moltbook, that was also, like, got popular for a moment.

John: Yeah.

Brian: Which I think was debunked to be fake. Yeah. Still, we're living in interesting times.

James: Yeah. I mean AI is helpful. Like, I think I was using it before where I just. I couldn't figure out, I think it was jq, I couldn't figure out something with jq to pull it out. I threw it to Claude and he was just like, oh, hey, here's that.

So as an augment, I think AI is helpful. As a crutch, like if you're fully relying on it and you don't know what's happening underneath, that's a problem.

John: This is why I have AI write all of my regexes.

James: You know, we should probably throw a Claude or Gemini at Ingress NGINX and see what it does.

John: There you go. Haha.

James: That's the March release. Haha. Yeah, people would hate that.

John: Just let the bots maintain it. Right? That'll work.

Well, again, thank you both so much for coming on to tell your story, maintaining an important piece of software. We're going to close this one out. Remember, listeners, stay ready.