1. Library
  2. Podcasts
  3. To Be Continuous
  4. Ep. #1, Introducing Edith and Paul
To Be Continuous
45 MIN

Ep. #1, Introducing Edith and Paul

light mode
about the episode

In this first episode, Edith and Paul discuss why people are doing Continuous Delivery, and what the major benefits and barriers to practicing Continuous Delivery are. You may also learn not to ask Paul about his PhD thesis.

For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.

Making People Happier, Quicker
Incentives and Decentralized Authority
Feature Flags and Dark Launches
People, Process and Permission
Risk and the Cost of Moving Slowly
Iterating on Real Users
Prescriptive Workflows and Escape Hatches
On-Prem, SaaS and Gatekeepers
Typos, Theses and Breaking Gmail

transcript

Paul Biggar: Hi, I'm Paul Biggar, founder of CircleCI.

Edith Harbaugh: I'm Edith Harbaugh, CEO and co-founder at LaunchDarkly.

Paul: And you're listening to To Be Continuous, a podcast about continuous delivery and software development.

Edith: You can get in touch with us anytime at our Twitter handle, at continuouscast.

Paul: The show is brought to you by Heavybit. To learn more, visit Heavybit.com. And while you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.

Edith: In this first episode, we talk about why people are doing continuous delivery as well as its benefits and barriers. Also, why you should never ask to see Paul's PhD thesis.

Making People Happier, Quicker

Paul: Okay, Edith. What's your favorite thing about continuous delivery?

Edith: Just making people happier, quicker.

Paul: That is not where I was expecting you to go with that.

Edith: Well, I think it comes back to, I'm an engineer and part of an engineer is you want to see stuff out there. You wanna see people using your product, you wanna see them happy, you wanna hear if they're not happy. You wanna fix bugs, you wanna ship features.

Paul: So one of the really nice kind of happiness things that we saw was support tickets. So people would come in and they'd have a support request and we'd be able to ship a feature change or whatever it was to that, that went out like twenty minutes later. And you'd be able to say to them, "Oh, sorry, but that should be fixed now."

Edith: Yeah, I mean we do the same. We're a young company and if somebody complains we're like, "Oh, we'll fix that right away."

Paul: Nothing defuses a Twitter rant like, "Should be fixed now."

Edith: Yeah. I think part of that, though, is a consequence of us, quite frankly, being very small and not having as many customers yet. Like when I was at TripIt, we had ten million users.

Paul: Right. Well, I think that's probably a different kind of situation. But I presume they did continuous delivery.

Edith: We did a weekly release and then we would do patches.

Paul: Okay. So this is mobile software.

Edith: This is mobile, yes. We had a whole different release train for mobile, mainly around the Apple store.

Paul: Oh, right, right, right. Yeah, the great disruptor of continuous delivery.

Edith: Yeah, I'd say the number one thing that stops continuous delivery on mobile is just Apple. I think there's all this urge and want to be more reactive and mobile, and there's this huge "thou shalt not pass" sign that Cupertino push up, which is basically like, "Your app might be great, but it's not happening."

Paul: So I wrote this blog post, I think it was on PandoDaily or something, that talked about how Apple was responsible for this glitch, this Kindle glitch that went out. So Amazon released a new version of Kindle that I think deleted all of your ebooks or something like that. And they weren't able to turn it around super quick. They got a special exemption, they got turned around within a day.

In the end it was fine, but really, the problem and the reason that that was caused is that the lack of a continuous delivery process meant that they couldn't just ship the new code and have it turned off and then gradually turn it on and, you know, the sort of thing that we do on the web but that they weren't able to do in mobile.

Edith: Yeah, I think it's funny because mobile has this reputation of being so much faster and more modern. But in terms of delivery to end users, it's five, ten years behind and it's all Apple.

Yeah, it's all Apple. Like, we saw that so many times at TripIt, we would have a fatal bug out in the field. And we would have a patch ready. And Apple would say, "Well, you know, we gotta review this." And then we were a good customer. We were the number one travel app.

Paul: Developers can write a fix in five, ten minutes once they know the thing. I mean, it depends on the fix, but it's very often five or ten minutes. And any time that you add on for that is a thing where more customers notice, more customers are affected, more customers' workflows are disrupted, and it's just pure overhead.

Like, Apple is just introducing overhead, but also not having a continuous delivery process in your organization, releasing quarterly, having someone who needs to click the button, something along those lines, are all more things that can just add barriers to fixing customer things.

Edith: Yeah, so this goes back to something we were talking about before, about how you said everybody wants to ship stuff faster. And I said, no, there are a lot of people whose

Paul: whose job is shipping things slower?

Edith: Yeah. So I mean, Apple does not really care if you ship stuff faster. Like, Apple gets dinged if stuff is

Paul: It's broken or

Edith: Or bad. Or at least that's their viewpoint. But then they put up all these processes where you're like, "I have a broken build, just let me patch it."

Paul: So this is one of the great sort of falsehoods of software delivery, that trying to make sure that it's perfect is better.

Actually what happens is trying to make sure that it's perfect means that shit never gets shipped. And actually shipping things and iterating on it is what makes it better.

And startups understand this, the whole Lean movement was built around this idea. Agile is built around this idea, and everyone is pretending to be Lean, everyone's pretending to be Agile.

Edith: Well, it's kinda like you ask somebody, "Are you agile?" What are they gonna say? No.

Paul: Yeah, right, right. I tend to ask people, are they agile with a capital A? And that implies that they follow the full Agile process, and nobody follows the full Agile process.

Edith: Yeah, I think, so we'll go back on one of the major tenets of Agile, which is people over process. And I think one of the holdbacks of effective continuous delivery is there are people whose bias is to have stuff be perfect, as you said.

Paul: Right, right. And they like processes that enforce that. So I mean, Apple obviously likes a process that enforces that. Well, like a human that makes sure that, I don't know, no nudity or whatever's against the rules on the App Store. But I think they also have like business reasons for not doing it.

So what everyone would like to do is just ship a web view in a native container and then just like update it off the web. And Apple doesn't let you do that because they're trying to make you build native apps and they're trying to make it, you know, all in a thing that they control, where they control the ecosystem, where you can't just easily take an app and move it off to Android or whoever else could be in that ecosystem if Apple allowed that. So they have like a very strong business reason to keep people from having a really good continuous delivery workflow.

Edith: Yeah, and it all comes back to money. They want to monetize off the App Store. Apple wants money? No, really? They didn't give you that computer for free?

But if everything's off web views, they can't enforce their 30% cut.

Paul: Right. They can't control the ecosystem. You may as well ship it to Android then.

Edith: Well, not even that. It's like if you sell something through an app, they get thirty percent right off the bat.

Paul: Okay, right, right. And they can't enforce that.

Edith: If you sell it via a web view, but just by, say, you had a web view and you're checking out, there's no way for them to take the thirty percent cut.

Incentives and Decentralized Authority

Paul: It's funny how many different sort of... I tend to look at the world through incentives. It tends to be one of the lenses through I look through the world. And whatever incentive you get someone with, there tends to be a lot of incentives not to do continuous delivery.

Even though, if you're a developer or you're a product manager, doing continuous delivery is like the most awesome thing in the world. But there's a couple of anti-patterns that I see. And one of them is fear. People are just afraid that things will break.

And then the other one, I guess this is a kind of fear as well, but there's this anti-pattern in product management of like, "Why wasn't I shown this? Or why didn't someone check with me?" You also see this in ranters on Twitter.

And I think that if you have continuous delivery, if your things going out all the time, then there isn't always necessarily a human who can validate that, like, "Oh, we're doing it right," or at least if your process isn't set up in such a way to allow this to happen.

Edith: Yeah, I completely agree. I think a lot of continuous delivery is accepting kind of decentralized authority.

Paul: That is a really interesting concept.

Edith: So the whole idea of a release train that was a monolith was that you had these many layers of approval. And everyone had to be checked as well.

Paul: There's a human, there's a release manager who's validating that it's good enough at each time.

Edith: And even beyond that, then the marketing reviews it, then you have legal or timing right, and it's very hierarchical. Whereas continuous deliveries, anybody could push at any time. That's very decentralized.

Paul: Right. So continuous delivery is anyone can push at any time, but it's not necessarily everyone releases every single push or at any particular time. There's kind of two models that... there's kind of two related words that people get confused with. There's the continuous deployment model of every piece of code that you push is shipped, and then there's the continuous delivery model of every piece of code that you push is possible to be shipped.

Edith: Yeah, I think it's shipped versus shippable.

Paul: Yes, yes. And I think web versus mobile is a nice example of this. Like, certainly most of our customers, every version of their iOS app that they build on CircleCI is shippable. There's a binary that you can download to your phone, that you can upload to your over-the-air updater or whatever it is, that is actually shippable, and Apple is putting that barrier in you to prevent it being shipped.

Edith: Yeah. That goes back to the decentralized versus hierarchy.

Paul: Oh yeah, if anyone is hierarchical, it's Apple.

Edith: Well, and then it goes back to, who does this ultimately serve? Because then you have all these people with broken phones out in the real world.

Paul: Right, right. We probably shouldn't keep going on about Apple, but I think it's an interesting model that they tend not to get things too broken. It allows you to make sure that people aren't really violating the rules too much. So you can't sneak in viruses. I mean, there can still be viruses, but they're harder to sneak in, which you would get with continuous delivery or in particular a web view model of continuous delivery.

You can't sneak in things that are against the rules, gambling and porn and whatever else Apple doesn't want in its ecosystem. And the expense of that is innovation. I'm doing air quotes for people at home around innovation. But you just can't ship things as fast.

People who are working on your ecosystem can't ship things as fast. And you'll find that a lot of startups who are trying to do something super interesting that doesn't really require being on a mobile will do it on the web because they can just ship way, way faster.

Edith: Yeah, it's basically, to your point, perfection is at the cost of speed.

Or my old boss, Gregg Brockway, who was the TripIt co-founder, he said a perfect plan tomorrow is not as good as a good enough plan today.

And he said that to me because I was... I built this new feature, I didn't feel it was good enough to ship, and he gave me some really good advice. Which was, let's just ship it. If it's broken, we'll fix it.

Feature Flags and Dark Launches

Paul: I guess the obvious thing that relates to that then is feature flags, and to sort of point towards your company slightly. Edith, of course, the CEO of LaunchDarkly.

Edith: Thanks, Paul. Paul is the CEO of CircleCI.

Paul: And feature flags as a service, but feature flags, I think, were really the innovation that made continuous delivery properly possible. And the idea that you're just gonna ship code separate from shipping features.

Edith: Yeah, it goes back to shipped versus shippable. Like, you have everything all bundled up and kinda ready to go even out in the field, but hidden.

Paul: Right, right. There's a bunch of features that we have that are maybe eighty percent shipped, that we sent them out to customers and we determined that it just wasn't good enough for some reason. Maybe there's a software bug, maybe there's an edge case that we hadn't considered that actually comes up in production. But I don't think that we could realistically do continuous delivery if we weren't shipping things in their off state.

Edith: That's cool. Can you talk about an example of a feature like that?

Paul: So there's this thing that we're rolling at the moment, and it totally exists, and like anyone, any customer could use it if only they knew about it.

Edith: It's like the secret sauce at In-N-Out.

Paul: Exactly, exactly. But I mean, we're not telling them about it. But it's gonna be the super awesome feature, but it's a big feature. And the foundation is built and the foundation is like part of every single build. And if you know the super secret invocation, you can get it.

And so we're going to start by rolling that out to customers who actually need it, and we're going to validate it against the first five or ten customers, and a very standard sort of product management thing of validating the business case, validating the customer case, make sure that the thing that we're shipping is a thing that's really valuable to our customers before we announce it, before we tell 100% of our customers.

So maybe 10, 20% of our customers will have experienced it before we announce it. And actually a whole lot of customers will experience it without knowing that they experience it. And I think that's kind of the ideal way of shipping things.

Validate the product first, validate the technology first, make sure that it all works, and only ship it to customers once... or, well, only have the marketing version of the launch long after the code is validated in production.

Edith: Yeah, I mean we did the same thing at TripIt. So one of the biggest features I ever worked on was scraping people's inbox for emails. So TripIt basically takes emails, strips them of interesting things, and then gives you a beautiful itinerary. You used to have to forward stuff to plans at TripIt. What a lot of our customers were asking for was that we could just scan their email and get their emails out for them.

Sounds simple until you think about it. And you're like, well, actually this is a pretty massive undertaking. Like, you have to get the right emails. Like, people, we found, they would have like ten thousand, twenty thousand emails. You have to pick the right ones and only the right ones.

So we dark launched this. Like, we only did it to people who opted in, and they gave us really good feedback. You know, we made a lot of really simple mistakes at the beginning, 'cause there's a ton of logic. Like, we would pick up, like, Turkish Airlines had like a travel to Istanbul, and we thought that people were traveling to Istanbul. And we'd make a trip, and so we'd have to get smarter and smarter and smarter about our filtering. And then we got to the point where we're like, "Okay, this is good enough that we can release it to our entire, you know, million plus base."

Paul: So there's a bunch of validation techniques that... do you know Marty Cagan?

Edith: Yeah.

Paul: So for people at home, Marty Cagan is like this product guy who runs Silicon Valley Product Group. He's well known in the Valley as like the product guy. And he runs a seminar and a bunch of Circle people went to the seminar, and he talks about product validation techniques. And the product validation techniques are basically the things that enable you to do continuous delivery really well. There's two of my favorite ones, and one of them is the button on the app that doesn't actually do anything.

Edith: I love that. Well, because if nobody ever pushes it, don't build a darn thing.

Paul: Exactly, exactly. So in order to do that, you really don't want to show that button to everyone, because what if everyone loves that button? You want to ship that blank button out to a small number of people and then a larger number of people to make sure that the volume doesn't exceed it. 'Cause if you show it to a thousand people and a thousand people all click it, you don't want ten million customers who would really love that feature being like, "Oh, fooled you, the feature doesn't actually exist."

Edith: Well, let's say forwards, what usually happens is nobody clicks. Because I've run that experiment many times. And nobody clicks it. And then you do stuff like you try a different wording.

You're like, "Let's make that button bigger. Let's make the button be the entire page." And you're like, "All right, nobody cares." "Like, let's make that button be a pop-up that, like..." You know, I've pushed hard on features that just turn out just to be dead ends, but better to find it out early.

Paul: Yeah. So that's one that works really well with continuous delivery, and the other, I like this one almost entirely for the name of it, the Wizard of Oz feature. So this is where you build a button or something like that, and someone clicks it, let's say it's your check your inbox for trip things. You click the button and you get told, "We'll get back to you in 48 hours with this feature."

And then there's a human who's like... I guess that they're not gonna be aggressively filtering through your inbox, but maybe it doesn't work in that case. But someone is like checking at this use case. They're uploading the spreadsheet manually. They're entering the data, they're validating it.

But it's really not possible to launch that kind of feature at scale either, because if ten million people click it, you really don't have the ability to satisfy the volume that could possibly be there, which you're almost certain isn't there. And so you need to scale it up to the level that you can actually tolerate.

Edith: Yeah, and I think this ties back to Lean.

I think people have a misconception of Lean, that it means lazy or like just throwing shit at the wall. When Lean really just means let's not be wasteful.

Like, let's not build out this whole feature until we've established a need for it.

Paul: One of the core tenets of Perl, the language, and Perl, the community, is laziness. And they mean it in the same way as Lean. Don't build shit that you don't want. What's the other phrase that comes up from Agile? You ain't gonna need it.

Edith: Yeah. And I say this because it's so painful. Like, I've built features that nobody uses and it's just sad.

Paul: Yeah. Like, people get angry at the features. You've never experienced this? That there's a feature that nobody uses but that somehow is like deeply embedded in your code base, and people want to rip it out all the time, but they can't for whatever reason.

Edith: Oh, you mean engineers internally?

Paul: Engineers, yeah, engineers want to rip it out, and maybe they can't rip it out because there's some important customers using it, or because there isn't a new workflow for it, or maybe just like the political thing, you know, someone worked hard on that and no one internally wants to say, "This is a loser. No one uses that feature."

Edith: Yeah, and I keep coming back to, I think software is really more about people than processes or tools.

Paul: But continuous delivery is about the process.

Edith: I think it's about people buying into the process, that it's better to have proof points, it's better to be

It's better to admit that you can be wrong than put all your effort into trying to be perfect.

People, Process and Permission

Paul: So one of the areas that people tend to come up with problems with continuous delivery is this idea that the code that they're gonna ship is known or not known. Like, as in, you don't need permission. You don't need to go through a product manager or whatever. It's just whatever code gets shipped actually gets deployed out. And so how do people deal with this kind of problem?

Edith: Oh, is this back to the decentralization?

Paul: Back to decentralization.

Edith: I think you have to have some, and I keep coming back to people, you have to know and trust your engineer and know and trust that they're doing the right thing. I mean, and I think this gets back, and I'm not gonna even touch this debate for right now, like the whole offshore versus onshore.

Paul: Yeah, let's not touch that one. Yeah, that's just like... We'll just move on. I mean, so perhaps a different way of stating it is, you have team members in remote time zones sometimes, and the PM isn't online at that particular moment to validate that particular feature needs to go out, but there's a customer who's demanding it, and it's like, what do you do?

Edith: And so, I don't think there's any right answer to this one.

Paul: Right. Like all people problems.

Edith: It's like, how good is the developer? How important is the customer? Is this really something that needs a judgment call from a PM, or is it just that the PM needs to feel like they need to be there?

Paul: Right, right. This is something that I personally suffered a lot of, the whole "How come this wasn't run by me" sort of thing. And it's a really difficult thing when like a product is your baby to see stuff going out that isn't necessarily the right feature or that isn't necessarily done in the right way, or, I mean, some cases it's just the way that you would do it versus the right way.

Edith: Yeah, and I mean this, I'll say again, Paul, this isn't much to really do with software. It's more just about overall management and decentralization.

Paul: So I think the reason it's important is that customers

Edith: I'm not saying it's not important. I'm just saying, this is in fact, I think, one of the biggest decisions you can make as a company.

Paul: Whether to decentralize?

Edith: Yeah, I mean, so you can end up with a

Paul: I don't think it's necessary to decentralize?

Edith: I don't think it's necessary, but if you don't, you could say, "Okay, I'm gonna be the Salesforce model," which is very top down. Or it could be Facebook, which is extremely bottoms up, or I can come up with something that's

Paul: even in a bottoms up organization, like you're going through product managers, you're working with people who are actually validating that customers actually want these features. And especially if you're an organization that values A-B tests and experiments and feature flags, there's a process that you can create where everyone is happy with the things that you ship.

Edith: Nobody's ever happy all the time, Paul.

Paul: Okay, okay. I think what I'm trying to say is...

Edith: And I think I'm being devil's advocate.

Paul: You are. You're just totally fucking with me.

Edith: Not totally, maybe just like...

Paul: Eighty percent.

Edith: Maybe like 10% rollout.

Paul: That was a slow burner. So my sense is that really there's kind of this overlap between continuous delivery and product processes.

Edith: Absolutely.

Paul: That is like quite a complicated one to navigate. And when people are switching to continuous delivery, often the problems that they're gonna experience are not technical problems. It's not just the fear of what might break if we deployed six times a day, but it's like, how do we overcome the... or what do we need to do?

What do we need to like set up, sort of the people equivalent of the script that you check into your repo that does the deployment, to make sure that we are doing things safely and that we're doing the right things for our customer and for our organization?

Risk and the Cost of Moving Slowly

Edith: Yeah, and also how much risk are you willing to accept to your students?

Paul: So risk is super interesting, for like the whole continuous delivery thing is all about risk.

Edith: Like, and so I think part of continuous delivery is saying the biggest risk is to move slowly.

Paul: Yes, yeah.

Edith: Like, I think, for example, pacemaker, you should never do continuous delivery on.

Paul: Because the risk is super high.

Edith: Yeah, you're installing a medical device in somebody's body.

Paul: Right. I used to draw this... I remember when I was doing the first seed round, in like two thousand and twelve or something, for CircleCI, I draw this graph of what level of safety do you need for... I think the question that comes up is like, what level of test coverage should you have? And it's not a question that you can necessarily answer without knowing the complete context.

Like, if you're making medical devices, sending people to the moon, that sort of thing, then yeah, that's the sort of thing where you write provably correct code, where you run like... Why can't I think of it?

Edith: Static analysis.

Paul: There's a particular subset of static analysis that...

Edith: Paul, by the way, got his PhD in static analysis.

Paul: But yet it's only been like five years, and I can't think of any of the words that apply to it. So there are particular subsets of static analysis that are really like programs that are provably correct. If you're flying on an Airbus, you're flying on software that has been built by like the leader, the guy who invented entire fields of like provably correct programs.

On the other hand, if you're making a consumer app, if you're making an early stage startup, hell, if you're making Facebook, since, you know, that's what the internal culture is, you can ship things with a much greater degree of risk, and the risk that you really... or a greater degree of kind of software or breakage risk or outage risk or whatever that is. Because what you're saying as an organization is that the real risk that we want to prevent is moving slowly, is being stagnant, is not getting products to market, is not getting products to our customers.

Edith: Yeah, exactly. I mean, so I remember, I had a dude who's now very senior at Hortonworks, he drew a triangle which was quality, cost, and time. And he said, you just move around these axis.

Like, you can have an extremely high quality thing that takes... that never ships. Or you can, like, decide we're gonna skimp on quality because we want to ship faster.

Paul: So I'm gonna disagree, actually. I think that the idea of the extremely high quality software that, you know, takes a very, very long time to ship, I think that that's not possible. I think that when you take a very long time to release software, what you end up with is, you know, to get code to perfect, you have to ship it, you have to put it into real-world experience, and you have to iterate on.

Edith: So, Paul, I have to say I was about to disagree with you, but now I completely agree.

Paul: Okay, good, good.

Edith: Because I think the lesson I have learned is that you cannot internally test the use cases that the real world sees. Like, particularly, and to go back to mobile, particularly with the rise of mobile. Just that there's so many devices out there in so many situations with low battery, low signal, you just can't.

Paul: And there's entire startups that are dedicated to like simulating your mobile device with like all these kind of different conditions.

Edith: And people are out in the real world and they suddenly have something you never thought of. But like, actually, I heard something really interesting. I chatted with a Facebook director of engineering. And they said they could not simulate Facebook's load. The only way that they could really see if something could stand up to Facebook's load

Paul: is to put it into production.

Edith: Yeah. Wow. They're talking about, you know, Facebook has billions of users all over the world.

Paul: I hear they're quite popular.

Edith: I hear the kids dig it.

Iterating on Real Users

Paul: So on the topic of things that the kids dig, it seems that if you're ever gonna make something that's popular, it's never gonna be popular on the first go. There's always gonna be like a nugget of something that you got right, and in order to really get from there to something that grows rapidly, that is the quintessential Silicon Valley success story, you need to be able to iterate. And to ship new things every day, ship new experiments every day, be able to validate what works out there.

Edith: I completely agree, and validate on real users.

Paul: Yes, real users are tough people. So actually, so I was chatting to the product folks earlier today about this kind of topic, and dogfooding was the issue, and this is a little bit of a tangent, but so we're developer facing company, we're very technical company, all of our first eight or nine or ten or fourteen or whatever people are engineers. And we use the product heavily from the start.

But when you dogfood, you assume that everyone has this same workflow that you do. They experience the product in the same way that you do, and you're not actually getting what the customer really experiences.

Edith: Yeah, I hear you, because at LaunchDarkly we dogfood our dogfood our dogfood. It's like this chain of turtles. So LaunchDarkly is feature flags as a service. And so we use feature flags ourselves. And so something we actively try to guard against is we're like, "Okay, we could build a lot of stuff right now for us because we're the most advanced user out there." But we need to dial it back and build more and better onboarding.

Paul: Right, right. Yeah, yeah, onboarding, I guess, is... you have a business there that needs to be.

Edith: Yeah, so I am very pro-dogfooding, but I think you just always have to be cognizant that if you dogfood your stuff, you're probably the power user. Which is not a... it's just a statement of...

Paul: Yeah, yeah, no, no, and I mean, it's exactly what I was getting at, that you're not the exact target user. You're not learning at the right rate. And that the stuff that you end up shipping is stuff for you, which is great, but not necessarily stuff for your customers, because they have different workflow or whatever it is.

Edith: But I do believe in dogfooding your own stuff and figuring out the pain points, and like, I think...

Paul: Oh yeah, yeah, no, no, I think dogfooding is great. The success of a product org, especially at developer facing companies, is realizing that, or making the entire company realize, that the workflow that they use is not the same. I think that's... or whatever customers, the informing people about the edge case in the product that are not experienced by the main product team.

Edith: That's really interesting, Paul. And actually, this might be just me being selfish, but some feedback I've gotten on LaunchDarkly is that we should be more prescriptive about what workflow people should follow with feature flags.

Paul: Oh, interesting.

Edith: And I wonder if you get the same at CircleCI, about being prescriptive about how to do continuous integration.

Prescriptive Workflows and Escape Hatches

Paul: So when we were starting, the most popular thing and the best sort of designed developer product out there was Heroku. And Heroku had this logic of, you will get an amazing experience if you conform, right? And the big one was, you know, read-only file systems.

People would just not conform. You can't use Heroku because of read-only file systems, and Heroku actually changed the whole world around that model. We felt that we weren't able to do the same thing.

One was that we were a small kind of startup, and we didn't have the same sort of traction that Heroku had at the point that they were announcing this, but also we felt that every developer is a special snowflake, every developer workflow is a special snowflake.

So the kind of product principle that we came up with is, you know, if you follow best practices, and every developer community out there, Ruby, Node, you know, subsets of those, they all have like ways of using the tools in the ecosystem in an idiomatic way. So if you follow those best practices, Circle will just work. And it'll be this amazingly beautiful experience.

And if you deviate, we still want you to use the product. And so we provide escape hatches for almost everything. And we're working hard to make sure that we have escape hatches for the things that are left.

Edith: Yeah, I think that's really important. I think we fall in love with our own products, and we think people want to turn every knob and twiddle every dial when really they just want a lot of stuff to work.

Paul: Yeah, they just want it working out of the box. And it's funny that there's so many ways to implement feature flags, and we've ended up, you know, we've had homegrown feature flags since before you guys were around. And we've just kind of ended up in this situation of the feature flags that you have, or I mean, really the product that you use constrains your workflow.

So we have three different ways of doing feature flags, and we're always trying to make our continuous delivery process fit into those three buckets. So like, I have the ability to ship a feature flag that runs on a specific machine, or for a specific user but only randomly, or for a specific project but manually set.

So those are kind of the three things that solve like ninety percent of our use cases. But we ended up with... we want to do continuous delivery in a way that doesn't really work on those three kind of ways of doing it.

And so instead we need to do something else. So we find different ways of deploying and different ways of getting the product out there to customers that fit within those things. And sometimes it's not pretty.

Edith: What's a use case that you had to jury-rig?

Paul: So we were disabling a field from our API. And we were pretty convinced that no one used this field. So we set up a feature flag for the field, and then we turned it off for our own project.

Ideally, we would have scaled it up. So, you know, it wouldn't have just been listing for ourselves, listing for... let me explain what I did and then what we would ideally like to have done.

So turned it on for ourselves, and then waited a day or two, and then turned it on for random smattering of ten customers who are building right now. Waited a couple of days, no complaints, turned it on for everyone. Waited a week or two, no complaints, and then we actually deleted the code.

But that manual process of, first we'll turn it off ourselves, then we'll pick ten customers and actually type the project names into our ChatOps thing in Slack.

Edith: Oh, you're one of those chatbot people.

Paul: Yeah, yeah. It's awesome. I couldn't recommend it more. But so, yeah, we do that, and then

really what I want is the... you know, we scale it up to 10%, to 20%, to 30%, but we can only do that on a randomized basis. We can't do that on a static, like, once customers are in, they're able to do it. So we weren't able to use that particular feature flag mechanism for it.

Edith: Well, Paul, you know you can use LaunchDarkly on a feature-by-feature... No, on a feature-by-feature basis.

Paul: Yeah, yeah, I know.

Edith: So I mean, you could keep your homegrown system and then just use us with really important features.

Paul: Oh, that's an interesting idea. Yeah, I hadn't thought about that.

Edith: I mean, you could just use us for those features and keep your old system in place. And then gradually all your features belong to us.

On-Prem, SaaS and Gatekeepers

Paul: Right. Oh, it'll get there. The other one, we were talking a lot about SaaS software, and obviously continuous delivery is this thing which came about because of the prevalence of SaaS software. But we're launching on-prem software. So, you know, as well as CircleCI, you can buy CircleCI Enterprise.

And that's like literally things that we ship that run in your own VPC or that run on-prem. And we can't really ship the same things to do testing.

For a start, we're not allowed to ship our analytics code in there. So we don't run Mixpanel in the stuff that we ship to enterprise customers. And so we can't tell who's using what parts of the features, and it makes the whole continuous delivery model much more difficult.

Edith: Oh yeah. I mean, so we talked before, it's like so with Apple's a gatekeeper, on-prem is a gatekeeper. Like, really, the rise of SaaS.

Paul: Really, that's what they're trying to be. Like, they're deliberately trying to be a gatekeeper.

Edith: Yeah, well, I think continuous delivery and SaaS are so tightly intertwined because one enables the other.

Paul: Right. There's this blog post that I read kind of in the dawn of SaaS, and it was by Joel Spolsky in the Joel on Software blog, where he talks about switching Fog Creek from a downloadable thing that runs on someone's thing to being actually on the web and being a web application. And one of the things he talks about is being able to ship things that customers don't see yet. Being able to ship things that fix customer bugs immediately after they start, and being able to get notifications about error messages, or about errors or whatever it is, immediately, instead of being sent by a bug report or being sent by an actual customer via an email that they catch, replicate or whatever.

Edith: We used to be so blind, Paul.

Paul: It was fucking awful. Like, trying to write software in... Oh, so I worked in the games industry briefly, when I finished college the first time. And shipping code that runs on a console that is going to be shipped like on a gold master disc six months from now, and that you're never really going to be able to validate.

That was like... games aren't even shipped that way now. But like, that was a fucking awful time of trying to understand all the different ways that software could break and all the different ways that your code might just not work when it literally shipped to millions of people at once.

Edith: Yeah, and to tie back to something we were talking about before, that's when you got this feature and scope creep to the point where like Duke Nukem never shipped.

Paul: Yeah, it's a particular problem in the graphics engine, or in anything that involves like a state-of-the-art graphics engine. Things will just fall back until they don't ship.

Edith: Yeah, well, 'cause then, and then all of a sudden, I mean, it goes back, and then you have all these issues with, well, people's computers themselves have moved on.

Paul: Right, right. The expectations are different as well. Like, if you're shipping Duke Nukem and it's been seven years now, the thing that you're gonna have to ship is gonna have to be amazing. Yeah, it'll have to be life changing, or else people will wonder what it is going on.

Edith: Yeah, but instead you've just been trying for the past seven years just to like

Paul: ship anything. Yeah, no, I'm very, very glad to be in the SaaS business.

Edith: It's liberating. I think analogy is between the transition between books to newspapers to online.

Paul: Right, because people are... the news cycle changes as a result of it.

Edith: Yeah. If you can publish multiple times a day, you just publish it.

Paul: Yeah. And of course you can continually go back and edit those articles in case you

Edith: Or put a retraction or a correction, unlike a book. Like, once a book is out in a bookstore, you know, it's there.

Typos, Theses and Breaking Gmail

Paul: Yeah. So the biggest thing that I've ever written, or the longest thing I've ever written, was my PhD thesis. And I have a paper-bound copy in my living room that I never look at. But occasionally someone will come and they'll be like, "Oh, here it is," and they'll open it to a random page, and looking over their shoulder, I see a typo.

Edith: Oh no!

Paul: And whatever page they open, there's some typo. I've read that thing fifty times. But somehow I didn't manage to get them all out.

Edith: Did you? That's really funny. Are they terrible typos or just mini ones?

Paul: I mean, it's things that a spell checker would have caught sometimes.

Edith: Oh, did you make it in some like terrible LaTeX or something?

Paul: Of course it was in LaTeX.

Edith: Yeah, of course. So it's probably beautifully formatted,

Paul: It looks amazing. Garamond font.

Edith: So it looks beautiful, but it's riddled with

Paul: minor punctuation errors.

Edith: Yeah, I remember, this is a total aside, but I wrote an econ thesis and I was really proud, and I took it home to my parents. And my parents, my mom has a PhD in English, my dad is an editor, and they started pointing out like, "You used an em dash when you should have used an en dash." I was like, "You're supposed to be admiring like my deep economic thought, not copy editing."

Paul: So apart from doing a therapy session for that, what else have we got in the continuous delivery vein?

Edith: Oh, I just was back thinking about my thesis.

Paul: And the deep parental scarring. Looks like we're all out of things to talk about this week.

Edith: No, I think we opened it up. You asked me what I liked about continuous delivery. And you said I was surprised by my answer.

Paul: What was your answer again?

Edith: I liked getting product to people faster and making them happier.

Paul: Oh yeah, no, I thought you were going to talk about employee morale.

Edith: I mean, I like employer morale also, but I don't think that's the... And, well, I think they tie together. I think employees are happy because customers are happy.

Paul: Right, right. Personally, having worked at Mozilla during the awful Firefox 3 to Firefox 4, you know, conversion, or the Firefox 4 launch, which took eighteen months, and you know, things just kept slipping and things just kept being added, and feature creep, and there were these things that the web was waiting for that were being held back by Mozilla being unable to ship Firefox 4.

And people were not happy. Like, the entire organization was particularly unhappy. And when we switched to the release train model, it's kind of a few months before I left, I think. And actually, I hold the singular distinction of being the first person to fuck up the Mozilla continuous release train model.

Edith: That wasn't where I thought you were going to go there, Paul.

Paul: No. I forget where I was going, but I think I can probably get there. People were unhappy. That was a good thing.

Edith: How did you fuck up the release train?

Paul: So I shipped a feature just before the deadline, and that's the whole thing that you're supposed to avoid with continuous delivery. It's never about shipping before a deadline, but no, I was determined to get this feature in. And this feature was one where we had argued about it considerably online on Bugzilla, and then, you know, I said, "Look, I'm just gonna ship this, and we shall see." And the feature was gonna be an amazing one.

So in JavaScript, there's no way to get a sub-millisecond precision. Alright, but there's a feature called Date.now, and Date.now returns the number of milliseconds since whenever.

And I said, "Okay, we can make this a floating point number, and that can give us sub-millisecond precision." Unfortunately, people were using this basically as a random name generator, and they were creating DOM nodes that had the name of the number of milliseconds.

And it was a terrible, terrible idea doing it, but DOM nodes can't have dots in the name, so I turned it to a floating point number. Everything broke, and the first thing that broke is Gmail. So Gmail upload use this feature. Feature, I spit on that feature. Used the thing, and so I broke Gmail for about a million people who are using our nightly release.

Edith: Well, Paul, I'm not gonna give them your address so that they can kinda talk about how they really needed that email.

Paul: But because continuous delivery, it rolled back. There was no big deal.

Edith: And I think that also plays into a bigger point that we were making, that until you push a feature out into the field, you can't really know how people are using it.

Paul: Right. Yeah, I mean, so actually in this particular case I think we could have predicted it. In retrospect, there were some things out there. But yeah, once it goes out, it settles the question for once and for all. Lots of people who knew what they were talking about were saying, "This is going to break the web," and I said, "No, no, it'll be fine." And really, we just put it out there and it broke the web and we rolled it back.

Edith: Yeah, I mean, that's one of my major tenets, is users always beat testing. And users beat assumptions.

Paul: Right. Products need to be validated, code needs to be validated.

Edith: And the sooner you get them out to real people, the better.

Paul: Did we talk about risk yet?

Edith: We talked a little bit about risk. We talked about pacemakers.

Paul: Oh yeah, yeah. So the thing that's important, I think, with risk and continuous delivery is that smaller units are less risky. You can ship out a tiny, tiny piece of code and validate that it works or doesn't work with significantly less risk than a whole quarter's worth of work.

Edith: And I think it's risk at every level.

Paul: What do you mean?

Edith: It's risk at market level. So I think you were talking very narrowly about just like

Paul: code risk and ability of, like, breaking a customer or being able to... like, you can't roll back a quarter's worth of stuff, but you can roll back one commit.

Edith: Yeah, but I think, to me, it goes even bigger. Like, the biggest risk, and we talked about the biggest benefit of continuous delivery, is the biggest risk is always just you're building the wrong darn thing.

Paul: Right, right. Tell that to your CIO.

Edith: Like, well, you spent eighteen months or twenty-four months just going in direction, like to your thing about the game discs, to build a game that nobody wants or cares about.

Paul: Right, right, right. Oh, well, on that depressing note. We are gonna be back in two weeks, I think.

Edith: Yeah, two weeks.

Paul: And we're gonna be talking about more continuous delivery stuff. We don't know exactly what, so send us a note.

Edith: Edith at launchdarkly.com, and I'm on Twitter at Edith underscore H.

Paul: And I'm on Twitter at Paul Biggar. And we shall see you in two weeks.

Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of CircleCI, and Edith Harbaugh of LaunchDarkly.

Edith: To learn more about Heavybit, visit Heavybit.com. While you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.

Subscribe to Heavybit Updates

You don’t have to build on your own. We help you stay ahead with the hottest resources, latest product updates, and top job opportunities from the community. Don’t miss out—subscribe now.