Ep. #3, Origins of Continuous Delivery

Paul Biggar: Before continuous delivery, there was often this idea that when you shipped the code, you shipped the product. There are rules about how one has to dogfood one's own software.

Edith Harbaugh: The dirty secret of product management is there is no such thing as a professional product manager.

Paul: Hi, I'm Paul Biggar, founder of CircleCI.

Edith: I'm Edith Harbaugh, CEO and co-founder at LaunchDarkly.

Paul: And you're listening to To Be Continuous, a podcast about continuous delivery and software development.

Edith: You can get in touch with us anytime at our Twitter handle, @continuouscast.

Paul: The show is brought to you by Heavybit. To learn more, visit heavybit.com. And while you're there, check out their library, home to great educational talks from other developer company founders and industry leaders. In this week's episode of To Be Continuous, Edith asks me, what are my favorite things about continuous delivery?

Product and Technical Validation

Edith: So Paul, what's your favorite thing about continuous delivery?

Paul: So my role is partially product manager and partially engineer. And so the kind of two favorite things I have about continuous delivery. One is the product manager stuff, which is you're able to ship a basically broken feature to one customer, two customers and whatever, and do your product validation and get validation early rather than after six weeks of building the perfect feature you get your validation after three hours of, you've got like two half-written lines that you know will work for one customer.

The other side of it, as an engineer is being able to roll out features without having to calculate every single edge case and every single customer possibility, and roll it out to a very small number of people. And I guess it's kind of validation as well, but it's more technical validation rather than product validation.

You're able to find out operationally, does this piece of code work, rather than having to do a rollback or be worried about, have I written enough tests? Have I got enough coverage? Feature flags and slow rollouts and that are kind of my personal favorite part of continuous delivery.

Edith: Yeah, it's super frustrating. I was talking to a friend who, they spent literally three thousand dollars in server costs testing out whether this feature would scale and then they shipped it to their customers, they found out that it didn't matter because nobody wanted to use it. So they spent literally three thousand dollars on Amazon Web Services that were just throwaway.

Paul: The largest cost that I think exists, and people sort of ignore this cost a little bit, is the cost of the engineers that are involved in shipping these things.

So if you build a six-week feature that took one engineer six weeks or two engineers three weeks or however it splits out and you're paying an engineer, let's say in San Francisco, then you just spent ninety thousand dollars on that feature.

Edith: Oh my gosh. And people don't think of it that way.

Paul: Right. It's the same as meetings. No one thinks that this meeting is costing us fifteen thousand dollars an hour or fifteen hundred dollar or whatever it is.

People think like, oh, we need to get this done. We need to build this feature. It needs to be the right color blue. It needs to be right. It needs to be perfect, rather than, the cost of this is astronomical, and how can we make this cheaper and how can we get more bang for our buck?

Edith: Yeah, I mean to go back to lean, it's like how can we reduce waste? Like how can we cut all the fat out of this being built and make it more efficient? But a lot of the pushback I get around continuous delivery and lean is like, why can't you just build it right the first time?

Paul: I think there's a bunch of reasons you can't build it right the first time. And the first of them is the real world is complicated and the real world has error conditions that you don't consider and it has, so I'm talking from a technical perspective here. But basically if you try to push any feature you will immediately find that it doesn't work for a bunch of customers for technical reasons.

There's exceptions that you weren't expecting. Clearly that's why they're called exceptions. There's a bunch of like bugs that are going out. There's edge cases that you didn't consider.

And then from the product side of things, there's exactly the same things. There's edge cases you didn't consider, there's customers who think about the product in a different way than you think about the product. So "Can I build it right the first time?" I think is just, it's a sort of a naive position. It isn't possible to build it right the first time, no matter how prescient you are, no matter how good of a product visionary you are.

Edith: And what I found myself is that sometimes I would overbuild a feature and miss the mark. You know, I thought people cared a lot more about something that it turned out they cared very little about. And back to what you said, that's extremely expensive, not just in the raw cost of the engineer, but in the opportunity cost of what you could have been building instead.

Launch Dates and Death Marches

Paul: Right. Before continuous delivery, there was often this idea that when you shipped the code, you shipped the product. And so people would have a marketing launch, or well, a launch, but a launch is a marketing launch, that they would aim to coincide at getting the code into the code base possibly for the first time before that launch. And this is always the most disastrous thing I've ever seen.

Edith: I have my own stories, but let me hear yours.

Paul: I don't particularly have any great stories, just I've seen it. I've seen people's launches that are like, oh, this doesn't work, or people talking about their death marches of like, we're launching this in the morning, I need to get it finished tonight.

It should have been finished last week or it should have been finished last month and in like five customers' hands for the previous month. And you should have validated, and why you trying to launch a thing that isn't validated?

Edith: Yeah, there's this whole school of stuff of the death march and then how the product managers and the marketing engineers kinda go in this death spiral of artificial deadlines and must-ships.

Paul: Right. The two work, I was gonna say well together. The two work horribly together. Like the deadlines, or artificial deadlines in particular cause, is it artificial deadlines? It's more, I think the

Edith: I'll give you an example. So I was working at an enterprise software company and we had this absolute must-ship of October 31st. So, you know, all summer this seemed a long way away until suddenly it didn't. You know, it's August, it's September, and we're like, damn, it's on top of us.

And really what it was is, nobody was really waiting for it with October 31st. It was just that marketing didn't want it to slip into November and then December and then have everybody be away. And we just shipped this completely half-baked thing which people could not migrate up to, 'cause that was literally the thing we threw away, is like, okay, we don't have time to build everything.

So we launched this thing that none of our base could use. Because there's no, it wasn't like that we had a half-baked migration path. It was literally, we said, fuck 'em, they can't migrate. I was in that meeting.

Paul: Right. This is awful.

Edith: Yeah, and it nearly killed the company because we just burned our entire base.

Paul: So how often were your releases in the

Edith: Oh, in that time we were considered fast because we were once a year.

Paul: So, you know, clearly if you were doing daily releases or weekly releases, there wouldn't be that, any of these problems at all. You could have released the thing long before October 31st. It could have been behind a feature flag in June. It could have been ten customers in July. You could have had all ten customers saying, well, we're not gonna use this, there's no migration path, and then realize that the migration path was the most important thing.

Edith: What you said of the code is the product, it's like, well, we said it wasn't. I mean, I think that's why I'm such a big fan of continuous delivery, is because I think it forces you to be more realistic. There's no mythical future out there. The future is now. The future is the people using your product.

Paul: Right. I've seen this idea when I worked for a telecoms company that would remain nameless, that if you hit the deadline, then the fact that what you were shipping was a really bad idea wouldn't be your responsibility.

Edith: Yeah. Garbage in, garbage out.

Paul: Something along those lines, yeah. So that marketing gives you an unreasonable deadline. Product management doesn't push it up in front. The engineers are writing code according to someone else's schedule.

And if you write the code that goes out, then it's not your fault that it's gonna be a disaster. Even though you can see it be a disaster and maybe the people on the ground can see it being a disaster. The people higher level are sort of like, I hope this isn't a disaster, and everyone else thinks this is gonna be awesome.

Edith: The healthcare.gov story.

Paul: Exactly. Healthcare.gov is a prime example of this. And if there had just been a sort of a continuous delivery thing, let's get this out its front, and what it builds is that the people on the ground end up with the same, not necessarily on the same page, but the same incentives as the people in the marketers, the product managers.

Edith: Yeah, so I visited a customer last week and they'd gotten LaunchDarkly. And it was kind of funny and a little sad when they said what they were most excited about using feature flags was just to show features to their executives.

Paul: Wow. So they would build things that the executives asked for, only for the executives?

Edith: Well no, so right now the executives had no early look at features, so by the time they saw them, if they were bad, there's just all this

Paul: Oh, you know, that doesn't sound so bad.

Edith: No, I was excited.

Paul: Right, yeah, okay. No, that sounds good.

Edith: Yeah, 'cause they wanted to get, because right now the executives were at the very end of this long release process and they didn't have any way to give them an interim peek.

Paul: Right. There's a big difficulty in terms of executives and management being involved in the product management process.

Founders as Product Managers

Edith: Yeah, I think it's fascinating that you called yourself a product manager.

Paul: Why is that?

Edith: Well, I mean you're also founder and CEO of...

Paul: Sure, sure, sure. I've engineered things so that my role is far more product than, I don't know, finance and management and that sort of thing. So the primary thing that I worry about now is, is the product good enough? How can we serve our customers' use cases?

And it ends up being a lot more like product manager than... Now, that is to say, we have professional product managers who actually know what they're doing who help with this.

Edith: The dirty secret of product management is there is no such thing as a professional product manager.

Paul: Fair, fair. But there's definitely people who have done it before and know how to validate and prioritize and make sure that releases don't slip and all that sort of thing. But the idea that an executive or that a PM would only see it towards the end is a really bad idea. And the idea that they're going to like completely spec it out up front is also a really bad idea.

I think kind of the secret of product, or of good product, which most really good product managers know, and it seems that very few executives know and also very few engineers know, is that it's successfully balancing the company goals with the actual dirty details of the implementation.

So there's engineers who will tell you that they're in the best position to build the product because they are actually in the code, they're in the weeds, they see how things can actually work. And there's executives who will tell you that engineers have no idea how to build a product because they don't understand the business case behind the product. And the truth is somewhere in the middle.

Truth is that the business case is, it's not everything, but it's a lot. And if you try to formulate the business case without understanding the real product metrics and how people are using the product and how the code is architected and orchestrated, then you can't possibly specify the product without that data.

Edith: Yeah, and I'll also add that as an engineer you could fall in love with your own product. Like we talked about this before, that you always know your product the best. And you're always thinking of some way to make it this little tiny bit better when it could be something that your actual users just don't care about.

Paul: Right. There's a bunch of features in Circle that were basically implemented the wrong way. And that was largely because I fell in love with a particular concept or an abstraction or way of doing things. And it turned out just to be confusing to customers.

Edith: And I think the only fix for this is, as you said, to show or to talk to actual customers. Like, I'll give an example, I talked to a LaunchDarkly customer. And I thought they were gonna ask for a lot of stuff, but what they actually really wanted was a Slackbot integration. And it would not have occurred to us at all because we use HipChat.

Paul: Okay, right.

Edith: But they wanted basically a chatbot so they could roll features out to different users right inside of Slack.

Paul: So one, that's an excellent idea. Our feature flags are all keyed from Slack.

And I think that there's a real danger to dogfooding. If you dogfood really, really well and you get a lot of insight from your product that way, then you don't develop other techniques for getting feedback from your product and getting feedback from your customers.

Edith: You become convinced that you are the customer.

Paul: Right. And you're never the customer.

Edith: You're a customer.

Paul: Right.

Edith: So back to your product manager, you are a customer.

Paul: Right.

Edith: With very specific needs.

Paul: And often you are the most knowledgeable customer. Right, I mean you are always the most knowledgeable. Like we have probably used features that none of our customers even know exist.

There's features that we built for other customers to use, but for ourselves first, behind a feature flag, that actually never got launched, and if they become part of our workflow and customers are saying, oh, there's some kind of thing, how do I solve that? I was like, how is that a problem? We don't experience this problem. Oh yeah, we never launched that feature.

The Limits of Dogfooding

Edith: Well Paul, I'm gonna devil's advocate you a little bit about some of the pushback I get in continuous delivery. And 'cause when you're talking about the customer one right now, the classic one is Slack, which came out of Stewart Butterfield himself using chat and wanting to build a better one.

Paul: Right. Sorry, I missed the, what was the question there?

Edith: So that's held up as an excellent product and it was all dogfooding.

Paul: Yes, so I think these are kind of different stages of the product. So here's a good example. Slack has excellent onboarding. The Slack team onboarded once at most.

And so in order for that to have become the amazing product it is, someone somewhere has a job of making sure that onboarding is awesome and they are not dogfooding onboarding every day. Or the whole team is not dogfooding onboarding every day.

Edith: That would be kinda horrific if every day they had to come in and reinstall.

Paul: Exactly, exactly. And then Slack has tons and tons of integrations and you add rooms and like the experience around all these things is really, really good. But no doubt Slack is only using one video integration or they're only using, you know, they're not using Google Docs and Dropbox. Or maybe that's a bad example, but they're not using the Microsoft tools and the Google tools or whatever.

They're not using multiple continuous integration services. And so how do they know that the integrations with those services are amazing? They must have a process that lives outside of their own dogfooding.

Edith: Yeah, I totally agree. I think if you get too far from dogfooding, you end up with a product that nobody, not even the people who built it, love, and that's very dangerous. Like you know, the joke is like nobody use the Microsoft Zune. Or my friend who worked at Nokia who carried around an iPhone because he didn't want to use the Nokia phones.

Paul: One of the reasons that people start to use products, this happened a lot when I was at Mozilla. People didn't want to use Chrome at Mozilla because, while they felt that Chrome solved problems for them. So in particular memory usage was a particular problem and for people who used a large amount of tabs Chrome was just a better solution at the time. But there was a loyalty aspect to it. You didn't want to use Chrome because you want to be loyal to your company or to your team or to, you know, the mission that you believed in.

And as a result, people don't experience other products and they don't see how much better the world could be, and they just kinda get stuck into the, you know, we're using this and we're putting up with the shit in this because we have to.

Edith: Stockholm Syndrome.

Paul: It's a little bit Stockholm Syndrome, definitely. Every time you tell your customer a hack, you know, that's a small amount of Stockholm Syndrome. You're saying there's a way of doing this, and the way of doing this is, well, it's a little bit hacky.

Edith: Which is not always bad because sometimes then the customer feels like they have this insider track. Like customers like feeling like you're giving them something special.

Paul: There's two sides to that. So at the start, yes, they love the specialness. I talked about this in a previous week where you want your customer support team to be able to say, oh, thanks for the report, we just fixed that. Or, you know, thanks for highlighting this use case. We just built it in the last ten minutes just for you.

And that's great until you get about six months into using the feature and they're still using the hack and they're worried that it's gonna go away. And we had this with a customer recently. We built something for them, a special way to control notifications. And we didn't want to build it into the UI because we were going to change how the UI worked and so we put it like somewhere that we didn't document and that was in an experimental section of their configuration.

And now they're like, okay, well this is in the experimental section. We're worried that it's gonna go away. You know, we're a big customer. How do we make sure that this doesn't go away on us?

Feature Flags as Long-Term Controls

Edith: Yeah, that's a way that a lot of people are using feature flags actually.

Paul: Go on.

Edith: So originally we thought people would use feature flags just for rolling out a feature and what you said at the beginning, for rolling them out to different users, verifying that things didn't break, things didn't implode. You know, the dragons didn't come down and set fire to the universe. What we found is that people were also using feature flags for really long-term controls. And it was precisely that use case.

It's like you have this customer who wants access to a feature. And you want a way to know that these five or six customers have access. And that's something that you want to know so that when you're doing new updates, you don't actually overwrite that, but it's not something that you wanna, it's not something you want at the code level, it's something that you want at higher level.

Paul: One of the dirty secrets of enterprise software is that you have dirty hacks for customers.

Edith: Oh my gosh, everywhere. Everywhere.

Paul: So many people have told me about the if statements that they have. If customer equals Google or whoever, then we're gonna do it this way. And feature flags provide a nice little abstraction above that.

Edith: And the abstraction is, and this is what our customers are using it for, is that then at least everybody else knows that there's this thing in there.

Paul: Right, right, right. So, and then you also get metrics around that abstraction so that when you decide to sunset something, and you should be sunsetting these features or finding better ways of doing it or something that actually solves their use case, you should be moving them away from that abstraction, and having a counter, a graph, a recency indicator of when these features are used, it's essential to actually getting those features sunsetted.

Edith: Yeah, or even out in the light. Like what would happen in enterprise software is you'd have one config file somewhere that one person knew about and that person would leave.

Paul: Oh wow.

Edith: And then all of a sudden everything would break and nobody even knew why. Or like stuff would be, you know, always implemented at the wrong layer. No matter where it's implemented, it's always at the wrong layer.

Shipping Buggy Software

So I'm gonna play devil's advocate again. The pushback I get is we shouldn't ship buggy software.

Paul: I would argue, without shipping, all software is buggy. Every piece of software that you ship is buggy. You just don't know what the bugs are yet.

Edith: So we put people to the moon. There are no bugs on that.

Paul: I mean if you want to spend ten years or an entire decade, you know, shipping your feature, you can ship it without bugs. If you're in a thing that can tolerate, well, let me put it this way.

There was the Challenger disaster. There's the multiple car recalls that are happening at the moment. So even people who have incredibly long cycles with strong validation and static analysis and all these things, even they can't ship non-buggy software.

Nobody can ship non-buggy software. The only reliable way of getting software that is in any way reliable is to ship an early version, which is known buggy. And if you don't ship an early version that's known buggy, you're gonna ship a late version that's unknown buggy.

And when you ship that, you find what are the actual things that are happening in practice.

So I shipped a feature last week. It was a long-run feature, like, you know, it took a lot of building and we shipped it behind a feature flag and all the tests passed and everything and I just turned it on for one customer, which was us, on one branch, and there was an immediate obvious bug.

And well, this is great because I didn't ship it to ten thousand customers to discover an immediate obvious bug, I shipped it to one customer and that customer is me, and I can go fix the bug now before we ship it to any more customers.

Edith: Yeah, I mean we did the same at LaunchDarkly. We were shipping something new and we do absolutely everything with feature flags. Because we have to. And we found that

Paul: There are rules about how one has to dogfood one's own software.

Edith: Well there was a rule and then we got really busy and we got very sloppy and we stopped doing it and then we tried to do a big release and it was a, you know, it wasn't a disaster. Big release.

Paul: Wow.

Edith: What?

Paul: I find it hilarious when continuous delivery companies do big releases. I mean everyone does it at some point, but it's still funny.

Edith: Well, for us, it wasn't huge. I think it was like two weeks of accumulated stuff.

Paul: Okay.

Edith: Which for us was quite big and then

Paul: You don't deliver every day?

Edith: It depends on what we're working on.

Paul: Okay.

Edith: So we do like mini pushes, but most stuff is behind a feature flag.

Paul: Of course, yeah.

Edith: So the two-week release was kind of our come to Jesus moment of, we're like, wow, that was a big pain. Like 'cause we hadn't feature flagged stuff adequately. We had to spend a lot more time at QA. It was just, it was stressful.

Paul: Yeah. There's a bunch of features that, like, any time that there was real stress in the product team was when someone had built a feature without feature flags and someone said, yeah, I'm not sure about this. Can we A/B test it? Can we push it out to only select customers? You know, and this is things that were either big product changes, so people have been working on them for weeks and were very happy with them and, you know, really wanted to get out there.

And so it led to a lot of frustration and it led to just people being generally unhappy and, you know, harsh words being said and

Edith: No.

Paul: I mean harsh for, like, you know, for, still professional. Harsh as in like, I really want to ship this today. I really feel that you shouldn't ship this today.

Edith: Oh, Paul. If you've ever been in enterprise software, there are many harsh words said, and usually they're along the lines of like, "This customer promised us seven hundred and fifty thousand dollars this quarter, where the fuck is my release?"

Paul: Right. So there weren't quite those harsh words because there wasn't any customer who promised us seven hundred and fifty thousand dollars. If there had been, then I'm sure words could have been harsher.

Edith: I like what you say, when you descale the stakes, it gets a little bit more civilized.

Paul: Developers get frustrated when they can't ship their stuff. The only thing that, you know, causes people to be really unhappy is the idea that they're blocked behind a thing and they get frustrated.

Day-Sized Work and Yammer's Hypotheses

Edith: Yeah. I met some guys from England that, from Geckoboard. They were here for a lean meetup and they took it to extremes. They said they broke everything down into day-long and no longer.

Paul: So what, you have to ship at the end of the day?

Edith: I don't think it was at the end of the day, but it couldn't be more than a day's worth of work.

Paul: Okay.

Edith: And they said this was very good for morale, because like everybody always saw their stuff continually getting out.

Paul: Yeah. And so they shipped it behind feature flags?

Edith: Yeah, they had built their own system.

Paul: Right. I guess they must just have gotten good at building a day's unit of work.

Edith: That's what they said. They said that they were very much into Kanban and they just tried really hard to scope it that way.

Paul: Gotcha. So at the start it was difficult and they just kind of got into the groove and

Edith: Well, for employee morale, for the reasons you just gave. You know, and so it was funny 'cause they were so bought into it that like if I'd told them about the old days where releases took years, I think their heads would have just kind of popped off in a very British way.

Paul: I like this idea of one day. I was reading about, I think this was Spotify.

Spotify? No, actually this wasn't Spotify. Whoever it was, every project was a two to ten person team and two to ten weeks. And those were the rules of how things got shipped.

Edith: Yeah, Yammer did the same at the beginning.

Paul: That was Yammer. That was Yammer.

Edith: Oh, it was?

Paul: Yeah.

Edith: One of the product managers, Ron, he did a guest post on our blog about how methodical they are. Just everything must fit in, must fit, must be data-driven, must be hypothesized.

Paul: Hmm. Oh, the hypothesis is an interesting thing. So every project would have a hypothesis for what the data was going to show?

Edith: So you can read more on my blog. He says it better than me, but you know, he talked about how he was trying to get people to upload more pictures to Yammer. And the hypothesis was that if there were more pictures on Yammer, there would be more engagement.

So they did a lot of improvements to their photo uploader to make it a lot easier to upload photos. And they let the experiment run and no more photos were uploaded.

Paul: Okay.

Edith: So they were gonna not ship the new feature. Just because it didn't improve days engagement.

Paul: Okay.

Edith: But they finally did ship it just because it got rid of some technical debt.

But literally if something doesn't move their engagement, they don't ship. Like no matter what their feature is, if it doesn't improve engagement.

Paul: Right. It's interesting for consumer companies that have to focus on engagement like that. 'Cause I think that there's a lot of B2B companies who would very sternly say, you know, engagement isn't as important as customer experience, or something that's qualitative rather than quantitative.

Edith: Yeah, they're very strict and they are B2B. Well, Yammer's kind of...

Paul: It's definitely on the line. Like, it's mass B2B, has more B2C characteristics, I think, than B2B.

Edith: Yeah, but it was interesting because they are so very strict about it. And he's, they actually have the analytics group as an entirely separate department from product and engineering to keep them honest.

Paul: Oh, interesting. Right. Yeah, it's very easy to skip the rules when it's like, we, yep. We know this one is gonna work. Let's just ship this and we don't need to test it. We don't need to validate it.

Edith: Or we shipped it and we tested it and it didn't improve, or maybe it actually made stuff a little bit worse. But we have this sunk cost of, we already built it. You know, so it degraded engagement by five percent, but what's five percent?

Paul: Right. Yeah, so I think, yeah, that's definitely my favorite things then about continuous delivery, the product validation and the technical validation.

Edith: Yeah, and just doing it quicker.

Paul: Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of CircleCI, and Edith Harbaugh of LaunchDarkly.

Edith: To learn more about Heavybit, visit heavybit.com. While you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.