1. Library
  2. Podcasts
  3. To Be Continuous
  4. Ep. #2, What is Continuous Delivery?
To Be Continuous
40 MIN

Ep. #2, What is Continuous Delivery?

light mode
about the episode

In this episode, Edith asks Paul ‘What are your favorite things about Continuous Delivery?’

For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.

Before Continuous Delivery
Feature Flags and Continuous Integration
Test-Driven Development and Customer Expectations
Lean and User Feedback
Evolution and the Wright Brothers
Distributed Systems and Services
The Consumerization of the Enterprise
Fad or Status Quo?

transcript

Edith Harbaugh: Are you saying it was literally big in Japan?

Paul Biggar: Continuous delivery is something that humanity has sort of been doing in various different facets for a long, long time and it's just, you know, now is the first time it's being applied to software.

Software isn't shit anymore.

Edith: Paul, there goes our clean rating.

Paul: We never had a clean rating.

Edith: The dirty builds just sat up.

Paul: Hi, I'm Paul Biggar, founder of CircleCI.

Edith: I'm Edith Harbaugh, CEO and co-founder at LaunchDarkly.

Paul: And you're listening to To Be Continuous, a podcast about continuous delivery and software development.

Edith: You can get in touch with us anytime at our Twitter handle, @continuouscast.

Paul: The show is brought to you by Heavybit. To learn more, visit heavybit.com. And while you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.

Edith: In this episode, we talk about what continuous delivery is and where it came from. Everything from SaaS and test-driven development to lean startups and human evolution.

Before Continuous Delivery

So Paul, where did continuous delivery come from?

Paul: So this is one of those things that evolved over quite a long time, but I think that the major sort of catalyst of continuous delivery was the fact that we moved to SaaS and that SaaS now exists.

Edith: So what was it like before there was continuous delivery? What was it like when you were starting out?

Paul: The first company that I really worked for was a games company and I think that a lot of them are kind of in the same place now. But you would release software on a disc that's shipped to customers via stores. And you had to have a gold master at some point.

And so we were a startup, so it was quite advanced and we had a daily build server. And we actually were selling middleware, so we shipped software to those companies. But those companies would ship at the end of the day, they would ship a disk that went out once. And there was no updates at the time. It was 2003, 2004, something like that.

So there wasn't any way to automatically update on customers' PlayStation 2s, I guess. Even games now have a little bit of continuous delivery, but back then you got your chance to ship it and that was it.

Edith: Yeah, I just had a brutal flashback to one of my first summer jobs, which was installing Windows 3.1. And this is back when you actually had, it was a set of 35 discs. So my job as a summer intern was to go to every computer in the office and like just literally sit there, like disc in, disc out, disc in, disc out. And if something went wrong in the install, I had to start over at the beginning.

And so it was like a two-hour long process just to get everybody up to 3.1. So in those days, there was no chance of a fix pack. I think everybody's gonna be on this version until probably they got another sucker summer intern in there.

Paul: And so if you imagine people installing even modern or relatively recent version of Windows where you have to think about the service pack one or service pack two or what level are you patched to, that was basically the old way of doing software.

And the first time that I remember realizing that there was a different way to do it was in a Joel on Software blog post where he talked about Fog Creek and how FogBugz had moved to being in the cloud. And one of the major things that they could do is that there was a bug, they could fix it. And in the cloud it would be fixed.

Edith: I think it's good though to realize that the reason why there was so much emphasis back then on making sure that a build was correct was because there was such heartache and pain if it was wrong.

Paul: Yeah. It made sense to hold up the release for new features or for something like that if it was promised to a customer because there was no alternative. It would be three or six or twelve months later when they got the next opportunity to go out.

And I think frankly, building software in the cloud and having the possibility of doing a release at any time you like has actually just completely revolutionized how people make software.

Edith: I completely agree. I mean, to go back to the old days, all the emphasis was on extremely high quality and extremely stable. Because you basically had one chance. You were lobbing this thing out into the world.

Paul: And even things where they had like quarterly release cycles or something like that, it was still, it felt a lot like one chance. So I worked at Mozilla and I was there on the switch from Firefox 3 to Firefox 4, and Firefox 4 took eighteen months to come out. And there was things that were ready to be released, you know, the first week of that release cycle, and that had to be held up for all these unrelated releases.

I think there was problems with the JavaScript engine and there was memory bloat and that sort of thing kept pushing it back. And Firefox, during the time that I was there, switched its process and it switched to what they call a release train model where basically they ship a new version every six weeks. And it doesn't matter what's shipped or what's not shipped.

They'll hold it up for total, like, life or, you know, build breaking, it just doesn't work at all. But basically everything else would make it go out. And the cadence and the velocity that they were able to get since that happened has been massive.

Edith: Yeah, I mean, so that was the first evolution, was it used to be let's try to cram everything in, and then you'd get this vicious cycle where it's like, "Well, let's just put this one more thing in," and then you'd have to add on all the QA time. So that was the other reasons why releases would take so long, is you had so much QA.

Paul: Interesting.

Edith: I talked yesterday to a guy who'd worked at Microsoft and he said, you know, quality was, things had to be rock solid for Microsoft, going back to my disk example.

Paul: And these are people who have unit testing, automated tests as part of their builds.

Edith: But even beyond that, you have to have a person actually try it and then you have to actually go out and try it on all these different systems.

Paul: On all the different systems, right. So one of the things that made it possible for the browsers to move to the release train model is that they had the alpha channel and the canary channel or whatever it is they called it. In Firefox it's the Aurora channel. That's the super, actually that's not even the super beta stuff. They even have a nightly channel.

So they have like a million people using the nightly. And that tells them whether things are broken and how they are in fact in production. And I guess it's kind of the first evolution of feature flags.

Feature Flags and Continuous Integration

Edith: Yeah. I mean, so let's talk about that for a second.

Paul: We probably need to, aren't we gonna define things in this one?

Edith: Yes, this is the definitional episode, Paul.

Paul: Right. Let's start with what's a feature flag. Because I think we've already talked about six things that we haven't defined. But what's a feature flag?

Edith: A feature flag is basically the ability to, I'd say, hide code but yet have it public. So it used to be that everything that's shipped was visible. So a feature flag is just basically an if-then that wraps a piece of code and says it might be shipped, but nobody can see it until I, the software team, decide that it's visible.

Paul: Right, and so some examples of that could be in a browser that you change a preference and the preferences "use the new version of this." Or another example might be that a feature goes out in like Facebook that's shown to 1% of customers in Northern California who are in this demographic.

Edith: Yeah, and it's very liberating. So the very liberating thing about a feature flag is that if something's not working, you don't have to do a redeploy.

So to go back to the old dark days where you had all these disks out in the world, that feature flag is useless. Because you're gonna ship all these disks out to people and they're gonna install it. A feature flag is useful when people are consuming stuff as SaaS.

Paul: The way that I think about a feature flag is that it separates the act of shipping code from the act of actually turning on that code.

Edith: Yeah, deployment from rollout. So it's deployed, it's available, but it's not visible.

Paul: Right. So now that we know what they are, what were we actually talking about?

Edith: We're working our way through the evolution of software. So we'd gotten up to release trains. You should talk a little bit about how continuous integration fits into that.

Paul: So continuous integration, I guess to define it, it's basically the act of having an automated build server that runs every day or at some sort of cadence and tests that your software works.

So CircleCI, my company, makes continuous integration as a service. And what happens there is whenever you push to GitHub, we'll automatically check your code out, we'll build it, we'll run your tests on it, and we can also deploy it to your customers, which is called continuous deployment or continuous delivery, depending on exactly how you think about the world.

Edith: And you make this seem so simple, but it's really revolutionary. I mean, I remember the days without this.

Paul: Sure, sure. I remember, in my first job we had a daily build server and so we would find out every day whether or not the code actually compiled. Even though we could get it all compiling on our own machines, or some modules of it or whatever, compiling on our own machines, there'd be all sorts of build artifacts and dirtiness in the code that meant that just code didn't actually work when it was ready to be shipped to customers.

Edith: Yeah, or even worse, I remember it used to be a thing when you broke the build. It was very shameful. It was kinda, you'd have to sing a song or you'd have to do something embarrassing, and that just doesn't really happen anymore.

Paul: Right, right. My co-founder used to forget to add new files to Git. So it constantly wouldn't work, although it was perfectly working on his machine, there'd be a file missing somewhere.

Edith: The classic excuse of, "Oh, well, it worked on my machine."

Paul: Exactly, yeah.

Edith: Yeah. I didn't know about builds my first summer job. Not my first, first summer job, but I was programming Visual Basic. And I was making a war game, basically. It was for a defense contractor and I was trying to justify like a space-based laser versus an airplane-based laser.

And if you had all these missiles coming over from Russia. Okay, this is a long time ago. How many could you shoot down? The thing worked perfectly locally. And then at the end of the summer, they wanted it off my machine. I was just like, "Oh."

Paul: Okay, right, right. You need to put on a plane or in space or one of those things.

Edith: No, they wanted my actual files off my machine.

Paul: Right, yes. In order to put it in space or on a plane.

Edith: No, it was just a simulation. I wasn't actually controlling missiles when I was nineteen. Though that would have been cool.

Paul: I had this absolutely terrible release process at this place that I worked at. It was O2, which is a really large phone company in the UK and in Ireland, and I think they got bought by someone recently, but whatever. And they were launching this feature called i-mode. And i-mode is this massive thing in Japan that completely failed to take off anywhere else. It was amusing because everyone...

Edith: Are you saying it was literally big in Japan?

Paul: Literally. It was big in Japan. And so they tried to launch this in Ireland, and everybody in the whole world knew that this was going to be a failure, but that was how the process worked anyway.

And the release process was they had all this code somewhere that was going to be deployed and attached in servers, and when they did that, there's a manual, I was part of the manual team that manually validated with handsets, clicking through handsets manually, that this stuff actually worked. There was no automation. The entire release process had to be repeated dozens and dozens of times as they fixed bugs and they reconfigured, and it was just a complete clusterfuck. Everything that you imagine.

Edith: Paul, there goes our clean rating.

Paul: We never had a clean rating.

Edith: No, the dirty builds just sat up.

Paul: When I moved to doing software that had any sort of continuous release cycle, it was liberating, because you actually know the path from what you're doing on a day-to-day basis to it arriving at customers, and everyone, every developer in the place can know what that path is. And everyone knows how code actually gets shipped. You don't have to keep it in a continuous state of, "Oh, maybe one day I'll ship this," or whatever.

Edith: Yeah, I mean, I think to go back when you had these longer and longer and longer release trains, or even pre-release train, people got very lax. 'Cause it just was like, "Oh, I'm just gonna sit in the corner and code away." And sometime in the distant future it might ship.

Paul: Right. And then as you're coming towards the end of the quarter, everyone is hammering things out as quickly as possible. And maybe there's a few things that shipped early in the quarter that never got cleaned up or never got, you know, quite ready.

Edith: What do you mean by shipped right now?

Paul: So I remember some times where you'd merge something into master because you needed to get it tested, but it wasn't quite production ready. It's out with the beta testers or something like that. But in order to actually ship to hundreds of millions of people, you need to remove whatever hacky thing you put in place.

Test-Driven Development and Customer Expectations

Edith: Yeah, so I think one of the things that has driven continuous delivery is just the rise of test-driven development.

Paul: That's interesting. Yes, I mean, I think it's obvious. You end up in a situation where when you're trying to move to continuous delivery, you're saying, "Oh, I'm afraid." And we talked about the fear last week.

And in order to get past that fear, you need to have automated testing. And I guess in the old days there was manual release cycles, and you can't do manual release cycles ten, twenty times a day. It's just not possible.

Edith: Or you can't depend on having like a fleet, like to go back to the Microsoft example, you can't rely on having two QA engineers for every developer who will, you know, just fling out code and they'll find everything.

Paul: But even with two QA developers, you can't have two QA developers when you push code every five minutes.

Edith: Yeah. They'll never keep up.

Paul: Exactly, yeah. And then they'll never identify the chains that actually cause the bug and so on.

Edith: Yeah. So I think the things we've talked about so far, SaaS has unlocked continuous delivery, test-driven development has unlocked it. What else do you think is really a foundational thing?

Paul: I guess the other one is kind of this change in customer expectations. Customers, as a fact of some people doing continuous delivery, everyone else has had to do it. So once you start to experience it a little bit and you start to wonder, why do other sites go down for, you know, six hours on a Sunday night, like this doesn't make any sense. Why do I have to download new versions of this software versus clicking the update version, or why do I even have to think about this? And I think for the installable software market or the desktop software market, Chrome was that sort of killer app that everyone realized that this has completely changed the game.

Edith: Yeah, it's completely changed the game. I mean, I think even now asking people to download like a plugin seems too onerous. They don't want to do it. They just want stuff to work.

Paul: So I mean, it's almost tied to a conversion funnel. If you expect people to install 31 discs to upgrade to the latest version of Windows, people are just not gonna do it. You're not gonna achieve the conversion rate that you're expecting. There's too many alternatives to your apps that are out there.

Edith: Yeah, it's interesting. So Sean Byrnes, who's LaunchDarkly advisor, he wants to stream everything, including his OS.

Paul: He wants to stream his OS?

Edith: Yeah.

Paul: Yeah, yeah, that makes sense. Containers, something there.

Edith: Containers.

Lean and User Feedback

Paul: Right. So we were talking about continuous integration, we were talking about feature flags. What else is in this sort of space of continuous delivery or important things?

Edith: The belief that users always know best.

Paul: Interesting. So it ties to sort of product management and validation and all that kind of ecosystem.

Edith: I think there's been a steady realization that you as a company can have theories and users have facts.

Paul: Right, right. So you basically have to put out software. You have to continuously deliver software to even have any idea if it's a thing, that feature, that users want.

Edith: Yeah, and I think this ties really into lean. So the whole idea of less waste. It used to be, you know, you would make a massive release, and I'd say Windows 95 was huge. You know, they actually had like a theme song for it.

Paul: Yeah, no, I remember, it was "Start Me Up" or something.

Edith: Yeah, and that gave summer employment to interns for years, 'cause they had to install all those damn disks.

I think with lean there's this idea that, you know, I don't know everything, and instead I'm gonna break stuff down into little chunks and deliver those out and get feedback.

Paul: Right. Is that a cause or an effect of continuous delivery?

Edith: I think it's a flywheel.

Paul: Flywheel, what is it?

Edith: I think knowing that you can get feedback faster makes you want to get fast feedback faster. I think there used to be this idea that a product manager or CEO is a god or goddess and they knew everything and then the world was just waiting for them. And I think more it's, you know, as a product person you try your best and you put something out and then you see if people like it.

Paul: So I think for anyone who's built any sort of lean startup or lean product or something along those lines, there is a sort of careful line between we know best and user feedback. And, you know, famously there's the kind of Apple-Google distinction of, Apple is just like great designers and they tell the world what they want, and Google A/B test the color blue forty-three times or whatever they did there. But I think for most startups you kinda have to tread this line very carefully of knowing what you want to ship and knowing how you're gonna change the world versus actually getting that feedback and actually seeing if in the real world people actually respond to it.

Edith: I think that goes back to you have to have a theory

Paul: You have to have a theory that you're testing.

Edith: Yeah, I mean, I don't think any startup can afford to say I'm gonna build ten thousand products and ship them all and see which one works. But they can say I'm gonna break my idea down into little

Paul: chunks and iterate and deliver them and see how people like them.

Edith: Yeah, I mean, and this is the whole genesis of lean versus fat startup.

Paul: So how did people do lean before continuous delivery?

Edith: That's a really good question, Paul. I mean, you can still do lean with a weekly or even monthly release. I mean, so I was at an Internet of Things startup and we did a monthly release.

And every month, so I was the product manager, we would think about, you know, what are we trying to prove this month? Is it that we want to do better onboarding? Is that we want to try to monetize more?

Paul: And so you had to keep things along a particular theme or something to manage the complexity of it?

Edith: It was more, you know, we were small. We were a small startup of eight people and I didn't have a huge staff. So I would say, okay, what do I really want to accomplish this month? And it was a really good forcing function.

Paul: So how did you, in the monthly release concept, how do you deal with bugs? Something goes wrong.

Edith: Oh gosh. So this goes back to why we had a monthly release, was it was an Internet of Things company and we had physical firmware.

Paul: That's what you shipped, actual firmware?

Edith: We had firmware, we had middleware, and we had software. So I'll tell you what it was. It was a Fitbit for plants. So it's basically a sensor that you put in a plant and it'll tell you, is it getting enough light, is it getting enough water?

Paul: This seems like one of the tropes that people make, like, it's a Fitbit for plants, it's an Airbnb for dogs.

Edith: We were before there was Fitbit, because I actually, when Fitbit came along, I was like, "Oh, that's cool."

Paul: Now I can't have done this for you.

Edith: I was like, "Now people understand what we do." And it was actually one of the things that made me happiest about my Fitbit for Plants company, which calls EasyBloom, is I was describing it to Oren Teich, who sits next to me. And I had had this persona that, you know, gadget guys in San Francisco would love this. And Oren's like, "Oh yeah, we had 10 of them at Heroku. We had one in every plant." I was like, "Awesome." It made me so happy. It was like when you meet one of your personas, they are exactly your persona.

Paul: So I saw this thing in the Science Museum in Dublin, or in the Science Gallery in Dublin, which was a plant that was connected via an iPhone that was sitting in the plant cage with it to the stock price of, I forget what it was, but the equivalent of Home Depot or something, where you would actually buy the plants. And whenever the stock price went down, the plant would be deprived of water, would eventually die and would automatically order a replacement for itself from the Home Depot, thereby causing stock to go back up.

Edith: This seems very Restaurant at the End of the Universe.

Paul: It was, I mean, it was totally a science project. It was intended to be amusing and make you think about the world. But I mean, it is very literally a fast feedback continuous delivery cycle.

Edith: Yeah, did you read any of The Hitchhiker's Guide to the Galaxy? Yeah, with the animal that runs around with the fork and knife, like, "Eat me, eat me."

Paul: Oh, the cow in the Restaurant at the End of the Universe, right, yeah.

Edith: Yeah. So back to why we had a month release cycle is because we had actual firmware. And if you've ever shipped something out to a lot of gardeners, like, that in and of itself is a feat, to get them to use it. And then they didn't want to ever update their firmware.

Paul: Of course not, no.

Edith: Because it's a, you know, it's a pain. So we had to have a month-long release cycle because if we made any change to the firmware it had to be, you know, good.

Paul: Okay. But I presume they wouldn't even release it every

Edith: We would try to release the firmware as little as possible. The middleware as little as possible. The website, we had a lot more latitude.

Paul: Gotcha, gotcha.

Edith: And then we had all these units in the field that we had to, so this goes back to continuous delivery. We couldn't guarantee that they would be at a certain level of firmware.

Paul: Right, right. 'Cause they would never be upgraded. They're in a plant somewhere and no one's gonna plug in the USB cord or whatever to their plant to upgrade the firmware.

Edith: Well, they would do it at their own schedule, which was basically the same as me and my iPhone, which is never.

Evolution and the Wright Brothers

Paul: So as you were talking about the plants there, I started thinking about this parallel to sort of evolution and survival of the fittest and the sort of continuous feedback kind of idea there.

It seems like something like continuous delivery is something that humanity has sort of been doing in various different facets for a long, long time. And it's just, you know, now is the first time it's being applied to software.

Edith: Yeah, I mean, so we touched on this the other day, but newspapers used to be very good at continuous delivery.

Paul: Like they'd have like three editions or

Edith: I think it's a very human thing to just crave the latest and greatest. To not want to settle for the past, to want to improve, to want to get better things.

Paul: I guess it's fairly obvious then that the cost of making those updates is always involved. So I'm thinking here of, you know, that they did an update to Star Wars and Apocalypse Now and that sort of thing. And they did those updates twenty years later because doing an update of a movie or of an album or something like that is a very, very expensive thing, and you only want to do it for something that is so massively successful or there's some kind of update for it. Or there's some kind of demand for that update or some kind of ROI on the update coming out.

Edith: Yeah, I mean, so it's fascinating, because you know who has a huge ROI on software is Wall Street. So I talked to somebody who had been in technology at a Wall Street firm, and I am not gonna say who, but they were doing continuous delivery ten, twelve years ago. They didn't call it continuous delivery, but they were, a lot of Wall Street is just basically trading on proprietary algorithms.

And so they were shipping those algorithms like multiple times a day. Because that's how they made their money. If they had a slight edge, they made more money.

Paul: Right. It reminds me of how the Wright brothers. Have you heard the story about how the Wright brothers, you know, why they got to powered flight first?

Edith: Because they cared more?

Paul: No, no. So this is a really, really interesting story. The Wright brothers were the worst funded of all the companies that were trying to get to powered flight.

Edith: And there was the original lean startup.

Paul: Right, right. They had dozens and dozens of competitors, all of whom were way better funded than them. And basically what they did that was different was they would iterate really, really quickly. So they would try an experiment, the plane would break, they would fix it and try again, like multiple times a day.

Whereas their competitors could, you know, I say release, but you know, release once a week or try a new experiment once a week or once a month. And so continuous delivery is the thing that made the Wright brothers, who otherwise just wouldn't have had a shot at all at being the first to powered flight, that they were the ones who got there first.

Edith: Yeah, and I think continuous delivery is really, what it really unlocks is just more iteration.

Paul: I think iteration is basically at the core of success and innovation anywhere that innovation is going to be required.

Edith: Yeah, and I think that's really key, and we touched on this in the past episode. There's some places like Wall Street where innovation, it is very much worth...

Paul: Right, it's life or death.

Edith: Or millions or trillions of dollars. Same diff. And then there's other places where you really don't want innovation. Like, you know, when I go to pump my gas, I just want that gas to come out.

Paul: I'm not sure I agree with this. Oh wait. Yeah, so those gas machines or the things that you put your credit card in or whatever, they're just about the worst UX I've ever seen. So if they had, you know, some sort of rapid iteration, or if they gave a shit about customer experience.

Edith: But why should they care? Nobody's gonna go to a gas station because they have a slightly nicer pump. Your thing is all about incentives, but are you really gonna say I'm gonna go a mile out of my way 'cause like the UX is better when I get my gas out?

Paul: No, no, no. I think you're totally right. I don't think that a sort of cost-benefit analysis would reveal that there's a benefit here. But it would be better if they actually iterated on that software.

Edith: So going back to the financial services firm I was talking about. He also said that they had like insane memory leaks. And so he tried to fix the memory leaks and his boss is like, "Well, nobody cares. We're not gonna make any more money if we, you know, fix memory." And he's like, "We restart the servers every weekend. Works fine." And he's like, "But..." And his boss is like, "It's not gonna make us any more money."

So everything in software is a trade-off.

You could skew towards innovation and go towards continuous delivery, or you could say, you know, "Nobody's gonna buy more gas if that gas station has a new UX every five minutes."

Paul: Well, theoretically they could. People could A/B test their UX and, you know, try to offer messages to suggest that they fill it up fully or that, you know, they upgrade to, you know, explain to them why the higher octane... I actually know nothing about cars, I've never owned one. But higher octane something, something, something is better.

Edith: So wait a second. You made fun of me for not updating my phone and you've never owned a car.

Paul: Oh yeah, no, I have a life philosophy of owning as few things as possible. I have a bike, a phone, a computer, and a bed. And I think everything else I could live without.

Edith: My bicycle was stolen this weekend.

Paul: I feel your pain.

Edith: I told myself that bicycle are cattle, not pets.

Paul: I lost a couple of bikes and I love those bikes.

Edith: Cattle, not pets, but I still miss that bike.

Distributed Systems and Services

Paul: So cattle not pets is something that comes up in the distributed systems context quite a lot. And I guess in continuous delivery as well. So what effect has the fact that we're all making distributed systems now, versus, you know, a piece of downloadable software, what effect has that on the rise of continuous delivery?

Edith: Well, that's specifically why I kinda try to make that joke, is that I think a lot of continuous delivery is the belief that hardware is disposable, that you don't have to run your own data center to get good hardware to get good software out there.

Paul: So the rise of the cloud is the... So why is that a catalyst exactly?

Edith: There used to be this theory that you had to be very vertically integrated, for lack of a better word. That you used to have to, even after you bought into SaaS, you would still have your own data center to run everything.

Paul: Like Salesforce.

Edith: Yeah, like Salesforce. And I will call out Salesforce as they were the first company to explicitly get people to buy into the idea that software in the cloud was a good thing. That was their original logo.

Paul: Yep, yep. Still their logo.

Edith: Yeah. But now they emphasize the cloud more, a lot. It used to just be the software with those slash through it. And then people had this belief that that also meant no more software salespeople, and that was completely wrong. So I mean, what do you think about it?

Paul: So I think fundamentally, without the ability to use, I guess it is hard to update a monolith.

Edith: Yes.

Paul: And fundamentally the thing that is created by SaaS is essentially you have a bunch of outsourced services, whether it's your hardware provider or your infrastructure provider or your payment provider or your database provider or whatever. They're all just separate services, separate departments with kind of defined interfaces. And the rise of services, I think, is one of the major catalysts of continuous delivery.

Edith: Yeah, I completely agree. I mean, it used to be you had to be very vertically integrated, like back to my Internet of Things plant company. We built our own billing system. Not because we wanted to.

Paul: What year is this?

Edith: 2009.

Paul: In 2009 you built your own building system?

Edith: Well, we tied into PayPal, but we still had to keep track of who'd subscribe to what.

Paul: Right, okay, yeah, I understand.

Edith: 'Cause we tried to upsell people to different services.

And funny side story. I love testing. What I found is that if we put a PayPal logo on, people were much less likely to buy from us.

Paul: Interesting.

Edith: And this is in 2009, so things might have changed. So I tested with and without a PayPal logo when people checked out. And what people said was that the PayPal logo, because I asked them, yeah, it made them think that we were flaky or fly-by-night company. That we weren't able to process our own credit cards. Because at the time, in 2009, there are a lot of flaky companies who use PayPal.

Paul: I read a similar thing by Patrick McKenzie, and I forget whether it's his Bingo Card company or one of the other companies that he has, but they had a PayPal logo and he added a Google Wallet logo, or Google Payments logo or whatever it was, and it increased conversion across the board. Except it didn't increase... No one actually used Google Payments, but it increased conversion across the board.

Edith: Oh, funny.

Paul: Yeah. Even amongst the PayPal users, because it made them feel like this is a real company that has like multiple different payment options. One of them is by Google, and not just shitty old PayPal.

Edith: Yeah, at the time, and no offense to PayPal, at the time though it was seen as the refuge of people who couldn't get a merchant account.

Paul: Yeah. I mean, that's how I view it now. If I go to a site that only has PayPal, they could have integrated Stripe in twenty minutes. Why are they using PayPal?

Edith: Yeah, Stripe is the best. I mean, we use it at LaunchDarkly. And it's one of those things where it used to be you had to build so much to just build anything. Just to build a very simple feature, you had to build all this infrastructure, like you said before about having to build your own build system.

Paul: It seems like there's a connection there between, now that all these services exist that you can use, there's now actually time to focus on what the user cares about and giving a really good user experience and A/B testing the onboarding flow. Whereas, you know, before, if you had to keep the machines up and you had to build your billing and you had to make sure that the money was still coming in and all that sort of thing, you didn't really have time to put, or at least it became outrageously more expensive to put time into the actual user experience.

Edith: Yeah, it used to be just you had so much to do just to do anything.

The Consumerization of the Enterprise

Paul: One of the things that has marked this era of enterprise software is that software isn't shit anymore.

Edith: There goes our clean rating for sure.

Paul: Enterprise software used to be sold top down, and the users never got to play with it before it got installed, and it was always just like complete shit. And there's this movement called consumerization of the enterprise.

Edith: Yeah, I mean, that was the whole premise of TripIt. Concur acquired us for $120 million because they're like, "Wow, these people are selling bottoms up."

Paul: Right, right. And people actually care, these people actually care about the quality of the software.

Edith: Yeah, I mean, at TripIt, so I was product director at TripIt. People would say, "I love TripIt."

Paul: And not, I presume, Concur.

Edith: Not so much, no. Not unless you were the CFO. You know, because that was their buyer persona, as a CFO.

Paul: Gotcha.

So presumably the reason that there's an overlap between the rise of kind of SaaS enterprise software and continuous delivery being a thing that people want in the enterprise is that high quality software, or high usability in particular, but I think high quality is a more general case, has become in demand since users started getting used to like really high quality software in the consumer space.

Fad or Status Quo?

Edith: So I guess kind of my thesis, and maybe this is, I think this might be a topic for another show, is so right now continuous delivery is the hotness. And at least that's my thesis, that what you said, innovative companies want to be doing continuous delivery. So is this a fad? Are we gonna see a backlash? I mean, I think that's the topic for

Paul: For future?

Edith: Yeah.

Paul: I think we can deal with it now. No, there's not gonna be a backlash. It's just gonna become the status quo.

Edith: So I'm gonna go pump my gas at a gas station that, you know, is continually updating the UI.

Paul: They're definitely going to be continuously updating the UI because they don't want to, you know, when they eventually have a new BD deal that means that they have to sell ads or something on the gas pump or in flashing lights, you know, they're not gonna want to send humans out with their USB sticks to upgrade it.

I don't think it's in the short term, but I don't think we're gonna see any backlash against continuous delivery as a concept.

We're already seeing backlash against SaaS and against things that have like auto-upgrading, you know, security flaws basically. I think that if people back away from SaaS or back away from, you know, things with various security vulnerabilities, they're gonna try to say how do we get continuous delivery to keep working in the model that we're in.

Edith: Wow. That's a really broad statement. And this is why I like talking to you, Paul, as you lay out this brave new world, and I like thinking about it.

Paul: Well, the problem I have is I don't actually understand the old world all that much anymore. So living in San Francisco, it's, yeah, everything is kind of this brave new world. I don't own a car because it's on-demand.

Edith: I don't own a car. But when you get City CarShare you still have to go to the gas station.

Paul: Sure, sure. But I guess what I'm saying is more around living in the future. You know, you rent cars by the hour or you press a button in your app or, you know, something comes in your watch or whatever it is.

Edith: Well, the streaming world. I mean, so that's my whole thing. It used to be like I had a Lonely Planet guidebook for San Francisco. Now there's Yelp.

Paul: Yeah. Or just the Internet in general.

Edith: Well, just the Internet in general, where there's continual updates and you don't have to have a physical artifact. Or I remember when I would tape songs I liked off the radio.

Paul: Oh wow. No, I remember that in high school.

Edith: Yeah, 'cause otherwise you didn't know when you would hear it again. And now if you're out and you hear a song you like, you just SoundHound it and you download it.

Paul: Okay, I haven't heard of that one. SoundHound is a Shazam?

Edith: Yeah.

Paul: Okay. So on that note, there's a backlash happening against Apple Music in the last couple of days.

Edith: There's always a backlash in Apple Music, it's just like, what is the current? I mean, there's the whole U2 thing, because nobody wants Irish people on there.

Paul: That's right, that's right.

Edith: And then there was the whole Taylor Swift thing. Like, Apple Music is continually like, "Who can we piss off this week?"

Paul: So I was gonna try to make it about continuous delivery, since I think that that's what this show is about. Apple is one of the least continuous delivery companies out there.

Edith: How can you say that when they forced U2 on everybody?

Paul: Didn't it take him a while to sort of step back from that? It was like weeks before you could even release, or you could not have U2 on your

Edith: I still have U2.

Paul: Well, that might be... I do as well. I don't actually know how it got there, but it's there.

Edith: They delivered it. They just don't, so there's continuous delivery, but they don't know how to roll it back.

Paul: Right, right. I think there's a whole show to be told about rollbacks and that kind of thing. And that's probably something not to get into now. What else was there in the kind of overall state of things you need to know about continuous delivery?

Edith: So we talked about things that fueled it, which was the moving away from installing software. The moving away from having to own a platform. The rise of test-driven development, the belief of lean that trying out little units of value was better than big units.

Paul: Yep. And then there was SaaS and there was Salesforce and there's always Apple. Yeah, I think that's the majority of how continuous delivery got to where it is.

Edith: Yeah, I think continuous delivery, and I think you put it very well, people demanding more, like you said, people demanding more from their software. Like, people demanding a better experience, faster, quicker, now.

Paul: Yeah. So that was tied to the consumerization of the enterprise, the rise of actually useful enterprise software and not shit.

Edith: Yeah, or just, you know, and I think what fueled it is the consumerization of enterprises, people saw all the wowy consumer products and said, "Why am I going to work every day with this thing that's awful?" I think that's all we have time for today, Paul, but we're excited to hear from more of our listeners about what they think about the show. And you should tell them about our new Twitter handle.

Paul: Oh, we are @continuouscast. Alright, @continuouscast, twitter.com. And yeah, we wanna hear what you guys wanna hear us talk about.

Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of CircleCI, and Edith Harbaugh of LaunchDarkly.

Edith: To learn more about Heavybit, visit heavybit.com. While you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.

Subscribe to Heavybit Updates

You don’t have to build on your own. We help you stay ahead with the hottest resources, latest product updates, and top job opportunities from the community. Don’t miss out—subscribe now.