Ep. #11, Pricing for Developer Tools

Peter van Hardenberg: There's a tremendous amount of basically wasted and unfun energy that goes into making sure you never make any mistakes. My favorite thing about continuous delivery is the ability to make mistakes and move on with them.

That separated a slow-moving core from a fast-moving periphery. Pricing is product. People really understand your product through its price. Deploy speed becomes one of the most significant factors to the success of your business.

Paul Biggar: This sounds like it'd be tough on a Game Boy cartridge.

Hi, I'm Paul Biggar, founder of CircleCI.

Edith Harbaugh: I'm Edith Harbaugh, CEO and co-founder at LaunchDarkly.

Paul: And you're listening to To Be Continuous, a podcast about continuous delivery and software development.

Edith: You can get in touch with us anytime at our Twitter handle, at ContinuousCast.

Paul: The show is brought to you by Heavybit. To learn more, visit Heavybit.com. And while you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.

In this episode of To Be Continuous, we have Heroku's Peter van Hardenberg talking about continuous delivery in the context of organizational structure, databases, and pricing.

The Freedom to Make Mistakes

So Peter, it's traditional to ask you, what do you like best about continuous delivery?

Peter: Well, I'm increasingly old and I've done a lot of jobs, and so I remember delivering Game Boy cartridges to the market. And a Game Boy cartridge is a physical device that goes into a Game Boy, hence the name. And that process involves physical manufacturing, and you get one shot.

And so there's a tremendous amount of basically wasted and unfun energy that goes into making sure you never make any mistakes. My favorite thing about continuous delivery is the ability to make mistakes and move on with them.

Paul: Nice.

Edith: This would be a great time for you to introduce yourself.

Peter: Yeah, hi, my name is Peter van Hardenberg. I'm a Heroku old-timer. I've been with Heroku about six years now, and I co-founded the Heroku Postgres team and have built a lot of products since then.

Edith: So it's funny you're talking about the Game Boy, 'cause Paul and I were just talking about that.

Peter: About Game Boys.

Edith: Well, 'cause I said one of the things that had fueled the rise of Zynga was continuous delivery.

Paul: That was the first thing we found about continuous delivery that was actually bad.

Peter: So I guess they're probably still continuous delivering somewhere in the great startup in the sky. They're still around, aren't they?

Paul: They're still around. They're still public and

Peter: FarmVille will never go away, I'm sure.

Paul: I think they've moved into other lesser things.

Peter: Developer tools.

Paul: Imagine.

Edith: So what do you like?

Peter: Gamified developer tools. Well, actually, it's how all developer tools are becoming now.

Edith: GitHub is gamified. How many check-ins do you have? Can you make a nice pattern out of it?

Peter: It's a bit like a game, except if you're losing the game, you lose your job.

Paul: Or you just can't get new jobs.

Peter: Well, that's actually probably more accurate, yeah. Yeah, it'd be a weird company if they let you go because you didn't have a long enough commit streak.

Paul: On the open source projects that weren't your job.

Peter: I like that, actually. Let's make that a game.

Paul: We were talking in the last episode as well about open source funding models, or the anger over lack of open source funding models.

Peter: That's a tough one.

Paul: That would be an interesting one. You get fired if you don't have enough contributions.

Peter: Yeah, just put it on your quarterly review. Current GitHub streak.

Paul: Let's put this suggestion on Hacker News and see if people agree with it. I can pretty much picture how this is gonna go.

Peter: I try not to read the comments.

Edith: So what do you think is liberating about the ability to make mistakes?

Peter: Well, I mean, more important than the ability to make mistakes is the ability to try things, right? I mean, I'm so accustomed now to being able to see in great detail, because of a SaaS model, sort of the reality of what my product is in the market. And then vice versa, I can turn that around and say, well, basically live and in real time, what's happening if we put something out there?

Do people care? Does it help with adoption? Is it suddenly throwing a bunch of errors? And if there's a problem, you can roll back very quickly.

You know, we run a database product, and so availability is really, really important, but only for a very small core of the service. People are generally pretty understanding as long as your sort of problems and outages are brief, small in sort of blast radius, and quickly remediated.

And I think that they prefer to get features versus stability sort of at the periphery, and at the core they need stability over features. So, you know, if you're a Stripe, right, you need to keep the payment processing online, but if some ancillary piece of your API has a problem, it's probably not as big of a deal.

Paul: Dashboard is less important than the payment going through.

Peter: Well, and responsiveness and progress are so important. And I think because we're in such a changing market, the ability to drive forward is at least as important as the ability to keep things stable, but you can't do both in the same place, basically, right?

I think continuous delivery is cool because it separated sort of a slow-moving core from a fast-moving periphery.

Continuously Delivering Databases

Paul: So when you're creating a software product, you really want every single component of it to be continuously deliverable, which means you want to be able to migrate from the thing that you are doing now to the new thing. And so there's an interesting thing about databases, because databases, I think, were fundamentally not built for this model.

Peter: They're heavy.

Paul: Right. We had a problem recently where we deleted an index. And whoops, that index was in use. And we thought it wasn't in use, but it was in use.

Peter: You should have been using Heroku Postgres, it would have told you.

Paul: Oh, so that's interesting. So the thing that I wanted was to be able to, you know, partially delete that, or, you know, let's run all the queries without that, and then after an hour I can actually delete it. What does Heroku Postgres offer you?

Peter: Well, so we have a number of diagnostic tools where you can see exactly how often an index is being used. In addition, though, next time you're trying to delete an index, wrap that in a transaction, so say begin, then drop the index, then do whatever, and then say roll back. And that way you can actually test as though you didn't have the index, because Postgres MVCC model means that all that stuff is totally transactional.

Paul: So when you say do all the stuff in between, what do you mean?

Peter: Oh, whatever queries you wanted to run. Also, and we're really getting into kind of like minutia and inside baseball here, but you can actually disable indexes specifically sort of at a session level as well.

Paul: Oh, interesting. This is all Postgres or Heroku Postgres?

Peter: Yeah, that part's just Postgres. And then we built the tooling around it so that you can do these things kind of the easy way, but I'll tell you, if you read that 3,000 page Postgres manual, you'll find some real gems.

Paul: Well, so part of the problem might be using a thing that's not a real database.

Peter: Oh, are you not using a real database? Oh, that's right.

Paul: I mean, we are using a real database. We're using many databases, but that particular one was not on a real database.

Peter: Well, you know, even baby databases grow up to be real databases someday.

Paul: Well, fingers crossed that that particular database that starts with an M and that we shall not discuss.

Peter: Oh, we don't like to speak ill of anything.

Paul: No, no, not at all. Especially Mongo.

Edith: No, you said the word.

Peter: You said the M word. So coming back to sort of continuous delivery of every part of the stack. Yeah, it's interesting. I mean, we've had some interesting challenges with databases in particular. And I actually gave a talk at Heavybit a while ago about kind of how we treat databases less as part of a service in sort of the classical sense, but more like a factory.

So basically what happens is when people request databases from us, we stamp one out, and then, you know, we try and keep it pretty low entropy from that point on. So we have a lot of tooling and support services around the database, but the actual database itself, once it's given to a customer, we mostly try and keep our hands off it.

Now, if there are security issues or, you know, performance issues or whatever, we'll go in and we'll kind of warm it up and do some work on it. But even still, we mostly prefer to actually replace the database. So what we do is we create a replica, bring it up to speed, and then we transition the load over to it. And that's kind of a form of continuous delivery for data services, which has been surprisingly powerful as an abstraction.

Edith: So you're basically treating them as cattle, not pets.

Peter: Oh, absolutely. It's all about the cattle and model. And when you're farming a very large crew of livestock, you can't really afford to give each one their own personality.

Paul: This reminds me a lot of, I think, what started out as an IMVU way of doing data migrations. Or at least, sorry, of doing schema migrations without locking a whole table. Well, for our people at home, I'll describe this.

So when you want to add, let's say you want to add a new value and it has a default, to a particular table in your schema, there's the possibility of that, if it's a particularly large table, for it to be locked while that happens. At least this used to be the way. I think modern databases are much better at this now.

Peter: I believe for some default values, it's now the case that you don't need to do a table lock, but that might be pushed until 9.6 for Postgres, at least. But yes, that's a common and challenging operation.

Paul: And so what IMVU started doing was instead of adding to a schema, they said that for every table, that schema is immutable. And instead they created a new table, and then they migrated the data from that table. So every time you want to read from the user, you're reading from user table 38 at the moment. And then you start to read from user table 39. If the user isn't in user table 39, you fall back to 38.

Fall back to 38, find, migrate it, write it into 39, and then do the thing. And then you have a background process.

Peter: If you have any listeners who are listening from the world of the internet and would like to try implementing this technique for themselves, Postgres has something called schema search paths. You can actually basically make each schema its own version, and then you have a search path. So if the table wasn't in the newest version, it would use the last version's search path.

Paul: So, wow.

Peter: You could set a version on your code, then basically, as you make new versions of the tables in the new schema, you can kind of move them forward. This actually sounds like it would probably work.

Now, it's really easy to say that from sitting around a table here in beautiful San Francisco. But if you have any listeners out there who do give this a try, have them send me a note.

Paul: I was not expecting this to be a session about continuously delivering databases and the intricacies of modern versions of Postgres.

Peter: You know, I have to say it's one of the hardest things, is basically how do you deal with your data?

I mean, data is sort of fundamentally heavy. I mean, I think about it as having mass, which is to say, you know, if you've got a few terabytes of data on a disk somewhere, just getting it out of that server and into another server is gonna take hours.

Paul: Right.

Peter: Just querying the data, loading it all into memory, that's gonna take time. And so this is really a challenge for continuous delivery, and people have tried all kinds of schema versioning and replication strategies and sharding strategies.

But, you know, the old question: okay, you deploy a new version of your code and it needs the new column. But the new column's not there yet. Okay, so you move the data to the new column and then you deploy your code. But then the old code doesn't have it.

Paul: Right, right.

Peter: And these are hard problems, and no one's really done a great job of answering them. So far, sort of the state of the art is, you know, you continuously deploy the new version of your code that supports both schemas. Then you continuously migrate your database, and then you deploy yet another version. Now without continuous deploy, this is, you know, even harder.

Paul: This sounds like it'd be tough on a Game Boy cartridge.

Peter: Oh yeah, let me tell you. I think we had 4K of RAM or something like that. So at least it was fast. But yeah, I think data and continuous delivery are both challenging interactions for sure.

Paul: Well, we end up with this problem in the testing world when large customers, in particular large enterprises, use the old way of testing, which is you actually use a dump of your production database as part of your test suite. And one of the major challenges we get when we're looking at older companies who are coming to the new world is, you know, your server's over here, my server's over there, and I want to get a five gig database dump onto your server to run my tests. How do I do that?

Peter: Right. And it's trying to like move a bathtub through a straw. So how do you do that? You just wait, right?

Paul: We wait until they stop doing that, and then they can become our customers.

Edith: So do you turn customers away then?

Paul: I mean, we don't turn them away, but we don't really support that use case. They're gonna have a miserable experience.

Peter: Yeah. I mean, there's data migration again, right? Just coming back to the physics problems, there are things that are getting better. So,

Paul: The speed of light is improving?

Peter: The speed of light has gone up thirteen percent in the last year. Did you know that?

Paul: I don't know if you're joking. I presume you're not.

Peter: It absolutely has not. Yeah. They call it a constant for a reason. I suppose it could.

Paul: I thought that might be something about, like, you know, just

Peter: dark matter. No, but it was very convincing the way I said it was.

Edith: Is this Hindenburg's principle?

Peter: Yeah, it's uncertain.

So anyway, the logical replication scene is getting better. Amazon's just launched a new service. It's three letters long.

Paul: Really?

Peter: Yeah. Oh God, hang on. DMS, Data Migration Service.

Paul: Okay.

Peter: And so through black magic and probably by not working very well, it will move all your data from one database to another in sort of a streaming online way, which is really cool if it works. But, you know, moving from Oracle to MySQL or other people.

Paul: Oh shit. So you can talk to this service as if it's MySQL and it will pull it from

Peter: The way it works is you basically set up a migration job, and if you think about it, it basically queries one database and writes to another. And it's a little bit of a hack job in the sense that, you know, you have to run like a desktop client if you're moving from a local machine. It's all very complicated and kind of baroque, but

Paul: So it's an Amazon service?

Peter: It isn't... Oh yeah.

Paul: You've used Amazon before, I gather.

Peter: But yeah, no, it's what everybody's always wanted, which is to move their data from one place to another without having to think about it. And of course it's never gonna work. But, you know, it'll help, it can help.

Things could be less terrible. I think ultimately that's what got me into databases, was the understanding that no matter how impossible it would be to make things good, at least we could make them a little bit less bad. So the soul crushing misery of writing software could at least be slightly less crushy.

Edith: I feel like writing software is one of the greatest joys in life because you get to create something from nothing.

Peter: So you don't work in databases.

Paul: I'm not sure you work in software.

How Continuous Delivery Went Mainstream

Edith: So what are the biggest changes you've seen in continuous delivery?

Peter: Oh, ubiquity, certainly. It's rare to meet someone these days that... Well, maybe I'm just lucky.

Edith: You live in San Francisco.

Peter: Yeah, I live in San Francisco.

Edith: And you hang out at Heavybit.

Peter: And I hang out at Heavybit with a bunch of other crazy people. But I guess I would say growth then, right? I mean, we're seeing large enterprises practicing continuous delivery. We're seeing, you know, small shops practicing continuous delivery. It's not sort of a bleeding edge concept that only sort of a small cadre of like sort of avant-garde developers are doing.

It's really, I think it's going to the mainstream. Which is exciting, because, you know, the thought of going back to the old ways of doing things is sort of terrifying.

I mean, I was trying to come up with like a sort of a mental model for how the cost of software integration is changed by continuous delivery. And I'll spare you all the math, but I actually did do some, I promise. And basically I concluded that if you believe that more complex changes require more testing and that there's more cost from integrating those changes, which I think is fairly uncontroversial,

basically deploy speed becomes one of the most significant factors to the success of your business.

So your ability to put software out there, the amount of time it takes you to go from this is good code and we're ready to deploy it, to this is out there in the world, the shorter you can make that time, the faster everything else goes. Because if that time is long, then you're gonna have a problem. You're gonna have many changes and it's gonna be a lot of that combination.

And it's gonna create an incentive to bundle more and more changes together. Which then means you'll take more manual testing, and there'll be more integration overhead. So you'll put more checks and balances in place. And you get kind of a horrible feedback loop.

Paul: This is Firefox 4.

Peter: This is Firefox 4?

Paul: And many, many.

Peter: Well, and this is all the classic enterprise software. This is basically how you end up going from, you know, hacking away on your laptop to doing annual releases.

Edith: Annual or semi-annual or biannual.

Peter: I mean, how often is there a Windows version?

To be fair to them, it's a pretty big product. Though I did actually...

Paul: I think it has a lot of users as well.

Peter: Yeah, were you in the Windows 10 beta? I was in the Windows 10 beta. It was kind of neat.

Paul: I haven't used Windows since 2003.

Peter: You should give it a try. They're making a bit of a comeback. It was kind of cool being in the beta, because it felt a lot like a continuously delivered software package.

Edith: They wrote a blog post about it. Like, they actually had different rings of beta releases.

Peter: Yeah, yeah. And so I tried to get to the innermost ring I could, because it was just a, you know, a game laptop. I was fine with it if it got trashed. And it's just kind of fun to be close to the middle of that.

Paul: But did it get trashed?

Peter: Just the once. But I think that might have been my fault.

Edith: Yeah, it was good. In the article they talked about how they stopped actually giving release dates. Because they said giving release dates would just cause them too much angst. Like, 'cause if they said something had to come out on March first, and they would defer features or, like, split up features.

Peter: You can cut scope or you can move dates. Those are your options.

Paul: You can just delete dates, and then you're sorted.

Peter: Right. Well, yeah, I mean, it was funny. I was talking to some engineers at Heroku recently about ship dates, and I've been on death marches before, and the games industry is quite famous for them. And I was saying, well, surely you must be opposed to having sort of shipping dates for projects in the early stages. And I was actually surprised to hear that they didn't.

I think that maybe the pendulum swung so far the other way that to some extent, I think creating a date kind of gives you the focus and clarity to

Paul: There's certainly benefits to shipping dates.

Edith: We had Kevin from Acompli and he said they shipped every Friday.

Peter: Yeah. I mean, that's not really continuous, I would say, but it's fully on him. It's on the right path.

Edith: Well, he was mobile, so that's when they cut the build to give to Apple.

Peter: Yeah, you know who doesn't have continuous delivery?

Edith: Mobile.

Peter: Apple. I mean, if you're on Android, you can just ship all day long. You probably shouldn't, but you could.

Paul: No one'll notice anyway.

Peter: Oh, Android users exist.

Paul: I know, but we're in San Francisco. Hang on. The audience can't see it, but for the record, I am holding up an Android phone.

Oh, wow.

Edith: And you're wearing a LaunchDarkly t-shirt.

Peter: I am wearing a LaunchDarkly t-shirt. What an amazing coincidence. It's a beautiful shirt. If you don't have a LaunchDarkly t-shirt, how do you get one, Edith?

Edith: Info at launchdarkly.com.

Peter: That's great. You should get one of those.

Paul: Yeah, and if you need feature flags as a service, also launchdarkly.com.

Peter: This is what I've heard. Yeah. What are feature flags as a service?

Paul: Oh, we just do this every episode.

Peter: Oh, that's fantastic. Well, it's not really important.

Paul: If you're a listener of the podcast, you will not be unaware of what LaunchDarkly is or how feature flags as a service work.

Peter: That's great.

Edith: So you came out of the gaming industry, you saw the death march, and then you just said something interesting about how the pendulum might have swung too far.

Peter: Well, I think, I mean, for me personally, and maybe my own pendulum swung too far. Having been in kind of a very conservative and very kind of, you know, single release, get it right kind of world, coming to the world of sort of online service software, I really appreciated the ability to try to experiment, to, you know, put things behind a feature flag and give it to a few users or give it to a lot of users and to do those kinds of rollouts. I think I got so used to that that maybe I kind of came to reject the idea of specific dates in general and maybe took it too far.

I mean, I wouldn't say that I'm happy to sit around for a year on end, but I think I'm coming to realize that dates are valuable scoping tools. And you can always slip a date, right? I mean, you can't always slip a date, depends on your partners, but you can usually slip a date if you have to, but it's a helpful and clarifying tool to set you on a path towards delivering a larger project.

Paul: Well, I think that there's a lot of people who came from the school of, you know, we'll ship it when it's ready, who maybe didn't have the sort of constraints that actually allowed them to ship more quickly.

Peter: Yeah, and ready is impossible to achieve if you don't have something that's really forcing you to converge.

And you see these kind of... I've seen friends who had startups that kind of tried to take that mentality, but because they didn't have that date, they didn't have that drive and pressure to get out the door. Maybe continuous delivery just depends on like incredible impatience as well.

Paul: Well, I think part of the source of this sort of like, you know, we'll ship when it's ready, comes from companies and sort of workflow processes that we've seen that look to be successful but aren't necessarily replicable.

So there's a lot that open source has to answer for in terms of the way that they ship software, and, you know, the PR that has to be perfect and, you know, the semicolons have to be in the right place before they'll get accepted, even though in a company you might take it because it actually makes the world better.

And then the other influence I see on this is GitHub. A lot of people took their lead from how GitHub made software. And GitHub has a unique set of circumstances that other software companies and small startups don't have, which is that, you know, they're massively successful and have, you know, just so much revenue that they're able to make all those sort of mistakes that your small company isn't able to make.

GitHub and the Tyranny of Structurelessness

Peter: I think GitHub had a significant learning experience organizationally.

Paul: Well put.

Peter: The everybody works on whatever they want and nothing has a date kind of mindset was very appealing in a kind of egalitarian, like, we trust each other, no one has to be the boss kind of way. But I think that as the company grew, it became probably increasingly difficult for that to really work, because it turns out that you don't necessarily want someone bossing you around, but it's really helpful to know what you're supposed to be working on.

So GitHub is interesting, and they're an unusual company in a number of ways, but not least among those is that I think their kind of punk rock ethos didn't necessarily scale as far as they'd hoped it would. And I think that while they built a lot of great software, I think that, you know, people talk about the tyranny of structurelessness, right?

Paul: That's a wonderful paper.

Peter: And I think that GitHub today has come to appreciate the incredible usefulness for everybody in an organization of knowing what you're supposed to be working on when, and who's supposed to work on it, and who to ask when you have questions, or who makes a call when there's a disagreement.

Edith: It's like the saying, people who say we don't have a software process.

Peter: It's just not documented.

Edith: And if you say we don't have a management structure.

Peter: Then it's just not written down.

Edith: Yeah, it's still like, oh, so-and-so is friends with so-and-so, and then that's actually even more insidious. So if you think you're a peer, but you're not.

Peter: I'm a huge believer in transparency. I mean, a well-run company will have a formal power structure or a formal reporting structure that mirrors kind of the informal one.

Paul: You mean the closer the formal structure is to the actuality, the better?

Peter: I would say so. I mean, it's never good if you have someone that's kind of secretly in charge but not really in charge, and kind of meddling but doesn't really have the authority to do things. One, you should probably recognize that person, or two... Things are not behaving well if you have that kind of thing.

Edith: That's kind of like the concierge in mafia terms.

Peter: Right, exactly. Consigliere.

Edith: Peter says it better.

Paul: So going back to GitHub, I think that GitHub's organizational woes were, you know, things that they overcame, and it sounds like everything's working pretty well now. But they're the sort of things that if GitHub hadn't been massively successful would probably have killed it.

Peter: Certainly. Well, I think we only see the survivors. So if you look at open source projects, right, the 99% of open source projects that are started never become a 25-year-old Postgres database, for example.

And I think that also emulating late stage mature projects with an early stage project is a huge mistake. For the same reason that you wouldn't go download a copy of IBM's employee handbook for a two-person startup. You don't want to use the same software processes for a small project that you would for something that spans 200 engineers. It just doesn't make sense.

Paul: The size that you have allows you to do things like continuous delivery much, much easier.

Peter: Well, and I think continuous delivery has different values in large and small organizations. It comes very naturally for a small organization. I think larger organizations have to make a bigger investment, but I think they can see sort of commensurate returns when they do.

A good example would be Etsy, who deploy thousands and thousands of times a year with an engineering team of, oh, I can't remember, 150 or something like that. And, you know, that's just their model. They kind of came up that way. And, you know, they're not afraid to make changes and occasionally break things in sort of service of trying to improve the product.

A Canary Release for Pricing

Paul: One of the things I wanted to ask you about is you've just gone through this big pricing change at Heroku, and you gave a talk at Heavybit about it last week. Heroku is a company that obviously does a lot of continuous delivery.

Peter: Absolutely.

Paul: And I'm curious about, you know, big change like pricing. How does that fit into that whole model?

Peter: Yeah. Pricing is difficult because it's something that's very important to all your customers.

Edith: Perhaps the most important thing.

Peter: Perhaps the most important thing. As I said the other day, pricing is product. People really understand your product through its price.

The pricing page is one place where you really have to be very careful about being completely accurate all the time, because no one wants to find out that they're not getting what they thought they bought.

So when you're making changes to your pricing, at least for us, this meant basically touching every part of the company: the marketing, the sales team, you know, documentation, but also a lot of internal systems. And so with a pricing change, you tend to roll this out gradually. And so, you know, when we did our pricing change, we actually went through a number of sort of ever increasing groups of users in order to improve the process and to improve the final product.

And I know we spoke about this the other day, but of course, for the listeners, you know, the early alphas of the product looked wildly different from the final pricing, and indeed, I think we learned a lot through that process, and people were pretty happy with the end result because of the things that came out of that.

But sort of throughout the system, we were constantly shipping small and incremental changes and usually burying them behind sort of opt-in beta flags and so on and so forth. You know, one of the great benefits was by the time we went to GA, we've had thousands of users in the new pricing giving us feedback, you know, testing all their software against it. So we were extremely confident on that day. What we really spent the GA day doing was just making sure the blog post was how we wanted it to be. We were very confident about the software.

Edith: So you've done a canary release of your pricing?

Peter: Oh yeah, absolutely. Yeah.

Paul: And people were able to opt into the new pricing?

Peter: Yeah.

Paul: Or opt out, or one of those.

Peter: Right. So in the beginning it was just strictly invite only, and people got the option to opt in. As we kind of expanded it out, we gave more and more people the ability to opt in. Eventually we gave everybody the ability to opt in. And then for our GA, what we said was new users and new apps just got it by default from then on.

So this worked pretty well for us. It helped. And I think because it was largely a price cut, we sort of moved some things around, which was sort of challenging, but in the end, almost everybody's prices came out lower, which, you know, if you're listening to this and you weren't one of those people, I'm sorry.

Edith: You can write info at launchdarkly.com for a t-shirt.

Peter: Or you can tweet at me at PVH and I'll apologize.

But, you know, by the time we went to the public, the final kind of release, we knew who was happy, who was unhappy. We knew what worked and what didn't. And some people have been using it for six months and been very happy with it. So yeah, I think you can't always do that, but when you can do it, it's great.

Paul: So it sounds like it was very much a continuous delivery process, like the same.

Peter: Yeah. Continuous pricing.

Paul: Obviously a lot of it was, you know, actually software and actual product changes, and a great deal of it seems to also have been sort of analysis and data collection. So I mean, it sounds like it's just another part of the product.

Peter: Yeah, I mean, your pricing is just another part of the product.

Edith: Yeah. Just like your email communication is part of your product.

Peter: Yep. And if you're not treating it that way, you're doing it wrong.

Edith: Or even just like your support, and even your tweets are part of your product.

Peter: Yeah. One of the hardest things, I think, is as a team scales, making sure all those pieces move together. You know, some of it is good data.

A lot of it is just learning to work in the large, right? I mean, when the person who writes the tweets doesn't know the person who wrote the software, how do you make sure that the message carries through? But these aren't new challenges.

Blame and Human Error

Edith: Well, a famous story I heard about Steve Jobs was he got into an elevator with an engineer, and he asks the engineer what he's working on, and the engineer kind of hems and haws and says something, and Steve Jobs is like, that's absolutely wrong. That's not our message at all, you're fired.

Peter: Wow. Poor management style right there. Don't do that, if you're listening to the podcast.

Edith: Well, and when I turn it around, I'm like, it's Steve Jobs' fault that he is not communicating down

Paul: Absolutely.

Edith: down the chain. Like, it's not the poor engineer's fault to have the vision. Like, it's Steve Jobs' fault that he's not saying this is what we're doing and passing it down.

Peter: The burden to communicate is on the person doing the communicating, not the person who's trying desperately to understand you, apparently in order to save their own job.

Edith: Yeah. So he's like, I can't have somebody working for me who doesn't know what we're doing. It's like, well, that's

Paul: So I'm gonna take this on a slight tangent here and say that it's neither their faults, and that one of the major problems I see in organizations is that they look for faults rather than looking for how can this be better in the future.

Edith: Oh, of course.

Peter: Oh yeah, and I'm gonna put a plug in.

Edith: Of course I agree with you, Paul.

Peter: My favorite book of all time, I think, is The Field Guide to Understanding Human Error. This is a fantastic read. If you like planes crashing and people dying on the operating table, this is the book for you.

Edith: Hey, I was looking for a Christmas present for somebody.

Peter: That's perfect. So basically what this book is about is trying to understand why perfectly good pilots drive airplanes into mountains.

Edith: Oh, there's a name for that. Controlled flight into terrain.

Peter: Right. Uncontrolled? Oh, controlled flight. Yeah, because they've got their hands on the control.

Edith: Yeah, and because, like, they used to think there was no... It's like, literally they will steer directly into a mountain.

Peter: And it's fascinating for our industry, because, you know, obviously medicine and flight have significantly more investment into understanding those fields than software. And clearly no pilot wants to fly into a mountain.

Edith: Well, the consequences are pretty binary.

Peter: Yes. But yet it happens all the time, and people classify this as pilot error in the sort of formal write-ups. And, you know, over the years that doctrine of kind of blaming the pilot began to break down as people began to understand, for example, user experience.

And so a great story from that book is, you know, there used to be this real problem where they would drop planes on the runway. So you'd be going, you'd be in a plane, you're getting ready to take off, and suddenly the plane just falls on the runway. What happens is the pilot has lifted up the landing gear.

And they would blame the pilot for this and say, oh, it's pilot error. And actually what happened was finally somebody went and looked in a cockpit, and there are two levers. One that would control, I think, the flaps, and one that would control the landing gear. They were the same size and shape, and they were right beside each other.

So if you weren't looking at the handle, it was easy to grab the wrong one. Oh, and to make matters better, on some models of plane they were the other way around. So this is unbelievable, right? And, you know, it's a pilot error. The pilot grabbed the wrong handle, but it wasn't a systemic approach to understanding the problem.

Now, the bad news is if you read this book, and you really should, they get into a number of different models for kind of trying to understand why failures happen. You can kind of look for the, you know, break the chain model, where, oh, if the pilot had grabbed the right handle, and then if you only make the handle right, then the pilot will always grab the right handle. But there's all kinds of different models. There's like an epidemiological one, where there's sort of latent risks in the systems, you try and identify them before they happen rather than after.

But basically all of these models have different kinds of defects. One is people go around chasing things that, you know, will never happen. Another one is, well, you break the chain, so this particular problem won't happen. But there's actually 50 more ways this problem could be triggered, right, and you just end up complicating the operating environment.

The bad news is that, you know, the author concludes that personal responsibility is an important part of safety, which is sort of an unsatisfying conclusion if you're a software person and you need to automate things. But these are hard.

Paul: As you're saying each of these, I could think of a bug analogy for them. Like, you could be running your fuzzers to find all the bugs that could ever happen, and maybe you want to fix them or maybe you don't. And where's the ROI?

Peter: Yeah, it's a hard thing to know. Anyway, it's an excellent read, and it's certainly very inspiring for me. I mean, in our game, automation is the only way to scale, so, you know, the author concludes that actually not automating and trusting pilots is the right way to scale, but we need to find our own solutions here in our industry.

Cargo Culting Unicorns

Paul: Well, the solution that we tend to have is we look at what unicorns did and we try to emulate that.

Peter: Well, and as I've been fond of saying, you know, cargo culting off unicorns is not the path to success for anyone.

Paul: I have a similar idea.

Peter: So I mean, basically you kind of have to understand your own business. And those unicorns didn't get where they were by cargo culting off somebody else either. If you want to succeed, you have to look at the fundamentals, you have to think about what's important, and then you think about how you're gonna get them things. It's not really rocket science, but it is very difficult.

Paul: My version of that saying is that advice is not for you. And it's basically, you know, when you look at basically anything that goes on the First Round Capital blog about how some unicorn did something, how Slack grew overnight or how Stripe does whatever, it's like, well, that was Slack, that was Stripe, that was, you know, from our example earlier, that's how GitHub makes software. And, you know, maybe it worked for them, maybe it didn't. We've no way of knowing, because they're a unicorn, and the rules that apply to them don't apply to you.

Edith: Well, not even that. A lot of advice is temporal.

Paul: Well, yeah.

Edith: Like, something that was very appropriate for that moment in time could be completely different.

Paul: That moment, that place, that specific set of circumstances.

Edith: Yeah, like six months later the world has moved on.

Peter: Right. Well, and more than that, I think there's also attribution error going on. Sometimes things are happening for reasons nobody understands, right?

Companies become insanely huge not out of any particular merit, though, you know. Actually, here's a great bit of science. I love studying this stuff.

Edith: We all do, that's why we're here.

Peter: The research project basically got a bunch of people to listen to music through their service. And what they did is they had several different universes where you could see what other people were listening to, but they segmented all the users. So, you know, you could be in sort of universe one through six, but you'd see what other people were hearing. And then there was a final universe where you couldn't see what anyone else was hearing. So basically it was sort of like a simulated radio environment where you could hear what was popular.

And what they found was that in each of these different sort of groups, different bands rose to the top of popularity. And what determined popularity was popularity.

Edith: Yeah, that's so true.

Peter: And the control group obviously didn't have this problem, but what they found was basically a band had to be good to be eligible to become really a breakout. But which bands broke out totally depended on which ones kind of got some early momentum.

So, you know, maybe Slack is popular, you know, they've done an amazing job of being worthy of and deserving of their success. And I think they're a fantastic company. And I have lots of friends there. So, you know, I'm a little biased in their favor.

But let's not forget that just because something worked for them doesn't mean they know why it worked or how it worked or that anyone else could do it.

There's also a big element of dumb luck and being in the right place at the right time. You know, you have to figure out how to capitalize on that opportunity when it comes.

Paul: So if I'm analyzing Slack, I can probably say that, you know, it was it's the polish, the design.

Peter: It's the folksy tone of the error messages.

Paul: Right, right, right. And in fact

Peter: It could be all of those things or none of them.

Paul: It could be all of them. It could be none of them. It could just be that people were sick of HipChat, and this was the next thing that came along and it was good enough.

Peter: I mean, I'm plenty sick of HipChat. I'll tell you that right up. It's a kind of mean thing to say. Sorry.

Paul: I really loved HipChat.

Peter: And I really loved Campfire once too.

Edith: Well, that goes back that Slack is the hottest right now. Soon something else might be really cool. Well, and, 'cause that's what we just talked about, is like everything goes through, is kind of fad.

Peter: If you're so lucky as to get that period. And certainly I think Heroku, I hope, is a little less boring than it used to be, but there was a time when people loved Heroku strictly because people loved Heroku, and not of any merit of our own.

Edith: We were talking about this about languages.

Peter: Oh, programming languages, yeah. What's the next new thing? Elixir, Crystal?

Paul: Elm.

Peter: Elm. Oh geez.

Edith: That used to be an email client.

Paul: It was, yeah.

Peter: It still is.

Paul: Elm is the new shiny thing. Well, one of the many new shiny things.

Peter: So what is so new and shiny about Elm, for the listeners?

Paul: Well, it's Haskell for JavaScript.

Peter: I'm sorry, you lost me at Haskell.

Edith: That's like somebody pitched me an idea of Uber for pedestrians, and I'm like, I don't even know what that means.

Peter: Haskell for JavaScript. Wow. That is either the best or worst idea I've ever heard. And I'm pretty sure it's going to be the latter.

Paul: I think it's closer to the best.

Peter: Okay, make the pitch. Why should we try Elm?

Paul: Well, so I haven't tried Elm, so I'm just literally

Peter: The hype is strong...

Paul: The hype is very strong.

Edith: I hear Paul has a couple other business ideas too.

Paul: I think static typing solves many of the problems that people have with writing JavaScript. That's it. That's all I got.

Peter: Sounds good.

Paul: Sounds worthy of a try.

Peter: I think the real problem with JavaScript, having written a tremendously large amount of JavaScript, is not knowing how you got to where you are in the call stack, because everything is asynchronous. How did I get here?

Paul: Almost a synonymous problem.

Peter: How did I get here, and why is this argument undefined?

Edith: That's like the meaning of life.

Paul: I mean, this is what static typing solves.

Peter: Kind of. I mean, you can still pass around null values.

Paul: Right, right. Well, I mean, maybe.

Peter: Maybes, right.

Paul: Oh, this is exactly the kind of problems.

Peter: Now you know why you're screwed, but that doesn't necessarily help.

Paul: First step to solving the problem. It's no longer in the 37th stack frame, it's in your checker before it went out to your customers.

Peter: So one would hope. I'm sure that Haskell and Elm and every other language will give you plenty of rope to hang yourself still.

Paul: I'm sure.

Pitfalls and Code Review

Edith: Yeah. So Peter, speaking of rope, what are some of the pitfalls or drawbacks of continuous delivery? Are there any gotchas or things that you'd warn people about?

Peter: Well, I think success from continuous delivery comes from, one, having good data to act on. If you're not following the important things and then following up, then, you know, it doesn't matter how often you deliver, you're not gonna see much benefit. You're just essentially random walking your way around the market.

Two, you have to be actually prompt with your continuous delivery. If you have things hanging around in branches indefinitely, right, your cost of delivery just goes up and up. You pay the integration costs, you just pay them continuously.

You know, short-lived branches, I think, you know, some people say, oh, you have to develop on master. I think short-lived branches are fine, and, you know, everybody's used to pull requests these days, so it's not a big deal. But I think those are probably the two biggest things.

And I think a good culture of code review, prompt, thorough, thoughtful, friendly, right? You're not being reviewed, your code is. That's sort of the third piece.

Paul: So I'm not convinced about the code review part of it.

Peter: Go on.

Paul: I guess I'm just not sure what the problem is that is being solved by code review.

Peter: I think that traditional software models have a much larger sort of quality assurance phase and tend to have more formal organization around software releases, the what are you doing, the how are you doing it, how's it being structured.

There's a great book, actually it's not a great book, but there's a great chapter in a mediocre book called The Architecture of Open Source, where the Berkeley DB authors write about creating Berkeley DB and sort of the lessons learned along the way.

And I hope I get the quote right, but it was, patches and bug fixes erode software architecture.

And so when you think about what that means for continuous delivery, what you have is many, many small things happening all the time.

Paul: Continual erosion of the architecture.

Peter: Right. And the idea behind good code review is that it sort of gives you an opportunity to reflect on that and comment on it and have other people understand what's going on. And it's certainly not essential in the small. But if you're talking about success over the long term

Paul: Well, I don't want you to think I'm against code review. Like, I think every single pull request in the last, like, three and a half years, maybe, at Circle has been code reviewed before it got merged. But I find myself sometimes just, like, just wondering, you know, what exactly are we doing with this code review?

You know, is this a syntax thing, or, I mean, very often it ends up being product reviews are being conducted at the end of, like, you know, five days of coding. And, I mean, before we had a real product process at Circle, this was a real problem. That someone would build a thing, and that's when it would first come out for product review.

Edith: So you had a process, and it was you built it and then reviewed it.

Paul: Well, yes, right. But the product review happened very, very late in the product cycle, and it resulted in people being very unhappy when

Peter: A huge amount of time.

Paul: Right. Some of the early product decisions that they made, there was disagreements as to whether that was the right product decision. And then that led to problems, and if you're of the sort of GitHub slash open source view of, or maybe, sorry, unfair to give it to GitHub. I think this might be a Linus Torvalds thing of code talks. Yeah, something like that.

Peter: Bullshit walks. Yeah, I think the PR model, sort of historically in the open source world, it's very common for you to labor for weeks on end on a patch, drop it on a mailing list, and then be told to go stick your head in a pig. Because no one cares. Obviously, in a work environment where, one, you're paying someone for that time, and two, they will quit, and then you'll have to find someone else if that happens too many times.

Paul: Or you just don't hit your own goals as a manager or product owner or whatever.

Peter: Yeah, or as an engineer. And I mean, the open source model is you tend to have one or a small number of basically dictators who decide at their will.

Paul: Hopefully benevolent ones.

Peter: Yeah. Well, dictators rarely remain benevolent.

Edith: Yeah. I think to Paul's point, it's really just about being lean with even development. And I'd say when you build something, you have endowment effect, like you spent five days on it, and you don't want to throw it away even if you're wrong.

Peter: Yeah, so I have kind of a mental checklist on projects about whether or not something is ready. And, you know, you kind of run it more or less formally depending on what the project is and how big and how many things it touches.

But, you know, is the product good? Does it solve a real problem people have? Does it do it correctly? Is the engineering good?

Is the code well written? Is it well architected? Does it fit into the architecture? If it introduces new abstractions, are they the right ones?

If it introduces no new abstractions, should it? You know, that kind of thing. SRE, is it gonna break? Is it gonna perform? You know, does it have security holes?

Is it good? Is it hardened, right? Will it go bump in the night when too many users try to hit it at once?

And if you've got those sort of things, right, like, will it work? Is it built right, and does it solve a customer problem? Then you're probably ready to go.

Paul: So that introduces a lot of friction into the small one line bug fix that you were about to

Peter: Well, no, but I mean, you don't This isn't like a... I would run that sort of checklist personally if I spent more than a week on something.

Paul: Sure, no, I'm being devil's advocate here, but suggesting largely that you know, a large, very frequently code review concerns often form roadblocks.

Peter: What percentage of downtime, Edith, do you think is caused by single-line bug fixes?

Edith: I'd say, like, if I had to guess, I'd say the less damage you think it's gonna make, the less time you spend reviewing it. And therefore the more likely it is to break something. And I say this in my own experience of, like, you know, the famous phrase of what could possibly go wrong. Like, if you're doing a complex data migration, you spend a ton of time on it.

Peter: Yeah, and you get good reviews because, you know, that's sort of the opposite of the bike shed problem, which is, like, a complicated patch, people will be thorough with. But, like, a couple of lines, it's a doc string.

No one notices you forgot the comma on the end of the line. You push the code. Hopefully you're running continuous integration. So it goes red at least.

But the more subtle errors can slip past you, 'cause, oh, you put a seven instead of an eight, and now we provision the wrong things relative to what the pricing page said, and I didn't realize, 'cause it was just a one-line change.

Paul: So, I mean, the example from earlier, like, you know, are we good to delete this index? It's like, yeah, I mean, we're not using it, right?

Peter: No one's using it. Yeah. So I guess at least code review means there's a second pair of eyes. Someone is taking a second, that didn't write it, to think, does this make sense? But of course, I learned back when I used to get a lot of code reviews. Of course, I'm more of a product manager these days, but

Edith: Everybody's a product manager.

Peter: Everybody's a product manager now. But yeah, no one's a product manager, there's too many of them.

So back when I used to get a lot of reviews, I just knew who to ask for what kind of review I wanted. So, you know, if I was, like, worried about the architecture, there was someone I would ask, and they would give me, like, a really, you know, they would always really critique the style and thoroughness of the patch. And if I wanted someone to check it for, you know, obvious bugs, I knew someone I could ask, and they'd do, like, a technical edit on it, basically, and, you know, read each statement backwards individually to make sure it still made sense. And you know, if I just wanted to ship the damn thing and I was hoping for the best, then I knew who to ask for that.

Edith: That's actually really good advice, Peter. I mean, that's something that we talk about a lot, is the trade-off between time, quality, features. And you're also trading that off even in your code reviews.

Peter: Yeah, and, you know, it's sort of subconscious, right? But when you ask someone for a review, if you've worked with them for a while, you know what they're going to review and what they won't.

Edith: Yeah, it's like, I mean, I love quality, but sometimes you just gotta ship stuff.

Peter: Yep. Sometimes you just gotta ship it.

Edith: Because the biggest risk is always just that there's no business value.

Peter: Yeah, and sometimes, I mean, you can put things out there and then see if there's value, and if there isn't, you'll take them away again.

Puns and Emoji

Edith: Well, Peter, we really enjoyed having you here today. Do you have any parting thoughts around continuous delivery or why people pun?

Peter: I don't know that we talked about puns on the podcast. I think that was before we started recording.

Edith: I was hoping you would talk about them now.

Peter: Do I have any thoughts about continuous delivery or why people pun? I think punning is one of the great joys of life.

Edith: Totally agree.

Peter: It's that little bit of wordplay. You know, actually, I was thinking about this on the way over. It's not really a pun, but I was trying to figure out what the emoji representation of a podcast is. So, like, microphone, cell phone, broadcast tower, how would you represent a podcast as an emoji?

Paul: I think it's the little pile of poo.

Peter: Well, that's already taken by everything else.

Edith: Emojis are basically a reduction back to hieroglyphics.

Peter: Yeah, I love it.

Edith: Like, we came up with all these letters so that we could have complex thoughts, and now we're just going back to hieroglyphics.

Paul: How did the ancient Egyptians represent podcasts?

Peter: I think it was Eye of Horus, ankh, and then a foot. So I guess we'll close on that then.

Paul: Thanks so much for coming by.

Peter: Thanks for having me.

Paul: Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of CircleCI, and Edith Harbaugh of LaunchDarkly.

Edith: To learn more about Heavybit, visit Heavybit.com. While you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.