
Ep. #8, Continuous Delivery in Mobile Development
On this episode of To Be Continuous, Edith and Paul are joined by Kevin Henrikson, Partner Director of Engineering at Microsoft, to discuss whatâs necessary to be successful with continuous delivery in mobile development.
For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.
transcript
Paul Biggar: What's your favorite thing about continuous delivery?
Kevin Henrikson: What I like the most is just the fact that it lets you get good things done quickly, but also correct mistakes quickly, right? The train keeps moving, but the size of what's on the actual boxcars is smaller. There's a marketing strategy and PR strategy and how that aligns with continuous is tough.
Paul: Hi, I'm Paul Biggar, founder of CircleCI.
Edith Harbaugh: I'm Edith Harbaugh, CEO and co-founder at LaunchDarkly.
Paul: And you're listening to To Be Continuous, a podcast about continuous delivery and software development.
Edith: You can get in touch with us anytime at our Twitter handle at continuouscast.
Paul: The show is brought to you by Heavybit. To learn more, visit heavybit.com. And while you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.
Edith: In this episode, we talk with Kevin Henrikson, an engineering director for Microsoft Outlook. Kevin talks about succeeding with continuous delivery and mobile development.
Why Weekly Sprints Work
Paul: Alright, Kevin, so what's your favorite thing about continuous delivery?
Kevin: Good question. I think so, Acompli started kind of the approach when we first started with just a couple people. And our deal was like, hey, you're trying to get this thing to work, send email. So it was like, keep checking it in.
Get closer and closer to kind of the definition of it was working. And then as we started to experiment farther out, we said, what's the right sprint cadence?
And we'd come from kind of traditional enterprise companies where you ship every six weeks or every four weeks. And we're like, oh, we can go faster. People are doing this agile thing.
It's like two-week sprints. And we tried two weeks and realized that most of the work came in on Thursday afternoon or Friday kind of thing, because the end of the second week. And so, well, what if we just do one week, then most of that work will come in the first Friday?
And sure enough, it worked, right? 87% of the work still comes in on the last day or kind of completes on the last day or gets to the point of ready. And so what we liked is that it gave us a much smaller kind of chunk of deliverable, right?
So like the train keeps moving, but the size of what's on the actual boxcars is smaller.
And so if you have a problem, it's easier to roll back, easier to fix. Or if you have a problem, you can just patch it quickly because you're used to moving quickly, right?
Edith: Wow. Oh no, go ahead.
Kevin: No, I was gonna say, so in a traditional world where you ship something every six weeks, you're like, oh well, we made a mistake. We need to go patch this. Like, the patch felt like an exception and you're like, your muscle memory for patching was not there, right? So you're like, we do it now. Is it worth it?
And then it's like, well, if you're five weeks in and you discover that thing, it's like, well, the six-week thing is out there, where if you're shipping every week, you're like, oh, let's patch it. We're like, oh sweet, we shipped yesterday. We just click the button again and it builds and it goes, right?
So you've built the muscle memory to move quickly, right? And so I think what I like the most, kind of going back to the specific question, is just the fact that it lets you get good things done quickly, but also correct mistakes quickly, right?
Edith: Wow. So this is a good time for you to introduce yourself.
Kevin: Cool. So, Kevin Henrikson. Currently manage the Outlook kind of non-Windows teams at Microsoft, so Outlook for iOS, Android, and Mac. Previously I was VP of Engineering at Acompli, which was a mobile startup focused on iOS and Android.
Edith: Yeah, so you had a lot of good thoughts right then. One of the things I wanted you to talk more about is why do you think 70 for 90% of the work is always on the last day.
Kevin: Yeah, I think people are kind of deadline driven. I think generally they say, hey, they want to work to the last minute. And I think software development ends up being this thing where you're never done. And so the only way to define done is to actually draw a line and say this is done, and then still some things still miss that line.
And so the tighter you give the window for things to get done, the more often they'll hit that window, right? Because you've kind of broke things into smaller chunks, right?
It also, because software is kind of this long-lasting thing that never is kind of done-done, that you end up with, if the chunks are small, people are able to estimate and think through. It's like, oh, is that gonna take me a day? Very rarely does somebody think something's gonna take them a day that takes months.
But very often, if you think something will take months, it may take many, many months, right? Because it's hard to guess how, because there's so many variables and things, so many things are gonna change in the environment or in what you're doing before you get there.
So I think the short answer is if you start with a smaller window and more discrete small tasks, those tasks are easier to estimate, they're easier to consume. They're easier if you get disturbed, like, oh, I'm gonna go down and get a soda, or somebody asks me a random question because some fire comes up. I'm still like, I can kind of make it up because I was only expecting to spend half my day working on that anyways. And so I think that kind of works out in our favor.
Paul: So there's an implication with what you said there that the task sort of grew because it was two weeks rather than, I mean, so I guess the phrasing is, when it was two weeks, was it that a one-week task had grown to two weeks? Or was it that things got cut when it got shrunk down to one week, and you ended up with technical debt or something along those lines?
Kevin: I think it was a couple combinations, right? So I think if you feel like you have two weeks to work on a task, you will spend more time kind of polishing or potentially doing things that aren't on the critical path to getting it done. And I think that's natural, right? Like maybe it is you're paying off too much technical debt, right?
In the startup world, you have to kind of have that lever. And even in just kind of general software development, you have to have a lever.
We're not launching the space shuttle here. There's room for some wiggle room and imperfection.
And I think breaking it into small pieces lets people focus on, hey, I'm only gonna do this and I have this much time to do it. Where you say, hey, it's got two weeks, you may be deep into the second week and saying, I just don't like my approach, I'm gonna change it. And you basically end up throwing away a week's of work, where if you're thinking that, hey, the deadline's coming in a week, you're gonna make that kind of epiphany like, oh, I made a mistake, or I've taken the wrong path, much quicker.
So I think it allows you to kind of break your decision making and optimize on getting it done, because the window is kind of quickly approaching, being a one-week sprint versus two.
Apple Sets the Pace
Edith: This might sound like seven-minute abs, but why not two-day sprints?
Kevin: Yeah, so I think two days are great things, right? Or one-day sprints, right? No, how do you continue to make it faster? So for us there was sort of one artificial gate, which was the fact that you really can't ship quicker than about a week with Apple. And even then, it sometimes slips by, 'cause four or five days, and during a peak season if there's lots of apps, like on Christmas holiday or around a new platform update, you'll see a spike in the review times.
And so Apple kind of, and we were an iOS app first before we ported to Android. And so that ended up driving us towards a little bit to a week, because it was like, if you shipped on Monday, the app would get in the store sometime end of that week. And that was a natural cadence to kind of start the next one. So it's really hard, without kind of pushing the emergency release button, which Apple only lets you do once or twice a year, to ship more than once a week.
And then we kind of let the rest of the team fall behind that, right? If you're only shipping the app once a week, let's keep the new chunks of new work or big bug fixes to that path.
On our service, and the way our architecture is, and we can talk about that more, but we spend as much, 70-some percent of the logic or the ant is down deep in the service, because that we can update all the time. So hey, we push a release out, I want to fix something, we fixed in the service the same day, right?
Edith: Oh, interesting.
Kevin: And so that allows us to have a much more kind of agile approach, because the service is constantly being updated, but the app is kind of gated in one-week chunks.
Paul: So what you're saying is that the biggest impediment to you shipping frequently is Apple.
Kevin: The platform. Yeah. The platform, basically Apple's review, and specifically it's the fact that Apple has a review process, which is great, right? That Apple review process generally keeps the bar of apps better and kind of puts some good discipline inside of the wider range of development people that are coming to the Apple platform.
But it also, when you want to move quickly, you can't, right? You're basically, the platform itself kind of enforces this four-to-seven-day lag in shipping.
Paul: So you're now running teams that are shipping on non-Apple platforms.
Kevin: Correct.
Paul: Did you guys make any changes to your release cadence as a result of that?
Kevin: Yeah, so I think on Android, for example, we still kind of keep to the one-week chunk of, we plan a sprint, build for a week, release to our beta group or dogfood users, and then release to production. But Android gives you this advantage where you can do percent-based rollout. And so with Android, we end up releasing usually two or three times a week.
So you'll release kind of the release on Monday at one or one to two percent, kind of walk that up three, five, ten percent. Usually at five or ten percent, you can pick some new signal of crash or behavior that you want to adjust, make a hot fix, push another release, start at 10%, 20%. Ah, you see another one you want to tweak, hit 20%, and then finally from 20 or 30 you end up going all the way to 100.
Paul: Do those Android users help you on the other platforms as well? Are they finding crashes that also exist on the other platforms?
Kevin: So today we're 100% native on both, so there's no shared code between iOS and Android from the mobile app perspective. Obviously from the service, it's 100% shared code.
And so what we've actually done in the past is when we launch a new feature that we think is either controversial or has a high chance of risk, for example, when we launched IMAP support, we launched generic IMAP support on Android first, knowing that we could patch it quickly and adjust if things went wrong, simply because there was no way to test the millions of random IMAP servers in the world and get any kind of focus thing. When we talk to Gmail or Yahoo or iCloud, single point.
Paul: That's really interesting, actually, the idea that because the rollout or the sort of safety features are only supported on Android, that people will end up building features there first.
Putting the Logic in the Service
Edith: How deliberate did you have to be about putting stuff in the service versus in the apps, or was this a lesson you learned?
Kevin: So we had always thought of the service as the place where we wanted to do as much of the logic that could break or as heart, right? You just don't want to do it twice. You don't want to implement ActiveSync twice, you don't want to implement an IMAP stack twice.
And it also allowed us to pick a protocol between our cloud service and the mobile app that was optimized for mobile, right? So think of the shape of the API, looks closer to the screen of the app, right? There's a request for, like, get my inbox, and then make a change, and where if you think of IMAP or the myriad of email protocols that we talk today, they don't have that very linear type of approach of, like, give me this screen, give me that screen. They're thought of in a way where you're gonna sync all the mail down to your desktop client and then manipulate it locally and then sync changes back up, in some, either connected or disconnected state.
And so the service was basically set to put complex logic, but also to allow us to get leverage, because everything we could build in the service was one less thing we'd have to do twice on the client. Because we picked a path that we didn't basically have a lot of shared code in the client. There's essentially none, just the API.
Paul: And it's the same service that sits behind all of the different clients on the different
Kevin: yeah, so we have a single cloud service that basically powers both the iPhone client and the Android client.
Edith: That's interesting. So it was at TripIt and we had a lot of similar issues. Like, we were at one time one of the biggest OAuth users of Gmail.
Kevin: Okay, awesome.
Edith: Because people would sync their Gmail with us and their calendars. Did you find there was a difference between the UE or the app experience that Android and iOS wanted, and how much did you bifurcate that?
Kevin: For sure. So a lot of people ask, like, hey, did you launch a feature first on Android or iOS, or do you always have to make them come out identical? And my answer to that's pretty simple.
It's like, most normal people carry one phone. They're either an Android user or an iPhone user.
And so they don't know that you released it first on Android or iPhone, because you didn't tell, they don't carry both phones.
And so we think of that from a feature point of view. When they're done, they're done, we ship it. Now if we're trying to line a press event around something, we want to make sure that they're both there before we do the press event. One may come out a day or two before the other. But generally speaking, we don't try to legislate, like, hey, you have to do it this way because it works on iPhone.
And in fact, we actually took a different approach where we said, hey, we wrote it one way on iPhone, and then when we went to Android, it wasn't just like, oh, go port all the screens over. It was like, what would this screen look like for an Android user? So we want our apps on the mobile device, and most mobile users want to look at an app and feel like it fits in with the rest of their phone.
And so the way that Android navigation works, the way that people use the back button or not, is very different on iPhone and Android. And the fact that there's widgets and the way that the widgets are exposed on Android, and the way that most people want to integrate deeper into the platform. And so those kind of things we let go with the platform.
And so our view is write it in native code, because it allows us to take advantage and move quickly to the platform-specific features, but it also lets us tailor the experience and the UI and the UX to whatever that user base wants.
Paul: It's interesting that you talk about it that way, because in the old days, it's kind of ten years ago, when people were making multi-platform applications, there was a tendency of people to try to make it all match to the same. Firefox is a good example here, or Chrome, or kind of one of those tools that people use on all platforms was designed to look the same on every machine, even though people didn't have that multiple machine. And I think one of the nice thing that's happened with the mobile ecosystem is that people have thrown away that concept and they've made it actually feel native for each of the platforms.
Kevin: Yeah, and I think that also goes to feature set, right? You would think of a traditional app as, like, oh, if it works this way on Mac or on Windows or on the web, you would expect a hundred percent kind of feature parity between all those platforms. And today on mobile we go and select what you actually use. And there's features that exist in mobile only, right?
Because mobile has all this additional context of your location and where you are and what you're doing. Are you running? Are you driving?
There's other sensors in the devices today that allow you to build specific features that are richer. And so we always thought of, at a company dialogue is, rather than sending more email, send better email, right? Like, don't say I'm running late, say I'm running late and I'm here, and send a quick tap of a button to put the map and show you that, oh, I'm twenty minutes away, or I'm on the train, or I'm at the train station, and so people can understand, like, you don't have to type out how many minutes it's gonna take.
You don't know, right? Like, are you walking, are you driving?
Edith: Well, and plus people lie all the time. Like, "I'm on my way" has a pretty loose meaning of anywhere from "I might leave my house in thirty minutes" to
Kevin: Yeah, and depending on the context, right? What that person's thinking. So yeah, good point.
Feature Flags and Flighting
Paul: So one of the interesting things that you said there is that it might take a day or two. When you're trying to launch something, they might actually land a day or two apart. And on this show, one of the things we try to do is we try to pitch Edith's business. Which is feature flags as a service.
Edith: I never knew you did that, Paul.
Paul: It would seem to me that kind of best practice there is to release the code and not have the code be a block around the launch. So the code is there, but it's behind a feature flag. Do you guys do that much?
Kevin: So we have feature flags, we have flight, we call it flighting, same idea. And so we launch all kinds of dark features on the server in particular. So there's all kinds of things that we've built that are currently in production on the service that the production app may not take advantage of, right?
Great example is, like, we launched avatar support. And it fit with the UI of Android better. And so we turned it on there and didn't turn it on on iOS because we're still working through how we want that UI to lay out, right? And so in that kind of case, and maybe we'll turn it on for some percentage of users and test it. And so yeah, we basically think through, like, what are the features that make sense?
I think the one piece that we're a little different than people that kind of go crazy with tons of feature flags is that we actually try to think through, like, is this something we're gonna want to potentially toggle on or off, rather than adding a flag for everything, simply because when you end up with a large number of flags, the interdependencies and the combinations become very hard to test, even at the scale we're at with millions and millions of users, you just can't get to the combinations, and then you make a mistake and developers are like, oh well, you didn't have XYZ default on, you weren't running experiment three, right?
And so those kind of things. And so for certain things we don't feature flag.
And the fact that, people used to ask early in days, were you guys do A/B testing? And we're like, yeah, we do A/B testing every Friday. We ship a new build, right? And so as a startup, we were A/B our way, we would just make the change.
And now the user base is bigger, right? With millions and millions of users using the app, we need to be a little more thoughtful in terms of how you roll that out. And so in this case, we are using some more flighting-type feature flag stuff to enable the users to see features in kind of waves.
Edith: So let's walk through this. I think I heard it called shipping Friday. So every Friday you shipped. What would you do if there was a bad release out on the field for a week?
Kevin: I mean, it'd be very hard for it to ever show up in a week. We would notice it. Our process basically is we plan the day on Monday, review kind of the previous week's work, code Tuesday through Friday. We ship to testing on Friday.
Test Friday night, Saturday night, Sunday night. We have a team that basically does testing overnight on those three days. Monday morning we get the report and then we decide to push it. We push it to our beta users.
And so there's a lot of gates. Kind of there's a set of QA and kind of manual testing that happens over the weekend. There's a beta test, kind of dogfood, that happens the first few days of the week. Then we kind of push it live.
And like I said, we can split the platform, right? So if it's an Android bug, we usually can fix it within an hour. It's very quick to push to a new Google app or Android app. With an iPhone issue or iOS issue, we have to determine the severity of it.
And if it's something that needs a critical fix, we'll push the red button with Apple that says we need expedited review, which usually can bring the time down from a week down to about 24 hours, but no guarantees, like, it's just best effort. And so that's what we do.
Now, there's been many cases where we disable features, right? Like, IMAP, for example, we turned on, we started seeing a lot of problems. We turned it off to a few countries that we saw more problems from, and it was very popular IMAP servers in those countries we didn't support. And so we just disabled it on a geo basis. And so there are places where we've turned off things where we had to put a feature flag in place to do that, but it really depends on the feature.
Some big UI changes are hard to feature flag, and so those you have to deal with. But fortunately those cause a lot less crashes, because UI is just not as instable of code as the kind of core sync code.
Edith: Is there ever something that you really wish that you'd put a feature flag on that you didn't?
Kevin: Yeah, I'm sure there is. I think we shipped iOS 9 on the day iOS 9 launched and had a crasher. So there was a migration bug. But I don't know how you could have turned off iOS 9, right? So some of these things you just, and then we push the button and it took us two days, so we had to eat one-star reviews for two days, which was super painful.
Edith: But they hurt.
Kevin: Yeah, they're pretty permanent.
Edith: Well, they hurt, and then they write mean things and it just stings. I got, at TripIt, we were overall five, but when you get a one star you feel the pain personally.
Kevin: Yep, yeah, you get thick skin.
Dogfooding Every Check-In
Paul: So you were saying you're releasing something to testers over the weekend, and then it goes out to the beta channel the next week. So if the testers gave you a bad report and you needed to pull it, then you end up in this situation, I presume, of instead of a week's changes going out, you're gonna have two weeks' changes going out, and if that's bad, it might stack up with three weeks changes, and then suddenly you're out of continuous delivery, you've got a big risky release going out. Any experiences with that?
Kevin: Yeah, so we've had that frequently.
So usually what happens, though, is because you've only written code for four days, we usually always say, like, if it takes us four days to write the code, we should be able to fix it within four hours.
And so we fix it Monday and just ship it and take the risk that, hey, we thought we'd, extra code reviews, very aggressive look, and then we ship it to dogfood, so we very rarely skip an entire week where we pull through that. We almost always find a way to ship it, and everybody kind of rallies around that as a cultural thing. Like, hey, if there's something wrong or a crasher comes through and we can't fix it, it's kind of all hands on deck to sort that out.
And the one piece I did miss is we internally dogfood every check-in, right? And so the way that our system works is when you check in code, whether it's on the app or on the service, within about three minutes it's pushed to your phone. Or the service has been updated. And so the service is constantly updating with every check-in.
And so things that are very critical, and so we say, well, what's a critical bug? Well, oh, I can't send email, or I can't reply or forward, or a button doesn't work. That stuff's found within minutes of it being checked in, because the dev team's relatively large and we're all running that app, and so we're seeing those kind of things, like, whoa, I don't like this, jump on chat and tell everybody, like, hey, let's go look at this.
Paul: Gotcha. So you just stop shipping new things if there's a problem. You, like, all hands on deck to fix the actual problems.
Kevin: Yep. And like I said, usually when you've only had four days of change, kind of at the maximum, usually within a few hours we can figure out what's going on and either disable it, right? Like, for example, in the case of when we shipped the iOS 9 crasher, it was with Spotlight indexing. It was the new kind of indexing feature in iOS 9. We just disabled Spotlight indexing, cut a new build and we shipped it within an hour.
So like most, again, it's never perfect. The software's an imperfect trade, but most things we can figure out soon enough, or be able to kind of roll it back or disable it.
Paul: Right. One of the big releases that I was part of that went badly on this kind of thing was the Firefox 4 release. So it took eighteen months to go from Firefox 3 to Firefox 4, and the main reason was, there'd be a big feature that would go out and the big feature would be buggy and would need a lot of work, and so it just wasn't ready to go out. And in the meantime, other teams are working on their own thing, things are getting included.
And in the end, something that was supposed to be, I think, a six-month release ended up taking eighteen months before it went out. And there was stuff that was ready, new CSS features, the sort of thing that you don't want to wait eighteen months to be released. That was the catalyst for completely changing to the continuous release model.
Edith: So you talked about dogfooding. How much do you dogfood? Do you require everybody to only do email on their mobile?
Kevin: We don't. So we've actually, there were cases where we could block, or, we don't, though. Fortunately, everybody uses mobile email, and so it's a pretty light ask. It's not like we're building an app that people would not normally use on their daily basis.
So I think the biggest thing is that generally developers on the team don't have the email volume that matches our user base. Generally developers don't get crazy long, hundred and twenty message threads and gajillion attachments and super long complex HTML and the messages. And so I think the bigger problem is not about people using the app. It's about the representation of the data matching our kind of wider user base.
Edith: Yeah. I asked because Facebook legendarily does not allow people to use Google products. So for example, they use Microsoft Bing instead of Google.
Kevin: Yeah. No, I think companies have those rules. I think for a long time Microsoft, they would only pay for your cell phone if you used a Windows mobile phone. So I think these are common ways of trying to encourage folks to use what's, and I think those are great. I mean, they work, but I think there's always exceptions to the rule.
I think that you also can come blind to competitors by only focusing on your own stuff, and you end up with tunnel vision of saying, hey, all I know is how my stuff works with everything else, and I've never worked in a kind of quote unquote real world universe where everything's different. Where you live in the home, yeah, Mac, you have Windows, you have this phone, I have Android devices. With Android in particular, right? There's how many different versions of Android that are out there in the wild today, on all kinds of different hardware, and even from the same vendor, we can see bugs between Samsung devices, right?
And so I think we encourage people to use what they have. People are allowed to expense or buy whatever phone they want. Lots of the team members, especially kind of those of us in support that are dealing with customers, have multiple phones.
We have different tablets and phones that are our personal devices that are connected, and we're using them, kind of switching between them, because that's what customers are doing, right? They're looking at all these platforms. And so being familiar with them is one, but also just being able to catch issues.
Polish Versus Speed
Paul: So one of the questions, or one of the things we talked about with Kris last week, was the trade-off between doing continuous delivery and delivering polish. And Acompli had this reputation as being this incredibly polished product that people loved using. It was the best. The best Gmail app, I think, on iOS was the description I've heard.
So how do you feel that trade-off works? Where you have something which has such a high degree of polish, how does it affect what you're shipping and how you ship?
Kevin: First of all, thank you. It's awesome. Yeah, I think we got lots of accolades for our polish, and I think we always thought that we could do better.
The founders were all kind of like plumbers, right? We all worked on infrastructure and backend systems. So UI wasn't our thing.
So the thanks really goes to the folks on the mobile team, the kind of experts we hired in that space that really just like to hone their craft. But it was a constant tension, right?
There's always this notion of the startup lever of, you have to get function and utility, and we're building a utility, we're building an email client. If it's missing big features, people just can't use it. If delete is missing and reply all is missing and certain features are missing, you can't use it. And so no matter how polished the send button is, it doesn't matter if you can't do reply all and it doesn't support BCC and those kind of things. And so there was always this tension between how do we get polish and utility.
And I think we could have got a more beautiful app, but potentially a less functional app, and who knows how that would have affected the outcome if we would have spent more time on polish, and vice versa.
But I think what I've learned now is, so another team joined Microsoft just after we did, called Sunrise. It was a calendar. And they were like world-renowned design, right? Like, a really, really beautiful design.
And you asked them how they kind of went to their sprints and delivery, and their model was more like every six weeks, seven weeks. And it was done when it was beautiful, was kind of their mantra, versus us. It was done on Friday.
And so they brought, now we've kind of have a combined team of working together, and they've brought this notion of, oh my gosh, they basically spend all this time getting the design perfect and honing it, and then, but we're able to ship it every week. And so the way that we do that now is the design starts about two or three weeks ahead of the actual work. And so we're allowed to get the designs much more time to kind of allow that polish to happen and those designs and mocks to come back with very, very high fidelity. So that when we implement it, we know what we're doing.
Where Acompli kind of did design and implementation almost in the same week. We would design stuff, spec it on Monday, design it on Tuesday, build it on Wednesday and Thursday, quick kind of put some paint on it on Friday, and then ship it on Monday, right? And so it just didn't give you time to kind of get to complete beautiful. And I think now that with the Sunrise and kind of integrating that process in the team that they brought over is really upped our game on the UI side.
Paul: So if you're looking for beautiful, or if you're looking for functional and beautiful, I guess, then shipping constantly would seem to be a helping hand there. But they literally just, like, spent six or seven weeks working it internally.
Kevin: Yeah, they would work it internally and continue to polish it and get the feature in the UX just right. Until they decided, hey, now this new chunk of work is done.
Paul: Would that involve, like, user testing?
Kevin: User testing, data research. And then it also helped because, as a startup, you become immune to the press.
If you're releasing every week and you're like, hey, look at this, look at this, look at this, the press stops writing about you.
And so Sunrise did an amazing job of kind of packing up that six week with something a little bigger and dropping a bigger hammer and saying, this is something big and amazing, write about it. And so a lot of the press and kind of attention they got was because they were able to kind of drop these slightly larger releases than a traditional one-week kind of bite. That gave them that additional attention.
Press Without Version Numbers
Paul: I think that's really interesting. I remember one of these sort of software marketing 101 things that I think Joel Spolsky talked about years and years ago was that the biggest thing that they could do to get press was just increase the version number. So FogBugz 5 or 6 or whatever just got a twenty percent or thirty percent customer increase just by existing.
And now with continuous delivery we've kind of killed that. No one talks about the new version of almost anything. I don't think anyone's ever talked about the new version of Facebook. There's the new newsfeed and things you can yell about, but it's not like we've gone to Facebook thirty-seven now and it's amazing.
Edith: I disagree with you, Paul.
Paul: Okay.
Edith: I mean, I think it's just that what's gone away is the notion of versioning, but yes, Facebook does a very good job about announcing new features. So does Twitter. Like, they push this Moments, and the number is an arbitrary thing, but they're definitely still pushing functionality.
Paul: So yeah, I think there's two things I'm saying. One is that releasing features feels like less than releasing a whole new version. And then the second thing is that companies of Facebook size and that sort of thing, and Google size, do good jobs of it. Whereas the small startups, it is very difficult for small startups to be able to do the same thing when they have that continuous delivery.
Edith: I think it's just hard for small startups to get press, period.
Kevin: Yeah, and you basically need, you talked about moments, right? And we always talked about, like, what's the next product moment. And even inside of a company like Microsoft, that's important too, because with Microsoft you have different kind of competing events, which is there's always lots of things to talk about.
And aligning your kind of moment with, oh, they're launching Windows 10, or they're launching Office, or they're launching new Xbox. There's very, very large events that are like, oh, the NFL's switching to the next version of Surface. These are huge product moments, and like, oh, I want to talk about my little feature in my app, right? And I say little, but even at the scale, right, with millions and millions of users, it's still in the scale of, like, the NFL switching to the new Surface Pros, like, that's pretty cool, right?
And so you have to kind of, there's a marketing strategy and PR strategy, and how that aligns with continuous is tough, right? I think we struggle with that all the time, of trying to pick when to turn things on, and what things you group together, and what's the story.
'Cause we always figured as a startup, it's not about the feature or the moment or what version you are, it's the story you're telling. And the version was the easiest way to tell the story kind of ten years ago.
And today the story is about, like, oh, we're launching this new capability or this new feature or this new thing, right? And that's the story now.
Paul: I think it's very easy for companies like Apple, and I see AWS doing this as well. They just have this event that everybody pays attention to, everybody listens to, and so they get to announce a list of features, but they I remember Amazon launched some changes to CodeDeploy in the last re:Invent, and I don't know if people really care that much about the new features that has gone into CodeDeploy, but it got a mention, because it got packaged in with all the other fifty other changes that had happened to AWS, and all of the press and everyone was paying attention to it that week.
Kevin: Yeah, I think they do a great job of focusing their moments and having kind of a train of moments, right? And say, hey, this is gonna be train week, and they get people to come show up, and they've got a writer that's assigned to sit in every session and write about it, they're gonna write about it, right? Like, they're nothing else they're gonna do all day, right? And so I think, yeah, but they're able to create that.
Paul: You almost have to justify the fact that you've sat in this session for the last hour. You have to write something even if it's the worst thing ever.
Edith: Yeah. Well, it goes back to, I mean, a lot of startups get paralyzed by fear. They're like, what if we release a feature and TechCrunch writes about it? I'm like, TechCrunch is not going to write about your feature.
Kevin: Right, right. Generally speaking, right. I mean, it's just too many other things to write about. Because they're more interesting in writing about the startup you'd never heard of that got five million bucks or eight million. And that's more interesting than your feature, because they've already wrote about you, and people roughly know what you do, and you're not completely changing the world.
Winning Over IT
Edith: So on a detour from press back to something interesting you said, you told me that your app was loved by users and by IT. So in this world of rapid builds, how did you also win the love of IT?
Kevin: Yeah, so our view, coming from the enterprise world, is that you don't get to choose what software you use at work. And we felt that that was kind of a big sea of change that's gonna come over the next couple of decades, is that people, and especially with mobile, are getting to choose what apps they run on their phone and using them for work, whether IT has sanctioned them or not, right? You look at the Dropboxes of the world, obviously.
Paul: Consumerization of the enterprise.
Kevin: Consumerization, right? And so then as you do that, you say, okay, well, let's go build things that people love. But I think, again, Dropbox is a great example of building something that people love, but they've had trouble convincing IT that it's the right solution, where Box, I think, has done a better job of that, because Box went down a much more IT checklist, right?
And so when I look at it, there's a security and IT checklist that's required to get to the IT person. But there's no checklist or kind of easy way to get users to love you. And so we spent the better first year, first year and a half, focusing on getting users to love what we built and iterating like crazy and moving continuously to get them there.
And then the plan and the kind of the thinking was, we had all these enterprise trials and one very large enterprise customer with Acompli before Microsoft acquired us, was, hey, we're starting to work down the enterprise checklist, because they're gonna tell you what they want. It's very clear. I need this SSL thing and I need this HIPAA thing and I need this FedRAMP thing. They're gonna tell you what those requirements are, and they're not fun, but they're very explicit. In fact, your choice trying to negotiate, hey, can I do a smaller version of that?
So I think the mix there is, once you get it in there, one of your sales angles is, hey, oh, by the way, your users are already using this for home or on their work accounts. And they're gonna love it when you roll it out. And it's by far the best thing you'll trial and kind of a bake-off, right? And so that was our strategy to go to market, was to say, hey, let's make it so users love it, spend a bunch of time on that. Use that for press and branding and then go to the IT folks and get them to buy it.
Edith: Did they ever challenge you on the weekly builds and how that would affect the stability?
Kevin: So they didn't. I mean, I think the challenge was more around security and data locality and things of that nature. Initially, the speed of change didn't seem to be a huge deal. And mobile in particular, like I said, the actual thing that the users were touching, people are so used to mobile apps updating that that's not a problem.
I think it was more around data security and compliance around the data platform, and we had strategies to deal with that, either keeping the data, accessing it through a secure line or VPN or dedicated lines or using dedicated tenants in AWS or Azure and those type of things. So there was ways to kind of get around the compliance thing, but the key thing was really where the data was and data compliance and less about how quickly you're updating it.
Paul: Awesome.
Shipping Weekly Inside Microsoft
Edith: Well, thanks, Kevin, for coming in. Do you have any final thoughts around software development or continuous delivery?
Kevin: I don't. I think it's cool.
I mean, my thing is that the faster you go, things break, but the right things break.
And so I think when we got acquired by Microsoft, everybody's like, well, no other team can ship every week. This is, like, good, that was fun. That was fun as your little startup thing. And I had this deck that I present where it's like, hey, we ship every week and we drink beer on Fridays and release on Mondays.
And they're like, that's cool. And it was like a novelty thing. And so they were like, it'll just never work.
And so things that were like, it's gonna take a week to sign the build because we have all these special requirements. And we said yes, but at the end of the day, we're gonna push a button and it takes twenty-two seconds. That's what it used to be for.
And so they're like, no, no, we'll give you a special deal, it'll be like one day. And we're like, no, that's not possible. We need twenty-two seconds. And we finally got it to work.
But those kind of things, we had to break these internal processes and reviews, and a lot of times it wasn't just a process but a team involved to kind of help, that you can't have humans involved in some of these processes. So a lot of the continuous thing is how do you automate, and say, look, we'll do everything you're asking us to do, but just let us codify it so that we can push the button and it does it, and we'll trust that we're doing what you asked. And that's the way we kind of got back to that kind of high-speed delivery. And we're shipping seven days in a very, very large company.
Edith: Wow, I'd love to hear a little bit more about the tools that make all this happen.
Kevin: Yeah, so there's all kinds of craziness in there. So a lot of the tools that were involved were tools that most of us know today. It's Perl scripts and shell scripting, and it's PowerShell and .NET, and it's just code, right?
I mean, what does it take to sign an Apple build, right? You have to run Apple's code signing tools and have to have the cert and you have to have access to it, and so a lot of it was networked VPNs to be able to have the right access to the certs on the right boxes when you're making this. You used to have to use a smart card, so we need a two-factor auth to sign the build. And so you have to literally stick your smart card in my laptop, my Windows laptop, to get the cert to sign the Android build, right?
And so those kind of things, we said, well, look, there's got to be a way to automate this. How do we go get two developers? We're going to do two factors. It's going to be me and Paul, and we're gonna both say the build shipped and it's been reviewed, and then the automated process kicked off. And so those kind of things we just had to kind of reenvision.
But the beauty of setting, like, no, no, the goal is seven days, we're gonna ship every seven days, and just finding all the things that broke and kind of going after them and just breaking them and saying, nope, we can't do that. We can't do that. We need this access.
We don't need that access. This step was like, oh, we need to review, like, political correctness, right? And make sure that there's no kind of offensive words, 'cause we ship to so many different countries, and how do we do that? And it's like, well, there's tools for that, and that requires submitting it to this team and waiting. It's like, no, no, no, well, at the end of the day it's gonna be a code and run, like, how do we just ship that over there and automate it?
So it ends up being an automation thing. It's just much like DevOps today. There's teams all around the world with one or two people running thousands and thousands of servers with automation. And so if you kind of box the requirements into, it has to go quickly and it has to just work, you have to just kind of write the code to close that gap. And so that's kind of what we did.
Paul: So when you were making that transition, I guess when you went in, you were shipping once a week.
Kevin: Yeah.
Paul: And did you just refuse to break that once-a-week thing, or did you submit to their thing and then try to get it back to once a week?
Kevin: So we refused. So when we first got acquired, we were still shipping as Acompli and under our own rules, and we kept shipping, and then in parallel we knew we were making the switch to rebrand as Outlook. And so as we started the rebrand, it took us about six weeks to go from rebranding to get to Outlook to ship.
But we knew that as that was going, like, it's taking us six weeks to be like, well, where's the cert at? Well, how do you rename this?
What font should you use? Where's the logo file? A lot of things were like one-time things. And so those did take a while.
But I think the first couple weeks we were probably shipping more like every 10 days. So it was a little longer, and there was a lot of manual elbow grease and escalation emails and phone calls and things that were just kind of more elbow grease than required to kind of get us through the process, where we got it to point now where it's like, it wasn't fun developing the first, right.
Paul: Did you have any stuff where on your end it was automation, but the API that you were hitting was a manual process?
Kevin: Multiple, yeah. I mean, send us an email and we'll sign your build, and we're like, no, no, no, we need, well, what are you gonna do? So it was literally kind of interrogation, like, okay, when I send you an email and I ask for this request or I fill out this form, yeah, what actually happens? And then you have to sit there and pull on the string and say, okay, and then you copy to this, and then, okay, so you have a password for this thing.
Like, how does this thing work? Okay, great. And then you're gonna copy it to here.
It's like, okay, great. So it ends up needing to go into this directory that only you have access to. Okay, we need access to that directory and we'll copy it there.
Right. Okay, great. And then the tool automatically picks up and signs it. Oh, okay, perfect. So a lot of it was figuring out the human mechanics of it for sure, and then figuring out where the code is that we can go write it.
Edith: So, Kevin, you felt so strongly that the seven days versus ten was worth it.
Kevin: For sure. No, it was incredibly valuable to our kind of getting to where we were in Acompli. And we felt that if we had any chance of kind of hitting the goals that we had set for ourselves and that the board had set for us when they acquired the company in terms of growth, that we had to get to seven days and basically be able to iterate as quickly. Otherwise the targets we set were gonna take us three or four times longer, because for every time you lengthen the development cycle, I'm a pretty firm believer now that it takes you that much longer to actually get your results, right?
So the whole game is how many shots do you have on the goal, and if you can do fifty a year, you're a lot better off than the team that can only do twenty, right?
And so I think that was the fundamental thing, is that we can make fifty changes a year, and the size of the changes is not as important as how many you can make and how quickly you can iterate on that.
Edith: Well, great. We really enjoyed you sharing your stories.
Kevin: Cool. Thanks for inviting me.
Paul: Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of CircleCI, and Edith Harbaugh of LaunchDarkly.
Edith: To learn more about Heavybit, visit heavybit .com. While you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.
Subscribe to Heavybit Updates
You donât have to build on your own. We help you stay ahead with the hottest resources, latest product updates, and top job opportunities from the community. Donât miss outâsubscribe now.
Content from the Library
Third Loop Ep. #15, One Year of Progressive Delivery
On episode 15 of Third Loop, the Progressive Delivery team discusses why AI makes human judgment more important than ever. Celebrating their bookâs birthday, they revisit alignment, autonomy, abundance, and automation through the lens of AI agents. From evaluating models to recognizing good design, they explore what faster software creation still demands of people.
Third Loop Ep. #12, Shipping at the Speed of Understanding with Steph Hippo
On episode 12 of Third Loop, the Progressive Delivery team speaks with Steph Hippo, Platform Engineering Director at Honeycomb, about what happens when AI accelerates development beyond the pace humans can absorb. They explore safe defaults, evolving code reviews, and the feedback loops that help teams understand what they ship. Steph makes the case for reinvesting saved time in better decisions, software quality, and human connection.
Third Loop Ep. #8, Sounds Like Alignment with Melinda Fekete
On episode 8 of Third Loop, the Progressive Delivery team sits down with Melinda Fekete to discuss FeatureOps, developer experience, and modern software delivery. The conversation covers feature flag lifecycles, runtime control, experimentation, chaos engineering, and why progressive delivery is becoming even more important in the age of AI-assisted development.

