1. Library
  2. Podcasts
  3. To Be Continuous
  4. Ep. #9, Empathy’s Role in Software Development
To Be Continuous
51 MIN

Ep. #9, Empathy’s Role in Software Development

light mode
about the episode

In this episode of To Be Continuous, Paul and Edith host Sam Stokes, cofounder of Rapportive, to discuss empathy in software development, the tendency for micro services to become mini services, and the act of constantly updating software in a constantly updating world.

For more on Continuous Delivery you can follow To Be Continuous on Twitter at @continuouscast.

Shortening Feedback Loops
Surviving an Overnight Hit
Early Integration at LinkedIn
From a Startup to LinkedIn's Teams
When Microservices Become Mini-Services
Resources Instead of Services
What Testing Is For
Confidence as Common Ground
Empathy Between Product and Engineering
Continuous Deployment for Hadoop

transcript

Sam Stokes: LinkedIn adopted a version of the microservices model. It's not a way of writing software, it's a way of running software. If you squinted it, kind of is continuous delivery.

Paul Biggar: I can't tell as you're describing this whether this is a good thing or a bad thing.

Sam: There's a strong tendency for a microservice to grow up into a monolith again. A natural thing is to collapse a lot of those responsibilities into a single mini-service.

Paul: Hi, I'm Paul Biggar, founder of CircleCI.

Edith Harbaugh: I'm Edith Harbaugh, CEO and co-founder at LaunchDarkly.

Paul: And you're listening to To Be Continuous, a podcast about continuous delivery and software development.

Edith: You can get in touch with us anytime at our Twitter handle at continuouscast.

Paul: The show is brought to you by Heavybit. To learn more, visit heavybit.com. And while you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.

Edith: On this episode, we're joined by Sam Stokes, co-founder of Rapportive. We talk about empathy and software development. How many services are the new microservices. And constantly updating software in a constantly updating world.

Shortening Feedback Loops

Paul: Alright, Sam, so what's your favorite thing about continuous delivery?

Sam: I like to think about it in terms of shortening feedback loops. A lot of things that startups do can basically be looked at as doing something and then waiting to get some feedback on it and then deciding whether to do more of that thing or to do a different thing.

And continuous delivery is, people talk about sort of reducing the batch size and then also about shipping stuff faster. And clearly having some kind of automation. And continuous delivery is going to help you with both of those things.

Paul: So is this specifically on the, sort of on the journey of a small startup towards product market fit? Is that kind of the sense that you're thinking of it as?

Sam: I see it as applying to teams of different sizes. So for sure in a small startup you've got, you need to iterate fast, you've got all these product ideas.

But I think even if you're a small team in a big company, as I was recently working on, you still have a lot of the same pressures. You have product managers who want to try new ideas. You have customers who are sending in feedback all the time, some of which you want to listen to, some of which you want to tune out.

It's sort of in some ways a lot like being in a startup, at least ideally it would be. And a lot of the problems you can run into are basically about not being able to close those loops quickly enough.

Edith: Sam, you're somebody who knows a lot about loops, so now might be a good time to introduce yourself.

Sam: Sure. My name is Sam Stokes. Until recently I was working as a staff software engineer at LinkedIn. And right now I am, I guess, a freelance software consultant. Put it that way.

Paul: I feel there's a little bit of history missing there.

Sam: All right. Going back in time a bit, so I was a co-founder of a startup called Rapportive. That was a plugin for Gmail that when somebody emailed you would show you a social profile of that person in the right hand side next to their email. So that you could get some insight into who you were talking to and put some more humanity into your communications with people, was sort of the way we saw it.

Paul: That's still the only plugin to Gmail that I use.

Edith: I use it too. And it's for exactly that reason because I do a lot of customer development and it helps so much to empathize. To see that this is a real person. You know, to see their face and you feel a connection.

Paul: It's interesting this with kind of customer communication. One of the tools that we use is Intercom, and once I started using Intercom it was the same sort of feeling I got the first time I used Rapportive. I feel that there's a human at the other end of this and I feel I know who that human is, and it just changes how you communicate with the people.

Sam: Yeah. It's really motivating to hear and to see other examples as well. We really felt like email and sort of a lot of business communication in general can be sort of cold and formulaic. And yeah, just bringing that human element, even just seeing a face. I haven't used Intercom, but I'm guessing they have some similar ideas.

Paul: Similar, when you send out messages or reply to someone that there's the person's face there.

Sam: Right. We had a sort of plug-in API in the, it was a sort of private beta version of our product. And one of the things people used it for, so this would let you plug into the sidebar. People would plug it into their customer database, and so they could see something like, they get an email from a customer, it shows them how long they've been a customer, how much they've spent in their store, what version of the app they're running, that sort of thing.

Edith: Was your original idea to do that sort of empathy or is that something that just you iterated into?

Sam: The empathy was where we started. There were lots of ideas about taking all information from all these different sources, public sources like, well, truly public sources like Google, quasi-public sources like a Facebook profile if you're friends with them on Facebook, and then private sources like stuff in your company database.

All stuff that you would be able to search for if you could be bothered to go and do the searching, but doesn't necessarily bring itself to you while you're talking to that person. And yeah, it was very much, what should you know when you're talking to this person in order to make them feel like they're interacting with a real person and an empathetic one.

Surviving an Overnight Hit

Paul: So I remember when you launched Rapportive it was like, or it felt to me like it was an instant hit. And maybe that was just the overnight success that comes after years of trying. How did that go and what was the kind of software delivery process of that?

Sam: That's a great question. I guess there's two parts to that really. So how did it get there and what was our process that allowed us to survive it?

So I guess, how did we get there? There was a certain amount of luck in that process, as there probably always is. We'd built our prototype and it was a pretty minimal prototype. You know, it had the functionality.

There was like this awful demo website up. And the only reason there was even a website up was we were trying to raise a seed round at the time.

We were applying to YC. Y Combinator requires that you have a demo site that's public. They don't like it to be possibly protected. And so we had this thing up where you could install it and we had like 20 users.

One of our friends decided that he was going to email a friend of his at one of the blogs, I think it was ReadWriteWeb, and say, you know, hey, you should check this out. And his friend checked it out. And it turned out that one of the demographics that really liked the product that we built was journalists, because they get a lot of email from people they don't know and they want sort of an extra nugget.

I guess that was one of our takeaways, is like, if your startup is really useful for journalists, that's a great way to get some early press.

Edith: That's what worked for TripIt, because journalists travel a lot. So TripIt just got all this free press basically because journalists used it, loved it, blogged about it. Unfortunately, journalists do not do continuous delivery so much.

Sam: So how do we survive that? Because what happened in that 24-hour segment was that we went from 20 users up to 20,000.

Paul: Wow.

Sam: So I guess one part of it was our CEO had made the original platform choice. His name's Rahul Vohra, he has a computer science degree, he's an awesome technologist, but his focus is not generally on writing the software.

But he'd been writing the prototype and he was like, I need the quickest thing that I can get to get this to market, so I'll put it on Heroku. Which turned out to have been a life-saving decision, because when the 1000X traffic spike arrived, we just added some more dynos. So that was how we didn't burn down.

What did we do after that? I guess in terms of software process, we were pretty young as a company. So it was pretty much hack away, read all the tweets. But there was a lot of what you might call customer development, more than software development at the time.

So we pretty quickly decided that we were going to try and respond to every single tweet and every single email and so on, to like try and live this idea of a good customer experience. And I think that helped to sort of amplify the viral effects of that press spike, because people were sort of retweeting the things we were saying and getting a generally good feeling about us.

And so a lot of those tweets were like, oh, I wish it did this, or I wish it did that, or I wish you could see Twitter profiles in there. So that it was a little bit of sort of customer support-driven development. Which obviously doesn't scale, but it can be a pretty good way to build some goodwill early on. And I think a lot of our users were sort of quite passionate about using us because they felt like they got that personalized attention.

Paul: So did the, well, when you scaled the dynos, did it just work?

Sam: It worked perfectly until we hit the rate limit for the API that we were getting all our data from.

Paul: The LinkedIn API?

Sam: At the time we were using a service called RapLeaf.

Paul: Oh yeah, yeah, I remember them.

Sam: It wasn't their main product, but they provided an API which, so basically the thing that Rapportive does, which is take an email address and produce social profiles about that email address. At the time, that was entirely just a UI wrapper around RapLeaf. We ended up sort of building our own version of that.

And so we'd been using their API on a sort of demo basis, because again, 20 users. Yeah, we immediately had to learn about things like request queues, so that our app wasn't just falling over waiting for the API, and then frantically biz-dev-ing to try and increase our rate limit while at the same time figuring out the best ways to get around the rate limit without upsetting the biz dev deal.

Yeah, we didn't really have much of a sort of continuous delivery process at the time, but I guess not as many people were talking about it. This was 2010. I think the first blog posts had come out from IMVU and so on, but I think it wasn't as, yeah, there weren't all startups supporting it.

What we did have was the Heroku deployment model, which was basically like within five seconds of writing the code you can deploy it to production. Which if you squinted it, kind of is continuous delivery. Because you're saying all of the bits of your deployment that you have are automated.

We didn't really have many tests at the time. So the tests we had were automated because there weren't any. Our deployment was absolutely, you do a git push. You don't have to like move any servers around or anything like that.

Paul: It seems like the testing model around, or the continuous delivery model around a Gmail plugin is probably going to be a fairly difficult process anyway.

Sam: We never really had a good answer to, you know, what happens when Gmail changes the UI. The surprising part was it didn't happen that often. We thought it would be breaking every week, and we'd need all kinds of canary accounts and end-to-end testing to monitor that kind of thing.

In the end, it happened rarely enough and we noticed quickly enough that we could usually just fix it on the spot. So even when we got better at testing and that sort of cycle of things.

Paul: We had a similar problem or, I guess, solution with GitHub. So GitHub used to be super flaky around two, two and a half years ago, when they were still on Rackspace. And whenever something would go wrong with GitHub, we would find out immediately.

All of the builds would start failing. And so before they knew that there was a problem, we knew there was a problem. And we would often like declare on Twitter, oh, we think GitHub's having a bit of an issue, and then two minutes later their status would tweet it, and then we would retweet their tweet.

Edith: So they were using you for their free testing.

Paul: Well, our customers at least.

Edith: No, I mean that you were acting as their tester and telling them that.

Paul: Well, I mean it was very, you know, automated on our end, but we didn't have necessarily automated fixes at the start. It kind of happened every, you know, two or three months, and they would swear that this is the last time.

Edith: It's always the last time.

Paul: Yeah. And in the end it happened like six or eight times, and by that time we had built a thing that allowed, you know, certain fixes, but it wouldn't be quite, you know, a fully automated process that you would expect if it was happening all the time, because it always seemed like it was gonna be the last time.

Sam: And it's probably always happening for a slightly different reason or in a slightly different way. It's actually, it's an interesting question to me in the subject of continuous delivery in general. People always talk about every time you commit, you run your tests. Every time you commit you deploy.

But that's actually only one of the kinds of events that you're dealing with in a software lifecycle, right, when something's actually running. I mean, sure, you're changing the code, but also the world's changing around you.

And I feel like you always end up needing a little bit of both. You need something that's gonna trigger when you change it, but you also need sort of an eye on everything you depend on.

And of course even figuring out what all those things are can be interesting.

Paul: So I wrote a blog post about this on the Heavybit blog, I think, a while back, and it was about third party APIs, you know, how do you test third party APIs? And the conclusion that I came to was that, so if you run the tests as part of your test suite, then if something breaks, your test suite will be broken. But also some things will just like randomly break in production.

And then of course you can't deploy because your test suite is broken. So you're in kind of the worst of both worlds. And so what I considered is that you need something which, you know, is just permanently monitoring.

Like whatever API things you do, constantly hits your personal accounts or your test accounts for those things, and then stub it out in your test suite. And that way you kind of get that best of both worlds. Your test suite doesn't fall over and you get an actual monitoring event when the third party service goes down in production.

Early Integration at LinkedIn

Sam: I can tie this to an experience from LinkedIn actually. So LinkedIn has the model where there's a staging environment and a production environment. And the staging environment is, I don't know if it's technically called a staging environment because it's not exactly production like, but what they actually call it is early integration. Because it's the environment in which teams integrate with other teams.

And see, the idea is you deploy your stuff into EI, early integration, as soon as you can, so that if any other team depends on you or vice versa, you can find out there before the thing hits prod.

But what you've effectively got in EI is this constantly changing environment full of dependencies. And you have exactly this problem, and different teams were choosing different solutions to, well, let's write some integration tests which actually hit the dependencies and then we'll run them in our test suite when we deploy to EI, or in the build step.

Paul: I can't tell as you're describing this whether this is a good thing or a bad thing.

Sam: I think there wasn't a clear conclusion, but my conclusion is that it doesn't work for the same reason that you're saying.

You need to know that if your test suite fails, it fails because of something you did. And then you need something else that can fail because of something someone else did. But if you're ever unsure which of those things it is, then the signal is lost for both systems.

Edith: Yeah. When I was in London I talked to somebody and they don't have a staging system at all for kind of these reasons. They're like, it's just easier to put everything into production. They feature flag it and just know whether it's an issue or not. They're like, maintaining a staging system is sometimes even more work, even much more work.

Sam: It honestly is, like, there was actually significant amounts of effort put by every team into just keeping their stuff running, the EI instance of their stuff running, and to some degree making sure that if you had any data that your service needed to run, that there was a representative set of data in EI that the teams that were depending on you could use.

This isn't production data, right? For sort of user privacy reasons, this is all fake data, which means you now have to maintain a fake data set for every single service, and very few teams had the bandwidth to do that.

Edith: And then also then you always have to remember, oh, let me go to staging, and then it's just this drudgery.

Sam: Well, the tools have ensured that you did it in that order.

Paul: Well, so I think the problem is having a single staging. I don't think that there's necessarily anything wrong with the concept of deploying something and possibly pushing a small amount of traffic into it or pushing some.

Edith: I think you should just go directly to production and feature flag it all.

Paul: Well, okay, I mean I guess to a certain extent they're the same thing. That you can have a staging environment and, you know, a small amount of your traffic is mirrored into that, which is essentially a feature flag situation.

Sam: Well then, I've heard the word used for that is the dark canary. It's canary in that you've deployed it into a real production machine, but it's dark in that the users aren't actually seeing it.

But the interesting question then is, what if that is write traffic? Like it works great for reads, you can just mirror them. But if you're mirroring writes, now you may have double writes, or you may have, you know, maybe there's a bug in the thing you just deployed.

Edith: That's why I argue for just put it all in production. Just the effort to maintain a system to test on is huge.

Paul: So what we do is people can spin up their own staging environments. And it'll just spin up a whole staging environment. You can do some tests on it. So people do that very often for load testing, or they, you know, put traffic against it of their own creation.

One of the problems that we often run into is, you know, how do builds interact with other builds? And if you're doing anything that involves LXC or any of the more platform as a service kind of things or queues, those are very difficult things to do in production. Or at least if you do it in production and you fuck it up, then you're in real trouble.

So those are things where we'll have people spin up staging instances. We've set a script, you start up your own staging instance, but you can have multiple staging. They don't talk to each other. It's not an early integration situation.

Edith: That's interesting. That was actually one of our early company ideas, was just to allow people to do builds on demand. We called it Continuously. Anyway, enough about me.

From a Startup to LinkedIn's Teams

What was the biggest difference you saw between moving from like a startup style up to LinkedIn? What was the difference in cultures?

Sam: I'd say a big part of the difference is in a startup, at least at the size we were at, we were five people at the time of our acquisition. You feel like you're one integrated unit that sort of, you know who's doing everything and to a certain extent everyone is doing a bit of everything, and certainly you always know where to go if you want something done and if you want something prioritized.

As a startup scales and certainly when you're a big company, you very much specialize, you sort of split off into these groups that do things. And so as I sort of alluded to when I was talking about this early integration environment, that's just teams within the company.

So LinkedIn adopted a version of the microservices model. Which as I've heard it described is, it's not a way of writing software, it's a way of running software. Because what microservices gave to LinkedIn was the ability to divide their software by the team structure.

So rather than saying we have this gigantic monolith and this is your bit of the code base, but it's still got to be deployed in the same way, teams could make their changes in an environment that was somewhat controlled, and then they controlled their deployment schedules. They could go continuous deployment if they felt like their tests were ready for that. They could do their manual deploys if they couldn't, or if they had some extra special deployment requirements.

And so you had all these different teams. Often, LinkedIn's a pretty big company, so often in different buildings or sometimes even different time zones, that would be doing the thing that you were currently thinking about. Oh, we got a URL and we want to see the contents of the news article that that URL represents. You may have to talk to at least one other team, if not two other teams, to get that.

Which is a really interesting and powerful model, because it means you can make progress on all of those things in parallel. You get these very interesting problems of sort of knowing what's going on between these different teams.

There's the technological drift of, oh, you changed this thing and now we're not compatible, but there's also the really subtle mission drift of, at the start of the year we all decided that we were going to do this thing, and we were going to build this feature, which required all three of our teams or all nine of our teams to work towards this thing, and you're gonna implement this feature, which we're gonna depend on in three months, so that the third team can do this thing.

And then it turns out that two months in, the first team realized that to do exactly that was going to require some significant refactoring they weren't ready for. And so they slightly changed their plan, but they didn't realize, because it's hard to tell, that that change was going to disrupt other people. So they continued along the track that they were going on. And then when Team 2 was ready to start integrating, they find that it doesn't support this thing that they were depending on.

Paul: Interesting.

Sam: And so you then get these sort of escalations or backpedaling, or, I mean, you get some very good discussions coming out of it, because usually it happened for a good reason.

Paul: There's an interesting parallel here in Amazon's, one of their learnings of having a microservices model, or at least a services model, is that when something goes down, it often takes a long time to find the team. There's an escalation that follows to another escalation, to another escalation, as people try to figure out what service is actually responsible for the outage. It seems a problem of communication structure.

When people change their things, it's a new person that has to be talked to, a new team that has to be reached, and maybe you go through several teams before you find out that the API endpoint that you're using is actually served by another team, which is served by another team.

Edith: It's turtles all the way down.

Sam: In some ways, the team model, actually, you can see it as an abstraction layer. So in theory you would actually have teams which in software terms you might call an adapter or a proxy, because you would have teams a large part of whose job was to effectively route calls between different other teams.

So the team I worked in was doing a lot of interesting things, but one of its roles was to integrate work from lots of other teams. And so we would get questions from all around, because as the integration point we were the most visible. But a lot of the questions we would get were things we weren't fundamentally in control of.

And so we'd be like, oh, you need to go ask that guy, or we would ask that guy for you. It's an interesting place to be, especially when people don't necessarily realize how much you're delegating things down to other people.

When Microservices Become Mini-Services

Paul: Why was that the split point for the service or the team? So I'm interested in the question of why do people create microservice, or when do you create a microservice, right? And in this case, or LinkedIn's case, it seems you create a service for each team. And so like, why did that team exist?

Sam: There's an interesting corollary to create a service to each team, which is if you really need a service which performs an integration role, you sometimes need to create a team for that service. Or at least that is a natural cultural assumption.

Paul: Yeah, yeah, yeah. If that's how your company works, that's how it works.

Sam: I don't think that's exactly what happened here, but what you do have is, sometimes you have a team that needs to do things, but the service they own isn't really micro anymore. Sometimes you have a service which is in an important request path or whose job is fundamentally gluing things together. And then its job can become more complicated than you can necessarily represent in the sort of idealistic Unix model of, oh, it just does one thing and that's all it does.

So let's say you're taking data from a bunch of different content sources and you want to blend them together in some way. And so you could say, oh well, we'll just have a service that does that sort of scatter-gather and aggregation, and then we'll give it to another service to apply some ranking on top of that. And then we'll give it to another service to apply business rules like filtering and abuse detection.

Except it turns out if you're going that many hops, first of all there's the inefficiency of having to throw this data around. Sometimes it's large amounts of data. Especially if what you want to do is some kind of relevancy ranking on that data. Relevancy is being a broad term for figure out what the user really wanted to see.

In the ideal case for relevancy, that requires access to all the data you could possibly have about the user, the article, users like that user, and so on. And if you need to bring that into more than one place, that's usually pretty hard, even to get into one place. And if you need more than one service with access to that kind of thing, then that gets a bit nightmarish. And so a natural thing is to collapse a lot of those responsibilities into a single mini-service.

Edith: So microservices become mini-services. And then do they eventually just grow up and become services?

Sam: One of the really interesting things I saw, so LinkedIn started off with monolithic services, moved itself to microservices, iterated back and forth along the spectrum a few times, there's a strong tendency for a microservice to grow up into a monolith again. Because as soon as you have 20 people committing to the microservice, well, it's a monolith now.

Paul: Yeah. So I think there's this interesting question that you see a lot of people who are new to microservices not really know, where do you put, you know, what is a microservice, where do you put the service boundaries. And there's a lot of pragmatic things that you can do.

So you can say, like, there's the team boundary, there's the, you know, very obvious API boundary, there's things that have different deployment schedules, things that have different reliability or uptime requirements. I think those are the kind of major ones.

But it's interesting to hear kind of where that breaks down, that like, if you have a model of your one team, one service, then you're not gonna have a microservice anymore. You're gonna have teams managing a bunch of different microservices, and then they have to become a bunch of different micro-teams to manage those microservices.

Resources Instead of Services

Sam: I think the way that LinkedIn structured it was actually pretty interesting in that regard. And that's kind of why I said sort of microservices at the start. So the actual model LinkedIn used was based on, they sort of went all in on a RESTful model.

And so what they actually talk about is REST endpoints. And so you would have an endpoint, which is defined pretty much by the name of the URL of that endpoint, or, I should probably say, resource.

So you might have a resource which is members, and you might have a resource which is articles. And there's a registry that says, if I want to access this endpoint, what physical machines do I go and ask? And services query that at runtime.

So if you want to get something out of the members resource, you do that lookup at the time that you're making the request. What that means is you're not actually talking to a service, you're talking to a resource. And the choice of whether, for example, more than one resource might live in a single service, that's up to the team that owns that resource.

Paul: Oh, I see.

Sam: And so if that team wants to have three similar resources live in the same service, because that works for their team model, their iteration speed, their uptime requirements, etc., they can do that.

Paul: Standard computer science solution, just add another layer of direction.

Sam: And yes, so you had some big services, some very small ones that just owned a single resource.

Paul: And it wasn't clear whether you were hitting a big one or a small one.

Sam: Right. And I guess the view was you didn't really need to know.

Paul: Right, right. In fact, it's great if you don't know.

Sam: And that sort of helped with knowing who to talk to as well. Because you could use the registry to figure out, I think there was an owners list for every resource. So you could say, I'm hitting members, members is returning an error.

Paul: It's almost odd to have registries keyed by URLs. Like you would think that the URL is the destination.

Sam: Not sure I see the distinction.

Paul: So you would think.

Sam: Oh, I see what you mean.

Paul: Yeah, a URL is the place that you go.

Sam: I guess there were two levels. There was the resource, which is sort of a logical URL. Which it would be named after, I mean, it wouldn't even have a host name in it. It would be, they called the service D2.

It stood for dynamic discovery. And so the URL would be literally D2 colon slash slash members. And that would be the whole thing.

Paul: Well, so I do think this is actually very interesting, because when people tend to build their microservices, you know, every service has a URL. And if your URL, you know, has an actual machine name, or maybe not a machine name, but, you know, it's something with a DNS name.

Sam: It probably has a load balancer name.

Paul: Exactly. It has a load balancer in front of it. And then, you know, if you're providing these endpoints to users under some other URL scheme, and you have to have an actual load balancer in front of it rather than a, you know, thing that can be contacted asynchronously.

Like, there's a thing through which traffic is routed rather than a thing, you know, which tells you where you're going. I guess the difference between a load balancer and a DNS resolver.

Sam: Right. And this went along with also having a library that everyone other company used which did client-side load balancing. So almost nothing sat behind a physical load balancer.

Paul: Oh, interesting.

Sam: And so part of this D2 resolution was figuring out, what hosts should I use in my client-side load balancing?

Paul: So it told you the set of hosts and then the client-side load balancer, like, picked randomly between them or something.

Sam: Yes. Or it may have told you the path in ZooKeeper to subscribe to so that you'd be notified if a host went down, et cetera.

What Testing Is For

Edith: Well, this was a lot of microservices. So I had a macro question for you, Sam, which is you have some interesting ideas around testing.

Sam: I have some interesting ideas. I think a lot of people miss the point of automated testing.

Edith: Paul might know a couple things about this.

Paul: I'm excited to see what this is.

Sam: I think that the word testing confuses people, because it sounds like a thing that you do to a piece of software or to a product in order to find things that are wrong with it, or a thing that you do to prove that the thing works.

And then of course there's lots of quotes about how testing cannot prove the absence of bugs. And I think that even though everyone agrees testing cannot prove the absence of bugs, that's still what people sort of talk about when they're talking about testing. And I think that causes two kinds of problems that come from the same root problem, which is people don't understand the value they're meant to be getting out of testing.

I mean, this is, I'm overgeneralizing, but I think a lot of people think they're doing testing to find bugs or to ensure quality. And so one thing people do as a result of that is they've seen testing being done somewhere and there were bugs anyway. Or the quality wasn't great anyway.

And so they concluded, well, I don't need to do testing. It's clearly just a waste of time and it's not going to find my bugs, so why should I even bother?

Paul: So what is the purpose of testing?

Sam: There's this great quote from a kind of obscure source, which is one of the people on the Debian project who run Debian Linux.

Paul: Software known for its reliability.

Edith: Paul, did you make a joke?

Sam: What they were trying to do was, the title of the post was a proposable for an always releasable Debian, because Debian as a whole, the project that maintains an operating system and tens of thousands of packages, was trying to move to a continuous delivery model. They used to do releases every few years. And people loved them for their stability and at the same time mocked them for how old their packages were, because they released every few years.

And they were like, well, we have tens of thousands of things, they interact with each other, we don't own any of them. How the hell can we possibly

Edith: It's open source.

Sam: Right, it's open source. And so this was a proposal to adopt a lot of the ideas around continuous delivery by saying we're gonna have a trunk that consists of a known releasable version of every package. And we're gonna write a big suite of tests that essentially sets, the phrase he used was a minimum level of quality. A minimum accepted level of quality, I think.

So it's not to prove that there aren't bugs. It's to set a bar that everyone understands and can rely on and replicate. And so if you do find another bug, okay, sure, but you can add it to the test suite. But in the meantime, everyone knows what is known to work and where the gaps are.

So to me, testing is about establishing areas of confidence and about increasing the confidence that you have in what you've built and how it interacts with other things.

Edith: Kind of like the known knowns, the known unknowns, and the unknown unknowns.

Sam: Right. If you have a test and it passes, that's a known known. If you have a test and it fails, you probably shouldn't ship that, but at least you know it fails. If you don't have a test, that might be fine because you might know that you don't have a test in a certain area.

Edith: Yeah, we used to call that, like, 40% coverage or 50% coverage.

Sam: Right. There's always some part of your system that's really hard to test. And sometimes it just isn't worth the effort to figure out how to test that. But it's useful to have tests everywhere else so that you know if something goes wrong, it's probably in that area.

Edith: When we were just moving fast, our QA would just complain that we were like, we only have time for 50%, you pick.

Sam: Right. Yeah, because you have to prioritize your effort in it. And I think just to complete the thought there, I think that idea of testing being about increasing and establishing areas of confidence makes much more sense in the context of continuous delivery.

And I think that's part of why people have missed the point, because continuous delivery is also kind of new. People are doing testing in the context that the tests are going to be run occasionally by developers when they feel like it. Or maybe by a QA team whose job it is to effectively do a really slow continuous delivery process and gate releases.

Paul: So I'm a tiny bit confused by, I guess I don't quite get the implication. In fact, I'm not 100% sure what it means to increase the, I'm sorry, say it again.

Sam: Establish areas of confidence and increase your confidence in.

Paul: So what does that mean? Confidence that it works?

Sam: If I was to take an extremist viewpoint, I would say the point of testing is to allow you to do continuous deployment. In which case it's confidence in your ability to release this stuff.

Paul: Gotcha, gotcha. Yeah, I don't think that's too controversial at all. I don't think I've heard it said that way. The other ways I've heard it said are, you know, reducing the risk of deploying software, which I guess is very similar.

Sam: To me that's the flip side.

Paul: Yeah. Or to prevent regressions, which is again a very similar concept there.

Edith: I think every time you release there's some risk though. Like every time you change something, you're by very definition increasing the risk.

Sam: But do you know where you're taking risk and do you know how much risk you're taking? Often the answer is, we'll see what happens.

Paul: So generally when you're taking that idea, it's not just, or, you know, the idea of increasing the confidence that you have in your software, it's not always testing that's the solution to that. Like monitoring is a thing that I find is particularly, you know, slow rolling and feature flags, which I don't know if you know, but a company provides feature flags as a service.

Edith: I am super excited.

Sam: Is that a company called LaunchDarkly by the chance?

Paul: This is now a meme though, almost every meme.

Edith: No, thank you. Thank you, Paul. And I'll also point out that CircleCI is now using us for their own feature flagging.

Paul: It's awesome.

Edith: The station break is now over. Back to discussion.

Confidence as Common Ground

Paul: So the monitoring and feature flags and that sort of thing are all, you know, part of this idea of, you know, can I reduce the risk? You know, what is the confidence I have that the software goes out?

Sam: And this is exactly the conversation I wish people would have with, for example, their product managers when they say we need to write some unit tests for this, and the product managers say, well, why? Because you have bugs anyway and we don't have time, we have to ship now.

I wish people would talk about the package of things you do to increase confidence. We're going to need some testing here, we're going to need some monitoring here.

Paul: Right. Well, I think as well, when there's that conversation, and I feel that there's kind of parallels in the security world, that you're having that conversation with PMs and there's this sort of moral reason for doing it. We're developers and we're the sort of developers who write tests or who value security or whatever those things are. And we refuse to release something which, you know, isn't properly tested or whatever that thing is.

And so that conversation with a PM inevitably goes sour, and I think that a lot of, you see comments on Hacker News where people hate product managers or scribe all sorts of, you know, malintentions to product managers, which is never a feeling I've had.

Edith: Never?

Paul: No, I've never, anyway. Side note.

Edith: Uh-oh, Paul's like, I refuse to get dragged into this.

Paul: But if the conversation is not, you know, we write unit tests because tests are important, or, you know, 80% coverage is the, you know, thing that God wrote on a stone tablet and we shall do it. It was, you know, this helps us, you know, maintain confidence, and we know that next week's release is going to be on schedule because we are confident that we won't be breaking these releases.

Sam: That's exactly it. I think the reason I like using the word confidence here is I think it's common ground. Because the developer wants to be confident that what they're being asked to support will work to the degree they think it's going to work.

The product manager wants confidence that what got built is actually what they wanted. They want confidence that they'll actually hit the milestones that people have told them they're gonna hit. And all of those things are impacted by things like technical debt and a complete lack of testing.

Edith: Yeah. I think you're talking about product managers kind of as a monolith themselves.

Sam: That's fair.

Paul: Is there a microservice model of product management?

Edith: Well, no, then you have this axis where you're like, it's actually this, the three-axis, you know, quality, time, features. And at any given time a good product manager is trying to balance between all three.

Sam: Right. And I think you can have a conversation like, hey, Jane developer, we really need to get this extra feature into this coming launch because there's a customer that really wants to see it. Now I'm Jane developer, I'm not gonna change my voice.

Well, I don't really think we're gonna have time to, sorry. I don't really think we're gonna have time to get this feature because it's kind of late in the cycle and we'd have to do a rush job.

Well, it's gonna be really important to hit the window, and it's also, it's just one customer, so we don't necessarily need it to be at 90% confidence. Is there anything you could do to get it done at a 50% confidence level for now? And we'll acknowledge that we have to fix that up later if we want to take more customers on.

And that might be the end of the conversation, or Jane might come back and say, well, I think we could do that, but it might impact some other part of the system, because we'd have to make a change to this service, and I'm not sure it would stand up under the load, so it would actually reduce my confidence in the other parts of the system that affects all customers.

Yeah, neither answer is right or wrong, but that's something that both sides can take part of.

Paul: It's interesting your use of a scale of confidence there. Because you do see this concept. You hear people talking about a minimal viable product, which is, you know, a lower confidence model. Tracer bullets is another one.

Edith: MVP is one of the most misused terms. I hate it at this point.

Paul: So it's interesting that this kind of provides a figure that you can adjust. And, you know, there isn't just, is this a minimal viable product, there's, you know, do we have 10% confidence that this works? Do we have 50% confidence that this works?

Sam: And I think the numbers people will pull out of a hat, but people have some common ground on vague numbers. Everyone roughly knows what they mean by 90% confidence.

Edith: And I think it also comes down to the culture. I mean, I've been at companies where, you know, everybody says we're fine with stuff breaking, and then stuff breaks, and then, you know, there's a lot of, and then that's when people start not trusting, oh, it's fine to have 50% coverage, and saying we can have 50% coverage, but if this thing breaks I get yelled at and perhaps even fired.

Paul: And I think a lot of this comes back to, you know, the team dynamic and sort of the whether you have a good faith belief that the promises that you're making are believed to be given in the same, or are understood in the same context in which you're giving them.

If you have a boss, a product manager, a colleague, you know, whatever it is, and you tell them, you know, 50%, and they hear oh great, that's gonna be 100%, and I'm just making whatever promise, it's not gonna work. That's gonna end up being a disaster in the future. But if, good faith I think is a very good term for this.

If your team works and people are, you know, good colleagues to each other, and there's a good team dynamic, and people hear what you say when you say it, I think that works very, very well.

Edith: Yeah, I agree.

Sam: I guess one interesting thing is people have tendencies to overreport or underreport their confidence in something. And that can depend on the culture and also on their personality. You have people who are always reluctant to sign off on anything because they know there's more they could do. Or they're used to having their promises taken at more than face value in the past.

Paul: There's the standard trope where someone says, you know, how long will it take to deliver this feature, and they say two months, and then someone says, you know, how about one month? And takes their estimate and just, you know, asks them to cut it in half.

You know, if instead the conversation is, well, you know, we can get 90% confidence in two months, and someone else says, well, yeah, I don't need 90% confidence. I actually need 35% confidence. Would we be able to get that in a month?

Then maybe the response is actually, you can have 35% confidence tomorrow, or, you know, something else along those lines, where people actually are working towards a common goal, rather than there being conflicts in between what people are asking for and what other people are delivering.

Sam: And that's huge if it's no longer two differing interests trying to find a compromise, but if it's two people with different skill sets working towards the same, working towards a business goal, or even figuring out what the correct business goal is in the face of these constraints.

Paul: There's a framework that I used early in Circle where often when developers are talking to each other, they talk about implementations. So they say, I think we should do it this way, and someone else says, I think we should do it this way. And, you know, people throw out lots of reasons why they should do it their way or their favorite way or whatever.

And the way that we overcame this is we started by stepping it back a little bit and we had three levels. We had problems, also known as goals sometimes, a solution and implementation. And so whenever we'd be discussing things on the implementation level, we'd take a step back. You know, what is the problem that we're trying to solve? Get on the same page with the problem that we're trying to solve.

And then decide, based on the problem, what is the solution that we're trying to achieve, and that can include things like confidence levels and that sort of thing, and then when you get down to discussing implementation of that, it tends to actually be relatively trivial. Because you've agreed on the problems, or the goals or whatever, you've agreed on how you're going to achieve them, and then, you know, the implementation just kind of falls out naturally.

And it's kind of what you're saying about implementation is kind of the conflicting schedules trying to reach a compromise and the problems and the goals is more, you know, everyone on the team getting aligned around a common purpose.

Sam: Alignment.

Edith: Yeah. Alignment, particularly so back to the time. Sometimes it turns out that there wasn't really an understanding of when something was needed.

Sam: Right, yeah. People don't even necessarily communicate that on either side.

Empathy Between Product and Engineering

Actually, I wanted to ask you, Edith, since you stood up earlier for the internal reality of product managers is not just a monolith. What do you wish you heard from the teams you've worked with that you weren't necessarily hearing?

Edith: Well, so it's funny, so I started off in engineering. So I started off in engineering and then I moved over to products. So I've been on both sides.

Paul: And that's what every product manager says.

Edith: Well, you're a product manager now, Paul. So when I was an engineer, I was just like, oh my God, here comes a product manager again asking me for more stuff. You know, we're already totally overbooked, realistically a week behind schedule. We're lucky if only that. And also I was working at bigger enterprise companies, and also what happened is we would get a lot of bugs from the field.

So it was like this constant thing of not only did we have to build new features, we had a platform burning out from under us because we were installed software, and we had to do an update for, like, you know, every time somebody came up with a new app server, we would have to do a major rev.

Sam: And probably a convincing reason why each of those things was urgent.

Edith: Yeah. And then so the product manager came up, it's just like, just no. It doesn't matter what you're about to ask for, just no. And then I became a product manager and I was like, oh.

Sam: These things are actually happening.

Edith: Yeah, and I think the best thing to semi-answer your question is empathy. And I think it's also very good to have some tension between product engineering.

Sam: Oh, that's interesting. Why is that?

Edith: You know, you gotta ask for stuff. Like, I remember once, geez, we were like a month late on something and the VP of marketing asked for one more thing.

And we were already a month late and I was just like, no. And then he said, you know, very nicely, he was Irish. He's like, I don't know 'til I ask. You know.

Paul: Oh, okay.

Edith: You know, like, in marketing or product, sometimes you don't know how hard something is. Like, and so sometimes in engineering we would be like, why don't you just ask for that? That's actually really trivial. When you're on the other side, something you think is hard could be easy.

Sam: Right. I've even seen the extreme version of that, where asking for the next thing makes people realize that they misunderstood what you initially wanted for the previous thing. And now that you're asking for that thing, they're like, well, why didn't you say that in the first place? We can do this whole thing so much simpler.

Edith: Yeah, exactly. And so I think that's, that's again, a go back.

I think all problems are basically not tools, but just empathy and having a team.

But unless you ask for something, you don't know.

Continuous Deployment for Hadoop

Paul: So we're running out of time here. So at double speed if possible. Edith, you had a question about Hadoop?

Edith: Yeah, Sam, you said that you could do continuous deployment in Hadoop, but I wanted to hear more about that.

Sam: The interesting thing about Hadoop is you're sort of deploying to an environment in the same way that you're deploying servers. The only difference is instead of deploying something that's going to be sort of constantly being hit, you're deploying something that might run immediately, or you might even be deploying something so that it can run on a schedule.

And so if you squint at the problem, it's actually very similar. You may have a staging environment where you want to run your Hadoop jobs first. You may have a battery of tests you want to run.

Where it gets interesting is a lot of these things are sort of new to the Hadoop world. So a lot of the ways people are writing Hadoop jobs don't actually have great testing support, or it's at least a new thing. And so this whole field is sort of figuring itself out.

So to make it a bit concrete, at LinkedIn, a lot of Hadoop jobs are written in Pig, which is this sort of SQL-like dialect for writing MapReduce jobs, basically. Which sounds great because it's nice and easy to read and write. It's sort of, you don't have to think too much like a programmer, except until you imagine that people are writing hundreds of lines of basically SQL. And that this thing is powering some important product.

But because it looks like a SQL query, it's not obvious to figure out how to test this thing. And so people are starting to write unit testing frameworks that can test a Pig job. People are moving to frameworks like Scalding and Spark, which have better sort of unit testing built in.

And then you've got some tests. So the question is, well, who's going to run them? And when are we going to run them? And are we going to make sure that we don't deploy stuff that doesn't pass the tests? And so now you're in what looks very much like a continuous integration situation.

And then I guess another interesting place where especially continuous deployment starts being actually really useful. You have this very common problem with Hadoop jobs, that the standard workflow is somebody writes it, they test it on some data they have, then they build it up into a big toggle and upload it to the grid. And so they just took the toggle they had on their machine and ran it on the grid, and let's hope it did the right thing. Let's hope it had everyone's changes in.

And maybe it ran, and these things run for hours overnight, it failed. Well, how do we debug that? Has anyone got the code that ran then? Often the answer is no. And so if a team wants to get away from that situation of people just uploading whatever they've got lying around on their machine, and maybe have the concept of versioning, for example.

So you can say, oh, it was build number one three two. Oh, that's this version in the code. It's really useful to have a centralized pipeline that these builds go through that can assign version numbers to things.

And so actually moving to a sort of trunk model with continuous deployment pipeline is in some ways even more valuable for Hadoop flows. Because I feel like teams doing any kind of deployment have already got a sort of solution to that problem. They probably know what's running in production.

Edith: That's fascinating.

Paul: It sounds like exactly the problem that the rest of the world has dealt with coming up to it. People used to deploy whatever the hell was on their machine and moved into a standard CI pipeline and promoted things and so on.

So it seems like they're, it kind of feels like the work that they have to overcome this is probably, I feel like people will move on to the next thing after Hadoop rather than actually solve the problem.

Sam: Well, I'd say there's a lot of work going on in this area at LinkedIn. So LinkedIn has a big continuous deployment set of tooling. And it's actually been written in a reasonably generic way, and so they're working on extending it to work directly with Hadoop jobs and fitted into the same pipeline and the same workflow as everyone else is using.

Because there are the sort of company leverage benefits of all the teams using the same tools, even if, oh, we're writing real-time services and we're writing these offline data crunching things. It's still quite nice if you can move somewhere between those teams and they still know how the workflow works.

That's the other big thing of continuous deployment, is you never need to educate someone about how to deploy your stuff again. And the more different your deployment is from what people are used to, the more useful that is.

Paul: I wish we could keep talking about this forever, but I think that is time, unfortunately.

Edith: But it was just getting good, Paul.

Paul: That's the same problem every week.

Edith: Same problem every week. Thanks, Sam. We really enjoyed you coming by.

Sam: Yeah, I really enjoyed being on the show. Thank you for inviting me.

Paul: Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of CircleCI, and Edith Harbaugh of LaunchDarkly.

Edith: To learn more about Heavybit, visit heavybit.com. While you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.

Subscribe to Heavybit Updates

You don’t have to build on your own. We help you stay ahead with the hottest resources, latest product updates, and top job opportunities from the community. Don’t miss out—subscribe now.