Ep. #7, Effective Team Structure

Kris Gale: If we are shipping continuously and learning continuously, know that you're going to have more information in the future. The first step is being able to ship increments of value continuously.

I love losing institutional knowledge. Institutional knowledge often defends complexity. And I really, really like losing complexity. Let the code shed the complexity.

Edith Harbaugh: You never steal anything, you just popularize ideas.

Kris: This is completely stolen.

Edith: No arguments here.

Paul Biggar: Hi, I'm Paul Biggar, founder of CircleCI.

Edith: I'm Edith Harbaugh, CEO and co-founder at LaunchDarkly.

Paul: And you're listening to To Be Continuous, a podcast about continuous delivery and software development.

Edith: You can get in touch with us anytime at our Twitter handle at continuouscast.

Paul: The show is brought to you by Heavybit. To learn more, visit heavybit.com. And while you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.

Edith: In this episode, we talk with Kris Gale, co-founder and CTO of Clover Health, formerly VP Engineering at Yammer. Kris talks about effective team structure for continuous delivery.

Continuous Delivery and the Learning Organization

So Kris, what do you like best about continuous delivery?

Kris: So I like continuous delivery because I think it's sort of the first step of building a learning organization. It's sort of the fundamental capability that a team needs to be able to learn collectively about what it's doing in the world.

The first step is being able to sort of ship increments of value continuously. The second, I think, is to be able to measure the result of any incremental change you've made, so you can back out of a bad path that you're on or sort of refine what you're doing and know that quantitatively.

The next step then is to move from sort of historical corporate planning, so this is our six-month milestone, our two-year roadmap, our marketing commitments for Q3 of next year, into sort of an assumption that if we are shipping continuously and learning continuously because we're measuring what we're doing, whether it's quantitative or qualitative measurements, the next step is really being able to assume that in your planning and know that you're going to have more information in the future than you had in the past.

So your plans that you made six months ago are necessarily lacking information that plans you would make today would have. So sort of institutionalize that. And then kind of the last part, and this is all enabled by continuous delivery, the last part is building organizational fluidity.

So as an organization, whether it's your project planning structure or your accountabilities, your workflows, are fluid. So if you learn something new, you're not locked into, oh, we have a six-person, I don't know, signup flow team, and we just discovered that we have no levers for moving engagement right now in the signup flow. So we're like committed to that and we've got this implicit prioritization that's built into our org chart. So to have organizational fluidity where you're adapting the organization to what you're learning continuously.

Edith: Wow. You know, that's fascinating. Paul and I have talked a lot about, I think, more of the mechanics of continuous delivery. We put it as kind of the last step to software org, but you're really talking about how that changed this entire organization beyond software. So this would be a great time for you to introduce yourself and tell us more about you.

Kris: Yeah. So I'm Kris Gale. Let's see. Most recently I was the VP of engineering at Yammer.

I heard you talking about Yammer on a previous episode. I'd love to dig into any of that if you've got questions.

Currently, though, I'm the co-founder of a company called Clover Health. So what we're actually doing is offering health insurance for seniors with the idea that we can fill gaps in care and actually prevent hospitalizations. We're not trying to like be people's doctors, but we're trying to sort of look across their consumption of healthcare and say, oh, here's an opportunity to inflect the probability with which this population of people will be admitted to the hospital.

So for example, making sure they get treatments regularly, get follow-up visits with specialists, that particular biomarkers don't deviate outside of a specific range. And all this stuff's really interesting to me because the healthcare industry as a whole doesn't change what it does continuously.

We're sort of locked into what people learned in medical school or what people learned on their first day at the job at, say, a big health insurance company or something like that. But to be able to power workflow with software, you're able to continuously iterate and continuously refine and optimize the intervention protocols that are leading to these prevented diseases.

Edith: Yeah, I mean, I heard that you're pretty good at data loops and that this is just basically using data loops for good, I think.

Kris: What's a data loop? So the way we were talking about data loops, we're actually talking a little bit before we walked up here, is this idea that there's something you intend to do. There's data that you collect in doing it. There's your objective assessment of that data to determine whether you've like added or removed value or improved something.

And then closing the feedback loop to get back now to actually modifying what you're doing informed by that new data. So that's sort of the data feedback loop, is from doing something to measuring it and deciding to do something differently the next time.

The tighter you can close that feedback loop, the more opportunities you have to optimize in a short amount of time.

Paul: So it's kinda what you might call the shipping loop if you implicitly believe that data is a key part of shipping.

Kris: Yeah, yeah.

Paul: And I would say that I think I do.

Edith: What do you both think? No arguments here.

Paul: Well, I think it's interesting because I think a lot of people don't actually have measurements as part of their feedback loop, and especially, I think this especially happens in a small organization where you have a startup that sort of comes out of a founder's vision or something like that.

And so one of the questions that I'm very interested about is, you were doing a big thing with lots and lots of people. And now you're doing a very small thing with, I presume, a small handful of people, and how much of what you've learned, you know, still applies at that very small scale?

Kris: So yeah, a lot of the project workflow strategy that we came up with at Yammer was really about how do you maintain that agility, that velocity of value delivery, an ability to react to new information and not be locked into, well, this is just the way we do it, in the face of new information at scale.

So Yammer's whole process was really about how do you take these advantages we had when there were like four or five of us in the really early days and not have that big bureaucratic company, oh, we know we should be doing something better, but we're not, because of momentum, inertia, bureaucracy, all these things. Like, how to break that.

So what's cool about starting over with a much smaller team is, you know, we're just there again. We're back at that five-person size.

Edith: You're a lot bigger than five people.

Kris: We are now, yeah. So like nine months ago we were, like, sixty-five people now already. The team's growing really, really fast. So it's been really helpful to me to have been through the process of scaling a software engineering team and seeing what that looks like. So it's like, oh, okay, I've seen this problem before, not worried about this.

Pulling the Boss Card

Paul: So when you started, did you immediately set out, you know, organizational goals and, you know, mission and values and that sort of thing that would help the company scale out?

Kris: Yeah, no, that's a great question. So I'm obviously like a really big believer in the way we approach this problem space at Yammer. But I didn't want anything to be dogmatic. I didn't want anything to be...

Paul: You wanted it to grow organically.

Kris: Yeah, I wanted it to grow organically to a certain point. But there have definitely been times where I stepped in and I said, no, I'm buying a physical board to track our projects, because just the way we've been doing this with sticky notes, I know that it needs this other dimension of information capture. I know that we're suffering because it's not there, so I'm gonna like actually pull the boss card and do this.

There've only been like three or four times I've done that in the last year, but it's things where I've like, okay, I've seen where this goes off the rails if we don't address this right now.

Paul: And is that a result of knowing sort of directionally how the company should be doing things, or setting things up in some way that you didn't get, that you didn't end up in these sort of bad places, or was it all sort of retroactive and, you know, a couple places that you pulled the boss card?

Kris: Yeah, it was mostly like observing patterns, saying like, okay, we made this mistake three or four times, and I'm noticing that our process or our approach here is divergent from a known good approach that I'm aware of. So let's at least, I still want to give the team full empowerment to iterate and own the process themselves, but let's at least just like pull the boss card, get it back on the rails.

And then say, okay, look, see how this works. Just give everybody a chance to work in a way that I know to be okay for a few iterations, and then kind of give them the keys again and say, okay, take this where you want.

Edith: Well, so Kris, what I'm dying to know now is what are those three or four key things where you gotta play that card?

Kris: Yeah, so one is single project focus, really, really important to me for software engineering teams. So what I mean by that is if you are on a project, that is your only accountability. You are not being split between three deliverables.

The other is that everybody work in cross-functional teams. So that means not giving, for example, I might tell you, build out the back end for this feature and I'll do the front end. So in order for us to do that, we would have to agree to the abstractions ahead of time. And again, I'm pretty opposed to planning and locking in planning, because the information that we have now when we're trying to lock in those abstractions is going to necessarily be incomplete.

Edith: I love that quote, because it's so true. I think you put it more, it's like you always know more in the future. It seems so obvious.

Kris: Yeah, I was actually, so we had Parker from Pivotal Labs over at the office yesterday. And he said that there was a saying they had at Pivotal that I really, really liked.

He said somebody there would always say, and I wish I could remember who it was, would always say, "You will always make better decisions with more information, and you will always have more information in the future."

I was like, that's so much more concise than any way I've ever said that, but I think it exactly captures it.

Edith: But on the other hand, my old boss Gregg Brockway, the founder of TripIt, he always said, "A good enough plan today is far superior to a perfect plan tomorrow."

Kris: Absolutely.

Edith: He's just like, you just gotta start moving.

Kris: Yeah. But I think the key is not locking yourself into that good enough plan. To allow yourself the mutability of changing that plan over time. So you're not constrained by that.

Edith: Yeah, but that was TripIt. I mean, I think he'd seen so many because he came out of the travel industry where people just got paralyzed. Like completely paralyzed. Oh, we need to plan for five years and what business travel will be like then. It's like, well, nobody knows.

Paul: Totally. I think the kind of key point of that is that in order to get that information you have to get moving today with whatever information you have available. Otherwise you're just not gonna have that future information. You're gonna be in the same place.

Edith: Well, look, sometimes, though, it's painful, but sometimes you move backwards then. Like you don't always move forward. Sometimes you start moving and then, looking back, you're like, actually we took like three steps back before we started moving forward.

Kris: It can definitely happen. Well, so to complete that thought, when we assign teams, we would always assign a cross-functional team with a business objective. So it would never be I give you a back-end task and I take the front-end task. It'd be collectively, let's accomplish this business goal.

So hopefully we're iterating day by day on the abstractions. So we are collectively accountable to one another and to this business goal we're trying to deliver, not to some specific set of tasks that we broke off when we first decided to assign out the work. So that's one of the other ones, is always cross-functional teams, always single project focus within those.

Rotating Cross-Functional Teams

Paul: So there's an interesting thing that you're talking about there, that it's not just cross-functional teams, but it's also that those cross-functional teams get put together for a specific project. Is that right?

Kris: That's absolutely right.

Paul: So are those teams that will have worked together in the past? Is there explicitly trying to make sure that the people know how to work together?

Kris: Yeah. So this is one of my other like core principles that I am pretty, pretty set on, which is rotation, is this idea that the cross-functional teams are rotational and ephemeral. So that would mean that maybe the three of us come together to ship a feature. We're all within the same organization, but when we're done shipping that feature, we're gonna break up.

You might be the backend person that goes on to another iteration of this feature area. Or maybe, you know, the two of us work on some other feature. Maybe it's three completely different people.

But constantly rotating people around, I think, is really important in an organization.

I have totally heard the arguments about, you know, you can build high trust teams if they're stable and they stay together. But I'm trying to optimize for trust across the entire organization.

So across Yammer we had, you know, 220 people in engineering. They would all rotate through these two to ten person cross-functional teams together, which meant that you basically within three to six months would work with a pretty significant portion of those two hundred and twenty people and would start to build trust across the larger organization.

Whereas had we just torn off like four people and said, you're a team, they would, you know, and I've seen this throughout my career in engineering where it's like, we do everything right and that team has no idea what they're doing.

Edith: It's their APIs are bad.

Kris: Yeah. They use this weird language or this approach or this style guide or whatever. So it sort of forced that rotation so that we're all sharing best practices among one another and building maybe lower trust, maybe not getting to that very, very, very highest level of trust among a group of four people, but across the entire organization getting to like forty or sixty percent of what's possible versus zero percent if you don't ever work with those people.

Paul: So I'm interested in structurally how that ends up, because one of the things you got is when you split people up by team is you end up with a sort of a microservices architecture. And, you know, this team works on this repo or this project or whatever, works on this service with this API. And when you have those teams that are forming and splitting up and they're ephemeral and it's only a small period of time, architecturally what does that do to your application or your architecture? Yeah, across your entire product.

Kris: Yeah, no, totally. And it was actually very intentional that we ended up with that kind of architecture for working this way.

So for two years we had a big monolithic Ruby on Rails code base. And for two years we said, this is a big problem, we should break this up. And everyone agreed, and everybody was on board with breaking it up. And two years later we had never done it, because we continued to work in such a way that supported that sort of architecture.

So the key for us was to say, well, one of the, so something to dig into here a little bit is these cross-functional teams are made up of representatives of what we call functional teams. So front-end engineering might be one of the things.

Paul: Right, right, right.

Kris: We had one called Core Services, which actually owned the service-oriented architecture that you're describing. Or those little microservices that collectively made up our service-oriented architecture.

Paul: And this is a sort of a matrix style, where you're a member of a team, but you're also a member of the back-end team or you're a member of the front end or the core services or whatever.

Kris: Yeah. So I might be a member of Core Services. My manager is one of the Core Services managers. But all the work I do is going to be within the context of a cross-functional team with a front-end engineer, maybe a client's engineer, whoever's necessary to get end-to-end with a value proposition.

And then I'm always going to be a Core Services member when I go into these cross-functional teams. But it's the fact that all the members belong to functional teams that makes that team cross-functional.

Paul: Gotcha. So Spotify has a fascinating article or case study about this.

Kris: Yeah, I've read the paper on this.

Paul: Yeah, so I think there's like four different dimensions of their cross-functional team. So you might be on the front-end team and then you're on some sort of guild that's interested in a particular kind of area, maybe design or some particular part of that, and then you're actually on the team, and there's another layer that escapes me right now.

Kris: I don't remember all of the details, but it's a similar concept.

Edith: It reminds me a lot of being a consultant.

Kris: Yes. So I've heard this from a bunch of people. Actually, the conversation I was having with Parker from Pivotal was that this is actually very similar to how they did what they did, which was consulting.

Edith: Well, because consulting, I was a consultant, and it's very much based on a customer. So you're, you know, a body shop, to be blunt. Customers gonna pay two hundred dollars an hour for people that meet their needs.

So it's kind of like how do we assemble the A-team of, you know, a front end, a back end, a product manager, a designer to suit their needs. And it's very mercenary. But yet it works.

Kris: Mercenary in, and do you mean that in like a pejorative way?

Edith: No, I mean it in a very positive way. I think what you were talking about, how companies start to have silos where you said, like, we're the six-person signup team. And we know signups, so we're gonna protect the signup turf whether or not anybody really needs signups anymore. Like versus what is the overall need of the customer.

Kris: Right. One of the things we were optimizing for at the really high level was to ensure that all of our engineering cycles were going to efforts that the whole company would agree is our top and most important effort.

Edith: Not, I like working on signups, but...

Kris: Yeah, so we're constantly fluidly reprioritizing. So going back to those sort of four phases, which by the way I stole from this guy Nate Fink, who worked with us at Yammer. He actually took over my job there running the engineers.

Edith: You never steal anything, you just popularize ideas.

Kris: This is completely stolen. His framework of these four ideas. I think it's just like a great way to put it.

The fourth is this organizational fluidity that allows you to reallocate staff to the things that are most important based on what you've most recently learned. So it was really important to us that if our CEO walked into the room and said, what are the engineers doing?

We could point at a single document, or in our case it was like a physical project board, and be able to point to the like ten projects that were staffed, all the names of people, and be able to say we're only doing these ten things because we have single project focus. We will do these in a small iteration. And then after that we can reprioritize.

A week from now, some people will be rolling off one of these ten and we can say, okay, what's the next most important thing for us to put resources towards?

Knowing When a Project Is Done

Paul: It sounds like you always completed the projects once they started. Is that right? It sounds like there wasn't a great deal of churn or just like, you know, this project is now important, so we're gonna kill this one and move this one on. It sounds like people actually finished.

Kris: So I wouldn't say we always finished it. If during the course of doing the project we discovered, oh, this is like, for example, we would not say this is a five-week project, so do five weeks of coding and then ship it at the end. When it was working the best, and I think the best tech leads that ran these projects were able to do incremental validation throughout the five weeks.

So it might be, you know, three days in, we're dark launching a feature. And we know that Edith's very happy that I said that.

Edith: Thanks, Kris.

Kris: So a couple days in we made dark launch an early, early iteration of it. So the product manager can for the first time like actually click it live with live data, and maybe there's no CSS on it, maybe it's like, you know, just like in a really sloppy form, but we'll be able to see like, okay, is this something we want to continue to invest in?

And if no, or if we do an early release of the feature to a small slice of our users and the metrics are really bad, we may just kill it without getting to the end of what was initially spec'd as the feature.

Edith: Yeah, I mean, a real issue we had at TripIt was, so we had the same sort of structure where we had, we called it the most important tasks list, which was like the projects, and it was always about thirty projects, of which the top seven actually had people on them. And then what would happen is if you were on the top seven, you didn't want to let your people go. So these projects, like, to your thing, like how do you know what a project is done? Like it's like, well, I have my engineers, I'm not giving them up.

Paul: Yeah. How were projects specified? It sounds like they were just very long-term projects that didn't die.

Edith: No, well, they were meant to be short, so there would be something like improved sign-up flow. Like, let's pick on that one. So that would bubble up to like the number three project, and then whoever was the PM would just be like, oh, I have people, I'm gonna just keep going. So how would you make sure that these teams were short-lived?

Kris: Yeah, so one of the requirements of the product spec that would be the initial seed of a cross-functional project was that that product spec have a definitive and finite done, whatever that is. Maybe it's like we come in with a hypothesis that if we make this tweak to, I don't know, message replies will drive more engagement. So done is like when you've made that tweak and either confirmed or disconfirmed that hypothesis, not keep flailing around until you find something that works. So that was really key for us, is to agree ahead of time to what done is.

Polish and the Backlog

Paul: One of the things I like about project-based teams is the idea that the project-based team can keep an eye on the priorities within that thing. You know, whether it's, you know, we're gonna fix this very, very small bug, you know, a CSS alignment sort of thing, or, you know, focus our efforts on a monumental rewrite, or and it sounds like you got the monumental rewrite taken care of, or it's obvious how that works, but how do you take care of the, you know, we shipped it and it's kind of like 95% perfect. So we'll call this done, but, you know, there's all these like small tasks.

Edith: That's what I meant, what happened at TripIt, and that's why people clung.

Paul: Right. So how is it handled at Yammer?

Kris: Yeah, you know, it's interesting. And I think actually in complete honesty, I think we were not great about polish. I think there were some things that some teams would just, you know, polish and polish and polish, other companies, in a way that produced a more polished result than what we did.

I think there was a potentially logically coherent and rational argument that if we couldn't measure the impact of that polish, like if there was no way to quantify that it had value, maybe we were actually doing the right things. Maybe the right thing to do was instead of doing the remaining, you know, four or five percent of this project that's gonna make it feel really, really slick, maybe putting those engineering resources into starting to like validate or invalidate some other hypothesis in some other area is gonna deliver more business value faster.

Now I would argue, like, I'm not saying that I know that for sure.

Paul: Right.

Kris: That's sort of the position that we took. Though on some features where polish was important, the definition of done could include that polish. Like it was fair for a PM to say, no, no, no, this isn't done until this is right in these ways and these pixels are lined up in this way and then this all works and this edge case has to be dealt with. That's part of the spec.

Paul: I guess the thing that I'm interested in is where the final results of it, you know, come out of, you know, let's put this out and wait for three weeks worth of user data or feedback or whatever. What happens to the team, and where does the polish get actually done? Is it a new project later down the line or how does that get handled?

Kris: Yeah. So my favorite way of addressing this was really just to let the PM team not keep a backlog. I'm actually pretty opposed to like formalized backlogs. We were talking about this a little bit earlier. But to have a general idea of, the next time we revisit this feature area, I'd really like to clean up all these things.

Edith: I'd actually really love to hear why you're against a backlog.

Kris: So backlogs are design inventory. We were talking about like the definition of lean. I think like lean, at least in my mental model of it, is about minimizing inventory.

Edith: Definitely.

Kris: I think if you have a backlog, you have committed to a bunch of design inventory. You're saying that at some future point we're going to do all this stuff.

One thing that's nice about not maintaining a backlog is you can actually choose never to revisit something. So for example, in some of these polished questions, you might say, well, this interface, it would be really nice if we fixed this thing or whatever. But then you do an iteration that just blows away that whole feature area. So you never actually have to do that.

Edith: That's interesting. I sometimes like maintaining a backlog just to acknowledge that we're explicitly not doing it, is kind of an anti-requirement. So that in every meeting, because somebody's always gonna pipe up, well, why don't we do this? It's like, that's in the backlog.

Paul: Right. In organizations that use bug trackers, or where their entire thing goes into bug trackers, it's very useful to have the bug that is marked as won't fix, so that you can always just refer to people, you know, we decided not to fix this for some reason.

Edith: Because otherwise you could spend like half an hour in a mini rat holing on, why aren't we doing this?

Paul: Right, right, right. Or you can start coming up with the same ideas again and again because you've blown away the institutional knowledge that comes up from having that recorded somewhere.

Edith: We've acknowledged that this issue exists and we've decided that we're not going to do anything.

Paul: You're looking sort of cynical on this.

Edith: Yeah. Kris is like...

Kris: I almost think like deciding not to do something is potentially as harmful, in that sort of like we've made a long-term plan way, as designing to do something.

Edith: This is why you're a great guest, Kris.

Kris: This is that argument you want to know.

Edith: Kris has a new perspective.

Kris: I like the rebubbling of things for reconsideration later when we know more. It's like, yeah, we decided not to do that, but we decided to do that with incomplete information in the past.

Edith: That's fair.

Kris: In the future, we may have learned that some other feature area, this was actually really significant, or something that looked like this feature was meaningful for our core metrics. So now the next time this bubbles up, we've got a fresh perspective on it. And we can reconsider, oh, okay, are we gonna do that or not?

Losing Institutional Knowledge

Paul: So one of the things I've seen that comes up with this is a sort of a constant back and forth between two non-optimal options. So say, for example, you know, we can combine these things or we can separate these things. And when they're separated it really looks like they should be combined. And when they're combined it really looks like they should be separated.

And if you lose the institutional knowledge of, you know, this is why we combined them the last time, then you often end up with, you know, a whole load of like just spinning your wheels, getting nowhere on that. How do you avoid that?

Kris: I think, yeah.

Paul: I guess you're not saying we're just going to throw away all the institutional knowledge.

Kris: So this is another contrarian position that I like to take, which is I love losing institutional knowledge.

Edith: Kris, you're the best guest ever. You're also our first guest.

Kris: So the most ridiculous one. So I have some really good examples of, and I'm actually probably less informed about the, so the question you're asking is really about product decisions. Like product design decisions, UI decisions?

Paul: I think it applies both to technical decisions and to product decisions.

Kris: Okay, so let's talk about the technical decisions. This is where I feel really passionate.

So there were a bunch of cases. So I built most of the original messaging infrastructure at Yammer, and I have strong opinions as a software engineer. And there were parts of the code that I'd be like, hey, if you're ever in there, ask me about it. There's a specific reason it's like that. Like, I can justify all this.

Or I would document it, or there'd be really verbose code comments that said, don't touch this because it's like this, because of this. And actually what I found was over time, that's super disempowering for future people. They don't own the architecture decisions. Some past person who wrote something down is dictating how they do their job now.

Paul: And they're not in the company anymore, and you can't actually talk to them.

Kris: Or maybe now they're in some management role and they're not available in that moment where you need to change the code because they're off doing a one-on-one with somebody about something really important or talking to HR or doing something, you've disempowered the engineers to do stuff.

So a lot of times what would happen is institutional knowledge often defends complexity. And I really, really like losing complexity.

I would rather people go to the code and say, I don't understand what's going on here. All the tests pass when I strip out this thing. So I don't know, maybe I broke something. There was no test for it. Maybe it's a weird edge case that just never comes up.

And let that bubble up again. Let the code shed the complexity. Be easier to maintain, easier to understand, easier to like intuit about. And fix that bug if it comes up in the future.

Or what really happens ninety-nine percent of the time is, by not having that institutional knowledge, the codes are simpler. And that weird edge case that came up that one time that you wrote an elegant solution for is actually just slowing everyone down, and you're better off for not having it.

Edith: So this is basically the buffalo herd of code. Like, you know, you know the buffalo herd, so they're an American animal, the buffalo, and the theory is you kill the weaker and the herd moves faster.

Paul: Oh, interesting. Okay. So the opposing view, and I think something that most people of my generation will have been educated with, is this Joel Spolsky wrote this thing about, you know, the Netscape 4 rewrite or something, where they just tossed it all away and then they, you know, had to rewrite the FTP module and they had to integrate with the fifty kinds of FTP servers, and no one had that technical knowledge, institutional knowledge, whatever it is you'll call it. So there's, you know, I'm feeling some sort of like level of anxiety based on what you're saying there, and, you know, it seems intuitively wrong.

Kris: I think if you are forced to stop and rewrite an entire code base, you've done something wrong upstream. Like, yeah, we can try to optimize for that case for like

Paul: Sure, sure. But I mean, what you're saying is we can take this module or this small thing and we can, you know, just discard the information that was involved in building it.

Kris: What I'm talking about doing is getting to a situation where everyone is empowered to make fresh decisions to continuously refactor and continuously modify the code without being disempowered by previous decisions, and potentially, yes, trading the fact that we may temporarily lose some functionality. And maybe the way to reintroduce it actually ends up being cleaner because now we're free to do that refactor or that cleanup or that simplification without, you know, that institutional knowledge is just so, so often used to like, don't touch that because, don't change that because, or, we can't move on to this new framework because.

And I never want those things. I would rather make the little mistakes and come back.

Measurement and Taste

Edith: Well, I think a key part of this is having a really good measurement system to, you know, to not just, I think Paul's fear comes from like not just trample across your code paste and then toss everything back out, but to have a lot of, at Yammer, you had a lot of measurement, you had a lot of analytics. So you knew that if you changed something and suddenly your conversion went down twenty percent, You even knew it.

Kris: And tests are obviously important for this. Like a well-tested code base, like the tests are to a certain extent like that institutional knowledge. The code should do this in this case or the product needs to do this in this situation.

Paul: So there's an interesting trade-off there between, you know, you're talking about tests as one way to fix this and Edith's talking about measurements. It occurs to me that tests are a sort of a binary version of a measurement.

And the measurement is an actual business value, whereas a test is a thing that someone wrote at some time in the past, but that is black and white and irrevocable, and you can't break the test.

Kris: No, I think you're right. I mean, the measurements are essentially like a form of test that just aren't so binary. Like did we move engagement? Did we break our conversion?

Paul: And it seems like the tests are much lower value than the, by definition, because the business value is measured in these KPIs or whatever the things are that you're measuring.

Edith: Yeah, I'd love to hear you talk a little bit more about Yammer's measurement framework.

Kris: Yeah, Yammer really cared about, engagement was our most important KPI. We were sort of banking on the fact that, as a kind of bottoms-up enterprise tool, that the more people that used it and the more often they used it and the stickier they were with the product, the more likely their companies were to buy the product in the future. So that was really it. It's pretty simple. It was like, does a user come back and continue to use the product if they've been exposed to this or if they're in this group or if they have this functionality?

We obviously looked at other things like signup rates, and we could kind of break down so that when we saw that we had moved the engagement numbers, we might say, well, is that because fewer people are making it through the signup funnel? Or does it mean that people who make it through the sign-up funnel and do this one thing then don't also do this, or, you know, whatever. So the really high-level thing, the thing we cared about most was engagement.

Edith: So that gave you kind of the freedom to say it's fine to refactor code as long as we're not touching this actual thing we care about.

Kris: Yeah. I mean, ultimately, like every project, it's a question of like there are different levels of where you're looking at different things. So if you're refactoring code, you're really looking at tests. I don't think that engagement numbers are a useful tool for, because the latency in getting that data back is too slow. You can't like, I'm gonna try to refactor this and I'll push it to production and I'll wait a week for, you know, an A/B test result to come back.

I just think that's why tests are so helpful, is because you can run them instantly. You can know just like, did I break it? Whereas the engagement and the KPIs and those things were really around, did this increment of product change make the product better or worse? And better or worse was measured in terms of what we believe to be the most consistent leading indicator of value, which was eventually sales.

Paul: Was there a sense of taste that was involved in that decision? So what I'm trying to get at, you were talking about polish a little bit earlier. And when you look at, say, a company today like Slack, one of the things that really, really works for Slack is that, this polish. I presume that there's a human in there who's actually making that decision. And when you're talking about the measurement and the engagement and sort of the lack of polish, I'm interested in, you know, how was the taste measurement done?

Kris: Taste measurement, yeah. So I don't think quantitative product development is a substitute for product vision or product, yeah, vision I think is probably the best term for that. I think polish is very much probably a part of the Slack vision. I don't know the product team there, but certainly as a user it seems like that's the case.

Yammer was very much about, so David Sacks was our original product person, the original CEO. And there were certain things that I would consider polish, not polish in the pixel perfect sense, but in, this feature really should be this simple. Or this should be as easy as this. That was very much part of his mentality when he was running product.

We had other product leaders throughout the years who were optimizing for different things.

But I don't think data ever replaced anyone's vision of what Yammer should be.

I think different people had different visions and probably differently valuable visions of where their product was going or what it should be. What the data really allows you to do is, on your way to that vision that you have, being able to check your own assumptions and your own hypotheses.

So it's not like, should it be polished or not, is not what you're gonna get out of the data. It's gonna be like, okay, so I thought that working on this part of the thing would be more impactful, but I'm not moving the metrics at all.

So somebody else had this idea that maybe actually we should be working on invites. That'll be a nice viral hook. Let's see if that actually moves the numbers. And you can say, oh, okay, yeah, we're moving the numbers a lot.

And sometimes like a negative test result was great signal for us. Like we tanked engagement by touching this. So we know this is a lever.

We know that, okay, let's continue to invest in this. All right, let's try different versions of touching this, because it's in somebody's product vision that this feature area needs to be improved. But like our first instinct actually made it worse.

Edith: What was your first instinct? How did you tank?

Kris: So the inbox feature in Yammer was one where, I believe it was David, David Sacks, was sure that like the product needs an inbox. And we built an inbox and we tanked the engagement numbers. The first version of it was negative for engagement.

So if you were really letting data drive and just had no vision and had no sort of belief about what's right about the product, then you would just say, okay, well, inbox is bad. We'll never do an inbox. But we were like, no, no, no, we know this needs an inbox. This version of it isn't what we needed, so let's try another iteration.

And we tried a few iterations in inbox before we got something that was positive for engagement. So it wasn't about like, what should we be doing? It's about what is the right version of what we...

Edith: Yeah, that gets back to my original question about, you know, how do you know when a project is done?

Kris: Right. So those were all separate projects. So that was interesting, is we didn't just keep the same inbox team together for multiple flailing iterations of inbox. We said, here's a hypothesis we're testing. Testing that first inbox hypothesis was a project.

So when that was done and when that was shipped, that team, if my memory is correct, it was years ago, that team broke up, and then that information from that failed test...

Edith: But the test was a success.

Kris: Well, failed in the sense that the engagement was negative and we didn't roll it out. But that information then went back into product planning to say, oh, okay, we still believe in inbox. So now with this test result, now here's what we believe to be a better iteration. So now we'll prioritize that against everything else we could be building.

So you can imagine, like, if that failed six times, that would probably lower itself in priority, you know, over time. But I think we got it right on the second or third try.

How Yammer Staffed Projects

Paul: So I have a question about the structure of that. So Yammer, I don't think we've mentioned this, but famously Yammer had two to ten person projects for two to ten weeks, right?

Kris: Yeah, that was our scoping principle.

Paul: And so you're talking about these teams that are formed and then broken up, and what I'm interested in is what was the structure that, you know, said this is gonna be the team, this is gonna be the thing, this is gonna be the product vision. Like what was the extra on top of all this?

Kris: Yeah. So step one was that the product team would obviously have some vision about where Yammer's going. So they'd say, here's an increment of product change we can bite off and build.

Paul: So the product team wasn't part of one of these projects. They're like sitting to the side scoping and organizing and that sort of thing.

Kris: Well, they would start there. The product manager who built a spec that became a project would actually join the team.

Paul: Gotcha.

Kris: So the product team collectively has the vision about where the product's going. Then a particular PM says, well, here's a thing I can design and have a team build. So that PM would do a spec.

And then we go through a process we called spec review, where the product team would actually, well, they had a process called product review where they would present it to the rest of the product team. Does this make sense? Does this fit our vision? Do we all believe this is worth it?

Paul: How many people is the product team?

Kris: So I mean, it grew from one person to, I think, at its peak it was maybe ten people.

Paul: Okay.

Kris: That sounds right.

Paul: So not a massive team?

Kris: Not massive, no.

Paul: Okay. And was there more to the organization then that, you know, designed the projects and measured them and that sort of thing?

Kris: Yeah. So let me walk through the sort of product lifecycle. So it starts with, here's a thing we want to build, here's an incremental product change. Here's the hypothesis we're testing by doing this product change.

That would then be approved by the rest of the product team. Yes, that's something we think we should go forward with. But then it would be floated to the engineering team.

We actually did this on Yammer, the tool itself. We would post it to a group called Product Review or Spec Reviewer. I forget the name of it.

Edith: Yammer on Yammer.

Kris: Yammer on Yammer, yeah. So we would post it in that group and then the engineers would chime in and basically say, okay, given what you want to build, we think it does fit within our scoping constraint. We think we could do this between two to ten weeks.

And it's gonna require back-end work, front-end work, and someone from the client's team. So at least staff it with one person from each of those functional teams.

Sometimes we'd say there's just so much back-end work to do here. We need like two or three back-end people. And that's how we determined what the staffing requirements for the spec were.

So then essentially the spec would just be in the hopper, and it wasn't a prioritized backlog. It was sort of the set of like a few-ish things we will do next when resources free up.

So then every week we would walk over to what we call the big board, which is our product planning board, where all these projects are written down along with these magnetic tags that represent each of the people who staffed on the project. And every week things were finishing up from the prior week.

So we'd say, okay, these four people are freeing up, these eight people over here freed up as well. So now we've got three specs that are approved. We know the staffing requirements. They're ready to kick off as soon as people are free.

So let's just start slotting people into those slots, and then the project would kick off.

And so another member of the team would be analytics, would join to do the quantitative piece of this. So product, engineering and analytics.

Paul: Okay.

Edith: What would you do if there wasn't people exactly free, like to your example, if something needed a lot of back ends and they're not free right then?

Kris: Yeah, so this is the constant question people have, is like, there's not always exactly, you know, 1.0 people units of work in back end and front end and whatever. So they're trying to optimize for efficiency.

I think anytime you're optimizing for efficiency in a research and development or product engineering org, you're probably doing something wrong or making some bad trade-off.

So if you're like, well, you know, there's only three quarters as much front-end work here as there is back-end work. So we'll let that person go early.

What inevitably happens is you're getting towards the, I'm buttoning it up, I'm getting ready to ship it. Oh, we just need to make this one front-end change. And that person's off with some other commitment.

So now you're blocked. So the throughput of the entire org, it seems more efficient because your capacity utilization is closer to 100%. Well, what really happens is you spend more time blocked. So you go slower.

Paul: We're not allowed to take that guy off the project that he's gone to because those are the rules.

Kris: Right. So like I'm not a front-end engineer, and if I'm on a project that's got a little more front-end engineering than back-end engineering and I'm paired with somebody that I know to be a good front-end engineer, I can contribute value. I mean, I know how to write code. I may not know the idioms of a particular JavaScript framework.

But if you do and we're working together, you can walk me through that and say, just stub out a subroutine that does this kind of thing or munges the data in this way, and I can be useful. Which is one of the values of cross-functionality and not just like siloing tasks into people's workflow.

Rebuilding Healthcare Software

Edith: Cool. Well, so I have a big picture question, which is, you know, Yammer, huge success. Your most recent company, Clover, just got, I called it a whopper amount of funding, a hundred million. How does this kind of org scale up with so much money?

Kris: Yeah, Clover's interesting because we're a health insurance business, so it's very capital intensive. I think, like, different from a SaaS business, we're probably gonna need a little bit more money right now at this stage.

Edith: Well, you have it.

Kris: Yeah, thankfully. Really what's interesting about Clover is it doesn't stop with software. Because we're not selling software to people, we have nurse practitioners, we have social workers, we're doing home visits, we have medical assistants that are doing clinical data abstraction out of medical charts.

There's a lot to do in a lot of different areas, but I think this model like really helps us to do that, because we can say, you know, we don't have standing teams that are just working on the data processing for clinical abstraction. We've got engineers that maybe, yeah, they could do a couple iterations on that, get the team of MAs spun up and pulling out important clinical signals from charts, but then they go off to work on home visit logging so that the nurse practitioners, when they're in somebody's home, have a good tool on their iPhone for tracking all of the clinical observations they make in the home, or working on the data pipeline or data visualization or whatever.

And we're sort of fluidly moving around because, you know, as big as the team is right now, and as many resources as we have relative to a startup of our age, we're essentially trying to rebuild the whole software stack that powers healthcare in the US internally just for our own purposes. So we're constantly like reprioritizing and shifting, so that organizational fluidity is super, super important to us right now.

Paul: This seems like one of the key areas where you can dump the institutional knowledge of, you know, the healthcare industry as a whole, because it's so backwards, and rethinking it from scratch is probably gonna be a major advantage.

Kris: You know, it's funny, one of our engineers, Jasmine, actually just recently said this in a thread. We were having a conversation about something we learned on a thing. And she was tasked with organizing a bunch of, for us it's provider data, so the doctors that are in our insurance network.

And she was like, the big lesson here is don't trust institutional knowledge in healthcare. It's misleading, it's based on old assumptions. A lot of people have never even dug in to understand how this data flows.

So yeah, no, you're absolutely right. Being really good about shedding industry institutional knowledge has been so, so important for us.

Edith: It sounds like a really interesting and important endeavor. What's been the most surprising part to you?

Kris: Surprising part of working in the healthcare space? There's definitely been a lot.

One thing is that, so in enterprise software, if you were building something that had like self-service where people could just pay with a credit card, in the year 2015, you would probably never build your own credit card processing.

Edith: You'd use Stripe.

Kris: You'd use Stripe or something like that.

In healthcare, we thought we would walk in with a bunch of that. So taking lab test results in this industry standard HL7 format, oh, we'll just use the off-the-shelf tools or the open source tools or the dominant IT vendor to do this. Almost every single time the answer's been like, no, no, don't, because all of everything is bad.

And it's hard maybe for like enterprise software people to believe me that the right answer is rebuild it yourself, because we've all been conditioned like out of that not invented here sort of mentality. But when you have like upstream vendors that have no idea how to reliably give you a data stream, or are dropping data, or are misclassifying certain, you know, clinical things, it's really, really important to get it right and to get it reliable.

So that's why as an insurance company we've had to build such a big software team, because data reliability is so important, and data feedback loops. I mean, I have a team of engineers that literally had to learn how COBOL encodes integers because some of the upstream data formats, and probably not generated by a COBOL system still, Probably, I'm hoping, but maybe. So it's just amazing how bad the state of healthcare tech is. But I think that's like such an opportunity for entrepreneurs who want to get into the space.

Paul: Well, it's funny, I see this sort of thing where someone has made like, you know, a big generalization. I'll give a concrete example. When we were building Circle, Circle is built in Clojure, and so there's very few libraries available. There's libraries for HTTP, but there's not libraries for, you know, connect to GitHub using OAuth or something like that.

And so we started building the front-end part of it in Rails, where there was all these libraries for all these sort of things, and we found that the time that we spent integrating those libraries was massive. Whereas the thing that we were actually trying to do, which in this case was connect to GitHub via OAuth, was, you know, three API calls.

Kris: Right.

Paul: And so the time that it actually took to integrate these off-the-shelf, the vendor component, whatever the thing is, they're built for a different problem than you're trying to solve. They're built for like a very abstract problem, and you're only trying to do a very specific subset of it. It's often trivial to write that subset rather than try to integrate the thing that's there already.

Edith: Well, Kris, we were really thrilled to have you as our first and best guest. Do you have any final thoughts?

Kris: No, no. I think we covered a lot of ground. I really appreciate you having me here. This was a lot of fun.

Paul: Yeah, this has been awesome. Thank you.

Edith: Yeah, thanks, Kris.

Kris: Absolutely.

Paul: Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of CircleCI, and Edith Harbaugh of LaunchDarkly.

Edith: To learn more about Heavybit, visit heavybit.com. While you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.