Sean Byrnes
Business Analytics: Are We There Yet?

Sean Byrnes is the CEO of Outlier.ai and founder of Flurry – a leading analytics and advertising service for mobile applications acquired by Yahoo in 2014. In the past he worked in the innovation group at Verizon Communications developing new business lines based on emerging technologies.

Collapse
00:00:00
00:00:00

Introduction

Thank you. I'm excited to be here tonight. I'm going to talk about effective analytics for running your business, mostly because numbers are usually very scary.

I think in the early days of founding companies, your goal is to try and find ways to get your product built and get customers. And thinking about numbers is something that seems in the abstract later on, but it becomes very important very quickly. I'm going to talk today about how numbers can actually help make you better at building your business instead of being this abstract concept that exists.

First off, who the heck am I? I founded Flurry years ago, back when mobile was a bad idea. And then mobile became a good idea. And then mobile became the best idea anybody had ever had.

I was very lucky to be along for that journey. It is now part of Yahoo, which is fantastic. I had decided to start a new company in January called Outlier, so I am back being a founder again of a two-person team. I tell you these things so you know that I live and breathe what I'm telling you. I'm not going to sell you anything. I have nothing to sell you. I'm just here to tell you about how I use numbers and how numbers can help you.

I also blog about startup companies at seanonstartups.co. If you'd like to hear more about my random ramblings about startup companies, you're welcome to check it out.

Business Analytics to Reach Your Vision

First off, before we even get forward on talking about numbers and analytics, let's agree on a definition. What are analytics? I love this quote, which is, "Every line is the perfect length if you don't measure it," because that sums up exactly what's important.

When you're starting a company, you have a gut intuition about how your business is going to run. You think, "My customers are really happy. I need to find more of them," or, "I think this channel is going to help me acquire a lot more customers." And in reality, those assumptions and those beliefs that we have about how our business is running may or may not be the same as reality.

At the end of the day, analytics, the goal of it, if we do it properly, is to help know that we actually do, to know what's going on, that we're not just making guesses, we're not making assumptions.

That's what we're going to talk about today, how numbers tell us what exactly what's going on. Now, what can analytics do, and what can they not do? This is an important part of our definition. What they can do is they can help you measure how you're doing. They can help you inform your decisions. They can help you make sure you identify opportunities and challenges that you didn't know existed.

What can they not do? They cannot fix your problems. They cannot motivate your people. And they cannot run your company despite the fact that people keep telling you that they can. Lots of people will try to sell you lots of services that claim to do these things. I assert that, in fact, that is impossible.

Our goal is to talk about analytics, how they make you better at running your own business and not run your business for you. To that end, I will talk about four things. I will aspire to being as entertaining as I possibly can be, because I know this can be a very boring topic.

We're going to talk about Key Performance Indicators, which is usually the acronym KPI. We're talking about measurement and tracking. We're talking about selecting the tools to use to measure whatever you've decided to measure. And then finally, measuring success, which is, in fact, why we do this.

We want to be successful, at least that's why I do it. I don't know if maybe you would disagree. So let's start off with section one, KPIs, also known as "Where are we going?"

What are KPIs?

What are your goals? That sounds like a very simple question. But, as a founder of a company, I imagine your goal is not to start a company. Because, congratulations, if that was your goal, you've already succeeded in achieving your goal.

So, in fact, what exactly is your goal? What are you trying to get to? Is it more customers? Is it higher engagement? Is it some sort of milestone to prove that you're not a crazy person just building stuff in your free time for no one? What is the goal?

I will keep coming back to this because this is the most important question that exists for your business.

Everything else that people will tell you about, from KPIs to OKRs, to anything else that comes up, what your goals are are the most important thing.

You're the only person that can decide that for your company, and you need to figure out what that is. Because that becomes the lighthouse, in the vast, foggy ocean of startup companies, that tells you where you should be going.

If that's the case, we have a vision, which is why you started your company. We translate that into goals, which is where you're going. And those become KPIs. It's very important that it happens in this direction. It's happens. It's important.

Everything has to derive from your vision. Your vision sets goals. Your goals are what we measure. It's really that simple. That is what a Key Performance Indicator is, is a way to measure how we're doing towards the goals you set to reach the vision to build your company into a big success, whatever you've decided success means.

Now, it's very important that these are not progress indicators. There is no such thing as a key progress indicator in startup companies. These are Key Performance Indicators. What is the difference? Key progress indicators tell you how great you are at working, and nobody cares about that because we all work really hard, right?

Key Performance Indicators tell us, "Are we actually getting closer? Are we at our goals, or are we not at our goals?" Those are not the same thing.

For example, what would be the difference? A progress indicator may be our burndown rate on every iteration that we're doing. A performance indicator may be how we actually sold $10,000 this month, or not. It's very important to not confuse the two because it's very easy to make yourself feel good by all those internal progress metrics that we use every day.

I will give you an example about how you would do this properly. Let's say we had a vision to start a company. We're all going to start a company, and it's going to be the first company to solve thirst problems by delivering lattes by drone.

We've decided that everyone needs a latte. They need it wherever they are, and the only way to do it is to deliver it by drone. So, we're going to deliver lattes by drone. That's our vision. Now, frankly, this means we're not a very good entrepreneur. But anyway, that's our vision.

From that, what goals might you have? Well, you might want to reach maybe 100 deliveries per day in San Francisco, reach 20,000 paying customers and launch in three more cities. If we do that, we will know that our drone-delivery-latte service is on to something.

From that we need three KPIs. We need to know how many deliveries we're making, how many paying customers we have and how many cities have active customers. If we knew those three things, we would know how we're doing towards our goals, which would tell us how we're doing towards our vision.

That's really, at the end of the day, as simple as it can be in deciding what you need to track for your business. You work your way down from your vision, which was our lattes, the goals we think we need to reach to prove that we're on to something and the KPIs are derived from that.

How KPIs and OKRs Relate

You may be asking, "How do KPIs and OKRs relate?" Because just last week, Kris was up here, and Kris was talking about OKRs. And you're like, "Well, this is confusing, because he told me OKRs are the world, and now you're telling me KPIs are important."

In fact, they're basically two sides of the same thing. They're not completely unrelated. But when you think about OKRs, OKRs are your objectives and your key results, and that's how you reach milestones.

As I'm describing it, KPIs are in fact the milestones that you're setting for yourself. There are two ways you typically would interact these two. There is, "Your OKRs inform the KPIs," so, "What do I need to track to know if I hit my key results to hit my objectives?"

That is the bottoms-up way of thinking about it. And it makes a lot of sense in the early days of your company where your objectives are the most important things.

Your objective may be, "We need to finish building the product." Well, at that point, that's your objective. Your key result is measuring, "Does the product work?" And then the KPI is essentially, "Does the product work?"

But there's another way of thinking about it, which is also top-down, which is, "Where do we need to go?" If you define your milestones numerically, in terms of metrics, then your OKRs become ways to try to reach those metrics.

In our drone example, if our goal is to hit 100 deliveries a day, all of our OKRs then are designed to help us achieve that specific goal. That goal is a metric, and that goal becomes measured by our KPI.

I am a big fan of the latter, because I find that when I'm setting my own goals for my company, it is much easier to think top-down in terms of the numbers, the analytics, and setting those goals, because they're uncompromising. There is no opinion in it. Setting a numeric goal like a certain number of customers a month or a certain goal we're hitting, it is the stark reality.

Reaching Informed Goals

The question is, "How do we reach that?" If you started thinking about what you can accomplish, if you think from the inside out, "What can I reach? How many people? What features can I build this week?" you end up in a world where you're not really optimizing towards a future goal.

I recommend your KPIs are informing your OKRs, meaning that you've picked milestones that are numeric, that are quantified.

But even if you do that, how many KPIs can you have? I recommend having only one. Actually, that's not true. You usually need to have more than one. There are very few businesses that you can quantify with a single numeric milestone. Usually five is the most I would ever recommend.

Basically, the lesson here is you don't want to have lots of them. Because if you have lots of milestones, lots of numbers you're tracking in your KPIs, this is what your dashboard might look like. And this is confusing because it doesn't make any sense.

What you need is more signal-to-noise. You need to know where you're going, and you need to be able to talk about it very simply and concisely. Most business have five KPIs that they track: their customer acquisition, their customer engagement, their customer retention, their total revenue and their total cost.

In most cases this is all you need, because if you can quantify these five things, you know exactly how you're doing at any point in time. For your business, what does it mean to acquire a customer? Remember what it means to be engaged? They may be different. But in general, these five categories are the things that people track.

What would this be for our drone company? Our acquisition would be the number of first-time customers per day. "How many people start getting drone deliveries right now?" Our engagement may be, "How many times do people order?" It may be less than one, maybe a half, maybe a few times a week. But we need to know how many times are people coming back.

Our retention is, "How many people keep ordering lattes by drone 30 days later?" That's an important question. "How much revenue do we make? And how much does it actually cost us to deliver?" You can see very quickly through these five numbers exactly how our business is doing, where we're going and exactly how we might achieve the next level of growth.

This is how you think about KPIs in your business. And then, when you can map those easily into OKRs, "Who needs to do what to be able to achieve goals we've set for ourselves, based on these numbers?"

Good KPIs, Bad KPIs

It is possible, though, to have bad KPIs and good KPIs. So, what is the difference? A bad KPI is something that you can actually control, something you can affect yourself. A good KPI is something that is what I call a second-order effect, something you cannot, in fact, change yourself.

The reason I say this is, if you're going to set a goal for yourself, and you can actually go there and kind of tweak it a little bit to reach that goal, the temptation to do that is overwhelming in the stressful environment of building your company.

You will do it because everybody does it, right? And so the easiest way to bypass that is to never give yourself the temptation by focusing initially on goals that you can't actually cheat at. What do I mean? Well, a bad KPI would be unique visitors to my website. Why is that? Because you can run a bunch of ads and have people go to your website. You could do that. You just spend money and people go to your website. It doesn't mean anything.

You could have total downloads. Again, you can get people to download your software. You can pay them if you have to. Or, you know, total revenue, because you can play the most amazing accounting games you've ever seen to make your revenue look a lot higher than it actually is.

You're like, "Oh, I would never do that." And everybody who's actually done it also said that they would never do it, and they've also done it.

So, what would be good KPIs? Total purchases. Because you can't actually force somebody to purchase your service. A purchase is actually a really good action.

A conversion rate. "How well am I converting these website people that are coming in, from wherever they're coming from, into actual customers? Profit, because profit doesn't lie nearly as easily as revenue does, and it makes it a much more stark contrast. You can see your net profit. You can see how much cash is in your bank after you've done the revenue.

If you pick KPIs that are not directly affected by you, but things that are these second-order effects, you find that it's much more hard. It's much harder to achieve them, first of all, but secondly, it provides an enormous focusing factor on what's important.

Because once you no longer have the temptation of cheating, you have to build the business. You have to do the things that are fundamentally the hardest, which are the most important.

And you can't avoid it. So that's why I recommend choosing those. Let's move on to the second section.

Numbers Lie

We've talked about KPIs for like, you know, forever. That's fantastic. That's great. I've picked my KPIs. That was really easy. In fact, I already had a dashboard set up before you even started talking, so you just wasted my time, to which I say, that was your fault and not mine.

Let's talk about actually measuring things, which I also call "numbers lie." It might be surprising for somebody who talks a lot about analytics talk about numbers lying. How do I know that numbers lie?

I know that numbers lie because some of the businesses that are the most data-driven in the world lose the most money.

There are numerous occasions of hedge funds that literally lose a billion dollars overnight. And so if numbers didn't lie, this should not be possible. But in fact, numbers lie all the time. And your numbers will lie to you.

Even if you pick your KPIs perfectly, even if you spend months meditating about the perfect KPIs, it doesn't mean they're telling you the truth. There's lots of reasons why. I call them all different kinds of bias. I only have time to talk about two of them today.

I'll talk about collection bias and interpretation bias. But you should take my word that there are dozens of different reasons. I will leave you on your own to figure those out, but I will talk about these two.

Collection bias, what is that? Let's assume we are a business, and we have a conversion funnel. We have people visiting our website. They turn into leads because they express interest, and they trial our product. Maybe they test our API or download our STK, or whatever it means to test it out. Eventually we convert them into a sale, and that's our funnel.

We're measuring this using whatever tool we measure it with. And it looks fantastic. We're turning 10,000 visits into 5,000 leads, 500 trials and 50 sales. It's brilliant. That's awesome. We feel really good about ourselves. We throw a big party. We raise a whole bunch of money.

We have an, "Oh, look at us, we're great." In reality, what actually happened was we're not tracking these things correctly. We actually had 100,000 visits, which only generated 5,000 leads, which only generated 500 trials and 50 sales. That doesn't look nearly as good.

It was all because we actually weren't tracking something correctly. Of course, you're all engineers. So you're thinking, "Well, of course I would track it correctly. I'm not going to make this mistake." Some of the biggest companies with the most people, the smartest people, make this mistake all the time.

It's not only about what your numbers are telling you, it's about how you're tracking them, the definition of what you're tracking.

What is a visit? How do you define a visit? Does everybody in your company agree that this is how we track a visit? Are you sure that, every time you do a code deploy, you haven't changed the tracking code and broken it on certain pages? So certain pages of your website are no longer reporting in?

This is actually taken from a real example of a company that did not realize that their tracking code was not on all their pages. In fact, it was only working on 1/10 of their pages, and they were tracking 1/10 of the visits they actually got.

How do you avoid collection bias? At what point are you collecting the wrong data to generate these KPIs, to answer these questions? There's lots of things. The first is don't try to collect everything. There's a lot of temptation as an engineer to collect lots of stuff and think about it later.

The problem is the more you collect, the more difficult it is to agree on the definition of something. It's surprisingly difficult to agree on what the definition of a lead is. So what is a lead? I have no idea. It depends. But the more things you collect, the harder it is for you to keep track of what everything means, and the easier it is to break it.

Bias Versus Clear and Consistent Tracking

The second is make sure you clearly define whatever it is you are tracking. Somewhere. Anywhere. In a Google Doc, in a Wiki, on a napkin that you tape up on the wall. Everybody should know where to go to figure out what a lead is or what a visit is in your system.

What's conversion? What's an engaged customer? You need to know what these things are, and everybody has to agree. I have been in meetings where people have talked past each other, talking about what an active customer was. They did not realize that their definition didn't agree. It was different.

Finally, you should be testing all of your data collection the same way you test features. If you have continuous deployment, that's awesome.

Are you tracking the same analytics every time you push code out, or are you just testing the features of your own product? Because, I guarantee you, if you don't test it, it will break. You won't know it broke, and you'll realize too late that all of your numbers are garbage, because you broke it, and you don't know when. Now, that's collection bias.

Let's talk about interpretation bias. What is interpretation bias? Let's assume these are your numbers. These are your metrics. It looks really noisy. I don't know what that is. And so somebody comes to us and says, "How is this metric doing?" And we're like, "Uh, hm. Let's add a trend line because that's, like, you know, what you do to this stuff. Boom, there we go. Up and to the right. What's up? That's awesome."

Unfortunately, that actually doesn't really look like it fits the data very well. Instead of a straight line, let's look at a second-order polynomial. "That's much better. In fact, it's up and to the right, so we should go raise money because investors will throw money at us, right? That's great!"

Except that also doesn't look like it fits the data very well. So, let's go to a fourth-degree polynomial. That's looking good. That's looking much better. Although, what is better? What is a better fit? I don't know. Is this a better fit than this or this? I don't know.

There's a point where the best fit is actually the data itself, and that's what we're trying to derive value from.

What you should hopefully be learning from this is that a lot of what you learn about your data is vulnerable to what you're looking for. I could easily make a case that we're growing, and I can make an easy case that we're not. It depends on what conclusion I want to come to.

You have to be very careful of reading things in your data because often your data will look like this. It won't be clear what's happening. You're going to need to actually come to conclusions and figure out what's going on. And so the most dangerous thing is to render interpretation bias where you're reading something in that isn't really there.

Especially in the early days, when you don't have a lot of numbers, the numbers are very noisy. You can run to conclusions that are totally off-base. How do you avoid interpretation bias? The first is let the data tell you what's happening.

If you go into your data and say, "I need to figure out if that feature is generating exponential user engagement improvement," you have probably a way to figure that out and justify it. Or you can go in and say, "Let's figure out what's going on with user engagement."

If you go in with the conclusion, numbers are great. They lie to you, as we've already covered, and you can justify anything you come up with.

Better Analysis Sources

The second thing is seek out independent analysis. By the way, asking salespeople if they hit their quota last quarter, they'll tell you yes every day of the week. Somebody who does not have a vested interest in an interpretation of the data is the best person to tell you what's going on.

Maybe that's you, or maybe it's your adviser or your investor, or somebody else, but definitely not somebody whose reputation is on the line.

And finally, multiple data sources are great. If your sales are up, and that's great, and you think it's because of a feature you added, but everybody's sales are up because it's the Christmas season, you should probably think about whether or not your conclusion matches what's going on.

Which brings us to the third point, which is that let's assume we picked the right KPIs. Let's assume we're not biased, and so our numbers aren't lying to us. How do we figure out what tools to use? Because, of course, some of you may be analytics companies, but not all of you. If analytics is not your core competency, obviously you're going to let somebody else do that for you, because you're too busy building your product and selling it to customers, the things we do.

I call this "Don't Buy 200 Hammers." Why do I say that? If you think about business intelligence, today you are in a blessed time where there are many options ahead of you. There are, in fact, two kinds of analytics.

There are what I call descriptive analytics like Google Analytics or Flurry. Just drop in a JavaScript tag or an STK, and it gives you a whole wealth of information with no effort, just a simple integration, gives you lots of it. But usually the knowledge it's giving you is not specific.

There are event-based analytics companies, like Mixpanel or Keen IO, where you can track anything about anything, You can have it track analytics about your toaster or about whatever it is that's going on. And those are great. It's super flexible.

You're like, "Well, that's great. I'll just pick Keen IO. Maybe I'll toss Google Analytics on the website, and I'll be done. But this is not the universe of business intelligence. It also includes your sales and CRM system, your customer support system, your financial accounting system, your billing system, your advertising system and your ops system. These are all parts of how your business is doing.

Very quickly what you realize is that, "Hey, I'm actually using all of those. How do I know what's going on?" And that, of course, is the most important question, because now the answers to my questions that my KPI has posed to me may be spread all over the place.

My revenue is over here. My costs are over there. My customer engagement's here. My acquisition is over here. And how do you figure out what's going on? You start to drown under the tools you're using and the data that you are collecting. The answers are all there somewhere. They're just getting so hard to find that it's not practical for you to, every day, log in to every one of these systems, export an Excel spreadsheet and try to figure out what's happening.

How do you avoid this kind of overload? Focus on what you need. We said at the beginning, "What are your goals?" Even though you're using all of these tools, they're tools. They aren't telling you what to do.

Figure out what you need to know and focus on that exclusively. Don't get distracted in the fact that these systems are collecting data that you might not need.

If you focus on everything they collect, they collect everything. You'll spend all your time sifting through it. Instead, "What do you need?" and focus on that. Don't let the paradox of choice overwhelm you. Just because it's collected, just because the data exists somewhere, doesn't mean it's important to you.

I will give you another example about why this can be bad. There are some services which are a tax on success. And you're like, "That's not possible. I'm smart. I can do math!" And that's true, but think about some of these services. A lot of them will charge you per event. Maybe that's an analytic service where you're tossing an event, and it charges you per event. Or it could be one of these other services where I pay per payment every time a payment happens.

Let's assume there's some service I'm using which is one penny for every 1,000 events, which sounds like a great deal. And it is a great deal. That's really cheap. So 10 events per customer engagement, a customer uses it three times a day. I have no idea what our product is. It could be anything from an API, to a payment server, to an STK.

It costs us, with 10,000 users, only $90 a month. This is awesome because that is an order of magnitude less than a desk cost in San Francisco. So we were like, "What's up? This is a deal." Unfortunately, what ends up happening is, all of a sudden, you're successful. And now you have a lot more users, and it costs you a lot more money.

The number of companies that are successful and get bills literally for millions of dollars from their cloud providers from one month to the next, because of success, is surprisingly high. So just understand the cost of what you're using and what you're tracking, because even though it seems cheap at first, these things get expensive very quickly for the same reason that your products get more expensive the more you use them. It's because that's how you make money. Just be very careful you understand the cost that goes into them.

Now, in summary, I want to bring up that your vision is more important than your metrics. No matter what I tell you, your vision has to dominate the numbers.

The numbers will never tell you what's going to happen. The numbers only tell you what already happened.

Your vision is the only thing that will tell you where to go. Your vision needs to be there, because the numbers will always tell you to do micro-optimizations, to do these small things. Your vision has to carry you through.

In the early days, the numbers will always be bad. By definition, everybody starts with zero customers. It's how it works. And so your vision has to be the dominant thing that you follow, but your metrics help you get there. Your metrics are a tool for getting there.

What About Growth?

Wait, hold on. What about growth? We are startup companies in a startup ecosystem. How do we know about growth? Because we can't raise money without growth. We won't be successful. We need growth. Growth, growth, growth, growth, growth.

I will talk about growth because growth looks like this. This is absolutely impossible to plan for. It is not possible to plan for an exponential growth curve. And so if it's not possible to plan for it, but we need to have it, what exactly do we do? What we do is we set a goal.

We broke our vision down into milestones. We set some goal. Our next goal is 10x higher than that one. And our next goal is 10x higher than that one. And our next goal is 10x higher than that one. This is how you break down your business and realize that, "I can be a high-growth engine. I can raise money, and I can succeed in this environment."

I didn't need to plan for exponential growth because what I realize was, wherever I am, all I need to do is set my next goal at 10x higher.And that sounds very silly. It sounds like, "Well, that's not really going to help me." But what it does is, as you're setting out these milestones and thinking about the KPIs and breaking those down into OKRs and whatever other acronym you want to use, it makes you realize that the things you do at every stage of your business will change, that the metrics and analytics that you track will change. The way that you get there will change.

When you have 100 customers, and you're thinking about getting into 1,000 customers, you realize that the way you got to 100 customers is not the same. Once you have 1,000 customers, the way you get to 10,000 customers, that's also not the same. From 10,000 to 100,000 customers, that is definitely not the same.

Every time you go through one of these exponential growth you realize that you have to shift gears in your business.

Maybe your numbers are smaller than this. Maybe you go from one customer to 10 customers, to 100 customers, and that's a big deal. But with every step along the evolution of your company, your tactics, your analytics and your metrics are going to change.

As a founder, as running your company, you need to realize that you need to understand where you are, but you also have to realize that your next goal is not just to get more of the same. Even if what you've been doing is working, you have to find a way to do it 10x more, because if it doesn't, you can't generate an exponential growth curve.

If you can't generate exponential value creation curve, you cannot raise venture capital funding. Which may be okay. There are many businesses that don't follow this that are cash-load businesses. Walmart is a great example. They just don't raise venture capital money.

Let's review. Choose five KPIs. I would prefer you choose one, but that's probably impossible. So we'll stick with five. Make sure your metrics are accurate. If they're not accurate, you should know that they're not accurate. Use only the tools that you need, because otherwise you'll get overwhelmed by all the data that you have at your disposal. And your vision is the most important thing at your disposal because it is the most important thing.

So my name is Sean. I have a company called Outlier. I have no product to sell you, but I will in a few months. What questions can I answer for you?

Q&A

Focusing on the Right Tools

In my experience, what I've found is a few things. One is I don't actually find it's useful to, I'm not a huge believer in, for example, real-time analytics. Because I think that it gets very distracting to check your numbers every hour or a few times a day.

I try to set times where I do think about the numbers, because there's so many other things going on with your business if you don't time block it. For some businesses, it's once a week. For some businesses, it's every day. For some businesses, it's every month. It depends a lot.

If you're a hardware company, and you're dealing with supply chain issues, you may need to measure your analytics in a very long time frame. If you're selling a payment system, you may need it measured every hour. It depends on what your business is. But figure out the cadence. When does it make sense to check the numbers? That's the first thing.

The second thing is if you do start with this concept of starting with your vision, and knowing what your goals are and what KPIs derive from that, come up with these five numbers. Those are all you care about. Everything else is superfluous. In that world, you just need to make it as easy as possible to get to those five numbers. Even if it means you have to log in to more than one system, it should not take you more than a few minutes to figure out what those five numbers are at any given time.

You could have a dashboard. You could have an email alert. I mean, there's lots of ways to get to the root of what those five numbers are.

But the key, the hardest part, in my experience, is blocking out everything else, all the numbers that aren't those five numbers, everything else that is more nuanced.

It's hard to block it out because there's so much of it. I have this phrase that I use called "analytics anxiety" where you have so much data, you know that there's something there, and you want to spend the time to go through it. Because the smart people who have technical skills, we believe we can make use of those numbers.

The hardest thing is not spending all of your time doing that and realizing that you have to trust in your judgment as to what you've decided is important. As part of that, it also is very useful to revisit, "Are these the right KPIs for us?" on a regular basis because you can't just blindly select some KPIs and assume it will always be true.

If you're doing what I recommend, and you're focusing on those, then you end up in a world where you worry that you're missing out on something. You worry that there's something you're not tracking, that listening to your KPIs is really important. But if you give yourself a regular cadence to revisit what those are, and maybe it's monthly, maybe it's quarterly, you can be sure that, "Listen. I have sat down. I have thought about it. And if I know these five things. I know how we're doing."

As long as I know that, and I know that I have a chance to revisit in the future, you don't go through the cognitive load of trying to re-address all of your assumptions every day or every week or every hour. Because, at the end of the day, there's so much going on.

These, again, are supposed to be the lighthouse on the ocean. They're not supposed to be the radar pinging you every five minutes, because then it's almost worse.

It makes the problem worse, because then you're not focusing on your business. You're focusing on that pinging noise in the background. Eventually, you just want to throw it out the window and let it leave you alone because it's getting so annoying.

That's how I approach it. I have found that that works very well. It's the same reason that taking a step back from your business is useful. If you can't set a cadence for yourself for when you check it, it's very hard to have the discipline of not doing it all the time. I realize that probably doesn't help everybody, but that's how I deal with it.

KPIs for Evaluating Personal Performances

Again, these are not tactical tools. Your OKRs, in my world, in my belief, and Kris may disagree with me, is that my OKRs are designed to help make sure everybody in the team is driving towards my goals that I have set, these numbers that I have set as the next milestone.

If I have set the next milestone as 500 customers, everyone's OKR, from the engineering team to the sales team, to the marketing team, are designed to help push the company in that direction. From that perspective, I would never sit down with somebody and say, "Listen, we have this metric of 500 customers. We haven't reached it, so you're fired," well, unless I didn't have any cash flow, and then it's kind of "layoffs," which is not the same as firing.

Your personal performance is in pursuit of that goal, but it is a shared company goal. Again, this is a different way of thinking about your business. This is thinking about your business top-down. You're thinking strategically. You're thinking about where you're going next and mapping that into the tactics.

I refer to OKRs as just tactics. Some people have these systems of OKRs which are individual OKRs, team OKRs and company OKRs. You can think about it that way. But I think what happens is it lends itself to a bottom-up way of thinking. You start thinking about what is possible, and you start rolling it up slowly.

I think about my OKRs, what can my team accomplish, and, "What does that mean for what the company will accomplish if those teams are successful?" I find that I would rather think top-down because if I'm going to work as hard as I know I'm going to work in my company, I want to make sure that, when I get to where I'm going, it's the place I wanted to be.

I will tell you, I meet lots of companies that worked very hard for 12 months to 18 months, and they're like, "I'm ready to raise money." The milestone they reached was not a milestone that lets them raise money. It's a very difficult realization to have, that all that time all that hard work you're putting in has put you in a position where, in fact, you can't take the next step.

There is no more capital to be raised because they didn't hit the right milestones. I would rather work backwards. This is why I prefer the top-down method, because I want to set the goals that I'll be looking at next. When I work backwards, and I set those tactics, I know if I'm working on something today, and I have to work all night, I know that it's not going to be wasted effort. Because I know it's helping me get towards this goal which is a worthwhile goal which will help me get to the next milestone in my business.

That's why I like this. I am not in any way saying the other ways of thinking about businesses are not good. There are many successful founders and entrepreneurs and executives that think about things differently. This is just how I think about the world.

Non-Human Mistakes

I would actually argue the collection bias, in many cases, is actually not a human mistake. You just simply broke it, and didn't continue to test it, which I guess you could argue is a human mistake. But in many cases I find that it's just simply like breaking a feature accidentally or inadvertently.

Other ways that numbers lie. One is, if you think about some of these hedge funds, why does a hedge fund lose a billion dollars overnight? It comes down to the fact that they had assumptions that they built into their technology which no longer held true.

For example, one of those companies assumed that the Swiss Franc would never float. The minute the Swiss Franc floated, I lost a billion dollars. The question is what assumptions are built into what you're doing?

One basic example I'll tell you is that, at Flurry, we had people using the APIs. And so you start tracking metrics around API usage. You assume certain use-cases have helped people. We'll use the APIs. All of a sudden, when those assumptions fall flat, you realize that the metrics you were tracking, and the milestones you're going for, are completely irrelevant, because they're using it in an entirely different way than you expect it.

Let's say that, as to give an example, you have an API business, and you consider an engaged customer, somebody that's hitting your API 20 times a day, for example. Unfortunately, most of your customers all hang at five times a day. Does that mean you actually have less engaged customers than normal?

In that case, I would actually say it depends. Are those people hitting it five times a day but cacheing their responses and actually using your API hundreds of times a day? Just cacheing it more often than you'd expect or more than you'd allow?

Oh, my God, they didn't. They broke the rules. They didn't follow my instructions. They're not supposed to cache their responses, and they're doing it anyway. Is that a success or is that a failure? I mean, I would argue that it's somewhere inbetween. People are using things a different way than you'd expect. Your assumptions are no longer true. And you have a chance either adjust your assumption or to change the technical infrastructure.

"Numbers lie" is a fairly quaint phrase. But what I'm trying to get at is that a number is only so good as where it comes from, and your understanding of what it means, and isn't collected from what you expect.

Everything derives from this idea that, if a number is collected in a way that you didn't expect, that assumption is false. If the meaning you're attaching to it is different than reality, that assumption is false.

It boils down to the assumptions that you're making that underlie all these numbers. And that's why they lied to you. It's not that a five is not a five. It's always a five. But what that five means may change based on your interpretation of it.

KPI Evolution at Outlier

The difficulty is we're so early, as I mentioned at the beginning, numbers don't really apply to us yet. We don't quite yet have customers, although I aspire to have that change soon. We don't have revenues because we have no customers. And I don't yet have a product for those customers to buy, which is why I have no customers and no revenue.

So we're a little bit early for measuring our business numerically. However, I think that the way I think about my business and these KPIs, you remember the five categories I talked about: acquisition, engagement, retention, revenue, and cost. They also happen to kick in at different points in the company's life.

The earliest one, the one that I think is the most important, is engagement. When you're first going out with a new product in the market, the question is engagement.

Do people use it? Do they use it a lot? Did they make it a part of how they operate? Because if that's not true, everything else is academic.

It doesn't matter about your acquisition. It doesn't matter about your revenue. If people don't engage with it, then they're not happy. And if they're not happy, they won't be customers. From there, I think about, OK, that's great. We were able to make people happy. How do we retain them? Because almost anybody will try a new product. They might even try a new product and use it a lot, but do they keep using it over time?

That becomes the second most important thing for me, because if I can make people happy and continue to make them happy over time, now I know I'm on to something. Now it's worth reinvesting more in to. If I can't prove those two things are true, I really need to keep working and iterating until I can prove those two things are true, because everything else, if I can't prove those two things are true, is academic.

Then the next question, can I actually acquire more customers at scale in a way that lets me grow very quickly? Because if I can make five people happy and retain them long-term, but they're the only five people that I can, I'm out of business. Well, that's unfortunate. Now I need to make sure I can acquire them at scale.

Then revenue and cost come in because revenue and cost are two sides of the same coin. Those are your business. At the end of the day, a business is defined as something that generates more money than takes in, which basically is defined as the fact that I pull in more revenue than my costs. It's often very difficult to achieve.

I'm making a flippant case about it, but if you're on to the first three, if you figured out scale-block customer acquisitions, if you've figured engagement, you've figured out retention, then the goal is building a business on top of that foundation.

Assuming you haven't done something crazy like acquiring people for $50,000 for a $500 product or paid them money to be retained, it should be possible to build a real business on top of what you're doing. That becomes a goal that you have. But you can't do everything at once.

If I had these five metrics to begin with, I would be paralyzed with fear of not knowing where to start. I'd start working on one, and then I'd jump to another, and then jump to another, and jump to another and I'd do nothing very well.

The reason that I think about this order of stuff is it gives me very clear focus at every step. My goal right now as a small, early-stage company is to focus on making a product that will engage the customers, that they will love to use every day.

If I can do that, I will make something that will retain them and make them happy for long periods of time. If I can do that, I will find a way to acquire them at scale. If I can do that, I worry about my cost and revenue structure and make sure the business scales along with it.

Why 'Start Over' With Outlier?

The short version is that if you do this and you enjoy it, and you enjoy the uncertainty, it is unlike anything else. It's something that you love to do.

There are few jobs that I've ever had, which were not founding companies, that challenge you in the myriad ways you get challenged building a company, that are as hard to do as building a company, but as rewarding, not even when you succeed, but rewarding along the way when you achieve the milestones that you're achieving because you're starting from nothing.

As a founder, by definition, you're starting from nothing. The minute you achieve anything, you've created it from nothing. That, in my opinion, is very fulfilling, and I enjoy that. The second is that I think that if you are a founder, it's because you have visions. You have problems in the world that you believe need to be solved.

The reality is very rarely is somebody else going to solve your problems for you. And so you find this burning need to make sure this is maybe not solved, but at least you give it a really good go. You want to make sure it happens.

I will tell you, though, that I meet with a lot of early-stage founders, people who think about quitting their jobs. I think poor reasons to be a founder are if you want to get rich, if you want to get famous or if you want to get powerful. Those are very bad reasons, because it basically never happens.

If you want to get rich, you should go work for a hedge fund. They pay you very well. You make gobs of money. If you look at the highest-paid people in America, none of them are founders. All of them are hedge-fund people getting paid a-billion-dollar salaries a year.

If you want to get powerful, you should go into politics. It's much easier to do. If you want to get famous, go on reality TV. I hear it's very easy to do these days. In reality, people found companies because they love to do it. Loving, solving problems, working with great people and trying to do something that is very difficult to do is important because, frankly, between you and me and the camera and all the people watching it online, the chances of success with any sort of company are very low.

If you think about the success of the company, it's a random variable. Statistically speaking, the chance of success is basically zero. Very few startup companies ever succeed. And why is that? It's not because we're stupid. It's not because we're not working very hard. It's because, in reality, it's very difficult.

It's hard to build something that generates more cash than it takes in. It's hard to build towards exponential growth and value creation. Not every business can do that. And, in fact, along the way, as we've talked about, there are so many things that can go wrong, you may fail at any given point.

If you realize that failure is just something we live with, then our job as founders is to defy reality and try to create success in an environment that's very harsh to us. In those dark days when I struggle, when I realize how hard it is building companies, what I realize is that it is not in my power to be successful.

What's in my power is to do the best job that I can, to position myself for the best I can do, and to believe that, if I do everything the way I think I can do it, if I achieve the goals and the milestones that I set out, I have given myself a chance. And that's what I love to do.

Thank you very much. It's fantastic.

You've been here a while...

Are you learning something? Share it with your friends!

Want developer focused content in your inbox?

Join our mailing list to receive the latest Library updates. After subscribing, tell us your preferences to receive only the email you want.

Thanks for subscribing, check your inbox to confirm and choose preferences!