June 14, 2016
Beta Testing for Early Developer Teams
UX designer and former Treasure Data product manager Luca Candela discusses defining your feature tests, cohorts and research experiments. H...
My name is Bill Lapcevic. I run business development at New Relic. I've been at New Relic for quite a long time. I was the third employee there. The reason that they hired the BD guy third is because, really, New Relic was founded on a partnership with a company called Engine Yard. We'll get into that story in a little while, but it's an important one because it gives you an idea of how valuable partnerships can be, even when you're very small, even when you haven't launched your product yet.
Before that, I was at a company called Solidcore, where I developed their alliances program, and I was there about a year-and-a-half. They were in the change control management space, so infrastructure, just like New Relic. And before that, I was the 13th employee at Wily Technology, which later got acquired by Computer Associates. It's interesting because Wily is in very much the same space as New Relic; it's in the application performance management space. But, the partnerships were a little bit different, and we'll touch on a couple ways they were different.
Wily is more in the enterprise and on-premise space, and New Relic, for those of you who know New Relic, is a software as a service-based application performance management tool. That actually opens up a huge world of integrated partnerships that on-premise products don't have, and is one of the reasons that I think New Relic is so wildly successful with its partnerships.
Down below that, just a couple of companies that I've worked with before, in partnerships at these companies, they range from BEA and IBM, where we built for Wily two enormous partnerships, to New Relic, where we've built 150 partnerships; some very shallow, some very deep, all very important in their own way.
Okay, so, I mentioned that I was the third employee at New Relic. Let me start by giving you a glimpse of the before and after of New Relic business development and New Relic partnerships organization. We have two guys who handle what we call key partnerships. These are our biggest, most hairy complicated partnerships. Amazon, Microsoft, Google, Rackspace, VMware. They're really complicated. They could suck up all the time in the organization. It takes a special skill set to really be able to manage them appropriately.
We have what we call top-tier partnerships, which are like key partnerships in that they're really, really valuable, but they're smaller organizations. Engine Yard is an example of a smaller organization that's a top-tier partner of ours. Acquia, if you've heard of Acquia. They're a big Drupal platform as a service and hosting provider. These are organizations that, they're just not as complex, but they're still very important in terms of revenue, lead generation, all the things that New Relic values out of a partnership.
We have emerging partners and hosting sales. There's a whole other class of partnership where they're smaller organizations, but they're still valuable because they have big blocks of customers that we want to access. Because there's so many, we can't put the resources and time into those partnerships that we would, like we do with an Amazon or a Microsoft.
So we take a more programmatic approach there â€” a lot of emailing, a lot of generalized value proposition stuff. But, those partners get a lot of value out of New Relic and out of the partnership, and so they're more than happy to provide our product, through our partnering motion, which I'll talk about in a minute.
The organization is bigger still. New Relic has Platform which is a new offering that we've added and it has Mobile, which is a new offering that we've added. Each of those have a separate set of partners and a separate partnering motion. Then there's the partner response center, partner marketing, and business enablement engineering. You can see that the greater BD team right now is something like 10 or 11 people that are handling partnerships within the organization. These are my resources that I have to work with. We're adding because we're going into Europe, so I need a BD person for Europe, and we see an opportunity with systems integrators, so I need somebody to build a program there.
This is not to say that your business development organization is going to start out and be this big and this complicated this fast. Remember that New Relic is five-and-a-half-years-old, we're about 330 people. In fact, I guess that makes us 110 times the size that it was when I joined. Having a business development organization like this, it's a big BD organization, but as I said, partnerships are really critical to the success of the company.
Let's talk a little bit about why partnerships are important, now that you know that it is. What I've tried to show in this chart is that over time, adding sales people to increase your revenue becomes problematic. It increases cost, because each salesperson comes with a pretty much fixed cost, and territories eventually grow smaller and smaller, and productivity drops. At the beginning, you have huge amounts of greenfield opportunity, and it's pretty easy to find those customers who are going to pay.
When you get to the size of New Relic, the productivity starts to drop for each sales rep. Each sales rep has a smaller territory. They often have fewer leads. The leads are maybe not as easy to cherry-pick from. If you have 1,000 leads, you can pick the top 100. If you only have a hundred leads, picking from the top 100 becomes a challenge.
You know what that is? Jupiter. Why would I show Jupiter? And why would I show Voyager 1? Well, I was thinking about this, and actually, this is a really great analogy for why partnerships are important. There's this really great thing that happens when you try and get a spacecraft the size of Voyager 1 into the outer reaches of our galaxy. You can't put enough propellant on it to get it out there by firing rockets the whole way. So what did they do? Anybody know? Gravitational slingshot. What they actually do is they leverage the gravity of Jupiter, this huge body in space, to slingshot the smaller body to a greater speed. They actually do this a number of times with a number of planets just to get out of the solar system.
The point is, though, that you can do the same thing for your organization. You can leverage a bigger company, an IBM, an HP, or when you were the size of New Relic, starting in Engine Yard, which had been around for 3 or 4 years longer than we had. You can leverage their brand, their marketing momentum, their people, their sales channel, and their customer base to slingshot your organization much faster than it would have grown all by itself. It turns out that if you do that, you've got a really great board chart.
The point of this is that partnerships can be an exponent on the growth of your company. Just keep that in mind as we go through this. That's why partnerships are important, because they need to be an exponent on the growth of your company. If they're not, they're kind of a waste of resources and time and energy. Which, going back to the first statement,
"There are a lot of bad BD organizations and bad partnerships out there that are a drain on your resources."
Back to New Relic. At New Relic, business development is actually responsible for 15 to 20% of the leads and revenue that come into New Relic every year. We've been at that run rate for about 2 or 3 years. It's a pretty impressive amount of business that are coming out of partners, especially when you see the size of our business development team compared to the size of our 100+ person sales and sales operations team. That's leverage. There's value there. We did 428 leads. We created 428 net new selling opportunities for our sales people in 2011; 2013, we're on target to do 5,000 leads. That's about, I think I calculated 11x growth.
So how do we do that? Let's talk about the mechanism by which we go out and get these leads. We're going to talk about identifying partners in a minute. Once we have a partner, we do a couple of things. New Relic, we identify. We look for partners with big blocks of potential customers.
For New Relic, a customer is an organization or person that is deploying a web application and has some need for performance management. Typically those are developers.
We typically sell to developers, some operations.
Where can we find big blocks of those, that we can go out and through a single relationship gain access to thousands and thousands of customers? Well, if you look at Amazon AWS, they have hundreds of thousands of customers. Let's say 400,000. Rackspace, 150 to 200,000 customers deploying web applications on their cloud. Microsoft, 150,000 customers, not to mention all their on-premise customers that are hundreds of thousands of more. Just with those three potential partnerships, we could actually get access, better access, than we could reach with our own marketing efforts to almost a million customer opportunities.
Now, even with partnerships, that takes time to penetrate, but the access is there, and that's really exciting. Your salespeople will get really, really excited when you announce that you've launched a partnership with Microsoft, or you're built into Azure. Think about that. Lots of leads.
The next thing we do is we need to provide some value to the partner. Otherwise, why would they partner with us? They don't want to just give us access to their customers.
There are a lot of things that partners will do for you, and just giving access to customers for the sake of your company is not one of them.
So we let them actually give away a paid version of New Relic, New Relic Standard, for free to their customers forever, unlimited. As many people who want to use it as they want or on as many servers as they want. They actually are giving away something that per server runs, a customer who comes direct to us, about $50 a month per server. Partners love that.
They love being able to augment their product offering with something from you where they don't have to spend any money, but they get value. They get marketing value out of that, customers are attracted to getting extra tools when you join a cloud provider. On top of that, we then let the partner use the paid New Relic product for free â€” not just give it away, but they use it to help support their customers.
We identified early on that partners have a problem. Customers are calling them up all the time saying, "I put my app on your environment, and it doesn't work, and it's your fault." They had no way of saying, "No, it's not our fault. It's not our infrastructure. It's your code." The developer says, "Oh, it can't possibly be my code," but it often is.
We found a way to entice the partner to use our product. They need visibility, just like the customer does. And, by giving them that visibility free of charge, they're willing to put us everywhere. They're willing to do marketing and integration work to drive our product into their customer base, which we'll talk about too.
Then we layer on, as I mentioned, integration and marketing. In my opinion, in this day and age, especially with a SAS product, and especially with cloud, if you have a partnership that doesn't have a tight integration, technical integration where the two products work together, it's not interesting. Marketing partnerships are just not that interesting.
You'll see with most of our partnerships we actually build our console into the management console of the cloud vendors. We have created billing APIs to allow us to push revenue through the partner. When a New Relic deal is done the partner actually collects the money and takes the top-line revenue.
We have APIs that allow for frictionless adoption by the customers through the partner. Customers can show up, press one button, like with Heroku. If you've ever used New Relic on Heroku, you press one button, everything's installed. It's all running, it's all integrated. Easy for the customer. Partner gets value because they then access that data as well so that they can support the customer. Customer gets value, New Relic gets a potential sale. That's a real win-win-win.
Then we try and convert those free customers to paid customers, and either through the partner and through marketing, or through our direct sales force, we access those customers who look like they fit the qualifications of somebody who would buy our profession-level product.
We sell it, and then the last one, we actually give some of that revenue back to the partner. The partner provided us a lead, we have to pay them. It's a pretty high-value partnership for the partner. It's a really high-value partnership, as you saw on the slide before where 15-20% of our revenue comes from these type of deals. Pretty good stuff.
But why do we invest in this partnering motion? I mentioned before that marketing can get you access to these leads over time. And, again, if you know New Relic, we've got a pretty damn good marketing organization. I bet half of you have a New Relic t-shirt or a helicopter, or have seen the billboards around town, or the "Data Nerd" bus, or any of the promotional stuff that we do. We do a ton of promotional stuff, and it's all brand-building.
We want people to know who we are. Those drive a ton of leads. In fact, if we only are doing 15-20%, then 80-85% are coming through direct marketing. Why wouldn't we just pour more money into direct marketing efforts, into our standard marketing efforts and say, "Don't worry about building up this separate organization, this different way of doing things. Why wouldn't we just rely on marketing to deliver this? That's what lots of companies do."
Well, let me give you some numbers. $300. I mentioned at New Relic,
A lead is somebody who deploys our agent and has that agent report data back to our service.
Marketing spends, between helicopters and SEO and advertising and email campaigns and personnel costs, and everything else, $300 for every single deployment of New Relic's product â€” whether it sells or not. It's a lot of money. Business development spends a hundred, and that's largely in fixed personnel costs.
Now, if you're the COO of the company, where would you like to spend money? The $300 lead or the $100 lead? Assuming that the partnership is such that they close and convert within a reasonable percentage of each other. Business development. That's why the team is growing so fast.
We've found that we can actually be more profitable in business development leads than we can on direct marketing leads. What's more, remember I said all the things that marketing spends on, that also is a blended number that includes organic search, where we don't spend any money, but people actually look us up online through various content, etc. If you take organic search out, marketing spends $700 a lead. Now, can you see why it's compelling to spend $100 instead of $700? Yeah. You make a bucket of money, you save a bucket of money.
Okay, so let's move along. How do we spot the right partnerships? It can be really complicated.
The first piece of advice I'll give you is know your own business, and know what you're trying to accomplish.
At New Relic, when we launched, we launched with Engine Yard. The reason we went into an agreement with Engine Yard where they gave away our product for free to all their customers, was that there was an incumbent in the space, Ruby on Rails market, a company called FiveRuns.
FiveRuns already had about 100 customers, 100 paying customers. They had all the buzz. There was no New Relic at the time, so no one had anything to compare FiveRuns against. Our biggest hurdle was, how do we break into that market when there's an incumbent that everybody knows and that many people use? The partnership.
By launching with Engine Yard as our primary partner, they gave us a way to 400 customers pre-launch. We launched with 4 times as many customers as FiveRuns had. It's that graviational sling-shot. We leapfrogged FiveRuns, and by the time they knew what happened, we had doubled that â€” 800 customers.
Then people were coming from other places. Then other partnerships started popping up because they wanted to do what we had done with Engine Yard because they saw it was so valuable.
These are all reasons that companies think that they should be doing partnerships. You have a friend who works there. You want to generate leads. It sounds pretty good. You want to drive web traffic. Yeah, that's great. Okay, revenue? Sure. Near the golf course? Yeah, for most. It's something to blog about? Yeah. That's great.
In my opinion, this boils down really to 5 things: Credibility. Awareness. Revenue. Defensibility; and Utility. Let's talk about each one of these in turn for just a second.
Credibility. Remember the Engine Yard story. I'm going to keep coming back to this. We did not have credibility because we didn't have a product in the market, but when Engine Yard basically made us their de facto APN product, at launch we had instant credibility. Engine Yard was sort of a lighthouse company in the Rails community back in 2008.
To my Engine Yard friends, you're still a lighthouse in the Rails community. They gave us instant credibility by effectively sponsoring us to all those customers. Those customers didn't have to ask questions about the viability of our company, or whether the product really worked or not. When Engine Yard said, "You know, you ought to use this," that was good enough. Really powerful.
Awareness. Again, launching that partnership with the marketing, the value proposition, and the word of mouth that created, created a huge amount of awareness. We followed that partnership up with a partnership with Heroku. You all know how big and important Heroku is in terms of the developer community. That was the one where everything went exponentially big for us. But Engine Yard was the starting point that fired us into the outer reaches of the solar system.
Revenue, of course, right? Lead generation. What good are leads if they don't convert? And the whole goal of running a business, some would argue, is to make money. We want to increase the company revenue opportunity.
Defensibility. How many of you have seen situations where you're doing really well in the market, and suddenly a competitor pops up out of nowhere hand in hand with some big company, wrecks the market for you? It happens all the time. We actually, at New Relic, we live in fear of this happening. There's a great series of books, Innovator's Dilemma. This is exactly the innovator's dilemma.
As companies grow bigger, they have a harder and harder time innovating, and smaller companies with really smart people who don't have the baggage of a big company can come in at a lower price point and eat you alive. Partnerships are a very legitimate way to defend against that because those competitors are going to try and leverage bigger companies. If you're already there and have a solid partnership, they can't penetrate those bigger companies.
We have a partner, who I won't name, out east. We have a competitor who showed up and tried to get in the door, but because they trusted us so much, that competitor, even though they shared the same investor, couldn't get in the door. They couldn't get the reps and the engineers on board with using their product. They continue to use New Relic to this day because we've proved value, we've proved trust. Defensibility, it's unheralded, but it's important.
The final one here is utility. How do you find ways to extend your product? How do you find ways to take data that you've collected and have it used in other places so that you increase the stickiness of your product? You have to build relationships with those other companies so that the technical integration works and that it's maintained, and that it's successful. That's extending the utility of your own products through partnerships.
Now, an important note, though, is that these are very much stage of company based, and it's really your stage of growth. Early on, we weren't focused on revenue and defensibility, and we certainly weren't focused on integrations that extended the utility of our product. We were mainly focused on credibility and awareness. Those were the company needs at the time, and those are what tied into the company strategy at the time. Then we eventually worked our way down. Now we're doing all of those things. It's just another way of showing that.
At first, credibility and awareness â€” our Engine Yard and Heroku relationships are great examples of that. Later on, we started collecting more metrics, we started getting more refined, we started growing our sales team, and it became more and more important to focus not on just general awareness, but revenue-generating leads. This is where we started focusing on metrics like conversion rates and churn and number of leads and penetration. We'll go into some of these in a minute. Some examples of companies that we work with in this vein are Amazon and Microsoft and Rackspace, but there are a host of other companies.
And then, as I mentioned, we launched Platform. Platform is an example of a business development-led partner set that extends the reach of our product. It extends the utility of our product. We have Redis and Resque and Microsoft Azure SQL and MySQL and 70 other plug-ins that partners have built to extend New Relic's visibility into their products. We had to build partnerships around each of those. Admittedly, those are much lighter weight and much more tightly-defined partnerships, but they're partnerships nonetheless.
How do you maximize a partnership? You've decided now on one or two partners, like New Relic did with Engine Yard and Heroku. They map to your company's strategy and your needs of your company, whether you're trying to gain awareness or credibility in the market. Or if you've already leapfrogged into a revenue-generation mode, how do you make them really valuable?
Who knows what that is? Yeah, if you have a two-year-old like me you know who this is and you hate him. This is Barney, and one of the things that I ask you to please avoid as you develop your partnerships are "Barney relationships." You know what those are? "I love you, you love me, we're a happy family," in a press release, and then nothing else happens, right?
Going back to the first statement, those are crappy partnerships. Those are partnerships that are a waste of energy and a waste of time because whether you know it or not, people get spun up on the concept in your organization and in the other one, and then when nothing happens, they get really frustrated, they're distracted, and they think the business development guys throw press releases out the door so that they can distract people while they go and play golf. Please, no Barney partnerships. They're useless.
So then, what are keys to successful partnerships? How can you develop successful partnerships? Well, I'm going to give you 5 of my favorites. I'm sure there are other important things to know about partnerships, but we can talk about that in office hours.
These are 5 of my favorite that I think are really important. Number one, have a compelling reason to partner. Don't just say, "Boy, that company has a lot of market traction. We need to partner with them." Why? To what end? Who cares? Solve a real problem.
With Engine Yard, with Heroku, with most of our partners, we're solving a real problem not only for the customer by distributing our product to them, but also for the partner by giving them that access to that data so that they can stop the finger-pointing at their organization and get to the root of the problem so that they have a happy customer. Super valuable. That's why these partnerships work.
Again, when it's a Barney partnership, there's no real problem to solve. One plus one must equal three. It's another way of saying the same thing, right? These have to be additive. You can't just have one plus one equals two, or they could come and get that product and they could come and get your product from separate locations.
There's no partnership there. There's no value add, right? It can't be a marketing only partnership. You can talk all you want about how this thing is valuable, but unless you can actually deliver that value, it's not really valuable.
Then finally, a word of warning:
Revenue, in and of itself, is usually not the value proposition that will form a powerful partnership. It's usually something else. It's usually a solution to a problem, and revenue, while it's a problem for both companies, is not a problem that their customers worry about, the partner's revenue.
This is one of the reasons that when you get into a scenario where you're trading leads, you're working on things together, again, it's better to have a real reason that Amazon solution architects wants to take New Relic into an account for us. It's not because they're going to get paid very much money on that deal. It has to be something valuable.
In New Relic's case, it's that they get to go home on Friday because there's a happy customer, not a customer who's complaining that their app isn't working. That's a powerful statement. The fact that you might get $10 a month is not a powerful statement.
Number two, cover all the bases. Don't make this a BD to BD partnership. I tell my team that if they're not working with marketing, product management, sales, support, services, the product organization and the executives as well as all the field technical people, they're not doing their job. If they're just working with their alliances or business development counterpart â€” Barney partnership.
Because BD people like to talk to each other. They're really great at bars together. I have great stories. They talk about how well they played the 9th hole, or the 19th hole, as the case may be. They just don't get anything done because we don't have any resources.
I have 9 people. I can't build product, really. I don't have a marketing budget. I've got to pull off of the marketing team's marketing budget. I have to have value. The more legs of the stool, then, the better, right? A one-legged or a two-legged stool just isn't going to stand up if you're not working across the entire organization and building value across the entire organization of the partner.
You've got a really rickety foundation for a partnership. Work with the executives, work with those functional areas. Get the flywheel spinning. Get this thing moving so that if the marketing guy happens to cross the sales guy and says, "Have you ever heard of this company?" They say, "Yeah, we worked with them on this account and it was really great!" That's the feedback loop that you want.
Build trust. Can't emphasize this one enough.
Make every interaction you have with a partner a quality interaction. If they don't trust you, that partnership is doomed to failure.
When we work with Microsoft, or we work with Rackspace, we're working with technical people who are talking to the customer. But, they don't really know the partnership, per se. They just know that they like working with us, and that when they do, they get value out of that interaction.
When that happens, they will never work with any other company. They will always recommend us when they find reason to. Those are champions for your organization. You can't have too many champions at a partner. Build them. And the only way to do that is by winning their trust and being their friend.
No surprises. Who likes to be surprised? Nobody in an alliance likes to be surprised. Surprises are really bad. Surprises are, "What do you mean you just went and talked to my boss? And you didn't tell me." "What do you mean you changed the pricing on me without telling me?" Surprises are bad. Stay away from them. Spend a lot of time keeping all the stakeholders on both sides informed.
And then last, something I call the Golden Rule. It's really simple.
Treat your partner like you would like them to treat you.
I learned that when I was little, only it wasn't about partners. But it works. When the tough decisions come, when those moments in time where the partner is pissing you off and your company is rallying around canceling that partnership, just remember to treat them like you would want to be treated in the same situation.
I've seen a lot of partnerships that were doomed to failure turn around and become viable, valuable partnerships because people took a breath and said, "Well, if I were in their shoes, what would I be acting like?" You start to understand what they're doing and why. Then you can figure out how to adjust the partnership to handle that.
Just a reminder: credibility, awareness, revenue, defensibility, and utility are all stages of partnerships. They're additive. You can do all of these, but typically when you're small, you're looking for credibility and awareness. Then you grow into revenue and defensibility, and then you grow into utility. Chances are, if you're not following those five keys, you're probably never going to get to the revenue, defensibility part.
All right, let's talk about life of a partner. I'm not going to spend too long on this, but I know some of the feedback was that you wanted to know about the timeframe that it took to make a valuable partnership. Let me start by saying Engine Yard, first partner, still partnered, and is one of our top 2 or 3 revenue-generating partners. Hundreds of thousands of dollars a year come into New Relic through our relationship with Engine Yard.
Here's how it went. Credibiliy and awareness. We launched New Relic through them. We built an agreement that allowed that distribution of our product as a bundling agreement, and there was a lot of marketing that went along with it because there wasn't an integration at that time. More credibility and awareness in 2009. We finally did the integration work. We built some APIs, we allowed easy deployment. There was more marketing and sales process experimentation. We started experimenting with how best to work with the Engine Yard sales people to develop these leads and to secure paying customers across both companies. Then, we had an amendment to the contract. You're going to see this as a theme.
In 2010, we moved more into revenue and defensibility. Competitors started arriving. They started pressing Engine Yard to move to their product solution, and so we had to provide more and more and more value. We tried to penetrate their account base.
We looked at how many customers do they have? How many are on our free product? How many are on our paid product? What can we do to drive each of those layers to get more and more penetration? More integration work. You're going to see an expert tip in a little while about integration work. And, a contract rewrite. We redid the whole thing.
In 2011, more of the same. A contract rewrite, an amendment. Now we were able to actually access their customers directly, so that we could actually call their customer any time we needed to, to sell. And that boosted the relationship further. But it was a change of the partnering motion. More penetration.
They launched new products. They launched their platform as a service, and so we had to figure out how do we fit in there. Then we started doing these executive-level summits, where we brought all the stakeholders together at a bar, talked about how we could make the partnership better.
Then finally, the last year and a half or so, we've continued more of the same. We've had more subtle efforts around frictionless adoption, more executive summits. We've trained their support organizations, and we've had discussions about how they can fit into some of our new products like New Relic Platform and New Relic Mobile. I do want to point out,
Relationships change over time. That's why we had to rewrite contracts. Be prepared. These things are not static.
So, an alliance exec, a partner exec, and a customer walk into a bar... I'm going to show you a picture of a traditional business development model for accessing customers. Then I'm going to show you a little bit under the covers how New Relic does it, and why it's more effective. It's kind of an expert tip, if you will.
On the left we have New Relic, and the happy alliance exec who works for me. In the middle, we've got the partner, and we've got the functional areas of the partner, the execs, the marketing guy, the product guy, the sales guy. Then, on the right, we have our target. We want those customers. We want to sell to those customers. That's where the revenue is going to come from.
In a traditional model, you have the BD guy, the alliances guy talking to the execs and the various functional areas, and then expecting those senior people in the organization to somehow filter that down to the sales reps in the field, and the sales reps say, "Oh, well, my VP wants me to work with New Relic, so I'm going to introduce him to all my customers, and then the customers would buy." Does that sound like that's going to be very effective? The answer is an unqualified "No."
It turns out, the problem is that execs at the partner, just like at our company, don't really talk on a day-to-day basis with the customer, and so if you're trying to get your product delivered to that customer and try to get in front of that customer, the execs don't have the power ultimately to do that. All the execs can do is not block you from working with the field teams, the sales people in the field, the field technical people. One alliance exec is going to have a real hard time managing all the strategy stuff and all the tactical stuff that goes with partner field teams. But, that's how you get access to the customers, so what do you do?
We built this thing called a partner response center. The partner response center has a couple of jobs. It's got an inbound role, and it's got an outbound role. As I describe it, I think you'll see how it fits many of the 5 keys to partnering. Its outbound role is to find these guys, to find the sales reps, the field technical people, the solutions architects, the support guys, the services guys, the people that are usually ignored by alliances organizations and to get to know them and to get to trust them, and to have them trust us.
Explain who we are. Explain why we get them home on Friday night. Not how they make money, because the support guy probably doesn't make any money from a customer transaction. He has a totally different reason for being. Then, find more of these people, and then leverage that into finding more of these people. Build contacts into advocates and into champions.
At New Relic, we define a contact as somebody we know of, an advocate as somebody who knows of us and, if asked, would speak nicely about us, and a champion as somebody who knows of us and without us doing anything, is out there promoting us â€” usually to the point where they're delivering us leads.
The response center's job is to put anything in place that's necessary to get these field people to be champions for New Relic. They have an inbound task too. Their inbound task and part of building trust is to be there for these people when they're needed.
When a Rackspace lead tech, the guy who runs the infrastructure for their biggest customers calls us and says, "You know, I really need to talk to a salesperson," or, "I really need some training on this feature," or, "I need some slideware so that I can show my customer," or, "I just want to talk about the reorg that's going on in my organization because you tend to know more about who's who in my organization than I do."
That's the PRC â€” the partner response center. It's an amazingly simple concept, and I can't emphasize enough, it's an amazingly powerful concept. You are building a way to create a channel of trust that an alliance exec can't do, and to maintain that and grow it over time. Again, very defensible.
What's more is then you get a groundswell of support from the field talking to the execs. The alliance exec is talking to the execs, the field is talking to the execs, the execs are very happy that the people that they're talking about partnering with are getting such rave reviews from the field. And that creates this virtuous circle which is a flywheel, it reinforces both sides.
The execs are happy, which makes the field people happy, which makes the execs happy, and all of them start promoting your company to the customer base. Get that flywheel spinning in a partnership. If you can do that, it's an unstoppable thing. That's going to be very effective.
All right, moving along. We're going to talk about measurement.
Measure, measure, measure, and when you're done, measure even more.
I'm going to talk about a few of the things that we measure at New Relic based on the stage where we were â€” credibility, awareness, revenue, defensibility, and utility. What do we measure about a partnership to make sure that it's being successful, and to know when we need to make changes? In the credibility and the awareness stage, you're probably early on, you're probably a small organization.
Too many metrics is just going to bog you down, but we found that the most important ones were how many new leads were we generating for the sales team, and how many paying customers did we have at those partners? We just wanted to see those numbers grow. We didn't have any specific goals. We were just doing everything we could to grow those numbers as fast as we could.
When we got into this revenue and defensibility stage in, say, late 2009, early 2010, the metrics shifted and got more complicated. Now we wanted to know from a defensibility standpoint how many of those customers at our partner were using our product because the more that were using our product, the less likely it was that we could be displaced. What's your penetration rate? What were your conversion rates from a revenue perspective?
Just adding leads is not good enough. Now you need to know how are they affecting your sales productivity? Are they converting? I could give 10,000 leads to our sales organization, but if they converted less than 1%, I've just messed up my sales organization. They will spend all their time working on useless leads. At New Relic, we target somewhere between 10 and 12% deploy to paid customer conversion rate for partner and for marketing leads. We want them to be the same. We want the salesperson to have the same experience with a partner lead as they have with a marketing lead.
Average selling price. Conversion rate's great, but what happens if marketing's customers are coming in at $5,000 a year and your customers are coming in at $150 a year? Not so valuable.
And then, how do you measure partner velocity, and the potential of a partnership? It's a little bit of a black art. It's a lot of gut feel, but there are ways to actually chart it so you can explain it to your organization.
And then, finally, where we are now, we look at even more deep things like conversion rates by sales team, so leads that are going to our enterprise team, to our SMB team, to our growth team. Are those converting at different rates and why? What's the cost per deploy?
Those numbers I showed you before, I didn't have those a year and a half ago. We just weren't thinking that way, but as you grow, as budgets become more important and as expenditures become more important, knowing how much your spending to get these leads becomes critical.
And then churn by partner. In the SAS business, it's as easy to install as it is to rip out, so what we want to know is, are partners delivering customers to us that last as long as, or longer than, somebody who just shows up at our website to try our product? Important metric. We're not really there yet. I don't really have that information yet, but that's what I'm working on right now, churn by partner.
I'll give you a couple charts, and these are charts I use in my board write-ups. A piece of advice:
When you're a BD guy, pick a couple of charts that you show each and every time to the board that map to these stages of your company. Really important, because the board doesn't want to see a different type of metric each and every time. Just pick some, and they'll be very appreciative to you.
This was our first half. We had 6,470 sign-ups, not including our Heroku relationship which does about 1,200 sign-ups a month. Those converted into 2,776 deployments of our agent, so basically, 2,776 leads. That's a 43% conversion rate. Marketing converts at about 25%. Big thumbs up for partnerships. Those converted into 338 deals, transactions, net new customers paying us money. That's about 12%.
That's right about where marketing is. I'm not going to tell you right now, but I will tell you what the total ARR or the ASP is, but the ASP is very similar, if not a little bit higher than marketing leads. The only problem with BD leads right now is we're not getting as many as marketing, but then I don't have a multi-million dollar budget to spend. This is a great way of looking at your business from a partnership, a funnel.
The next one, and I believe, personally, that everything can be explained in a 4-quadrant chart, is partner velocity. How do you tell how far your partnerships are progressing over time? I have my team do this quarter over quarter, and what you're seeing is two bubbles for each partner with an arrow that shows the distance, the velocity. Where they were at the start, where they are at the end of the quarter. The axes are revenue, and ubiquity or penetration.
Now, you can use this in a really interesting way. I'm going to talk about that on this slide which is partner potential. If you plot the vector between where the partner is now and where you think they can be on these two axes, you kind of get a handle, apples to apples regardless of the size of the partner, who you should be spending your time and energy on.
Like, if you look at these bottom ones, there is a company AppFog, the bottommost right. Lots of signups, no revenue came out of it. We thought maybe they could do a little more revenue, but we didn't bother spending too much time with them. And it turns out that was a good thing because now they were acquired by some company, and they've kind of disbanded, and the partnership isn't there anymore. Partnerships do disappear over time.
Meanwhile, Engine Yard is at the top and Acquia and Joyent â€” three of our more powerful relationships, those are the green ones. You can see, I've drawn a diagonal line there. If a partnership doesn't have a vector that's going to put it on the other side of that line, I'm not going to spend time on it. I need it to be on the right side of that line for it to be valuable enough for me to distract my company, distract my team, to spend time on it.
Anyway, again it's a little bit of a black art, but it helps you slot the partnerships, helps you decide where you're gong to spend your time and energy and resources. It helps you decide who you're going to promote to your executive team as important. Very important. By the way, you can show this to the board too, so they can get a feel of where this thing is going after all.
All right, expert early stage company hint. Integrative parternships are sticky, I mentioned this before. New Relic actually created an organization called the Business Enablement Engineering organization. Here's what was happening, and I'd like you to just as you look at partnerships, think about this as you grow your company.
We were stealing from our engineering organization because we wanted integrative partnerships. So, every time I wanted to have a really technical discussion, I had to get the senior VP of products on the phone. I had to get our top three engineers to work on partner integrations instead of features.
It's not a very good model, and lots of partnerships get bogged down because of that. Lots of BD organizations find they just never get integration access because they can't steal from the features which always take priority. And they should.
What did we do? We went out and we actually built a secondary engineering organization that reports to each line of business, but it started with BD. I have two engineers and a project manager whose job is: build partner features and build partner integrations, maintain them and maintain partner APIs that allow us to do frictionless adoption of our product through all our partnerships.
One button press, deploy our product, set up the account, integrated console, excellent customer experience, high value for the partner. We can't do that unless we have ongoing engineering resources, but I don't have to fight my product organization for those resources. I can prioritize what's important for my partners. That's huge. I can't emphasize it enough. Expert tip:
If you're the founder think about, at the right time, hiring an engineer for your partners. It's really, really valuable.
The last thing I'm going to show you here is this channel conflict is a good thing. I hear all the time, "I don't want partnerships because then our salespeople are going to be all upset, and there's channel conflict and we don't know what to do." Listen, channel conflict is never bad. Why? Because it means you have channel conflict.
It means you're reaching customers with your partnerships. That's a great thing. What it means, though, is that you have to go and make the right adjustments. What this chart is, is an effort internally â€” again this is an internal chart that you guys are getting to see. It's a way of showing where partners work or salespeople work.
If you look at the top right and the bottom right, those are partnerships. Those are the big blocks of customers over 5,000 customers at one company. On the left-hand side, there's a little partnering, but it's largely our sales organization selling to individual corporate entities.
That one chart saved me hours and hours of complicated discussion with the sales team. Make sure that you can easily define where your partnerships start, and where they stop so that your salespeople know what to work on and you can start to eliminate channel conflict that way. But as a reminder,
It's good to have channel conflict. It means you're growing. If you don't have channel conflict and you have partners, you're probably not doing something quite right.
Just a reminder, choose some goals for your partnership. Structure the partnership to match those goals. Measure, measure, measure, measure, and then measure again, and then go back and adjust, adjust the partnership. You saw that Engine Yard partnership, we changed it 4 times in 4 years, 4 new contracts. We had to adjust. We were making it more efficient, more effective, and it's still a viable, very valuable partnership for us.
I worked for a guy who gave me this analogy:
When you're building your partnering strategy, it's like you're at the bottom of a mountain that you're going to climb. You look at that mountain and you say, "Okay, well I'm going to go to the top." We all know where we're going. And you say, "Okay, I think the best path is up that, like, sandy area, and then around. Then I can work my way up." So you pull yourself up to that first ledge and you look again. Things are different. Be prepared to adjust. You might not want to go that direction. You might suddenly see an easier route.
Partnerships are not static things. What do you do when a partnership goes bad? This was a question that somebody asked. First of all, remember that only a diamond is forever. Seriously, partnerships come and go. They'll all end some time. Here are some signs that trouble might be brewing for your partnership. This doesn't mean end the partnership, but this means you need to take a serious, long hard look at your strategy, at your partner motion, and how that partner is interacting with your company.
Are your metrics all headed south? You know that commercial with the monkeys, where they have the chart going up and the guy comes and he flips it, and now the chart's all going down? It happens.
Number two, it's not a stool if it only has one leg. Get the marketing organization, the product organization, the field organization, and the execs all active in the partnership on both sides. That way, if marketing guy decides that it's not that interesting, you can fall back on the product support and services organization to put pressure on him to maintain that partnership.
Resource black hole. Monitor how many resources you're spending on a partnership. How many people, how much time, and how much engineering time, how many features are you creating. You've got to be careful. You can easily overspend. BD people like to overspend, just saying. Go to market process friction, we talked a lot about that.
Pay for play. Has anyone ever tried to partner with someone who said, "Yeah, you can partner with us. It'll cost you $500,000." Don't go into those partnerships. That is not a good sign. If what they want is you to pay them to be a partner, there's probably a better partner for you out there.
BD lovefest, we talked about. BD people love BD people, and we love to talk to each other. If you have a BD person in your organization, make sure they're talking to other people. And then the resource black hole. Most of all, stay on target.
Partnerships work. They do work. They are hard, they are hard, they are hard. They take time. They take people thinking about them all the time. They take instilling these concepts into your culture that partnerships are a gravitational slingshot for your organization. But if you do, you'll blow up the Death Star. And summary, but we've already summarized.
That's all I had. I'm happy to take questions, thank you.