August 16, 2016
SF Metrics Meetup: OpenTracing, and Druid
Heavybit member company Librato recently hosted the SF Metrics meetup in our San Francisco Clubhouse, where speakers Ben Sigelman from Light...
I'm really excited to give this talk. Not least because, for me, this is the capstone on more than a yearlong project.
My name, as Dana said, is Peter van Hardenberg. I've been at Heroku for six years. Heroku Product, these days. I started the Heroku postgres team. I've done a lot of different things at Heroku including rolling out one of the most significant changes to our pricing we've ever gone through.
I'm going to give this as a tale in five acts. We're going to talk about Heroku's original launch pricing. We're going to talk about how the market evolved between 2009 and 2015, and then we'll get into the actual act of changing pricing in a deployed product.
I'm going to try and walk a bit of a tightrope here. I'm happy to answer questions along the way or at the end. Actually, I suspect Ted will want questions at the end. But I'm going to try and balance specific commentary on what was happening at Heroku with general principles that I hope will be applicable to all of your businesses, whatever stage you're in.
We'll talk about the rollout, and then we'll go through some conclusions.
I want to open with one simple thought.
Pricing is not a last-minute, "What do we charge for this thing?" kind of question.
This is a common misconception, I think. Everybody knows that pricing is hard, but I think people under-appreciate how significant pricing is to your product.
On some level, your customers understand your product through your pricing, right? This is where they ask the question, "What am I buying, and how much does it cost?"
You can play a little loose with the truth on a futures page, but if you misrepresent what your product is on the pricing page, and you say you get 100 licenses, and you get only 80, or whatever that is, you're going to have a big problem.
So, let's take a look at Heroku's pricing page around when the product launched. This was sort of early days. This pricing page was much loved. It looks quite dated now because it was very old. The basic pricing model was well known throughout the industry today.
It was five cents a dyno. Your first one is free. You can add databases for an additional charge, and you can pay for some additional features. And that was great.
This aligned really well to the market in the state it was in at the time.
There was a real focus on simplicity and ease of use. It drove adoption very effectively because we removed any barrier to entry for new users and we really spread through a market.
And we'll talk about why that's important, and who that might be important for, a little later.
But there's some other interesting notes here, which is the pricing model was well suited to market conditions at the time. Ruby was a single-threaded web server, which meant that the scaling model was not to add more threads or worker processes. You added more dynos, lightweight containers to scale, and we handled it with a load balancer.
Apps were hosted in a single process. There's a webpage, model view controller in a database. That classic, three-tier app that's not really state of the art today.
There's APIs, front-end pages and micro services. But at the time, it was a really good fit for a community that was growing quite a bit at the time, and was not well served by anyone else, and that was the Ruby on Rails community.
Back then, we had almost no abuse. Nobody knew about us to care. And besides, we didn't really have many serious customers anyway. We'd just launched, right?
We just introduced pricing. Basically, if anybody gave us any money, that felt like a win. And they did. We grew really well. We had great metrics, everyone was excited, and it was wonderful. Of course, that couldn't last forever. Fortunately, we kept growing.
I'm going to talk about Heroku circa 2015 and what happened along the way. Our product expanded into a number of other languages and other communities. Each of those communities brought new complexity to the product as we tried to expand the edges to serve new communities.
In the beginning, it was pretty easy. Python looks a lot like Ruby. But a good example of how thing started to change for us was that when Node started to come on the scene, all of the sudden this model where you could explain the pricing very easily to a customer, which was, "You need one dyno for each concurrent request you want to serve."
Suddenly, you could serve a thousand concurrent requests with a single dyno. And all of a sudden, one, this meant that, effectively, our price had dropped a thousandfold for that audience.
There's a floor on this function, but the scaling model changed radically. And also customer comprehensibility was degraded by this. How many dynos do I need? Well, it depends. And it depends is the death of all pricing conversations, but it's really hard to get away from.
There were a number of other changes. Microservices became a theme of how people developed software. We started to split our software into smaller components. Not to say no one was doing this before, but within the community we were part of, a greater consciousness of this way of de-factoring and out-factoring software became more and more common.
This led to these sprawling architectures on Heroku where what was originally an app suddenly was one of many components in an app.
And so this became ever more complex to manage and ever more complicated for us to understand, price and serve. In addition, as our user-base grew, people brought us harder problems.
This is great, this is awesome, it's what we always wanted, but the amount of RAM you need to serve up a circa-2009 Ruby on Rails ERB script versus running a headless phantom testing framework, or doing any kind of heavy-duty transcoding or rendering in the background, all of the sudden, these workloads produce really new challenges for us.
So we started to respond in terms of adding new feautures and elaborating the product. Abuse, of course, became more and more widespread as more and more people found us and more and more people realized what they could and could not get away with.
As we grew in the market, we also began to see friction from this model in our enterprise sales. I'll talk a little more about that because it's important, even though the focus here is really about change in our self-serve model.
Our business strategy was evolving. But the core of it was, first, keep that onboarding experience simple. Whatever language you come from, it should feel native. If you're new to the platform, you should be able to start quickly. If you're experienced with the platform, you should be able to work effortlessly.
At the same time, we realized that in order to grow our business and to continue to keep the trust of even individual developers running their hobby projects, we needed to support ever greater production workloads with ever more sophisticated needs.
If I'm deploying my own personal blog, I don't care that there's a five-second downtime as the dynos cycle out and the new code spins up. But for people who are running applications that could take 60 seconds to boot, that want to deploy 20 times a day, that had users using their product around the clock, this became difficult to serve, and this was an ongoing thing.
The problem, of course, is as you serve that use case, you degrade and complicate the use case for the individual. So that was an increasing tension. We've always viewed these small customers, the individual developers, as our strategic advocates in our sales model.
We'll talk about that a little more later. And it's from a strategic perspective, we wanted to create a clearer value access to our pricing. The Heroku model was about the number of dynos you ran. And that was really the decision that you as the buyer made. We'll talk a little about the value axis later.
One takeaway from this evolution of market I want you to think about is that no matter how simple your pricing model may be now, the longer you survive, the more simple fixes you put in. You're just creating yourself a mess.
It's inevitable, but adding simple stuff will make your simple stuff complicated. So, as of 2015, our pricing looked like this. Dynos were still nominally five cents an hour. Except now there was a whole bunch of different dyno sizes.
There are five-cents-an-hour dynos, 10-cents-an-hour dynos and 80-cents-an-hour dynos, which cost 16 times as much but had 12 times the RAM, but were on dedicated hardware. The messaging was ever increasingly more difficult.
There was a single axis. Well, actually, there were two axes, the number in the type now. And so it was getting harder and harder for customers to model what they needed in their head. If you were well adopted into the platform, you could figure it out.
On top of this now there were worker processes along with the web processes, one-off and interactive, cron jobs, scheduled jobs. All this stuff added up to an ever more complicated model, and we moved away from the one dyno free to a 750 hours per month of credits.
People could run their one-off jobs and lightweight things without making that free. Because some people of course would game that and then try and run hundreds of dynos worth of processes without paying, if there wasn't some cost to that.
Our databases as well expanded: hobby, standard, premium and enterprise. We moved beyond just postgres to postgres and redis. We had all these different sizes. And then, of course, there's all the legacy stuff where your existing customer base is on old plans with different names. It was an enterprise business model with a whole different set of skews, with different discounting matrices, different sales motions and different sales people.
We were selling "Technical Account Managers," "Premium Support," custom terms for certain customers. And then, of course, we have the hundred add-ons all with their own business model that we've tried to shoehorn into some kind of relative sanity, but each with their own play on that.
The end result was just really challenging. The upshot of all this was that if you were a sufficiently motivated user, you could get 75,000 dyno hours per month free and the whole time be complaining about how Heroku is too expensive. This is craziness, right?
The amount that we would see people consuming, large amounts of our product, joyfully every day, while complaining about how expensive we were, and never giving us a dollar, was really distressing.
We're going to come to how we started to pick this apart and attack this, in a minute. I'm going to take a little sidebar, and we're going to talk about value axis. The value axis is what you're selling.
When you go to your pricing page, presumably you have some sizes, numbers or licenses or whatever, right? You've got a little bit. Some. Maybe a lot? And as the price goes up, you get more. What are you selling, exactly? And what are they buying?
The classical Heroku value axis was based on gigabytes of RAM. You got one 1X dyno, you got half a gig of RAM. You got four 1X dynos, you got two gigs of RAM. If you've got eight 2X dynos, that's eight gigs of RAM. And the price goes up.
What people concluded, logically, was that they bought RAM. And so, they shopped around, just like you would. They went and they said, "I can get RAM cheaper from DigitalOcean." This is like, the most painful thing to see for a product. The idea that it's a product that's in a completely different category, with a totally different user model, offering different features. They're wildly different products admittedly solving similar problems.
But to have that kind of comparison, where no one's even thinking about what the product offers beyond how much they pay per gig of RAM, was really tough to see out there. And it wasn't universal by any means, but it was emblematic of the challenges and how our business model had become strained as the market had shifted from where it was in '09 to today.
Now obviously, if your company is earlier stage, you may not have accumulated this much skew. Probably, you already have a significant amount of skew, because pricing is always hard, and no one gets it right. You've always introduced problems the minute you put your first pricing out there. That's fine, that's normal. But it is a pain.
What we wanted to do was move to a model where, essentially, the value axis became two-dimensional.
The first choice we wanted you to make as a user, as a customer, was, "What do I value?" and not "How much do I need?" We wanted to segment our users into, " I care about my personal productivity," "I want the ability to scale," or "I care about compliance, trust and the more ephemeral performance," and other kinds of things.
Maybe we accomplished that, and maybe we didn't. But whatever that axis is for your business, it's important to think about how a user will view and experience this. And when they go away, and they look at your pricing page, when they go and look at your competition, what mental model are they building to compare? What do they think they're buying and what do they think they're getting?
Okay, enough of that. Let's talk about, now we've identified these problems, how we attack them. This is totally vague advice, but I'm going to try and actually make it a little more concrete as we go.
My recommendation is this: work from first principles.
Okay? You're not Heroku. You're not Stripe. Actually, a couple of you are Stripe. You're not GitHub. You're not AWS. You're not RedHat. You're definitely not Slack. You need to think withthat brain you have or those brains you've brought, together, and think, "What do we really care about?"
Don't just cargo-cult a business model off us or anybody else. I mean, you should look for references, but the point is to understand why they work and what they're accomplishing, not just to parrot them.
Let's get into some specific examples. Our strategy, as a platform, is to acquire a lot of individual users. We think this is important for Heroku because we need the world to use Heroku, so that the world will align around Heroku, so that all these individual developers will buy small Heroku accounts.
They'll spend just a little bit of money with us. They'll feel invested in Heroku. We know that they like Heroku because they're paying us. The most sacred act of capitalism is to open your wallet.
If we can get even just a dollar from a developer. It's not about how much. It's about the fact that they're willing to pay something. And that indicates that this is more than just a dalliance. Or you use it because it's free but that they recognize at least some value. We think this is very important from an emotional perspective.
Then once they're invested, they're adopted and they're loving the product, then they will bring that product to work with them. Or they'll go to work, and they'll hate their job because it's terrible, as most jobs are. But they will want to use Heroku because everything else sucks.
And so we see this all the time, right? People go rogue. They deploy Heroku inside big companies because they have a problem to solve, and they just can't take dealing with whatever outdated, archaic spinning up an instance and managing it yourself. Or if you're really unlucky, calling IT and asking them to stack a server for you somewhere,whatever nightmare scenario they're in.
The idea is that these people become our advocates and our champions.
Whether we're coming in from a traditional enterprise engagement model at that business where we are engaging through a CIO and then selling down, or whether these people are championing us up and we're opportunistically collecting and harvesting that, and encouraging and developing those leads, the idea, either way, is that these people are important.
This is a lot like how GitHub operates, or DropBox or Google Apps or Slack, or a lot of other businesses. There are really important differences between each of these businesses and their models.
But as a general shape, this is freemium, right? Now, freemium works if you have a large user base that's important and valuable to you. Because it helps you to sell up. So you need a product that's useful to individuals for this to work. If your product is only useful to businesses, it's not as valuable.
How do you engage that audience? You need to work from these principles for your own product. We know we have a problem. We're going to try and make a change. We set out some sort of ground rules before we started, as best we could.
First, we knew we wanted to make one change, not trickle out 20 different changes over a long period. We wanted to say, "It was this, now it's this." You can stop thinking about it. You have to digest this, and then we'll move on.
Second, there's a certain amount of inertia in any system, and we wanted to try and constrain the scope of what we were changing as narrowly as possible and really execute thoughtfully in one space. We had tried a number of previous projects. There are some fantastic people at Heroku who spent a lot of time and energy on those projects, and ultimately they didn't ship, and there are a variety of reasons for that.
One of the goals in trying to approach this was to cut the scope down more than those projects had. Some of those had tried to create bundles or packaging on top of other pricing changes. We really wanted to just narrow the scope down. We have significant revenue. It was important to the business that that continues, obviously.
As we were working, what we knew was that we were really most concerned about this self-serve audience. We were selling reasonably well into enterprise and businesses, but where we were really having trouble was getting the everyday developer to adopt and consume. I mean, they were, but they were complaining, and we wanted that to get better.
Here is an example of the goals that we set out for this project. This is an early revision. I found this in some Google Drive. At a high level, I recommend that as you work on a project like this. You separate the goals from the implementation and make that explicit, no matter whether you're two people or 250 people. This is a very useful tool.
In this case, what we have is a number of things that we're setting out to do. We want to put a compelling price point for individual developers. There's something in there about encouraging best practices in prod, and making everybody use two dynos. While that was a nice idea, we ended up cutting that. It was too complicated and hard to explain. And honestly, it seemed nice, but it turned out to not be that important.
The other thing that we did was kept a list of things we were specifically not fixing. Heroku's SSL pricing is a thing people complain about, but that was a separate product domain. It was a separate set of problems. Not unimportant, but we needed to draw a box around something we could ship.
There's some other examples in there as well. And this is what, at some point, our product proposal looked like. Over time it got simpler and simpler before it finally shipped, which is good. Then we went through a bunch of iterations on this, but I just wanted to give you a glimpse of what that might look like.
Throughout the lifetime of the product, there was always basically two things happening. There was the goal, and there was the implementation. People would come to me, and they'd say, "You should solve X. I really hate it how you have to..." And I'd go, "Hold on. You've seen the deck, and here it is. Let's talk about it."
"So, do we have the wrong goal, or do we have the wrong implementation? Why is it this price? It should be that price. These are our goals, do you deny that the proposal implements them? Are we breaking something we haven't documented, or are we trying to do something we don't need to do? Can we pick a simpler solution?"
This was a constant feedback loop that was very actionable, right? One of the really challenging things when you're looking at pricing, if anyone in this room has done so, is that someone comes with an idea, and you realize that you're fixing a problem, and that's great, but what you've forgotten is that you had that idea three weeks ago, and this actually turns out to break this other thing.
Or it looks really good, and then you spend a bunch of time and you make some changes and you work towards it. And then you go back later, and you go, "Oh right, but now the individual developer has a speed bump on the pricing curve. Having those things explicit, separate, and documented is a very valuable tool, please do that.
All right. Let's talk about actually shipping a change like this. No plan survives contact with the user base. The first stage, even before we talked to anyone, was we came up with a hypothesis. We modeled it very pessimistically. If you are like us and in the SaaS business, oh my god.
I do not know how anyone runs businesses without this much data. It's so nice. I could literally, for each and every customer, say, "If I change the price like this, what exactly will happen to their bill?" and, "What percentage of users will see what kinds of price changes?" and, "What would our total revenue impact be, assuming that my model is any good, which if course it was not."
But it was a pessimistic model, and the actual reality was better. You do the kind of modeling that's useful, especially as you iterate on things. Because you can constantly gut check and go back in truth against your model.
The first contact with external people was a set of informal interviews with friendly users and friends. We had actually fully fleshed out mocks of all the pricing pages. We did not share them. This was good advice from Paul Kopacki, and I'll give a shoutout to Vikram Rhana, on our product marketing team, who was a really close partner through all of this.
I mean, pretty much everybody in the company worked on this before it was all done. But Vik and I spent the most time maintaining the goals documents.We went out and talked to customers. And we had these mocks, but we didn't share them, because if you share them, it feels very real.
Suddenly, people are going to have a different kind of emotional reaction, and at that stage, we wanted a gut check. Like, "Hey, does this make sense? We're thinking about a thing." Internally we knew that we were very committed to making a change. But we wanted to get the loosey-goosey first look.
And so we talked to our, basically, friendlies, and that was really helpful. It was a great way to practice talking about the change. It was a great way to hear people's initial responses. It was good for gut check. Where it was bad, of course, was that these people are already deeply invested in the product.
If they're talking to you, an insider in the company, they already feel special and kind of invested. And you get a lot of benefit of the doubt.
You're not going to find out what the rage on Hacker News is going to look like by talking to someone who is spending $20,000 a month on your product. Nor who you go to drinks with every month or so.
Haters are my favorite, though. I love the haters.
Don't get me wrong. Nerd rage on the internet is a powerful force for good in the world. If you know how to harness it, right? If someone is telling you your product does something it doesn't do, or doesn't do something that it does do, they're not wrong. Don't argue with them. You're wrong, right?
The burden is on you, the person putting a thing in the world, to make sure people get it. You've mis-positioned. You've miscommunicated. You've mis-documented. You didn't ship. It doesn't work. Figure it out.
The nice people won't tell you, right? They'll muddle along. Haters are just people with that frustration trigger on a hair trigger. They're wonderful for many things. I mean, obviously it can be draining, and you need to learn to compartmentalize your rage. But it's a really great thing to have those people out there and to figure out how to use that.
We implemented an alpha, and we made some changes to our initial proposal based on surveys and informal feedback. We eliminated some features, moved things between tiers. And so we ran a small pilot, it was invitation only.
At this point, the model had given us enough data that we could predict, because it's a pricing change, the outcome is fairly deterministic. People's prices go up, or people's price stays the same. There's a mode of qualities to pricing. I hope you've watched Michael Dering's wonderful talk on the topic.
But you know, we could say, okay, let's put this in front of a hundred people whose price might go up, and let's put this in front of a hundred people who are small businesses, whose price will go down. Let's see if it goes down enough that they feel good.
We ran a bunch of cohorts, and we looked at what happened. And of course it leaked. The wonderful thing about a leak, which was always inevitable, was that it gave us optionality that if we were really wrong, it was much easier for us to step back from it. Because it's not a public beta.
We could say, "Oh no, this was just an alpha. We were running a test," and quietly put it in the dustbin and start over from scratch. Fortunately, that didn't happen. That would've been very sad. But it kept the cost of pivots down, and it maintained the option for us to change strategy very late in the game.
We got the Hacker News thread, people took screenshots of the new pricing page, and people wrote very thoughtful blog posts about what they thought. And the message was exactly what we'd hoped, which was, if you're a paying customer, this is great news. And if you're not a paying customer, it's not going to be as much of a free ride anymore, but there are pricing options that are pretty fair.
Not everybody was happy, but by in large, we thought it was a fair response, and that was encouraging. We had a few things wrong. So when we went into beta, we actually made some more significant changes. We simplified things even further. And one of the big changes, just as a shoutout to a very specific detail, in an early version of the project, we'd blocked adding domain names on free applications, custom domains.
Seems reasonable, right? You have to buy a domain, so you should pay your hosting provider. If you've already spent money on this project, why would you not pay your hosting provider? What we found, actually from the alpha, where we started to get response, and what become clear sort of in retrospect, was that, well actually, hobbyists will often buy domains before they write a line of code.
You're out, you have a funny idea, you register a domain name. The next day you sit down, and maybe you start programming.
This was the kind of example where testing in these growing concentric circles, exposure and commitment to the product, gave us qualitatively better feedback over time.
The beta really was opt-in for everyone. New users could opt out, but they saw the new pricing by default. And this started to give us confidence that you're pricing has two dimensions: there's what you put on the pricing page and how people experience it, and then there's how people perceive it on the pricing page and how people experience it when that bill hits their credit card.
You're running a ton of applications on our platform for free versus you're complaining about how expensive we are. The perception of our pricing was not well pitched for that audience, even if the reality was they were paying nothing or very little and getting a lot of value from it. So our goal was to align those things.
The beta was our mechanism put it in front of all new users by default to see how they behave. That gave us the comfort to keep moving forward further. Finally, we went GA. Again, we made a few more tweaks.
Assailant One was an earlier version. You could run your apps for free for 12 hours a day before they needed to sleep. We ended up extending it to 16 hours. Maybe it was 18 hours, because people said, "Oh, but if I have a long school project...," "Oh, people on the East Coast, on the West Coast."
Again, we have all the data, so we could see that actually making it 18 hours meant basically nobody's apps would be affected unless they were already running 24 hours a day, constantly, anyway. So that further targeted the change, and it defanged some complaints, and people were more positive as a result.
Our goal was never to extract revenue from this audience, right? It was to get them over that penny gap. We wanted to be as generous and friendly as we could in this process, because we're not trying to make money with these guys. Again, this is our strategy.
We're not going to make our big numbers off seven-dollar-a-month dynos. We just want you to spend a little bit, because we want you to show that you get value.
Ultimately, GA, we launched it. It became the new default. Every new app and every new user only had the new pricing. Existing users still had the option to migrate at their preference. At that point, all the details were finalized. Of course there's an asterisk, because I was sitting in a meeting the other day, and some of the details are still being negotiated. But that's the nature of work.
For us, the right strategy around this, because we were aiming to execute a pricing change but we also wanted to give people the flexibility to engage with us at their comfort, was we said, "Here's the new pricing. We encourage you to move soon because you'll save money, but we're going to give you time to think this through, read the materials and come to your own conclusions. You can migrate when you're ready."
Ultimately, we have sort of a cut-off date, but we're trying to run campaigns to move as many people as we can before that, and when that comes, we'll do the hard work of reaching out and moving people over and so on. This also gives people who may not have had the benefit of the change work out for them lots of time to think about what they want to do and to get more comfortable with what's going on.
A good pro tip if you are communicating with users about a pricing change: the fact that everybody else is getting a pricing cut does not make them feel better about their own change in pricing.
Heroku's pricing went GA in August, the new dynos. So we have enough data now that I can say a few things without giving away too much of the goods. We saw a huge increase in new customers per month. This was a top-line goal for the project. We wanted more people to pay us a little bit.
We've seen dramatic increase. It's now been long enough that the initial spike, we think, of people being startled by the news has passed. At this point we think people move over because they want to pay us. They feel it's the right thing. The messaging's there. That's communicating as it is.
And we have genuinely differentiated price points for individuals and hobbyists. One of my favorite things about this whole change is we've separated the world of dynos into two classes: hobbyist dynos and professional dynos. They have different features.
If you're a business, you can use hobbyist dynos. And if you're just a student, you can use professional dynos. But if you're a business, and you're using hobby dynos? You just feel, maybe, a little bit embarrassed about that. And it's just those little cues all through the product.
This is the emotive nature of pricing. It's not about saying you absolutely can't live without this feature. We want people to use what feels right for them.
But we're sending these signals about how we think about and how we intend to support and evolve those products over time and who we're targeting with those products. We do now have that value axis I talked about. If you go to the Heroku pricing page today with the piano keys, you'll see.
One cool ancillary benefit is that we've managed to isolate where we put product complexity into this professional space. We took a lot of the features that were previously keyed off weird things, like if you have two dynos, or a professional dyno, then you get access to this.
And we said, "Professional dynos have this, hobby dynos don't." And this is nice now, because the pool of people who don't really need to think about these features aren't confronted with understanding them. So we lower the cognitive burden on consumers of the product.
It wasn't all sunshine and roses. It took a very long time. It was in active development for more than a year. It was the third attempt on the project. One of the PMs made a joke about castles sinking into the swamp, from Monty Python.
It was really painful for us to implement this. Pricing projects are always painful. They have a very cross-cutting nature, essentially. So, it involved bringing in people from all over the org on a rotating basis.
This month we need help from this dashboard team, and next month we need to talk to the metrics people to make sure that their product maps in it the right way. It was very difficult because it meant you were often starting from scratch.
You have to re-justify and re-explain these things if you want people to really be bought in.
That was very tiring. It would have been nice to put together a team that could do this, but the nature of our company and the nature of our product is that there's a lot of different code bases. Andpeople don't necessarily have the skills to move everywhere.
We did successfully reduce that free dyno gaming. But of course, within a few days, new games had begun. And I won't do them the justice of describing them here, but people are very clever, I will give them that.
Let's talk about some conclusions. Pricing defines your product. Pricing defines your product. People relate to your product through how they pay. That is important. People who buy your product relate differently than people who don't.
Have an explicit strategy. Don't just do what Stripe is doing. It may work for them, it may work for you. You will not know why or how. And if it stops working, you won't be able to fix it.
Think about who your audience is. Think about what they need. Think about how they buy. If you don't know, guess, test and learn. Just like with anything else about your product.
Yes, it's hard. Yes, it's a black art. Yes, it's emotive and technical and economical and everything else. But you have to have a strategy and be explicit about it. Design your solution. Design it, iterate on it, go back again, know what you care about and know what your users care about.
Think about what's important to your organization. Support with data. Get lots of feedback, model your changes, and iterate. Of course, your pricing will never be perfect. Pricing is always terrible. You will never be happy. You work at a startup.
I have a brief anecdote if we are still under time. I think I have a moment. I'm going to take it. When I started this project, people told me dynos cost $35 a month, and that was too much. We ran a bunch of surveys to find out what they should cost, what they thought they should cost, and got a lot of different numbers. Eventually, we settled on seven and 25.
So, everyone always told me dynos cost $35 a month. When we shipped the new pricing, suddenly I hear they cost $84 a year. My conclusion is developers will multiply numbers once.
I don't know why this happened, but no one ever quoted me an annual price for dynos before we shipped the new pricing. Now that the pricing is quoted monthly, I hear about annual prices very frequently. Don't know why. There it is.
All right. I'm going to open the floor for questions now. I'm happy to be more specific, or less specific, about anything you might want to know. There's so many different facets to this. I've really only scraped the surface of one small piece of self-serve pricing.
We didn't talk about positioning or broader go-to-market options. We didn't talk about Slack hardly at all. We didn't talk much about enterprise sales. License software is a whole different model. If there are only four of you, and you're in charge, you can just do whatever you want. That's not a question. But you still should think this way, I think, regardless.
In our case, basically, our customers had price cuts. We spent so much energy arguing about this and talking through every scenario and modeling financial impacts, likely media, stories and whatever else. Ultimately, we concluded that we could be patient, and we could be generous, and those would be our motivating goals.
We felt people would be the most comfortable if we gave them the most freedom to move when they were comfortable. I said comfortable too many times there. We wanted to let people do this at their own pace as much as possible, and then we're running campaigns to move them.
There were people who were angry that we only gave them, I think, eight months to migrate, six months after rumors of changes started going around. One well-known internet personality was so incensed that he was being asked to pay seven dollars a month that he moved off the platform, started running it himself and wrote a very angry blog post all about this.
I'm not going to say anything in particular about his account, I will just note that when people asked him how much money he saved by moving off the platform, he was conspicuously silent.
Yeah, you know, it's debt. You're accumulating code paths that aren't run, for fewer and fewer people. You're building up. On the postgres side, now we have, I think, 50 defunct plans that people are still buying.
It creates problems, because you forget that you're trying to ship some new product, and you're like, "Oh, we'll put it on these plans," and you just forget these other people are still out there.
We changed the plan because people need different things. Because this was broadly a price cut for almost everyone. It was very easy to basically put an end date on it.
I didn't talk about this, thank you for reminding me. The Uniqlo CEO famously had a map of New York, with Apple Store locations on it, when he was looking for where to put the Uniqlo stores. Someone said, "Why do you care about Apple?" And he said, "Well, the customer only has one wallet."
And so for us, it's very easy when you're working on pricing changes to stay down in the micro, and think like, "This is going to do this or do that. Looking at the macro across a whole account for us, people buy a lot of different dynos and databases and everything else. Remembering this either amplifies or diminishes the total impact.
For us, because we let people opt in or out per application, people who have a diverse portfolio, they'll move all the apps that save the most money right away. And even though we've modeled their total account will go down in spend, the apps that they can run for free without paying that extra seven dollars, they're just going to leave on the old pricing indefinitely, because why wouldn't you?
Ultimately, it was an engineering and business model thing. Like, do it gradually. Be generous with it. But we wanted to put the old thing away and not have to deal with extra code paths, API and extra fields in the billing models, and the monthly revenue reporting had to have a separate accounting system. So, pay down debt.
We've spent a ton of time on our invoices. Like, a ton. They're interactive with graphs. I don't know if you're a Heroku customer, but you can see over the month how many dynos and which categories and what the names were of your one-offs.
I think it's very important. I don't know if it really matters as much as we've invested in it. But I think that getting the details right is really important in anything you do, and that's a holistic customer experience.
I think it's worth doing well, if you do anything. But that's kind of a smug non-answer. I haven't run any A-B tests to give people ugly invoices and nice invoices and see if that changes their attrition, but we try to make them nice and we listen when people complain if they're confusing.
What problem are you trying to solve? If you are selling to a business user rather than to an individual developer at home, a free trial is a natural and common thing. And it's something they're comfortable with. It gives you a scope of engagement and an opportunity to drive a conversion. Then you can close that lead if it doesn't work.
Your pricing in that context is part of a larger sales motion, which is, how do you prospect? How do you generate leads? How do you close customers? There's all kinds of different models from very low touch, the kind of self-serve.
We have a Pachinko machine for email, to the very high touch. We only land customers by getting an introduction to the CIO, flying out there, doing meetings, sales enablement and sales engineering. So, it depends.
Do you have a particular problem you're trying to solve with a free trial? Yep, so, if they bought the hardware, why are you giving them a free trial? Seems like without the software, the hardware's probably useless. So, you're a hardware company with premium services you sell. You're a software company that has a hardware leader. Great. I mean, that's a very reasonable market strategy.
It seems to me, just based on what you're saying, with no deep reflection or great understanding of your business model or specific problems that you're facing, that what you described suggests you should have some free tier. Because you have people who have already opened their wallet. They're probably cheap to support, I imagine, if they're at a small scale. And you should have a separate professional tier.
The hobbyist, trial tier should probably be resticted in features and functionality and maybe a free tier for the professional, commercial-grade, production-grade, whatever you call it or however you think about that.Seems not unreasonable, right? Like, who's buying? What are they holding at the time? How do they think about it? How long do they need to find out whether it's valuable?
If for you, the time when people pay is when they go to production, then not setting up a time-boxed engagement around that may be better, because projects often take indeterminate amounts of time to reach production. You may want to give people a taste of what that might look like and scope it that way. Again, it depends, but it sounds like it may be a useful tool sometimes. It introduces product complexity. Do you have the sales support to actually harvest those leads? It depends.
Your customers are not the same people as your engineers because you have a lot more customers than engineers, I hope. And they are much more diverse in their needs and interests than your engineering team will generally appreciate, or your product team, or your management or whatever. That's just normal.
We're bad at empathy and scale of humans. How do you avoid pissing people off? Do right by your customers. Right? If you are part of a team that ships a product, you need to have reasonable and fair limits in place around what you're giving away for free. And not everybody on this project was always fully bought in to every dimension of it.
Having been with the organization for a very long time and in a lot of roles, I got a fair amount of benefit of the doubt from a lot of different people when they may not have agreed with me. Constant and regular investment in communication, right?
Again, it's sort of a bogus answer, but it's true. When I heard someone didn't like what we were doing, I would say, "I'm going to clear two hours. I'm going to come. I'm going to bring the deck. I'm going to sit with you at a whiteboard. We're going to talk about this. We're going to take as much time. You can tell me anything you don't like about it."
We'll look at the goals of the project. We'll talk about why it's important. We'll put the piece you're doing in context. This is really hard because it was cross-cutting across a whole organization. And at the end of the day, not everybody bought in.
But I'll tell you, I'm not going to complain about any particular teams, but there's one team that's having a really hard time with the project. They were kind of unhappy about the change we were making and the work that we were doing. I don't know if it was ever totally resolved or eliminated, but in the beginning, the work that they did was user-hostile. Because they thought that's what I wanted.
They'd seen "it's a pricing change," and "we're changing what we're giving away for free. We're giving away less." So they weren't approaching from a position of empathy for the customer. And so they would do things in weird, unnatural and almost punitive ways because they thought someone's twisting our arm and making us do this gross thing.
That was the opposite of what we needed and the opposite of what we wanted, which was to craft great experiences and to be fair about what are we giving you for free and what are you buying. You know, the end result is, "Is the specific mechanism around the differentiation between free and paid at that tier perfect?" Probably not. We can probably do better.
But we looked at it for years and never found anything better. So, ultimately, we shipped something. But what we saw over time with that group is that when I kept coming back to them, being like, "Hey, how can we make this friendlier to the user, and maybe we should be a little more thoughtful about the experience here."
Eventually it started to dawn on them that although we were changing things, we were trying to be really good to the users about it. At least I'd like to feel that way. The end result was, I think, pretty good, pretty fair and pretty generous. The early iterations were not so.
How do I avoid engineering attrition? If you allow people in your organization to hold the whole company's business hostage because they're not comfortable with your business model, you can never survive. You should invest deeply in building a team and building that empathy, earning the trust, in communicating about the goals and why they're important. But you can't let yourself be held hostage. Ultimately someone has to make a call and you have to move forward.
Maybe they're going to make the call, or maybe you're going to make the call, but not everybody's going to make every call, and at the end of the day, you have to pick something and execute. We lost a couple of engineers that worked on the project. I think it may have been a contributing factor. I don't think it was the sole factor.
It's a tough thing, but at the end of the day, you have to do right by your customer and your business, and roughly in that order. I think, ultimately, it's a better product for it. And I think that most people were happy with the outcome, both among customers and the staff. It's tough.
Cutting price is usually a popular choice. The metrics we were tracking, we collected feedback at several stages throughout the gradual rollout to ever larger audiences. We looked at user behavior. NPS is interesting, but I care a lot more about action.
So, we looked at attrition and adoption across different cohorts and still are tracking it very closely. We're happy with our new-user signup numbers, and we're extraordinarily happy with our new customer numbers. We've been very pleasantly surprised with overall revenue growth as a result of this.
The theory was, basically, our pricing and positioning were wrong, and evolving that would make it easier for more people to adopt. And that appears to be the case. It's really hard, right?
There's so much noise in all the signal. It's really easy for someone who spent a year on something to be like, "I guess it worked because we didn't go out of business." The net of it is, the metrics we tracked performed a little better than we'd hoped, and that's good. Those were some of the numbers.
We had to rewrite a lot of dev center. Our pricing showed up in a lot of places. We changed the names of dynos. They're kind of gross now. It's like standard 1X, and it should just be 1X. But because there's still the 1X for legacy reasons, I can't make it a synonym until the old one's gone, speaking of reasons to retire old things.
Ultimately, this project took us a year. At Heroku, we make PMs responsible for documentation, and our dev center team are editorial. So they could come back and say, "You need to rewrite this." Me and other people working on the project had the document as we went, and we tracked down the docs that needed editing. I mean, it was sort of a natural part of the process because people would ask you questions. We spent a lot of time working with our marketing team, rebuilding pricing pages and feature pages.
There's not enough time to talk about it in 30 minutes, but it's part of really a re-positioning of what dynos were and how we wanted people to think about them. So, rebuilding dub-dub-dub and our marketing content was a huge part of the project.
I mean, not everything when you're working on pricing is necessarily a highly enjoyable patch to write. We need to rename this thing in the database. Can we just not do this? Can we keep the name? No? But we have other things that are important to work on.
I'm like, "I know, but all of the other work, and let me show you all the other work and all the other pieces that have been moving for a year now." You're arguing that maybe this piece of work isn't important. And individually, it's not. But it's part of this big network of things where if we do these other changes but we don't follow through on this piece, the whole web comes undone.
Or maybe it was fine, and we cut scope. It's not about just bludgeoning people into doing what you want. But usually that helped. Usually that worked. Often it would lead to improvements in the plan, changes in the rollout strategy or things like that as well.
We didn't do A-B price testing. This comes back to "the pricing is emotive." What we were looking for was that things would not get worse. But this was not about tuning an existing model and testing, "Keep everything the same, but make dynos $32 or $37." Keep everything the same, but test the word production against the word professional.
Blow up the entire pricing page, redesign the dashboard interface. We were moving to a new locale. There's definitely room for optimization within this space still. In particular, I get emails every now and again like, "How do I upgrade to 'hobby'?" If you knew how long I tried to make this easy.
But when you're making a big jump like that in fitness space, A-B is going to be really messy and noisy. Besides, you're doing it for a strategy reason rather than trying to tune something you have. You want to make sure that you're not tanking things.
But really, A-B testing is not a great tool in that space. And you could argue with the beta, where we sort of exposed all new customers to it. Pricing in particular, right? You can have this price or you can have this price. It's very hard to get those things pulled apart the way we sell where it's at a per application basis.
Would people broadly, on their whole account, choose one model or the other? Well, we're not asking them to make that choice right now. How often do they engage with the property in that way? Do we want to write all the software it would take to explain that decision to them? Ultimately, we just said, "Nah, this doesn't seem like the right tool for us right now."
Deal sizing, sales enablement and discount matrixes and compensation for salespeople and how that gets split out. There's a whole other important, interesting, separate can of worms. But again, in the spirit of, "Don't cargo-cult your business model," there are many reasons why you shouldn't put enterprise prices transparently on your business page.
These people frequently are tasked with and skilled at negotiation. And so they want to have a negotiation with someone, because that's what their job is, and they're used to that. So by not giving them that option, they're uncomfortable with that sales process. And also, businesses, they want to see success, they want to talk to your reference customers. Depends on your market and your price point and many other things. But don't take my word for it, think about how it applies in your business.
I'm very comfortable with the idea of not listing your enterprise price transparently. In fact, for some products, you don't have to list any price transparently.
I think people relate to products they pay for very differently from products they don't. There is something important about having a good way for people to buy the product and then how those people behave.
I think in a year or so, our old data was very hard to tease apart: what, when, why and how people were paying for things. And I think it'll be much clearer now.
I don't care about that money. I mean, I care. I want to run that business not at a huge loss. It's a sustainable position. You know the old saying, "If you're not the customer, you're the product." I wanted people to have a way as individuals to buy Heroku that felt comfortable to them, where they felt like they weren't part of a marketing program. That seven dollars. We're not making a ton of money.
We could sell a lot of them, and the money's nice. It's more than the product costs. It's a sustainable position. If everybody only bought those, we would still make money. But it's not about the money, it's about offering something to that audience so that they can be engaged financially in the business. You don't want to be Geocities, you know?
I'm pretty good at postgres. I've run a postgres product for a number of years, so my SQL-fu is pretty strong. I wrote very elaborate database queries that basically brought together a ton of different data about and around customer invoices and run rates.
Then I could model what I predicted their behavior would be, and then what their old bill would be, what their new bill would be. And then I put that into a data clip, which is a Heroku lightweight BI tool. Then I loaded that into a Google spreadsheet so that then I could take all that data, pivot it, crunch it, you know, type in different constants easily and share that within the organization.
So that's tactically the answer. I also spent a lot of time writing queries and then saying, "If it was this much, how many users would go up in price and how many would go down?" Okay, if we make it this many hours per day, how many free customers would have a bad experience, would have to change their behavior or would have to remove Pingdom?
Okay, now how many people are using Pingdom? And just crunching and crunching away on all that. It wasn't perfect. I think that for such a major change, it was pretty well accepted. And I think a lot of people saw their bill go down, which always helps a lot. All right. Well, thank you all for your thoughtful questions, and I'll be around. (audience applauds)