June 14, 2016
Beta Testing for Early Developer Teams
UX designer and former Treasure Data product manager Luca Candela discusses defining your feature tests, cohorts and research experiments. H...
In this Heavybit Fireside Chat, former Keen.io Director of Marketing Alexa Meyer is joined by Segment CEO and co-founder Peter Reinhardt to discuss open source as a valuable way to find product/market fit. What began as an attempt to kill an idea became an unexpectedly successful business strategy for Peter and his team.
If your tech is based in open source or you’re considering fostering an open-source community around your product or company, then this is the talk for you. Hear Peter explain how Segment helped analytics.js evolve from a community project to enterprise tool. Take away pricing tips on things like what to charge, who to charge and when to charge. Check out the video and transcript of this Fireside Chat.
Alexa: Hey everyone, I’m Alexa. I was previously the Director of Marketing at Keen.io, which was based out of Heavybit quite a long time ago. It’s an analytics tool for developers. Now I’m working on something completely new in the mental health space, but open source was a big part of our business and marketing strategy at Keen.
I’m super excited to be here with Peter Reinhardt, the CEO and founder of Segment. We’re going to dive in a little bit into how analytics.js, one of their open source projects, played a big role in them finding product/market fit.
For those of you who don’t know, Segment is a customer data platform with over 15,000 customers. They’ve raised over a $100 million and they are doing quite well. I’ll kick it over to Peter to just open up, tell us a little bit about yourself and the journey from analytics.js to Segment.
Peter: I grew up in Seattle and then went off to study aerospace engineering at MIT. You may be wondering how does an aerospace engineer end up at a software company, running a software company. The short answer is that my roommates at MIT were both in the computer science department, and we decided in our junior year that we really wanted to start a company together.
So we applied to Y Combinator with this idea to build a classroom lecture tool. The idea was to give students this button to push to say, “I’m confused,” and the professor would see this graph over time of how confused the student were.
I might have this graph up here right now. A bunch of professors were really excited about it. We got into Y Combinator with this idea and raised about $600k coming out of Demo Day, and we were really excited about it.
We put it on a bunch of classrooms in Boston as the fall semester got going in December 2000 or September 2011. And it was a total disaster. Basically, all the students opened their laptops and they all just went straight to Facebook. This was horrifying because we had literally just gotten the checks from our investors.
So we had to go through the painful process of calling back all our investors and explaining to them that this was a terrible idea, and basically decided that we wanted to pivot into being an analytics company. And so we tried to build a competitor to Mixpanel, or Google Analytics, or Kissmetrics, or Keen.
It turns out this is a really hard market to build a product in. There are literally dozens, if not hundreds, of analytics tools, and it’s really difficult to differentiate.
Without a differentiated value, it’s really hard to make any money. So about a year after that first failure, we still didn’t have any paying customers of our analytics tool, Segment.io. That was a pretty tough point for us. That was December 2012.
If you pause there, rewind back to the very first week of Y Combinator, we’d been like, “Well, we should have analytics on our classroom lecture tool.” And so we Googled it and we found Google Analytics, Kissmetrics and Mixpanel, and we were trying to figure out how to install these different tools, trying to figure out which one to use.
We basically noticed that they all collect the same data, and we couldn’t figure out which one was going to be better than any of the others. So we decided to build a little abstraction that could basically take in the common user data that all of these tools needed, and then could translate and fan that data out to all those tools downstream.
We built this little wrapper, called it analytics.js and then we set it aside and kept working on our classroom lecture tool. Then this little library got cleaned up a little bit, four months later it got cleaned up a little bit more, four months later cleaned up a little bit more.
By this time we were trying to sell our own analytics product. We kept encountering this objection in the sales process, which was, “Well, I already have Mixpanel installed.” Or, “I already have Google Analytics installed.” Or, “I already have Keen installed.”
Then my co-founder, Ilya, had this really good idea. He’s like, “What if we took that little library that could send data to those three tools, and we added ourselves as the fourth tool that it could send data to? So every time we encounter that objection, we’ll just send them this library and they’ll be able to send the data everywhere.”
So we started using this, we started sending it out to customers, they started installing it, starring it on GitHub, submitting pull requests, things like that. A few more months went by. We started to realize that our analytics tool was still going to fail because the people who were using this little library were still not using our analytics tool. They were just sending data to these other tools.
Finally we get to December 2012, we realize that our analytics tool has failed. We have about a $100k left of our $600k in funding.
We get to this decision point of, what are we going to do with the final $100k? What’s our final attempt to build a company here?
My co-founder Ian was like, “You know what? I think there’s a big business behind analytics.js, this open source library.” And I was like, “That is literally the worst idea that I’ve ever heard. It’s 500 lines of code, it’s already open source. Explain to me the business proposition behind this library.”
We fought about it all day long and I went home and I was racking my brain. It was like, “How do I kill this idea?” I came in the next day and I was like, “All right, guys, here’s what we’re going to do. We’re going to build a beautiful landing page. It’s really going to pitch the value of analytics.js, this open source library, and we’re going to put it up on Hacker News and we’re just going to see what happens.”
I’m thinking this will totally kill it and it will prove to them that there’s no good idea here because nothing ever gets voted up on Hacker News. And so we did that and much to my surprise, it went straight to the top.
It got about 300 upvotes on Hacker News, got a few thousands stars on GitHub, a few thousand e-mail signups. People were reaching out to us on LinkedIn, demanding access to a hosted version beta that we said existed but didn’t. So open source library sort of launched the company, if you will. I think we’re going to talk a little bit more about that. Sorry for the long answer.
Alexa: It’s great. Just to get a quick show of hands, how many of you have either built something open-source related to your product, or are considering it currently? So, a lot of you. That’s great, it’s probably why you’re at this talk.
Peter, could you tell us a little bit more about how both Segment and that initial community around analytics.js has evolved over time, and why?
Peter: Frankly, most of the community interest happened in the first 48 hours, where we basically went from 20 stars on GitHub to about 1,500 stars on GitHub. The cool thing was, the Hacker News community validated the idea, first and foremost.
You know, they could go and trawl through the code and say, “Okay, this is abstracted well. This is well-written code. It’s clean. This is something I would want to put on my site. I feel confident in the engineering behind this library.”
A lot of this initial value of the analytics.js community was just in validating that this was the right way to build a solution to a real problem.
Frankly, since then, the library has continued to get a steady stream of attention. But even today, it’s at like 3,800 stars, just to give you a sense. So it’s not like in the last five years it’s gone on some exponential curve in terms of interest in it. It was really an initial spike and then a long, slow rise.
Unlike other open source libraries where the open source library really contains most of the value, analytics.js is a little bit unique in that it doesn’t totally solve the problem that developers have. And so you really need a hosted version in order to solve that problem.
The library has continued to be a lead gen source for us, if you will. I think it gets about 75-100 page views a day on GitHub. And the community, I think, has been really helpful in issuing support requests and occasional things to flesh it out over time.
But really, almost all the interest happened in this bang at about 48 hours, right at the very beginning.
Alexa: So you have that initial interest, and a lot of that early community has contributed to your success in some way. But did you build out a funnel to continue to support and grow that, or was it more of an emergent outcome?
Peter: It was more of an emergent outcome. I think there are other open source libraries, like say a database, where you have to open source the entire database in order for someone to give it a try. Analytics.js is a little different because you can see it and inspect all the code, because it’s only 500 lines.
In order to actually use it, you need to deploy it to a CDN. And then when a marketer wants to update one of the integrations to send data to a new tool, then you need to recompile the library and re-deploy it to the CDN. So it’s different in the sense that there’s a lot of heavy lifting there that an engineer has to do to actually use the library, but that they don’t really want to do.
That very naturally becomes the genesis of a hosted version where a marketer can push a button and our hosted version of the library changes automatically.
I think there’s something unique structurally about it as an open source library that makes it really amenable to driving traffic in a funnel towards a hosted version of a product.
Alexa: It looked like a lot of people in this room are either considering an open source strategy or already have one. Do you think what happened with Segment, analytics.js is a repeatable, reproducible strategy that other founders and engineers hoping to build out successful paid, hosted developer products could replicate? And should they follow that?
Peter: I was thinking about this a little bit. There’s a really good blog post. How many of you know TomTonguz, the blog? Oh, okay, there’s a lot of hands there. Good, so he had this article recently, I think he was reflecting on the past year and how almost every company is really kind of a special snowflake.
I think it’s a really good observation. Most companies really are pretty unique in a lot of ways when it comes to the core product strategy. So I worry a little bit that there’s something structurally about analytics.js which is not reproducible, or at least not reproducible on purpose.
Maybe to be a little bit more helpful, I think what’s interesting about analytics.js is that it was sort of a minimum testable product in the sense that it tested some of the assumptions of,”Is there value to be delivered here?” And, “Is that something that people are interested in, but it was not the actual thing that people would want to buy?”
Which I think is different from a lot of other open source projects where it’s really basically the entire kit and kaboodle and you can do everything with it, except maybe there’s a few features that are blocked off around the edges.
There’s a really famous or well-known diagram of what a minimum viable product looks like. They show what it’s not, which is two axles with wheels on either side that are just disconnected, and some bolts lying around. That’s not a minimum testable product because no one can actually go try it.
I think that’s not what analytics.js was. Analytics.js was much more like the other diagram, which you may have seen, which was a Razor scooter with a vision bubble of a car. You know, analytics.js could basically provide some value and really test that assumption, but could not really solve the entire enterprise problem that we now address through a hosted service with data governance and data sources from other places like Stripe and Salesforce and ZenDesk, collecting data from mobile, and so on.
I think if there’s some way that you can use open source to test the demand for a product or test the minimum viable thing, then it can be a really good way to build a fanatical set of users that really go deep in understanding and contributing back to what problem you’re trying to solve for them.
Alexa: That’s really interesting and I want to touch a bit on some of the theoretical approaches to open source from a business perspective. I think it was a past Speaker Series talk here at Heavybit. MySQL’s Marten Mickos was saying how there’s basically two main open-source approaches.
There’s a foundational approach where you have an Eclipse or a Mozilla foundation-type model. And then there is one where the company and the actual open source project are one and the same, so something like MongoDB or JBoss. Do you agree with that assessment, or do you have a different opinion?
Peter: I definitely agree with that assessment. I think there aren’t that many successful, truly open source models for how to build a company around a product that is really, truly open source. And we looked at that early on.
This is one of reasons why I didn’t think analytics.js was a good idea, because if you look at those two models and you look at analytics.js, it’s not complicated enough to have the JBoss or Red Hat model where you provide support around some really complicated system that needs to be run in production.
It doesn’t really fit the other model, either, of a consumer product that has ads or other things baked into it. So those are really the only models that have been proven out for open source. And analytics.js didn’t fit either of those.
I’d say actually it was very purposeful on our part to pivot out of being an open source company.
It’s certainly where we came from. We still open source a lot of things. I think we have, maybe, a 1,000 or 2,000 open source repos, some of which have more stars than others. But we very much pivoted out of that on purpose, towards a real business model that we felt had legs.
Alexa: I agree with you as well. Given that that’s something that’s worked for a lot of companies, to use open source to do that initial test, build that seed community, figure out if there’s product/market fit and validate that, and then shifting your focus to a more hosted and paid business model. What early indicators did you look for as a signal that that was something that was working?
They would tell us how many of the top million sites were using the open source version of the library. It is shockingly small compared to the hosted version.
It was pretty apparent from early on that a very, very, very small number of hardcore engineers actually want to use the open source library and everyone else wants to use the much easier hosted version.
Other systems that are maybe deployed in server backends or other things would be way harder to measure, like how many people are using the open source thing versus a hosted service. But we were lucky, in a sense, that analytics.js could be detected.
Alexa: As founders that are navigating that decision, “Go open-source first, test product/market failure, build out the hosted version and try to test that,” what would you recommend?
Peter: I think if there is a natural way to build a small open source library that tests product/market fit, then I’d totally do it. You’re going to get a fanatical developer community that’s excited about the fact that it’s open source.
They’re going to issue support requests, they’re going to give you lots of feedback. You’re going to get more feedback than you want, probably.
So I think it can be a really, really valuable strategy. But if it’s not natural, I wouldn’t force it.
Alexa: Obviously we know what happened with Segment. You guys are super successful. You landed on a SaaS business model, versus a fully open source model. I’m curious, did you ever consider FOSS foundations or vendors in your early business decision-making, or was the primary focus always on contributors and users?
Peter: I’m sorry, what do you mean by a FOSS foundation?
Alexa: Open source software, “Free Open Source Software” foundations and vendors.
Peter: Oh, so just like going to a model, just offering the entire time for free?
Alexa: And then operating more on a support base.
Peter: I don’t think we would have solved people’s problem that way. We were most interested in helping companies solve the underlying problem, which was, how do I get my data out of all these systems into all these other systems?
There are some open source, like truly open source, products that help people do that and they just don’t have that broad of adoption. And I think it’s because there’s enough complexity in setting it up and maintaining it in your own data center or your own cloud, that it doesn’t really solve people’s problems.
I think in order to actually build something that solves the underlying problem for companies, you actually have to go all the way to a hosted service.
Peter: I guess we sort of considered it but it wasn’t a natural way to help people.
I think that’s a big benefit of building a library in that ecosystem, in the sense that you’re going to get a lot of early usage and people bang on things to see what works.
There are definitely other communities, like my father-in-law is a physicist at UCSB. They program in Fortran. That community is not an early adopter of new libraries.
Alexa: What were some of the early challenges of this approach? And how did you solve for them?
Peter: I think the biggest challenge is when you’re coming from the open-source world and shifting into the business world, you have this mentality that things are free. Or at least that your ground assumption for building a product is that your product is free.
I think that’s actually pretty bad, as far as trying to build a company, because you’re not really grounded on the value that you’re providing. You’re just grounded on the fact that you offered it for free at some point.
So I think, especially for us coming from the MIT community originally, where there’s really strong free, like, “software should be free,” from Richard Stallman, as a sort of cultural aspect of the MIT community. We were really, really adverse to asking people for money for the service that we were providing.
It actually took a sales advisor grinding us down to get us to understand that it was okay to ask companies to pay money to use this service that was providing them value.
Basically all this guy did, which was amazing, he basically sat me down right before we went into our first or one of my first sales meetings with him, and he said, “Peter, you have to ask for $120 thousand a year.” And I was like, “Holy shit, are you kidding me?”
At the time, we were charging maybe $120 a year for using analytics.js, the hosted version. And he’s like, “No, you have to ask for $120k.” And so I asked for $120k, and I got argued down to $18k a year, which was pretty embarrassing. But nonetheless, that was 150x what we had previously been charging, right? So he got, whatever, 80% off and I got 150x.
I think I was constantly surprised by how much people were willing to pay, which was basically an estimation of how much value I felt they were actually getting out of it. And so as we got more and more confident about asking for those prices, within five months of these, asking people for this amount of money, we were getting $120k a year on some deals.
Making the shift in your pricing estimate, from free open-source to what a real enterprise software company needs to ask for, is a pretty tough transition mentally.
Alexa: Yep. Going back to the beginning, to round this out before we open it up for questions from the room, are there things you’d do differently now that you have the hindsight? Looking back when you had that $600k, you were $100k left, you did this open source thing. What would you do differently, if anything?
Peter: We’d do so many things differently. The first was, it just took us a really long time to figure out what product/market fit would feel like. I don’t know exactly what stage your companies are at, but you know, we spent a year and a half basically searching for product/market fit when it probably should have taken us about six months.
I was always really impressed and inspired by the guys from Codecademy, you know, the online learning-to-code thing. I think they went through 12 product ideas in eight weeks. And they legitimately tested all of them. It took us a year and a half to do three.
Literally, Codecademy was created four days before Y Combinator Demo Day. And in those four days they actually found product/market fit because they iterated so many times and they grew from zero to 300,000 users in four days and then had the most amazing Demo Day of all time. So I think there’s something to be said for the pace of testing new ideas that we were really bad at.
Alexa: Getting one last question, you were saying, what was holding you back from testing more quickly at the time? Was it stubbornness around, “No, this is going to work, we’re going to keep going?” Or was it fear?
Peter: Stubbornness and the tricking yourself into believing that someone’s saying, “Oh, that’s cool,” is validation of the idea. Everyone will want to be nice to you and
it’s very easy to trick yourself into believing, yeah, there’s real product/market fit here. And you just waste six months chasing ghosts.
Alexa: Well, thank you so much. That’s great. I’d like to open it up to the audience if you all have questions to ask Peter.
Peter: About 48 hours. We launched analytics.js and it went, “Whoop!” Then on Hacker News we got all these email signups. Then we sat down the next day and we were like, “Okay, what does this mean?” And we were like, “Well, clearly there’s product/market fit for something here. Clearly there’s demand for the hosted version,” because people were literally reaching out to us on LinkedIn.
I have a screenshot of this guy who’s now a faculty at MIT. He reached out and said, “What does a brother have to do to get access to the beta? I will tolerate bugs like you wouldn’t believe.” It’s like, “Okay, clearly we should.” It was not ambiguous at all.
And so analytics.js launched on December 12, 2012. Then we basically, immediately the next day, started working on the hosted version because we said it exists and it didn’t. We built it in five days and launched it on December 17th. And then we just focused entirely on that.
The weird thing with analytics.js was that it got all this star attention, it got like all these people who were excitedly chatting about it.
I would be willing to bet that no more than 500 people have ever actually used open source analytics.js in production. It’s not large. Way, way, way, way more people have used the hosted version.
By the end of December 2012, the hosted version was launched. So soon after, and it was so much easier to use that, everyone just cut over to that. By December 30th, by the end of that December, we had 70 companies sending data through the hosted version, off of the beta list of a few thousand signups. It was very, very fast.
Alexa: Were you charging them at that point for the hosted version, or no?
Peter: Oh, God no. We were terrified of charging people.
Alexa: Okay, because I was going to say, when developers are evaluating something that’s free versus paid, and they’re like, “Well, it’s free and I can just do it myself.” But if both were free, then that–
Peter: It took us 10 months, maybe, until we decided to ask people to pay money. We were terrified of this.
We literally wrote, I have them all, we wrote handwritten emails apologizing to people that we were going to ask them for money.
I can’t believe how afraid we were of asking people for money.
Alexa: But it was worth it, once you finally did it?
Peter: Yeah, exactly.
Alexa: Would you advise people to ask for money a lot sooner?
Peter: From day one, yeah.
Alexa: What have you learned about adoption from small companies versus large companies that can pay for Segment?
Peter: The way that we get into these companies, or into all companies, is pretty similar. But the willingness to pay and the reasons why we get into those companies are a little bit different.
In really small companies, say, less than 20 people, we only really get in if they are investing for the future.
Because a really small company doesn’t use that many tools. They have an analytics tool and maybe an email marketing tool, and you can just install those directly yourself.
But if you are a small startup and you’re looking ahead to where like you’re going to have a bunch of ad pixels, you’re going to have a whole bunch of other data sources, then you might iFornvest in installing Segment or analytics.js early on.
Startups are a unique beast in terms of our marketing, because they need to be expecting this growth curve for it to make sense to them. But even there, we get in through developers. Developers who are like, “Oh, this is the right way to do it, we should just use Segment.”
In SMB world, we’ll just call it 20 people to 200, we still come in through a developer. But then it tends to sort of percolate up through the organization until you get to the VP of engineering or the CTO. And then they’re the person who actually buys it.
Basically, they’re doing it because the marketer, the CMO or VP of Marketing, is constantly pestering them and saying, “I need this tool. I need this tool. I need this tool.”
At some point, the VP of Engineering throws up their hands and is like, “This is ridiculous. We need a better solution.” And then an engineer is like, “Ooh, ooh, ooh, I know.”
Then in the mid-market and enterprise, the same thing tends to happen in a business unit. But then there’s like a different complicated mechanism by which we spread across business units and up inside the organization.
I think we were looking at analytics.js and we were looking at, just how complicated is analytics.js? It’s like, it’s just not that complicated. I mean, it really is like a thousand lines of code. So if we just provided service for that, it was a little bit unclear what service we would be providing. Like, how would we, what would we do?
Then we were talking with people about what problems were in the way of them actually using the library, which is sort of, what services would you offer? And it was entirely, “Can you host it on a CDN for me?”
Then that naturally flows into a SaaS model, as opposed to naturally flowing into, like, “Well, this thing is so complicated, we can configure it for you,” you know? That would maybe go more towards a services model like Red Hat.
So it’s the product that forced us to split there, as opposed to it being a really cerebral decision.
You want to understand the technical inner-workings of this thing that you’re about to add into your code base.
From that perspective it’s been really important and critical for it to be open source and be visible and understandable throughout all of our sales processes. Because when someone we’re selling it to, even a large company, and they’re like, “Oh, well, how does your code work?”
Because they’re used to a bunch of terrible vendors that have APIs that are always crashing. We’re like, “Oh, here’s our library. It’s open source. Check it out.” Engineers are like, “Oh, man, two thumbs up. This is great.”
All of a sudden you get this buy-in at this really important layer in the company and it really accelerates growth. So I think it’s been very important even though no one uses it, which is weird.
I’d say we have two moments in the company where the pricing has really dramatically changed. The first moment was the one that I described earlier where this sales advisor was just like, “Ask for more, ask for more.” And really that was just value. There was no rhyme or reason to it. We’d just go into the company and we’d say, “Well, given your volume…” and, “Here’s our price.”
We just threw out whatever because we had no idea.We didn’t know if we could charge people a million dollars or only a hundred dollars for it. The only way to find out was to come up with some fabricated reason for the price that you’re presenting. So that was pretty shoot-from-the-hip, testing the waters on value.
Later, more recently, maybe a year and a half, two years ago, we made a much more thoughtful, concerted structural change to our pricing. Which was to switch from API calls, how many pieces of data are you sending through Segment, to how many users are you tracking through Segment. That was a much more thoughtful, analyzed–
We hired a pricing consultancy firm to come in and run user surveys and then compile all that data and then we ran some experiments. That was a much more concerted effort and was also very successful, doubled revenue growth rate, basically.
I lost track of your original question, but pricing is super impactful and early on, I think you’re just trying to understand the value.
I think this is a big problem for a lot of open source projects and associated companies. It’s not a problem for analytics.js, and it’s not because we came up with some clever hack there.
It’s just because the structure of the library is, no one wants to host it on their own CDN. I don’t know if there’s magic there, but we don’t really encounter that.
For better or worse, we didn’t really think that much about it and we just made everything MIT-licensed, which is a very permissive license. Once it’s out there as MIT-licensed, you don’t really take it, I mean, it would be a really bad thing to take it back. So pretty much all of our libraries are MIT-licensed.
It’s interesting because we have actually one competitor that, literally, they take our open source library and use it as their product. It has that downside I guess, that people can just compete with you on it.
There’s no anti-competitive terms or anything in the license, it’s very permissive. On the flip side, I don’t know how a company like that really attracts talent when they aren’t even using their own code.
So anyways, I think it sort of works out in the end. But yeah, we’ve never really thought long and hard about what the licenses are that our stuff is open-sourced under. We probably should have.
The question is, do we have external contributors not coming from Segment? Definitely, yeah. On analytics.js, I think there’s probably, I would guess a hundred external contributors. Basically the library is in a pretty polished state, so there’s not a lot of activity on it now. But yeah, from the early days, there’s probably a lot.
Alexa: Thank you all for your questions and your attention. From my perspective, being in the analytics industry for the last three years, it’s been really fun to watch Segment’s growth and success.
I’m going to close it off with a more general question. What advice would you have for early-stage founders in general, or ones that are considering using open source as that initial product/market fit validation strategy?
Peter: I guess I would just say
you can probably spend a really long time chasing your tail searching for product/market fit.
I would be very, very careful about how you think about and understand whether you have product/market fit.
And if you don’t, be really, really honest with yourself about it. Otherwise, you can waste a big chunk of your life searching for something that isn’t there. So really, really focus on finding product/market fit. That’d be my main recommendation.
Alexa: Thank you so much. It’s been a lot of fun, and thank you, everyone, for coming.
Peter: Thank you.
Interested in joining Heavybit? Our program is the only one of its kind to focus solely on taking developer products to market. Need help with developer traction, product market fit, and customer development? Apply today and start learning from world-class experts.