August 24, 2017
API Discovery Meetup: Clarifai AI and Algolia’s Airplane Mode Search
In these talks from the API Discovery Meetup, learn how you can use Clarifai for convolutional neural net training, and let users search eve...
Alright, thanks. Happy to be here. I recognize definitely a few faces in the crowd.
I'm the co-founder and CEO of Crittercism. We're a mobile application performance management company. It's interesting because APM has been this acronym that's been around forever. It seems like the newer age of developers and DevOps people actually don't know what it means. It's interesting because that's the target market we're going after is enterprise companies. There's definitely been this transition that we've gone through and I can talk about that. Part of those transitions have also involved the messaging and marketing which we can get into as well.
Just a little bit about Crittercism. We've been around for about three years now. We first launched our product Summer of 2011. It's essentially an agent or an SDK that's embedded into mobile apps. It sends off real time diagnostic data about performance issues. Developers or operations folks can log into our portal where they can consume the data, we help them triage, troubleshoot. We do a very in depth root cause analysis, if the user was trying to buy something and it didn't work, if maybe they couldn't add something to a cart. We provide a ton of data to recreate those circumstances and then the hope is they can quickly turn around a fix.
I guess the other things to mention here are our revenue. I deleted some of the private data. We haven't released revenue numbers, but we have grown pretty significantly and Gartner has also recognized us as indeed being the leader in the industry. That's a little bit about us. We do support iOS, Android. We have something for HTML 5, we support some third party platforms, something for Windows phone.
It's definitely been an exciting journey along the way. We've grown very quickly. What I'm going to do is actually walk you through year by year starting in 2011 and just share some of the learnings as we've grown and what's worked and what hasn't worked.
If I was to show you the beginning of our slide decks today it might look something like this, but back in 2011 it was more like this: We have the cute critter up there. It says "Support Platform." You should take notice to the word platform because when we originally started the company, it seems like a little thing, but we'd call ourselves a tool.
No one wants to be called a tool for many reasons. Platform sounds bigger, sounds sexier and that's how we were positioning ourselves.
You might think support, what does that have to do with performance management? Early on we thought it was one product. It was really two products. We had this crash reporting thing and we also had this widget for if you're using a mobile app it allowed you to submit support tickets essentially through the app. We ended up taking a look at our data and one of them was growing much more quickly than the other which was the crash side. It took us much longer than we should have to actually deprecate one of the two, but we indeed ended up doing that.
I'm going to get to how we actually grew ourselves and made the transition. I just want to set the stage for how we got some initial traction and maybe some of that will be relevant to the stuff that you guys are working on. Really, 2011 in summary, we were just focused on building product, getting lighthouse customers on board and doing unscalable things.
In the early days, we just wanted to prove out the product market fit. Here's an early graph. One of the beginning slides said that we were on something like 800 million devices. Back in 2011, I think the number says 100K and then it goes up. We're in the tens of thousands, hundreds of thousands devices at this point. It was just friends and family that we were reaching out to.
We had gone through an incubator similar to what you guys are doing. We just tried to get as many people as we could to use the product. We were sending e-mails in the form of, "Hey, so and so why don't you come and try out the product?" We had a beta list. We created some artificial demand by setting up that list.
We found early on, actually, to be this magic number of five. What I mean is we set up a way that you could add team members. We found that for whatever reason as soon as any one of our customers added five or more team members that there was just exponentially more usage in that account. Actually, I think that was a great early learning for us.
When we were setting up our pricing tiers we never wanted to discourage adoption. I think if you look at some products, they sell seats, like number of developer seats or what have you. But I actually think it's not the right approach necessarily.
You really want an organization ALL in on your product. Especially the Valley is very fluid. Many people will join a company and move onto something else, and you want those people to be your advocates. That's exactly what happened to us.
Early on we got the early customers that did manage to come and use our product. We found that one person would be at a very large video streaming provider then go to a financial services company. And guess what? We maintained that relationship and they brought us into that account as well.
We also found that doing something as simple as adding Google OpenID sign-in was huge for us, making it that much easier to get customers on board. Actually funny enough, we've been wanting to add GitHub for a long time and we haven't. I still think we should. We just haven't gotten it scheduled for whatever reason.
There are some flip sides to that. When we first launched, we launched Summer of 2011. We announced our funding and at this point I had mentioned it was a lot of long tail developers and small companies using it. We quickly had some great inbound enterprise interest once we announced our seed funding. Part of it was that we had some great product market fit. You know, to quote Paul Graham, "the hair on fire problem" we were attacking which was apps were crashing.
At this point, by the way, we hadn't transitioned to a full application performance management platform. It was still basic crash reporting which was really our wedge into the market. We managed to find that wedge which was great. Companies were just looking at app store reviews. They were doing these crazy workarounds. We heard stories of LinkedIn hiring contractors to use their apps in Kansas to see if it was working correctly. People were throwing phones in microwaves to simulate faraday cages.
You see people doing silly things, and you know you have a great product when they don't have to do those things anymore.
To circle back to the point I was making before about adding team members, adding Google sign-in. One issue is that once you do that, it's a little bit harder to see what company they work for. You'll get a bunch of Gmail addresses. It does ease adoption.
The other thing by the way is that as you grow and you start adding sales reps, you'll need a way to be able to identify those accounts. There are some automated software you can use to do that, but you'll also need a way to route those leads to different territories as you start building out territories. From the beginning we actually didn't ask for a phone number. Having an area code would actually let you know what territory that customer is coming from.
There's always dropoffs when you add more fields to a sign-up form so obviously that's something to pay attention to. I think we did the right thing early on not having that as a required field. I don't even know if we had it at all, and we would just kind of manually reach out or maybe we did set up some automated marketing campaigns. Today, you do need to enter a phone number. It's a small point, but it actually helps us immensely when we can't identify the account or we need to be able to route a lead.
We ended up getting our first big accounts towards the end of that first year that we launched, December of 2011. At that point they're actually in trial so they didn't start paying until January of 2012. We did have zero revenue for this year, but that wasn't the goal. The goal was getting lighthouse customers on board, getting our metrics up.
Early on, again, it was a lot of manual stuff. It was stuff that wouldn't scale. We also didn't really know the price to charge and we felt like there wasn't a lot of past precedent to look at.
One funny story. Well, it's funny looking back, it was not funny at the time, was with Smule. They're a game developer. They make Magic Piano and a bunch of other great apps. The Ocarina app, if you use that one. They were another shock to the company. They were introduced to us through an investor, really interested in the product. At the time we had no idea how to price this thing. There was someone else at the company that was there for about two months, and he moved onto something else.
We ended up working on this crazy spreadsheet where we would factor in price elasticity and just some crazy calculations that looking back made no sense whatsoever. We ended up giving them a quote for something like 40K a month, something ridiculous. Something so absurd that the CEO called me up and grilled me. Not only because that's an absurd price for the state of the product at the time, but also because they were a referral through an investor. They were expecting some kind of discount, and certainly that wasn't what they were expecting.
We learned a lot from that. We learned we couldn't charge 40K a month at that point which was great. By the way, that has changed for us now as we've added more product and whatnot. At that time we at least knew a ceiling. For Netflix, we were lucky we didn't run into that case. We did present a pricing schedule that was based on volume. It made sense for them — as they added more subscribers, they grew, everyone made more money. That became actually one of our biggest accounts for a long time until the past 12 months. They were no longer our biggest account.
One other side note was actually one of our first paying customers was in Europe which presented some compliance stuff that we had to deal with. At the end of the day it was some great logos that we put up and some great metrics we were able to generate to get us to a series A which we needed to sustain the company.
I realize I'm probably running a little bit long on time so I may not touch on everything here. I guess the last thing I'll mention about product market fit, before I start talking more about sales, is that the pain point we were addressing was really focused on all developers. It wasn't focused to a particular niche or to a particular vertical.
I think that was really helpful because we would with open arms welcome any application that wanted to be on the platform. We really had to try to search out the communities.
Actually, when Heroku, and thanks to him for Heavybit, they did a great job of really engaging with the Ruby community. It was a very tight knit community. This wasn't the case with mobile. It may not be the case with the types of developer communities that you all are targeting.
We really had to search them out and we did some crazy things like, and you can't steal this one because I think we're still going to do it, we sent the development teams packs of marshmallows with marshmallows guns and critters. Of course, we ended up getting tweets with people holding the guns and doing crazy, inappropriate things with the critters which was exactly what we wanted them to do.
The other thing I guess is that it's hard enough to get people to add code to their existing code base, so that's part of your business model. We tried to make that as easy as possible. We would literally put the exact Find-A-Code in the doc so they could just copy and paste. It would be auto generated based on their data. I think that worked out really well for us as well as just dogfooding our own product.
My wife is an engine developer. She uses our product which is interesting when you go home and there's bugs and your wife is yelling at you. It was still a great way to test it out early on because we were focused on building a product for developers, but we didn't have our own mobile app. So that was kind of a great path to testing our product more deeply.
Alright, let's switch gears a little bit.
By the end of 2011 it was time for us to hire our first non-engineer. I think many of you have teams of engineers. We got to the point where we really felt like we needed someone out there. I was running around trying to do sales, marketing, BD all by myself and I was still coding at that time as well as my co-founder. It just wasn't scaling and it was time to stop doing some of those unscalable things and hire non-engineers.
We hired a generalist which worked out for a while. We can talk more about that towards the end of the session. But the fact is, if you do hire a generalist, you're going to eventually have to hire a specialist. You'll have to hire that VP of Sales, the VP of Marketing. Someone who's an operations person. That can create potential conflicts if you end up hiring someone that comes in early on to just do a little bit of everything.
We also hired a VP of Sales. We got really lucky, it was through our network. I think if I was to give some advice on how to find your first VP of Sales, it needs to be someone who's a leader but someone who's just also a really great rep. Someone who's a great outside rep that can really put some automation in place, put some systems in place, but also get out in the field, know how to convey the value of your product.
These people can be very hard to find. I would take your time. It's really important and it was one of our most important hires I think.
We also hired a VP of Marketing. They all came in around the same time, but it became clear that this was going to be a very exciting market. We needed to raise awareness and that was the reason.
Here's a timeline in terms of how we structured things. We had three employees, just the co-founders in January. By the end of the year we had seven. At that point we had jumped from basically zero to 25 million devices. That's compared to the 800 million number a year today. We had some great early traction and we moved beyond that initial build product to get to lighthouse customers phase. I mean of course, we're always building product, but that initial phase to 2012.
And 2012 was really all about land and expand. We wanted to get into every major account we could. We obviously still didn't know what the exact price should be. We were still doing some testing. All we knew is that there were companies out there like Turner, AT&T that had hundreds of groups and mobile apps within those companies, huge amount of brands.
What we wanted to do is get at least one or two of those brands using us and then upsell the account from there. By the way, that's exactly what we did.
Our VP of Sales at the time, this is kind of interesting, since we didn't really have a sales team we just assigned him a straight commission — a percentage off of the sales that he was doing. Then some of the initial people that he hired, who were junior inside reps, really helped him get additional leverage as we got more and more interest for the product. That only scales so far.
You can't just take a percentage off your top line and give it to someone. It worked early on when we didn't want to bother coming up with a commission plan or any type of structure. Eventually, we did have to setup an on-target earnings, an OTE, that included a base comp plus a commission.
He also did a great job of moving us away from just selling features. Early on when we would do a demo we'd say something like, "Look, you don't have to go to the app store to see all those bad reviews." Well, actually, we didn't even say that. It was even worse than that. We said something like, "Check this out. You can get some sweet real time crash reports. You can check out all your performance issues. It's great. You can click this button and it does this, this and this."
What he did a good job of was coming in when he's talking to a customer, saying stuff like, "Don't you hate it when the CEO's app doesn't work, and he comes and he maybe prints out a bad review and he shoves it on your desk and he says, 'Hey, fix this,' and you don't have any data to go back there and nothing is actionable. You don't know how many users it's impacting. Using our product, you can actually take the top 10 issues, put them into your sprint, cut off 50% of your problems right away." That was a much different sale than saying, "Look, you can filter by app version and you get all these data, and look, you get some diagnostic data." The story approach worked much better.
Of course, we also put some more automation in place. I think I was using Google Docs or Excel spreadsheets or maybe I was trying out Zoho at the time. We just really weren't putting the leads that we were getting into a system that sales reps could utilize. So we moved to Salesforce which is not cheap, but it's worth it. Every sales person basically today knows how to use it. It's what they're used to using if you're hiring someone that's experienced. At the end of the day, it's just not worth having them train on something else.
I guess the last thing I'll mention is that this land and expand approach was great, but we did run into some problems with the contracting early on. We had our own paperwork, we had our own contracts that we got that were form letterheads from attorneys or whatnot. A company like Turner, they want to work on their own paper. Yes, as an aside, if they do want to work on their own paper you could try to command a higher price which some companies do. I know some analytics providers in the Valley do that.
This in particular is bad because we didn't actually have a contract attorney. Let me take a step back. We did have a contract attorney, but we were using our corporate attorney. We were using the same attorney that we used for fundraising. We ended up racking up a huge bill, something like larger than our series A fundraising bill, just to close this account. We use Wilson Sonsini. They were nice enough to actually take some money off the bill when we showed them what had happened.
The point is as you start having to work more and more with customers under contract, get a contract attorney. Get someone independent. It won't offend whoever else you use. It's completely normal. We just took too long to do it and we wasted some money because of it.
The other thing to mention is that for a long time we just had basic and enterprise. Part of the reason was we were still testing out pricing, we weren't sure what it could be. Eventually, we set up a long tail, kind of a premium upgrade tier. At that time this was targeted towards developers that wanted to pay us but were more of the independent type.
The corporate customers, the enterprise customers would go for the enterprise tier, which I think also worked at this time. It did give us actually at least a few hundred paying customers which was great early on in terms of the metrics or data that we wanted to show to investors and everyone else. It did create some problems early on which I'll get into in a second.
The other thing to mention is that we did billing in arrears. I don't know how many of you are developing — you have developer-focused products. If you're not selling seats, you might be selling based on usage or volume. This became a bit of an accounting headache later on. We'd have a 30-day period. At the end of the 30-day period we would send them a bill.
The problem was we started doing this in a way that whenever they signed up it started that 30-day period, which may not sound too bad. But if you think about it, if you're trying to close out your books for a month, you charge someone, let's say someone comes on board at the end of July. That means you really won't know until September what to charge them because you have to wait until the end of August. Because of that the books for July take forever to close and we had to do some fancy estimations of what we thought the average contract value would be for that customer.
It just created a bit of a financial headache. We ended up moving everyone to just the first of the month. We did some proration and we still build an arrears for the volume, but made sure we started at the first of the month.
I mentioned I'd get back to some of the problems that the premium tier introduced. By this time, Fall of 2012, we did have at least a six figure contract or two, and our ACV was starting to go up. But you could imagine, if you're a customer paying six figures you'd go to our pricing page and you see $25 a month. You may kind of turn your head. What are these guys doing? What am I paying for? Is there some price discrimination here? There is an anchoring risk.
I'm a big fan of putting pricing up on the website, but you have to realize that if you still have a tier that's not exposed for enterprise customers and it's way orders of magnitude above what's on the website, that will become a problem.
Even today, ours says: "Starting at 100 a month," which is true. That's in a low volume bucket, so apps with higher volumes will end up paying more. It still creates problems for us. One is that it's not clear what the volumes are for every customer and not everyone always reads every line of text on your website. The others that we do have different pricing for, and this is kind of an aside, we have consumer apps and employee apps that use our platform. We do have different pricing and they are radically different. At the end of the day just be aware.
Let's move on.
At the end of 2012, we were about 16 people. This is the breakdown in terms of what the different organizations looked like. We also had grown pretty rapidly in terms of devices. We grew from zero to 25. Now at this point we're at 335 million devices using the platform. It was at this time that we started really building out some of our field teams. At the time we tried this pod approach. I believe it was something like two inside reps paired with an outside rep or maybe it was one inside and one outside. I can't remember exactly what it was at the time.
Whatever it was, we just did a poor job of conveying what the commission would look like to our board. It caused a lot of problems. It took us a very long time to come up with the exact commission. All of these just created this unnecessary confusion. The good news was we had real revenue coming in. We were still trying to figure out how to structure our sales team and how to comp them.
Let's jump to 2013. This is the year that we really built out our executive team. One of the things I think I personally waited a little bit too long to do was to do just that. I talked about early on of running around and doing different jobs. I was still doing that. If you're the CEO it becomes clear. If you're spending time with option agreements and doing a lot of the business development work for the company or the marketing work, you really need to be at a higher level than that thinking long term. You need to be working with people who are in charge of that, setting the company goals and vision and going out in the field and talking to customers.
There's a lot more that you should be doing than some of the day to day details that just distract you from running the company. That's one thing I would have done earlier on.
The other thing was we did finally get a great high velocity sales model in place. I'll talk a little bit more about that as we go along here. We also expanded our sales territories.
One other note is that we started making this transitioning. When we first started the company it was very much focused on dev, DevOps. We started to target more and more of that IT operations persona which is really the proper place for a product like ours in particular where we focus on applications out in production. Those are the folks who manage them.
It depends on the type of product you're developing. IT operations can tend to be a little bit used to the ways of doing things. Engineering will typically take over some new product initiatives, mobile being a great example. Mobile is very solid within companies today and we're just starting to see IT operations get more and more involved. But one thing to keep in mind is they typically have the bigger budgets and they're actually the persona that eventually should be using our product once that IT ops team merges with that mobile team.
I'm telling you this because I'm hoping that it somewhat translates to the target markets that you're all going after. It does take a different set of collateral, a different sales process, different messaging on the website. I'll talk a little bit later about some of the things that we did well there and some things that we didn't.
I just want to touch on this executive team concept. Pretty much all of these people were hired middle of last year. I'm talking about bringing in real enterprise sales professionals, marketing professionals.
Just to give you an example, our chief revenue officer he was at Wily and Mercury back in the day which were two APM companies. He led sales at Splunk to a significant revenue and he did the same thing most recently at AppDynamics. Splunk is a multibillion dollar public company. I'm sure AppD will IPO this year if not next year.
You really need someone like that who just has a huge amount of experience in the space that has commanded sales teams, grew revenue to a huge level.
Now we did have an existing VP of Sales and this individual we wanted him to stay on. We actually gave him the VP of the west to control. But at the end of the day, some people who come early on into a company to grow revenue, they want to go off and do it all over again and actually that's exactly what he did. He went to another company to lead sales after we brought in our CRO. It's a very normal occurrence. The rest of these folks have similar backgrounds. Our VP of Engineering co-founded a $60 million infrastructure business.
I think if you're in a hot market, you're seeing some great traction, you should be able to attract the top talent out there.
I wanted to compare and contrast one other thing that I mentioned early on which was this transition from talking about features and stories to problems that customers are running into out in the marketplace, and then finally to an ROI type of sale.
This is one example slide where we talk about problems that customers have dealing with their applications out in mobile. I think the more interesting slide in my mind is something like this to illustrate what I mean by really having a benefit-driven sale or talking about the ROI. These are actually the reverse of the benefits, but you can extrapolate them from here. You can see the impacts of not using a product like ours. What it can cost you. In some sense, a lot of customers fear a lot of these things and they're running into a lot of these problems today. You want to be able to easily illustrate those.
Another example of how to construct ROI we have a slide like this where we show prospects. What was it like before you used our product? What happened afterwards and how do you measure that impact? Something like this, if you're going in there and asking for half a million dollars, it's much easier if you can say, "Well, we save you 10 FTEs. What are 10 engineers worth to you?" I'd pay half a million dollars to save 10 engineer worth of time.
I think we talked a little bit about this. This was just really a summary of some of the things I've talked about. I did want to touch on what the current sales structure looks like. I mentioned we have a chief revenue officer. He's done a great job of really pulling in his crew of experienced sales professionals and if you bring in someone like this they're going to really leverage their network. I think him alone has hired something like 14 people.
What he did early on was put some support structure in place — hired a VP of Sales in the east, a VP of Sales in the west. It gives him time to focus more. Similar to how a CEO needed to hire their lieutenants, he needed to hire his lieutenants. We also hired sales operations. We were starting to grow big enough to a point that we did need someone to focus on things like commission structures, creating reports in Salesforce, integrating Salesforce with all of our other systems. It's very important. You need a lot of metrics.
You obviously don't want to always be afraid of taking action if you don't have data. I think that's one thing we probably early on were a little bit guilty of. We kind of threw our hands up in the air and said, "Well, we don't have the data." Sometimes you just got to move on without having it.
In any case, if you are using something like Salesforce you do need to connect that with everything whether it's something like Marketo or a subscription billing provider, etc.
RSM, regional sales manager inside sales rep. We actually did end up going with a team approach. What we did was we paired an outside rep and an inside rep. Each one of those was actually assigned 75% worth of a sales engineer time. The nice thing about doing something like this in our stage of the company is that typically you'll see some conflict between inside reps and outside reps.
Traditionally, you'll see a dividing line in terms of: If this is a 50K deal or below maybe the inside rep will handle it, or above the outside rep will handle it. That can create a lot of friction if there's a dividing line because you want to be able to grow these accounts over time. That kind of goes against the land and expand strategy. If I'm an inside rep I don't want to grow an account and then lose it to the outside rep. This model gets rid of that friction and it also means that a good inside rep will not tolerate a bad outside rep and vice versa.
We also have, there's multiple acronyms for it, BDRs, ADRs, SDRs, business development reps, sales development reps, accounts development reps. Those are more of our hunters and individuals that look at incoming leads and try to qualify them before they're passed off to the field teams.
One other note on this, as you start expanding your sales force you will get kind of this clash. It's important that you maintain culture as you grow.
The sales people will tend to be more of the suits and you'll have your engineers or the hoodies. Inevitably you'll need to pay attention to this. And that's more of an analogy, I'm not literally talking about how you dress. Although it's funny because in the Valley, I feel like I need to apologize when I wear a blazer. It's because I was at meetings earlier by the way, or I would have dressed more casual for this.
The point is just make sure you pay attention to culture as you're growing the sales team because there's that analogy that they can be coin operated or they can be more aggressive than your engineers are used to.
In some cases you need people like that. You need people who can get out there in front of a customer and who's loud and everyone likes him or her. They're able to get shit done and sometimes those people are different than what you're used to.
I'm just nearing the end here. I just wanted to point out this is our pricing page today and we still have that anchoring issue even at $100 a month. I guess the last thing I'll mention is that we've grown very quickly. In terms of headcount we were 16 a year ago, now we're close to 60 people. Really, we made the investments once we realized that this was a huge market opportunity.
I think that a lot of people make a mistake where they think I can sit back and focus a little bit more. "Let's do entirely inside sales or transactional. We'll just automate everything." That's great and all but the next person is going to go ahead and invest in the market. Get people out in the field talking to customers. And guess what? You're going to be left behind.
If you do operate in a market that's a land grab, then I suggest you go out and grab it. Today, these are all paying customers that we've managed to get as a result of implementing a lot of this.
In summary, I think a lot of companies will sometimes have this discussion about should we go after the long tail? Should we go after the enterprise? I don't think that's the right discussion. You can't ignore developers in this day and age. Instead of having that discussion, you need to figure out what is the type of product, who are the users of that product and then focus it that way. If your target is developers, that's great. You're going to want to get corporations, large companies on board using it. I don't think you should be worrying about it if it's long tail or otherwise because they're always going to have side jobs and whatnot.
I mentioned most of these. One of the ones I didn't mention was that our pricing, it was initially monthly, charged in arrears, we've actually been making a big transition of getting paid up-front deals. This is huge for cash flow reasons and I wouldn't recommend doing that until you actually figure out your pricing. I think everyone is always iterating on pricing but we've gotten to the point that we're pretty comfortable asking for a deal 12 months up-front which is huge.
We closed something like over 80% of our deals last quarter up-front. If you're building out an expensive sales force or investing heavily in the market, it can mean the difference of many months, if not a year, in terms of runway that you have on top of what you estimated previously. I highly recommend going ahead and doing that.
That is the end of the presentation. I would love to take questions.