June 18, 2015
Testing and Dev/Production Parity in Docker
DockerCon is just around the corner, and as you may have heard "Containers are the Future" ;). With that in mind, let’s take a look at som...
My name is Ivan Kirigin and I'll be talking about growth tactics and strategy. You can reach me at firstname.lastname@example.org. I think there's going to be some time for Q and A and if you have a question that I don't answer in time or whatever, do get in touch. Happy to help out.
A brief history; I started out with robotics, but then did a start-up called Tipjoy. Worked out at Facebook and then Dropbox and now I'm doing a new start-up. I tried some consulting, but I think the way I'd like to organize this is to start out with some conclusions, the way I view the world, some lessons learned and then project those back to give a bunch of examples through all these companies.
measure, test, deploy.That's pretty succinct.
I actually think this misses out a lot because it kind of make sense. You know you measure what's going on with your users. You run tests. Then you deploy what works. It sounds simple, but operationally there's a lot more detail that helps out with that.
So to delve into some of my opinions here, the first thing is to align everyone to a goal. This is actually something that is implicit of Facebook. They left that off the list because everyone knew they just wanted to get everyone on the world on Facebook. And it's actually surprisingly important because you have Google Analytics, you have Mixpanel, you have some internal metrics on some transactions, you have maybe some performance stuff like site speed or some monitoring. You are inundated with numbers.
They're all over the place. It's really confusing and everyone looks at different things. So picking something very simple like one, ideally maybe two numbers that you can all rally behind.
It is surprisingly important to get everyone in the metric's mindset. Then once you have this goal, what you need to do is understand what's going on. So put yourself in the mind of the user and say they hit this landing page or before that they might see an ad or blog post. Then each step in the funnel of which they use your product, you have to instrument that to know what's going.
Besides the actual events, you want to also understand who these users are. Maybe they're coming from a certain location or they're a certain kind of user. They signed up at a certain time. So all these details to understand this, the very first step is just to measure to really get a grip on whatever is going on.
With that, there's a natural extension to come up with a bunch of ideas that can help drive growth. If you have some invite flow where you're trying to have a social network that you want to grow. Obviously, if you are able to increase the number of invites, maybe with some suggested invites, that's like an idea out there.
If you just look at all of the things that are going on in your service, you're going to be able to come up with dozens, or should be able to come up with many, many ideas of what could drive this. Drive you to where it's your goal.
The problem is, and I don't care if you have one or ten or a thousand engineers, you're going to have this problem of having many things you think you want to do than you can actually get done. It's a huge problem and so you have to triage. You should triage by cost benefit.
This sounds obvious, but it's surprising how often all of this stuff is not applied where you take an idea and you say, "Okay, for it to do that what would happen?" If you model it out. If you did increase that invite count or whatever, what would the resulting output be. The nice thing about having a goal at the start is that you can anchor it all on everything shooting back to that goal.
What would actual impact on the actual thing you're trying to do as a company be? On the cost side it's interesting because certainly you can look at how many engineers that you have and how long it would take to get things done.
Also it's really important to start building things that you think are really important now at the top of your list in parallel. Then you have sort of operational constraints, as well. For example, you might have a single designer and you just practically can't do three front-end tasks at the same time.
This is where you want to get a grip on everything that's going on in your service, come up with a bunch of ideas, and then pick the best ones and try to drive them as fast as possible and in parallel.
And the final step just to close a loop on this is you have measure what happens. So you do that task and you see this is the result. This is actually really important, if for nothing else, than to build it intuition for that part about measuring the cost benefit because sometimes you're just suing you're gut on these things because you can't practically pick a reasonable number. So you're building a really good intuition for what would happen if you did a certain thing. That's really important there.
There's a bit of a problem, though. This advice, you pick a goal, you drive things toward that goal and you do that as fast a possible. Even people that say they're very data driven, whatever, they don't do it. Why is that?
I think there's a few reasons for it, but one of them is just a bit hubris about like some product features, maybe some aesthetics or some vision that you have or something that you really identify with the you really won't let that change and so you have to actually be really humble to say like everything's on the table no matter what.
So "murder your darlings" is one of my favorite phrases. It means that, basically, if you have something that you are really enamored with, really identify with, it's probably a sign of something you should take out.
This is true in writing, and I think also when it comes to good ideas around start-ups. You want to be open to everything.
Another aspect of this is that this breaking apart all the ideas you could possibly do, and you want to get it done in parallel as fast as possible, it actually makes you have this mindset of doing small changes. This is actually really good, generally, but that shouldn't be to the exclusion of larger changes. And actually, a lot of the things you do that might move the needle the most are actually really big, and it takes a great deal of creativity to understand that.
If you think about what a proper pivot is, not just scrapping what you're doing and doing something new. It's looking at what your user is doing, and trying to understand how better to make something that people want. And that actually takes leadership from the top.
Another problem here related to this incremental approach is that engineers typically want to slice and dice task lists and get things done in very small chunks, maybe have like short sprints, and you shouldn't let how you want to go about doing things affect what you do.
That's just totally backwards. So I think you need to have a great deal of leadership and creativity to drive this in a very data driven way, that really has everything on the table, once again.
Related to this is, I think, the science of motivation, which is fascinating stuff. And Dan Pink gave a great TED talk about this, the link is on the bottom there. I highly recommend you check it out. His book, Drive, is not very good. Just watch the TED talk. That's all you need. It covers everything you need to know.
People are motivated by just a few things; autonomy, mastery and purpose.
And we talk a lot about how to run a good company at startups. Some companies have really good shipping culture, and they have just disparate teams that can work independently. Obviously that comes with the autonomy, and if you want to ship at all, that comes with the mastery. And if you're a mission driven company, then you have the purpose behind it.
But I think the people that, if you could design your company in a way that inpiduals have these characteristics to their jobs, then you're going to get things done so much better and so much faster.
I would say certain products are very conducive to this - like Facebook. My thing is that facebook.com is a big collection of different products, so one guy can just hack on a little part of it. And I think that's dramatically impacted how well they've done because not every product is conducive to that.
Dropbox, for example, it has this really big deal, Don't lose people's files, and there's a lot of scalability and other things that go with that. So you have to, by necessity, keep things more in lockstep. You have less autonomy and you move a bit more slowly, but definitely try to achieve this because in the end, it's important for having happy people too.
So a lot of people that talk about growth hacking, they list these strategies or tactics that are really just anecdotes that worked for them. And I don't think that's very helpful because you might not be in the same situation. You might be able to learn from the patterns, but, really, it would be better if you just saw the patterns behind it. And one of the only ones I've actually come up with is, if you want something, if you want your users to do something, ask them. And if you want them to do it a lot, then automate it.
Asking could be like send them an email or send them another one or make the prompt to take some action, make that button bigger or a brighter color, or remove everything else and this is a very easy way to think about it, if you want something to happen, just ask. But then on the other side of that, if you can be creative enough to find a way to remove that decision altogether, that could be huge because then there's no like conversion rate, there's no decision to make. And I think an example later for Dropbox is a really good example of this where they just automated some core part of the product, but it's a really interesting pattern because I've seen it applied a few times now.
So now I'm going to talk a bit about the different companies that I mentioned before, and try to say like some things that I think they did really well, and poorly, and how that relates to all these ideas, just to get a better sense for what I'm talking about.
iRobot is a great robotics company. They make something called a PackBot. It's about yea big, and it's a bomb disposal robot, and the way it works is that you would have somebody teleoperate that thing next to an IED on the road in Iraq, and then they would set off a small pipe bomb and the IED would go off, and the day is saved.
So during the Iraq war, business was booming, so to speak. And they sold 10,000 robots, which is a pretty good number. But if you think about it, there are only specialists that use this robot - explosive ordnance disposal teams. And they sold only 10,000. There are actually many, many millions of soldiers, right?
They built something else called the Warrior, it's a badass robot. It's huge. Yea big. Big enough to drag somebody.Big enough to have a huge arm on it. It can knock down a door. It's also small enough that it can go upstairs and go through doorways, which is really important. And everyone at the company is like, "Oh my god, this robot's amazing."
If you think who it's going to be sold to, it's sold to anybody that's in a squad - the Navy, the Army, the Marines, whoever. It'd be any soldier's assistant robot. It could do all these awesome things. But then they decided to cut the funding from it because initially the funding came from the government.
Then that dried up, and they decided not to spend internal money on internal research and development dollars. They thought that the stock price would get hurt because the R&D ratio to revenue would be too high. And just anecdotally, my options were underwater the entire time and they would be still today.
So while they maintained their stock price, like they did a very poor job of long-term driving growth. And I think the core problem here, they didn't start with a proper goal.
The goal for public companies cannot be, in any reasonable way, cannot be "Oh, let's make sure our stock price is maintained." That's not a goal to align a company behind.
How about like selling the most robots? That would also mean making the most money.
Does anyone here know who General Atomics is. Anyone heard of that company? No? That's kind of surprising considering you've probably heard of a Predator. This robot which is incredibly popular, has killed many people, and has sold billions and billions of dollars worth of robots. And the way General Atomics went about making this was so badass.
They just saw the future, and they thought, "We're going to need to make this." And they didn't go through the procurement process of like if you've seen the stories of these fighters that take just billions and billions of dollars to make, and years and years, they just built it.
They saw where it needed to go and they built it. They made an incredible amount of money on that because they focused on the long term value for the company, they focused on this goal of making something amazing, of making as many robots as possible.
At Tipjoy, I already mentioned this a bit, Tipjoy was all about micropayments. And the idea was, we could see the idea in a few echoes from different companies, so we focused a lot on interaction design because we thought that to make micropayments work. We just needed to just make it really easy.
Just make it as simple as possible, and we also had some interesting things for an API for developers. That's kind of like what Stripe is doing to make it really easy for them. And also for like Kickstarter, I think there's some echoes of a community there, that we really wanted to support things we loved.
Overall, it's nice to see all these other startups that are doing better than we did about tackling this space. But when I think back about the problem here, actually payments are difficult because you have these merchants and then these payers, and to get that ecosystem working is quite difficult.
We would focus our measurements on performance, so to see like how many transactions or how many just like the total number of users, signups or growth, and we did not dive deep enough to really understand what was going on.
For example, we had this widget that would go on a blog, back when blogs mattered, you could easily give a blog post like a quarter if you wanted to. And in retrospect there was definitely a viral loop there to say, "Okay, somebody becomes a payer. And they're actually a blogger. And they install it on their own blog." And that would have been better if we had actually tried to understand which kinds of blogs and which kinds of users would have really had an impact there.
And, similarly, we did stuff on Twitter. So it's pretty cool. You could tweet "P $2 @ Ev" or whoever, and that would be a payment. It'd be like a command line thing - a social payment. But we had a lot of charities that used this, but we didn't really look at really what kind of users on twitter would have really driven things a lot. And I think this is a core problem about just not measuring enough to know what would really drive growth and so that's why I think we didn't do well.
For Facebook, there's a lot of really interesting stuff on Facebook. Obviously, they did a lot really well, they have a huge, huge user base. So just to talk about a few things. One of them was the study they did that found, they really wanted to understand why people stick around because if you really want to drive retention, you can't just look at the rates, like to see when people come back, you have to look at the why.
And if anyone's working on an analytic startup here, that's the one thing I would love. I want to give you all my data, retention rates and whatever, and tell me what is it that they did that made them stick around? It's like the billion dollar question.
Facebook found that if you have ten friends, you're much, much, much more likely to be retained. And so they built this feature, PYMK - people you may know. You might have seen this, it's like on the sidebar, the suggested friends.
And I don't know if you noticed, but you totally don't recognize these people, they're like strangers to you. Probably it's because you have hundreds of friends and you actually don't need this feature. Who it's actually for, is for them.
They have five, six, seven friends, and they're trying to get this welcome committee of people to come on board and friend them so they stick around. And it's remarkably effective because when they just passed that threshold, then they were retained users. That was really awesome to see that they find the true levers of what they could do to help drive towards their goals.
Another anecdote I mentioned it's really cool to sort of build things in parallel as fast as possible, and Facebook had this really big gong they would ring. It's really cool to help with their shipping culture. They would ring it when they had a big launch like a newsfeed redesign or a profile redesign, and it's a really fun thing to do.
Except that the growth team bragged that they never rang the gong.
The point was that they were always so focused on the iterative approach to solving this growth problem, that they thought if they ever rang the gong, that would mean they did something that just took way too long, too big of a project. And the quick hits are definitely the way to focus there.
Another interesting thing about Facebook is the way, internal to the company, they started taking their approach about how they went about user growth, and applying it to the advertisers. For credits, the virtual currency, they have payers. So you've probably received an email from Facebook that says, "Hey, you're tagged in a photo by a friend." That's an amazing reengagement mechanism.
Sending that email triggers you to come back to Facebook, makes you a more active and retained user. For advertising, if you've ever received a newsletter from Facebook, "Hey, you should check out this new ad feature, we have an API" or whatever it says, maybe it gets some ad credits, they're thinking about that newsletter in the exact same way that the rest of Facebook is thinking about these notifications. That you can drive these hooks to get people to do more with the system and to drive engagement.
That's when I first really started to think that these methodologies could really be applied to almost anything because an ad system and a social network are entirely different, but you can use the same tools to drive things. For credits, I think that they had a few problems there. And it hasn't really captured the opportunity of what it could do.
I think they basically picked the wrong goals. I think for a payment system you need to look at just maximizing transaction volume. And instead they made a high price point that really excluded a lot of people, it's like 30%. They didn't have any subscriptions, they focused on facebook.com, not on the open platform, they were very aggressive with their developers, which was, in my opinion, totally unnecessary and the wrong way to treat your customers.
It's really like a missed opportunity because each of those things, if they were to change it, that would lead to a better system with more engagement and more people paying through them. And so, hopefully, they can still do that because I think it just makes sense for a company with, basically, everyone on the planet to get your payment system in there, as well.
For Dropbox there's a bunch of cool things. One of them is how they measured things on activation. What they did, they looked at what every single paying user did, besides sign up. What's the thing after that that every single paying user did? They have this milestone that says, "Okay, you sign up and then you activate."
And they found that if you have the desktop client installed and you sync a file, that's their definition of activation. And I think it's a really good way to measure things because it's a way of anchoring and dividing up your user base to understand it so much better because, for example, if you sign up from a mobile device and you haven't activated, that actually changes the way you use the product pretty dramatically, and your understanding of the product is really different.
Another great thing they measured was around referrals. This is the referrals page and I've looked at this page a lot because it's just this bottomless well of new signups. If you can optimize this page, it's really high leverage for Dropbox. And there's a bunch of things here.
There is bulk email importing. There is something below that that lets you just type in email addresses. You can post to Facebook. You can post to Twitter. You have that global link on the bottom that lets you sign up. All these different channels that you can go through there.
Any guesses for which feature, if it were used, like which feature is most effective? Adam, you can't guess? No? Bulk imports, yes. But it was not the most important one, actually, because it wasn't done as often. There was a hesitation to give your password, etc.
Actually just raw email is the most important, and second would be the global link on the bottom which you might not think.
But it's email and control of these links is just really important, compared to Facebook and Twitter, which have a good reputation for sharing. I think Dark Social is what people are talking about today. It's why email is still so huge.
So what we would do for any one of these channels? The problem is how do you reason about this? When you have five different things on this page, what should you focus on. And you've just got to break it down.
So, for example, for Twitter, you could look at like a given signup cohort, you look at people that signed up last week, and you say, "Okay, these are our active users. How many of them visited the referrals page? How many of those visitors tweeted? How many of those tweets got clicks, or how many clicks did they get? How many people signed up as a result of those clicks?"
And the cool thing is, if you take the product of all these ratios, you get the number of new signups for the number of users that are in your pool, and that's the definition of viral coefficient. It's like you have a certain number of people, and then how many people do they bring in within a window of time.
Viral coefficient's kind of tricky, like if it just monotonically increases. A given set of users does not, over time, have invited a total number of smaller people. It only goes up. You have to be careful with your balance of timing here. But if you do this, not only is this obviously a set of ideas.
Okay, let's drive more people to the referrals page and that will make the first number go up, or maybe let's do some copy text or some editing on the actual tweet, that'll get more clicks. So it drives you to say all the things you could be doing to make this one channel better. Then because of that place in this whole funnel, you know exactly the number at the end that would be output.
So it's a way of talking about things like bulk importing versus your Twitter channel. And the answer is this unitless ratio of how awesome it is - this viral coefficient.
And once you break things down and can reason about them well, then that's what enables you to do things like measure the cost benefit. So the answer for any of these tasks would be something that you can compare and so that would let you compare a different task you need to do.
I mentioned about asking and automating, and Dropbox made this product feature that was called Camera Upload, and it's pretty awesome. It would be awesome if like everybody just took their camera and plugged in their SD cards. Then opened up that in their folders and dragged them to their Dropbox and then waited for that to sync and then, cool. That'd be amazing if people actually put all their photos in Dropbox and so they automated it.
You plug in an SD card, it'll suck up all the photos and put them in your Dropbox. It'll ask permission, it'll be nice, it'll scale well too, so there's a lot of effort that went into it. And it did the same thing for your phone, so if you plug in your phone, but also on the iOS or Android applications, if you just take a photo and open the app, and it's even better on Android, if you just take a photo, it'll upload that to Dropbox.
This is amazing for engagement. It's like surprise, surprise, if you just make something automatic, you just get thousands and thousands of photos in there. I think the first time I used this feature, when I plugged in my camera, I uploaded, what was it, six gigs in 20 minutes? So when you think about the three tiers and the conversion to subscribing, sending out referrals, it's a huge amount of engagement, it's awesome.
There was a problem, though, which is that it took forever. It took way too long to build. And it's debatable whether it took like a normal amount of time or too much or too little, but I think this actually has to do with a big problem around agility versus quality.
I think there's a one-dimensional scale here, where if you focus more on agility or on quality, and you take longer to get something done and you get it right. Then by sort of definition you're doing less in that same amount of time, and that's a problem because maybe some markets or some competitive environments are such that you actually need to move faster.
This is something you need to be very well aware of, and Dropbox would apply the same product development process that makes the desktop client so awesome to everything else. I think that sometimes you just need to move fast and break things.
So one of the products I worked on was Shmodel, which was a new sharing model. It was a concatenation of those words. One of my favorite features. So what they had before we made this was public links, and that was an easy way to get a link to just anything, any one file in your public folder. They also had shared folders which were more persistent collaborative space, and shared folders were... people talk a lot about like the referrals in Dropbox, and those were huge, but the shared folders were actually equally huge.
They're a really amazing channel for growth because if you think about somebody that's invited to join a shared folder, there's all this content already there, and they have to sign up and get the desktop client to get all the files. It's really great for not just signups but activation because you have all that content, and the persistent collaborative space, retention.
But Shmodel was something new. It was the ability to link anything in your Dropbox, not just a single file but a whole folder, and also it lets you have like a web experience to see that stuff. So there's a collection of photos or even a whole hierarchy of folders it was a really great experience. But I think even though that was good and really fun to launch, there's a problem in discovery of this feature.
And so I would argue, as far as the scale of agility versus quality, any effort we made to make it really good on the web experience side or what have you, it wasn't worth it because there's just the core problem of getting more attention to just creating these links. This relates to some recent updates that kind of predict where Dropbox is going here with one of the hardest product design challenges that they have.
So Dropbox is amazing, it just lives in the background, does exactly what it should, and that's awesome. And from an engagement perspective, Dropbox is horrible, it just lives in the background, you never see it. So it's like, you can't drive things the way you'd like to. And this is a recent change they made in the top right there, of a new menu that'll open up, hopefully, at the right time, to have more chrome.
Also if you look at the mailbox acquisition, just as a caveat, I don't know anything that they're planning, but I can predict that certainly they'll make an iPad app at some point. And also if you think about the way you use mail, it's entirely different than Dropbox. It's something you have open all the time, and that level of engagement is entirely different than something that just works in the background, let alone that you care about your email as much as you care about your files.
I'm very pleased with this, I think it's going in a great direction, but it really highlights this issue of what I call surface area. If you could think about the engagement points that you have with your users as touch points, like a surface and if you could expand things out, then you're more likely to have engagement. And so Dropbox, just as a desktop client, doesn't have many touch points. If you expand that out, it's an easy way of thinking about driving engagement.
I'm working on a new startup. It's called YesGraph. I won't talk much about the product. I don't think we have time besides to say that it's an enterprise product with a subscription. that implies a bit about what we can do to grow. This is just the ways I think we're going to grow, I'm pretty excited because there's so many of them. Just to delve into it a little bit.
If you have a high price point for a product, like an enterprise product, then there's a bunch of things you can do like ads and press and sales that you can afford to do, basically. Ads for example. You have to compare your customer acquisition costs from an ad channel to the lifetime value you have from the users from that channel.
So it's pretty straightforward. If you have an ad and you pay for a user. And they come in and they give you more money, that's good. You should put more money in that ad channel. You have other issues like the time it takes to get that money.
Maybe if you're not like rich with VC, flush with cash, then you don't have that much money to drive this. If it takes 12 months to earn back the money you get eventually from that user, you can't really drive this as much as you want.
Another issue with ads generally is just saturation. There's, I think, a limit to the amount that you can get from ads. For press, sales, and blogging I really want to highlight that even if something is a human process, you can still instrument it and understand it and drive it.
For press, people pay for PR agencies, but I think they do a few things, some of them poorly. They try to tell you the impact that press hits have, but you actually can measure that much better than other folks can, like a third party.
You can see all your referrers, you can see your direct traffic increasing, and then you can see how those users converted to paying users or what have you. That's the benefit side. You can see what happened from those different traffic sources.
On the cost side for press, press is all about relationships. And, so, what you want to do is get in touch with many journalists. Tell them your story. Try to convince them that you're riding on some kind of trend, that there's something interesting going on here, and that's basically what PR folks do. They're just very good at getting in touch with this huge network of journalists and it's something you could certainly outsource, but it's not that hard for you to do and you can measure it the same way.
You could say, you know, just track the conversations you have and see. For different categories like tech journalism or print, which ones had the most hits and that drove to the benefit. And, really, just organize your time around that because I think a lot of people, especially, in early start-ups, they think, "Oh, I'll get on TechCrunch." Maybe that's dialing down a bit.
It's totally the wrong mindset because it's as if there's like one thing you need to do. Now you need to be on TechCrunch and everything. You need to like drive it as much as you can and then results what the results is. And TechCrunch posts so much now that I don't think the result is what it used to be.
In sales this is especially true because for sales it's a lot like ads, but you're not paying to place an ad, you're paying for a human to sit there and, you know, be on the phone and write emails and what have you. You need to look at the most important thing, I think, which is the time it takes for a sales operative to make their own wages. Right? Like how long does it take to be efficient enough at sales that they'll be ramped up and start bringing in more sales than it cost them to be hired.
This is something that you can certainly drive as far as the quality leads and what you do with them and how it takes to go through the funnel of sales. But also on maybe your onboarding. Like does it take three months to ramp up? Maybe you make that two? What is learned in that three months to make it shorter? And then your hiring process. Think about that as well.
Like what kind of people are you hiring, and if you're scaling it out really far you can actually do a really good job of identifying exactly the kind of sales people work well. Maybe it's different for large organizations or smaller. That's where segmentation matters. But just because it's a human process you're talking about here with sales doesn't mean you can't instrument it and drive it in the same way.
So for blogging, what we're planning on doing is a fair allotment where we can log in a broader sets of things. There's going to be a lot of sort of earned media that we'll create and there it's a lot like press. And we're going to put time into making this content.
Then we can see, for example, the hit rate of how am I going to get on the top Reddit or Hacker News or we're picked up by other downstream publications and see how much that buys us as far as the time it takes to put effort instead of maybe on some product side or some engineering effort. But on actually creating content and then the benefit we get from that on the other end.
Virality is something we already talked about a bit and so I'm very happy to in a position to where we're kind of a social product, but also in the enterprise. So there's some viral channels to it, as well. And in the same way that we might optimize like the referral space for Dropbox.
We can invite. We can optimize the invite flow, and onboarding, and all that and the time it takes for people to become viral, as well. There's a lot that we plan on doing, and it's a question of optimizing all of these and comparing them to what the impact will be. But, overall, it's pretty exciting to have so many options that are on the table for most of you, I would be, as far as the kind of products you make.
That's it. It's a bit of a whirlwind, so I'm happy to answer any questions to delve deeper into any of this stuff. And, again, if you don't have a chance to ask or chat with, email email@example.com. Feel free to get in touch.
I think it's a matter of your role. Right? So it's kind hard at a start-up because at my start-up we have three people who are actually here. There's Kenneth and Vincent. So chat with them. So, we do different things, you know. So, Vincent and I are engineers and if there's anything as far as like something out, we need to balance that out.
I'd think you'd look at the whole process as a whole. And people do this anyway to look at their agility to see if you're working on enough things, and if things are getting done fast enough. And the biggest thing you could is hire. You know, because you can't easily change your habits when it comes to like the different roles there.
I do think that there are cut out roles for a growth team or for engineering team as a whole. And I think there's like a data engineering side that's more about managing and warehousing. There are all the data and managing graphs and answering questions. Or building systems that can answer. There's a component on product management that's really, I think, product managers. It's fairly easy compared to engineering.
As having done both, I can say that it's much easier. And there you can spend, actually, a fair amount of time modeling things out and trying to triage all the things you could do. And then you have just like product engineers that build things. And, certainly, as you get more sophisticated you could have more data scientists to really drive the kind of deep insights you need. But I think most people have much lower hanging fruit they can work on.
Well, oftentimes if you don't have very sophisticated data tools, it's actually an engineer that would need to answer that question. But as you get out there and there's some open source things too, like Hive, for example, it is this big distributive data base you could run queries off of and you can run like a jobs that can be... anyone that can know SQL can answer that question.
And so we had some folks at Dropbox that I think were awesome that were on like the biz ops side that work from an iBanking background that could do some of this data crunching to answer these questions. And at Facebook, they had some analysts, too. So these are not product people. They're not engineers. But they could still answer the kind of data questions that drive things. But I think it's hard to answer outside of any context, but I think, generally, there's usually some really simple things you could do to start.
Like one thing I would do if I was coming as a consultant to a company or if I was advising on what one role they should do to make them more data driven is take all the things you're planning on working on and estimate the impact. And just to get like that data driven culture imbedded within your company, because I think if you were to take your task lists that you have right now, a lot of it would be fairly unjustifiable. You know?
There's even an anecdote from this where, I think it was Twitter that had this issue where the sign-ups were going down. So registrations on web landing pages were going down pretty dramatically, and they also had a shitty Android app. So it's like, well, what's shitty mean? How bad is it? What does that mean for like our long-term engagement with those kinds of users. Really hard to estimate. But on the other side you have sign-ups. That's like really easy to say we're fucked if that continues.
And how do you triage that? The answer is that, at the very least, you have this common currency to say, you know, somehow try to compare these things. Unfortunately, it's not necessarily cut and dry when you're dealing with some things that are aesthetic. But unless it's like a core part of your brand to be well- designed, I would kind of shun the aesthetics side because you can point to like some kind of retention or maybe work of mouth thing if it's like really good experience.
You can usually point to something like that to say this aesthetic thing is actually going to pay off in this way and that. The actual cause of the sign-ups going down, by the way, was that the user name space of add user name was getting filled up. So all these people, like, would be putting in things and like not finding any hits. So they changed the order into putting your name and email address.
They had your email and then from that they had a bunch of suggested names that you can go for. So you just took up that hang-up. But that took some analysis to find out what was going on.
That's another thing. It's often very reactive. It's like shit's on fire, what are you going to do? You have to solve it and to that you have to understand what's going on.
Yeah, I think people get hung up on the growth side of it, but a lot of things are fairly analytical and data driven. So I think you need to go back to more core skills.
Like managing large amounts of data if you ever worked with like any reasonable data base that's large scale. That's a really good skill for the data side of it. And like I mentioned, the analysts that had like an iBanking background, they're really good.
I think the primary characteristics of those people and also any good engineer, I think, is the speed at which you can learn new things. So that's actually good news. Right? It's not like you're looking for some special growth, dude. It's really all about just the core skills you'd expect in a start-up - like learning fast and moving fast.
I do think, as far as design and product, I think one insight from Dropbox I really like. It's only tangentially related to your question, but people call Dropbox, like, very clean, very beautiful, simple interface. They really praise it a lot. And the good news is that they didn't need some Da Vinci-like designer.
They just smoothed out all the rough edges, which is an incredible amount of work. When it comes to some, you know, Swedish version of Windows 98 was having some bug. Solving that problem times 10,000 is actually how they made it good. So that's really good news, as far as, like, hiring the right product people because you just have to put in the hours.
Yeah, the nice thing about spreadsheets is that they're so flexible. So if you have a list of anything, put it in a spreadsheet. But that's not going to get you the data input. Right? So it's usually really good to have a simple data store for raw events and then some tools around managing that. And you probably want to roll your own.
I found, for example, Mixpanel, and this is kind of a controversial opinion so I don't know if it will be proven true, we'll see how it goes at YesGraph, the effort that it takes to pump all your data into a third-party tool is basically the effort that's required to just manage your data anyway. You're not going to have a user waiting on a response from Mixpanel when you ping them. So you're probably going to have some kind of queuing system to do this in the background, which is basically all you need to do to instrument yourself anyway.
Then on the other side you have processing that data and there's a lot of tools for that. And some of them I can recommend, but I think there's actually a bit of, the opinion gets kind of controversial, I think, you know, I talk about technical debt when it comes to you just move fast and you get things done and you like accumulate these problems. I think there's actually something related to that that I would call process debt.
And if you, for example, just like throw a line of JS, if that's all it takes to get your data into a tool. You don't know how to answer certain questions. So take Mixpanel, I'm going to harp on them because they're doing well, and so I can. To answer a question like on a social tool. How many friends does somebody have a week after sign-up and how does that correlate with retention - like a basic histogram? I have no idea how to make that in Mixpanel. I don't think you can.
But when it comes to managing your own data, the costs, like the process cost around answering the question, "How many hits did we get on this web page?" is basically as hard as answering that question about other data. It's just you have to get it all in one place, manage it, do the math on it, and spit out an answer and maybe put it on a dashboard. So I recommend rolling your own.
As far as tools for that, I mean, we've been using MongoDB, but I don't think that will be necessary soon. I think that... I just learned that Postgres has a really good key value store within it, so we'll use that, probably. And, I mean, Vincent probably can talk more about this than I can because he's the one that set up our Mongo for our task tracking.
Then when it comes to managing data, it's surprising how much scale matters because adding up a bunch of numbers is trivial. Obviously, there's no like data signs, no math on that, but doing that when you have a billion numbers is actually really hard. So anything is easy until you scale, or anything as hard as scale. And then you have to have more and more complex systems.
For visualization, I can recommend Rickshaw or Highcharts, which is probably a better layer than D3. But, then, something on the Web is good because you can share that easily. Another tool, a lot like spreadsheets, is email. For example, one of the cool things about a Dropbox, which is a daily stats email, and I think this is kind of common these days, but just to know like what's going on. The core.
You pick a few core numbers and the numbers that drive those core numbers, "How're we doing day to day. Let's get it up on the same page." So having a dashboard that's takes data, massages it, creates some kind of cache of it and then puts a graph on a web page, that's actually not that hard to just say think data massage it and generate an email every day, take a snapshot of the page. So these are all really good tools.
There is one thing I'm missing when it comes to data and it's really answering the harder questions easily. Like it's just really hard to do that, and there's a bunch of tools in whatever language you want. And even if you want to break out of what you're probably using production, like [R], there's like even more tools there. But some of these questions are just really, really hard to answer. Like the deeper questions like I talk about like why something is going on.
You often don't know what to ask. So you could answer any questions and you're just like, "Well, I don't know what to look for."
So one thing that I heard, at least to solve that problem, but I don't know how much I trust this. You can take all these things that you think might be a driver of retention, say, and for every user you can have, like, they did this or not that. So it's sort of a profile of a user by whether they did it or not. If you have that times a million users, and it's also a big matrix, and then you have whether they were retained after months is like another vector.
There's a bunch of ways to solve that system of equations. Ya'll remember AX equals B. Right? So, yeah, you can actually get like a set of like coefficients for each one of those things to see what might be driving the retention. But that assumes linear systems.
So this is another thing. It's like you have pick them. It totally assumes a certain model. So the math here is hard and there's no clear way to answer things. Oftentimes for any given learning task you have like the average voice of a dozen algorithms that you don't understand well. So it's like, yeah, you have a bunch of answers, but what are you going to do? It's really hard.
And this is where I am hopeful that more advanced analytics tools will be able to come and solve this problem. But I think it's basically a solved problem to manage your data and look at a graph of it. And you can do it yourself or have a third party. I expect for us my plan is to do it more ourselves so that we're actually really accustomed to working with our data so when you want to answer the harder questions, we're ready.
Yeah, I would say it's a lot like what I described about building intuition. You just need to try it and look at it.
For example, I'm sure a lot of you do this. Every time there's a popular app; you should open it up install it, and see what it does. Just try to get good understanding what it is about it that makes it a good experience. Also, when you do it yourself, I think that you build that experience too. So it's really practice that gets it.
From my experience, and sort of it's kind of obvious. I didn't take like a class on this. I didn't read a book. It's actually a distributed set of experiences that I had that came together to make what I know and everyone's different. That's generally how you get good at anything is practice. That's the good news. Your brain is not static. It's like you just work on something and you get better at it. So it's good.
But I think there's actually a pretty large community around this growth stuff and some of it is a bit, again, distasteful. It's just there's something very arrogant about calling yourself a hacker - it's weird. But I think, overall, there's actually a lot of data out there about like different tactics to try. There's examples that you can get left and right. And as a user from products. If you're just thinking about things in that mind set.
Let's take something totally different like the brand, like the tone that you, like, the feeling that you have when you read the copy text of a landing page. That is nothing to do with growth. Right? It's just something totally different. And how do you get good at that? Well, you write and then you read all these different pages to see what's the feeling that you have when you read that? And it's just something you can get good at by doing it. You just kind of have to know to look for it.
If you just try to think about what they try to make obvious and drive it. And, especially, services that have grown really fast. I think LinkedIn is a good example. It's actually a disgustingly ugly service when you think about this hodgepodge of features they're thrown together. I'm competing with LinkedIn, but that's beside the point. But they've done a really good job on the data science side and driving things.
Like a really good example recently is what they did for endorsements, which I think the signal to that is very, very low, but the product design to make something clickable and usable has been praised. So if you just keep your ear to the ground, I guess, about examples of this that a really good design, like Growth Efforts, that's a good one that comes to mind, and there's a whole community of people that try to answer this. Quora is a good site, as always.