May 25, 2016
The Evolution of Dark Launching on Snap CI
Badri Janakiraman shares the story of the evolution of dark launching on Snap CI
Awesome thanks! I currently work at Heroku. I run product for our Ecosystem group, which is our APIs, our CLIs, add ons, and languages, anywhere we have a third-party community that we create, or we engage in one. I don't have a background in marketing, at all. I'm a developer, and probably a pretty bad one. That's why I do product things.
I stumbled into marketing by accident. I'm really, really lazy by nature, that works well for being a product person. But about three years ago I was evangelizing Postgres internally at Heroku, .
We run Postgres as a service, one of the world's largest fleet of Postgres and it's a great database. I worked with it previously, but I sounded like a broken record, internally, talking about why Postgres is great.
Because I was tired of having this conversation over, and over, and over, I said, "Okay, I'm going to quit having this conversation and I'm just going to basically write an email that I can send around internally." I spent 30 minutes on a BART ride, writing this post, published it the next morning, and it hung around in the first spot of Hacker News all day long.
From there, I got recruited to the Postgres team. At the time, I was doing other product stuff at Heroku, and the GM said, "Okay, you're doing this marketing thing better than we are, and you're doing it in your spare time."
So, out of that, I'm saying, "I don't do marketing. That's a whole other group of people." But, he actually convinced me to do it. The rest of this falls on,
"How do you take content and then utilize it for a broader, altruistic way of thought leadership and really marketing without it being the big M marketing, with banner adds, etc?"
This one post, just as an example, started that. Since then, it has really seen steady, consistent traffic of about 50 thousand people over the past three years, doing nothing else for it. I imagine if we were actually trying to do something with it, we could do a lot more.
On content, has everyone watched, or listened to, or was at Danielle's talk around content marketing? Okay, a few hands. If you haven't seen that, start there. That's the perfect place to start for content. She covers a lot better than I can, but I'm going to give you a couple quick Cliff Notes if you're going to start there.
Sourcing content. This is the big area I want to hit on. This is the number one thing I hear from marketing people and also anyone that's trying to build up a body of user base and readers,"How do I get my engineers to write content? Where do I get content from?"
The obvious place is start with your product. Talk about the product benefits. Hopefully you're doing this. This is step zero. If you're building a product and not talking about the benefits of it, I'm shocked you're building the product. You should be talking about the problems and what it's solving.
From there, there's a few others. Support tickets is a favorite one that I love. Support tickets are great beginner content. As you build your product, you become deeper, and deeper, and deeper in it, and you start to lose that beginner mindset.
She talks really well about how you only have that beginner mindset once. You've got great insight into all of your newbies via support tickets. What does that beginner mindset look like? How can you take those things and turn them into not just a reference or a support guide, but how can you turn them into actual content for other beginners, more broadly and recruit people in?
Email, this is one of my personal favorites. Internal emails among engineers. If engineers are reading it and talking about it, it's probably good to be shared more broadly, because other engineers are going to find it useful.
There's one employee from Heroku that every email he would send, I wanted to take it, put it in a JIS, and post it to Hacker News because it was valuable, thoughtful content. This takes a little bit of cleanup to make it publicly consumable, but there's not going to be that many trade tickets hidden in your engineer's email and these are great engineering leadership.
More generally, engineers, especially if you do a bread and circus, or show and tell on a weekly basis or if they're doing code reviews and swapping tips. All of these things are great places to source content.
I walked by our conference room this week, and all of our buildpack maintainers where talking about crazy bash tips they'd found within working on their buildpacks. To me, this is that thing I want to read about. I want to hear about how Bash is horrible, amazing, and powerful all at the same time. I would love to read 'Fifty Things that you Didn't Know Bash Could Do or that You Should Never Do in Bash, but You Can'.These are the places I look to source content and really where it starts.
A few specifics. Talk to your customers. This one I picked up from one of the Heroku founders, James, who's hiding in the corner there.
Use you, not developers. If you ever say 'developers' and you're building a developer product, you're doing it wrong.
If you use you, talk to them. There's a one-to-one relationship there. You want to talk directly to them. At the end of that line there's a person, and a body, and a developer that you want to have a more direct relationship with.
Be classier. Danielle talks about this, but as a company and as a product, you have a target on your back. You're known. You can't kind of hide in the anonymity of the internet. You have to take that high road. There's also a nice validation if you do do this. You feel better about yourself along the way. It's nice, and kind of helps maintain that journey.
Be altruistic. There's a lot of easy ways to hate on products, and developers really do like to hate on certain things. But being altruistic, talking about a grander vision while it's not the popular opinion, they'll usually line up and follow it. This works pretty well.
Onto evangelism, for just a quick minute.
My approach to evangelism is pretty simple. Basically, help people. The goal is to help them. Make them successful.
If you make them successful, then they're going to want to appreciate support and like your product. It's a lot harder for them to ask for support then it is for you to reach out and give an answer.
When we launched Python support at Heroku, a thing I did that I was a little skeptical about at the time was give out my cell phone number. So, all of our first 50 users I gave my cell phone out to. I think I've had one person text me since that day.
That general gesture of saying, "If you need me, you can get to me," is huge for customers. If you think about this from an engineer perspective, a lot of your engineers are wearing pagers. Why can't customers call you, if they need it? This extra sign of effort goes a really, really, long way.
I would basically give them any method they wanted to get in touch with me. Cell phone is the highest form of touch they could get, then IM and email. Basically any possible way.
It definitely went a long way. Some of those first adopters have gone on to write the first books about Heroku, built add ons, and have basically continued to stay in that ecosystem and reengage over, and over, and over. Prior to that, they had no real relationship with me, or the company.
Another piece to that is when you start to build that personal relationship, they're going to like you as a person. Or at least hopefully they do. If not, well, be a nicer person.
Personally, I'm not a people person at all, but I find this does a lot to help improve the relationship people have with your product. If they like you, they're less likely to leave your product. Just out of a sense of loyalty and friendship.
This works really well at that creating advocate stage. That's where I kind of view the evangelism piece. If you can create those advocates, you get an economies of scale versus doing this all manually yourself.
So onto a little bit more of the meat. You've watched Danielle's talk, created awesome content and gotten fans that are creating awesome content, but what do you do with it? There's a lot of places content can live, and most people take and shove it into their existing website. In some cases, this can be great, but you're losing a lot of leverage here.
The first thing here is around timing. How many of you have heard this from marketingor if you've worked with a big marketing department, "Every launch has to be a tier one launch." If it's a tier two or three, they're OK. What can we add in there? What can we ship new to make this a tier one launch?
The other piece, as soon as we decided no matter what it is, it's a tier one launch, is, "When do we launch this?" It's always Tuesday through Thursday. Tuesday is the biggest day for traffic. If we can get it on Hacker News' number one spot on Tuesday, we'll have this many page views and we're going to have this many sign ups, and that's great.
That is pretty much every marketing department to over generalize, but pretty much every marketing department that I've ever worked with.
Now in reality, shipping and launching things can be broken up a lot more. Expanding on launching, I still actually hate Monday.
Monday is not a good day for launching. Your engineers are going to be worried about it through the weekend. You're going to be thinking about it. You're not going to sleep well. It's just going to create other problems that are unrelated to audience.
Friday, I actually love. The way I approach this is, if I can't get it on Hacker News on Tuesday, but I can get it on Hacker News on Friday, that's great for reach. If Twitter volume is busier on Tuesday, but quieter on Friday and if I can launch a tier two or tier three- a smaller product update- I can get eyeballs that I wouldn't otherwise get. Other people are competing for those, so this is a really nice opportunity.
The other one is weekends. I actually love launching things on weekends. Not because I want to be working on the weekend, but because I can make an announcement, tweet about it, and post it to Hacker News. I've done this before where I went up to Sonoma, and came back to a lot of responses and engagement.
This depends on how much you want to engage your customers during this time. Do you want to be discussing it in conversation? If that's the case, obviously you may not want this. But for smaller, interesting engineering blog posts, thought leadership, etc, it's really easy to do. Weekends are quiet, but there's still developers reading and engaging with your content on the weekends.
You can produce all this content, too ahead of launching your product. So, some of this thought leadership stuff, you can be building up a stream of. Don't necessarily wait on this.
A little more tactically on the third party channel piece, I view this two ways. There's beginner guides and then advanced content. I've seen a couple of different approaches, really different approaches for each of these, and you should be doing both.
On the beginner content piece, really what you want to do is have your content live on a third party property. I say there's one exception to this if you are an 'e-learning' site of some kind, like Code Academy, etc. You want all this learning content to live on your site.
Heroku was really interesting from the early days. We had this charter of wanting to be the place you'd come to learn to build apps. That was great. That was altruistic, and that was big. If you were building a web app and wanting to build it the right way, we wanted to be the place to teach you how to do that.
Today, I would argue that's not as much as what you find on our dev center, our documentation, our guides, because it's simply hard. You focus on product documentation over time.
As your product grows and becomes more complex, you're going to want to focus on that, and then how do you get other people engaged and contributing? There are some counter examples to where this can work. But for the most part, over time, I see as your product grows, you're going to move away from saying, "Alright, let's teach everyone about REST APIs," because it's a hard uphill battle.
This doesn't mean that you can't have an influence on it and your name can't be tied to it, whether sponsoring a third-party site or creating a third-party site that's not under your brand. Heroku actually has a great example of this in 12factor.net where it's clearly tied to Heroku, if you know anything about it, but it's not a heroku.com property. There's a lot of these.
I'll talk a bit about how this actually works, worthwhile for you in the retargeting bit, in a minute. I did one of these as an accident. The Postgres tutorial is 200 pages, and it's archaic. No one reads it and it hasn't been updated in 10 years.
I started to say, "Let's go update the docs in Postgres," then I decided I don't want to touch this horrible documentation thing, so I thought let's create a new one. I sat down on a flight and wrote Postgres Guide over a couple days. It's up on github, it's open source, anyone can contribute.
There are a couple of pieces here.You can clearly see the Heroku tie right in there, call out to Heroku Postgres. There's a tie out to myself. There's advertising on it whether you like advertising or not. If you want it, there's a free opportunity to put your advertising directly there.
The other thing is, there's retargeting on here. So this is a really focused audience of people that care about Postgres. This is a perfect audience to target for Heroku Postgres or any other Postgres activities. You could imagine doing this for Rails, Ruby, or all these other things as well that are a captive, targeted audience, that we can pixel and retarget to and do interesting things with.
So, I spent some time. I created some custom documentation, but was this worth the effort? I think that's what it really comes down to. How much is it worth to spend time doing any of these things? Over two years, it's seen 150 thousand unique viewers and 400 thousand impressions. The amount of time I've invested in this, was about 16 hours of initial effort, and then occasionally checking some pull requests.
Net total, I have not put more than 40 hours in here. To have a potential reach of 150 thousand people that we know are interested in Postgres, that we know we can target our product to, to me, that seems like a pretty good return.
John Sheehan actually talked about a show and tell, they've got a lot of these properties that they retarget to. Their cost per acquisition of users is two dollars, just for retargeting to these sites because it's a very captive audience, well worth the investment and the cost.
Just a quick look at growth over time, too. It wasn't an initial Hacker News effect. To be fair, I did chop off the initial Hacker News effect, which was right before that at about 15 thousand views. But since that time of doing nothing. It's continued to grow because other people have talked about it, contributed to it, and feel it's a worthwhile altruistic thing because we're improving documentation for Postgres as a whole.
A few other examples. There's one company that does this amazingly, Google. Google makes it really easy for you to come to one of their projects and not know they have anything to do with it. Angular, I think, is the highlight of this. Maybe Polymer will get there soon. But if you go and look and Angular, you have no idea it has anything to do with Google.
To contribute to it, it's a pretty archaic CLA. But overall, you can go there, read about it, and engage with it, and think, "Wow, this is a great community project." Meanwhile Google is very heavily engaged in it.
12factor.net is a Heroku example. This is, I guarantee it had broader reach, because it wasn't on heroku.com, but talks about the principles that really inspired the platform and what we were trying to create at the time. Much of 12factor is codified in Heroku, but it's not directly a Heroku property. People come and see that as thought leadership and say, "Okay, they obviously wrote this. They know how to run a platform. I trust them with this sort of thing."
I've also seen this on the Postgres side of things as we've evangelized and talked about Postgres. People come to us and view that as a check box of 'we know what we're doing' because we're talking about this beginner and expert content out there.
In terms of process of how does it look, I say it starts with the beginner content. From here, you can pixel people. Basic retargeting pixel, hopefully that makes sense to everyone here, but basically just pixel a site. Then you have that audience and you can do whatever you want with it. The pixel works at about a thousand people.
If you can get a thousand people a month to your content, which is not that hard to do, then you've got a worthwhile audience you can target to. From here, there's a bit of a dirty word to developers, but you can target this pretty separately.
You can basically say, "I'm going to target all these other properties about a webinar," so you're not alienating the developers that don't want a webinar. You can also call this a webcast, or whatever you want, to make it friendlier to developers, which actually does help and work, and really get a higher level of engagement.
People to follow up with emails. If you want to start doing direct sales around that, you can also target directly for your product. You can do this from within the webinar. You can do this from the pixeling. You can do this from a soft touch around the beginner documentation, as well.
Then on the webinar, people will notice that you're the people creating that content, that you are espousing the best practices that are out there and view that as thought leadership, which is easy to do. There's nothing mischievous about this because you're being very transparent about it, but most people won't dig in deep enough to say, "Oh well, clearly they're doing this because of this because this, and it works." They also appreciate it because you spent the time creating that content. They'll give you a pass on that side.
On the other side with advanced content.
Beginner content is one that I think most companies, as we build our products, start to pass. Advanced content where we stay more excited about, because it's the things that challenge us, and challenge engineers. Creating that outlet for it is a little hard.
The first thing I would say is create an outlet of your own for some of that advance content to live on. This can be engineering.mycompany.com and just let engineers run with it. Don't restrain this, just encourage content to get out there.
It's really easy when someone sends an email saying, "Hey, can you turn this into a blog post? I'm happy to review it." Even if they're not up for doing that. You can do this on their individual blogs and offer to support and encourage some of that thought leadership.
One note here, if you don't already have a newsletter, give the engineering blog the ability to subscribe via RSS, and also via email. Accept emails. With mail temp, you can easily go from RSS to deliver an email every time there's a new blog post.
Although, if you've watched Danielle's talk, you already have a newsletter because it is a perfectly captive audience, so just use that. If you don't have that, this is an easy, lightweight thing to start with around your blog and basically sending updates to it.
In terms of curated news, this has been an interesting kind of growth over the last two years. Really, as Google reader has died, I feel like that's put a good kill to RSS pretty well. There's a lot of examples out here.
Postgres weekly is something I curate and started doing a couple years ago. I work with Peter Cooper on this at Cooper Press. He runs a ton of newsletters. He's built a silent empire of dev focused newsletters that reaches to over a 100 thousand people every week. He handles all of the administrative pieces.
Coming back to Postgres weekly, we're one of the more niche ones. We reach out to about seven thousand people every week. 50 percent of those open on a weekly basis, and about 20 percent of that seven thousand will click on some article that week. I'd say fairly high engagement for not a lot of effort.
On working with these newsletters, I'd say give the curators a heads up. At the end of that, there's always a curator that'schanneling this and handling that channel. Give them a heads up. Get to know them. Send a friendly email.
These days, it actually works wonderfully in that I don't have to pay attention to what's going on with Postgress, people just tell me. They're basically doing this job for me, so that I can say, "Yep that's good. Nope that's not."
Don't overwhelm them on every blog post. If you let your marketing person send them every single blog, it will start to fall on deaf ears.
A good way to judge this is if it's already been on Hacker News, I'm paying attention to that, too. If it's already been on Reddit, I'm paying attention to that too.
Don't send me that newsletter or that blog post. It's there and I've already seen it, it's already going to be included. Then if you send me every single blog post again, it's going to start to fall on deaf ears.
The other thing I'd actually encourage here is own the channel. I feel like a lot of companies think it's a lot of work to invest in itand a weekly newsletter.
Owning the channel means you can set the high bar. It means you can have good quality content. I know when I write content, it's great quality because I wrote it, so I can put it in the newsletter.
This way you know what's happening. You're already following the space. It's pretty low effort to come in every week and say, "Alright, I'm just going to add some sentences, follow the space, here's the links that go in this week."
What's the process look like? It's actually pretty easy. I warned Peter that he's probably going to have about 20 to 30 requests saying, "We need a weekly newsletter around hypermedia APIs." These get pretty neat show. There's probably a market out there that's super targeted that it could make sense to have that kind of newsletter, but you're already following those channels.
I spend about 15 minutes each week, and it's Tuesday night, I actually need to do this tonight, I'll probably do it on the BART ride home, I'll just go through and add some text and write some words to describe the articles. That's it. The rest is all taken care of. It's a pretty lightweight process.
If you're not going to own that channel, which I say is really good and powerful to do, at least be aware of them and engage with them. Get to know the curators. It makes your life easier and their life easier, so they're happy to support you in that way.
It really does start with content. Listen to Danielle's talk. Hopefully the tips in the beginning were fairly helpful and to the point. Beginner content doesn't have to be on your brand or on your product property. I'd argue that it probably shouldn't. You'll have broader, bigger reach, if it doesn't. Google's a great example of this. You can look at the example of 12factor versus where it is in dev center.
Postgres guide. I don't know where it could have lived that it would have gotten more traffic inside a Heroku property, but people know that connection and appreciate it. I will say, it's much more powerful if you do own the expert content and the beginner content.
With beginner content, there's a lot of places for it to live and that expert content, it's harder to have that thought leadership if you don't own the channel. It's worth investing in and it's actually not that expensive, I think as you saw. 40 hours to create a beginner content market. Then the expert content, it's on an ongoing basis, but lightweight overall.
Great, thank you.
I use Perfect Audience. I'm a big fan of them because of the way you can segment different kind of groups and you can do interesting things. I've actually used Perfect Audience, not just for retargeting to new sign ups, but to product features.
If I can create someone stickier and if I know that if they use this feature, they're more likely to have a higher lifetime value, I can note ones that haven't, retarget them for using that feature and then once they're done, they're out of that segment.
I don't think it damages brand, but what I've seen is, it's really hard to continue focusing on that, as your product grows. That will start to become a secondary concern. I've seen companies kill it completely because they think, "No, we need to focus on the product." It's not costing anything to just leave it there.
That beginner content is good for a really long time. Beginner content doesn't become as outdated as the expert content does.
The Postgres Guide, I've literally not touched in two years, aside from merging some pool requests that show up. The traffics only grown.
The other piece is that your brand is going to lessen the impact of that beginner content. People are thinking it's about your brand, it's about your product versus about learning something.
I don't think it wasn't the original intended journey. We definitely got there. So what are the journeys that people take? There's a variety.
There's some that will come in and basically they see that and there's a link right there to Heroku Postgres and they're thinking, "Awesome! I don't want to worry about managing that." Then click. Sign up, right away.
There's also a longer tail of, well, for me this one's been harder to measure, so I would welcome ideas that people have and thoughts on how to measure this. Sales will pull me into a call, I'll jump in and sales will introduce me and they'll say, "Oh yeah, I was reading this just yesterday, awesome. Thank you for pulling in this expert." And I have no idea what I'm doing, but you saw my name, so you think I know something.
It's that validation of, "Oh look, these guys are out there, leading the community. Who else would I want to run my Postgres?" It's possible that there are a ton of other people in the world, and there are, that know how to run Postgres, but because we are creating that content and our names are there, people associate with those names. They recognize it as you engage with customers over time.
That's really on the higher end one. So I think the higher end one, it's been interesting where it's been harder for me to measure that, but it's been a common sentiment that I can tell helps with closing sales deals, or customer retention because of that.
Good question. Now, because this is all Postgress-centric, this is not a core part of my job because I'm not running product or marketing for Postgres. At the time, I would consider this a core part of my job. If I went and did this, if I went and did Postgres guide for two days, and it's now gotten to 150 thousand unique people to it and we drove a thousand signups, to me, that's probably an OK two day investment.
I'd have to actually sit down and run the math, but it feels right. I'd say yes. Product marketing, this is part of my job and now that I'm on other teams running product there, it's a little different. But it's still something I think I have flexibility to do.
I'm trying to think of things I can talk about here. Often times, the things that didn't work, it wasn't that they didn't work, it's that the coordination with marketing from engineering was really heavy. I've seen that time and time again, where marketing wants a certain type of thing and caliber and thought leadership but engineers just want to write something and ship it.
I've seen engineers almost quit before every blog post. I think the push of 'engineer create this blog post', is the wrong direction, versus the pull of, I probably say this being more of an engineer than a marketer, but observing what the engineers are doing, talking about, and working on and pulling that and creating content out of that. I feel that is the right way.
I see every time, marketing's tried to really push, "Let's do this around the content. Let's have this big push." It hasn't been nearly as successful. It's been painful. I'd have to think a lot more on other things that haven't worked. There's been a lot of small ones. Not every post is huge.
I think I learned the hard way about shipping content and product on Tuesday and Thursday, if everything doesn't hit then and if it doesn't hit, that was your shot. It's hard to go back and launch something the next day, once you've already launched it.
Things like Fridays and weekends are hugely useful as long as you don't create a huge process around them and a burden on your team.
I think the content, it's a pretty solved model, in terms of the content. The standard webinar says teach people something, talk a little bit about your product. You know they're a captive audience and they're there for a reason.
Teach them something that is a pretty straight-forward model. I think it's really more around the naming of your webinar that alienates developers and it's strictly that. It's strictly a dirty word to developers.
I think webcast is the best thing I've heard. I've seen debates about this on Twitter, among people that run them and there's not a great answer, but webinar is not the answer.
I have not done those as much because there's a higher effort to go and create that content. Most of the content I focus on and create is a really low effort to create and turn out.
Video, in my experience, is just high effort. The webinars I've done, are because people pulled me to do them versus saying, "I want to do a webinar, please let me do a webinar."
It's about that lower effort content that's easy to create. That's where I focus more.