March 3, 2017
CircleCI Office Hours: End to End Test Automation That Doesn’t Suck
On January 25th, 2017 Heavybit member CircleCI held their monthly Office Hours meetup at our San Francisco Clubhouse. The evening's presenta...
Every aspect of developer and enterprise software startup organizations has been touched by the Covid-19 crisis. In-person DevRel activities, like hackathons and conferences, aren’t currently possible, which leads to a shifting landscape of available engagements and higher expectations from leadership. This panel discusses how DevRel teams are adjusting to the shifting landscape and how you can equip your team for our new collective reality.
Patrick Woods: I’m really excited today to have this conversation, and it’s really cool to see so many founders and DevRels and a mix of other commercial and engineering teams in the audience. I’ll ask everyone if you can just share a little bit about who you are and what you’re working on, and then after that for the interest of context, share a little bit about what DevRel means for your context. That would be really helpful as we go through the rest of the conversation. So, Jessica, how about you kick us off?
Jessica West: I am the Head of Developer Relations at LaunchDarkly, a feature management platform. When you think about developer relations in general– If you look at the title it’s relating to developers. So in DevRel we’re always talking to people and looking for people that are developers that know how to practically speak with other developers, and helping devs better understand our product and keeping that feedback loop open.
Allowing developers to say, “Here’s how I’m thinking about a product. What do you think about this?” And for us to then as an advocate, to advocate for them to our company and bring that feedback loop and then come back and say, “Here’s why this is a great solution for you,” and “Here. We solved this,” or “Let’s help you find a different way.”
Things that we work on– In a number of ways we partner with our integrations team very heavily, and helping with those partnerships and making sure those federations work really nicely. Then activities ranging from creating demos to webinars and helping with docs. The thing that a lot of us have seen for developer relations is assisting with events speaking and the lovely booth duties. It’s just a little bit of everything, from the code, content and community perspective.
Patrick: Awesome. Thanks for that. Jana, would you like to go next?
Jana Boruta: I lead Experiential Marketing at HashiCorp. I’ve been doing developer relations, community-building events and in-person events until March for a decade now, and I’m currently at HashiCorp. I’ve been here for the last five years and we build infrastructure tools, so we have a really big open source community and we also have an enterprise version of our tools. Right now, I’m in Mendocino on vacation. We just finished a big event two weeks ago. I think Jessica explained DevRel pretty well. That’s how we approach it.
What’s really cool is at HashiCorp we actually have a whole org that is dedicated to developer relations, and under that is my team. We do the experiential marketing for our practitioner audience, we have the community team with the developer advocates and the web education, because really developer relations is all about content, education, and just helping our users learn how to use our product.
Patrick: Cool, awesome. Thanks so much for dialing in from your vacation. I can’t imagine a more relaxing thing to do on vacation than hang out with us. Derric, how about you?
Derric Gilling: I’m the CEO of Moesif, we’re an API analytics platform and we help companies treat their APIs and products themselves, everything from guiding customers to getting that usage data. For us in terms of DevRel, we really see two big pieces. One is education, I think that’s really important in terms of helping our developers, whether it’s around content to webinars to speaking at conferences.
The other big piece is authenticity, and the reason why I believe in this so much is because we’re not tied to other goals, such as sales which is commission-based. DevRel is really there just to be that liaison between the community vs the engineering and product teams.
Patrick: Lots of juicy stuff here to unpack. I’m the co-founder and CEO of Orbit. Orbit is a community experience platform that’s focused heavily on developer communities. I’m so happy to be talking with the crew today, as Orbit sits at the intersection of DevRel and community, and lots of other exciting things.
To kick us off, I think let’s start broad. In light of the changes over the past few months, how has the practice of DevRel changed given the context that you shared about your company? What’s different now?
Jana: Where do we even start? It’s interesting, DevRel, you have a collection of a bunch of different activities. Whether they’re online, your learn platforms, like forums, to in-person. Whether it’s speaking at conferences, booth organizing, meet ups, things like that. Everything shifted in February and March, we had to redo all of our DevRel plans and shift everything to be online. We had to really think about “How do you make these online experiences engaging and interesting?” Because we still have our goals to hit. It’s a fun challenge.
Jessica: For our team, the events were just a portion. As a smaller team we don’t have the full resources that HashiCorp does, so we have each advocate focusing on their own area. We just had a portion that was the in-person events, which has shifted, obviously.Everything was very jarring at the beginning of the pandemic, but as far as transitioning to more virtual events, it was a nice opportunity for us to go to attend those.
It actually helped some of our advocates that aren’t able to travel as much participate and be engaged in those types of events, and send proposals, because they could actually go speak at that event because it’s online vs in person.
We saw a mix of some people that obviously did a down turn for the first month or so when everything first started, and then we saw an uptick of people that were actually able to go submit a talk and speak at an event that couldn’t travel before. It was a percentage change, but it wasn’t a big strategy change.
Derric: Just to add to that, the educational piece is always there when it comes to DevRel. But the way that you talk about that stuff before, you had to mobilize the entire team in from sales to marketing to anyone else that needs to do event planning for a large event like that. Now we’re focused on micro events where it might be just one or two people and that’s it. They plan their event independently and get to actually create more of a relationship with those folks that are on that webinar or that podcast, or whatnot.
Patrick: Great points. Thinking about the education piece, it seems like previously for a lot of companies, the in-person workshop was a big part of the educational effort. Has that changed? Do we have the ability to create meaningful connection and educate people in an online setting? How is that different now than it was in a very in-person world?
Jana: Even before the pandemic hit, we were building out our learn platform because in-person trainings, they don’t scale. We would have training day at our conferences or workshops, but you could have max 25-30 students and we only have so many trainers. Versus building out a learn platform that thousands of people can go through and go through the different modules. A hybrid, it’s great.
Jessica: I was going to say, echoing what Jana said, for us the education piece– We didn’t really use events as an education point, we had a workshop that the developer relations team was in charge of for our big event, but that was just one event through the whole year.
For us, our education is a lot more with online content through tutorials, demos and written content. It scales much better and it lives a lot better over time. We’re trying to focus in on saying, “this is that piece of content and the ROI”, so that you can see that lives better over time. Whether that be a blog, technical documentation or a tutorial.
Our team actually was working on a type of work called Guides, which is our learning platform, and we’re working on that in Q1 and we launched that in Q2, which is really exciting. That’s one of those first pieces that we’re working and engaging and helping our community learn and engage, and that’s through best practice of how you’d use feature flags in general and then tutorials. It’s a blend that wouldn’t have been in our documentation side.
Derric: When it comes to these educational pieces, if you think about an event you usually only have maybe a couple of talks that they’re able to do. It might be a little bit higher level or a little bit broad when you start creating stuff that’s asynchronous, someone can take a look at your piece of content offline on their own time, you can actually really target. You can have a whole story behind this piece of content, and then this piece content can fit into the next piece of content and create the whole story behind it, which is a little bit different than what we’ve seen from in-person events.
Jessica: If I could add onto that, which brings up a really good point for building that story, having written content and having the demos, that increases our scope to talk with a lot more people. The diversity aspect, some of them wouldn’t have a chance to go to an event, or even from a language perspective they can translate that. It may not be the best, but they can run it through Google Translate.
Or if your company has the ability to put those in different languages. That’s a lot easier to digest from an education point and having people understand it and go through different series of data on their own time and at their own pace, rather than being in a workshop and trying to catch up with the instructor, feeling behind because they got stuck in some debugging area or maybe their computer wasn’t set up and they could be at a really high anxious point.
Having content in a non in-person format, like an event, makes it a lot more digestible and more accessible.
Derric: Not just wider in terms of audience, now you are targeting a global audience, but also different socioeconomic backgrounds. Some of these events are pretty expensive. If you go to AWS you have to fly there and the tickets themselves are not cheap. Now you’re able to reach an audience that can be really anywhere in the world and has an interest in learning your technology or trying something new.
Jana: I love that point. We converted our in-person event, our European event, and we’re expecting about 1,000 people. Yet we had 8,500 people registered from 105 different countries. So all of a sudden, when you take this in- person event that is really expensive to organize, and you’re flying your employees out and you’re flying speakers out to this digital event. It just becomes accessible to more people around the world. It’s a beautiful thing to discover.
Jessica: Not to mention the people that are working full time jobs during the day and then they are learning things at night, so that really goes back to education piece, which I think is so core to developer relations.
Jana: I actually think after this, again, it’s so unfortunate what’s happening in the world but this shift to really, “How do you build these digital experiences, improve your digital experiences?” It’s been such a great shift, I think we will always have, at least for our conferences, always have a really built out digital component.
So many things from the content we produce, we made about 23 sessions and we have sent everyone speaker kits and now we have these really highly produced videos of our customers speaking, of our ambassadors speaking. It’s on our YouTube, it’s on our resource page, and that extends the reach of the conference so much farther.
Patrick: I’m interested, it seems like the scale of educational content and training materials is unparalleled online versus offline, what does that mean for the community aspect of DevRel? If you think about code, content, and community as some of the three big buckets that DevRel folks think about. Asynchronous education is super effective, but are we losing anything in terms of community by a shift to purely online?
Jessica: I can take that one. I think for community, it’s something that we have to make sure that we distinguish. Community is not the same as in-person things. We have a community that’s online, and through that if you think about more than 10-15 years ago. When the internet first came out we had the IRC, those chat rooms that people were coming in there and talking and finding that community. Napster is one example, there was a channel, a whole community created around sharing music because they couldn’t afford to buy the CDs.
I think it’s really important to not convolute those. You’ve got great tools like Discourse and you can have your own community and sharing information and answering those questions. I think that there’s so many portions to the community, so with these events that we’re seeing in a virtual manner, it’s still part of your community but it’s not the whole part of the community. Making sure that I think that those things aren’t suffering, they’re just changing.
Jana: Yes. It’s like, we of course have been building online communities since the 70s. \There’s been different versions of it, which looks different now, but there’s an aspect of community and those in-person connections that I do think are missing. It’s hard, I wish we were all sitting in a room together with the attendees. There’s those connections, so I do believe that at some point people will want in-person to come back. I haven’t quite figured out how to really get those connections in a digital way that you get in person.
Derric: Besides a Zoom happy hour?
Jana: These people are exhausted. For us, we built a custom platform for our digital conference and so we tried to build in community engagements that way. So it’s chat, people will be able to chat and ask questions. We had this meetup function that you can suggest a hosted meetup where we thought of that as like hanging out at the Hashi cafe, like at our coffee shop at our conferences. We would suggest a meetup and then people would join. But again, it’s tough to replicate that connection you have in person.
Jessica: We’re not going to be able to replace that, because we are humans and we create that connection. But until we have that line in a safer manner, it’s just changing. I think that we will have that balance once we come back to whatever normal looks like, or whatever cadence or what a post-pandemic world looks like and where we are. I think we’ll be appreciating those in-person events in that context even more so.
Going back to what Derric had mentioned earlier, we’ve been doing more micro events versus these big, large scale ones. I think we’ll see that not only virtually, but also in-person as well because it’s just more of a safety point. So if you think about some of these big events that we’ve all attended, and I think about being squished and just being moved down the hallway for an hour, trying to get from one point to the main stage. That’s a lot and the thought of that is a point of anxiety. I think that we’ll see these events really come back to what we think about community and coming together, locality, and making sure that you have a really authentic connection.
Jana: Personally, I don’t know about you, but I don’t want to attend AWS re:Invent. Does anyone actually want to attend it? For me, I’m like “I don’t want my team and I to have to organize a 10,000 person conference with a massive expo hall.” I also think they’re just wasteful. We’ve got all the signage just thrown out, all the carpet we cut. This is such a beautiful time for us to reset and just do things better and figure out what these new ways of doing things are. But yeah, I don’t want to go back to 10,000 person conferences.
Patrick: Let’s shift gears a little bit to some of the nuts and bolts. We’ve got a lot of founders, got a lot of DevRels wondering, “What do I do? How do I do it?” So maybe let’s actually start with two related questions. Let’s start with “What are you measuring now? Has that changed in the past quarter or two? What are the top two or three things that are really important that you’re reporting out on that you’re running budget for?” And then we can go into the nuts and bolts of some of the tactics that you’re using to make those OKRs and objectives come to life. Does that make sense?
Jessica: That’s funny, I just gave a talk on this yesterday, about measuring efficacy, and so I feel like it’s very top of mind. Events are just a portion of it, we haven’t changed the way we’re measuring those. That’s why we’re measuring things from the OKR perspective. We are looking at all the different activity, so we’re looking at how the content is performing for those unique visitors coming in with our specific content that we’re writing and putting out there.
What’s the funnel for them from the behavior of– Do they go from the guide to a sign up page? Do they go from a guide, do they leave? How long are they on that page? All the way through looking at our SDK adoption and the attach rate for integrations, and looking at the whole story of where they come and engage with us, whether it’s from online through Twitter, LinkedIn or with our website, and then all the way through account sign-up.
We look at that as a holistic point from our dashboards for developer relations, and then bring that in. For events, the things that have changed is it’s a little bit easier because we can see the number of registrants and attendees really easily for going to the talk. Before, you had to have a guesstimate of that person. A speaker would go on stage and say, “I had 50 people at my talk, I think.” Now we can see very clearly there was 138 attendees. So it didn’t change measurements specifically, but it made it better because they have a very clear idea of what was being measured and then we can pull it into the impact.
Derric: This also allows you to iterate on content and what you’re doing for DevRel activities much faster. Before, if you’re going to an in-person event you have a dump of the X number of attendees, and then you had to try and figure out “What did they actually do? Did they get any benefit from this activity?”
Nowadays you can get very narrow into how they’re able to look at this piece of content, then this piece of content, and once they saw that piece of content they want to actually get it for a spin.
This is where the more advocacy side is merging with stuff like developer experience, thinking about “What does that first time experience look like? How long does it take for them to get started with the SDK or create their API keys and that type of stuff?” Which was very hard to measure before.
Jana: For us, we just changed the weight. Each activity for DevRel has a number to it, and so we saw DevRel activities per month going down, but just a portion of their work was speaking at conferences, attending conferences and us having booths. So we just added more weight to writing a blog post or adding a learned module or responding to an open source issue, things like that, just to justify it.
Jessica: I think that brings up a good point too, for measurement. It’s really easy to measure the number of activities, like “We did five blog posts, we did five events.” But I think for advocacy and for a lot of things, it’s a really more impactful if you are measuring impact. That’s a better story for you to better understand what your developers develop, advocates are doing, and also for your executives to understand the value that your team is bringing.
Just adjusting that weight, not only just adjusting the weight but saying, “We wrote a blog post. Cool, but that one blog post brought 20,000 people to our website and at one event they had 10,000 developers.” Even though those are only two things technically, that drove 35,000 developers or unique visitors to that content and brought that awareness to your company. I think if you were measuring before by numbers, this is a great time to switch it to the impact, because you can see that in a stronger manner.
Patrick: Yes. Speaking of impact, what are some examples? How would you measure that? Is it page views, is it sign ups? I’m sure it will vary per context, but I think showing impact is essential. How are you all thinking about demonstrating that in your contexts?
Derric: This is really interesting, because it is dependent on the product. Is your product an API? Is your product maybe open source tool, or something like that? But in terms of trying to understand what is the value that someone got out of that piece of content? They may need to take a look at a couple of different pieces of content, but eventually they want to do something after that. Whether that is signing up on your website or generating an API key, or testing out your platform.
Now you’re able to build out that full journey from the initial piece of content and how many different pieces of content that it took. Because some content does really well in terms of top of funnel, but then there is the other piece of content that is really educational and very on point. However, you have less total page views but it’s much better in terms of value that you’re bringing into your community.
Jessica: We have everything that we’re measuring and then we’re looking at that impact and saying “Here’s what we’re driving towards the top of the funnel to close.” That goes along with really what our company goals are. What does your company need most right now? Is that more top of funnel or do you have a lot of leads coming in but they’re not closing?
That means people aren’t understanding your product, so you need to focus your efforts there rather than just awareness. We have a variance, there’s a certain percentage of our time that we’re spending on top of the funnel, and then there’s a different percentage that we are spending closer to the bottom of making sure people understand that.
For teams out there looking at what to measure and how to spend those efforts. Take a step back and think “What’s most needed for your company right now? Is it high level awareness?” Who’s your competitor? What are they doing? What are they doing that’s hurting you that you can help with? Is it really going back to the understanding and education?
Derric: With all this stuff moving towards online, it gives you a better opportunity to do that evaluation.
You can take a step back and instead of just looking at the three talks you did at one conference, now you can have certain content that is actually focused at top of the funnel. Just getting that exposure out there.
Then the other piece of content is really more around, “OK. You know about this product, but what can you do with it? How should I leverage it?”
Jessica: Analyzing and iterating is a much quicker response rate than we can do right now, vs on the plane. Because it’s just getting on the plane. That was a lot of work. Now I’ve got to get my passport, now I’ve got to go do this, and I have to get over jet lag. I’ll come back to it. Right now we can look at the event that we’re doing virtually and we can talk about “Here’s how many people had attended, here’s an event recap, here is some of the things that we’re doing.”
Because I was not on a plane I was able to knock out this educational content, and I was able to actually get some deep focus time to do that demo that was lacking, or the blog post associated with it. Then I did a Twitch stream and I could see these three things that I did, and I can measure the impact and who and what it resonated with and where they jumped off in the video or how long they spent on that blog post. I think Wistia is a great example of looking and engagement from your videos for seeing where people rewatch, where do they stop, where do they get off?
Jana: That’s been a really nice shift, because in-person events, apart from doing a survey you just don’t have data like you do when you set up all the data points on your websites. Now we’re like “OK. On average, each person that signed up watched a talk for 20 minutes. So that means–” We have all these data points and now we can adjust the program based on them. “OK, so we made the day too long. It should probably be shorter.” Or “People’s attention span based on the average, they could only watch for a max 20 minutes.” Adjusting these programs based on what we’re finding, the data and how people are watching or consuming content or the experience.
Patrick: I’m wondering if we could drill down and maybe each of you could pick one tactic or technique that you’ve really seen be incredibly effective over the past few months. It could be something with micro-events or the fact that your team could send more CFPs because they’re not flying around. What’s one big thing that you’ve been really excited to drill into over the past few months, in terms of your goals and accomplishing them?
Derric: For us, we can actually put more investment in terms of making sure that a particular marketing event has the most impact and it is helpful for an audience. What I mean by that is we can loop other folks, such as partners or customers that might have a little bit more of– I would say the numbers behind why it is that you’re doing what you’re doing, whereas when you go to something like an in-person event you get so busy with the planning of the booth, planning on getting there and all that type of stuff. It takes away time from maintaining those relationships, and we’ve noticed that we’ve been working with a lot of our partners much more recently because of this.
Jana: The biggest shift and what I’ve loved is that we always have product releases, product announcements at our conferences.
But the really cool shift this time was, we spent a bunch more time working with product and engineering and our corporate comms team, and we’re able to just enhance those announcements.
We actually had to have all of the content announcements done a couple weeks prior, because we were recording the keynote a couple of weeks before, and they were able to see the keynote and create these little snippets. We were able to brief press and analysts earlier, so it actually made our releases a lot stronger because it took coordination across the whole company to prepare for this event. We had great content, press wrote about us, so that was a massive shift versus spending so much time preparing for that announcement in-person, but what about after that?
Jessica: Yeah, I think one thing that we’ve been doing a little bit more is analyzing the data that we’re looking at for how people are hearing about us, and what kind of content they’re landing on. And what we’re doing for our technical documentation and guides and education points, so that’s a stronger focus point that we have right now. Then going to Jana’s point, we had made a beta release and were working through a couple of things for our feature workflows.
That was really strong, because we had all the time to work on making sure the messaging was all correct ahead of time, we had customer panels and they were really excited about “What is the reason? What’s the best use case?” And because we had more time to prep around that, that strengthened that release, which then led to a few follow-up pieces of content that we have going through it, and some demos and thinking through that whole process of “OK. We have this community aspect, how do we support that with our coding content?” Just that trifecta for what we do, our “Superpowers” so to speak, for DevRel. And “How do we help make our company stronger from that one?” It allows us to focus in on that.
Patrick: It seems like some of the conversation in the DevRel world earlier in the pandemic was, “What are we going to do? We can’t get on planes.” Based on the conversation today, it seems like there are tons of advantages to a largely online DevRel program. Other than community and the in-person aspect of getting on a plane, are we missing anything or is there anything we’re leaving or losing as a result of not being able to be on the road as much?
Jana: Yeah, I really think it’s the relationships you build at attending conferences, making friends, making those connections. Whether it’s third party or your own, but I think just the benefits outweigh the negatives, especially just the region and what we’re able to develop now. It’s cool. We’re almost thinking of running a HashiCorp TV show, because I think people are a bit Zoomed out. So it’s like, “How do we create more interesting content? How do we do Twitch streams and do live coding?” Again, just the reach is so much farther. It’s been really exciting. March was really brutal, but it’s been an exciting couple of months.
Jessica: I think, in general, part of the strong– I don’t want to say “Visceral reaction,” but in some ways. As an industrym in developer relations, I think we’ve over-indexed on associating events with developer relations, so this allows us to go back to that one.
Again, at its core, events is part of it and it’s part of the community, but it’s not the only part. So it allows us to actually have deeper connections and a deeper understanding of what we’re doing, and what we’ve got to do to developer relations in the first place, and what DevRel was when it really started before we had the ability to get on the planes and do things like that.
So, that’s my two cents on it.
Jana: I’m sure we all believe this, but DevRel is becoming more important than ever. Sometimes you’re like, “What does this mean? Is this a fad?” But at least at HashiCorp, that we have a whole org dedicated to this, it validates that this is something that companies need. More than ever, with these activities we’re doing, the reach is important.
Patrick: This has been very educational for me, so thank you all for being so candid and openly sharing all of your tips and tricks. It’s been a lot of fun, and thanks to Heavybit for producing it.