June 26, 2014
PagerDuty’s Favorite Dev Ops Blogs
PagerDuty’s Ops Engineering team lead Arup Chakrabarti knows how to scale complex architectures. During his time at PagerDuty, the company...
I'm Gina, as Tim said. I have 15 years of communications experience. I actually do some regular PR type stuff, too- write speeches, I dabble in a little bit of everything. You might say I'm a saddist because actually, my favorite thing to do is communicate in difficult situations, mostly because I think it's a time when it's make-or-break and it's most important. That's why I'm here.
The real reason that I'm here is because shit happens to everyone, happens to people, happens to bad companies, happens to good companies. I'm not standing in front of you guys to tell you how to prevent shit from happening to you and I'm not going to give you five steps for doing the exact right thing when it does.
What I really want to help you guys do is re-frame how you think about communications and think about communicating in difficult times. I know it said Crisis Communications, and that sounds very dramatic, but to me, a crisis is anything that could negatively impact your reputation with the people that you care about. It doesn't have to be a huge scandal. It can be something that may be minor to other people but is important to you.
I think the thing to take away here is that everybody's crisis might be a little bit different, but there are some principles that you can use to avoid getting yourself into a worse issue.
Today I just want to share six practical tips, again, these are sort of simplified. If I had time, I'd sit down with each and every one of you, get a sense of what you do everyday, and figure out how to help you frame it.But in absence of that, in this group, we'll just go through six practical tips that you can follow that will help you weather the storm.
The first is Be Authentic.
This doesn't sound very practical at all, and it actually is something that starts now. As you grow, one of the thing that I challenge you guys to do, is figure out what your authentic voice is, what your values are, which again, sounds really esoteric, but what's important to your customers? What do you pride yourself on? What do you tell them that you're going to do for them?
My advice- do that, don't let it get away from you, and as you grow, make sure that everybody who works with you understands what those things are. I think that the biggest challenge in a crisis is for a company who was kind of portraying one image, and somehow lost track of it along the way and wasn't really living it.
That's sort of the difference between, we have a difficult situation and we have something that could be really, really, really bad. So, authenticity I think, is something, do it now. If you claim that your product quality is the most important thing about you, it's your differentiator, you'd better be doing the testing, you'd better be doing QA. Make sure that if there is a product flaw, it doesn't expose all these things that you were doing that don't really match up.
Then finally, authenticity comes into play when you are dealing with a crisis. This is sort of an interesting one, I think, for you guys, because I know you spend a lot of time probably short, I think Dana said, GitHub, doing all these things, that when something bad happens, it's the time to be sort of human. Think about the people on the other end of your communication as just people, and be as sensitive as you possibly can, try to think through what they're going through because they'll judge you on the smallest things.
During crisis situations people take cues from everything so it's really important that once you get there you start thinking, "OK, what is my audience going through, and how can I at least try to relate to them a little bit?"
Number two, Plan for Your Top Three Risks.
I have worked with companies that have books with 70 pages of different scenarios that might happen. It gets really exciting when you get into the 60s and 70s, anthrax scare, all that stuff. I don't think that's the best way to do it because inevitably what happens to you is not going to fit within the confines of that playbook, and it will be very ironic, it's the 71st thing that you weren't prepared for.
But I think that you can create a framework for yourself based on what your top three things are for your company. Normally I would say, it's based on your business, it's based on your customers, but I think for those of us who are here in this room, you can work off a pretty basic starter list. I think Tim kind of went there a little bit.
If I was to think about general things that you guys might face, I would say data loss, maybe a security threat, service availability issues, or product quality.
I think you can start there, and figure out what that means for you and it'll give you a good way to put a framework in place and start at least changing the way you think about things. You guys are innovators, you're optimists.
I'm not here to turn you into pessimists, I'm just trying to get you to kind of think through, "OK, these are some things that I could face," andthey may be crisis, but they're not that far off, they happen to some great companies. So think through what this means to you, and if this isn't your list of three, go ahead and create your own, but just put it in your mind and sort of think through it.
Number three is Know Your Communications Channels.
It's amazing how many times I've dealt with a company and we kind of walk through everything, and we're like, "OK, so let's go tell the customers, oh shit, how do we tell them? What do we do?" It sounds really simple, but do you have email addresses for people? Do you have phone numbers if you want to call them? How well do you know customers on an individual level and then how well do you know your broad outreach stuff?
So we'll talk a little bit more about this but the impact should dictate your strategy here. If two people are impacted and you can have a crisis or a difficult issue that only affects a couple of your customers or your users, you don't need to post a blog to talk to those two people. You should have a way to get to them directly.
If you look at what the target is, if it's targeted outreach, if you're going to connect with individuals, do you have their email addresses? Do you have their phone numbers? Do you know how to reach them and setup a meeting? It's kind of common sense in that way of, you know, let the impact of the issue dictate how you respond to it.
Then, sometimes, you're going to have to talk to everybody as fast as you can. There are things that you can do now, as you're getting started, that will help you when this time happens. I'm sure a lot of you guys have a blog, or you're getting a blog started. A blog post is a good way.
I love status sites so having a service availability page, where people can check the status of your service. You can make it super simple, but it's a great place where if something happens, you can easily just go post there and the people who need to know, will know where to find it.
Also, we've come across things with companies where maybe you can put a little message, have something that you can put in the login page so that as soon as they go to login, they're going to know what to expect, if the service isn't working, if they can't get to it, or if there's some kind of issue.Homepage banner.
Then finally, social media. I personally would say social media is your last resort. It's kind of a broad way to communicate, even just to your customers. If you have no other way, and you know you have a following then I think it's fine. It wouldn't be my first choice. I think that in these types of situations if you can target it to who's impacted, that would be the best thing.
And then don't forget employees. How many of you guys have more than three employees? OK. So, it's entirely possible that one of those people may not be in the room, right? Even if it's one person. One person who doesn't have the message and doesn't understand what's going on could really screw up your communication strategy here.
I think employees are super important, they should have a heads up. Everybody, even if they're not involved, should be aware of the situation. You should tell them before you communicate externally. If they're the ones that are going to have to talk to users and customers on the phone, they need to be prepped with talking points. It's kind of simple, take whatever statement you've made, turn it into talking points.
And then, everyone else should know how to respond. If you don't want them to do anything, tell them not to do anything. If you have a statement you want them to use on social, give it to them, but don't leave them hanging, they could do the wrong thing without trying.
Number four, Get the Facts to Customers Fast.
This is particularly important if there's anything that's actually impacting their experience. People can get a little paralyzed trying to figure everything out and get all the facts and have the story created. In reality, sometimes the best thing to do is say, "Hey, we're aware that there's an issue and we're working on it."
I think there's an inclination to wait a little longer and feel like that's not OK, but I can tell you that customers will be happy to know that you know about it. They'll stop tweeting to see if they can make you aware of it. They'll stop calling you. They'll feel satisfied with that, as long as you let them know, "Hey, this is where we're at now," and you can update them regularly.
You don't have to have your first communication be perfect. You should give them the facts, don't communicate assumptions, but just put them out there, and actually, this is a good example from Heroku, "We're experiencing an issue, we're aware of it, and we're investigating it."This was the first communication that they put out, and obviously they updated it, but it's a great way just to say, "Hey, we know about it." It eases people's minds.
That's a first step,getting the facts. At some point, and I think I was talking to Ken, who was talking about a postmortem, at some point, you're going to have to do some kind of broader communication if you've had a issue. People are going to want to know what happened. They're going to want to know why it happened.
So you need to put together some kind of response. I think the best way to think about it is the four Rs- Responsibility, Regret, Restitution, and Reform-Four Rs.
Responsibility. I would say, if you're looking at a response, at a minimum, one sentence on each of these. Don't leave anything out and try not to put in too much more.
Accept Responsibility. No one wants their vendor to push off blame on someone else or pretend it didn't happen, or it wasn't our fault. Just accept responsibility.Everyone knows, and you're better off establishing that and moving forward.
Restitution. People are going to want to know how you're fixing it. What are you putting in place to make this go away?
And then Reform. What are you doing to make sure that it doesn't happen again? Have you learned your lesson? If you look at something like, if you had a human error, if someone did something, what are you putting in place? What kind of checks are you putting in place so that that person doesn't do it again? That's important.
And then finally, Regret. Your lawyers might tell you not to say sorry, I think saying sorry is not only acceptable, it's kind of required. No one's going to move on if you don't apologize. It's sort of the first step. It's not going to prevent people from being pissed at you on Twitter or feeling a little bit burned, but it's the first step in moving forward.
I think that apologizing has become this sort of thing, like the New York Times dealbook actually has something where they track companies responses to crisis, and call them out on it and see if they've done all the things they say they're going to do.So I think as important as this response is, you've got to actually do everything you say you're going to do in it, and expect yourselves to be held accountable.
So, What Not To Do. Don't wait too long to communicate. Don't spend hours trying to get a statement just right. This is the combination of speed and accuracy. You've got to balance it out. If you feel in your gut it's getting to be a little bit too long, it probably is already too long. Balance it out, don't wait too long.If you do, and there's a conversation going on, and your voice isn't in it, you're going to have a really hard time getting it back.
This one's tough. Don't post an overly long or complex blog. Usually when people see something that's super long and there's a lot of details and paragraphs, they just assume you're trying to hide the facts with a bunch of stuff, or bore them into stopping reading it. It just pisses people off.
Go to the four Rs, start there, and try not to add in a bunch of fluff. At the end of the day, your customers probably aren't super interested in all of the engineering background that went on. They kind of expect you to take care of that. They want a sense of the overall issue and a sense of how you're fixing it, but you don't really need to go through every single thing that happened.
Pick and choose. Keep it short enough and if you really need to, have someone who's not technical read it. I spend a lot of my time when there are technical issues going through and going, "I don't get it, I don't get it." I might not get it, but at the same time, I'm also forcing people to figure out, how can you communicate so anyone can understand it?
Particularly if it's a public blog post. If you don't have one of me, get your friend, your wife, your mom. Get someone to read it and see if they understand it. It's really hard to do, but you need to work at that.
And then, finally, being inconsistent is a big problem. Not only is it going to confuse people if you have different messages in different places, so if you post a blog somewhere and a status page update somewhere else and they're all inconsistent, and you've recreated the wheel andcrafted unique messages, it's going to confuse people.
Keep it simple, one message, different channels. It also will save you a lot of time. Don't recreate the wheel on this stuff.
Number six is kind of important- This is Not a Media Opportunity.
I'm in PR. I love helping people place amazing stories, it's a lot of fun, but now's not the time. When you've had an issue, you need to focus on your audience, which is probably going to be your users or your customers. Communicate directly with them.
If I'm going through an issue and a customer reads about something in an article instead of reading about it from me, I consider that a big fail. People want to feel like they have a connection to the company, they don't want to go through the press to hear what you have to say.
It's really important that you focus. Don't worry about what the press think about it, it is what it is. Focus on doing the right thing by your users and people will see that. Particularly if you can post a blog, or you can use your status site, that's a channel that everybody can read.
Develop a short statement if you are getting inquires from the press. A statement is fine, and if you have something that's already posted, there's nothing wrong with pointing them there.
In fact, that's what I would recommend because these are times when I think you want to keep what you're saying pretty tight and be able to focus on fixing the issue and not really spend a lot of time doing interviews. It's not good for anyone.
A note on turnaround stories. There could be a desire to go out after you've gone through all of this, and go, "Let's place this great story, how we've turned it all around." It's super romantic in theory, but it's really risky. I think that if you have experienced a difficult situation, and your customers have let you move on from it, and they're trusting you, there's really no point in rehashing it. Let your actions tell the story for you.
If you had a product issue, fix it, let people know you fixed it. Communicate to your customers. But I would caution you on trying to place some kind of turnaround story. It's risky and it's not where your head should be. Now is the most important time to focus on doing what you're doing so I wouldn't recommend it.
Now that we've gone through all the tips, I want to put them into play with a couple of specific situations, and, ideally, when we're going though this, you guys are going to kind of know the answer anyway, but I think it helps to sort of align them to things.
Service interruption. First thing you do, "OK, it's down, we're experiencing an outage." Let's sit down and figure out what is going on. We don't know yet.
First thing you do, post some kind of initial message. Get it out there, let people know that you're aware of it, and you're working on it.
Continue to work on it. In that span of time, you should have enough information about the issue to gauge, at least at a high level, how long it might last. Once you know that, you should establish some kind of cadence of communications.
If this is something you think you're going to be able to solve in two hours, you can probably update people every 30 minutes to an hour. If it's something that's going to go a span of days, that would be a lot of updates. My recommendation is figure out what it is based on the situation, and establish a cadence.
People just want to know when they can hear from you again. It's like when you go out and tell someone, "I'll be home at midnight." It's probably better to just be like, "Hey, I'll call you and let you know when I'm going to be home." It makes them feel a little bit better.
Communicate facts as you have them. Again, don't be afraid to post a little interim update, letting people know where you're at, but again, facts. So, chasing a lead on what it might be is not a fact. Figuring out what it is and having a new time frame in terms of when you're going to get it fixed, that's a fact that I would communicate.
Then finally, when this is all over, people are going to expect you to publish either a root-cause message or some kind of postmortem going through the situation in a bit more detail, giving them that background. You can use the four Rs to go through that. Depending on your audience, you can dial it up or dial it down for how technical it is.
A lot of people rely on vendors, right? You probably have an infrastructure that is not your own. At the end of the day, your customers don't care.By having an issue with a vendor that causes you an issue, you can't really point to the vendor, it was your choice and you should just own it. It's fine to say, "Our X vendor is having an issue," I would not advise you calling them out by name. That's my opinion. Don't call them out by name, genericize it.
If you have a page on your site or somewhere where you talk about the infrastructure you have, people are going to know anyway, you don't need to say it, it just looks like passing the buck. And I would say also, if your vendor is causing the issue, you have to kind of incorporate them into your whole process as you know and understand what they're communicating so that you can be aligned. That's how far I would take that.
Security. Number one, obey the law, whatever the law is at the time. If you have to notify people of something, you have to notify people of something- just do it. If you don't have to communicate, let the impact dictate your strategy.
You might have a security issue that impacts two of your users. You can probably pick up the phone, right? As we talked about. If it impacts everyone, you're probably going to have to communicate more broadly. I really think the best thing to do, people get really wrapped up about, "Oh, something happened, let's get the blog post going, let's get our response." You really have to understand the impact and do what makes sense.
You wouldn't necessarily communicate to 150 people if only two of them care. Use that common sense around it, and let that determine your strategy. If customers are impacted and you decide to communicate broadly, try to reach those customers first and just give them a heads up so that they know.
If you do have to post a blog because a majority of your users are impacted and some aren't, just let the ones who are know, otherwise you're going to be flooded with phone calls from people wondering, "Is it me? Is it not me?"
The other thing with security that's interesting is a lot of times your customers may have customers who are impacted by it, and so, as with your own vendors, it's important to work with them to help them figure out what they're going to say to their own customers and align on that messaging. That's something that you can do to help them out if you've caused them or their users pain.
This is another one that's interesting, be transparent, but don't put your users at risk.
If there's a security vulnerability there may be a desire to communicate that you're aware of it, but if you do that before you fixed it, you're just inviting people to exploit it. Think through, "OK, is this something that is going to put our customers in danger if we communicate it publicly?" If you can wait a couple days and fix it, that's okay, I don't think anybody's going to ding you on that.
And then, clearly communicate user actions. If people need to do something, tell them. People get the most upset and post the most negative comments on security related blogs if they don't have a clear idea of what they're supposed to do next. People just want to know what they should do, if anything. Just tell them. Then, put an executive, put yourself, make this from a person, not the company, and offer people a place to go if they have questions.
This might make me unpopular, but I think if you post a blog on something like this, you're really communicating out. I would use a direct email address if people have questions or a phone number rather than having people post comments on a blog that you're not going to answer in front of everyone. This is really an opportunity to push out a message and then handle customers one-to-one, in my opinion, people might argue that.
So, trolls are not a crisis, but they can rear their heads and stir shit up that might turn into a crisis. As you guys know, don't feed the trolls. It's really hard when someone says something negative about something you've built, or you, to not respond and not retaliate, particularly if you think they're wrong or they're false.
Just don't. Avoid it or you're going to dig yourself a deeper hole and you're going to be stuck responding to someone who has nothing but time on their hands to torture you. The one thing I will say about this is a lot of times these trolls are pointing something out that maybe is true. So if they're pointing something out, you might want to think through like, "Oh, is this really an issue? Maybe I should investigate it."
If that's the case, go back to all the best practices we just went through. You're not communicating with this person, right? Your audience is the people who are impacted by it. Communicate with them. Don't respond directly, it's not going to help you.
There was an issue earlier this week where someone responded to a negative comment from a journalist and it started a war. It's hard because these are emotional things that are typically happening. You have to take the high road.
I would actually advise that as you grow, from a communications standpoint, overall, never attack individuals. Take the high road. It's not worth getting into an argument and people eventually see through it. So that's my final slide before we move into Q and A.