November 2, 2016
The DevOps Journey At Sauce Labs
CircleCI held their monthly Office Hours at our San Francisco clubhouse on Oct. 19th with special guest Joe Alfaro, VP of Engineering at Sau...
What are the responsibilities of modern security teams and how should you distribute these responsibilities from the start? Shruti Gupta of Brex and Lisa Hall of PagerDuty discuss how founders should approach security before hiring a dedicated professional, identify the key traits of a great security team member, and the tenets of security org design.
Shruti Gupta: I’m Shruti, this is Lisa.
Lisa Hall: Hello.
Shruti: Do you want to introduce yourself first?
Lisa: Sure. My name is Lisa Hall. I’m the director of security at Pager Duty. Been there for about a year now. Previously was at Twilio, GlassDoor. You can probably find me on LinkedIn to find all that info.
Shruti: Awesome. As I said, my name is Shruti. I work at Brex, we’re a Fintech. So for us, security is about protecting data and also protecting lots of money that we have. In the past, I’ve built several security programs from the ground up, including at Airbnb, Instacart, Open DNS, which was a security company that got acquired by Cisco. I’m really excited to talk to you about this topic because I feel I’m really passionate about it.
Shruti: Awesome. So, Lisa, at what point do you think a startup should consider having a full-time security employee?
Lisa: Now. The time is now. No, really. I think it’s really important to get security in early. I imagine it more like
cardio. Sorry, but cardio is not going to hurt you. It’s only going to help you. Whether you decide to be a bodybuilder, or you just want to walk around the block without getting tired. Cardio is probably not going to hurt you. The earlier you get it in the better, even if you don’t have a dedicated security team or security projects. You get security in early.
We partner with IT and facilities and you’re building out your first office. It’s really great to have a security point of view when you’re doing that. “What kind of badges should we have? What should be on the badges? How long should we retain that data?” Everybody needs computers, you’re building out IT. It’s only going to help you to have a security minded individual in there during that time. Same with human resources. “Should we run background checks? How long should we do this? What should we worry about?”
So I cannot stress enough that it will not hurt you, I promise. It’s like good security, hygiene.
If you want to look at metrics, I think generally people most companies hire security within 30 to 100 people. It generally sits at the 1-2 % headcount, overall headcount. Depending on the size of your company, you’re going to want to go for 2%, go for 50%.
That’s standard, but again, I don’t think it’s going to it– It will not hurt your team to get security in early.
Lisa: What do you think?
Shruti: Yeah, I totally agree. You brought up a really interesting point about building security even before you hired that full-time person. How would you suggest our friends to do that? Maybe hire a consultant? Do you have any recommendations on some of the resources that they could tap on, either from a consulting perspective or make friends with other security teams in the industry? What are some of the strategies they could follow?
Lisa: Before you have a security team? I think generally most companies have security evangelists within your company and you can tap into that resource first and use those connections to find that security resource and that first security hire will help you build that type of thought and those type of people within whatever part of the company that you’re focusing on. What what’s your opinion on that?
Shruti: Yeah, makes sense. I’ve always seen that to work really well. All of my previous companies, there were all these a few people who deeply cared about security more than the other people, and they had pretty good knowledge
as well. Maybe that was a coincidence, but setting them up for success to be able to set a strong foundation proved to be really useful.
Lisa: I think that works too, when we talk about educating our developers. Maybe we only have one security person on the team, we can really tap into “Oh, I know the developers on each team that care about security, we can train them and get them to be coaches for their team.”
Same goes for almost every part of the company. I have people in our people team that I know care about security.
They can go to them for that. Getting those pockets of security and really focusing on that helps make sense.
Shruti: Have you seen any cultural indicators or business indicators that would say “Now is the right time to hire a full-time security person.”
Lisa: I think it really does depend on the company, but a lot of times customers drive that, especially now with
a lot of focus on compliance. Everybody’s talking about GDPR and CCPA and all these other things. It’s not a bad lever to use to get that security thing going, because security is generally not viewed as revenue-generating. But if you can tie it to “We enabled customers, we enabled sales, we are building compliance,” you can use those tools to build up your
Shruti: Makes sense. Yeah. Very helpful. In my previous experiences, I’ve also seen it also depends on the nature
of the company. Whether you’re in a more regulated business or whether you’re doing something that demands a higher sense of security for the assets you’re protecting. Then it varies quite a bit. I’d say it’s great to start early. My husband had a company before and he’s someone who’s not in security. It’s very interesting to know his perspective on how he thinks about security.
Actually, there’s a lot of other practical things that entrepreneurs have to worry about. They can be essential to the business surviving first. My personal feeling is that I think, first, it’s important to build something that’s worthy of securing. Then also planting small seeds as you go along. Then when you feel like you have something meaningful,
then hiring a security person to be able to help you along that journey to scale it up.
Lisa: I totally agree. That brings us to our next topic, which is really once you have this built or you have part of your idea of what you want security to be. Where does it live? Where do you put it?
Shruti: Yeah. In my past experiences, I’ve seen it to be more successful when security is in engineering and they report to the head of engineering directly. I’ve seen it set at multiple places outside of security, sorry, outside of engineering. Within engineering. It depends on what kind of a team you’re trying to build.
A modern [security] team is very proactive and does a lot of engineering work. If you want your team to be that, then it’s kind of hard for them to be successful unless they’re in engineering.
Then where in engineering is pretty important as well. I think I find it the most set up for success, manager reports to the head of engineering, because there’s no biases, or no resource restrictions on when you are directly reporting
to the head of engineering vs. if you were reporting to maybe the infrastructure team, or maybe you’re a part of the
infrastructure team or a part of the compliance team that reports to some other function. That’s what I found to be very successful.
Lisa: Yeah, I have to agree with that. I’ve always seen the most successful in security is heavily embedded with engineering. A lot of the work is partnering with engineering. I think that it’s hard to get leverage, especially if you’re talking about top down. “These are our objectives. This is what we’re going to do.” It’s hard to have outside perspective.
I also am a big fan of security being enablers and not being the stick, but being more of the carrot. I think we’re better equipped to do that when we live in engineering, because we can say “We want to help you. We want to build things better.” Sadly, I’ve also seen it not successful and sometimes successful, live in more the compliance, legal
side. Where it’s a little bit more like the stick, like “Well, this is cause, this is this thing we have to do, we have to do these things, but there’s a reason why there are regulations and engineering can kind of steer it a little bit better, I think.
Shruti: It makes sense. Awesome. Along the same lines, what would you describe as a modern, or a successful security team in a small– in a startup?
Lisa: Hmm. I think I probably would go, first, to a pitfall of what people think security would be in a startup and your first security engineer, or your first security hire. I think a lot of companies set out to have that unicorn security hire, where “This person can solve all of my problems. Or actually, you get all of the problems.”
I think it’s better to have an open mind about how the security can really enable each part of the business and be a partner to business. I like to use the term of how that role, whether it’s one person or 10 people, are consultants to the business and they give our best advice and we try to tell you what we think is right. It’s really up to the business on how we want to run the business and what we choose to do and not having it be that person where it’s like “You are now here. You will solve all of my problems for security and everything else. Our customers are going to be happy. We’re going to be compliant. Everything’s going to be great.”
Shruti: Make sense. I’ve always worked for companies that were growing really fast, like Airbnb, Brex, Instacart. It was really essential to build a modern culture around security. Otherwise, we would not be able to scale with the business. Some of the things that I always kept in my mind while bringing a culture of security was one, as Lisa mentioned, finding unique creative ways to support people instead of saying ‘no’. I think saying ‘no’ is like the easiest thing you can do, being on security. But that really doesn’t solve the problem.
It’s also very essential to determine how much risk you want to take on and be aligned with the business leaders on that. Every time you make a decision, keep in mind that nothing is going to be perfectly secure, but you just need to know the threshold or the risk that your business is willing to take. You should help them make that trade off and find the best solution. Education’s really important as well. I think the security team, which comprises of a few people out of the many more people in the company can really not make or break security for the company as a whole. They are just like fuel to the fire.
But as entrepreneurs, or as leaders of the company, it’s your responsibility as well to make sure that everyone
understands that security is a shared responsibility and that the security team feels supported to help other people. Help the company add more security instead of just being the ones responsible for security.
Lisa: Yeah, I sense a theme here. I’m a big fan of the say ‘yes’ security type of team and lovable security. I think when you’re first building that, it’s so essential to have a team that are our partners and that can do that. You could easily get a very specialized security person, you’re like “I need this person that codes in this, and that’s going to build this thing.”
But security is so broad, it’s, I think, essential to have that type of security culture being built. This person or these people will be responsible for growing the security culture for this company. It really could be such a great starting point there, with your first few hires.
Shruti: Yeah. Makes sense. How would you think about the proactive use of more engineering proactively to build a strong foundation vs. the reactive piece of fighting incidents when they actually happen and how would you think of splitting between the two?
Lisa: I think, in any [security team] there’s going to be a little bit of both. You have to decide how– Hopefully it’s not too much reactive. You really have to be able to get somebody in who can plan that. Even if it’s an engineer, you’re hiring director -level or whoever got this, long vision. Understanding your risk and being able to focus on what that landscape is for you, because it’s different for every company.
Then knowing little baby steps, I think a lot of times people just want to eat the whole elephant, kind of thing, and you can do baby steps to get there. It’s kind of back to the cardio, you don’t have to lift that 200 pound weight, or whatever, right away.You can do tiny bits. Part of that is recognizing “We will have a little bit of reactive security. We’re going have– Someone’s going to try to hack us. We’re going to have alerts. People on support are going to do things.” I don’t know.
You’ve got people all over the company that you’re going to have to react to. At the same time, we have to have that long-game vision of where we want security to be, where we want the program to be, and how you can get there, even if it’s in tiny bits.
Because you only have a three person team or whatever you have.
Shruti: Yea, makes sense. Awesome.
Lisa: How would you approach the same?
Shruti: Yeah, I think I would approach it similarly, where I think I would hate for it to be very reactive because that’s just not solving the problem. It’s just fixing the patches. Investing a lot in building a solid foundation is what I would rather invest on, as compared to just always firefighting. It’s really frustrating as well. Like if you were in security, you just always want to deal with fires all the time when you know there’s so many holes you can fix.
What I’ve done in the past is spending a lot on engineering, tooling and automation, such that an education, such that the load was more on the proactive side, rather than the reactive side.
Lisa: Yeah. I really like what you’re saying there. I think education’s great. You can, again, spread security across the
company. I’m a huge fan of automation and tooling. It’s also one of those things I’ve seen done horribly wrong.
Where people are like “Well, we could hire for this, or we could just tool it out.” And you get a tool that you don’t have anyone to operate or run and it just kind of sits there and goes into the abyss. Or it creates more security fires for teams because they’re responding to things that maybe aren’t even necessary.
Shruti: Or worse, you could have a bunch of unactioned alerts and when you actually have a breach–.
Lisa: What? No, that’s never happened, what are you talking about?
Shruti: Then you’d be in more trouble than you would be if you didn’t have those tools in the first place.
Lisa: Okay. I’m curious on how early teams can promote security work. A lot of times maybe the CEO, or somebody, thinks “We need security. Hire a security person.” This poor security person is they’re all on their own. How do we promote security as a thing? How do we promote it across the company and within maybe specifically, development teams?
Shruti: Yeah. That’s a great question. I think there’s a few strategies. One is at the strategy level. Security can be a very qualitative thing.
A lot of times people feel like there’s no event, then that means we were secure. But that’s really not true, because having a bad event depends on so many things that are not in your control.
Even if you have a stellar program and what you’re protecting is really useful, then you might– It’s just about how hard someone is trying and then also your security posture. Not having a breach is not an indicator of how secure you were. But it can be quite hard to speak the same language as someone who is not in security. Having some sort of a metric to determine what your risk posture is, or how secure you are, in broad sense, is super useful. I’ve seen that in the past.
What I have personally done is I’ve created a scorecard, a custom scorecard to Brex and that has a few hundred criteria of what I want in an ideal perfect world for us to have. Then I’ve ranked us in each of those
categories and calculated the total amount of score.
That just helps me speak the same language with other people outside of engineering, or even within engineering and demonstrate how we are making progress and adding more maturity. It’s not an exact score, but it’s a north star to keep a track on and prioritize work and speak the same language.
It also promotes the democratizing of security, because then you involve a lot more people and they understand what’s going on. You make it more of a community thing, rather than just security owning all the burden and
other people not feeling that they understand what the security team is doing.
Some of the other things that a non-strategy level, from a tactical perspective that I think could be useful is presenting your work and demonstrating what’s the cool stuff that you’ve been working on, gets a lot of buy in.
Also, if there were any bad vulnerabilities that were found confidentially and patched, then teams could present those as well. Those get a lot of attention, because then engineers do realize that security is not a hypothetical thing. It’s a practical thing. There’s really risky vulnerabilities that can do a lot of damage.
Lisa: Yeah, I totally agree. I’ve seen that, too, where security can partner with developers on that. A lot of times people– You write code, you love it. You want to make sure it’s good, so people do take it seriously. You can partner with developers on making that an important part of their process as well, is important. The metrics I also agree with, because a lot of times you have to communicate that up, then that’s a good way to do it.
Shruti: Makes sense. When you hire candidates, what do you look for in them? What are some of the cultural aspects you look for and also some of the other knowledge-based things that you look for?
Lisa: Well, I look for them to be able to do everything and solve all my problems. Yeah, I don’t. I think the biggest thing is really I’m looking for people that say ‘yes’. I think the days of being the ‘no’ security, “No, you cannot do that.”
are over. I don’t think that wins any favors on any team. Being able to partner. I don’t really look for people, depending on what we’re hiring for, but I don’t look for people who are just really specific in anything in particular. If they think about security a certain way and are learners and have a “Let’s enable the business, kind of attitude.” I think it goes a long way. Also, even if you are super well-versed in one thing, tools change, the environments change.
Being able to adapt is really important and understanding that maybe the biggest risk in the industry is phishing today, or whatever you want to say. Maybe at the company the biggest risk is this other thing. But being able to see how that changes and varies and being able to adapt to that is really important. Otherwise you get stuck in this world of “This is the one thing we focus on. This is the most important thing.” Little do you know, behind you everything’s changing and you get to be able– So adaptability, maybe that’s it.
Shruti: Yeah, totally agree. I think the attitude and culture fit and the capability of working and collaborating with other people I think is really important. I believe one vulnerability here or there, doesn’t make or break a security posture, unless it is the one that actually causes a massive thing and then you can’t recover from it. But most of the time, for practical purposes, I think culture is really important and adaptability is really important as well.
Lisa: Thank you so much for having us. We really appreciate it.
Shruti: Thank you.