October 6, 2015
Ep. #4, Scaling Your Engineering Team
In this episode, Edith and Paul talk about who owns code, how to scale your engineering team, and cowboy coding.
In episode 30 of The Secure Developer, Guy speaks with Justin Somaini, a security industry leader and Founder of Somaini LLC. They discuss how security theory has changed over the past 25 years, and how AppSec can be improved by educating the developer community.
About the Guests
Justin Somaini is the founding partner of Somaini LLC and a well known Security Industry leader. With more than 25 years of information security experience, he is able to bring his expertise to resolve and benefit leading companies today. Justin was previously CISO at Yahoo!, and was Director of Information Security at Verisign.
Guy Podjarny: Welcome back to The Secure Developer, happy to have you back with us.
Today we have Justin Somaini.
Justin has a
very long and extensive history in security, ranging from being
the Chief Security Officer in a variety off
Coming from Symantec and Yahoo! and Box and SAP. I'll let him tell his story in a sec, but welcome to the show, Justin.
Justin Somaini: Thank you for having me. It's great to be here..
Guy: So, Justin, before we dig into the many interesting questions that we can debate here. Tell us a little bit about how did you get here? Who you are a little bit, some short version of that history.
Justin: Yes. I've been in security for
25 years, and started out many
moons ago at Pricewaterhouse as a penetration
tester. During that process I realized
that if I was ever going to be a great consultant, I might
actually want to do the job at least once in my life.
I got the amazing opportunity to come over to Charles Schwab to run security operations, and then realized that this whole fixing security was a lot more difficult than just consulting on it.
So, I stuck with it and from there I went to VeriSign
where I ran all security for five years, went over too
Symantec after that as their CSO..
Then over to Yahoo! as their CISO, to Box as a chief
Then just recently left SAP as our global
Then of course, as we do in the valley, we have a tendency to participate with VCs and security companies, so I've been doing advisory roles and advisory functions for VCs and security companies as well.
I try to be very active in the industry that I love.
Guy: That's excellent.
All those things also keep you fresh a little bit. I like
to say they help you learn in parallel, and not just sequentially.
So, it's quite a journey.
Maybe we start a little bit with that historical
perspective, you've gone from these different companies that are all so different
and different times.
How would you say the security landscape has changed in the ways you've been tackling, maybe the same title, across a decade or two?
Justin: Yeah, we have gone through a
lot of maturity and a lot more is
still yet to come.
But over the past 25 years, when I started security
was very much what I would call a very simple audit-and
"We have a
firewall. You know what? Nobody can ever really
execute anything on a web page."
Literally that was a statement that was made to me way
back in the day.
It's very sophomoric thoughts about security,
even though we had luminaries in security theory back in
But as we've matured and grown, I think that
we've really grappled a
deeper technical base of what's going on,
hence application security and driving it deeper
into the nuances of coding and
development, and how coding is really done in
spite of the acceleration that software development has gone
through over the past 15 to 20 years.
But on par and parcel from a network standpoint, an education standpoint, from an organizational standpoint. We've done those as well. Some of the things that I think that will mature, but maybe a little bit later, if we look at global law enforcement.
I remember probably up until maybe 15 years ago law enforcement was not deeply focused on the cyber problem. It was a bit of a joke in a lot ways.
I hate to say this, I have a lot of deep love an
affinity for law enforcement globally that deals with this
problem, but there were other challenges that they were
That's a very different story today.
When you look at the FBI or Interpol or Europol or
some of the other agencies around the globe, they have
dramatically matured to deal with the problem as that threat has
So we've seen a lot of changes, but quite honestly, I
think we're maybe in the first quarter
of maturity in application security, or in security
as a whole.
Not unlike they're still robbing banks, and that's been around
for a while, and we look at other industries
such as legal or finance that have been around for
a very long time.
In spite of that, they're still maturing themselves, at a slower pace of course. I think we have a long way to go.
Guy: We're still at the beginning.
changes are obvious
in the landscape as some tech has
changed, like mobile devices or others.
But what would you say are the beacons of change, or what are the key drivers of difference in today's world in security versus maybe a decade ago?
Justin: When I look at security I have a tendency at a
meta-level that you're referring to it, I take a step back
and I think about--
There's a big difference between security theory and applied security.
So if we take
security theory about how we attach
confidentiality and integrity and availability to
data and its transactions, in a mainframe
world that's fairly simple.
You apply it in the mainframe, it's a centralized model, you've
got green screens that are basically just windows into
that data and transactions. It's
fairly easy to implement a CIA model to that.
What we saw with a transformation from mainframe to
client server, the world dramatically changed,
and theory didn't change but how it was applied did.
So, where that data was and where the transactions
were, now shifted to workstation
servers and multiple servers running those
services, workstations' data being handed back and
We had an explosion of complexity
simply because the data and transactions moved, and we did not have the
mechanisms to apply CIA to it.
If we move forward from client server into client
cloud, now those services are on the Internet with
a more insecure and hostile direct
access environment, shall we say.
But really underneath the hoods what you're seeing is an
interconnection of those services as we look
at Web 2.0
and some of these other API integrations that you have.
That transaction and data service model
becomes even more complex.
Now as we move even beyond that, and what I would call
containerization, while the
concept of the service doesn't
necessarily change in a great degree in a
container model, but what we do have is an exact
aspiration on the operational side that we saw with
We have many containers
with many data and transactions, and how we manage and
govern them becomes even more complex, and it's being done
in a multi-cloud space around the globe with a whole bunch of different
So, how we leverage
technology to do things has
a significant effect on how we apply CIA to
govern those things, and so that's the basic concept of how I
look at technical security to start with.
What you see is you're able to predict a
lot of the challenges that we face if you keep
an eye on what is the new technology that is going to
be adopted eventually by businesses and
organizations as they try to accelerate their growth and
But containers came onto the market, and now it's flushing, and now everybody's trying to run and keep up. We can also do the prediction with serverless, a whole bunch of things are just starting to take hold.
Guy: I fully relate to that, I feel like for
starters there's many moving parts and a lot of the
safety of controlling the environment has gone away, because now everything is going to
be all interconnected and mobile and on the internet.
Before we go to the next challenge, what have you
seen as strategies?
Like, successful strategies to help tackle that? It feels like definitely a big battle. What do people do, or what have you done maybe, that you've seen work to conceptually try and adjust to this fragmentation?
Justin: Yeah. The security management model that
I try to approach, I'm a big believer in a
general statement, the centralization of a security function
so you can get some leverage.
But getting to your point--
You're never going to be as agile as the lines of businesses or the development teams or the functions in a centralized model. So, how do you have a distributed or integrated security model to be able to deal with a day to day handling of issues as they come up?
As a result of that, generally what you have-- This is more
pronounced in application security than it is with security
operations, as we have shifted to an Agile
model, the velocity of change, or releases,
has dramatically shifted.
Which is for security to be able to handle this
velocity, which is where they struggle to a great
degree in AppSec.
But one of the models is having embedded individuals in those lines of
But secondary to that
is really finding ways to not only
educate and empower the development teams,
but also make it clear that because of the models and the
decisions that they've taken there is a significant
accountability that they have in the security model as
This is where the first problem really
comes in with basic security teams and
developers, because they are two very different
camps or mindsets and beliefs.
We've grown up in security in this operational model, where I
create a policy and a standard, and somebody is just going to build it
to that standard.
That's what DBAs and system administrators do when they build service, that is not how development works.
Guy: Definitely not today.
Justin: You have a problem and the developers are part of the
process to engineer a solution.
Most security people have never checked in code.
They don't understand what a CICD pipeline is.
If you talk about new technologies or new capabilities like mesh
networks, or even containers to a great degree, it's just
not really part of their normal ecosystem.
So the first step in any security build out
for AppSec is to really establish
leadership or a partnership with the development
One, a general understanding an
agreed upon problem set, security coming to the
table with proposed solutions but not a
dictation on what the answer is, but
a series of answers so that the development leads or development
teams can be part of that conversation.
But being part of that conversation also means that they're now enrolled, that they're adopting, and that means you're going to have greater efficacy in integration and implementation of those. Like static analysis tools, and things like that.
Guy: Yeah. I love the-- Going even back to an early start of that
answer and talking about embedding security knowledge
But it sounds like it's the security champions even, or the likes.
But you're talking about also just as equally important as almost the embedding of
developer knowledge within security, also ensuring that people are
abreast on it.
those shared conversations and when, like in Agile there is
no like "There's some glorified design
meeting that happened that security needs to be a part of."
How in practice does that work? Do you take developers and make them a part of the security team? Do you send your security people to learn how to code? How have you seen that work?
Justin: There's two, what I would say, parallel paths.
One is "How do you get just as much automation of
security into that CICD process as
possible?"--with the developers, o
f course because they're the ones that are actually going to have to deal with the output.
So if they're not part of this, it's not going to go well. In my experience you have really an opportunity to teach developers how to fish, and if you teach them well what comes back to you, security, is exponentially more than what you put into it.
Let me explain.
If I can teach a developer the basic concepts of security
that we look for, and how we look for them in general,
they're able to shard that into the myriad of different
ways that things are actually developed and
implemented and coded, etc.
To be able to come back into that process and actually give
advice on better solutions.
So education and training is really big, but it's very difficult
You have labs. Hands on experience is probably the best one
to do, and so you have those mechanisms and it's a train the
I'm a big believer in that.
Train one developer on the security and have them
host a lab for 10 or 15, and eventually you've got to go through that
But on a day to day basis I believe that
having an individual that is
born from the development community be
identified in the security team
as the security lead, for lack of a better
word, or the security accountable individual for that
LOB, that effectively is part of those scrum meetings.
That's part of that leads.
That's doing the hand-to-hand discussions,
and walking through, and working with the developers
Because I've found that developers in the
community that they have on a working relationship day
to day is very powerful on being able
to get their buy in, get their proactive approach, and get their
focus. Versus a gate check that they have to
So I want people embedded as much as possible. Usually identifying those individuals and having them be a part of it is the best way to go.
Guy: Yeah. What I love about that perspective as well is that it's very analogous or very parallel to how DevOps operate. You've got those sysadmins, ITs that were outside the team.
You look at what happened to those sysadmins, they became DevOps automation and they became SREs. They're probably paid double.
They're embedded into the application while most of the activity, most of the ops activity is handled by the dev teams, but the proficiency is built within-- It depends on the org. But in many times within the org, basically let's apply this for security and create security's equivalent of an SRE.
Justin: It does have some problems though.
So taking a step back and maybe not droning on this too
much, but Agile has come in
years ago but has had a lot of
changes, not just on how software is developed. That's
very easy to see.
the impact on the business, what we call digital
transformation on how we digitize the entire
supply chain, and pull customers in and put
suppliers in, and actually have those analytics dramatically
change the products that are actually being delivered.
Netflix is probably a great example of this.
On the security side it's had a significant impact as well, that I
think we're just now starting to realize, but can
take some insights from newer companies.
As Waterfall to Agile
started to approach, we saw higher velocities.
So what does security do?
We need to get tools in, we need to do training.
But at the output we still have the same
vulnerabilities, or the classes of vulnerabilities.
OWASP Top 10 haven't really changed that much, right?
So, what do we do?
We're gonna put it in an SDK and give it to developers, and give
them standards, we're gonna have third parties come in. And yet what we see is the
same classes of vulnerability are still there.
Cross-site scripting is still probably the biggest vulnerability, and everybody knows how to solve cross-site scripting.
Justin: The question is, why is it still there?
What you have in businesses,
you have this
problem where developers are caught
between a rock and a hard place.
They are on one side being paid and accountable for
feature functionality of a service, and on the other side you
got security saying "You need to be very clean in your
code." So you need to dedicate time to both, and it
just doesn't work.
Which is why we have cross-site scripting issues, etc.
Because the developers are going to go where their paycheck goes. Feature
What I see security doing is
moving to solve this problem, is moving what I would
call from a governance and advisory type function,
to one of governance and product delivery.
The reason for that is, let's take something like
cryptography or data at
REST encryption. Historically
you have two organizations in a security team, an enterprise
security and an application security team.
They would go out saying "You need to have a peak AI
environment, here's your standard end build to solve that
problem." What we see now to be
agile, to be the product security, the security teams are saying
"Stop coding data at REST solutions.
We will do it and provide you a service, an API service, to be
able to leverage it."
That means that enterprise and AppSec teams are combining into
one team, to be able to do the PKI and the
APIs all the way up the stack to be a production
service to the various applications.
That's a very different organizational and skill set model than we've ever seen before, and I think that's a significant change that we're going through right now.
Guy: That makes a lot of sense.
You're tapping into some of the advantages,
there's a lot of conversations about the advantages of DevOps from a security
perspective, and the advantages of agility.
Probably the most well touted one, or most touted one, is
The fact that you can patch faster, but you're highlighting a different one which is, "If the system is more library-oriented and micro service oriented and the likes, then I can interject, or the security team can interject more elegantly by being a component of the system that is security conscious."
Justin: Yeah. When you look at born-in-the-cloud
companies, security started with one
developer solving a problem.
Being in the valley, of course, you talk to tons of
It's one developer saying "I need to
create an identity and authentication mechanism for our service"
"OK, then how did it grow?"
"I needed to create a logging API."
Then, "I needed to create a crypto."
And then eventually we had customers
saying "We need to be certified, so we got some of those governance people."
But it was born a very different model than what
other companies that have been around a long time are going in
So, I think there's a lot of lessons in regards to why they have done it, and the scalability that they have as a result. I think it's an incredible opportunity.
Guy: It makes a lot of sense, and for that you need to transform not just how you work, but actually the makeup and the skill set inside of your security teams to be able to provide these types of solutions.
Justin: Yeah, I would say a security team's staff will rotate about a third, maybe a half, from process management that we do today generally into developers. As a result of that they're developing internal tools as well to be used. But, what does that do to the security workforce?
Justin: I think that's a big impact that will be going on for 10-15 years.
Guy: On the flip side, it might be an answer to the infamous security talent shortage that really does get resolved. Not that workers are that plentiful, but a slightly more varied talent pool there.
Guy: We talked a lot about application security, but when we've spoken
before you were mentioning how application
security extends more, or will extend more
when you compare it to endpoint security, that's an
area that is not as mature as it should be
compared to the risk.
What's your view on the AppSec industry, if you will?
Yeah, I think the AppSec industry is absolutely horrible. The reason why I say that is, not that good people haven't tried, but it's a very difficult problem to create a business.
For example, how many firewall vendors, end point
do we have out there?
Go to RSA, how many of them are dealing with
application security versus anything else?
Yet, when you look at the problems that we face as an
90%, 95%, 98% perhaps
are at the code level.
It makes no sense whatsoever.
The only reason why it makes sense is that it's so much easier to
create a network control access
point than it is to do a mature static
It's very difficult to do AppSec from a vendor standpoint.
Because of that, what you see in organizations is that the solutions that a vendor can put in has no context to the application development service or business in which a company has. It makes it very difficult to create a one size fits all solution when our environments are pretty different, especially for cloud services.
Let me put it that way. So what happens is that the internal
teams in that company need to take
that on, since the product delivery statement that I made about
security teams. I wouldn't say all of them are doing that
necessarily, but that's what should happen.
This is why I say application security is pretty
bad. On one side, we have a complexity and a problem,
and a difficult business model to create a vendor in this
space to solve the problems.
On the other side, we've got developers that are paid for
feature functionality and are not able to dedicate-- Or even
security teams to get the resources to do all the things that need
to be done inside the application to solve the problem.
The third problem I would say is, if we look at any of the other
solutions that are out there to do WAF
application firewalls and things along those
lines, while it's an effort, it's what we will
call a secondary control versus a primary control.
The primary control must be in the
application where it has context, it
understands what that is versus being outside of it.
I think that it's great that we create it initially into the market to get
something there as a whole, while we have a long term
solution on driving it inside the application.
Generally companies haven't done this, so we are only left with a WAF that ultimately causes problems sometimes, right?
Guy: Yeah, it's the same notion that you started with talking about how it was easier to protect the mainframe than all these moving parts. The application has a lot of moving parts. The network there's one of it, or there's five. It's just easier to tackle.
Justin: I mean, nothing for nothing, but how are you able to protect a modern day application that is containerized, is multi-cloud, it's got connections up and down your supply chain and developers are pushing releases at least once a day?
Guy: Maybe the good news is that there is an opportunity to try and crack that in terms of the need of the industry, but the complexity is very much there.
Guy: Is it safe to add to that, there was a shift in people?
I definitely talk a lot about that, which is
one aspect of it, is the complexity.
But I guess I wonder whether
we are talking to the wrong
people, whether the solutions that are built-- I
personally built a product called AppScan Dev Edition that
had developer in the name, but that was pretty much it.
I mean, it was a really good product and it succeeded financially, but it
really was an auditor's product built into a developer environment.
Do you see that happening? Do you think that's indeed an important part of the solution or is that secondary?
Justin: If I understand your question correctly, it's how do
we make sure security is acted
upon by the various LOBs, in this case, the
Versus driving security to the networking or
ops team, which can't really solve the problem?
At least that problem, on the application side.
I think that that's absolutely the case, but that gets
into a very different problem which is, how do you
drive a culture of security into the executive
Second to that, how do you ensure that
security is a business driver, so
that it is important to the executive management team?
Which also requires a security person to be business
We generally have security people,
not to make it overly complex, but we need to be
more participatory in the executive management
team. To do that, we need to at least
have a modicum of understanding of what a business
"What is a funnel? What is marketing? What are conversion rates? What are sales cycles?" That's generally unknown. What that allows you to do is to use all the tools in your belt.
Customer requirements, demands, competitive
analysis of other competitors in the market
from feature functionality standpoint,
solicitation on deal flows and the sales
cycle and how you can shorten that, and bring that to
the product managers and say:
"With better security, we
can increase net promoter score.
We can shorten the sales cycle.
We can actually possibly drive top line revenue, if we have
really lock solid security implemented and have a
lean-forward approach on how we open the APIs
to other partners and networks, and all these other things."
I believe in that model very significantly. It's what I did at Box, as a Chief Trust Officer.
Driving that competitive differentiation on security, it has a massive impact on the culture and what the developers really see as important, because it's coming from the business.
I mean, I consult with my clients a lot on that one topic and it is very rarely-- I've never heard anybody really talk about it, and I feel very passionately about needing to drive that.
Guy: Yeah, I agree.
It's hard, anything that requires an org-wide
change is not something to be taken lightly.
You also hear a lot about that even in the worlds of
marketing needing to adapt to Agile.
It's almost like everything is adapting to this DevOps, Agile
change. Because the whole business is becoming one big feedback
cycle, and security needs to be part of that
group as they adapt the business.
Also, at a certain cadence that is now not just based on
feedback of users.
One of the key challenges is that security's feedback cycle is bad. It doesn't hurt until you're breached, and then and it hurts really bad. It's hard sometimes to know if you're doing it right.
Justin: I really like the way you just put that feedback cycle.
Honestly, I've never heard that.
One, it makes a lot of sense and it relates obviously to a
whole bunch of things.
Yeah, I think if I was gonna phrase it a different
way-- What I was saying a different way, using your words, "How do we
get security part of the feedback cycle of the
They are still focused on hearing about
what Gartner is saying, what customers are saying, and what the competitors aren't
How do we pull, tease out the security-isms of those three and have it be part of lifecycle? So that now they pay attention and drive it inside their words?
Guy: Hallelujah. Now we need to actually get that done.
Actually, with that there's one other bit that I want to make sure that we cover before
we do it, and this is actually a decent segue.
talk a bit about corporate cultures and security within those environments.
You've seen big and small and also over time,
what would you say working with Symantec,
Yahoo!, Box, SAP.
How does the corporate culture play into it? Maybe to try and give it a practical sense, how did you need to adjust your approach? To change what would work, given that surrounding?
Justin: Culture is an interesting thing, and there's no
one-- Take any company, there's no one culture.
But all of them have it, so when you look at
security and you approach about implementing it, what you're really
talking about is "How can I
move the needle for somebody else?
How do I understand what their challenges are, what they're trying to
do and get them onboard with the problem that we're trying
to solve and know their role into that?"
Culture is a massively important thing to not only break into that behavior change, but also solidify security as a core value of that organization.
When you look at a Yahoo!, being in Sunnyvale
and one of the bastions of the
internet back in the day, and coming around.
At that particular time going in it was you had
Facebook, you had Google especially coming in, and a lot of
competition that they once dominated.
There was a bit of a turnaround within the company culture
that needed to be aware.
I had five CEOs within 12 months, not a fun
To get to the developers,
you had to understand the challenges that they were facing
when they were going through their own transformation in the business, and
how can you help solve some of their problems, if you're
asking them to help you solve some of theirs?
It's the "We're in this together, how can we
I think that statement holds true in any other company, at Box for a great
You know what? We're
starting out, we're a startup.
We're breaking through, we're going to go IPO.
We've got massive competition from Microsoft and Dropbox and the
We have limited, massively limited
resources and money, and all those other wonderful things.
We're in this together, and we need to fight.
You can't have that "We're in this together,"when you're doing a keynote
and then walking away, issuing a standard.
That doesn't happen.
You have to be in the thick of it, day to day. Listening, hearing and working, and rolling up your sleeves and working with them. That's the first thing about culture.
The second thing is, the development culture
vs. any other team's culture is very different.
Most security people don't understand that as well.
Where the developers, they're the creatives of
technology, for lack of a better word.
They're very accustomed of being presented
a problem, and they're tasked with
figuring out how to solve it.
Versus what we've historically done in security, which is
security comes up with a policy and standard, i.e.
what the answer is, and we give it to somebody else to just implement.
That doesn't work very well in development communities.
So, what you need to do is you need to go in with not
only a clear definition of what the problems are, but
probably a couple of options on how to solve it and
solicit the input advice and guidance
of the development community on that ultimate answer.
If you don't do that they're going to reject it.
It doesn't work.
You don't understand the development process, the codes,
the languages, all the other complexities that they deal with.
Honestly, at the end of the day, a business is either a product or a sales company. It's not a security company. You don't have the political power. So, you got to be in there, you've got to win the hearts and minds, and to do that you've got to check your ego at the door. Because everybody else knows it.
Everybody else knows you're not as smart as you think you are. Generally, they're smarter than you, in the development space.
Guy: Well, lots to unpack there, I think we might need to
do another episode.
Because there's so much more to even chat about there.
A lot of great insights here, going from of starting comments
about security needing to move from this governance and advisory,
to governance via product delivery.
The need to build security expertise within dev, whether it's the security
champions that make that a part component.
We've talked a lot about how you have to figure out the security business
And from there to understand "What's the value?
How do you turn security into a business differentiator,
and get that into the feedback loop of the business, and fundamentally do all of
that within the context of the culture? To understand
what makes the surrounding tick."
Which it sounds like the end goal is the
same. It's still those same elements of embedding into the
fabric, but maybe the business drivers and
the what's doable and what's not changes
per the organization.
Not easy tasks, but I think really useful perspectives to run forward with.
Justin: I really enjoyed this. I ramble on for 10 minutes, and you succinctly and accurately say "Yeah, OK. One sentence, this is what you're saying." That's fantastic. Thank you.
Guy: Communication matters. So, before I let you go here, I like to ask every guest on the show if you had one tip or one pet peeve you want to talk about to try and give to a team that's looking to level up their security calibre, to do security better. What would that be?
Justin: I love my industry.
I love security.
I feel that I'm possessive, "It's my industry," so to speak.
"We're all in this together." But within that
environment there's a lot of collaboration.
We share a lot within the security
industry. We're open, we're accepting.
So the one thing that I would say to a developer,
is just reach out and say "I want to
learn. I'd like to be a little bit better."
You will be amazed.
To anybody, myself, anybody on LinkedIn, anybody that you see. You will be amazed at the response of welcome and participation that you see to level up education or anything that you might need.
Guy: That's a great tip. Seek out to learn, and you shall find. That's also-- You just tee'd this up, but I was just going to ask if somebody had further comments, feedback, questions for you to ask on the Twitters or others, how can they find you?
Justin: Justin@Somaini.net is my email address, I'm on Linkedin and my website's down but I'll be building that up again. I don't really do Twitter, but I'm on LinkedIn a bit.
Guy: That's good to know, you're a-social network. Justin, this was a pleasure. Thanks for coming on the show.
Justin: Thank you very much.
Guy: Thanks to everybody for tuning in, and I hope you join us for the next one.