Ep. #15, Self-Protecting Software with Jeff Williams of Contrast Security

Grant Miller: All right, Jeff. Thank you so much for joining us.

Jeff Williams: Thanks for having me, Grant.

Grant: Let's just jump right in. Can you tell us a little bit about your background and what you're working on now?

Jeff: Sure. I started writing software in high school, way back in the early 80s, and accidentally got into security along the way.

My first job happened to be on a classified project for the Navy where they had a really high level of security, so Orange Book Level B2, if you or anybody out there remembers that.

I learned a lot about security and I really enjoyed it, so I got on that career track.

Grant: Your first project for the Navy, was that while you were in high school or was that just after?

Jeff: No, it was just after. It was my first job out of college.

Grant: OK, great.

Jeff: Yeah. I worked in a skiff, in a safe underground with a bunch of, like 50-100 people. It was interesting.

Grant: That's crazy.

Jeff: It took us three years I spent on that project, and you DevOps folks might be interested that we never released software one time.

Eventually Congress killed the line item for that project, and so it's on a shelf somewhere with the Ark of the Covenant.

Grant: Oh my gosh. That has to be a little bit infuriating.

Jeff: Yeah. It's like my best programming years, gone.

Grant: All so what did you get into after that?

Jeff: Like I said I got into security, so I got hired by a security consulting company and went off and I became a security consultant for 10 years or so.

That was right at the beginning of the dotcom boom. I got to do AppSec for lots of the early dotcom companies.

That was really interesting, and it was when that field was evolving so I feel like I got to participate in the very early stuff, like SQL injection and cross-site scripting and all those standard vulnerabilities.

We were just figuring them out.

Along the way I got pulled into OWASP, the Open Web Application Security Project, and I wrote this thing that I called the OWASP Top 10 and it was a little side project, I put it out there and it just went crazy.

People started downloading it, OWASP got slashdotted and then a little while later they asked me to take over as a global chair of OWASP.

So I really got entrenched in the AppSec community.

Grant: Interesting. Yeah, that Top 10 list is still pretty well-circulated.

Jeff: It's funny, I feel of two minds about it. In one sense, I like the fact that it's out there and it raises awareness of AppSec.

But on the other hand, it's really not the way I think about application security at all.

I'm really a high assurance person, not a checklist of the top vulnerabilities kind of person. I don't feel like the Top 10 has really changed anything.

In fact, it's the same stuff that's in there after I wrote it in 2002, and it's 2019 now. It's still the same stuff, that's crazy.

Grant: Yeah, that's insane. Is that while you were at Aspect that you were working with those dotcom companies? Or--?

Jeff: It was actually at a company called Exodus Communications, which was a big data center company during the dotcom boom that eventually went bankrupt.

When it went bankrupt, Dave Wichers and I spun out Aspect Security and we took the consulting work and created our own practice.

That company did really well. It grew for almost 15 years and then it just sold to Ernst and Young a couple of years ago.

Grant: Amazing. That brings us to your current role.

Jeff: Yes. While I was at Aspect, Arshan Dabirsiaghi and I had this idea for a different application security tool that worked based on instrumentation, like a New Relic or AppDynamics for security in that research at Aspect.

Eventually we spun that out as a separate company called Contrast Security, and that's what I do now is I'm the CTO there.

Grant: Yeah, OK. I want to dive into a bunch of these different areas.

The first piece that I'd love to talk about, you were just mentioning that you wrote the OWASP Top 10.

I also saw that you wrote an ISO spec for security, so I'd love to get your perspective.

Maybe first let's start with the ISO one, around these standards and certifications because I think a lot of our listeners, they hear from their customers that they want some SOC-2 compliance or ISO 27000 certification.

There's so many different competing standards, so how should an application vendor think about those options and what should they be doing?

Jeff:

There are a ton of standards, and there's really no very well accepted standards in application security.

That ISO standard you're talking about, that's called the systems security engineering CMM.

We did that in the mid 90s and it's a great document, but it never really achieved very widespread adoption.

Like a lot of other security standards, it's out there, but people don't really use it.

I spent a lot of time in the high assurance world doing classified work and so on, and those standards are great.

If you're a security engineer you should read the Rainbow series of documents because it's fantastic.

Those guys knew more about security 30 years ago than a lot of companies know today, but it hasn't changed anything.

There's all these standards, like you mentioned the NIST standards are probably the ones that have the most traction at this point.

So if you're looking for a standard, I'd probably recommend you go look at the NIST cybersecurity framework if you're a CISO, to give you a good sense of the kinds of things to work on.

NIST 853 has got all the standard controls in it, and so on. Those are good standards.

Grant: I think it's actually important even if you're not a CISO, if you're a product manager who's building a tool for enterprises to use I think reading these specs is a really great way to get an understanding of the requirements that your customers are going to have.

If you're trying to sell to the Global 500, these requirements are pretty well spelled out in these documents. Wouldn't you say?

Jeff: Yeah, for sure. Big enterprise customers are going to demand that you're compliant with things like NIST, and SOC -2 type 2, and things like that.

So it is a good start, but I think you have to interpret those requirements for what you're really trying to do.

Grant: Sure. Once I read the ISO spec I understood the playbook of a lot of the CISOs that I was trying to sell to.

Questions you would be asked, you are like "Yeah I know about that." It's just a common way to think about the problem.

Jeff: Yeah, we actually have someone who does that pretty much full time.

Just handle these compliance checks from our various customers, because we get these huge questionnaires that have a thousand questions that we have to answer about how we do security.

You're going to want to get organized about it.

Grant: Yeah, for sure. I think having a pretty clear idea of what those areas are is a great way to do that.

OK, so if we think about all these different criteria like the Rainbow Series or NIST, do you think you should wait until your customers are asking you for a specific criteria?

Or do you think you should just start with one and say, "We're going to try to follow ISO 27000," or "We're going to try to follow some standard that's out there?"

Jeff: Actually, what I really think is the most efficient way to do it is to focus on the big rocks early and make sure you're on a path where you can get compliance with those requirements over time.

But you're not going to be able to do everything when you're a little startup with no funding.

You've got to be strategic about which things you do, so you may choose something like access control or logging or something that's-- Maybe it's single sign on.

Whatever you think is going to be the most important thing, and then as you get bigger and you get more security folks on board you can complete your story.

Most organizations are willing to work with you on that stuff, it's not like when you go in and you get these thousand question questionnaires if you fail one they're going to just say, "Nope, can't use you."

A lot of it will work with you to make sure that you meet their requirements.

Grant: Yeah, that's a great point. I always joke that these vendor security questionnaires are filled out with rose colored glasses.

"Every laptop we have has the latest and greatest version of every browser and operating system on it."

And it's like, "Is that really true or is there some laptop sitting in a back room that doesn't have that?"

How do you think about the fool truthfulness that most vendors fill out these questionnaires with?

Jeff: I think there's a little bit of license there. We're a little different.

When I was in a consulting company, imagine we had 40 or 50 consultants all with their own laptops and all going onsite to various customer locations.

We were doing manual code reviews and manual pen tests, so we had access to their source code and their most sensitive systems.

I knew that if any one of those laptops got compromised, that was a existential threat to my business.

I put it in a very extensive process there to make sure that all the protections were in place, we audited them continuously.

We really took it extremely seriously. We brought some of those practices over into Contrast.

So we're pretty tight about it, but I'm sure that there are some things that we've answered that aren't completely split across every system that we have.

Grant: Right. This, less being about Contrast, because you obviously take this super seriously. This is a part of what you do.

But I think just the industry as a whole, the response to vendor security questionnaires are to me less about the actual implementation, they're more just "Cover your ass if you're the enterprise, and make sure that someone attested to all these things that you want them to have said they've done."

Jeff: The other side of it is that it does make companies do more of the right thing.

Grant: Yeah, I think that they help drive in the right direction. That's a great point.

Jeff: What I find is when they ask the questions, they'll ask a question, "Do you do XYZ?" Maybe technically the answer is "No" because you're doing exactly what that question asked.

But if you dig underneath the question and think about, "Why are they asking this? What is the concern that they're thinking about?"

Then usually you can say, "I understand now. Here's what we do to address that risk."

And you explain to them what you do and why you do it, and they're perfectly willing to listen.

Grant: I think to your point if you think about security, and you're doing things, and you're threat modeling, and you're spending time understanding what the different vectors are and how you could be compromised and you're trying to address those pieces, then you're right.

But I think a lot of companies out there don't really understand the extent at which they could be compromised in their software could be or their employees could be, and they just don't really-- I say that because I know that for my first company I had no idea.

I was just-- We were using every application we saw and it didn't really matter. We'd put data anywhere. I probably was reusing the same password.

It was just things that I did because I didn't start my career in an air-gapped environment, I started my career at a consumer software company.

There's just a lot of people out there that don't really get it right away, so I think the industry as a whole needs to continue to move forward and educate and continue to push people to really adopt these security best practices.

Jeff: I think the game's changed. It's become an environment where you have to be secure to sell to these big enterprises.

They're seeing the same breaches we're seeing. They're not going to trust their enterprise to some little company that's not doing the right security stuff, and I get it.

When you're small company it's very difficult with your limited funding to not only build the thing you're trying to build, but also do all this security work.

It's a real tradeoff challenge that you've got to address.

Grant: Yeah, there's a lot there to do. Before we jump off this topic I would love to just hear a little bit of the story you mentioned that the OWASP Top 10 list just took off.

What inspired you to create this? Then you said it got slashdotted, which is which is a great reference and a throwback to how things used to go viral.

Just tell that story really quick, because I'd love to hear that.

Jeff: Yes. Look, at that time OWASP was a mailing list.

There were like 20 or 25 people on it and someone recommended that I joined, so I joined this list and they were building a guide for writing secure code.

That was an interesting project. Then they started talking about building an environment where people could experiment with vulnerabilities, and I had already built something just like that for the training that I did in AppSec.

So I thought about it, and I was like "If OWASP builds one everyone's going to use that and my thing will just die on the vine, so why don't I just donate it to O-WASP?"

So I donated this project, I called it Web Goat. It's just a safe environment for people to experiment with application security without hacking into Facebook or something.

The response was unbelievable. People just loved it, and I realized as a consulting company donating stuff to OWASP sends all the right messages about what we were doing.

It says we're experts, it says we're passionate, it says we care enough to contribute this stuff.

So I really got hooked on it, and then later that year when I had this idea for the OWASP Top 10, I was like "I could publish it myself and nobody would read it or I could publish it through OWASP and everyone will read it."

So I did that and that project took off, and it was very rewarding for me personally to try to make the world a better place. But it was also really good for Aspect.

I encourage every company to contribute to open source efforts because it really does send the right message about your company. It's a much better way of advertising than wasting $150,000 dollars on a booth at some big conference, take that money and invest it in some open source tool and you'll probably get much more interesting business out of it.

Grant: I love that. I think that's become a very popular way for companies to go to market , around open source and around content.

These are super critical to the go to market strategy today. You were doing it 15-20 years ago, that's pretty amazing.

Jeff: Later I learned about content based marketing, and I saw a lot of people trying to do it, and basically they're just writing blogs about other people's content.

Which I think is really weird. Content based marketing should start with content, you should you should make something interesting that nobody knows.

It could be a study, it could be a paper, it could be a tool. Make something cool, then talk about it. That's content based marketing to me.

Grant: Not just a hot take on somebody else's things?

Jeff: Right.

Grant: That's funny. That's a great point.

Jeff: So OWASP kept growing, and I was asked to take over as the global chair shortly after that. I did that for nine years and we grew OWASP to-- Now it's huge.

There's hundreds of chapters around the world, there's dozens of conferences every year, lots of training and great tools and projects going on.

I stepped away about five years ago when I started Contrast, because there's a limit to how much you can do in a day.

Grant: But you were acting as the global chair for several years?

Jeff: Yeah. For almost nine years.

Grant: That was while you were also running Aspect, which was more of a consulting organization?

Jeff: Exactly right, yeah. Consulting is really interesting.

If you're going to build an enterprise product, consulting in whatever that field is for as much time as you can is amazing, because I feel like I spent years and years working directly with the people who are going to buy my product.

I got to learn exactly what their concerns are, I learned exactly how their use cases work across hundreds of companies.

That's really what allowed us to build Contrast.

Without that experience I don't think we would've gotten the product right, and I don't think we could have even known what the right features were or how it should work.

Grant: At Aspect were you also recommending different application vendors?

Jeff: Yeah, it wasn't something that was a big part of our business. We didn't resell any products.

We tried to use some of the products that were on the market, like static analysis tools, and dynamic scanners, and WAFs in solutions that we were working with our clients on.

But I never found any of those tools particularly useful. In fact, I think they might do more harm than good in some cases, so the service that we provided was largely manual penetration testing and manual code review.

Which is fantastic, and it actually I think scales better in a lot of ways than the other tools in the AppSec space.

Grant: Interesting. So you weren't trying to resell or integrate different technologies, and you weren't selling any technology either.

You were just selling consulting services?

Jeff: That's right. It was just consulting.

Grant: Then why not introduce a product alongside of Aspect? Why spin out a new company in order to do this?

Jeff: If you want to spin out a company you've got to have a really good idea.

We did a little bit of product, like we had an e-learning product that we made and sold at Aspect, but when we had the idea for Contrast we realized it really needed to be a separate company.

It just doesn't work business-wise to have a product company and a consulting company smashed under one entity.

The business models are very different, the selling motion is very different, the marketing is very different. It really just needs to be a separate company.

Grant: OK, sure. But you were able to take your years of experience and all the knowledge that you gained, that's what you're saying, as a application vendor to have worked as a consultant for so long and have run a consulting company.

In seeing so many different enterprise use cases you felt particularly well prepared to solve this problem with these enterprise customers.

Jeff: That's exactly right, yeah. Because we'd work side by side with them solving the problem that they needed a product to solve.

The only problem was doing manual consulting is that it just doesn't scale, so we needed automation but we couldn't find it from the existing scanning tools.

So we were always looking for a new way to solve that problem, and when the job instrumentation API came out in 2009 we thought, "This could be it."

Grant: Interesting. Let's talk a little bit about it, because you have a fairly particular perspective on application security and the future of application security, which I think is super relevant for the various application vendors out there.

Let's dive into a little bit of that, and maybe provide some context and background, and then your perspective.

Jeff: Sure. So AppSec is pretty hard, I think of network security a little bit like Lego because it's constrained in the way you can fit things together.

But AppSec is more like clay, you can build anything with software.

So there's many more degrees of freedom in screwing things up, and I see the future of AppSec as moving away from scanning and firewalls.

I think that approach has run its course. Those tools don't have enough information to be really accurate about AppSec, and because they're not accurate it means you have to have experts involved.

If you have experts involved it's really not automation, it's just experts with tools and it doesn't scale.

So the biggest problem I see in organizations is the ability to scale AppSec, and as software gets released faster and faster the workload on the security experts goes up and up and up unless we figure out a better way to automate.

So for me the transition is about getting from this outside in approach of scanning and firewalling, to an embedded approach where the security is happening within the thing that we're trying to protect. That make sense?

Grant: Yeah. So it sounds a little bit like it's harder to tack that on later, and it needs to be built more into the actual applications?

Jeff: There's two pieces. There's the security defenses, which you have to build in and you got to build them in over time as you build the product.

But then there's the verification of those defenses that has to be happening continuously as you're building the product, so what we found was if you're analyzing just the source code you get one view of an application.

If you're looking at the HTTP traffic that a scanner would put out you see another view of the application, and if you're looking at libraries and frameworks you get another view of what's in that, and if you're looking at a WAF or something in a different view of the app and none of those different views have all the information.

It's a little bit like that parable of the blind man and the elephant, each person can only see one part of the elephant so they make all these mistakes about what it is they're looking at.

That's really what we see in AppSec tools, is all these point tools that do their thing but they make a lot of mistakes.

What we thought is "If we could get inside the application, the same way that a profiler or an AppDynamics gets inside the application, then we can measure all that stuff.

We could see the code, we could see the libraries, we could see the data flow and control flow, we could see back end connections, and configuration, and libraries and frameworks.

We'd have this huge information advantage so that we could accurately see vulnerabilities and report them to the folks that need them immediately.

Then we can also identify attacks the same way and prevent them from exploding applications."

This isn't just happening at the application layer, this trend to secure things from the inside is happening across the whole stack, so we see it in cloud products like Threat Stack.

We see it in OS products that are moving inside the application, even some of the traditional scanning vendors like Tenable.

Now instead of scanning from the outside, now you get an agent that runs within the platform itself.

Grant: Cool. So if I'm an application vendor and I'm selling either a SaaS product or distributing it privately for folks to install, how should I think about--?

Should I be integrating this with my application when I am running it in the cloud? How do I do this?

Jeff: I think there's-- Again, there's two pieces.

There's what you do during development to make sure that the thing that you're producing is secure, and then there's what you do in operations to make sure that your application is defending itself in the environment that it gets deployed.

During development you want to make sure that you get those tight feedback loops on security, so as soon as your developers make a mistake with security, let's say they introduce a SQL injection flaw.

There should be instrumentation that detects that vulnerability and reports it right to them through the tools they're already using, and hopefully it takes -- It comes back within like a second.

That's the most cost effective way of doing security, and then if you do that process you'll be producing applications that are relatively free from vulnerabilities.

We're not doing that today. The average application has 26.7 vulnerabilities, which is a horrific number.

You'd never fly on an airline if when you checked an airplane it had 26.7 safety problems.

Grant: True. You'd never take off, you'd be sitting on the runway the whole time.

Jeff: Yeah, of course. But we put out code that's like that all the time and these are the apps you use every day.

Online banking, your healthcare apps, everything. It's all like that.

If you do what I'm suggesting and get that process in place so you can get instant feedback on the code that you're building, developers will fix those problems in stride and check in clean code and you won't build up this backlog of vulnerabilities.

I know organizations that have 30,000-40,000 vulnerabilities in their backlog that they're not even getting to.

So then if you do that, you should be getting into production with something that's relatively clean, which is great.

But if you're out there in production people are using your application, maybe it's on prem somewhere and it's not getting updated frequently.

A new vulnerability could get discovered in some library, or a new class of vulnerabilities could get discovered that nobody knew about when you were writing the code, so you want to have that runtime protection in place so that your application can defend itself against both known vulnerabilities as well as novel attacks.

That seems to me, that belt and suspenders seems like a really good balanced approach to application security if you do those two things.

Grant: OK, great. Just for reference, Contrast plays in both those worlds?

Jeff: Yes. We're a little unique there. Using traditional tools, you need to buy a SaaS tool and a DaaS tool, that's a static analysis tool and a dynamic scanner.

You'd probably also need to buy an open source analysis tool like Black Duck or something, and then you'd probably need to buy a WAF to protect it in production.

Contrast does all of those things because we work from inside the application, we analyze the app as you're building it and we analyze all your libraries and frameworks and tell you if you've got any known vulnerabilities, and then in production Contrast protects the application against being exploited.

We make vulnerabilities like SQL injection and expression language injection and XXZ and a bunch of different other kinds of vulnerabilities impossible to exploit.

Grant: Should those vulnerabilities have been--?

Did you already detect them in the developer environment, but the team just didn't have time to fix them and so they're deploying it even though they know there's vulnerabilities and your software is being the gatekeeper to protect those vulnerabilities before--?

Even though they know that they're an issue?

Jeff: That's a common way to use Contrast, is to give you that breathing room so you can deploy even with vulnerabilities and then go back and clean those vulnerabilities up later.

But I think even if you banned applications from going to production with vulnerabilities, there's still a great use case for putting some runtime protection in place because of these new vulnerabilities that come out.

There's dozens of new CVEs that get reported every week. If you're application is using one of those libraries, then you need to fix it within about a day.

If you think about the Equifax breach, I'm sure you read about, they were using an old version of Struts and it had an expression language injection vulnerability in it.

They took several months, three or four months actually to get on top of that. In the meantime the attackers breached their application, and it was a total host compromised so the attackers could do anything they wanted on those hosts for quite a long time.

Those are the kinds of things that you need to be able to respond to almost immediately, because as soon as those new vulnerabilities come out we see a huge rise in automated attacks across all of our customers that get attacked by these things almost immediately.

People are weaponizing these things and turning them into scanners to just search the internet for those vulnerabilities right away. I think you've got about a day.

Grant: Great. So from a vendor's perspective, if I'm using tools like Contrast, can I use this to help demonstrate the security of the application that I'm hosting or delivering to my customers?

Is there some type of--? Because one of the challenges with an annual pen test and white box code review is that this is a point in time analysis.

So with something like Contrast, am I able to produce something that is very up to date as of the day they requested it, and show them consistently that this is our--?

Almost like uptime, "Here's what our uptime was for responding to CVEs," and things like that.

Jeff: Yeah, exactly. I think this idea that you get a security test done right before production deployment, that's an ancient concept.

I think really what you want today is continuous application security verification and reporting.

A lot of government standards are moving that way, actually government is ahead of the game here in terms of the continuous compliance movement.

There's a number of standards, like the Sands stuff and some of the NIST things are moving towards compliance.

PCI standard is also moving towards continuous compliance, and the idea is you shouldn't ever really be untested and you should be continuously testing for security problems.

So yeah, Contrast provides that evidence where you can be continuously up to date dashboard on every one of your applications in parallel.

Grant: That's cool. I'm debating it with myself if there's an interesting concept, which is obviously there's things like status page for uptime.

You can show there's different outages different companies have, so if Salesforce was down and Google Calendar was down, and everyone's showing their status.

There's obviously status pages at a company, and showing the status of uptime.

There's an interesting level of transparency you can provide if you publicly share some of these dashboards around the continuous security and CVEs or other aspects of your security to your customers for their information.

Obviously there's a level of challenge there, because if you have CVEs you don't want to say "We're running this CVE in production," because then it's going to make you a target.

But do you have any thoughts on making that more available to customers more publicly?

Jeff: Yeah, I'm trying to make that day happen.

I've put forth a lot of ideas around that concept, so I just wrote a piece for Forbes that challenges companies that says "Who's going to be the company in your sector that decides to do security right and does it publicly?"

A little bit like Microsoft's trustworthy computing initiative from the early 2000s, they turned the company around and they made security public in a lot of ways.

I talk a lot about "Doing security in sunshine," it's really important to do it internally anyway.

Security doesn't thrive when people turn it into shame and blame, and that's when I think security gets bad culture-wise.

It's really best to be very open about security internally, at least to talk about "Here's what threats we have. Here's who's attacking us. Here's what the attacks look like. Here's where we're vulnerable."

Everybody should know that , that gets your company aligned on security and makes it-- It drives the importance of it for developers.

I strongly encourage you to take some of the attack data, that hopefully you're seeing application layer attacks and you've got some visibility.

Take that data and make sure developers can see it every day, so they know how important security is.

That culture I think is really powerful, so I encourage you to do what you can about making it visible internally and hopefully someday, eventually, I'd love to see a day when companies are public about their application security efforts.

Right now think about your online bank, what do you really know about how they do security?

Remember, you're trusting them with your finances and a lot of your personal information, maybe a lot of your transactions.

They know all that, and you're trusting them, but my guess is you have absolutely no idea who wrote the software or how they were trained, how they tested that software for security, how it was deployed, whether they have known vulnerabilities in their open source, how often they do security testing. I bet you don't know any of that. That's really scary.

I call that "Blind trust," and it's dangerous.

Grant: That's funny. I don't think about it very much for myself personally, I do think about it a lot for the software that we use at Replicated because I think my personal finances I'm not as worried about as I am about the information that we have from our customers, or that we could produce internally.

I think that's a much more-- It's orders of magnitude more valuable, so I'm very conscious about how we use that software and what we do for security at Replicated internally.

Jeff: Interesting. I bet most people think about their personal stuff first.

Grant: Yeah. I guess at the same time, as a founder, most of my personal wealth would be tied up in Replicated as well.

Jeff: Yeah. But it's not like credit cards, credit cards you're covered for $50 bucks. But if somebody breaks in your online bank and drains your account, there's a good chance it's just gone.

Grant: Yeah. That's why you've got to hold it all in a crypto wallet, in Bitcoin.

Jeff: Yeah, there you go.

Grant: Now that we know more about Contrast, which is super helpful, and just generally the perspective around security and application security and how vendors should be thinking about that.

That's all really useful information, and I'd love to dive into a little bit more of some of the Enterprise Ready components of Contrast.

The first piece I think is always interesting is just understanding how when you started this company, how did you first go to market and how did you get adoption?

Then I think that informs some of the other decisions you made around features to build, so can you talk a little bit about the origins of the company, how you got it out there, who your first users were?

Jeff: Contrast started a little bit differently from most startups because we were being spun out of a consulting company, so we already had a bunch of potential customers in mind and they were all very large enterprises.

The largest Wall Street banks, and some of the biggest international financial institutions like World Bank, and IMF, and the Federal Reserve and places like that.

When we built the product, we built it for those enterprises out of the gate.

We didn't start with an open source project and then make a commercial version and move up from small companies to big companies.

We went right for the biggest companies, so they pushed us really hard on those enterprise features right away.

We had to add enterprise logging, and access control, and single sign on, and multi factor authentication, and all those things. We had to build those really early in our process.

Grant: Did you self-fund? Did you raise money? Because oftentimes building those features can be a little bit costly.

Jeff: Aspect funded the first year of development. We had a team of roughly 10 people working on the product for a year, and Aspect did that-- I think of it like a seed round, essentially.

Grant: Sure.

Jeff: We did well, we had some nice sales to some large organizations and that really put us in a great position to go raise an A round.

Which we did, we raised an $8 million dollar series A in 2014.

Grant: OK, great. So you were building this internally at Aspect, this makes a lot of sense. When I asked earlier, "It didn't make sense," you spun it out of Aspect.

You started it there, you used those relationships, you seed funded it with this experience, and then you had 10 engineers on it.

That's a pretty good sized development team. How many design partners or early customers were you working with at that point?

Jeff: I wish we'd been smarter about that, but this was really our first steps in product development.

It's a little bit hard if you're building a revolutionary product, it's a little hard sometimes to work with design partners.

At least in the early phase, because we had a vision of a different way of doing application security.

We weren't a scanner, and everyone's just looking for a better version of what they're doing today.

We had a very different idea and it was very difficult to show it in a partially finished forum, because people don't get it.

They're like, "Where do I push the scan button?" I'm like, "We're not scanning. We're continuously monitoring in the background. There's no scanning."

"Yeah. Can I stop a scan in progress?" And it's just like, those kind of questions--

We did a lot more of that on our own just with an idea of what we wanted it to be then I probably would do today.

I'd probably work more closely with partners if I had it to do over again.

Grant: Yeah, I get it though. It's hard sometimes and you're trying to bring something that's not really well understood in the market, you're trying to create a new category and that's a very difficult thing to do with early partners.

You have to really find folks that truly get it from the beginning, and they're not just looking for a better static code analysis tool.

Jeff: Yeah, and it's tough in the early days when you have this vision where the product doesn't really work as well as you'd like.

That'll alienate your early partners pretty quickly, so sometimes I think you got to just put your head down and build a flying car and then show it to people and then they'll get it.

Grant: Sure. That first year while you were incubating it, did you have users or did you wait until you had raised--? When did you get the first people really using this?

Jeff: We definitely had users, so we sold a few very large banks and we could've done better on customer support.

I think that's really where I would have liked to spend more time, "Let's get a partner, let's get them using the product and then let's really support them to make sure we make what they want."

But we were really still instrumentation for security is tricky and not every language has great instrumentation support, so we had a lot of technical challenges to solve in those early years.

We spent our time doing that, and thank goodness we got enough commercial interests to allow us to raise an A round and eventually go on to grow the company.

Grant: OK. In terms of founding team, who else was super involved early on and what were the different roles?

Jeff: Arshan Dabirsiaghi was consultant at Aspect, so he and I did the early work on just getting the concept to work.

Then a number of other folks at Aspect joined the team there. Folks like Harold McGinnis and Edin Sinai, and Dan Fidler and Shane Dabirsiaghi also.

So we had a nice team of folks that helped us get that product off the ground.

Matt Austin joined us a little later, and those guys are super security researchers and great developers. They really helped us move the technology along in the early years.

Grant: This is mainly a technical team, or are there anyone doing sales and marketing early on?

Jeff: We leaned on the Aspect sales team a little bit. Actually, a lot.

So Pete Deane was instrumental in helping us sell Contrast to some of the New York firms.

It was interesting because we had that Aspect infrastructure in place, we didn't have to do things like finance and accounting and HR. Those things were all in place.

Grant: Yeah, that's great. So now it's a year in and you have a couple of customers that are using it and you raise $8 million.

Then is it go and just scale, go to market? Or is it still you were figuring out the product at that point still?

Jeff: I think there was still a lot of figuring out product market fit in those days.

We had our concept, but it was still a lot of evangelism and a lot of legwork to get customers.

Really, we just had to show it to customers and once they saw it, customers generally loved it.

But it's a lot of face to face meetings to get them to see it and to get them to try to understand it, so it's some work but we think it's so much better than the legacy SaaS and DaaS scanners that-- It's very compelling, once you actually see it.

Now interestingly the first product was just the Assess product that just finds custom code vulnerabilities, and later we added our library analysis piece and then slightly later we added the runtime protection piece.

We rolled those products out, I'd say maybe every year we had a big release.

Grant: Nice. Those first discover, and then it was discover in different external libraries, and that it was protect?

Jeff: Exactly.

Grant: Yeah. That makes sense, it'd be harder to go the other direction.

Jeff: We weren't the smartest about how we built all the product out.

We've had a number of features like for instance, we had this gigantic rule editor built into the product for creating custom rules, that you could create custom rules and it's all through the web app.

It was way too early for that, nobody ever used that feature. We have a few customers using custom rules now, but it's really not that well adopted.

We pulled it out of the website and you can still do it, but it was just too much extra fancy features to maintain.

Grant: So when you're developing those, because that's three different important feature releases and new product introductions, do you think about that in terms of a product go to market?

There's an alpha, a beta, a GA? Or how do you think about releasing those different features, are you doing the same iteration with early design partners?

Jeff: It's a lot different today than it was back then.

Back then we would just build it and we'd put it in front of our customers and try to get feedback.

Now it's much more, we're a dev ops shop now so we release six or seven times a day. We've got almost 100 people on our development team and they all work on different parts, so there's a team for the job agent and a team for the dotnet agent and Ruby and Python and Node.js.

We've got a team focused on our OSS product, we've got another team on our Protect product.

We're pretty organized now and the features roll out the way you'd think, we've got a well-developed roadmap.

We work with customers to get their requirements and then we build an MVP version, we get feedback and then sometimes we release it under a feature flag so people can opt in and then eventually it becomes GA.

But it wasn't always that sophisticated, that happened over time.

Grant: Cool. I'm guessing in the beginning it was just like, roll it out and show it to some customers, and someone's building a piece of collateral to describe it.

But now you have a much more formalized process.

Jeff: Yeah, so we put out a new roadmap every month.

Both internal and external facing roadmap, so we're constantly thinking about how we adjust what we build to meet with customer demand.

Every time we talk to a new customer it's all in JIRA. If they ask for some feature we note it in there and we keep track of who's asking for what, and that really helps drive our roadmap forward.

Grant: OK. I'm guessing you had some idea that you wanted to be able to offer the Protect product, or was that all just based on customer demand? They were asking for it?

Jeff: That was still pretty early.

I think we had this idea that it would just make sense to have one product do application security across the entire lifecycle from the first line of code all the way through production and operation, rather than having to buy four or five different AppSec products and have a team around each one of them.

We saw the same advantages in the Protect environment that we see in the Assess environment, like having more information just makes you more accurate.

Really I think we had this idea that it would just be a much better mousetrap, so we decided to go build it.

We did talk to people, but I think it wasn't something that they brought to us and said "We really need this."

Grant: Yeah, OK. I think that's a really interesting point, which is your roadmap is combined with trying to execute on the vision for where you want the company to go and offering you want to have and how you want to be perceived in the market, vs. how do you want to take customer feedback and then use that as the impetus for delivering the next product?

Jeff: At this point it's a really complicated set of priorities. The sales folks want a certain set of things, the product teams they've got a vision.

Me as a founder and AppSec evangelist, I've got certain things that I want to be able to talk about that are consistent with our story. All those concerns have to be balanced in the roadmap.

Ultimately if it's not in the roadmap, it doesn't get done. We have to have a single source of truth in terms of what we're actually going to build.

I can't go to the engineering team and say, "Look. I'm a founder. I just really need this feature. Can you do it?"

That's not how we do things. It's got to be balanced against all our other priorities.

Grant: Yeah. That makes a lot of sense. Cool. So going a step back, when you first introduced the product I looked at your pricing page and you have these two different tiers.

You have the community here, which is free, and then you have the enterprise plan. Did you--? I think you said you started off with just the enterprise plan. Is that right?

Jeff: That's right. We just launched our community edition less than a year ago.

Grant: OK. Four years?

Jeff: Four years.

Grant: Yeah. This was just a an enterprise offering, "Contact us for pricing information?"

Jeff: Yeah, that's right.

Grant: OK. And then last year you decided to roll this out to really increase the top of funnel and get more folks trying out your software, and-- Is that right?

Jeff: I'd say really the biggest concern is changing the way the market thinks about application security.

If you invent a flying car in your garage and you just drive it around your neighborhood, or maybe you just sell it to rich guys and they drive it, nobody is really going to find out about it.

We want to make sure the world knows there's a better way to do application security, so we thought about it and we said "What kind of community edition can we offer that would make sense?"

And we said, "We don't want to do a feature restricted version of the product, because then nobody's going to commit to using it. We want to give them the full thing."

We said, "How can we somehow make that so that we're not just giving the product away and that our investors won't get really mad?"

So we said, "Look. We'll give it to you, the full thing, for one application."

That'll allow small enterprises that just have one app, maybe it's just their website or they're doing some online commerce or whatever, they can use Contrast for free forever. That's great.

For bigger organizations it gives them a way to easily try the product, and individual projects can build it into their pipeline and really get a ton of value.

They can get that continuous compliance really quickly, and if they want to do more, and if they like it and they want to do more they can they can talk to us about a commercial license for the product.

Currently, it's only Java, in one app in Java that you can use Contrast on, but we're going to be releasing support for the other languages that we support very soon.

So dotnet, Node.js, Ruby and Python and dotnet core as well. Those will all be coming out shortly.

Grant: Cool. I always call this product assortment, which is how do you decide which features to offer in which plan?

It sounds like generally you wanted to make most of the features available for free, do you hold back any features like security audit logging or role based access control, or your single sign on?

And only allow those to be used in the enterprise version?

Jeff: It's not that we're holding them back, it's that there's some features that don't make sense if you've only got one app.

Access control for instance, if it's just one app team you don't need a complicated enterprise access control model, things like that.

We don't have LDAP integration and a few features like that in the community edition, and it's all SaaS, so we're trying to make it really easy for folks so they can just use the SaaS. They don't have an on prem option.

Grant: OK. Our thesis is that you actually are better off holding some of those more complicated features back, because that way you're not confusing the smaller companies and smaller teams with features that are really only required once you're doing this at massive scale.

Things like RBAC, just granular RBAC doesn't make sense when you're doing this for a couple of folks.

Jeff: But we didn't want to do a time limitation, because then I think people won't commit to using it in their infrastructure.

If you've got a product that's going to time out in 30 days or 60 days or whatever, why would you build that in your pipeline?

You're just going to have to rip it out. And that's not what we want people to do, we wanted people to really use Contrast and make it part of their new way of doing application security.

Grant: Yeah, that makes total sense. You also mentioned, we talk about deployment options.

You say you have SaaS-only for the community edition, but you do have an on prem version for your enterprise customers.

Was that from day one, or how did that come about?

Jeff: It was from day one, because remember we started with all these Wall Street financial institutions and many of them at that time required on premise, particularly for security products. That is changing.

Now the majority of our customers use our SaaS, that actually is the vast majority of our customers use our SaaS.

Even many of the very large financials, but there still are entities that they just won't do that.

They're just not going to use a SaaS product, so we still support the on premise product and it's exactly the same. It's one codebase, it's just a different way of deploying.

Grant: I'm guessing your application is written in Java as well, based on the community edition and the focus on Java early on.

Jeff: It is, and we're pretty happy with that choice.

Our product has two pieces, it has our agents that run as part of your application. So for Java, it's a .jar file, for instance.

That's where a lot of the complex security stuff happens, all within your running application.

But the Contrast team servers, the other piece, that's the piece that's either on prem or in our SaaS.

That's more straightforward, it's a bunch of APIs and we use Spring and a bunch of standard enterprise dashboard kinds of technologies.

Grant: Cool. You're bringing this out to market, so you see the future of Contrast as wide adoption across companies large and small, or do you think you'll continue to have a strong focus on enterprise customers?

Jeff: We're growing really quickly, so two years ago we were probably 30 or 40 people and now we're almost 200.

We're going to be another hundred by early next year, so we see the market is really currently where we sell to is anybody building web apps or web APIs.

We still sell to a lot of large enterprises, but we also have a team that focuses on companies with less than a thousand employees.

They're doing really well and we're seeing that market develop, so that's exciting.

But ultimately this is thinking long term, I think it's crazy to deploy applications without instrumenting them for security.

What I would say is everything that's complicated in the world, we instrument it so that we can know what's going on inside it.

Your car is totally instrumented, industrial factories are instrumented like crazy for sound, and fire, and vibration, and everything.

Temperature, space shuttle, airplanes, all super instrumented. But software, which is arguably the most complex thing man has ever built, is almost not instrumented at all.

Log files are a terrible form of instrumentation, so we think there's a real need for instrumenting applications. Security seems like the perfect use case for that.

It's a little crazy to deploy these complex applications, bet your enterprise on this application and have really no idea whether it's vulnerable, whether you've got components that are vulnerabilities in them, and if someone's attacking them in production.

Most oppositions don't know any of that and that's nuts.

We're trying to change the world to a place where people actually know whether their stuff is secure or not. It's a realistic goal for companies to know where all their code is running across Dev and QA and production. They should know, they should have an inventory that tells them exactly where all that code is, exactly what's in it.

They should have a bill of materials for each one of those apps and know which back end connections each one of those apps is making.

They should have a point of contact for each one, and they should have a live dashboard that shows them "Here's what security looks like for that app."

It should say, "You've got 12 vulnerabilities in dev, but in QA only two, and in production you're being attacked from these places, and they're using these kinds of attacks. They're targeting these business functions."

You should know all of that for all of your applications all of the time, and it's possible to do that in parallel across very large portfolios.

We have many customers that have many thousands of applications that are getting all that telemetry all in parallel, and when you start thinking about that, all those applications secured in parallel.

You can really see the weakness of the scanning paradigm where you have to scan one app and then move to the next one, scan it, produce a PDF report that goes to some experts, scan the next one, and so on.

The strobe light visibility just doesn't cut it in 2019.

Grant: Yeah. The speed of development, the speed of everything we're going, I think that application vendors have to think about delivering a very secure application.

Then I think another part of it is using other applications that are secure, we always talk about the daisy chaining.

If Contrast is using some other vendor that's insecure, that's a huge area of vulnerability.

We all have to be cognizant of the security of all the tools we use, else it's just the weakest link ends up being the place where things break down and data is leaked.

Jeff: Yeah, that's right. Nobody is writing all of their code all the way down through the operating system these days.

Grant: They're barely writing the code even to do the application, especially when the node world is infamous for this.

Embedding so many different libraries to perform small functions.

Jeff: Sure. And that totally makes sense to me. Why should anybody write replace all function?

You should use one that somebody has written and tested. I always tell people, "Never write security mechanisms yourself. It's really hard."

I ran a project at OWASP called the Enterprise Security API where we said, "Let's take the top 120 methods that developers would need to write secure code and we'll just write good versions of them, we'll test the hell out of them, and we'll provide them for free to anybody so that they can use these as a starting point."

It's really hard. We wrote, even simple things like validating strings and doing encoding and decoding and canonicalization, those things end up being really pretty tricky when you get right down to it.

So we spent several years building great library to do that.

Grant: This is one area where I think that security has done a much better job than most other areas of development, and even dev ops.

I think there is a lot of other areas of software development have a "Not invented here" syndrome. "I didn't write it, so I don't trust it."

I think in security it's often the opposite, where it's understood that if you wrote your own encryption library, you probably didn't do it correctly.

You should go with the open primitive that's available and leverage that, so you have done a great job establishing the patterns and building the primitives and making those available.

Jeff: For sure for encryption and hashing, but much less so for things like input validation and even access control.

We see so many people writing their own access control mechanisms, it ends up being a big mess of boolean logic.

I think the message hasn't been received across all security controls equally, and so many people write their own encoding methods.

They're like, "I just need to HTML entity and encode this string, so I'll just encode the less than and greater than signs and maybe single quote."

But then they forget about double quotes, or they forget about certain edge cases on Unicode characters, and things like that. It ends up being really hard to get it perfectly right.

Grant: When you talk about access control, are there specific tools or libraries that you think folks should be looking at for when they're developing their internal access control options?

Jeff: Yes. Generally I think people should stick to the options that are in the frameworks that they're using, so if you're using Spring then there's some annotations you can add for access control.

If you're using other frameworks then you should use the functions and features that are built into that framework. If you start trying to build your own or wedge in some--

That was ultimately I think was the undoing of the Zappi project was that it was all standalone mechanisms.

If you're using a framework it gets really hard to use the standalone mechanisms because they're not integrated into the framework.

You can't just put an annotation, instead you've got to make a bunch of complex method calls.

So really, it ends up being the frameworks that end up winning here, so folks should really learn how to use their frameworks, study them and come up with a secure coding guideline that says "Here at Company XYZ here's how we do access control.

You use this annotation in these locations, and you have to-- If there's a access control failure then you need to catch that exception, or test the result or whatever you do, and then make this long call and take this action to put up this error screen on the page."

You've got to spell that stuff out so that everybody does it consistently in every place through the product, otherwise every developer is just going to do it a different way and you're going to end up with these log files that are full of garbage and error screens that are all over the place.

You should just have a strategy for doing each of these things.

Grant: Yeah, I'd love to find a way to get a guide like that open sourced, someone's internal access control guidelines, and provide that as a resource.

That seems like something that other companies could model off of.

Jeff: Here's the thing, I've come to this and I've probably written 50 of these secure coding guidelines over the years, I don't think people really read them.

I think mostly they go on the shelf and people write code the way they're going to write it, so my thing now is we have to take that security guidance and turn it into code.

It has to be automated as part of the software pipeline.

One way of thinking about Contrast is as a platform for creating those guardrails around application behavior, so I'd love people to take those rules and articulate them as Contrast rules that then get enforced automatically.

You can easily-- If you had an access control pattern like the one I just said, you could easily put that into Contrast and say "In every one of our APIs we have to make this access control check with this invitation, then we have to make this log method call, and then if there's an error then you have to do this thing."

Those kinds of behaviors are easy to verify in in Contrast, and I think that's how they ought to be verified, because that's a way that developers get instant feedback.

They'll get something that says "You blew the access control policy. Here's what you need to do to fix it."

They'll just do it, fix it, and check it in clean code and then "OK." you're done with it. There's none of that downstream work on tracking, and risk rating, and putting in a risk register, and making a ticket, and fixing it, and retesting it, and all that. All that work just goes away.

Grant: Yeah. Getting things surfaced immediately rather than having a long feedback loop I think is the key to addressing those issues.

Jeff: Probably everybody's seen that chart that says "If you fix it in dev, it costs 1X. If you fix it in test, it's 10X. If you fix it in production, it's 100X."

I think that chart is wrong, because I think it goes up really fast. If you wait more than like, I don't know, 30 minutes after the vulnerability is created and checked in then the cost goes up dramatically.

It's not a straight line, it's a really bumpy curve.

Grant: Yeah, that makes a lot of sense. If someone wants to find you, you mentioned you've written a bunch of these secure coding guidelines.

Obviously we mentioned the ISO one earlier, are there any that you've written more recently that you're really--? That you think folks should take a look at?

Jeff: I haven't done it in five years because I've been building a product company, so any of the guidelines I wrote back in the day are probably not super relevant.

They're also-- They were built for private companies.

Grant: OK. Got it.

Jeff: The good news is they're not that hard to create.

There's probably 15 major security areas that you want to deal with, and you could start with something like the OWASP ASVS project.

That's the application security verification standard, and it's got a real nice list of the kinds of things that you want to cover.

It's a little more in-depth than something like the OWASP Top 10. So you could start there and then go through each one and say, "OK. How are we going to achieve this goal in our framework?"

You might be using React and Node.js on the server side.

Maybe you're using some framework, and you just say "OK. Given the technologies that we're using, what's our pattern for implementing these different features?

How do we do authentication, and access control, and error handling, and logging, and input validation, and escaping, and encryption?

All those kinds of things? How do we do back end connections? Are we going to put everything over SSL? Where are we going to store the keys?"

You've got to figure all that stuff out, and then once you have it, you can turn those into-- And it doesn't take that long.

Once you have that, then you can try to think about "OK, how could we automatically test this as part of every commit?"

Grant: That makes sense. Related to this, one of the really important patterns or permatives we see emerging is Kubernetes, so we see more and more folks deploying applications to Kubernetes.

When they start that maybe they're deploying services, and it's focused on reliability, but I think a lot of the Kubernetes ecosystem recently has had a focus on service mesh.

Service mesh, be this Istio or Linkerd or Console. They all have some amount of claims around security and system to system communications.

Do you have any thoughts around--? Does part of this move from--?

Maybe this is the framework concept you're talking about, does this move up a level from language-specific frameworks into deployment-specific frameworks?

Jeff: I think that it's a pretty good place to enforce some of the security in the stack.

It's a nice place to put in rules about saying, "You got to have all your patches applied and your host up to date. You've got to make sure you don't connect to arbitrary systems out on the internet. You can do some of the zero trust rules and enforce those there."

But in terms of the application security, it's not a particularly good place because Kubernetes can't really see what code is running in the container.

It can just see that there is a workload there, and they're going to run it, but they won't have access to see something like SQL injection, for instance.

That's just going to be a request coming in, the code is going to handle it and it's going to create some malicious query, and it's going to send that to the database.

Kubernetes is not going to have anything to say about it, so I really think what you need is an instrumentation-based approach to security across whatever stack you're using.

I'd love it if at the application layer you chose Contrast to be that instrumentation-based approach, but if you're using a container, maybe you're using some Kubernetes-like product or maybe you're using a Docker-like product like Twist Lock or Aqua.

Maybe if you go down the stack maybe you're doing something in the operating system or at the cloud level, maybe you're using Threat Stack or Carbon Black or one of those.

But the products that are within the thing that needs to be secured, I believe is the right approach to security. I think we're past this approach of scanning things from the outside.

In the old days scanning used to be the efficient thing to do, because you'd set up a scanner and then you'd just scan stuff. It was all very simple. But modern enterprise architectures aren't easy to scan, there's networks everywhere and there's internal systems and external systems and containers and cloud and it's all over the place. So now the easy thing to do is to just make an agent part of the stack that you're deploying and have it go everywhere, wherever that stack goes.

I don't care if you move that stack from an internal data center to an external one, or to a container or whatever, it doesn't matter. The security goes with the workload then.

Grant: Yes. Your point is, if you're trying to apply security at the deployment runtime level, it's a little too late.

You should be getting in their earlier, instrumenting the application code, and there's obviously security at every layer that needs to be applied.

But starting in the foundation is really important.

Jeff: Yeah, I think that's right.

Grant: Cool. Jeff, this has been so good. I really appreciate it.

This has been an amazing overview of how folks should think about their internal application security, the security of their vendors.

There's so many different areas where I'm thrilled for people to dive into more. So, thank you very much.

Jeff: It's my pleasure. This has been great.