March 3, 2017
CircleCI Office Hours: End to End Test Automation That Doesn’t Suck
On January 25th, 2017 Heavybit member CircleCI held their monthly Office Hours meetup at our San Francisco Clubhouse. The evening's presenta...
My team works closely mostly with our newer investments, anywhere from bringing together founding teams, to hiring the very first engineer, to hiring the first recruiter. So we've hired hundreds of people over the last couple of years in consumer and enterprise companies and we also work with some later stage companies like Docker and Airbnb.
Before that I ran operations for Rypple which was bought by Salesforce and became the work.com platform. I spent five years at Mozilla. I joined when it was 20 people, ran recruiting and HR, and grew that to about 350. I've been recruiting for 16 years which is longer than I'd care to say, but I've seen a lot of fun things, and it's been a great ride.
I'm going to talk about debugging recruiting. When Dana approached me about the talk, it was really about companies that were beginning to scale and how to understand the problems that you're actually having from a recruiting process and how do you actually fix them?
Recruiting breaks down into three basic areas, the first being sourcing. How do you get people into process? That's either referrals, people you reach out to on your own, people that are applying to your website, or if you're working with agencies to help build a pipeline.
Second stage is evaluation. That's essentially how do you figure out if you actually want this person to work with you? That usually involves giving some sort of technical evaluation and understanding their work history and how it's relevant to what you're doing. Doing drinks or dinner or whatever it be to assess culture fit. Then doing some deep referencing to make sure they're the right person for your company.
The very last step is conversion, so essentially, being able to get someone to join your company.
I'm going to start with the very last piece because it's the most important. If you don't actually have a good conversion rate, the rest of the things upstream don't actually matter. So healthy conversion really is around 75-80% acceptance rates on offers that you're actually making.
If people aren't accepting the offer, it's a huge amount of wasted effort.
Now one of the biggest issues that you'll typically see around conversion is the person was actually not ready to receive an offer. I just got off the phone with someone today that accepted. About a week ago when we got on the phone, they had already made an offer. The person had not signed an NDA, did not know exactly what the company was doing, and did not know what they were going to be doing in the company. They weren't sure how to value the company because they didn't know enough about it. Those are all problems that are easy to solve.
It's a huge mistake. The reason it's a mistake is because people make an offer when they're ready to make an offer, when they decide that they like someone, instead of waiting for them to be ready to accept an offer.
A lot of that is being able to go into the pre-closing process, and making sure you're answering every single question that that candidate has before you actually extend the offer.
One of the other big issues that we run into is compensation. People just haven't done their due diligence about what to offer someone, or more importantly, how to talk about the value of what they're offering them.
Other issues in conversion is essentially organizational problems that show up in the interview process, or just a sloppy process in general and making sure that you're able to understand what those issues are as people move through your process.
So how do you actually go about fixing conversion? One of the things that we recommend, and I'll get into a case study in a second, is going through and talking to every person that says yes and no. The people that say yes are actually even more important than the people that say no, because something happened through the process that got them to really engage and be excited about the company. The more that you understand that, the more you're able to pull that forward into the process.
You're looking at all the different issues in what I call your "conversion funnel", which is all the steps and touch points in the process, understanding where the problem is, either at the beginning and talking to the right people, a gap in your evaluation, or something that comes towards the end at closing or compensation. Then, once you identify the gap, make sure you remove it.
I'll talk a little bit about Airbnb. About a year ago, their conversion rate was about 60%, or a little bit shy of 60% at the acceptance rate. So I spent about an hour talking to their head of engineering recruiting and he diagrammed the whole process and everything looked great. Every phone screen to on-site looked good, every on-site to offer, all of that was great. But the conversion was just terrible at 60%.
They went through and they interviewed every single person that accepted, and they found out two things. One, compensation was off, so they fixed compensation, and two, they weren't really selling the story very well.
So from the very first call to every piece in the process, they began selling the Airbnb story a little earlier and earlier, and then going through all the steps for pre-closing, of making sure that all of the questions were answered ahead of when the offer was actually delivered. Their acceptance rate went from 60% to 80% for the entire year.
So moving into the next statement in the evaluation process and so understanding if you have a leaky bucket.
Aside from having offers accepted or understanding everyone that makes an offer, who are the people that you would have liked to have hired but they actually never made it that far because they decided to opt out.
One of the key things to figuring out how to fix your evaluation problem is to actually diagram your process from end to end. If you don't have a process that's consistent then that's a problem in itself.
You should be able to understand all the different pieces in the funnel and knowing when people are falling out. They're also bringing and selling early into the process, and again, it's the whole pre-closing piece of not just evaluating in one way so that you can make a decision, but putting you in a position to actually get the person you want.
So I wanted to diagram a basic process for you guys. This is what I've been using. I used it at Mozilla. I've used it at a number of companies that I've worked with, but basically it's just a basic recruiter screen, going to a technical phone screen, and then the first on-site.
The first on-site, you go through the basic technical evaluation pieces, but you also leave time at the end to figure out what unaddressed questions the person has and you make sure that you answer questions for the individual.
I know it's very basic, but a lot of companies don't set aside time to answer, "What concerns or questions do you have?" or, "What is it like to work there?" and showing them what the company is really about.
After that first call, always do a check-in with either the recruiter or the hiring manager to get a sense of what they think after they've had a chance to really debrief and think about the process. What questions are outstanding and making sure you address those questions in the next round of interviews.
Then going into a good offer delivery. So again, that goes through being consistent about process and understanding where people are falling out. From an evaluation perspective, if you're able to map this, you'll both know if you're wasting time in any one of these steps and also know where you're losing people. Then you can figure out why you're losing them.
So the beginning part is around sourcing and talking a little about how you get people into the top of the funnel. That is, making sure that you, for the company that you are, are targeting the right people and you're telling them the right story.
By making sure that you're both getting to who you want, or having a sense of who it is that you really want, and then once you have them, to be able to tell them why it matters, that is the best way to actually improve your conversion.
So a little bit about the top of the funnel piece. Making sure that you're targeting the right people and knowing who your target candidates are. Not all companies do this particularly well. They might think that Google engineers are the perfect people for them and in a lot of cases that's usually not the right answer. So try to figure out, where do people that would want to work here come from, and make sure that we're talking to those kind of people.
The next piece on the top of the funnel is just being able to systematically drive your internal referrals. So if your employees are not referring people, you should really find out why. The expectations around it should be in the 30-60% of hires should be coming internally. If you're not getting that, then there's a problem. So either they're not comfortable referring their friends, or they're not out there proactively talking to people.
Hiring managers must source. Realistically, everyone must source. Everyone at the company should be involved in recruiting.
We see more and more of this recruiting being pushed to the edges. We run a community of designers and product managers. We run another one around big data and one around security. We were doing a dinner with a bunch of designers at well-known companies and three of the topics of the conversations were all around recruiting and hiring.
What it tells me is every hiring manager and every team member is being asked to be involved in the recruiting process. Make sure you're able to drive that into the organization.
Bring on a recruiter and work with agencies. I think you need to have the right DNA for your company to be able to recruit. There are a few companies that do that well. But I think having someone that's going to quarterback the whole process is key to making sure that no one falls through the cracks, that you're actually hitting all the steps in your process, and that you're getting good metrics on what things are working and what isn't working.
The last piece I think can be helpful. It's not particularly helpful for early-stage companies, so it takes a while to get to a scale where your blog posts actually matter. It took a while for Stripe to be able to get there. It took a while for Mozilla to be able to do that.
It's a good behavior to try and bring in as soon as you can. Really talking about the interesting things about your company and being able to have that shared publicly, that's then shareable because it allows you to have these scaled conversations. So instead of one-on-one telling people that you're great, you're able to at least show them and express things that are interesting. It allows you to have one-to-many conversations.
Storytelling is an incredible key. It's another thing that I've found that startups don't do particularly well.
A lot of people will do their investor pitch, and they'll do that really well, but you really have to apply the same amount of effort and energy into talking to candidates and teaching other people in the company to do that.
I really believe there are essentially three stories. So what is the market story? Why does the company exist? And why does it matter from a financial perspective?
The second is the technology story. So what is unique and interesting from a technical perspective of what it is that you're trying to do?
The third is about relevance. How do you make a tailored story to each person that you're talking to? One that allows you to have a directly relevant conversation with that individual so it doesn't look like they're just one of a million engineers that you'd be happy to take. There's something very particular about that person that makes you want to hire them.
It's figuring out what is unique about your organization and learning how to talk about it, then practicing that story over and over and over again.
I worked for a company called Guidewire Software, I don't know if people know that, but I worked with them pretty early. They went public a couple years ago. They make insurance software.
There's not very many things that are more boring than insurance software, but we came up with a really good narrative about why we were able to hire these great engineers. We really focused on the ability of transforming an industry and what that means. Don't lead with insurance.
You lead with, "How do you change or revolutionize a new industry?" So you need to figure out what that is for your company and how do you actually create a story that makes sense and that actually resonates. You know you're ultimately trying to get people to care. How do you get them to care about your particular problem?
The last thing is getting everyone involved in telling the story and spending time with anyone who's in the recruiting process to help them talk about the company and how to talk about the company, and how to have it done in an authentic way.
I think there's a little bit around training and having the CEO, or whoever it may be, talk about the company,helping people talk about why it's amazing to them, and doing that in a setting that allows people to share that both internally and express that externally during the interview process.
That's the basic talk that I have right here, and I'm happy to go into more detail on any of those things.
The question was, "How do you filter people in versus filtering people out?"
So the early days of Mozilla, we only knew how to hire people out of Open Source. They had hired poeple that had been part of the project or part of the community and people that they knew. So we spent some time thinking about what is it that we actually look for. We can't just cannibalize our community and hire everyone out of it, so we need to figure out what is it that makes someone great for Mozilla.
These are certain things that we broke down. One is understanding that a technology is just a tool. So we were looking for people who were pretty voracious about the languages they used. They weren't just myopic about working with one thing. They had to have this broad interest from a technical perspective, of what it took to solve the right problem. That was one of the things that we would look for.
The second piece is, they didn't have to necessarily contribute to Open Source, but they had to care about Open Source and what that meant.
The third was getting people that were very mission-oriented into the company.
Once we started to broaden out what someone would look like that was relative us, we actually started to hire at a much higher rate than we did before, and much more successfully.
The first question was about sourcing, "What tools do you use and how do you do that effectively?"
I'd say LinkedIn is still the best and most widely used tool, but I think the beginning part of sourcing is how you think about targeting. I just went through the exercise with an EIR that we had to identify founders for them, and so we went through asking, "What are the universe of people that we're actually interested in from the sector perspective and from the technology perspective?"
So we had a list of probably about 50-60 companies that would actually make sense, then we started going after that list. We used LinkedIn and other resources to be able to do that.
That was a much more heavily targeted exercise for identifying people ahead of starting to do sourcing, and understanding what the universe of people looks like. From a tools perspective, Connectifier is one that we use quite a bit. We started using that and we played around with most tools, like Entelo and TalentBin, they're somewhat useful.
But I think discovering people is not that hard. It's actually relatively easy to identify people. The hard part is actually getting them to talk to you. To figure out a way of getting them to engage with you in a conversation. Then once you engage in the conversation, getting them sold a little bit, and then getting them into process.
That first conversation is not an evaluation conversation, it's, "How do you get them to have one more meeting?"
I think the second piece is, "How do you not hire people that are too senior?" Most startups are looking for "player-coach." I've always been an individual contributor first and a manager at the same time, I think is incredibly important. It's essential for startups.
I think one is the targeting piece. Two is, I'm kind of biased against job postings. Perhaps you're going to get lucky, but I would tell you that at Mozilla, less than 3% of the people that we hired actually applied. The rest we all directly sourced.
To get the person that you want, the one that is going to be the most successful, you must figure out what that target looks like and then start going outbound. But it's time consuming and most people are trying to run a company or ship product. I wish I could tell you that there's an easy shortcut, unfortunately there isn't.
The question is actually, "How do you find a great internal recruiter?" I actually spend about 20% of my time talking to recruiters just because it's by far the single most leveraged hire that we could make. It's like wishing for more wishes, I stole that from someone but I don't remember who I stole that from. Most of the people that I have hired that have done well have spent time in an agency and part of that is you just have to, you have to hunt.
Base pay at agencies is really low, so they're all paid for success.You need to find the right person that is both a bit aggressive and has been successful at an agency, but gets tired of it. Most of the life expectancy for people at agencies is between two to three years just because the work gets monotonous.
The first piece is if someone's been at an agency at some point. The second is finding someone that tells a really good story, and getting them to. They don't necessarily have to tell your story as well as you do because it takes time to figure that out, but have them talk to you about companies that they've worked with in the past.
I would never hire a recruiter that doesn't source. A lot of Google recruiters don't source, they're mostly "managed process."
I think fundamentally that's one of the most important things that you need, someone that can go out there. So, where to find them? I would look at some of the top agencies. Start to build relationships with the account managers that you've had there.
Sumo Logic is one of our companies and I found their recruiter that way. There's quite a list of companies that have hired their recruiter that was once at the agency working closely with them, and a lot of that is building a relationship with them and answering their questions. If you find someone that you like, just start building the relationship with them.
How do you actually engage people that are not engageable at the moment?
On average, before someone joins one of our companies, we have known them for at least a year. The person that just joined to run recruiting for one of our companies, I've been working on since 2011. My longest hire is ten years, and a lot of that is how do you actually start to connect and build a relationship with someone.
The time horizons to actually hire people is getting longer and longer. There is a subset of people that are ready to go and so they put themselves on Hired or they go out to market. Conversion on those people is really low because they're out there to get as many offers as they can. So how do you get people that are not as active? It just takes a really long time.
It's not, "Hey here's why you should talk to us." It's "Here's why I want to talk to you".
If you have their GitHub and you can be very specific about your message to ask them a question that gets them to engage about themselves is actually the way to get them to talk. Then you don't talk about you until you've started to talk about them, because people love talking about themselves. They don't really want to hear about you as much.
So you must find a way of starting to build a relationship. Then seeding in the story about what you're working on and why it's interesting. Expect that it's just going to take a long time. So I write, I put a slide up there about process and all the steps in process, and that's for people that are active.
In a lot of cases we know who we want to hire well ahead of that. They circumvent that whole process because we've met with them ten times and we've already referenced them, and we've done our back channels. We know most of the things about them, and it's mostly us trying to show them why this particular company is where they should be. Then start to build those connections.
We spend a lot of time on it. We also do a lot of research. When we craft a message to the person that we're interested in, for example if you know you have 100 people and you know who that list is, if you can get your conversion up from 5% to 15-20%, that's a huge difference.
People get the, "Hey, let's talk. Let's have coffee." email 50 times a day. They don't get the "Hey, I saw this blog post that you did a while ago that talked about performance on X, Y and Z, and we've been having this performance issue. I would love to meet you and pick your brain about this interesting thing that you did because I think it was fascinating."
People have a hard time turning down personal emails that you clearly took the time to actually do. To do your homework and then personally write to them. It's easy to delete bulk emails.
Secondarily, just follow up. We ran this experiment on sending automated emails. The email response rate on the first email I think was 20%. Second was 15%, and then the third email was 35%. We did this over a week-long process. We live in this new world where people check email on their phone and they're busy. They see it and they were going to respond and just forgot.
You might find hitting them a couple times annoying, but it actually gets people to re-engage. After a couple emails, it makes it hard to ignore.
The question was about tools for managing process.
In the beginning, when we have a new, small company, we start with a Google Doc, just to track. But you quickly outgrow the Google Doc.We use a CRM because we have to. We've been around for three and a half years. We have 8,000 people in it and ATS no longer makes sense. Most people have been using, Greenhouse? Sorry, Greenhouse has been doing quite well.
How do I feel about coding tests? Coding tests are a filter. I don't believe they're neither good nor bad. They're just good or bad based upon how you choose to insert it into your process. If you have thousands of people applying every day to your website, which is not a problem very many companies have, having a filter at the front of that to allow certain people that are either serious or that do well on the test to go through, I think that works really well.
If you have a test in your process that's towards the end, which is like a badge of honor that everyone has to do, and you explain it as that and don't make it too arduous, it's not like three days of work to do and you explain why it's a piece of your company, then I think that that works well. If you make every single person take a test before you come in to interview, that's usually a bad idea. It's not really solving the purpose that you want.
Most people don't have the problem of having too many people apply. I think it could be useful. You just have to be thoughtful about how you implement it.
That is a good question. How do you hire when you don't know engineering people and when engineers don't know that person firsthand? Try going to second-degree connections. I know that that's not always easy for engineers to do, but try to be able to find ways of extending the network out of the people that you know immediately.
So you might have to go one degree away. An engineer might have a girlfriend, boyfriend, husband, wife, that is in an adjacent industry. Getting over staring at your shoes and end up geting the introduction in that particular way.
The question was about funnel optimization. For each hire, how many people should you be doing outreach to? It depends on how well you do your targeting. If you're doing just pure outbound, blasting email, generally your response rate's gonna be anywhere from 10-20% and that depends on the background of the person doing the outreach.
Joe Recruiter response rate would be much lower. The CEO or an engineer will tend to be a little bit higher. So if you get 20% response rate targeted, maybe half of those people actually want to interview. If you can get an offer to one to two of them, you can hopefully get one. Roughly it's about a hundred people for every offer.
Those are my numbers, and I've been around recruiting for a long time. I can get people to respond to my emails pretty well. The less of a profile that you have, or the less of a network you have, your response rates actually go way down. The amount of outreach actually goes up with that.
Again, like the outreach, the person doing the outreach really matters. You might have to have your VC start sending emails for you, or letting you, giving you the credentials for their gmail account.
So how much can you lean on your VC for recruiting? I would not base the future of the company on whether or not your VC can do your recruting for you. My goal has always been to help our companies be self-sufficient. A lot of that is teaching them DNA for actually being effective at recrutiing. Then hopefully connecting them with a great recruiter.
When there is a specialized candidate that you really want, then you can pull them in. But depending on who the VC is, the response rate might not be much better.
I think that if you can get people excited about your team, and you can get them excited about what you're working on, that's actually much better.
In a lot of cases you may not even have to go into the full interview process because you've learned about them, maybe you've gotten a chance to work on some things together, or you picked their brains about problems that you're working on. While you're building this relationship you've been referencing them at the same time.
You know that they're good, so maybe you have to do one or two generic interviews to get it through, but the due diligence is not going to be nearly as big if you know the person's walking off the street, versus you've spent six to eight months actually talking to them.
Generally it's all a timing thing, so getting someone at the right point and catching them on the right day. At my previous company, I won't mention which previous company, but I was meeting with my old boss andwe were sitting down for lunch, and he caught me on the right day when I was pissed at my previous boss, and so he says, "Come in and talk to people".
A week later I had accepted a job and gave notice.
The question was spear fishing versus casting nets. I'll say it a little differently. It's more of building the relationship. Part of that is I have a list of people that I know or have heard to be amazing. So I spend the time finding out what that fish likes to eat, and where that fish likes to swim, and you know what that fish dreams of when they're not at work.
I put myself in a position to actually deliver that. Part of it is really getting to know people for an extended period of time. I think that's part of why I've been in recruiting for 16 years and why I have these relationships with people from companies that I've worked with, or people that I interviewed that we didn't even hire that still come back because you were kind to them.
You spent time with them. You answered their questions and you were helpful to them in those pieces. The more you're able to do that, the more your network actually extends and the more effective you'll be over time.
So the question is, "How do you figure out if they are or are not interested?"
So at the beginning part of the process, it is my job to get them interested. If I can't tell a story that gets that person interested or engaged in what I'm doing, there's something wrong in the way that I'm telling the story. That goes back to fixing those pieces.
So the second piece of, "How do you figure out if someone's actually interested in working with you." There's a list of questions and a book called Getting to Yes which is a famous sales book. But essentially how do you actually relieve and remove all the barriers from getting someone to to say yes, or to accept?
Understanding what are all their decision criteria. What is it that they really want in their next job? Really understanding those things and making it obvious that those things are available at your particular company.
So it's on you to sell them. Then it's on you to understand what makes that person tick. Being able to then express those things in the process, that gets them excited about what you're doing. And if people aren't excited, they'll just tell you, "I don't care about robots" or "I don't care about working on drones" or whatever it may be. It's just a question of getting people excited about the right thing.
I do see people that, in some cases, don't care that much about the thing that they work on, but they care a lot about the people that they work with. So if you can expose that to people, and if they get excited about who you are, how you do things, and hopefully the problem, you can be pretty successful.
I wish that I could tell you that there's a magical shortcut. There's good people in a lot of different places.There's more concentrations of them depending on where you are. We did a lot of work at Waterloo. You're going to have to find one person that you think is good, and then you start to expand from there.
There's no magic honey pot of recruiting, or at least if there is I haven't run into it.
A lot of it is figuring out what does the universe look like of people that I want? How do I go find a universe that isn't as full with a bunch of people messing with it so I can start to build a relationship. Every school, even the ones that you've never heard of, probably have one to five people that are amazing. They're there because their grandma's sick, or they don't want to move away from home.
How do you find the one TA for the upper division computer science class at this one particular school? That's likely the one that you want. One shortcut that I found specifically in university is to look for the TA for "Operating Systems and Compilers" because that's usually the hardest class at the school. So if you can find the person who's the TA, they're probably someone that you're going to want. They'll probably pass your engineering test.
How do you not lose to Google? The conversation can't be about money, because you're going to lose that every single time. It has to be about impact.
How do you express what it's going to be like to work with you? I think everyone knows what they're going to get when they go to Google. They're going to get put into this pool and they may or may not get something interesting. But they'll get frozen yogurt and salad, or backrubs, whatever it may be today at Google.
If you're really attempting to sell the impact of what you're trying to do, that's the key to trying to beat Google. You may not be successful. That goes back to the targeting piece. If they're considering you, Google, Uber, or they're talking to the companies that are going to pay as much as possible, then they're not the right people.
Mozilla was a non-profit. We would go up against Facebook and Google pretty consistently, and we would win. I think there were maybe five people that we didn't get in that time. Because we sold mission. I think that was a hard thing for them to kind of counteract. I hope we did the right thing for them because that was in the pretty early Facebook days, so I think I probably prevented some people from becoming millionaires. But I think they really loved their job. I think they loved Mozilla.
The question is, "How do I actually find information about closing candidates, and the specific questions that I ask."
Number one is figuring out how they make the decision. That is, "Do they decide by themselves? Do they talk to their friends? Do they talk to their parent? Do they talk to their wife or their girlfriend? Who's involved with that decision?" If someone has a small child, it's unlikely that they're going to commute from San Jose to San Francisco every day, because it's just not going to happen. It's figuring out who's in the decision-making process.
How long is this process going to take? Getting a sense of what does the timing look like. Either you have to speed it up or slow it down depending on when that person's timeline is.
Then the last piece is trying to figure out what it is they value. It ranges. It could be they value being home at a certain time so they can put their kid to bed. Or they value a particular technology or a particular language or they value getting to go speak at conferences. Figure out what it is that they actually care about and how they make that decision.
How does someone decide on what they're going to do?
So if you have five offers in front of you, they're all valued the same, how do you actually decide where you want to work? It's not that different from person to person. They want to know what they're working on. They want to know how that is going to affect their career, and what the path is for them.
Startups are not particularly good at this. The more you can understand what a candidate values and then echo that back during the process, the better. Most importantly, deliver on it.
When I moved from recruiting to HR, I felt recruiting is this promise that you make and then the HR piece is making that promise true. If you get to know people and get to know what they want, then deliver on that promise, and you do that for all of your employees, it's going to reinforce and get you more people to come in. That's what gets people to love their job.
But back to your initial question, you have to figure out how they decide. Then make sure that you address those points in the process. Be honest that you can actually deliver them.