May 2, 2019
Ep. #29, The State of Open Source & Docker Security
In episode 29 of The Secure Developer, Guy sits down with Liran Tal, Developer Advocate at Snyk, to discuss the state of open source, Docker...
Hi. So, I'm Ali. As I said, I work on the Internet. I've been working on the Internet for a while. This is apparently my seventh start-up. The first one did super-awesome. We sold and got $147 million in cash, so that's really exciting. The rest of them, not so much. I'll get to that in a minute and now I'm at my seventh one, it's pretty awesome.
I'm sure that you guys have a similar set of stories, we've all been kind of doing this for a while in various spaces. What I'm going to talk about tonight is something that keeps coming up as a recurring theme in all of the companies that have been out today, which is supporting our customers, and how we do it, how we turn it in to something that helps to sustain our business, and how it feeds back in to our product.
The first thing that I'd like to point out is that support is probably not your end goal.
If you guys are sitting here, my understanding of this community is that you are probably developers. Perhaps product managers, maybe sales, but most of us don't go to college going, "I'm going to learn how to support a product. I'm going to make a product just so that I can have people who I can support with it. I just want to answer emails all day. It's going to be so amazing."
While we are making things because we want people to use them, we want to solve their problems, the fact that we need to support people as we use those things generally isn't the thing that we have in mind when we start a new project.
I'm sure that none of you left your confusing elects or your weird features, your gross bugs in the product just so that people would have to talk to you. But it's there, and we all have them, and people need to talk to us.
While it's not part of the business that you want to deal with necessarily, it's a necessity of your business. It's like hiring your accountant. It's like hiring your cleaning service so that you don't come in to your own overflowing trashcans every day. It's just part of getting work done.
It's making sure that your customers get the support that they need to use your product successfully.
My first job dealing with support was my first job out of college. I graduated after studying humanities and computer science, and went into a job as a Java developer at a company that was basically doing monitoring as a service before Nagios existed.
I was googling last night and I ran across this picture and I had to bring it up. This is our office, and we had zigzag fish tanks all through the office as our cubical separators because we were Freshwater software and it was super-cool. This was about the year 2000. Nagios started in '99, we had launched this product in '98, no '97.
This was concurrent with the time that people were starting to figure out monitoring for their services as a service. Whenever we hired someone for this team, they all had to do support, so every single developer on our team did support. We all were responsible for all the email that came in and we all had a phone on our desk.
We provided phone support from nine A.M. to six P.M., Monday through Friday. and it was Mountain Time so no one knew what that was, and they would call during off-hours and we're like, "Oh, we're on this special time zone. You don't know what it is? You missed us by an hour."
But we all had phones on our desks and we all answered it, and we all did support. So this is how I got my introduction to support, I was fresh out of college, writing code. So there's me. I was there. Those are some dudes on the phone. I was looking at that and, like, "Whoa, we had a lot of people needing support that day, that totally sucks."
But the real reason I wanted to show this, is that laptop. That thing is fantastic. We used to do work on that. That's amazing. So, aside from that laptop, which I'm sure was pretty awesome, the tool set that we used for starting this up, we had a homegrown Filemaker Pro database. And, like "ew," but this thing was awesome.
Not only did it serve as our support tool, but it also served as our CMS. So, every customer we had linked to the company or everyone who contacted us, we had linked to the parent company that they were part of. We could see every support request that they had made. We could see all of the billing activity around the customer itself.
Everything was there for all of us. It was really useful for us, as developers to look at a customer, you know, somebody calls and you're like, "Okay, well this person is about to close a 70 thousand dollar deal, so let's be very circumspect about this," because we were still pretty small at the time.
Or, you know, this is somebody who has called us ten times in the last week with the same problem and doesn't like the answer, and the fact that we had this database that we could all use, that we all contributed to, and that we could all pull from at a moments notice, actually made our support job a lot easier.
The next tool that we had, and this is actually really smart, we used name-support email addresses, so I was Aliemail@example.com. And on the edge, our mailserver would intercept an ali-support or pete-support, and shove it in the database and send a copy to our personal inboxes. So we could still manage all of our correspondence as normal email, but we gave these email addresses out to customers, and then we'd know that without ever having to think about it, that all of that correspondence would also be captured for the rest of the company to use.
And like I mentioned, we were all responsible for support. One thing that was fantastic about everybody on the development team being responsible for phone support is that everybody got really, really careful. It was like, "Okay, if we screw this up, the phone is going to ring." And I think that it's even worse now than it was back then, but it's like, oh my god, that phone's going to ring, this is the worst thing, I do not want to talk on the phone, that was pretty effective in a few ways.
So, start-up number one, that was awesome. We did a pretty good job of supporting customers. We had people who loved us. The product still exists despite being sold in 2001 and subsumed into a toolset that was subsumed into another toolset, that now belongs to HP and we just discovered the other day it's still there. We're like, "Aw, that's so cute," and people are still using it.
Let's talk about two through six a little bit. Really quick, because it's boring. Commercial real estate transactions in the cloud before people did anything in the cloud, so this is not the kind of customer that's like, I'm going to take a risk and put all of my data on someone else's computer. That didn't work out. We had a desktop platform to automate testing of Peoplesoft installations.
An IP-based payments platform. It's actually the backbone of what Square managed to build itself off of at the very beginning, and then they were like, "Hey, we don't need to use you as a middleman.We're out." So, that was number four.
Number five, an open source desktop music player. It's difficult to make money off of music and open source software, so combine those and that's bad.
And the last one was really smart ideas about browser extensions that never really even made it to market. So, you know, this stuff happens. We all have our successes, we have our failures, so that's mine.
Start-up number seven, which is where I am now, is Glitch. Glitch was a web-based MMO, that took place in a Flash client. It was basically a very complicated website with the really, really, really complicated Flash widget in it.
Our engineering team is in San Francisco, we had a co-founder in New York City, our art team and our world-building team were in Vancouver, we had customer support scattered all over the US and Europe, we had, let's see, music in Toronto.
When you have a game like this, and you're sharing media assets and code and you have people you need to talk to, you know, communication and collaboration become key.
And this was all we communicated through IRC. We had our own IRC server. On your first day you got the password, and you were expected to be on IRC. But the sucky thing about IRC is that if you are offline, you don't know what you missed.
So we built a log bot, and the log bot shoved everything into a database. And then we built a webpage to see what was on the log bot, so you could be like, "Okay, well every morning I'm going to go in and see what happened over night."
But sometimes you're on your phone and you're out and may need to see what's going on and so we made it responsive, and you could view it on your phone. And then we were like, "Ah, but sometimes I need to reply," so we added a little input box. And you could be like, "This is Ali and I have something to say," and then it would show up as log bot talking, so you had to say who you were.
We needed a way to share all of our files- all of the art and the music and the design, there's just so much stuff that goes in to building a game assetwise. So we took our CD and we bolted this file upload functionality onto the side of it and you could upload a file and we would put the URL in IRC and everyone could click on it, and boom they could see you file.
We needed a way to share a code in our log files. Glitch did crazy things like if you squeeze the chicken 13 times it would run out of grain and people would be like, "This chicken's not giving me grain any more." So we'd have to go to the log files and we were like, "How many times did this person squeeze this chicken today? You squeezed it 13 times." But these log files were just huge, so we put Pastebin in and then we crammed all these logs in to Pastebin, then we could share those the same way we did files.
So this thing, IRC plus all this stuff, it's a total Frankentool. We just grew it out of bits and parts and mashed it all together but it worked great. This was the foundation of how we got work done. This was the thing that enabled us to get as far along with the game as we got.
But Glitch was not economically viable, so we shut it down. We announced in November 2012 that we were going to shut down the game and we took it down in December 2012. So what we did, we took our remaining money and we decided to write the product version of the tool that we developed for building Glitch and that is now Slack.
You guys may have seen this slide, we did feed Mark Andreesson this graph, he doesn't have access to all of our data like this, but he is one of our investors, so he gets this data. We started inviting people into Slack, like very close friends and family who we could rely on to give us good feedback in late April of 2013. We had been using it ourselves for about two months at this point.
So we had a really, really, really small user base up until we went to this thing that we called "Preview Release" and "Preview Release" happened on August 14, 2013. And this is nice, people came in and we got lots of users and it was fantastic.
I'm going to show you what happened with support. This is a graph of our support load over the same period of time and here are some numbers. Whoops. Yep, I can do this, I'm good. Yeah!
In the month of July we had, I think, 30 support requests to deal with. We launched in mid-August and by the end of August we had 300 support requests during the month of August. And then it went up to 800 during the month of September, 880. So this number just kept going up and we did not know at the time what to expect.
We were like, we're going to do some press, and we'll tell people that we have this thing and it's nice and they can use it if they want, and it's free and we're still figuring it out. So we put that out there and were like, well maybe 1000 people sign up, maybe two.
And 8000 people signed up in the first 24 hours. And by the end of the week 14,000 people signed up. We had no idea what to expect and we did not expect this. This was way, way, way beyond our most optimistic expectations. This was something that we could not have planned for.
The things that happened from a support perspective is basically a 20x increase in ticket creation over night that sustained itself for three solid months. And when this happened, we realized that our existing support tool was completely inefficient for dealing with this, we had gone with something that was nice and friendly, and kind of not too expensive, not too cheap.
We couldn't understand our queue volume. We couldn't bring in other people to help. We couldn't sort it by what we had dealt with already. It was terrible. We were like, this tool is not going to help us manage our growth.
One thing that you see here, ooh, I have a laser pointer, yeah, right here, this dip, so we had 14,000 people we needed to invite into Slack. We trickled out the invites very steadily all the way through mid-November and when everything starts to level off here, that's when we finally got through our invite backlog. And people who were coming in were finding us organically, but there was no longer a pent up demand for people to get on the platform.
So one thing that I did, when we hit this point, and we're like, "Oh my god, our tool is not going to support us and our growth is crazy and we're going to have to do things in the future that cause a lot more people to come in," is, I was like, "Okay, we're switching support tools."
So, I spent October evaluating support tools. I spent November writing scripts to migrate all of our existing data and practicing the migrations. And December 25, woke up, pressed a button on my script, went to Christmas dinner, got really drunk, came home, sobered up, and was like, "Yes, it's done." And that was how we moved to a new support tool.
We added some staff to help us with support in August and November, and January, and that was a relief. Every time we hire someone that's a relief.
And one thing that isn't reflected on this graph is that Twitter is also a part of our support universe. People can contact us through email, or through the product, which is the same thing, that all goes in to ZenDesk. We also have a lot of people talking to us on Twitter. Right now, we're dealing with at least 150 tweets per day. And at this point, Stuart and I are the only ones dealing with Twitter, and Stuart's our CEO. We had other grown-up things to do.
So, things that we learned in this first growth phase. I'm going to go through these in a minute.
We learned about triaging our support queue. It helped us feel good about knowing the amount of work we had to do and also helped us make sure we were prioritizing what needed to be done first, very quickly.
I talked about this, choosing a tool that we could grow in to as our customer base grows. I desperately wish we had done that. And giving ourselves the abilities to throttle growth if it had the potential to overwhelm us.
So, triage. What we do, and this works for us and it may not work for you, is every time something comes in we categorize it as one of these things. It's either a problem, it's a question, it's feedback, or it's a feature request.
The thing here is that when you offer support, you are providing yourself as an empathetic ear for another human being. This isn't just about, you know, "Oh, I am having problems with your product," and you're not the person being like, "Let me help you."
Somebody needs your help. You are helping them in some way. And you are having a very fundamental human interaction at that point. So when we triage, we take a step back, and it's not "How do I feel about this? Do I know that this is a problem? Do I think this is a future request?" but, "How does the person who emailed us feel about it? What are they thinking? What's the answer that they need from us?"
It's like, why did they take time out of their actual lives, as an actual human, who has actual things to do, to write this to us? Because everybody has a reason. And the fact that people take time out of their actual real lives to do this matters to us. It matters a ton.
So we want to treat everything that we get, we want to reflect back the feeling with which it was sent to us. And that kind of sounds hippie and weird, and I'm not a hippie, weird person, but it matters. We are all interacting here as humans.
So a quick example of that one, we had a team that migrated to Slack from IRC based on their CEO wanting to migrate, and we immediately got several support requests, "Oh my god, Nick, it's broken. You have to fix this. We have to use "/nick" to change it to "john" and "johnafk" and "john on the phone."
They were used to using their nicknames to communicate to one other what they were doing and they felt that it was a problem so we dealt with these as problems. To us, it's a "future request." It's something that we explicitly don't support for a lot of reasons. Names have a lot more weight in Slack than they do in IRC.
But we still treated those as problems because the people on the other end were having genuine problems. And that was a tough one to deal with because this is a different platform and you hadn't necessarily bought in to this, someone else bought into this for you. So let us explain to you why it is the way it is and let's help you find ways to address this need otherwise.
Some of these can be difficult, but by triaging them, we make sure that we all collectively have an understanding of how people feel when they write us; what kind of answer they should expect to get, how quickly they will expect to get an answer from us, and I'll show this in a minute, we have some data too.
Then this next little deeder down here, the "about field," this is a dropdown in Zendesk and it has really, really top level stuff for Slack. Like, "notifications" is a category, and "email" is a category, and "messages" is a category and it's designed to be extremely short and easy to figure out, like, what is this ticket about? And put it in there.
We haven't been doing this for long, maybe a month and I have made plans for crunching data on it, but right now it's just kind of another field that we're using. I can come back in six months and say whether it was useful or not.
I believe, after this experience, that this is one of the ways in which you can do yourself the biggest favor. The good support tools are not that much more expensive in absolute terms from the cheaper ones.
If you choose a tool that you can grow in to, you don't hit a barrier at the very moment that you need your support tool to shine.
And that was really, really hard for us. We hit our twentieth growth moment, and I had no way to manage it. It was hacks like, well if we use these tags and use this automation, and then we share this list, then we can handle these things this way, but it sucked.
This is not how we wanted to manage support. We were dropping things. We didn't know how long it was taking to reply to people. We were missing replies when they came in. It was just terrible and had we just spent money on the more expensive thing at the outset, it would've saved us a ton of time.
So the next one. This one was super, super useful to us. Give yourself the ability to throttle during your growth periods.
And what we realized early on from that crazy chart is that people had their greatest support need in their first 30 days of using Slack, and that is something that has sustained itself. I just recently ran some numbers and 60 percent of our tickets come from people that are within their first 30 days of using Slack. 35 percent are within their first seven days.
So, when we know that we are on-boarding people rapidly, we know that our support queue is going to explode. So we give ourselves some tools to throttle how quickly we on-boarded new users.
Everybody who signed up to use Slack actually went in to an invite queue, and we're like, "Hey, we'll send you an email super soon and we're sorry. And here's some credit by the way." Like, "We're super sorry we can't have you right now, but here's some money you can use once we do have you." People are generally pretty cool about that.
We also provided a line-skipping mechanism for people who didn't want to wait. As long as they could get the referral code from someone else who was already in, we'd bump them to the front of the line, they could get in immediately.
But this invite queue is crucial, because what it meant was that we could go, "Okay, today support is definitely very behind, we have one person who's sick perhaps, let's just not send out invites today. Let's give ourselves a chance to catch up. Let's give ourselves a chance to get this under control, and then we'll send out more invites."
It's kind of scary when you're trying to start up a product and get market fit, and to be like, you know, we're just going to let these interested people sit there for a minute. But we would rather let the interested people have a good experience when they come on-board, rather than rushing to get them on-board, and giving them a bad experience.
So, let's go back to this chart. This is the second version of the chart, and on this little blow out thing here, you see the original chart from the last slide. And this is our active user growth chart for the first year of using Slack. So this is to August 13th, I guess.
And just, if you're curious, we define an active user as someone who used the product today and also used it last week. And using the product is launched it, sent, read messages.
So, this was really hard.We did our offical launch on February 12th. This is when we got out of preview phase that we were in previously, and we were like, everybody's welcome. Come on in. This is a real thing that we're doing for real money.
So, let's look at our chart again. This really sucked. This was really, really hard. Basically everything doubled. And it did not let down for a long time. So, the enormous spike was once again due to on-boarding a bunch of users.
And even though we did our official launch, and we're big kids, we once again employed our throttling mechanism to make sure that we weren't on-boarding what would've been 13,000 people in one day. We scattered those out throughout, I think, five days. But it still just kept coming and coming.
So the things that helped us here. The things that we learned from the last time, were, use your throttles, you know this is going to suck because you're on-boarding a bunch of people and you're going to need a lot of support.
This is our first real test of our new tooling and our triage process, and it worked great. One thing about having a triage process is that when you're dealing with a spike like this, it feels like you're never going to get through it. You just look at your support queue, and it's this undifferentiated mass of stuff. It feels heavy, and it's like, I don't know how, as a human, I can get through all of this stuff.
And once we started triaging, it started feeling a lot more manageable, because we were like, "Oh, well really there are only these 10 people who are having problems and everyone else just has a question because we did a crap job of documenting this. We can handle this."
So the fact that we had, at this point, a triage process to come in and be like, "Okay we know how bad the situation is, and it's not really that bad," gave us a lot of power and a lot of comfort as we moved through this to handle that spike which, I'll say it again, it really sucked. That was really, really hard.
We added more staff in March and May and August. We also have two people who help us do Twitter now and at this point, Twitter is easily 200-300 tweets a day.
Looking back, it's stupid to say this, but it took me two PR cycles to correlate the fact that "Oh, PR cycles mean a lot of users, maybe we should prepare for that." Now I'm very well versed in what articles are going out, when, and when I should expect embargoes to lift and how all that works.
Unless you have some reason to believe that it will be different this time, the thing that happened last time is most likely what's going to happen again. And I really wanted that to not be the case, because I really wanted it to be easier, but it was basically the same.
It was like, "We're going to do this press and maybe this many people will sign up." And it was this many people and, you know, I've learned twice now. I know for the next time. Let's see here. This isn't going to get easier.
Every time we do a PR cycle, I am now prepared for the fact that it's going to be hard. Our team is prepared for the fact that it's going to be hard. And we're kind of ready to pull on our big boy pants right now and dig in and get it done.
I was at XRXL last week, and someone said, "The only way out is through." And when you're sort of at that, if you have the privilege of having explosive growth, which you have a lot of problems, and you can't complain about them because then you sound like an asshole, but you still have problems and the only way out of those problems was to go through them.
Like, we couldn't just not have the growth. We had the growth, the growth was there, the growth was with us. It was like, okay, we've got to deal with this shit. So, we dealt with it. And it can be more manageable. So data, I here you guys like data, so I put some in here.
So to shift topics a little bit, knowing our patterns is one of the things that helped us manage the onslaught better. Knowing when tickets are created is hugely useful to us, because we have people using Slack all over the world, and it's like, well, if we have tickets created at 6AM, and everyone in San Francisco gets in at 10:30 because this is San Francisco and we don't get in to work early, what does our support queue look like?
So we started, this is Zendesk good data integration and this is one of their default reports. And I can just go in and be like, if you can't read that, the blue is opened between six AM and noon, the green is opened between noon and six PM, and the yellow is overnight, so six PM to six AM.
And it's fantastically split in thirds. This is our data from August. So now we have two people in Europe who handle most of the overnight stuff. This is San Francisco, so the six PM end time doesn't really work, we're usually working until nine or ten. And then we have some people on the East Coast, and they start taking over that morning shift.
And now we basically have 24 hour coverage and as long as we make sure that people are around during different hours of the day, just based on our natural sleep cycles, we take care of it. This is the same data but split up by hour.
So, once I started taking a step back and looking at this, it was like, "Okay, we're getting completely pummeled between eight AM and noon," and this has been an extremely useful time to realize that Europe is about done with their work day, and the East Coast is right in the thick of the beginning of their work day.
So now that we have four people attacking this every day at the exact right time we don't really have problems anymore. It doesn't get out of control, it's there, it's manageable, everybody is getting really prompt replies, everything feels really good, nobody feels overwhelmed, because everything is manageable. And this is another good data chart that just comes default, which is awesome.
So, one thing about our product is that we're doing something that people use to communicate with their co-workers. If it goes down, we can actually disrupt an entire company's workday, and that's incredibly not cool. That is something that I take with the utmost seriousness.
For us, if somebody's having a problem, it is extremely important that we answer them promptly. We have to do this if they're having a problem. So having this sort of 24 hour coverage is very important to us, and also back to expectations and, you know, did someone have a problem or a question, like, "What are they expecting out of you?" And as long as you understand the pattern not only of when things come in, but what sort of things you're getting, you can handle this load a lot better.
And that's what this slide is. It's a little bit hard to see here. I apologize. The green wiggle is the number of, oh hey, I have this really close, right here, the green wiggle is the number of problem tickets created. These are genuine bugs or people having issues with the software, that's the lowest wiggle.
The yellow wiggle is the questions. So people basically just going, "Can I?" or "Does it?" or "How do you?" And recently that's the top wiggle. And then that darker purple wiggle is that feedback and features queue.
So, when you look at this it's like, "Oh, you know, a third of our tickets are problems, and everything else we can deal with without involving development."
So, moving on to that, oh wait, one second, because I hear you guys like tools. So our toolset right now is really simple. It's Zendesk, and we offer email support, and built in product support. If you guys want a live demo of that, it's something we built for ourselves, I can attempt to give you one, and it may fail, because, you know, live demos.
We use a tool called Respondly for Twitter support. And the thing I like about Respondly is that it gives us the tools to treat Twitter like it's a very lightweight support queuel. It's like Zendesk. Here is a quick screenshot of what we see in Respondly when someone sends us a tweet.
And this is hugely important. So this goes back to very first job and understanding who we're talking to. Every tweet we get the name, we get the at name, we get their bio, we see how many followers they have, and this is amazing, because we know exactly who we're talking to and most people in this industry put their company in their bio.
So I can immediately go, "Oh, I know what team this person is on and they're having problems because some other dude disabled the Slackbar Response yesterday." You can start tying together all the stuff that's happening just based on knowing who they are and where they work.
Another thing that we really like about this tool is that it makes it really easy to follow people, which is this dude up here, and to fave tweets. We fave a lot of tweets because we think everybody is pretty awesome. It makes it really easy for us to reply to them in archive, and this prevents us from overlapping in terms of somebody has replied to this person and we accidentally sent you a duplicate reply. Doesn't happen anymore.
And finally, there's basic collision detection. So I can see if Stuart's viewing a tweet and be like, "He's got it, I'm not going to answer it. Moving on."
If you do have a Twitter support presence, Respondly is great for us, I would recommend you look in to something like this where you can understand not just who you're talking to but also get the context around it. And my screenshot doesn't show this, but in this undifferentiated white mass of their previous conversations with that user you'll see them too.
We use Zendesk for everything else, and I'm not going to show you a screenshot, because I took one and started blurring things out that you shouldn't see and I'm like, screw it, you can look at it on the Internet.
I'm done with that stuff, and I'm just going to talk generic advice and smart things that I've learned on the Internet, working on the Internet.
The first thing is that not every support request is a request for product support.
You will get people who have a lot of unrelated things they want to know about, PR, or marketing, or business development, or some sort of legal thing, or, I don't know, like physical space operations, resumes, pitches, people want to sell you their products through your support queue, which, no, don't do that, and if you're fortunate enough to have a support team, this is not the thing that they need to handle.
Much like you don't want everybody on your team distracted by other things that are not their job, you don't need your support team distracted by your business decisions, your product decisions, figuring out if you need to get a different trash service because of this thing in your support queue.
So the thing is if that stuff comes in to your support queue it's good, because it means that you're easy to find, and if your customers need to reach you they can. Product decisions, they don't belong in your support queue. Your support team can not make product decisions on behalf of your product.
And one thing about Slack is that we make opinionated software and making opinionated software means that we can say, "Here's our opinion why this feature works this way," and everybody automatically has an answer for a customer question related to that without having to be like, "Why did we do this?" The software has the opinions built in.
You should use your support team as a shield. Your ultimate goal here is not to have customers never contact you at all costs.
Your goal is to make sure that there is a way for that information to feed back in to your organization. That your organization makes your customers feel cared for. And that you can manage customer expectations and defer them if necessary.
And the final thing, most of us here are engineers and we're problem solvers and we got into this because we like to solve problems. We like to make things better with computers. And you're going to want to solve all of the problems that you hear about. But you cannot solve every problem.
You are going to need to say, "Is this a problem? "Is this customer's issue, is this a problem I want?" "Is this a problem that I want to take on for my business, with my money, and my time?" And it's fine to say no.
You are building a business with a set of problems in mind. It is very easy when you start dealing with customer stuff, to be like, "Oh, but this problem is really interesting, because it's way more interesting than dealing with the billing code that I'm writing right now." But you've got to deal with your billing code.
So, final thing is just to stay focused on what you're building. Your support queue has a ton of valuable information from all of the people who are using your product. It's also incredibly distracting.
Choose your problems. Solve the ones you want to solve. Form and communicate opinions throughout your company on what you're solving, what you're not solving, and why. And then everybody can move forward quickly and easily and together. And that's all, thank you.
The question was, "If a whole bunch of stuff that doesn't support is going into your support queue, how do you determine what's garbage, what shouldn't be in there, and where it should go instead. Is that basically it?"
So, we don't have a perfect nice nugget of an answer for you there. It's been a lot of learning and discussion amongst us at Slack. Our support person will be like, "I don't feel comfortable answering that," and someone will be like, "Oh, that's not for you. We understand that this is a product decision." And then, as we've started all pattern-matching against those. Like, "Oh these are product decisions." Then we can be like, "Let's figure out how to organize this chunk of stuff and take it over to the people to make the decision.
And the way that support for us becomes a shield for that is that we make sure that the customer knows that they were heard, that we are considering what they asked us to do, or the question that they had. And then we can still feed that back into the product organization which can act on that data in their own time.
So, it's not like, "Hey, everybody drop everything. We have to discuss about a custom emoji issue right now." Which happens a lot. There are a lot of custom emoji issues. And so by using support as a shield to recognize a product question, like something that they should not make a decision about, and then tell the customer, "We've got you, everything is cool," and then let the organization do what they will. We actually manage to move a lot faster and get a lot more customer stuff out that way.
For future requests and feedback, we do, so let's take integrations as an example, we have, I don't know, like 80 integrations at this point, and they are all different and they are all unique flowers that require a lot of hand-holding and support.
And we look at one and if the support team doesn't know what to do with an integration, we're like, "Here it goes into the integrations bucket" and that's how we're starting to segment that stuff out.
We also have separate queues for billing and accounts, so when people are like, "I need to process a P.O.," we're like, "Oops, that goes over to the billing people." So we are actually using groups inside of Zendesk to split out things that belong to others.
The question was about fixing and closing bugs and whether customers participate in fixing and closing the bug. And it depends. It depends on severity. So if someone says, "Hey, I noticed that your Sprite is shifted two pixels to the right." Then we'll be like, "Oh, thank you. We'll fix it." You know, close it out.
If it's something significant, and by significant, I mean impacts utility of the product, impacts experience with the product, is a severely diminished experience, then we absolutely go back to the customer and we're like, "We made this change. How do you feel about it? Is this what you want? Is everything fixed for you?"
So for actual problems, we go back to the customer and we close the loop with them. We have our own internal bug tracking system. So whenever we log a bug internally on behalf of a customer, we paste in the Zendesk link and it immediately gets flagged in our system as a customer facing bug.
So I can easily go through and be like, here are all of our customer facing bugs. We just closed on this customer facing, let's follow up on all of these tickets and go through that way. So it really depends on the severity, but if it's anything that we think someone might want to know about then we'll follow up with them.
So the question is, how big is the team and when did we decide to bring in more people?
Right now, the support team is at seven people, plus me, I do a fair amount of support work when I can. We have two more people coming on board very soon. It has been a a case of, putting out feelers, and hiring when we feel like the right person has come along, quite honestly.
I'm now doing a better job of getting into a groove of hiring people regularly as our growth continues very steadily, we obviously need to hire people more regularly. We have a better recruitment/management pipeline now.
When you grow like we did, there is just some stuff that you suck at managing and managing our recruitment pipeline was awful. I was terrible at it, because I'm terrible at email in the first place and I literally had 550 resumes to go through and I was just like, "ah!" So, people got responses from me like two months later. I was like, "I'm sorry, I'm still kind of interested." And they're like, "I got a job ages ago."
So, to be honest, the hiring has not been the most stellar part of what we've done for growing our customer support organization. The goal here is to have a constant pipeline of excellent people and to bring them on-board as soon as we need them.
What we're looking at a lot now is metrics and if we hire enough people, that we can always reply to everyone in under four hours, then we should hire a couple more people. And then we'll probably be in a good state. We have no bones about hiring too many people for this position.
The other thing that we haven't talk about at all is that the support team does QA when they are not doing support. Was how do we decide when we needed to have 24/7 support and how did we train those employees?
There wasn't a moment when we said, "It's time to do 24/7 support." It kind of emerged as we're like, "Wow, we wake up every day and we are completely slammed. Then we spend the entire day getting through everything we had. And then we go to bed and we're exhausted." And once you sort of go through that grind for a while, you're like "This can be better."
As far as training, we have a very unique platform because everything we do for our business is in our product, which is highly searchable. So it's very easy to say, "Here are our processes. Here are the things that are your responsibility. Here's how you search. Go find some answers."
And for remote people, like the people in Europe, they tend to do some time in their normal time zone, and then really late in their day to have some overlap with us for a little while. So they have two or three hours with the rest of the US team.
But it really helps, because we have people on the East Coast. So, it's only six hours difference, versus the nine with me. And the East Coast people have been around for a long time. So we're all very helpful to one another. There's not an official training program. It's very much lead by example, and we have a lot of examples available.
First question, we actually never used the Twitter integration in Zendesk and it's really heavy weight. You have to do a search for a thing, like your own mentions and then turn them into "twickets." And then a "twicket" is a thing with an ID that you manage in the same way that you would manage any other email.
And you get the entire screen taken up with it. You have all of the actions. And it's just a lot. It's really slow. And I think that if we had 20 tweets per day, it would be fine. But when you have a couple hundred tweets that you need to get through and I have two hours a day and the CEO has one hour, it just didn't cut it.
We needed something faster, and we tried Tweetdeck. Which was fine, almost, kind of, when there were two of us. But we had to click through each one to see if there was a reply. It was a mess. So we've tried different tools.
We've settled on Respondly. We started using them when they were very early on in their product development cycle, so we've been able to work with them to get some of the stuff that we need and they've been awesome and responsive.
So, the second part, what other channels do we have? And it's really small. So we have feedback at Slack.com and it also works for help and support and PR and anything you might think of will generally work.
That's the one that people think of, but the main source of our tickets is actually the in-product help. So, if you launch Slack and you look in the upper right corner, there's a question mark. And you click the question mark and you can log a question. And it's kind of like Intercom. We built it on the back of Zendesk. It's on the Zendesk API.
You go to a page, you can see your open tickets. You can see responses from us and send responses to us. And one thing that we've done there is that we've actually mirrored all of that data into our own database. So even if Zendesk is having an outage, or even something's just a little bit slow, the customer never sees that. It's always the exact same fast response that they get with the rest of our product.
And that's also allowed me to do some slicing and dicing of data. For example, figuring out how long customers are around before they send a help ticket. I have all of this in our database, because I have the user information, and I have all of their help ticket information. It's like, "When was that user created? When was that help ticket created? Boom".
So, there are a lot of benefits to mirroring your help data internally. I couldn't have predicted that that would be awesome but it is and there are other things that I've sort of taken off and done since that data is available. More like fun, "I wonder about..." side projects, rather than strategic initiatives. There's a lot of cool stuff you can do when you have all of that stuff locally, rather than going out to the API to pull it every time.
Do I triage? Yes. We don't triage according to whether someone's paying or not. So everybody just has a problem or a question. But we do answer in order or paying. So if I have 10 paying customers with problems, and 10 without, the 10 who are paying go first, unless there is a significant product problem, like, if someone's team is somehow broken, which doesn't really happen, but let's just say it does. Let's say that someone can't login to their team. And they're not paying. Then, that will take precedence.
Thank you so much.