In this Speaker Series, Datadog’s VP Product Design, Stephen Boak, outlines his approach to product discovery from early exploration and interviews to developing long-term partnerships

  • Introduction
    • Product Discovery Strategies with Datadog
    • 2011-Present Timeline: From Boundary to Datadog
    • Datadog & Heavybit
    • Your Connection to Customers
    • Recruiting and Managing Design Partners
    • In-depth Example: Dashboards & Personas
    • Product and Design Briefs: Courting Your Team
    • How to Recognize When a Launch Fails
    • Press-Releases & Development: Courting Your Customers
    • Bundles & Upsells: Defining Team Organization
    • Conclusion: Final Thoughts & Closing Statement
  • Q&A
    • Questions to Always Ask Your Customers
    • Customer Surveys for Gathering Information
    • Avoiding Leading Questions with Customers
    • When to Include PMs
    • Sample Size
    • Breaking Through to the Day-to-Day User
    • Buyer vs User
    • Teams Without Designers
    • Design Research for Bundling and Upselling
    • Data to Inform Design Decisions or Improvements
    • Sourcing Design Partners for New User Categories
    • Staying Aligned in the Early Stages
    • Realizing 1 Feature Should be 3 Features
    • How Your Reliance on Data Changes as You Scale
Download slides


Product Discovery Strategies with Datadog

I'm here to talk about product discovery, and how I've run product discovery processes, and how we do this at Datadog. And some of the lessons that we've learned by doing this effectively, and maybe not so effectively at times, and hoping to share some of those takeaways with all of you.

And before I freak anyone out with the title of the talk, these are birds. These are birds of paradise from the Planet Earth series, and all of the weird, unique, and nuanced courtship dances that they do for each other were kind of the inspiration for the title and the theme of this talk, and all of the weird, and unique, and nuanced conversations that we have to have with people both inside and outside of our organizations to get software built in the right way and for the right people.

2011-Present Timeline: From Boundary to Datadog

First a little bit of background about me. So I've been at Datadog for four and a half years, but I've been around the developer tool space for almost a dozen years now, across four different companies. Before Datadog, I was a co-founder of a Heavybit company and spent some time in the program. And also for a brief time, did a podcast called Don't Make Me Code about developer experience design. I have just been around the space and the customers and developer tools generally for a pretty long time now, and really have fallen in love with the space.

Datadog & Heavybit

I think people sometimes don't realize quite how big we've gotten, we're over 2000 people now. When I joined four and a half years ago, we were about 200 people. So we've grown quite a bit since then. The company was founded in 2010, though. So a lot of the bedrock principles of who our customers and who our user base is, these were things that were pretty well established by the time that I arrived. But the focus on DevOps, the SaaS business model, all of these things are very much part of our business today and the bedrock of the company and the product. Because I'm going to be talking about them today, I wanted to make sure that I mentioned them. So I'll start with the obvious, you're probably a founder or an early employee of a devtool company and you may well be a developer yourself. Many companies that go through Heavybit are dogfooding their own products, we at Datadog, certainly do this. Our own engineers use the product every single day.

Especially at the early stages, you're just really close to the users and the products. Often at the early stages your product does one thing, hopefully does it really well. And you as a founder are maybe dogfooding that product or really close to it. This is actually the thing that made me fall in love with the space to begin with. As a designer, I'm not a user of Datadog or most dev tools in general, but all of our engineers are. That kind of proximity to a group of people who are passionate about the tools that they're building, these engineers, who care deeply about the experience of the software they're building and who have a deep vested interest in it and skin in the game. This is really the thing that's made me passionate about the space and why I've stuck around for so long. The engineers who care about the UX of what they're building and who also have been great partners to me as a designer, and who give me great feedback about the products that we're building.

Especially at the scale that many of you are probably at, chances are that making software will never feel as close and as personal as it does to you right now.

And so that's something you really should be leveraging as a strength.

Your Connection to Customers

If you're here, you're also already connected with customers. If you haven't got any customers yet, that's a different talk. But in the early days, you're working really hard to define the core assumptions that are going to make up your understanding of your business, your product, and your users. These stay with you essentially forever. These bedrock assumptions at Datadog is still very much with us today and everything that we build connects up to these assumptions, every product, every feature that we build connects up to this understanding of who these people are.

You probably know these things as personas. As a designer, I've actually grown to dislike some of the dogma around this, the fancy slides with a photo and somebody's fake name and job stories that articulate all the specific things they do as a part of their day and everything. I think it's a little bit overdone and a little bit too specific, but the way that we mix and match these assumptions and this understanding of the people that we're building products for at different times, this is undeniably a crucial part of building any product or business.

I don't want to discount that. So what this looks like at Datadog, many of our users are DevOps professionals in practice, what this often means is their developer is managing their own infrastructure. One of the key phrases that we've used from the beginning of the company is "Datadog breaks down silos between dev and ops." And now some other silos that we're trying to break down, it's kind of a core mission statement for us. Many of our customers are cloud native, they're using cloud-based infrastructure or hybrid infrastructure. The things that come along with that, the dynamic ephemeral environments, the use of a wide range of different software languages, and technologies, these are all kind of principles about our customers. Last but not least, they trust us. They trust our expertise. I wanted to specifically call this one out because at more times than I would expect I have had in my career in this space to fight against the assumption that our customers are experts, that they don't necessarily need or want our advice, that they know what they're doing in their jobs.

And at a macro level, there may be some truth to this. Of course, our users are experts in their fields. They do know what they're doing, but at a micro level, all of the different technologies and languages and frameworks that we integrate with, the way that software development changes so rapidly over time new technologies and frameworks are introduced all the time. And so we really do feel like the stewards of best practices.

F or pretty much our entire history, and through this day, some of the most common questions we get asked are "what are the best ways to monitor X? What are the best ways to do this?" And our customers really do look to us for these best practices. And of course, all of this is true of Datadog's own engineers, it's one of the reasons I wanted to call out what this looks like at Datadog is because software engineers at Datadog very much fit this model.

Recruiting and Managing Design Partners

And so what we actually do when we go and recruit partners, when we're building a big new feature or product is, and this is one of our most well-worn tried and true processes. We use this for virtually everything that we build at Datadog, I've used this at other companies as well. We go and first recruit and identify a small number of customers that are interested in the problem that we're trying to solve.

This might be big users of a particular product that we're building. They may have filed a lot of support tickets about a particular problem with us. Increasingly we rely on our sales organization and CSMs who may know that they have a customer concerned about a particular problem. And we use all of this to recruit customers into kind of a private beta.

I really want to pause here and emphasize the people that we find during this process. And especially at the early stage when Datadog was a 200 person company, and I was really starting to form relationships with some of our early customers, the people that were generous with their time, that were thoughtful, that had a lot of great insight, and really did help us in our product building, these people really were special. And so it's not only important to make them feel special as a part of this process, but it becomes kind of a positive flywheel.

As you devote more time to these people, to finding them, and to spending time with them, you can go back to them over and over again.

They are a bedrock of, and have always been a bedrock of our business and finding them and recruiting more of them is really crucial, I think, especially in the early days. And so we are bringing these people in, we're showing them ideas. The design team is showing them mock-ups with ideas, we're getting feedback on them. And then when we have built software and we're ready to show it as part of a private beta, we bring these people in and also make them feel really special as a part of that process.

We treat their feedback as high priority. We're trying to iterate very quickly during these early phases and to show them that they really are important into the process of developing in these early stages. And then once we feel confident enough that what we've built is going to satisfy a wider audience, we go release it more widely. And again, this is still a crucial part of our software development process today. We actually call these people design partners.

In-depth Example: Dashboards & Personas

So I want to get now really granular and talk about at least one example of this in our dashboards product. You have the persona, as you would expect, the software engineer at a software company, and they are troubleshooting an issue with a service that they're working on. And so all of these core assumptions, this is a DevOps engineer, they're cloud native. They trust us for advice. And this very specific use case of wanting to troubleshoot a service that they're working on is the one that we focused on at this time, a couple of years ago with our dashboards.

At the time, our dashboards pretty much looked like this, or a flat list of widgets, all of these widgets on the screen are the same size, they're laid out in this extremely, rectilinear way. You have very limited control in this space over arrangement and other things. The things that we are hearing from customers at this time were like, one, "I configure these dashboards in a certain way. I basically make one dashboard for each service that I work on, but I also include clusters of widgets for dependent services, dependent infrastructure."

So there's a kind of a relational model of dependencies here in the board. And then further, actually people are designing these things in a particular way. Like they sort of visualize it as a two column or three column matrix and they visually cluster and organize things in ways that make sense to them. And they put the most important metrics on the top. And so they really are designing these almost like a CMS system to suit their needs. And so we wanted to build features to better cater to these needs. So you really can see the customer persona come out in this, the cloud native DevOps engineer that has a service oriented or micro-service view of the world, they are building these clusters of widgets for dependent services and infrastructure. T his model really does tie back into that, those core assumptions of our users and their needs. And so again, just to kind of pull this out, you can't really see it right away, but people are clustering these widgets visually, and they want tools to be able to do this better.

If you were looking closely and we did look closely at a lot of the boards of the customers we're building, we could see these structures emerge in the way people were building them and laying them out. And so I also wanted to emphasize the 'trust us' piece of this, because it wasn't something that we particularly pulled out as a part of this exercise. It's just been a part of our process for a very long time.

But whenever we build an integration, and Datadog has over 400 integrations with other technologies now, we're almost always building some kind of out- of-the-box dashboard, something that guides customers and shows them "here are the metrics that we monitor for this service or for this technology. Here's a dashboard that lays them out in a sensible way." And so we're really trying to give them a good starting point, a set of recommendations about what is available to them, how they can organize that in a way that makes sense, and really trying to start them off on the right foot.

This has always been a part of how we build new products. And so what we ended up building in this case is a few different things. First, we allowed customers to literally cluster widgets into these groups. At the top here, you can see in the control panel somebody can create a new group and they can put widgets into that group. Then in addition to that, we revamped a lot of our out of the box dashboards, these starting points to include these groups.

So now not only could people have a dashboard as a starting point, they could actually look at individual widgets or clusters of widgets. And then, we built a clipboard basically, command C, command V. So I can just hover over a widget or a cluster of widgets and press command C. And then as this little looping video is showing, I can just paste that into my own dashboard. So we're trying to give people better organizational tools. We're trying to improve our own out-of-the-box content. And we're trying to make it faster and easier for people to get started with new technologies and to create dashboards, to monitor those technologies. And we tried to make this all part of a seamless experience. Then what we said when we shipped the blog post for this was a lot of that same language. We know that dashboards are a shared resource for engineering teams. We know that people wanted and always want tools to make dashboard creation and building up the knowledge around these things easier. And we want this process to be a frictionless part of troubleshooting because ultimately these boards are the way that people understand the health and performance of the services that they're building People are not only copying and pasting and bringing in content from our dashboards, but once they've built up a few, they're starting to share them across their own boards. We've actually kind of looked at this sort of genealogy of dashboards in Datadog and how people will start with what we build out of the box. And then a team builds something for a service, and that becomes canonical and people start copying and pasting things into their own boards out of this, and so making that whole experience as seamless as possible is really what resonated with people and ultimately how we communicated it publicly.

I now want to get into the dance and the different ways that we had to communicate both internally and externally to get this done. So here again are the birds.

Product and Design Briefs: Courting Your Team

And first I want to talk about the kind of inside- out, or internally-driven approach for courting your own team. Many of you are product leaders and leaders of your companies and so it's your responsibility to align everyone internally on what you're building. This again comes back to these clear personas and use cases and defining what successful outcomes and workflows look like as a part of this.

We didn't always have these. When I first joined Datadog, we didn't have any process for building product briefs or writing down these personas as part of those briefs. And it was only after some pretty big failures. And for my team in particular, the design team, we felt these failures in terms of iterations.Getting through version seven, version eight, version nine of the design, people are still arguing about whether the solution works, whether we're satisfying the needs that we have.

Because none of this is written down, it was really hard to get agreement on this. It wasn't until we got to maybe 250 or 300 people that we really started implementing this as a part of our process, writing down briefs, making sure we had alignment on these things. And it was some pretty spectacular failures that led to this. I think our biggest launch failures have really always come from a lack of clarity of who the persona is, and why we're building something. And when we've had to go back and rethink those things is when the pain is caused from this.

How to Recognize When a Launch Fails

And so in a little more detail, what these failures have looked like is, if we're modeling the persona in the wrong way, if our understanding of the persona is not quite right, actually something I wanted to say earlier in this process or earlier in this deck that I didn't have in the slides, but I think is really crucial is in these recruiting conversations, when you're talking with people, far too often there's an inclination to go in with a sense of what you want to build already.

It's kind of one of those pieces of fortune cookie wisdom: just listen to what your customers are saying. I think it's a really hard thing to do in practice. Often internally you can kind of rally around a sense of what the thing is that you want to build next. And you come into these conversations with customers already having a lot of inertia and a sense of what you want to build. I think this too is a principal cause of this kind of pain.

If you go in assuming that you have the right answer, we're just modeling things in the wrong way and getting the wrong answer here. You ultimately start showing through this to people and you'll hear things like, " I don't get it," or "I don't understand what this is. I don't understand how it works." And this confusion is really the most common theme that we've seen when we're not getting this right.

Just people, they don't have a lot to say other than, "I don't really understand what you're doing here." And what this ultimately leads to is tear down. So as opposed to design iterations, we've now invested potentially person months of developer time building something, and we have to tear it all down and do it again And so not only can that failure be painful, but I think the sunken cost fallacy and all this can make it even worse.

If you sort of assume, "I've already invested a lot of time in this, I think fundamentally we're on the right track, but we just need to change a few things," maybe that's right. But if it's not, you go from sort of a medium-scale failure to an even larger failure. And so really being willing to tear it all down, so to speak and being willing to admit that you've gotten these assumptions wrong, especially at these kind of middle stages is also crucial in not making the failure even larger than it already is. So then the other dance that we're doing is the external.

Press-Releases & Development: Courting Your Customers

We don't formally practice this at Datadog. I've heard this called press release driven development or outside -in, I know this is an Amazon thing too, and they call it working backwards, but it's the more customer-driven approach to this. You're trying to court your customers from the very early days. You're trying to imagine "how is a customer going to react to this?" And so you start by writing the press release. You write it in the way that you think is going to resonate with the right kind of customers.

We have tried this a few times at Datadog, and I think this can be a really helpful exercise, but a couple of times in particular, what happened was the internal team wrote only a press release. So absent the formal product brief, absent the things that I think our teams internally, especially designers really need to be effective, you can't build products from just the press release. You need the specific understanding of the persona and what you're actually going to build.

I think both of these things serve a really interesting purpose, but in particular, I found myself advocating against press release driven development. At least if your thing is, "we're only going to write the press release and the team is just going to build from that." So if you're already doing this, or if you're interested in doing it, I would just urge you to also, if you really care about your designers and engineers, you're also going to write this kind of internally focused language and develop proper briefs.

Bundles & Upsells: Defining Team Organization

This also is a really crucial part of the core business dynamic for us, and I think for many of the companies in this space too, which is to bundle and upsell. This persona is not static. It gets augmented and built up over time. So for 6 of the last 10 years, Datadog was an infrastructure monitoring company. We've since added application performance monitoring, and logging, and now lots of other products too. And we sort of move things into the fold we roll in now a complete picture of observability includes infrastructure, APM, and logs. We've marketed this as kind of the three pillars of observability. I think now it's more like nine or 10 pillars with the different products that we're launching, but all of this gets folded into our customer model. The looping video that you're looking at here is a little piece of an integration between our application performance monitoring and logging products, where if you're looking at a trace, you can see the actual logs being generated by that one trace, and then you can pivot over to our launch product and actually view those logs in the explorer. And so this seamless, connected experience between products is really a part of how we bundle and upsell software.

It's also a big focus of the design team these days is when we're building a platform of interconnected products, we want the experience to be consistent, connected, and so it really does feel like one seamless platform experience as you move around. This is selling off that value. Now you can view your traces and your logs together, and you build up this model. Now this complete picture of observability includes all of these things. You extend that, you upsell, you grow the business based on an extension of this model of the persona.

Last but not least this also defines the way that we build our teams as we add more and more products and personas that these products are for. It actually has informed how we build say the design organization, that when we're figuring out how to structure a team of designers, we want them working on products that are related. We want them serving the same kinds of customers, in part so that they can work well with each other. They can review each other's work really effectively. They have a similar and shared understanding of the customer and the product that they're building. This is not just true of design, but we're all really structuring our teams in this way, around the customers that they're serving, because these tools go pretty deep. It takes a pretty long time as a new person in the organization to understand "here's the customer that we're building this software for, and here's what they need." And so we want people to be able to spread that knowledge and to build it up with the team over time. It really does inform how we're building the organization and not just the product.

Conclusion: Final Thoughts & Closing Statement

So in conclusion, again, when you're starting with the product discovery process, you can begin by recruiting people that are close to you, very likely you're dogfooding your own product. And so you recruit a small number of people based on these core assumptions about who your users are and what they need. You recruit them in to become design partners based on these strong matches.

Over time, you bolt on more and more of those assumptions and you add new features products for them. This is a big part of how we upsell and how we add value to the product over time. And this not only informs how we build products. It informs how we build teams, and we organize ourselves around the needs of particular kinds of customers. And this is basically where we are today is a 25+ billion dollar company and how we organize our teams and our products.


Questions to Always Ask Your Customers

We do have things like a customer advisory board, a set of people that we're meeting with regularly, it's the same group of people. And so we're getting pretty consistent feedback from them on what we're building and overall strategy. I don't want to make it sound regimented. I mean, part of the goal is actually to recruit a diverse and varied group of people. I myself have been led astray by limiting this reach too much at times, by focusing too much on software companies, for example.

There was a time when I would go and talk to, some of our biggest users who were software companies and a lot of them were singing the same tune. They wanted Datadog to behave more like GitHub, like a CMS. "We want to automate everything we build, we want tooling. And then we want this to be managed like code." And this was a few years ago, but at that time they were a relative minority of customers who were very vocal about what they wanted, but it was then really important to go and talk to other different kinds of customers who wanted very different things.

I think there's a fallacy in actually trying to make it a consistent thing, or at least a too narrow a thing. Certainly as we get more and more varied customer types, we want to bring more of those into the fold. And in part there's a learning process as we're building new things as to who really is going to latch onto something. And so you don't actually know in the beginning who those people are necessarily going to be.

Customer Surveys for Gathering Information

We still have a rule today, we don't do surveys. I think in part it's impersonal. I think one of the nice things about enterprise software is the power law distribution. The 80/20 rule of customers, that 20% of your customers account for 80% of your revenue. And even today Datadog has 12,000 plus customers, but you'd sort of do the napkin math on that. And it's not a crazy number of people that are disproportionately contributing to the revenue and the health of the business.

Because these people are using the tool every single day, they live in it. And I mean my experience talking to customers directly is they're willing to be pretty generous with their time because they have so much skin in the game, they care a lot about it. And the survey approach has always felt kind of cold and impersonal and like a half measure, as opposed to really investing the time, and sitting down with people, and seeing what they have to say.

Avoiding Leading Questions with Customers

First, I'll be frank and say, most of my career in software has been at pretty small companies that have not had formal research organizations or practices around this. And this is something we're formally building out pretty recently here. Building up a formal research practice that tries to do things like this in a more consistent and measured way. And so I don't know, I wish I had a better answer to this.

I've always bounced between product management and design. I am not trying to throw shade on PMs, but I've always felt like PMs have been more susceptible to falling in love with the thing that they want to build next, the next feature that they have in mind. And I've always felt like maybe it's my ignorance as a designer and not being as closely connected to the individual customer that we're dealing with on a given thing. Maybe it's that I have less vested interest in a particular thing that we're building.

I've always felt like the thing that actually has given me leverage in these conversations is not caring quite as much about the exact thing that we're building. And that's really all I can think to say is, don't care so much about what you're building upfront and allow yourself to learn.

When to Include PMs

We don't have a greater structured way of doing this. Depending on the day, depending on the thing we're doing, it'll be a varied mix of people in the room. We've never tried to practice this in a formal way.

Sample Size

It cannot be one person. I think one of the most interesting things about these interviews is you would think, they're saying something and everybody's going to interpret it in the same way, but actually one of the most interesting things that comes out of this is when people do interpret what a customer is saying in different ways and the sort of debates that that leads to, and even follow up questions that you can go back and ask and other things like that. And so in general, I think one product manager and one designer will often be in the room, but no matter what you're doing, you have at least two or three people from the company in the room, so that you get that varied perspective. And you don't just filter down to one person's perspective on something.

Breaking Through to the Day-to-Day User

I guess we're starting to see more of this disparity now, I actually feel like this was much less of a problem, so to speak when we were smaller in scale, I think this sort of ground truth around dev tools companies, this land and expand model too, that you get in the door. O ne of the keys to Datadog's success has always been the kind of turnkey nature of the product. Anyone can go to the website, sign up, install an agent and get going. That individual engineers and teams are so empowered now to dictate buying decisions that historically, we've just seen so much energy from the teams that are getting value out of it that the buyers have mostly, I don't know, gone along for the ride.

We are seeing more of this disparity now, especially like in organizations that have already adopted Datadog, and there's a decision about whether to adopt a new product, or maybe they're using some competing product to do something else in the platform. We also have a kind of disparity between manager and maker use cases. There's sometimes a diverging set of user needs where a manager or director overseeing several engineering teams.

I think you could see this in our SLOs product, a manager or a director will really care about certain things when looking at SLOs across teams, how do they manage resources to keep those SLOs up? How do they trend over time? They're actually looking at different time periods. So we have had to make hard decisions. I don't know if that directly translate to buyers though. We definitely have to kind of consider these different use cases.

We do have to hone our language so that the language resonates with the buyers. They're also, we're up against like Gartner and Forrester and these analysts. And so that the buyer language, I think often aligns with the analyst language and so selling to that and actually crafting features and marketing language that kind of gets us ingratiated with those agencies is also a principle of how we build software. And so I think that speaks to the enterprise buyers, and the way that they evaluate software is the analysts and the frameworks.

Buyer vs User

I'm struggling now to think of a conversation where this has happened with both parties in the room. I think generally what will happen, I mean, more often, I think in these kinds of organizations it begins with CSMs and maybe a product manager going into the room. I actually don't have much visibility into this exact conversation, but, I know that CSMs and PMs are kind of, they're doing early demos.

So you get everyone into the room, everybody sees the same thing, but you've got language in there kind of catering to both sides. You have the analyst language that caters to the buyer, so to speak. I wish I had a better answer to this. I don't know that we're purposely trying to like segregate those two groups. I think what often happens is they're in one room in the beginning and then maybe it becomes two different conversations as we're developing the prospect where time is part of the sales funnel.

Teams without Designers

The founders of Datadog, there's always been kind of a bias against overdoing this, focusing too much on the science of it and not enough on just the talking to people and learning part of it. And, I certainly understand and agree with the principle that there is a rational sort of scientific way to do this. You disconnect yourself, you try not to ask leading questions, you do this, but I mean we're just starting to develop this out now at the scale that we're at. And I don't think that focusing a whole lot on the science early on would have helped us all that much. I think that the conversations and the meaningful conversations were really the most important part of it early on.

Design Research for Bundling and Upselling

So I guess what I would say about this, as we grow more and more where we now have some new products where there's honestly a question of, "is this an extension of our current core user base, or is this actually some fundamentally new kind of user?" And I think when it comes to bundle and upsell at least the question of whether you're bundling is an important one. And I think for an early stage company, what I would say is you probably want to buy as heavily towards something that's proximate to your current users. Something that you really can easily bundle in as opposed to something that's more distant. And because that's harder, it's more work, it's a different persona. It's maybe a different product experience. I think we're starting to see this with say, Datadog security product. We're now kind of evangelizing, not just DevOps but DevSecOps and folding security into this. There are honestly questions of "is there now a persona that just exists entirely within the security product and may not care at all about most of the rest of the Datadog platform?" I think that we could only really consider questions like that at a bigger scale. And I think it would have been dangerous early on to try to build something that was too far removed. I think it helps that we actually have always tried to find things close to the core business to grow.

Data to Inform Design Decisions or Improvements

We have an internal tool. We use Metabase, we have a bunch of our own internal databases. We are tracking, who are the biggest users of given features. We are tracking page views. And so we have a pretty good sense of activity and how people are using the platform, looking at user funnels and learning who the biggest users of particular features are. It's been interesting, the transition to Datadog becoming a public company and the compliance and security around this, views of this kind of information have gotten a lot more restricted.

When we were a much smaller company, more people had more visibility into this kind of thing. we even had tools that are very tightly locked down now, but tools to kind of ghost into customer environments and actually see how things were configured. And so you could actually kind of like act as if you were that user. I think all of those things have historically been helpful in helping us understand customers.

Particularly for this dashboards exercise that I talked about, it was great to actually have customers show us the dashboards that they built, to even go and look at some of them ourselves and not just to have people tell you what they're doing, to actually go and see it for yourself. To actually be able to see their environment. I think this is true of pretty much every enterprise tool. They're kind of like an empty shell in the beginning.

The personalization of the experience to the team and the company is so crucial that when we tried to show a sort of generic Lorem Ipsum style mock-ups of things, they just don't resonate, like I don't see my environment in this. And so for some of the biggest roll-outs we've had, some of the big visualizations of the things that we've done, we've gone as far as working with customers to plug their real data into something and putting it into a sandbox environment to show them just so that we have something that resonates with them.

Sourcing Design Partners for New User Categories

Here, we've done a mix of things. I think the best case, or I don't know whether to call this the best case scenario. I mean, one scenario is that we are moving into a product space that's, somebody may be a customer of Datadog and some other tool. So they might be using Datadog for infrastructure, and maybe Splunk or something for their SIEM or security monitoring. And so those hybrid customers are a great starting point finding them first and having a good list of this.

Our sales organization has a really good list of this. The customer is using Datadog for one thing and some other company for another thing. S o we can go and find those people and talk to them. And I think in some cases we've even gone outside defined users of these products that don't use Datadog at all, and trying to get a better handle on the value that they provide.

Because I've kind of oscillated between product management and design in my career, I've always been trying to, I've always wondered where exactly is the line here. And these kinds of scenarios are very product driven. Is it a product manager going out and understanding what is the business model? Who is the customer base? Where does the value in the billing, the pricing model come from and all these kinds of things?

As opposed to say, large incremental improvements to features that we're building like our dashboards product, it's maybe the biggest and most widely used part of Datadog. And there's a ton that we can do by talking to existing users as a really rich vein of feedback and bug reports and other things that people have. And so that can be a much more design driven or sort of internally driven exercise as opposed to things like market research and finding people who aren't even necessarily users of your product.

Staying Aligned in the Early Stages

Let me see if I can get at this because it's something we struggle with all the time, the docs and written communication in general are crucial. I mean, especially as we've gotten bigger and bigger, you can't expect in any rational way to get everyone in the room at the same time. but the problem we often have is the doc sometimes already represents some kind of alignment. A product brief goes out. It means in some group of people there's already kind of a line on what that is. A design review document goes out, maybe it's a pretty explicit pitch for one direction.

We are also very much trying to solidify the ideation phase more so that people aren't just giving you feedback on your ideas, they're giving you input. And I think if you read, like there's a lot of evangelism in the design world about this, this kind of design sprint style processes where you get everyone into a room and you whiteboard and you generate ideas. COVID has been an interesting, kind of experiment in all this because the whiteboard is not there anymore, but the ideation phase still needs to be there.

I do very much see this as a role of the design team to work with product managers when there is a debate. And sometimes there are these really heated debates about who is going to be the user for this thing. Is it going to be one kind of person, type A, or type B or whatever. To very explicitly surface that debate in some kind of concise documents, we're trying to choose between this and this, here are the workloads that cater to those different things.

Maybe we're missing the thing entirely, like in at least one case I can think of, there was just another scenario that we hadn't even thought of. And it was someone in engineering leadership that came through and identified that. And so the docs in some ways close doors rather than open them. And so I would encourage all of you to not only have those kinds of things, that document what has been done, but to explicitly try to invite in ideas via concise, I really can't emphasize the word concise too much, if these things become book reports, especially as you get bigger, there's a whole lot of doc fatigue that goes on. And so working hard to bring ideas in without overloading people.

Realizing 1 Feature Should be 3 Features

The way you said it, that feels like a standard part of product building. Sometimes we've realized we have bitten off more than we can chew. Even to be honest, some of that dashboard stuff that I showed, that did not ship at once that shipped over, I want to say a year actually. And so that bedrock truth, the knowledge that led to us shipping that stuff did not change in that year, but other priorities came up other things shift. And so those three main things that I talked about, the revamping of the out of the box content, the clusters, the copy paste. Those did not ship at once. It wasn't some giant launch.

That's one example, but this happens all the time. Just like we have this idealistic view of where we want to get to, but this is in fact, as opposed to sort of an agency model, one of the things I love about working embedded in product companies is this perfectionist ideal. You're trying to strive for the perfect product, but you never quite get there and you spend years and years trying to get there. And you iterate and break things down into small pieces to be able to shift them.

How Your Reliance on Data Changes as You Scale

I, and I know others inside, we think we're not using data enough. We still heavily, and I think this has just been in our DNA as an enterprise software company, as a devtools company, we heavily bias towards qualitative forms of research and talking to people. And in many cases, I don't know that we have the numbers to do split testing and these kinds of things that you might see other types of companies. And I do think there are places where we could be making better data informed decisions. And I think we, if anything have an over-reliance on the qualitative approaches to research. So I know that I've seen this happen in other places, this over-reliance on data. And if anything, I think we had the opposite.

This was really fun. Thanks for having me.