Ep. #93, Adaptive Sampling Strategies with Mike Goldsmith

Mike Goldsmith: So cost control has definitely been an interest of mine recently. I think I've worked on a few different projects, and working with customers and people in the community that just send a huge volume of data. And a huge volume of data has really got a high level of fidelity, but it occurs a huge amount of cost as well.

And a lot of that comes with redundant information. So when you've got redundant information, you're having something that looks identical, and you'll send it 100,000 times, and it doesn't have a lot of value. You just need to know that that thing was sent at a particular frequency, and then you can extrapolate from that to say how often something happens.

But having an identical record of something thousands upon thousands upon thousands of time isn't adding value to you. And there is a cost incurred, and that's wherever you're sending it, whether it's different backend analysis systems or if you're sending it to like localized ones inside your infrastructure, there's a cost of both processing it and then doing something with that or storing it and then analyzing it.

So having different ways to keep the fidelity, so understand what's important and why, but not always exactly the number of things that was done. And yeah, there's a few different mechanisms available now that held control at the different ways.

Martin Thwaites: So fair to say that you like the hard problems.

Mike: Yes, I do like the hard problems. I remember when I was at university and my university lecturer said something along the lines of, "You're one of those people that likes to chew barbed wire, aren't you?"

Martin: Okay, so building stuff for cost control in the Collector is like chewing barbed wire. Is that the quote of this podcast?

Mike: I think that's a very good analogy, yes. It's a very difficult problem. On the surface, it always seems not too difficult when you're talking about, "Well, I just don't want to keep all of this data."

But there's really interesting problems once you start to do it at a really big scale. And some organizations generate petabytes of data per hour, per day, whatever that looks like. Applying those interesting things over that amount of data in a way that then doesn't become cost prohibitive itself is a very big problem.

Martin: So pretty much like the iron triangle thing, isn't it? You know, it's like you want good telemetry data that's, you know, just the good stuff. You want to get it fast and you want to get it cheap.

You've got to think about memory constraints and CPU constraints, and you're constantly moving between all of these boundaries, which is a really interesting problem space, I would imagine.

Mike: It is. It's very interesting. And as you're right there, you've got the cost, the space, the efficiency.

If you do some cost controls just really heavy-handedly, the cost of running that cost control could not be that much different from just sending all of that data.

So you have to be really intelligent and controlled about what decisions you make, how you apply them, are they doing enough for you? And yeah, as I said, we're introducing more mechanisms now that are available to be more efficient and more understandable and higher and easier to tune. Just using the hammer for everything isn't always a good option.

Martin: I mean, I suppose this is probably a good time to let people know who you are and what it is that you do.

Mike: Yeah. I'm Mike Goldsmith. I'm an engineer at Honeycomb. I've been working in and around OpenTelemetry since it started in 2018, 2019.

Typically working on OpenTelemetry tooling, so inside the SDKs at the specification level, and most recently an awful lot of work in the Collector trying to make that more usable, more accessible, and solve more problems for more people who are operating it.

Because realistically now, if you are an OpenTelemetry user, which most people are or want to become, if you've got any amount of real, actual live data that you want to monitor through a pipeline, you need a telemetry pipeline, the Collector's going to be it, and operating it is not a simple task.

Doing things in the right way is not always the most simple, so that's definitely an area that I've been focusing on to try and make it easier, have more control, and yeah, that's definitely something interesting.

Martin: I suppose there's lots of different right ways, though, isn't there? Like, right is not one thing, because if right was one thing, then there'd only be one config, and there wouldn't be any kind of configuration if there was one right way to do this.

So, you know, there's lots of different ways that people can configure the Collectors to build these pipelines, from tiering to adding different kinds of components and all of that kind of stuff. So, you know, the right way is way more nuanced, I would say.

So you building like lots of things in the Collector is really just about choice. Is that right?

Mike: Yeah, that's right. So the Collector is intentionally built to be in a composable state. So you pick the pieces that are important for you, whether you're going to be receiving metrics or logs or traces and where you're going to get them from. Are you going to get them from Kubernetes? Are you going to get them from your application that's running Go? Or do you run a Python Django app or whatever that looks like?

Telemetry comes from lots of different sources and then therefore naturally looks differently. And what you want to do with it might be different as well. So there's lots and lots of variances in there.

So the Collector is intentionally composable, both on how you receive the data, what you do with it when it's in the Collector, and then how you send it onward if you're choosing to do that, whether that's in storage or a backend analysis system, whatever that looks like.

But yes, it's intentionally composable because what you want to get out of the Collector is nuanced in that you've got to decide what's best for you.

Martin: Cool. I mean, you said the right way. You didn't say there's lots of right ways to start with. So what do you consider some of the right ways, in inverted commas, to use the Collector?

Mike: I think I touched on it earlier. Don't keep everything that looks identical. I think keeping everything for the sake of keeping it is a bad choice. It adds additional costs.

Martin: But what if I'm a telemetry hoarder? Like I have an attic full of boxes of telemetry.

Mike: Then you need an intervention.

I mean, there are scenarios where you need to keep access to everything. That's not unusual.

Definitely worked with people in the past and companies in the past, organizations that require by law to keep everything. Like there's some organizations that require to keep everything for a certain amount of time.

But that doesn't mean that you need to put it into your most expensive pipelines. So typically people keep everything and then they think everything then must go into my analysis system. Typically people pay their premium when they're talking to an analysis system.

So whether you're going to do investigations over it, you want to do your metrics, your dashboards, whatever that looks like. If you send everything that you ever have generated to those systems, it's going to be a painful, expensive experience.

Knowing what to send to what system and being able to make good educated decisions around how much of that you send and where you send it is really a big part of how you operate the Collector to be efficient and organized well.

Ken Rimple: So one of the areas I'm wondering about, you're talking about someone comes in, they decide they wire all of their telemetry in through the OpenTelemetry Collector and they're sending logs and metrics and traces. Where are the typical pain points that people run to early on? I mean, everyone's different, but what are your typical ones you keep seeing over and over again?

Mike: I'm trying not to be too mean. They're using the wrong telemetry signal.

Martin: But everything's a metric. Everything's a metric, Mike.

Mike: Everything's a metric. Everything can be a metric.

It's the whole, going back to the whole thing, just because you can does not mean that you should. And yeah, I'd say that's probably one of the biggest observations that I see is that people are so heavily metrics-driven.

And metrics has a place, it's got value, but it definitely does not tell you all of the story. And it can't tell you all of the story because of the way that they're captured. It doesn't give you context of why something happened. It tells you that something happened and how many times that thing happened.

So it's not infrequent that I come across organizations, pipeline setups, all of those things, where they rely on metrics to tell them the story. But metrics can't tell you that story.

Logs, traces are a much better mechanism and give you so much more context and value when set up correctly. There are some caveats about that. So they can tell you a much richer story. Metrics can very easily tell you that something happened and how often it happens.

It's the logs and the traces that are the things that tell you why something happened. What was the context? How do you get to the point that you're at right now? And all the time, does that analysis change? Can you tell the same story or, repeatedly, can you tell what might happen in the future because you can see a trend appearing? Telemetry that is rich in detail can tell you that story much better, and you can make patterns from, and metrics, as I say, already tells part of that story.

Martin: I think it's important to say that metrics aren't bad. Like, metrics aren't a bad signal. The problem is when the entire world is a nail and all you've got is a hammer. Like, this is the problem.

And, you know, there's a balance, isn't there? Everything's a metric. Great, okay, we're just using metrics for everything. But then there's also, well, let's not use any metrics. And then you've got the observability pillars where people are like, oh, if you want observability, you need logs, metrics, and traces.

I think the important thing that we're trying to get across here is use the right signal for the right job.

You know, I always talk about metrics as being two things. There's the big M metrics, and then there is the little M metrics. We always need little M metrics. Those are your graphs that you build. Those are the visualizations that you're monitoring to tell you about things.

And then the big M metrics are our time series aggregates. They're a data type that we've thought about up front, that we've decided that this is a thing that we need to graph in this particular way under this particular scenario because this particular thing might happen, otherwise known as known questions.

But that idea is fine. If you've got a dashboard that you need to produce in real time that has real value, metrics are the thing that you should be building. That is why metrics exist for those known questions.

The problem, as you put it, like being able to see what happened within a user's request, being able to narrow down those individual user requests and see what's common about different user requests — that's why you need logs and you need traces. You can do things like exemplars. There's lots of ways to do it. You don't necessarily need metrics, need logs and need traces together in order to achieve "observability" — 100% observability.

But you do need to be able to tell a story. You need to be able to build graphs and visualizations that allow you to be able to look at things in aggregate. All of these things are things that you need. How you get them is more interesting. So yeah, building those pipelines and relying on one individual signal is a problem.

It's about using the right signal at the right time and also being efficient about using that signal, which I think is the real purpose of why I wanted to bring you on, which was to talk about all of these signals — doesn't matter which one — can get expensive. Metrics can get expensive, there's reasons why metrics can get expensive. Traces can get expensive, there's reasons why traces can get expensive. Same with logs, mainly with people putting a log line that says, I was here.

That's how logs get expensive, but they can all get expensive. But what I think is that you've been spending a lot of time in the areas for each one of these on how do we make them better, or just how do we tell people how to use them in order to reduce the cost of these, to increase the performance of them, that kind of stuff. So let's start with logs, because I think logs is one of those ones where most of the people who listen to o11ycast will be either using them extensively or have used them extensively in the past. So how do we reduce our logging volume? How do we make logs more useful?

Mike: Yeah, for sure. So of the three signals, I would probably say that logs is the most voluminous because of what you said there. People will log something for the sake of logging it, and those logs then don't tell a story. So people say, well, I've logged something here, then I've logged something here, and I've logged something here. They all have fragments of the story, but that means that you have a very high volume of things that can come through. So in a very big system, the volume of logs that are being generated can be very, very large.

So a couple of the tools that I've added to the Collector and supporting the Collector, there's two parts that I'd like to talk about. And the first one is the log drain. So the log drain is an algorithm that is based from an old Python library.

It was through a research paper. It's been ported to Go, and then therefore we can add it to the Collector.

And what this does is, as logs come through, it analyzes and says, what's the shape of this log? Can I discern what parts are variable and what part is static? And eventually, over a learning period, it will then create a template. And then that template, it will apply over the log and say, this is the template that I had for that log.

And what that lets it do is it will let you say, well, what parts are variable? And then can I capture those variables as attributes? Because then I care about what that is.

So for example, if you had a log that said, user Mike logged in at IP address, like 192.168, whatever that looked like, the log template might say user, static; Mike is a variable, therefore I'm going to make that a placeholder; logged in at, all again, static; IP address, variable. So it will create two variables in that template, which is my username and my IP address. The rest of it is static.

So that means that the next time that it sees that log, it can say, I can apply the same template again. And now Martin has logged in at a different IP address. Ken has logged in at a different IP address. The template doesn't need to change. It's just the two variables that are part of that.

What that lets you do is you can then say, well, if I can see that the template looks like this, all I really care about is that I recognize the template and I want to know what those two variables are. The rest of the log value, the rest of the log frame, the template that you've seen, now becomes less valuable.

So I can do things like, well, if the user is this, then I don't care about it. Or if the user is that, I don't care about it. If it's an IP range that's internal to me, I don't care about it.

I know what that value means, and therefore I can make educated decisions on it. In that example, it's quite simple, but you could have a template that says lots of different information about it. It's got lots of different items in there.

But what's really interesting is if you start to pair it with things like the second component that we've recently worked on, the log dedupe. So what that does is, earlier we were talking about, if you see a very high volume number of things that all look the same, but you really want to care about knowing that it happened and how many times.

When you've got variable parts in your log body, so the log to string text, it's not very easy to say, well, this one looks identical to that one, which looks identical to that one, which looks identical to that one, because those variables are part of that framing. If your log body has Mike and Ken and Martin in them, they're not identical. It's very difficult to check which ones are the same.

So when you use a template with a log dedupe, those two things then say, well, this is the template framing. I know the attributes. This is the log dedupe.

And if I attach the log dedupe to the template, I can then say, this log frame, this log shape happened 100 times. I don't need to know that. I don't need to have 100 copies of it.

I can have one copy of it and tell you that it happened 100 times. And those two, in that scenario there, it's very simplistic, but you've reduced the log volume by 100 times and you haven't lost any fidelity.

Martin: But what if I need to keep those? I mean, it's a really good example, like, you know, user logged in. I mean, a better one would be user failed to log in, because that's normally a compliance log.

When you look at, say, regulatory systems, you need to keep the failed logins as a very important signal that you need to keep. But I need to keep all of them now, because now it's a regulatory signal. So I can't use a log dedupe on those.

How do we handle that then? Because now all of a sudden log dedupe doesn't really matter, because I can't dedupe any of these. What's the answer to that?

Mike: So going back to the log template, you know what values were in those placeholders. So you get the log template, you're attached to log dedupe after it, but if you know that, between the log drain and the log dedupe, you need to make a more informed decision, you could filter on something or you could route it past that and ignore the log dedupe.

In the log dedupe, you can even add conditions and say, do the dedupe based on this value. So you could say, what's the login status? Was it success or not success? And you could add that as part of the deduplication logic.

So you can say, I always want to keep these things, or I don't want to keep these things, or I want to keep them at different rates. You can do that between the log template, giving you the variables, so you can make educated choices.

You could add transforms in there. You could add filters. You could add reroutes. Or in the dedupe, you can then add the specific attributes as part of the deduplication algorithm so it knows how to dedupe against them.

Martin: So the reroute was an interesting one. Does that mean that what I can actually do is send them to different destinations? So I can send all of my data to one destination and then through a log dedupe process have a lesser amount of information going to another destination?

Mike: Yeah, easily enough to. So with the log template, you can then select a routing connector. It would be in the Collector. And then based on a particular attribute being set, which you know has come out of the log template, you can then say, if the user was successful, send it to the Log Dedupe and take one in 1,000 because I know that it's successful and I don't care about them.

If it is not successful, keep them all. Just bypass them, send them to S3, wherever you want to go, some sort of storage system. You can make choices before Log Dedupe.

Log Dedupe is a tool that is used to intentionally reduce volume. If you know that you need to keep something, don't pass your data through that. That's a choice that you can make.

But having the log drain, I think, is valuable enough to always be available to every log pipeline. Then Log Dedupe is something that you apply when the right time is there.

Martin: Yeah. And this is a real key cost control at that point because we're able to store less data in hot storage in this idea of something that is a operational analytics platform, something that you need to get real fast questions from.

But then you've got 100% of your data for forensic analysis. All of this stuff might exist in a different pipeline. Maybe you've sent all of your logs to one thing, but then only a subset have gone into hot storage, which allows you that split. So you have actual ways of being able to configure different things.

One of the things, not something that you created, but something I talked about earlier this year, was around generating metrics from this data. So using the raw data to generate some metrics data while keeping the raw data and splitting the raw data between hot and cold storage, essentially generating multiple signals out of that one different data store.

I see this as kind of like the thing that I don't see platform and SRE people doing, which is looking at their pipelines, building their pipelines, maintaining and updating their pipelines to make their backends more efficient.

There's this idea that using a vendored Honeycomb backend removes the need for SREs and platform people to actually do work in the Collector space. And I think actually, no, it makes them way more useful because now their job is to clean the telemetry. It's to implement log dedupe and log drains in order to do it. Side note, I do think that log drain should be called AI log templating because then it would have sold more things. Maybe it would have got some VC money for doing that.

Mike: Yeah, I should do that next time. Next time I create something, AI is definitely going to be somewhere in its name.

Martin: Cool. So that's how we look at logs then. So to summarize then, we can use a log drain mechanism to template logs that aren't already templated. It is probably a good distinction there because a lot of people are using structured logging frameworks already, which will already do the message templating aspect, which is what log drain does.

Log drain, I see, is a tool for SREs and platforms to, let's just say, help the developers who aren't building properly structured logs. But then we've got log dedupe, which is probably useful for both camps. We've got the log dedupe that would work for somebody who's already building message templates in their application.

Maybe they're using the structured log from Go or .NET that actually build structured logs with message templates. Log dedupe is still useful for them, but it's also useful if you've done the log drains. So that's, to me, those two core. Is there anything else that we should consider in that sort of space around how we make logs more useful?

Mike: So for logs, we've touched on the right signal. And something you just mentioned there was the signal to metrics connector. And that is really interesting.

So if your logs are telling you something that could be a metric, and the metric is the right way to describe that, and your system can infer that knowledge really well based on the metric, signal to logs connector allows you to convert across signals in the Collector, which is, I think, probably something that not a lot of people know.

That same information, that same context can move between the different signals. And yeah, the signal to metrics connector has another place in that as well. So if you want to take something that is a very high volume log but it could really have been a metric because someone thought that it would be really useful, then that is another opportunity there to reduce the volume because you could do a signal to metrics connector, generate a metric from it, which has got a much lower interval.

It tells you the right message. It's meant to be a metric, not a log, and then just filter out the logs afterwards.

So then you're converting many, many logs that are very repetitive and tell the same thing many times into a lower cardinality metric.

Martin: Cool. Okay, so that's logs. We've solved logs. Logs have now been solved with the AI log drain processor that everybody's going to buy because it's got AI in it.

So let's talk about some of the other signals. I think metrics is not something that I know you've worked on, but there are ways that we can reduce things inside of that. So I know of two particular ones that you can use.

So there's a way of changing the granularity and the interval of metrics. And there's a new component called the cardinality guardian, which somebody contributed recently to help reduce the cardinality of some of these metrics as well. So that's really the core of metrics, right?

Mike: Yeah, so metrics is probably not something that you want to over-reduce the cardinality. Sorry, the interval is definitely something you want to control, how often something is recorded and then sent through your pipeline.

But the number of things that you are monitoring is probably not something that you really want to over-reduce because that affects your fidelity of what that metric can tell you.

If you aggregate and aggregate, you're probably in for a bad time. And I think that's quite a well-known industry standard.

So the two processors that you mentioned there, the interval processor in the Collector is already available, and what that helps you do is if you've got a very, very fast scraping process or whatever records a metric very, very frequently, like once a second, once every five seconds, the interval process can be used to aggregate that into a larger window. So it will aggregate all the numbers up into a 30-second window or a 60-second window. So out of your telemetry pipeline, you receive the same fidelity, it's the same total value, but at a lower frequency. And that's really useful.

Martin: Yeah, and I suppose my stance with this generally is, especially we're talking about application metrics over infrastructure metrics, if you're wanting something at a per second interval, you should probably be using sample traces and sample logs rather than using your metrics for that. Metrics are about upping that cardinality.

I think everybody really knows about it. Once you look at those two processes, they're really, really simple to set up and get going.

So I'm not going to dwell too much on those because the big reason why I wanted you to come on was to talk about the new processor, which is the dynamic sampling processor, which is a new tail sampler that's being added to the Collector by you as a port of something that we've been doing for quite a while at Honeycomb. So tell us a little bit about the dynamic tail sampler.

Mike: So the Dynamic Tail Sampler is a new component in the Collector we've been building over the last couple of months. And effectively, it is something that Honeycomb has operated and used for a very long time with its own proprietary tool, Refinery.

So what the Dynamic Sampler does is it's a tail-based sampler, so it takes in the full context of a completed trace. It makes decisions based on the whole shape of the trace, exactly what happened across everything that that trace did, all the services it touched, all of the different instrumentation spans that it touched, did it go to a database, did it go to an external service, was there an error somewhere in there. It can have all of that context to allow you to make really educated, full understanding of what happened in this trace, and then make its sampling decision based on it.

So there's three main sampling strategies that are available to people in OpenTelemetry right now. The first one is head sampling. Head sampling is typically done at the SDK level, but can be done in the Collector.

And that applies a flat percentage: of the number of traces that are coming through to me, I will take X percent of them. So you can take 1, 5, 50% of whatever you see, but it does it at the trace level. And it doesn't take into context anything that happened inside that trace.

That's really good for cost control. It's not good for when you find those things that are really important, but don't happen very often in your system and your sampler totally eats them up.

The second option that's available in the Collector is the tail sampling processor.

That is a good step towards being able to make better choices around what's happened. Again, it uses the full context of what has happened in the trace. It happens after the trace is completed, but it doesn't quite tell all the story. It doesn't give you all of the knobs that, I think, something like Refinery and what Honeycomb want to see, which is things like, if you have something that has a high cardinality value on it, which Honeycomb's very well known for wanting you to do. You've got user IDs, or you've got session IDs, or you've got a wallet ID, or something that changes very frequently, and you want to be able to say, well, I want to care about that particular one. I want to care about all of these things, but I want to make sure that I get a good representation of all of the variations of what's happened in my system.

You don't have as many options to apply that. And that's why we built the dynamic sampler. So the dynamic sampler uses Refinery's algorithms. Those algorithms are based on dynamic sampling.

So you tell the sampler, in the shape of my telemetry, these are the attributes that I care about. I care about service names. I care about user IDs. I care about wallet IDs, session ID, whatever that looks like.

So tell me, whenever I see these things, I want to bucket them separately. And then I want to sample them separately. And that's the big thing.

So every time you see a variation across these one or more attributes in your trace, you bucket them. And then you will be guaranteed to get at least one of them every time that that sample happens. If you start to see more of them, so if you see 10 or 100 or 1,000 that match the same key, the same bucket, then it'll start to increase the sampling threshold on them. The threshold then converts into say, oh, I want less of them.

So you can achieve either a flat percentage. So you can say, I want to receive roughly 10% of everything that comes through, but I want to see the full distribution, of course, all of the variations of this key. Or you could do it through throughput to say, I want to have X number of spans come through per minute, per second, or whatever that looks like.

Martin: Yeah, so you've got a lot of different things. I think one of the examples I've been using recently for how what I call trace fingerprinting works is think about a checkout flow where you have, say, 10 different loyalty algorithms and discount algorithms that people might go through.

It's not just, here's a code, and you put in the code, and it adds 10% or 20%, and it's the same algorithm, it's just a different value that you have in there.

If you have one algorithm that works differently if they're a pro customer than if they're a loyal customer versus somebody who's an ultimate or premium customer, and you have different levels, and all of these algorithms work differently, as it stands right now, with the current tail sampling algorithm, you would have to have a section for each of those in your config in order to say, yes, I want to sample at 10% of these, and you still don't get a real good distribution. Whereas with the dynamic tail sampling, you're able to say that this algorithm is something I care about, and I want at least one trace that corresponds to each one of these algorithms by putting that in your fingerprint. I think that's really important because if it's a 1 in 20 chance of somebody hitting this other algorithm, you're 1 in 10 sampling because, well, we've got so much volume we need to sample at 10%, or we've got so much volume that we need to sample 1%, the chances of you hitting that 1 in 20 path through your code is essentially non-existent.

I'm sure somebody can do the math, and it'll probably be in the notes of the podcast, and we'll get a comment on the social saying, well, the actual chance of this is 1 in blah, blah, blah, blah, blah. Yeah, I'm not a maths person, so I couldn't tell you.

What I can tell you is, realistically, there's a chance that you're not going to get it. Whereas if that is important, dynamic sampling allows you to be able to put each one of the different algorithms into a different bucket and make sure you have at least one of each of them. So you're able to see them.

I think that's, to me, what has really changed. It really changed my idea of what tail sampling is. So being able to bring that into the Collector has been really, really interesting.

Mike: Yeah, definitely. We've observed for a long time that with Refinery and then how those algorithms work and being able to do dynamic sampling, they answer a lot more questions and they give you a much higher level of confidence. And I think that's probably one of the biggest things you talk about when we talk about sampling strategies.

How confident are you going to be to answer a question based on that telemetry? Because that's the direct conflict, tension between a sampling strategy being implemented and then how good of an answer can you get out of whatever comes out of that sampling strategy.

The head sampling, good luck if you've got a very low chance of something happening and you've got a high sampling threshold.

You might see something very infrequently, and you'll never see it unless you go hunting for it. With the tail sampler, it is possible, but it is much harder. You've got to know how to fully organize every policy, understand how every variation works, how those different policies are applied, how those sampling strategies are applied. So it's a very high bar to getting it correct, making sure that you've still got a high level of confidence. With the dynamic sampler, it reduces the complexity and increases your confidence because you know that you're going to get variations of everything.

You are going to get every combination possible, at least one of them, and then you know that you've got something to start with, somewhere that you can answer a question confidently, because you know if the sampler has seen it, you are going to get one of them, which means that every variation of your fingerprint or your key attributes or whatever you want to call them on your trace, you're going to get some of them.

So yeah, from my perspective of working with those algorithms for a long time, when somebody sees what the dynamic part of being able to bucket things and then independently change the sampling threshold, the sampling rate on those individual buckets, when you see that light bulb moment, they go, oh, wow, I know what I was missing now. And that's like a big thing for them.

Martin: Cool. So how do people try it out? Is it live yet? Is this something that people can just turn on in their collectors and it'll just work?

Mike: So it is available to use and try. So we have a Honeycomb Collector distribution. It's part of that. It's in that process. So if someone was using the Honeycomb Collector distro or wants to go and find it, they can use it and connect to it and start to use it now.

For the official OpenTelemetry Contrib distribution, it's not available there yet. We expect that to be in the coming weeks. We're just going through some final refinements and making sure everything's exactly as we want, because once it's part of the Collector distro, it's got a bigger audience, and therefore you want it to be a little bit better, a little bit more confident in it.

Martin: I thought people loved breaking changes, though, Mike. That's what developers live for is breaking changes, right?

Mike: I love breaking people's code.

Martin: So right now it's alpha. It's in the Collector repo, so people could build their own collectors around it right now.

Whether you would recommend they do that right now is a completely different question, because you are going to break their code in the next couple of weeks. So really, depending on when this comes out, it may or may not be a solidified thing. So wait for beta or ideally wait for, obviously, release.

So just to finish up then, what's your biggest piece of advice for people who are trying to control the costs inside of their telemetry pipelines right now?

Mike: I think the thing that we started with is probably the thing to just be aware of, which is use the right signal for the right thing. So make sure that you're using traces, metrics, logs, and it's telling you the right story.

And if it's not telling you the right story, why is it not? What are you using it for? And how can you move to something that might be telling you the right thing?

Once you're confident that you're using the right signals, then you've got access to tools to make sure that you're seeing the right amount of things. So a lot of the time you don't want to see everything, but also you don't want to not see something that's really important. So using the controls that are available in an intelligent way, and making sure that you are aware of what you're doing, and making sure that you keep coming back to it.

So something that you mentioned earlier is that having people use it, like build a telemetry pipeline, but it's not something that you can just build and then forget. It's something you come back to. It's something that you should actively be reviewing and understanding. How does my telemetry shape? Are the assumptions and the things that I've got in place then, do they still apply now? Are they going to apply in the future?

I think a lot of people have this almost misconception that instrumentation and a telemetry pipeline is something that you can just build once and forget. And that's not the case.

It's definitely something that you should be reviewing. You should be understanding how it works. Is it doing the right thing? How are those tools evolving? And are there better solutions for what I have now? And can I benefit from them?

Martin: Awesome. Okay. So where can people find you if they want to come and talk about this sort of stuff? Are you on socials? Are you on Slack?

Mike: Yeah, so I'm on LinkedIn. I'm also on the CNCF Slack. So if you go to CNCF Slack, I'll be there, part of OpenTelemetry. Lots of different OpenTelemetry, different channels that are available there, whether that's in Python or Collector or JavaScript or all sorts of different places.

Martin: Awesome. Well, thank you so much for being here. This has been amazing.

Ken: Thank you.