1. Library
  2. Podcasts
  3. O11ycast
  4. Ep. #28, People Problems with Austin Parker of Lightstep
O11ycast
37 MIN

Ep. #28, People Problems with Austin Parker of Lightstep

light mode
about the episode

In episode 28 of o11ycast, Charity and Shelby speak with Austin Parker of Lightstep. They explore topics like rethinking human error, purposeful and intentional training, DevRel expectations, and underrated management tools.

Austin Parker is the Principal Developer Advocate at Lightstep and a maintainer on the OpenTracing and OpenTelemetry projects. He is also the co-author of Distributed Tracing in Practice.

transcript

Austin Parker: If you've ever been the person trying to bring in the new tool or the new technique or the new anything into an established dynamic, then you're going to feel that pain, I think.

What you'll realize pretty quickly is this is all, like all technical problems, ultimately it's a people problem.

A lot of it comes from fear, I think a lot of it comes from -- There's the classic "Who moved my cheese?"

When you go into established processes and established practices and you start moving things around or you want to move things around, that engenders a lot of fear.

Charity Majors: Or exhaustion.

Austin: Yeah, exhaustion and fear. Because everyone's been there before.

I think it's easy for us to get into this habit of, I think it's a thing that I have to self talk myself out of all the time, which is "It's so obvious."

Charity: The bad vendor habit.

Austin: Yeah, "The bad vendor habit." Because we're immersed in this as people that talk to developers for a living, and talk about what our company does and what our product does.

But to us it's so obvious that this is the best thing in the world, and "Obviously you should use it."

But if you are-- If your day to day is go in and you're doing work for whatever qualifies as work--

Charity: Your core mission is not this. Like, this is completely orthogonal.

In the ideal world you would focus 100% of your time and energy on your mission.

Austin: Absolutely.

Charity: All this other stuff is a distraction.

I also feel like there's something interesting here in that if you're asking someone to learn something new or to change their process or tool, what you're offering them has to be an order of magnitude better than what they have in order for you to honestly say, "Yes. I don't care what you're doing, this is worth your time. This is worth your energy. This is where--"

Austin: Right. It has to be 10, 20, 50 times.

Charity: It has to be an order of magnitude better for it to be really worth the investment, and I really think that all these tools and everything--

It is just in the last maybe two years that I think that it's actually approached that 10x.

I think that we can look at the door of support metrics and we can go off on a whole tangent there.

Let's just say that it is better, so you're this engineer and this is where ownership is so key.

People-- I think often engineers feel like they don't have a lot of power, but an engineer's power is wrapped up in your ability to do things .

Your ability to build and do, that gives you ownership, that gives you credibility. If you don't build it, nothing exists.

Austin: No, I think that's true. I think it also, if we look at it from another angle, it's also a statement about there's another type of building and that's the relationship building.

That's the professional convincing people about your position, and I think that's actually one of the better things that the DevRel community--

Or, why you should listen to us is not necessarily because we are the builders and doers.

We do build things and we do things, but the real value is "Look. We are interfaces, and we are here to connect you to those stories and give you those convincing anecdotes and give you the things you need to take back to inspire hope."

Because I think that's how you combat it, you combat fear with hope.

Charity: Yeah, hopefully. This guy said his team, the platform team, they're on it.

They're investing in this stuff, they're seeing the benefits, they're sold and they're having a hard time making that leap.

They've been pushing all of the articles and all the stuff that we say over to the other teams around them and the other teams just aren't nibbling.

I think there's a lot of context here that we don't know.

"How senior is this person? How long have they been there? How much credibility do they have? What are they known for being an expert in? How good of a communicator are they? What's the relationship with the other teams, the managers? How much tool cross pollination?"

There's so much here that we don't know.

But like, this question really starts to zero in on just how critical the senior engineer, however you define that, if they don't want to adopt this stuff it's never going to get traction.

Shelby Spees: This questions aligns with my own personal experience.

I'm not that far removed from being the person trying to push exactly these changes and showing the senior people in my team, and they're like "It's all hype."

They don't trust vendors, things like that.

In retrospect I realize the way I was communicating this stuff probably needed work, but also I didn't have the bandwidth and I didn't have the time to sit down and do the work to actually show them the difference.

I think that it's really common where people are too spread thin to even just start making the changes that will make them less spread thin.

Where we just don't have that slack in our teams and we're just piling tasks and piling work on top of ourselves.

Charity: If you're fairly new to the team, you are just beginning to establish your credibility and you're very--

You don't want to go out on the limb and you're very conscious of the "You're just starry-eyed. You're too young to know better. You'll be scarred," that attitude is super real.

In some ways it's super legit and in some ways it's not, so I think that there are a few strategies.

If it's working for you, if the engineering superpower is doing things and if you have the cycles, in theory the whole premise of this stuff is that if you bring it in it makes your life better.

It gives you time back.

It is an investment, not a cost. So if your team has brought something in and made it work, I think that you're going to have so much more credibility than any of us out here talking on the perimeter. We can help inform you about how to communicate it, but maybe doing an internal lunch and learn, or-- It's weird how infrequently we really track what we ourselves are doing.

Like, how often are we getting paged? How long does it take to recover? Are these trending good or bad?

I think that there's a lot that we can do as engineers, instead of just throwing up our hands and going "They're not doing what I told them to."

We can actually show the evidence that it is making an impact, the ways that it's helping, the ways that it has helped give us so much more time back.

This is why we look for champions, and any time that we're looking at an account we look for the champions, because we know that they have so much more credibility than we do as vendors.

Austin: I think another way to-- It's apocryphal even, but the story of someone that built a bot to track how much time people are in meetings.

You keep the little running counter, and then the joke is always "OK. Now you attach that to everyone's salary, and you see exactly how much this meeting cost."

You can do a similar thing by looking at on call, looking at incident response, looking at "OK, how many people are we pulling in to each incident? How much time is this taking away from--?"

Charity: "How costly is this for us?"

Austin: Literally, "What does this cost?"

Most of the time you'll find, I think if you do the math, any amount of adding observability and trying to modernize how you think about this will save you money.

Charity: Quickly, or immediately.

Austin: Very quickly. I did this at-- It wasn't with observability it was with testing at an older job where I came in and it took several years to get to the point where I had the credibility to do this.

But to what you said earlier, we had this very old legacy test system that was just frustrating enough to cause a lot of problems, but not frustrating enough to get--

Charity: That netherworld region.

Austin: Right. It was never quite-- It was skipping off the surface like a stone.

Every time it got down, it would be right back up and it's not enough of a problem to address it.

At that point, it was like "OK. This is someone else's responsibility, but I can write code.

I can go in and I can do the proof of concept and show it and be like, "Look. We can do things better."

That's maybe what I would say, "Start small."

Maybe you're the platform team or maybe it's not your responsibility or you don't have ownership over it, but if you have the code and you can go and you can see "This is a Java service, I can just throw open telemetry in there a nd boom, it works."

Start proving that out in small, isolated cases. Then that's the best evidence. Evidence that it can be done is the best evidence of all.

Charity: It really is. Giving them a template almost that they can use as a springboard, because that first one is always the hardest.

Another thing that I feel like often engineers feel like influence and authority just sort of happens magically, but that shit right there, that is what creates influence.

That is what creates internal authority, showing them-- Shaming them almost and showing them how they can do their own jobs better.

That is a truth that accrues over time and earns you credibility. This seems feels like a really good time for you to finally introduce yourself.

Austin: Yeah, sure. I'm Austin Parker, principal developer advocate at Lightstep. I'm a friend of the pod, I don't know.

Charity: We got deep in that real quick.

Austin: We did. I was going to say, "This must be a really heavily edited show, like they do the interview segments and then--" I'm much more lackadaisical.

I also helped write a book on distributed tracing, and I guess I'm most famous now for running a DevOps conference in Animal Crossing.

Just goes to show you that your brand doesn't quite turn into what you think it's going to be.

Charity: You know what? I hate unicorns.

Austin: But you have so many pictures with them on them.

Charity: People just started attributing them to me, and I love aggressively feminizing male-dominated spaces so I just took it and went with it. But I've never been a unicorn person, and now I am. Personal brands are funny that way.

Austin: Now you're the unicorn person and I'm the Animal Crossing person.

Shelby: I really like what you both-- Not just about the branding thing, but just the way that influence doesn't happen how you expect.

I remember early in my career I would join a project, I would join a team and I would be like, "All of this is wrong. We should know better."

I didn't have the credibility, people believed in me and I was very lucky that I had a lot of people going to bat for me, but I didn't have the credibility or the experience to be able to show them how to go about making things better.

I realized I needed to be a little bit more patient about that and get some more experience under my belt.

So I'm wondering, how do you balance gaining that experience and learning how to speak the language and spending time on showing the differences without ending up just getting used to the way things are and becoming the cynical senior who doesn't want to change anything?

Charity: That is such a good question.

Austin: That is a good question. I think that's actually one of the things that makes tech the absolute hellscape that it is, because--

Charity: I don't think it is.

Austin: It is in a lot of ways, though. The societal aspects.

Charity: Relative to the world at large I think there's a lot to be said for tech.

Austin: Absolutely.

Charity: Now, you could just argue that the world is shit.

Austin: Yeah, I will agree with you there. Categorically, technology is generally good and has pulled a lot of people--

Charity: It's young and it's fast-moving and it has the ability to correct its mistakes.

Austin: More so than a lot of systems.

Charity: If I look at banking or finance or the entertainment industry, I recoil in horror. Like, I would so much rather be in tech.

Austin: Absolutely. But I think what hurts it a lot though, is there is this attitude almost of "I paid my dues. I did the scut work for a long time, and now I get to call the shots."

For the most part, it's not the case e verywhere. I don't want to say it's universal, but--

Charity: I've been fortunate because I have not really been in environments like that, for sure.

Austin: I think it depends on the organization, too.

Charity: Surely it does. The future is here, it's just very unevenly distributed.

Austin: Yeah. Maybe we'll have eventual consistency and justice in tech organizations.

Charity: Absolutely. It's a glorious future.

Austin: But I don't know, you see posts online, I saw there was a post going around from HN the other day where it's some person bemoaning that they're an L4 at Amazon.

Charity: I saw that. That was hilarious.

Austin: It's like, "I'm so depressed." You can go look it up, I don't want to reconstruct that anonymous post on the internet .

Charity: Totally. To Shelby's point though, I think that-- Here's the thing.

Nobody likes somebody who comes barging in the front door and tells you everything is wrong.

You just can't come in like that, because there's always so much context and so much history, and it just is going to rub everyone the wrong way if you don't even make an attempt to understand why--

Because they have reasons for doing these completely unreasonable things that they've done.

Austin: Absolutely.

Charity: They probably know that they're unreasonable, so this is just a mistake that I see people making all over the place.

I think that the right thing to do is to come in humble and come in curious.

Put your curious hat on, like "Why is it like this?" Really try to understand and try to be useful.

Another thing that I see people-- Based on just failing at their jobs, is before they do the shit that they were hired to do they're already looking around and they're taking shit off of other people's plates and critiquing it.

That is just going to annoy fucking everyone, so number one, focus on making sure that your own queue is dealt with and that you don't have to be a superhero.

Level up, learn your shit, pull your weight is what I'd say.

Pull your weight and do what your team is counting on you to do before you start opining on other people's work , come in curious and genuinely trying to learn instead of starting with the "This is terrible."

Start with a, "Why is this terrible?" It's a subtlety that makes all the difference.

Austin: You have to keep that up, too. That's the challenging part.

Charity: Which is why it's wonderful to have more junior people on the team.

It's wonderful to have a range of distribution skill sets so that you never get locked into the television.

I think there is this really interesting dialogue that can happen between the senior folks who've been there a while and you're immersed in it, but new people come in and they have fresh eyes and they remind you of it.

Austin:

I think the most important thing is to always have a process that's able to learn from itself. I think when I talk about observability, I like to talk a lot about the important part of this is not necessarily the tools or the technology, it's the cultural phenomena of looking at data and then being able to explain things.

Charity: Sell two technical systems.

Austin: Yes, sell two technical systems and having a feedback loop.

New people are the lifeblood, they are the input and they are the signal to the feedback loop of "How are things working?"

I think people that are successful leaders and people that are successful growing in an organization or growing at scale are people that are always willing to learn and are always happy to learn, and want to.

Charity: The more senior you get in your role in your org and your position, the more you have to be conscious of performing this.

Not just being it, but actively and openly performing it. Asking the questions.

This is something that I started to realize that I was doing really badly a couple of years ago, which was just that I knew that I was open and curious and stuff, but other people didn't know that was in my head.

I wasn't doing a good job of asking the questions, thanking people for coming to me.

I was happy when people brought me feedback, but I didn't tell them thank you.

I didn't act happy, so they didn't know that I was happy.

Leadership is in so many ways almost less about what's in your head, and more about what you're performing so other people can see, and doing that consistently.

Shelby: How people perceive you.

Charity: How people perceive you, yes.

What you say and what they hear are not the same thing, and you have to be constantly conscious of that.

Austin: Ironically enough, the more you lead and the more you do all this stuff, the harder it becomes because you have to fight harder against people's deference to authority.

Charity: They've just made up their minds about you.

They have a narrative about you in their heads, they have decided what it is that you're thinking and feeling and saying before you open your mouth. It can be very hard to dislodge that or rattle it or change their perception of the thing.

Austin: We could all be better at this.

Charity: God, yes.

Austin: Everyone can always grow and learn on this axis.

Charity: A pro tip that I got from a leader who I really respected a long time ago, I asked him, "How did you get better at management stuff?"

He paused and he said, "I took an improv class. It was better than every book that I've ever read on leadership."

It's all about that active listening and being very fully present and never saying no, always say "Yes, and."

I've been meaning to do that ever since, and it's been ten years now, but you know.

Austin: It's like how I recommend the book from Sydney Decker, The Field Guide to Understanding Human Error.

It's about systems and it's about why systems fail, and that's what I recommend to everybody that wants to understand systems thinking and how to handle incidents and how to build incident response.

Charity: Shelby, I'm kind of curious about your-- You've been going through this process of a new job, a new role, a lot of actively leveling up and stuff.

This process of building influence and the shifts in the perception of you, do you feel like you've learned anything interesting?

Shelby: Yeah. So, I've been teaching for such a long time and I've always been a know it all. I would speak authoritatively about things that I wasn't necessarily-- What's the word--?

Charity: Authoritative.

Shelby: I didn't have authority about, to the point where less than a year into my first software job I was designing and running a workshop for the users of the Python tool I was working on, who are all rocket scientists.

I was teaching rocket scientists as a baby developer, and what's happening and it makes sense that this is what happened, is as I learn more and gain more experience I'm realizing how little I know.

I've circled back around, and Charity, you've given me advice about this before where my lack of experience is itself a thing I can share, it's a thing I can perform and it's a thing I can use to help other people.

That's been the theme of my career, because I love teaching.

I taught English and I love helping people, and I especially love helping people do their jobs better and enjoy their jobs more.

As I figure things out, I'm like -- There's a part of me that feels like I should have already known this, and there's a part of me that's like "If I didn't know it, then other people probably didn't know it. I can help them." I can help them by clarifying and connecting the dots for them, so I've been focusing on that.

Because otherwise I get nervous and I'm like, "I haven't been living in server rooms for the last two decades, so I'm not qualified for anything."

Erasing all of those thoughts because what I am qualified to do is help people where they are at and get to the next step in their observability journey or their DevOps journey, or whatever.

I think the perspective I have that can benefit the tech community and the DevOps community is something we talk about a lot, where the future of DevOps is observability.

It's these modern release engineering practices and it's about freeing up our teams, and Charity, I really appreciated your post about the future of SRE Jobs and stuff.

I've been thinking about how the future of DevOps involves a lot of training and growing people.

Because in tech we're constantly learning, we don't get to stop learning.

There's always going to be new technologies and new concepts and new approaches and new ways to integrate all of that, and every time you go on a new project or start a new job it's going to be different.

We've been so bad at training people, we basically don't train people anymore.

Charity: I don't understand how training would even happen.

Nobody knows the answer, nobody can train you in the answer because nobody fucking knows the answer.

I feel like everything that I've learned I've pretty much been self taught, and this is where I think systems is very different from hardcore computer science.

Every system is different. It's a fingerprint. It is unique, it is complex, you can only apply past lessons from other systems to a certain extent, and then it takes a great deal of wisdom to know how far that goes.

I feel like training gets dropped around as a panacea for "We should know how to do this," but we don't know this.

The beauty of tech is that we learn to feel comfortable in that zone of not knowing and partial knowing, and needing to move forward anyway.

I guess I'm just skeptical about anything that involves training as a solution, because I just don't believe in it.

Shelby: I use the term "Training," but I think it's more just accepting that people aren't going to have the answer.

People don't have specific experience in the exact thing that you're looking for.

I'm arguing that the tendency to only look for senior engineers is hurting our future in running software, and you yourself have said if your team is healthy and you hire somebody, they should be productive within six months.

Charity: Sure.

Shelby: So that's basically what I'm arguing, is like--

Charity: I absolutely think that we should open the barn doors wider and take more risks on people and trust that people can learn things, which is why it breaks my heart every time that I see someone call themselves a Ruby engineer or a Java engineer or something, I'm just like, "You are not your framework. It doesn't fucking matter."

Austin: Yeah, it doesn't really matter. Six months is enough time to teach anyone a new programming language, I would say.

At least to the point where they can be productive.

Because the dirty secret of programming is that a solid 50-60% is Googling things and the other 50-60% is searching JIRA and confluence.

I tend to agree, it's not a training problem.

If anything the desire for training comes from this broken model that I've seen a lot. I don't know if it's just--

Charity: You can teach somewhere to pointy clicky in the tool, but you can't teach that and think about something.

You can just give them the opportunity to figure it out and answer their questions when they have them.

Austin: Exactly. The analogy I've been using is "I think we do a disservice to programming as a discipline by thinking of it as engineering."

Stepping aside from engineering as a protected class of our field or whatever, I think of a team of people developing software like a kitchen. It's like a chef and a bunch of sous chefs, or it's a bunch of chefs.

The sort of traits that are very handy to have as a chef-- Like, yes.

You know a ton of theory and you know a lot of stuff, and a lot of it could be completely self taught, but it's ingenuity.

It's creativity, it's collaboration, it's "How do we manage sudden shifts in demand? How do you deal with all these rushes? The menu changes over time."

Charity: Everything changes, and I think the appetite for risk goes both ways.

There's the managers and the companies who don't want to take risks on people because they don't know the exact framework or whatever, but it also cracks me up whenever I get it.

I talk to someone who's looking for a job who's just like "I totally want to work at a startup. But what I want is to have my jobs planned like 3-6 months in advance, to work from 9- 5 every day and not have anything change in the roadmap." And I'm just like--

Austin: "That's not a startup."

Charity: That's not a startup, right. To circle back to that first question about that first engineer though, I gave this poor dude just a 30 second answer.

At the end of it, I was like "You can try these things. But also, if you're not getting buy-in from your tech leads and you've given it a shot, move on and get a new job. There are people out there who are hungry for people like you who are looking for ways to help bring them into the next generation of computing and the next level of operating humane services. Don't hide your light under a bushel. Don't waste it on people who don't want to be helped."

Austin: Yeah, you always have to be willing and able to walk away and go on to the next thing.

Charity: There is a divide right now, and you see this-- I keep going back to the door report.

But the DORA metrics 2016-2017, they're the elite category which was the top 2%, and then 2018 was the last year they published it of course.

Do you see it just start to shoot up? It grows from 7% to 22% and it's achieving escape velocity it's going up so steeply.

Austin: You would have to imagine that if they were still reporting it, it would be 80-90.

Charity: Absolutely. It's a lagging indicator. These are the teams who have seen something different is going on.

Something like an order of magnitude better is going on. And yeah, it's a little bit disruptive.

It means if you aren't trying new things in tech, you're losing ground.

There's no such thing as standing still. But everyone else is slowly losing ground, but there are teams out there who have identified that shit can be radically better.

I feel like that thing that we have to fight the most is just this numbness, this feeling of--

Austin: Complacency.

Charity: Or just being ground down.

Just cynicism, like "This is just how it is to work with computers. It's just going to suck. We've just got to bite it, eat the midnight pages and we just have to spend half of our every day grinding through shit that has no core business value."

We're just floundering and trying to figure out how to orient ourselves and whether that was the right bug and how to reproduce it, and now we're working on the wrong thing because we can't see what the fuck we're doing.

Our feedback loops are so long that we've lost all the context by the time we realize there's a problem.

This is well charred ground at this point, shortening those feedback loops and making them tight, introducing software ownership, this stuff is stuff that we are figuring out how to do it and not everyone's interested.

Austin: Yeah. A lot of that you combat through, like we said before, just doing it.

A lot of that you combat through the natural selection of these things.

There is a lot of flexibility in all of this, and I think what you're seeing more and more is that there are no golden geese left.

Whatever you are doing today, someone will come and they will look it up.

Charity: Yes, they will outperform you.

Austin: You will get outperformed, you'll get out-innovated, you will get left in the dust.

Charity: Shipping is the heartbeat of your business, as the Intercom folks say. It is the heartbeat.

And people who are like, "Yeah. My heart can beat once every couple of months,"like, that's not great.

Austin: What's wild is I think this is what I'm seeing, I think it's starting this realization that isn't just a software only concern.

That's not just a "You need to be deploying on Fridays and every other day, because deploying doesn't actually mean anything, it's just how you work."

It's a way of doing business and it's a way of talking to your customers and it's a way of actually doing things in the open, being afraid to fail, not being quite so polished and groomed and whatever else, it's going to be a transformative change.

Charity: Doing blameless postmortems across your organization, and for business folks too.

Austin: Also democratizing data .

Charity: Fuck yeah, democratizing production.

Austin: Yes. All of the stuff I harp on so much, it's all the stuff that right now you were probably collecting the same data fifteen different times for slightly different purposes.

You can just put all that shit in there at the point of writing the code, you can just have observability, it's not that hard and then you're just presenting different views to people.

Charity: That's the thing. It's not only it's not that hard, it's so much easier.

Austin: And it's 20 times easier than what you're doing.

Charity: It's so fucking easy.

Austin: You're duplicating tools, you're improving performance--

Charity: You're saving money.

Austin: You're making sure that the people using your site on a browser are a lot happier, because they don't have 20 different analytic libraries. They have one.

Charity: This is one of the things that I love so much about LaunchDarkly folks, is that they aren't just engineers.

They're like, "No. Everybody does a better job when they can get close to production safely and have control over sales engineers in the field, flipping the thing in the middle of a sales demo." Turn it on for their customer.

Austin: Flip the switch.

Charity: You don't have to be one of the poor people writing code to have an interest in being able to look at and examine or modify production, it makes you better at your job.

Austin: It's a requirement, right? We're well past the time of this being some newfangled experiment, I think the evidence is borne out--

Charity: It's not an argument.

Austin: That there's no argument here.

There's just like, "Are you doing it or are you not doing it? And if you're not doing it, then you've got a target painted on your back."

Charity: I think that the next big hump is CV. I think that CI, the industry is like "Yeah, we've got it."

But in CV, the idea that as soon as you merge your code back to main that it automatically goes out within a few minutes.

It doesn't have to be visible to everyone, we're not stupid, but this is where the whole constellation of tools comes in.

You use your feature flags that you get, but if your code is not in production it may as well not exist.

It is decaying. It is rotting, it is stale. You should get that shit out of there fast and you should have this ticking clock in the back of your head, and this muscle memory of "Don't even look at it."

Austin: Yeah, I agree.

Charity: Anyway, I have one more question that I wanted to run by you which is the book end parallel to the one about the engineer, which was one from a newish manager.

Let's say he's like, "I drank the Kool-Aid, like I'm all in and I really want my team to be engaged in some of these newer ways. I've dropped the articles in the Slack or whatever, but they keep telling me they're under water. They don't have time to work."

And they're the engineers, like if I was still an engineer I feel like I would know what to do to start doing it myself.

Get my team on board, but as a manager it feels like pushing on spaghetti, like you're pushing on a wet noodle.

What do I do? As a newish manager I'm at a loss.

Austin: They don't let me manage people. No, I don't actually.

That's a good question. I think that when you're a manager, even a new manager, a lot of what you do really ends up in incentives.

That's your lever, is incentivizing or disincentivizing behavior.

The dumb naive answer is you make it fun, people enjoy things that are enjoyable, so incentivize people doing the right things.

The other incentive you have as a manager, and I was actually going to say this for the platform question earlier, checklist and process.

Process is a remarkably effective tool to wield in virtue or vice, and in this case if you think of observability as a virtuous thing, which I do, then use the ability you have to set process to gradually introduce these concepts. Things like production readiness, checklists, observability checklists. If you have a run book, adding things into run books and making it part of retrospectives and incident review and trying to block out time.

As a manager, you have a pretty big amount of control over people's time in broad strokes, so you can start to say "OK. We're going to have a couple hours, or a Friday, or a hackathon or something."

I think the one thing I would say not to do is don't tell people to do it when it's not work time.

Charity: Jesus Christ. Of course.

Austin: Like, don't encourage people to go do stuff after hours.

Charity: I like the way you're framing that. I think that one under acknowledged tool that managers have is the job ladder.

The levels, if they say "No one gets promoted to senior engineer unless you are writing and shipping maintainable services, unless your on call metrics are sustainable. Unless your mentoring--"

Whatever. You can say, "No one gets to be staff engineer unless you are keeping up on the trends in industry that will help us keep ahead of the waves of--"

I think that stuff is slow moving and it doesn't feel as satisfactory, especially with engineers, but it is deeply powerful and it can move mountains given time and the right coaching.

You can also, something that I've used before, is the rule that whatever you praise you will get more of.

Austin: I was going to say, money. Not necessarily salary, but money for tools.

Charity: You can pay for tools. You can sign the check, for sure.

If you can take aside one of your more impressionable engineers and convince him or her that this is the coolest shit ever, maybe she'd be willing to do a hack week on it and you can get excited about stuff.

Then you praise that shit to high heaven and you're like, "Look how awesome this is. This is great."

Austin: Modeling.

Charity: Yeah, you just set someone up to succeed at it and then you point to how successful it was, and you keep-- Here's a mistake that I see managers do .

They praise it once, or they point it out once, and never again.

If week after week your teams are in a better place, you have to keep bringing it up and keep pointing it out and keep going "Remember how much this used to suck? Look how much better this is."

Austin: And not just with your team, not just this year, but also to your other managers.

Charity: You evangelize that shit.

Austin: Both horizontally and vertically, so make sure that all the other managers know.

Make sure that your boss knows and that their boss knows.

Charity: Excitement is so contagious and so many managers don't use it.

Austin: If you are excited, they'll be excited.

Shelby: Yeah, no. I really like the idea of just taking someone aside and adjusting priorities for them, because that's the real power you have if you're a manager, as you can set priorities.

Maybe not top level, but you're running a team and so--

Charity: You can get them excited and clear their plate.

Shelby: So you're in a much better position as a manager than as the mid-level engineer who's excited about a thing, and they're too swamped with work to even make time to try things out.

Charity: This was the other piece of advice that I had for that mid-level engineer though was "Do you have management support?"

Because managers should be all over this shit. The thing is, you can't pitch it to them in engineering terms.

But if you're a mid-level engineer and you take the stuff to the engineers and you put it to them in terms of headcount and how much more time you'll have for your time to recovery. You have to speak manager language, which comes down to money eventually, but God the case is so powerful.

It can be made and you can use all these materials that you don't use that you're dropping the slack to show "Where are the inefficiencies in your system? Where are the weeks of engineering time that are going unpredictably?"

Which is the worst kind of planning.

Just all of a sudden your roadmap is derailed because you have to do this thing because nobody could understand your systems for the last week, get some managers convinced and they can put their tools to work and maybe make some progress.

Austin: Be aware of what you're already doing.

One thing I've seen people up is-- Especially at a bigger, more complicated organization, duplicating or triplicating turf wars can be a thing.

But I think there's a lot of really strong-- That can both be good and bad. Because sometimes--

Charity: You've got to go looking for allies.

Austin: A lot of times those allies can be unexpected. The people that might be--

Like, if you have some big internal monitoring things that someone built and dropped on everyone and then walked off, go find the person that's responsible for it now because there's a pretty good chance that they don't want to deal with running Nagios Dashboards or janitoring Grafana.

Charity: Never assume that someone's not an ally.

Always give them the chance to surprise you, because some of the most surprising allies will come out of that.

But you've got to give them a chance and not write them off.

Austin: That's true.

Charity: Austin, this has been an absolute delight.

Austin: It has been.

Shelby: Thanks so much for joining us.

Austin: Thank you for having me on.