In episode 33 of o11ycast, Charity and Shelby are joined by Katy Farmer of CircleCI. They discuss learned helplessness, understanding complicated systems through direct experience, and championing devs to fail gracefully.
About the Guests
Katy Farmer: I definitely got here through a series of accidents wherein I learned that I was really good at learning.
So I learned to code maybe only five years ago and the resources were really hard to understand.
And like probably most people who've learned to code in the past five, maybe even 10 years know that the resources can be pretty academic.
I remember reading a book and the second chapter was on recursion.
And I was like, "I don't know what this is. And I don't know how I got here."
So once I learned that no one really remembered what being a true beginner was like when it came to coding, I realized that I had an opportunity to teach other people where the beginning was.
So I've kind of got into DevRel in order to teach people that it's okay not to know.
And it's okay to really be at zero.
And like, what does it look like to know absolutely nothing about DevOps well like just look at me, we'll do it together.
Charity Majors: You're like a talk show host. The talk show host of software.
Katy: Yeah, I want to be the Ricki Lake.
Charity: Let us adventure to this far away land where the natives speak in ones and zeros.
Shelby Spees: I mean, that aligns a lot with my experience too is I remember just sitting down to do tutorials or reading textbooks and being like, "This isn't starting from zero, this is starting from some place beyond where I'm at."
And so that's been a lot of my career too, is just trying to bridge that gap from, I've never thought about where software comes from.
Charity: It's definitely what you did here. How you showed up--
Like you showed up and Liz being Liz, it's DevRel for a particular segment of people, and then Shelby comes up and is like, "Oh, what if we don't know everything from Google?"
And brought us along that.
Shelby: Yeah, I tweeted the other day that like, when I read the SRE book, I didn't know what a load balancer was.
Charity: Oh honey! Oh, I forgot, I thought babies knew what load balancers were.
Katy: It took me ages.
Charity: I thought everybody knew that.
Katy: It took me so long to figure out what a load balancer was.
Charity: What! Oh.
Shelby: And it's like, it sounds self-explanatory except if you've never thought about like when you type google.com into your browser what happens and the whole like going back and forth and you've never--
Charity: There is one server, one golden server out there in the cloud. And it answers all of your requests.
Katy: I'm a big fan of that goal so far.
Shelby: You know it's like when an alien comes to earth and you have to explain like what everything is and that's kind of how I felt learning all this stuff.
But before I tell my whole life story-- Katy, why don't you go ahead and introduce yourself?
Katy: Yes, Hello, I am Katy Farmer, pronouns she her.
I am the technical community manager at CircleCI, and I'm just very excited to be here talking to you all today.
Charity: You also have an adorable bow in your hair. I wish everyone could see this.
Katy: Thank you. It's the only way I can deal with not being at conferences is just to jazz it up a little bit.
Shelby: But yeah Charity, like what you were saying.
Most people don't know what observability is. Most developers, most infrastructure engineers most DevOps engineers have never heard of observability, don't know what it is and so we need to meet them where they're at.
And that's been, Honeycomb's challenge since you first started talking about observability.
And I think there's that whole segment in the CI/CD space as well.
It's just like I, as a developer, I write my code and throw it over the wall and you know the DevOps movement's been around for like 15 years now, t here's still people who are scared of their BuildConfig.
And so just making that more approachable is a great effort.
Charity: Is that true? Are people really afraid of their BuildConfigs? Why?
Katy: Because they're YAML.
Charity: YAML fucking sucks. That's just different.
Katy: I'm a person who's pretty like neutral when it comes to all like programming languages and such, but--
Charity: All languages but YAML.
YAML is not a language, it's like the inbred dog of your redheaded stepchild.
Katy: Yeah, I think that a lot of times people the first config they come across, is like a highly customized five-year-old config.
And then they say, "Oh, I can't do that. This tool is too hard to use."
And obviously not only has the product changed on the amount of time and like no one has done an overhaul of this config in a certain amount of time.
I think it has more to do with like not maintaining your config file and will not auditing it and like keeping it up to date.
Shelby: It's, I mean, surely you say this all the time that like your building deploy process is like the least loved code in any organization.
And so it grows a lot of barnacles and it becomes very hard to read it and debug it.
Charity: And people forget that config is code for starters.
And one thing that I learned very early on in my career at Linden Lab was whenever you're editing config files, leave fucking detailed comments about why you put this variable and what it means, because you will forget.
And then it'll just be like six months later, you'll be looking at a config value and you're like, "I wrote that. What did I mean, when I wrote that?"
Katy: I still think that about like college essays I wrote that were about the Great Gatsby, which is not a subtle book.
And I read the paper and I'm like, "what am I talking about?"
Like having an English degree means you're like really good at making things up so--
Charity: Symbiosis man, just like reread it. You're present context.
Shelby: I wonder if like that.
I mean, my background is in linguistics and like social science and having to just explain everything I did as I went along.
And then I learned eventually that like, I'm not good at remembering, like why I did a thing.
And so I went into programming, just documenting and commenting and writing like really strong commit messages, like for my own benefit.
Because like, I'm not even talking about me six months from now, but it's like me five minutes from now.
I have no idea what I was doing.
Charity: Most of my Git commits from early heading home.
We're just like, fuck fuck fuck. Fuck Terraform. Nope, fuck this harder. It's like not detailed.
Katy: At CircleCI, when people use Circle, like the most common Git message I see is "fixing Circle" which just makes me happy.
Shelby: And you have to debug your build too. I mean, I've been there.
And like, what I experienced on teams before is this sort of learned helplessness among Devs where traditionally they're not given access to the updates, the build config, or any of that.
And so they can't debug it.
And then even if they do get access they've never been allowed to before it becomes this like scary thing.
And it's like no, it's not that different from regular code.
And especially with tools like Circle, you just edit it and run it, a nd it's an external Dev environment.
Charity: Almost nobody configures your shit from first principles.
It's all cargo culting, it's chains of cargo cults. And literally I'm sitting here thinking have I?
No, I have never like begun configuring my SQL by reading the main page and like selecting the correct variables.
Fuck no, you take the defaults, if they don't work, you Google something, you copy paste it, you try again, you swear at it, you kick it and rinse and repeat until it's somehow completes like that's--
Katy: That's the scientific method.
Charity: They're not designed, they evolve.
Katy: Depending on where people are like as developers like what they work on, sometimes the people who are in charge of the config can be a little precious with it.
Like the people who wrote it are like "Don't change anything."
Charity: Oh yeah.
Katy: But I think it's because they feel that it's fragile or that their understanding of it is fragile.
Charity: Well, yeah it's just like we used to with the production systems.
Like, "I got into work, nobody touch it. For the next year, don't touch it. Like it's working now, but we don't know how we got it to this place, and we don't know what will happen if it breaks again."
Shelby: I have a confession to make that I've totally been that person.
Just like nobody touch anything, we're code frozen for the rest of Q4.
Don't change anything, and like even at the time--
I mean, I saw the commits on me getting backed up, and then I felt the pain of those giant deploys and stuff, but I was still I was, "Oh, don't change anything."
And I had to unlearn that. And so it's like, I know as possible, I've done it.
Charity: It's deeply ingrained in us to slow down when we feel scared.
When we feel nervous, we feel anxious we slow down and try to get control.
It's encoded into our DNA like we balance on two wobbly little feet.
We're just, "Ah, falling, slow down."
But the problem is that with the software, that's the wrong impulse.
Because with software, I think we have to have a different metaphor where it's like a heartbeat.
You need your heartbeat to be pulsing, like regularly to just clean the system, get all the oxygen pumping, clean everything out.
You do not want your heart to just like save it all up but go boom boom.
Shelby: I thought of the most ridiculous metaphor the other day where it's like your business is a car and it's driving and a code freeze is putting one of those wheel locks on your car.
Katy: That thing of like-- That theft prevention thing.
Shelby: Yeah. And then your Devs are sitting in the backseat.
And your car is still driving. You just can't change direction.
Like people think they're putting on the brakes but they're not, your business still is running, your services are still running.
And if you can't make changes, then your hands are tied.
Katy: I actually just read something from one of my colleagues this morning that was a really, just a nice sentence that I liked.
And they said, "Big changes require small changes."
Which I just liked does this idea of like I think, especially because I went to a bootcamp and they were pretty Ruby focused, they definitely kind of hammered home agile, like do things in these small pieces.
It's really hard to do it turns out.
Because like you said, it kind of feels unnatural to us because I'm afraid to do everything.
I'm like, when you're new, you're like, "Ooh what will break? What will happen?"
And I think that there's some power in maybe making your developers feel safe. If they mess up, they mess up small too.
Charity: I really enjoy having a culture where you celebrate people, t he first time they get done in production. It's like your first blood, man.
Like you're not a real warrior until you brought down prod.
Shelby: I think there's a confidence gap there that a lot of people have.
And I think that's one of those things we talked about, what makes a senior engineer, and I think that's one of those unfortunate hazing rituals almost that you need to learn that like yes, you're going to break things.
Or even as a beginner it's like you learn that the red error message.
It doesn't mean you're a failure. It means you have an opportunity to fix it.
Charity: There's a confidence gap. Sometimes there could also be an overconfidence gap.
Shelby: Oh sure.
Charity: There's both sides of the line. And that's why this is interesting and hard.
Katy: Yeah, to me there's a little bit of a gap too with like regular like "I'm learning to code."
And then once you get into DevOps, it feels like--
Charity: By DevOps, I think you mean reality.
Once you get too deep into production code, that's different.
Katy: Yes. When you realize that there's not just-- Like going to make my own metaphor here which is, I have an English degree.
And most of that was just reading books and then writing down what they were about.
And that's like learning to code in a bootcamp. Like I read it and here I go, I wrote some of it.
Charity: Isn't it pretty, isn't it great? It compiles, it's lovely.
But what are the real world impacts of that code?
And then once it got into all this tooling, like honestly that's the thing and like it's missing from bootcamp curriculum 'cause they don't have time to teach it.
And then you're like, "Okay well, I wrote this really nice Ruby class" and they're like, "Cool, but you're not in charge of that. You're in charge of deploying or whatever it is."
And I think that we don't have the same learning resources.
Charity: Something like kids who go through college d on't learn fuck all about this either.
Katy: Yeah no, they don't either.
Charity: Nobody does.
Katy: Correct. Nobody teaches the tooling and that's--
Charity: I almost don't know that it can be taught.
It's constantly changing, there is no canonical like anointed set of tools.
It's in a constant state of flux.
Production systems can't really be taught it's the real world, it's like, you kind of just got to throw people in the deep end and offer them you're a pole.
So when they start to drown, it's the only way I know to teach people.
I have not yet found a way to learn about complicated systems without being deep in a complicated system.
Charity: You kind of, it has to be nerve wracking. 'Cause it has to be real.
There is a simulator for this shit, maybe someday they will be.
I don't know, I think that, living on the knife's edge is just kind of part of the charm of production systems.
Knowing that what you do has an impact.
I find it very hard to motivate myself to writing code or just like write something.
Like why? Like why would I do that? Like what is the problem I'm trying to solve?
I'm a little too far over on that side of the scale where I can't, I wish I could write code for fun, I can't, I need a problem to solve.
Katy: Yeah. I feel like the most success I've had like as a developer advocate, or even now as a community manager is like finding out that what people don't understand about a product is usually an underlying philosophy that no one has explained to them.
Charity: Say more about that.
Katy: So we'll go back to like Shelby and I were talking about load balancers, right?
Like and I tried to just Google what a load balancer was on my second day at InfluxDB.
I was like, "People keep saying this and I don't know, what this mean."
But when I googled it what comes up is just a list of products.
Sometimes you learn a product and how to maybe integrate the product, but not what it does.
So I could see like I tried to set up Kafka and was the saddest person much like Kafka. And--
Charity: English major joke.
Katy: Yeah. But still didn't know what it was for and I didn't know what problem I was solving with the tool I just learned the tool.
Shelby: Yeah, that was very much my experience where it was, okay there's all these products.
And I can maybe map the product name to this terminology that I just learned.
But I don't know like the so what? Of like why do I need this?
Why do I care? Why would people choose one over the other? Where does it fit into the system?
And like I'm very much, I'm a systems thinker almost to a fault where like sometimes I can't make forward progress until I understand where things fit in the overall picture and it slows me down and it's very frustrating sometimes.
And so when Google's search results are basically just a bunch of ads and it's like this doesn't help me make sense of the problem this thing is trying to solve.
And I mean, yeah, you can turn your coworker slack your coworker and be, okay help me understand this.
But when you're the expert, but when you've been doing things long enough you forget what the hump of learning that initial part is.
And so I think, Katy we've been working in tech for a similar period of time and so, and I think that's where our perspective is really valuable.
Because we remember how that pain of having to learn a thing for the very first time.
Recent pain is the most important pain.
Shelby: Yeah, and so I try and hold on to that when I talk to people about observability and throughout my career, I've had moments of like, "Wow, when did I even learn that thing that I'm able to just rattle off now?"
And when I tell people like structured events, and it's like, "Okay, wait, wait, first principles, why would a CS student care about structured events or something that?"
And not everybody, I don't need to explain it that way to everybody, but my ability to explain it that way, it allows me to meet people where they're at.
And so I'm saying all this, it's like I'm not trying to pat myself on the back.
I have to remind myself that that's valuable.
Katy: Yeah. Same. And I think part of the difficulty for me in learning tooling, learning the real world is that it's all very contextual.
So it's like, this probably happens with Honeycomb a lot.
I assume it happens a lot with Circle and most places I've worked, which is that someone will say, "Okay I think there's something wrong with Circle."
And then they'll give me a list of the seven or eight different tools, services, languages that they're using.
And you have to figure out where the problem is.
And it's somewhere in the intersection of these 400 things.
Shelby: Integration bugs are the hardest bugs to debug.
Katy: Sure, yeah. Where does it live?
Shelby: Where does it live?
I've never heard of half of these things and I don't know what they do.
So let me try and figure out what they do and you'll learn very quickly what products map to what systems and what problems they're trying to solve usually.
But it's like there's definitely sort of a cliff that you have to climb before you're able to get to the point where you can help people with that stuff.
So it's really, it's like getting thrown in the deep end on that is one of the best ways to learn the entire ecosystem.
Katy: Yeah. I mean, that's the only way I've done it.
And I think in that way Charity is absolutely right that like--
The only way I have successfully learned any tooling is by someone kind of throwing me into the middle of it and being like, figure this out real quick.
And I think, I hesitate to tell people to dive into the deep end or whatever on things, because I don't want to put people in a situation where they're going to flounder or fail.
But at the same time, how do we balance that learning that first time people encounter the real world, making sure that they both can experience it enough and interact with it enough to actually learn how things work without setting them up to fail?
Katy: Yeah. So I just have this very fond memory of going to the public pool when I was very little and my mom would go into the deep end and put her hands underneath me while I practice swimming.
And like that's kind of what you want is like you do need to be in the deep end but you don't want to feel unsupported.
And I think that's a lot of setting up people for success or failure is the support there.
Shelby: This is something that I'm really excited about like, a Honeycomb we're investing in the design side and charity's talks about this already sort of publicly.
And I think there's so much in observability that supports this first experience in production and this first experience on a new project, on a new service in production.
When you change jobs and all of your experience, a lot of it stays relevant but for anything that's domain specific, that's specific to this company, you don't have any of that.
And so now that it's possible to make that learning process better for people, it's such a waste to not have good observability.
It's like you have money and you're setting it on fire for every hour that your engineers are spinning their wheels and having to ping people on Slack like, "what does this thing mean again?"
"Where is it live? Like yes, like have really good documentation commit your code, make sure your obstructions make sense."
And are very well curated and all of that is a worthy effort and observability supports that knowledge transfer.
Katy: Yeah. I remember when I first started learning about observability, I definitely heard the word but I just thought it meant you can see it.
I thought it was way less specific than it is.
And then when I started actually learning it and using open telemetry, stuff like that, I was, "Oh this actually makes a ton of sense."
Like what would you do if you couldn't see this data?
Like how long would it take you to get here? Would you ever get here? I don't know.
And so when I was in college my wife and I both worked for the university.
She worked as a programmer and I worked in like kind of the I.T help desk, but there, and I'm sure other people in academia can call in or whatever.
There it's so bad.
The education systems need observability and DevOps tooling really badly and don't have it. And they just have this house of cards.
And at the time they had literally one programmer it was my wife.
She had graduated the year before.
So I think about the huge benefits of observability when it comes and it seems like such a simple solution, but it's an unknown concept.
Shelby: I feel like there's people who--
Like for me, I didn't live in the world of metrics and logs for a very long time before I discovered observability.
And so I was just like, "Oh, this is great. Why wouldn't I want this?"
And I think there is sort of a contingent of people who've spent years and years.
And they have the sort of scars of having to debug things where you just have some squiggles on a dashboard and some--
You're tailing some logs or whatever or trying to search through a mountain of logs and find this needle in a haystack.
And if you weren't the person originally read the code you have no idea how did you debug it, unless you're like that wizard who's been around long enough that, like when the squiggles look like this it's Redis or whatever, and there's no way to teach people that.
And that's the thing where I'm just, it's a huge waste.
Yes that experience is super valuable, but we could just skip all of that and then let people debug things better.
Katy: Yeah, so I think I heard once that you can and should use your gut instinct but you can also teach your gut instinct.
Like you can hone it so that it is better.
So that it's like when you look not just at the shape of the graph but when you look at the data you go like, "I have a feeling this is related to Redis because of--"
And then you actually have a reason, because the last time this happened it was Redis or the last time it happened no one found it for three months.
And then it was this other thing.
I like the idea that like there's some instinct involved that can be learned over time so that you become more like Liz.
And this is something interesting about doing what I like to think of doing DevRel backwards, which may be Shelby you can play it on, which is that I thought like you get into DevRel once you are Kelsey Hightower.
And you're like, "I've had all the ideas, and now I will evangelize those ideas."
And I feel like I did it backwards. I was like I have no experience and no knowledge but you can learn with me.
And so it was definitely felt I was doing it a little bit backwards in that way.
Shelby: I'm really heartened to see communities like Egghead who bring in sort of beginner programmers and learners who will develop content and deliver it very sort of early into their careers.
And so we're starting to grow DevRel from this early learner community.
And I think it's awesome.
That's sort of where I've been in a lot of my teaching career is like it started out I was in high school, and I really liked French class and I was good at grammar.
I wasn't great at vocabulary, but some of my classmates really struggled with grammar.
And so my French teacher would send me out in the hallway to tutor them on the grammar section.
But unfortunately this was during the vocabulary review.
And so I wouldn't do as well on the vocabulary tests but I got to help my classmates do better on the grammar side.
And so that's how I learned that, hey I really like tutoring.
And, I dove into the deep end on that in college.
And afterwards I taught English for a while.
And so you don't have to be the expert. You don't have to have written the book on a thing in order to teach it. And sometimes, beginners are the best people to teach a thing because they remember all those pitfalls, it wasn't that long ago.
Katy: Yeah. I really liked that as a philosophy that you can kind of like you just bring people with you along the learning journey.
Charity: That's easier said than done though. Like I'm an introvert.
I cannot learn something new and perform social interaction at the same time.
Or like, I'm someone who, if I need to work my brain I need to have no distractions nothing in my visual, like frame of reference, anything I need to learn it on my own.
And I don't really learn from other people either.
Like I can't, it's a process I guess.
Katy: Yeah. I have very strong learning preferences.
I don't--One of the big struggles for me in learning to code was that I don't do really well with learning from videos.
Charity: Yeah I can do so with the videos.
Katy: It's so hard for me to pay attention.
Charity: Same. Anything that has an audio visual track, honestly, it's just distracting to me. It has to be written.
Katy: Yeah. I'm like, I have a degree in reading just write it down for me. Like, I'll figure it out.
But I think to through the link I like making things in different mediums because I know there are people who prefer that.
But I think too that like even just being a beginner in some ways helps experienced developers who are beginner in the area, feel better about it.
Because I think as I'm sort of like doing community outreach now and like, my job before was a little bit more on the "let me show off what the product can do" side.
And now it's a little bit like "let me show off what our users can do."
And I'm like happy to let them teach me how to do something and to let them be beginners.
I think it's refreshing for them to have someone be like, "Well, I don't really know how to do that. Can you show me?"
And it makes them feel better about learning new things also because I know sometimes like culturally, socially, it's very hard for experienced engineers to say "I don't know how to do that."
Shelby: I'm very lucky that in my first job I got to organize a workshop on the library was helping build for the users of that library who were PhD holding rocket scientists like literal rocket scientists, aerospace engineers.
And so I'm surrounded by people who are like, I mean I think at that point I had been at that job for like nine months of my first full-time office job and teaching these people who've like 30 years of experience in their field.
They're like big names. And I got to not only teach them, but I got to have meetings with some of these where they're learning this thing for the first time.
And they're asking the question that I'm like forming in my head.
And it turns out like everybody is a learner at some point in time.
Shelby: People change jobs, people change projects.
People always are going to have gaps in their knowledge that they have to fill.
And so just for me it's just like I'm so passionate about information accessibility, because like there's always going to be somebody who can benefit from that.
And whether they've been around for a couple of weeks or they've been around for 30 years.
And so it's just like my whole thing with the observability is it's another form of information accessibility that speeds along that process.
Katy: I have learned more about how systems work from observability, like tools and data hen from someone explaining it to me for 45 minutes.
Charity: Say more about that.
Katy: Something that was really hard for me to understand is I would say like, "Okay I know what our product is, right?"
Like it's CircleCI yeah, we do the thing, we pass the builds, but understanding the entire system I'm like, "Oh, well that's impossible."
But when you see something go wrong in observability system, you're like suddenly I'm like, "Oh, I can see a map. I can see a map."
Charity: A lot of this is about taking the shit out of our heads and making it like democratically accessible.
Charity: Because everyone's been working on systems for a long time has this mental model of the system that might be incred--.
This is why it really helps to be an early engineer it'd be one who put it together the first place, that we rely on all the time even though production is always shifting, always changing.
Any map that you have in your head is going to be out of date I assure you. And when you can take that out of your head and put it into a tool where you can then interrogate it, number one it's so much less of a cognitive burden on you.
Number two, it's more accurate and up to date.
Number three, it's available to other people too.
So you're going to either have our dirty cache in the head by our own out-of-date model of the system, or we could put some labor into increasing our visibility in the system so that we can assemble an actual up-to-date view that we can then interact with.
And so can the rest of our team. And it's a better world.
Katy: Yeah, I remember seeing my first graph of like, "Oh here's all the services that were touched in that request.
And here's where the error happened." And I was like--
Charity: Terrifies you the first time you see that. You're like, "Oh, what the fuck."
Katy: Like "Whoput that there, why did it go there?"
Charity: Yeah, this is happening every day. Thousands of times a second.
Katy: Yeah, having that model laid out in front of you is really like for me, it was just like a game changer.
Like I felt like, I am kind of a systems thinker.
And when I can't visualize the system, I get stuck.
Like Shelby was saying, I sometimes have a hard time moving forward.
Katy: And then you just see it in front of you.
And you're like, Oh, well now I don't have to-- I can see the system.
And I know like that makes it easier to navigate.
Charity: What are you optimistic about, where do you see progress?
What do you think is going to become better for people this year?
Katy: So one thing that I think about a lot is the adoption of observability and sort of like, even CI/CD tools like regionally.
So companies that are based in San Francisco almost always have them but if I go to DevOpsDays Nashville a lot of those companies are just beginners in DevOps.
Like they're just getting up board.
Katy: Yeah. And in my experience going to conferences it has been very regional with the adoption just because they haven't had access or the funds or a team.
So my hope is that we keep casting that net wider.
And like you don't need to be a tech company in order to use these tools.
Like every place I've worked has been a product for developers.
And I would really like to see these practices go out to-- Every business is kind of a tech business in some ways.
They have a tech team. And I would love to see them just become more accessible to those smaller teams who have fewer resources.
Charity: Every business is now a tech business.
I've been thinking about this a lot too.
I honestly, I have this rant that's been boiling up in my head for a couple of days now that the colossal failure of our entire technical management class, let's put it that way, because you talk to engineers it's very difficult to find a dozen engineers who are not convinced that CI/CD is a way and that they should have automatic deployments.
And yet it's very difficult to find a dozen companies who have let them do it.
And this to me is like fucking managers, man. It's all her fault.
Katy: Yeah, I wonder a lot about like how much of a burden is on a single engineer to justify this like--
Charity: Engineers can move mountains. I have a lot of confidence in engineers, but it's the engineering managers and directors, VPs.
It's their fucking job. It's their one job is to make sure that their time is being spent on the right things.
And it's hard not to say that this is not the right thing given how most of shitty most of these these pipe-- I'm going to stop rambling, this a whole different podcast.
Let's wrap it up here. Thank you so much for coming, Katy. It was really nice to have you.
Shelby: Thank you, Katy.
Katy: Thank you both.