Ep. #51, AI for Best Practices with Soya Park

Rachel Chalmers: Hi, friends. Today I am thrilled to have Soya Park on my show. Soya is co-founder of a Stealth AI Startup (Akify), building enterprise platforms that capture tacit knowledge from company leaders. She completed her PhD at MIT CSAIL where she researched human AI collaboration and methods for extracting implicit expertise from domain experts. Pretty topical stuff these days.

Soya was also a postdoctoral researcher at Emory University and has worked with Microsoft and IBM Research. Her work focuses on using AI to surface the institutional knowledge that lives in leaders heads but rarely makes it into systems. Soya, it's so great to have you on the show.

Soya Park: Thanks for having me, I'm so excited for my first podcast.

Rachel: I hope we make it a really great experience for you. We're a friendly show.

Soya: Awesome, awesome.

Rachel: In your career you've worked with tons of companies and tons of corporate leaders. How much of a company's most valuable knowledge, its institutional best practices, never makes it into any written system?

Soya: Yeah I would say mostly, almost all of it. And this is something that we keep noticing as we talk to our design partners. This is something that comes up again and again. The same pain point basically.

So what basically happened is that these directors and leader of the company, and they're amazing, they're like superheroes holding these companies together. Then they know how to work the company, they know how to get things done, certain tasks, what to ask the employees but none of them really lives at the enterprise stack.

And the companies we are working in, they are really tech'ed up companies. They're extremely tech savvy, they're really efficient in using their enterprise stack, like project management tools such as Jira, Slack or even AI Copilots. And as they're using these tools heavily, what they say is like this type of best practice doesn't really fit into any tools.

So all these leaders, what they've been telling me, which is a bummer, is that, "there's nowhere to put it so I just put it in my brain."

So yeah they're not in the tool currently. So yeah that's something we, as Akify, really want to solve.

Rachel: So is this like the VP of all of engineering knowing who to ask to solve which problems or is this director level program manager just trying to push a job to completion? Or is it both?

Soya: So I think it's a different level as we talk to different levels of engineers. I think it's both. As a director, they only operate in the high level. So yeah all this high level knowledge is still valuable to learn their company, and the TPM they are the ones that have a more practical knowledge of how to actually get things done. So I think this best practices really exist in any different levels.

Rachel: And where do they leave traces? Is it in email, is it in Slack? Where do you extract that knowledge from?

Soya: Yeah so we do look into enterprise Slack like yes project manageable tools often have a really good structure, like what they put out, how they structured a big task and also we deal with the software company so GitHub and also some companies are Bitbucket. So all these code repositories have ingredients. I would say that's not a best practice as it is. Like it has to be processed that all the ingredients are living on this enterprise stack.

Rachel: What is the most surprising pattern you found that leaders didn't even know they had?

Soya: I guess the most surprising thing that I found is that, I think the fact that they just don't know how great of a decision they're making I think that's honestly the most surprising for me.

When I was a PhD student at MIT, I was funded by IBM through a PhD fellowship so they let me come to work with them. I did an internship and a normal semester as well, so yeah I worked with them a long time.

I had the privilege to look into ML teams and observe how large organizations launched AI products.

My job was back then as a PhD student researcher was to-- what Akify is doing right now, I did it as a person. So I basically asked ML engineers, product teams or legal teams, to really give me, yeah, "Can you send me your email, Slack, or PowerPoint? Or even project management board , like if they use Trello, can you send it to me?"

And I also did an interview with them. But my job is to really, like what Akify is doing right now, is try to really see these jewels. What are the best practices going on? What are the best practices and how it looks like.

I guess it's just a massive amount of data. As it is, it doesn't mean anything. It's messy. Multiple teams and people are involved in it, so it's really hard to make sense of it. At the beginning it's just like raw materials and uncut stones, but once there's the right touches and processes, it becomes a really valuable thing.

So my job is to connect the dots. And I was able to do this because people like me, we know software engineering and how the ML workflows look like so we are able to pick and map these processes within these Slack messages.

So what I found is all these things that they make decisions on daily is amazing. It was totally mixed in at this situation and all. And the thing is, they don't know they are doing great. That's what really surprised me. "You are doing all these great things and why don't you even know about this?"

And I point out things like, for example, when they're building product at the beginning, they're kind of trying to teach machine learning knowledge to their stakeholders and they have amazing slides. So they really try to understand and onboard their clients to their machine learning models.

And I kind of point out things like that and to them it's like, "hey you did this at the last project and it looks like it's really expedited the process and you didn't do this this time even though it's a pretty similar project. Why didn't you do it?"

And they say, "honestly I didn't know there were some good things to do and I guess I just forgot that I did that the last time." So things like this. This is like a very missed opportunity. It's like you're amazing, you should do something you've been doing and yeah that's honestly how we started with this company.

Rachel: That's super interesting that you can find in all of this structured and unstructured data evidence that supports continuing some practices that otherwise people might not even know had that level of efficacy. That's very cool.

So you've got this huge mass of data and luckily natural language processing has come along in the meantime and effectively it sounds like what you're building is a huge knowledge graph, a multidimensional space that's identifying connections between inputs like a really good deck on machine learning and outputs like the project went faster. Is that accurate?

Soya: Yeah, exactly.

I think all this massive data just needs the right eyes and right lenses to look into it.

I guess that's the first part to how to process initially. And the second part is really confirming with them. It's like, we have this proactive AI aspect which is because company best practices change pretty frequently because based on the technology out there or the clients or product or landscape.

So I would say that there's two things: There's initial processing and then also continuously learning with the proactive AI. That's the two really important pieces.

Rachel: Yeah, you can bubble up those identifications as emerging best practices and propagate them throughout the organization. Super cool. Why did you choose to layer on top of existing tools rather than build an entirely new productivity platform?

Soya: Yeah, so this is something that I quickly learned. So my dissertation and all is I built this piece, these productivity tools, and we had to do this literature review as a part of the process. And there is an old paper basically saying, "introducing new tools is something that you should not do because all these tools and suites that companies are using, that's a part of the organization decision. So if you try to introduce a new tool that's basically undermining those decision processing."

So basically this a very important organizational habit. So what they're basically saying is you should always try to preserve it. And interestingly, your tool might not be a good idea because it's really forcing them to change it. So yeah, you have to really be living between this already existing tools and infrastructure and try to read them.

So I guess I knew that from the beginning and that's something that I tried to put in to my heart. It's like, "oh, we should not build this one of these new tools but try to really weave this in existing tools."

Rachel: Yeah, there's probably better ways to design a human car interface but we're all so familiar with the existing controls that changing them would be a huge lift.

Soya: Mhm.

Rachel: So it sounds like your competitive moat is this methodology for capturing all of this data not so much the language model itself. Is that right?

Soya: Yeah, that's right. Mhm.

Rachel: What makes inferring these implicit best practices so difficult?

Soya: Yeah so with these current AI, LLM reasoning models, they're amazing. They're honestly very amazing in how a lot it could be done. But I guess understanding best practices of organizations has been something that predates the gen AI. The first article I saw about best practice was something in a Harvard Business Review article in the 90s. It came from the 90s.

Basically the article is talking about why best practices are so hard. Once you nail this best practice it's such an amazing thing. It's going to help companies to scale and repeat the amazing process again and again. And that's the reason that I got into best practices. I was wondering why companies are not doing this, why is best practice not is a thing?

So I think it has been something organization researchers have beem thinking about this for a while. I've tried to really bring this to light. And I think there's a couple challenges.

With best practices, the nature is very implicit. So what is good or bad about organization is very hard.

And this is something I said before. It's something you need to really have process semantics to understand and also domain expertise, in this case software engineering, to really understand this process. So that's like a unit of some-- like a person with expertise.

And next thing is that with best practices, you need deep reflection. And reflection is something that doesn't go well in the enterprise setting.

Rachel: Well we're all too busy. We don't have time to take a step back and think.

Soya: Yeah, yeah, yeah exactly. It's like they don't have time to. It's like okay so we finished the quarter, so let's think of what we did amazing last quarter. Like they don't have time for that. They literally have deadline after deadline. So all this is just very expensive and implicit.

Best practice is like a unicorn. It's impossible to get it, but once you get it, it's amazing.

Rachel: You're building AI that effectively learns manager preferences and best practices through these feedback loops which is great when it's a virtuous circle, but negative reinforcement also exists. How do you prevent your tool from just reinforcing biases?

Soya: Yes. Gosh, it's an amazing question. Part of the reason, there's a lot of workflow builder out there. I feel like every company, enterprise setting, productivity tools--everyone has a workflow builder and automation. And that's something, the question you ask is like something that's really critical.

I think that's why, honestly, the automation in the current words is not as explored as much, is because of that. Not having this good feedback reliable feedback loop, I think it's really hard to do and some things that require for automation because companies and environments change all the time.

And the way we resolve this is through my co-founder Lydia's background. She's a CS professor at Columbia University and her background is really how to build a reliable AI system over time and so users can really build a trust with AI.

And so yeah she really knows a lot about this and we are able to build a system which is, what we do is we proactively check in with the employees and we try to have this feedback loop within the company's daily workflow.

For example, software companies do stand up meetings and the managers have a dashboard to clearing blockers and we see those as amazing feedback touch points to collect this feedback. They are already in the context, trying to do this work. And yeah it was like we are able to really collect this--

With this, we asked the questions to really clarify what's going on at this point and we were able to collect this knowledge with zero friction. And with our deployment, we see an improvement in the accuracy, automation accuracy.

And yeah after a week of using our tool through our POC, our customers said, "oh yeah I actually trust the automation enough that I can actually use it." So after a week they are able to generate their reliable automation.

Rachel: Very cool. And you're getting recognition. Capital One, Apple, Microsoft are already interested. What is one objection? What's one pushback that you still hear from prospective enterprise buyers?

Soya: Yeah I think like a common theme is, "how much could it be better?" Because rightfully they're already bombarded all about these productivity tools and they're promised the same thing: "we'll automate your process and make it more productive."

So I think that's something we've been seeing when we're onboarding new clients. I guess they have a skepticism, like, "hm, yeah, I don't know how much you can do this," b ecause they're promised before.

So we see that this is something that they have initially, like skepticism. But as time goes by, our product is all about the scalable and building trustful AI and slowly introducing it by following their best decisions.

So after a week, people now see that, oh, it's like they're very slowly introducing it so they're comfortable enough to follow this automation. So we know they're going to come around after a week. Like we keep seeing the same pattern, so we're going to come around. But initially there's something like repeated skepticism we keep seeing.

Rachel: You're doing a great job building the trust. How do you stay current? I mean this market is so fast moving and the models are changing so rapidly. What are you reading? What are you listening to?

Soya: So I'm not a reader. I should read more. But yeah, I'm always saying the joke, when I was a PhD student and everything, my job was to read. I read papers all the time and now I want to take a different approach when I'm learning new things.

So since I'm in San Francisco, I go to Luma events. I talk to people and I go to these events, demo events. And companies are introducing new products and people are coming to the events. I learn so much by going to the events.

And Lydia and I, one commonality between the two of us, as when we were a both researchers, we love organizing these events and we enjoy giving these talks. So our jam is to go into events, sometimes even organize it. So yeah, that's how we learn.

Rachel: It must be nice to be an extrovert. I'm here in my introvert cave, like reading everything I can get my hands on. But yeah, the Luma calendars are crazy. There's like five things on every night that look really great.

Soya: Oh yeah. So I always hand pick. I cannot go to all of that. I try to always do quality over quantity. So yeah, I always talk to just three people. And once I talk to three people, I just get out of there.

Rachel: If you had 12 clones you couldn't do all of them. But that's a really nice rubric: T hree good conversations and then you can go home. I like that. I could do that.

I'm sold on your platform. On your expertise. I'm going to make you prime minister of the solar system. For the next five years, you get to say how everything goes. What does the universe look like in five years time?

Soya: I guess, like how I started working on this project and all is, yeah, I think everyone is already doing some many great things, but there's not enough AI or employees to really tell them that something you're doing is a great thing.

So what I really want to do is this type of cheerleader, positive reinforcement layer to each organization that really tells you, "hey, you did this and that's amazing. You keep doing that." So that's what I want to do is have this AI cheerleader layer living and seeing what you are doing and it's really encouraging you, you should keep doing those things.

Rachel: I would love for us to have more institutional memory as well. Like 10 years ago, everyone was talking about Project Aristotle at Google, which demonstrated that diverse teams have fewer blind spots and are more productive than monocultural teams.

Apparently in the current political climate we've just conveniently forgotten all of that. I would love for people to be constantly reminded that that is actually demonstrably correct and economically reasonable. And denying it, it's only a matter of time before we have to relearn all of those lessons.

Soya: Yeah, I love that. That's something that even yesterday, we talked to design partners. Using their words, it's like, yeah, as you being in the organization, you even forgot what is a good or bad. It's like something because you get used to. It's like, yeah, you need just a constant reminder. Someone has to tell you and really point it out.

Rachel: Yeah that's a really great point. Okay, last question, best question. I know you were nervous about this one, but don't be nervous. This is our signature question. If you had a starship equipped to fly 100 years, with generations of astronauts going to a very distant destination, what would you name it?

Soya: Yes, this question is, I'm very nervous about it because I'm born and raised in South Korea and I just read a cultural article about it, that apparently like a space thing is not a thing in Korea. Like I haven't watched Star Trek or Star Wars until I moved here for grad school.

Rachel: Wow, you're so lucky! You get to watch them all.

Soya: Yeah, I did that within the past five years. So I did it like as an adult. So a spaceship is something that I haven't being thinking about for a long time. T his is something that's very new to me.

But I did come up with some keywords that we are really excited about at Akify. Something like, yeah, I keep talking about this cheerleader, like, a good reinforcement. And also a continuous learning is something that's really important to us. So continuously help you do these good things.

So, yeah, I tried to brainstorm how can I use these keywords to make it more spaceshipy? I tried to look into their name convention and everything. And I asked ChatGPT about it. And one word that sounds like a spaceship is, I think, continuum. It's continual. It's continuous. That's a word that I like at Akify. So I tried to make it as a name. Continuum. I think that's a spaceship name.

Rachel: That's absolutely beautiful. One of my favorites. And you've put more research into that question than like 90% of our guests. So, no surprise that you came up with an excellent name. Can I recommend, if you're just catching up on spaceship media, The Expanse, which is a favorite of mine.

Soya: Oh, I did watch that. Yeah. That show takes some time to get into it, but I did it.By end of season one, I was hooked.

Rachel: Yeah. I can just see the Continuum flying alongside the Behemoth. It's perfect.

Soya: Another recommendation. Did you watch Andor?

Rachel: Did I watch Andor?Of course.I have friends everywhere. I loved Andor.

Soya: Andor was amazing.

Rachel: What a fantastic show.

Soya: Yeah, I did in reverse order. I watched Andor and went back to the old Star Wars movies.

Rachel: Yeah, unfortunately, Andor is the pinnacle of all of the Star Wars properties. It's the absolute best written, most thoughtful thing that's come out of that franchise. So downhill from here on, but Andor is a great point of entry.

Soya: I love it. Yeah, I have a lot of friends who didn't watch Star Wars before, but they got into it because of Andor.

Rachel: Yeah, yeah, there's some real dreck in there, but Andor is a jewel. It absolutely is.

Soya, what a joy to have you on the show. Thank you so much for taking the time.

Soya: Yeah, it was so much fun. Thank you.

Rachel: I hope to have you back again when you've learned even more about this brave new world.

Soya: Yeah, will do.