Ep. #42, Ziggy Stardust with Katie Hallett

Rachel Chalmers: Today I am so pleased to have Katie Hallett on the show.

Katie's part of the founding team at Lightpanda along with Francis and Pierre.

They built BlueBoard, an eCommerce analytics company, later acquired by ChannelAdvisor.

Their experience developing its scraping infrastructure and navigating the limitations of Chrome Headless inspired their vision to redesign a web browser for the AI era.

Katie, thank you so much for coming on the show.

Katie Hallett: Thank you for having me. Pleasure to be here.

Rachel: But Katie, I love my web browser.

Chrome and I have been together for half my life.

Please don't pry my web browser from my cold dead hands.

Why build one for machines?

Katie: Yeah, so no stress there. We are not taking Chrome away from you.

Rachel: Thank God.

Katie: Chrome is probably here to stay for as long as we're still using web browsers to interface with the web as humans.

Like there's a lot of new projects popping up for consumer facing browsers, which are built kind of largely on the Chromium code base.

So like Perplexity's Comet Project, or Arc.

But what Lightpanda is building is truly adjacent to that.

We're building something from first principles for machines that don't need to see a web browser. So they don't care about how much it looks great and how user friendly it is.

Rachel: What kinds of jobs do these scrapers, these headless browsers do? What were the problems you were running into at BlueBoard?

Katie: Yeah, so Blueboard was a B2B SaaS company in the eCommerce analytics space.

So we were scraping millions of product pages, grabbing data like the price, the stock availability, consumer reviews at scale so that consumer brands could get insights on their distribution essentially.

So that was a very, I guess, you could call it traditional data extraction scraping use case where simply put using a browser that's designed for humans is just a super heavy way to go around.

That is like it's a hack basically. So the problems that we had at scale were just, it's like you start running down at a scale, it's expensive, it breaks, you need like a super complicated infrastructure to host it all.

It's just not fit for purpose to put it simply, I suppose.

Rachel: Can't you just run Firefox headless and then use Puppeteer or Playwright?

Katie: You can, yeah, that's exactly what people do.

So I mean there's slightly different use cases between headless Chrome and headless Firefox.

Headless Firefox is more generally used on the testing front, but Playwright and Puppeteer to programmatically control a web browser in headless mode is the defacto solution right now, which everyone is using.

It's what we were doing 10 years ago and it's where every browser infra company building agency is relying on today.

And we built Lightpanda with the Chrome dev tools Protocol API which means same like Playwright script that you would send to Firefox or Chrome in headless mode, you can just plug and play that, switch it up to Lightpanda and it will execute on Lightpanda the same way.

Rachel: So what's in Lightpanda? Is it like a Parser it's just interpreting the HTML?

Katie: Yeah, so Lightpanda is, it's a full web browser with one big difference right now, which is that we don't do any of the graphical composition.

So we still have a JavaScript engine, which is V8.

We have an HTML five parser from the Netsurf project that we're reusing and a couple of other C libraries.

But we're doing full JavaScript execution, which essentially you can kind of see us as a layer in between like a fully fledged web browser and a basic HTTP request.

So like a kind of curl with JavaScript execution if you like.

But you have access to the full living dormitory and it is a full web browser just without the graphical elements.

Meaning something like taking a screenshot would be not possible in Lightpanda.

Rachel: So I've got my Puppeteer script, I can fire them into Lightpanda just like I was firing them into headless Chrome.

What's the saving, like how much faster is it? What's the smaller footprint that I get from Lightpanda?

Katie: Yeah, so firstly it's--

Before you even get to the execution, it starts up about 30 times faster than Chrome, which in itself opens up loads of possibilities when you're running at scale, like serverless use cases.

And then execution it's about 10 times faster than headless Chrome while using somewhere between, depending on the machine, like five and 10 times less memory as well.

So the throughput of all of that is somewhere around a hundred times more efficiency, which starts to get really impactful when you're doing things at scale.

Rachel: What kinds of things are your users scraping? What are they pulling off the web using Lightpanda?

Katie: Yeah, so I mean Lightpanda is like a super horizontal project and our goal is to be really the layer that sits underneath the entire web automation stack and is like to machines literally what Chrome is to humans.

So that's everything from like your individual developer who has some really niche use case on one specific website right up to the AI labs who need to grab, like, who need to index the whole web for fine tuning and training.

And then everyone in between who has like AI agents that they're building or even traditional web scraping use cases in the eCommerce space like our previous company.

Rachel: Yeah, I really need an agent to just download all of my tax file information and send it to my accountant. Could you get on that?

Katie: I'm sure. I'm sure that's something we're going to be doing for you in the future, whether you know it or not.

Rachel: Fantastic. Where did the name come from? I love it.

Katie: Yeah, so Lightpanda was like, that was the classic ideation session that happens at the beginning of a project.

We chose it for a few reasons, some of them quite pragmatic, like domain name things, but also like there's a bit of a history with browsers and animals as mascots.

Like the Firefox is the Red Panda I believe.

And so yeah, we're a lighter panda that's surfing the web.

Rachel: Totally makes sense. So it's written in Zig, which is a new one on me.

What is Zig and why is it cool?

Katie: Yeah, so Zig is a pretty new, relatively new system programming language.

At the time we were starting Lightpanda three years ago, it was really pretty unknown.

Now there's a handful of pretty well known projects, so it's kind of getting more of a reputation in the developer community.

And I mean for what we're doing, building a web browser, there wasn't a ton of choice for programming languages.

We needed something which was a low-level system programming language. So the choices were essentially C, C++, Rust or Zig.

And Zig has, for our specific use case, a lot of really cool features. So firstly like C, C++ less modern languages.

So Rust and Zig offer a lot more safety features, which is super important for a project like ours. So that kind of ruled out C and C++.

And then Zig is just in terms of the DNA of our project, which is all about the speed and performance has this comp time feature, which versus the other three languages we could have chosen from is essentially like it opens up the possibility to do meta programming IE like code generated by code, which just makes the whole architecture of the project so much more efficient.

It combined with all the C libraries that like completely effortlessly that we're using say like V8, NetSurf libraries, et cetera.

And it's simpler than Rust, which was more of a personal preference.

Like Rust could have been a totally great language to do this project in as well.

Rachel: You've sold me, I'm a Zig fancier now.

Katie: Oh that sounds cool, right?

Rachel: Do you use any AI tools in your own workflow?

Katie: Yeah, so I'm the only non-developer in Lightpanda.

So I actually use AI a lot in my kind of ideation processes for creating content and even just thinking through like how to approach a problem.

I use a lot of deep research in market sizing and prospection and things like that. Kind of like a creative partner.

I've never been great at going from zero to one without another person to bounce ideas off.

So I would say that was probably my primary use case right now.

In terms of all of our go to market and outbound, we're in like hyper personalized outreach mode.

So I'm not using it yet on the SDR side, but I'm interested and I've had some great conversations with people that are doing that because my whole background has been in yeah, like go to market and customer success.

So it's interesting seeing how that whole sector is moving right now.

Rachel: Do you worry that LLMs are going to replace engineers?

Katie: I mean, I don't, I think engineers have been the first to adopt and that's been a trend on other things.

And so that we're seeing how this is playing out in the way their roles are transforming first.

And so I think their roles are definitely going to change and different types of people will probably flourish. If you were a developer now versus 10, 15 years ago, I definitely think the approach is different.

But I fundamentally am optimistic about the outlook for, I think more code means more code means ultimately more developers.

Rachel: More customers will Lightpanda.

When you say you think different people will flourish, I want to dig on that a little bit.

What kinds of people do you think will thrive in an AI enabled world?

Katie: Well, I mean--

I think it requires a certain openness of thought. So embracing change and accepting that where once your role as a developer, to take that example, was like 9:00 'til 5:00 writing lines of code, now instead you are orchestrating code gen agents and then you're reviewing that code.

So I think embracing that change is not necessarily something which, and I obviously that's a bit of a broad brush generalization.

I mean there's a lot more nuance than that, but I think if you're willing to embrace the promise that AI can actually elevate your profession, then I think that that will enable you to flourish as a kind of broad generalization.

Rachel: Yeah, that makes sense.

This one's a little selfish 'cause I've got a couple of college kids in the house.

What advice would you give to college graduates based on what you've learned from working with AI the last couple of years?

Katie: Gosh, yeah, I love this question and I actually found it really thought provoking 'cause I have two kids as well who are far from being college graduates.

And I candidly don't feel that well placed to dish out advice.

I asked a couple of other people as well their thoughts on this and I think it's a really tough time for people to be graduating college. I really don't envy them. I mean it was hard in different ways for us I think.

But I suppose what I kind of came down to was a couple of things.

On the engineering side, choose something and specialize in it and get really good at your thing, which is something which you could argue is true in a lot of walks of life anyway. And then otherwise, there are some things that are going to hold true regardless of the cultural era that we're in, right? Like AI is here to stay, but humans are still also here to stay.

Rachel: We hope.

Katie: Surround yourself by the kinds of people that you want to be like, right?

Like you are the sum of the people who you spend the most time with.

And I think the hardest thing is getting into that first job out of college.

So just relentlessly to see that. Like I remember being there and sending like hundreds and hundreds, probably thousands of outbound messages.

And so I still think that looks the same whether you're using different tools and once you're in there, once you start building your network, I think that the mindset still is the most important thing.

Rachel: Yeah, LinkedIn turned out to be an incredibly powerful tool for me.

I just like at one point started putting everyone I met into my LinkedIn network and that's been one of my better investments.

Katie: Yeah.

Rachel: Katie, what are some of your favorite sources for learning about AI?

Katie: So I tried a lot of the different kind of daily digests.

My favorite one that I've found so far is the newsletter from TLDR. They have an AI one specifically.

Rachel: Yep, I love that one.

Katie: And I've just found that it's the best summary.

It always has everything and I like the way that they write.

That's probably my primary source for kind of daily news.

I think it's good to honestly keep your sources pretty limited. Obviously being in San Francisco, I'm attending all the meetups and stuff and getting other people's input, but you can quickly fatigue from it all.

So I would recommend finding the one source and then not using too much else.

Rachel: Yeah, it's kind of remarkable like for everyone saying San Francisco is over.

If you're on any of the events list, there's like a dozen gen AI specific events every night here in the city. It's exhausting.

The youth, they have so much energy.

All right, if everything goes exactly the way you'd like it to for the next five years, I'm making you God Emperor of the Galaxy.

You get to determine how things pan out. What does the future look like?

Katie: I would love, I mean I'm going to keep this fairly micro because obviously I could get very idealistic if we go too macro, but.

Rachel: People do.

Katie: I mean, I'd love world peace, but hey.

Rachel: Yeah.

Katie:

I would love to see a world where the promise of AI agents is realized. And AI optimism, as well, is proven to be not crazy.

So I mean concretely where agents are reliable enough to work safely alongside us and it's not hype or novelty, but it's an actual tool that elevates human consciousness.

And I'd love for you to have a browser teammate that's running on Lightpanda.

Rachel: That's doing my taxes for me.

Katie: Right!

Rachel: Now, I'm giving you your own starship. It doesn't travel faster than light, sorry.

So it's going to take more than one human generation to get to Alpha Centauri. What are you going to name your ship to the stars?

Katie: I'm going to call it Ziggy Stardust.

Rachel: Oh, that's fantastic. Bowie reference. That's the first time we've had that one.

Katie: It's kind of become our nominal mascot for Lightpanda because I have a little David Bowie Panda now.

And Zig's logo is like the lighting bolt. So it's a nice metaphor.

So for our project, like a lot of people have said to us countless now that we're insane for trying to build a web browser from scratch.

Rachel: Aladdin Sane. Big Bowie fan?

Katie: Huge Bowie fan.

Rachel: He was so far ahead of his time.

Katie: Right? And I would love for him to be there when my kids see Lightpanda ruling the world, generations from now.

Rachel: Yeah, bring back Bowie. That's a strong platform.

Katie, it's wonderful having you on the show. Thank you so much for your time.

Katie: Thank you so much.