
Ep. #98, Open Source Data Integration with Michel Tricot of Airbyte
In episode 98 of JAMstack Radio, Brian Douglas speaks with Michel Tricot of Airbyte. This conversation examines data distribution connectors, lessons on improving flexibility within tech stacks, and tactics for leveraging community resources as a startup.
Michel Tricot is Co-founder and CEO of Airbyte. He has been working in data engineering for the past 15 years. As head of integrations and engineering director at Liveramp (NYSE: RAMP), he grew the team responsible for building and scaling the data ingestion and data distribution connectors, syncing 100s of TB every day.
In 2020, he co-founded Airbyte, the new open-source data integration platform, with the vision to commoditize data integration pipelines across all industries and organizations. After just 1 year, Airbyte grew a community of over 5,000 members, got deployed over 16,000 times and raised 180M from Accel, Altimeter, Benchmark, Coatue and YC.
transcript
Michel Tricot: I cannot even enumerate how many sources you might have. And that is the problem with data integration and with existing solution generations. You don't know, every company has data in places that you can't even think about.
The reason for open source is actually to figure out what is the framework and the platform that you need to build, and then how do you enable people to have the flexibility to pull that data wherever it is.
Brian Douglas: Hey, this is Brian and you're listening to Jamstack Radio, a bi-weekly series where we discuss the Jamstack, a new way of building websites and apps that are fast, secure, and simple to work with. Jamstack Radio is brought to you by Heavybit, a program dedicated to helping startups take their developer products to market.
For more information, visit heavybit.com. If you're interested in being a guest on the show, or if you'd like to suggest a topic, find us on Twitter at Jamstack Radio.
Welcome to another installment of Jamstack Radio. On the line, we've got Michel Tricot.
Michel: Hey Brian, thank you so much for having me. Yes. You got it right. Michel Tricot.
Brian: Perfect. Okay, perfect. Yeah, thanks for joining us. You're actually joining us to talk about something I'm not even familiar with too much. I've definitely managed large amounts of data and put together ELTs.
It was more like a senior engineer was doing it years ago, and I was sort of watching on the sideline. That was kind of my experience. But y'all are working on something pretty cool which I haven't said what it is, Airbyte. Do you want to introduce yourself, tell us who you are and why you're here?
Michel: Yeah, of course. I love to talk about data, I love to talk about Airbyte, but on my end, I've been in the data space for the past fifteen years. I worked at many different scales of data, whether financial scale, internet scale, IoT scale, and build engineering team over there.
I was head of integration and director of engineering at LiveRamp. So went from a very scrappy startup up to the IPO and had the chance to experience this massive hypergrowth and grow the team to support it.
And yeah, early 2020, January 2020, my co-founder and I both had our battle scars around what does it mean to actually have access and desilo data, give access to companies to all the data that they own. And that's how we started Airbyte. So obviously we went through a few iteration on the product.
Brian: Okay, excellent. Yeah, you probably had tons of users before the IPO. There's a lot of stuff that's happening on there.
Why Open Source Data Integration
And I'm interested in the idea of the open source data integration model. Why open source? How is it open source? What are we talking about here?
Michel: Yeah. Basically Airbyte is a data integration platform. What does it mean? It needs to be able to pull data from any kind of data sources that you might have, whether it's an API, whether it's a file.
And the thing is, I cannot even enumerate how many sources you might have. And that is the problem with data integration and with existing solutions general, which is you don't know, every company has data in places that you can't even think about.
The reason for open source is actually to figure out what is the framework and the platform that you need to build and then how do you enable people to have the flexibility to pull that data wherever it is, giving them the tools and giving them the ability also to build something that can be reused by other companies.
To take a very simple example, I don't know how many companies are building a Salesforce connector internally. But in the end they only the same data. It's always gonna go to BigQuery, Snowflake, Redshift, S3, whatever.
Why should you have so many people working on the same thing?
And at that point, what we're doing is really offering an open source platform for people to just build it once and then reuse it across multiple companies, sharing the maintenance also of these connectors.
Because when you're talking about data connectors, it's always an external system that you have no control on, they might change APIs, they might change data model. You want to make sure that your connector can adapt very quickly to it. And then the open source model is great for that. People that are users and contributors can actually jump in, fix it, dispatch the new version, and boom, every company has access to that data again.
Brian: Okay. So the open source part of it is the consistencies. This is something I did a couple jobs ago. I had to connect some data from GitHub and I was pretty junior. Same place I was actually watching this ELT stuff.
I didn't realize the SDK that GitHub had, the Octokit community SDK, so I built an entire library of a brand new SDK from scratch based on the needs of the company. And then I went to another company and did the same thing because that's all I knew.
And then when I realized, oh, I could just use the SDK because by then it had been mature enough that it's like, oh I can consistently pull repository data or pull issue data or whatnot and put it in our dashboard. In the same vein, you have a paved path for folks to say, okay, I need data from these properties. Let me check to see if Airbyte already has those connectors.
Michel: Yeah, exactly. So either you can leverage an out-of-the-box connector that is being maintained by us and by the community. Or we also have toolings, CDKs. And then if it's missing, then it's just a matter of getting to work on it with the CDK.
The advantage that you have by integrating with something like Airbyte is that you've done this GitHub scripts. So when you do them, first you need to build a script, then you need to run it, then you need to monitor it, then you need to realize when it's broken, when you need to update it. Building everything around it takes a lot of time and it takes a lot of engineering time, data engineering time.
The Airbyte platform allows you to just run, orchestrate, monitor, give you observability on your data. So you never have to think about it, it's just focus on what has the value, which is pulling the data out.
Brian: That focus, I'm curious, who's your target market then for folks to leverage something like Airbyte?
Michel: When we started with Airbyte, we had to make sure that we can also build the contributor community and not just a user community. So we really focused on the data engineering use case so that our users are also our contributors. And this way we can really bootstrap all these new connectors within the repo.
But then this data is here for someone, it's here for data analyst, analytics engineer, etc. So it's really about providing an internal platform into company that is maintained by data engineer, but the data is actually used by other parts of the organization on warehouses and everything. And how do you enable and remove the friction between these two teams?
Brian: Okay. Excellent. I'm curious of what other out-of-the-box solutions that people can sort of kick around today.
Michel: We have basically the most common APIs, whether you have Stripe, GitHub, Salesforce, Facebook, Google, etc. We also put a lot of effort on databases, which is how can you pull data from MySQL, Postgres, MS SQL, and feed that into your data warehouse so you can start joining this type of data into your warehouse. Then also file-based type of system.
But I cannot tell you how many we have. The thing is it grows every day. I think in the past year we've built over 170 connectors and it just continues to grow right now.
Brian: Is this powered by Airbyte engineers or is this actually community folks who are supporting these?
Michel: It's a bit of both. We obviously did the first ones. Because we also needed to learn what kind of pains people would have and build a tooling around it, but today it's a shared responsibility.
Building With the Community
Brian: Okay, that's great because you mentioned 2020 when you all launched the official company product. But growing an audience, growing a community around the product itself. I'm impressed with folks who create movements around solutions.
And it's not more of, I gotta go knock on your door and say, hey, you know what? It's about time we connect metaverse APIs or whatever. I'm not sure. Now you can actually lean on the community of what they want and what they need.
So I imagine there's some sort of support forum or maybe in the open source repo people are like, hey, this is what I want. And then are you then showing on the ropes and saying, okay, here's the framework, here's the CDK. How about it?
Michel: Yeah, exactly. We have two main forums for interacting with the community. First one is obviously GitHub. The second one is our public Slack.
That's where we really have most of the conversation happening. Troubleshooting, feature requests, just community members helping each other on CDK development.
One thing that I really like about having and working with the community is we're building data infrastructure.
So there is a lot of things that we need to do slowly for reliability reasons. But we're still a startup, so we need to move fast.
And what we get from the community is actually we get a ton of insights about the product, how quickly we can build new features because we have sixteen thousand companies that have deployed Airbyte today. So we get this pause of what is the market looking for. And although in execution and engineering we need to make sure that it is reliable.
Brian: I guess my question is if you're leveraging the community for these CDKs, what are folks paying for? Like cloud offering, one of the things that people are getting out of that?
Michel: So that's something we've always been very transparent with the community, which is Airbyte open source resolve the problem of individual or small teams. Everything that goes above that should be something that Airbyte gets paid for.
The first thing that people talk to us about, okay, I love Airbyte, it's running on my laptop, I want to productionize it. I don't have the time, I don't know how to operate it. What can I do? And that's when we want to be there and say, okay, either you're technical enough yourself or we have an alternative for you, which is Airbyte Cloud.
And for us, that is the first step toward monetization. And we launched Airbyte Cloud in private beta last October. That is the first step to monetization. And after that it will be around potentially additional feature or more like team type of feature. That's the idea.
Brian: Okay. The team features make a lot of sense to be able to manage and maintain this thing together. I guess the genius of this is that the folks who need the connections are probably because they do have a large amount of data. They might have a side project, yes, but in reality, the need to process this data and introspect it is really come from the needs of employees at company.
So I definitely am with you with the business model. Definitely wish you best of luck when you do GA that and share it.
A Thousand Paper Cuts
Zooming out at the problem that you're solving today, I'll just throw an idea out there. I've always been maybe shy about is when I do connections to these data sets from other services like Salesforce. I don't really like doing simple things like testing or mocking out stuff and getting examples.
So I'm curious, have you come across this problem if people need to test to see if the connector is still connecting? Do you raise flags, or I assume there's some monitoring as well to let you know when things are not connected?
Michel: One thing that we're building behind the scene is what we call a connector factory. It's really about identifying all the things that make a connector reliable, whether it's about having access to a sandbox account, whether it's about documentation, whether it's about technical support, how much do they know about the connector? It's about how many community members do we have that expertise around that connector.
All our connectors are monitored on a continuous basis based on the sandboxes that we have. And every time we discover an issue, that is something that we add to that particular sandbox.
We like to say that building connectors is a thousand paper cut problem. First connector you will have will work fifty percent of the time. And then as you have more users and as you discover edge cases, you just start encoding all of that, and at the end you get a very, very reliable connector.
That's buried in our approach.
Centralizing Fragmented Data
Brian: Okay, cool. I'm curious, what are some other examples for things that people would want to start doing with this out of the box?
Michel: Yeah. Let's say you're working for your marketing team. They want to understand attribution model, pulling data from Stripe, pulling data from Google, Google Analytics, Google Ads, Facebook Ads, etc. Getting even data from your own customer database, bringing all of that together, and that will allow you to understand where are these customers coming from, how much are they spending.
What can I attribute that customer to, whether it was content, whether it was ad, whether it was organic, etc. So that's for a marketing use case.
Now we're also working with other open source companies and one way they use Airbyte is let's build our GitHub and community dashboard. And they start pulling a lot of data from GitHub and they build their dashboard on top of it with generally Snowflake, BigQuery, or Redshift in the middle.
Brian: Okay. A thought I've had for a while because I do DevRel at GitHub. At the moment YouTube is our biggest interest at the moment.
Understanding data from YouTube is all on the YouTube dashboard or paying for a third-party tool who's done all the connections and now charges you a very large fee to be able to introspect that data, especially at the size of GitHub and their YouTube channel.
So all we want to do is we do want to identify if there is somebody who is commenting on YouTube, can we cross check that with somebody who's in Twitter and maybe shout out somebody who had a great question in the YouTube comments? We can figure it out and then do something with that or tell a story for validating the work we're doing every day and trying to get the word out on features.
Michel: Yeah, and ultimately that's really the value of centralizing data and accessing all this fragmented data is it allows you to measure better and it allows you to show also what you're doing and the impact of what you're doing and it also gives you what are the next steps.
If you realize that this particular type of video is not working on YouTube, you should just maybe just shut it down and invest more into another type of videos. Having this centralized view with all these data sources is making company better than others. How good are they with reacting and anticipating with data?
Brian: Yeah. It is a very interesting problem to be in and I'm pretty sure you're on the forefront of the excitement at the moment.
So speaking specifically to DevRel, we've migrated from going to tons of conferences, being at events and essentially to saying, hey, conference organizer, let us know, give us the list of people are here, will give us the scanner for badges, etc. That was the norm for DevRel for the longest time and that's changed a bit.
Now we're doing live streams from Twitch and Twitter. Some people are doing that. LinkedIn is another great place to do live streams. I'm not sure if I'm just unveiling all the DevRel secrets right now, but we want to know, okay, people on LinkedIn, they're engaging.
If we want to recruit after that, at what point in the conversation or in the talk that we have a spike in views and reactions. Let's do that again. And that's all data that way we're really sort of peeling back under the curtain and getting that from LinkedIn directly. But it'd be great if we could also build our own features on top of that data as well. And I think it's a big need.
I think there's another, this one's for free, but this whole movement around open source. There's a lot of projects that are open source that just need to have community contribution data. And GitHub does have the insights tab, but sometimes what we provide as well is not the questions you want to answer.
You have another question. You're sort of trying to skate where the puck's going. And by having access to your data, you can basically skate wherever you want on the ice.
Michel: And that's exactly what you're saying, is there is no product that can guess what you're gonna do with data and what you actually want to get from the data.
So if there is a way to have access to the raw data and you put smart people on top of that data, they will figure out what matters.
And that's a big shift that we're seeing in the data world today, which is, I mean, people call that the modern data stack, but it's about enabling data analysis on top of warehouse. Making sure that you have on the left side massive amount of data that is flowing through your warehouse and then just leveraging these people to either create dashboards, analytics, etc. That is becoming the foundation for how companies are making decisions today.
Brian: I'm intrigued. I'll be paying attention to Airbyte for sure. Hopeful folks who listen to the conversation will also check it out.
If there's something that I assume is of need, y'all have a Slack channel. And you also have an open source GitHub repo where they can sort of raise their hand and say, hey, this is what I would love to see out of this.
Michel: Yep, exactly.
Picks: TikTok, Wordle and Brisket
Brian: Cool. So with that, I do want to transition us to picks. These are things that we're jamming on.
Could be music, food, whatever. Sometimes people use technology picks. Sometimes people go right off the rails and talk about this random stuff.
But Michel, I've actually got some picks, so I wanted to jump in because it came up before we even hit record, I was actually running late because I was actually producing a TikTok. So GitHub actually has a TikTok account. So at the moment we're underscore GitHub. If you just type in GitHub, you'll see us where we've got the most followers that have the name.
It's been an interesting experiment. For DevRel, I think what we like about it, it's a new audience. And I don't know if a lot of people realize how new this audience is. It is definitely a younger crowd, a lot more college students, a lot more even high schoolers, because you only have to be 13 to have an account.
Sharing messages and content to an audience that we're not used to, which is professional developer and beyond, that's kind of our go-to-market audience. It's been intriguing as the approach. So we were trying different things. I'll probably end up doing a DevRelCon talk about this down the road.
But it's worth paying attention to. I know there's some issues with data collection for TikTok. So hopefully they figured it out. But I mean, I've got a second phone to do all that business with too as well. So yeah, I just want to shout out to TikTok.
But also I wanted to shout out this thing that I've been messing around with, which is Wordle. Every day I walk into a website that's not even called Wordle.com because someone else is squatting on it.
What's amazing is this engineer, I think his last name is Wardle. His wife liked word searches and word puzzles. So created this quick app and it's quickly becoming the to-do app for everybody.
I saw Evan You do the Vue Wordle. I saw the Remix team also do a Remix Wordle. And do it live streamed as well, both of them. And then we've seen some other Wordles, like knockoff Wordles, and I think this is a really good testament.
The engineer himself, he wasn't looking to make money or promote anything. So he just letting it grow as much as he'd like.
And I love that story because I don't think everything you have to do has to be a big splash and acquire users and take on VC funding or anything like that. Sometimes you could just have fun with a thing and then see it grow and I imagine by the summer no one will care about Wordles. It's a moment right now on the internet.
Michel: You know, the team is always comparing there. Well, those, cause I would say as a non-native speaker, it can be a challenge sometimes. Yeah. But I'm getting there.
Brian: The concept again, the two app, you just have five letter words, add a nice little short word database, let people have at it. And it's like, I love the constraints of the problem. But I eventually will be building my own Wordle just because the constraints. It's like a nice little weekend project.
Michel: It doesn't matter what the ambition you have, it is about what kind of value do you also bring to people. I mean, Wordle is just, oh, it's my lunch break, let's do the Wordle. If they enjoy it, yeah, that's a pretty good objective.
We have different type of objective. We rely on VC money to get to that objective, but we've spent a lot of time trying to understand what are the individual we're working with, what do they need, what would make their life better, what do they sometimes dislike to do completely and how can we take the burden on us and our community to just let them do things that they prefer.
I don't think there is a small project. It's just about the impact you have and the value you bring to people.
Brian: Yeah, indeed. Curious, do you have any picks for the listeners? So yeah, do you have any GitHub repos or anything else that you've been kicking at?
Michel: I am working on an open source project, so where I spend my time is our GitHub repo. But besides that, I just bought some new thermometer for my smoker. I cannot wait to grill some and to smoke some.
Brian: Oh, what type of smoker do you have?
Michel: I have the Smokey Mountain.
Brian: Okay, I'm familiar. I don't have a smoker or a grill. I just moved to a place that has a backyard and I've yet to actually have any sort of barbecue. So maybe I'm failing as an American.
Michel: You beat me to it. And my next big step is let's do a long brisket. That's why I prepared the thermometer so minimizing my chances of failing the brisket.
Brian: I don't know if you're into TikTok, but if you want to fall into a barbecue TikTok, it is a thing. People showing off the best meats for sure. TikTok's a weird place, so tread lightly.
Appreciate you taking time out of your day to chat about Airbyte, folks. Sounds like a pretty sweet project to embed into your sort of ELT's data infrastructure warehouses as well. So check it out and keep spreading the jam.
That's all the time we have for today. If you're interested in being a guest on the show or if you'd like to suggest a topic, find us on Twitter at Jamstack Radio.
To learn more about Heavybit, visit heavybit.com. And while you're there, check out their library. It's packed with amazing talks on sales, marketing, product, and general management from founders of developer tools companies and other industry leaders.
Subscribe to Heavybit Updates
You donât have to build on your own. We help you stay ahead with the hottest resources, latest product updates, and top job opportunities from the community. Donât miss outâsubscribe now.
Content from the Library
Open Source Ready Ep. #46, Donât Ask Claude to Push the Button with JT Archie
On episode 46 of Open Source Ready, Brian Douglas and John McBride sit down with JT Archie. When an agent skips a command or grabs the wrong branch, a better prompt may only get you so far. JT explains how Steps uses structured pipelines to give agents clearer inputs and boundaries, then joins the hosts to explore SQLite, tool permissions, and the limits of fully automated development.
Making AI its own Primitive in Ruby
RubyLLM was built so that Ruby coders could use it to build with AI, which is becoming a language-agnostic primitive. Builder Carmine Paolino explains.
Open Source Ready Ep. #45, When Agents Outrun Human Review with Dotta of Paperclip
On episode 45 of Open Source Ready, Brian Douglas and John McBride speak with Dotta about Paperclip, an open-source platform for managing teams of AI agents. They explore what happens when agents produce more work than humans can review, why permissions need to extend beyond a separate computer, and how engineering judgment changes when writing code becomes easier.
