Ep. #14, Databases for Humans and Agents with Hannes Mühleisen

CL Kao: Welcome to the Data Renegades podcast. Today, our guest is Hannes Mühleisen, co-creator of DuckDB. DuckDB is the analytical database that runs inside your own program, in your browser, on your laptop, and it has spread from notebook into production pipeline across the industry.

Hannes built it with Mark Raasveldt in the database research group in CWI Amsterdam, co-founded DuckDB Labs to carry it forward, and then AWS acquired the company in August 2026. Hannes and I will pull two threads. What does the small data argument mean now that it lives inside the largest cloud? And then what a database should look like when the machine can write more SQL than human. Welcome to the show, Hannes.

Hannes Mühleisen: Thank you for having me. Absolute pleasure.

CL: So I guess first of all, for a listener who's new to your work or just assume DuckDB exists as it is already in their whole life, can you introduce yourself?

Hannes: All right. So my name is Hannes, as you said. I live in Amsterdam in the Netherlands, which is a small country. I've been working on databases for the past 20 years or so. And about the last eight years or so was on DuckDB.

I was a researcher in the beginning. Then I was a professor and now I'm an entrepreneur and now I'm an employee again, which is a kind of weird process, I have to say.

CL: Right. So from this research institute, CWI is kind of the top research institute in the Netherlands, right? And then you started the company carrying forward for the DuckDB research into the industry and then now part of AWS. So what's the through line here? It's just obviously database, but what's your unique take that this needs to happen?

Hannes: I think the through line could be many things. One of them is Mark. I've been working together with Mark, who you mentioned, for I think 10 years or so at this point. So he started as a master's student and then became a PhD student and then became a postdoc. And he became a co-founder and now is my fellow colleague.

So that's definitely a constant. We have been a very creative partnership, I think, with the two of us over the lost decade to work on databases. So in a way, Mark and, of course, many others in the team are a constant, but he has always been there.

My general approach to systems work I think also hasn't changed at all, where, which is uncommon, we really look at product, at user experience, that kind of stuff is really relevant. And it's kind of funny because we also look at performance and we're pretty good at performance, but we will always look at user experience. And I think that's pretty unique.

And that's also a long through line. And my office is also in the same place. The CWI office is 200 yards from here.

And our company office was here where I'm sitting. And now this is an Amazon office. So, hey, I'm also in the same place in lovely Science Park, Amsterdam, which is a terrible, terrible place.

CL: Okay, so DuckDB came off on the research lab, but it's, I guess, unconventional. It has more code than paper. There's a lot of research and that performance focus.

I lived through where it was launched and got popular. It was really amazing. I don't know how often, it's like week, you got exciting new usability things. So there's definitely that focus usability. That's very rare in the database system.

But at that time, I think you're still at CWI. How do you convince CWI to let you disappear into working with a more product or industry thing? Because it's unusual.

Hannes: Yeah, we didn't ask for permission. You would be surprised to hear that. Even though I'm German, I do like breaking the rules, which is why I don't live in Germany and I live in the Netherlands, where rule breaking is much more accepted.

Like they say in the Pirates of the Caribbean, the rules are more actual guidelines. And this is true, especially in Amsterdam. I think the rest of the Netherlands, maybe not so much. But Amsterdam, as long as you don't inconvenience someone else, you can get away with a lot.

So when Mark and I, one fateful night, decided that we're going to start a database system, we didn't tell anyone. We just started doing it. It was a bit awkward in the beginning when we went to the research group meetings, and they asked you, what have you been working on?

We were like, yeah, this thing that you would only be working on if you built a database from scratch. And it became a bit obvious after a year or so of that. But we didn't tell people in the beginning.

Because it's always difficult to start something new. There's always an institutional tendency of stomping out maybe the new thing, because it happens a lot that people come up with new ideas, and it's not always in the line of the research group or the institute or the funding scheme or whatever. So we just decided we didn't care, and we did it.

So we went rogue. And what helped in going rogue was that they couldn't fire me anymore because I had just gotten tenure weeks before that. And Mark had all the papers for his PhD, but still had time on his contract that I couldn't fire him either.

And so we had this kryptonite of basically saying, we work here, but you have to pay our salaries, but we can kind of do what we want. In the more positive formulation, I want to give them credit. The CWI is a great place that leaves people alone to think, right?

They gave me tenure and said, you know best what you will do. And that's what we did. But we still didn't tell anyone.

This was a total gamble of both of our research careers. Had this not worked out, it would have been over. As you mentioned, we didn't write any papers for years. Had this not gone anywhere, it would have been over, right? We would have to do something else.

CL: Okay, so I'm curious, though, you seem to mention that this is a big conviction, well, what we got to do is do this right. And then, so were you already imagining what kind of success looked like, as in everyone doing any sort of analytics, kind of not quite SQLite, will be using DuckDB? Do you have that kind of picture in the beginning? Well, this is where it's going to be.

Hannes: Wow. No, we had some hopes. We did say, let's build it in a way so that in the very least, it's not getting in the way of it becoming popular.

Our ambition, and I'm an ambitious person, as you can imagine, our ambition was for everybody to run DuckDB eventually. We were also realistic enough, however, to know that that is not very likely, but we're going to try nevertheless.

That's my personality of leaping and looking later. But I think the design of DuckDB was definitely made to appeal to many people, right? So it was definitely designed to not be something that three oceanographers will install on their server farm and then use that to process oceanography data.

No, we made a database system. We realized that they are universally applicable. And so we did everything to not be in the way. But we couldn't imagine the attraction it would get very quickly.

CL: You're, I guess, very modest. What I'm hearing is that, well, you have this directional thing. Well, what does it look like?

Well, everyone will be using that. But you leave yourself out of blocking that success. As in, how do we...

Hannes: How do we not screw this up?

CL: Yeah.

Hannes: Exactly. But we did think about some things. We did think about user experience, right? We said, hey, the user experience we envisioned influenced the design. It usually doesn't go that way.

We said, this needs to be trivial to install and run. Okay, so it had to be an in-process system because it needs to be trivial to install and run. So there's already a design decision that we took because we needed to be interesting to as many people as possible.

CL: Right. What's really interesting is that focus of user experience for a system like database system is super rare. Was that an idea that came from something else or just, we got to do it this way?

Hannes: I think it came from us talking to statisticians, them saying that they hated databases. And we were like, why do you hate databases? Not databases, like database management systems.

When I say database, I mean a database management system. When a statistician says database, they mean a table, but I mean the system. They said they hated database management systems.

And I was like, why? What's wrong with databases? And by talking to them at length over many years, we found out what they didn't like about it. And installation was one of the things.

Hannes: You couldn't have a database system as part of an analysis script because it just wouldn't work. The username would be different, the host name would be different, the port would be different. All of that wouldn't work.

You couldn't have a MySQL in there, even though you might want to use MySQL or Postgres or whatever. But that wouldn't work because everybody's setup is going to be different, the port would be taken on their machine. They wouldn't have root permissions, so they would have to do something else. Their username would be different. It couldn't be part of a script.

And that was one of the things that we fixed very early on, that DuckDB can be part of a script. How we learned, we learned by talking to people, is a very uncommon thing for computer scientists.

We talk to the people and we realized that one person's screw up is another person's opportunity. Because nobody in the database community had ever cared about that sort of thing, there was a huge amount of opportunity for us. We are very grateful to everybody who didn't care about the users in databases.

Because otherwise, we wouldn't have a company or a project or anything, right? It would not have happened. So, ironically, somebody who got this right was MongoDB.

MongoDB showed up and said, we need web scale and we don't need these pesky schemas and transactions. Okay, fine. But lots of the user experience they got right.

They said, okay, here's this binary. You download it, you run it, it works. It's insecure AF, but it works. So they prioritized some things that made it easier for users. And we looked at this and we're like, oh, single binary, awesome, we'll do single binary as well. No dependencies, awesome, we'll do that as well. So we had some inspiration, but that was Mongo, that was not a traditional database. It hadn't appeared yet in the Oracles of this world.

Still hasn't.

CL: So the statistician you talked to kind of highlighted the tension because obviously they love SQL, but they hate having to...

Hannes: No, no. They don't like SQL either.

CL: So why didn't you solve that problem either?

Hannes: Well, we have worked on SQL a lot. We pioneered friendly SQL. That's a DuckDB thing, right?

CL: Yeah, that's what we want to get into as well. Because like I said, the weekly release is all that. It's all, wow, you don't have the trailing comma anymore. You can say from something. So this is all great.

Hannes: You can do everything.

CL: I want to touch on a little bit first on the overall journey, because I think a lot of listeners will be interesting, because you call yourself a reluctant founder, right? And you turn down venture money for years. And then there's always this tension, like doing deep engineering work or research versus being a company, right?

So I guess the question is really, what did running DuckDB cost you that the engineering never did? What's the difference in life for you?

Hannes: So what did it cost me? Yeah, that's an interesting question. I was a researcher, right? I was sitting in my office not far from here every day thinking about database architectures.

In a way, I still do the exact same thing, right? I think the brain loop I'm running is still the same: think about databases. How can databases be better? Repeat.

What did it cost me? I think it's fascinating. I think if you work in the government research lab, your emotional ups and downs, they are quite predictable.

It's like paper acceptance or research grant acceptance or research grant rejection or paper rejection or promotion or no promotion. It happens, but it's a very predictable rhythm, right? I guess if you win a Nobel Prize, you don't know in advance, but in general, this is very predictable. You will know the notification date for your paper and then you will know, right? I think something that was different in running a company was the emotional ups and downs were quicker together and more violent, if you want.

One moment you hear that DuckDB has been adopted by a company XYZ and you go like, yes, awesome. And the next moment, one of your employees has a serious problem at home and you have to deal with that and it's very intense.

Or you're yourself struggling. I definitely had started going to therapy when I started a company just to get emotionally ready for the onslaught every day of questions, decisions, right? You can imagine there's a lot of that. And it's very different from the life of a researcher, to be quite honest.

It's different. And now that's all gone again, because now I'm an employee at one of the biggest companies in the world, which is something I'm still adjusting to, but so far the emotions have been good there. So, hey.

CL: Well, thank you for sharing that. Back to the database side of things. You posted about the lost decade for the opportunity that the data system had, right?

Tell us more about that. What was the whole point or what do you want to change? And then has it achieved?

Hannes: Yeah, I can dig in my thing here. I didn't prepare this, I swear. But I have this laptop here that is in my office.

CL: This will get into a computer museum at some point.

Hannes: No, I don't think so. So this was my first MacBook Pro that I bought. I was unemployed at the time. This was actually not a great purchasing decision when unemployed, but it just came out in 2012.

And it was the first Retina MacBook Pro. I don't know if you remember this one. And I bought it because it was so awesome. I'm not using it anymore, but it's in my office, so my children, when they're here, can watch YouTube.

And one day I was wondering if it could run DuckDB still and how fast it would be. And it turns out it can run DuckDB very well. And it can run queries and benchmarks very fast, even compared to today's computers.

And then that brought me to this idea of the lost decade, because what this laptop had, what people realized at the time, what a big thing it would be, although Phil Schiller, I think, in the keynote talked about it at length, that it was the first laptop that had an SSD by default, at least the first mass. Historians are going to murder me and say, oh, there was a Dell before. Yeah, fine. But the first mass market, and it's such a game changer for data processing.

Had DuckDB existed in 2012, it would have been a total game changer. But what people did in 2012, again, they would go to these distributed things, even for medium-sized data, because nobody could imagine that your laptop could deal with it.

And that's what I wanted to show, that, hey, the local small data thing, the local firsts, that world could have been there in 2012 because the software, yeah, that was possible because obviously it can run. And so somebody could have written it back then. Maybe not with C++11, but maybe the C++ 07 or whatever it was there before, right?

So I think it was really fascinating. Again, I'm really glad no one did because otherwise I would have not had my opportunities. So, in a way, I'm happy about it.

But I think it was really fascinating to see how fast this thing, what a game changer this architecture was. Not the screen actually, the screen didn't matter, but the SSD together with a good CPU. Let's say DuckDB had been around at the time, history could have turned out very differently of databases.

CL: This reminds me of a story because this is kind of underlying hardware change that the software has not caught up, right? You would imagine that in efficient economics, someone should have already implemented that to max out the capacity. You would think. I remember in early 2000, and I think we have multiprocessor for a long time.

But my friend, Artur Bergman, who later founded Fastly, was complaining that, why does copying rsync file from one host to another so slow? Because we're not saturating the network. What's going on? Because there's single-threaded or something.

So there's always a lot of the gap between the underlying architecture change and how the user can actually get the benefit and experience that. So I'm super pumped what DuckDB actually made it happen for that decade kind of recover rather than another decade missing.

Hannes: Yeah, we've gotten back. But I think that's really cool. We got back to this. I think it was possible to go back.

I think it also happens when game consoles, like the first games on a new console, they never really push it to the max. It takes them like five years to get to a game that really pushes the console to the max because it just takes them so long to figure out how the hardware really works, what really can be done. And I think it's a bit the same.

What you mentioned with multi-processing, I think is also really fascinating in database terms because for many systems, the multi-processing is tacked on afterwards. They have an engine and then they say, ah, yeah, we need to do multi-core. Ah, yeah, it's through this weird thing where we have this--

CL: Special model locking things.

Hannes: Yeah, we have an exchange operator and our operators pretend parallelism doesn't exist. In DuckDB, we were like, yeah, no, multiprocessing is a thing. We can't ignore it. So we're going to make this thing in such a way that all the operators are parallelism aware. That gets you much better performance.

So there's also generations of software engineering that went through this stuff, of course.

CL: Okay. From this small data where DuckDB fits in the memory or the browser that does a lot of interactive...

Hannes: On disk.

CL: Yeah. So I think from there, there's a period of time that you all focus on DuckLake and then make it multi-machine again. And then this is, I guess, counterintuitive, but it is a very interesting take. But tell us more about that.

Hannes: Yeah. DuckLake is something that we've come up with maybe one and a half years ago, maybe two years ago at this point. So in my mind, this is already ancient history.

A side note here: DuckDB, as you said, runs at such a speed that something that we kind of dreamt up two years ago is already, I don't know, done. You have to imagine, I'm thinking right now what we're going to release next year. So you had to imagine where my head is, right? It's somewhere else. But I will be happy to talk about DuckLake.

Now, DuckLake was a reaction to the Iceberg people basically building this table format for blob storage. We were like, great, that's a great idea, and then they added this catalog server on top that had the database to manage commits, and we were like, so you don't want to use a database, but now you're using a database.

Does this maybe have design implications on your previous design? They were very adamant in the beginning to not have a database, and then they added one later, but they refused to revisit their design assumptions from earlier that were very adamant about not having a database. And so we just said, guys, you should revisit those design decisions, and that's DuckLake.

We said, we have a database, let's use it. Let's use it for all these annoying tiny tables that Iceberg has to store in these wonderful Avro files that manage all these pointers in this metadata that you get to the final data and manage the deletions and manage the versions, and they have the JSON thing on top, and it's like manage the schema and all of that.

And there's also kind of small bits of data that are extremely painful to keep consistent, to mutate. And it sounds a lot like a database. It sounds like you need a table with a foreign key.

CL: Right. Everything is a database.

Hannes: Obviously, I like databases. So, if all you know is a hammer... But it sounded like all these mutations and consistency for Iceberg, for this metadata zoo, were very, very, very difficult and dangerous.

And then we were like, guys, this is a foreign key. We have 50 years of system design, how to enforce a foreign key. Can we just maybe use it instead of reinventing it, but this time on object storage, right?

That's DuckLake. That's the whole point. It's kind of a rant, if you want, a system design rant in the form of a competing idea. And to our great surprise, it's actually quite successful.

We got a lot of praise for DuckLake, I think. We have a collection of emails from people that were like, yes. It seemed to have hit a nerve. The Iceberg people, I'm told, are also thinking a bit about going into that direction now. Shockingly, after all these years of being adamant.

So that's kind of fascinating. And so DuckLake is an alternative for managing your data on object storage in a sane way, right? Where you have transactions, where you have insertions, deletions, updates, schema changes, all of that in a sane way. And it makes it possible to do that with a catalog database that just has tables in it and could be any database, really. Could be DuckDB, could be Postgres, really, we don't care.

CL: Yeah, this reminds me of, Mike Stonebraker was on a podcast earlier, and he emphasized that simplicity is always the key, right? Why are we reinventing this tiny file spreading all around?

Hannes: Yeah, Mike Stonebraker is, of course, our dear leader in databases. He's our only living Turing Award winner in databases. And very, very deservedly, of course.

I have a fun story about this. I went to the Computer History Museum. Have you ever been?

CL: I have not, actually.

Hannes: It's not far from you, I think. It's down in Mountain View. You should go.

Because they have a section on databases. And they have the original Hollerith machine, blah, blah, blah. But they also have a wall or a corner or whatever on relational databases, including an informational video.

And it's so fascinating because, of course, Stonebraker is in it. I think, pretty sure, is in it. But I had such a religious feeling in that moment because I was like, where I live in Europe, there's not a single museum of databases, I can assure you. So it was really so funny to go there and be like, oh, yeah, yeah, tables.

Yeah, that's it. And then it talked about Oracle and how IBM did the whole go-to-market fail with the relational model, and then Oracle jumped in on it, and then all that development and how Postgres showed up. And it was really awesome to see.

But yeah, it's tables all the way down. The idea that a database will need something else to database is kind of absurd, right?

Of course, you store your internal stuff in a table, right? You have something to manage tables in a sane way that can manage. Why would you not use it?

It's almost like saying, okay, we're going to build a compiler, but we're not going to build a compiler in the language that we're compiling. We're going to build it in something else because for some reason we don't trust the language you're compiling.

Maybe this is true for the JavaScript people. Fine. But it would be weird to have a C++ compiler not written in C++, I would argue. Right? It would be strange, let's just say.

CL: It's the first thing you do, self-hosted, right?

Hannes: Self-hosted, exactly. So if you're a database and you need to store information about tables, where could you put this information about tables? Tables.

CL: Yeah, of course. This is obvious, right? Postgres got this schema for all the things, right? Information schema and all that.

Hannes: Yeah, they do. Is there a difference between if something looks like a table or if it's also stored as a table? So the way I'm thinking about this, for example, it's kind of hilarious that the Parquet file format, the beloved Parquet file format, internally uses Thrift to store the information about the table that it stores. Could also have been a recursive thing, right?

Where you say, hey, it's just also a Parquet structure that stores the information about Parquet. But I know this, at some point, has to stop, but this is generally, I think, a safe pattern to apply recursion. Who would have thought?

CL: Sure, yeah. Or I guess maybe that's a weird idea for people like us.

Hannes: Yeah, I know. By the way, sorry, another fun side story. We have to rip out recursion in a bunch of places in DuckDB.

You don't know why? Why? Because in the infinite wisdom of the C++ compilers, the binaries they create have limited stack space.

CL: Sure. And then you're now tail recursion and then things will blow up.

Hannes: And then people send us queries that have a million opening parentheses. And then we ran out of stack space.

And so we had to spend a bunch of time making a lot of stuff, including our parser, from the obvious and clean and nice recursion abstraction to a not so nice, but extremely robust iterative construction. So even though we like recursion in our code base, recursion is problematic, let's just say. Just a side note. Sorry, I thought this was funny.

CL: Yeah. So we were, I guess, on DuckLake and digress a little bit. But I'm curious, though, because you seem to have a habit of contradicting yourself in previous things.

But still make things awesome, right? Tell us about Quack. It's like, we don't want client server, but, well, there is something you can use as well.

Hannes: Yeah. That's true. I'm okay with that.

I've made my peace with contradicting my past me. I think you make the best decision you make under the circumstances of the information, and then when that changes, you may have to adapt. And I would rather adapt than being a dogmatic grump for the rest of my life. I've met these people, they exist, that refuses to accept the fact that the world has changed.

I'll give you another example. I was very anti-agentic coding for the long time. And then, I don't know, a couple of months ago, somebody was like, you're being an idiot. Just give it a try. I will sit next to you until you have tried.

And I was like, okay, fine. And now I'm like, yep, that's it. So I can change my opinion.

On Quack, I think people should be able to change their opinion. You will still be able to survive. The internet sometimes is very harsh and people are afraid of being judged or whatever. But I'm like, guys, whatever. New information has come from the heavens.

So in this case of Quack, I said long time we don't do client server. I've written a research paper about how client server is awful because the client protocols of existing databases are all awful. The reason we caved, or I caved maybe, because this is something I worked on myself, is that lots of people started building this as a sort of plugin for DuckDB or something around DuckDB.

At some point, I think there were seven or eight GitHub projects that all basically retrofitted a client-server RPC kind of thing on top of DuckDB. And at that point, you have to be like, hmm, something is going on. There's a need. Obviously, there's a need.

And why is there a need? There's just use cases where the in-process model breaks down. And the most important one is concurrent updates to the same database.

So what DuckDB does is there's a DuckDB database file. And the DuckDB process or a process that runs DuckDB can lock that database file for modification. And then DuckDB has the whole MVCC and everything. It's not a single-sided thing. In that process, you can have a thousand threads modifying that database file at the same time.

Not a problem. It's all safe. But it didn't work to have two processes operate on the same database file.

And this works on SQLite because SQLite cheats and just locks and then the other side busy loops until the lock is lifted. And then it does its change in the other side busy loops. SQLite has zero in-memory state about the database. But that also means it's very slow.

So if you wanted to be faster in transactions, changes to the database, you do something like the ARIES-style logging, right, where you have a write-ahead log where you commit your changes to, you flush them to disk, and you have your in-memory state of your database that you modify. And then at some point, you checkpoint and then truncate the log and flush the changes that you made in memory back to disk. And that's all great. If you crash, you restart, you recover from the log, blah, blah, blah.

This whole thing, right? This is something that Postgres does, DuckDB does, MySQL does, Oracle does, everybody does the same thing because it turns out to be a good idea.

And if you have that setup, you cannot have two processes with write access to the same database file because both of them will make changes and they will write to different WALs. How is that going to work? And then they both have ephemeral in-memory states. Let's say the easiest case, you have a primary key and you insert the same key from both sides.

Well, that will not work. Now you have two databases. You effectively have to split. It doesn't work.

And so for this case, if you have multiple writers wanting to make changes to the same database at the same time, you need some sort of RPC thing. And so that was the original motivation to do Quack. Even if it's just on a localhost, it works really well, right? You can have a DuckDB process that acts as a server on this database file and you can have a thousand client processes, Python scripts, whatever, send their changes to this one thing and they will all serialize it, commit them, do all the persistent thing and everybody's happy.

And then later we said, all right, we're going to expand. The first version actually used local Unix sockets instead of HTTP. And later we expanded it to HTTP and found out that HTTP is as fast as Unix sockets. We were like, fine, we'll also do HTTP locally. But it was really built around this kind of user set, we want to do this.

We said we can't. At some point, it's not about me being right, it's about solving people's problems.

Again, not standing in the way of DuckDB's success. That's a very important pattern. And if I have to look stupid for five minutes, then so be it.

I'm perfectly happy with that. I look stupid all the time. I have small children, so...

CL: Okay, that's super cool. We'll get to that precision, but tell us a little bit about, because when DuckDB was getting popular, there was MotherDuck. And then I think there's some collaboration in between, but they also have hosted.

But now afterwards, you have Quack. And then how does the relationship work? And then what's it like to work with the different entity and commercial setting?

Hannes: Yeah, I think that worked out really well. So we have always had this split between Duck Labs, formerly called DuckDB Labs, the company, the commercial entity, and the DuckDB Foundation, which is a nonprofit that is the home of the DuckDB project. And DuckDB Labs was always separate from that. So we never had the IP in DuckDB in the company. The IP was always in the foundation.

And so that made it easy for us to say, if you're going to work with a commercial company like MotherDuck, we would do that with our DuckDB Labs, the company hat on. And I think that worked really well because you can do things companies do, like make contracts and charge people and provide services and do all of that. And you're not compromising your open source project.

And that's what we've done with a bunch of people, not just MotherDuck. We have had many, many partners over the years, including Amazon, by the way. They were one of our partners as well. And we always interacted with them through our commercial entity.

And that gives you a nice separation of concerns where just because somebody has a contract with the company doesn't mean they have control over the foundation.

And so that was the way we designed that. And I think that has worked out really well. There are a lot of companies now that run on DuckDB there.

You would be surprised. I can't talk about many of them, regrettably, but there's quite a lot out there. And yeah, we're interacting with them, of course.

CL: I wanted to maybe talk a little about what does it mean for the industry about this AWS acquisition. There's a history for big companies sponsoring open source project or acquiring that. And then usually there's a strategic thinking about, well, this enables consumption of their other thing. So what's your take there, what can you share, and then what's the kind of big picture here?

Hannes: Obviously I didn't make the decision to get acquired. That's the data, they made that decision, so you should ask them. I wasn't part of the company when that decision was made, let's just say. And it's only been a month. So I'm still getting started here. So for AWS, it's definitely something that they saw a lot of DuckDB use internally and externally. Because of course, we send a user agent when we talk to S3.

It says, user agent, DuckDB, so and so.

And so I have heard that this has popped up in their internal telemetry. And they're like, hey, we got a ton of accesses here from DuckDB to our storage service. Maybe we should talk to them.

And that's actually what we did. We worked with Amazon for almost two years before we got acquired as one of our development partners for feature prioritization in DuckDB. And we did a lot of work with the S3 tables team.

It's really great collaboration. And so when they then one day showed up and said, hey, Kevin, maybe talk about an acquisition, we were like, obviously, of course we can talk. We know you. We talk to you every week, right?

It wasn't super strange. And we could already understand why they cared and the way how they were thinking.

So I think this was, maybe the grassroots success of DuckDB has somehow made it to the corporate radar. If I had to guess, again, I wasn't involved. But I think that's how this got started, if I had to guess. So Amazon is very customer-focused. So if a customer comes and says, hey, I want to use DuckDB to talk to your service, they will take that seriously, right?

CL: So I think the database, well, it's just so fascinating because all the things centered around SQL hasn't really changed, but it also changed how this is used.

Hannes: It changed a lot, I would say.

CL: And then I think one thing you talked about before is that if you write pandas, you're writing the plan for how you want to query, but SQL defines the result first, right? And then do you think this is why SQL stopped becoming the machine interface or agent interface for getting data out of, because it starts with the result. This is what I want, right? And then go figure out how to get it.

Hannes: I'm not sure. I think SQL is just enormously part of the training data. So I'm not sure it's the best what an agent would pick if they had to design a query language. They'll probably see that in five years or five months, depending, I don't know.

But I think it's just that the inertia of SQL is gigantic, which was, by the way, also the reason why we picked SQL when we started DuckDB. We could have done something else. Lots of people did their own query language. SQL was declared dead many, many times over the years, but it's been around since 1972 or something like that. And it's stood the test of time in many ways, right?

And my general take is that what works for a human is also generally okay for an agent.

I think there's especially these text-based interactions. And you could also argue that a terminal is a terrible thing for a human and why have you ever forced anyone into it? Fair point. But that interaction of putting strings into a box and then the strings being interpreted and there being a formal grammar, I think that's generally something that works well.

It's probably easier for an agent to create a fairly compact SQL query than a 10,000 line C program to do the same thing, right? Especially if you think about all the complexities of scheduling, parallel processing, competent engines being around like DuckDB, right? There's something the agent doesn't have to invent of a Parquet parser, it can just use ours.

So I think there's just two things. One is that the training data on SQL is just immense. And the other thing is that what we forced humans to do kind of works for the machines as well, I think.

CL: And then I think there's an emphasis in DuckDB where the error message should be human readable, right? And then now it's machine readable.

Hannes: Don't tell me about it, man. So this is something, when I was a wee lad of 16 or so, I was in the local MySQL user group in my hometown. And I was always remembering this error message from MySQL. Maybe you know it. You have an error in your SQL syntax.

CL: Yeah, certain code, blah, blah, blah.

Hannes: Some code. You have an error in your SQL syntax, check the manual that come with your version of MySQL. And they wouldn't even tell you which version it was.

It would say, you have an error in your SQL syntax, check the manual that came with your version. Which one? We don't know.

Of MySQL, period. This error message, again, lovely that other people do some weird things because it creates opportunity for us. Awesome. So we spend a lot of time on friendly error messages. We say, oh, this table doesn't exist, but there's this other table that sounds just like it.

Do you mean this one? And actually, with DuckDB 2.0, we're going to ship a whole new SQL parser, completely written from scratch, to be even better at error messages, so we can do partial recovery and say, hey, you almost made it, but we expect this here. This query is so close to being correct, and that doesn't work in the traditional parsers that are based on an all or nothing kind of thing. So one of our guys, Daniel, he worked really hard on making a new parser. That was a research paper we published two years ago or so on how we should rewrite parsers for databases. We did that, and the result is going to be out in two weeks.

But it's going to be out in two weeks, and we're going to have a new parser. Just to be friendlier. And we also just merged, this was another one of my projects in DuckDB 2.0. We'll have an agent mode where we detect that we are running under an agent such as Claude and then change our behavior.

For example, we use half the tokens in showing query results because it turns out the agent doesn't care about formatting. It's a machine. So we can make a much more compact representation. We use the standard error stream to report progress.

We don't truncate. We tell you about truncation. We give you some way of exploring which functions are there.

There's a whole blog post coming out about that, about agent mode. And we just realized we went from user experience to agent experience. I'd like to claim copyright on the term AX. We've heard it here first. Because it is exactly what's relevant.

Because in the way the tool decision isn't done by the human anymore. People say, I want to analyze this Parquet file. They don't say, I want to use DuckDB to run this query. That's last week's AI use.

This week's AI use is, just open this, figure out something from this file. And then the agent makes the decision. So we have to make DuckDB better for agentic use cases.

And that's part of what we're doing. And luckily, it happens to be the case that what works well for humans, again, tends to work well for agents as well with some deltas, which is what we try to capture in the agent mode.

CL: So what I'm hearing is that your focus on the user experience now suddenly paid off because, well, there's a lot more users than agents.

Hannes: Absolutely, absolutely. I think that is an accidental win for sure. What we did made it easier for agents to use DuckDB, for sure. We get an error message that says, maybe try this.

CL: It can recover properly.

Hannes: Yeah, it can recover.

CL: Without looking at the manual.

Hannes: Yeah, the space is not like a plane with some peaks which are working queries.

It's more like it has gradients. You can be more or less wrong in your query. And that's good for agents to find out.

CL: Right. There is so much I want to unpack here. And then I think we don't have a lot of time left.

But I guess earlier you mentioned that you are now converted to agent decoding. What's the difference in using agents for coding the system like software and doing data analytics like using DuckDB to achieve certain goals? What's your take on it? Is it the same thing that you said, delegating, or are there very different working patterns?

Hannes: I think they're different in a way because I don't do a lot of data analysis, I have to admit. Data-based people don't have data problems. We just don't. Now, which data sets am I looking at?

Test cases, maybe. No, I don't know.

But I think that the systems work is more complex than running a SQL loop, I would argue. Your context for a SQL loop, it can be huge, but in general, I don't think it's that huge. You have your schema, you have some metadata about you, maybe some samples of your data, fine. But if you want to make a PR in DuckDB, a million lines of C++ that somehow are relevant to this whole thing.

And I'm quite impressed actually by specifically Claude, which was what I'm using. How I can say, do this obscure thing in DuckDB, and sometimes I have to tell it about it and be like, Google for this, or here's the doc page, or here's the file that you look for. But most of the time it's figuring it out on its own.

I'm quite impressed by the complexity that it can handle at the moment, I have to say. And I think that's different from a SQL prompt. You don't need a gigantic model for a SQL prompt.

You can use one of the smaller ones. I've had some success with locally hosted models, actually, just to create SQL. It worked really well. I don't think I want to write C++ with those models, though.

CL: Sure thing. So I think that we definitely live in the future now. It's like, wow, you can get a machine to generate code and queries or plan the theme to do things. So this is super exciting to see how DuckDB fitting so well, making sure the agent's using the best tool.

So thank you so much, Hannes. And then before we wrap, we're going to put in Data Renegades rounds, quickfire questions. It's like production's outage 2am, only short answers. Are you ready?

Hannes: I'm ready.

CL: All right, first one. Duck species that deserve release codename and hasn't had one yet. Spoiler.

Hannes: The Mandarin duck, it's Aix galericulata. It's a very good-looking duck, but it hasn't had a release yet.

CL: You say you didn't run out of queries, but when you do, what data set do you reach for a sanity check for anything new in the day-to-day system you created?

Hannes: Always TPC-H. It's very boring, but I know the schema by heart, so I don't have to think.

CL: Right. Most surprising place you found DuckDB running in the wild?

Hannes: Electric car batteries for telemetry reasons. That surprised me.

CL: Oh, wow. Okay.

Okay. What's one hot take about a database you're willing to defend on the podcast now that you also have a boss?

Hannes: I have a boss, yeah. I'm going to get yelled out for this one, maybe. So we started DuckDB because everybody was using really old database tech from 20 years ago.

But it's actually still happening. The number might have even gone up. Postgres was started in 96. What can I say?

CL: Okay. Your one non-negotiable in how a database system should be built?

Hannes: It's transactions because change is eternal. Everything flows. And if you pretend that the change is something that you can be eventually consistent or somewhat consistent about, you're mistreating your users, in my opinion.

CL: Okay, that might be a hot take as well. Someone in the data or database system more people should know about?

Hannes: I would say Thomas Neumann. He's a professor from the Technical University of Munich. He's a very smart man. And not many people know about him. So Thomas Neumann.

CL: Amazing. And what's a burning topic that people aren't talking about enough?

Hannes: There's so many. But my pet peeve is database protocols because they are just neglected and nobody looks at them. And they're terrible.

CL: Might be ripe for innovation.

Hannes: Except for Quack.

CL: Okay, what's one question we should have asked, but we didn't?

Hannes: You didn't ask me if I would do it again.

CL: All right, you got to answer it now.

Hannes: I would. I think the experience has been great. I have to say I'm really grateful to everybody in the team and everybody out there who has been part of the ride.

It's been fun. Jordan said once that if what you are doing grows exponentially, then every day is the craziest day of your life. And so today I'm on your podcast. And it's also true for today, I would say. Not just because of the podcast.

CL: That's so amazing. Thank you so much for being here with us, Hannes.

Hannes: Thanks so much, CL.