Ep. #46, Canned Monkeys with Don Marti

Rachel Chalmers: Today, I am so happy to have my old friend Don Marti on the show. Don works on web ecosystem and open source business issues, including collaborative research on software incentivization and the impact of advances in consent management and tracking protection technology.

He has worked as VP of Ecosystem Innovation at Raptive, as a strategist at Mozilla and editor of Linux Journal. Don started the California Authorized Agent program at Consumer Reports Digital Lab that led to the development of CR's Permission Slip service.

He's written for Ad Exchanger, Linux Weekly News and other publications, co-authored a paper on the economics of software quality for the Journal of Cybersecurity and a book chapter in Advances in Advertising Research. He's been a speaker on open source software, user data targeting and signaling effects at technology and advertising events including Ogilvy Nudgestock, Southern California Linux Expo and the MSIX Conference.

Don, thanks for coming on the show.

Don Marti: Yeah, thanks Rachel. Thanks for having me on the program.

Rachel: For our listeners, Don and I go way back to the Software Freedom Day back in the 90s. Do you remember this, Don?

Don: I remember we had a phone conversation when I was leading a protest against Unisys regarding the GIF ("jiff") compression patent.

Rachel: Ah, it's "gif," we're gonna have to fight. End of a 20-year friendship. It's only "jiff" if you say it "ja-raphic."

Don: All right, all right, that's fine. We, I think, were "the first protest march over a mathematical algorithm" was the quote in one of the news sites at the time.

Rachel: It was so fun because I was a cub reporter fresh out of Australia and you were one of the first people I talked to who really was immersed in issues of software freedom. We didn't even call it Open Source then. It was before Open Source was coined as a term. But the rights of users to own and manipulate and iterate on software, it really opened my mind. It was a hugely formative time for me.

Don: Yeah, I remember at the time we were doing install fests where you could bring in your computer to the Cow Palace in San Francisco and we would install Linux on it for you. And we actually just did another one of those and I managed to get some of the old crew back together because of the whole end of support for Microsoft Windows 10. There's a new Linux install fest scene happening.

Rachel: Is it at Cow Palace?

Don: No, it wasn't at Cow Palace. We did it at the Alameda Public Library.

Rachel: Right, right. Again, for listeners who aren't familiar with San Francisco, Cow Palace is literally what it says on the tin. It's where they hold the Grand National Rodeo. So taking it over for an interactive LAN party is my kind of fun.

So, Don, this is our, what, third, fourth major platform shift in our lifetime. Now we have AI. Finally, HAL is here to throw us out of the spaceship. You've described LLMs as passing the Turing Test by bullshitting, which I love. Can you expand on that?

Don: Sure. The original Turing Test is really useful, but it's evaluating really two things. It's evaluating the ability of the AI to produce some plausible text, and it's also evaluating the ability of a human tester to see intelligence in some text input.

So because our brains are wired to see faces on Mars or interpret patterns, as there may be a person behind this, then some of what we get from the original Turing Test is the evaluator meeting the AI halfway. And so in a lot of situations, doing some plausible text is exactly what you need. If you're writing a thank you note or an essay for a class you're not very interested in, then...

Rachel: Or a letter of resignation.

Don: Or a letter of resignation. In corporate life, I would routinely use the AI for generating self evaluation for these semiannual processes that everybody goes through and nobody pays attention to.

But once you get outside that kind of core area of "give me something plausible, where what's important is the fact that this text exists," then you need to evaluate it by other standards.

And so what I'm really interested in is what are other benchmarks that can be applied to the output of an AI? And of course you've got things like flight simulators or continuous integration systems for software. And I would argue that markets are another kind of next generation Turing Test.

Rachel: Say more about that. I'm not really sure I follow what you mean there.

Don: Well, one way to define intelligence is: Is this system able to act in its own interest in some way? And so if you use market design to connect the interests or the score of the system to some output that you want to produce, then you can use a market as, call it an intelligence test.

So if you take a prediction market where humans and bots trade on "will this outcome come true at some point in the future," and then you give your AI some either real or play money on that prediction market, then what I would propose as kind of a Turing Test 2.0 is the AI passes the test if it can pay interest on the money you staked it with and it can pay rent for the computing resources that it uses. And then on that market it can earn enough income to stay solvent for an agreed upon period of time.

Rachel: Okay, I'm uncomfortable with that for reasons that are probably a bit career limiting for a VC. Certainly with all of my investors' money, I'm absolutely committed to delivering an agreed upon rate of return. However, I hate to see the many faceted thing that is intelligence flattened into maximization of financial resources.

The bee I have in my bonnet about the Turing Test is that I have a humanities degree and I see a lot of STEM majors saying, "oh, this machine is imitating poetry and fiction so convincingly."

And I read the output and, compared to what I've been trained to read and hold as a high standard in expression, it's drivel. I think a lot of people have too low a standard for what constitutes genuine communication. And that's why I think claims that the models are passing Turing Tests are obviously implausible.

Don: Yeah, I think you could define a Turing Test that was based on "communicate with someone who knows about a subject," about that subject for an agreed upon number of words. And there's a really interesting blog and I'll give you the link for the show notes that is done by a historian who covers ancient Rome.

And he's writing about the use of AI for student papers and puts in an example of a paper generated by AI to the assignment of, "explain why ancient Rome was so powerful on a military basis for the size of town that it was."Realistically, you're starting with a run of the mill town in the middle of Italy somewhere. How did they manage to put together such powerful armies back in the days when they were a republic and consolidating their control over that area?

And The AI came out with something that was really plausible to me because I don't know enough about that period of history. But, as a professor who actually studies that period, the author was able to find the substantial factual errors in what the AI was coming up with.

Rachel: I love that.

Don: Yeah. And that's Dr. Bret Devereaux is the blogger.

Rachel: I remember back in the day when you and I first met, the Economist wrote an article about the emergence of Free Software and Open Source. And I think you and I talked about it and about how everything the Economist writes seems so brilliant and insightful until they write about something that you're actually deeply knowledgeable about. And then it seems like, I can see how you would think that, but no.

Don: Yeah, and I honestly think there are two kinds of people. There's people who hear about generative AI and they say, "oh, this is interesting. I'll ask it to write a report about something that I understand really well to see how it does."

And then there are other people who say, "oh this is interesting. I'll ask it for a report about something I don't know much or anything about to see what it comes up with." And my theory anyway is people divide themselves into two groups based on which of those they tried early on in their experiences with it.

Rachel: Yeah. And I definitely did the first. And it's really formed how I think about what these transformers are useful for. I think part of what's a little bit mind melting about AI is a lot of it is massively overhyped. People treat it as an oracle, which it is not. And that's all very reminiscent of, to me, the blockchain world, which is consistently undelivered and feels like a pyramid scheme. Another career limiting observation for a VC.

But there's still this kernel of stuff that's really quite compelling and interesting where you can use it to generate your resignation letter or generate code that you can then iterate upon. And I do think it's the generative rather than the receptive side that's really interesting because you can train it on a body of knowledge. And that extension into the world, the ability to refactor an old code base into a new language, there's something really compelling about that core of the technology.

And then right over the other, there's all of these hyperbolic claims that people are making to prop up the market. And distinguishing between the two requires a lot of media literacy. But the existence of language models is eroding media literacy across the market. It's kind of a very ironic and fitting end for our civilization, I think.

Don: Well, it's sort of eroding what you might call the kind of GI Bill, mass literacy, mass education system. So the one thing I notice about the big cheeses of AI and big tech in general is how much they avoid what they're doing for their own families. So if you look at the families of the big cheeses in Palo Alto and San Francisco, you're gonna see very strict rules on screen time. You're gonna see private schools that are extremely manual and extremely instructor intensive. And that's way different from any other industry.

Rachel: Yeah.

Don:

If you go and talk to all the CEOs and top execs of the companies that make skis, they're taking their kids to the ski slopes and starting them young. Right? What is so different about this industry is the massive difference between how they see their space, their family space, versus what they want for the rest of us.

Rachel: And I don't know about you, but that feels like a big betrayal to me because coming from a humanities background, I got into software in the first place because it was that moment of the web democratizing access to the world's information and creating new opportunities and creating new connections across the world. It feels like we've turned our back on all of that possibility and moved instead to a model where we just give all our money to these eight guys.

Don: Yeah, and that's so much the difference between the web boom and the AI boom. I mean, I was here in the Bay Area for the web boom and I remember interviewing people from all kinds of different backgrounds.

I was in a Linux users group with a guy who had been a union welder and moved on and became a Linux systems administrator. I interviewed a person who dropped out of a grad program in history to do Linux consulting.

I worked with a manager who had actually been the gunner for that R2D2 looking machine gun on an aircraft carrier and decided not to stick with the Navy and come to work on Linux stuff. Of course, there's so many people in the arts and literature who knew enough HTML to become tech people.

So the web boom was so employment-positive and brought so many people into the tech workforce. Meanwhile, the AI boom is kind of the opposite. AI investments tend to be linked to layoffs kicking people out of the tech workforce.

So when I see this new report from Deutsche Bank about watch out, the AI bubble burst is coming, I keep wondering, what if that is actually good news for the broader economy? Because AI projects have been displacing so many other uses of capital that are more hiring-positive.

Rachel: I want to go back to what you said about the GI Bill because I think you tapped on a really deep vein there. One of the huge benefits of the post-war peace dividend, along with the Eisenhower highway system, was this enormous broadening of access to education, not only here in the US but in Australia as well, where we had free education and in England.

And I think that created a couple of generations that had a shared consensus reality around the principles of a modern liberal education. I think you're right that along with neoliberal dismantling of other public goods, AI is the latest flowering of the dismantling of that consensus reality.

Don: Yeah. And it's not just consensus reality, it's how many hops are you from real reality.

Rachel: Yeah.

Don: So going back to ancient Rome, I guess I was reading the foreword to a famous Latin textbook and it was originally written because so many US veterans were coming home from Europe and going to college on the GI Bill and for some reason they developed an interest in history and they're signing up for Latin class. So there was a mini Latin boom.

Rachel: Well, a lot of them fought in Italy, so they'd seen Rome for the first time.

Don: Yes, yes. And a lot of the territory that they fought over during World War II was the same territory that the Roman armies marched over.

Rachel: Yeah. History doesn't repeat itself, but it does rhyme. Later in the same post, where you talked about AI passing Turing tests by bullshitting, you talked about the cost of benchmarking as the biggest obstacle to an open ecosystem for AI. So what workarounds are there?

These handful of private companies have invested billions in training models. And we have some pieces of those, but the benchmarking is still super expensive to implement. How can we mitigate that risk?

Don: Well, there's two ways. There's the evil way and there's the way that I would recommend.

Rachel: Tell me the evil way, Don.

Don: The evil way of course is you just train on real interactions with real users, like the famous Microsoft Tay. You use unsuspecting participants in social media or in the chat features of video games as your source of training data.

Rachel: And in case anyone's forgotten this, it's hugely memorable to me because a friend of mine was closely involved in the project. Tay was a Twitter bot, definitely not named after Taylor Swift, that was just supposed to be your fun girly friend. And it took what, 48 hours before she was regurgitating Nazi propaganda.

Don: Yes, yes. And so there's this tremendous temptation to take shortcuts in not just getting text for the training set, but also in using real human interactions as your reward function. So if you want to train a large language model in a not very ethical way, you have it participate as if it were human and use its ability to attract interaction from unsuspecting humans as your benchmark.

Rachel: And then it gets red pilled and radicalized.

Don: It gets red pilled or radicalized or it vanishes into some high engagement, high deception, local maximum.

Rachel: 4chan?

Don: Well, 4chan or I mean look at the copy that works in email, spam or on Facebook. If you come up with a plausible story that some magic thing has been invented and these people over there are denying it to you because they want to control everything, but I'm going to tell you how to get it, that kind of story really works. It resonates.

And so you'll end up with large language models that are really good at telling people what they want to hear, what gets them coming back for more content, but content that's further and further away from reality.

Rachel: So please tell me there's a way to do this that isn't just feeding off of and then reinforcing the human capacity for self delusion.

Don: Well--

You have to find an AI benchmark that is low overhead and that can be connected back to reality in some way that's not just fair, but that appears fair from the outside. So that's where various software quality metrics and continuous integration systems and automated testing are really promising. Because you can say, I'm not just evaluating this AI on its ability to produce plausible text in general, I'm evaluating it on its ability to solve a particular problem.

Rachel: Yeah, I like the idea of flight sim. I mean, I keep coming back to your prediction marker. I totally buy the logic, but money is also an illusion. It's also a human artifact.

There is physics out there, there's a physical world, there's an environment with limited resources. I kind of want to push AI into those climate tech and health tech directions where, yeah, the consequences are life or death. But that seems to me to be a more powerful disincentive than losing money. A matter of temperament maybe.

Don: Yeah. And if you dig into accounting, accounting is a form of structured storytelling.

Rachel: Yes.

Don: And so you could say, "hey, AI your task is to tell me a plausible and entertaining story."

Or you could say, "AI, your utility function is to produce a story that is valuable to these investors because it fills the narrative conventions of accounting and earns money."

Rachel: I mean we're literally doing that using AI to generate the thesis for new venture funds. I see that happening every day. But if I think about insurance, that's a space where actuarial tables literally touch grass every day.

Insurance companies are not in denial about the physical reality of our world. I feel like making that the litmus test, making optimizing insurance outcomes a litmus test might be more generally beneficial.

Don: Yes, yes. If you had an AI that you could make a bet with and say, "all right, AI, you win, if I have more years of healthy life" and then that AI is then in a position to make lifestyle and investment choices for you.

Rachel: Damn it. Insurance is a prediction market. You beat me. You talked me around.

Don, what are you working on now?

Don: Well, what I'm working on now is trying to solve two problems with one solution.

So right now, a lot of organizations use a lot of software that they don't have that good of an understanding of because it's assembled from open source components from everywhere shout out to.

Rachel: The XKCD with the whole infrastructure of modern software and the little domino holding it up. That's this one guy in a flyover state.

Don: Yeah. And that one guy in Nebraska cartoon is actually the optimistic one because you don't know who that one guy is.

Rachel: Yeah.

Don:

The software component that's gonna break your business process is maintained by somebody two or three levels deep on the dependency graph that you're not aware of.

So we've gotten to this really uncomfortable situation where extremely large business risks for extremely large software using organizations are caused by human factors that they're not aware of. Particular issues of compensation, burnout, time investment for individual people can have these more than butterfly wing effects on large software systems. And we see that over and over.

Rachel: Incidentally, this is why I love investing in infrastructure software, because the reverse is also true. When infrastructure software is built generously and imaginatively, it lets teams that use it be generous and imaginative.

Don: Yes, yes. And almost all software teams want to do the right thing and support the code that they depend on at an appropriate level. And the problem is you're going to go to your boss and say, "we depend on the following open source. And we may have done a big effort to try to quantify that. We need to support this in order to prevent bad things from happening."

And management is going to come back to that and say, "oh well, yes, you want to do something nice, but you're not showing me a quantifiable risk that I can justify putting money into."

Rachel: So this is another place where we could ask some super intelligent AI, "tell me a story that I can tell my boss so that I can get compensated for working on this incredibly important and highly leveraged thing."

Don: Yes, yes. And the stories that we're trained to believe and to be able to take action on are market stories. And this goes back thousands of years. Many generations of farmers have participated in futures markets in order to hedge their risks of a bad crop or take profits on a good crop.

And people whose business depends on some commodity like grain or fuel will participate in the futures market in order to reduce their risk and to understand what's coming for them in the future.

Having a market price that moves in response to news is an incredibly powerful decision making tool.

Rachel: Yeah. Are you using AI tools in your own workflow? And if so, which ones?

Don: I have gotten started with the Zed editor, which is a really interesting constructive environment that will let me use multiple LLMs. So far I have not actually committed any LLM generated code, but I'm getting suggestions and visibility into things.

And the other part, of course, is that some of these newer editing environments are capable of hooking you up to other software quality tools. And in some cases, I think people are upgrading their environment to include AI. And the AI is sort of the pumpkin in the pumpkin spice here. What's really making the big difference in productivity is some of the tools that come along with it.

Rachel: I love that. It's stone soup.

Don: Yeah. And I'm reminded of what Linus Torvalds wrote about microkernels back in the day, where he said that yes, the microkernel developers are claiming that they can do microkernels in a high performance way, but really a lot of their performance optimizations that they're applying to microkernels are things that you could go back and apply to a regular kernel.

And so there's, I think, a really positive movement toward better tools for software quality in general. And the fact that some of those tools are being applied to help deal with LLM output is really a bonus.

Rachel: That's a perfect segue to the next question. Are you worried that LLMs are going to replace engineering? What advice can you give to people who are currently graduating into the worst job market I've ever seen?

Don: Oh, well, I do have a list of general advice for college graduates. But first of all, I think I would really personally start to be worried about LLMs if bot traders were dominating the public prediction markets.

So right now we've got this sort of re-flowering of the public prediction market scene at the same time that we have a bunch of AI developers looking for opportunities to brag on what their particular LLM can do.

And so I would start worrying about LLMs coming for me and the kind of work that I do when I see bot traders consistently beating human traders on public prediction markets. That would be my red alert alarm.

Rachel: And it's not happening yet?

Don: Not yet. Not yet. The so-called superforecasters are still human.

Rachel: Interesting.

Don: Yeah.

Rachel: What other advice do you have for college grads?

Don: Well, number one, don't let yourself get expensive.

Rachel: Perennial good advice.

Don: If you spend too much money every month, that limits your options a whole lot.

Rachel: But Don, I really like horses. It's a problem.

Don: Well, that's the thing. You made that choice. That's a good thing. That's a good thing. I could probably say there's stuff I like too, but it keeps your options open and there's a time in your life when it helps to have your options open and be able to take a couple weeks away or move or work on a project or start a thing or whatever. So it helps to be as inexpensive as possible. Yeah.

Rachel: The longest I took away from riding was when I was establishing my career, that's for sure.

Don: Oh yeah, yeah. Advice number two would be for new college graduates. Just finagle your way into things. There are a lot of websites out there looking for content and so if you tell a website editor that you can go cover some conference or event for them, you can then turn around and tell the event organizer that you're on assignment for this website and get yourself a media pass and get in and learn something and meet people and try to stick to that all important Freegan diet.

Rachel: Does this still work? I mean this was certainly how you and I got invited to a ton of conferences back in the day, but is it still a thing?

Don: I'm seeing students of certain professors doing it. So I think there are some professors out there giving the advice to their students of if you want to be ready for the real world, you should use your college affiliation to impress the website and then use that to get into a conference.

So there are also more formal opportunities to go to a professor. And very often professors have something really boring that they want to have done for their research like count the specs on this microscope slide or whatever. And if you do that, then you're a co-author on the paper and you can again finagle your way into things.

Rachel: Another hack I've seen, now that I think about it, is people starting podcasts to interview people who have the job they want. I've seen somebody who wants to become a fund manager just talk to every fund manager in the business on a podcast and that's been a really great way to build his reputation.

Don: Yeah, yeah.

Rachel: What else should the grads be doing?

Don: Oh.

Absolutely build stuff. Don't sit back and wait for somebody to tell you to build stuff. There are so many companies handing out free tools and easy to get materials and stuff that you can thrift or dumpster dive for and hack on.

Go to a makerspace.

Rachel: Start a makerspace.

Don: Yeah, go to a makerspace. Start a makerspace. There are a lot of options to develop your skills and then in the process of developing your skills, you're going to end up with good questions and people who know a lot about stuff like getting good questions.

So build the thing as far as you can. When you get stuck, you'll have a question and then that's your excuse to meet somebody who you're going to be better off meeting.

Rachel: Yeah. What do you think of conventional job hunting?

Don: I have always been terrible at conventional job hunting and I can only imagine it's getting worse. All the candidates are using AI to generate resumes and HR is feeding the resumes into their own AI. All I can think of is there's gotta be some way around it and you got to somehow get Internet-famous and then get people to come to you.

Rachel: Yeah, it's a rough time. What are some of your favorite sources for learning about AI?

Don: Oh, definitely look at the release notes for useful open source projects. Definitely read LWN, which is kind of like the super release notes for Linux because it's such a big project.

What I like to do is look at release notes, look at developer blogs, and then work backward to the tree that bore the useful fruits.

Rachel: Yeah, I can't claim that I'm doing this, but my partner Michelle, when we're looking at companies we might want to invest in, she goes and reads the GitHub repos and all of the comments and all of the published papers. It's a super rich way to get insight into what people are working on.

Don: Yeah, yeah, the reputation graph is still out there and we don't have as good tools for working with it as we used to, but it's still out there.

Rachel: Don, you've convinced me of the value of your prediction markets. I'm going to make you God-emperor of the galaxy. Everything is going to go the way you say it should for the next five years. What does the future look like?

Don:

I want to open up a news site in the morning, and the number one story is a person I've never heard of getting in the news for being right about something. And somebody who I've already heard of being wrong about something is not on the front page.

Rachel: Stop. This is the most beautiful vision of the future anyone's come up with. I'm sold.

Don: Thank you. Thank you.

Rachel: So your prize is a starship. It's gonna take generations to get to its destination in the galaxy. What are you gonna name this beautiful ship?

Don: I'm gonna find somebody who knows Latin and give it a classy name in Latin, obviously. Because if you really think about how long the ship's gonna take, you're gonna need a name that's gonna endure.

And then since I'm a Charles Stross reader, I would have to use the name, some variant on canned monkeys, but, turn it into Latin.

Rachel: That's beautiful. Monkeys in a tin. That's all we are.

Don: That's the premise of the whole "canned monkeys don't ship well" argument.

Rachel: If any of our listeners are fluent in classical Latin, please let us know what canned monkeys would be in Latin. And Don, so wonderful to have you on the show. Thank you so much.

Don: Thanks for having me on the show, Rachel.