Ep. #5, The Voice AI Turing Test with Aoden Teo

Amir Zohrenejad: Super excited, Aoden, to have you on the Lab Notes podcast. Why don't you introduce yourself and tell us what you're working on?

Aoden Teo: My name is Aoden. I'm the co-founder and CEO of Miso Labs. At Miso Labs, we're working on highly emotive and realistic voice models. Our first voice model, the Miso 1, is open source on GitHub. It has about 3.2k GitHub stars. And right now we're working on our next model, which is a full duplex speech-to-speech model.

Amir: For people coming from more of the LLM space, or especially folks that are using coding agents, clearly those have in certain ways passed the Turing test now. But voice AI has not passed the Turing test in my view. When I'm interacting with anything that's clearly a voice model or voice AI, generally it's pretty frustrating pretty quickly and I can tell it's not a human. Why have we not passed the Turing test yet with voice?

Aoden: Yeah, absolutely. The key problem is we passed a Turing test kind of with scaling laws, right? If you look at LLMs, they've scaled massively. And if you look at the parameter count at which we kind of passed a Turing test, I think you might say, okay, maybe GPT-3.5 for most people. Most people wouldn't say GPT-2. And so you're going to say, okay, well, that was around 175 billion parameter kind of like Densum model. And with voice, almost all voice models are incredibly small. And so you're really looking at GPT-2 parameter counts, and so you sort of are getting GPT-2 error results.

And the reason all the models are very, very small is essentially you have a big latency constraint.

Especially if you're in a text-to-speech paradigm, for people who haven't maybe played around with voice AI before, the way, if you're using a text-to-speech model, generally things would work is that you have a transcription, then you have an LLM to generate text, and then you speak that text out with a text-to-speech model. So not only is latency already bad, you're going to multiply that by a factor of three usually. And so in the end, the models have to be very, very small to run very quick if you use a traditional, standard LLM transformer architecture.

And that forces you to be in a parameter count where it's just really infeasible to pass during tests, even in language, right? An 8 billion parameter or a lot of audio models or even less than that is going to be really, really hard to get good results out of it.

Amir: Got it. And so I think you kind of already alluded to some of this, but as I understand it, there's kind of cascade models where there's kind of three models chained together, speech to text and some kind of LLM and text to speech. And then there's voice to voice.

Can you just give us a lay of the land of where things are right now? When people are using voice AI right now in production, which one are they using? What do you think the future is going to be and what is stopping us from where we are right now to getting to that future?

Aoden: Yeah, absolutely. So basically, most production usage today happens in the cascade thing. So that means you're using a transcription, LLN, text-to-speech.

And the central reason for this is that the Melimbo one is much easier to do this in from training models. And so before you have good voice-to-voice models, this is going to dominate.

And number two, let's say you're building an enterprise customer support agent. What you want is that your agent should be able to understand all your enterprise processes. You should be able to give it a document that explains how to do everything.

And it's just way easier to hand over the document to an LLM. Now, there's been so much research done in LLMs that this is really easy and really trivial.

And so because you get to have the intelligence of the context from the LLM, the kind of text-to-speech pipeline, it's much easier to set up in production use cases like in customer support.

Whereas voice-to-voice, first of all, it's very experimental. There's not that many models out there that do it. So people don't generally use these things in production. However, they're obviously much better.

There's a couple of things. Let's imagine that you had a perfect text-to-speech model, which you kind of don't because of the latency constraints that I described earlier, but let's say you did. You kind of would still probably fail the Turing test if you just used a text-to-speech model there.

If you imagine the way a human being talks to you, the way I speak to you is not just conditioned on the text that I'm saying. It is also conditioned on the tone of voice that you've used.

Human beings match a lot of things from their interlocutor. They match their words per minute really, really closely.

And if models don't do that, maybe they're really realistic, but you're going to have some weird feeling. Like, this is really strange. This person doesn't need to be normal.

You could have a perfect text-to-speech model. But if it doesn't understand, if it's just not conditioning on the right things, then that's bad.

And then also how you say something and what you say are two very, very correlated things. You might not choose to say a particular sentence if it was going to be read out in a particular way. If you're going to yell something, it might change what you're going to say than if you're going to whisper something.

And so having these be two separate models is probably, in the end of the day, very anti-bitter lesson. And also there's clear reasons why you want to combine them to one thing.

But then, it's very obvious how to take all the research that you've done in LLMs and apply it to this cascade model thing. You just take it wholesale, change out your data sets basically, and then you've got it.

With voice-to-voice, it's clearly a much better paradigm, but there's a lot of things that you have to do differently. There's tokenization, which I think is one of the most difficult challenges.

There is: how do you get text context into these things? Because your voice-to-voice model actually does have to be able to take in text. There's no way you can deploy a production voice-to-voice model, but it's not able to read your enterprise processes for customer support, or it's not able to read your preferences and stuff.

So even your voice-to-voice models need to be able to interact with text in some way. And so that's all much more experimental. And so far, the research there is a little bit behind.

Amir: Okay. I want to dig in. So it seems your view is that voice-to-voice is the future.

Aoden: Yeah.

Amir: So I want to dig in, and I know your technology uses voice-to-voice going towards full duplex voice-to-voice models. So your guys' technology, you take the audio wave and you turn it into discrete pieces. Is that correct?

Aoden: Yeah, exactly.

Amir: And I know as I was reading, there are some approaches that try to take continuous and stay continuous. Can you walk us through how your approach works, creating these discrete chunks, what it means, why this approach versus continuous approaches?

Aoden: Yeah, absolutely. In some sense, I'll say that all the approaches are sort of isomorphic to each other. So the continuous versus the discrete approach, in some vague sense, they're basically the same approach.

So in practice, you can choose whichever framing helps you think better. But you can do different things in both spaces.

So the key idea with, let's say, discrete tokens is you cut your audio up into 80 millisecond frames.

80 millisecond frames is arbitrary. That's just what we do. You could do shorter, you could do longer. I think actually probably shorter is better.

And the basic idea here is what you want to do is you want to quantize this space. In practice, by the way, you're running this on a computer, so it's already kind of quantized. You're trying to essentially compress it. If you were to try to quantize this into a vocabulary and you want to get very, very emotive voice, that would be highly intractable.

If you've played around with Transformers, the vocabulary size is one of the dimensions of the LM head at the end. So if you have like a hundred trillion parameter vocabulary, you need to have at least a hundred trillion parameter model. So you need to do something clever to avoid this problem.

And essentially the idea here is you exploit kind of the combinatorics of a vector. So the idea is you cut into 80 millisecond frames. And what you're going to do — actually, you can do many things here — but what you're going to do is essentially learn multiple different vocabularies.

So in our case, we learned 32, so we'll just use 32 as a placeholder. So you learn, let's say, 32 different vocabularies.

And then what you do is the output sound is the sum of the token chosen from each vocabulary. And you essentially just run part of the model over and over again to get the full token. And that gives you essentially your vocabulary size to the power of 32 possible tokens without having to blow up your parameter count, essentially. And that's essentially the key idea.

It's very similar to something like diffusion, right? You're moving closer and closer to the correct answer. That's essentially what you're doing when you're generating the full token.

And in some sense, that's essentially what you're always doing, continuous approaches as well. And because everything does end up being sort of quantized on the computer, these things are very, very equivalent. But the key idea here is you have to do some kind of tokenization. And in particular, you have to do some kind of compression.

Compression is obviously super important in language models. Like byte-pair encoding, when we do tokenization, is essentially an aggressive compression strategy, so we don't generate letter by letter, which would blow up your context window, also to make it harder to learn things. So this is some version of compression. You're essentially compressing a continuous space down into a frame of 32.

But you want to do even more compression than that because suppose you have an 80 millisecond frame. And think about, if I'm going to say, "Hello, how are you?" That's a lot of 80 millisecond frames. And that contains very little information.

And so you can imagine your model reasoning out over that. It's going to be very token inefficient. It's going to blow up a context window.

And if you use your traditional solution to this, which is by parent coding, it doesn't work because, in our case, we have 2048 to the power of 32 tokens. And that's kind of just enough to be emotive.

I actually think we probably picked too small a number. It probably wouldn't even work. But 2048 to the power of 32, that's an insanely big number.

And the problem is that if you are going to try to compress this, you actually never generate the same token twice. So by pairing coding, you're looking for the most common pair.

And with voice, there's not even a notion of the most common token because you never actually generate the same token twice.

So this is another challenge if you design some kind of speech-to-speech model. You can get away with this in text-to-speech because you have some text and so it kind of saves you, but with speech-to-speech you really do have to solve this problem where you have to come up with some other strategy for tokenization.

Amir: Because your input space is so much bigger than text-to-speech, right, where emotion is not in text, right?

Aoden: It's not in text, which is the problem, but also it makes it so much easier.

Doing byte-pair encoding to compress your tokens being such a successful idea is not obvious to me. Obviously now we know this works, but I don't think it was that obvious at the time. And I think that with audio, the number of people who've thought about LLMs is several orders of magnitude larger than people thought about audio, so the solutions that we have are very, very, very new. You can essentially do what's called bag of words, which is essentially some sort of latent space reasoning. So this is a train time thing, but what you can do is, instead of predicting one token at a time, you can try to predict the sum of the embeddings of n tokens, usually 16. But in audio, you can actually push this even further. And you can actually try to do this at inference. And so you're essentially allowing the LM to learn some sort of compression.

Amir: This tends to work a lot better. So I think, and you're already talking about this, the technique that you guys are using is residual vector quantization. So you're taking that 80 millisecond clip and then you take it through, I think, 32 different vectors. And then in each of these vectors, is it just numbers that come out while you're doing the training, or do each of these 32 layers correspond to something that we can abstractly think of, like certain forms of emotiveness or anything like that? Or no, it's just—

Aoden: That's a good question. Basically, you learn these vocabularies. You have no guarantee that they're going to have anything nice in them.

Amir: It's a black box.

Aoden: It's a black box, basically. There is something you enforce. You enforce that the magnitude of the vectors goes down exponentially. So you do enforce the fact that the zeroth codebook, the first position of the vector, has more of an impact than the 32nd one.

And so in some sense, you're refining the error over and over and over again and getting closer and closer and closer. This is very important if you want to do good inference, because you want to be able to get away without generating the entire token. And you can't do that if the 30-second codebook is really the important one. And also, it's just much easier for the model.

Then the nice thing, though, is that you can actually see, did I accidentally learn something interesting? Do these actually have semantic meaning? And it turns out that they generally do.

You can obviously just generate the first codebook and see what happens. And it turns out that this kind of produces, essentially, phonics. So very low-level, kind of messy, but almost completely emotionless phonics.

And then as you do the middle codebooks, they tend to do tone of voice. But it's like talking to someone over the phone on a bad connection. So you get the emotiveness, but it's very fuzzy is what tends to happen. And then the final codebooks tend to refine the sound, so it's like sound quality.

But that's not guaranteed. That just seems to be what happens, which is nice. It tells you maybe there's something interesting about the approach. And it helps them. If the codebooks happen to learn something interesting, then that's much easier for the model to learn.

You train these codebooks separately for the model. So it's not like you take a big model, do gradient descent, and do great incentive in the codebooks as well; you separately learn good codebooks on speech.

It's kind of like tokenization, right? You separately do byte-pair encoding. You don't do that with the model itself.

Amir: And so there's tokenization, and then the analogy would be like an embedding space, right? So in the embedding space, when they want to explain it conceptually, they say, if you take France to England, and then you add the vector that goes to London, that will go to Paris. Are there similar mappings that exist?

Aoden: Yeah, absolutely. So you take this vector, and this vector is not really an embedding, right? Every point in this vector is an integer. And then you can compute an embedding from a token, essentially by storing separate embedding lookup tables for every vocabulary.

And then you have an embedding now for each position of the vector. So this has now become a vector of vectors. Then you can sum them all. And that gives you kind of the embedding of the full token.

And then basically, you can essentially do the same thing. And you can do vaguely mech interp steering type things with this, but this is not really mech interp.

It's like what you scrub. So you can actually take a man's voice saying something, take a woman's voice saying something, subtract them, and do this a bunch of times so you get an average vector. And then you can shift the embeddings by this amount. So you can do accents as well.

So you can take a British accent versus Australian accent, subtract it too, and you can actually find the embedding that represents Australian-ness.

So yeah, that's super cool. That's pretty fun.

Amir: Switching gears a little bit, I think in terms of that concept of the Turing test, one area that, and it's gotten better, but still is kind of lacking, is interactivity when it comes to voice models.

So I was actually preparing for this reading, I think it was a paper from the team at Gradium. They were talking about, okay, a few things that are missing are things like smooth turn-taking, back-channeling, which, I guess, I didn't know was the other person going, uh-huh, which I sometimes do on these podcasts. I'm like, I shouldn't do it because then I have to go filter it out. I go filter it out when someone else is talking to the audience.

Back-channeling, there's overlap handling and interruptions. And I was reading another paper from the Hugging Face team where they were kind of reverse engineering the OpenAI voice model where they were talking about a voice activity detector model that sits separately.

Can you just talk about overall interactivity? Are these the three things? Is there a finite number of things you have to solve for for interactivity to work? Or is it no more than these three things that I just told you?

That kind of voice activity detector thing, that seems like every time there's noise from somewhere, it would interrupt. It seems like a very hard thing to do. It's probably a bad solution.

Can you just talk about interactivity? What's the state of the art? What are the problems? And how are you guys looking to solve this?

Aoden: So interactivity, there's probably a lot of things in addition to that. Those are the three things that are obviously describable. But there's probably a lot of other things. Like turn-taking is a very rich thing, I guess.

It's not just about not interrupting someone — maybe you do want to interrupt someone. Sometimes it's about if you're interrupting someone, your tone of voice has to change. So the model doesn't know what's interrupting someone. There's lots of subtlety there.

Interestingly, I'm very bitter lesson-filled. I think that essentially all the problems here come from the fact that you're usually having a model behind the scenes. And then separate models that control your turn-taking.

These models have to be tiny because they're adding latency. And so they're bad. And they're not all in the same model. They probably need similar context.

Basically, if you take a model, just standard transformer model, and try to predict the next audio token. So this is the thing you would try to do when you try to do speech-to-speech. And you do nothing else, totally naively. You just try to predict the next audio frame and you start trading it with gradient descent. And you look at partial models before you've converged.

The first thing it learns is turn-taking, which I think is really interesting. It will be babbling, but give you highly realistic turn-taking.

It's not trying to learn one side of the conversation here. It's trying to predict the next sound of a podcast. It starts to do like, yeah, yeah, yeah. And laugh over, before it's speaking English. It's like vaguely English sounding sounds.

Does that mean that's what people do the most? It's the easiest thing to learn. So it's actually not hard to learn.

So the issue here is that if you're doing some kind of text-to-speech, or even if you're doing a speech-to-speech model that's architected in a particular way where it's handling the sound, but it's not handling deciding when to speak and so on and so forth, it's like an unsolvable problem, kind of. But the models themselves learn this very easily. This is not a really complicated thing.

And the key thing, in my opinion, is just a better lesson. You have to let the entire model handle everything. So the model also has to be handling when to speak.

Various VAD detection systems, right? The way these things kind of work is sort of like if I were talking to you.

But the way I talk to you is like most of the time you're anesthetized. And then suddenly every so often I zap you, you wake up and you listen to the whole conversation.

That's just not a very good way. You would not be a particularly good conversationalist if that was what was going on. And so I think essentially the problems are from a very awkward design. And the reason you have these kinds of awkward designs is you're trying to apply as much as possible LLM research wholesale to speech, as opposed to building your own stuff adaptively.

So those problems are a huge part of passing the Turing test, in our view. But I think you essentially just solve them with a bitter lesson. Your speech-to-speech model has to understand those things standardly.

As you do this, do the parameter sizes — do the models have to get bigger?

So for turn-taking, interestingly enough, even very small models learn the turn-taking. But obviously, for speech-to-speech, you do need larger models because you're handling some intelligence. The way I think about it is that a speech-to-speech model just needs to be big enough to cheat with Cluely.

So the idea is the model will talk to you and secretly be looking things up in the background and talking to you. Kind of like an engineer cheating with Cluelly. But if you're not smart enough, it's easy to build a really tiny model that is not smart enough even to do that.

And so that's the main challenge of speech-to-speech is you just want to be able to do reliable tool calling, reliably understand what you're supposed to do. There's obviously some challenges with reasoning with speech-to-speech, with the compression I described before.

So if you think about when I'm reasoning, or, well, when ChatGPT reasons over text, it reasons with a very small number of tokens. Whereas if I'm doing a leak code interview, let's say I'm trying to figure out some binary search problem and I start talking through my solution, I'm like, okay, so what we need to do is we need to figure out whether you're left or right and blah, blah, blah. The number of tokens I've expended to describe that is unbelievably high. So your tool calling will be inefficient.

One of the things that we've noticed really helps to do with the models is to essentially separate the model's thought process from its speech process. Every time you generate an audio frame, you can also add an additional token to the audio frame that doesn't correspond with any speech. And then pre-training, you ignore this. You don't train on loss for it or you set it to zero all the time.

But then in post-training, you can actually allow it to use this token to reason, essentially. And the point here is that it can be reasoning much, much faster than it is speaking.

And so that's kind of what people do, though. When I'm talking, let's say I'm trying to solve a hard problem or I'm trying to look something up in the background, my thoughts are way ahead of what's going on in my mouth. And I think that's a very, very important part. I think that's something you can play around with.

Amir: Yeah, that was going to be my next question. And part of what you're describing, I understand part of it, maybe I'm kind of struggling to understand. My brain is moving behind my mouth in this case.

And the question is mostly around: if you build a really good duplex voice-to-voice model that can't look up anything and it still has to be really small, it can be very emotive and talk to me, but it will be kind of useless.

Aoden: Yeah, and I also think it wouldn't be emotive because emotiveness does scale quite extremely with the parameter size. So you want it to be pretty big.

Amir: Yeah, but for it to be useful, putting in an LLM and being able to do things like tool calling or data retrieval, let's just stick to data retrieval. Let's just say RAG.

Let's just say there's a corpus of data that the retrieval is pretty fast, let's say that won't add a lot of latency, that this voice-to-voice model has to be connected to to retrieve some data, to put it in there as it speaks.

How does that work in a voice-to-voice model? Will you still have a speech-to-text that will still go do stuff in the background and will it just say, hey, hold on a second, and then wait until it comes back? Or how would you...

Aoden: Yeah, I think for sure the end result of the model is going to have to be able to interact with text, right? So even if the user is interacting with voice, the model does need to be able to understand text.

The nice thing about this, though, is that text is a really well-studied thing. So you can actually just reuse weights and stuff very effectively. But you do need to have some way for your model to read text. And you do need to have some way for your model to essentially output some text-like thing, right? That's a tool call.

And you want it to be able to do this kind of in the background. You don't want it to just be blocking. So reading something should not be a blocking process because otherwise you introduce terrible latency where it's like, oh, give me a second. So you don't have to be blocking.

Essentially, one way to do this is to tie its ability to do tool calling to it generating audio. So, for example, this is one of the main big advantages of adding in an additional token where it's like, this is the token where it's reasoning over, it's not being trained on perplexity, but it's able to use this, for example, to do tool calling. As it talks, it has to talk to be able to do things. And so that's when we can do tool calling.

And then in terms of taking in text, there's many different approaches that we've tried. They all generally kind of work because you can always just reuse text embeddings from stuff like Kimmy and things like this. And these are more than enough to do data retrieval. And so that's pretty easy.

There's some things you can play around with where you put them. You can inject them into the very start of the model. You can put them in the final layer and add in various different mechanisms.

Generally, these kind of final layer mechanisms seem to work a little bit better. But we've tried a whole bunch of things. I don't think we've exactly decided on one.

But yeah, you 100% need to be able to interface with text, right? It's like you're trying to build a human being, right? Human beings, it's a huge disability, for example, if you're not able to read, if you're blind or something. And so you don't want your model to have that. You want the model to be able to read.

Amir: Okay. So in your current release, you call Miso TTS the most emotive model in the world. How do you measure something like emotiveness? Are there kind of objective benchmarks on emotiveness, or is it all subjective? How do you make sure you get more emotive?

Aoden: Yeah, certainly it's all subjective, right? The goal of a voice model is to imitate a human or do something for a human. And so a human has to be involved somehow in the evaluation process.

We pick the word emotive. It's generally marketing speak. I can create a very emotive voice model that just screams at you all the time.

It's not clear what that word means. For us, the key thing is Turing tests. Basically, that's kind of the key benchmark. And there's various different ways you can measure Turing tests. These can't be done verifiably — you need to actually pay people to come in and stuff. But the way we think about it is that emotive voice models need to pass a Turing test. And then Turing tests also are kind of a continuous test.

Basically, you can sort Turing tests by how long the context of the Turing test is. It's probably very, very easy to pass a half a second Turing test. And it's really hard to pass a Turing test over the course of a year.

If you're talking or listening to something for a year, probably something weird will come up. And so the way we think about it is, I think this is a very useful framework of benchmarking things, is you look at the length of the Turing test that you can reliably pass with people. And that's essentially how we re-benchmark things.

Amir: But then are there different, let's say, axes of different emotions, different accents?

Aoden: There are. You can play around with these things with various mech interp things, or you can just do the raw embeddings.

You can actually try to make the model more emotional. You can actually do it with sparse autoencoders. You can boost things that are associated with different emotions and happiness and stuff. So you can do that. You can measure emotiveness.

I'm very less than pilt. I don't know if you can tell. And basically what you want to do is just throw tons of compute at something and have it figured out. You don't want to force it to do particular things.

And I think emotiveness is really cool. It's very important. One of the things that we think about in terms of emotiveness: it's very appropriate to capture the context-appropriate emotion.

One of the things we really focused on with Miso 1 is that if you ask it to generate a voiceover for a TikTok, it should do that in a really, really very different tone of voice from if you ask it to read out a eulogy, right?

So for us, what we think about in terms of the importance of a modernist is a mode of range, as opposed to really, really nailing one thing, but then being really, really bad for everything else. And for that, I guess it's important when you benchmark things that your Turing test involves lots and lots of different scenarios.

You can train a model to be really, really good at one specific thing and pass one very specific Turing test. But in our view, when you benchmark these things, you want to be as creative as possible and find as many different things and see if over a wide range of different contexts you're able to pass Turing tests. And for me, that's kind of what emotive means, right? It's like emotive range.

Amir: Going further up in the stack, now in agents built on LLMs, there's a lot of talk about what belongs in the harness, what belongs in the model. And I assume, obviously, most voice agents right now that are being released are being built on kind of that cascade model.

But overall, and you can take this on any direction you want, whether you want to talk about different applications of voice AI, but in terms of what belongs in the model layer, let's say what belongs in the harness layer, any views you have when it comes to voice AI and applications built on voice AI?

Aoden: Yeah, absolutely. The way I think about it is that most things probably are the model layer. If you think about what harnesses really are, there's kind of things that allow the model to interact with the world.

You can actually get a lot of improvements in the model from the harness. But generally speaking, I think those are temporary fixes. Like your model doesn't reason enough, so the harness will push it to reason more. Or the model is bad at figuring out which file it wants to open. And so you have some heuristic that forces it to do better.

But broadly speaking, I think where harnesses really shine is essentially giving the model access, easy access to different things and being able to interact with the world.

And so that's where I think that's true in LLMs. Although, when you build harnesses, a lot of people focus on trying to make the harness improve the model's performance. I think in the end of the day, most of the performance-based things should be model-based.

And so I think that's true of the voices as well. So voice emotiveness, that probably has to be in the models themselves. The model's ability to read and appropriately tool call and stuff like this has to be the model itself.

But harnesses are giving the model access to Fable so that it can look things up and stuff. That to me seems like a harness, essentially allowing the model to interact with the world. And a good harness gives the model the easy ability to interact with an arbitrary number of things.

Amir: Sure. To push on that, obviously, tools to interact with a lot of the outside world are going to be tools that are kind of plugged into the harness.

But what about something like memory? Obviously, as agents become longer and longer horizon, for them to be more than task doers or idiots and others, they have to have some kind of memory that kind of improves.

And obviously, I don't think each of us are going to continually post-train and fine-tune our model. So, when it comes to something like voice AI, is there anything in something like memory that has an interplay with voice AI that's different from the way people are thinking about it for coding agents and LLMs?

Aoden: Yeah, I guess my opinions on memory is that clearly we haven't solved memory. Clearly the models are very bad at memory. It's catastrophically bad. And so it makes sense to do stuff like various different memory harnesses that have to be in the harness because you have no notion of memory at all.

The way I think about this, though, is it's very likely that the models themselves will eventually solve this problem, right?

Amir: Okay. Tell us more.

Aoden: If you believe in AGI, right? Surely this will get solved. Either through something like better memory systems or with continual learning and things like this. That has to happen eventually.

So someone will figure it out. And at that point, that will probably be subsumed into the model layer.

So I personally wouldn't start a memory startup because that seems like a very anti-bitter lesson thing to do. And I like the bitter lesson.

You do like the bitter lesson. But obviously in the meantime, the harnesses have to handle memory.

In terms of voice, I think it's just very, very similar. Maybe in terms of when the agent's talking to a user, you want to record that somewhere.

And I guess that's now audio and you can always transcribe it and put into text pretty easily offline. So I think maybe there's a little bit extra that you might want to try in a harness. But I don't think it's that different.

I think with voice, the nice thing is that people talk a lot more and they write. They give you a lot more context. And so if you can get a user to be using voice, you suddenly have way, way, way more context that you can personalize things for with them.

In terms of applications, assuming that soon we're going to have kind of cross the uncanny valley, pass the Turing test and voice AI, any applications that you're excited about that these are going to unlock beyond things like telemarketing and customer support that already can exist? I mean, the way I think about this is that voice is essentially your pathway into emotional intelligence. A huge number of human jobs depend on this, for example, sales.

One extremely big application I think is if you can kind of do to sales what OpenAI and Anthropica have done to coding, not only is this very meaningful for sales, this is probably very meaningful for zero person companies. Because if you look at what, let's say, your average startup founder is doing today, if you talk to them, how much of your code do you write by hand? The answer is probably very little. So what is your job? Then they probably tell you some version of, what I do is I go and talk to customers.

I understand their problems. I sell to them. I talk to investors to raise money. And I talk to hires to join. Fundamentally, those are all emotional intelligence tasks. And the models don't seem to be scaling on this. And so voice is kind of your gateway into this. Emotive voices is the beginning, but you want highly persuasive voice. You want all sorts of different emotional intelligence tasks.

I think if you crack this, the big step change will be that suddenly you can plausibly have a company that entirely runs on its own without any human involvement.

Another thing I think really big is education. If you look at the models today, they're way smarter than the average tutor at math. But most eighth grade math students learning, I don't know, quadratic equations for the first time are not learning this from the models. And it's essentially an interface problem.

And so highly emotive voice, I think, is going to be a really, really big game changer for various applications of emotional intelligence like teaching. I think language learning is obviously a really big one. There's obviously lots of AI language learning already. But I think that will just put on steroids to be able to learn a new language in a month.

Amir: You guys are a kind of foundational model company. In terms of the process of training a voice-to-voice model and how that is similar to or different from training in LLM in terms of the different stages, pre-training, post-training, alignment, making sure it doesn't say certain words or certain things, what are those steps that you guys go through?

Aoden: It's essentially exactly the same. You have pre-training, which you have a big run, and then you have mid-training, which is basically pre-training, but you're trying to mix in some post-training data.

And then post-training, I think the biggest differences are reinforcement learning from human feedback makes less sense in audio because the speed at which I can generate speech is around the same as which I can review it.

Somewhat not true. You can listen to things on two times speed, but it's very hard to tell if the voice sounds good on two times speed. So you do less reinforcement learning from human feedback. You don't do none.

Yes, we do a lot of reinforcement learning. As I mentioned, we're interested in persuasion-based tasks as well. So we do reinforcement learning directly on persuasion-based tasks. Mafia is a good game to play.

Interesting. Are you guys sitting around the office playing Mafia all the time?

We pay people to come in and play Mafia. And it's a very signal-rich system because if someone gets voted out, every voting round provides a learning signal.

And it turns out, you don't have to pay people very much to play Mafia. No, they do it for fun. And then you give them a reward if they win, so they play hard. That's one of the ways you can post-train against persuasion.

Amir: You guys are an open-weight, open-source company. What was the decision to go do that overall?

In the kind of LLM space between all these open-weight models right now that people are going to post-train, fine-tune themselves for their specific tasks versus kind of the closed frontier labs, there's a lot of chatter, let's say, on the Twitter sphere. Anything different about voice? Do you think people are going to post-train and fine-tune their own voice models?

Aoden: I think I just kind of like open-source stuff. I don't have very deep thoughts. I have thoughts on it, but they didn't really influence the decision to make them open-source. I just thought it's a cool thing.

You should make it open-source so that people can play around with it. I think especially smaller models, it makes a lot of sense to have them be open-source because people can actually run them locally, play around with them, fine-tune them. Maybe a trillion-parameter model. I think an open-source trillion-parameter model is a very different proposition than an open source 8 billion parameter model.

The way I think about it is that there's lots of ways that open source voice models can be abused, obviously, for scamming people, impersonating people. But it doesn't have the same catastrophic risk as open sourcing a foundation model that could make a virus that drives us to extinction. So the risks are much lower with voice.

I think the rewards are probably a bit higher. Because these things are so latency constrained, you do benefit users quite a lot in terms of, if they want to, being able to run it locally if they have the money for the hardware. But then also, I think voice is pretty difficult to run locally.

So I think one of the reasons people don't open source models is they spend a lot of money training them and then they're afraid that nobody will actually pay them any money. But voice is quite hard to host very, very well. And so I think people are very willing to pay for hosted options.

And then there's very big economies of scale, particularly in voice, because your batch sizes and things like this, it makes it much easier to run economically and fast. And so I think the economic incentives for voice models tilt more towards — open sourcing something doesn't hurt your bottom line as much.

Amir: So actually I wanted to dig in more on that one, because I was going to ask, obviously something that's as latency sensitive as voice, is the inference going to move closer and closer to the end users?

Is it going to be device? Are we going to have a worldwide CDN running Miso Labs kind of models for inference that have to be close? Because with voice, network speed and network latency do become relevant, right? So imagine a few years from now, we have really emotive, interactive voice-to-voice models. What's the inference stack for that going to look like?

Aoden: At least in the near to midterm, you're probably going to have to run these on the cloud, especially because I think these are getting much bigger. How much bigger? So we're training a 150 billion parameter model.

I think even probably in the next three years, the models will be very large, like 500, 600 billion, probably. So these are going to get larger, there's clear returns from making them larger and larger. And so until it's plausible to run a five, 600 billion parameter model locally, which maybe happens someday, I think they'll likely be on the cloud. I think the inference stack, though, is you definitely have to have a very different kind of inference stack.

First of all, I'm sure some great companies will be built for this kind of inference stack, where you need to have local GPUs. You don't want the GPUs to really far away. The ping time between here and Singapore is really high. And so that would eat up essentially all of your latency.

And so obviously, if you get all the voice providers together, and they're all buying from one inference stack here, I think you could plausibly support GPU clusters near every kind of major city, and the request gets routed to the nearest GPU cluster. I think that will be very, very important.

I think real-time AI, even if it goes beyond voice as well, like robotics, have the latency constraint as well. I think most things will move over to being real-time. And so inference of this type is going to be really, really important too.

Amir: The final question that I want to talk about a little bit is languages, foreign languages. As you go from one language to another language, are there things that carry over? Because you guys are starting with voice-to-voice, if not like there's text in it at all, is training each new language starting from scratch?

Or is there anything in the base model that kind of carries over? Especially different languages, even the emotiveness might be different. Tell us more about different languages when it comes to very realistic voice AI.

Aoden: If you're training, there is some transference from language to language. So even if you train a model entirely in English, and then you train a model entirely in Spanish, but you use the English model as your base model, you'll get some transference. It's better than training from scratch.

However, what we've noticed, especially with speech-to-speech, is that having a lot of different languages in the pre-training set is just very, very advantageous.

Amir: It is advantageous. The bitter lesson strikes us again.

Aoden: Yeah, it does.

So basically, if you measure, you just do English, and then you do English mixed with other things, even at the same amount of data, it turns out that you perform better on English just having other languages in there. I reckon it's some edge of stability thing where it's like the data is more diverse or something. But that does seem to happen.

So in pre-training, you kind of want your base model to have seen a lot of languages. Then maybe you do supervised fine-tuning only in one language because that's your main market. But then at least you have a base model that can plausibly really easily be fine-tuned into many different languages.

Amir: Well, those are kind of the main questions that I had. I think this has been very interesting. We're super excited for what you guys are building.

And it seems like we're about the cross the uncanny valley and kind of the voice AI Turing test about to be passed. So super excited to see what's next.

Aoden: Thank you so much.

Amir: Thanks for joining.