Ep. #18, When Code Outruns Your Tests with Jake Schuenke and Kevin Clark

Meet Jake and Kevin of Phantom

Eden Full Goh: Hey, everyone. Thank you so much for joining. Really excited to have Jake and Kevin from Phantom here. Thanks, guys, for making time.

Kevin Clark: Thanks for having us.

Jake Schuenke: Great to see you.

Eden: I think this is a good opportunity to let everyone get to know you guys a little bit.

Kevin: Perfect. I'm Kevin, currently staff engineer at Phantom. Started my career in QA, moved to being an SDET, then moved to being an engineer. I've done front-end stuff, back-end stuff, have a lot of experience in fintech as a whole.

Phantom's my first crypto company that I've been a part of. The quality roots stay strong with me where I care about testing a lot almost everywhere I go.

Jake: Hey, everyone. I'm Jake. I started my career managing a fleet of Jenkins servers, running a bunch of Selenium tests back when it was cool. The team that I joined wasn't called the DevOps team, but we eventually became the DevOps team.

I've always enjoyed being the oil in the machine, helping people do their jobs more effectively.

I've also worked on a handful of different backend teams, but mostly been in the DevOps space. Because of that, testing also is really important to me. It's always been a part of my job.

Eden: I know the term DevOps only really became more popularized in the last five or 10 years or so. What was the name before DevOps came along? I guess it was just part of engineering?

Jake: We were called the automated testing team. And then I was part of the culture shift. I remember the first time we asked, what if we ran tests on every commit? And people's heads exploded a little bit. So it was a big culture shift back then.

Kevin: And I think the term gained more traction when we started doing more infrastructure as code stuff as well. Five, 10 years ago was really when those started taking off more. Chef and Puppet were some of the first ones. And that's what I think you started seeing a lot more of that type of role exist.

System engineers or reliability engineers or other names for it, I've heard. Infra engineers, another one, I don't know. We like slapping different names on stuff in this industry. Platform engineer is now a term that collects all of the different skills. I know Google's site reliability engineer is their DevOps role.

Eden: I think the industry has evolved a lot and we've come a long way. And even with the way that AI is becoming more prevalent, I hear new job titles and new team types popping up all the time.

So just so everyone can get a sense of the agenda that we have for today, I'll just talk through what we're here to cover. Really appreciate both of you joining. I think it would be really helpful for everyone to start more with understanding both of your backgrounds and your careers. I know we just touched on it in the intros, but definitely share experiences that you guys had prior to your current roles at Phantom.

I would love to hear more about the Jenkins servers he set up, Jake. And I think it would be great to give everyone that background. Then we have some questions around just industry trends that you guys have been spotting over the last couple of years and even months, I think weeks. Things have been changing so fast, honestly.

And I think the audience would be really excited to hear about your experiences scaling a QA program. And I think specifically just how it's different in finance and in crypto and for a company like Phantom, what you feel comfortable sharing. And then I would love to hear some reflections from you guys about lessons you've learned throughout your career at Phantom.

Lessons From Marqeta and Relativity

Kevin: For myself, prior to the job minus one from Phantom, I was at Marqeta. Most people have never heard of Marqeta, but they're a payment processor. They power Cash App, Klarna, DoorDash, Uber, a lot of the major fintech or fintech adjacent brands like Affirm, they also have a partnership there. Ramp is another one. They ended up powering a lot of these companies in a B2B sense.

And what I did there specifically was I was building out their credit offering that they wanted to give. But it's really hard to test money at the end of the day. Coming from starting as an SDET, my early career, Sling TV, for instance, that's really easy to test. At its core, it's like, cool, I clicked the video, did it play, and then you had to check for buffering and stuff. There's some more technical things with doing video streaming, but from a basic perspective, it's pretty easy compared to, as I said, anything that deals with money.

The moment you touch people's money, there's just more trust that's involved with it. And your company's reputation matters a lot.

And if you end up getting a reputation for, oh, it's always buggy or I've lost funds with them or whatever, it's a death bell. And so it's all the more important. Not saying testing is not important everywhere, but when you have the trust of your customer as a key part of your offering, I think it's even a little bit more. So Jake, you go.

Jake: I'll do it. My first job out of college was a company called Relativity and they did e-discovery and DockerD for court cases. So that interface for reviewing documents was super important to the point where seconds matter. And as they're discovering the most important documents for a court case, they tried to review millions and millions of documents in days sometimes.

And so the testing around that interface, that reviewer interface was super important. I think we had over 10,000 Selenium driven front end tests when I started. It was a bit of a beast. And I think that taught me a lot about tiering your tests.

Not every test is created equal. Some functionality must work. Some has to kind of work. And some is not as important as the rest.

And one of the things I built in my time there was a test flakiness measurement system, which is still relevant today as always. And that was actually a custom plugin built into the Jenkins server. And so I think a lot of the testing philosophy, as we move to more of a test on every commit, thinking about targeted testing and when to run certain tests compared to others, also became a skill set. And so that's where a lot of my testing experience came from.

I moved on later to a company called Pivotal, and I think eventually were bought by VMware. That was a lot of Kubernetes experience, being more of an infrastructure engineer. Then I eventually jumped over to a little startup that did live stream shopping. Also some interesting infrastructure issues, building tooling there, mostly custom dev environments, to now I'm a Phantom. We're managing all of the front-end tooling for our devs and making sure our quality is of the highest standards before we roll out a new release.

Kevin: Fun fact about Pivotal for everyone. They're the ones that were one of the leaders for TDD and pushing that as a thing. Yes, I'm a convert, true believer in TDD.

Eden: Oh, yeah. I didn't know that about you. That's awesome.

You guys have had a wide range of experiences in and out of the industry. Before you joined Phantom, were you guys involved in fintech and crypto in a personal or professional capacity? Because I know, in some of the work that you do, the tests that you have to build, the infrastructure that you have to build, it's very nuanced. And it's very specific for your use case. How much of prior experiences is translatable to Phantom, or did you have to pick up a lot of new knowledge coming into Phantom?

Kevin: A lot more of it was transferable than I thought it was going to be. I think a lot of the things that just in general make you a good engineer are abstracted away from any specific language or tech stack or whatever. And I think the same is true about testing strategy things as well.

I will say crypto has its own difficulties. In some ways, it's easier to test than a normal fintech application. In a lot of ways, it's harder to test as well. It just depends on what lens you're looking at.

But one of the things I did, for instance, at Marqeta was we had to load test our backend for credit. You have to build something specific to load test this in the sense of think how a credit card works, right? You swipe your card. There's an average number of swipes that people make a month, anywhere between 12 to 14. And then you have to do a whole bunch of stuff on top of that. You make a payment every month, maybe you get interest charged, you get a statement generated, and then the cycle continues again, right? And compared that, load testing something like some sort of messages API, like Twitter, right?

You can just bombard that endpoint with a million requests and that's what it's meant to do. That's how that endpoint works and there's not all these ancillary things that you have to add on to get a good relative test to how it would look in production.

But in any sort of fintech system where oftentimes there's multiple steps that have to be completed in order, you have lots of third-party integrations that you're dealing with, oftentimes it gets really difficult to test as close to prod as possible.

So you either have to invest heavily into your testing infrastructure, which can be a hard sell, because, well, it takes engineering hours to build the thing. And then if you do a half-off job building it, you have a really crappy, flaky test, right? So there's this balancing act that you have to do to try to maximize the quality you can get in a lower environment versus how much effort you're putting into your tests and effort you're putting into your quality.

Jake: Yeah, I definitely agree with Kevin on the whole is that the philosophy and strategies are pretty universal. Then the specific implementation of those strategies can be radically different. And particularly that balance of, well, I want to test in somewhat real world scenarios as left in the SDLC as I can that can be quick and make my devs happy. But then I still want to make sure I test the final as production, hooked up to everything, all of our third parties as possible. But then making that somewhat performant as well is the same challenge everywhere.

Also, personally, I barely owned any crypto before I joined Phantom. So on a personal note, it wasn't part of my world. And I thought the onboarding was going to be much more difficult, but all the same principles apply as would be previously in my career.

Eden: Yeah, it's a whole world, and now you guys are experts.

Testing in the Age of AI

Well, let's definitely talk about Phantom. How long have you guys been there, and what are you guys building right now?

Kevin: I've been here a little over a year now. I might have just crossed two years. Yes, it's two years. Right now, we're building all the stuff we're talking about, right?

AI is a crazy productivity multiplier, and more productivity means more code, means more builds, means more of everything downstream from it pushing the commits up. And so how do we scale out our CI? How can we keep our tests up to date with all these features that we're pushing? How do we get good, clean signals so we can release more often? Did I miss anything with that, Jake?

Jake: No, I think, like Kevin said, AI is changing how we work. So we're just trying to adapt our testing strategy to that new world.

But same principles apply. Have to say TDD never died. It was always cool. It's coming back in a big way.

And I think the same idea of doing testing as early in the planning process as possible, getting it all the way left is just as important as ever, especially with downstream coverage. And there's a lot of evidence to suggest that building that test coverage early to validate what's your agents are generating also more valuable than ever. So that's the challenges we're tackling right now.

One Wallet, Many Chains and Surfaces

Eden: And so for our audience who wants to maybe get a little more familiar with the Phantom platform, I know you guys have a native iOS app, a native Android app, a Chrome extension, there's a web app. What are other products and integrations that you guys have, and as your team as DevOps, you have to worry about all of that. I get to work with you on the mobile stuff, but I'm sure there's a lot of other stuff that you guys touch that I don't even have to think about.

Kevin: I definitely do not feel justified in taking credit for any of this. Surrounded by amazing engineers at Phantom. But there is so many integrations under the hood for almost anything that we're doing, right? Phantom's a non-custodial wallet at its core.

And really what that means is from a positioning of what we're trying to do a lot of times is be the thing that connects the user to whatever action that they're trying to accomplish on any sort of chain that we support. I think it's both the breadth of integrations that we have, but then we also have to deal with all these different permutations of that same integration.

Ethereum works different than Solana does. And we should test both because we care about both, right? Multiply that for every chain that we support in our app.

Multiply that for every provider that does a thing, like any on-ramp provider that we have. That's probably something that we should be testing. So I think that's where some of the challenges for testing Phantom comes.

And this is where, as I said, I can't take credit for it.

Our engineers do a great job at really, really walking through the app they're building the products out, making sure that the features that they're making work and they do what they're supposed to do. And then these different products also have to work with each other as well, right? Our browser extension needs to integrate seamlessly into our web app. The mobile app needs to be able to go into dApp browsers seamlessly. It needs to feel like there's no transition if possible. And so the UX experience of how that works as well, that's where another shout out.

We have some of the best designers I've worked with who this is their lifeblood. They think through this to such a deep level and it just impresses the hell out of me.

The QA Program Before and After AI

Eden: I'm curious, in terms of the timeline of the QA program, since you were both here before AI really started taking off, I think in the last six to eight months, 12 months, what did the QA program look like? And just your software development lifecycle before AI?

Because I'm sure there was more manual testing involved. You probably still had automation. And what does it look like now? And were you guys early adopters? Immediately when the best cutting edge models came out, you were already on it the first week? Curious about that process.

Kevin: I got to say, Phantom is very on the bleeding edge of trying to use AI in every way that we can. And this is where I don't have a concept of before AI, for me being here a year. AI was already a big part of the culture when I joined, so I'm going to defer to Jake.

Jake: It used to be a lot more manual. We used to have way more manual tests and we would rely on that, and a new feature would come out and we would have the engineers upstream mention new manual regression tests that we would add to the suite. You would have our manual testers validate those new features, usually. We also have a really good dogfooding culture at Phantom where we're testing things in-house really early in the process. And so that was what we would rely on as a safety net to make sure everything was working smoothly before we roll things out to our customers. Today, we have just way more automation.

We've partnered with Mobot, of course. And y'all have helped us with a lot of coverage, particularly around Ledgers that used to be very, very manual. And as someone who had to execute a lot of those tests for a few releases, sometimes very frustrating. And then additionally, we just have way more automated test coverage as well that we're continuing to build out day over day. But again, the same idea of the principles that still apply, that's still a feature team feeding in the requirements downstream to make sure we protect everything before it goes out to prod.

Eden: Maybe take a second to dive into the consumer journey and using the Phantom app and the Ledger experience and how that works for anyone who might not be familiar. Why is the Ledger such a key part? When you're developing, writing tests for it, thinking about how it integrates with everything, is it top of mind for you guys?

Kevin: Well, I think the principle of it is what's really important to Phantom, which is, as I said, non-custodial wallet, right? And this is, compared, thinking about a bank: you deposit your money into a bank and then that institution is the thing that's holding on to your money.

For anyone on this call who doesn't know, this is what a Ledger looks like. This is a Ledger Nano. And it just has this little fold out. And there's two buttons and there's a screen.

This has all my crypto on it. And so this is my bank. And I can be in possession of my bank.

And that's, from the consumer perspective, why Phantom's attractive, because Phantom doesn't hold on to a consumer's crypto.

And why using something like a Ledger is attractive is, okay, cool, I have this piece of hardware. It's secure. It's really difficult to hack into it. I'm not going to be bold enough to say it's impossible. I want to say it is, but I don't know, I didn't write it. I didn't build the thing. But protecting that's really important at the end of the day, part of that consumer trust that we're talking about, where the more control that you can give to the user, the better in that sense.

Jake: I think it also changes the game around how important quality and a good user experience is for us, because it is portable. Somebody can go and use a different app to manage their money.

That's part of the whole design. And additionally, people that typically are trading on Ledgers are people that are more integrated into our ecosystem and more invested in the success of crypto as a whole. So we try to make that a first-class citizen in our QA process. And I think something to call out, Mobot has helped us with a lot.

Testing with actual Ledgers in the warehouse with the robots was a use case that we didn't have total coverage on. And now it's something that I think we cover very well in our testing process.

Kevin: It's probably one of the more high confidence areas that we have at the moment.

Speed, Release Cadence, and Hotfixes

Eden: What are the things that are on the top of your minds for this year that you're hoping to focus on for the team?

Kevin: I think a big one is speed. Without degrading coverage, in a sense, right?

We want to be able to have the best signal possible for a given build with it taking the least amount of time.

And this is both from the sense of our release management process, right? The quicker we can test things, the quicker we can identify issues, and the tighter our feedback cycles are.

But for AI iterating on something, it'll sit there, push a build, it'll send it up. It wants to check, okay, cool, did this work or not? Oh, there's issues, let me come back and fix it. It just helps improve that cycle as well. And so the quicker that you can make the turnaround time, the quicker the whole process becomes, to go from idea to that feature shipping out.

Eden: You guys are currently releasing on mobile once a week right now, right? How often are you releasing for web or the Chrome extension or some of the other platforms, back end? And are you thinking of shortening that?

Kevin: For the most part, we're CD everywhere else. Other than Chrome extension, that follows once a week as well because that goes through a review process with the stores. But for all of our web, our edge function footprint and our back end footprint, everything's CD. So as quick as things can be validated, more or less.

Jake: As far as speeding up, I think the week-long process is pretty solid right now. I could see us speeding up in the future. But right now, I think it works really well for us to make sure we balance the speeding quality.

Kevin: I think in the case of hot fixes, if you take the store out, any review time with the store, I think we have our time to get a fix out to under four hours at this point.

Eden: How does the hotfix protocol differ from the normal release management on a weekly basis for stuff that's gated bi-weekly? So the Chrome extension and the mobile apps.

Kevin: Anytime you're hotfixing something, hopefully you're fixing a problem. I think that's the key thing. And that's at least what our process is here, right? We really only hotfix if there's an incident involved.

And just to be completely honest, it's a pain in the ass to fix mobile apps or extensions because you have the review process that you have to go through. So the protocol for the most part is you really want to make sure that the change is as targeted as possible. You want to introduce the minimum amount of code delta into the build in order to fix whatever issue that you're trying to fix.

That way that you can also make your testing as targeted as possible. Still do a broad smoke test of all the features and whatnot. But then we really try to focus on doing targeted testing on the area of change, right? So if we're hot fixing something for our swapper, for instance, we'd focus our testing on that part of the application.

Jake, I don't know if I missed anything with that.

Jake: I think the targeted part is the most important and how it differs. We just usually don't have time to run the full suite.

But again, where Mobot's helped us out, we now have a subset of our tests that we'll throw over to you all for assistance. And that was another big ease of mind, I think, for Kevin and I. Sometimes hotfix, you're really balancing: we want to give this fix out to our customers to protect them and their money. We also want to make sure we don't break anything.

As anyone in testing knows, knock-on effects are a thing, and unforeseen consequences that even the smallest change can happen. And so setting that up with you all, where we throw over our app and you run a subset of the test very quickly and give us a response back, it's just another safety net that we've added to make sure we don't break any of that.

Kevin: I think it's interesting too because the mentality is a little bit different than if you have a true CD backend. Because if you just have a fast pipeline to go to production in the middle of an incident, sometimes the correct thing to say is, let's ship it and see if it stops the bleeding. And you can make that decision because you can roll back instantly because you have the full control of the environment.

But that's where for anything that's running where you don't control it, so these mobile app, browser extension, these client applications, you don't really control when they update. You're just hoping the Play Store or whatever store they're getting it through just does it correctly.

And then if you make it worse with your fix, if you did a bad fix, it's really hard to roll back, if not impossible to roll back. And then they're just stuck there more broken until you get the next fixed up.

So I think that's where the biggest difference with front end client perspective compared to more modern CD.

Jake: And going back to the previous question of speeding up our process, I think the iOS store would be the thing that sometimes the reviews take a few days. It's really hard to get a fix out quickly. And having that on the tail end of your whole testing process is just a real challenge to consistently hit if we were to speed up, let's say, half a week. I think that'd be a real challenge for us.

Eden: I've definitely heard reports from folks in my network that the review times have been getting so long because everyone's generating all these AI apps. And I guess it's a lot of work for Apple and Google. They've got to probably increase the pace of that review. They must have a review agent, I would think.

Kevin: You would be surprised. They have automated checks, but it's still people that ultimately make the final review decision. At least that's my understanding of it.

Jake: For what it's worth, their expedited review is still pretty fast. For anyone who doesn't know, they have a way you can request an expedited review after you've submitted a new version of your app. And that's on the scale of a few hours.

And that's what we leverage for our hotfix is to try to get that time down. But of course, we don't want to use that system and then possibly get locked out of the store or something like that.

Phantom's Release Checklist, Layer by Layer

Eden: One more question about Phantom specifically, then I want to take a step back and talk about trends and industry stuff you guys have observed. But when you talk about the full suite or the full release management process, what are the checklist items? There's probably unit tests on every PR, there's component tests, API tests, there's your UI automation that runs on the mobile app, there's the process you have with us.

What are all the components? Is there still exploratory testing you guys do for your team? You mentioned you dog food a lot. What are all the different components?

Jake: I'll start with, I guess, the PR level. I'll even go further.

Our local testing. We enable our engineers to be able to build a full simulator build of our app. And they can even kick up to one of our third parties to get a mobile build for a real device. They can iterate on that locally, make sure everything's solid there.

Once we get up to the PR level, we have, like you said, unit test, component tests, all the normal CI checks. And we also do smoke tests on a real device right now before that gets merged into main.

And then as part of our release process, at the beginning of the week, we cut a release branch, we create a production build where we ship off to Mobot to run our full regression suite. We also have our automated regression suite. We also have our manual testers start their manual regression suite. And all that culminates into us finding bugs throughout the week.

And then once all of the most important or anything that's critical is fixed, we then submit to the store and roll out from there. We also have a tiered rollout system to make sure that everything's good, that we can quickly notice problems in production that we might have missed.

Kevin: The automated regressions, they run every night as well. And we're working on making that run more often.

And then on our backend side, same thing. There's a couple tiers of tests. And then we also have synthetic tests running in production to give us good signal of if things are working or not in production.

Load testing, we have that as well. I think this is one of those interesting things where AI has changed the equation a little bit with how you can approach a testing strategy, where before, let's go five years ago, you still wanted to automate as much as possible and you still wanted really good coverage numbers and whatever sort of metric that you're trying to measure your test coverage with. But one of your goals really as any sort of large scale engineering org is you want to reduce the maintenance costs of your tests as much as possible because it can be such a suck just on your resources. AI changes that equation, where you can actually justify having a little bit of redundant test overlap in different layers now. And almost, I think that's good. I don't quite know yet. We're still on the figure it out stage with how we're going to evolve as an industry here.

Because AI changes can be so thrashy, and they can impact so much stuff, and you might not pick up, oh, this test code changed with this code under the hood too. And okay, cool, CI passed, but then regression failed.

And so I think that's where one of these places now where it's okay, cool, we might want multiple layers of testing the same thing in slightly different ways to catch these different permutations of issues as they come up.

But you still have that, as I said, the maintenance costs with it. So you have to make sure you have AI automation to cover for any, oh, the selector changed and now five tests are failing.

Teaching AI to Write the Right Tests

Eden: How automated is that, having an agent maintaining tests for you guys, process been? How much coaching and nudging do you still have to do and refinement do you still have to do?

Kevin: It's an area that we're improving on. It's hard to get AI to write good code just in general. It has to have the context of what you want it to do.

So just thinking through a backend application, if I have an OpenAPI spec that defines my API that I want to build, that's pretty good context for the AI to know, okay, cool, this is what my data models need to look like, this is the shape, the code, it can just run with it. But then think about testing it. Okay, cool, give it the OpenAPI spec and it sees all the different APIs I have, but it has no context of how to use the thing. And so it can write an end to end test and it probably will give you something that works.

But is that the correct test for it? Maybe. Oftentimes within end to end testing, front end or back end, you have to do a series of steps to get into your correct test state up front. And as humans, we shorten that to, okay, we need an active user or whatever. But really, to have an active user could be 15 API calls that you have to make under the hood to get into that state.

And if the AI doesn't have context about that, it will never generate that code correctly.

So this is where I think the test cases, manual test cases themselves, are this huge boon to letting AI be able to generate these things correctly, in my opinion, because it gives the steps that it needs to know on how to get through it. But then you're still missing other bits of context. It has to know about selectors, has to know about any sort of library code you have to encapsulate the test. If you're doing a page object model type style, it needs to be able to know what is what, how does it relate to the app.

If you're doing some sort of visual based regression, you have to have a source of truth for things. So it's evolving for us, and it's a complicated process to do well, I think.

AI for Triage and Migrations

Jake: I'll add on two things that I think AI are helping us with a lot. One is triage. For example, that nightly run we're running on the daily, we got some feedback from our engineers, like, hey, I have to dig through these logs.

I have to go through these screenshots, watch the videos. Half of these, sometimes there are infrastructure issues that just invalidate the whole sweep that's run, et cetera. I've just spent a lot of time triaging all this.

And so we had an agent just do that initial triage, call out hotspots, call out where they should start. And it just reduced their morning cup of coffee activity from potentially hours to 15 minutes. And then they can focus on, all right, what are those two or three selectors I need to update? What's the flow I need to fix? Just saving them a ton of time.

Now, letting AI just completely fix that test itself, hit or miss sometimes, it still means that nudging, but that still saves us a ton of time with just the collecting information step.

Kevin: I would say on that one, it's really good at giving him first pass. Most of the time, it's 70 to 80% correct, right? Still needs a human to finish it, get it across the line. But I would like that to be, when that gets to 95%, 99%, then cool, great, you have a self-healing flow.

Jake: The other one I thought about was migrations. Something that Kevin and I even talked about in our fun discussions is how we're running the test, the framework we're using, as important anymore. And a migration that used to be, back in the day, a three month that we need all hands on deck, making sure these tests are good, running them, is now, hey, let's run these side by side, migrate a handful every week. And it's a lot. And we can do that for a 10% performance gain or stability gain, which again, would have been a full department sweeping effort, is now just, hey, we can just do this over the course of a few weeks.

And we still have to babysit the AI as it's doing the migration, but the overhead cost is way less than it used to be.

Multiplier, Not Replacement

Eden: I guess a lot of the learnings and everything you're navigating right now, do you think a lot of this is generalizable to other teams and companies outside of crypto and fintech? And do you have any recommendations about scaling a program specifically in your industry or outside of your industry based on AI and everything that's evolved in the last few months?

Kevin: My opinions on this are actively forming and they change day by day as new things come out. But I do think there's a lot of different ways different companies are approaching the usage of AI.

And I think ultimately it's a productivity multiplier and that's undeniable. It is an incredibly useful tool and that is undeniable.

It can write the same code that I can write quicker than I can write it as long as I guide it in the correct way. Right? And so it doesn't replace humans.

We're not at AGI yet. And I think we're seeing that in the industry. You can look at Ford recalling workers it laid off because the AI started acting up after they left, right?

And so I think when you're looking at your business, you have one of two options with AI. You say, I want to cut down my workforce and maintain my productivity at the same level. You can accomplish that. Say you have a thousand employees, go to 500, you can still stay at the same productivity level that you're at.

Or you say, I want to increase my productivity two to three X. I'm at a thousand employees, but I want to act like my output is at a two or three thousand employee company.

You can also do that. And I think the companies that are getting in trouble are the ones that are trying to do both, where they say, I want to increase my productivity and cut my workforce.

And maybe there's some specific industries that works for. But I think we're seeing a lot of companies pull back. And this is last month or so, hot off the press, is what's going on around the industry. And so I think that's changing a little bit.

Jake: Yeah, I would agree. I struggled to make a sweeping statement that I think is correct because the state of the world changes again day by day. I would encourage any engineering department to experiment with what works for you and your culture and your processes, because every department's different. And there's so many variables around what your STLC looks like, what your test coverage is now, even the languages that your frameworks are using.

Because AI needs the training set to be effective, that can be all the difference. A solution that works for us at Phantom might not work for someone that's written in a much lesser used language throughout the internet. So what works for us is we're just constantly trying to be data-driven and not afraid to experiment in small ways that don't affect the velocity overall. And it seems to be working for us in the long run.

Kevin: Maybe one more generalization I would add is when you add AI into your process, you have to be prepared to add it everywhere. If you just add it, okay, I only want it generating code, and you two to three X the amount of code that you're generating, but you don't scale up the rest of your SDLC to accommodate that, you don't scale up your ability to test that 2 to 3X, you don't scale up your ability to review that 2 to 3X, you don't scale up your ability to monitor that 2 to 3X, you're going to find something that breaks in your process. Something along there will break at some point. You have to be holistic about how you approach it.

Eden: That actually segues into my next question, which is, as you have been trying these tools and new QA strategies and continuing to build your QA program, is there an example of a lesson learned or something that broke that you think you could share?

Jake: The first one that comes to mind to me, and I've seen this from other people, is PR reviews. When you're pumping out a ton of code, to steal Kevin's metaphor, the pipe, okay, we can generate all this code, but now all of a sudden, we need to review it quickly and more effectively. And all of a sudden, that became our bottleneck. And so at the beginning, that was something that we had to, all right, can we have an agent give us a breakdown or a summary of what's changed here? Now we're exploring, can we be more targeted with our testing based on agentic analysis? And I think that was a big challenge where we had to start to really be creative of, all right, how do we get PRs just in front of people? Because there's so much happening now in Slack that it's hard to keep up with. And we've started to find a ton of remedies for that. And then it's sort of like whack-a-mole. There's another part of the process that needs to be scaled and figured out. Because if you scale your output to 4X the size of your company, all of a sudden, everything else has to 4X.

Audience Q&A: Weird Bugs and Cheap Phones

Eden: It looks like there's a couple of great questions in the Q&A here. So in the last few minutes of our time together, we'll transition to this. First question, it looks like, is: what's an example of a challenging defect you believe was found due to your process and tools? Any fun stories there? A cool bug you found?

Kevin: We've made some architectural changes with our app under the hood this year. And a big part of this is to really try to go after performance wins in the application itself. So I think the more we've instrumented the app and the more that we've added tests around the app, there's been some really weird performance scenarios that have actually come up, where just the way that the homepage loads, for instance, the bottlenecks that we would find of, okay, we have to make 30 API calls or whatever it is to get all the data to load the homepage. But then if feature flags failed, for instance, it would sometimes just cause the app to get in this really weird partial loading state.

It's hard, because we actually find a lot of issues early. So it's hard for me to point at any one specific weird or fun one.

And this is also where just another huge shout out to the devs. They're really good at finding a lot of these things even before it gets into the release process itself, I would say. So not a really good answer necessarily, I think. But I don't know if you have one, Jake.

Jake: I think the interesting one for me going back to performance is, we try to support as many phones as possible, particularly in the Android ecosystem. And that can be incredibly challenging. We support every device under the sun, every version, and that just leads to a lot of strange bugs.

There was one the other day. We had a specific version of Android on two devices. And it's just easy to be curious.

I'm like, why aren't those two phones on this specific thing? Are we crashing all of the sudden? And I think those in particular are very interesting to me.

Kevin: On that note, I want to flip it back to Eden. Have you ever heard of a Tecno Spark before we asked?

Eden: No, that was the first time. I think it may be heard of the brand, Tecno. But yes, actually having a company ask to test specifically on this very basic Android phone was a new true.

Kevin: I believe the phrasing I used was, we need you to test on the shittiest phone you have in the office.

Eden: And it turned out we didn't have enough crappy phones. So we needed to specifically go get one that was even crappier for your needs.

There's a couple other great questions here. We may not have time for all of them, but I'm also going to try to just interpret the meaning of some of these. I think one attendee wanted to understand there's a lot of complex industry knowledge, niche domain knowledge, knowledge base that testers and automation engineers need to have when you're onboarding new employees and when you're training an AI agent. So at Phantom, what is the process if someone comes in as a new team member or an agent who doesn't know anything about crypto? How do you teach them all of that and get them embedded in the program?

Kevin: I'm laughing a little bit because I think this is just true of fintech in general as well, where there's just a dictionary of language of made up words and acronyms that you don't use in a normal sentence, and you use multiple times a day when you're talking about this stuff. So I think Phantom does a pretty good job at, here's the basics of crypto, if you're not familiar, just from internal training that we have.

But just generically, it's kind of got to read and learn. Many times when I was onboarding, I asked Jake, I'm like, what does this mean? Just having that person that can help guide you through the random acronyms is mostly what it is if you're not familiar with it.

And then I think having that culture where it's okay to ask questions, right? That's always something I try to foster, at least. I think expecting everyone to know everything is really difficult.

And I think that's even more true with AI, right? It's really easy just to have AI explain a thing to you now. And generally, it's pretty good about this sort of stuff.

Jake: I'll extend what Kevin just said. I personally use agents for a lot of onboarding. I have in the agent definitions to be verbose and explain things and assume that I don't know.

I actually just recently came up with a skill called PR Like I'm Five, which will explain the architecture and the background and domain of the PR you're trying to review. Because there are times when I'm like, what even are these words that are on the screen? And that's helpful. So I encourage everyone to use the AI friends to help explain things. That's very powerful.

Kevin: On that note, it's really useful if your company sets up a core knowledge base of all the common stuff and then you can point your AI at it, right?

That's one of the things that we have at Phantom. I can't take credit for it. Very talented engineers set it up.

But we have a core MCP server that you just connect to and it has all our MCPs on it. And it has access to all the data through that. It makes it more consistent for everyone to have the same set of information.

Hexagonal Architecture and Slow Unit Tests

Eden: Next question is, have you guys used hexagonal architecture? And if so, have you found it helpful for separating fast running unit tests from slower tests?

Kevin: Hexagonal architecture. No, that's a new one for me. I'm looking it up right now. What is this? I didn't know there's a name for this.

Eden: We've got some industry experts in the audience today.

Kevin: Yeah, you do. So I didn't know the name of this. I'm looking at one picture, so I could be horribly wrong. But this honestly just looks like good design in a lot of ways.

And it seems about having specs. I assume there's a bit that says that everything should be driven by specs. I sure hope so, or this makes less sense.

Eden: The second part of the question was separating fast-running unit tests from slower-running unit tests.

Kevin: Yes, and that's just a thing regardless. Okay, I'm going to harp on unit tests for a second a little bit.

Anytime I see a slow running unit test, it actually makes me think that unit test is wrong. And that unit test probably shouldn't be a unit test, if that's what it's doing.

There are some cases where you have slow running unit tests, right? Especially when you get into parametrized testing. And you're like, cool, I want to do this one function with a thousand permutations. But each cycle is really quick if you do that.

I think when you have slow running unit tests, it's actually a sign that you're testing too much. And it can often be, this should really be what everyone call a functional component or an integration test. And I think there's a lot of value in, especially when thinking through backend systems specifically, depending on your framework, but being able to test slices of my application where I don't have to spin up the whole thing.

So that's my hot take. I think a lot of unit tests are just bad.

One of my AI skills I have, it's actually an agent. And a version of this agent is part of our PR review agents that we have, but it's my angry test reviewer. And I have it be very critical of any unit tests that are mocking too much or are not actually testing actual logic.

AI is bad at generating unit tests. It will give you this test that it mocks every single call. And then you look at what's actually being tested. It's like, okay, I have this one if condition. That's the only thing that's not being mocked in this function now.

Jake: Related to this though, with the recent big UI rollout that we did, our front end is much more modular. And so we're exploring, at the PR level, instead of running that end-to-end test, we can build a smaller version of our app with just a single module.

Kevin: We also have one of our engineers made a library called VitestMobile. And we can actually run our unit tests inside of the app running in the simulator. And it gets this really powerful component test.

So we're a React Native app. So we can render the full component as a part of that unit test in the environment that it's running in. And that's been a really powerful tool for us in conjunction with more modular UI to just have a lot better coverage, I think, at this fast layer.

Last Question: iOS Versus Android

Eden: All right, I think we have time for maybe one more question. And it looks like this question was a follow-up from maybe us talking about crappy Android devices earlier. Audience member asked, do you see clusters of different types of defects on iOS versus Android?

Kevin: Oh my gosh, yes. Yes, we do. iOS generally doesn't have issues. And you can thank Apple. They have very, very tight controls about the environment.

Android, these phones that have these shitty MediaTek CPU-GPU combos causes the most pain. And it's because you have this quad-core, 1.2 gigahertz processor that's trying to make a ton of network calls on startup. And it's also in conjunction with often where these phones are, they're in the parts of the world with crappy internet.

And so it's slow internet, slow CPU, man does it cause a lot of issues, and it's mostly, as I said, the phones with these CPUs. And Tecno Sparks are one of them. There's a couple other manufacturers that use them as well.

Jake: I will say it is frustrating to have all those different use cases. Also really interesting sometimes. And I think it really connects to our mission of giving everyone access to crypto and these markets and our apps. So we're very dedicated to trying to support as many phones as possible because of that. But man, sometimes the weird scenarios we see are wild.

Kevin: No joke. We were trying to reproduce a crash on a Tecno Spark. This is six to eight months ago.

And no joke, we were considering putting the phone in the oven for a little bit to try to get it to heat up, to worsen the performance, try to get it to crash and emulate being in a hot environment. That's the extents that we've been trying to go through to replicate some of these issues that we see from our analytics.

Eden: I think in seeing how you guys work, one of the things that's really impressed me and stood out is how committed you are to ensuring accessibility across device models, user demographics, countries. I've heard of teams that, oh, yeah, we're just dropping support for iOS 17 and below, not even going to try or whatever it is.

But I think you guys have made a very deliberate decision to be inclusive. And that is extra effort. But I just wanted to say how impressed I am with how much you guys have approached that and how Phantom approaches everything in general. And it's a pleasure to get to work with you guys.

Kevin: Thank you, Eden. And likewise, we really appreciate the partnership with y'all. Jake is the one that sang your praises earlier, but I think quality is important.

As I said, the trust that the consumers put into our app working for them is really important. And so any step that we can do ahead of time to try to make sure that we don't ever violate that trust is something that we want to do.

And Mobot is critical as a part of that.

Eden: Thank you guys for joining. If you want to learn more about Phantom, feel free to check them out. Get a Ledger, install the app, try it out as a consumer.

If you want to learn more about Mobot, check us out at our website. And if anyone needs testing with real robots, we're happy to help.