Why On-Device Inference Needs Custom Observability

The Unique Challenges of Mobile Compute

A significant focus in modern AI has been on large language models with billions of parameters. Billions more parameters mean bigger training investment, and presumably, more-impressive performance. But there are emerging use cases for mobile AI inference, like offline mobile text generation and the rise of general-purpose generative models for edge that hint at a growing need for reliable mobile AI performance.

Chris Karani, a longtime software engineer with specific expertise working in the mobile space, has built a variety of open-source, mobile-specific AI projects like Wax and Swarm, and is now focusing on building a mobile-specific monitoring and release-gating platform RYNO after observing the unique challenges that on-device inference poses to software performance on mobile.

Below, he explains why mobile inference needs a new breed of observability, and why on-device inference may necessitate a different way to think about app development: At the edge, moving inference on-device effectively introduces a variety of new hardware-specific variables, such as device generations, OS versions, and thermal and memory constraints.

How Traditional Observability Can’t Cover AI at the Edge

Karani explains that as a builder in the mobile space, he initially relied on popular tools like LangFuse and LangSmith for observability, but found that neither provided the monitoring coverage his projects needed. He found himself building RYNO, a flight recorder and release gate for AI in production on mobile, which focuses on privacy, safe execution traces from real devices, and turning production failures into regression tests to improve future releases.

“The project combines the hardware state with the end AI behavior to let teams debug and figure out why model failures happen when the apps are actually in production,” the founder explains. As an example, Karani cites a team he spoke with whose mobile app that identifies plants out in the wild and diagnoses them for issues.

Over time, the app team noticed negative reviews in the App Store from frustrated users who found the app misidentifying plants, but when the app’s developers conducted the same tests internally, they noticed no errors. “What they didn't realize was that a lot of their users were out using their apps in high ambient temperatures. This causes a variety of issues that the existing tools just couldn't catch.”

“For instance, once inference is running on a phone in production, changes in thermal state, memory pressure, and available compute can change how that workload performs in production. You can see slower execution, memory-related failures, or different behavior from what you saw during testing. RYNO comes in and is able to combine the hardware state with the model output and help teams debug why their models are failing when they're out into production.”

Closing the Loop for Mobile: Ragequitting, UX, and Regressions

For consumer-level mobile applications, this type of failure can lead to the ultimate cost: Users ragequitting the app and immediately deleting it. While the founder’s aim is eventually to target optimization opportunities for any edge computing setup, he admits that mobile is his current focus. “In mobile, most apps have a high uninstall rate.”

To avoid grossly inaccurate outputs due to hallucinations, Karani explains that his platform runs AI jobs across five stages: Capture, explain, replay, promote, and release.

• Capture: Utilizing the open-source, lightweight, privacy-centric Terra SDK, which can be embedded into mobile apps, and is built on open telemetry to prevent vendor lock-in, and records OS version, memory pressure, thermal state, and records OS version, memory pressure, thermal state, device class, and the compute resources involved in inference, such as CPU, GPU, or Neural Engine where available
• Explain: Utilizing the local macOS app’s ability to plot a visual timeline of device physics and model execution, identifying factors like correlating changes in device state with changes in model execution
• Replay: Having captured structured metadata, template IDs, token counts, and the precise physical hardware context at the time of errors, developers can use a replay button to recreate the conditions surrounding the failure to artificially reproduce the same environmental stress for testing purposes
• Promote: Teams can turn a one-off production failure into a structured, repeatable evaluation
• Release: From here, the platform connects to a team CI/CD pipeline to run various evaluations on different devices with that specific eval case prior to the next app release to prevent further regressions before they make it to production

The founder adds: “A lot of these issues also come in simple OS updates. Everything was working fine, then Apple launches a new update that introduces unexpected changes. And sometimes getting that stress test when you're testing within a controlled environment is difficult. So a lot of these issues can only be caught when the AI is actually ‘out in the wild.’”

“Sometimes you have to actually catch these failures once they're actually already out there because these systems are so non deterministic. They're not like traditional, deterministic software where you might have a really tight unit test case, so certain errors don’t reach production.”

“Because there's 100 million ways that the AI can actually fail. So, one of the best approaches is having this sort of loop that we've created to catch them quickly in production and ensure that they don't persist in the future.”

Building for Privacy and Memory Optimization

Karani explains that privacy was a key part of the project due to its initial focus on iOS. “We believe that the on-device AI on iOS is some of the best in the world right now. The Apple Neural Engine lets users run these really awesome models. But for iOS and mobile users, privacy is very important. It’s common even for experienced teams to accidentally violate user privacy while trying to collect the debugging context they desperately need.”

“This is how teams are silently trying to capture model regression with internal tooling. They captured a prompt and its output, and those contain extremely sensitive data, so you end up violating user privacy. To fix that for RYNO, we actually have a ‘never log’ list. No screenshots, no user prompts, no contacts lists, no physical addresses, no chat transcripts.”

“We use a minimal telemetry schema. Instead of logging what the user has typed, we log a structured shape and the operational health of the mathematical transaction. We've been using a lot of tools to ensure that teams are able to get the metadata that they need to reproduce these failures while still avoiding violations of user privacy in the process. We think this is super-important for on-device AI.”

Aside from privacy, the founder notes that going lightweight was an important architectural consideration to run successfully on edge. “Because of the memory pressure issues with the Jetson Linux, we made a key decision early on to build the core of the SDK in Zig. This gave us a lot of control over memory pressure and it gave us full control over how we actually want to architect memory use for the SDK, because on-device AI uses a lot of resources on mobile.”

“So we needed to build an SDK that's extremely lightweight. Because we’re running an observability SDK on constrained hardware, it would need to stay lightweight enough to not create another source of memory pressure competing against the model it’s observing. We think that for edge AI, debugging is something that is going to become a big problem. And it's totally different from how current tools actually enable teams to receive telemetry.”

The founder admits that part of his inspiration came from his own analysis of the AI space, which increasingly seems dominated by billion-dollar vendors pushing trillion-parameter models. “We believe wholeheartedly that edge AI is 100% the future. Smaller models are improving at a rapid rate. Vendors like Apple and Google keep pumping a lot of money in the space, and we think that the kind of benefits that edge AI provides are undeniable.”

“First, there’s privacy. Second, there’s improvements to latency, and third, is cost reduction. As the models keep improving, people are going to start realizing and taking this extremely seriously as we move forward. The space is still very early. But the world I see in the future is one in which we'll have on-device AI everywhere you go. There’s already on-device AI in cars, and we'll have robots that need on-device AI, for television, mobile, and so on.”

“Yes, the Cloud will always be where true intelligence, the larger computational intelligence, lives. But more and more companies are going to keep moving towards leveraging the on-device AI space as we move forward. And I think that's why Apple has also been putting a lot of effort year over year into the Apple Neural Engine and every chip that they release, prioritizing the inference power that they can actually pull out of the on device models.”

Founder Lessons, Enterprise Adoption, and the Future of Compute

For founders looking to build on-device AI products, Karani suggests that his own deep understanding of the space has been fundamental to his progress. “I think having a good understanding of how the hardware state actually works is important. The computation and the OS system attributes during inference time...that’s a different problem space that a lot of engineers need to understand if they're going into the on-device space.”

“Inference is one thing, input and output of prompts is another. But for a lot of these models, you have to understand they're not in a problem space that is similar to running on a large data center in Atlanta with perfect resources all the time. These are models that are running on-device with varying, different pressures in constrained environments. Understanding those environments is key to actually building the right systems for AI solutions.”

On the question of why device-specific telemetry still doesn’t seem to be mainstream at the enterprise level, the founder points back to how the tool ecosystem for edge computing is still in its early days. “There’s a lack of tooling. A lot of the enterprise companies we speak to are concerned over their lack of telemetry and understanding of how models behave when they’re actually out into production.”

“For instance, we were speaking to a medical company that was trying to launch a very privacy-centric, on-device model to different customers. But what they found is that they just couldn’t risk getting an incorrect diagnosis from the model (and not being able to understand why it happened). Existing tools just didn't cater to them. The space is so early and emerging and a lot of the focus is on cloud models.”

The founder notes that while high-end cloud models get a lot of attention, they’re also currently leading to a lot of untenable costs. “I think that's what's going to push enterprises more into on-device AI. And as [on-device models] improve, the solutions will become simpler, but we need to provide them with the kind of tooling that enables them to ship on-device AI reliably to their users.”

Will there be an inflection point at which everything flips from cloud to local? Maybe not. “I think people are going to have a balance of the two. Teams will leverage on-device models for certain tasks and route to cloud models when they need heavy computational work to be completed. I think the future will be a hybrid of the two. I don’t think on-device AI is going to completely negate the need for Cloud, but I think cost and performance are going to push people to try to understand how to leverage a hybrid of the two. And the majority of the work we do today will be pushed to the on-device models for inference.”