Is Early Detection the Key to AI Performance Engineering?

More AI-Generated Code = More Software Performance Problems?

Studies suggest that some 88% of organizations have adopted AI, with some hyperscalers claiming that as much as 75% of their new code is AI-generated. Does more AI-generated code mean more performance issues and software regressions? Without guardrails, yes, says Arthur Pastel, co-founder and CEO of CodSpeed, a performance intelligence platform built specifically to help teams detect regressions and fix performance issues earlier in the SDLC. Today, CodSpeed is used by teams at OpenAI, Cloudflare and Vercel as well as leading open-source organizations like Mozilla and Pydantic.

The founder, a lifelong coder, explains that the project arose from a longtime interest in software performance. With the growing popularity of the Rust language, Pastel observed that while many interesting tools were being built, there was still no way to measure performance continuously in CI, which inspired him to build the first version of what became CodSpeed, and to found the company.

CodSpeed founder Arthur Pastel discusses software optimization at EuroRust 2025. Image courtesy EuroRust

Making Performance Visibility Continuous

Pastel suggests that the core problem that drove him to design CodSpeed is a lack of continuous visibility into performance during the software development life cycle. “Prior to CodSpeed, you really just had point-in-time performance data points.”

He explains that on a team, one member might run benchmarks, which are local and not committed, and because they are run on one specific machine, they aren't easily comparable with other benchmarks.

The second problem is a lack of continuous gathering of performance data. “For example you might have some production performance data, like in a continuous profiler or a production observability platform, but you don't have that data on a specific set of input data. So essentially, you can’t really track it over time and see how it evolved. (And that’s before you introduce factors like regression.)”

“This is the first problem we solved. Then we quickly figured out that people don't want to spend time fixing performance issues themselves. So we went further: We built a tool to surface the problems, and now we also remediate them and improve performance.”

Performance Engineering Matters Everywhere, Even in OSS

Pastel notes the importance of regression analysis goes beyond massive commercial software packages and is growing in importance among the open-source community. “It is really easy to convince people building open-source projects [of the importance of performance engineering] because they’re building foundational blocks.”

“So for example, if you introduce a regression in Pydantic (which is used by LLM providers) or in a really low-level library used everywhere, even a small performance regression will cascade everywhere.” Pastel points out that popular open-source projects are getting hammered with many more pull requests than before thanks to AI contributions, sometimes as much as 10x, which makes visibility into the performance impact of contributions extremely important for maintainers.

“Otherwise, in the beginning, we primarily spoke to organizations that were really sensitive to performance changes, which might be for UX or cost of infrastructure.” Orgs running infra at scale will generally be concerned about how performance affects their cloud bill, but LLMs have introduced more variation into the quantity (and quality) of things teams can “generate.”

“If you don't have any guardrails, then your performance might get completely out of hand, and you won't realize it until you're actually running on a production database.”

Accounting for Benchmark Noise and Memory Usage

Pastel explains that his team’s approach involves building a CPU simulation tool to address the 'noisy neighbors' issue of running on a shared VM whose measured performance might quietly change depending on what neighboring VMs are running. “It can be really painful, especially in overprovisioned environments like GitHub Actions and a lot of CI runners.”

He explains that a common practice to reduce costs is to overprovision instances, which can lead to extreme variations in benchmarks. “For example, you might measure something that, at one point, is 50 times lower than the baseline, yet the code doesn't change at all.”

The CodSpeed team's approach is to simulate the CPU to generate a model that, while not perfect, would represent actual execution. “We give a super-fast feedback loop of how the performance changed between two versions of code. And we built it to be really portable, so you can bring it from one computer to another, one server to another, and the results will be comparable as long as you have the same architecture.”

While CodSpeed works on x86 and ARM architecture, his team is actively working on bringing the project to be completely architecture-agnostic. “To do this, we forked Valgrind, which is a dynamic binary instrumentation tool. So instead of simulating and emulating a whole system, we integrate at the binary level, and we have some integration libraries which make it easy to build the benchmarks and the performance test.”

Pastel explains that the initial goal was to make it as easy as possible to build performance tests so that users actually write them, as even code-level issues (like compilers mistakenly trying to remove benchmarks) can get in the way.

“One of the main problems here can be when you observe more macro-level phenomena. For example, let's say you have thread contention with a lot of thread scheduling, in which case a simple CPU model is not always enough, though we want to push our modelization as far as possible. However, while the next steps may include having some proper isolated or bare-metal instances, at some point you still need to validate the macro behavior.”

“On top of simulation, you sometimes need to measure on real hardware. We have actual physical instances that we can provision on demand in isolated data centers where we can measure performance,” says Pastel. Even physical concerns like the height and location of your server within a data center can yield completely different performance patterns.

“The idea is that using CodSpeed, if you have a GitHub Actions workflow, with just one line, you can get an actual physical instance that's dedicated to you, without paying thousands per month just for dedicated hardware. We share those costs and make it affordable.”

“The last instrument, because our idea is really to build an ecosystem of instruments that can be plugged to your code, is the memory measurement instrument, which we built because we also had a lot of people complain about memory consumption. Often, if you optimize for speed, it will be a trade-off and you will just consume more memory. Depending on what you do, reducing your memory footprint may also be a requirement.”

“Memory shifted our thesis a bit to be more about optimizing performance, which we define as the amount of resources you use. So your goals might be optimizing for disk or network usage, and ours is to help you optimize your software to reduce the footprint it consumes within your own system.”

“If you introduce a regression in...a really low-level library used everywhere, even a small performance regression will cascade everywhere. If you don't have any guardrails, then your performance might get completely out of hand, and you won't realize it until you're actually running on a production database.” -Arthur Pastel, Founder/CodSpeed

How Technical Leads Should Think About Performance Engineering

Pastel breaks down the value of performance engineering for orgs into a few layers. “First, performance issues might just be hiding some bugs, possibly due to an algorithm not being properly implemented.” Code correctness can therefore be part of performance management, as can making sure the resulting complexity of an algorithm is what was intended.

“Then, if everything runs faster, then you need fewer servers and less compute to run something, which means you cut your costs, which is one key benefit.” Pastel observes that orgs running cloud infra at scale generally prefer tools that are fast and don’t overconsume memory, which is why teams he works with often optimize for speed first, then memory usage second after they start comparing memory footprints.

Lastly, performance engineering can directly affect user experience. “Depending on what you ship, whether it’s a Web application, even if it’s a backend server or something, the overall latency of your program will define a lot of the user experience. As long as there is a user interaction involved, UX can be improved because you can make things faster.”

Pastel gives examples of how CodSpeed helps front-end teams detect performance regressions in CSS rendering, as well as optimizing at the kernel level and with JS execution. “There are many things that can be optimized to just make the overall experience for the end user way more pleasant, which is something that's pretty important as well.”

For teams building software, performance engineering also provides multiple benefits. “First, with AI, you’re shipping more code than ever now. You're spending much more time reviewing, so you have much less time to focus on performance. And the idea is that, at first, you need guardrails on the most important parts of your application, whether that’s the time it takes to load a full page, or to perform the most basic interaction on your server via your back end.”

Pastel argues that as software teams have less time and resources to devote to improving the performance of their products, they at least need some kind of background job optimizing performance. He points out that developers now generate and review code with LLMs, but even frontier models have no runtime performance signal, neither in their training data nor as feedback when they generate code. They can't measure the code they generate, so they're left to guess at what performant software actually looks like.

“What LLMs need is a proper harness to measure performance, which is what we've built. We initially built this for humans, but now it works for agents as well. Essentially, we help the agents build a really simple harness around any kind of code, any kind of language, then let them query the data in a proper format.”

Agents don’t collect or consume performance data in the same way humans do, but the team still needs to surface them for users. As a result, they developed an MCP server, a CLI, and skills to let users run their own auto-research loop, to minimize the cognitive burden on already-overstretched development teams.

Why Aren’t Agents Already Doing This Stuff?

Pastel suggests gaps in today’s agentic tooling and infrastructure that prevent them from being proper guardians of software performance. “CodSpeed can run on open-source software and the tools we built in sandboxes in a super-cheap way. But agents would still need to interpret and aggregate the data, which is the first tricky part.” He points out that optimization work is hypothesis-driven. Even the world's foremost performance experts form a hypothesis from an observed code pattern, write the change, and validate it by measuring. Agents need that same measurement loop.

“But then, there’s also a whole infrastructure play, which is much harder. You need to have isolated machines to run more-macro tasks.” Pastel explains that CodSpeed operates a fleet of bare-metal machines tuned for measurement stability spanning various hardware configurations, architectures (x86, ARM), and GPUs. Agents can request an instance on demand and run benchmarks in a completely isolated environment, on the hardware that matches their production target.

Pastel also believes that the evolving economics of compute resources is constraining something that technical teams used to not consider a constraint at all: Available memory. “Just getting RAM now is getting harder. We never really took the time to optimize software in the past because it was super-easy to say, ‘My software is consuming a lot of RAM, but people will just get laptops with more memory, so it's completely fine. Same with CPUs; if things were a bit slow, you would just get a bigger instance.”

While there are faster CPU and GPU units in development, Pastel points out that compute and RAM are becoming scarcer and more expensive, requiring teams to treat them as constrained resources and focus on performance.

For decades, Moore's Law meant slow software could wait for faster hardware. That escape hatch is closing. As hardware gains slow down, performance has to come from the software itself.

“We never really took the time to optimize software in the past because it was super-easy to say, ‘My software is consuming a lot of RAM, but people will just get laptops with more memory, so it's completely fine. Same with CPUs; if things were a bit slow, you would just get a bigger instance.”

How Founders Should Approach Performance Engineering

For startup founders, Pastel notes that agents can process thousands of times more data than humans in the same time period, which means that both your startup’s agents, and those of your customers, could be dragged down significantly by performance regressions.

Pragmatically, he admits that performance engineering may not be a priority in the very earliest stages of a software startup. “Performance might be taken as an afterthought in the beginning, and it’s fine to [focus on building] a simple MVP, because first you want to validate that there is a need for your product.”

However, the AI era has created another wrinkle that makes performance significantly more important, especially in competitive spaces. “For customers you hope will adopt and actually keep using your product, migrating from one thing to another has become super-cheap. Essentially, you can just have an agent tasked to migrate from one solution to another. I think this is why a lot of SaaS products are at risk, because we don’t have the strong lock-in and switching costs we had before.”

“So, performance is still a key differentiator, especially when you think about adoption and retention. It’s still a big deal in my opinion.”

Performance Engineering in the Enterprise

Pastel sees performance engineering as only becoming more important for enterprises, especially on the infrastructure side at scale. “Enterprises have already realized the margin gains they can get from optimizing their software. We see adoption growing today, and it will only accelerate in the future.”

He also argues that the advent of new AI tooling is helping orgs shift performance left as well. “Previously, performance would come down to, ‘We’ll see what happens in production.’ And if there was an outage, someone would wake up at night and try to fix it. Now we see more and more AI SRE companies that fix such issues, rather than having someone being paged to wake up in the middle of the night.”

“We’re building something to fix issues before they even happen, which we believe is better, because you don't want agents doing stuff in your production instances when you are an enterprise. If you can catch issues before production, you can validate with a human before anything happens.”

“It just happens in a much more natural cycle, rather than thinking, ‘Okay, there is a fire, I will try to fix the fire, push in production again, and iterate with a super-long development cycle.’ With agents, we really can ship in minutes while before, especially for enterprises, the release cycle might take weeks, if not months.”

How Will Performance Engineering Fit into the Future Stack?

Pastel is confident performance engineering won’t be going anywhere. “You can't train models to optimize performance by design. Performance depends on what you actually run in production: Your data, your infrastructure, your product goals. That context has to be measured, not learned.”

The founder is skeptical that even a perfect agentic future will account for all the potential outcomes of performance engineering, even if agents eventually come to possess all the right tools to communicate and validate performance changes. “Even if we manage to have some super-optimized languages, and we have new state-of-the-art models that allow us to have some crazy breakthrough to do a lot of optimization, there is still the structure of the code, which will still be an issue and cannot be optimized by a compiler.”

Pastel suggests that performance engineering, like security, will be a discipline that may never become out of date. “It won't disappear because you still need to have guardrails and things to make sure that your app is [performant] and consuming the least amount of resources possible. This problem is not ‘done.’”