Will AI Manage Software Performance with Humans in the Loop?

Performance Engineering: Part Brute-Force, Part Hard Decisions?

Software startups often launch quickly to beat competitors and gather early customer feedback. At that stage, performance should not necessarily be the priority, founder Tomás Senart argues. But once a product is exposed to scale, or latency and infrastructure costs become material, the economics change and teams need a repeatable way to keep performance from drifting. The creator of the popular Go HTTP load-testing tool Vegeta has carried those lessons into his startup Perfloop, and shares them here.

Tomás Senart discusses his learnings in Go interfaces at GopherCon 2015. Image courtesy Gopher Academy

From Load Testing to Go

As an engineer at SoundCloud, Senart had been tasked with replacing a significant chunk of a legacy system that needed to be loaded and tested. “At that time, the only thing available that was not a [heavyweight] Java ecosystem tool was Apache Benchmark, which had an array of usability issues and functionality limitations. We had a concrete problem and a culture of building things from scratch. Those two things together led to Vegeta.”

While the project has become popular in the Golang community, the founder clarifies that there was more to its success than the inherent speed and transparency of the language itself. “I think you can do performance engineering in any language ecosystem. With Go, because concurrency was such a core part of the design of the language, it made building a load-testing tool very natural at the time.”

“But you can build a load-testing tool in a way that will ‘lie to you’ in the results. So it wasn't just about building something quick, it was about building something correct. Azul Systems CTO Gil Tene is famous in the performance engineering world for many things, including the concept of coordinated omission: A phenomenon in load testing (and many tools fall into this trap) where the client that is sending the requests paces itself according to how the server is doing.”

“That is a bad way to test the system, because you're basically stopping to hammer the server at the exact point where you'd get the [indication] that it's not doing well. So you need to keep the rate constant. That was a key design decision at the time to build Vegeta in this way: To not suffer from that problem.”

The founder suggests that the most important lesson from his earlier work wasn’t that performance data is lacking, as most teams have dashboards full of telemetry and a benchmark they’ve run at least once. “The problem is that the benchmark you ran once is already stale. Performance is not something you set and keep, it is an envelope you have to hold against constant change.”

“Every code change, dependency bump, traffic shift, and architectural change nudges it, so the loop from measurement to a verified improvement has to run again and again just to stay in place.” Senart explains that the measurement process is inherently slow, which makes it easy to deprioritize and fall by the wayside.

His startup Perfloop closes the second half of the loop: After measuring, Perfloop assesses how systems behave under load, across the entire loop. “It finds the performance opportunities, proposes changes, and proves whether they actually moved the needle, with the numbers to back it up.”

Providing proof, the founder argues, validates automation across the rest of the process, rather than forcing teams to take things on faith: An important nuance as benchmarking loops need to be run far more often than can be done manually. “The system carries the ‘relentless’ part, the finding and the proving, while the person stays in charge of what is worth holding and what actually ships.”

How Much Should Tech Leaders Prioritize Benchmarks, Really?

Addressing the excitement around AI evaluations, the founder notes that the most popular AI benchmarks can and do become contaminated. “That's why people started creating private eval sets that they run so they're not gameable. That’s one approach, but it has its own problems, because there is no transparency into the quality of those benchmarks. It's a moving target, honestly: How to have transparency while not being gameable or having evals end up in the training data of models themselves.”

Senart draws a line between viral LLM benchmarks like Deep-SWE and benchmarks that teams specifically use internally to build their own products. “If you're building these capabilities in your company and in your product, the only thing that matters is your evals. It's not going to be the public coding agent evals for the coding models that matter the most.”

“The benchmarks that last are the unglamorous, private ones tied to an outcome you actually care about, because there is no incentive to game a benchmark whose only audience is you. Public benchmarks help you decide what to test. Your own codebase tells you what is true.”

"The only thing that matters is your evals. The benchmarks that last are the unglamorous, private ones tied to an outcome you actually care about, because there is no incentive to game a benchmark whose only audience is you. Your own codebase tells you what is true.” -Tomás Senart, Founder/Perfloop

Performance Engineering Lessons for Founders

Senart suggests that the adoption of agentic coding agents is fundamentally changing software, and not necessarily for the better. “First, agents now write a large and growing share of code, with a very wide range of outcomes. Many teams are moving faster than their ability to review every line, so they are shipping performance slop: Inefficiencies and waste that no human deliberately introduced and no one is specifically watching for.”

“Second, the load on production systems is changing shape. It is not only human users anymore. Agents call services at machine speed and volume, so the same systems get hammered far harder than they were designed for.” The founder observes that bot Web traffic has surpassed human traffic, and Web-based infrastructure is only the first beachhead to be pounded on by agents.

Commerce is another rapidly-growing space for agentic workflows for retail and checkout processes, and enterprise services appear to be the next frontier. The founder argues that agentic traffic is no longer a hypothetical future condition. It’s an inevitable thing that will happen, and make performance even more crucial.

In addition, it’s no longer feasible to wait to be ‘bailed out’ by eventually better hardware. “Now GPUs, memory, and compute are expensive and staying that way, so inefficiency shows up directly as a cost on your bill.”

The above factors lead to more performance debt that accrues quicker on infrastructure taking heavier workloads. “What is missing is a way to keep up. Occasional expert attention or a periodic consultant audit cannot match how fast the debt now accrues. That is the gap Perfloop is built to close.”

The founder clarifies that while you can use agentic systems to support performance engineering, AI doesn’t nullify the fundamentals. “You still have resources, you have things that drag on them, and you’ve got to make the most of them. You need to unpack that, one layer at a time, and be methodical. That's what performance engineering is.”

“You still create hypotheses, you go test them, and you look at the hard data. It’s just that the experiments can go faster now. You have agents which, with the right harness, can do that kind of work rigorously and not just guessing and going in the wrong direction, then you can parallelize so many of these hypotheses for testing.”

Senart offers two final pieces of advice for founders. “First: Do not pay for performance you do not need yet; if nobody uses your product, its performance does not matter. Second: The moment you are exposed to scale, the economics change in your favor. Performance work is methodical science, and the old constraint was that you could only afford to test the top one or two ideas. With the right harness you can now run many of those experiments in parallel, with rigor, and test the other ten you used to ignore.”

Getting Performance Engineering into Enterprise Customers’ Hands

Senart concedes that his startup’s approach to automating performance engineering isn’t necessarily mainstream among enterprise engineering orgs yet, due to a number of factors. “The first is trust, which is the hard one. You are asking a large organization to let an autonomous system reason about and propose changes to code that runs their business. They will not do that on a verbal promise.

“They need proof that the system finds real problems and that its proposed changes actually work, and they need that proof to come from somewhere they cannot dismiss. This is exactly why we are doing the work in open source, in public.” The founder notes that the maintainer of the widely-used Go library parquet-go has publicly agreed with his hypothesis that closing performance debt will come down to utilizing the kind of concurrent optimizations he’s building today.

“The second blocker is the usual enterprise gate: Compliance, security review, governance. There is nothing special here, it is just work you have to do to be allowed in the building. We are starting SOC 2 now for that reason.”

“The third blocker is not so much a blocker as it is inertia: Simply the reason to bother to do anything. Enterprises are actually a strong fit, because their scale and internal complexity produce a lot of recoverable inefficiency. In a large system, the performance you have left on the table is real money on the bill. Once trust and compliance are cleared, that recoverable cost is a very concrete reason to engage.”

“What it takes [to get into enterprises] is portable, verifiable proof on code that looks like theirs, compliance work done honestly, and a value story told in the language they care about, which is cost and risk, not benchmarks for their own sake. For us, this is the route, not the first stop. The immediate motion is the public open-source work and self-serve adoption. The enterprise conversation opens once that public credibility is built.”

"[For enterprises], the first [blocker] is trust, which is the hard one. You are asking a large organization to let an autonomous system reason about and propose changes to code that runs their business. They will not do that on a verbal promise. The second blocker is the usual enterprise gate: Compliance, security review, governance. The third blocker is not so much a blocker as it is inertia: Simply the reason to bother to do anything.”

What the Future of Performance Engineering Looks Like

Senart understands the current excitement about AI agents in software, but isn’t sure that performance engineering will simply become another agentic guardrail. “Guardrails are part of it, but the real question is where the human sits on the spectrum between specifying everything up front and steering every step (and both ends fail).”

The founder offers the two extremes: The ‘dark factory’ that is prompted once while humans step back, without human judgement or context along the way. The other is today’s common practice of endlessly babysitting coding agents edit by edit, which preserves human judgement but can’t scale, turning senior engineers into bottlenecks.

“What actually amplifies a senior engineer is in between, and it is the harder thing to build. The human stays on the decisions that carry judgment, which tradeoffs to accept and what is worth holding, while the system runs the steps underneath and surfaces only the calls that need a person. Present where judgment is required, absent everywhere else. Not steering more, steering higher.”

“Performance needs exactly this, because the target keeps moving. You cannot freeze the judgment into an up-front spec when the envelope shifts with every change, but you also cannot re-derive every step by hand each time. So you want a system that does the relentless step-level work and pulls the human in precisely when a real tradeoff is on the table.”

“That is what Perfloop is built for. The whole loop runs as an MCP server, so the pieces are callable on their own: A person or an agent can point it at one slice of code, or hand it a rough hunch about what might be slow and get back a benchmarked case. Autonomy is a dial the customer controls, from scoping, to directing, to reviewing, to fully delegating, which is really just a way of choosing how high on that spectrum you sit.”

“My honest bet is most teams land at ‘direct’ or ‘review’ for anything that matters: Let the system carry the steps and the proof, but keep their hands on the decisions that need judgment. So performance engineering won’t shrink into a guardrail. It will remain a loop that humans steer from above, with far more reach than before.”

The founder notes two additional factors that will affect future software projects. “First, the set of teams exposed to scale is expanding fast, because agent traffic drags services that used to be ‘safe’ into the deep end. Second, for the teams already at scale, the economics changed underneath them: The debt piles up faster, because agents write more code with less scrutiny per line, and it no longer depreciates the way it used to, because you can no longer count on cheaper, faster hardware to absorb it.”

“Performance debt that used to quietly depreciate now accrues interest. When a problem goes from occasional to continuous, how you handle it has to change too. A once-a-year expert audit cannot keep up with debt that compounds. Perfloop is my bet on what replaces it: The rigor a great performance engineer brings, available to any team that needs it, with the human keeping the judgment and the system carrying the grind. The open-source work is where we are proving it in public, one verifiable win at a time.”