The Unique Challenges of Mobile Compute

A significant focus in modern AI has been on large language models with billions of parameters. Billions more parameters mean bigger training investment, and presumably, more-impressive performance. But there are emerging use cases for mobile AI inference, like offline mobile text generation and the rise of general-purpose generative models for edge that hint at a growing need for reliable mobile AI performance.

Chris Karani, a longtime software engineer with specific expertise working in the mobile space, has built a variety of open-source, mobile-specific AI projects like Wax and Swarm, and is now focusing on building a mobile-specific monitoring and release-gating platform RYNO after observing the unique challenges that on-device inference poses to software performance on mobile.

Below, he explains why mobile inference needs a new breed of observability, and why on-device inference may necessitate a different way to think about app development: At the edge, moving inference on-device effectively introduces a variety of new hardware-specific variables, such as device generations, OS versions, and thermal and memory constraints.

The RYNO dashboard specifically monitors on-device inference.

How Traditional Observability Can’t Cover AI at the Edge

Karani explains that as a builder in the mobile space, he initially relied on popular tools like LangFuse and LangSmith for observability, but found that neither provided the monitoring coverage his projects needed. He found himself building RYNO, a flight recorder and release gate for AI in production on mobile, which focuses on privacy, safe execution traces from real devices, and turning production failures into regression tests to improve future releases.

“The project combines the hardware state with the end AI behavior to let teams debug and figure out why model failures happen when the apps are actually in production,” the founder explains. As an example, Karani cites a team he spoke with whose mobile app that identifies plants out in the wild and diagnoses them for issues.

Over time, the app team noticed negative reviews in the App Store from frustrated users who found the app misidentifying plants, but when the app’s developers conducted the same tests internally, they noticed no errors. “What they didn't realize was that a lot of their users were out using their apps in high ambient temperatures. This causes a variety of issues that the existing tools just couldn't catch.”

“For instance, once inference is running on a phone in production, changes in thermal state, memory pressure, and available compute can change how that workload performs in production. You can see slower execution, memory-related failures, or different behavior from what you saw during testing. RYNO comes in and is able to combine the hardware state with the model output and help teams debug why their models are failing when they're out into production.”

Closing the Loop for Mobile: Ragequitting, UX, and Regressions

For consumer-level mobile applications, this type of failure can lead to the ultimate cost: Users ragequitting the app and immediately deleting it. While the founder’s aim is eventually to target optimization opportunities for any edge computing setup, he admits that mobile is his current focus. “In mobile, most apps have a high uninstall rate.”

To avoid grossly inaccurate outputs due to hallucinations, Karani explains that his platform runs AI jobs across five stages: Capture, explain, replay, promote, and release.

• Capture: Utilizing the open-source, lightweight, privacy-centric Terra SDK, which can be embedded into mobile apps, and is built on open telemetry to prevent vendor lock-in, and records OS version, memory pressure, thermal state, and records OS version, memory pressure, thermal state, device class, and the compute resources involved in inference, such as CPU, GPU, or Neural Engine where available
• Explain: Utilizing the local macOS app’s ability to plot a visual timeline of device physics and model execution, identifying factors like correlating changes in device state with changes in model execution
• Replay: Having captured structured metadata, template IDs, token counts, and the precise physical hardware context at the time of errors, developers can use a replay button to recreate the conditions surrounding the failure to artificially reproduce the same environmental stress for testing purposes
• Promote: Teams can turn a one-off production failure into a structured, repeatable evaluation
• Release: From here, the platform connects to a team CI/CD pipeline to run various evaluations on different devices with that specific eval case prior to the next app release to prevent further regressions before they make it to production

The founder adds: “A lot of these issues also come in simple OS updates. Everything was working fine, then Apple launches a new update that introduces unexpected changes. And sometimes getting that stress test when you're testing within a controlled environment is difficult. So a lot of these issues can only be caught when the AI is actually ‘out in the wild.’”

“Sometimes you have to actually catch these failures once they're actually already out there because these systems are so non deterministic. They're not like traditional, deterministic software where you might have a really tight unit test case, so certain errors don’t reach production.”

“Because there's 100 million ways that the AI can actually fail. So, one of the best approaches is having this sort of loop that we've created to catch them quickly in production and ensure that they don't persist in the future.”

Building for Privacy and Memory Optimization

Karani explains that privacy was a key part of the project due to its initial focus on iOS. “We believe that the on-device AI on iOS is some of the best in the world right now. The Apple Neural Engine lets users run these really awesome models. But for iOS and mobile users, privacy is very important. It’s common even for experienced teams to accidentally violate user privacy while trying to collect the debugging context they desperately need.”

“This is how teams are silently trying to capture model regression with internal tooling. They captured a prompt and its output, and those contain extremely sensitive data, so you end up violating user privacy. To fix that for RYNO, we actually have a ‘never log’ list. No screenshots, no user prompts, no contacts lists, no physical addresses, no chat transcripts.”

“We use a minimal telemetry schema. Instead of logging what the user has typed, we log a structured shape and the operational health of the mathematical transaction. We've been using a lot of tools to ensure that teams are able to get the metadata that they need to reproduce these failures while still avoiding violations of user privacy in the process. We think this is super-important for on-device AI.”

Aside from privacy, the founder notes that going lightweight was an important architectural consideration to run successfully on edge. “Because of the memory pressure issues with the Jetson Linux, we made a key decision early on to build the core of the SDK in Zig. This gave us a lot of control over memory pressure and it gave us full control over how we actually want to architect memory use for the SDK, because on-device AI uses a lot of resources on mobile.”

“So we needed to build an SDK that's extremely lightweight. Because we’re running an observability SDK on constrained hardware, it would need to stay lightweight enough to not create another source of memory pressure competing against the model it’s observing. We think that for edge AI, debugging is something that is going to become a big problem. And it's totally different from how current tools actually enable teams to receive telemetry.”

The founder admits that part of his inspiration came from his own analysis of the AI space, which increasingly seems dominated by billion-dollar vendors pushing trillion-parameter models. “We believe wholeheartedly that edge AI is 100% the future. Smaller models are improving at a rapid rate. Vendors like Apple and Google keep pumping a lot of money in the space, and we think that the kind of benefits that edge AI provides are undeniable.”

“First, there’s privacy. Second, there’s improvements to latency, and third, is cost reduction. As the models keep improving, people are going to start realizing and taking this extremely seriously as we move forward. The space is still very early. But the world I see in the future is one in which we'll have on-device AI everywhere you go. There’s already on-device AI in cars, and we'll have robots that need on-device AI, for television, mobile, and so on.”

“Yes, the Cloud will always be where true intelligence, the larger computational intelligence, lives. But more and more companies are going to keep moving towards leveraging the on-device AI space as we move forward. And I think that's why Apple has also been putting a lot of effort year over year into the Apple Neural Engine and every chip that they release, prioritizing the inference power that they can actually pull out of the on device models.”

Founder Lessons, Enterprise Adoption, and the Future of Compute

For founders looking to build on-device AI products, Karani suggests that his own deep understanding of the space has been fundamental to his progress. “I think having a good understanding of how the hardware state actually works is important. The computation and the OS system attributes during inference time...that’s a different problem space that a lot of engineers need to understand if they're going into the on-device space.”

“Inference is one thing, input and output of prompts is another. But for a lot of these models, you have to understand they're not in a problem space that is similar to running on a large data center in Atlanta with perfect resources all the time. These are models that are running on-device with varying, different pressures in constrained environments. Understanding those environments is key to actually building the right systems for AI solutions.”

On the question of why device-specific telemetry still doesn’t seem to be mainstream at the enterprise level, the founder points back to how the tool ecosystem for edge computing is still in its early days. “There’s a lack of tooling. A lot of the enterprise companies we speak to are concerned over their lack of telemetry and understanding of how models behave when they’re actually out into production.”

“For instance, we were speaking to a medical company that was trying to launch a very privacy-centric, on-device model to different customers. But what they found is that they just couldn’t risk getting an incorrect diagnosis from the model (and not being able to understand why it happened). Existing tools just didn't cater to them. The space is so early and emerging and a lot of the focus is on cloud models.”

The founder notes that while high-end cloud models get a lot of attention, they’re also currently leading to a lot of untenable costs. “I think that's what's going to push enterprises more into on-device AI. And as [on-device models] improve, the solutions will become simpler, but we need to provide them with the kind of tooling that enables them to ship on-device AI reliably to their users.”

Will there be an inflection point at which everything flips from cloud to local? Maybe not. “I think people are going to have a balance of the two. Teams will leverage on-device models for certain tasks and route to cloud models when they need heavy computational work to be completed. I think the future will be a hybrid of the two. I don’t think on-device AI is going to completely negate the need for Cloud, but I think cost and performance are going to push people to try to understand how to leverage a hybrid of the two. And the majority of the work we do today will be pushed to the on-device models for inference.”

The post Why On-Device Inference Needs Custom Observability appeared first on Heavybit.

On episode 11 of Third Loop, the Progressive Delivery team sits down with Chad Fowler to explore regenerative software and what architecture looks like when AI makes code cheap enough to continually replace. They discuss Phoenix Architecture, Pace Layers, observability, architectural constraints, and why the system itself, rather than any individual implementation, may become the real asset in an AI-native future.

The post Ep. #11, Durable Systems, Disposable Code with Chad Fowler appeared first on Heavybit.

On episode 58 of Generationship, Rachel Chalmers sits down with Anastasia Marchenkova. They explore what it will take to move quantum computing from promising hardware to useful, production-ready systems, including better orchestration across quantum, classical, and AI compute. Anastasia also discusses open-source infrastructure, the limits of AI automation, and her larger vision of making powerful computing and scientific tools more broadly accessible.

The post Ep. #58, Quantum Uncertainty with Anastasia Marchenkova appeared first on Heavybit.

On episode 4 of Lab Notes, Amir Zohrenejad speaks with Junchen Jiang about why KVCache may be better understood as reusable, AI-native data rather than a temporary inference optimization. They explore how LMCache and CacheBlend can reduce redundant computation, move context across distributed inference systems, and help support increasingly complex AI agents. The conversation also covers multimodal workloads, open-source infrastructure, and the future of AI systems research.

The post Ep. #4, The New Big Data of Inference with Junchen Jiang appeared first on Heavybit.

On episode 10 of Third Loop, the Progressive Delivery team speaks with Honeycomb co-founder and CTO Charity Majors about observability, AI, and the changing economics of software development. They explore a future where code is increasingly disposable and regenerable while architecture, constraints, production behavior, and promises to users become the durable artifacts that matter. Along the way, Charity shares her perspective on responsible AI adoption, engineering in nondeterministic systems, and why production remains the ultimate source of truth.

The post Ep. #10, The System Is More Than the Code with Charity Majors appeared first on Heavybit.

On episode 12 of High Leverage, Joe Ruscio sits down with Dexter Horthy of HumanLayer. They explore the promise and limitations of autonomous coding, including why today’s models excel at bounded programming tasks but struggle to account for the long-term consequences of architectural decisions. The conversation covers dark software factories, code review bottlenecks, program design, technical debt, and the continuing importance of experienced engineers.

The post Ep. #12, The Limits of Lights-Out Coding with Dexter Horthy appeared first on Heavybit.

How Can Agents Be the Future if They’re not Reliable?

Some reports suggest that as many as 25% of enterprises have adopted agentic workflows. Isn’t this the era of AI agents? Why isn’t the number higher? Aside from predictable enterprise-level concerns (49% of enterprises are reportedly holding off due to security concerns, for instance), agents haven’t proven to be reliable workhorses at scale, particularly for long-horizon tasks. Not yet, anyway.

Agents still can’t “remember” their learnings from previous successful runs, still hallucinate when their context window becomes full, and still have a number of other issues. Wasn’t the perfect agentic future supposed to feature technical teams commanding fleets of infallible, well-trained agents to do their bidding?

Open-source creator Mike Hostetler believes such a future is possible if systems combine strong guardrails with fault-tolerant infrastructure designed for large-scale deployments. Here, he explains why he built his project, Jido, not using Python, the de facto programming language for AI, but using Elixir, which was inherently designed to support maintainability and massive concurrency.

Mike Hostetler discusses how AI is changing the business of building software on the Software Without Borders podcast. Image courtesy Software
Without Borders

Resolving Nondeterministic LLMs and Deterministic Algorithms

Hostetler, a veteran developer and founder, got his start working on the jQuery ecosystem, from which he built a career, a startup, and a lifelong curiosity about technology. He explains that his introduction to Elixir came from a tour of duty working in distributed systems, which inspired him to think through the challenge of being able to run multiple agents with real-world reliability. “I envisioned a future of ‘10,000 agents per human.’”

The creator notes that his original vision was running 10,000 agents on a single Raspberry Pi. In other words, in a resource-constrained environment with no direct access to LLMs. Models were getting better, but capital expenditure from foundation labs continued to skyrocket.

Hostetler suggests his goal was to make the best of both worlds: Capitalize on the improving performance of state-of-the-art models, combined with performant infrastructure that wouldn’t break the bank. “The intersection of these two ideas is: Agents needed to be easy to build, easy to deploy, easy to orchestrate and coordinate. And then, 20% of your agents could use an LLM as the thinking step. But it needed to be just as easy to build 80% of your agents on what I call classical AI algorithms.”

The builder clarifies that ‘classic algorithms’ also include logic that predates LLMs: “Everything from a finite state machine to a behavior tree to hierarchical task networks. Video game AI as well as LLM AI. And those could work in concert and then be easily and inexpensively deployable for a user, which gets into the real pragmatic piece.”

Hostetler clarifies that despite Jido’s explicit “AI optional” clause, his approach to AI is less skeptical and more pragmatic. “I love the nondeterminism of LLMs. I love pushing those boundaries. Just by happenstance, having put my own open-source project out there, I got to know Geoff Huntley and tried the Ralph Wiggum loop before it was publicly available. Loved it, and had Jido running loops.”

“But I quickly observed that [with successive loops], you end up compounding quality issues and you run into these challenges.” The builder’s observations led him to start experimenting with combinations of agentic loops and classical, deterministic AI algorithms, based on a self-taught course of study.

“I've used my computer science degree more in the last two years than I have in my entire career, which is just so much fun. So I’m not ‘anti-AI’ at all. What I want to put forward is something that lets you use both modes of tooling in the same context, instead of having a ‘developer mentality’ that requires a hard switch between ‘Here’s an agent that has an LLM as its brain’ versus an agent that has a finite state machine as its brain, and you have to build a bridge between the two. Jido makes both into first-class citizens. I knew as an engineer, as an architect, I would want both.”

How Can Agents Master Long-Horizon Jobs?

Hostetler admits to reading, and speculating about, the news headlines about agents seemingly getting closer to the holy grail of succeeding on complicated, long-horizon tasks in a single, zero-shot run. “Yes, I read the headlines, and I see the METR reports. I would say I have hypotheses about the topic, but I acknowledge the limits of my hypotheses.”

“We see new model drops like the [recalled-and-re-released] Fable 5, which can reportedly do a long-running task for eight hours, uninterrupted.” The creator notes that it can be hard to separate reality from hype, but notes that foundation labs may have different incentives than the rest of us.

“I would think there is a lot of pressure to embed the capability behind the walls of the model; to make Fable better via academic RL, or to have a lot of infrastructure running between the API call and the model itself. You see them pulling in things like their web API tools, and doing everything to make Anthropic a walled garden for any corporate customer.”

Hostetler contrasts the lofty ambitions of foundation labs against his day-to-day work, which requires him to service ERP systems maintained by tech leaders across the country who need reliable performance, day in and day out.

The creator suggests starting from a strong harness, but also looking into reliability. “A really concrete step is to start with a very high-quality model harness if I'm going to build my own agent. Yes, admittedly, Elixir’s philosophy is ‘let it crash,’ but you don't really want that to happen in practice. Your focus should be: How do you recover from the crash? How do you deliver high-quality software from that?”

“Elixir’s philosophy is ‘let it crash,’ but you don't really want that to happen in practice. Your focus should be: How do you recover from the crash? How do you deliver high-quality software from that?” -Mike Hostetler, Creator/Jido

Squaring the Circle: Autonomous Agents vs. Reliable Uptime

The builder recognizes the gap between LLM-powered agents that often fall down for a variety of reasons and Elixir-based systems like Discord and Pinterest that are expected to provide continuous uptime for millions of users. “Without getting too technical, this really was a key design decision of Jido itself.”

Hostetler explains that the project was designed to accommodate both builders of independent, solo agents as well as those building for coordinated, multi-agent use cases. “And also, there’s all the infrastructure concerns for when you go to deploy: Where does it live? How is it persisted? It's persisted with what I consider to be two schools of thought.”

“The first [persistence story] is that I persist my BEAM system back to a durable store, like a Postgres database which is centralized. No issue there. And I would say that more than half our users use this. The other key story is what I call the fully distributed, fault-tolerant approach for scale that you get into with the really big deployments, such as WhatsApp or Facebook Messenger. Multi-region, multi-clustered...the big stuff.”

“Jido was geared for [large-scale deployments]. We could get into the specifics of how Jido was created to live within the BEAM itself, but it is possible. And there are nuances here, too. People come to the project and they sometimes feel like it's overbuilt. Case in point, when you send a message into a Jido agent, we use a dedicated message envelope called a Jido signal, which, when you're in a single-node use case is really and truly overkill. And yet it's the standard built on the CloudEvents spec (which is independent of, and exists outside of Elixir).”

“We do this because you can't have it both ways: You can't have your typical simple JSON payload alongside a robust, enterprise-grade, transport-agnostic CloudEvents envelope and do the same thing. You've got to pick a lane. And so, Jido picked the ‘robust and scalable’ lane. I do work with some teams that are now asking me about the multi-cluster use case. I’m working toward a demo of a multi-region cluster independent of a durable store with resilient agents.”

“To clarify, everything I just discussed was outside the agentic loop. Within the agentic loop of determinism versus nondeterminism, there are guardrails within which Jido has to operate. It can't just go do everything because it was meant for the Raspberry Pi use case. Because Elixir and BEAM can natively cluster, you could easily have a deploy to support a use case for something like, let’s say, Internet of Things (IoT).”

“For example, you’re going to have a central cluster that then works with Jido agents that are deployed in an IoT setting that talk back to the central cluster, and you'd have two agents communicating with one another. You could have that modality in a larger deployment. As an example, one of the big successes in the Elixir world is the robot control project Beam Bots, which adopted Jido natively, so we’re seeing use cases for non-clustered low-level IoT all the way up to your typical Claude Code agent harness.”

How Startup Founders Should Think About Agentic Products

When asked how he would advise startup founders thinking of launching agentic products, Hostetler offers a handful of suggestions. “For your first level of planning, I would say: Focus on the default. How you define what an ‘agent’ is could be colored by your Claude Code glasses! Make sure that as you're defining an agent in terms of what you bring into your infrastructure, and that you have a tighter, concrete definition of what that is.”

“Second, the other mistake that I've seen (and have made personally) is that everybody wants to let the LLM ‘take the wheel.’ At the point when you're designing infrastructure to build a company on, you have to draw clear boundaries between nondeterminism and determinism, and create gates to bring it back and forth. And this is outside of the agentic loop.”

The creator suggests that customer expectations for agentic projects have grown over time. “After a few years, we’ve seen the patterns that ‘work,’ whether it's natural language or dynamic UI, we know the patterns to create user value and what users expect. The innovation has shifted from, ‘We can do just about anything,’ to ‘The user expectations are pretty much there, so how do you deliver on that?’”

Hostetler describes an interesting project a friend built in Elixir that effectively replicated Claude Code’s functionality, but for posting on LinkedIn, using a dedicated agent with a specific ID. “Persistence wasn't dynamically allocated, but the agent would persist, and then it would hibernate and rehydrate when you wanted to edit a piece of content.”

The project let users edit posts normally via the LinkedIn GUI, but also supported giving instructions to a coordinator agent. “The key is, he made that persistent throughout the entire experience. So, no matter which modality you were interacting with, there was a kind of predefined team structure for how the agents would interact with your content. And it just struck me as this really innovative mental leap, that any agentic products I see aren't thinking big enough.”

The creator suggests that the main challenge he’d offer to startup founders and product owners is to think bigger. “I don't see a lot of product owners thinking about [larger ideas] because they're really stuck in the mindset of, ‘Claude Code is my agent. It can do everything. I have to always use Claude Code.’ I say: ‘No, you really need to break out of that mental model.’”

“It just struck me...that any agentic products I see aren't thinking big enough. I don't see a lot of product owners thinking about [larger ideas] because they're really stuck in the mindset of, ‘Claude Code is my agent. I say: ‘No, you really need to break out of that mental model.’”

How to Think About Enterprise Adoption

When asked how he would advise ambitious startup founders looking to sell AI products to enterprises, Hostetler draws upon his own day-to-day experience. “I would say that we're early. I would acknowledge the timeline displacement, Ray Kurzweil’s riff on the William Gibson quote about the future being here, just not evenly distributed.”

“What's happening in enterprises is that there are engineering teams who are using the latest tools, and they're on that curve somewhere. But the operations teams are just starting.” The creator notes that he works with operations teams that are just starting to encounter the same challenges that engineering teams faced 18 months ago: Worrying whether AI will replace them, realizing AI likely won’t replace them, then trying to figure out how to fit into this brave new world.

“We're early in that phase [where people are trying to find] pragmatic use cases. The issue isn't so much the tools themselves. What's interesting to me is the layer above them, which I wrote about in my blog on orchestration. To me, orchestration is the cornerstone, because there's nothing actually orchestrating the tools.”

“I think that corporations are set up in silos. Sales isn't talking to marketing, who isn't talking to ops. Ops isn't talking to product. So, the pragmatic things that we need are places to deploy AI experiments, because the thing we've emphasized internally isn't so much AI as an end state, but rather, AI as iteration.”

The creator recounts a story of a team he works with using Claude Code to build a dashboard. In a completely unintended consequence, Claude Code used the Google Workspace connect to deploy the dashboard to Google Sheets via Apps Script to create a public URL via, of all things, a Google Sheets backdoor. “I burst out laughing. There's an entire product right there, in that space between a non-technical user and a model’s instincts to propose unexpected solutions.”

Hostetler also points out a strong business opportunity in governance. “During my tour of duty in corporate, I spent two years as a VP of enterprise technology. I managed all the IT and compliance stuff for a publicly-traded company.” The creator notes that “AI governance” could easily go beyond basic compliance frameworks like SOC 2.

“Every LLM call in your organization will eventually be proxied and logged. But nobody's doing this at the moment. Think about just capturing it as one problem. Analyzing and fingerprinting.” The creator predicts that sooner than we think, organizations may need to be “AI compliant” with whatever new version of the SOC2 standard that arises.

“Another [enterprise use case] I’d look at is insurance. Cybersecurity insurance with AI, mark my words, in the next two years, that will be a ‘thing.’ And this won’t just be your CrowdStrike-style, on-device stuff. It's all the things that get exfiltrated from what you build. Nobody's talking about that. The strategy there is very immature. Admittedly, this isn’t a very sexy space, but I think there's a lot of money to be made there.”

The Future of Elixir and Durable Systems

The creator is realistic about how his project and the greater Elixir ecosystem figure into the future of software development. “A non-Elixir shop is going to have a hard time adopting Elixir. So I can see Jido becoming an agentic sidecar that eventually makes it easy for TypeScript devs to spin up an agent with Elixir-level durability.”

“This might be for a market segment that could be ‘one step below’ the Temporals of the world. To clarify, I’ve used Temporal before, and it’s fantastic, though expensive. But thinking pragmatically, there are a lot more TypeScript devs in the world. Python will be its own thing. TypeScript devs are going to need an SDK to easily call Jido agents, and then Jido agents will operate async, and it'll just be self-contained with a dashboard to spin it up.”

The creator acknowledges that popularly-used languages and frameworks like TypeScript and Node.js are excellent for what they are, and that they solve specific problems. “But you need to be able to spin things up inexpensively that companies built on TypeScript can solve either by running in Docker via their infrastructure, or run out of a cloud where they can spin up platforms of agents, with one agent per customer (or dozens of agents per customer).”

“And to be able to do that practically is just going to be overwhelmingly expensive, even with Cloud costs. And you need to be able to stitch it all together, potentially using non-deterministic agents with deterministic agents to deliver a strong product experience. It’s a very early picture, but that's where I see the market now.”

The post How to Make Agents Durable for Concurrent Systems appeared first on Heavybit.

In episode 54 of The Kubelist Podcast, Marc and Benjie sit down with David Crawshaw. David shares how a weekend WireGuard experiment became Tailscale, and how a series of unsuccessful developer-tool experiments eventually became exe.dev. The conversation offers a candid look at product discovery, technical failure, cloud economics, and building infrastructure for AI agents.

The post Ep. #54, The Age of Personalized Software with David Crawshaw appeared first on Heavybit.

Not every employee is a superstar. In large organizations, you will encounter order takers. Despite targeted coaching, repeated instruction, and close evaluation, order takers require constant supervision, learn slowly, and fail to internalize the goals (org, team, and personal) that they’re meant to carry to their daily tasks. Powered by language models and the harnesses around them, AI agents are touted as a way to replace order takers. Companies like Oracle, GitLab, and Elastic are enacting huge AI-related layoffs and the numbers just keep growing.

But despite the promises of a pure AI utopia, we still can’t trust agents to work towards goals. Most agents require an order-taker’s level of supervision, coaxing, and constant prompt-response. A human manager must still own a goal, break it into tasks, give the models context for each task, and triple check outputs and next steps. Even when an agent uses tools and skills across multiple tasks, a human for the most part, is still responsible for goals and outcomes.

From Order Takers to Outcome Owners

A task has a clear input and output. A tool can help complete it. But intelligent humans are not hired simply to complete a collection of tasks. They are given an area of responsibility and expected to deliver against goals or outcomes.

Take a software engineer. Refactoring a module or fixing compile-time errors are tasks. Making sure the backend scales with seasonal traffic patterns while minimizing cloud spend is an outcome. The engineer needs to understand the architecture, decide what needs to change (if anything), weigh tradeoffs, sequence the work, then monitor outcomes and adjust the plan when they find something unexpected. To understand seasonality, they might study historical trends or ask others in the organization. As new patterns emerge, they update their understanding and remember it to avoid future pain.

Consider a sales rep. Writing a follow-up email after a sales call and logging it in the CRM are tasks. Generating $500K in qualified pipeline is an outcome. The rep needs to understand the customer needs and product value, identify and target accounts, personalize outreach, handle objections and find patterns towards a quick close. If the pipeline runs dry, they need to diagnose issues and determine whether it’s their own missteps, or an issue with another team’s outputs, such as product or engineering.

Current agents are very good at completing tasks, especially when the output can be verified immediately. But they are surprisingly bad at making informed decisions over long time horizons.

Longer Context Is not Enough

Most agents implement some variation of ReAct: the model reasons about what to do, takes an action, observes how the world responds, and reasons again. For a while, practitioners believed the path toward long-horizon agents looked straightforward:

model + tools + loop + infinite context = autonomous agent

The corollary was that if the context window became large enough, the agent could keep every prior thought, action, and result in view, loop forever, and learn in context.

This is dead wrong. More time = more mistakes. Recent studies find a 30% accuracy drop when long-context models are asked to retain information across sustained interactions. Over longer horizons, agents also suffer from what researchers call "behavioral state decay": requirements, prior attempts, diagnoses and open sub-goals begin as priorities, but at some point stop influencing the agent's decisions.

Long context gives an agent a larger transcript. It does not decide what remains important, which beliefs should be revised, or whether the current plan still serves the original goal. Essentially, long context just offers an agent more time to make mistakes and lose the plot.

This forgetfulness is expensive. Repeatedly looping agent jobs also incurs higher token costs. We’re seeing increasing concerns from orgs like Uber and ServiceNow that have burned through annual token budgets in a matter of months as the “free lunch” effect from tokenmaxxing has come to an end. Token costs have gotten so bad, that there’s a growing trend of inference theft leading to eye-opening losses via public gateways. What organizations need to make agents live up to the hype is the ability to build and run them over long-horizon jobs without breaking the bank.

The Five Missing Systems

Long-horizon agency isn’t as simple as a model running in a loop for a long time. We’re missing critical systems that provide continuity of intelligent behavior across changing evidence, context boundaries, process failures and learning cycles. We’re just not there yet, but I’m hopeful.

We believe continuity depends on five systems:

Organizational memory. Most agent memory is scoped to a single session. Long-horizon agents need useful knowledge to survive beyond that session, evolve as facts change and become available to other agents in the organization. Examples of companies working here include Mem0, Mubit, and Antfly.

Goal orchestration. Distant outcomes do not provide enough feedback to guide each decision. Agents need to break them into intermediate goals that make progress measurable without losing the relationship between a local reward and the original objective. Examples of companies working here include LangChain, CrewAI, and Paperclip.

Durable execution. Long-running work must survive crashes, expired credentials, delayed approvals and irreversible interactions with the outside world. The execution layer must preserve state and know whether a failed operation should be retried, reversed, escalated or stopped. Examples of companies working here include Temporal and Diagrid.

Learning. A transcript records what happened; learning changes what happens next. Agents need a way to turn outcomes into better playbooks, instructions, memory and, potentially, model behavior without allowing every noisy experience to rewrite the system. Examples of companies working here include: Letta and Modiqo.

Toolchain Ecosystem. Agents need additional tooling to extend their capabilities into real-world use cases where they can genuinely move the needle. Among others, a competitive ecosystem of agentic tooling that offers better harnesses, secure sandbox environments and verification, comprehensive monitoring, the ability to support bespoke, custom builds for individual use cases, and large-scale governance and security need to emerge. Examples of companies working here include: Galileo, AgentOps, Halluminate, Bronto, and Strix.

Delineations between these system categories will blur. Memory, for example, may eventually absorb some of what we now call learning. Some capabilities may become native to models. Others will remain properties of the harness, runtime, or organization around the model. The primitives powering these systems are still being designed and implemented.

Got Opinions or Want to Hear More?

If you’re interested in exploring these further or you’re already building in one or more of these categories, join us on Sept 24 in SF alongside Glean, Hex, and Baseten at DevGuild: Long-Horizon Stack. In future posts, I’ll explore each of these systems and how orgs can transform agents from order takers to outcome owners.

The post Long-Horizon Agents: From Order-Takers to Outcome Owners appeared first on Heavybit.

On episode 41 of Open Source Ready, Brian Douglas and John McBride sit down with Phil Estes. They explore why the definition of a container remains surprisingly fuzzy, how microVM-backed sandboxes could support the next generation of AI agents, and what AI-generated contributions mean for projects like containerd. Phil also explains why the future of open source depends as much on trusted maintainers and healthy communities as it does on code.

The post Ep. #41, Containers at the Edge of AI with Phil Estes appeared first on Heavybit.

Can You Make AI Infrastructure Invisible?

Depending on which people you talk to, individual developers spend hundreds (to thousands) of dollars monthly on AI services, with some predicting that AI tool costs will simply exceed the cost of human developers soon.

What if the infrastructure and tooling needed to build with AI were free of charge, and wasn’t locked behind a dozen different vendors’ paid plans? That was the question that led developer Mike Kuykendall to create the open-source project Shimmy, a lightweight API replacement for local LLMs. Below, he explains his thought process and his vision for AI software built entirely on free infrastructure.

From Frustration to Lightweight Infrastructure

The start of Kuykendall’s journey was his own frustrations with working across the many different AI-powered software tools he had been using. “It honestly started as a simple shim, because I was getting sick of all the ways to connect with OpenAPI protocols.” The creator was surprised at the project’s popularity, which appears to have come from many other developers also being tired of having to log into dozens of different paid services.

“I just wanted something that was completely private and offline, where you can control your own inputs, outputs and everything else.” Since then, the creator has moved the project to his own Rust-based library Airframe to avoid having to depend on other vendors’ command-line interfaces. “Every time I turned around, I would try to do something else and immediately say, ‘Oh wait, this other thing is owned by someone else at that level.”

“I never really was looking for anything out of it other than just to make a good space for people. That's why I’m trying to make it a ‘free forever’ concept. If I can build products that use [this foundation] that would help people, I could sell [those products]. But the underlying service should be a place where we could all come together and just use AI for free. The other thing is, if you look at every service out there, everybody is just completely throttled on tokens. That's an enormous cost for people who want to do development now.”

“I think that companies want everyone to pay for inference, and you shouldn't have to. The thing is, everybody's already got bundled costs.” The creator points out that the cost of utilizing high-end foundation models continues to increase, while Chinese AI labs are turning out increasingly competitive open models...which aren’t always supported natively within mainstream AI development toolsets.

“Now the users are left with this mess. It was getting to a point where I was paying four different vendors and I wasn't even maximizing all the stuff I had. I asked myself, ‘Why can't we just have an agreed-upon substrate we just all go to?’”

“Why not make the substrate free? Shimmy is free, always will be free. I'm not going to charge you. You have it. You use your model, I make sure it's up to date and your model helps you succeed.” Kuykendall suggests his primary audience is software developers, who could eventually use tools built on a free infrastructure substrate.

If you look at every service out there, everybody is just completely throttled on tokens. That's an enormous cost for people who want to do development now.” -Mike Kuykendall, Creator/Shimmy

Freeing AI Infrastructure from Garbage Collection

Kuykendall makes the emphatic point that despite the domineering popularity of Python for AI use cases, he designed his project to be 100% Rust-based for a reason. “Garbage collection [in Python] is ridiculous. It’s bad engineering.”

The creator explains that he began learning Rust to challenge himself, but found it to be an excellent working language whose ownership and borrowing system inherently forces developers to manage memory safely at compile time, but also matched AI software development well by minimizing non-repeatable, nondeterministic outcomes.

“I also found that the garbage collection brought along with it a bunch of stuff you don't need. If you change your thinking around whether you need to retire some variables or let them gather in heaps, you can just use them and borrow them appropriately.” The creator notes that Rust’s static type system, which verifies every variable at compile time, was both more sensical and helped him create invariant pipelines that reduced unpredictable outcomes as he built.

How Free Infrastructure Cures Tokenmaxxing

Kuykendall suggests that a largely free AI infrastructure substrate should be the solution for headline-grabbing token budget problems from orgs like Uber and ServiceNow. “Just recently, almost every AI company cranked up their rates,” notes the creator, in reference to how major AI vendors recently repriced tokens that caused users everywhere to burn through their token budgets early.

“Last month, I ran out of tokens for every one of my top services on day 15. And I was staring down at the remaining two weeks, thinking to myself, ‘What am I supposed to do now?’ I was forced to go try to ‘do it cheap.’” For his own purposes, the creator went off and did what he imagined many other developers are doing: Trying a list of free alternatives, including OpenCode, Pi, Hermes Agent, and others.

“I think basically the corporations right now are trying to recoup their money. They realize that they're just going to have to take it from the people already using it. That's when I realized if I'm going to be a developer and I'm going to use this service, I need a more-economical way.”

Having already canceled many of his pricey AI subscription services, Kuykendall reflects on how other developers like him may be making the same choices...and ending up with the extra hundreds of dollars saved from those canceled services, perhaps ready to spend elsewhere. “I’m working on other products on top of shimmy. Because that's where the money should be, in my opinion. Application should be where we put the money, not the inference.”

An Inference Tipping Point for Engineering Orgs, Coming Soon?

Kuykendall foresees a not-to-distant future where engineering teams take a hard look at their AI bills, come to the same realization he did, and look to make a change. “I think the [largest AI vendors] understand that they overinvested.”

“In the meantime, if I had a young up-and-comer startup right now, I would use an inference stack that was free. I’d try to get the best compute I could get for my dollar. I think there is going to be an ‘API apocalypse,’ or whatever they’re calling it on social media, when everybody starts seeing viable alternatives they can use instead. And I think we're already getting there.”

The creator notes that open models like Alibaba’s Qwen family and Z.ai’s GLM models can already perform well on software development when coupled with high-end hardware, a stark contrast to some of his previous experiences. “I worked at the IRS for several years, with no AI available. We had to do everything ‘the hard way.’ There were times when I’d think: ‘I could save three hours if I could just use AI right now!’”

According to Kuykendall, we live in very different times now, with highly performant open models and high-end hardware both within reach of individual developers. “Long story short: What I wanted was for nerds like us to lay the groundwork so that it was just invisible infrastructure, for AI work, development, or anything else.”

“It was also important to build the project offline. There are capable builders in Asia, some of the biggest drivers of growth on my GitHub repos, because I think they’re under the Great Firewall.”

What I wanted was for nerds like us to lay the groundwork so that it was just invisible infrastructure, for AI work, development, or anything else.”

Lessons from Building Many Projects with AI

While AI can be a powerful assistant, Kuykendall warns other builders to avoid lulled into a false sense of security from sycophantic models that agree with you a little too much. “You have to separate yourself from this stuff, or at least watch for AI having an emotional impact on you. Try to get away from that and make it a source of data instead.”

“I've been in AI development pretty much from the moment it started. And I noticed while doing testing that tests would just happen to go the way I wanted them to, by a point or two. Always. The AI was quietly influencing things. So for founders, I would recommend that first of all, they should spread out their compute and find many points of view. Every one of these [models] is differently trained and they're going to give you a different answer. And a lot of times you have to watch your logic, because it'll tell you exactly what you want to hear.”

The creator recommends putting models through their paces, at minimum asking why ideas might be good and why they might be bad, and potentially pitting other models against each other. “You have to try to look at it as an empirical and scientific process, and try to take the human emotion out of it.”

What AI Infra Will Look Like in the Future

Kuykendall envisions a future AI infrastructure without the many layers of interface that block users from getting things done. “When I first started really spitballing it early on, I remember thinking: Why am I wasting my time with a computer, that's running a browser, that's running a window, that's running a thing that I connect to a server, that runs the AI? What's all this intervening stuff about?”

“Some of my early, kind of crazy experiments were running AI on hot metal and then having an operating system that's just an AI with tools. Because I found that I would rather have that than have to poke around for that one thing, or that one file. Why don't we just change the idea of an operating system to basically be an intention system? Why not take the AI, surround it with tools, and give it the ability to talk to me and others?”

“I think that's the next thing that would be really transformative. Taking out all the shell layers I don't need between myself and an AI with compute. I need to be able to talk to it. I need to have rewrite ability. I know this because I've already built more than a dozen AI tools. Then you put that in a stack and run on hot metal. I’d be saving a lot of money just doing that.”

The post Can You Make AI Infrastructure Free Forever? appeared first on Heavybit.

On episode 9 of Third Loop, the Progressive Delivery team explores the complicated relationship between AI, automation, and human creativity. Kim Harrison, Adam Zimman, and Heidi Waterhouse discuss AI’s ability to lower technical barriers and reduce toil, along with its tendency to strip away context, reinforce sameness, and confidently produce answers it does not understand. Along the way, they encounter whale dictionaries, ancient scrolls, AI-written memos, autonomous tractors, and the dangerously comfortable “Yes Bubble.”

The post Ep. #9, Constraints, Creativity, and Competition appeared first on Heavybit.

More AI-Generated Code = More Software Performance Problems?

Studies suggest that some 88% of organizations have adopted AI, with some hyperscalers claiming that as much as 75% of their new code is AI-generated. Does more AI-generated code mean more performance issues and software regressions? Without guardrails, yes, says Arthur Pastel, co-founder and CEO of CodSpeed, a performance intelligence platform built specifically to help teams detect regressions and fix performance issues earlier in the SDLC. Today, CodSpeed is used by teams at OpenAI, Cloudflare and Vercel as well as leading open-source organizations like Mozilla and Pydantic.

The founder, a lifelong coder, explains that the project arose from a longtime interest in software performance. With the growing popularity of the Rust language, Pastel observed that while many interesting tools were being built, there was still no way to measure performance continuously in CI, which inspired him to build the first version of what became CodSpeed, and to found the company.

CodSpeed founder Arthur Pastel discusses software optimization at EuroRust 2025. Image courtesy EuroRust

Making Performance Visibility Continuous

Pastel suggests that the core problem that drove him to design CodSpeed is a lack of continuous visibility into performance during the software development life cycle. “Prior to CodSpeed, you really just had point-in-time performance data points.”

He explains that on a team, one member might run benchmarks, which are local and not committed, and because they are run on one specific machine, they aren't easily comparable with other benchmarks.

The second problem is a lack of continuous gathering of performance data. “For example you might have some production performance data, like in a continuous profiler or a production observability platform, but you don't have that data on a specific set of input data. So essentially, you can’t really track it over time and see how it evolved. (And that’s before you introduce factors like regression.)”

“This is the first problem we solved. Then we quickly figured out that people don't want to spend time fixing performance issues themselves. So we went further: We built a tool to surface the problems, and now we also remediate them and improve performance.”

Performance Engineering Matters Everywhere, Even in OSS

Pastel notes the importance of regression analysis goes beyond massive commercial software packages and is growing in importance among the open-source community. “It is really easy to convince people building open-source projects [of the importance of performance engineering] because they’re building foundational blocks.”

“So for example, if you introduce a regression in Pydantic (which is used by LLM providers) or in a really low-level library used everywhere, even a small performance regression will cascade everywhere.” Pastel points out that popular open-source projects are getting hammered with many more pull requests than before thanks to AI contributions, sometimes as much as 10x, which makes visibility into the performance impact of contributions extremely important for maintainers.

“Otherwise, in the beginning, we primarily spoke to organizations that were really sensitive to performance changes, which might be for UX or cost of infrastructure.” Orgs running infra at scale will generally be concerned about how performance affects their cloud bill, but LLMs have introduced more variation into the quantity (and quality) of things teams can “generate.”

“If you don't have any guardrails, then your performance might get completely out of hand, and you won't realize it until you're actually running on a production database.”

Accounting for Benchmark Noise and Memory Usage

Pastel explains that his team’s approach involves building a CPU simulation tool to address the 'noisy neighbors' issue of running on a shared VM whose measured performance might quietly change depending on what neighboring VMs are running. “It can be really painful, especially in overprovisioned environments like GitHub Actions and a lot of CI runners.”

He explains that a common practice to reduce costs is to overprovision instances, which can lead to extreme variations in benchmarks. “For example, you might measure something that, at one point, is 50 times lower than the baseline, yet the code doesn't change at all.”

The CodSpeed team's approach is to simulate the CPU to generate a model that, while not perfect, would represent actual execution. “We give a super-fast feedback loop of how the performance changed between two versions of code. And we built it to be really portable, so you can bring it from one computer to another, one server to another, and the results will be comparable as long as you have the same architecture.”

While CodSpeed works on x86 and ARM architecture, his team is actively working on bringing the project to be completely architecture-agnostic. “To do this, we forked Valgrind, which is a dynamic binary instrumentation tool. So instead of simulating and emulating a whole system, we integrate at the binary level, and we have some integration libraries which make it easy to build the benchmarks and the performance test.”

Pastel explains that the initial goal was to make it as easy as possible to build performance tests so that users actually write them, as even code-level issues (like compilers mistakenly trying to remove benchmarks) can get in the way.

“One of the main problems here can be when you observe more macro-level phenomena. For example, let's say you have thread contention with a lot of thread scheduling, in which case a simple CPU model is not always enough, though we want to push our modelization as far as possible. However, while the next steps may include having some proper isolated or bare-metal instances, at some point you still need to validate the macro behavior.”

“On top of simulation, you sometimes need to measure on real hardware. We have actual physical instances that we can provision on demand in isolated data centers where we can measure performance,” says Pastel. Even physical concerns like the height and location of your server within a data center can yield completely different performance patterns.

“The idea is that using CodSpeed, if you have a GitHub Actions workflow, with just one line, you can get an actual physical instance that's dedicated to you, without paying thousands per month just for dedicated hardware. We share those costs and make it affordable.”

“The last instrument, because our idea is really to build an ecosystem of instruments that can be plugged to your code, is the memory measurement instrument, which we built because we also had a lot of people complain about memory consumption. Often, if you optimize for speed, it will be a trade-off and you will just consume more memory. Depending on what you do, reducing your memory footprint may also be a requirement.”

“Memory shifted our thesis a bit to be more about optimizing performance, which we define as the amount of resources you use. So your goals might be optimizing for disk or network usage, and ours is to help you optimize your software to reduce the footprint it consumes within your own system.”

“If you introduce a regression in...a really low-level library used everywhere, even a small performance regression will cascade everywhere. If you don't have any guardrails, then your performance might get completely out of hand, and you won't realize it until you're actually running on a production database.” -Arthur Pastel, Founder/CodSpeed

How Technical Leads Should Think About Performance Engineering

Pastel breaks down the value of performance engineering for orgs into a few layers. “First, performance issues might just be hiding some bugs, possibly due to an algorithm not being properly implemented.” Code correctness can therefore be part of performance management, as can making sure the resulting complexity of an algorithm is what was intended.

“Then, if everything runs faster, then you need fewer servers and less compute to run something, which means you cut your costs, which is one key benefit.” Pastel observes that orgs running cloud infra at scale generally prefer tools that are fast and don’t overconsume memory, which is why teams he works with often optimize for speed first, then memory usage second after they start comparing memory footprints.

Lastly, performance engineering can directly affect user experience. “Depending on what you ship, whether it’s a Web application, even if it’s a backend server or something, the overall latency of your program will define a lot of the user experience. As long as there is a user interaction involved, UX can be improved because you can make things faster.”

Pastel gives examples of how CodSpeed helps front-end teams detect performance regressions in CSS rendering, as well as optimizing at the kernel level and with JS execution. “There are many things that can be optimized to just make the overall experience for the end user way more pleasant, which is something that's pretty important as well.”

For teams building software, performance engineering also provides multiple benefits. “First, with AI, you’re shipping more code than ever now. You're spending much more time reviewing, so you have much less time to focus on performance. And the idea is that, at first, you need guardrails on the most important parts of your application, whether that’s the time it takes to load a full page, or to perform the most basic interaction on your server via your back end.”

Pastel argues that as software teams have less time and resources to devote to improving the performance of their products, they at least need some kind of background job optimizing performance. He points out that developers now generate and review code with LLMs, but even frontier models have no runtime performance signal, neither in their training data nor as feedback when they generate code. They can't measure the code they generate, so they're left to guess at what performant software actually looks like.

“What LLMs need is a proper harness to measure performance, which is what we've built. We initially built this for humans, but now it works for agents as well. Essentially, we help the agents build a really simple harness around any kind of code, any kind of language, then let them query the data in a proper format.”

Agents don’t collect or consume performance data in the same way humans do, but the team still needs to surface them for users. As a result, they developed an MCP server, a CLI, and skills to let users run their own auto-research loop, to minimize the cognitive burden on already-overstretched development teams.

Why Aren’t Agents Already Doing This Stuff?

Pastel suggests gaps in today’s agentic tooling and infrastructure that prevent them from being proper guardians of software performance. “CodSpeed can run on open-source software and the tools we built in sandboxes in a super-cheap way. But agents would still need to interpret and aggregate the data, which is the first tricky part.” He points out that optimization work is hypothesis-driven. Even the world's foremost performance experts form a hypothesis from an observed code pattern, write the change, and validate it by measuring. Agents need that same measurement loop.

“But then, there’s also a whole infrastructure play, which is much harder. You need to have isolated machines to run more-macro tasks.” Pastel explains that CodSpeed operates a fleet of bare-metal machines tuned for measurement stability spanning various hardware configurations, architectures (x86, ARM), and GPUs. Agents can request an instance on demand and run benchmarks in a completely isolated environment, on the hardware that matches their production target.

Pastel also believes that the evolving economics of compute resources is constraining something that technical teams used to not consider a constraint at all: Available memory. “Just getting RAM now is getting harder. We never really took the time to optimize software in the past because it was super-easy to say, ‘My software is consuming a lot of RAM, but people will just get laptops with more memory, so it's completely fine. Same with CPUs; if things were a bit slow, you would just get a bigger instance.”

While there are faster CPU and GPU units in development, Pastel points out that compute and RAM are becoming scarcer and more expensive, requiring teams to treat them as constrained resources and focus on performance.

For decades, Moore's Law meant slow software could wait for faster hardware. That escape hatch is closing. As hardware gains slow down, performance has to come from the software itself.

“We never really took the time to optimize software in the past because it was super-easy to say, ‘My software is consuming a lot of RAM, but people will just get laptops with more memory, so it's completely fine. Same with CPUs; if things were a bit slow, you would just get a bigger instance.”

How Founders Should Approach Performance Engineering

For startup founders, Pastel notes that agents can process thousands of times more data than humans in the same time period, which means that both your startup’s agents, and those of your customers, could be dragged down significantly by performance regressions.

Pragmatically, he admits that performance engineering may not be a priority in the very earliest stages of a software startup. “Performance might be taken as an afterthought in the beginning, and it’s fine to [focus on building] a simple MVP, because first you want to validate that there is a need for your product.”

However, the AI era has created another wrinkle that makes performance significantly more important, especially in competitive spaces. “For customers you hope will adopt and actually keep using your product, migrating from one thing to another has become super-cheap. Essentially, you can just have an agent tasked to migrate from one solution to another. I think this is why a lot of SaaS products are at risk, because we don’t have the strong lock-in and switching costs we had before.”

“So, performance is still a key differentiator, especially when you think about adoption and retention. It’s still a big deal in my opinion.”

Performance Engineering in the Enterprise

Pastel sees performance engineering as only becoming more important for enterprises, especially on the infrastructure side at scale. “Enterprises have already realized the margin gains they can get from optimizing their software. We see adoption growing today, and it will only accelerate in the future.”

He also argues that the advent of new AI tooling is helping orgs shift performance left as well. “Previously, performance would come down to, ‘We’ll see what happens in production.’ And if there was an outage, someone would wake up at night and try to fix it. Now we see more and more AI SRE companies that fix such issues, rather than having someone being paged to wake up in the middle of the night.”

“We’re building something to fix issues before they even happen, which we believe is better, because you don't want agents doing stuff in your production instances when you are an enterprise. If you can catch issues before production, you can validate with a human before anything happens.”

“It just happens in a much more natural cycle, rather than thinking, ‘Okay, there is a fire, I will try to fix the fire, push in production again, and iterate with a super-long development cycle.’ With agents, we really can ship in minutes while before, especially for enterprises, the release cycle might take weeks, if not months.”

How Will Performance Engineering Fit into the Future Stack?

Pastel is confident performance engineering won’t be going anywhere. “You can't train models to optimize performance by design. Performance depends on what you actually run in production: Your data, your infrastructure, your product goals. That context has to be measured, not learned.”

The founder is skeptical that even a perfect agentic future will account for all the potential outcomes of performance engineering, even if agents eventually come to possess all the right tools to communicate and validate performance changes. “Even if we manage to have some super-optimized languages, and we have new state-of-the-art models that allow us to have some crazy breakthrough to do a lot of optimization, there is still the structure of the code, which will still be an issue and cannot be optimized by a compiler.”

Pastel suggests that performance engineering, like security, will be a discipline that may never become out of date. “It won't disappear because you still need to have guardrails and things to make sure that your app is [performant] and consuming the least amount of resources possible. This problem is not ‘done.’”

The post Is Early Detection the Key to AI Performance Engineering? appeared first on Heavybit.

On episode 3 of Lab Notes, Amir Zohrenejad sits down with Qizheng Zhang to explore one of the fastest-moving areas of AI research: recursive self-improvement. Together, they discuss Meta-Harness, context engineering, and why the future of AI may depend as much on the software surrounding models as the models themselves. The conversation also examines evaluation, agentic systems, and the limits of today's autonomous AI research.

The post Ep. #3, Exploring Recursive Intelligence with Qizheng Zhang appeared first on Heavybit.

Does AEO/GEO Matter More or Less than SEO?

It’s easy to find many deeply unhelpful statistics about the changing state of developer marketing. For instance, 33% of internet users reportedly discover new brands via internet search, yet Google AI overviews have apparently reduced clicks to top search results by 58%.

So what does any of that even mean?

Has SEO become worthless for startups? Has visibility in AI overviews (known as AI overview optimization, or AIO) or in LLM results, also known as generative engine optimization (GEO) or Answer Engine Optimization (AEO) replaced it as the most important way to sell developer tools?

Developer content expert Adam DuVander suggests that the numbers and rankings don’t necessarily correlate to value for your startup, especially if you haven’t positioned your company correctly. While he built the website LLMRank.fyi to help founders get a point-in-time sense of their visibility in LLMs, he argues that before even looking at rankings, founders should focus on strategy first.

Understanding What LLM Rankings Are (and Aren’t)

DuVander suggests that the website is the result of work he was already doing with his clients, startups that are, unsurprisingly, concerned about how highly they rank in LLM responses when users ask about products in their space. “We can see how these [rankings] change and when they don't. And I think both of those are interesting.”

The content expert also cautions against overindexing on any single data point, as LLM results are still mercurial and non-deterministic. “I heard a story of a startup on a retreat, where the team all opened up their laptops and typed in queries about their product into their favorite LLM, all writing different prompts from their individual points of view. And of course, they all ended up with different answers there.”

“As much as we all wish that a developer would type in the words that are in the H1 on our homepage, they often don't. (That's a whole other topic, because you should be able to use your website’s H1 to speak to that developer audience.) But the ‘good news,’ if you can call it that, is that [LLM visibility] is on a long timeframe. If you’ve worked in SEO, you’re used to long timeframes, but this is potentially longer.”

The content expert suggests that while LLM visibility takes time, it may be a good idea to start looking into the topic, but not to simply index on how highly their own products rank. Comparing differentials against competitors, particularly between different types of prompts, can highlight opportunities for improvement.

“The more interesting questions are: Where are you ranked above the companies that you consider to be your competitors? And where are they ranked above you? If you look back at the language for those, you can start to see narratives or gaps. Maybe this [prompt] was about a particular feature or a use case, and you're doing ‘better’ there.”

“That means that you likely have put in some effort there, possibly from content on your site. It might be outreach to devs, or devs who are talking about you positively in this area. And then there are prompts where you don't show up at all: Potential areas to fill with the same sorts of activity that you're already doing.”

Are LLMs simply a new marketing channel? “From the standpoint of: ‘We know that devs are using this to ask these questions, get recommendations, and potentially make decisions, all before they see your website, if they ever do, sure. But everything that goes into that recommendation are all the other channels that you already are working on: Your blog, documentation, and work in the community. What your DevRel team is putting out on GitHub. All these things are already existing channels, and this is a way to pull those together.”

“[LLM visibility] is on a long timeframe. If you’ve worked in SEO, you’re used to long timeframes, but this is potentially longer.” - Adam DuVander, Founder/EveryDeveloper

Why Positioning May Matter More to LLMs than Search

DuVander points out the differential between scouring for keywords, the way SEO experts did with Google, and using LLM prompts. “I've seen some AEO/GEO dashboards that just include SEO keywords. But thinking like a developer, not like a marketer or an SEO, and using [developers’] words, you should think about the use cases you have. I really think this is an opportunity to look at your positioning and make sure it's super clear.”

“I think it's okay if you don't show up at all in some of those searches that aren't right in the wheelhouse of a tightly positioned startup for the ‘ideal dev.’ Some of that basic work of knowing who that ‘ideal dev’ is is in the minds of the founders a lot of times, but especially as the team grows...that can shift as the product changes. You should be asking: Are we still as focused on that ideal dev as we could be?”

The content expert also notes what appears to be a semantic difference between SEO and AEO/GEO. In SEO, traditionally, the goal was to rank highly for a core keyword, which would then generally ensure ranking for any derivative sub-keywords. Ranking #1 for the term “data pipelines” would generally mean you would tend to rank well for associated keywords like “data sources” or “data ingestion.”

“That seems to be less of the case for AEO/GEO, since context is so important to an LLM. Add a couple of words to your prompt, and that changes the context. Now, you might be completely absent from sub-keyword LLM searches. And actually think that’s a good thing for founders who are positioning themselves in a way that differentiates them from another startup that might seem to be a ‘leader.’ It’s an opportunity to be able to kind of own that ‘zone’ and that positioning. SEO might have made that harder to do. The LLMs seem to be more willing to open that window.”

What Should Founders Do First? LLMs.txt, Markdown, Content

The content expert recommends that founders start out by recording their visibility in LLMs over time to build a baseline of data they can use to compare. “From within there, some of those gaps will point to areas you'll want to try to fill. That’s assuming that you're running these prompts multiple times and you feel confident the results are coming from ‘developer language.’”

“I know there are still a lot of questions about LLMs.txt and whether AI uses it. To me, one of the benefits of that is how teams can get on the same page about what they’re putting forward, and which things matter. So I think creating [a LLMs.txt file] that has categories that really show some thought put into it is worth doing.”

“If we're just talking about automating something that essentially looks like a site map? And I've also seen some teams just dump all their blog posts in a giant list. I don't think that is very valuable at all because it doesn't show that you've pointed toward a particular area. In everything you're doing, you should be looking to give those signals about what matters because if it all matters, we know that nothing matters.”

DuVander recommends thinking through website use cases where human interactions are still important. “What are the ways that you can streamline the process for [a human visitor] to get back to their agent that has maybe made this recommendation.” The content expert notes that adding markdown is a specific step founders can take to make their content LLM-readable, the way Stripe does for its documentation site.

Going forward, DuVander then recommends founders look at the potential gaps they’ve identified and build a plan to fill them in. “If this is a positioning issue where you're not showing up as highly as you would expect to, then I think you should make that a focus. You could have a series of blog posts or other content about that topic. Make a campaign of it. Don't just say, ‘OK, we'll make our one definitive page and be done with it.’”

The content expert clarifies that making progress against LLM queries where startups are underrepresented may require a continuous drumbeat of content and campaigns going across different channels, whether they be blogs, podcasts, hackathons, or other activities.

“If this is a positioning issue where you're not showing up as highly as you would expect to, then...you should make that a focus. Make a campaign of it. Don't just say, ‘We'll make our one definitive page and be done with it.’”

How to Think About Rankings in Competitive Spaces

DuVander admits that, like with SEO, AEO/GEO visibility against monolithic competitors can be challenging. “First, I’d recommend thinking about timelines, because it may take months or quarters to be able to really see any movements [in your rank] if you're invisible now. Having a super-entrenched competitor seems like it might still be as big a deal as being the top SEO result.”

For startups going head-to-head against tough competition, the content expert recommends digging deeper into areas where founders can make traction. “I would make sure I knew the use cases where we could win against that competitor. And I’d want to make sure it wasn’t just us saying that. Talking to developers, you can find there might be plenty of community chatter on places like Reddit, not even necessarily about your startup or your competitors, but just about the use cases.”

DuVander clarifies that Reddit (and every other community channel) isn’t a magic silver bullet, and reminds founders that such communities are often hostile to blatant promotion. “These channels have existed before LLMs and I think that it matters where your audience is. So if your audience is on Reddit and discussing it, yes, they won’t like it if you come in and talk about your products. But your own community might be talking there.”

The content expert suggests that one approach to community spaces is to highlight projects your community has worked on, such as by sharing projects they’ve successfully built using your platform. “But also, you should be looking at places that aren't Reddit, since your audience might be hanging out in other places.”

DuVander explains that rather than fixating on being everywhere, a better strategy is to find the few channels that matter most for your community. Particularly for smaller teams that don’t have the bandwidth to try to conquer Reddit and Hacker News and Product Hunt and Dev.to (and all the rest).

Preparing for the Future: Meeting Devs Where They Live

DuVander suggests that on some level, marketing may matter more than ever in the age of LLMs. “The big thing that has changed is that the workflow of most developers has changed. So this isn’t just about ‘another channel where you could be recommended,’ which would be a different conversation. But despite the changes, I don’t think established channels go away.”

“You still want to show up when the developers search. When they go to Google, maybe they get a recommendation in one window and they think, ‘Let me see about that.’ There are still devs who are going to be doing their research and wanting to make sure that they're making an informed decision.”

“Yes, if they trust this LLM to write code, they probably trust it to make recommendations too. But yeah, they're going to still be doing those other things: Going to conferences and hackathons. Classic DevRel sorts of places. From the conversations I've had, it's much more gearing around marketing now: Being able to build that authority and trust with the audience that is needed to show up in the LLMs and be recommended.”

The content expert also suggests thinking ahead to the next step: The deep, in-the-weeds context that’s already contained within the agentic tools developers work with every day, and the hidden opportunities to partner with related communities. “What else is true in that developer's context and that moment? They are in their own codebase. The agent knows the languages and tools they're using.”

“I think that's an underexplored area. You’re not just getting to a use case, but also: What language is someone building in when they do this? What other tools are there? So those are our partnership opportunities [for those other tools]. Which goes back to the idea that everything old is new again. These aren't new ideas, but it's about framing things in a way that is aware of what a developer’s context is, and what their agents’ context is.”

“I’m thinking about partnerships with other [complementary] tools that would make sense alongside yours. If both companies are talking about the use case, and their devs are talking about it, you start to build a lot of the training for that. LLMs now see these two connected together: They see ‘Your Product with Python,’ or ‘Your Product with Rust.’ And the LLMs are more likely to make that recommendation if it's from that context of knowing the language that developer is using. So that's I think an underexplored area, taking not just the use case, which already is a slice of the larger approach, but going even further.”

The post How AEO/GEO Differs from SEO (and Whether It Matters) appeared first on Heavybit.

Performance Engineering: Part Brute-Force, Part Hard Decisions?

Software startups often launch quickly to beat competitors and gather early customer feedback. At that stage, performance should not necessarily be the priority, founder Tomás Senart argues. But once a product is exposed to scale, or latency and infrastructure costs become material, the economics change and teams need a repeatable way to keep performance from drifting. The creator of the popular Go HTTP load-testing tool Vegeta has carried those lessons into his startup Perfloop, and shares them here.

Tomás Senart discusses his learnings in Go interfaces at GopherCon 2015. Image courtesy Gopher Academy

From Load Testing to Go

As an engineer at SoundCloud, Senart had been tasked with replacing a significant chunk of a legacy system that needed to be loaded and tested. “At that time, the only thing available that was not a [heavyweight] Java ecosystem tool was Apache Benchmark, which had an array of usability issues and functionality limitations. We had a concrete problem and a culture of building things from scratch. Those two things together led to Vegeta.”

While the project has become popular in the Golang community, the founder clarifies that there was more to its success than the inherent speed and transparency of the language itself. “I think you can do performance engineering in any language ecosystem. With Go, because concurrency was such a core part of the design of the language, it made building a load-testing tool very natural at the time.”

“But you can build a load-testing tool in a way that will ‘lie to you’ in the results. So it wasn't just about building something quick, it was about building something correct. Azul Systems CTO Gil Tene is famous in the performance engineering world for many things, including the concept of coordinated omission: A phenomenon in load testing (and many tools fall into this trap) where the client that is sending the requests paces itself according to how the server is doing.”

“That is a bad way to test the system, because you're basically stopping to hammer the server at the exact point where you'd get the [indication] that it's not doing well. So you need to keep the rate constant. That was a key design decision at the time to build Vegeta in this way: To not suffer from that problem.”

The founder suggests that the most important lesson from his earlier work wasn’t that performance data is lacking, as most teams have dashboards full of telemetry and a benchmark they’ve run at least once. “The problem is that the benchmark you ran once is already stale. Performance is not something you set and keep, it is an envelope you have to hold against constant change.”

“Every code change, dependency bump, traffic shift, and architectural change nudges it, so the loop from measurement to a verified improvement has to run again and again just to stay in place.” Senart explains that the measurement process is inherently slow, which makes it easy to deprioritize and fall by the wayside.

His startup Perfloop closes the second half of the loop: After measuring, Perfloop assesses how systems behave under load, across the entire loop. “It finds the performance opportunities, proposes changes, and proves whether they actually moved the needle, with the numbers to back it up.”

Providing proof, the founder argues, validates automation across the rest of the process, rather than forcing teams to take things on faith: An important nuance as benchmarking loops need to be run far more often than can be done manually. “The system carries the ‘relentless’ part, the finding and the proving, while the person stays in charge of what is worth holding and what actually ships.”

How Much Should Tech Leaders Prioritize Benchmarks, Really?

Addressing the excitement around AI evaluations, the founder notes that the most popular AI benchmarks can and do become contaminated. “That's why people started creating private eval sets that they run so they're not gameable. That’s one approach, but it has its own problems, because there is no transparency into the quality of those benchmarks. It's a moving target, honestly: How to have transparency while not being gameable or having evals end up in the training data of models themselves.”

Senart draws a line between viral LLM benchmarks like Deep-SWE and benchmarks that teams specifically use internally to build their own products. “If you're building these capabilities in your company and in your product, the only thing that matters is your evals. It's not going to be the public coding agent evals for the coding models that matter the most.”

“The benchmarks that last are the unglamorous, private ones tied to an outcome you actually care about, because there is no incentive to game a benchmark whose only audience is you. Public benchmarks help you decide what to test. Your own codebase tells you what is true.”

"The only thing that matters is your evals. The benchmarks that last are the unglamorous, private ones tied to an outcome you actually care about, because there is no incentive to game a benchmark whose only audience is you. Your own codebase tells you what is true.” -Tomás Senart, Founder/Perfloop

Performance Engineering Lessons for Founders

Senart suggests that the adoption of agentic coding agents is fundamentally changing software, and not necessarily for the better. “First, agents now write a large and growing share of code, with a very wide range of outcomes. Many teams are moving faster than their ability to review every line, so they are shipping performance slop: Inefficiencies and waste that no human deliberately introduced and no one is specifically watching for.”

“Second, the load on production systems is changing shape. It is not only human users anymore. Agents call services at machine speed and volume, so the same systems get hammered far harder than they were designed for.” The founder observes that bot Web traffic has surpassed human traffic, and Web-based infrastructure is only the first beachhead to be pounded on by agents.

Commerce is another rapidly-growing space for agentic workflows for retail and checkout processes, and enterprise services appear to be the next frontier. The founder argues that agentic traffic is no longer a hypothetical future condition. It’s an inevitable thing that will happen, and make performance even more crucial.

In addition, it’s no longer feasible to wait to be ‘bailed out’ by eventually better hardware. “Now GPUs, memory, and compute are expensive and staying that way, so inefficiency shows up directly as a cost on your bill.”

The above factors lead to more performance debt that accrues quicker on infrastructure taking heavier workloads. “What is missing is a way to keep up. Occasional expert attention or a periodic consultant audit cannot match how fast the debt now accrues. That is the gap Perfloop is built to close.”

The founder clarifies that while you can use agentic systems to support performance engineering, AI doesn’t nullify the fundamentals. “You still have resources, you have things that drag on them, and you’ve got to make the most of them. You need to unpack that, one layer at a time, and be methodical. That's what performance engineering is.”

“You still create hypotheses, you go test them, and you look at the hard data. It’s just that the experiments can go faster now. You have agents which, with the right harness, can do that kind of work rigorously and not just guessing and going in the wrong direction, then you can parallelize so many of these hypotheses for testing.”

Senart offers two final pieces of advice for founders. “First: Do not pay for performance you do not need yet; if nobody uses your product, its performance does not matter. Second: The moment you are exposed to scale, the economics change in your favor. Performance work is methodical science, and the old constraint was that you could only afford to test the top one or two ideas. With the right harness you can now run many of those experiments in parallel, with rigor, and test the other ten you used to ignore.”

Getting Performance Engineering into Enterprise Customers’ Hands

Senart concedes that his startup’s approach to automating performance engineering isn’t necessarily mainstream among enterprise engineering orgs yet, due to a number of factors. “The first is trust, which is the hard one. You are asking a large organization to let an autonomous system reason about and propose changes to code that runs their business. They will not do that on a verbal promise.

“They need proof that the system finds real problems and that its proposed changes actually work, and they need that proof to come from somewhere they cannot dismiss. This is exactly why we are doing the work in open source, in public.” The founder notes that the maintainer of the widely-used Go library parquet-go has publicly agreed with his hypothesis that closing performance debt will come down to utilizing the kind of concurrent optimizations he’s building today.

“The second blocker is the usual enterprise gate: Compliance, security review, governance. There is nothing special here, it is just work you have to do to be allowed in the building. We are starting SOC 2 now for that reason.”

“The third blocker is not so much a blocker as it is inertia: Simply the reason to bother to do anything. Enterprises are actually a strong fit, because their scale and internal complexity produce a lot of recoverable inefficiency. In a large system, the performance you have left on the table is real money on the bill. Once trust and compliance are cleared, that recoverable cost is a very concrete reason to engage.”

“What it takes [to get into enterprises] is portable, verifiable proof on code that looks like theirs, compliance work done honestly, and a value story told in the language they care about, which is cost and risk, not benchmarks for their own sake. For us, this is the route, not the first stop. The immediate motion is the public open-source work and self-serve adoption. The enterprise conversation opens once that public credibility is built.”

"[For enterprises], the first [blocker] is trust, which is the hard one. You are asking a large organization to let an autonomous system reason about and propose changes to code that runs their business. They will not do that on a verbal promise. The second blocker is the usual enterprise gate: Compliance, security review, governance. The third blocker is not so much a blocker as it is inertia: Simply the reason to bother to do anything.”

What the Future of Performance Engineering Looks Like

Senart understands the current excitement about AI agents in software, but isn’t sure that performance engineering will simply become another agentic guardrail. “Guardrails are part of it, but the real question is where the human sits on the spectrum between specifying everything up front and steering every step (and both ends fail).”

The founder offers the two extremes: The ‘dark factory’ that is prompted once while humans step back, without human judgement or context along the way. The other is today’s common practice of endlessly babysitting coding agents edit by edit, which preserves human judgement but can’t scale, turning senior engineers into bottlenecks.

“What actually amplifies a senior engineer is in between, and it is the harder thing to build. The human stays on the decisions that carry judgment, which tradeoffs to accept and what is worth holding, while the system runs the steps underneath and surfaces only the calls that need a person. Present where judgment is required, absent everywhere else. Not steering more, steering higher.”

“Performance needs exactly this, because the target keeps moving. You cannot freeze the judgment into an up-front spec when the envelope shifts with every change, but you also cannot re-derive every step by hand each time. So you want a system that does the relentless step-level work and pulls the human in precisely when a real tradeoff is on the table.”

“That is what Perfloop is built for. The whole loop runs as an MCP server, so the pieces are callable on their own: A person or an agent can point it at one slice of code, or hand it a rough hunch about what might be slow and get back a benchmarked case. Autonomy is a dial the customer controls, from scoping, to directing, to reviewing, to fully delegating, which is really just a way of choosing how high on that spectrum you sit.”

“My honest bet is most teams land at ‘direct’ or ‘review’ for anything that matters: Let the system carry the steps and the proof, but keep their hands on the decisions that need judgment. So performance engineering won’t shrink into a guardrail. It will remain a loop that humans steer from above, with far more reach than before.”

The founder notes two additional factors that will affect future software projects. “First, the set of teams exposed to scale is expanding fast, because agent traffic drags services that used to be ‘safe’ into the deep end. Second, for the teams already at scale, the economics changed underneath them: The debt piles up faster, because agents write more code with less scrutiny per line, and it no longer depreciates the way it used to, because you can no longer count on cheaper, faster hardware to absorb it.”

“Performance debt that used to quietly depreciate now accrues interest. When a problem goes from occasional to continuous, how you handle it has to change too. A once-a-year expert audit cannot keep up with debt that compounds. Perfloop is my bet on what replaces it: The rigor a great performance engineer brings, available to any team that needs it, with the human keeping the judgment and the system carrying the grind. The open-source work is where we are proving it in public, one verifiable win at a time.”

The post Will AI Manage Software Performance with Humans in the Loop? appeared first on Heavybit.

On episode 57 of Generationship, Rachel Chalmers sits down with Navneet Kaur, founder of FemTech India and TechThrive Ventures. Together they explore why women's health innovation must extend beyond Western markets, how AI is reshaping femtech, and what founders need to understand about trust, culture, and personalization when building global healthcare companies.

The post Ep. #57, Attention is Currency with Navneet Kaur appeared first on Heavybit.

On episode 8 of Third Loop, the Progressive Delivery team sits down with Melinda Fekete to discuss FeatureOps, developer experience, and modern software delivery. The conversation covers feature flag lifecycles, runtime control, experimentation, chaos engineering, and why progressive delivery is becoming even more important in the age of AI-assisted development.

The post Ep. #8, Sounds Like Alignment with Melinda Fekete appeared first on Heavybit.

On episode 92 of o11ycast, Ray Myers joins Ken and Jess to explore how observability, reliability engineering, and formal software engineering practices are becoming even more important as AI coding agents take on larger roles in development. Rather than viewing AI as a replacement for established engineering disciplines, Ray argues that techniques like continuous delivery, testing, specifications, and formal verification provide the confidence needed to safely harness AI-generated code.

The post Ep. #92, Confidence Is the New Bottleneck with Ray Myers appeared first on Heavybit.

On episode 2 of Lab Notes, Amir Zohrenejad sits down with Hanchen Li to explore the systems that make modern AI agents faster, more efficient, and better at learning from experience. They discuss KV Cache optimization, long-context inference, prompt learning, continual learning, and why better benchmarks may be just as important as larger models in advancing AI research.

The post Ep. #2, The Infrastructure of Intelligence with Hanchen Li appeared first on Heavybit.

On episode 40 of Open Source Ready, Brian Douglas and John McBride sit down with Orhun Parmaksiz to explore Ratty, his experimental GPU-rendered terminal emulator that combines traditional command-line workflows with interactive 3D graphics. They discuss the future of terminal user interfaces, how AI is accelerating developer tooling, and what recent supply chain attacks on Arch Linux reveal about the evolving security landscape for open source software.

The post Ep. #40, Terminal Innovation in the AI Era with Orhun Parmaksiz appeared first on Heavybit.

On episode 7 of Third Loop, the Progressive Delivery team speaks with Nathen Harvey about the intersection of AI, DevOps, and user-centric software development. Together, they examine what DORA's research reveals about high-performing teams, why shipping faster doesn't automatically create more value, and how organizations can build stronger feedback loops with users. Along the way, they discuss agentic systems, platform engineering, and even AI-powered kitchen appliances.

The post Ep. #7, Exploring DORA and Progressive Delivery with Nathen Harvey appeared first on Heavybit.

On this debut episode of Lab Notes, Amir Zohrenejad is joined by Ren Wang, a researcher and PhD student at UC Berkeley, to explore the state of physical AI and why robotics has progressed differently from large language models. They discuss data scarcity, world models, simulation, dexterity, and the challenges of building robots that can reliably operate in the real world.

The post Ep. #1, Why Robots Are Hard with Ren Wang appeared first on Heavybit.

On episode 39 of Open Source Ready, Brian Douglas and John McBride speak with Zach Smith, creator of Kplane, about rethinking Kubernetes for an AI-driven future. Zach explains how virtualized control planes could enable isolated cluster experiences at massive scale, while the conversation explores developer experience, AI tooling, and the possibility that future AI agents may each require their own cloud.

The post Ep. #39, Agents Take the Wheel with Zach Smith appeared first on Heavybit.

On episode 56 of Generationship, Rachel Chalmers sits down with Mark Brocato, founder of Mockaroo and creator of Fabricate, to explore the evolution of synthetic data in the age of AI. Mark shares how a simple internal QA tool grew into one of the most widely used synthetic data platforms and discusses how agentic AI is transforming software development, testing, and data generation.

The post Ep. #56, Vibe Coding for Data with Mark Brocato appeared first on Heavybit.

Giving AI Data Ownership Back to Users

Researchers have warned for years: Frontier models are potentially running out of data to train on. However, the apparent lack of data hasn’t stopped AI labs from endlessly launching newer, shinier models with endlessly improved capabilities, which, in turn, makes switching costs painful for users who have their data locked up in a previous-gen version of a competitor’s model.

After significant experience working in general software development and then in localized inference, veteran developer Terence Pae began working on turning his own Mac into a personal AI harness, freeing him from vendor lock-in from increasingly commoditized models and giving him significantly more control and ownership over his data and permissions. Now his open-source project Osaurus has passed 140,000 downloads and 6,000 GitHub stars, all with zero paid acquisition.

What if your MacBook was your AI harness? Image courtesy Osaurus

Why Vendor Lock-In Has Made AI Development Painful

Pae explains that the project came from his own concerns as a developer who was heavily dependent on third-party harnesses. “I'm such a heavy user [of AI], and it’s easy to not really think about all the data I might be sending out and the potential implications there.”

“As AI evolves and grows more powerful, it’s constantly taking in all this information from users to make models better. But at the same time, as an individual user, I’m wondering: Where is my privacy? Where is my data going and where is it being stored?”

The founder also notes that vendor lock-in benefits the frontier labs while making switching between models painful. “These AI companies are building moats, and they’re not using their data to do that. They’re actually using your data. The thesis with Osaurus is that as a user, you should be owning everything that is related to your data, your code, your context, your tools, and everything else. All you really need is the inference, whether that’s running locally on your hardware or in the Cloud, so you’re not dependent on a specific provider.”

Lessons Learned from Decentralized Infrastructure

The founder notes that deciding to go with a building-in-public approach to open source was the key to growing his project, and that his work in decentralized infrastructure led him to focus on local-first AI development. “There are many ways to build AI harnesses. For example, you can write code that runs on the Cloud, which depends on having a certain infrastructure. And you need a database to host your user's data. But Osaurus is, itself, a client and a server that runs locally, and spins up a SQLite database locally. Everything is self-contained on your device.”

Pae draws on his experience with peer-to-peer systems, where every node acts as both client and server. "Each node can connect with the others and create a network where no individual node has to rely on a centralized server. And if an AI is meant to be owned by the user and designed for the user, it also needs to act like a node that doesn't depend on a centralized service. That software should still work even if the company behind it gets hit by a meteor."

Pae's design approach assumes the future of AI won't always have a human in the loop. "Right now, we're relying heavily on a chat interface with agents and having a human in the loop constantly interacting with them. But I believe the next evolution is going to be more about agent-to-agent communication. For agents to be truly autonomous, they'll have to rely on other agents."

The founder’s experience working with cryptography informed his design. “Basically every agent will have an identity with a public key and a private key, so that agents themselves can verify each other. Being able to use a signature to verify each other has many benefits. But the key benefit is just making sure that this agent is talking to the right agent.”

“If my agent is reaching out to somebody else's agent, it needs to be able to verify that that particular agent is owned by that person. Designing in those specific protocols helps us to get to where we need to be in terms of agent-to-agent collaboration. Those are the basic building blocks I'm setting in place, but the future vision is about giving agents the autonomy, the guardrails, and the ability to act on your behalf without compromising or being compromised by other systems.”

I believe the next evolution is going to be more about agent-to-agent communication. For agents to be truly autonomous, they would have to rely on other agents.” - Terence Pae, Founder/Osaurus

Is On-Device Harness + Compute a Viable Alternative?

Pae acknowledges the growing challenges of cost management in AI operations, including organizations burning through massive token budgets utilizing third-party vendors. “I think [on-device] could absolutely be an alternative. In my opinion, the current model of scaling and building all these data centers to power all the demand in the world doesn’t seem very feasible.”

“I think the rate of agentic use is only going to go up from here. As a software engineer, my AI costs used to be about $15 monthly. Now, between cloud subscriptions and API usage, I'm paying $3,000 to $5,000 a month. That's just me as an individual engineer, and it's exactly the cost curve that makes local inference worth taking seriously.”

The founder suggests that while token economics seems increasingly likely to affect SaaS-style pricing, AI computing at the edge may also be competitive soon. “Local AI has improved almost exponentially in terms of intelligence per watt. For example, the recent Gemma 4 models run under 24GB of RAM and match what frontier models could do only a generation ago. That capability now fits on a laptop.”

Pae also points out that not every use case demands a trillion-parameter, general-purpose model. “You may just need a good-enough, smart-enough model to help automate things. And it's the harness's job to really bring out the potential of the model to not only increase capabilities, but also reduce some of the dependencies as well. In the future, we’re all going to be less dependent on the Cloud and more dependent on local infrastructure, in my opinion.”

Getting On-Device AI into Enterprise: Models, Tools, Education

While the promise of having full control over your data for daily AI tasks, without interruptions from cloud service outages, seems like an easy sell to individual developers, the founder concedes that risk-averse enterprises may need more nudging to actually adopt local-first AI.

“I think there are three issues preventing enterprise adoption. First, the local models themselves: There’s constant innovation happening here. The models may not be fully ready today, but maybe soon, possibly in the next year. Second, there don’t seem to be enough tools and harnesses that can work with local models. Major frontier labs like Anthropic and OpenAI not only have models, but also have tooling and infrastructure to be efficient and creative with their offerings.”

“Third is a lack of education: People may not know that these capabilities are good enough to use today. We speak with a lot of enterprises, including family offices and healthcare clinics that deal with sensitive data. The most common theme is there’s just no trust in the Cloud.”

The founder shares that cloud providers and their 30-day data retention policies still seem to rub enterprise customers the wrong way in terms of risk exposure to court subpoenas and the like. “As a company, that’s a huge risk. As an enterprise, you lean towards trying to have no risk at all.”

In contrast, having a private AI instance means sensitive work can stay on local infrastructure and never touch the Cloud. Osaurus supports both local and cloud inference, but the user decides what leaves the device. “Unlocking private AI with a full-featured agentic harness is going to really unlock the productivity that these enterprises are looking for: Having a private AI infrastructure with a frontier-level model, with a frontier-level harness that they can actually use internally.”

The founder also asserts that not all enterprise customers are stodgy, middle-aged, middle managers who are terrified of using AI at all due to risk. “The people we talk to definitely want to use AI, and want to simplify their workflows. They're really curious about it. A lot of the people that we chatted with use AI personally, but they don't use it for work. So they know AI’s capabilities.”

Pae recalls starting initial conversations with enterprise customers with his own preconceived notions: Would enterprise users seek very specific things, similar to familiar tools like Microsoft Excel? “Actually, most of the feedback we got was that [enterprise customers] want a ‘general AI,’ a general harness that is able to do what ChatGPT does. The people we talk to are looking for something more broad in terms of applications.”

Three issues are preventing enterprise adoption: the local models themselves, not enough tools and harnesses that work with local models, and a lack of education."

Building AI Products for Startup Founders

Pae is direct in his advice to startup founders looking to start companies around AI-adjacent products: Don’t build on top of someone else’s walled garden. “I would say treat inference almost as a building block: You can have a query, and then it returns a specific output in a function. Otherwise, you don’t want to build software in a way that locks you into a single provider.”

“You don’t want to build products in a way that’s super-dependent on specific functionality [from one vendor]. You want to build [components] in a way that is easily replaceable. Inference should be treated that way as well. Building an agent? You should design it in a way that you can easily replace your inference provider, and in my opinion, also have the ability to swap in local models as that may make things a bit more defensible.”

“Vendors like Anthropic and OpenAI may have some really cool features, but not every AI vendor will have the capabilities to build something similar. If you offer functions that can work with many different providers, you can also build in a way that targets specific audiences, such as the rapidly-growing open-weight market in China.”

“Being model-agnostic has worked out for us because there are different communities using these different open-weight models. We’re not really bothered if Anthropic suddenly launches a feature similar to ours, because they aren’t likely to also support MiniMax or other models.”

“In a larger sense, I’ve been thinking that software isn’t exactly ‘defensible’ anymore, since you can potentially create anything with enough attention and time. If we treat software like a true commodity, I think that's not going to be scalable for the long term. What is truly defensible is the community that you build around your software and also the transparency you can provide. The community that forms around you gives you that ability to push forward.”

The post What if the AI Harness Was Your Computer? appeared first on Heavybit.

On episode 13 of Data Renegades, CL Kao sits down with Josh Wills to explore how AI agents are reshaping software engineering, data science, and data infrastructure. They discuss verification, benchmarking, pre-training data pipelines, multimodal AI, and why understanding a problem may matter more than writing code. Josh also shares lessons learned from decades of building large-scale data systems and his journey from management back to hands-on engineering.

The post Ep. #13, Building Trillion-Scale Data Pipelines with Josh Wills appeared first on Heavybit.

On episode 6 of Third Loop, Kim Harrison, Adam Zimman, James Governor, and Heidi Waterhouse sit down with Betty Junod. Together, they explore how agentic coding tools are enabling non-developers to build software for themselves, creating a world where the ideal customer profile might be just one person. They discuss personalization, Progressive Delivery, open source, and what happens when users become builders.

The post Ep. #6, The Weird Edges of Taste with Betty Junod appeared first on Heavybit.

Moving Memory From Individual Apps to Centralized Infra

Despite all the enthusiasm for AI agents, recent reports suggest that only a fraction of enterprises have bothered to put them into production for a variety of reasons: Reconciling how most orgs’ data is locked up in legacy systems; constraints around data architecture and flows; and the usual enterprise-level governance and compliance concerns.

But agents themselves still struggle with a variety of performance issues that keep them from being the durable, reliable workers of the future they’re supposed to be. One glaring example is agents’ memory, or lack thereof, between jobs. Even after successful runs, agents by default begin their next jobs with a severe case of amnesia.

While some orgs have taken the approach of building agentic memory into specific apps, there’s a growing need for a wider, infrastructure-based approach for agents that can remember how to do things correctly. Open-source builder Heinrich Krupp tackled this challenge with his project MCP Memory Service, and explains his approach below.

MCP Memory Service provides persistent semantic memory for agents. Image courtesy mcp-memory-service

MCP: Generally Useful, But Where’s the Memory?

Krupp, an infrastructure and data management veteran, explains that his day-to-day work with enterprises on cloud infrastructure, delivery, and provisioning resources led him to start experimenting with MCP in December of 2024.

“My immediate reaction was: ‘This is genuinely useful!’ I had been working with AI tools for a while and had noticed a consistent gap. There was no memory layer. So I sat down with Claude, brainstormed the scope, and started building.”

The builder’s design thinking prioritized a privacy-first approach and open-source tooling, which led him to disqualify ChromaDB as a vector store for being too heavyweight and tack toward sqlite-vec for performance.

“I juggle a large number of projects with constant context switching, and I needed a reliable way to store important facts, decisions, and discoveries so they’d be accessible in the next session. When you’re managing Terraform configurations, security compliance, and customer-specific architecture decisions across 50-plus enterprise accounts, losing that context at the start of every AI session is a real tax on your work.”

After experimenting with different versions of Anthropic Claude, the builder found a fit with Claude Code’s hooks, embedded commands that users can build in across a Claude project’s lifecycle. Using this feature, Krupp built “Memory Awareness Hooks” to automatically inject context at the start of new sessions and keep learnings persistent. “This effectively gives Claude Code a persistent brain. That was the breakthrough that made the whole thing feel complete.”

Building Memory that Doesn’t Start from Zero

Krupp notes that his agentic sessions always seemed to start from nothing: No matter how many explanations he’d enter into the prompt, after a few dozen tool calls, the context window would fill up, he’d restart, and begin all over again.

“The tools that existed at the time either treated memory as a cloud subscription (proprietary, per-call pricing, data leaving your infrastructure) or as a simple markdown file you maintained manually, which is just organized forgetting with extra steps. Nobody had built memory as actual infrastructure that was self-hosted, semantically searchable, accessible via a standard API, with a lifecycle that includes consolidation and forgetting.”

The builder suggests that AI memory seemed like a personalization feature for chatbots, not a shared data layer for multiple agents, tools, and sessions. “The decision that shaped everything else was treating memory as infrastructure rather than a feature.” Krupp set about designing a REST API first (with 76 endpoints) so any agent, client, or language could interact with it without the need for specialized SDKs. “MCP support came on top of that, not instead of it.”

“On the storage side, the core tension is retrieval precision against latency and cost. We use local ONNX embeddings (sentence-transformers’ all-MiniLM-L6-v2, running fully on your hardware). No API calls per query, no per-token cost, no data leaving your infrastructure. The trade-off is embedding quality versus a frontier API model, but for most memory retrieval tasks the local model is more than sufficient, and you get 5ms read times against sqlite-vec.”

Krupp also designed a hybrid backend that bridges the gap between local and cross-device. “You want local speed for reads but you want cross-device sync. So the architecture separates concerns. sqlite-vec handles all reads at 5ms, a background process syncs to Cloudflare for durability and multi-device access.” By separating the MCP server and the network service at runtime, the builder avoided ‘database locked’ failures.

“The toughest trade-off was memory granularity. Storing at the turn level gives fine-grained retrieval but spreads a session’s semantic signal across many entries, which hurts benchmark scores on things like LongMemEval. Session-level storage scores higher on those benchmarks but gives you less precision when you want to know exactly what was said about a specific topic.”

Krupp argues that for memory granularity, chasing benchmark scores misses the point. “In production, fine-grained retrieval with a good quality scoring system outperforms coarse session storage on the tasks that actually matter.”

“The knowledge graph layer was an unexpected evolution. It started as a way to link related memories, but once you have typed edges (causes, fixes, contradicts, supports) you can query causal chains, not just similar facts. That’s a qualitatively different capability.”

Job-to-Job Memory vs. Context Rot

The holy grail in agentic engineering appears to be performant, reliable long-horizon agents that are immune to the context rot and hallucinations that set in with prompts that are too long and complicated. Krupp clarifies that MCP Memory Service’s agentic memory isn’t the solution to this challenge, but to a different one.

“Memory at the MCP level doesn’t solve in-session context rot. That’s a different problem involving attention mechanisms and context window management. What it does solve is the inter-session problem: Why does every new session or new agent start without knowing what the previous one learned?”

“The practical mechanism is what we call context providers, automatically injecting relevant memories at session start based on the current project, the current files, the recent activity. In our setup, this reduces context token usage at session start by 65% or more. The agent already knows the architecture, the recent decisions, the known failure modes. You don’t re-explain; you continue.”

“For multi-agent pipelines, the knowledge graph becomes the coordination layer. One agent stores a learning with a specific tag. Another agent retrieves it without any direct inter-agent communication protocol. There’s a user in our community running a five-agent cluster where they use the memory service as both shared state and an inter-agent messaging bus. Cluster agents write to a sentinel tag, the local agent polls that tag. That pattern emerged naturally from the tagging system; nobody designed it explicitly.”

The builder clarifies that memory isn’t the only missing piece for consistent agentic performance. “You also need good task decomposition, clear agent handoff protocols, and checkpointing within long runs. Memory is the persistence layer. It doesn’t replace the orchestration logic. But without it, every long-horizon task is fighting against statelessness at the infrastructure level, which makes everything else harder.”

Memory...doesn’t replace the orchestration logic. But without it, every long-horizon task is fighting against statelessness at the infrastructure level, which makes everything else harder.” -Heinrich Krupp, Creator / MCP-Memory-Service

The Role of MCP in Agentic Memory (and General Agentic Ops)

“MCP matters because it gives memory a standard interface that AI clients can discover and use without custom integration work. Before MCP, connecting a memory service to an AI tool meant writing glue code for every client. Now Claude Desktop, VS Code with Copilot, Cursor, and a dozen others can connect to the same server with a config entry.”

Krupp cautions that MCP itself may not make sense to be the lynchpin of software architecture for a number of reasons. “The REST API is equally important and more universal. If MCP evolves, is superseded, or simply isn’t supported by a particular client, the memory infrastructure still works via HTTP. Twelve MCP tools sitting on top of 76 REST endpoints is the right architectural layer separation.”

“MCP does some things well, like discoverability, tool schema negotiation, and integration with AI client ecosystems. But MCP doesn’t replace a stable HTTP API for programmatic access, agent frameworks that operate outside MCP-aware clients, and enterprise environments that need REST for their existing auth and proxy infrastructure.”

“In the long term, I think MCP will become a standard the way REST became a standard, ubiquitous but not special. The interesting competition will be at the memory content and quality layer, not at the protocol layer.”

How Startups Should Think About Agent Memory Economics

Krupp suggests that the right price tag to stand up an early, no-frills agent memory system is $0. “The default setup I use of an SQLite-vec backend with local ONNX embeddings costs essentially nothing beyond the server it runs on. No per-call API charges, no vector database subscription, no data egress. For a startup in early development, that’s the right starting point.”

“The economics change when you need team sync and multi-device access. The Cloudflare backend adds real costs, but at typical startup scale, it’s in the range of single-digit dollars per month. That’s worth it when you have multiple developers or agents sharing a memory layer.”

“The comparison that matters for startups is against the alternative: calling a frontier LLM to regenerate context on every session. Memory retrieval at 5ms with zero API cost replaces a non-trivial number of LLM tokens per session. If you’re doing active AI development with many sessions per day, the token savings pay back the infrastructure cost quickly.”

“My practical advice is that teams shouldn’t over-engineer the memory layer early. Start with semantic search and basic tagging. Don’t build a full knowledge graph before you have 1,000 memories worth organizing. The consolidation and quality-scoring systems are powerful but they’re optimization layers; the core value is simple, persistent, semantically searchable storage. Get that working first, understand your retrieval patterns, then add complexity where it actually helps.”

The builder adds: “Self-hosting changes the unit economics entirely. Proprietary memory APIs charge per storage operation and per retrieval. At scale, that adds up. Open-source, self-hosted infrastructure with local embeddings is a fundamentally different cost structure that startups should think about seriously, especially if memory is a core part of the product.”

How Builders Should Approach Agent Memory Economics

Krupp reiterates that the biggest mistake that builders make today is treating agent memory as an individual app feature, rather than as centralized infrastructure. “When memory is a feature, it belongs to a user, lives inside an application, and gets designed around a single interaction model. When memory is infrastructure, it’s a shared layer that multiple agents, tools, and workflows can read from and write to. The architectural implications are completely different.”

The second biggest mistake, says the builder, is storing everything without a quality model. “Raw storage without scoring creates retrieval pollution. You end up with a database full of session noise, test outputs, and transient observations that contaminate your search results. Memory needs a lifecycle: Store, quality-score, consolidate, and eventually forget. The ‘forgetting’ part is uncomfortable for developers, but it’s essential. Stale memories, outdated facts, and superseded decisions actively harm retrieval quality.”

“The third biggest mistake is conflating vector similarity with causal knowledge. Semantic search finds related content well. It does not tell you that A caused B, or that Fix X resolves Problem Y, or that Pattern Z contradicts Pattern W. For complex technical domains, you need typed relationships, not just cosine distance. That’s why the knowledge graph layer matters.”

Krupp recommends that teams that are specifically building AI products prioritize memory isolation early. “Who can see what? If Agent A writes a memory, should Agent B be able to read it? How do you scope memories by customer, by project, by context? Getting the tagging and identity model right early saves painful migrations later.”

When memory is a feature, it belongs to a user, lives inside an application, and gets designed around a single interaction model. When memory is infrastructure, it’s a shared layer that multiple agents, tools, and workflows can read from and write to. The architectural implications are completely different.”

Getting Agentic Memory Enterprise Ready

As an enterprise architect himself, Krupp reviews the core requirements for enterprise software usage: “Data sovereignty, auditability, and isolation. Each one implies a specific technical answer.”

“Data sovereignty means the memory service has to run on your infrastructure, using your compute, with embeddings generated locally. No API calls to an external service for retrieval. Every sensitive architecture decision, customer interaction, or compliance note that goes into memory stays inside your perimeter. Local ONNX embeddings are non-negotiable here.”

“Auditability means knowing who stored what, when, and why. The service includes an audit log plugin. Combined with the x-agent-id header that auto-tags memories by agent identity, you have a clear lineage trail for every stored fact.”

“Isolation means multi-tenant memory where one team, customer, or agent cannot access another’s memories. This is solved via tagging and scoped retrieval, but it requires deliberate design at the application layer. OAuth 2.1 with Dynamic Client Registration (RFC 7591) handles authentication and client provisioning, the same auth standard enterprises expect.”

Krupp notes that agentic memory is still missing a few key pieces of the puzzle for enterprises. “What’s genuinely missing at this point is formal certification and compliance documentation. SOC 2, ISO 27001 alignment, GDPR data residency statements exist as architectural properties but not as third-party certified artifacts, or at least, not yet.”

“For any enterprise that needs checkbox compliance before adoption, that’s the current gap. Nobody has demanded it yet; the project’s users have either been comfortable with self-assessment or have been developers who can evaluate the architecture themselves. That will change as the project matures.”

Evolving an Open-Source Agentic Memory Project

Krupp concedes that there are many approaches to solving agentic’s amnesia problem, and that there isn’t necessarily a clear-cut leader at the moment. “I periodically compare the project against other memory layers as they become popular. When something interesting appears, I analyze it, usually with Claude, and ask: What can we learn from this? What should we adapt?”

“That process has generated a number of RFCs and design discussions in the project. One issue proposed transitive closure and abductive inference for the knowledge graph. Capable contributors pick those up and implement them. The project has become genuinely community-driven in that sense.”

“I think that what made this community possible was investing in infrastructure that has nothing to do with the code itself. Things like thorough documentation, a comprehensive wiki, deployment automations, periodic housekeeping, and clear contribution paths. Projects that don’t invest in those things don’t attract serious contributors.”

“If I could, I’d like to mention one last thing about the project. Being honest, I think it deserves more support from sponsors. The maintainer workload remains significant even with extensive agent automation, including steering the project, reviewing contributions, maintaining quality standards, and responding to the community. If the companies and developers who depend on this infrastructure valued it at the level they’d pay for a proprietary alternative, the project would be in a very different position. Open-source sustainability is a real problem in this space, and memory infrastructure is no exception.”

What’s Next for Agentic Memory

The builder is realistic on the place of memory within the future AI stack. “Basic vector storage will be commoditized. It’s already happening. The differentiation will move to the quality layer (how memories are organized, scored, and consolidated) and to domain-specific memory schemas that encode deep knowledge about particular fields.”

“A healthcare AI’s memory is not interchangeable with a software developer’s memory. The schema, the relationship types, and the forgetting policies are all different. But durable memory will become table stakes for serious AI products. The competitive advantage will be in the quality of what you remember and how intelligently you use it, rather than simply having the ability to remember anything at all.”

“Also, a less-obvious learning I discovered while building this project is that the most useful patterns for how to use memory in AI systems have come from the community, not from me. A user running a multi-agent cluster discovered that the tagging system could double as an inter-agent messaging bus, no custom protocol, just sentinel tags.”

“Another user put the service behind a Cloudflare tunnel with a self-built auth proxy and made it remotely accessible across all their devices. These weren’t designed features; they emerged from people solving real problems with a flexible primitive. That suggests something broader about how to think about AI memory. We may be able to accomplish more if we don’t design for a specific use case, but instead design for composability. A memory service that exposes a clean, flexible API will find use cases you never imagined. The ones you do imagine will probably be the least interesting ones.”

The post How to Think About Agentic Memory Job-to-Job appeared first on Heavybit.

On episode 53 of The Kubelist Podcast, Marc Campbell and Benjie De Groot sit down with Anurag Goel. Anurag shares his journey from early employee at Stripe to founder and CEO of Render, one of the fastest-growing application platforms in cloud infrastructure. They discuss Kubernetes, platform engineering, bare metal, AI agents, and what the future of application deployment looks like.

The post Ep. #53, Render and the New Cloud Stack with Anurag Goel appeared first on Heavybit.

For now, AI agents are autonomous entities to which users can delegate simple tasks: Monitor your calendar. Sort emails. But for now, when the time comes to combine a variety of tasks, or execute a complicated multi-step process, agents generally need orchestration and additional human supervision to get any semblance of reliable performance.

While some analysts predict significant adoption of agentic into enterprise apps this year, we haven’t quite arrived at the exciting future state where agents can handle complicated, time-consuming tasks on their own with little supervision. Yet. Open-source creator Moaz Muhammed explains his approach to agentic orchestration in the CodeMachine project.

From Spec-to-Code to Architecture

The creator explains the origins of his project as being a spec-to-code project, initially intended to help users build enterprise-grade projects without the friction of using complicated coding tools. “We started with one hard-coded workflow. However, we ended up having the project build about 90% of itself via dogfooding.”

“The initial idea was having a workflow to build architecture, then start converting the architecture into a plan. Like agents in a workflow, with the first agent writing architecture and related decisions like which stack to use: front end, back end, and so on. Another agent took the MD file and converted it into a plan. A third agent converted the plan to tasks based on the size and scope of the project.”

Moaz notes that the original architectural spec seemed to be a hit with early users and may have gone on to influence the structure of subsequent AI projects. But over time, feedback started to vary, with software engineers taking issue with the original projects’ waterfall-based methodology.

“They said, ‘We want something faster.’ Some projects would take hours to build. I even built one with about 80,000 lines of code, which proved that CodeMachine was at least capable of building huge projects. It wasn’t perfect, but a good starting point, because CodeMachine provided a real and organized scope, with every file in its place, everything well documented and explained without human supervision.”

The creator explains that as the project evolved, he shifted to more of an engineering focus with clear architecture. “This was a very big problem [for AI developer tools]. When you started a project, you couldn’t scale it, because you weren’t really building. You were ‘vibing.’” By shifting focus on building full architecture for projects, Moaz was able to avoid generating spaghetti code projects.

“From there, we pivoted from being hard-coded to a single workflow into being a workflow infrastructure for any kind of workflow. That was actually a huge transformation.”

Growing Adoption with Tool and Framework Choices

Moaz explains several deliberate choices he made over the course of building the project, such as focusing on command-line interface over the more-common browser-based chat window. “Not only was the terminal popular in well-known coding tools, but developers building software tend to use a CLI-based terminal interface. They’re familiar with it, and I’m familiar with it. Personally, I love the terminal so much as an interface.”

The creator suggests that some design decisions not only made the project more appealing to engineers, but also made it more marketable. Popular and trendy tools and frameworks can help attract new users despite strong opinions and preferences among technical audiences. “I used to use Node,js, then switched to using Bun, which was becoming an extremely popular alternative even before it was acquired by Anthropic.”

Of course, the choices weren’t all for marketing: The goal was to stay on top of a trending stack while delivering real value, especially since some engineers will always be hesitant to adopt new technologies due to potential bugs and lower reliability. The creator suggests that the strongest projects are those that balance engineering with marketing.

“It's not enough for a product to work. Engineers need to enjoy using it and feel connected to it. The most successful projects understand this human side, making deliberate choices that are not only technically sound but also appealing to the developer community.” So, the decision to move from Node.js to Bun was not purely technical. It also helped the project align with emerging trends and attract attention from engineers who were excited about the new ecosystem.

Between Human-in-the-Loop and Full Autonomy

The creator tactfully avoids weighing in on whether there exists an ideal balance between a fully autonomous agentic fleet or, on the opposite end, a tightly-restricted agentic program with one or more humans in the loop. “I can say that the philosophy of the CodeMachine project sits between [those two extremes]. It depends on the use case.”

“I designed this project to act as infrastructure for workflow agents for any coding task. For example, you could use it as a full pipeline to migrate from ancient languages like COBOL to a modern language. Some workflows might need permission or some sort of user interaction at times, but you can set the interactive portions of your project to control every autonomous step.”

“It's an orchestration platform that lets you run long-running workflows in any configuration you want between interactive and autonomous.” The founder also concedes that there are well-founded concerns about agentic security, but emphasizes that security is up to individual builders.

“Users will interact with this project like infrastructure. For those who prioritize security, local models are a solid option. CodeMachine is fully local. Any data it stores stays on your device and never leaves it. The only point where data exits is when it's sent to your provider, such as Claude Code or Codex. So while the project itself poses no security concerns, full end-to-end security comes down to keeping your setup as local as possible.”

What Agentic Means for Future Developers

The creator is optimistic about a future that enables builders, rather than focusing heavily on technical expertise. “I think the future is going to belong to people who have a ‘builder’ mindset, who can think in terms of architecture and how to build things brick by brick.”

The creator suggests that the future of software will belong to builders who embrace this different set of skills. “People who take the initiative to build things, whether using Claude Code or other tools, but also have the curiosity to ask why Claude and other tools make the choices they do. People who are curious about the why will build the next great era of products.”

The post Why Orchestration May Be the Future of Agentic Development appeared first on Heavybit.

On episode 91 of o11ycast, Ken Rimple and Jess Kerr sit down with Janaki Vivrekar. Janaki shares how Amplitude is building AI-powered analytics agents, why evaluation frameworks are becoming essential to AI product development, and how teams can use observability techniques to improve agent performance over time. The conversation explores eval-driven development, production feedback loops, and the challenges of helping AI systems reason about complex business data.

The post Ep. #91, Every Failure Becomes an Eval with Janaki Vivrekar appeared first on Heavybit.

How to Actually Sell AI Products in Competitive Markets

AI in today’s news headlines is all about big numbers: High performance scores of models on key benchmarks. Millions in funding raised for billion-dollar startup valuations. But what about sales figures? There’s a great deal of noise being made about how well AI startups are doing, but not much in the way of specifics on how they’re successfully going to market.

EverWorker CRO John Roberts admits that the current sales environment for AI products is fraught: On one hand, organizations do want to realize the mythical productivity gains AI should be offering. On the other, many have been sold a bill of goods; low-rent GPT wrappers that aren’t exactly performant, or inscrutable products that no one bothers to learn how to use.

Below, Roberts outlines his suggestions for succeeding in AI go-to-market:

• Focus on Outcomes: Tech stacks are far less important than identifying valuable results for customers and actually achieving them
• Understand Changing Buyer Psychology: Buyers are no longer afraid of missing out. They’re afraid of f***ing up
• Provide Actual “Technical Support”: Your team’s job isn’t just to convince customers to buy; it’s to get customers successful
• The Strongest Path to Success is Success Itself: The strongest selling signal today is wildly successful and happy customers who won’t stop selling your product for you

Outcomes, Not Tech Features, as a Focus

While Everworker’s promise is providing “AI workers” that fit its customers’ use cases, Roberts suggests his team’s approach focuses less on tools and entirely on outcomes, whether for scaling outbound outreach, building sales playbooks, or running content and SEO programs. For revenue teams, part of the initial approach is diagnosing and triaging breakage points.

“If you can understand what your revenue target is, we simply walk back from there. Okay, that's your target. What about inbound leads? If you have them, how quickly are you following up with them? Industry standards being that if you're not following up within less than five minutes, your conversion rates are going to fall off a cliff. And if you are following up fast enough, what’s your conversion rate to meetings once you actually reach out?”

The Founder’s AI Checklist: Outcomes, Bottlenecks, Starting Small

Roberts outlines a plan that founders should build before buying a new AI product. “There’s definitely a checklist I would put in front of founders [looking to adopt AI products]. What’s most important is understanding the outcomes that they're trying to achieve and the bottlenecks they're facing today. If you don’t know either, it’s very difficult to propose a solution.”

“Compare this to the classic SaaS sales model, where teams buy a tool, then find themselves stuck six months later trying to figure out how to implement it. At EverWorker, I think our competitive advantage today is we actually started as a service-first organization. When we think about an engagement, we're always bringing in forward deployed engineers and business analysts that will actually help build the business case.”

“Make sure that the bottleneck you're thinking about today is actually the bottleneck that should be solved first, and then pass it over to the technology side to build a solution to solve that.”

The CRO also cautions startup founders to avoid going too big, too quickly. “Start small, get a quick win, and use that quick win to build buy-in internally, so that people feel secure that this isn’t something to replace everyone’s jobs.”

“I think that's the biggest concern that we have today: That CEOs might have already bought into the idea of being ‘AI first,’ but you get a level or two down into the organization and someone else on the team might say, ‘Wait! This thing is going to replace my job, so I won’t do it.’ The only way that you get over that is by starting small, seeing the success, letting people inside of the organization own that success, and then scaling from there.”

“What’s most important is understanding the outcomes that [customers are] trying to achieve and the bottlenecks they're facing today. If you don’t know either, it’s very difficult to propose a solution.” -John Roberts, CRO / EverWorker

“Our approach involves walking customers through a very prescriptive process of setting up. The people that were doing the job [manually] today need to be the ones that are auditing the output of the [EverWorker agents] before they give the all-clear to let the agents run free and continue to scale.”

Roberts notes that his team’s mindset of adopting agents isn’t a “set it and forget it” approach, but rather, training customers to become zookeepers who manage the care and feeding of their agents while vetting outputs and measuring outcomes.

Why Selling Teams Are Becoming More Technical

“If you're focusing just on hiring, you need to have somebody that can understand the business process. That's the really important piece today.” The CRO muses that some of these ideal systems experts may have started at high-end consulting firms but were disillusioned by how corporate the environments were, and wanted to get their hands dirty and accrue deep domain expertise in technology.

“Those are the type of people that can really step in and say, ‘I can manage this agent. I understand the business context. I understand how this integrates and impacts other parts of the business, and I understand the outcome that I'm trying to drive towards. I can put all of that context into an AI worker and get the outcomes much faster without actually having to do the work.’”

Does that mean the future belongs to solutions engineers? “Look at how Microsoft had layoffs recently, and replaced those sales roles with SEs. People want answers. They want somebody that knows how to solve the problem. They don't care about any of the other stuff. Just solve my problem, provide the outcome and then move on.”

Why Selling AI Products Needs a New Playbook

The CRO suggests that today’s AI sellers are focusing too narrowly on the traditional Challenger Model, which focuses less on relationship-building and more on challenging buyers’ status-quo beliefs about software. “No one questions if the future is the place that they want to go. Everybody has bought into the idea that they need to change.”

“Where people are stuck today is part of a different story written by the same folks who wrote Challenger, which they covered in Jolt Effect. There's a lot of research on this, but basically, the fear among buyers is no longer the Challenger-style fear of missing out. It’s FOFU (fear of f***ing up). Everyone is stuck. They're frozen because they don't know what decision to make.”

“You need to figure out how to help customers overcome that fear. And there's a lot of things that you can do to help folks do that. But that's where deals are stalling today in the AI world. It’s why you start smaller, take risk off the table, give [customers] opt-out language, and lean in with services. You make sure that they understand that you're there to support them.”

“The fear among buyers is no longer the Challenger-style fear of missing out. It’s FOFU (fear of f***ing up). Everyone is stuck. They're frozen because they don't know what decision to make.”

“I always tell my team that before they actually pull the trigger and buy something, customers feel like they're going to jump out of a plane by themselves. I tell my team: ‘I want you to strap yourself to them and make sure they feel like you're jumping out of the plane with them. Those are some of the tactics that help win in this day and age.”

How Do AI Startups Scale Successfully?

The CRO concedes that the AI space can still feel a bit like the Wild West as so many things are new and so many norms have not been established. So how can AI startups build enough success to scale? “The only way to succeed is to have successful customers. You can't spend your way to success on LinkedIn. You can't email blast your way to success. You just cannot grow a business that way.”

“Because of the trust issues organizations have, and because of FOFU, the only way to really scale and grow a business nowadays is to have wildly successful customers. Customers who go and tell their friends, ‘This is what I'm using. It's working. I'm getting a ton of value out of it!’ And then hopefully they go tell 10 other friends. Successful customers is the only way to win.”

The post How to Think About Selling AI Products appeared first on Heavybit.

On episode 11 of High Leverage, Joe Ruscio speaks with David Crawshaw about the shift from traditional developer infrastructure to agent-native computing. David traces lessons from founding Tailscale to now building exe.dev around the idea that “agents need a computer.” The conversation explores why VMs may be the right primitive for AI coding agents, why deployment friction becomes unacceptable when software can be generated in minutes, and how agent-driven development changes code review, production workflows, security, and the economics of personal software.

The post Ep. #11, Why Agents Need Computers with David Crawshaw appeared first on Heavybit.

On episode 52 of The Kubelist Podcast, Marc Campbell and Benjie De Groot speak with Hugo Santos. They discuss the evolution of Namespace Labs, why traditional cloud infrastructure isn't always optimized for developer workflows, and what it takes to build a vertically integrated platform for builds, testing, and developer productivity. Hugo also shares lessons learned from nearly a decade at Google and years of startup building.

The post Ep. #52, Beyond the Hyperscalers with Hugo Santos of Namespace appeared first on Heavybit.

On episode 38 of Open Source Ready, Brian Douglas and John McBride speak with Graham Christensen, CEO of Determinate Systems, about the evolution of the Nix ecosystem and why more organizations are embracing reproducible infrastructure. They discuss secure package management, enterprise adoption challenges, open source business models, and how AI tooling is rapidly reshaping software engineering workflows.

The post Ep. #38, Reproducible Infrastructure with Graham Christensen appeared first on Heavybit.

How to Think About Using Social Media as a Founder

It’s harder to promote your startup on social media channels than it used to be. Sure, it’s still free of charge to post to social channels like LinkedIn and Twitter/X, but publishers dramatically reduced organic reach of social channels years ago to “encourage” users to pay for promotion. Today, success on social media isn’t about hoping you “go viral.” It’s about these strategies:

• Starting From a Clear Goal: Thought leadership is different from generating leads, and requires a different approach
• [Eventually] Scaling Your Social Presence: While founders often start by focusing on a single channel, they often go multi-channel eventually, and post regularly
• Effectively Engaging: Successful social interactions are user-focused rather than transactional
• Bridging into Real-World Interaction: Where applicable, social can help you to build toward face-to-face meetings
• Preparing for Scrutiny at Scale: Everything posted online lasts forever!

To understand how founders should think about social media both in the short and long term, we sat down with social media veterans Carter Hostelley and Karri Carlson of the agency Leadtail, which has helped startups grow their social followings since 2012.

1. Set Goals: Building Reputation vs. Building Pipeline

• Strategy Before Numbers: There should be an ultimate goal behind the follower count or post shares. What’s yours?
• Choosing an Approach: Building in public, building niche reputation, or engaging with top influencers can work if you have a plan.

The Leadtail team understands the urgency of startup life, but suggests founders take a beat to consider their ultimate goals for social media. “If a founder tells us: ‘I need to get 5,000 followers on Twitter,’ we’ll say: ‘OK, but why?’ Are you looking to build a following so you can raise funding? Or to become popular? Or to get people to try your product so you can accelerate your learnings? Or to hire the smartest people out there?”

“Is your goal really to hit some number this quarter, or is it to build a brand over time? You can literally buy followers if you just want to play that game. But we recommend having a strategy behind your presence on whichever platform.”

As an example, the team discusses “Tech Twitter/X,” an active ecosystem for highly technical founders in arenas like AI and cybersecurity (on a platform that used to be much more active itself.) “On Twitter/X, where we’ve seen a few different ways to be successful.”

a. The Journey Sharer (and a Potential Self-Intro to Investors)

A common approach to Tech Twitter is treating it like a public journal, where founders candidly share their wins and losses (and their commitment to success for potential investors). “You share the process of building your product, as well as your challenges raising money, or hiring the right people, or what happened when the product crashed.”

“This approach can be part of a strategy on how to get funding. How to be seen as somebody in the community who investors want to know about. Because you're demonstrating that you are a responsible steward with their money and that you're able to build things successfully and you're able to hire successful people.”

b. The Thought Leader Following/Deciphering the Trends

“Another approach is to continuously share your thoughts and opinions around where things are going, while ideally having opinions that are not just the same thing everyone else is saying. The idea here is to be opinionated, but not too abrasive.” Leadtail admits that this approach can be challenging for engineering-focused founders who aren’t usually given to doling out hot takes.

“But you can still share your views on trending topics. And for those who aren’t as comfortable commenting publicly, the strategy is really to engage around the technical questions that are happening on other people's posts as they share what they’re seeing, and to do it in a highly productive way.”

“This is a bit of a different approach that focuses more on finding all the conversations about this thing about which I am knowledgeable. Every time someone encounters that topic, you want to be in the thread. That's how people are going to come to know you. Every time someone looks at a conversation that's about this corner of this universe, there you are. That's how you’re going to get known, and that's how people are going to become interested in who you are.”

c. The “Reply Guy”

“A third approach we’ve seen founders take, particularly for super-technical folks, is identifying a bunch of influencers talking about the technology space you’re in, and looking to build a following by replying to those influencers. The way to succeed here is to find a specific group of high-profile folks who are knee-deep in this stuff and actively engaging in discussions.”

“You’ll want to make that list of influencers you reply to be a manageable size (we recommend no more than 20-25 or so for starters), and look to engage with them regularly. You’re essentially monitoring them and looking for opportunities where it makes sense to dive in, but to do so in a way that is genuinely authentic and helpful.”

2. Scale Your Posting (and Channels)

• Choosing the Right Channel: Different channels have different strengths. Busy early-stage founders may want to start small
• Scaling Across Channels: Eventually, successful founders tend to adopt multiple platforms, including Twitter for direct engagement, LinkedIn for video, and blogging platforms for long-form analysis
• Posting 2-3+ Weekly: A cadence of 2-3 posts weekly isn’t a bad place to start, and can be a combination of prepared programming and commentary on ad hoc trending stories

Which Social Channel Is Right for My Startup?

The Leadtail team confirms that it makes sense for new founders to focus on a few social channels, or just one channel, in the earliest days. “Yes, the vast majority of folks are really anchoring around LinkedIn, but for certain areas, like AI and cybersecurity, especially for early-stage founders trying to engage with VC, Twitter/X is very active.”

However, the social media veterans suggest that successful founders generally end up spreading their presence across multiple channels for strategic reasons. “Many years ago, we’d talk about supplementing your social presence with a content presence on Medium.com. But it’s about considering how the combination of these things work: You update people on product status in one place versus having deeper industry commentary somewhere else.”

“And the general track is that founders start to see some success, raise their Series A or Series B, and at that point, it’s recommended they have a strong presence on LinkedIn, which can be a good home for video content. While their thought leadership sits on Substack. And their conversational updates, the direct engagement and meeting of community people, takes place on Twitter/X.”

How Often Should Startup Founders Post to Social Media?

“You may be pursuing a few of these scenarios regularly (except the reply guy scenario, as that will depend on your diving into conversations as they happen). If you’re sharing your journey or pursuing thought leadership, you might want to consider at least 2-3 posts per week.”

“And if you can set up a cadence that people start to get used to, for example, ‘Every Monday I share my learnings as a new startup founder,’ and even group those together using a thread, you can start to build an audience over time.”

“If you find yourself delving into more of the thought leadership side, then you may want to time your social posts around your Substack newsletter posts, if and when you plan to run that. You can be episodic and try to schedule ahead if you're talking about big topics, though some of your posts may be driven by breaking news, which may kick up your posts to 3-4 per week.”

3. Engage by Centering Around Other People and Provide Value

Social media for startup founders has rules of engagement. “Number one, you have to make it about the person you are engaging with.” The Leadtail team notes that it can be tempting to try to jump ahead and book sales demos with someone interested enough to post a reply, but that can be premature.

Regardless of whether you’re trying to be the candid, tell-all founder or become a thought leader, the replies you post should be valuable, helpful, and engage people where they are. If you have deep expertise, you can share it in a way that isn’t overbearing. If you have real-world experience with someone else’s question, your insights can speed them along their journey.

“Generally speaking, we recommend that you do not immediately jump in and offer to book a demo. It’s better to make the conversation about the other person, and to focus on being as helpful as possible.”

4. Look for Organic Meeting Opportunities as Appropriate

• Avoid Rushing into Sales Conversations: A response to your post usually isn’t the same thing as confirmed sales interest
• Low-Stakes Meeting Opportunities: Some conversations may pave a path to a low-stakes meeting, such as at a local meetup

No one wants to follow the pushy salesman who didn’t take the hint. Being overly transactional or promotional doesn’t make sense and doesn’t build followings. What might make sense is, one you have a conversation going, looking for an organic opportunity to progress the conversation off of social, potentially in a low-stakes manner.

“After you engage successfully with other people, you might start figuring out how to set up a time to connect at the next local meetup that's happening...the one you were both planning to attend anyway. Alternatively, maybe you jump onto a texting thread.” Regardless, the Leadtail team recommends having an engagement strategy for different types of people on different topics, at different levels of interest, to move from engagement to conversation.

“Meetups are ‘back’ in the startup community. So it’s not uncommon to respond to a conversation with a lightweight follow-up: ‘Hey, this has been a great exchange, but are you going to be at such-and-such event? It’d be great to meet up in person.’” (The social media experts also note that, at least for the time being, this type of chatter tends to work better on Twitter, while direct messages on LinkedIn are generally seen as sales pitches.)

5. Prepare for Scrutiny as You Grow

• Technical Audiences May Be as Passionate as You: Everyone has deeply-held beliefs, so interactions should be respectful, even among differences in opinion
• Your Online History is Forever: Avoid online arguments and negativity that could hurt your brand, since they won’t go away

The Leadtail team notes that there have absolutely been founders who have succeeded on social media by bringing in their own unique personality, being playful or snarky, so long as that was genuinely who they are. However, no one succeeds for long by being consistently negative and attacking people who disagree with you.

“As you grow your following, you still want to be productive and helpful. And for technical audiences, you’d expect people like engineers to be inherently helpful and focus on solving problems. But they may also express a much higher level of disagreement, depending on their experience or conviction. So you’ll want to be aware that not everybody in your community will focus on wanting to be helpful above all else.”

“This is the curve: You start posting on social media, and at first, all you get is crickets. Then one day, you post something, and somebody actually engages with you. You can’t believe it! You want to go tell all your friends somebody engaged. And then you start to get your first negative replies.”

Reminder: Everything You Post Online Is Forever

“This is even more important today than it was 10 years ago: Everything you're doing on social media is public, and will be public forever. We often coach executives on this regarding sensitive topics like politics, but it can also be around technology opinions. As a tech startup founder, you want to be sensitive to the fact that you’re addressing an audience of brilliant engineers who may have very deep convictions about what they do before you slam that new engineering framework in public.”

At the very earliest stages, a founder’s social presence may be virtually indistinguishable from their startup’s. “This is the thing founders have to be aware of. You may hold whatever strong views, political or otherwise, on your personal account, but that reflects on the company you're building and its culture.” It’s often a good idea to work with your co-founders and investors to build an online presence that reflects the kind of culture and brand you want to build.

“Playfulness is one thing, but having a super-snarky troll-like founder is another. Most founders don’t ultimately want to become the next great internet troll. They want to grow their business and raise funding. So over time, marketing teams eventually start doing routine cleanup across their socials, and they’re trying to be aware of the brand they’re building.”

“You can get into trouble when working with executives who get pushed into a place of visibility they didn’t necessarily seek for themselves. They can sometimes be very sensitive to any criticism thrown their way. So it’s a good idea to have training and a plan in place to make sure people on your team aren’t feeding the trolls.”

“Obivously, there’s a balance to strike. It’s possible to not be opinionated enough to break through the noise and just repeat the same thing everyone else is saying. So yes, founders can and should have real opinions, but that doesn’t mean they should be abrasive and troll-like. It’s OK to disagree.”

“We always say: The focus is the future because anyone can have opinions about the future. Who knows what will actually happen? And by the time the future is here, you’re going to have new opinions to share anyway.”

The post 5 Social Media Strategies for Early-Stage Startup Founders appeared first on Heavybit.

On episode 12 of Data Renegades, CL Kao sits down with Maxime Beauchemin to explore what happens when software teams start working alongside armies of AI agents. Max shares the thinking behind Agor, his collaborative platform for orchestrating coding agents, and reflects on how AI is reshaping engineering, product, design, and data work. They also dive into the future of agentic workflows, technical collaboration, and what human roles may look like in the years ahead.

The post Ep. #12, Agent Orchestration at Scale with Maxime Beauchemin appeared first on Heavybit.

How Do You Get Change-Resistant Enterprises to Adopt AI?

For some time, enterprises have resisted change, with internal departments using a specific software platform as their center of gravity: The ERP for finance, the CRM for sales, and so on. Each with painstakingly built custom configurations (with new team members not allowed to touch anything!). So what happens in the age of AI when everything speeds up and the pressure is on to modernize, even at risk-averse enterprise orgs?

Founder Prateek Jannu’s project Coasty focuses on what he considers to be AI’s most immediately valuable use case for enterprise: Computer use, which defers actual point-and-click functionality on desktops and browsers to autonomous AI. Says the founder, computer use lets enterprises gain immediate value from the speed and versatility of AI, while not disrupting existing enterprise tech stacks (especially the customized Netsuite instance that took the finance team years to perfect).

Addressing Enterprise Inertia and Resistance to Change

Jannu notes that the Coasty project came from observing how, while many academics have been excited about the potential of AI, enterprise orgs seemed to recoil in horror. “A lot of companies, even among the Fortune 500, did not want to build MCP servers, or anything they thought could destroy their systems or accidentally leak sensitive information.”

From there, the Coasty project launched as a computer use agent with high performance scores 82% on the agentic OS World benchmark when tasked with looking at users’ screens and figuring out how to interact as a keyboard and mouse would. “A score is just a score and doesn’t necessarily imply anything with regard to long-term tasks,” the founder admits, but it may be an indicator of potential value for a variety of use cases for form fills, accounting, hotel management, and others.

The founder notes that building AI computer use has taken a different path than previous-generation workflow automation and screen-scraping tools due to changing interfaces and other external factors. “We use five different systems with a variety of models, which is important for enterprise systems because UIs also change.”

“If an accounting team is using a legacy system that gets some kind of software update, or alternatively, if they’re filing state taxes but next year, there’s a new option, some new provision for an updated state tax rate which a simplistic workflow automation tool might not pick up. We’ve built our models to look for and pick up such changes and understand variances between steps that previous-gen solutions might get stuck on.”

AI Features Enterprises Care About: Speed vs. Accuracy

“When we started out, we noticed other teams in the computer use space seemed to be focusing on speed, perhaps because teams have looked at speed as a factor when they think about process automation. ‘If it’s faster, it must be a better option,’ was the thinking, and why would anyone pay for something that’s slightly slower?”

“This was a hard decision for us: Can we compromise on speed right now but guarantee that we can be more accurate than any system out there? That's a major trade off.” Jannu notes that while having low latency definitely demos well, but a computer use tool that’s fast but inaccurate is virtually pointless.

As his team reviewed their options, they realized that in the AI age, as code becomes faster and cheaper to produce, software will eventually change daily, which will mean that UIs and APIs will too. “If you focus on speed, you’re going to break at some point.” Meanwhile, enterprises run on decades-old software that can have fragile configurations or integrations, causing them to be extremely hesitant to change vendors with any frequency.

“Once your AI is logged in to their software, [enterprise teams] don’t want to make significant changes. If you did go in and make changes to their tools, they wouldn’t renew. They wouldn’t even adopt. So we decided to focus on accuracy instead of speed for enterprise personas.”

Enterprise Challenges: Speed and Decision Tree Errors

Moreso than building for any specific interface (desktop, browser, or terminal, all of which Coasty supports), Jannu suggests that significant challenges include decision tree errors and time thresholds. “Anytime you give a task, you expect it to take a certain amount of time. Small tasks are relatively fast, but as users build confidence, they start testing boundaries and make bigger asks.”

“They start requesting more long-horizon tasks trying to test how well Coasty can do, tasks that may take closer to 10-20 minutes. Sometimes, tasks can go off trajectory. If I request the creation of a certain kind of database, and it chooses the wrong table in step two, it's going to be completely wrong for the entire trajectory. So it’s really about how to get those first five to 10 steps right. And even if the system gets something wrong, how can we get back on track?”

“One thing we focus on with this project is self-correction: If there’s a mistake, Coasty can go back and rectify that. One thing that was really hard to pin down was controlling and auditing which step the system gets wrong.” The founder explains how the process involved building dozens of testing layers to detect errors and revert to previous steps, a process that remains complicated.”

Hard Lessons: Enterprise Users Expect Their Own Flow

“We also learned that enterprise users want an easy way to provide logins for their agents, but the challenge is doing that safely. We added what we call agent credentials which we store in an encrypted format, but the agent never sees the credentials, and will prompt users to enter them as needed.” The founder suggests he expected this behavior: Of users to instruct agents to take action until they ran into a roadblock and needed something, like credentials.

“We realized that people were prompting their agents more like they would a human, because that's how these users were expecting it to work. For anyone trying to implement computer-use agents as well, I'd highly recommend trying to see what users expect from the system: Sometimes they may expect a full, end-to-end prompt, but sometimes they’ll say: ‘Hey, just come back to me and ask if you need credentials.’”

“This is something that we did not expect. And a lot of users got turned off because of that. We had to learn the hard way. And we had to talk to a lot of users to find out about these issues, whether their issues were related to costs or functionality, before we realized it was something as simple as users expecting agents to come back and ask them for credentials.”

What’s Missing in Computer Use: Security and Safety

The founder admits that the ultra-viral OpenClaw project kickstarted adoption of agents for personal use, but suggests that agentic has seen a noticeable delay on security to accompany all the exciting advancements. “If you look at how software had a boom in the early 2000s: A lot of compliance companies came up with their own standards.”

“But at the end of the day, the larger companies, including the Fortune 100, came together with the goal of figuring out how to identify whether software is ‘safe’ enough for use by big companies or clients. And they came up with SOC 2 and similar methodologies.”

“I definitely think there's going to be at least some kind of standard in the future, a security framework specifically for agents that could also cover computer use or general agentic frameworks. It might involve stress testing agents with jailbreaking problems, putting them through some kind of verification system to ensure it isn’t vulnerable to prompt injections or other kinds of attack.”

The founder suggests that an industry standard might emerge once enterprise adoption of agents hits critical mass, which doesn’t seem to have happened yet. “A lot of these companies are adopting [agents] to a very small degree. Maybe it's one small team using it for a certain reason. But if you want the whole organization to use a certain kind of AI, the company needs to make sure that if something does go wrong, someone is responsible.”

“There's going to be a sort of framework or a standard set for it, ideally an open standard so people can go audit it. For our project, we maintain a really slim internal security standard so if something does happen on our end, we’re ready for it.” The founder suggests that every agentic software project should stand up their own security standard along with a battery of internal tests.

“If you’re building agentic, try to test it. Break it. There are a lot of good open-source repositories with guides on how to do prompt injections or jailbreak certain system prompts.Try them on your own system. Check how it works. Is it failing? How? If you're trying to take your software to a production-grade level, try to break it as much as you can beforehand.”

Computer Use as an Enterprise Modernization Opportunity

“I think there are a lot of enterprises that depend on legacy systems that do not have APIs or MCPs built. Maybe 95% of them do not. I believe [computer use] is the best way to get started with AI [for enterprise customers] because a lot of companies are under pressure right now to ‘implement’ AI.”

The founder notes that while some early adopters are diving into building MCP servers, most companies are still pondering implementation and potential security issues. “[Enterprises] have a lot of these questions. What's the best way to go about [implementing AI]? Try to build computer use. At the end of the day, your software does have a login screen of some kind. People use it with a keyboard and mouse. And if humans can use these things, computer use AIs do the same.”

The founder suggests that as models only continue to improve, computer use will as well. “If today, we can perform a one-hour task reliably, maybe within a year, we can perform a three- to four-hour task reliably. Lots of companies have processes that take hours. With successful computer use, you can see efficiency improvements of 2x or 3x, even as you run agents as background tasks.”

“A lot of business units rely on tools that don’t have MCPs or AI-capable APIs. For example, I know of several accounting tools that don’t have MCP support, and won’t for some time? Maybe upwards of five to ten years? But whatever you do with a computer, with a keyboard and mouse, can be done right now with computer use.”

The post Computer Use: AI’s Most Enterprise-Ready Use Case? appeared first on Heavybit.

On episode 5 of Third Loop, Heidi Waterhouse, James Governor, Adam Zimman, and Kim Harrison dig into the hype around AI-generated code and ask a deceptively simple question: if code is “free,” what does it actually cost? From token pricing and maintenance overhead to user value and developer judgment, they explore why software is never quite as cheap as it seems.

The post Ep. #5, Free, Like Puppies: The Real Cost of AI Code appeared first on Heavybit.

On episode 55 of Generationship, Rachel Chalmers sits down with Sunil Dhaliwal to explore how AI is reshaping developer infrastructure, venture investing, and computational biology. Sunil reflects on building Amplify Partners during the rise of cloud computing, why the best technical founders “live the problem,” and where he sees the next generation of AI tooling emerging. The conversation also dives into personalized medicine, AI infrastructure consolidation, and what remains fundamentally human about investing.

The post Ep. #55, Faster Hypothesis Disproving with Sunil Dhaliwal appeared first on Heavybit.

On episode 51 of The Kubelist Podcast, Marc Campbell and Benjie De Groot sit down with Kyle Galbraith. Kyle shares the story behind Depot and explains how the company evolved from accelerating Docker builds into building an entirely new CI platform designed for the AI era. The conversation explores BuildKit internals, remote caching, microVMs, AWS infrastructure, and why modern software development may require rethinking CI from the ground up.

The post Ep. #51, CI Is the New Bottleneck with Kyle Galbraith appeared first on Heavybit.

On episode 10 of High Leverage, Joe Ruscio sits down with Steve Krouse to discuss the rapidly evolving relationship between AI and programming. Steve shares lessons from building Val Town at the center of the AI tooling wave, why he believes better abstractions will define the future of software, and how engineers can avoid becoming passive operators in an increasingly agent-driven world.

The post Ep. #10, The Human Brain in Software Development with Steve Krouse appeared first on Heavybit.

On episode 37 of Open Source Ready, Brian and John speak with Stormy Peters about the evolving relationship between AI and open source software. Together, they unpack the growing challenges maintainers face, why traditional “good first issues” may be disappearing, and how AI tools are changing the way developers contribute to projects. They also discuss open-weight models, inference costs, and why community health still matters more than lines of code.

The post Ep. #37, Is AI Killing Open Source Software? with Stormy Peters appeared first on Heavybit.

How We Went from LLMs to Agents to Skills

In late 2025, Anthropic declared skills, modular, reusable, file-based instructions that encapsulate domain expertise, to be an open standard. Since then, hundreds of thousands of users have reportedly installed a variety of skill.md files to improve their agents’ abilities on a variety of tasks, including coding with specialized languages or performing non-coding tasks like spinning up documentation or creating datasets.

While performing his usual daily duties with Claude Code, Turkish developer Yusuf Karaaslan began experimenting with skills, and eventually ending up designing the Skill Seekers project, which processes and translates a variety of data sources (including GitHub repos, PDFs, videos, and others) into agentic skills. Below, he explains what led him to create the project and how thousands of users worldwide are finding value from agentic skills.

From Context Limits to Skill Libraries

Karaaslan, a game developer by trade, notes that his initial foray into working with skills came from trying to work with the back-end of the popular online game storefront Steam. He started with Claude's official skill-creator skill, feeding it every relevant link from Steam's documentation one by one and asking it to produce a complete skill for the inventory system — the goal being a direct answer instead of a guess. It didn't work.

"No matter what I did, Claude would skip most of the input or cut it down far too aggressively, and it kept guessing at what I was asking for. My read was that because I'd asked it to handle both the heavy data gathering and the processing in one shot, it was optimizing for cost and cutting corners on the part that mattered most. Even the cut-down output was a little more confident than nothing, which was the clue. So I sat down and started building a separate system to do the heavy data lifting, the part that doesn't actually need AI in the loop. That was the first prototype of Skill Seekers."

The contrast once he piped that prototype's output back into Claude was hard to miss. "Before, the answers were always hedged. 'Maybe try this, maybe try that.' After I added the Skill Seekers output to the context, the tone changed completely: 'You can't do it the way you want with Steam inventory, you need an external backend.' Much more confident, much more direct, and actually correct. Even when I challenged it on details, it stayed true to Steam's actual behavior. The funny part is that all this happened the same day Anthropic announced skills. The first version of Skill Seekers shipped that same night.”

From there, Karaaslan kept iterating on the prototype, prompting Claude Code to extend it with web scraping and working around internal context limits until each new data source slotted in cleanly. "Until a certain point, my focus was on feeding in as many data sources as possible to build a richer, more complex skill: The need to download a dynamic website, or the need to scrape a codebase. Once I felt we'd covered almost all the major data sources and some of the more niche ones, I started backtracking to improve the system architecture. At that point I hit Claude's internal limits again: This time around analyzing the whole codebase. So I fell back on my usual trick: I had Claude generate full class diagrams, package diagrams, and flow diagrams for the key systems. With those UMLs in hand, I could pinpoint exact pain points, architectural problems, and more. From there, I designed the new architecture end-to-end and replaced the live version module by module.”

The developer’s iterative approach led him to map all connections and flows within the project, and eventually unify everything with a single interface, ensuring key functions were connected and removing duplicate branches. He explains that he relies on three primary skills himself for design, implementation, and PR reviews. "I actually developed these skills side by side with Skill Seekers itself, based on my own observations along the way. And I just open-sourced my AI workflows too.”

“Using my design skill, I start by using a card that simply explains test-driven development, and from that, I generate UML diagrams of how that should work. I then review, accept changes, and create issues and tasks, then create a development report of any issues. After everything is complete, I will send the output to the PR review skill. In this way, I have full control over the project.”

Accelerating AI Learning by Modeling Human Learning

Karaaslan suggests that his approach building skills started in much the same way he, himself, learns new things. “Before using AI, [to learn new development topics] I would read documentation, take notes, and learn from example projects. At first, I mimicked this process.”

The developer took a similar approach to eventually making Skill Seekers a project capable of ingesting data from a variety of different formats. “To implement how I learn things by watching videos into Skill Seekers, I would get code from the screen, collect timelines, and copy transcripts and screenshots into Skill Seekers. I was just mimicking how I learn without AI.”

The developer notes that after the release of the project, he observed how others would use it, noting that users would want to review new additions. “People would want to adjust some stuff, add some stuff, remove some stuff. I wanted to automate the process, so I created a skill for that, and afterwards, we created the workflows, and gave the skill to that workflow. Exactly what we would do manually. It’s about mimicking, optimizing, and automating.”

What Skills Add for Developers

Karaaslan suggests a variety of development use cases have emerged from the project. “It started as a helper, but it has become a data management system for any kind of input that can be structured and reproduced, and with minimal cost because once you create a new skill config file, you can run that config again and again with a single command-line prompt, or just by telling Claude to do it directly via MCP.”

The developer lists a variety of use cases, including an integration partner that uses the project to power its marketing and helper bots with skills that ping codebases to answer user questions about them. An academic AI research startup uses the project to scan new research papers for usable skills that the team can then use internally.

“Like I said, it's all just data that AI can understand. You can update to new versions easily and connect to your codebase in one click. So you can create your own tools, your own skills from your codebase. You can give it official documentation, your own implementation, examples, maybe some video, and it all gets merged into one big skill that includes everything.”

“After you create skills, you can add a workflow to create an agent and system calls. I use that feature often because you can control the context. It's actually a helpful tool for context engineering.”

“In my workflows I have an automated skill that's triggered every time I push to the main repo. Every time I want to do design work I have a ‘design repos’ skill that I create with Skill Seekers focusing on the architecture and API. Whenever I run my agent skill, it automatically uploads that skill, and every time I run it, I know I have the latest version of my architecture as knowledge within the context inside the agent.”

How to Think About Skills for Beginners

“Let's say you’re not very fond of AI, and you just want to do your work as a developer. Developers always think in terms of architecture, how we do stuff, how we structure stuff. If you want to use some new back-end framework or a new language you’re not experienced with, you can stay focused on system architecture while Skill Seekers builds a skill that turns your agent into an expert on your tech stack. The best part is that Skill Seekers has an MCP, so you can just casually tell Claude to create skills for your project. No extra steps.”

“Let’s say we wanted to start a new project and it's in an area we’re not familiar with. We might want to do some light experimentation before building because the cost of learning seems too high. Previously, we might avoid focusing too much on learning entirely new frameworks and languages. We would tend to just stay with what we know now.”

“You can tell Skill Seekers, ‘I want to do this project, I want to use this technology, I want to try this new framework, I want to try this new library,’ in casual, natural language. The MCP will trigger and generate the skill for the latest, top-of-the-line skill for you. You can just say, ‘I have an idea’ or ‘I want to try this functionality of this language or this library,’ and you start writing anything that you can test.”

"It's not too different from being an engineering lead. You say, 'I have a vision. I want to do this thing,' and it creates a team for you that knows that field and executes on your vision. You have a problem, you have a solution, you have a tool, you have a limitation, you have an output. This is always the same."

Real-World Use Cases for Skills

"The biggest use case I see is builders using skills to power support for their GitHub repos across the different language versions they need to maintain. Each version has its own skill, and every time they push an update, an automatic trigger refreshes the bot's knowledge. Doing that by hand is incredibly time-consuming."

Karaaslan's most-used example comes from inside his own company, where the team builds with Unity. He started by setting up an internal skill marketplace so developers could share work with each other, then began authoring skill config files for the technologies the team uses day-to-day. It didn't take long to spot the bottleneck. "Even with Skill Seekers doing the hard part, the surrounding loop was repetitive: Create a config, generate a skill, package it, and manually upload it to the marketplace. The friction was in everything around the generation step."

The developer explains his process of layering in fixes, one at a time. "First, I added a shared config repo so anyone on the team could publish or pull a config through their own Skill Seekers install. Then, I added packaging and uploading to the marketplace as native functionality inside Skill Seekers itself. The flow worked end-to-end, but I hit a new wall. The volume of data I was sending Claude during the local enhancement step burned through my usage almost immediately."

The fix was to stop tying Skill Seekers to any single model. "I made it agent-, CLI-, and LLM-agnostic. Now I trigger the whole pipeline from the command line: Scrape, restructure, combine sources, run the local AI enhancement step on whichever model I want, package, and push straight to our internal marketplace. I tend to use OpenCode with Kimi for the enhancement step. I tend to get more usage with it, and in my own tests, the skill quality came out on par with Claude."

That decoupling is also where Karaaslan sees the project's longer-term direction. "My vision for Skill Seekers is for it to become a universal data layer for AI systems. Model-agnostic, source-agnostic, pipeline-friendly. Whatever the input, whatever the agent on the other end, the skill is the same shape. That's the point."

The post Translating Data into AI Skills appeared first on Heavybit.

On episode 90 of o11ycast, Ken Rimple and Jessica “Jess” Kerr speak with Cory Ondrejka. Together, they unpack the rise of agentic AI, the shifting identity of software engineers, and the growing importance of measuring real-world impact. Cory shares his concept of Outcome Engineering and how teams can adapt to a world where building is fast but validation is everything.

Come find the Honeycomb team at O11yCon on May 21st!
https://events.honeycomb.io/o11yConSF2026
Use discount code SFCast2026 for 20% off

The post Ep. #90, Outcome Engineering in the AI Era with Cory Ondrejka appeared first on Heavybit.

On episode 4 of Third Loop, Elisabeth Hendrickson and Joel Tosi join the hosts to discuss systems thinking, software delivery, and why organizations often solve the wrong problems. They explore their upcoming book Signals and Levers, unpacking the CREATE framework and the illusions of progress, predictability, and control. The conversation also dives into AI, user trust, feedback loops, and what it really means to improve delivery.

The post Ep. #4, Signals and Levers with Elisabeth Hendrickson and Joel Tosi appeared first on Heavybit.

On episode 9 of High Leverage, Joe Ruscio sits down with Simon Willison to unpack the rapid evolution of AI coding tools and what they mean for software development. They explore the shift from vibe coding to agentic engineering, how coding agents are reshaping workflows, and why experience still matters. The conversation dives into trust, security, and what breaks when code becomes cheap.

The post Ep. #9, The AI Coding Paradigm Shift with Simon Willison appeared first on Heavybit.

HR Mistakes That Founders Still Make

Despite the accelerating influence of AI, there are still four key errors that new startup founders make today on hiring and building HR culture:

• Not Hiring for Fit and Culture: Leading to costly turnover and rehiring cycles
• Hiring Fast and Firing Slow: Which means poor culture matches and unnecessary team friction
• Not Building in Proper Legal Safeguards: Which turns early attrition into ownership nightmares
• Delegating Away Culture Building: Which prevents founders from ever building a cohesive team

HR veteran Darlene Harper advises a variety of early-stage startups. She explains that she continues to see founders make these mistakes and offers alternative approaches to prevent attrition and other painful HR problems:

1. Hire Early Teams for Fit (not Friends & Family)

Harper recommends hiring early-stage teams based on:

• Qualifications and Knowledge Level: Do the candidates have relevant experience and education?
• Startup Stage: Have they worked at early-stage startups? What are the largest and smallest orgs at which they’ve worked? And are they the right people today, or for a later stage?
• Intangibles: How well does the candidate work with the startup’s expected culture: Are they adaptable to changing schedules, do they learn quickly, can they manage change?

Harper relates anecdotes she’s heard about founders increasingly struggling with early employee retention. “This seems to be a real issue for founders now. It may seem surprising, but it’s understandable: All of a sudden, founders find themselves with funding. They have to hire somebody, but may end up with someone who isn’t necessarily the best fit for what they need.”

“I recently heard about a startup that got millions in funding and went broke within the next year, in part because they had hired all these PhDs from prestigious universities. People with great pedigrees, and of course, they demanded big salaries, but they weren’t necessarily the people the startup needed right at the beginning. They should’ve been later-stage hires.”

“You certainly want somebody that knows to ask the technical questions, and you want to be able to check their background and references to confirm this person really has done what they say they've done, and that they have the knowledge and qualifications.”

From there, HR experts can ask more-probing questions. Harper notes that asking about the size of the absolute largest and smallest companies at which a candidate has worked can be especially telling, as well as how candidates handle ambiguous situations without necessarily having all the information needed.

“And you do need to be very careful about hiring people just because you know them. Also, if you do have a longstanding relationship, you don't necessarily want to risk losing that relationship because you're bringing them into a business situation.”

2. Hire Slow and Fire Fast

Says Harper, “The cardinal rule is to hire slow and fire fast.” It’s crucial for founders to manage challenging team members out of an org as quickly as possible, whether they’re not pulling their weight or otherwise aren’t a good fit.

“In a small startup especially, everybody else knows it’s not working out. Meanwhile, having a team member that doesn’t fit in your org is poisoning the environment for the rest of your staff.”

The HR veteran cautions that being able to hire slowly and let go of people quickly is also a function of having the correct paperwork and corporate protections in place.

3. Confidentiality/Disclosure and Vesting in Hiring Documents

The HR veteran is adamant that startup founders should protect their startups with legal documentation as part of the hiring process that includes:

• Confidentiality Agreement: An NDA (or similar) to protect IP and customer data
• Disclosure Clause: Part of the confidentiality agreement that protects against unauthorized disclosure of confidential internal information
• Vesting Schedule: Setting a maturity date on options builds a “cliff” that keeps co-founders from immediately walking away with a huge portion of the company

Harper cautions founders to ensure they have the necessary paperwork. “Most important is making sure [new hires] have signed that confidentiality agreement with that disclosure piece. And if you're giving them equity, and somebody doesn't work out, make sure that you've got that [person] on a reasonable vesting schedule.”

“If you hire a co-founder who doesn't work out and they leave after six months, you don't want them taking 20% of the company with them.” The HR expert adds that a 12-month vesting cliff is a good minimum from which to start.

Harper notes that while many startup experts still talk up the “move fast, and break things” mindset, there are certain core pillars of hiring for which you need foundational infrastructure. “If [early hires] have never signed that nondisclosure or confidentiality agreement, then they've got your IP and they don't have anything stopping them as they walk out the door.”

4. What to Own vs. What to Delegate to AI

• Tactical, Repetitive Toil: Very much fair game to delegate to AI
• Building Your Startup’s Culture and Communication Style: Very much needs to be founder-led

Harper suggests that founders should be open to utilizing AI to automate away busywork, citing an example of a current client considering AI to automate social media posts to improve their company’s visibility. “AI can absolutely do a lot of the things (and give you a lot of the information) that you might have needed to get from HR in the past.”

Reports suggest growing adoption of AI in HR organizations for use cases such as drafting job descriptions, optimizing job ads, and sending reminders to fill out various hiring documents. The HR expert concedes that when startups grow to a certain size, they often concern themselves with additional compliance and regulatory tasks that can also be delegated away.

However, Harper notes the foundations of a startup’s culture come from the way its founders communicate with team members, and should never be entrusted to AI. “What I wouldn’t delegate: Founders’ interaction and communication with their employees. Letting teams know what they need from them, holding them accountable, those types of interactions. Anything that comes from AI is not going to be the same as something that needs to come from you, the founder.”

“Teams need you as the leader to say: ‘This is what I want from you. This is how I'm going to hold you accountable. This is what's important. These are the priorities.’ Nobody should be relying on AI to do that because that's not something that AI can come up with. That needs to come from the leader.”

The post Four HR Mistakes Founders Still Make in the Age of AI appeared first on Heavybit.

On episode 11 of Data Renegades, CL Kao sits down with Michael Stonebraker, legendary database pioneer and creator of Ingres and Postgres. They explore the origins of modern relational databases, why specialized systems outperform one-size-fits-all platforms, and what today’s AI hype gets wrong about enterprise data. Stonebraker also shares hard-earned lessons on startups, research, and staying relevant over five decades in tech.

The post Ep. #11, Contrarian Bets and AI Skepticism with Michael Stonebraker appeared first on Heavybit.

On episode 8 of High Leverage, Joe Ruscio sits down with Scott Hanselman for a conversation that goes far beyond prompts and productivity gains. They talk about craftsmanship, learning by doing, the long-term talent pipeline for engineering teams, and how AI could either free people to do more meaningful work or simply accelerate existing inefficiencies. From pair programming to Star Trek economics, this episode examines the bigger human questions behind the current AI boom.

The post Ep. #8, The AI Preceptorship Model with Scott Hanselman appeared first on Heavybit.

The Tell

On March 19, 2026, Cursor launched its Composer 2 model, calling it "coding intelligence at frontier level." Within a day, a developer inspecting API traffic spotted a model identifier: kimi-k2p5-rl-0317-s515-fast.

Cursor is a ~$50B AI company targeting ~$6B of revenue in 2026, and Composer 2 is the flagship of its product roadmap. Anthropic and OpenAI both have competing coding agent products in Claude Code and Codex, so it is paramount for Cursor to own its own models. And yet it did not train a whole new frontier model from scratch. Cursor started with Kimi K2.5, an open-weight model from Moonshot AI, post-trained with RL for coding.

Cursor's VP of developer education later confirmed it, somewhat awkwardly adding that "roughly a quarter of the training compute came from that base, the rest is our post-training."

It Is Cool To Be Closed

In terms of momentum, as measured in the attention economy, the gap between closed and open-weight models could not be more stark.

1. In March 2026, Alibaba's Qwen team fell apart. Lin Junyang, the technical lead and the public face of Qwen globally, resigned abruptly. Yu Bowen, the head of post-training, left the same day. Hui Binyuan, the Qwen Code lead, had already gone to Meta in January. Three pillars of the team in ten weeks. The reporting suggests Alibaba is shifting toward consumer DAU metrics and horizontal, product-driven structure.
2. Meta's Llama 4 was a dud. The company reportedly planned a pivot to closed proprietary models (codenamed Avocado and Mango) under new Chief AI Officer Alexandr Wang. It has since partially hedged, saying it'll release open versions of the next generation, but the frontier-open-by-default Meta of 2023–2024 is long gone.
3. Ai2 lost Ali Farhadi, Hanna Hajishirzi, and Ranjay Krishna to Microsoft's Superintelligence team in March 2026. The Paul Allen-linked foundation that funds Ai2 is publicly pivoting toward applied AI over frontier open- model development.

Meanwhile our closed-model philosopher gods grace magazine covers.

No Jensen, but a shredded Demis Hassabis. That poor robot.

Green Shoots Sprouting Down the Stack

That Cursor is building on open weight is not an anomaly. A quick look at OpenRouter's top models over the past month shows that four out of the top five (and 12 of the top 20) most-used models are open weight.

OpenRouter Top Models, snapshot taken April 21, 2026

The numbers above are not a measure of total AI inference market share, as they don't capture workloads generated from applications developed by OpenAI, Anthropic, or any other model developer that also builds its own applications. The tokens processed by ChatGPT, Claude Code, and Claude Cowork would undoubtedly dwarf the total tokens processed by OpenRouter.

However, the numbers above are an additional proof point that application builders are moving away from the frontier labs' hosted offerings towards building their own solutions on top of open-weight models. Increasingly, the stack looks like the following:

This emerging stack reflects the fact that the AI ecosystem is maturing.

Most AI applications need models that are good at the very specific tasks their product does. Cursor does not need a model that is good at physics and biology, it needs a model that can design, write, edit, and reason about code in a real repository with real tools quickly, accurately, and cheaply. That is a much narrower target than "frontier general intelligence." Cursor now has built up a dataset from real-world use over time which it can leverage to effectively post-train. Training a base model from scratch means paying to teach it basic reasoning and tool use all over again. Calling a frontier API means paying for superintelligence your product will never exercise. Taking an open base and RL'ing it on your corpus of coding data and environments is the option that spends compute where it actually matters.

Where Open Weights Are the Only Option

The post has focused on LLMs and application stacks built on top. It should be noted there are other areas, such as on-device language models or Physical AI, where the only options are open-weight models. I would argue this is the case because these markets are still very early, and for all the players any progress will help all the ships rise. As these ecosystems mature, we can expect dynamics similar to LLMs to take shape.

Who Will Pay to Develop Open Weight?

Training LLMs costs a lot of money, and the base models are still improving rapidly even if the pace of progress from pre-training is slowing. Releasing it and getting no API revenue from it is not a business. And yet the application layer described above is quietly being built on this infrastructure and somebody has to pay for it. In the US, the traditional answer was Meta, but the new answer is increasingly no one.

Meta is retreating, Ai2's backer is pivoting to applied AI, Mistral has gone into services. Chinese labs have jumped on this vacuum and used open-weight as a wedge for distribution. The Kimi family of models now has the best open- weight models available. But similar to Alibaba and Qwen, one can only assume most of them will lock up their frontier models over time.

The obvious US player with the means and motivation to bankroll open-weight model development indefinitely is Nvidia. Thankfully, at this year's GTC, it committed $26B to open- source model development. It has the compute resources. One can only hope it will bring on a team that can turn this investment into strong models with good distribution.

Parting Thoughts

Open-source software was declared dead many times. Yet today it powers most databases, operating systems, and web browsers in the world. LLMs will not be different. Open weights won't own the frontier of capability and headlines, but they will win at the base layer. Expect a lot of AI products not coming out of OpenAI, Anthropic, and Google over the next 24 months to quietly look a lot like Composer 2 under the hood. And if you are working on open-weight models in the US, my hat's off to you. You won't get a spot on the cover of The Economist. You will have to be content with powering a diverse AI application ecosystem.

The post The Base Layer Will Be Open appeared first on Heavybit.

On episode 36 of Open Source Ready, Brian Douglas and John McBride sit down with Jesse Vincent. They explore how Jesse’s Superpowers project turns AI coding tools into structured, reliable development systems. The conversation dives into agent orchestration, prompt engineering, and what it takes to make AI behave like a capable software engineer.

The post Ep. #36, Managing AI Coding Agents with Jesse Vincent appeared first on Heavybit.

In this episode, the hosts unpack the thinking behind the name Third Loop and what it represents. Building on ideas from their book Progressive Delivery, they explore the gap between shipping software and having it truly adopted by users. The conversation dives into feedback loops, user agency, and why modern software requires a closer relationship between builders and consumers.

The post Ep. #3, Give It a Name: Why Software Needs a Third Loop appeared first on Heavybit.

On episode 54 of Generationship, Rachel Chalmers sits down with Vishakha Gupta to explore the hidden infrastructure challenges behind modern AI. They unpack why multimodal data systems are still fragmented, how graph and vector approaches can be unified, and what it takes to build production-ready AI pipelines. This conversation offers a systems-level perspective on scaling AI beyond prototypes.

The post Ep. #54, Human-like Memory with Vishakha Gupta appeared first on Heavybit.

On episode 50 of The Kubelist Podcast, Marc Campbell and Benjie De Groot sit down with Ivan Burazin to explore the rise of sandbox environments for AI agents, how Daytona enables instant, stateful compute, and why traditional infrastructure models fall short. Ivan also shares lessons from building early cloud IDEs and finding product-market fit in the AI era.

The post Ep. #50, Building Sandboxes for AI Agents with Ivan Burazin appeared first on Heavybit.

On episode 35 of Open Source Ready, Brian Douglas and John McBride sit down with Den Delimarsky. They explore the rapid evolution of the Model Context Protocol (MCP), from its early RFC days to a structured, industry-backed ecosystem. The conversation dives into governance, authorization, and how MCP is shaping the future of AI-driven developer workflows.

The post Ep. #35, The Evolution of MCP with Den Delimarsky of Anthropic appeared first on Heavybit.

On episode 89 of o11ycast, Ken Rimple and Charity Majors are joined by Bryan Cantrill. They dive into the origins of observability, the realities behind AI productivity gains, and the tension between cloud convenience and infrastructure control. The discussion highlights how major tech shifts often look obvious only in hindsight.

The post Ep. #89, Software Is the Killer App for AI with Bryan Cantrill appeared first on Heavybit.

So, AI Generated Lots of Code. What Does any of it Do?

For years, the all-consuming measurement of effective software development has been “developer productivity,” the ability of a team to deliver as much high-quality code as quickly as possible. Thanks to advances in AI-powered code generation tools, delivering large quantities of code quickly is now less of an issue.

But there are new bottlenecks. As AI tools spin up millions of lines of seemingly “good enough” code, the bottleneck has shifted to reviewing it. With the cost of generating code already approaching zero, one of the biggest future challenges in software may simply be understanding it. As new engineers enter the workforce and existing ones move on to new projects, they will inherit entire codebases that were not written by human hands.

To help humanity navigate this new future, a team of graduate students (Amir Amangeldi, Felix Wang, and Allen Zhang) set to work on building Noodles, an open-source project that helps users understand the reasoning behind different code branches with visual abstract syntax tree (AST) parsing. Below, the team explains why they built the project and what comes next.

The Noodles project visualizes what each branch of a repo does with AI-generated annotation. Image courtesy Unslop

The New Bottleneck: Understanding AI-Generated Code

The Noodles team is blunt. “The fundamental problem [in AI coding today] is that parsing AI output is hard.” The three creators explain that each of them relies heavily on AI tools like Claude Code these days, but have found it increasingly difficult to find time to review every last line of AI-generated code.

“We felt that we needed a higher level of abstraction to review changes. So we thought: Why not create diagrams that visualize what the changes are at a higher level?” The Noodles project emerged as a way to represent that abstraction, essentially as an additional, visual layer that stays updated as the codebase changes.

“Understanding AI-generated code, that’s 100% the bottleneck,” says the team. “And when we say ‘understanding code,’ that entails reviewing, asking critical questions, making probing assumptions, and testing. Which is harder to do.” Admittedly, it’s possible to delegate code review-related tasks to agents now, but that also effectively delegates away the ability to understand AI-reviewed code, to say nothing of how detailed or granular the picture could or should be.

“The other bottleneck is: As you scale and have more output, perhaps you are able to grasp what's happening. But how do you coordinate the different coding agents that are working for you at the same time? How do you keep track of the context of what's happening at any given point on your behalf?” Beyond the immediate issue of understanding what a specific code branch does, there’s a second, broader gap in trying to understand the larger context across everything your agents are doing.

Using Deterministic Tooling for Visualization

The Noodles team admits that their initial spec, which utilized LLMs for visualization, didn’t quite work out as planned. “To be frank, the output was different every time, which became difficult to track. So we thought: How can we make this as deterministic as possible?”

The team landed on AST parsing as an alternative. “It’s fully deterministic. Of course, it's not perfect, but it's able to go through classes and functions, and understand what's calling what consistently, and then represent that on a diagram.” The Noodles project was built to output a visual graph that displays connections across various functional nodes to explain what each connection means, and summarize changes at the PR and branch levels. It uses AI to group nodes together to arrive at higher-level diagrams as well as to enhance nodes and edges with helpful descriptions.

The team notes that the project was otherwise engineered around simplicity, leading to three different node designations for input entry points, process points (encompassing either individual functions or aggregate processes), and outputs, utilizing Mermaid diagrams, which turned out to be more popular than D2 diagrams.

Real-World Use Cases (and Limits) of Visualization

The team notes that the project is seeing the most use in collaborative settings among teams where not everyone has been exposed to new code or changes. “For teams that are still using pull requests, [Noodles seems like an] entry point that's helpful, specifically for large changes. That's the point where people want to understand ‘what happened.’”

“And that's why we have a free GitHub integration where you can integrate our unslop bot into your repository, and then analyze it, just like you would with a regular code review agent. Running an analysis spits out a link that you can click to see a diagram pertaining to the changes in this particular PR. So that's the dominant use case.”

The team notes that in the grand scheme of things, onboarding is one of the most important use cases that requires more understanding. “Passing on the foundational understanding of what’s been done before is absolutely key, so for learning in general, there’s absolutely going to be space for that in education.”

“However, architectural diagrams on the cloud level are probably not going to go away. The further you zoom out for a complex system that has multiple different servers, databases, and whatnot…those types of diagrams are here to stay. But for code-level diagrams, it’s not as certain.”

“It’s really a question of how we can adapt fast-moving AI tools for humans, whether they're developers or vibe coders or any other sort of human using AI. And there's definitely [a place for] visualization, but I think the bigger problem is what level of detail is too much to ask a human. To us, that ‘level of detail’ comes down to the human judgment that needs to be applied.”

“AI can predict a lot of things. But the places where human judgment needs to be applied, that's where some level of information representation needs to be passed on to humans. Whether it's in the form of a notification, including text or a diagram that explains to the human what remains for them to decide. Whether it's concise enough and delivers information in the right shape, at the right time, with the right level of scope. That's really the problem we're going after here.”

The team notes that, over conversations with its growing community, it has started to realize the limits of visual diagrams to help people understand code. “Folks are moving so fast with AI that hardly anyone has time to consistently stop and think about what's happening. They want to continue moving as fast as AI. So, having the understanding is great, but it does slow them down.”

The team utilizes its own tool to inspect its codebase, but finds itself wanting to move faster and find bottlenecks and workarounds more quickly. “The core question we’re struggling with now is how to have that understanding while moving faster. We’re designing our next project around that question.”

How Devs, Tooling, and IDEs Will Change

“With the barrier to entry into building things becoming lower, anybody can build. So the expectations of a human interacting with these tools is going to be different too. And so, the hardcore engineering practices around what it takes to understand or debug are going to have to change. The type of human that's engaged in engineering will be different.”

The team suggests that all the talk in the headlines of organizations flattening their hierarchies due to AI isn’t just talk. “All these roles of product managers, designers, and engineers collapsing into one role in fast-moving startups is a real trend. The developer tooling of the future is going to have to be much more approachable, much more simple.”

What will the IDE of the future look like? “We don't see it being a bunch of files that you have to click through, because the sheer number of files is going to be so large. And it’s not clear that a terminal is good enough either, because the terminal is constrained in its functionality. We believe it will have to be a standalone user experience that is able to balance between the chatting, the browsing, and the visuals in a very simple, unified manner.”

“So yes, the IDE of the future is certainly going to borrow some aspects of all of the things that have come before, but it will have to fundamentally change.”

The post How Visualization Helps Unblock the AI Coding Bottleneck appeared first on Heavybit.

On episode 10 of Data Renegades, CL Kao and Dori Wilson sit down with Toby Mao. They explore how real-world data problems at Netflix and Airbnb led to the creation of SQLGlot and SQLMesh, and why data modeling remains the hardest challenge in data engineering. The conversation also dives into how AI is reshaping engineering workflows, without replacing the need for strong architectural thinking.

The post Ep. #10, Data Modeling Matters Most with Toby Mao appeared first on Heavybit.

On episode 2 of Third Loop, Kim, Heidi, and Adam sit down with Kent Beck. They explore how Progressive Delivery extends ideas from Agile and Extreme Programming by focusing on safer releases, feature flags, reversibility, and observability in production. The conversation also dives into AI-assisted coding, experimentation, and what it takes to ship software users can actually trust.

The post Ep. #2, Features and Futures with Kent Beck appeared first on Heavybit.

On episode 34 of Open Source Ready, Brian Douglas and John McBride sit down with Sunil Dhaliwal to explore how AI is reshaping open source and startup building. They discuss why technical founders still have an edge, how taste is becoming a critical differentiator, and what it takes to find product-market fit in a world of abundant software. The conversation also dives into monetization challenges and the evolving role of open source.

The post Ep. #34, Technical Founders in the AI Era with Sunil Dhaliwal appeared first on Heavybit.

On episode 53 of Generationship, Rachel Chalmers sits down with Dr. Mehdi Nourbakhsh to explore how artificial intelligence is transforming the physical and digital worlds. They discuss the origins of generative design, why many AI initiatives fail inside organizations, and how leaders can move beyond experimentation toward real adoption. Mehdi also shares practical enterprise AI use cases and his perspective on how AI can augment human decision-making.

The post Ep. #53, The Era of Reimagination with Dr. Mehdi Nourbakhsh appeared first on Heavybit.

2025 was to be the year of AI agents, a prediction that may or may not have come true, depending on which people you ask. For a time, it seemed like the open-standard MCP would become the standard for orchestrating multiple autonomous AI agents (and perhaps it will), but not before it addresses its known security vulnerabilities.

For more perspective on putting agents into production for real-world engineering teams, we spoke with founder Dexter Horthy of HumanLayer to understand how agentic coding will evolve past its many questions about security, reliability, and token economics into full adoption into mainstream software development.

Evolving Agentic Products for Real-World Engineering Teams

Horthy acknowledges the breakneck pace at which agentic has evolved even in the past year. His own organization rolled out a "Research, Plan, Implement" (RPI) framework to help engineers break complex coding tasks into discrete phases: research the codebase first, plan the approach, then implement, rather than throwing an entire ticket at a coding agent and hoping for the best.

"We rolled this out last year, but I tell people: 'Look, I don't know if we'll even be using these same techniques in six months.'" Still, the founder notes that the RPI framework and his team's original work on context engineering and 12-factor agents seemed to become much more relevant as coding agents improved by leaps and bounds in the back half of 2025.

"I wanted to make it clear that there was no 'magic' or 'silver bullet.' Under the hood, we were fundamentally helping people do better context engineering, and that's what we've started productizing."

Dexter Horthy discusses the importance of managing context windows in software engineering. Image courtesy AI Engineer World's Fair

The Adoption Gap: When Expert Tools Meet Real Teams

The founder candidly admits that his team saw a consistent pattern when rolling out RPI across engineering organizations. Expert engineers who invested deeply in learning the tools started shipping like crazy, but when those experts handed the same tools to their teams, the results were inconsistent at best.

"We had people who loved what we did and became very good at it just by finding our stuff online, or sitting down and doing a workshop with us." But when the early adopters tried to share the new knowledge and frameworks with their colleagues who weren't as invested, those folks didn't get the incredible results that were possible with a lot of time investment, and the adoption didn't spread.

When Horthy's team investigated, they found specific, diagnosable problems. The first was bad research: A skilled engineer would naturally break a ticket into targeted codebase questions, but most people would just paste the entire ticket into the research tool and get back opinions instead of facts.

"The problem is, if you tell the model what you're building, you get opinions. The model starts making implementation decisions instead of proposing options to you around how to proceed."

The second problem was bad plans. The team's planning prompt had 85+ instructions, and buried inside were critical interactive steps like "present design options to the user" and "get feedback before writing the plan." For about half of users, the agent would skip all of that and just hand back a finished plan with all decisions already made.

"When we investigated, we found the difference between good and bad results was a single line: 'Work back and forth with me, starting with your open questions and outline before writing the plan.' I found myself standing in workshops full of enterprise engineers saying, 'Folks, yeah, here's the software, but don't forget to say the magic words.' It was, quite frankly, embarrassing."

The founder notes that this wasn't the user's fault. It was a tooling problem. His co-founder Kyle wrote a blog post citing research showing that frontier LLMs can only follow about 150 to 200 instructions with good consistency. Their planning prompt alone had 85 instructions. Add in the system prompt, tool definitions, and MCP servers, and the model was simply over budget.

"Solving these problems is actually still pretty hard. I don't think any of this stuff has been commoditized yet, though maybe it will be. But that's why we're excited about building collaboration features and changing the SDLC by creating a Google Docs/Notion/Figma-like experience around coding agents."

The Instruction Budget: Why Less Context Means Better Code

This diagnosis (that LLMs have a finite instruction budget) became the foundation for Horthy's fix. As Geoff Huntley puts it, the less of the context window you use, the better results you'll get. But it's not just about stuffing in too much information. You can also overload the model with too many instructions.

Huntley's Ralph Wiggum loop, an ultra-viral technique of wrapping a coding agent in a Bash while loop to periodically refresh its context, took off because it addressed the performance-eroding context rot that creeps in over successive runs. Horthy sees the lessons as more durable than the technique itself.

"The thing about Ralph is that we might not be using the same prompts in six months. It might not be the same technique in six months. This stuff is going to evolve. I think the most valuable thing about Ralph is the lessons that it teaches about context engineering."

"If what you're doing is grounded in understanding context engineering and the limitations of attention, you don't actually have to do any of that complicated stuff. If you just take care of keeping your context window small, you can get really good results."

Horthy suggests the broader conversation around Ralph ultimately reinforced a principle his team was already learning the hard way: giving the model fewer instructions, simpler tasks, and smaller context windows. "I don't know if I actually recommend that people should use Ralph to write production software."

"It's more about understanding how it works and why it's good and then applying those lessons for your workflow. For example, our implementer agent uses faster, smaller models for writing the code and running the tests, and then a bigger, smarter model spot checks changes, keeping context usage in the parent session low, allowing for resteering."

Geoff Huntley's viral Ralph loop is a clever hack, but does it represent the actual future of agentic coding? Image courtesy Geoff Huntley

Will Coding Agents Help Everyone Roll Their Own Everything?

While use cases for different verticals emerge frequently, AI's strongest proof of concept still seems to be in code generation. Yet as users attempt to build in more multi-step long-horizon tasks with their coding agents, they continue to bump up against the limitations of long context windows, including rot and hallucinations.

"I think the thing that everyone will agree on is that you will always get better results if you use less of the context. And 'using less of the context' is a thing that you, the engineer, have to think about. How do you want to do this, and how do you fit that into your workflow?"

Will models and coding agents eventually hit an inflection point past which organizations stop relying on packaged software and start simply rolling their own? "I think we're going to see a lot more hand-rolled versions of software. The 2010 ZIRP-era practice of 'get a SaaS for everything' is probably going to look very different in the coming years."

"I got really excited a few years back when Klarna claimed it would drop Salesforce and Workday and just build it all themselves. Those are the two vendors that nobody ever fires! I think they ended up like walking that back a bit." However, the founder suggests that orgs rolling their own internal tools may still be the right direction.

"I'm not going to say that the change will happen through vibe coding because I'm sure engineering will need to be done. There's a difference between spending 20 years building a piece of software and building an internal version in a hurry. There are just things that happen when you spend 20 years on something that you will not get for a while no matter how much AI you point at it."

Please Read the Code: Why Reviewing Plans Wasn't Enough

Perhaps the most candid reversal in Horthy's updated thinking concerns code review. In the original RPI framework, he advocated for reading plans instead of code, with the logic being that an engineer can't easily review 2,000 lines of generated code, but can review a 200-line plan.

"I was wrong. I am humble enough to admit when I was wrong." The founder explains that after six months of reviewing plans instead of code, the team had to rip out and replace large parts of their system. A 1,000-line plan tends to produce about 1,000 lines of code, so there was no actual reading "savings." Worse, the implementation would sometimes diverge from the plan, forcing the reviewer to hold both in their head.

"The new advice is: Don't read the plans. Please read the code. It's the same amount of work, there may be surprises that cause drift between the plan and the code, and the code is the thing that actually ships."

Horthy acknowledges the elephant in the room, namely high-profile open-source projects like Beads where allegedly nobody reads all the code. "These are very cool projects. But nobody gets paged at 3 a.m. if they're broken. Nobody gets fined millions of dollars if it's done wrong."

The founder describes how this realization shaped the team's approach to feedback loops. "How do you increase the odds that the code being reviewed is correct and needs no iteration?"

Rather than reviewing faster, the goal became aligning the agent before it writes code. The team now surfaces all codebase patterns that might apply to a problem and lets humans decide between old patterns that shouldn't be used anymore and newer approaches that should inform future decisions.

"LLMs are really good at finding patterns and following them. In a way, this is all they do. They take some input, some text, and translate it. 'Match the format of this code we found in the codebase in this other file over here.'"

Horthy also wrote a blog post about making testing feedback more context-efficient, giving the model deterministic signals about what's broken rather than dumping entire test outputs into the context window. "No matter how big context windows get, you always get better results if you use less of them."

No matter how big context windows get, you always get better results if you use less of them." -Dexter Horthy, Founder/HumanLayer

From RPI to QRSPI: Splitting the Monolith

Before going deep on coding agents, Horthy wrote 12 Factor Agents, which was arguably the first deep dive on "context engineering." The paper's core argument: Don't use prompts for control flow. If you know what the workflow is, use actual control flow. Classify the input, then feed it to a series of smaller, more-focused prompts with fewer instructions and fewer actions to choose from.

The irony was not lost. "We got on stage and said 'full-fat agents don't work, build workflows and micro agents, use control flow for control flow.' Then we turned around and wrote a giant monolithic 85-instruction prompt. It was time to drink our own Kool-Aid."

So his team split it. Their original three-step workflow (Research, Plan, Implement) became a seven-step pipeline: Questions, Research, Design, Structure, Plan, Worktree, Implement, each step with fewer than 40 instructions.

The key innovation is where the human review now happens. Rather than reviewing a 1,000-line plan or 1,000 lines of code after the fact, the new workflow front-loads alignment into two short artifacts: a design discussion (~200 lines) and a structure outline (~2 pages).

The design discussion captures current state, desired end state, patterns found in the codebase, resolved decisions, and open questions. The structure outline maps out the order of phases and how to validate along the way.

"You're forcing the agent to brain-dump everything it found, everything it wants to do, everything it thinks you want, and ask you questions about things it doesn't know. You get to do brain surgery on the agent before you proceed downstream. 200 lines instead of 1,000. That's leverage."

The acronym didn't work out so well (QRDSPWIP), so the team picked a subset and started calling it QRSPI.

The Future of Orchestration

Regarding the future of managing agents, Horthy cautions against making absolute judgements about whether one agentic technique or framework is uniformly "better" without considering the details.

What's more important is knowing when to decompose. "Sometimes you hit a wall, so you decide to decompose your workflow and make it more reliable, more robust, but maybe a little bit less able to handle everything in the input space."

"But once you get to know what the input space would be, you can give people a happy path that's much more reliable. And then you can always still eject out of the situation and say, 'Hey, something unexpected happened. Let's see if the LLM can figure it out.'"

On the question of whether agentic frameworks could be declared uniformly 'good' or 'bad,' the founder concedes that using them like properly-specified tools in a loop, with proper expectations, can work well...provided you're prepared to decompose longer workflows into smaller slices and manage context across each step. "These are tools in your arsenal. You need to know where on that autonomy slider you want to be."

The post What's Missing to Make AI Agents Mainstream? appeared first on Heavybit.

Acqui-Hires Then Vs. Acqui-Hires Now

Throughout startup history, an acqui-hire meant a company had failed. The product didn’t break out, revenue growth stalled and the board found a soft landing. A bigger tech company would buy the assets, but the transaction was for the team and not the business or product.

Most of the consideration (cash + equity) would flow to the founders through retention packages, some employees would be brought on, and investors typically got somewhere between zero and their original investment back. Effectively the acquirer was recruiting a hard-charging team with specific technical domain expertise.

But in the AI era, a new category of expensive acqui-hires has emerged, including Inflection, LoveForm, Windsurf, Scale AI, Adept, OpenClaw. And of course, there’s a regulatory angle here, as government agencies like the FTC have criticized acqui-hires as a shortcut to avoid antitrust regulation that would be an issue in a normal acquisition. (There’s still congressional scrutiny of acqui-hires going on in 2026!)

While some of these deals are not as egregious as hyped up in the press, the trend is real. If this trend continues, it will have a profound impact on the relationship between start-up founders, investors and employees.

The Four Levels of Acquisitions

Traditionally, acquisitions would fall into one of four categories, with some tending toward huge-money deals on down to a much smaller scale:

Starting at the high end, $$$$ deals with bankers, top-shelf liquor, and steak dinner celebrations:

• Business Acquisitions: Much higher-end deals, such as the Activision → MSFT merger
• Product Acquisitions: Product-based deals, such as the Instagram → META acquisition
• Technology Acquisitions: Focused on core tech, such as the DeepMind → GOOG deal
• Team Acquisitions: For core team members, usually lower-end, lower-spend affairs

Down to the low end, $ deals with interview loops, beers, and maybe a celebration at In-N-Out!

The Mega-Acqui-Hire

Mega-acqui-hires get a lot of press attention because they imply companies are paying numbers once reserved for elite athletes to hire engineers.

But when you look closely, these deals are not all the same. They fall into three categories:

1. Normal Acqui-Hires: The headline number is large, but not as a multiple of the amount raised. The structure is classic acqui-hire at a time when investors are plowing money into AI startups.
2. Team + Tech Acquisitions: The acquirer believes they're buying into a market that's about to be worth trillions.
3. Elite Athlete Contracts: Paying whatever it takes to put the best talent on the roster, regardless of whether the deal makes traditional M&A sense.

Google’s $500M acqui-hire of DeepMind’s founders seemed like a gamble, but it paid off. Image courtesy BusinessInsider

DeepMind → Google (2014): Team + Tech

Google's purchase of DeepMind in 2014 was arguably the first mega-acqui-hire, although "team + tech" is the better label, in my opinion. DeepMind had made genuine breakthroughs combining reinforcement learning and neural networks. The company had little revenue and no real product to speak of, but Demis Hassabis, Mustafa Suleyman, and Shane Legg had clearly built something that mattered. Google paid over $500M—an extraordinary number at the time—because it was buying both the people and the technical lead they had already created.

In hindsight, this was a great bet. It was a very early, very large bet that a small research team had found one of the most important seams in computing.

Inflection → Microsoft (2024): Normal Acqui-Hire, Big Numbers

Despite the headlines, Microsoft's "acquisition" of Inflection AI looks like a well-connected board navigating towards a soft landing.

Inflection had raised about $1.5B. Microsoft paid $650M directly to the company, not to the founders. Investors like Greylock reportedly made 1x to 1.5x on their capital. The flagship product (a chatbot called Pi) was clearly losing to ChatGPT, Microsoft was already the largest investor, and Reid Hoffman was one of Inflection's co-founders and a Microsoft board member.

Adept → Amazon (2024): Normal Acqui-Hire

Amazon's deal with Adept followed the same template. The headlines are eye-catching, but Adept had already raised more than $400M and its agent product had not found meaningful traction. From a distance, it looked like a frontier AI talent coup. Up close, it looked like another large soft landing: Investors got their money back, the founders got high-profile roles, and Amazon got a team it wanted.

This was another example of the classic acqui-hire playbook inflated by the amount of capital that has sloshed into AI. The boards can still navigate exits to help recoup their investment.

Windsurf → Google (2025): Team + Tech

The Windsurf deal is where the modern structure of acqui-hires starts to look different, but I would argue this is a "team + tech" deal. Google paid $2.4B while Windsurf had raised about $243M. More importantly, the deal was not simply "company gets bought, team joins acquirer." Google hired the CEO and key researchers, licensed the technology, and effectively took the pieces it cared most about, while the remainder of the business later found another home.

That is not a normal acqui-hire with big numbers. It is a premium paid for a team that had deep expertise in one of the most important emerging categories in AI. Looking back from March 2026, it is hard to overstate how well timed that bet was. In the subsequent year, code generation broke out as the dominant AI use case, and Windsurf had a team with both product intuition and technical understanding in exactly the right place.

Google was not rescuing a failed company. It was paying aggressively to bring the team and tech that exploded in the subsequent 9 months.

By 2025, it was clear that coding was the #1 use case for AI. Image courtesy OpenRouter

Scale AI, NFDG → Meta (2025): Paying Elite Athlete Money for Talent

Meta's AI acquisitions are truly a departure from anything that came before. The closest analogy is what happens in professional sports: The ultra-rich teams pay whatever it costs to get the best players on your roster to win a championship ring.

Take the Scale AI deal—a 49% stake at a $14B valuation. Scale has real revenue (reportedly over $1B annually) and a real product in data labeling and RLHF infrastructure. But Meta didn't buy Scale for the labeling business. It bought Scale to bring on Alexandr Wang.

The NFDG deal is similar. Meta didn't buy them to make money on the NFDG portfolio. The deal was to bring on Nat Friedman and Daniel Gross.

Meta is treating AI talent the way the Dodgers treat pitchers or Paris Saint-Germain treats star strikers.

I imagine the thinking is: At a time when Meta is spending $65B+ annually on AI infrastructure, why not pay a few billion more for talent?

So, can you buy a championship in this game? Meta started 2025 with a clear open-source model lead with the Llama family. It has now lost that lead to Qwen and other models from the Chinese labs. Yann LeCun has left and Alexandr Wang has been moved to head up one of Meta’s AI labs, while standing up a separate and totally different “applied AI” lab.

For comparison, Anthropic has made 3 acquisitions in the lifetime of the company: Humanloop, Bun, and Vercept.

LoveFrom, OpenClaw → OpenAI (2025, 2026): ¯\_(ツ)_/¯

These are hard to categorize since we know very little about the io tech and the OpenClaw numbers are undisclosed. My guess is these fall somewhere between "Team + Tech" and "Elite Athlete" acqui-hire. Although, the io announcement video certainly gave The Decision on cringe factor.

So, Are Things Changing?

Yes. A few trends are clear:

1. Acqui-Hires as a Preferred Outcome: As AI funding rounds have gotten larger, boards have been more likely to navigate to large acqui-hires for the AI talent when the business has not taken off. While not big outcomes for investors, this helps set a floor on the investment outcome and encourages continued large funding rounds.
2. Mega-Acqui-Hires Are the Exception, not the Rule: Very few companies (Meta, maybe OpenAI) are willing to pay elite athlete comp for AI talent.
3. The Definition of “Team + Tech” Will Change: With the improvements of codegen tools, we can expect to see the line between "team" and "tech + team" blur for the best engineers. In other words, the codebase will have little IP value, as a few people can easily rebuild it.

Misaligned Incentives

Mega-acqui-hires create tension between founders, investors, and employees that the traditional startup playbook did not really anticipate.

In a classic acqui-hire, everyone more or less lost together. The company did not work, the exit was modest, and there was not much ambiguity about what had happened. The newer structures are messier. If a large buyer pays hundreds of millions for a technology license while hiring the founders and key researchers into massive comp packages, a lot of the economic value can bypass the normal cap table logic. Founders may walk away in great shape while investors take a loss, or at best grind out a flat return. That is a very different emotional and financial outcome than a normal company sale.

Investors have noticed, and tighter language is already being added to NVCA docs around IP transfers, exclusive licenses, asset sales, and founder departures tied to employment offers from strategic buyers. The point will not be to stop every deal, but to prevent a scenario where founders effectively sell while leaving the cap table behind. That concern is no longer theoretical.

Employees are arguably in the worst position. They cannot insert tag-along rights into their employment packages. AI teams are more valuable than ever, but the upside is not necessarily distributed evenly across those teams. Some people are being paid like franchise players. Others are discovering that being adjacent to rare talent is not the same thing as being treated like rare talent.

After Windsurf’s founders were acqui-hired by Google, the rest of the team was scooped up by Cognition. Not all teams will be as fortunate. Image courtesy Windsurf

So What?

The conventional wisdom has always been: Do not build a company to be acquired. I think that advice still holds. Most mega-acqui-hires are boards finding soft landings to recoup their large investment and will not lead to huge windfalls for the founders. It is too early to tell if the elite athlete acqui-hires were just a failed experiment at Meta. They certainly appear that way as of this writing.

But AI has introduced a real paradox. At the exact moment we are being told AI will automate large swaths of knowledge work, certain knowledge workers are more valuable than ever. And the dynamics are very beneficial to founders who fit in this category. You can easily raise large amounts of money, miss on product-market fit, and still end up producing one of the most valuable assets in the ecosystem: A team that knows how to build frontier systems together.

The acqui-hire is no longer just a consolation prize. In a few corners of AI, it is becoming its own category of outcome.

The post The Acqui-Hire Is No Longer a Distress Sale appeared first on Heavybit.

Where Back-End Systems Are Headed in the Age of AI

While much has been written about AI’s ability to dramatically speed up developer productivity by spinning up thousands of lines of code, far less has been written about the effects of AI on back-end engineering. But the practice of managing server-side, database, and APIs is also evolving, and not just due to code generation.

Gartner estimates that within a few years, 80% of API calls will not come from humans, but from AI agents that autonomously scuttle forth and mercilessly hammer endpoints with endless data requests. As a result, back-end systems may need to become significantly more durable, but also more flexible to accommodate the wider variety of interactions from many different species of AI agents and tools.

Mike Piccolo is the founder of the iii (formerly known as Motia), an open-source, language-agnostic orchestration engine that his team built to help back-end systems and teams adjust to this brave new world. Below, he shares his thoughts on how back-end architecture needs to evolve, and how teams should think about the future to come.

In the same way that MVC and React were needed to unify server-side data management and front end, respectively, back-end systems may also need a new core primitive.

Repeating History: Front-End Complexity Moves to Back-End

Piccolo suggests his project echoes the evolution of technology from the digital transformation era, when server-side applications, previously built and maintained on poorly-organized PHP spaghetti code, began evolving to model-view-controller (MVC) architecture. “Every company was taking a bespoke approach to [their digital transformation],” the founder notes, leading to an unsustainable level of complexity.

“This is what necessitated MVC, which was an abstraction that tells you how to properly build this type of server. Everybody just switched to it, almost immediately. It was Rails, then Django, then Laravel, then .Net MVC, then Spring. Basically all the programming languages implemented it.”

MVC being a universal framework temporarily solved the problem of orgs having to reinvent the wheel until teams began pushing the logic to the front end, making more-complex experiences that led to the front-end framework wars. “You have Angular, Ember, React, Vue, and others. This was the next paradigm shift, which React solved in the correct way.”

Piccolo compares React’s success to MVC: A framework that’s also now in use everywhere. “They took all the complexity of building inside of the Document Object Model, including event listeners, the event loop, and all of these things, and they compacted it into a single core primitive, the component. And it made both sides of the house really happy.”

The React component gave implementation engineers a mental model to build frontends as complex as they liked, but was good for library maintainers, framework developers, principals, and architects because “they got to use these low-level primitives that React provided to build this ecosystem, including Next.JS, Redux, and TanStack.”

The founder suggests that a similar paradigm shift is happening in back-end development, with complexity shifting from front end to back end. “What that looks like is tons of complexity going into a system that wasn't built for it, and the complexity reaching a threshold.”

The founder’s mission echoes that of MVC and the React component. “We are building a new set of core primitives that will handle all of the complexity to build a generically intelligent backend.” Piccolo confides that the task was larger than it seemed initially, requiring multiple API frameworks, such as Python Flask, Express.JS, or Next.JS, potentially used with a Java server.

“That’s your APIs covered. But now it gets more complex. You need background jobs and queuing.” After adding services like SQS, or BullMQ or RabbitMQ, the founder notes that teams often look to a workflow engine or durable execution frameworks such as Temporal or Apache Airflow to handle the durable execution.

“And then, you need to add agentic frameworks. So you’re adding shared-state concurrency with something like Redis, and streaming with a service like Pusher.” The founder notes that with so many disparate systems, builders need to wire them together, while also building observability over top, making for extremely complicated infrastructure with huge amounts of surface area.

“Each of these things has its own domain, its own SDKs, and its own APIs. The only thing in common across all of them is that there are functions, workers, and triggers you can combine.” Piccolo explains that carefully observing those common features led his team to create the core primitives of the iii project.

“You send events in, and they trigger a particular function inside the queue, and workflow engineers do something similar but with an ordered workflow. Agents also have this type of recursive loop, but those are really just triggers for the next instantiation of the agent call. That’s the basis of what we're building, this core set of primitives that will provide common observability, common shared resources, and services across this whole stack.”

A World Where Millions of Lines of Code Appear Every Moment

The founder suggests that AI tools have improved so much in recent months that they seem to be starting to outpace top-level coders. “I have several senior- or principal-level architects on my team, and they’re extremely fast, switching screens so quickly you can barely see what’s going on. Previously, it was faster for them to code manually.”

“And in the last few months of using tools like Claude Code and Cursor, even my team is starting to adopt them and see value.” The founder notes that a growing challenge for orgs may be a massive influx of PRs that make catching up on code reviews difficult. Teams stuck with mountains of AI-generated code are experiencing the shock of having a year’s worth of code pile up on their terminals in just a few days, if not faster.

Piccolo suggests that this challenge will also be solved, perhaps via agents shipping better code and/or being better at reviews themselves. “Part of the solution is to reduce the surface area that these things need to build. If you ask an agent to build a Web app, by default, it's going to use React because it's the right abstraction.”

“The agent is capable of writing all the event listeners and all the DOM manipulation manually. But it won’t. It uses the smaller surface area, which is the correct abstraction. So if we do succeed in creating these core primitives in this abstraction, it will reduce the total cognitive burden. You'll need fewer tokens, and the agent needs to think less about it.”

Part of the solution is to reduce the surface area that these things need to build.” - Mike Piccolo, Founder / iii

Is Mass Adoption of Agents Right Around the Corner?

On the topic of large-scale agentic adoption, and where and when it will happen, Piccolo is reflective. Will the world of software soon be dominated by millions of autonomous agents hammering APIs non-stop? “There are different ways to look at it. The technology may be getting there, but enterprise adoption is a different question.”

The founder suggests that enterprises are not likely to be running free-rein Clawdbot instances within the next year. “I could be wrong! Things are moving incredibly fast. But just in terms of security, compliance, and legal concerns, [widespread] adoption for enterprise users just seems too far fetched. I'm highly technical myself, and I'm not comfortable running Clawdbot without really thinking through what I'm doing. I'm not going to give it my credit card yet.”

Piccolo concedes that users on Twitter and other social media platforms seem to be making great progress on getting agentic systems to work, but isn’t convinced that a long-term solution has emerged yet. “The problem is that we need this other layer for things to work. If you have durable execution and queues spinning in the background, and then you bolt on agentic? That makes things much more complex. It seems insanely hard to manage and observe and have confidence in.”

“We need this base layer of core primitives that provide the visibility, the explainability, and the durableness at the base layer below anything else we do, whether it be agentic workflows or durable execution workflows. All these things need to sit on top of a new foundation, in my opinion.”

What the Agentic Future Means for Back-End Teams

A long-running narrative in the AI space is how it will, inevitably, reduce the need for human engineers, and potentially shrink the sizes of development teams, even for back end. But Piccolo is skeptical of any significant change happening soon, at least at larger orgs. “[For an enterprise], you need to have a sizable team that is managing what you're building and ensuring that it's built in the proper way.”

“I think that a lot of the focus will move to the context engineering space, where that becomes extremely important to manage the system. That's already happening.” The founder also suggests that many orgs might not realize the immediate importance of system integrity (which used to be treated as code quality).

Piccolo explains that system integrity and code quality essentially resolve into the patterns from which AI agents learn, which has larger ramifications for any business. “You’d want to meticulously say: This is not the correct pattern. We like this other pattern and we want to keep it consistent across our codebase. That's the same meticulousness you need in this context space, which applies not just to the engineering side, but also to the business side. What is your domain, what does your company do? Your IT sides describe the code quality and patterns, and your business side describes how that translates into business value.”

The post How Back-End Engineering Is Evolving in the Age of AI appeared first on Heavybit.

On episode 33 of Open Source Ready, Brian Douglas and John McBride sit down with James Strong and Marco Ebert. They discuss the retirement of Ingress NGINX, one of the most widely used Kubernetes ingress controllers, and the factors that led to its deprecation. The conversation explores maintainer burnout, major security vulnerabilities like IngressNightmare, and the ecosystem’s shift toward Gateway API.

The post Ep. #33, Retiring Ingress NGINX with James Strong & Marco Ebert appeared first on Heavybit.

In this debut episode of Third Loop, James Governor, Kim Harrison, Heidi Waterhouse, and Adam Zimman explore how the concept of Progressive Delivery emerged from real-world frustrations with how the industry talked about shipping software. Drawing on experiences from companies like GitHub and LaunchDarkly, they explain how practices like feature flags, experimentation, and observability came together to form a new delivery model. The conversation also sets the stage for the podcast’s broader mission: examining technology through the perspectives of builders, users, and observers.

The post Ep. #1, The Story Behind Progressive Delivery appeared first on Heavybit.

On episode 7 of High Leverage, Joe Ruscio sits down with Peter Bell to explore how the CTO role evolves from early-stage founder to enterprise leader. They unpack what it really takes to scale AI adoption across an engineering organization, and why simply buying tools isn’t enough. The conversation dives into agentic software development, observability, context engineering, and what happens when production code is generated without direct human review.

The post Ep. #7, The CTO’s AI Playbook with Peter Bell appeared first on Heavybit.

On episode 49 of The Kubelist Podcast, Marc Campbell and Benjie De Groot sit down with Felipe Huici to explore how unikernels are reshaping modern cloud infrastructure. They discuss virtualization, containers, and how Unikraft enables millisecond startup times and massive workload density. The conversation dives deep into performance engineering, Kubernetes integration, and the infrastructure challenges emerging in the AI era.

The post Ep. #49, From Containers to Unikernels with Felipe Huici of Unikraft appeared first on Heavybit.

On episode 9 of Data Renegades, CL Kao and Dori Wilson sit down with Wes McKinney, creator of Pandas and co-creator of Apache Arrow. They explore how AI coding agents are reshaping software development, why data infrastructure remains surprisingly resistant to automation, and what semantic layers mean for the future of analytics. Wes also shares bold predictions about “radical accountability” in software and the changing bar for startups.

The post Ep. #9, Radical Accountability in Software with Wes McKinney appeared first on Heavybit.

On episode 32 of Open Source Ready, Brian Douglas and John McBride sit down with Glauber Costa to explore Turso, a Rust-based rewrite of SQLite built for the AI era. They discuss database reliability, open source licensing, and why embedded databases are becoming critical infrastructure for modern agents and applications. The conversation also dives into AI-assisted development and the future of software engineering.

The post Ep. #32, Rewriting SQLite for the AI Era with Glauber Costa appeared first on Heavybit.

On episode 52 of Generationship, Rachel Chalmers sits down with Piyush Agarwal to explore how developer behavior reveals far more about buying intent than traditional sales signals. They discuss why most dev tool GTM strategies fail, how to distinguish curiosity from real demand, and what it takes to engage developers at exactly the right moment.

The post Ep. #52, Serendipity as a Service with Piyush Agarwal appeared first on Heavybit.

During a discussion earlier this week with a friend working on an RL engine at a major academic lab, he mentioned his strongly held belief that an "RL infra company” can never be a viable standalone business.

I was shocked: not only because of who was saying it, but also due to the mindshare and investor interest RL commands these days.

But he has a point. From ~50 companies on Madrona's market map from a couple years ago, only 4 are focused on model training. Of those, one (Mosaic) was acquired, one (Cerebras) is mainly a hardware company, and Together and Modular now bill themselves as inference providers.

So, is RL/post-training infrastructure a venture-scale category by itself? Or should we think of it as a feature for inference providers and a workflow for the makers of foundational models?

RL Is Already Big Business

Billions are already being spent on RL:

Mercor claimed to be at $500M of annual revenue in September 2025.

Databricks reports $1B of AI revenue which includes RL from their Mosaic acquisition.

Anthropic and OpenAI are pouring money into post-training and RL as pre-training hits scaling laws and we run out of human-generated text data.

If you attended the NeurIPS conference in San Diego in December 2025, you would have felt like AI==RL.

At the same time there is no single large infrastructure company that is focused on RL or post-training infrastructure. Unsloth and Prime Intellect are the only companies I'm aware of dedicated solely to post-training infra.

Training Is Hard. Inference Is Eas(ier)

Doing inference reliably at scale is not easy. GPUs are scarce, context windows are exploding, and managing KV caches at scale is real engineering work. But it is, fundamentally, an infrastructure problem, not a research problem.

There are open models, open inference stacks like SGLang and vLLM, optimized kernels, routing layers, and cloud abstractions. If you have capital and good engineers, you can stand up an inference service.

Training a model is a very different process than simply running inference.

Training is a different beast, and RL in particular is messy, unstable, and expensive. You are interacting with environments, defining reward functions, and fighting regressions. While in theory models now have enough base intelligence to do long-form white-collar work with RL-based post training, the bottlenecks to delivering this vision are not generalizable algorithms and infrastructure.

The Environment and Verification Bottleneck

The current consensus is that we can deliver autonomous agentic AI from our base models through post training with RL, and that the blockers to delivering these agents are synthesizing the environments that mimic long-horizon tasks and defining verification methodology needed to determine correctness on these tasks. As tasks being automated become longer and more complex, defining what “correct” means becomes unclear and needs deep domain expertise. Reinforcement learning depends on reward signals, but in real work those signals are ambiguous, gameable, and can shift in long-horizon multi-turn tasks. Software development was the first domain to which labs delivered autonomous long-horizon agents (i.e. Claude Code), and I believe this was because the people who worked at the labs are the domain experts in writing code, and verifying software correctness is fairly straightforward.

Bottlenecks present business opportunities, and there is already a growing market around the problem of RL environments and verification. Frontier labs are the earliest and most concentrated buyers of RL environments and companies like Mercor and Halluminate are positioning themselves upstream as suppliers of expert feedback, evaluation scaffolding, and environments.

Companies like Applied Compute (and some RL consulting shops) are taking another approach: Instead of asking enterprise workers and recent grads to moonlight and create RL gyms, they are going to enterprises directly and leveraging artifacts created by their workforce to build "autonomous workers" owned by the enterprise. This makes sense: If you are a large bank or hedge fund, in order to prevent Anthropic or OpenAI from cannibalizing your business, you should capture artifacts created by your workforce (docs, screens and mouse clicks) to train your own autonomous workforce.

So, Can You Build a Company Around RL?

Yes, but it won't look like inference.

The inference providers won because they solved an infrastructure problem: Serve models fast, reliably, and at scale. Capital and good engineering get you there. But a standalone RL business cannot win on speed or scalability alone. The bottleneck isn't infrastructure, it's the algorithms and the complex data pipelines needed to see value from them.

A viable RL company needs to deliver something fundamentally new: continual learning from on-policy RL. Any such service will need to:

1. Include pipelines that synthesize RL environments from model outputs
2. Automatically build (or at least propose) reward signals and verification
3. Update model weights and biases in the background with little human intervention

Today, this is a research problem. Whoever solves it can create a generational company.

The post Can You Build a Company Around RL? appeared first on Heavybit.

On episode 8 of Data Renegades, CL Kao and Dori Wilson sit down with Scott Breitenother to explore how AI is reshaping the modern data stack and redefining the role of data teams. They discuss the evolution from spreadsheets to autonomous agents, the realities of data democratization, and why future workflows may revolve around one human working alongside multiple AI assistants. The conversation blends practical lessons from building Brooklyn Data and Kilo Code with forward-looking predictions about the next wave of data tooling.

The post Ep. #8, One Human Plus Agents with Scott Breitenother appeared first on Heavybit.

In episode 31 of Open Source Ready, Brian and John sit down with Matthaus Krzykowski, Thierry Jean, and Elvis Kahoro to explore how dlt and dltHub are changing the way developers build data pipelines. The conversation dives into DuckDB, LLM-driven workflows, and the growing shift toward developer-first data engineering. They also discuss open source adoption, AI orchestration, and what it means to be a “10x engineer” in 2026.

The post Ep. #31, Developer-First Data Engineering with dltHub appeared first on Heavybit.

Everything that follows is written through the lens of enterprise software. Large, revenue-generating software systems historically built and operated by teams of hundreds or thousands of developers.It has implications for other classes of software, but that is a discussion for another day.

Six months ago, if you had asked me how much production code would eventually be written by AI, I would have claimed a large percentage. LLMs are clearly a massive productivity boost for software developers, and the value of humans manually translating intent into lines of code is rapidly depreciating. I also believed, and still do, that humans whose primary job is to build and operate enterprise software are not going anywhere, even as their day-to-day work is fundamentally redefined by this newest abstraction.

What I underestimated was how little of that future work would involve reading code at all.

I am now convinced that for better and worse we are barreling toward a future where a large and growing fraction of production code is never read by a human. Not skimmed. Not reviewed. Not tweaked. I have taken to calling this Write-Only Code (shout-out to Waldemar Hummer of LocalStack for helping coin the term) and have been spending a lot of time thinking through what it means for us as an industry.

“AI writes the code” is already true inside many enterprise teams, but today it mostly means some form of AI-assisted pair programming. Humans decompose work into small tasks, hand them to agents, review the resulting pull requests, make edits, perhaps re-prompt, and iterate until the result is good enough to ship. This was a sensible practice and necessary to reliably ship production code with models that existed before late 2025. Subtly but importantly, in this workflow the requirement for human review also preserves the rest of the software development lifecycle, because human review remains the primary bottleneck through which all production changes have to pass.

That bottleneck is going away.

Recent step-function improvements in model capabilities are the first step in breaking many of the core assumptions underpinning the software development lifecycle (SDLC). These agents can now successfully handle much higher-level chunks of functionality. With emergent techniques that allow agents to plan, execute, and self-correct over long horizons, we are already seeing experiments demonstrate the shocking scope and complexity of working software they can produce. As these practices are adopted we will start producing software at a volume and pace where even if we wanted to, there will never be enough human bandwidth to review it line by line.

Unlike some AI maximalists, I still do not believe that “no one reads the code” implies “no humans involved.” Enterprises still require accountability. Someone must own the system, its outcomes, and its performance. Someone must answer for incidents, and ultimately be on the hook when things go wrong. Enterprises buying software are looking to their vendors for far more than just code.

The role of a software engineer has always been to sit between poorly specified, natural-language business goals and (mostly) deterministic machine behavior, while navigating a web of constraints. You can distill that role even further to “reducing risk.” Reduce risk that the business does not have the capabilities it needs to compete. Reduce the risk that the software is broken, that it cannot scale, or that it cannot adapt as business goals shift. That role does not disappear in a future where Write-Only Code flows directly from LLMs into production, but the culture, processes, and tools required for humans to fulfill it need a fundamental rethink.

History Doesn't Repeat Itself, but It Often Rhymes

Software development and delivery has a long history of eliminating one bottleneck, only to immediately confront the next. As we stand on the precipice of removing perhaps the biggest bottleneck ever, humans writing code, it is instructive to look at the last one we removed.

It is easy to forget that just a few decades ago the practical bottleneck for running new enterprise software in production was not writing code at all. It was hardware. Procuring servers, waiting for delivery, getting them racked, configuring networking, and then, months later, finally putting something into production. The combined emergence of continuous delivery practices and on-demand, ephemeral computing in the mid-2000s collapsed that constraint.

What followed was a new wave of tools and practices that assumed a fundamentally different posture toward production. DevOps and “pets vs. cattle” was not just a slogan, it was a rewriting of expectations. Production servers stopped being precious, named, and lovingly maintained. The idea that working servers could be stamped out programmatically, and explicitly not intended for human access, once seemed absurd. Today, if someone in a high-performing team shells into a production machine, it is considered tainted and scheduled for replacement by a pristine instance.

This shift moved the bottleneck squarely to developer velocity. Once infrastructure could be provisioned on demand and incremental changes could reach production as fast as they were created, the limiting factor became how quickly a human developer could translate business requirements into running software. This unleashed a new wave of productivity and, in my opinion, is why developers became the new kingmakers.

At the last startup I founded, this reality shaped nearly every decision we made about how we worked. I was maniacally insistent that any proposed change to our SDLC be evaluated first through the lens of developer velocity. Changes that safely improved developer velocity were effectively mandated. Changes that slowed developers down were heavily scrutinized and, if not rejected outright, at least tuned to minimize their impact. Over time, my team came to half-jokingly call this heuristic “Ruscio’s Law.”

What we treated as a simple rule of thumb is rapidly evolving from an optimization for elite teams into a prerequisite for survival.

From Review to Trust

For most of modern software history, human code review has served as the final backstop for confidence in production systems. Tests can be imperfect, monitoring can be incomplete, and bugs can be subtle, but in the end, skilled engineers who both create and review the code can reason about what it might do. Write-Only Code breaks that assumption at scale. If we are going to ship code that is never read by humans, we need other ways to gain confidence.

Much as humans no longer shell into individual production servers, I believe we will develop similar practices around unread code. Over time, we will treat “humans had to read this to be comfortable” as a smell in our code generation pipeline, or as an explicit, expensive trade-off reserved for truly mission-critical subsystems. A natural outcome of this shift is a “code reading coverage” metric, tracked much like test coverage. What fraction of production code has actually been read by humans, partly as a safety signal, and partly as a metric teams deliberately and safely work to drive downward toward an asymptote.

Pragmatic teams will not adopt Write-Only Code everywhere at once. They will identify where it is safe to begin and where traditional human review should remain. Understanding and controlling the Slop Radius, how far unintended behavior can impact before being detected or contained, will be a critical skill for teams to develop. As practices and techniques mature and the guarantees we can extract from automation improve, the surface area of unread code will expand and the scope of manual review will shrink.

The question for engineers is not whether this shift will happen, but what primitives will replace human authorship and review as the foundation of trust.

How the Engineer’s Job Changes

In the AI pair-programmer story, the human engineer is still primarily an author and reviewer. In the Write-Only Code story, that same engineer becomes a systems designer, a constraint writer, and a trade-off manager.

You spend more time shaping intent than shaping implementation. You obsess over interfaces, invariants, failure modes, and the conditions that must hold true. You decide what still requires human review and what explicitly does not. You invest in the tooling that makes “ship it blind” not a reckless act, but a competitive advantage. You reduce risk.

You also accept that the SDLC is being rewired even when the external inputs and outputs look the same. The business still asks for software and receives working software, but the inside of the factory is fundamentally different. That difference will decide who wins.

There is also a psychological shift that should not be understated. Engineers have long taken pride in crafting and deeply understanding what they ship. Entire books celebrate Beautiful Code. In a world of Write-Only Code, that pride shifts toward building systems that remain correct without requiring comprehension of every line. “I write code” becomes “I build software.” Human engineers will still be accountable for outcomes, but they will increasingly seek that confidence without ever reading the code.

In Conclusion

Write-Only Code is not a prediction about what we should want. It is a description of what happens when software production scales beyond human attention. The question is not whether humans should remain in the loop at the level of individual lines of code, but whether we are willing to take responsibility for systems whose behavior we can no longer fully inspect. We have been here before. Each time a bottleneck falls, the industry reorganizes around what replaces it.

The mistake would be to treat unread code as a failure of discipline rather than a signal that discipline itself must change. Human review does not disappear because it is unimportant, but because it no longer fits the scale and shape of the problem. The organizations that succeed in this transition will not be the ones that cling longest to familiar rituals, but the ones that invest earliest in new primitives for trust, accountability, and control. Refusing to adapt does not preserve safety. It simply ensures that adaptation happens accidentally, under pressure, and without intent.

The role of the human engineer has never been to type code for its own sake. It has been to reduce risk in the face of ambiguity, constraints, and change. That responsibility not only endures in a world of Write-Only Code, if anything it expands.

The next generation of software engineering excellence will be defined not by how well we review the code we ship, but by how well we design systems that remain correct, resilient, and accountable even when no human ever reads the code that runs in production.

The post Write-Only Code appeared first on Heavybit.

The "AI Engineer" job title is a recent invention, and companies are hiring a lot of them.

I've generally assumed an AI engineer is someone who can build high quality software applications that leverage LLMs at runtime. In other words, folks who do applied AI. I decided to validate that assumption by talking with a few folks in the space and scraping some job listings to build a better mental model of the job responsibilities and requirements.

The analysis below is from 31 job postings at 21 companies. The raw data is shared here.

More Resources on AI Engineering & Hiring Trends

• Article: What Makes a “Good” AI Product Manager?
• Article: AI Predictions for 2026
• Article: Language as a Proxy for Taste
• Article: How to Drive AI Productivity and Value for Engineering Teams

AI Labs Do not Have AI Engineers

If you go to the OpenAI or Anthropic careers pages, you won't find "AI Engineer" listed as a job title. OpenAI has an entire "Applied AI" division with listings for standard engineering roles (SWE, Android Engineer) and Anthropic has an "Engineering & Design - Product" team that focuses on AI native applications.

The technical positions at the labs broadly fall under the following categories:

• Research and model engineering: building the next models
• Infrastructure: deploying training and inference at scale
• Applied AI: building apps that leverage the foundational models

In other words, since the "Applications" orgs at these companies are focused on leveraging advances in the foundational models, all the engineers on those teams are Applied AI Engineers.

The Role Has Gone Main Street

A lot of companies that are not pure play tech are hiring AI engineers. I imagine these companies want to "leverage AI" or "add AI" to their products and services, don't have the know-how in-house and are bringing on AI engineers to help them do it. For example:

• Law Firms: Morrison & Foerster is hiring Senior AI Engineers to build their generative AI platform
• Banks: JPMorgan and Capital One both have many open AI Engineer roles, building agents with LangChain and LangGraph
• Defense Contractors: Booz Allen wants AI Engineers for national security applications
• Consulting Giants: Accenture and Salesforce have lots of open positions, probably to staff the same non-tech companies with their "AI initiatives"

A 100-year-old law firm is hiring AI engineers in house...

Keywords to get past the AI screening for your AI job

Looking at the skills the 31 job postings are asking for:

Skills breakdown

Programming Languages
─────────────────────

Python: 77%
TypeScript: 29%
JavaScript: 13%
Go: 13%
Java: 13%

Frameworks & Tools

──────────────────
LangChain: 29%

LangGraph: 23%

LlamaIndex: 19%

PyTorch: 16%

TensorFlow: 10%

Key Skills
──────────
Agents/Agentic: 71%
Evals: 61%
RAG: 29%
Fine-Tuning: 29%
MCP: 16%

• Python is table stakes. 77% of postings require it.
• TypeScript at 29%. That's the clear #2. Java, Go, and C++ are mentioned, but I assume it is for other parts of the codebase.
• LangChain is the framework of choice. LangChain and LangGraph together appear in 32% of postings. No other orchestration framework comes close. LlamaIndex shows up in 19% of postings, while AutoGen and CrewAI are barely mentioned.

Here's what's not emphasized in the job postings: PyTorch and TensorFlow. Traditional ML frameworks appear in only 10-16% of roles. This is another proof point that AI Engineer == Applied AI Engineer. The job isn't training models. It's building apps that call LLMs.

Agents, Agents, Agents

A whopping 71% of postings explicitly mention "agents" or "agentic workflows." It's clear that the industry has moved beyond simple chatbots, or at least they think they have.

RAG (Retrieval Augmented Generation) shows up in 29% of postings - and it's implied in almost every other one.

MCP (Model Context Protocol) is in 16% of postings. Hopefully it will go away soon and be replaced with CLI experts. Just kidding.

Evals

61% of job postings mention evaluations. Anyone who's shipped an LLM application knows that evals are everything. You start with evals.

Anyone that has built production LLM apps has learned this lesson the hard way. If you're interviewing for an AI Engineer role and they don't ask about evals, that's a red flag.

C.R.E.A.M. $-$$$$

Salary Ranges by Seniority
──────────────────────────
Entry $130K-$150K
Mid $150K-$250K
Senior $160K-$300K

The range is wide: $130K to $325K base salary. Startups pay higher than big non-tech companies.

Entry-level AI engineer roles barely exist. Only 3% of postings are for new grads. This is overwhelmingly a mid-to-senior role. Companies want people who've already shipped LLM applications in production.

So What?

"AI Engineers" are applied AI engineers, software engineers familiar with the tooling around LLM apps and how wrangle accuracy out of a stochastic compute stack.

Anyone who has made a career as a software engineer has had to upskill to keep up with new technologies throughout their careers. This is one more cycle in the same trend.

The post So You Want to Be an AI Engineer? appeared first on Heavybit.

On episode 7 of Data Renegades, CL Kao and Dori Wilson sit down with Bryan Bischof to explore his journey from pure mathematics to building real-world ML systems. They dig into recommender systems, surprising data bugs, and why truth-seeking should guide data teams. The conversation also covers modern data tools, visualization limits, and the realities of “self-serve” analytics.

The post Ep. #7, Truth-Seeking Data Systems with Bryan Bischof appeared first on Heavybit.

On episode 30 of Open Source Ready, Brian Douglas and John McBride sit down with Felipe Huici to explore Unikraft and the growing role of unikernels in modern cloud infrastructure. They discuss how unikernels differ from containers and traditional virtual machines, why millisecond startup times matter, and how Unikraft enables secure, scale-to-zero workloads. The conversation also touches on Kubernetes integration, open source governance, and where cloud isolation is headed next.

The post Ep. #30, Inside Unikraft and Unikernels with Felipe Huici appeared first on Heavybit.

On episode 6 of High Leverage, Joe Ruscio sits down with Carl Lerche, Principal Engineer at AWS and creator of Tokio. Carl shares his journey from Ruby and Rails into Rust, and explains why memory safety, fearless concurrency, and async runtimes matter for modern infrastructure. The conversation dives deep into the origins of Tokio, lessons from building foundational open source software, and how Rust’s guarantees are shaping the future of systems engineering.

Want 10% off your ticket to TokioConf? Use the discount code: Heavybit10

The post Ep. #6, Async Runtime For Rust with Carl Lerche of Tokio appeared first on Heavybit.

What Is an “AI Product Manager” (And Why Are They in Demand)?

After what appeared to be a shocking drop in product manager job listings starting in 2023, openings for product titles apparently increased 42% in 2025. Why is product management making a comeback?

Could it be that after a few years of teams using AI codegen tools to churn out large quantities of code faster, they’re re-learning important lessons about how the best products address customer pain points and provide a user experience that solves customer problems (and ideally delights them in the process) rather than “just doing things”?

Meanwhile, an emerging job title is the “product engineer,” which reportedly saw growth up to 400+ listings and counting for the previous year. Product engineers are expected to combine engineering analysis with product design,

We checked in with development veterans across the Heavybit community to dig into why product management is seemingly making a comeback in the age of AI, and what it means to be a good AI product manager or product engineer.

More Resources on AI Products & Startups

• Video: The GitHub Copilot Launch Story with Alyss Noland
• Article: Driving Productivity from AI Tools for Eng Teams with Laura Tacho
• Article: The Future of Software Documentation in the Age of AI with Heidi Waterhouse

Amit Eyal Govrin: The Future of AI PMs Is Blurry

Amit Eyal Govrin is the co-founder of Kubiya and a veteran of both the DevOps ecosystem and developer-focused go-to-market. He has previously served tours of duty at AWS, Cloudyn, and Panaya.

In the Age of AI, Context May Matter More Than Your Job Title

Govrin suggests that in the grand scheme of things, as more software products are built by AI tools, the skill that may come to matter most is not coding in a particular language, but rather, being able to utilize AI models properly. “When it's all said and done, there will only be one skill that matters: Context engineering. We know that LLMs are statistically based: Garbage in, garbage out. So if you have poor context, you’ll ship poor code, and you're going to have poor code quality.”

“We've worked with organizations that claim to be ‘AI native’ and sure enough, they wanted us to review and fix their broken code. Before, they were moving slower and were shipping X amount of code. Now, they have 10x or 20x more code, but it’s bad code.” The founder suggests that not being able to contextualize prompts for AI tools will simply lead to a bigger mess and a larger productivity drag as teams spend more time trying to fix garbage code outputs.

Amit Govrin discusses the need for engineering teams to evolve with AI on TheCube. Image courtesy SiliconAngle

Reflecting on what makes a “good” AI product manager vs. a “good product engineer,” the founder suggests that the semantics of job titles may not even matter in a few years. “I think the main cutoff is not going to be: ‘Did you come from the front end or back end or product?’ It's going to be your ability to be a context engineer and know how to instruct AI and delegate to it.”

“I actually think that a lot of the lower-level engineering roles, the juniors, are going to go away entirely if they don't know how to adapt across the stack. I think product managers who know how to prompt with context and know what they're asking for will become unbelievable engineers and replace entire engineering teams.”

Brian Douglas: Why AI Might Turn Us All into Product Managers

Brian Douglas is a veteran software engineer and developer advocate who has served tours of duty at Netlify, GitHub, and CNCF. He is also a longtime open-source software enthusiast, having previously founded OpenSauced and also hosts the Open Source Ready podcast.

Is Prompting An AI Like Writing a Product Spec?

Douglas reflects on how using AI every day has prompted him to rethink his own role, and how AI may be quietly removing the divide between being an engineer versus being a product manager. “As a former founder, I sometimes feel I'm ‘working myself’ into product management even when doing day-to-day engineering.”

The engineering veteran suggests that while using AI, building anything requires more product specs up front to properly guide models in the right direction. “I do way more planning and specs and GitHub issues. There’s more talking about the ideas, and getting the ideas on paper, because I know the AI will do so much better if you have a plan and you have a really well scoped-out task. But the AI would also do a better job if you had a design.”

For example, as front-end engineering becomes more AI driven, Douglas suggests that front-end projects may be driven more by AI-focused technical professionals who can direct AI tools with basic design sensibilities. “So there’s actually also design thinking. Where the boxes go, and what happens above the fold. All that stuff. For our Mission Control product, the front end was basically vibe coded by our CTO (who's not a designer).”

“Another engineer basically cleaned it up and made it mobile friendly because that wasn’t even considered at first. (Most folks who do front-end would start mobile-first, with ‘things in the box’ being the things we can use. And the desktop version only pulls from what the mobile version can actually give you.)”

Brian Douglas discusses open-source software strategy. Image courtesy Fintech DevCon.

AI Prompts: A Forcing Function for Thinking Like a PM?

Douglas suggests that in some cases, AI tooling may be reversing the typical sensibilities of building front-end products. “But when you do the inverse, you might just tell your AI to make something work. ‘Put stuff on the page and then we'll figure it out later.’ Any front-end engineer would tell you: ‘No, don’t do that. We're going to end up with some weird tech debt. Let's think about this stuff first.’”

The veteran dev notes that relying on AI tooling can feel like a forcing function that leads builders to consider foundational product considerations, including performance and security. “I’ve used some of the Chrome-based MCP tools, for instance. When you reach the Chrome console you hit the recorder. It'll tell you your largest contentful paint (LCP), your First Contentful Paint (FCP)...all stuff that you're supposed to be doing anyway.”

“Usually when you need to ship something, you don't really think about it because you're just trying to get it out the door. But now, this is becoming more of a process in which we’ll ask: ‘Before we go live, did we run it through the Chrome MCP? Did we get things like the page load speeds to see if we have any regressions?’”

Douglas compares shipping products using AI tools to GitHub Actions’ intended purpose of providing CI/CD for projects. “Before you hit push, you could actually have the stuff all run. So we're now seeing a shift where you’ll start thinking of things you wouldn’t have before: Security, speed, and performance. You can think about that before you push. Now after your agents are done, they’re introducing themselves into quality assurance and checking.”

Alexandr Kurilin: Engineers Should Already Have Product DNA

Alexandr Kurilin is a longtime software engineer, investor, and serial startup founder. He started his career as an engineer at Microsoft, then went on to co-found startups such as Freckle (acquired by Renaissance), ConceptualHQ, and now DoubleDusk.

Product Manager Thinking Has Always Been a Best Practice for Builders

Kurilin suggests that engineers with a product focus have always been important. “This has been a motif in my hiring process for a long time. I want very ‘product-y’ engineers. This might be a hot take, but let’s just say...there are firms that have a lot fewer PMs than you would expect. And that trend is only going to continue with engineering AI augmentation and the merging of previously separate disciplines.”

The founder suggests that regardless of any changes AI may be making to engineering, the best engineers tend to care more about the big picture. “For me, personally, I would never hire an engineer who simply doesn’t care about how the user feels about what they’re doing, how it impacts them, where the modules they implement will be used, and the context of it all. Why? You have to care about the layer next door, be that other engineers using your interfaces or the user.”

“It would be valuable for you, as a back-end engineer, to understand not just what the front-end folks are doing, but why they're doing it. And if you understand really well what use case, what job-to-be-done they're trying to solve, you can work together much better on actually solving what the user ultimately needs instead of just being given a spec and blindly executing. As an engineer, you're not just a coder. You're supposed to be a kind of psychologist, thinking about humans whose problems you're trying to solve.”

I would never hire an engineer who doesn’t care about how the user feels about what they’re doing. You have to care about the layer next door.” -Alexandr Kurilin, Founder/Double Dusk

How AI Is Changing Engineering and Product-Based Thinking

The founder notes that AI-driven change for software development may be the same change we’ve seen for years. “It’s about moving up the abstraction chain, which is just what computing has been doing for the last century. We’re just trying to give more and more leverage to software developers to worry less about implementation details and go higher up the value chain to get closer to the human problems they're trying to solve.”

“We’re at the beginning of the next evolutionary step: We don't care about the implementation details of individual code modules as long as we can verify that these black boxes are doing what we expect them to. If the coding agent can prove through tests and type systems that the module will behave according to spec, we don’t actually need to peer inside. If all of our validations pass, and the agent can keep maintaining that isolated blob of logic in perpetuity, why worry about the internals?

However, Kurilin notes with interest the rise of the “AI engineer” job title, and what it might mean in the age of LLM-powered products. “On some level, like with a lot of things in Silicon Valley, people are rebranding something already well-established. At one point, and at some companies, ‘AI engineer’ meant an ML and data scientist who moved into more of a product engineering role.”

“I actually don't think that's true anymore. I think for some companies that just means a ‘product engineer,’ who mastered how to wrangle LLMs into the product to add useful AI functionality for the end-user. It’s now all about iterating on the optimal prompts, curating evals, context management, tool calling, model selection, caching, cost management and much more.”

The post What Makes a “Good” AI Product Manager? appeared first on Heavybit.

On episode 51 of Generationship, Rachel Chalmers sits down with Soya Park of Akify to explore why an organization’s most valuable knowledge rarely makes it into formal systems. They discuss how leaders make excellent decisions every day without realizing it, and how AI can surface, reinforce, and scale those best practices without disrupting existing workflows.

The post Ep. #51, AI for Best Practices with Soya Park appeared first on Heavybit.

On episode 29 of Open Source Ready, Brian Douglas and John McBride sit down with Cameron Foxly to talk about ASCII Motion, an open source tool for creating animated ASCII art. Cameron shares how a brand design challenge at GitHub led to his first open source project and how AI tools helped him move from idea to production quickly. The conversation explores developer delight, terminal design, and why taste still matters in an AI-powered world.

The post Ep. #29, Building ASCII Motion with Cameron Foxly appeared first on Heavybit.

What Lies Ahead for the Year

2025 was a year of great expectations for AI, and arguably results that didn't quite meet expectations. Will 2026 be the year that AI returns value on all those investments? Will it be the year agents replace software developers? Heavybit’s community of founders, operators, and investors have shared their predictions for 2026: AI, software, and potentially a few slightly less serious subjects.

Interested in joining our community of technical founders and early-stage investors? Join us at an upcoming community event near you.

Corey Quinn, Co-Founder / Duckbill

Dropping Soon: The Other Shoe

“In 2026, there’s going to have to be some kind of reckoning in tying AI spend to business value that comes out of it. Because AI is only gonna suspend disbelief for so long before the bill comes due. So when the music stops, you’re probably gonna want to make sure you have a seat.”

Philip James, Engineering Manager/Freed

AI Models Go In-House Sooner Than We Think

“I have a longstanding bet on where AI is going: I think we’re going to see more companies start making small models available on their own hardware. I think it’s going to become more important for companies to own their own inference and models and start bringing that in-house instead of relying on the ever-growing foundation models. This might be a contrarian opinion, but I think [foundation models] are going to hit a ceiling just on hardware availability sooner than anybody thinks. So I think it’s going to get more important to bring that knowledge in-house.”

Amir Zohrenejad, Partner/Heavybit

2026: The Year of RL?

“I think reinforcement learning (RL) is going to be big. Working with AI is going to be more about working with post-training learning.”

Alexander Jung, Co-Founder/Unikraft

No End in Sight: More AI Deployments, Workloads, and Hardware

“We’re seeing really massive deployments these days. There’s lots of people who are generating content continuously: Workloads, things that are untrusted, and completely ephemeral. That’s going to continue to grow exponentially. We’re also seeing a lot of hardware being built out, which is an indication that [AI] is only going to become more abundant.”

Jeremy Edberg, CEO/DBOS

AI to Amplify, not Replace Engineers

“Last year, we heard the prediction that ‘all junior engineers are going to be fired and replaced with AI coders.’ That may be starting to come true for some people, but others are realizing the folly of that, and they’re doing the opposite. They’re actually hiring engineers and teaching them how to leverage AI to be better. My biggest concern was that we’d get rid of all the junior engineers and not have any senior engineers in a few years, but I’m hearing that bigger companies are actually hiring juniors out of college. I think in the next year, the AI coders are going to get better. The tools are going to get better. And we’re going to learn how to use them as tools to enhance what we do, not replace what we do.”

Dana Oshiro, Partner/Heavybit

Where Should Tech Investors Be Looking This Year?

“In terms of categories, everyone seems to be investing in inference, and as a result, there’s going to be a lot of new startups that we’re going to see on the hardware and edge side, [including] custom chips and advanced architecture.”

Chad Metcalf, CEO/Continue

AI: Less About Manual Intervention, More About Being Continuous

“We’ll see doubling down on actual AI use cases that yield some sort of value and [connected] to some sort of ROI. In code tools, that looks like continuous AI. If code goes so much faster, everything after it has to go faster too, and that means it all has to be continuous and a developer can’t intervene every single time there’s attention required.”

Priyanka Sharma, Investor and Startup Advisor

Will AI Transform Front-End Development First?

“My prediction is based on my recent experience building out the Web presence and positioning for my new company. Even though it wasn’t my job, I suddenly got into building it out myself. Initially, we were going to have a very traditional Wordpress implementation (and Wordpress is awesome, by the way!), but one thing led to another, and I was in Cursor building the site out. And it has been such a fun process. And by chance, I saw a conversation about this: Cursor released a post about how they have gone CMS-free. Sanity responded with what I thought was a very thoughtful and smart response. It’s the conclusion I came to myself in building this site: We need a CMS of some kind eventually. So I think the space around front-end development, around static websites...things that are not super-complex when it comes to functionality and how mission-critical they are...that kind of development is going to change dramatically in my opinion.”

Victoria Melnikova, Head of New Business/Evil Martians

AI’s Last Mile Problem...Solved?

“I think AI is going to get more intricate. I’m really looking forward to complete AI pipelines. Right now, it feels like we can get 80% of the way there, but we have to [work manually] to get to 100%. I’m really hopeful that we can resolve this in 2026, and go from scratch to a 100% production-ready state [with AI tools.]”

Simon Wistow, Co-Founder/Fastly

AI Will Transform Our Lives, But Not Completely

“I think two things are going to start to happen. One, I think we’re going to start to see an interaction between the AI companies and e-commerce companies in the same way we’ve seen between the publishers and the AI companies. We’ll start to see more agentic commerce and the e-commerce space really starting to have to reckon with how buying and zero-click transactions are done. The other thing that might happen is we’re going to start realizing that AI is going to end up more like the Web or the iPhone, in that they have completely transformed our lives and are part of every part of our lives, but...our lives are not completely unrecognizable. I think that’s how AI is going to end up. It’s going to infuse into everything, from kids’ toys to coding to commerce to whatever. But in some ways, our lives won’t be unrecognizable, they’re just going to be enhanced.”

Brian Douglas, DX/Continue

AI? You’ll Need Infrastructure for That

“My prediction for the year is that we’ll see this AI infrastructure wave really take off. Now, we’re going to need all the same tools we built for the last 15-20 years. So [for example], embedded Linux: Think of that for secure and AI runtimes. Things that are now CPU-bound now have to be GPU-bound. I fully anticipate that’s the next thing we should be paying attention to.”

Agree with these predictions? Disagree? Join us at our community event and let us know.

The post AI Predictions for 2026 appeared first on Heavybit.

On episode 88 of o11ycast, Ken Rimple and Jessica Kerr speak with Charity Majors about the shifting role of observability in modern software development. From AI-assisted instrumentation to reducing developer cognitive load, the episode examines how teams can move from reactive monitoring to continuous learning. It’s a deep dive into observability as a sense-making practice, not just a tooling problem.

The post Ep. #88, Metrics Are Good, Actually with Charity Majors appeared first on Heavybit.

2025 was the year AI stopped being optional. Coding assistants grew up, agents took over roadmaps, and the biggest players made bets measured in the tens of billions.

Below is our month-by-month recap of the most important headlines from DevToolsDigest... context for where we’ve been and clues for where 2026 may take us.

January

• Microsoft, NVIDIA, and OpenAI announce $500B Stargate Project
• DeepSeek sends tech stocks tumbling
• Introducing AX: why agent experience matters

February

• API economy trends for 2025
• How AI is changing developers (and the code they write)
• Clay achieves unicorn status with $57.5M raise

March

• Why Google made a $32B bet on Wiz
• HashiCorp officially joins the IBM family
• What the heck is MCP and why is everybody talking about it?

April

• 15 milestones that shaped Microsfot's vision for AI
• Shopify CEO manifesto says AI now mandatory for all employees
• 12-Factor Agents: principles for building reliable LLM applications

May

• Apple's famed design leader Jony Ive joins OpenAI
• Emerging developer patterns for the AI era
• Redis becomes available under the AGPLv3 open source license

June

• MCP seems viral
• The current state of MCP
• The 2025 InfraRed 100 Report

July

• Figma files IPO: mentions AI 150+ times
• Windsurf’s CEO goes to Google; OpenAI’s acquisition falls apart
• Cursor snaps up Koala in challenge to GitHub Copilot

August

• Cohere raises $500M to challenge OpenAI
• 95% of generative AI pilots at companies are failing
• Databricks eyes over $100B valuation with AI growth plans

September

• Introducing the MCP Registry
• OpenAI acquires Statsig for $1.1B
• Snowflake launches Open Semantic Interchange Initiative

October

• DORA report: state of AI-assisted software development
• Fivetran in talks with dbt Labs
• AI coding tools market update

November

• The state of cloud-native computing in 2025
• Platform for running AI models Replicate joins Cloudflare
• Forget AI regulation, data is the real wild west

December

• Linux Foundation announces the Agentic AI Foundation
• Sourcegraph and Amp become independent companies
• Anthropic acquires Bun as Claude Code reaches $1B milestone

Have 2025 reflections or 2026 predictions to share? We're always taking submissions at devtoolsdigest@heavybit.com. If you want the latest dev tools and enterprise software news delivered straight to your inbox, be sure to subscribe.

The post 2025: DevTool Industry Year in Review appeared first on Heavybit.

On episode 6 of Data Renegades, CL Kao and Dori Wilson speak with Roger Magoulas about the real bottlenecks holding data organizations back. From the origins of “big data” to today’s explosion of tools and pipelines, the conversation focuses on why understanding, semantics, and communication matter more than ever. The episode is a call to shift from constant firefighting toward curiosity-driven insight.

The post Ep. #6, From Big Data to Curiosity-Driven Insight with Roger Magoulas appeared first on Heavybit.

On episode 5 of Data Renegades, CL Kao and Dori Wilson sit down with Benn Stancil to explore how data tools evolve, and sometimes lose their identity. Benn shares lessons from building Mode, the risks of drifting from an opinionated product vision, and why most companies struggle to turn data into meaningful decisions. The conversation also dives into the future of BI in a world increasingly shaped by AI and unstructured data.

The post Ep. #5, The Identity Crisis of BI with Benn Stancil appeared first on Heavybit.

On episode 48 of The Kubelist Podcast, Marc Campbell and Benjie De Groot sit down with Justin Cappos, professor at NYU and a pioneer in software supply chain security. They explore the origins of modern package manager security, the real-world limits of SBOMs, and why systems should be designed assuming compromise. The conversation spans CNCF governance, in-toto, TUF, Git security, and the emerging role of AI in securing software.

The post Ep. #48, Unpacking Software Supply Chain Security with Justin Cappos appeared first on Heavybit.

On episode 87 of o11ycast, Ken Rimple and Jessica Kerr sit down with Lada Kesseler to explore how experienced engineers can work effectively with AI coding assistants. They discuss why AI feels like a fast, noisy black box, and how patterns like semantic zooming, feedback loops, testing, and observability can help developers stay in control. This episode is a deep dive into using AI without sacrificing clarity, quality, or trust.

The post Ep. #87, Augmented Coding Patterns with Lada Kesseler appeared first on Heavybit.

On episode 28 of Open Source Ready, Brian Douglas and John McBride reflect on the biggest themes that shaped open source and AI in 2025. From sustainability and security to MCPs, agents, and infrastructure, they revisit key conversations with guests and unpack how the industry evolved over the year. The episode closes with bold predictions for what 2026 may bring for developers, maintainers, and open source communities.

The post Ep. #28, 2025: Year In Review appeared first on Heavybit.

In episode 50 of Generationship, Rachel Chalmers sits down with Joshua McKenty to unpack how AI-driven scams, deepfakes, and identity fraud are already reshaping our digital lives. From scam factories and nation-state actors to broken trust infrastructure, Josh explains why authenticity is the core problem, and what it will take to fix it. The conversation spans cybersecurity, public policy, and the future of human trust in an AI-saturated world.

The post Ep. #50, Defense Against Deepfakes with Joshua McKenty appeared first on Heavybit.

On episode 27 of Open Source Ready, Brian Douglas and John McBride speak with Adam Hevenor, creator of Vibecheck. Together they explore the fundamentals of evals, how teams can run structured experiments across model variations, and why cost-efficient design matters more than ever. Adam also offers grounded perspectives on MCP adoption, Claude Skills, and the economics shaping today’s AI tooling ecosystem.

The post Ep. #27, Rethinking AI Evals with Adam Hevenor appeared first on Heavybit.

Making Semantic Text Management Lightweight

Much has been written about the ability of large language models (LLMs) to spin up large amounts of text, and still more is being written about how an overreliance on AI-generated text might do more harm than good. However, natural language processing (NLP) is a core part of the functionality of LLMs.

AI models require strong semantic text management capabilities to clearly understand natural-language prompts and to return accurate, intelligible responses. While training massive models can be compute intensive, it’s also possible to build lightweight NLP tools optimized to run on CPU, like WordLlama. The project’s creator, physics researcher-turned-data-scientist and engineer Lee Miller explains.

WordLlama is a lightweight NLP toolkit designed to run on CPU compute.

Tackling Token Economics and Training Costs With Embedding

Miller explains that the project came about from a few observations: One, that embedding packages for word embeddings for models with moderately large, million-word vocabularies can add up to a huge data footprint; and two, that LLMs use much smaller token-level vocabularies but are trained on trillions of tokens, forming strong semantic representations, useful for many NLP tasks.

Miller suggests his priorities were about making a project that was both as lightweight and as performant as possible. “I wanted to make sure the project didn’t have a lot of dependencies, such as requiring users to install PyTorch. Those can be pretty big packages that add a lot of compute overhead, especially if you’re considering any form of attention.”

“So I had to consider: How simple can we make the operations of doing a semantic sorting technique? In other words, can we call a sort operation on a list of strings but do it semantically (by word meaning) rather than alphabetically?” The creator explains that his goal was to avoid deep learning interfaces with high overhead altogether, converting everything into minimal formats like NumPy and Safetensors instead.

Getting NLP Projects to Work in Low-Compute Environments

Miller suggests that at least for the time being, there’s a stark difference between LLM projects built for CPU compute and those built for GPU. “If you start looking into GPU compute for your projects, it doesn't make sense to do things that are this lightweight anymore. It's a completely different area of computing altogether.”

The creator explains that in some ways, his goal was more about optimizing for low-compute operations rather than top-of-the-line performance. “Quite honestly, the model I’m using isn’t necessarily the best way to do what this library does, but it can do what it was designed to do without all the overhead that a lot of the bigger models require.”

Lightweight NLP and Semantic Text Management Use Cases

For example, WordLlama can perform fuzzy de-duplication, finding and removing two text strings that may be similar but differ in unanticipated ways, such as different versions of the same customer or lead across a startup’s sales records. “You could do a great job with a cross encoder or another transformer model–but there are other use cases that don’t need all that horsepower.”

The creator notes that users apply the project with RAG systems to process small text snippets for list sorting, or other data processing-related tasks, such as fuzzy deduplication, text clustering, or semantic text splitting by way of fuzzy matching, such as for sorting semantically similar custom survey responses.

“The idea is to add that little bit of a semantic layer to all the different Python operations that you might want to run in order to tease things out based on their semantic meaning, rather than the physical structure of the text string.”

Having really lightweight tools can be important in a lot of applications where all you really need is to do something semantically relevant to your LLM interaction. Ultimately, it’s another tool in your toolbox for helping you get good outputs from your LLM.” Lee Miller, Creator/WordLlama

The Future Value of NLP and Lightweight Compute

“One thing that I've found working with LLMs is that the more that you start doing with NLP, the more that you need a variety of tooling to help you utilize it effectively,” Miller notes. “Anything from ways to evaluate a model’s output to ways to provide the right context (or what I guess is now called context engineering)...these things determine what should go into the model on your next inference turn.”

The creator suggests that working with NLP use cases has also underscored the tradeoffs between relying on lightweight vs. heavyweight compute. “You might want to make different choices based on your application performance needs. LLM inference isn’t the fastest thing in the world, and you don’t want additional latency where you don’t necessarily need it.”

“It can come down to whether someone is using an LLM from their laptop to do context engineering, or some evaluation on the output. If they don’t have a GPU on their computer, do they really want to have to go load up something externally on GPU hardware just to do some basic semantic work on their inputs or outputs?”

“Having really lightweight tools can be important in a lot of applications where all you really need is to do something semantically relevant to your LLM interaction. Ultimately, it’s another tool in your toolbox for helping you get good outputs from your LLM. Otherwise, there’s a lot of opportunities in terms of efficiency and performance, where you don't necessarily want to throw the whole kitchen sink at the LLM, and can easily weed out things that might not be relevant for the context. And that's really where the context engineering piece comes in.”

How Founders Should Think About NLP and Future Developments

Miller cautions founders to beware the hidden costs of going straight to top-of-the-line LLMs. “There can be a tendency to go with the biggest model you can get, with the highest accuracy you can find, to try to head straight to the very top end of things.”

“But along the way, there's a lot of decisions you can make around prioritizing the information you put into your models that can really benefit the accuracy and improve your outputs as well. I’d advise teams to look into even some basic context engineering, or semantic splitting to filter out things you aren't going to want to put in your context.”

“These are what I might call ‘NLP-lite’ tasks. Depending on your use case, you don't necessarily need anything too heavy-duty for a lot of those in terms of models and compute. But looking into a package like this can really help with improving inputs to your LLM. So if you're working on building with AI, it’s a good idea to keep in mind that token input cost is only part of the picture. You can use context engineering to both reduce input costs and improve outputs by targeting more relevant information."

The post Combining Context Engineering With Lightweight Compute appeared first on Heavybit.

On episode 49 of Generationship, Rachel Chalmers sits down with Denise Koessler Gosnell and Kathryn Erickson to unpack their new book Tech Confidential. Together they explore burnout, leadership, motivation, and the surprising ways our emotional wiring shapes tech culture. It’s a candid, deeply human conversation about transforming toxic systems into something resilient, sustainable, and even joyful.

The post Ep. #49, Dumpster Phoenix with Denise Koessler Gosnell and Kathryn Erickson appeared first on Heavybit.

In episode 4 of Data Renegades, CL Kao and Dori Wilson sit down with Micah Wylde. They trace his journey from building fraud detection at Sift Science to architecting massive real-time systems at Lyft and ultimately founding Arroyo. In this conversation, Micah breaks down the real complexity of streaming systems, why schema evolution is still the hardest challenge in data, and where the industry might move over the next five years.

The post Ep. #4, Streaming Made Practical with Micah Wylde appeared first on Heavybit.

How Better Data Formatting Affects Dollars and Cents in AI

Text-based large language models (LLMs) process whatever text a user inputs as a prompt in token-based intervals. Each token is effectively a text-based unit of measurement (not necessarily an individual word) for how much context a model must process in a prompt to generate a response.

“What’s the best way to debug this recurring 500 error when customers place orders in my e-commerce system that includes inventory management, order processing, and payment processing apps?” has significantly more tokens to process than “What’s a good TV show on Thursdays?”

Longer queries with more text mean higher risk of context rot, a reduction in response quality as the context window gets larger and larger. They also mean incurring higher token costs to process prompts that are longer, potentially not efficiently optimized, or request complex multi-step tasks.

Vendors and researchers continue to work on making AI model operations cheaper by streamlining hardware-based compute usage for processes like training, fine-tuning, and inference. But there are real gains to be made on the prompting side as well. The popular open-source TOON project encodes the JSON data model in a more-compact text format that’s readable for humans and LLMs and efficiently conserves token budgets.

However, there’s more to building good AI applications than pinching pennies on tokens, as creator and self-taught engineer Johann Schopplich explains.

TOON is a data serialization format that’s human- and LLM-readable. Image courtesy Johann Schopplich and GitHub

Making JSON Human-Readable (and LLM-Readable)

The TOON project came from a need to conserve tokens for an extremely specific, text-heavy use case that required interfacing with human editors as well as with LLMs. Specifically, Schopplich’s team at the time needed to manage a daily stream of news updates for an economic and political podcast.

The creator explains, “CSV tables are pretty efficient and LLMs are trained on them, so in terms of efficiency, you can’t really top them. But we had a very specific use case for a media company that needed to read different RSS feeds from around the world. We needed to give the editorial team the scraped content from the feeds, and while we always had the idea of the team prompting with AI, we wanted to inject our aggregated data into the prompt first.”

“That’s why raw JSON wasn’t really an option. It wasn’t readable for newsroom editors, and it wasn’t really token-efficient, because JSON often repeats the same keys over and over. CSV tables are compact but flat-only and don’t handle nested data. YAML is human-friendly, but its flexibility and looser structure make it harder to validate and less predictable for LLM prompts.”

“So, keeping JSON’s data model, but designing a more-compact, deterministic representation on top of it, made the most sense.” TOON emerged as a way to both conserve tokens while also being workable for a non-technical editorial team.

In addition to saving tokens and being readable for a wider audience, TOON’s headers act as a kind of guardrail for LLM processing. “I think that’s another key feature: It’s token-efficient for tabular data. Like CSV, TOON declares field names in a header row. But it also adds an explicit length header for the number of rows.”

Providing both a field list and a numbered length header gives LLMs the ability to understand how many columns and rows are in a table, which would let a team quickly and accurately gauge how many news stories came in from specific regions, for example. The creator notes that in benchmarks across 11 datasets and 4 LLMs, TOON achieved about 73.9% accuracy versus JSON’s 69.7%, while using roughly 40% fewer tokens.

Extending JSON-Based Serialization Beyond Prompting

“The idea was not necessarily to create something completely new that LLMs don't understand. That's why it borrows syntax from YAML and the CSV-esque header structure. It feels familiar for LLMs and often performs best when you give [your LLM] a short primer.” The creator notes that there are already official and community implementations in TypeScript/JavaScript, Python, Rust, Go, Java, .NET, Dart, and more.

“If people use the encoder for the [programming] language they are most familiar with, then they can provide an encoder for non-technical people, so to speak. The structure of TOON is pretty readable; once a non-technical viewer glances at it, they can usually understand the structure.”

Schopplich notes that the project is apparently providing gains in unexpected ways. “Some people have reached out to me about how they store TOON instead of JSON in their vector database, saving space (and later token cost) when they re-inject that data into LLMs. It’s not what the project was intended for, but why not? I’ve also seen people replacing their JSON encoding function to just TOON encoding. That was another idea behind the project: To be a drop-in replacement for the JSON you’d otherwise send to LLMs.”

“If you have data in your storage and you retrieve it, and send this data encoded to your LLM, you don't have to think about which parts of the data should be in YAML, and which should be in tabular CSV format. You can just use the TOON encoder and you are pretty much good to go. That's also one of the main ideas for the project: To keep the holistic structure that JSON provides, just in a different format.”

Token reduction is one thing, but accuracy of data retrieval is another.” -Johann Schopplich, Creator/TOON

How AI Startup Founders Should Think About Token Economics

In terms of understanding token economics for startups, Schopplich concedes that founder life isn’t easy. Yes, cash-strapped startup founders, concerned about dwindling runway, may fixate on their product’s token economics, but it could be at the overall cost of product quality and overall user experience.

“Token reduction is one thing, but accuracy of data retrieval is another. Or, maybe we can reduce a piece of content into very small tokens, but the LLM takes a lot of time to process it [introducing unexpected latency]. One of the goals of the TOON project was to not compromise on accuracy. The accuracy per 1,000 tokens is actually on par with JSON, if not better because we have the length header as a guardrail.”

“So by all means, benchmark end-to-end. But not just the tokens you save. Also measure Time to First Tokens (TTFT): How long it takes for the LLM to start responding at all.” In other words, cost-conscious founders who overindex on minimizing token costs to save budget may end up with a product that is so laggy, or so inaccurate, or both, that no customer even wants it, regardless.

The post How Data Serialization Improves AI Token Economics appeared first on Heavybit.

In episode 48 of Generationship, Rachel Chalmers speaks with Sarah Novotny. They dig into why AI models fall short of true creativity, how the tech industry drifted into extractive incentives, and what real security and accountability might look like at scale. Sarah highlights lessons from Kubernetes, open source ecosystems, and political science to propose a more trustworthy technological future.

The post Ep. #48, Trusting AI with Sarah Novotny appeared first on Heavybit.

On episode 3 of Data Renegades, CL Kao and Dori Wilson sit down with Maxime Beauchemin. They explore the origins of Airflow and Superset, the evolution of open source in the data ecosystem, and how today’s tooling reshapes the role of the data practitioner. Max also shares a forward-looking perspective on agentic workflows and how AI is accelerating everything from BI to pipeline development.

The post Ep. #3, Building Tools That Shape Data with Maxime Beauchemin appeared first on Heavybit.

In episode 2 of Data Renegades, CL Kao and Dori Wilson speak with Simon Willison. Together they dive into the origins of Datasette, the evolution of data journalism, and the surprising ways open source tools shape global reporting. Simon also explains how LLM-based agents will redefine data cleaning, enrichment, and analysis. A must-listen for anyone building or scaling data teams.

The post Ep. #2, Data Journalism Unleashed with Simon Willison appeared first on Heavybit.

In episode 26 of Open Source Ready, Brian Douglas and John McBride sit down with Quinn Slack. They explore how Sourcegraph built Amp, why unconstrained coding agents represent a major shift, and why AI economics make traditional SaaS pricing impossible. Quinn also breaks down the rise of open source models, the future of developer tooling, and why ads might be the key to making AI accessible to everyone.

The post Ep. #26, The Economics of AI Coding with Quinn Slack appeared first on Heavybit.

In this debut episode of Data Renegades, CL Kao and Dori Wilson dive into the unconventional paths that led them into developer tools, civic tech, and machine learning systems. CL recounts his experience building early open source infrastructure, mobilizing communities through data transparency projects, and now shaping how we design software around LLMs. This episode sets the tone for the show’s mission: celebrating builders who redefine the boundaries of what tools can do.

The post Ep. #1, Introducing Data Renegades appeared first on Heavybit.

In episode 25 of Open Source Ready, Brian and John sit down with Rachel-Lee Nabors. They explore how AI agents are reshaping the web, from the decline of traditional browsers to the rise of agentic experiences powered by small language models and MCPs. Rachel-Lee explains why advertising models are collapsing and why the next web may depend on direct payments and open source innovation.

The post Ep. #25, Death of the Web Browser with Rachel-Lee Nabors appeared first on Heavybit.

In episode 47 of Generationship, Rachel Chalmers chats with Dr. Milette Gillow and Sedinam Simpson, co-founders of The Tech Bros, about their mission to make tech more inclusive and inventive. They unpack lessons from their first accelerator cohort, debate the future of AI, and share what it takes to build confidence and community in an evolving industry.

The post Ep. #47, The Tech Bros with Milette Gillow and Sedinam Simpson appeared first on Heavybit.

Most developers agree that code will be increasingly written by agents. If that’s true, then familiarity with a specific programming language's ecosystem will stop being a criteria for hiring software developers.

I ran this idea past a few friends who have hired lots of software engineers, and they disagreed.

The theme that kept coming up: The clearest window into an engineer’s taste is the language they build in.

Every language carries its own ethos, a philosophy of how software should be written. A Python developer values readability. A Rust developer values discipline and efficiency.

These preferences aren’t trivial. They reveal how someone thinks. It’s the kind of craftsmanship they aspire to as a builder. The languages you’ve built in signal who you want to be as an engineer. Like the stickers on your laptop, they tell your story before you even open it to show a demo.

Beautiful Craftsmanship

Every programming language encodes a set of values. Spend enough time building in one, and those values shape your sense of taste. What you optimize for, what you tolerate, and what you find ugly.

When engineering managers talk about “culture fit,” they’re often circling this idea: Does this person’s taste align with how we build things here?

Enough Talk. Let's Look at Code

Two snippets that count the frequency of words in a string: one in Python, one in Rust:

from collections import Counter

def word_count(text):
words = text.lower().split()
return Counter(words)

It’s beautiful because it looks like pseudo-code. A non-programmer could read it and get the gist. It trusts the standard library instead of reinventing the wheel, confident that the built-ins are good enough.

use std::collections::HashMap;

fn word_count(text: &str) -> HashMap<String, usize> {
let mut counts = HashMap::new();
for word in text.to_lowercase().split_whitespace() {
*counts.entry(word.to_string()).or_insert(0) += 1;
}
counts
}

Every allocation is explicit. Resource usage is tightly controlled. Nothing is left to chance.

Sure, these are toy examples. Python and Rust were built for different kinds of work. In the craftsmanship analogy, it’s more like comparing wood to bronze than comparing two statues. But the point stands: each carries a worldview about how things should be built.

So What?

When a team lead looks at the languages a developer has built in, they’re not just gauging how productive that person will be. They’re trying to understand the tribe that person identifies with, and whether it matches the tribe the company is building.

That tribe says more than knowledge of a library, linter rule, or test framework ever could.

In a world where AI systems can translate ideas into code across any language, taste becomes the more important filter for choosing teammates.

Programming languages may fade into invisible layers between us and our agents. But their ethos will outlive them. They’ve been quietly training us all along. Not in how to write code, but in what we find beautiful.

The post Language as Proxy for Taste appeared first on Heavybit.

What Goes Into Multi-Agent Orchestration

AI agents are autonomous systems which take action independent of regular human intervention. AI startups have been exploring agentic functionality for some time by way of agentic workflows that utilize agents to sequentially execute steps in a pre-defined process.

However, to scale their AI programs, organizations are experimenting with agentic orchestration, the process of managing multiple agents as each one executes its tasks. As one founder puts it, having multiple agents under the hood may turn engineering into a multiplayer discipline that doesn’t just emphasize human-to-human collaboration, but also requires simultaneous human-in-the-loop oversight of AI agents for a variety of use cases.

We discussed the potential of multi-agentic orchestration with researcher Salah Alzu’bi, who currently works on the open-source orchestration project ROMA.

Researcher Salah Alzu’bi discusses multi-agent systems. Image courtesy Open AGI Summit.

Approaching Multi-Agent Architecture

Alzu’bi explains that the project arose from a need to provide a common framework for multi-agent communication. “For example, deep research is a common multi-agent use case. And one project might take the approach of having a search agent that grabs a few queries, then extracts information, then has a verifier tell it what it’s missing, and loops again, while another repo might have a different system entirely.”

Alzu’bi suggests the ROMA project was born in part from the need to benchmark which frameworks had better performance and scaled better. To do so, the team needed to choose how it would assign tasks to agents for execution, which include common strategies like:

Flat Decomposition

Operating agents at the same level of abstraction without any hierarchical separation. Flat decomposition systems are used in distributed sensor networks and swarm robotics as they benefit from greater flexibility and generally not having a single point of failure. However, they’ll often have higher overhead for coordination.

Graph-Based Decomposition

A system that represents tasks and dependencies as a graph, within which agents handle nodes or subgraphs. Graph-based is used in task-and-motion-planning systems for robots as they can support complex dependencies and are a good fit for distributed optimization. However, graph-based systems tend to require in-depth dependency tracking.

Plan-/Goal-Based Decomposition

This system focuses on agents achieving goals rather than pre-defined subtasks, breaking down high-level goals into plans. It’s used in goal-conditioned reinforcement learning as it can adapt well to changing environments, but can require significant planning overhead and strong reasoning capabilities.

Hierarchical Decomposition

A hierarchical system breaks down tasks into a clearly-defined, tree-like structure of subtasks. This system is used in areas like automated workflows and human-AI collaboration as it can scale better and offer improved reusability, though it can have issues with improper task specification if agents get confused about their precise role.

The team settled on a hierarchical decomposition, utilizing a framework specifically designed to avoid task specification issues by breaking subtasks into three clear categories: Think, write, or search. “The idea is to use a meta-agent framework with recursive recursive hierarchical decomposition, which is what we feel is the most natural, intuitive way to think of these multi-agent systems.”

Architecture Choices: Context Rot, Parallelization, Hierarchy

“We were mostly inspired by Professor Jürgen Schmidhuber’s paper on recursive decomposition for long-form generation. The idea is to recursively break down tasks, which is a very natural way to think about executing tasks in sequence.” The researcher offers an analogy to hierarchical delegation within companies: CEOs toss an important contract to VPs, who break down the project into subtasks for different teams, though the approach has other technical benefits.

Pioneering AI researcher Jürgen Schmidhuber at the 2017 AI for Good conference. Image courtesy ITU.

“Another big problem that hierarchical decomposition addresses is context rot, the degraded LLM performance we see when the context size gets very big. Context rot can cause agents to stop performing even simple tasks well.” While techniques like context compaction (which condenses older parts of an interaction to improve performance) have emerged, Alzu’bi suggests that hierarchical may be one of the clearest solutions to the problem.

“Having a hierarchical structure allows for the root task to break things down. And then, each of these little subtasks doesn't even need to know what's above: It has the parent orchestrating that stuff for them. So all [individual agents] need to do is basically just keep solving the task at hand.” The researcher suggests that the approach helps local agents solve problems without a great need for observability and without a significant risk of rot.

“In particular, parallelization can solve things really well. You don't need to wait for sequential processing of tasks. For example, if you ask a multi-agent system to map a city, you can break that down into different regions. You don’t need to wait for one neighborhood to finish before starting the next — each agent can map its own area in parallel. Then at the end, the parent agent looks at all the regional maps, makes sense of them, and produces a complete city map.”

Finally, the researcher explains his team’s deliberate choice to not treat every task equally. “Not many multi-agent systems seem to do this today, but you don't really need a state of the art model attacking all your tasks at any given moment. For example, you can use a very small model that performs simple tasks very well to save cost and time (because latency can also be a factor). It’s about customizing different models and agents to do particular things within an intuitive framework.”

How to Think About Multi-Agent for Startups vs. Enterprises

The researcher suggests that multi-agent is still new enough to offer a variety of opportunities, though an interesting area for upside is going beyond single-purpose AI products like coding assistants. “I think of what we’re working on as a framework to quickly prototype, or even ‘vibe prompt,’ if you want to call it that, a high-performance multi-agent system for a particular set of tasks that might include search or knowledge retrieval. Just spin it up, change a few prompts, and you have your agent.”

However, building multi-agent for enterprises can bring other challenges, including the need for on-prem security and compatibility with a variety of models. “We designed our project to let you use whatever models you want for whichever use case within a certain budget, which is something people care more about at the enterprise level.”

The researcher notes that some customers may be sensitive to other operational factors like token budgets and latency. “We built our project to let users fix their budget and latency to desired levels, which we think helps solve some of those problems for enterprises with a preference for on-prem security and flexibility around model choices.”

How do organizations know when they’re ready for multi-agent? Or alternatively, can they get by with a simpler, single-agent workflow solution? “Every organization will need to do the validation themselves and think about the trade-offs.”

"Organizations should have their own internal evaluation suite of benchmarks they run their models against.” -Salah Alzu’bi, AI Researcher/Sentient Foundation

For example, single-agent workflow setups can offer faster, simpler deployments, lower overhead for deployment and maintenance, and less operational risk. However, they will hit context limits on larger tasks faster, will require execution that remains rigid and serialized, and perform poorly as tasks grow in complexity.

Multi-agent workflow setups can be more complicated to deploy, demonstrate noticeably higher latency, and incur higher risks at launch due to the additional components. But they can also handle significantly more-complex challenges with longer context windows, and they can evolve more easily over time by swapping in new or upgraded models and agents.

Alzu’bi notes that most enterprises he works with currently seem to focus on an expected set of metrics. “You have metrics like latency, costs, and model performance in general, as well as user satisfaction. But organizations should have their own internal evaluation suite of benchmarks they run their models against.”

Security, Governance, and the Future

At minimum, before implementing a multi-agent framework, the researcher recommends at least a few common-sense security and governance measures. “You don’t want your LLMs having access to your API keys, your passwords, or any of that. A simple prompt injection can leak everything.”

“I would also recommend an internal safety alignment check. At minimum, come up with a list of the most common prompt injection or security threats that LLMs are known to suffer from and test against them. Obviously, this is an area that’s still under development and there are new threats every day, but a good rule of thumb is: Minimize exposure. Don’t give LLMs access to things you care too much about.”

“At this time, I would also recommend human oversight with a lot of security checks. There was an incident a little while ago with an AI app deleting a user’s production database. I wouldn't deploy multi-agent systems to high-stakes environments with access to people’s bank accounts yet...not until we fully understand what the risks are.”

The post How to Approach Multi-Agent Frameworks appeared first on Heavybit.

Developer tools have many challenges when it comes to creating content and leveraging it to drive growth: aligning content strategy with product and go-to-market efforts, maintaining a consistent output of high-quality content, and ensuring their content gets distributed on and off-site.

But one challenge that I don’t hear as many people talking about is the importance of refreshing technical content to keep it up-to-date. In fact, refreshes are one of the biggest opportunities for brands that have been publishing consistently, and they’re a great way to drive awareness without spending a ton of money on new content.

This was true in traditional SEO, but it’s even more true in optimizing content for LLMs (AEO or GEO), as the mainstream models seem to favor newer content. With developers increasingly relying on ChatGPT, Claude, and other AI tools to discover new tools and platforms, it’s getting even more important to keep your content fresh.

Here's the thing though: if you've been publishing solid technical content, you're not starting over. The gap between traditional SEO and what works for LLMs isn't massive. You just need some tactical adjustments to how you approach content updates. Let’s dive into it.

Developers increasingly report daily usage of AI tools. Image courtesy StackOverflow

Why Refreshing Content Matters

One of the easiest ways to improve an existing website is to improve the content that's already there. Because you don't have to start from scratch, you'll spend less time on each piece of content.

Here's what we've learned working on content strategy for developer tools at Draft.dev:

Content decay is real. Nearly 60% of blog posts experience declining traffic within 12-24 months of publication. Your best-performing content from 2-3 years ago is likely underperforming today. Version numbers change, discovery algorithms evolve, competitors publish newer content, and technical details become outdated.

Refreshes outperform new content. We consistently see traffic increases of 200-500% for refreshed content while requiring much less investment than new content creation. Plus, updated content is typically indexed faster by search engines and LLMs.

AI changes everything. Because LLMs tend to favor more recently published or updated content, anyone who’s been publishing for years has to consider refreshes a core part of their content strategy. In my view, this is especially important in developer content where out-of-date information erodes reader trust and leads to bugs and wasted time.

What's Changed With AI-Powered Search

Before I dive into specific tactics, you need to understand a little bit about what matters (and doesn't) seem to matter in this new world of AI-powered developer discovery.

Freshness of ContentAI models favor recently published or updated content. We recommend considering technical content refreshes at least once a year for this reason.

Content StructureFormatting is more important than ever. LLMs aren't as sophisticated at crawling and inferring meaning as mature search engine crawlers, so you should update content to be:

• Skimmable (high-level summaries or bullet points at the top)
• Structured (use tables and matrices for comparisons)
• Self-explaining sections (be aware of chunking)
• Conclusive (key takeaways or FAQs at the bottom)
• Well linked (internally and externally)

Most of these structural points are good for readers and SEO as well, so this is just a win all around.

Schema.org MetadataStructured data helps machines understand your content. Ensuring your website includes simple organization schema, along with FAQ and blog posting schemas, can help your content be better parsed by AI. This requires some technical implementation, but it's worth the effort.

External Brand MentionsIn a world where LLMs train on massive datasets from across the web, brand mentions seem to weigh more heavily than backlinks. This means content mentioning your brand positively across a variety of platforms is key to showing up more often in LLMs, and more practically, it means that you should probably be publishing multiple versions of your content across multiple platforms.

We’ve found that clients getting more mentions in LLMs are publishing on:

• Newsletters that are published online
• Personal, partner, and user blogs
• Dev-focused platforms like Hashnode and Dev.to
• Relevant Reddit comments (not just link drops)

This is a great opportunity for smaller brands to catch up with established players who may have stronger backlink profiles but less focus on external content today.

Comparisons and RoundupsMore than ever, the goal for developer tools should be to guide LLMs towards the category your tool fits into and the specific use cases where it's superior. As developers interact with chat tools and ask progressively more specific questions, you want to make sure your brand is present in those contexts.

This is where direct comparisons ("Us vs. Them") and roundups ("10 Best Tools for X") come in. When refreshing this type of content, it’s important that you feed LLMs the most accurate and up-to-date information so they treat it as authoritative on the topic.

Identifying What Needs Refreshing

With some of the factors above in mind, the next exercise you need to run is to determine which content needs to be refreshed so you can prioritize your efforts accordingly.

Traditionally, we’ve used our clients’ Google Analytics and Search Console data to identify pages with:

• Declining traffic over the past 6-12 months
• Lost rankings for valuable keywords
• High bounce rates or low time on page
• Existing backlinks and domain authority
• Previous strong performance

I think this method still makes sense from an SEO perspective, but it might not be quite as relevant for GEO as it’s still hard to track citations and traffic from LLMs.

So, at this point, you have a few options:

• Use a tool like Scrunch or Airops to identify content that’s not being cited in LLMs as much as you think it should
• Look for any pages on your site that highlight key features, compare your tool to others, or share implementation details and make sure they’re kept up-to-date annually
• Simply make a plan to regularly review and refresh 1-2 pieces of content per week on an ongoing basis

You might think that having a dedicated LLM analytics tool is the way to go, but keep in mind that these tools are still imperfect and making very rough estimates, especially around niche topics. So, depending on the type of content you produce, the volume of output, and your resources available, I think all of these options are viable.

Video content can offer many benefits and potentially help GEO. Image courtesy HubSpot

Best Practices for Refreshing Developer Content

Once you decide on your method for determining which pages deserve a refresh, here are the tactics we use to update older content:

1. Do a Technical Review and Update

This is the most obvious and important step for technical content as software and its best practices are constantly changing. Checking version numbers, re-running your tutorials, and having an engineer review older pieces is one of the best ways to ensure accuracy and relevance.

Some of the most important things to review include:

• Technical accuracy
• Code examples and screenshots
• Statistics and benchmarks
• CTAs and brand messaging
• Dead or broken links

2. Add Rich Media

I’ve seen video and unique visual content improve search engine rankings, and have heard (although there’s still debate on this) that rich media helps with LLM citation rates as well. Either way, having multiple formats for content (video, text, visual) helps the user experience, so if it’s a high-value piece of content, it might be worth investing in.

3. Revisit Formatting and Structured Markup

If you haven’t updated this piece since LLMs became mainstream, I’d recommend looking at the article’s formatting and Schema.org metadata to ensure it’s following best practices. Little things like having a table of contents or some “takeaways” at the end of the article can make a big difference and take very little effort to add.

4. Merge Redundant Content

Another problem we see clients with large content libraries run into is having multiple versions of the same content spread across their website and blog. Sometimes it’s redundant comparison pages or landing pages, and other times, they’ve written four blog posts about the same core topic. Taking a bit of time to merge redundant content and redirect the remaining URLs can help ensure crawlers get a “single source of truth” for each idea on your site, ensuring their information is accurate and consistent.

5. Remove and Redirect Irrelevant Content

There are times when the easiest way to clean up your content is to simply unpublish it. This happens when features are no longer relevant, you end support for certain technology, or the opinions expressed are no longer the ones you hold.

Removing irrelevant or inaccurate content ensures that LLMs are less likely to feed users inaccurate or out of date information. Just make sure you redirect any of the URLs so you don’t lose backlinks or cause an increase in 404 pages.

6. Update Published Dates and Re-Promote

Finally, update the “published date” on each piece and re-promote it across your social and syndication channels if appropriate. A lot of teams miss this, but you’re giving this content a new lease on life, so you might as well increase its chances of getting found by telling people about it.

Results You Can Expect

As of this writing, we’ve worked with clients to update a lot of content, but the results are still variable, leading me to believe that there’s more to learn here. Some consistent trends have emerged though:

Timeline for ResultsMost content should see improvements in search or LLM appearances within 2-4 weeks as search engines recrawl and reindex the updated content. I’ve seen many pieces receive 200-400% traffic and citation increases within the first quarter post-refresh, so again, it’s worth doing for your high-value content.

Comparison to New ContentNew content can take a couple months to mature and start driving traffic, but refreshes often show results in a couple weeks. This depends of course on the volume of LLM/search traffic to the piece, but it makes sense as this content gets the benefit of any existing authority, backlinks, and indexed URLs.

Getting Started With GEO Content Refreshes

If you've already invested in content marketing, you're ahead of the game. AEO/GEO isn't about reinventing the wheel, it's about optimizing your content strategy for the way developers are using AI tools and making decisions today.

So here's what you should do:

1. Keep creating high-quality, technically helpful content. The fundamentals haven't changed, you still need to produce content that actually helps developers solve problems.
2. Update your content regularly. Especially tutorials and documentation. Aim for at least annual refreshes of your best-performing content.
3. Optimize structure and metadata across your content. Make sure your posts are skimmable, well-structured, and include proper schema markup.
4. Build brand awareness across multiple platforms. Don't just refresh on your blog. Keep an eye on content syndicated to Dev.to, Medium, and other developer-focused platforms.
5. Share with niche newsletters and communities. Re-promote refreshed content and make sure your communities and any relevant newsletters know about the content.

Finally, if you ever have questions about SEO, GEO, or refreshing content in developer tools, reach out to me on Linkedin. I’m happy to help.

The post Refreshing Developer Content to Maximize AEO/GEO appeared first on Heavybit.

In episode 24 of Open Source Ready, Brian Douglas and John McBride sit down with Ivan Burazin, CEO of Daytona, to explore how his company is building runtime infrastructure for AI agents. Ivan shares how Daytona pivoted from developer environments to powering the next wave of autonomous AI systems, and what it takes to make agents fast, secure, and scalable. They also discuss open source licensing, enterprise adoption, and dopamine-driven development.

The post Ep. #24, Runtime for Agents with Ivan Burazin of Daytona appeared first on Heavybit.

In episode 46 of Generationship, Rachel Chalmers and Don Marti trace a thoughtful arc from the open source protests of the 1990s to today’s AI-driven world. They explore how large language models blur truth and plausibility, debate ethics in benchmarking and market-based definitions of intelligence, and mourn the loss of the web’s early democratizing promise. Don discusses sustainable open source ecosystems, AI tooling, and practical advice for new graduates navigating an AI-saturated job market.

The post Ep. #46, Canned Monkeys with Don Marti appeared first on Heavybit.

In episode 86 of o11ycast, Ken Rimple and Jessica Kerr sit down with Amarilis Campos from Nubank. They explore how Brazil’s largest digital bank uses observability to build resilience and autonomy at scale. Amarilis shares how Nubank’s culture of “acting like owners, not renters” and its journey from metrics and logs to tracing with Honeycomb helped the company grow to over 100 million customers.

The post Ep. #86, 12 Years and 100 Million Customers with Amarilis Campos of Nubank appeared first on Heavybit.

How is Developer Content Changing in the Age of AI?

The cliche “developers hate marketing” suggests they don’t care for traditional marketing channels, like TV commercials or email signup forms. (And many don’t.) But it’s not impossible to market to technical personas, either. Instead of sitting passively through a demo, devs prefer to actually try software, or to explore articles, documentation, or podcasts, which has made technical content a powerful marketing tool.

But now that AI has gobbled up so much of the internet, is it sufficient for technical startups to write technical content, then sit back and wait for developers to wander in? As more people utilize generative AI chatbots as a proxy for search, how should founders think differently about content?

Developer content expert Oleksii Klochai of Wizard on Demand suggests actionable steps for technical startups to improve their content program for both SEO and GEO (generative engine optimization):

Transition Away from the TOFU Fluff

Unspecific, top-of-funnel content without any net-new information doesn’t drive revenue or differentiate your startup from competitors, even if it’s ranking #1 on Google.

BOFU Content

Specific, opinionated bottom-of-funnel content explaining how your product fits customers’ needs sets you apart for both human and LLM readers.

Utilize Zero-Click Best Practices

As you gain awareness from unclickable LLM results, focus on clearly-stated use cases, examples, and social proof on-site, as well as credible third-party mentions elsewhere.

Reconsider Distribution

Beyond posting just to your website, consider respectful participation in closed communities and experimenting with paid placements.

Re-Evaluate Measurements and Team Setup

Focus on measuring outcomes across a lightweight team that includes product experts and technical marketers.

More Resources on Developer Content

• Article: The Future of Software Documentation in the Age of AI with Heidi Waterhouse
• Article: Personal Branding for Founders with Karl Stewart
• Article: Educate Technical Audiences With Template Marketing by Al Chen

The classic SEO content funnel may be on its way out. Image courtesy HubSpot

Is Long-Form SEO Content Past Its Shelf Life?

Klochai suggests that for technical startups, the days of the “HubSpot playbook,” producing massive quantities of long-form articles to try to build your brand into its own publishing house, may be over. “Basically, people were saying: You should publish a lot. But I think there has been a huge shift.”

The content expert suggests that the shift in interest may have gone from top-of-funnel (extremely broad content intended to generate awareness) towards bottom-of-funnel (more-specific content targeted at buyers who are already in-market and actively shopping for a solution).

“I think this shift is a good thing that should have happened 10-15 years ago. LLMs have helped it in some ways.” Klochai points out that TOFU (top-of-funnel) content is often so high-level that it covers general-knowledge topics, which are already being covered in Google’s AI Overviews and in basic LLM responses. Frequently without citations.

BOFU (Bottom-of-Funnel): Unique Insights About What You Do

The content expert suggests that specialized startups can offer a unique perspective on technology, which probably should’ve been the focus of content all along. “For example, if you work at a Kubernetes monitoring startup, you have the opportunity to write about that topic and express opinions on best practices. Rather than a generic topic like ‘How to build a sales team.’”

“It might not even make sense for such a startup to even try to build a generic Kubernetes content resource, because that has also been done. The real opportunity, and a lot of the value, can be found in BOFU content.” The content expert notes that for a time, e-commerce vendors in particular focused on getting product recommendations from popular chatbots like ChatGPT.

However, recent reports suggest that ChatGPT is referring 50% less traffic to external websites. Klochai also notes, “At the time of this writing, the number of user sessions originating from ChatGPT is less than 1% of overall website traffic.”

“There is still value in being mentioned as a product recommendation, but that might take the form of a user asking: What products should I use for X problem? And you might appear as one of the recommended products, maybe without the link to your website. But maybe just being featured as ‘One of the Companies That Can Potentially Solve the Problem’ is what matters.”

“This is why people started paying attention to BOFU: If you want to appear in [LLM] product recommendations, these systems need to understand that this problem is actually something that you help solve. And if you don't have anything on your website, or online anywhere, that talks about your positioning in this specific way, then these systems have no way of finding that out.”

“The problem is that a lot of people don’t seem to really know how to do BOFU.” The content expert suggests that many orgs’ content programs have focused on high-level TOFU content for so long that they may have forgotten how, or never learned, to create lower-funnel content.

Klochai suggests that the new beachhead for technical content is greater focus on specific problem spaces. “People are starting to write very focused pieces about their problem space, about how their products fit into the market. I think people are going to continue realizing that BOFU opportunities are important to communicate what your product actually does.”

“And that it’s actually okay to be selling your product.”

Content for Audiences Doing More Research Than Ever

The content expert doesn’t argue with past research suggesting anywhere from 57% to 74% of sales prospects complete most of their research before contacting a vendor. The deep research that in-market buyers perform makes it even more important to help LLMs, and search engines, correctly identify what your startup does and what your product can do.

“Technical companies may be competing with a few established competitors, but relatively speaking, the most competitive SEO niches are gambling and medicine-related topics. Basically, what I mean here is writing, in clear language, about what your product does from multiple perspectives, not really worrying about saying the same thing twice.”

“That's how natural language works. If you really believe something, then it's okay to say it again and again on different pages. And that also communicates to whatever model or crawler looking at your site that this is actually an important thing, whether that’s synonyms or different ways of describing what you do.”

The content expert points out that another piece of the puzzle is how external websites talk about your company and products. “This seems like a bit of an unknown for LLMs at the moment, though there's a lot of chatter about how many responses are being picked up from places like Reddit, for example.”

“But you don't really have control over what is being said about you on Reddit. (As I understand it, there are currently lots of companies, even household names, that are manipulating Reddit by seeding conversations and doing all kinds of unethical stuff.) I'm not suggesting this is a good thing and I'm really hoping that platforms including Google and Reddit crack down on some of this stuff.”

“But if this is going to be the next frontier, then what do you need to do on your side in terms of advertising, PR placements, and marketing? To help people understand and be able to correctly explain to others what you do?”

"If you really believe something, then it's okay to say it again and again on different pages. And that also communicates to whatever model or crawler looking at your site that this is actually an important thing.” -Oleksii Klochai/Principal, Wizard on Demand

What Does Zero-Click Marketing Mean for Technical Content?

The content expert suggests that messaging and creative assets can both affect the success of a startup trying to get attention. “Is everything going toward zero-click marketing? Is all Web traffic going to continue dropping in the future? It’s hard to say and I’m still undecided. I don’t think Google-style searching will be completely replaced. Research suggests LLM users may be using Google differently, but they still use it.”

However, the content expert notes that in response to prompts, LLMs tend to present a rather limited amount of information, which may eventually drive buyers to your website even without a direct link. While OpenAI recently announced in-chat e-commerce, it’s not clear how well B2B software will fit.

“If you’re a dev using a coding tool like Cursor, you might ask LLM chatbot to generate some code for you, and the code ends up using a library that’s part of a paid service. That could eventually be an interesting funnel that skips the initial website visit. But generally, you’ll probably end up going to the website of that library or the company that created it, and you’ll be looking over their solutions.”

“So, I think having a website continues to be important from the perspective of publishing and documentation.” Klochai suggests that despite some portion of searches potentially going to zero-click behavior, media consumption habits won’t change overnight. “Our information needs are much higher and more intensive than search behavior. And that is not going away anytime soon.”

How Technical Audiences Are Consuming Content Differently

“I think a lot of developers still want to read, and like to read. But the way they discover what to read is changing quite a bit.” Klochai notes that developers, like everyone else, are on social media, including communities like Hacker News. “But that’s discovery, and people are still very much interested in depth of information.”

“If you’re the head of marketing at a technical startup, and you want to talk to technical personas, you have to think about the distribution angle. How are you going to get people to read what you wrote?” Previously, orgs might have focused on SEO and organic reach, which might have been fine. “I think that these days, you're going to have to be much more intentional about how people are going to discover what you wrote.”

“Do you post on Hacker News, then call it a day? Or are you going to do something a little bit more sophisticated? For example, an owned media approach that doesn’t depend as much on what else was being posted on that channel. Something that gives you a little more control and predictability over how many people will read.”

“This could mean building a newsletter. It could mean visiting multiple communities and making it a habit to distribute things a little more actively. Which may mean being a genuine participant in communities where people are hanging out, and not just posting your links. Actually coming in and answering questions and communicating with others, whether that’s in a Slack or Discord channel or anywhere else.”

The content expert also suggests that startups may be underutilizing paid promotion for content “Lots of companies run ads. In many cases, very few of those ads actually go to content. But a lot of the interesting ideas that companies actually want to distribute aren’t necessarily on landing pages: They’re in content pieces and blog posts.”

“There’s a lot of stuff you can do to open up your offerings, including why you’re using paid channels. If you’re using paid, you can do much more to actually push your content to people who might otherwise not naturally find it.”

The content expert notes that different personas may have different preferred channels, social media and otherwise. “Depending on what you want to do, you have to understand how those places work and how to get people to read your stuff in those channels. I think that is an important skill to have for a marketer these days.”

49% of searchers trust organic results more than paid. Image courtesy PageOnePower

How Should Startups Rethink Distribution of Technical Content?

The content expert admits that busy marketing leaders at technical startups need to make difficult trade-offs when it comes to resourcing the creation and distribution of content versus the expected outcomes. It’s important for marketers to properly scope how much time they can invest into content on a per-article basis in terms of creation and distribution, and that they can commit to doing it consistently.

“The question is: How do you find the right balance of getting the right people involved, with a reasonable amount of time invested? For example, distributing content to closed communities: You cannot just post promotional links there. It's more jumping in and commenting if there’s a relevant conversation happening.”

“But the amount of time that takes on a weekly basis? Understanding in some cases, like a free Slack community, you may comment once and after 90 days, all history is gone. And you have to go in and do it again. That is kind of one level of effort that you could be putting in.”

“For early-stage companies that understand certain things don’t scale, this kind of thing could be interesting. You have a chance to talk to people and maybe have some kind of deeper discussion, not just about product recommendations.”

Beyond genuine community engagement, the content expert suggests that startups might explore paid partnerships with active communities. “I think startups aren’t really thinking enough about opportunities to do some sort of sponsorship of these closed communities. Obviously, in an ethical way that does not create weird incentives or anything like that.”

Klochai points out that paid campaigns can help justify investing in sophisticated creative assets, but there are limitations and trade-offs. 49% of online users trust organic results more than paid search results (the other 51% being ambivalent). “It seems similar for social media: If a Tweet is promoted, you may trust it a bit less than a non-promoted Tweet.”

“But there are a lot of ad formats that appear to be merging traditionally promoted and non-promoted content. LinkedIn now offers ‘Thought Leader ads’ which look like normal posts, but say they’re ‘sponsored.’”

“We’ve found quite a bit of success with these types of paid programs that use platform-native content. It’s about not trying to interrupt people with whatever kind of format that you decide is ‘best,’ such as a shiny video. It’s about trying to create things that fit within that platform, and then leaning into the paid distribution on that platform.”

The content expert notes that while paid channels do require more of a budget, they can be more economical than investing years trying to build an organic following. Also, organic and paid don’t have to be mutually exclusive, and startups can absolutely combine the two.

"We’ve found quite a bit of success with paid programs that use platform-native content. It’s about trying to create things that fit within that platform, and then leaning into the paid distribution on that platform.”

How to Revitalize Your Technical Content Today

Klochai suggests that teams pause for a beat to rethink their priorities. “It’s much less about saying: ‘Let’s build out a library of stuff we think is a good idea.’ It’s about trying to understand whether the ideas you’re executing on are resonating with the marketplace in some way. Previously, it was easy [to see what ideas worked].”

“This might be one of the reasons why we ended up in that scenario of optimizing for high traffic. The way to see if content succeeded was to just pump it out and then look at the traffic. If the traffic chart was going up, you were succeeding. But in reality, that’s not really the case if your goal is sales and revenue.”

Klochai suggests that previously, the popular approach was to find what topics were popular in search, then write articles for the highest-volume topics identified by SEO tools. “I think these days, the approach needs to be focusing on which topics are actually core to our business. Topics about which we actually have something useful and interesting to say.”

The content expert suggests that despite being unintuitive, it may be more meaningful to deprioritize search volume for certain terms and focus on topics that impact your business, and your customers’ businesses. “I still see all these ‘listicle’ articles all the time for product recommendations. You see dozens of these blog posts that cover ‘The Top Tools for X.’”

“And they’re all kind of the same. So why are these articles even there? Because people looked at what competitors did and they tried to one-up them. And I don’t think that’s a good idea anymore. Maybe there’s some interest you can extract that way, but that’s an article written for bots. Not for reaching people.”

The content expert suggests that content may not just be about the structure of information, or the level of usefulness or informativeness. “Yes, those are important. But for some topics, it may be more important to have a different point of view, or to be entertaining, or visually different than everything else. What should you do if everything out there is a blog post that is just text and images?”

“It’s about focusing on your users and understanding what they really need. I think starting from there will improve the quality of a lot of the stuff that companies do. The thought process should be more about what users actually need, then connecting those needs back to what you can do. And thinking about what is different about what you can say versus other people. And thinking not just in text and images.”

The post Developer Content in the Age of AI appeared first on Heavybit.

In episode 45 of Generationship, Rachel Chalmers speaks with Allegra Guinan of Lumiera about the trust dynamics and design ethics of voice-based AI. Together they explore how human tendencies to anthropomorphize voice systems can both build and erode trust, underscoring the need for responsible design and diverse perspectives.

The post Ep. #45, Perspective Density with Allegra Guinan of Lumiera appeared first on Heavybit.

“A startup is a company designed to grow fast.” -PG

Why Moats Matter for Software Startups

Almost every startup burns cash while chasing growth. That is intentional. Investors fund these loss-making machines because they expect them to eventually become large, high-margin businesses whose future profits will dwarf their current losses.

Software companies are especially attractive to venture investors. They can scale distribution at the speed of the internet, and once the product works, each new user costs virtually nothing to serve. That’s how Microsoft sustains ~45% operating margins, while Walmart operates at around 5%.

But the same factors that enable fast growth also make software companies vulnerable to competition once they hit steady-state profitability. A small team can build a Microsoft Excel competitor, but no one can build a Walmart rival without billions of dollars in capital.

For a software business to deliver durable, outsized profits, it must develop intrinsic barriers to entry. In other words, it needs a moat.

Moats vs. Competitive Advantages

Some competitive advantages evolve into moats, but not all do.

For example, OpenAI had a massive first-mover advantage when it launched ChatGPT. But that edge didn’t translate into a true moat. Competitors like Anthropic caught up quickly.

Sergey Brin and Larry Page’s PageRank algorithm initially gave Google a superior search product compared to existing search engines. They were able to transform the high usage of their superior product into a real moat as they layered user behavior data (clicks and engagement) into the algorithm, creating a data flywheel that improved the product with usage. No matter how much money competitors like Microsoft poured into their search efforts, they couldn't make much of a dent, and the result was a cash cow that has financed Google’s empire.

Types of Moats in AI

There are many types of moats: Brand, economies of scale, data flywheels, high switching costs, regulatory barriers and patents, talent, etc. It is still too early to say which moats will prove to be durable for AI-native businesses. People don't even agree whether the model is the product, or a commodity. However, it seems there are differing moats emerging for AI infrastructure (foundational lab) vs. applied AI companies. On the AI infra side, the major labs are trying to build moats around economies of scale and talent. And on the applied AI side, it seems that moats based on brand and switching costs related to long-term memory and owning customer context may be emerging.

OpenAI is a great case study. It is increasingly two separate entities: A leading foundational lab building the world's best closed-source models and a consumer AI application company with products like ChatGPT and Sora. The lab’s moat comes from compute scale and concentration of top talent. The application moat, on the other hand, consists of things like the "ChatGPT" brand and the increasing memory and context it has about its users. If the Sora app takes off, it will be a classic social media network effect and data moat.

One area that seems surprisingly underdeveloped is the data flywheel for reinforcement learning and fine-tuning. Very few companies are meaningfully building continuous learning loops to modify their models' weights and biases based on user interaction. This may be because today’s LLMs built on the transformer architectures don’t support true continual learning well.

Moats in Early-Stage Funding Rounds

With money pouring into AI startups, moats often come up in funding conversations. That’s understandable. If investors write a check for an agentic framework startup, they want to know whether the first-mover competitive advantage will translate into a durable moat by the time OpenAI launches their own.

But at the pre-seed (and even seed) stage, founders shouldn’t waste time over theoretical moats around businesses that don't exist. The only job is to get to product–market fit.

Good investors at these stages will be looking for signals of founder-market fit (pre-seed) or early traction (seed), not for long-term defensibility. It’s fine if they ask about moats simply as a litmus test to gauge your strategic thinking. But if they pass because the moat isn’t “strong enough,” that usually means they don’t believe in the founder or the market, and they’re sugarcoating the rejection.

Parting Thoughts

If you’re an early-stage founder: focus on traction — and forget the moat.

The post Forget the Moat appeared first on Heavybit.

In episode 23 of Open Source Ready, Brian Douglas and John McBride sit down with Davanum “Dims” Srinivas to discuss the health and future of the Kubernetes community. They explore how corporate changes impact open source contributions, the importance of onboarding programs, and the challenge of sustaining long-term contributors. Dims also shares insights into Kubernetes’ evolving role in AI and GPU workloads. The discussion is equal parts career advice, technical insight, and open source storytelling.

The post Ep. #23, Kubernetes, AI, and Community Engagement with Davanum Srinivas appeared first on Heavybit.

On episode 15 of Platform Builders, Christine Spang and Isaac Nassimi sit down with Amrit Dhangal. Together, they explore his journey building Acquire from a scrappy live chat tool to a venture-backed customer service platform. Amrit reflects on scaling challenges, personal sacrifices as an immigrant founder, and the lessons he’s carrying into his new company, Aero.

The post Ep. #15, Everything’s Recorded Whether You Know It or Not with Amrit Dhangal appeared first on Heavybit.

In episode 22 of Open Source Ready, Brian and John sit down with Benji Kalman, co-founder of Root, to explore the intersection of AI, software development, and security. They unpack "vibe coding," its impact on API proliferation, and the hidden costs of increased technical debt. Learn why a security vulnerability is just a bug with a purpose, and discover how AI agents can be used not just to write code, but to automatically find and remediate vulnerabilities in open-source containers.

The post Ep. #22, AI and Container Security with Benji Kalman of Root appeared first on Heavybit.

On episode 14 of Platform Builders, Christine Spang and Isaac Nassimi speak with Anthony Presley. Anthony walks through his journey from early consulting to creating workforce management SaaS, highlighting how persistence and customer-driven development shaped his companies. Together, they dig into the complexities of restaurant software, the importance of open ecosystems, and where dining tech is headed.

The post Ep. #14, Navigating Restaurant Tech’s Roadmaps with Anthony Presley appeared first on Heavybit.

In episode 44 of Generationship, Rachel speaks with Felipe Huici, CEO and co-founder of Unikraft, about the powerful world of unikernels. Felipe breaks down how these lightweight, specialized VMs can achieve millisecond cold starts, enabling services to truly "scale to zero" without users noticing a delay. Learn how this efficiency is not just about saving money, but also about providing a secure and scalable foundation for the next wave of AI-generated applications.

The post Ep. #44, Why Unikernels Are Cool with Felipe Huici of Unikraft appeared first on Heavybit.

In episode 21 of Open Source Ready, Brian and John sit down with Chad Metcalf, CEO of Continue, to explore the rise of continuous AI. Chad shares his journey from early embedded systems and platform engineering to leading an open-source-focused company building coding agents. They discuss automation, developer trust, open source attribution, and how AI is reshaping workflows across the entire software lifecycle.

The post Ep. #21, Continuous AI with Chad Metcalf of Continue appeared first on Heavybit.

In episode 43 of Generationship, Rachel explores the frontier of AI-driven cybersecurity with John Amaral, co-founder of Root.io. Together they unpack the promise and challenges of agentic systems that detect, patch, and remediate vulnerabilities automatically. This installment offers a mix of technical deep dives, AI optimism, and a vision of security that works more like an immune system than a fire alarm.

The post Ep. #43, Security Woven In with John Amaral appeared first on Heavybit.

In episode 13 of Platform Builders, Christine and Isaac explore the intersection of AI and medicine with ER physician Nathan Murray. Nathan shares how and why he founded DocAssistant, an AI-powered scribe and decision-support tool for emergency medicine. This conversation examines the challenges of integrating AI into clinical workflows and offers a unique perspective on the future of healthcare tech.

The post Ep. #13, Mirroring the ER Brain with AI with Nathan Murray of DocAssistant appeared first on Heavybit.

Introduction to Training an LLM

Organizations of all sizes and maturity levels are adopting artificial intelligence (AI) quickly. While enterprises are notorious for slower adoption, their retention rates for AI applications are now hitting 63% annually, up from 41% the previous year. So while there are arguably overhyped, generic use cases that could fizzle out, AI is becoming stickier in some places.

If you’re at an organization that’s already leveraging generative AI, you may be tasked with exploring innovative applications for it. Large Language Models (LLMs) offer a range of opportunities, but while pre-trained LLMs like the ChatGPT family from OpenAI and the Llama family from META are excellent for general purposes, there’s a good chance your company will need custom LLM models that are a bit more bespoke.

Teams can optimize models for better on-the-fly performance using techniques like retrieval-augmented generation (RAG), or they can take the approach of training their LLM on their own data. (Note that a training project, the initialization of an AI model’s weights against a dataset, differs from a fine-tuning LLM project, which further trains pre-trained models and is often even more costly in terms of computational resources).

Whether you're building a chatbot with expertise fueled by internal documentation, generating domain-specific content, building a natural-language processing (NLP)-focused BERT model for a text data or text generation use case, or enhancing existing models with proprietary knowledge, this tutorial will walk you through the practical steps required to train a language model on your own dataset.

More Resources on LLM Engineering

• Article: LLM Fine-Tuning: A Guide for Engineering Teams in 2025
• Article: RAG vs. Fine-Tuning: What Dev Teams Need to Know
• Article: Synthetic Data for AI: Purpose and Use Cases
• Article: How to Create Data Pipelines

What Is an LLM and Its Key Components?

If you’re reading this, there’s already a good chance you know about LLMs, here’s a brief recap. An LLM is a deep learning model trained to understand and generate human language. LLMs are commonly built on the Transformer architecture, introduced in the seminal "Attention is All You Need" paper.

Key components of an LLM include:

• Tokenization: Breaks inputted text into manageable units/tokens (such as words or subwords) so it’s machine-readable.
• Embedding Layer: Maps tokens to high-dimensional vectors. These embeddings are updated during model training to capture task-specific meaning.
• Positional Encoding: Added to token embeddings to give the model information about the position of each token in the sequence.
• Attention Mechanisms: Allows the model to weigh the relationships between all tokens in a sequence, and dynamically focus on the most relevant components of the context for each token.
• Feed-Forward Networks: Process each position independently, and enrich the model’s capacity to detect and process complex patterns per the model’s algorithms.
• Output Layer: Converts the final token representations into probabilities over the vocabulary, which then generates predictions, such as the next token in a sequence.

When combined, these elements of an LLM create a pipeline that transforms your inputted raw text into outputted meaningful predictions with nuanced understanding and context-aware sensitivity. Now let’s talk about how to apply that to your organization.

Why Train an LLM on Your Own Data?

If you have a common medical problem, such as a cough or headache, you’re probably fine going to see a doctor who is a general practitioner. On the other hand, if there are recurring concerns about specific parts of your body, such as your knees or shoulders, you might want to schedule an appointment with someone who specializes in that field, such as an orthopedic surgeon. The value of your own LLM, trained to focus on your users’ needs, can be seen in a similar way.

While GPT-4, for instance, might be able to provide a broad overview of what your organization does at a high level — based on publicly available information — it’s not going to be able to do so with the same level of granular knowledge that is common amongst your co-workers.

By training an LLM on your own dataset, you can achieve:

Domain Knowledge: General models will struggle with niche terminology or specialized workflows. By training your own LLM on your own specific data tailored to your use case (like support tickets, codebases, or datasheets), you ensure the model internalizes domain-specific vocabulary, abbreviations, and context.

Better Performance: Generic models can be imprecise on narrow tasks. Fine-tuning on task-relevant examples, such as customer questions, code reviews, or internal documentation, significantly boosts accuracy, relevance, and reliability. For use cases where data freshness is critical, updating your data (such as via a data pipeline) using a real-time cadence may be necessary.

Privacy and Control: If you choose to host and train your own LLM on-premise or within a secure cloud environment, you’ll ensure that sensitive, regulated, or proprietary data stays within your infrastructure. This is especially important for industries with compliance requirements or IP concerns.

Customization: By fine-tuning the model's highly-sensitive behavior, you can better match your organization’s brand voice, formatting standards, or response strategies. This leads to a more consistent and aligned user experience across your product or platform.

Training an LLM on your own training data turns a general-purpose tool into a purpose-built solution aligned exclusively with your environment, goals, and constraints. That said, anyone who has successfully built, launched, and maintained in-house tools knows it can all seem easier said than done.

How to Train an LLM on Your Own Data: A Step-by-Step Guide

The success of training your own LLM using training data you own hinges on a variety of factors, but let’s outline the broad steps, using Python and open-source tools where available for simplicity.

1. Collect and Process Your Data

It’s a lot easier and efficient to train your model if your assembled training data is cleaned up and standardized. Below are steps and tools to get high-quality data ready (and build a custom dataset as needed):

Data Collection

Here are a few sources to either sample or start datasets:

• Hugging Face
• Kaggle
• GitHub Repositories
• DataCamp

For the actual contents of your data, you’ll need to figure out what items make the most sense for your use case, such as:

• Internal documentation
• CRM or support tickets
• Chat logs or Slack messages

If you’re building a customer support LLM, for example, you can start with a dataset like MultiWOZ, which is a collection of multi-domain task-oriented dialogues.

Preprocess Your Data

Once you’ve got your data, you’ll need to prepare it for training. Preprocessing your training data will result in greater efficiency and reduced latency. Here’s what you need to look for:

• Remove duplicates to avoid skewing your model
• Watch for data gaps and remove rows with missing context
• Filter incomplete examples, such as dialogues without responses
• Label your data if needed for supervised fine-tuning
• Eliminate HTML tags and special characters
• Normalize text (lowercasing, removing stopwords, etc.)

Finally, tokenize and convert your text to a numerical format using tools like Hugging Face's tokenizer:

python

from transformers import AutoTokenizer

sample_text = "Hello, how can I help you today?"

tokens = tokenizer(sample_text, padding="max_length", truncation=True, return_tensors="pt")

You may also embed the tokens or assess attention layers later in the model pipeline, but at this stage, it’s a good idea to focus on ensuring your data is in a standardized, clean text form that will play nicely with any API you use to feed data to your model

2. Build Your Model

At the heart of the majority of LLMs is the aforementioned Transformer architecture, but luckily you won’t need to build a full custom LLM from scratch. For example, you can use popular Python frameworks such as PyTorch and TensorFlow, combined with libraries like Hugging Face Transformers or Langchain to work with a prebuilt base model or customize your own.

Here’s an illustrative example of loading a pretrained model using Hugging Face Transformers:

python

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("gpt2")

The above makes sense if you're fine-tuning a full model. For efficient training, consider techniques like LoRA or adapters that let you train only a subset of the parameters.

3. Set Up Your Environment

You’ll need a proper development environment to train and test an LLM, which requires high-performance capabilities. Here are some examples:

• Python 3.8+
• GPU support (e.g., NVIDIA CUDA)
• Libraries: PyTorch, TensorFlow, Hugging Face Transformers, Tokenizers
• Cloud options: Google Colab, AWS EC2 with GPU, Paperspace, or Lambda Labs
• On-prem

Example setup using Hugging Face:

bash

pip install torch transformers datasets

To run on GPU:

python

import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

Step 4: Train and Test the Model

Now comes the actual model training process.

First, set the initial parameters:

• Epochs: Number of passes through the dataset (e.g., 3–5 for small datasets)
• Batch size: Number of samples per batch (e.g., 16 or 32)
• Learning rate: Controls how much the model adjusts per update

Example using Hugging Face's Trainer:

python

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(

output_dir="./results",

per_device_train_batch_size=16,

num_train_epochs=3,

logging_dir="./logs"

)

trainer = Trainer(

model=model,

args=training_args,

train_dataset=train_dataset,

eval_dataset=eval_dataset

)

trainer.train()

Optimization with Stochastic Gradient Descent (SGD)

Modern LLMs often use Adam or AdamW, which converge faster and handle sparse gradients better than SGD. It may make sense to use SGD only for experimentation or comparison, though if you prefer classic SGD, here’s an example:

python

import torch.optim as optim

optimizer = optim.SGD(model.parameters(), lr=0.01)

Test and Interpret the Results:

Your next steps should include evaluating your model’s loss, perplexity, and task-specific metrics:

python

trainer.evaluate()

To further break down the evaluation criteria, let’s look at each category:

• Perplexity: The exponentiated average loss. This is the go-to metric that tells you how “confused” the model is when predicting the next token. Lower perplexity means more confidence and accuracy in its predictions.
• Loss: How far off the model’s predictions are from the correct answers (ground truth). It's computed during training using a function, which is usually cross-entropy loss for language models.
• Task-Specific Metrics: Measure the model’s performance on the actual job it’s intended to do, rather than just general language ability. For example, If your LLM is summarizing text, you might use ROUGE or BLEU scores. If it’s answering questions, Exact Match (EM) or F1 score might be better. Each metric captures success in a way that generic benchmarks cannot.

As a general rule, during training, you’ll want to measure log loss and perplexity after every epoch/batch. At the validation stage, you’ll calculate all your metrics on a held-out dataset.

Step 5: Modify Parameters (Hyperparameter Tuning)

With proper tuning, your LLM will train faster, generalize better, and perform more accurately on its assigned custom tasks.

Hyperparameter tuning is the process of systematically adjusting the configuration settings that control how a machine learning model learns during training. Unlike model parameters (like weights and biases, which are learned from data), hyperparameters are set manually before training begins.

When training LLMs, tuning hyperparameters is crucial to getting better performance, especially on custom or domain-specific datasets.

Since LLMs are incredibly sensitive to how they’re trained, even small changes in hyperparameters can lead to noticeable differences in output quality or model accuracy. Once you determine the correct hyperparameters, you can:

• Speed up convergence
• Prevent overfitting or underfitting
• Improve generalization to unseen text
• Reduce training time and cost

Hyperparameters you can tweak include:

• Learning Rate: Controls how much model weights are adjusted during backpropagation.
• Batch Size: Number of samples processed before the model updates weights.
• Number of Epochs: How many times the model sees the entire training dataset.
• Weight Decay: L2 regularization that penalizes large weights.
• Dropout Rate: The probability of dropping a unit during training to help prevent overfitting.
• Warmup Steps: Gradually increases the learning rate from zero to its target value over these steps.
• Gradient Clipping: Prevents exploding gradients by capping the norm of the gradient vector.
• Sequence Length: Defines how many tokens the model can handle at once.

You can use tools like Weights & Biases, Optuna, or Ray Tune to automate tuning. It’s a good idea to not try to tune everything at once, which can muddle your results. Instead, it can be better to prioritize by learning rate, batch size, and epochs first, then work your way down.

It’s also helpful to evaluate your model using metrics like perplexity, BLEU score, or custom task-based metrics. It’s also a good idea to use a validation dataset to avoid overfitting:

python

# Example of evaluating perplexity

import torch

import math

loss = trainer.evaluate()["eval_loss"]

perplexity = math.exp(loss)

Final Thoughts

Training an LLM on your own data is no longer the exclusive domain of large research teams. With open-source tools, pre-trained models, and cloud-based compute, individual developers and small teams can create powerful, domain-specific LLMs.

If you’re exploring how to use LLMs to enhance your product or automate domain-specific tasks, this process can be your foundation.

The post How to Train an LLM on Your Own Data: Beginner’s Guide appeared first on Heavybit.

There’s a great deal of interest in whether AI can (or can’t) provide value by increasing the productivity of engineering teams. This interview with developer productivity expert Laura Tacho, CTO at DX and 10+ year veteran of software development, covers opportunities for engineering orgs to take advantage of AI tooling and actionable suggestions for AI startup founders to build stickier, more-successful products for engineering end-customers.

How Are Engineering Teams Using AI?

78% of surveyed companies use AI, though only 3% of engineering leaders report achieving “very high” productivity gains. The most widely-reported use case for AI is code assistants, including high-profile products like GitHub Copilot, Claude Code, Cursor, and Continue, but there are still adoption challenges that might not have anything to do with productivity.

More Resources on Dev Productivity & AI Adoption

• Article: Productivity and Mental Health Resources for Remote Teams
• Article: Enterprise AI Infrastructure: Compliance, Risks, Adoption
• Article: How AI Is Reshaping Enterprise Infrastructure
• Article: The Future of AI Code Generation
• Article: What to Know About Pricing Developer Tools

While coding may seem to make sense for AI use cases because of how familiar it is to devs, it may not be the best opportunity. Says Tacho, “We have to look across the whole software development lifecycle to see productivity gains. We did a study of 180+ companies, and we found that for developers using AI, the number one time-saving use case was stack trace analysis, not code generation.”

Why Don’t Engineers Adopt AI?

The CTO suggests that while the difference between inflated expectations and the disappointing reality of cleaning up after messy, nondeterministic AI is an adoption barrier, there are practical obstacles as well. “I would say training and enablement are probably the largest barrier.”

“It’s not intuitively obvious, even for teams with lots of enthusiasm and optimism, where the highest-value use cases are hiding for a particular organization. Some patterns need to be taught and coached, just like with any other tool. Companies that skip that step (and try to rely on the optimism and enthusiasm of an individual developer) may see some initial success, but then they're going to see that success taper off.”

What’s Really Missing: Targeted Goals & Outcomes

So yes, individual devs may not use the company Copilot account more frequently because they haven’t been drilled on prompting best practices. However, entire engineering organizations may not be getting as much as they could from AI tools because there’s a lack of strategy.

“I think where the rubber isn't meeting the road is figuring out: What are the business problems in the company that have the most to gain from AI usage?” The CTO clarifies: “There should be room for experimentation. There should be time to try something new and figure out: OK, what can we do with this tool? But when it's only that, and we don't have targeted goals and outcomes, then things can go off the rails.”

“The companies where I see the most adoption and the most durable benefit from AI are ones that came to AI with a specific problem. For example: Companies that needed to modernize legacy infrastructure, and hypothesized that AI would help them modernize legacy components much faster than they could have previously.”

“Organizations that want to see organizational benefit need to think about this on an organizational level, not just rely on individuals figuring out what to do.”

How Can Startups Make the Most of AI?

Tacho concedes that different organizations with different resources (and different amounts of runway) will need to take different approaches. “I think you need to look at it as a spectrum of targeted experiments vs. targeted use cases. That proportion needs to be adjusted based on your company’s size and stage.”

“For startups, those are the environments where the individual can have a huge impact. So it would make sense for people to be experimenting, to be rapidly figuring out where they can exploit the capabilities of AI for your business. But you should document that as you go and make sure it's not just one individual deciding to do one-off things. Build a playbook of use cases that aren’t ephemeral.”

The companies where I see the most adoption and the most durable benefit from AI are ones that came to AI with a specific problem." -Laura Tacho / CTO, DX

“Depending on the market pressures on your business, that's going to dictate how you think about a rollout. Obviously, for startups that rely heavily on individual rigor and urgency in order to get work done, the AI strategy is going to fit better with focus on individuals or small teams. Larger organizations that are dealing with decades’ worth of legacy software need to take a different approach.”

How Should Engineering Teams Adopt AI?

The CTO empathizes with AI startup founders, who face mounting pressure from a market that moves incredibly fast. “I feel like AI tooling is being commoditized at a rate I have never seen before. A few months ago, when people evaluated tools, they were looking at which model they could use. It was a differentiating feature. And now you can use whichever model you want. Sometimes it feels like things change hourly.”

However, the CTO notes the increasingly competitive nature of some AI product lines is at odds with the organizational benefit of keeping a unified stack, despite individual developers continuing to crave autonomy. “This is a tough one. The reality is that engineering leads feel that business pressure for vendor consolidation. No company wants to be forking over more money for an arbitrary number of vendors that all more or less solve the same problems.”

Tacho suggests that a viable alternative for startups is to offer developer stipends to allow them freedom of choice. “It’s definitely a reasonable alternative, especially in a startup environment that may be a bit more open with regards to governance, risk, and security. It’ll also depend on whether you work in consumer vs. another industry.”

The CTO notes that other companies do attempt to consolidate around single vendors, using targeted proof-of-concept campaigns and running bake-off comparisons between competing vendors when needed. “Some companies decide that since the market's so volatile that they’re going to trial something for six months and then figure out if things stabilize.”

“I think no matter what approach an organization takes, there are going to be shadow AI tools out there because these are tools that right now are not prohibitively expensive for developers to pay for by themselves, and that is the reality, and it happens everywhere.

How Can AI Startups Target Developers as Customers?

The CTO notes that while certain market factors appear to be changing, the top drivers for purchasing AI tools, or any tools, remain the same. “I just spoke to a bunch of other CTOs, and while cost was not as much of a factor in purchasing decisions last year, it definitely is this year.”

“But accuracy and security are still top drivers for purchasing decisions. I think there's some table-stakes aspects for any AI tool that engineering teams have come to expect. Beyond the table stakes, the more that tools can illustrate better ROI and costs, the better.” Tools that can make usage metrics and data readily available will play nicer with developer productivity tooling to paint a clearer picture of ROI.

“For a long time, the primary metrics we had were acceptance rate, number of licenses, and how much was being paid for them. But there wasn’t much transparency when it came to how tools were actually being used. So focusing on a really robust admin API is a great way to ensure that you can build a defensible ROI case and get people to keep their wallets open.”

Opportunities for AI Products Targeting Engineering Teams

“As mentioned, areas like general-purpose code authoring tools may be reaching a level of saturation. But beyond code generation, there are opportunities to target other use cases across the software development lifecycle: Requirements authoring, code review, or changelogs. For AI products targeting engineering teams, the most valuable use cases are the ones that are under-solved right now.”

Of course, many teams are still concerned about the non-deterministic nature of GenAI and the havoc that a hallucinating LLM might have on their codebase. However, the CTO notes that organizations that have built-in organizational resiliency tend to be the ones that make the most of AI.

“When you introduce AI, in all its non-deterministic glory [to resilient orgs], they're not falling over. They're not bottlenecked. Whereas companies who don't [have organizational resilience] and think that AI will be a magic solution to all the problems they've been avoiding? Those companies are getting a reality check. The fundamentals don't go away just because you have AI.”

Beyond code generation, there are opportunities to target other use cases across the software development lifecycle. For AI products targeting engineering teams, the most valuable use cases are the ones that are under-solved right now."

“I think developer experience is not an outcome of using AI tools really well. It's a prerequisite to use them well. What's good for an individual human developer is also good for an agent. There's extra stuff that you might want to do for agents, but the physics of what makes software or codebases easy to contribute to, easy to change...that's important for AI agents and for humans.

“AI is an amplifier. If you have parts of your system that are a bit garbage, they're going to be amplified garbage now. And if you have really solid engineering practices that have stayed ahead of the industry curve, then that's going to be amplified and you're going to get even better results.”

For AI startup founders specifically targeting engineering orgs, the slickest user interfaces and best new user experiences will only go so far. If those engineering orgs that AI themselves are struggling to maintain stable, resilient systems, they may struggle to succeed with any AI product.

“To those AI product builders who are concerned about whether their product will be sticky with customers, I might say: Maybe think about building tools that fix your customers’ rickety systems. Probably a less exciting problem to solve, but people have their wallets open for it right now.”

The post How to Drive AI Productivity and Value for Engineering Teams appeared first on Heavybit.

As AI becomes more of a reality in software development, teams are experimenting with a variety of use cases, including documentation and technical writing. In this article, veteran technical writer (and co-author of Docs for Developers: An Engineer’s Field Guide to Technical Writing) Heidi Waterhouse discusses some of the biggest mistakes teams make using AI for docs, as well as the best opportunities to use AI productively to support user journeys.

More Articles on How Documentation and Software Dev Are Evolving:

• Article: Who Owns the Documentation and Why It Should Be the Robots
• Podcast: Documentation Deep Dive with Megan Sullivan of Gatsby
• Video: Building Great API Docs
• Video: How Great Documentation Drives Developer Adoption

AI as an Assistant, not a Solution

Waterhouse elaborates on her recent session from the LeadDev conference, at which she spoke on a similar topic. “The focus of my talk was discussing things we know AI can do well, and how to make that fit with our understanding of what people need from documentation.”

The author suggests that the key to getting value out of large language models (LLMs) today is thinking about AI “as an assistant, and not a solution. Because thinking about it as a solution is what leads us down some unfortunate paths that are really hard to dig ourselves out of.”

Heidi Waterhouse discusses AI in documentation at LeadDev. Image courtesy LeadDev.

Are LLMs the Best Tool for Software Docs and Technical Writing?

Freshness: Using RAG to Ensure Docs Stay Up-to-Date Instead

Aside from hiring a good writer a year or so before launching a product to document it properly, Waterhouse suggests that product teams identify what can and can’t be easily documented, such as standardized OpenAPI documentation. “There's not a lot of distinction between writing a good test case and writing a good prompt. The things you do to test are the things that you need to document: How do I do this thing? And so we go into test cases and extract user stories from those, because we know that those work.”

The author points out that alternatives like retrieval-augmented generation (RAG) may be better suited to maintain docs freshness than using a LLM as a catch-all solution, even if it was trained on a large dataset. “Although we get better answers from very large [data] corpuses, I think it is dangerous because for nearly every LLM out there...their training corpus is at least a year old, and we don’t know exactly what it contains.”

“So if you have made software [changes] in that time, the LLM doesn't know about it and it's not retrieving fresh data. So I think you should have a small, specific, internal [focus to] look at all this stuff and use that as your context window, instead of a giant context window. That makes it both cheaper to run and more relevant.”

Indexing and Librarians: Better Docs Through Better Organization of Info?

The author notes that for documentation, the real use case that most users want isn’t longer, prettier notes to read, but the ability to ask a specific question in natural language and receive a contextually correct, helpful answer. “That's the ‘killer app.’ That requires some indexing. I haven't built this yet myself, because nobody has paid me to do it. But being able to ask any question and get an authentic answer: That’s what you want.”

To get to such a state requires proper indexing of product information, so that your system can return a specific, accurate response. In other words, a highly deterministic result that’s at odds with GenAI’s inherently non-deterministic nature. “For example, if a person asks about installation, here's the set of documents they want, and you’d refine it with the set of documents you’d return. So you’d need a natural-language sorting on pre-indexed stuff.”

“But that means that you have to pay someone to do the indexing with computer assistance. Indexing a whole set of things is difficult, but you can't just leave it to machines because machines don't understand anything. They do not ‘understand’ as they are not capable of ‘understanding’ as a concept. So I think that there's going to be an uptick in the need for librarians: People who understand semantic information and semantic relationships.”

The State of Docs vs. What Has Been Lost to Time

The author points out that the status quo of documentation isn’t reams of static docs that someone wrote years ago and no one has looked at since. “It hasn’t been like that for 15 years! But one of the interesting problems is that we've democratized technical writing and made it all markdown files, which means we've taken the expertise out of it.”

“I think back to the priesthood of Framemaker, RoboHelp, and other things we used to have. Extremely specific, context-sensitive help, where every page you were on in an application had an identifier, and if you clicked the little ‘help’ button, you would get help specific to that page. It would know where you were. There's no reason we can't have that with web applications. But nobody has been doing it. And I think just returning to something like that would be immensely useful.”

“Today, what we're writing is: Here's how to get started with what you're doing. Here's the reference documentation. There's a whole framework about what a reference doc is and what an instruction doc is and what an example doc is. And all that knowledge is difficult to transmit to founders because it's expertise. It's like asking someone: ‘Please be an expert in user interfaces.’

A Gentle Reminder: Docs Aren’t for Happy People

“I think that what we need to be doing is not just writing more documentation, which doesn't actually help people. I think what we need to be doing is making it easier to find answers to a problem that you're having. This is always the goal of documentation. Because I always say that anyone reading documentation is already pissed off.”

The author clarifies that people reading documentation don’t want to read documentation. They want to already be completing their tasks. “We need to do as much as we can so people don't have to read documentation. What if machines could read documentation and answer questions. That'd be great! But we need to design better software.”

In an ideal world, with better software, users wouldn’t need to fight a 20-page config guide to get it to work. “I tell my clients: ‘Have you made a Docker container for [your product]? Could you just give people a runtime environment that works for them so that they don't have to configure it?’”

“I think we're moving in that direction a lot for the things that we used to use documentation for. But in order to do that, you still have to know how things work.”

What we need to be doing is making it easier to find answers to a problem that you're having. Anyone reading documentation is already pissed off.

Are Smaller Models the Key to Better Docs?

Waterhouse notes that while GenAI is exhibiting potential for summarization use cases, and might be theoretically useful to gobble up reams of documentation and spit out answers, the solution isn’t quite that simple. “The thing that gobbles up data also has its own biases. It doesn't have a good semantic understanding of exactly what you're doing.”

The author suggests that an alternative might be smaller, lighter-weight models and retrieval tools trained on a highly-specific, frequently-updated dataset from which it answers questions. “But it's much clearer to think of it as a chat-based interface for an index rather than thinking of it as an answer machine.”

Weaponizing Skepticism

The author notes the problem of software documentation seeming to have all the answers...put into the hands of LLMs that still confidently give wrong answers. “Recently, there was a thread online where someone recommended feeding some Perl string into an LLM to ‘see what it parses as.’ It turns out the string parsed as a UNIX command to wipe your hard drive.”

“Now, that was obfuscated. And a human who has been on the Internet for a while knows not to run random unchecked strings. But [AI] doesn’t know that. It doesn't have any sense of danger. And I honestly think that we are going to have to code in a sense of danger. For example, how a prominent startup founder’s vibe-coding tool accidentally wiped their production database.”

“I think what we need to build into our tools is a sense of skepticism. (We can call it ‘testing,’ but testing is just weaponized skepticism.) We need to teach AI tools a sense of danger and a sense of skepticism before we let them loose in production.”

“Separately, we need to think about how AI will consume what we write in a different way than a human being might. How do we design documentation not just to be consumed by humans, but to be consumed by machine learning [algorithms]?” the author muses. “Honestly, the OpenAPI project seems like the start of that: This is a semantically sound, well-formatted thing that you can consume whether or not you're a machine. It’s predictable and you can count on it to have the same structure so that it's not going to give you weird answers.”

“Making our documentation accessible to machines is going to be valuable, because then we can use machines to ask [the documentation itself] questions.

Making Builders Future-Proof in an AI Future

Waterhouse also considers the impact of AI on engineering teams, including for junior developers. “Every time we make an advance in technology, we feel like we're going to lose those skills, and it turns out that somebody cares about preserving them. Also, even mechanization doesn't reduce the amount of work that needs to happen.”

The author cites More Work for Mother by Ruth Schwartz Cowan, a book about how laundry used to be an infrequent, time-consuming chore that required hiring washerwomen servants. After the advent of the washing machine, the position of washerwoman vanished from society, but because washing machines made doing laundry more accessible, families would do laundry more frequently, causing housewives to eventually invest the same number of hours per year doing laundry as they did previously.

“We're going to have junior testers who can run exponentially more tests, but somebody still has to write them. Those skills will be sustained, but they're not going to be ‘a whole job.’ The problem is somebody has to understand the fundamentals.”

“We watched this happen with the abstraction of languages up the stack. There are not a lot of people coding in C anymore because our abstraction layer above C is good enough. But there's a reason people are still using COBOL. It's because everything but COBOL has floating point errors for extreme edge cases of math that actually matter quite a lot. So we still need to have at least a few experts from somewhere who care about the deep internals.”

The author notes that older programming experts are retiring, leaving a vacuum for experts on computing fundamentals. “For a lot of these really sophisticated abstractions that LLMs rely on, we need someone who understands how to program the thing that programs the thing that makes the chips.” The current state of play in AI is marked by inflated expectations and budgets, but hopefully, market forces will iron out the issues.

Without experts in the fundamentals, software will have increasingly precarious dependencies. Image courtesy XKCD.

What Will Never Change: Who and What Docs Are For

The author cites Making Users Awesome by Kathy Sierra’s fundamental point about software: No one uses it because they ‘love using software.’ “People use software because they’re trying to accomplish a task or get something done.”

“And when we're writing documentation, every piece of documentation that somebody has to read slows them down on the way to getting something done. It may speed them up in the long run, they may understand something better, but on the whole we are diverting them from their task.”

The post The Future of Software Documentation in the Age of AI appeared first on Heavybit.

In episode 12 of Platform Builders, Christine and Isaac chat with Mike Boufford, CEO of Otti and former CTO of Greenhouse. Mike unpacks hard-earned insights on topics like leadership philosophies, management best practices, and the unexpected moments that shape company culture. This conversation blends candid war stories with practical takeaways for anyone leading technical teams.

The post Ep. #12, Earn Respect, Not Headlines with Mike Boufford appeared first on Heavybit.

In episode 20 of Open Source Ready, Brian and John sit down with data engineering and cognitive science expert Vasilije Markovic to explore AI memory and how we can build more intelligent systems. From the challenges of "context rot" to the practical applications of AI memory in construction, education, and finance, this conversation covers how to give your AI the context it truly needs.

The post Ep. #20, Exploring AI Memory with Vasilije Markovic of Cognee appeared first on Heavybit.

In episode 42 of Generationship, Rachel Chalmers speaks with Katie Hallett about why she and her team decided to build a web browser for machines, not humans. Katie explains the inefficiencies of traditional scraping tools, why Lightpanda is so much faster, and how AI is reshaping developer workflows. They also offer career advice to help new graduates stay ahead in the AI era.

The post Ep. #42, Ziggy Stardust with Katie Hallett appeared first on Heavybit.

At Heavybit, we invest in enterprise infrastructure. This is an evolution from our early days as developer tools specialists, but the core thesis remains the same: We back founders building for technical users.

What’s changed is the scope of that audience. Technical users are no longer confined to engineering teams at tech startups. They now power critical workstreams across nearly every major business on the planet.Take JPMorgan Chase, for example. Over 55,000 employees, more than 20% of the entire workforce, now work in tech roles like software engineering, DevOps, and data science. It’s a bank with a huge brick and mortar footprint, but its business runs on software. This shift is reflected in startup trends too: In recent YCombinator batches, the share of companies building for technical users has grown from around 10% in 2017 to over 30% today.

While it may feel like early winners have emerged with strong moats, we’re still in the earliest innings. Assuming foundational models continue improving at their current pace, many of the defining infrastructure companies of the AI era likely haven’t even been imagined yet. All the noise around vibe coding and AI codegen? Those are just tremors from an offshore quake. The real tsunami is still on its way.

Racing to Capture the Budget

According to McKinsey, in 2014 enterprises spent 75% of their IT budgets on labor. By 2024, that number dropped to 58%. Hardware spend stayed flat—the biggest change came from a surge in external software spend, as SaaS tools ate into internal headcount.

This trend will only accelerate with the rise of AI.

Two major categories of new infrastructure companies are emerging to capture this shift:

User-Down Startups

These startups give superpowers to existing technical personas, or aim to replace them entirely, by embedding intelligence into their workflows from the top down. Some are AI-native productivity tools (like Cursor), while others reimagine the workflow altogether (like Claude Code). Their main competitor is often just a new feature in an incumbent tool: Why adopt Nao when Hex just shipped Magic?

Tech-Up Startups

These companies build the underlying stack for AI-native applications. The first wave included evaluation frameworks and vector databases. But a new set of primitives is coming—tools that make the emerging intelligence stack usable by builders and deployers. Many of these companies will sell into the same enterprise infra budgets indirectly, by powering the top-down apps.

Winning Founding Teams for Each Segment

When evaluating early-stage companies, we focus on founder-market fit and total addressable market. But the signal I’m looking for is less “Can they build the product?” and more on whether the founder can demonstrate that they can win deals and build sales momentum. For me, that signal is different for each of the two segments.

For user-down companies, at least one founder must have deep experience in the role they’re trying to augment or replace. You can't sell DevSecOps tooling if you do not speak the language of DevSecOps. A prime example is Harvey, where one co-founder was a lawyer and half of the first 120 hires came from top law firms.

For tech-up companies, I look for founders who know how to build for developers: Great developer experience, smart open-source strategy, and real community building. These users hate being sold to, but they are drawn to great docs and communities that help each other debug in online spaces like Discord.

If you're building in either of these categories, I'd love to hear from you: amir [at] heavybit.com

The post How AI Is Reshaping Enterprise Infrastructure appeared first on Heavybit.

In episode 85 of o11ycast, Dr. Cat Hicks unpacks AI’s impact on software teams from a psychological and social-science perspective. Along with Ken, Jess, and Austin, she explores how AI magnifies long-standing tensions between solitary and collaborative models of development, and how fears about AI often reflect deeper issues like undervaluing collaboration or having unrealistic productivity expectations. The discussion also explores empathy, theory of mind, and pluralistic ignorance, highlighting why developers may prepare more for AI than for each other.

The post Ep. #85, AI/LLM in Software Teams: What’s Working and What’s Next with Dr. Cat Hicks appeared first on Heavybit.

In episode 41 of Generationship, Brighthive CEO Suzanne EL-Moursi joins Rachel Chalmers to unpack the “three-layer cake” of modern data architecture: composable stacks, agentic AI, and governance-first design. From integrating hundreds of data sources to enforcing real-time compliance, Suzanne shares how her team is tackling the grunt work of data so humans can focus on innovation. Plus, her vision for turning every organization—big or small—into a data powerhouse.

The post Ep. #41, Covenant with Suzanne EL-Moursi appeared first on Heavybit.

In episode 19 of Open Source Ready, Brian and John speak with Josh Rosso, Principal Engineer at Reddit and author of Production Kubernetes. From his early days at CoreOS and Heptio to running Reddit’s massive compute platform, Josh shares insights into managing Kubernetes at internet scale, the business realities of open source, and the risks smaller OSS projects face. Lastly, they dive into AI’s growing role in engineering and the challenges of keeping the internet human.

The post Ep. #19, Kubernetes at Scale with Josh Rosso of Reddit appeared first on Heavybit.

In episode 11 of Platform Builders, Christine Spang and Isaac Nassimi chat with Greg Galant. Greg recounts his early entrance into podcasting, social media, and eventually PR tech. He shares war stories from the early 2000s, including launching a podcast before “podcasting” was popular, creating the viral Shorty Awards, and bootstrapping Muck Rack into a multi-million dollar SaaS company. The conversation offers a unique lens on innovation, timing, media evolution, and how AI is reshaping earned media.

The post Ep. #11, How LLMs Are Changing PR Forever with Greg Galant appeared first on Heavybit.

RAG vs. Fine-Tuning: Advantages and Disadvantages

In the rapidly evolving world of artificial intelligence, the ability of machine learning models to produce coherent, relevant responses has become critically important. Foundation models like GPT-4 or LLaMA have shown impressive question-answering capabilities, but they come with a limitation: Their knowledge is frozen at the time of training.

More Technical AI Resources:

• Video Compilation: How to Make Open-Source & Local LLMs Work in Practice
• Guide: LLM Fine-Tuning: A Guide for Engineering Teams in 2025
• Guide: Synthetic Data for AI: Purpose and Use Cases
• Guide: How to Create Data Pipelines
• Interview: How to Properly Scope and Evolve Data Pipelines
• Interview: Best Practices for Developing Data Pipelines in Regulated Spaces

For teams working with generative AI (GenAI), retrieval-augmented generation (RAG) is a powerful architectural innovation that overcomes this constraint by enabling real-time access to external knowledge sources using natural language processing (NLP). In simple terms, RAG can enhance a model’s performance allowing it to retrieve relevant data (beyond the general knowledge it may possess from prior training) dynamically from the appropriate data sources and incorporate it into generated responses.

Let’s break down how RAG works, what makes it valuable, what complexities are involved, and the tools organizations are using to adopt it successfully.

What Is Retrieval-Augmented Generation (RAG)?

RAG is a technique that extends the power of machine learning models by combining them with a data retrieval system that can access external knowledge sources in real-time. Instead of relying solely on what was encoded during training, the RAG system can "look up" information from external data sources—like a vector store containing documents, articles, or enterprise data—before generating a response. RAG models effectively use ad hoc data retrieval to enhance a machine learning model’s performance.

This approach allows AI models to stay current, using ad hoc information retrieval to adapt to different domains and contexts. You can think of RAG as giving your AI a dynamic memory—one that can be updated anytime, queried efficiently, and used intelligently to improve outputs.

Without RAG, foundation models are like a closed-book: they only know what they were trained on. With RAG, they become an open-book: they can consult an external, always-updatable knowledge base as needed with domain-specific data.

How RAG Works: The High-Level Architecture

At a high level, a RAG pipeline consists of two main components: retriever and generator.

The retriever component searches a knowledge base to find the most relevant pieces of information (passages, documents, or snippets) based on the user’s query. Instead of full-text search, RAG systems typically use vector search—a method that finds semantically similar documents using embeddings generated by models like OpenAI's text-embedding-ada, SentenceTransformers, or Cohere.

Once the retriever returns relevant context, the language model (like GPT-4 or LLaMA) takes this context and the original query to generate a coherent and informed response. The generation is "augmented" with retrieved knowledge.

A defining trait of RAG is that knowledge is not stored inside the model’s parameters, but in a separate and often dynamic storage system—usually a vector database. This decouples knowledge from the model itself, which is a game-changer in terms of flexibility, cost, and maintainability.

Key Benefits of RAG

RAG unlocks several major advantages for teams working with large language models (LLMs):

Dynamic Knowledge UpdatesUnlike traditional LLMs that require costly retraining to learn new information, RAG systems can be updated on the fly with new data. Want your chatbot to understand today’s breaking news or a newly added product in your catalog? Just update the knowledge base, no model retraining necessary.

Lower Infrastructure Costs Compared to Fine-TuningThe fine-tuning process for large models is expensive in both computational resources and engineering effort. RAG allows you to avoid this by offloading domain-specific data and knowledge into a retrievable format. The base model remains unchanged, which saves time and money.

Easier to Iterate and DebugWith RAG, improving your system often means adjusting the data retrieval pipeline or curating better documents—processes that are faster and safer than altering a model’s weights. This makes development cycles shorter and more transparent.

Enhanced Factual Accuracy and TrustBecause RAG systems can cite their sources or refer to specific retrieved passages, the responses tend to be more grounded. This is essential for applications that require trust and traceability, like legal advisors, medical assistants, or customer support agents.

Domain-Specific CustomizationNeed a model to specialize in financial reports, technical documentation, or legal precedents? With RAG, you can tailor responses by simply controlling what data is stored in the information retrieval layer—no need for model-specific customization.

The Trade-Off: Added Complexity

While RAG systems are incredibly powerful, they introduce new layers of complexity compared to vanilla LLM applications. Some areas that should be given careful attention include:

Retrieval QualityThe performance of a RAG system hinges on its retriever. Poor retrieval leads to irrelevant or confusing context being passed to the generator, which degrades the quality of the response. Tuning retrieval involves selecting the right embedding models, refining the search index, and filtering results intelligently.

Vector DatabasesRAG typically relies on vector databases like Pinecone, Weaviate, Qdrant, or FAISS to store and search embeddings. These databases must be managed for performance, availability, and data freshness. Indexing strategies, vector dimensionality, and approximate nearest neighbor (ANN) algorithms all play a role.

Prompt EngineeringYou can’t just shove retrieved documents into a prompt and expect good results. Effective prompt engineering is critical–you need to decide how to format the retrieved content, where to place it in the prompt, how many documents to include, and how to structure the prompt to steer the model toward accurate, context-aware answers.

Evaluation and Feedback LoopsMeasuring the performance of a RAG system isn’t trivial. You need to monitor retrieval precision, generation quality, and hallucination rates. It often involves human-in-the-loop evaluation or advanced automatic scoring systems that blend search relevance with natural language understanding.

Popular Tools in the RAG Ecosystem

A growing number of open-source and commercial tools make it easier to build and deploy RAG systems. Here are some of the most widely used:

LangChain is a Python framework that simplifies building RAG pipelines by chaining together components like retrievers, vector stores, and LLMs. It offers abstractions for managing prompts, memory, agents, and retrieval workflows. It's especially popular for building chatbots and custom search agents.

LlamaIndex (formerly GPT Index) focuses on creating structured indices from unstructured data, making it easy to plug external data into LLM applications. It supports various vector databases and retrieval strategies and plays nicely with LangChain and other LLM frameworks.

Pinecone is a fully managed vector database that handles storage, indexing, and retrieval at scale. It’s known for high performance, ease of integration, and production-readiness. It supports metadata filtering and hybrid search (text + vector).

Weaviate is an open-source vector database with built-in support for semantic search, classification, and multi-modal data. It includes a GraphQL interface and allows for automatic schema inference, making it a strong choice for teams building custom knowledge systems.

FAISS, Qdrant, and Milvus are open-source options that provide robust alternatives to commercial vector databases. FAISS (by Meta) is a library for efficient similarity search, while Qdrant and Milvus offer full server-based vector storage with filtering and scale-out capabilities.

Compared to fine-tuning, RAG is potentially a faster, more up-to-date method of improving performance, but it’s not plug-and-play.

RAG represents a powerful shift in how we think about building intelligent systems. By decoupling knowledge from the model and allowing dynamic access to relevant information, RAG gives teams a flexible, cost-effective, and more accurate way to use generative AI in real-world applications.

However, adopting RAG is not a plug-and-play solution. It requires thoughtful design of the retrieval pipeline, selection of the right tools, careful prompt construction, and continuous monitoring to ensure relevance and reliability.

For teams building AI systems that need to reason over private data, stay current with external information, or deliver grounded answers with references, RAG is quickly becoming the gold standard architecture. As the ecosystem matures—with tools like LangChain, LlamaIndex, and Pinecone evolving rapidly—implementing RAG is becoming increasingly accessible and impactful.

What Is Fine-Tuning?

As organizations race to adopt LLMs for everything from chatbots to document analysis, a key question keeps coming up: How do we make these models work better for our data, use cases, and tone? One common answer is fine-tuning.

Fine-tuning is a powerful technique that customizes a pre-trained language model on a specific training dataset to achieve more precise, context-aware, or brand-aligned behavior. While alternatives like prompt engineering or RAG also aim to steer LLMs, fine-tuning offers a fundamentally different approach—changing the model itself, not just the inputs or supporting data.

Fine-tuning is the process of continuing to train a pretrained model on a new, typically smaller, training dataset that is specific to your domain or application. It’s like taking a general-purpose expert and teaching them a niche skill through focused practice.

To clarify how fine-tuning fits into the broader LLM development stack, let’s distinguish it from two related concepts: pretraining and prompt engineering.

Pretraining is the initial process of training a large model on a massive, diverse dataset of training data (like Common Crawl, Wikipedia, or books). This is where the model learns the structure of language, basic facts, and reasoning patterns. Pretraining is computationally expensive and only done by major AI labs.

Prompt engineering involves crafting instructions and examples in the input to guide model behavior. It’s a lightweight, zero-cost way to steer output but is limited in its ability to enforce consistent behavior or deep domain alignment.

Fine-tuning sits between these two extremes, offering more precision than prompting without the cost and scale of pretraining.

How Fine-Tuning Works: Custom Training on Target Data

At a high level, the fine-tuning process involves taking a pretrained model and continuing its training using a smaller, targeted dataset. The process of compiling training data typically includes the following steps:

Data Collection and CleaningThe first step is to collect a high-quality dataset that represents the task, tone, or domain you want to teach the model. This might include: customer service transcripts, medical case reports, legal contracts, internal documentation

Cleaning, labeling, and formatting the data (usually into input-output pairs) is critical to avoid introducing noise or bias.

Model SelectionYou then choose a base model. If you're working with open-weight models, you can download and fine-tune locally. If you're using a proprietary model, you’ll use the vendor's fine-tuning API.

Training the ModelTraining typically involves supervised learning, where the model adjusts its internal parameters to better predict the desired outputs given your inputs. Depending on your specific use case, you might also use reinforcement learning or instruction tuning.

Modern fine-tuning frameworks like LoRA (Low-Rank Adaptation) or QLoRA (Quantized LoRA) have made it easier and more memory-efficient to fine-tune large models on modest hardware.

Evaluation and TestingAfter training, you’ll evaluate the fine-tuned model to ensure it generalizes well to new examples. Metrics like accuracy, F1 score, BLEU score (for generation tasks), and human feedback are commonly used.

Open-Weight vs. Proprietary Models: Two Fine-Tuning Paths

Not all models can be fine-tuned the same way. The landscape splits into open-weight models and proprietary models, each with different implications for access, control, and cost.

Open-weight models are available for download and modification. You can fine-tune them on your own infrastructure or using cloud GPUs. This offers full control over the model and training process, which is ideal for teams with MLOps capabilities and strict privacy requirements.

On the other hand, proprietary models with fine-tuning APIs don’t expose their weights but offer APIs for uploading fine-tuning data. The provider handles the training process. A quick list of pros and cons:

Open-Weight Models

Pros and cons for open-weight models like the META LlaMA families, Mistral, Falcon, BLOOM, or Pythia:

Pros:

• Full control over weights and architecture
• No vendor lock-in
• On-prem deployment possible

Cons:

• Requires infrastructure and ML engineering expertise
• More complex to scale and manage

Proprietary Models with Fine-Tuning APIs

Pros and cons for proprietary models with fine-tuning APIs like ChatGPT, Claude, Cohere, AI21, or Google PaLM:

Pros:

• No need to manage infrastructure
• Easy to use with minimal ML expertise
• Automatically integrates with hosted inference APIs

Cons:

• Limited flexibility in architecture and customization
• Potential vendor lock-in
• Higher long-term cost for high-usage applications

Key Benefits of Fine-Tuning

Fine-tuning isn’t the right tool for every situation, but it shines in use cases where precision, control, and integration are critical.

Precise Behavioral ControlPrompt engineering can only go so far in steering output. Fine-tuning lets you bake in the exact tone, structure, and policy adherence you need—whether that’s avoiding disclaimers, matching brand voice, or using specific formatting.

Deep Domain AdaptationLLMs are trained on broad data, which can be weak or inaccurate in niche areas like biomedical research or legal reasoning. Fine-tuning allows the model to internalize deep, domain-specific knowledge that isn’t easily accessible via RAG or prompting.

Tighter Workflow IntegrationFor enterprise software that relies on deterministic behaviors (e.g., filling forms, structured output, or specific taxonomy usage), fine-tuning provides a way to hardwire patterns into the model’s behavior.

Multi-Turn or Task-Specific MasteryIf your use case involves multi-turn dialogue, classification, summarization, or code generation, fine-tuning can make the model more adept at handling those specific tasks.

Fine-tuning is more resource-intensive than RAG, but offers potentially better response times and gives users more control over output formats.

The Challenges of Fine-Tuning

Fine-tuning offers great power that can enable LLMs to grasp deeper nuances in complex fields like healthcare, but it comes with significant challenges—especially when compared to lighter-weight methods like RAG or prompting. (It can also be a more time-consuming approach to improving your model’s performance.)

High Cost and Compute RequirementsTraining large models—even for a few epochs—requires significant GPU time and memory. While adapter-based methods like LoRA can reduce cost, fine-tuning remains more expensive than prompt engineering or RAG. Introducing new data generally incurs high retraining costs.

Model Versioning and Lifecycle ManagementOnce you fine-tune a model, you create a new version that must be versioned, monitored, and maintained. This introduces lifecycle management complexity: Which version are we using in production? What data was it trained on? How do we roll back?

Risk of Overfitting or DriftIf your training data is too small or too biased, the model can overfit—losing its general-purpose abilities and hallucinating facts. Fine-tuned models can also "drift" from expected behavior over time if not retrained or evaluated continuously.

Evaluation Is HarderPrompt-based systems are easier to debug—you can tweak the prompt and re-run. Fine-tuned models are black boxes with learned behavior that’s harder to explain or adjust without retraining.

MLOps and Deployment ComplexityFine-tuning introduces classic ML deployment challenges: reproducibility, monitoring, rollback mechanisms, and model registry. Enterprises adopting fine-tuning often need mature MLOps pipelines to manage this complexity at scale.

Fine-tuning is a high-impact tool in the LLM toolkit, especially for organizations seeking precise, repeatable, and domain-specialized AI behavior. It offers more control and customization than prompt engineering and is ideal when RAG can’t capture the depth of domain expertise required.

But it’s not a silver bullet. Fine-tuning introduces infrastructure and lifecycle challenges that demand careful planning and evaluation. For many use cases, RAG or prompting may be sufficient and cheaper. The best approach often combines methods—using prompt engineering for flexibility, RAG for dynamic knowledge, and fine-tuning for precision and domain depth.

As LLMs continue to evolve, expect fine-tuning methods to become cheaper, faster, and more efficient—especially with innovations in parameter-efficient fine-tuning, quantization, and automated evaluation. But for now, teams should approach fine-tuning with clear goals, strong datasets, and the right tooling in place.

RAG vs. Fine-Tuning: Key Technical Differences

As LLMs become essential tools for enterprise AI, product personalization, and task automation, two strategies have emerged as the primary ways to adapt these models to specific domains: RAG and Fine-Tuning.

While both aim to improve accuracy, relevance, and utility, they do so in fundamentally different ways—with distinct architectures, performance profiles, and infrastructure needs. Understanding these differences is crucial for choosing the right approach for your use case.

Architecture and Data Flow

RAG

In RAG, the system consists of three main components:

1. Retriever: Given a query, it fetches relevant documents or chunks from a vector database using semantic similarity (embedding search).
2. Knowledge Store: An external, updatable source of truth (e.g., docs, articles, proprietary data) stored as embeddings.
3. Generator: An LLM that takes both the user’s query and retrieved context to generate accurate responses.

This modular architecture separates the LLM from the data it relies on. Updates to knowledge don’t require retraining the model—just refreshing the content in the vector store.

Fine Tuning

In fine-tuning, you take a pre-trained base model and continue training it on a domain-specific dataset. The model learns to embed that new information directly into its weights. Once fine-tuned, all responses are generated from this internalized knowledge.

There is no retrieval component or external database—everything the model knows must be learned during training. How RAG and fine-tuning differ:

RAG:

• Updating data: Potentially instantaneous via vector database
• Adding new knowledge: Relatively easy with dynamic document ingestion
• Traceability: High with citable context

Fine-Tuning

• Updating data: Requires retraining
• Adding new knowledge: Must create new dataset and retrain
• Traceability: Low as responses may be less explainable

RAG offers far more flexibility and auditability, which is a major reason it's favored in fast-changing domains like legal, finance, or customer support.

Performance and Latency

Speed and responsiveness are critical for real-time applications, especially those deployed in production or embedded in user-facing apps.

Fine-Tuning: Low-Latency InferenceFine-tuned models are essentially “pure” inference engines. Once deployed, they require no external calls to retrieve information. This means:

• Lower latency: Responses are generated quickly.
• Better for streaming and real-time chat: Great for voice assistants, live coding tools, etc.

For example, a customer support chatbot trained on support logs via fine-tuning can answer quickly, making it suitable for high-throughput environments.

RAG: Additional Latency from Retrieval StepsRAG introduces at least one extra step: querying a vector database to fetch relevant documents before generating a response. This adds:

• Overhead from semantic search
• Latency from embedding generation (if done at runtime)
• Complexity from context formatting and token limits

Even small delays can affect UX in real-time apps.

Latency Mitigation Strategies for RAGTo combat this, teams use:

• Caching: Store previous retrieval results or final outputs based on query similarity.
• Asynchronous retrieval: Begin document fetch while processing initial user input.
• Index optimization: Use fast approximate nearest neighbor (ANN) search algorithms like HNSW.
• Client-side embedding: Pre-compute query embeddings to avoid runtime delays.

RAG users may observe higher inference latency due to the combination of both retrieval and generation; have caching opportunities in both the retrieval and prompt output steps; and may observe lower token efficiency due to including full context.

Fine-tuning users may see lower inference latency (having to only use inference for generation); have fewer caching opportunities at the prompt output step only; but see higher token efficiency due to pretraining.

Choosing between fine-tuning and RAG can come down to your resources, your customers’ expectations, and your regulatory environment.

Infrastructure and Tooling

Both RAG and fine-tuning require significant tooling, but the nature of the stack is very different.

RAG Infrastructure: Modular ML + Search StackRAG relies on a hybrid architecture that integrates language models with search infrastructure. Typical components include:

• Vector database (e.g., Pinecone, Weaviate, Qdrant)
• Embedding models (e.g., OpenAI, Cohere, SentenceTransformers)
• Retrieval orchestration (e.g., LangChain, LlamaIndex)
• Prompt templating for context formatting
• Chunking strategies for indexing long documents

RAG’s complexity lies in retrieval quality, chunking strategy, and prompt formatting—not model training.

RAG is ideal for teams that want fast updates to data, high transparency, and minimal training infrastructure.

Fine-Tuning Infrastructure: ML Training PipelinesFine-tuning requires a more traditional ML training stack:

• GPU compute for training and inference
• Data pipeline for cleaning, formatting, and labeling
• Training frameworks (like Hugging Face Transformers, PEFT, LoRA)
• Model hosting platforms (like Replicate, Sagemaker, Vertex AI, Modal)
• Version control for models and datasets

Fine-tuning introduces MLOps challenges including versioning models and datasets, managing fine-tuning artifacts, evaluating and validating outputs, and handling model rollback. A comparison of infrastructure elements when using RAG vs. fine-tuning:

RAG

• Core resources: Vector database, embedding models, LLM inference
• Skill requirements: Search, embeddings, prompt design
• How to Manage Updates: Can happen in real time via database
• Tool examples: LangChain, LlamaIndex, Pinecone, Weaviate
• Operational overhead: Moderate (retrieval tuning, prompt management)

Fine-Tuning

• Core resources: GPUs for training, datasets, model hosting
• Skill requirements: ML engineering, training optimization
• How to Manage Updates: Retraining needed
• Tool examples: Hugging Face, LoRA, OpenAI/Anthropic APIs
• Operational overhead: High (MLOps, model lifecycle management)

How to Choose the Right Approach for Your Use Case

There’s no one-size-fits-all answer—RAG and fine-tuning serve different needs.

RAG and fine-tuning are not mutually exclusive—in fact, many production systems combine the two. A hybrid approach might use RAG to inject current data into prompts, while fine-tuning ensures that outputs are formatted, styled, or filtered in specific ways.

Understanding the technical trade-offs between these approaches is essential for building scalable, maintainable, and performant AI systems. Fine-tuning gives you surgical control over how a model behaves, but requires serious infrastructure and MLOps support. RAG gives you agility and explainability, but introduces retrieval latency and retrieval engineering complexity.

For teams building next-gen applications, the best solution is often not either/or, but when and how to use both together.

Questions to Ask Before You Decide

Before jumping into implementation, consider these four key questions to frame your decision:

1. Is your data frequently changing or relatively static?RAG excels when your knowledge base updates frequently—product catalogs, pricing, policies, customer support FAQs.

Fine-tuning is better if the core knowledge is stable—like legal definitions, scientific terminology, or internal classifications.

2. Do you need precise, tightly-controlled outputs?Use fine-tuning when consistent tone, formatting, or behavior is required (like structured summaries or form outputs).

Use RAG when flexibility is more important than rigid control.

3. What infrastructure and talent do you have in-house?RAG requires experience with vector databases, prompt engineering, and search tuning—but not GPU training.

Fine-tuning requires ML engineers, model versioning, and MLOps pipelines.

4. Are latency or cost primary concerns?Fine-tuned models tend to be faster at inference, ideal for real-time systems.

RAG introduces retrieval overhead and may require more tokens, affecting cost.

Use Case Fit: RAG vs. Fine-Tuning

Here are practical examples where each approach shines:

When RAG Works Best

• Customer support bots: Pulls from up-to-date documentation, product info, or service logs.
• Internal search and chatbots: Integrates with company wikis, handbooks, or internal portals.
• Knowledge assistants: Summarizing legal contracts, scientific papers, or research reports.
• Compliance review tools: Cross-checks against evolving regulations.

When Fine-Tuning Works Best

• Classification tasks: Spam detection, sentiment analysis, document type labeling.
• Form parsing and structuring: Extracting data into predefined formats (e.g., CRM fields).
• Tone-specific writing: Generating content in a brand’s voice or for a specific persona.
• Code generation: When outputs must follow specific patterns or conform to style guides.

Hybrid Approaches: Can You Combine Both?

Sometimes, the best solution blends both worlds. Hybrid architectures combine the dynamic power of RAG with the precision of fine-tuning—though this adds significant complexity. Some examples of hybrid patterns include:

• Fine-tuned retrievers: Train a custom model to improve document selection in a RAG system.
• Fine-tuned generators: Use RAG to retrieve context, then pass it to a fine-tuned LLM that specializes in response formatting or tone.
• Layered workflows: Use RAG for knowledge injection and follow with a fine-tuned model for post-processing (e.g., structure, validation, or filtering).

Hybrid makes sense when your use case demands both freshness and structure (like a regulatory chatbot that references policy but formats output for legal review). Perhaps you’ve outgrown prompting and retrieval tuning but still need flexible data updates. Or you have MLOps maturity and the engineering bandwidth to handle dual pipelines and monitoring.

It’s important to proceed with caution. Hybrid systems introduce multi-point failure risk, require cross-discipline skills, and increase debugging complexity. For early-stage teams, this can slow iteration and introduce maintenance debt unless the use case truly demands it.

Choosing between RAG and fine-tuning isn’t just a technical choice—it’s a strategic one. It depends on how often your data changes, how much control you need, what infrastructure you have, and what your users expect.

Here’s a simple way to frame your decision:

• Use RAG when your knowledge needs to stay current and traceable.
• Use fine-tuning when your behavior needs to be consistent, structured, or domain-specialized.

Explore hybrid models only if your system requires both agility and deep model control—and your team is ready to support that complexity. Ultimately, the goal isn’t to pick a perfect approach—it’s to choose the one that best matches your real-world constraints while leaving room to evolve.

The post RAG vs. Fine-Tuning: What Dev Teams Need to Know appeared first on Heavybit.

In episode 40 of Generationship, Salma Mayorquin of Remyx AI unpacks the shift from traditional MLOps to ExperimentOps—a framework for scaling insight in modern AI development. With stories from Netflix, Yelp, and Stripe, she explains how teams are bringing foundation models into production and what it takes to keep up. If you're grappling with LLMs, MLOps, or decision paralysis in GenAI, this one’s for you.

The post Ep. #40, ExperimentOps with Salma Mayorquin of Remyx AI appeared first on Heavybit.

In episode 18 of Open Source Ready, Brian Douglas and John McBride are joined by Christopher Burns to unpack the complexities of consent management and cookie banners. Discover how Christopher’s open source project, c15t, aims to simplify privacy compliance for developers, bridging the gap between legal requirements and practical implementation. This episode offers a fresh perspective on why consent is not just a legal burden, but a critical component of modern web development.

The post Ep. #18, Consent Management with Christopher Burns appeared first on Heavybit.

In episode 10 of Platform Builders, Christine Spang and Isaac Nassimi chat with Seth Besmertnik about Conductor's journey through the WeWork acquisition and its incredible comeback from near-disaster, offering a powerful lesson in resilience and leadership. The conversation then pivots to the future, exploring how AI is about to turn the web upside down. Seth provides a clear framework for how brands must adapt their content and SEO strategies to survive and thrive in this new AI-driven world.

The post Ep. #10, Building Leverage, Not Features with Seth Besmertnik appeared first on Heavybit.

How to Think About Data Pipelines in Regulated Spaces

Tech teams standing up new AI programs, or scaling existing programs, need to determine how to build and operate data pipelines, a key infrastructure component for AI programs at the enterprise., and without good data pipelines, that data will suffer from quality, accuracy, and freshness problems – no matter which vendors you rely on. In this article, we’ll cover expert insights on best practices for data pipelines in regulated spaces.

Roshan Nanu, PhD, Director of AI at Prompt Health, is an expert on building data pipelines to support AI applications in healthcare, a highly-regulated industry with strict data privacy and transport regulations. We spoke with him about why data pipelines are so important, walked through the best practices and common mistakes, and learned how tech teams can implement these ideas in any industry.

AI Is Only as Good as the Data Underpinning It

The first mistake tech teams tend to make happens well before the first data pipeline plans are even scoped. Once an engineering team takes on a data pipeline project, it’s tempting to assume the data in question is all from the same domain and follows the same standards.

Nanu explains that the data conversation has to start well before the project gets to the engineering team. “I'm talking to our sales team, our marketing team, and our billing team about their practices,” Nanu explains. Over time, he says, “It becomes more and more important for these teams to create good free-text data.”

Generative AI has, of course, become dramatically more sophisticated in the past few years, but it still abides by a classic principle: Garbage in, garbage out. The information in the data coming from your sales, marketing, and billing teams might be very meaningful, but if it’s not formatted for the data pipeline you’re building, that meaning can be lost.

Before you build the data pipeline, you need to zoom out and assess the quality of your entire data corpus. “With companies wanting to leverage AI and develop LLMs (large language models) to scale, we can only get reliability if we have a good corpus of data to start with,” Nanu explains.

Reliability for AI applications is critical. If reliability is low, otherwise-effective AI applications can face adoption issues. This is especially true for retrieval augmented generation (RAG) applications. If an enterprise wants to enable users to get precise answers to nuanced questions, users need to be able to trust those answers.

Nanu explains, “We can only make good RAG pipelines to get information to people if we have good documents to use as sources.” For the sake of AI, “good documents” go far beyond the quality of the information within them. Good documents also exhibit high data quality, being uniform and standardized, ensuring that terms are consistent and formatting is clean.

Nanu frequently has conversations with non-engineering teams, showing them how to adapt their processes to support AI applications that will, in turn, help them. “I’m telling them to standardize their processes, write down their heuristics and how they do things, and write down good documentation based on their processes and best practices,” he explains.

Investing in this work before building your data pipeline saves significant effort down the road. Rushing into it risks discovering months later that teams have been formatting data differently or entering it into inconsistent fields. Retroactive standardization is always harder than proactive standardization.

The Type of Data You’re Working With Matters

All data is not created equal, and the type of data you’re working with can be a huge influence on how you build your data pipeline. A focus of Nanu’s, for example, is automating note-taking for healthcare providers – an otherwise time-consuming manual process.

Different companies approach the challenge in different ways, using either scribes, intake-based generation, or note re-formatting/re-writing. The outputs may end up basically the same, but the inputs vary wildly.

“My dataset needs to have whatever I want as my input,” Nanu explains, “And then your output needs to be in a format that I can actually use.” The challenge is in the refinement, not the collection or transformation of the data.

With companies wanting to leverage AI and develop LLMs to scale, we can only get reliability if we have a good corpus of data to start with.
-Roshan Nanu, Director of AI/Prompt Health

In a healthcare setting, data meshes can be a helpful addition. A data mesh is an approach to data architecture that decentralizes data ownership and management, allowing self-service among different teams.

“We have a huge advantage in that we are an electronic medical record (EMR) company,” Nanu explains. “All of our customers’ data is stored in the same way, even if they don't create their data in the same way.” A data mesh helps ensure that the data from domain experts – physicians, in this case – can be stored correctly without overburdening those experts with data work.

That said, a data mesh is neither easy to implement nor a panacea. “It's still a very hard problem to create a data mesh that works well, that our AI developers can use and pull from for their individual use cases,” Nanu warns.

Figure Out What Your Data Really Means

When you look for technical guides around data pipelines, many assume you’re starting from square one. In reality, most tech teams inherit datasets and data infrastructure built years ago. AI might not have even been on the horizon when your system was first built, but now it’s a priority, which means you need to deeply understand your data before doing anything else.

“With developing use cases across manufacturing, across healthcare, and more, the big gap is actually less about what data is available – it’s about whether AI engineers know what it means,” Nanu says.

A developer and a physician have very different knowledge bases, and even physicians in the same hospital will likely have different ways of transcribing records. No amount of sheer AI experience will make mutual understanding simple; you have to cross the gap first.

Otherwise, Nanu says, “Your AI developer will ask, ‘How do I map these inputs to what I actually need to get?’ ‘What do all these terms mean?’ ‘What do these data columns mean?’” It can take a lot of time, he warns, but it’s worth it if it means turning that sometimes obscure domain expertise into a resource.

It can be helpful to break down the process of figuring out your data into multiple steps:

1. Procure the data
2. Cleanse the data as needed
3. Format the data so that engineers and data scientists can query it
4. Define the data so engineers and data scientists can understand what they’re modeling and trying to accomplish

The first step is impossible without the step of defining data. Defining datasets can feel like humble work, just understanding the data at hand, but it’s significant progress. “Just getting that domain expertise and having it available to your AI engineers is such a big step,” Nanu says.

Looking at it another way, this progress isn’t just a big step for your team but, cumulatively, a big step for the software industry as a whole.

“Right now, we're in a world where middlemen developed the software,” Nanu explains. “Your domain experts are using some piece of software, and you're brought in now to build on top of that, but you're stuck in whatever infrastructure your software engineers set up, without any thought to the fact that this data would be used later.”

Ingest and Store Data to Support Present and Future Use Cases

Once you understand your data and have processes in place to standardize it, your goal is to build a data pipeline that is consistent, reliable, and scalable. Companies have a lot of documentation and data, almost by definition, and your data pipeline isn’t ready for production until it can ingest, process, and store all of it.

“The next step is developing that consistent data pipeline, getting something in place to ingest your own company's documentation, your patient data, your sales data, your data from every other third-party software you're using,” Nanu explains.

Ingestion, however, is only one component. Once you can reliably and consistently ingest data across all of the sources your company uses, you need to be able to store it.

“Get all of that into a data warehouse,” Nanu says. Storage policies need to be clear and strictly reinforced. “Data needs to be in a format that your AI engineers can query for the use case at hand.”

Nanu emphasizes an upfront infrastructure investment because AI is such a rapidly evolving field.

“Right now, every time you turn around, someone has a new use case for AI they want to throw at you,” Nanu says. “Today it’s ‘Hey, automate documentation for providers.’ Tomorrow, it's ‘Hey, figure out if patients are going to show up or not.’ Or, internally, ‘Hey, get us better reporting to know what the sentiment of our customer support is.’”

The use cases for AI are already plentiful, but they’re still growing. An effective data pipeline, relying on effective data infrastructure, can pipe data in and make it usable for use cases that haven’t even been built out yet. “For all AI use cases, we need data sets that we can go to that are trustworthy,” Nanu says.

Support Your Data Pipeline with Automation

As mentioned, many data teams inherit their datasets and infrastructure, so an early investment in improving this starting point frequently pays off.

As you build your data pipeline, consider what you can automate. Once you get full access to your data and standardize it, you can start building automated processes that work from those standardized formats and allow you to avoid a lot of manual work.

“If you have a lot of that standardization in place, the most important thing is to decide on a cadence, to set up your ETL pipeline, and to get it running such that you are updating that data frequently,” Nanu says. If your data isn’t frequently updated and fresh, even an otherwise effective data pipeline can be unreliable.

It makes sense, but many feel tempted to delay automation. “Honestly, putting that in the backlog takes so much time to recover from,” Nanu says. “If you have the opportunity, if you're getting it set up, automate that data pipeline first so it's always up to date.”

Choosing not to automate your data pipeline is, in many cases, choosing to take on tech debt. “Once you're up and running, your company expects you to put out new use cases or make things shift faster,” Nanu explains. Without a mature pipeline supported by automation, he says, “It becomes harder and harder to carve out time to go and optimize that back end or go and improve on processes.”

Much of this comes back to foundational engineering best practices. Nanu has seen companies try to build pipelines with only data scientists, not engineers, “And it was the most inefficient thing I've ever experienced because no one on that team knew how to write good software,” he says.

If you have a lot of that standardization in place, the most important thing is to decide on a cadence, to set up your ETL pipeline, and to get it running such that you are updating that data frequently. If you have the opportunity…automate that data pipeline first so it's always up to date.

That said, there are limitations, and you can’t expect a perfect pipeline set up on day one. “Automation is not always feasible,” Nanu explains, in cases where the AI team is new or the company is first identifying its primary use cases. “You don't know your data markets yet until you go through a few processes. And it's hard also to get access to that data,” Nanu says.

Some enterprises can take four to six months to spin up an AI team and get them access to production data to work with and train on. In such cases, developers shouldn’t just sit and wait. Instead, Nanu says, they can “use an assumption of what they think the data might look like once they get it for their initial models.”

Companies will always have new use cases, and they will expect data pipelines to keep up and evolve. A stable foundation and robust automation are essential. “If you can, get it done early,” Nanu says. “If you can't, hire for it. Hire someone whose sole job is to come in behind that AI team and develop those backend systems to do that data and software engineering.”

Store and Manage Data Securely and Efficiently

Data privacy is important in every industry, but it’s especially important in highly regulated industries, such as healthcare, finance, insurance, and manufacturing. If an AI application were to regurgitate private information about a patient, for example, there wouldn’t just be reputational damage – there could be penalties, fines, and lawsuits.

As Nanu says, working from his experience in healthcare, “All it takes is 20 instances of private healthcare information to leak for a lawsuit. 20 individuals.” As a result, a ‘mostly’ secure application just isn’t enough. Tech teams have to thoroughly de-identify the data that’s fed into their models to ensure privacy risks are as unlikely as possible. But that’s easier said than done.

“It actually takes quite a long time to set up a robust de-identification,” Nanu warns. “And then you have to go through the extra step of getting a third party to review a sample of your dataset and your de-identification process to certify that it is sufficiently de-identified to no longer be protected under HIPAA. And they go through that data painstakingly.”

There are no shortcuts, Nanu says. Even if you purchase high-quality de-identification models from a vendor, you still have to build your own systems to do additional redactions and keep your data clean.

Of course, even properly de-identified data is only as useful as it is accessible. Once you have the data de-identified, you need to store it effectively and enable developers to query it efficiently.

This component of the pipeline is integral, and Nanu recommends assigning a whole person or team to data management. “From a logistical standpoint, it's definitely worth having a person or team managing your data,” Nanu says. “That’s probably the most important thing that you have.”

Working with someone who can focus on data management is crucial because it’s not work that you can take lightly. “When getting things set up, I would be very cautious about what resources you're using to do so,” Nanu says. “A lot of people are going to dump stuff into S3 and say, ‘Hey, it's good enough,’ but it's really hard to index and search that data.”

The other obvious option isn’t good either. “On the flip side,” Nanu says, “sticking it on RDS gets very costly, very fast for big data sets, and it's not overly efficient.”

There’s a tough balance to strike: You want to index your data properly, but you also want to store it cost-efficiently. “At this point, data is money, data is power, and data is what you need,” Nanu explains. “You need to be able to store all your data efficiently, but also query it efficiently.”

How to Think About AI Agents

The generative AI paradigm shift has blossomed into innumerable sub-trends, and right now, AI agents are capturing a lot of attention. Agents are relatively new, and they’ll require high-quality data to underpin them if they have any hope of working.

“If you have one agent that is reliable, then you’ll have to put a lot of time and effort into building that agent and training it and fine-tuning it,” Nanu explains. Until you have a robust data pipeline, agents might not warrant your focus yet.

That said, if agents are what excite company leaders and agents are what’s drawing a budget, then don’t shy away. Nanu recommends caution: “If you're stuck building agents, it really requires thinking through every step of your chain of thought pipeline. Don't leave a lot to chance.”

The more you leave to chance, the less reliable the agents will be, leading to a degraded experience. “You get an explanation one day that you don't the next day. There's a small update, you switched to a different GPU, and now you're getting slightly different results,” Nanu explains. That’s not an experience you can put in front of a customer.

The good news is that even the newest trends still point back to building a solid foundation first: Your data pipeline. But if you’re building for agents, you must pay particular attention to data diversity.

“There's just so much of a gap in reliability that you really do need a very large and diverse dataset in order to support good agents,” Nanu explains. “That means you need several thousand examples of realistic input and very high-quality output.”

Develop Failover and Fallback Plans

Many teams are rightfully wary of becoming too dependent on any given vendor, including foundation model providers. However, for early-stage startups and teams new to AI, buying into a foundation model, even at a relatively cheap plan, is a good start.

“A lot of companies want to retain ownership of models,” Nanu explains, “But those large proprietary companies, those source models, provide something very valuable at first, and that is speed and quality.” The idea of standing up your own models can be appealing, but it’s unlikely you’ll match the providers’ speed and quality, and you certainly won’t get close to that level quickly.

Generative AI is a scaling game, and until you have the data, it’s safer to risk some vendor lock-in. “Your fine-tuned, self-hosted, open-source 7 billion parameter model is going to perform absolutely terribly until you have that 5,000-or-more data point set,” Nanu says. “Until you get to that point, you're better off throwing your prompts at a foundational model.”

At this stage, the goal isn’t to launch the ideal AI strategy on day one; the goal is to get started.

That doesn’t mean, however, that you have to throw caution to the wind. Nanu acknowledges the risks and warns, “More and more, we're seeing vendor outages. And at any point, these closed-source vendors can raise their prices from 10 cents per million tokens to ten dollars per million tokens – they can do that overnight.”

A lot of companies want to retain ownership of models, but those large proprietary companies provide something very valuable: Speed and quality. Your fine-tuned, self-hosted, open-source 7 billion parameter model is going to perform absolutely terribly until you have that 5,000-or-more data point set.

As a result, you should balance the power you’re giving these vendors with the ability to fallback in the event of outages or other issues. Nanu, for example, uses models from these providers but has backups to keep them going if anything happens.

“We build out our datasets to train our own custom adapters that we own, that we host, and that we can fall back on if those closed-source models ever go down,” Nanu says.

That safety measure even backs out to how they use the models when they’re up and running. “That's why, even though GPT-4o has that 127,000 token context window, we actually keep most of our prompt engineering and our work under an 8,000 token context window just so that we can easily fall back to our open-source models on our self-hosted infra,” Nanu explains.

There’s No Free Lunch

It’s easy to feel behind the AI wave, but everything is still new, and best practices remain unsettled. “There's no free lunch,” Nanu says, “And at this point, all the systems for AI are all constantly in development, and are brand new. The whole industry is still figuring out best practices.”

Instead of waiting for a playbook that can get you from A to Z, focus on building the foundational processes of your data pipeline:

1. Build a plan with goals
2. Scope out data sources
3. Map out data processing processes
4. Determine storage
5. Determine data flows
6. Build / connect / test
7. Monitor and observe

Throughout, treat your data pipeline like operational architecture, similar to the approach DevOps teams take to other foundational pieces of software infrastructure. While there’s a lot of excitement around AI, standing up an AI program isn’t something to build on the cheap. A data pipeline is infrastructure, and you need to build it knowing that it’s something to maintain and evolve over time.

The post Best Practices for Developing Data Pipelines in Regulated Spaces appeared first on Heavybit.

In episode 84 of o11ycast, Ken Rimple and Martin Thwaites welcome Maddy Montaquila, lead PM for .NET Aspire at Microsoft. This episode dives into the "magic" of Aspire, exploring how it streamlines the developer experience from frictionless app startup to telemetry-rich development. Discover how Aspire makes telemetry a first-class citizen, simplifying debugging and enhancing observability for modern applications.

The post Ep. #84, Maddy Montaquila on .NET Aspire appeared first on Heavybit.

In episode 39 of Generationship, Rachel speaks with Simon Willison, founder of Datasette and co-creator of Django. Simon discusses the surprising resurgence of blogging, his coining of the term “prompt injection,” the power of learning in public, and how he uses GitHub issues as an external brain to manage hundreds of projects. This quick-witted and humorous conversation offers a pragmatic look at leveraging today's tools for maximum productivity and impact.

The post Ep. #39, Simon Willison: I Coined Prompt Injection appeared first on Heavybit.

In episode 17 of Open Source Ready, Brian and John speak with Docker founder Solomon Hykes about his latest project, Dagger, and its mission to fix the pain points of modern CI/CD. Solomon explains why DevOps is due for a systems-level rethink and how AI agents are changing the way software gets built and shipped.

The post Ep. #17, AI Native Software Factories with Solomon Hykes appeared first on Heavybit.

In episode 9 of Platform Builders, Christine and Isaac sit down with Manny Medina, founder and CEO of Paid, to unpack the tectonic shift happening in software. From the rise of AI agents that autonomously perform tasks, to the death of seat-based pricing, and why ARR and SaaS metrics just don’t make sense anymore—Manny makes a compelling case that we’re entering a new era. If you're building, investing in, or working with AI-native companies, this is essential listening.

The post Ep. #9, Manny Medina Explains: What the F Is an AI Agent? appeared first on Heavybit.

In episode 38 of Generationship, Rachel Chalmers sits down with Heidi Waterhouse, co-author of "Progressive Delivery." They unpack what Progressive Delivery means as an extension of continuous delivery, focusing on the crucial role of user experience in software development. Discover fresh insights on user autonomy, the pitfalls of uninvited AI, and how to bring empathy back into the software lifecycle.

The post Ep. #38, Wayfinder with Heidi Waterhouse appeared first on Heavybit.

In episode 47 of The Kubelist Podcast, Marc and Benjie sit down with Grant Miller, Founder and CEO of Replicated. This talk chronicles the triumphs and tribulations of Replicated and the many entrepreneurial lessons learned along the way. Grant and Marc also unveil SecureBuild, a new offering for open source projects. Lastly they share their firsthand experiences using AI-assisted coding tools and how they’re revolutionizing productivity.

The post Ep. #47, SecureBuild with Grant Miller of Replicated appeared first on Heavybit.

On episode 16 of Open Source Ready, Brian and John sit down with Mitchell Hashimoto, founder of HashiCorp, to discuss his journey after leaving the company and his latest passion project, the open source terminal emulator Ghostty. Mitchell shares the accidental origins of Ghostty, his pragmatic approach to technology, and his thoughts on the current state of open source business models. Lastly, they explore the complexities of AI development and the trade-offs of foundation governance.

The post Ep. #16, Building Tools That Spark Joy with Mitchell Hashimoto appeared first on Heavybit.

General-purpose large language models (LLMs) are built for broad artificial intelligence (AI) applications. The most popular open-weight models, which make their core hyperparameters publicly available, are usually general-purpose models prepared with pre-training, an important practice in using deep learning to improve a model’s analysis capabilities.

Off the shelf, these pre-trained models can explain a wide range of complex topics, handle rote tasks to free up their human partners, handle customer service queries as a chatbot, spin up text generation for specific content needs, handle summarization of large blocks of text data, and more. While this will not be a full tutorial, this step-by-step guide, we’ll cover how teams can use the process of fine tuning on already-trained AI models to boost performance and provide better outcomes faster. Fine-tuning is generally one of the core processes in creating a custom LLM of your own.

How Fine-Tuning Improves LLM Performance for Engineering Teams

Out of the box, LLMs made for any task may not be suited to any single organization’s specific tasks. (Some teams will look to more purpose-built alternatives, such as Bidirectional Encoder Representations from Transformers, or BERT models, which use self-supervised learning, or other custom models.) There are other barriers for startups that may be interested in using LLMs: non-deterministic outcomes that introduce unpredictability, the high cost of accessing popular models and maintaining the resources to host them, and the rapid pace of change in the LLM landscape.

There are several popular techniques for improving what an LLM produces: refining the prompts you feed it (prompt engineering), introducing data from new sources to the LLM (retrieval-augmented generation, or RAG), and changing the training parameters of the model itself (fine tuning).

All three approaches will get you better outputs. (RAG in particular has gained popularity for providing ad hoc performance boosts for models.) However, only fine-tuning large language models delivers the sort of persistent behavior change that consistently channels an LLM’s power toward your organization’s purposes. A model customized to your startup’s specific task (and your customers’ needs) can help you outclass your competitors.

Next, we’ll cover how fine-tuning works, different approaches to fine tuning, how to do it (from model selection to output evaluation) responsibly, and which AI infrastructure investments make the difference in fine tuning effectively.

What is LLM Fine-Tuning?

Fine-tuning LLMs means conducting additional training on an already pre-trained LLM, like OpenAI’s ChatGPT, Anthropic Claude, META’s Llama family, or your own LLM internally to improve its performance. In essence, fine-tuning extends the training of an LLM beyond pre-training, tailoring it for specific domains and use cases on subsequent prompts, as opposed to more-transient methods like RAG.

The fine-tuning process can involve connecting your pre-trained model to one or more data sources (such as a knowledge base of your own data) via API to flow fine-tuning data to the model (as well as to connect the fine-tuned model directly to downstream applications). The fine-tuning process can benefit from embedding, effectively providing a starting point within your own data to which your fine-tuned model can refer. It’s also a good idea to automate time-consuming data preprocessing and other preparation tasks.

Specifically, training LLMs updates and refits your models’ parameters—the weights and biases it applies to a dataset during the training process as well as the preset hyperparameters (i.e., learning rate, epochs, batch size) that guide how those weights and biases are applied—to provide better, fit-for-purpose responses. LLMs tuned for a specific domain or use case tuned with an appropriate learning rate deliver faster performance and better, more relevant answers that ideally require fewer re-prompts.

Poorly done, fine-tuning can also introduce safety risks. Most popular LLMs have built-in guardrails to prevent their misuse; fine-tuning can compromise them. In a study, researchers from Princeton University, Stanford University, Virginia Tech University, and IBM Research found that fine-tuning enabled LLMs from Meta and OpenAI to offer advice on bomb-making, run scams, and participate in other criminal activities.

Such risks can be mitigated with thoughtful planning and deliberate execution. Real-world fine-tuning success stories aren’t hard to find, particularly for teams that go beyond using the largest large language models and consider going smaller:

• LlaSMol, a Mistral-based LLM fine-tuned by researchers at Ohio State University and Google for chemistry projects, substantially outperformed non-fine-tuned models
• At Harvard University, large language models with smaller parameter counts fine-tuned to scan medical records for non-medical factors that influence health found more results with less bias than advanced GPT models trained on a larger, more-general set of parameters
• Researchers at three Chinese universities developed LegiLM, an LLM fine-tuned to interpret and support compliance with data privacy regulations
• Sentiment analysis and other natural language processing (NLP) tasks are also popular use case for LLM fine-tuning

Key Approaches to LLM Fine-Tuning

There are several broad approaches to LLM fine-tuning, including full fine-tuning, parameter-efficient fine-tuning (PEFT), and instruction tuning or reinforcement learning from human feedback (RLHF). Each has merits and drawbacks, but at least one is likely to suit your organization’s goals and budget, which are important to consider, particularly if training your own LLM internally.

Full Fine-Tuning

Full fine-tuning is a method for fine-tuning LLMs that retrains all of your base model’s parameters. Full fine-tuning offers the most control over model outputs, which may be ideal for developers looking to execute obscure, domain-specific tasks. (In contrast, few-shot learning attempts to train models on extremely few data points.)

It is, however, very resource-intensive. Meta’s Llama 3.1 family, for instance, has between 8 billion and 405 billion parameters, depending on which version you use. Fine-tuning a 405B-parameter model would likely demand an enormous amount of expensive computational resources, of the kind conventionally provided by high-end graphics processing units (GPU), and would yield a model as large as the base model. Fine-tuning often aims to scale general-use models down for domain-specific tasks.

Full fine-tuning also carries some risk of overfitting, which can cause your LLM to overindex only on recognizing its previous training data and fail to perform well against future, net-new datasets. Overfitting can be a particularly significant risk when you’re tuning an LLM for use on smaller custom datasets. In those cases, a fully fine-tuned model can memorize training examples and behave erratically when presented with unfamiliar data. Catastrophic forgetting—a phenomenon where an LLM overwrites general knowledge learned in its original training—is also a possibility.

Parameter-Efficient Fine-Tuning

Parameter-efficient fine tuning (PEFT), in which only a subset of model parameters undergo fine-tuning, addresses some of the drawbacks to full fine-tuning. With most parameters frozen, PEFT demands less memory and compute resources, reduces the risk of catastrophic forgetting, and takes less time.

Low-rank adaptation (LoRA) is a common PEFT method. LoRA is a method for fine-tuning LLMs that involves injecting low-rank trainable matrices into each layer of a transformer architecture, cutting the number of parameters that need to be trained for a downstream task. Quantized low-rank adaptation (QLoRA) is an extension of LoRA that compresses base model parameters to reduce the amount of storage needed for fine-tuning.

PEFT comes with trade-offs. Fully fine-tuned models typically perform better, and they offer more-complete control. But when computing resources or domain-specific data are scarce, PEFT can be a useful alternative.

Instruction Tuning and Reinforcement Learning from Human Feedback

Instruction tuning improves overall ability to follow instructions by training LLMs on a labeled set of instructional prompts and correct corresponding outputs. It’s often used in tandem with reinforcement learning from human feedback (RLHF). RLHF is a method to fine-tune LLMs that “rewards” a model for correctly responding to questions and commands.

Together, these techniques make models more predictable and helpful. They’re also inherently complex, given their dependence on well-crafted instruction/response datasets and human input.

The LLM Fine-Tuning Workflow

Fine-tuning is a complex process. Build in enough time for preparing up front and iterating on the back end to get the most value out of it. You’ll make important choices along the way: base model, tuning method, number of iterations, and more. Be sure to align them with your needs, infrastructure, and capabilities.

Step 1: Choose a Base Model

Start by selecting a flexible base LLM with published parameters. Open-weight models offer access to the weights and biases you’ll need to tweak as you fine-tune for your own specific task, although their underlying training data, algorithms, and model architecture remain hidden.

Size and inference cost will be major considerations—startups may prefer models small enough to tune, train, and operate on a single on-prem or cloud GPU. But there are other decision factors: the quality, objectivity, breadth of the LLM’s training data, and its overall level of fit to your use case, as well as its carbon footprint and other sustainability factors.

Here are four popular open-weight LLMs. Each offers several size options:

• LLaMA by META
• Mistral by Mistral
• Falcon by TII
• Gemma by Google

>> Click to zoom in on this infographic

Step 2: Prepare and Format Your Dataset

Preparing data begins with collection and formatting, but data quality is non-negotiable. Whether you’re training on your own domain-specific datasets, synthetic data, or third-party data, you’ll need to make sure you have a high-quality dataset that is accurate, unbiased, free of duplicates, and well labeled. You may also need to tokenize your data (depending on your model) and split your dataset into training, validation, and test sets. LLM training datasets typically use JSONL format:

Well-labeled data

Poorly labeled, incomplete data

There are good, low-cost tools and approaches to help you prep and format your datasets:

• Label Studio, an open source tool for labeling a range of data types
• Snorkel, an open source Python library for weak supervision with label functions
• Custom scripts are also useful for data preparation

Step 3: Select Your Fine-Tuning Method

This is a moment to revisit the fundamentals of your fine-tuning plans. Do you need to improve the base model’s ability to follow instructions? How much experience with LLM workflows does your team have? Do you have the budget and resources to support full fine-tuning?

All these questions should guide your choice of fine-tuning method. Consider how your answers match up with the attributes of common fine-tuning approaches:

Full Fine-Tuning

A high-cost alternative (due to steep compute and storage needs) requiring a large, task-specific dataset, significant infrastructure (including powerful silicon, long training times, and significant storage), large-scale full-model deployment, high complexity requiring deep ML expertise, but allowing for high customization that can deeply change model behavior.

PEFT

Significantly lower costs than full fine-tuning, requiring a small-to-medium-sized dataset, can run on consumer-grade GPUs with faster training, efficient deployment size as only adapter weights are added, low-to-moderate complexity, and medium to high levels of customization with focused adaptation.

Instruction Tuning

Moderate costs requiring training and curated data, medium-sized data requirements for a diverse dataset, moderate infrastructure needs, deployment of the same size as the base model, moderate complexity that involves task curation, and a moderate amount of customization with general task adaptation.

RLHF

High cost that requires labeling, fine-tuning, and reward modeling, manual labeling requirements for data, typically involves distributed training with custom environments, deployment of the same size as the model or larger depending on reward models, very high complexity that requires training reward models plus human evaluations, and high levels of customization optimized for desired outputs.

Step 4: Choose Your Framework and Tools

Choosing the right frameworks and tools is another important consideration.

Ease of use and broad community support make ecosystems like Hugging Face, PyTorch, TensorFlow, or Replicate attractive to startups, particularly those using a PEFT approach. Combining tools like transformers with LoRA-based tuning methods streamlines development and reduces the barrier to entry and can integrate well with reinforcement learning techniques.

A basic LoRA setup snippet using Hugging Face’s PEFT.

More-advanced workflows might benefit from structured training frameworks like Axolotl, LangChain, Unsloth, or TorchTune, especially when scaling up. Built on top of Hugging Face libraries, Axolotl is also well suited to PEFT techniques.

Startups should also look at compatibility with experiment tracking tools such as MLFlow, ClearML, or Weights & Biases, and consider deployment tools like BentoML or OpenLLM if real-time inference is needed. The frameworks chosen should ideally match the team’s technical skills and integrate well with existing infrastructure.

Step 5: Train and Validate

Use standard model training loops with checkpoints and early stopping. Monitor:

• Training/Validation Loss: How well your model learns from training data and how well it generalizes to net-new data
• Overfitting Signs: Such as huge gaps between high performance on known datasets compared to significantly lower performance on net-new datasets
• GPU Utilization Issues: Such as low or spiky utilization that may indicate bottlenecks

Step 6: Evaluate and Iterate

Don’t stop at “it trains.” Evaluation is the key to refinement. For smaller datasets, some estimates suggests 3-10 epochs can be sufficient, but every model and every dataset differs. It’s important to consider your data batch size against the number of epochs you plan to run. Larger data batches will likely require more epochs for your model to “see” and process all the data in your set a sufficient number of times. However, running too many epochs can lead to overfitting, not to mention potentially overspending on compute to run excessive training cycles that actually diminish your performance.

Common Mistakes Teams Make When Fine-Tuning

Fine-tuning should be an iterative process with frequent oversight and feedback cycles and an eye toward ensuring the highest data quality as inputs. Here are some common mistakes teams make during the process:

• Using Poor-Quality or Inconsistent Training Data: A bad dataset can lead to overfitting or underfitting, as well as performance drops that ultimately, hurt customer experience
• Choosing an Overly Complex Model or Method: Overcomplicated fine-tuning can lead to loss of performance on generalized datasets, catastrophic forgetting, and higher costs if retraining is needed
• Skipping Human Evaluations: Without human evaluation, models can miss important subjective factors such as ethical safety issues or societal biases
• Failing to Benchmark Against a Baseline: Without a baseline, it can be difficult to assess performance or the impact of any changes made during fine-tuning
• Ignoring Iteration Cycles and Feedback: Skipping iterative feedback can lead to lower adaptability to new data and models preserving unwanted biases.

LLM Evaluation and Optimization: Measuring What Matters

LLM behavior can feel unpredictable. Fine-tuning may seem like black magic unless you have a sound evaluation plan. Start simple: Test whether the model actually performs your desired task better than the base. Use both quantitative and qualitative metrics, tailored to your use case.

No single metric will tell you everything you need to know about your model’s performance. There’s no shortage of tools and measures to help as you broaden your measurement and optimization plans. Just a few examples include Perplexity, F1 Score, BLEU/ROUGE/METEOR, LM Evaluation Harness, trix, and Hugging Face's Open LLM Leaderboard.

How to Fine-Tune Responsibly and Build Trust in Your Model

Most teams hesitate to ship black-box models into production, as they should. Responsible fine-tuning means thoughtful experiments, grounded in user needs and focused on recording important learnings you can use to iteratively improve your model’s performance.

Build a Tight Feedback Loop

To ensure optimal performance, it’s important to build in a feedback loop that monitors performance deltas across successive epochs and against any data variations. By making iterative observations, your team stands a better chance of identifying issues of data quality, compute bottlenecks, and other obstacles standing between you and a highly performant model.

Right-Size Your Infrastructure

Choose tools and systems that match your scale. Don’t spin up a tensor processing unit pod when a LoRA adapter will do. As your organization’s needs scale, compute costs may become a larger issue, particularly as you fine-tune against larger datasets run across multiple epochs. Teams running enterprise-level workloads will need to be aware of I/O issues transferring data across their compute hardware, CPU/GPU bottlenecks, memory limitations, or mismatched batch sizes that lead to latency or timeouts.

Bake in Safety and Guardrails

Rather than wait for compliance headaches and lawsuits to find you, it’s a good idea to proactively build in safety measures that ensure your fine-tuning leads to a model that is both safe and performant:

• Filter Training Data for Toxicity or Bias: Removing harmful or biased content from your dataset before fine-tuning helps make your model less likely to output toxic responses, and more likely to be fair and unbiased.
• Run Safety Evaluations Before Release: It’s a best practice to run iterative safety evals that can include, but aren’t limited to, standard benchmark evaluations, automated evals, and even adversarial red teaming to root out potentially harmful outputs.
• Use Output Filters, Rate-Limiting, or Human-in-the-Loop Systems: Output filters can be a helpful final safeguard against biased or harmful content. Setting rate limits caps maximum requests at a time and can prevent unauthorized attacks. And adding human safeguards to better label data and evaluate outputs can prevent unanticipated issues.

Understand and Embrace the Weirdness

At the risk of stating the obvious, LLMs are probabilistic, not deterministic. Generative AI can and does output unexpected responses, hallucinate, or otherwise do unexpected things. Yes, it can be a frustrating and novel experience for engineering teams to operate a seemingly unpredictable system.

However, aside from adopting the mindset that LLMs simply behave differently than a deterministic codebase, it’s not a bad idea to note unusual behavior as potential opportunities for improvement. Unexpected responses that recur could be a sign of gaps in training data or a need for additional training runs, for example. Teams that learn to expect the unexpected can apply those learnings to fine-tune a more-performant model.

The Real Payoff: Why the Infrastructure Around Fine-Tuning Matters Most

The most immediate win from successful fine-tuning is a more-robust model that performs better on any given day. However, the infrastructure, feedback loops, and team processes you build around it are also enormously valuable, particularly as the AI space changes and your team is expected to continuously adapt.

More resources that may be helpful to teams standing up AI programs:

• How to Create Data Pipelines: A starter guide on the basics of building key data infrastructure for AI programs
• How to Scope and Evolve Data Pipelines: An in-depth interview on best practices for evolving data pipelines over time
• The Data Pipeline Is the New Secret Sauce: A deep-dive into the hardware infrastructure and CI/CD-like data discipline needed for AI programs
• Machine Learning Model Monitoring: How to approach monitoring for models in production
• Machine Learning Lifecycle: Idea to Launch: A starter guide on how to approach standing up an AI/ML program
• DevToolsDigest Newsletter: A weekly newsletter that covers the most important news for software teams
• The Humans in the Loop: A biweekly newsletter that bottom-lines AI news for busy software teams

The post LLM Fine-Tuning: A Guide for Engineering Teams in 2025 appeared first on Heavybit.

In episode 8 of Platform Builders, Christine Spang and Isaac Nassimi are joined by John Melas-Kyriazi, founder and CEO of Standard Metrics. John shares his journey from VC to founder and how he’s rethinking investor–portfolio company collaboration. From spreadsheets to vertical SaaS, and now to AI-powered platforms, John dives deep into building software that’s truly purpose-built for venture capital.

The post Ep. #8, The Beauty of Vertical SaaS with John Melas-Kyriazi of Standard Metrics appeared first on Heavybit.

In episode 37 of Generationship, Rachel Chalmers chats with Latrice Barnett and Kanika Chander from Majority Collective about their mission to empower underrepresented founders. The conversation covers the practical benefits of fractional expertise, the importance of defining clear company strategies, and a thought-provoking exchange on the opportunities and pitfalls of artificial intelligence in shaping the future of entrepreneurship.

The post Ep. #37, Optimism with a Plan with Latrice Barnett and Kanika Chander appeared first on Heavybit.

In episode 83 of o11ycast, the Honeycomb team chats with Dan Ravenstone, the o11yneer. Dan unpacks the crucial, often underappreciated, role of the observability engineer. He discusses how this position champions the user, bridging the gap between technical performance and real-world customer experience. Learn about the challenges of mobile observability, the importance of clear terminology, and how building alliances across an organization drives successful observability practices.

The post Ep. #83, Observability Isn't Just SRE on Steroids with Dan Ravenstone appeared first on Heavybit.

In episode 15 of Open Source Ready, Brian and John chat with Adewale "Ace" Abati from Block about Codename Goose, an open-source AI agent, and the underlying Model Context Protocol (MCP). They explore how AI agents are revolutionizing developer workflows, the concept of "vibe coding" for rapid prototyping, and the future of AI in productivity and accessibility.

The post Ep. #15, Codename Goose and the Future of AI Agents with Adewale Abati appeared first on Heavybit.

In episode 7 of Platform Builders, Christine Spang and Isaac Nassimi dive deep into the world of CRMs and SaaS strategy with Steve Oriola, CEO of Unbounce. Steve shares his insights on the historical shifts in CRM technology, the critical role of data in modern marketing solutions, and his approach to leading companies through significant transitions and acquisitions. Learn how focused product development and a strong company culture can drive success.

The post Ep. #7, SaaS Before It Was Cool with Steve Oriola appeared first on Heavybit.

In episode 82 of o11ycast, Ken and Jess chat with Morgante Pell, the visionary behind Grit, an AI-powered agent designed to automate developer toil and technical debt. The discussion covers the evolution of AI in coding, the challenges of building and deploying AI agents, the future of combining code awareness with production awareness, and Grit’s acquisition by Honeycomb.

The post Ep. #82, Automating Developer Toil with Morgante Pell of Grit appeared first on Heavybit.

In episode 14 of Open Source Ready, special guests Mike McQuaid and John Britton join Brian and John to share the story of Workbrew. They unpack their journey in the developer tools space, the evolution of Homebrew, the inspiration behind Workbrew, and their perspectives on the future of developer environments.

The post Ep. #14, The Workbrew Story with Mike McQuaid and John Britton appeared first on Heavybit.

In episode 36 of Generationship, Rachel Chalmers speaks with Christine Spang about the past, present, and future of enterprise software, with a particular focus on AI. Spang shares her experiences building Nylas and discusses the practical applications of AI in business, as well as the ethical considerations surrounding its use. Lastly, they explore the social and cultural impact of AI, pondering its effects on human connection and the perception of reality.

The post Ep. #36, Solarpunk with Christine Spang appeared first on Heavybit.

In episode 6 of Platform Builders, Christine Spang and Isaac Nassimi chat with Brad Rigby about Canopy’s strategic shift from a niche tax tool to a leading accounting practice management platform. This conversation explores the challenges and triumphs of this major pivot, the nuances of building vertical SaaS, and how Canopy is leveraging AI to streamline workflows and make accounting tech essential.

The post Ep. #6, Making Accounting SaaS Sexy Again with Brad Rigby appeared first on Heavybit.

In episode 81 of o11ycast, Charity Majors and Martin Thwaites dive into a lively discussion with Hazel Weakly and Matt Klein on the evolving landscape of observability. The guests explore the concept of observability versioning, the challenges of cost and ROI, and the future of observability tools, including the potential convergence with AI and business intelligence.

The post Ep. #81, Observability 3.0-vNext-final-DRAFT with Hazel Weakly and Matt Klein appeared first on Heavybit.

In episode 13 of Open Source Ready, Thorsten Ball of Sourcegraph joins Brian and John to unpack the real-world engineering decisions behind language choices, like Microsoft using Go for the recent TypeScript compiler rewrite. Explore the importance of continuous learning in tech, the benefits of knowing multiple programming languages, and how AI agents are poised to reshape developer workflows.

The post Ep. #13, Why Microsoft Chose Go: A Deep Dive with Thorsten Ball appeared first on Heavybit.

In episode 35 of Generationship, Rachel is joined by Brooke Hopkins to explore what it takes to make voice AI agents reliable, robust, and ready for real-world deployment. Drawing from her experience at Waymo and her current work at Coval, Brooke reveals how testing and evaluation are the key to staying ahead in a fast-moving AI landscape.

The post Ep. #35, Wisdom with Brooke Hopkins of Coval appeared first on Heavybit.

In episode 80 of o11ycast, Ken Rimple and Jessica Kerr chat with Kent Beck about the emerging world of AI-assisted coding. Beck shares his experiences with "augmented coding," discussing the benefits, challenges, and the evolving relationship between developers and AI agents. They explore how these tools are changing software development practices and what it means for the future of coding.

The post Ep. #80, Augmented Coding with Kent Beck appeared first on Heavybit.

In episode 5 of Platform Builders, Christine Spang and Isaac Nassimi chat with Scott Mitchell, former CTO of Salesloft, about the challenges and triumphs of scaling a high-growth B2B SaaS company. Scott shares his experiences navigating hyper-growth, from tackling critical infrastructure bottlenecks to fostering a culture of adaptability. They also explore the evolution of sales technology and the impact of AI on the future of sales engagement.

The post Ep. #5, Maxed out on AWS, Still Falling Over with Scott Mitchell appeared first on Heavybit.

In episode 12 of Open Source Ready, Brian and John welcome Ron Efroni and Ross Turk from Flox to explore the world of Nix, a powerful package manager and more. They discuss how Flox simplifies Nix for developers and enterprises, the challenges of adopting new packaging paradigms, and the evolving landscape of containers and AI in software development.

The post Ep. #12, Exploring Flox and Nix with Ron Efroni & Ross Turk appeared first on Heavybit.

Emerging Questions in Global Regulation for AI and Open Source

The 46th President of the United States issued an executive order to prescribe “safe, secure, and trustworthy development and use of Artificial Intelligence.” About a year later, within days of taking office, the 47th President rescinded it. There remain disparate views across technology, regulation, and business on the future of AI, all the way up the chain to federal governments and prime ministers.

At the 2025 Paris AI Summit, leaders from the US and UK declined to sign the diplomatic declaration on “inclusive AI.” US representatives claimed the agreement was too restrictive. UK representatives claimed it was not restrictive enough. As nation-states increasingly regard AI development as a precious resource and a competitive advantage on the world stage, it’s unclear whether regulation can keep up with the pace of innovation. OpenUK CEO Amanda Brock explains how:

• World Leaders Don’t See Eye-to-Eye on AI Regulation: Western governments can’t agree on how much AI should be regulated, but do want to see accelerated competition against international competitors like China
• AI Governance May Come Through Tooling and Foundations: There are relatively new public and private initiatives to govern AI through tooling and foundations devoted to public good
• Regulation Always Seems to Lag Innovation: While different nations are at different levels of sophistication in thinking through concepts like data and open source, regulation itself may struggle to keep pace with fast-moving innovation in AI
• The Concept of IP Will Struggle to Stay Relevant: Intellectual property is seeming like an increasingly antiquated way to deal with technology that innovates at such a pace and is built on gobbling up copyrighted materials
• “Winning” Will Look Different When Innovation Outpaces Copyright: In the new world, the best way forward may be finding a way to compensate creators beyond the restrictions of a copyright

World Leaders Don’t See Eye-to-Eye on AI Regulation

“In Paris, the messaging around regulation, particularly from the US and China was that the bureaucracy that comes with the EU’s AI Act must stop,” says Brock. “There was a very definite sense across the summit of a vibrancy, a pro-innovation approach.”

“I think that the US Vice President was very clear that [the US doesn’t] want to see prescriptive regulation across the world, and I think that was very directed at the EU. The EU were focused on a code to explain how to comply with their cumbersome regulation and their conversation felt very old school and like they were watching from the sidelines.”

She suggests that the AI regulation should be more nuanced. “I think there is a sense that there'll be regulation to the extent, and only to the extent, it's needed. But the Vice President talked about their being a ‘new sheriff in town,’ who clearly does things very differently.”

Brock continues, “I personally happen to agree on the particular point that bureaucracy and regulation don't work here,” suggesting AI is a potential source of great innovation, of the sort that probably shouldn’t be stifled. “Whilst there may be a need for regulation in specific use case of AI and in particular across already-regulated sectors like mobile, healthcare, or finance, AI regulation in most countries on a more broad-brush basis will only exist in spaces where it’s absolutely necessary.”

More Resources on AI and Open Source Regulation:

• Article: The Power User's Guide to Open-Source Licenses
• Article: Understanding Business Models and Defensibility in Open Source
• Article: Emerging Legal Challenges for Open Source in the Age of AI

AI Governance May Come Through Tooling and Foundations

“A really interesting development is the launch of ROOST, a set of open-source tooling, which is all about automating governance in AI. The approach speaks directly to developers by creating governance through tooling, or ‘tools not rules,’ as they put it.”

“The initiative was launched by former Google CEO, Eric Schmidt, and META Chief Scientist of AI Yann LeCun, and includes a number of tools that have been donated by big techs at its launch.

Schmidt specifically clarified they were not talking about open weight models, but rather, open-source software when talking about ROOST tools. In the light of accusations about open washing in AI, they did this to be sure that there was no confusion around what they meant by the term open source in the context of ROOST.

Schmidt’s clarification was notable as it also aligned to a theme across the summit, the first global summit to really focus on open source, and this was that the term ‘open-source AI’ was really not used at all at the summit.

Open weights were discussed frequently, beginning to clarify what had previously and wrongly been referred to as “open-source AI” and making clear exactly what the component being mentioned was. “It’s quite possible we’ll see regulatory language evolve in the months to come to focus on open-weight AI rather than open source.”

“Interestingly, according to research from Tortoise and the Tortoise Index, France is number one in Europe in open-source AI. Research that OpenUK has conducted shows that France is by far growing fastest in open-source software in Europe while the UK remains Europe’s largest open-source contributor and number one in open source by far.”

“At some point, unless we have a shift in our policies in the UK, they'll catch up with us.” Brock suggests a definite shift in France, with significant support from President Macron leading to a number of pro-open-source policies alongside the setup of a foundation called Current AI, focused on public good AI, led by Martin Tisné.

Brock suggests that Current AI is about “building a public good, rather than open source. It has $500M in public/private funding at launch to focus on working out the data conundrum and the inputs and outputs that create AI. So there is a really interesting piece there where the French have absolutely grabbed what needs done with both hands. They're looking to get to $5 billion in five years. That's a huge foundation.”

Regulation Always Seems to Lag Innovation

Brock offers that data governance remains a core challenge in AI, particularly as some nations don’t have as well-developed an understanding of data and openness as the UK. “Forgive me for being blunt, but the UK has a much more sophisticated understanding of data and what ‘open data’ means than the US.” Suggesting that the differential between thinking about and regulating different data types may contribute to the confusion, she agrees with Tisné that a focus on data is essential to resolving the challenge.

“It's not an easy thing to fix. You've got a landscape or different forms of information in the input data used by AI: Numbers, words, music, imagery, and so on. All going in and out of the AI, each with different rights attached. And it depends on whether you’re looking at rights based on contracts, or at individuals' personal and privacy rights, or regulatory rights how these apply.”

“You might be looking at confidentiality and contracts, data privacy in a regulatory regime, or healthcare information that's never going to be public. Some of this information isn’t in the legal category of data.”

“You have to understand what kinds of information you are talking about and you have to understand the legal rights. When we talk about data in the context of AI, it is nothing like software, and the term ‘source’ as in open source, becomes a complete misnomer and confusing.”

“We shouldn't be talking about ‘open-source data,’ but open data, because there is no source in it.” Brock notes that intellectual property regimes around data are also different. “And when we look at information or content used to train AI it’s broader than both software and data in their legal meanings, differentiating ‘open data’ from ‘open-source software’ is a big conceptual shift that doesn’t fit with existing software licensing structures used by Wikipedia or Creative Commons.”

“We really need to look at this landscape of rights, apply it to the use cases and then understand what the data or information challenges are. I think that there's been a disconnect because the leadership in the area of open data is European, not American, and we've not managed to pull that group together with the AI context. I suspect that that is the next step: to sketch out that basic landscape and start to build understanding for people.”

We really need to look at this landscape of rights, apply it to the use cases and then understand what the data challenges are.” -Amanda Brock, CEO/OpenUK

The Concept of IP Will Struggle to Stay Relevant

Brock offers that a holistic, top-down approach to classifying an endless sea of data may be needed. “Thinking about the data used at scale, ‘inputs and outputs’ is probably the term we have to shift towards, which includes not only what we would traditionally define as ‘data.’ This is bigger.”

“It is going to be a bit alien to software engineers, because it has a different set of legal rights. Personally, I think our intellectual property regime is no longer fit for purpose. I think it's been heading that way for a long time and that the AI boost of the last couple of years has provided the final nail in its coffin.”

“In the case of the software industry, I’ve always believed that open source is actually a correction of the application of copyright to software. We had collaboration in software from the get-go, but then lawyers applied copyright to software, meaning licensing was required to be able to use another’s code.”

“This led to people building deep war chests of copyright, silos in usage, and of course meant copyrighted code became extremely valuable. Without that you wouldn't have a few big companies owning our digital infrastructure. Without this application of copyright we would all have been collaborating on software development.”

“Arguably this is the way it should’ve evolved. Had that happened, there would be no need for open source, but effectively, open source at its heart corrects that application of copyright and returns us to the pre-copyright days of collaborative creation of software and de facto standards.”

“I view the actions of the open-source movement as a correction of intellectual property’s mistakes.” Brock suggests that the current point in time, with friction regarding the use of publicly available copyrighted works for text and data mining, sees content publishers and intellectual property holders trying to apply old-world sensibilities that aren’t keeping up with modern innovation.

“Winning” Will Look Different When Innovation Outpaces Copyright

When it comes to training AI and text- and data mining, “What we're seeing from the European Commission is this concept of building a license to use with an opt-out and having a registry of people who've opted out that you have to check against. Despite any new tools, that is just not going to be workable in the long term.”

“It can never be more than a stopgap and even the EU admits this. You might try to go down that route because you don't know what to do but ultimately it’s avoidant. It’s no more than a sticking plaster that will ultimately fail. The real issues will have to be dealt with.”

“I think in the same way for software, we have to use licenses because we have places outside of the US that don't have public domain as a concept. So you need to have a license to use copyrighted code. You need to have a license because copyright applies. If we say that that was a mistake of history and you took copyright away, there would be no need for that license.”

“I think we will shift gradually towards that because the concept of building the infrastructure, even with smart tooling, that can manage that licensing regime is just unthinkable. It’s just never going to work and it's also going to be a massive inhibitor to innovation particularly when other countries that are more successful in AI are taking a different approach such as how the US approaches fair use.”

“When it comes to managing content usage as an AI input, I believe that the winner will be whichever regime works out a way to reward the creators of content that doesn't involve licensing and royalties but enables their future to be one where content creators survive by having different business and funding models that are not IP dependent for their content provision to continue.”

The post Regulation & Copyrights: Do They Work for AI & Open Source? appeared first on Heavybit.

In episode 4 of Platform Builders, Christine Spang and Isaac Nassimi interview Ray Zhou, co-founder of Affinity, about his journey building the company, the evolution of the SaaS market, and the transformative potential of AI. Ray shares insights on scaling a business, the importance of focusing on the customer's problem, and how SaaS companies can adapt to thrive in an AI-driven future.

The post Ep. #4, Building Affinity: From College Dropout to SaaS Leader with Ray Zhou appeared first on Heavybit.

For Data Pipelines, Planning Matters. So Does Evolution.

A data pipeline is a set of processes that extracts, transforms, and loads data from data sources to destinations in the necessary format. While data pipelines have been in use for years in storage and analytics use cases, they’re becoming increasingly important for AI use cases, and will likely become even more crucial as organizations look to deploy AI agents that are dependent on high-quality, up-to-date data.

Development teams looking to stand up their first data pipeline will be best served by scoping their initial project against clearly-stated business goals. Over time, it’s also a best practice to understand how your tech stack will need to evolve as your team and your data corpus expand to avoid piling up technical debt. Data expert, DAGWorks co-founder, and Hamilton and Burr co-creator Stefan Krawczyk explains.

Working With Data Means Constantly-Shifting Terrain

Some important concepts to keep in mind include:

• Datasets changing over time: Unlike codebases, datasets shift as inputs change, and in growing companies, data to be managed frequently expands into larger datasets
• The need for standardization: As organizations and their datasets grow, the potential for changes and anomalies also grows
• The need for observability and quality checks: Over time, organizations will find it increasingly important to have visibility into data quality and potential issues

Krawczyk suggests that developer teams newer to data management should understand that managing a codebase (which remains largely static) differs from managing datasets, which change over time. “This really brings another dimension to understanding what's happening in terms of managing pipelines that feed into data, AI, and ML applications.”

The co-founder notes that many data projects start small–which is the best time to take a first-principles approach. “While there are many ways to author a pipeline, enforcing standardization can definitely help you. And you need to instrument for observability so that you can understand what's going on.”

The co-founder notes that as orgs grow, their internal datasets may power more operations, including machine learning models or customer-facing AI chat. “If you don't understand the connection between [your data and your programs in production], that's where you really run into issues.”

“In general, code and data are those two dimensions that change at different rates. And this is where it becomes important to understand that you need to build different mechanisms for introspection, quality checks, and that sort of thing to make [your data pipeline] work over time.”

Beware of Non-Deterministic Outputs and Scaling Pains

Working with AI/ML programs? It’s a good idea to:

• Set Benchmarks Early: AI models are infamously non-deterministic, so having metrics aligned with business goals from the outset reduces variability
• Instrument for Scale: Understanding that a data pipeline will become key infrastructure to operate over time, it’s best to not over-engineer beyond your team’s needs
• Understand and Plan for Change: Smaller startups need to balance shipping fast vs. implementation costs, while enterprises will take compliance into account

The co-founder notes that with the increased use of LLM models, which may give different responses even to seemingly identical data inputs, teams are best served by proactively determining success metrics aligned with their business goals. “A lot of people get caught up in these data changes. Then your boss asks you: ‘Why is this output different?’”

The co-founder notes that upstream changes can affect an entire system, which is why teams should have some measurement of business metrics, a “macro” metric, in place to gauge overall health. Without an overall business metric to guide decision making, trying to instrument for every single possible data discrepancy can lead to alert fatigue because it’s hard to gauge the importance of a particular “micro” metric.

“As long as you ensure you're measuring against business metrics, that’s how you can gauge your program’s health: Regardless of what’s going right, what is actually critical?” The co-founder notes that bringing in observability typically happens when issues arise, but being successful with data pipelines is about anticipating shifts, among data, customers, or even regulations.

“How confident can you be that when the policy team at your airline company changes something, it won’t impact your chatbot? You don't want it to hallucinate and say things that you’re going to be liable for, the way Air Canada’s chatbot did. And so the connection between that policy data and where it's used becomes critical. At a startup, that's relatively easy, because small teams are all in the same room, but at larger companies, you have different orgs in different places.”

Different companies at different stages will tend to have different problems. Larger orgs will often have more-complex problems that involve governance and customer privacy issues, as well as issues specific to their vertical. The co-founder suggests that the challenge of managing data pipelines has a lot of depth and variation across company size, vertical, and industry policies, with very few one-size-fits-all solutions.

More Resources on Data Pipelines:

• Article: How to Create Data Pipelines
• Article: The Data Pipeline is the New Secret Sauce
• Article: The Role of Synthetic Data in AI/ML Programs in Software

How to Approach Building a Data Pipeline From First Principles

For those starting out, it’s a good idea to:

• Avoid Too Many Assumptions: It’s important to inventory all data inputs and outputs and understand the impacts of upstream and downstream changes
• Standardize Like an Engineer: Your pipeline will become another piece of operational infrastructure, so the earlier you get organized, document, and plan, the better
• Start Simple: As mentioned above, it’s a good idea to avoid the overhead of over-engineered support systems before your pipeline itself matures

“When building a new data pipeline for the first time, the key thing to remember is: You don't know what you don't know. You need to not only understand the inputs to the data, but you also need to understand the outputs. And it will depend on your context and use case. If you’re an e-commerce company recommending clothes to shoppers, you'll look at the output of recommendations vs. the input of what customers actually brought.”

The co-founder suggests that having a software background and an operations mindset can be helpful, and that building a structured process of metrics and checks will be a great jumping-off point on which to build. “If you have that background, you can standardize how changes are made and updated and applied. Which lets you add in specific things in continuous integration (CI) or other quality assurance checks and balances.”

“All the spaces in the industry seem to be moving in this direction, because we simply have more engineering to manage. Maintaining these different moving pieces is ultimately a software engineering problem. So I would recommend starting simple, and don't over-engineer. You should be asking, ‘If you have a system and process that makes it easy for you to run tests and checks (and not only for the code) then does the data also meet the shape that I expect?’”

“A few years ago, there was hype around data contracts, which are just someone writing tests to make sure that: If I'm reading from the customer database and I'm going to consume or transform that data, or use it for recommendations or to input into a chatbot, that someone should be writing some sort of expectation check at some point in the pipeline. In the context of software development processes, this is potentially akin to a CI system, which you can use as a scaffold to add in more checks and balances as your needs grow.”

When building a new data pipeline for the first time, remember: You don't know what you don't know. You need to not only understand the inputs to the data, but you also need to understand the outputs. And it will depend on your context and use case.” -Stefan Krawczyk, Co-Founder/DAGWorks

How Teams Typically Start Their Data Pipeline Projects

Common first steps for data pipeline projects include:

• Identifying Data Sources and Destinations: Start with your sources, your processing needs, and your destinations and map around them.
• Pipeline Components Will Evolve: You’ll end up versioning components in your pipeline as your needs change (and as those individual products change)
• Scope Auths and Permissions: A significant portion of operating a successful pipeline is ensuring the parties that need the data can actually access the data

The co-founder notes that while much of the data management landscape is Python-based due to tooling primarily being in that language, more teams are using alternatives like TypeScript or JavaScript, but technology choices shouldn’t trump best practices.

“The classic case of implementing a pipeline is starting something that is ‘offline,’ meaning you have a production database, from which you need that data to power some sort of data-driven or AI/ML product. The ‘classic’ way you’d do this is taking a snapshot of that database, then dumping it into a data store like an S3 bucket. But, most data these days can fit into a single machine.”

“So then, you need to spin up some machine, which could be Kubernetes or an EC2 machine, or maybe even on AWS Lambda. There’s many ways to do this, considering computation may take too long, requiring you to script something in Java or Python or Node to transform it and output it elsewhere.”

“And that other place could be what we typically called our data warehouse, or it could be another data store, a vector database, or just a regular Postgres database. I think most developers are thinking about that full stack: They have an app, which then talks to a database. But all the data that goes into the database potentially needs to be accessed in a serving context that is not from your web API.”

Full-stack app development teams need to come to terms with data and schema being a commitment unto themselves, requiring documentation and versioning like an API. They’ll also need to manage encapsulation and authorization issues, as the majority of the time, data management is about enabling the right teams with access to the data they need.

GenAI Projects and Data Freshness Requirements

As more teams pursue GenAI projects, data pipelines are changing by:

• Being More ‘Online’: Rather than serving data from a standard database, more GenAI products require pipelines to serve live web requests
• Potentially Requiring Up-to-the-Minute Data: Depending on their use cases and customer needs, some GenAI products require a high degree of data freshness
• Potentially Requiring Different Data Update Treatments: Teams should scope data freshness needs and not over-engineer if data doesn’t constantly need to be refreshed

“Maybe I'm pulling context from [company] data, or pulling data from a document, I need to transform it, send it to an LLM, get a response back to some of the ‘What-Was-Traditionally-More-Offline’ stuff. This happens more for data with respect to machine learning (ML). It’s turning into more of an online process with GenAI, within the context of things like web requests.”

The co-founder notes that teams building new GenAI products need to understand their requirements for data freshness to build successfully. For products that don’t urgently require data with up-to-the-minute freshness, teams might consider offline processing systems, which are easier to maintain due to having fewer moving parts. For products that require constantly refreshed data, teams will need to look at alternative implementations to ensure their data flows and is received in a timely manner.

“So there are going to be levels of complexity. First, you’ll want to take that step back to ensure you understand your data freshness requirements. If your requirements are high, you might look into scheduling and cron jobs as an approach to pull, transform, and push your data to its destination.” The co-founder recommends not skipping ahead directly to a high-maintenance online streaming system for data pipelines unless product and customer needs demand it.

The Most Common Data Pipeline Mistakes Today

Some of the most common mistakes teams make with data pipelines include:

• No Benchmarks Tied to Business Goals: It’s difficult to succeed without a clear understanding of ‘what matters’
• No Standardized Authoring: Without standardized authoring, it’s not clear what effect future changes will have
• Technical Debt: Without standardized authoring, and over time as projects change hands, previous coding and tool choices can hinder future projects

The co-founder suggests that one of the most common mistakes that teams make with their data pipeline projects is not setting up a series of metrics and benchmarks from the start. “If you have no way to measure the impact, you end up in a ‘management limbo’ that leads to overpromise and underdelivery. (And if you don't know what success is, that's not good for any kind of project, regardless.)”

Another common challenge teams face is the need to ship quickly while lacking standardized methods to author data pipelines, leading to painful technical debt over time. “Depending on how you author your project, it can end up becoming very hard to understand. When you make a change to your tools or processes, what’s the impact?”

“So for smaller teams, you’ll see issues along the lines of a product developer changing the production schema upstream, not knowing that this column and the values in it actually are used downstream. But as teams get larger, if there’s an issue with the schema or a range of values, maybe I can go in and fix it, but I don't know whether someone else actually had dependencies related to that change.”

The co-founder suggests that not having an understanding of how data flows is often the cause of trouble. “If you don't have the flexibility for your level of scale and SLA needs, that's where the problems start, since you basically don't know what to change, or how to change it, without knowing the impact of making any changes.”

“When you're building pipelines, you're effectively stitching together computation in various ways as part of various processes. One processor takes some data in, transforms it, puts it into another process, and so on. This is what should happen logically. But, usually the way that teams get into technical debt is that they actually couple how they execute things with logic.”

“Here’s an example: Let’s say you started with Lambda and you set up your pipelines to all assume that you're running on Lambda. That will mean a certain way of authoring things. It’s arguably great for independence, but it brings in other challenges. Alternatively, if you set things up to run on PySpark, all your jobs might be written in PySpark SQL, but you might find yourself having to switch to more of an online process. This is where you can get a lot of technical debt: From which infrastructure you choose versus how it's implemented.”

When you're building pipelines, you're effectively stitching together computation in various ways as part of various processes. The way that teams get into technical debt is that they actually couple how they execute things with logic.”

How Lineage Tracking and Modularity Reduce Tech Debt

Teams can reduce tech debt for their data pipelines as they scale by:

• Building Traceability and Lineage Into the Process: Knowing the origin of data changes is crucial for triaging downstream issues
• Simplifying Metrics Into Dashboards Built Around Business Goals: It’s a good idea to set up reporting that filters out operational noise and focuses on business outcomes
• Using Modular Frameworks That Scale: Modular frameworks that scale well with different dataset sizes and operational contexts incur less debt over time

“My team is working on Hamilton and Burr, two Python-based open source frameworks. The idea behind both of them is that if you can draw a flowchart for your data pipeline, you can basically use one of the frameworks. The latter is about conditional branching and dynamic looping in the flow charts, particularly for teams working with agents; the former is about using transformation processes for time-series forecasting, retrieval augment generation document processing pipelines, or customer data to transform for ML models.”

“Both of the frameworks make things really easy to use and test, without you having to engineer for it. You can always see the lineage or tracing: How data flows as a flow chart so you can then see how things progress. It’s about helping you structure your code in a way that helps you understand the kind of connection that exists between different transformations.”

“This means one less thing to engineer for when going to production, because you can understand how this column in this table was created, or what this output meant and what was upstream.” The founder notes that going zero-to-one with data pipelines is relatively easy, but zero-to-N can become cumbersome quickly with multiple authors and needs for debugging and observability, which can eventually require a total pipeline rewrite.

“With our frameworks, we’ve built a lot of platform hooks to enable the ease of observability and data collection. They also come with self-hostable UIs to have an out of the box way to see how things operate, with the idea being that we can slowly start to consolidate and provide a single pane of glass that is a better experience: Having several different data quality tools, data monitoring tools, a catalog to see lineage and tracing.”

The co-founder notes that his efforts were informed by his own experience managing platform teams to avoid technical debt. “Ideally, people shouldn’t have to rewrite much logic to swap out where code runs, which leads to less technical debt and more ability for your data scientists or ML teams to collaborate because you can define better boundaries. The decoupling makes your projects more modular, which means you can run things in different contexts [like Jupyter notebooks, PySpark jobs, FastAPI].”

The co-founder suggests that as GenAI has seen greater adoption, the market has greater demand for responsive data pipelines that teams can launch quickly and tweak iteratively. Teams that don’t have to re-instrument and can quickly understand where their data issues lie see less operational burden. “But you will need to build up some sort of view of your data in terms of what the distribution is, and whether you’re getting the business outcomes you want.”

How to Think About Data Pipelines for Building AI Agents

Teams considering agentic projects should first consider:

• Getting Data Management Processes in Order: To run successful agentic programs, teams need to fully understand data inputs and outputs
• Being Able to Fully Scope Simple Workflows: Before diving into agent building, it’s important to be able to fully sketch out and understand the steps of the human workflow you want to replicate
• Iterating Gradually: AI agents given the power to make workflow decisions will need to be managed carefully

The co-founder suggests that approaching the building of agentic products should also start from a first-principles approach that starts from inputs and outputs. “This is why I think there has been a larger focus on tracing. If you think of distributed tracing, most people are using one of the foundational model providers, so that's an API call.”

“And so, you need to instrument the API call, but then you also need to instrument what's upstream and downstream.” The co-founder suggests that teams that haven’t figured out the basics of data management may struggle if they rush to build agents immediately.

“You really need to understand your data and what's going on with it before you can fully automate it. And so, for agentic-like projects, we’re often referring to using an LLM with tool calling. Here’s an example of a very specific agentic workflow: If a finance person works in accounts payable, they're always getting a certain series of PDFs. They always need to extract certain aspects of those files. Very deterministic stuff.”

“So for a PDF, we’d first target the extraction of various properties you would need from that file. In such cases, I would recommend focusing on explicit, clear workflows, in the same way you’d write out instructions for a human being on what to do.”

“And as your process gets stronger, and you understand what’s going on while measuring the business outcomes and metrics, this is where you can start to build in more complexity. And then, maybe you’d start incorporating agents that dynamically make decisions on what action to take next. (I teach a course on topics like this, so feel free to reach out with questions.)”

As your process gets stronger, and you understand what’s going on while measuring the business outcomes and metrics, this is where you can start to build in more complexity. And then, maybe you’d start incorporating agents.”

Final Thoughts and Best Practices for Data Pipelines

Some overall takeaways for building data pipelines and managing data for agentic;

• Don’t Get Wrapped Around the Axle: During the implementation process, it’s common to overindex on edge cases
• Don’t Try to Fit Everything Into an Agile Sprint: Data pipelines likely won’t fit into a traditional Agile sprint, and will require additional maintenance and monitoring
• Get Working Prototypes and Workflows Before Investing in Tools: Before buying the shiny new data product, it’s a good idea to fully scope out your use case first

The co-founder rattles off a list of final best practices for teams to consider: “For any business project, you need to have clear business metrics of the value your project will bring to help orient your first version against expectations. Also, it’s generally a good idea to never promise 100% of all the things we’ve discussed here, as they probably won’t fit into an Agile sprint context.”

The co-founder suggests that a common pitfall is how easy it is to find use cases for which their scoped data pipeline should theoretically work at 100% efficiency. “Where most people spend most of their time in building and bringing these things into production is working on those edge cases and figuring out: How do I transform the data in the correct way? Or, how do I tweak the prompt? Or, what do I need to observe? What are the extra guardrails I need to add to make sure we're providing the customer experience we expect?”

The co-founder reiterates: Data pipeline projects often don’t fit neatly into Agile sprints. “It’s very easy to build a prototype, but harder to productionize it. You need something more of a six-week kind of sprint model, where you allow for not only building a prototype, but also lots of data collection to really understand and measure what the pattern of edge cases is. Basically: What is this product getting wrong most of the time? And, how can we fix it?”

The co-founder notes that these decisions should precede technology choices. “You’ll notice I haven’t mentioned alternatives like vector databases yet. Because that's really about technology being a means serving an end. You need to really understand the failure modes first to really understand what solutions you need.”

It’s also important to realize that data pipelines will become part of your operational infrastructure. “This type of project is not a one-and-done. In traditional software development, it's very easy, within sprint models, to call a feature ‘done’ and leave it as is. You’ve written your tests, you deployed, it’s done. That's not true here, since you need to continue to monitor things, especially your LLM calls.”

“For LLM calls, you need to ensure that you are providing time in your engineering schedule for maintenance, looking at data, and ensuring that things are working. This is something of a carrying cost that is a little different than a regular software engineering delivery model. Overall, a data pipeline project will probably take you a little longer than a standard Agile sprint.”

The post How to Properly Scope and Evolve Data Pipelines appeared first on Heavybit.

In episode 34 of Generationship, Nathen Harvey brings data, humor, and heart to a conversation about AI, DevOps, open source, and developer experience. He and Rachel dive into how AI is influencing software engineering, the role of platform engineering, metrics for assessing performance, and broader reflections on engineering culture and career growth.

The post Ep. #34, Together with Nathen Harvey appeared first on Heavybit.

In this episode of Platform Builders, Christine Spang and Isaac Nassimi chat with Ben Rubin about the evolution of software development, from the early days of the internet to the current age of AI. They discuss the impact of AI on B2B SaaS, the challenges of building platforms in a rapidly changing landscape, and the future of user experience. Ben shares his experiences and insights on how developers and businesses can adapt to thrive in this new era.

The post Ep. #3, Building Platforms in the AI Era with Ben Rubin of Verify appeared first on Heavybit.

Why Synthetic Data Matters for Software

Running AI in production requires a great deal of data to feed to models. Reddit is now contracting with Google and OpenAI as part of multimillion-dollar training deals with each publisher. OpenAI transcribed millions of hours of YouTube videos despite dubious permissions. Meta considered buying the publishing house Simon & Schuster.

However, not all data can or should be made freely available. In highly regulated spaces, teams need to protect customer privacy while still using high-quality datasets. In such cases, teams have a few options. They can attempt a self-hosted, closed-circuit model using real data, but incur the risks of leaks and breaches. Alternatively, they can use data that is either partially redacted or synthetic, the latter offering most of the completeness of real data without compromising user privacy.

Adam Kamor is a veteran software developer and engineering lead with more than a decade of experience building software products at Microsoft, Tableau, and the synthetic data startup he co-founded, Tonic. In this article, he explains how synthetic data helps software teams by:

• Protecting Personally Identifiable Information (PII)
• Keeping Teams in Compliance
• Saving Budget on Inference Costs

How to Use Customer Data in Your LLM Product

Kamor suggests that data management for software teams building products with LLMs isn’t as straightforward as data in, data out. “The problem that engineers face is they may want to send customer data to LLMs to provide a certain experience to customers. But they need to be aware of what, contractually, and legally, they can and cannot do in terms of customer data.”

The co-founder points out that, for compliance purposes, using customer data requires teams to be aware of these important considerations, particularly if they go the route of using a third-party LLM:

• Your Sub-Processors: The connection between your data and your LLM is not always a straight line. Which companies and products form that supply chain? What do they do with data as it moves along the supply chain?
• Compliance: Many customer contracts require updates when there is a change in how and where data is sent. In highly regulated spaces, there are also regulations such as HIPAA and GDPR.
• What are LLMs doing with the data? Your customers, and the regulations in your space, may partially or completely limit the kind of data you can and can’t share with LLM providers, especially with regards to third parties using your customer data to train their models.

In other words, teams that work with customer data, particularly in highly regulated spaces, often start from a problem space of needing relevant, high-quality datasets while remaining compliant with privacy policies. They need to get to the end result of having performant, well-trained models that give accurate, customer-focused responses without privacy risk.

Engineers may want to send customer data to LLMs to provide a certain experience to customers. But they need to be aware of what they can and cannot do in terms of customer data.” - Adam Kamor, Co-Founder Tonic

Increasing Data Quality and Model Performance, Decreasing Risk

Kamor suggests that teams looking to work with large amounts of sensitive customer data have a few options:

1. Host Your Own LLM Privately

Self-hosting would let you safeguard all the data you use to train that model internally. Ideally, hosting an internal, closed-circuit model helps mitigate security, privacy, and regulatory issues, assuming your own model doesn’t get compromised. It should be noted, however, that in these early days of AI/ML, that while deploying and operating your own model may lead to strong competitive advantage, requires considerable expertise from MLOps teams, and can be a costly challenge in itself due to misaligned incentives.

2. Work With Data to De-Identify It, or Use Synthetic Data

For many organizations, particularly resource-strapped startups, self-hosting a model may not be practical. An alternative is to work with your data set, either de-identify sensitive data or use synthetic data.

You can de-identify private data by redacting some or all sensitive fields within your dataset. Typically, developers de-identify data by tokenizing (replacing sensitive data with unique, non-sensitive tokens), hashing (using cryptographic functions to convert sensitive data into unique hash values), or masking (obscuring sensitive data with fictitious data).

However, new research shows that de-identifying data is often not enough to protect user privacy. A study found, in an experimental healthcare setting, that “de-identification of real clinical notes does not protect records against a membership inference attack.”

In this form of attack, adversaries get the patient records of a target person and use that data to find out whether that target person’s information was in a given model’s training dataset. If the model is used to identify patients with sexually transmitted infections, for example, then mere inclusion in the training data could be damaging to patients.

In theory, synthetic data can address this risk while solving all the privacy, security, and regulatory issues already mentioned. With synthetic data, you can build an LLM’s capabilities without revealing anything private to the LLM – removing the possibility that the LLM could ever regurgitate anything private to your users.

The co-founder offers an example: “Let's say you want to train BERT summarization models, which are frequently used to identify the most important text from a specific passage to create a summary. This type of model is relevant to physicians to summarize a patient's medical history: When new patients come in, you can provide a summary of those patients for your records.”

“Next, you’d want to train your model on the medical histories of all the current patients in your medical system, and then use the model for inference on future calculations. But you do not want to train your summarization model on customer protected health information (PHI) because the summarization model might regurgitate that.”

“Any situation where you're training a model on the entire population (so you can use it for inference on individuals) is dangerous because the model might memorize things about individuals in the population. And inevitably, it will regurgitate it to individuals when it's being used.”

Benefits of Synthetic Data

Synthetic data is extremely useful when the data you want to use to train an LLM needs to remain private. Synthetic data is the best way to keep your customer data private without affecting the utility of the data itself, allowing you to train or finetune an LLM without compromising the quality of its outputs or the privacy of its inputs.

Protecting Privacy in Highly Regulated Industries

In highly regulated industries, such as healthcare and financial services, customer data is not only private but guarded by strict regulatory regimens, such as HIPAA for healthcare or the BSA in finance. HIPAA violations can cost organizations upwards of $2M.

Satisfying Legal and Contractual Obligations

The co-founder points out the need for legal and regulatory compliance as well. “So, how do I synthesize my data so I can still train an effective model and not miss my customers’ sensitive information...and also adhere to my legal contracts? Customers typically don't want you training models on their data, though sometimes there will be exclusions for when the data is synthesized or de-identified.”

Synthetic data lets you build an artificial dataset that reflects customer data without actually containing and revealing any of it. That way, you can build LLM products that work within the narrow problems customers actually have. All without touching their sensitive information.

Practical Considerations: Code Privacy, Inference Costs

The co-founder points out that synthetic data isn’t a silver bullet that protects proprietary code from being gobbled up by models, and that exposing proprietary code to commercial LLMs might not be that big a deal. “I don’t think code alone is that valuable, or that it’s necessarily a danger to a software business. The efficiency benefits you get from using AI coding assistants probably outweigh any of those risks.”

“If I gave out all the source code to [Tonic] to somebody, I don’t think it would hurt my business. It would take them a long time to figure out how to stand it up and maintain it. And it would take forever to figure out how to add new features. By the time they got all of this done, we’d be a year or so ahead. I’ve never been of the opinion that code itself is so proprietary and sensitive.”

For teams looking to curb inference costs, synthetic data isn’t necessarily cheaper or faster for running ML jobs. Organizations operating their own internal models might be better served using distillation: “If you're if you're concerned about curbing inference costs, then I would use the outputs of LLMs to train more-efficient BERT models, for example. If you properly train your your BERT summarization model, you can get super-cheap inference.”

For early-stage startups, the co-founder suggests that rolling your own models, a time-intensive process that can require considerable specialized expertise, might be overkill. “For a startup, there are worse things you could do than just pay for an OpenAI ChatGPT plan and move on with your life and try to get customer revenue. But if you're operating at a certain scale and you're genuinely concerned about inference costs, then go train more-specialized models and run them yourself, and you'll get your costs down without issue.”

“We've started doing that with some of our features. We're not using LLMs for all of our NLP tests. We’re using traditional BERT models and they're cheap. They run on CPU. They're great. But they take longer to develop and they require, frankly, more skill to develop correctly. Not the kind of thing some junior engineer whips up in 20 minutes with a clever prompt.”

For a startup, there are worse things you could do than just pay for a GPT plan and move on. But if you're operating at a certain scale and you're genuinely concerned about inference costs, then train specialized models and run them yourself. You'll get your costs down.”

How Modern Data Is Shaping the Future of Models (and Software Teams)

The co-founder notes that there is a great deal of press about the utility of “small” models, specifically that their tight training and tuning on a finite dataset seemingly makes them more capable for limited use cases. But small isn’t necessarily always the right move.

“I think the march to small models can’t just be because they're cheaper and faster. That won't be enough to justify them because the cost of LLMs will just continue to go down. Sometimes a bit at a time, and sometimes it’s massive drops like with DeepSeek R1. That alone is not going to be a strong enough driver.”

“These smaller, faster models need to have other benefits. They need to be able to go places that large language models can't go. Or because we only run on-prem and we can't make calls to clouds. There has to be some other driver to necessitate that, I think, because LLM costs will just keep going down.”

The co-founder concedes that the pace of change in the AI/ML space is incredible, and that standards for hiring software developers and data experts could evolve as AI assistants lower barriers to entry. However, he advises software leads to look for data specialists with fundamental knowledge that isn’t easy to fake on a test.

“When we hire data scientists, I want to see a mathematical seriousness to them. I want to see an understanding of statistics and probability and other complex math topics. The fundamentals are still important. The reality is that some jobs, which used to require PhD-type folks to do, can be done almost as well by an engineer that knows how to write code and good prompts.”

“But if you're doing serious work, you still need serious people that have the mathematical grounding to think about these things properly. Admittedly, maybe that sort of thing is less important than it was previously. It's all changing so fast, and my answer could change tomorrow. Sometimes it seems like there's a game-changer every couple of days.”

Takeaways on Synthetic Data and How to Think About Data Management

To summarize, the co-founder advises software leads to think about synthetic data first and foremost as a safer alternative to using potentially risky real data to train models. “Ultimately, synthetic data isn’t so much about writing safe prompts and chats with your LLM. What you’re trying to do is train your own model. So if you're trying to do that on potentially sensitive customer information, you likely need to remove customer PII first. That's the most important consideration for privacy.”

More Resources

• Article: Synthetic Data for AI
• Article: The Future of Coding in the Age of GenAI
• Article: The Data Pipeline is the New Secret Sauce
• Article: Enterprise AI Infrastructure: Privacy, Maturity, Resources
• Article: Guide to Synthetic Test Data Generation

The post The Role of Synthetic Data in AI/ML Programs in Software appeared first on Heavybit.

In episode 11 of Open Source Ready, Brian Douglas and John McBride sit down with AI expert Steve Manuel to explore the Model Context Protocol (MCP)—a framework that enhances how models interact with their environments. They break down why context-awareness is crucial for machine learning and how MCP is transforming open source AI.

The post Ep. #11, Unpacking MCP with Steve Manuel appeared first on Heavybit.

Heavybit is thrilled to be sponsoring Data Council 2025, and we invite you to join us in Oakland from Apr 22-24 to experience 3 days of cutting-edge technical talks from the brightest minds in AI & data, learn more at datacouncil.ai and use the code HEAVYBIT20 for 20% off.

Designing the Foundation Models Track

Tom Drummond: Hi everyone, my name is Tom Drummond.

I'm a co-founder and managing partner at Heavybit. We are one of the sponsors of this year's Data Council.

We're very excited to be involved and as part of that, we are talking to a handful of our track hosts to get a read on who they are, what they're putting on for everybody and give everyone a little preview of what they can expect to hear at this year's event in a couple weeks.

Today, I am talking to Dr. Bryan Bischof. You are the track host for the Foundation Models track, correct?

Dr. Bryan Bischof: Yes, that is correct.

Tom: I have to say, having looked at all the track host backgrounds, you have by far the most interesting background out of everybody.

Bryan: Interesting. Is it the Blue Bottle? Is that what got your attention?

Tom: It's one of the eclecticism. You've got a little blue bottle, a little fashion in there. You've got kind of all different aspects.

I feel like I should do a talk show host intro, ladies and gentlemen, it's Dr. Bryan Bischof, educator, scientist, author, world renowned, something or other.

Bryan: Yeah, I think that last one is probably the one I identify with the most. " Something or other."

Actually, I might just go ahead add that to my LinkedIn. Something or other.

Yeah, I've been really lucky. I've gotten to work at a lot of really exciting organizations, doing super cool stuff.

I've been very privileged to get to sort of apply mathematics to a bunch of real-world problems, not just in one space.

You know, I started out in time series stuff, making computers talk to one another, got to work on coffee stuff, which is a big passion of mine.

And then later, I got to work on logistics for clothes and you know, these days, working on sort of data stuff.

Tom: That's awesome. And so tell us, where are you today?

Bryan: At the time of this interview, I am a hedge knight, but in only a few days, I will be joining Theory Ventures as the head of AI.

Tom: Congrats. Yeah, that's awesome. Tomasz is great. So let's talk about your track, so Foundation Models. What did you want to achieve putting this track together?

Bryan: Yeah, so when Pete asked me to put together the Foundation Models track, one thing that first crossed my mind is cool, I'm going to have people talk about like training large foundation models and that lasted in my brain for about 15 seconds and then I realized no, that's horribly boring.

Most of the people that are coming to Data Council just won't find that content that exciting. Training large, like massive models, even if it's, post training or mid training, I just don't think that it's going to be that exciting for that audience.

Instead I put myself in the shoes of an attendee because I am almost like a perfect example of who would want to come to Data Council and so what would I want to see?

And so what I thought about is, okay, candidly, I often find myself wondering, what do the labs want me to know?

What do they want me to think and how do they want me to be using their models?

I've been really lucky. I've gotten to talk to folks at Anthropic and Google and OpenAI.

I've gotten to sort of like work directly with one of the speakers in my track, Nikunj, and I've really benefit from those conversations. I've learned a lot from talking to the API teams about how they think about their models and they think about the application layer.

So my sort of inspiration for the people that I invited is actually the exact opposite of what the track is called.

So my track is called Foundation Models, but my track is actually about the application layer and it is about what you should know about the foundation models if you're building at the application layer.

Tom: Awesome, okay, so tell everyone, if they come to your track, who are they going to be hearing from?

Featured Speakers & Topics

Bryan: Let's start at the top. We're going to talk about whether the model is the product or not. We're going to have Han (Han-chung Lee) from Moody's, he's going to talk about the fact that the model is the product and that is really where the value is being created.

Hamel (Hamel Husain) is going to argue against that point that the value is created outside of the model itself at the application layer and that's going to be sort of the aperitif for the rest of the track because then what we're going to get is we're going to get a parade of model builders who are very sort of both sided on this argument.

They're building the models but they're simultaneously going to be trying to explain to you how to use what they're building to create value. And this is obviously intentional.

My goal here is to help you think critically about where you fit into these things.

The next speakers are going to be from the major labs. So as an example, we're going to have Nikunj (Nikunj Honda) from OpenAI, he's going to be talking about the agent's API. This is a really big deal, and everyone needs to know about it.

Next going to hear from Ragho (Raghotham Murthy) from Meta. Now Ragho is going to be telling you what does the Llama stack look like. If you care about building local AI applications, you need to understand the Llama stack.

Moving on, we have Ravin (Ravin Kumar) from Google DeepMind. Ravin is going to be talking about LLMs as tools.

If you really believe that the value is created outside of the model, then you need to understand where the model fits in. I'm really excited to see Ravin's perspective on this.

And finally, I have to say the word multimodal or people just won't come, so we obviously wanted to have somebody from the really multimodal perspective and so we have Ethan (Ethan Rosenthal) from Runway.

Tom: That is a killer lineup. What's a good elevator pitch for the series? Like why should you come to the Foundation Models versus like one of the other six tracks that are going on?

Why Attend This Track

Bryan: Candidly, because, like, we all need to have a really clear understanding of our positioning relative to AI right now.

Whether you want to go and build a foundation model to drive new applications and capabilities or if you need to understand how to use the state-of-the-art capabilities in your application, candidly, like this is the right information from the horse's mouth.

You'll notice like those are some big brands and I'm not someone who's like super obsessed with brand, but I think in this case, when you've got a few players that are really driving the conversation and the progress, you need to hear their perspective, and this is going to be the most rich way to get that information.

Sure, you should go look at the API Docs, but I think this is going to be a real live in-your-face demonstration of what these positions are.

The one thing that you'll probably notice is missing from this track in a very obvious way is Anthropic. Where is Anthropic?

Let's say the word MCP into the microphone so that everyone is very excited and what I'll say to you there is the MCP talk is already on the internet and you can go find it today.

It was recently given at the AI Engineering Summit in New York. So go ahead watch that talk. Thanks to Swyx for organizing that talk.

That's where you can get the information on MCP. You can think of this track as the version of that MCP talk for all the other model providers and all the other pieces of the story.

Tom: So you're going to be touching a bunch on kind of agents and tool use and things like that?

Bryan: Absolutely, there'll be a lot of discussion of sort of like, how do you have agents in the loop, how do you think about, like, what are tools and how do you make your tools really valuable?

Bryan’s Own Talk: “Failure is a Funnel”

Tom: That's awesome. I actually noticed, I'm kind of curious if this is like connected, but you're actually giving your own talk on day one of the conference, right?

Failure is a Funnel in the Data Science and Algo's track, I think.

I feel like today not a lot of people apply kind of classic engineering rigor to building these applications.

Like is that kind of what you're going to be talking about in the Failure as a Funnel or is that different?

Bryan: Yeah, I wanted to really talk about something that I feel like I'm an expert in. So I obviously chose failure.

And so my thesis is that basically building applications using AI is much more like classical product development than people think it is.

And I don't mean in the sense of like the engineering road mapping, the engineering frameworks, engineering process, I actually mean product analytics.

And so where I think a lot of engineering teams, especially AI engineering teams are coming up short is the way that they integrate data science skills, data science analysis and thinking of these processes the way that we would traditionally think of sort of optimizing your product.

It sounds really banal to say whoa, whoa, whoa, take a break. Look at where people were dropping off and the use of your tool. But actually, it's surprising how rare people are making this connection.

Tom: I feel like most people are sort of stuck on, hey, I should just like reword my prompt a couple times or I should like change a couple things here or there, and it's trial by error.

Bryan: It's even deeper than that. You hear a lot of discussion these days about evaluation and I'm really excited about evaluation.

I've been pushing evaluation for the past couple years, but I think people are very limited in scope when they think about what that means.

They think of evaluation as "did it work, did it get the answer right?" But actually, evaluation is a multi-step procedure and one step at a time, you get closer and closer to the right outcome.

I'm going to do something very unusual in my talk. I can promise, I can't reveal the secret, but what I can promise is if you come to my talk, you'll see something that you've never seen before at a tech conference.

Tom: That is reason enough to be at Bryan's talk, 11:45, on day one, April 22nd, Oakland, Datacouncil.ai, come and get your tickets.

Use HEAVYBIT20 for a 20% discount. Look forward to seeing you all there.

The post Data Council 2025: The Foundation Models Track with Dr. Bryan Bischof and Tom Drummond appeared first on Heavybit.

In episode 2 of Platform Builders, Jamie Davidson, CEO of Vitally, joins Christine Spang and Isaac Nassimi to discuss the evolution of customer success platforms. They explore Vitally's journey from a customer data tool to a comprehensive platform, the strategic decisions behind product development and fundraising, and the impact of AI on the future of customer success. Davidson also shares insights on innovating customer engagement and building a platform that empowers both customer success teams and the wider business.

The post Ep. #2, Charging Customers was a Mistake with Jamie Davidson appeared first on Heavybit.

In episode 33 of Generationship, Rachel Chalmers is joined by Nicole Forsgren—developer productivity researcher and co-founder of DORA—to discuss how AI is reshaping the software development landscape. From coding with LLMs to evaluating trust in AI-generated outputs, they explore the future of developer experience and the enduring value of human intuition.

The post Ep. #33, Developer Experience with Nicole Forsgren appeared first on Heavybit.

Heavybit is thrilled to be sponsoring Data Council 2025, and we invite you to join us in Oakland from Apr 22-24 to experience 3 days of cutting-edge technical talks from the brightest minds in AI & data, learn more at datacouncil.ai and use the code HEAVYBIT20 for 20% off.

Defining a Clear Vision for the Data Science Track

Jesse Robbins: Hey, this is Jesse Robbins. I'm one of the general partners here at Heavybit. We are extremely excited to be sponsoring this year's Data Council.

And to kick it off, we are interviewing the track hosts, including Sean Taylor, who just joined from OpenAI. Welcome, Sean.

Sean Taylor: Yeah, thanks Jesse.

Jesse: So Sean, you are hosting the data science and algorithms track.

I hear that you have some strong opinions, which is how you're putting that together.

You want to tell me about those and what the track's all about?

Sean: Absolutely, yeah. I mean I think it's important to distinguish your track and have like a really strong opinion about what it should be about.

I wasn't just kind of looking for the best talks, I was looking for the best talks about what I think data science really is meant to be and how it's going to unlock value for the companies where we operate.

And so for me, the cornerstone to that is experimentation and causal inference. Those are topics that are kind of important in statistics and in machine learning, but we apply them in companies to make better decisions on a day-to-day basis. So I really wanted the speakers that we had to reflect the kind of the day-to-day work of someone who thinks that way and thinks that that's the way that we have impact on companies that we work at.

Jesse: One of the things that I know that you're highlighting is frameworks and methodologies that are actually being applied in modern research and data science environments.

You know, a really strong list of confirmed speakers. Can you tell me about who you're excited about and what they're going to be talking about?

Sean: Well, I'm excited about them all equally. They're all amazing people and I really feel very lucky to have such a great track.

We get a lot of submissions, but also I kind of go and seek out speakers to make sure that we have the set of topics that I want to portray there.

But these speakers are, I think, probably the best that I've had in all the years that I've been hosting. In particular, we have, Hadley Wickham is probably the star of the show.

He is somebody that I was a long time fan, like, maybe 20 years ago when I was starting my career, he was somebody that I looked up to and he's only gone on to do bigger and better things since then.

And he's built like, the majority of the tools that people use in practice. Like, he's had his his hand on on most of them.

So, I kind of invited him to come and speak about whatever he was recently been thinking a lot about.

And so, it's kind of exciting to kind of, you get a peek into the future from someone like him who's been thinking really deeply about how generative AI can be integrated into data science workflows.

Deep Focus on Experimentation and Real-World Applications

Jesse: Wow, that's awesome. Who else? Tell us about the other speakers in the track.

Sean: Absolutely. Yeah.

I think one of the kind of hallmarks of the data science track is that we talk a lot about experimentation. And having people who work on experimentation in big companies but also at startups has been sort of one of the themes.

So, we got Timothy Chan, he's head of data at Statsig. Statsig for people who don't know, is one of the most popular A/B testing tools on the market.

You know, used by hundreds, probably thousands of startups and big companies at this point. I'm a big fan of the tool.

He's been working on experimentation for as long as anybody and he gets to talk to people who are running experiments at all these companies all the time.

Jesse: At a massive scale?

Sean: And a massive scale. So, that's a really unique opportunity to get insight from somebody who's not going to have the same perspective as somebody who's just been at the same company for a long time or even kind of like, you know, deep academic expertise.

He's somebody who's really hands-on with all the customers of their platform. So, very excited for Timothy Chan.

Also, Joe Powers is a principal data scientist at Intuit, who will be speaking, he's really exciting, I think for the alternative perspective is having somebody who's deeply thought about one specific problem for a long time.

And so, he's worked at Intuit for a number of years now and he's going to be talking about Bayesian A/B testing and how they apply it there.

So, getting that peak in under the hood of how a big company that you know, can be a little mysterious how a company with like tons of resources and really hard problems, approaches solving them.

So, getting that insight from somebody who's been there for a long time and kind of like one dedicated use case will think, I think, will be a really nice compliment to Timothy's talk.

Jesse: One of the things that I loved about the conference last year was seeing really established practitioners, working really closely with people that are figuring out both what they're trying to do inside of their organizations and also how to justify or explain or even market internally to business users who might need to use or apply these technologies.

Can you talk about the Office Hours program and sort of the other elements in the conference that kind of really formalize the ability to connect people that know what they're doing with people that are trying to figure it out and make the case?

Creating Connections Through Office Hours and Community

Sean: Yeah, absolutely.

I think, you know, one of the hard things about work in like 2025 is that we don't get to peek over each other's shoulders, kind of see what people are doing in practice. So, that's kind of the role of a conference like this, is to like, get to tap on someone's shoulder, see what they're doing, have them share what they're working on and have that kind of deep interaction.

And so, one of the unique qualities of Data Council is that you get unique access to the speakers because we have office hours after the track, after the talk is over.

And then, in that Office Hours you can really spend dedicated time with that person. It's not just like a couple minutes after the talk, and it's a very social conference in general.

I think that the hallways are really great and you end up getting to spend a lot of time with everybody.

And that's what we're all looking for is kind of like validation of what are we working on? Is what we're working on kind of normal?

Can we learn from other people what they're doing and like figure out a way to improve what we're doing?

Data Council is kind of like one of the few places that we can do that these days.

Jesse: Last year, Roger Magoulas and I posted about how data science was having a DevOps movement.

A moment where community was starting to come together and formalize in a way that frankly we hadn't seen a lot, I think in part because what you just said.

Where you don't get to peek over each other's shoulders. I find that data science teams are in some ways more isolated than others, which is part of why Data Council is so important.

That's why Heavybit sponsors it. It' why we participate so heavily, why we are hosting a party.

It seems to be that this is a really unique event and opportunity for people that frankly was just needed sort of that ability to kind of reconnect and start working together again.

For you, what are your favorite parts about Data Council?

Sean: Yeah, I mean for me I feel like it's a reunion every time. So I really, I have specific people that I want to catch up with.

And so, you know, being in my track is really wonderful, but there's also lots of other people at the conference, so it being over multiple days and being able to space things out is really nice and there is a lot of downtime baked into it.

But I think for me, like the, you know, there's different concentric circles of community that you have for your profession.

You have community within your own company and you have the kind of like broader community and getting to be in person with people for this dedicated amount of time is really special and unique.

And so like, you know, carving out time to hang out in the hallway and just catch up and see what people are working on.

And that's usually what I ask people is kind of like, what challenging thing are you working on these days? Or like, you know, what's keeping you up at night?

And then, usually get a really interesting answer to that.

Jesse: When you selected speakers, how did that work? What was your process?

What was the analytics stack that you ran in order to find the right folks given the extraordinary quality of the people that you're bringing in?

Sean: It's a tough process because we do have open submissions for talks and we get a lot of great submissions, so it's really tough number one, to go through that and say no to a lot of talks 'cause there's a lot of people that probably could be speaking that we don't select.

We get so many great speakers. But also I want to compliment that with some people from my network that I think kind of fill in the gaps from what we got in terms of submissions.

And I tried really hard to like, get opinions from the community about what should be in the data science track.

So, I went on LinkedIn and I posted, "What should be in the data science track in 2025?" to try to like just solicit people's opinions.

And it was like crickets. Like, I couldn't get anybody to form a strong opinion about what is data science now after we have people forking off to be machine learning engineer.

There's all these disciplines that I think, I would say are kind of like forks of the original data science discipline.

So, what's left for us? So, I guess I just formed my own opinions about that and it's kind of what I think I see as most effective in practice at companies, which is people doing kind of like, deep analytical and statistical work.

People building tools that are useful to practitioners internally to the company. People practicing experimentation and causal inference that help improve decision making. When I can find people who can speak to that from some really recent and useful experience with real examples, I feel like that that's what people are and the track are going to enjoy and learn the most from.

Jesse: It's amazing. You know, one of my questions is like, what are the practical skills and insights that you want people to have? And you just described that to a T.

So, I'll ask you another one, which is, how much has AI changed the landscape in the last year since the last conference?

What is suddenly even more different now, particularly given that you just joined OpenAI?

AI’s Evolving Role in Data Science

Sean: So, two years ago I think was the original like, AI moment and I remember being at Data Council and the most packed talk was this talk about generating SQL from natural language.

I was like, okay, no one's going to be writing SQL ever again. And you know, spoiler alert. I write SQL every day.

So, I think that was kind of an interesting, kind of like evolution of how we thought about.

At first we were like, wow, it's coming for our jobs and there'll be nothing left for us to do 'cause all the queries will be written and it'll do all the analysis for us.

Jesse: The AI also doesn't want to write SQLs it turns out, yeah.

Sean: But now, having kind of like seen how things have played out that the AI systems are generating even more interesting data and generating more questions for us to answer for the the businesses that we work in.

I think that that's really my big insight is that experimentation is even more important today than it was a few years ago because it's one of the few things that gives us like real insight into how these systems are behaving in practice. So, that kind of rigorous empiricism is now more valuable than it was before AI existed.

Jesse: Yeah, you can synthesize and collate information all you want, but if you run experiments, that's literally the only way you learn things.

Sean: Right.

Jesse: So, we have to do new stuff now.

Sean: Yes, exactly.

Jesse: Sean, before OpenAI, you were at Lyft and Facebook and you have had a variety of roles from research scientists to being a manager and now back to like core startup work where you are at a core data science role 'cause OpenAI is small, right?

Sean: Right.

Jesse: One of the things that I know is a priority for the conference is figuring out how to get people who are earlier in their career, who maybe haven't had foundational data science roles at the biggest companies that use the most data.

Is this track that you're running and the conference generally something that's accessible only to kind of people at your level who you're already connect with on LinkedIn?

Or what are you doing to make it more accessible for people earlier in their careers?

Making the Track Accessible for All Career Levels

Sean: Yeah, that's a great question and I'm very sensitive to that and I try to make sure that we have a track that's open for everybody, interesting to everybody.

And I would expect if you're attending Data Council, no matter what your role, you'll find something interesting in the data science track.

But I do think I want to have a strong opinion about what's in there content-wise. But when I talk to all the speakers about what the talk should be like, we talk a lot about the level.

Like, how to kind of calibrate it to the right audience. Like, who's attending the talk, what are they going to want to see it?

Jesse: This isn't a poster session.

Sean: Yeah, and I think the speakers are, you know, they want to give a good talk and they want to make sure that they cater to the right audience as well.

And that means kind of having something for everybody. It means having, like, touching on advanced things that do kind of pique interest to people and make them want to follow up.

But also kind of covering the basics and making sure that you explain things at a really high level and talk a lot about the why and motivate the work.

And so, I think it starts with yeah, that framing for the speaker is about like, what is the goal here, what is the aesthetic that we're trying to hit?

And also curating for speakers that I know are likely to give good talks. Like they, you know, that they have a track record of giving talks that are going to be accessible to a broad audience.

Jesse: Yep. When I was putting together my own conference for the DevOps community, the big challenge was always like, you want to be memorable but for the right reasons.

And that means that you need speakers who are both experts in their field and entertaining and passionate, and are going to follow up with more great content and great conversations.

And I think that Data Council this year is really doing a fantastic job of increasing who's going to be able to come.

And I just think there's so much more interest now that people seem to care in a new way about both the value of data and experimentation and rigorous analytics.

Is there anything that you're glad you didn't include this time around? Like, what's on your no list?

Sean: That's a great question. You know, we had a really good talk from a couple years ago.

I also have previously worked on forecasting. So, I would lump that into data science work and I think that's still kind of uniquely something that data scientists tend to focus on.

But we don't have a forecasting talk this year. We had a really good one a couple years ago from the folks at Nixtla, which is a startup that works on forecasting packages.

I don't know if there's really anything left. Like, I just feel like maybe the interest in that has kind of declined a little bit or maybe a little bit of it is moving it over to Gen AI.

There are some new methodologies that are like, you know, pure AI approaches to forecasting. But that's one thing that kind of is cutting room for this-

Jesse: Who could have forecasted this outcome?

Sean: Exactly.

Jesse: So, right now we're in a really interesting and I think challenging time in certain industries around like proving the value of data science and analytics.

Can you talk a little bit about sort of how you see people making the case for what they do?

Is that part of your track design is really sort of highlighting like, this is how you talk about the value of this type of work and how you communicate that to other people in both business and other contexts.

Demonstrating the Business Value of Data Work

Sean: Yeah, I think it's a really challenging discipline and field to be a part of from that perspective because when you make a good decision, people really kind of take that for granted based on data.

So, there's a lot of like kind of, I mean, Duncan Watts has this book called "Everything Is Obvious: Once You Know the Answer."

And I think that that's really a common theme with data work is that the value is so internalized so quickly 'cause it's just kind of like a perceptual system for people.

They learn some fact about the world that they didn't know and then they kind of assume that they always knew it. And that's really tough for us to overcome.

I do think people do spend a lot of time trying to justify the value of this practice, even though it doesn't directly like, turn into engineering output or things like that.

But we have to tell stories. And I think that that's kind of like one of the things that we do with data is how do we tell a story both of what should be done, but also how the work that we did was beneficial. And that's kind of like what we do at conferences like this.

'Cause a lot of these are case studies where people are going to motivate the work that they did and why they did it, and why it was valuable for them or the company that they worked at.

So, I hope that there are some lessons there in the track. But for me personally, one of the things that I like the most about data science is finding the weird things, like finding bugs.

Facebook used to have this thing every week at All Hands called "Fix of the Week." Maybe they still do it, I don't know.

And it was always like somebody who looking at data, found some problem that they then fixed and generated more impact than most of the product teams could hope to in a year.

And I think that that's kind of the story to me is that like, if you don't look, then you don't know that there are problems. And I think data's one of the only ways that we have of knowing that there are problems.

Jesse: Speaking of that, are we going to get any hot takes on Airbnb and the no A/B testing conflict that they've managed to ignite?

Sean: Yeah, I mean, I can kind of see the perspective that you can't, that there's the like, you can't A/B test your way from a horse to a car and I'm very sympathetic to like, if your strategy's wrong, A/B testing is not going to help you.

So, I can see the argument. But I think almost all tools are kind of like, in the right hands can be very effective, and then in the wrong hands can be quite detrimental to what you're trying to achieve.

Jesse: Sean, what are the things that you're hoping personally to get out of the conference other than obviously the, having everyone say this was the best track?

Sean: I think to me, I still feel like I'm at the beginning of my career as somebody who works on AI.

I wasn't trained to work on AI, but I now work at Open AI and I'm trying to learn as quickly as I can how to be effective in that space.

So, in our track we have a couple talks. We have Hadley Wickham is talking about applying Generative AI to data science problems.

And we have Bryan Bischof who is Head of Data science and ML Engineering at Hex who's worked on--You know, Hex Magic is partly his creation.

And these are the kind of practitioners I want to learn from. Want to see what people are doing and see what people are thinking because that space is evolving really rapidly.

And how do we as data scientists evaluate and evolve and improve AI products is something that's kind of like my current focus.

So, I want to make sure I learn as much from the practitioners at the conference as I can.

Jesse: That is awesome. Well, I think that it's going to be really amazing. I am looking forward to these talks. I am really looking forward to the event.

We Heavybit are hosting our own party on the 23rd. So, hopefully you'll come there along with everyone from your tracks.

And once again, Data Council is coming up in Oakland, April 22nd to the 24th and tickets are available at datacouncil.ai.

Sean, thank you so much for joining me and can't wait to see you there.

Sean: Yeah, thank you.

The post Data Council 2025: The Data Science & Algorithms Track with Sean Taylor and Jesse Robbins appeared first on Heavybit.

How to Create Data Pipelines

Introduction to Data Pipelines

In today’s data-driven world developers and product managers rely on smooth, automated workflows to process, analyze, and leverage big data effectively, particularly as organizations look to launch data-hungry AI programs. Whether you're a data engineer, a software developer, or a business analyst, setting up a well-structured data pipeline is crucial for ensuring data accuracy, scalability, and using real-time data for timely decision-making.

The problem is building a data pipeline the wrong way can lead to inefficiencies, errors, and costly rework. Without a clear guide, you might struggle with slow or unreliable data processing, complex maintenance and debugging, scalability issues as data volume grows, and/or inconsistent or inaccurate data outputs.

This article will review what data pipelines are, why they’re important, how to create one, and common challenges when getting started, without having to get a degree in data science.

What Are Data Pipelines?

At its core, a data pipeline is a series of interconnected steps that automate the flow of data from its source to its destination. These steps typically include:

• Extraction: Gathering data from various sources, such as databases, APIs, and files.
• Transformation: Cleaning, standardizing, and enriching the data to ensure consistency and quality.
• Loading: Storing the processed data into a data warehouse, data lake, or other target system.

>> Click to zoom in on this infographic
Some of the typical components for data pipelines include tooling for collection, processing, storage, consumption, monitoring, security, orchestration, and networking.

Why Create Data Pipelines?

A data pipeline is a series of processes that extra, transform, load data into its destination as transformed data, properly formatted and structured. When designed correctly, it ensures efficiency, scalability, reliability, and real-time processing.

A data pipeline can automate repetitive tasks, saving time and reducing errors. It properly manages the types of data you need to transport and manipulate. It can handle increasing data loads without performance drops. It maintains data integrity from source to destination. And it can enable faster decision-making with up-to-date data.

A poorly built pipeline, on the other hand, can lead to data loss, bottlenecks, and unreliable analytics—costing your business time and money.

Effective data pipelines consist of:

• Data Collection & Ingestion – Collecting data from sources (APIs, databases, logs, etc.).
• Data Processing – Cleaning, transforming, and enriching raw data, either as batch processing or streaming.
• Data Storage – Storing processed data in warehouses (BigQuery, Snowflake) or lakes (S3, Hadoop).
• Data Consumption – Delivering data to data visualization dashboards, ML models, or different data-as-a-service apps per business needs.

Data pipelines are fundamental to an organization's ability to effectively deploy AI across its operations. AI and machine learning models thrive on vast amounts of high-quality data. Inaccurate, inconsistent, or incomplete data leads to flawed models and unreliable results.

Data pipelines ensure that data is cleaned, transformed, and standardized, providing the necessary foundation for accurate AI model training. Pipelines also enable organizations to implement data quality checks and validation processes, ensuring that only reliable data is used for AI initiatives. This is critical for avoiding biased or inaccurate AI outputs, which can have significant consequences in real-world applications.

How to Create a Data Pipeline

In today's data-saturated landscape, turning raw data into actionable insights requires a structured approach. That's where data pipelines come in. Let's explore the process of building your own pipeline, from initial planning to ongoing monitoring.

Step 1: Plan and Set Goals

Before you begin to build, it's crucial to define your pipeline's purpose. Proper planning prevents costly rework and ensures your pipeline delivers the desired outcomes. Some questions to ask could include:

• What problem are you solving? (Real-time analytics? AI training? Reporting? Improve sales forecasting?)
• What data do you need? (Structured databases? Unstructured logs? Website logs? Sales or marketing data? APIs?)
• How frequently should data update? (Batch vs. real-time?)
• What are the key performance indicators (KPIs) you want to track? (e.g. data latency, data quality, processing time)
• Who will use this data? (Data scientists? Business analysts? AI models?)
• What are your scalability needs? (How much data will you handle today? Next year? Will data volume grow?)
• What is your budget? (For tools, cloud services, and personnel)
• What are your security and compliance needs? (GDPR, HIPAA, etc.)

Example goals could include:

• Ingest daily sales data from three sources into a data warehouse for BI dashboards.
• Stream IoT sensor data in real-time for predictive maintenance AI.

Without planning, you risk building an inefficient or unsustainable pipeline.

Step 2: Select Data Sources

Your data pipeline's effectiveness depends on its ability to connect to diverse data sources. Common sources include:

• Databases (Relational databases like PostgreSQL or MySQL vs. non-relational databases like MongoDB or Cassandra): Connected with SQL queries, JDBC/ODBC connectors
• APIs (REST, SOAP, GraphQL): Connected with Python (requests), API clients
• Cloud Storage (Amazon S3, Google Cloud Storage, Azure Blob Storage): Connected with SDKs (Boto3, Google Cloud Storage, Bloblang)
• Streaming Data (Kafka, IoT): Connected with Kafka consumers, WebSocket listeners
• Flat Files (CSV, JSON, XML, Parquet): Connected with Pandas (read_csv), custom parsers

Don’t forget, it's a good practice to check for rate limits (APIs) and data formats (JSON vs. Avro).

Step 3: Create a Data Processing Plan

Data processing involves cleaning, transforming, and enriching your data. Tools for this step include:

• Python: Libraries like Pandas, NumPy, and scikit-learn provide powerful data manipulation capabilities.
• ETL Tools: Tools like Talend, Apache NiFi, and Informatica offer visual interfaces for building data transformations.
• Cloud Data Processing Services: Services like AWS Glue, Google Cloud Dataflow, and Azure Data Factory provide scalable and managed data processing.

A detailed processing plan should outline data cleaning steps, like removing duplicates and handling missing values. It should include data transformation logic such as data type conversions, aggregations, and joins. The plan should also include data enrichment processes, such as adding external data sources. And it should have data validation rules, like ensuring data consistency and accuracy.

Step 4: Determine your data output storage solution

Choosing the right storage solution is critical for data accessibility and performance. Some options are:

• Cloud Storage: Cost-effective for storing large volumes of data.
• Data Warehouses: Optimized for analytical queries and business intelligence. (Snowflake, BigQuery, Redshift)
• Data Lakes: Store raw, unstructured data for data exploration and machine learning. (Hadoop, S3, Azure Data Lake Storage)
• Flat Files: Simple and portable, but not suitable for large-scale data.

Step 5: Determine How Your Data Will Flow

A flowchart diagram helps visualize the data flow and identify potential bottlenecks. Data orchestration tools, such as Apache Airflow, Prefect, or Dagster, automate the execution of pipeline tasks and ensure proper workflow. These tools enable:

• Task scheduling and dependencies
• Error handling and retries
• Pipeline monitoring and logging

Step 6: Build the Connections and Test Code

Building connections involves writing code or configuring ETL tools to connect to your data sources and storage solutions. Testing is crucial to ensure data accuracy, integrity, and organization into correct schema.

Connection Examples:

• Python: psycopg2 for PostgreSQL, pymongo for MongoDB, requests for APIs.
• ETL Tools: Drag-and-drop interfaces to configure data source and target connections.

Testing:

• Unit Tests: Verify individual data transformation functions.
• Integration Tests: Test the entire pipeline workflow.
• Data Quality Checks: Validate data accuracy and completeness.

Step 7: Monitor Movement of Data

Continuous monitoring is essential for identifying and resolving issues promptly. Monitoring tools can track:

• Data Latency: The time it takes for data to flow through the pipeline.
• Data Quality: The accuracy and completeness of the data.
• Pipeline Performance: Processing time, resource utilization, and error rates.
• Alerts: Setup alerts to notify you of any anomalies.

Integrations with Data Pipelines

Integrations are the lifeblood of effective data pipelines. Without seamless connections between various tools and components, data pipelines become fragmented and inefficient, hindering an organization's ability to extract valuable insights.

Integrations automate the movement of data between different systems, eliminating manual data transfers and reducing the risk of errors. They ensure data compatibility, promoting data consistency. Real-time integrations enable the continuous flow of data, supporting real-time analytics and decision-making. They enable centralized monitoring and management of the entire data pipeline, simplifying troubleshooting and maintenance.

Integrations streamline data workflows, freeing up data engineers and analysts to focus on higher-value tasks. And they allow organizations to scale their data pipelines as data volumes grow and adapt to changing business requirements.

Here's a breakdown of the importance of integrations, focusing on key data pipeline tools.

Data Warehouses

Data warehouses (e.g. Snowflake, BigQuery, Amazon Redshift) serve as central repositories for structured data, optimized for analytical queries. Integrations include:

• ETL/ELT Tools: Seamless integration with ETL/ELT tools (e.g. dbt, Fivetran, Airbyte) is crucial for loading and transforming data into the warehouse.
• Business Intelligence (BI) Tools: Integration with BI tools (e.g. Tableau, Power BI) enables users to visualize and analyze data stored in the warehouse.
• Data Orchestration Tools: Integration with orchestration tools ensures that data loading and transformation processes are executed in a timely and reliable manner.
• API Integrations: API integrations allow for the real-time pushing of data into the warehouse, or the requesting of data from the warehouse for use in other applications.

Data Monitoring Tools

Data monitoring tools (e.g. Datadog, Prometheus, Monte Carlo) track data quality, pipeline performance, and system health. Integration needs include:

• Data Sources: Integration with data sources is essential for monitoring data quality and detecting anomalies.
• Data Processing Tools: Integration with data processing tools allows for tracking pipeline performance and resource utilization.
• Orchestration Tools: Integration with orchestration tools enables monitoring pipeline execution and identifying potential bottlenecks.
• Alerting Systems: Integration with alerting systems (e.g. Slack, email) is crucial for notifying users of critical issues.

Data Orchestration Tools

Data orchestration tools (e.g. Apache Airflow, Prefect, Dagster) automate and manage complex data workflows. Integration needs include:

• Data Sources and Storage: Integration with various data sources and storage systems is essential for extracting, loading, and transforming data.
• Data Processing Tools: Integration with data processing tools allows for executing data transformations and analysis tasks.
• Monitoring and Alerting: Integration with monitoring and alerting systems ensures pipeline reliability and timely issue resolution.
• Cloud Services: Integration with cloud services (e.g. AWS, GCP, Azure) enables seamless deployment and management of data pipelines.

Data Processing Tools

Data processing tools (e.g. Apache Spark, Pandas, dbt) transform and enrich raw data into a usable format. Integration needs include:

• Data Sources: Integration with data sources is crucial for extracting raw data.
• Data Warehouses and Data Lakes: Integration with data warehouses and data lakes is essential for loading processed data.
• Orchestration Tools: Integration with orchestration tools allows for automating data processing tasks.
• API Integrations: API integrations allow for the enrichment of data with outside sources.

Common Data Pipeline Challenges for Startups

For startups, data pipelines often begin as simple scripts and then quickly evolve into mission-critical infrastructure. Startups face unique challenges when building data pipelines due to their rapid growth, limited resources, and evolving needs.

Scalability Issues

Startups often experience exponential data growth as they acquire more users and expand their operations. Pipelines designed for initial data volumes may quickly become overwhelmed, leading to performance bottlenecks and system failures. Predicting future data volume is difficult, so over-engineering is just as bad as under-engineering.

When building scalable pipelines some important elements to consider for your strategy include:

• Cloud-Native Architecture: Leverage cloud-based data services (e.g. AWS Glue, Google Cloud Dataflow, Azure Data Factory) that offer elastic scalability and pay-as-you-go pricing.
• Distributed Processing: Use distributed computing frameworks like Apache Spark to process large datasets in parallel.
• Horizontal Scaling: Design pipelines that can be horizontally scaled by adding more computing resources as needed.
• Microservices Architecture: Break down pipelines into smaller, independent microservices that can be scaled independently.
• Data Partitioning: Divide large datasets into smaller partitions to improve query performance and processing speed.
• Asynchronous Processing: Use message queues (e.g. Kafka, RabbitMQ) to decouple pipeline components and enable asynchronous processing.

When aligning your strategy to a budget, carefully monitor cloud service usage and optimize resource allocation to minimize costs vs. the cheaper price, but dramatically limited scale of 100% on-premises hosted data. Consider serverless computing options (e.g. AWS Lambda, Google Cloud Functions) for event-driven pipelines to reduce costs. And choose appropriate storage solutions (e.g. object storage, data warehouses) based on data access patterns and cost considerations.

Some common bottlenecks that teams come across include problems around data ingestion, transformation, storage, and orchestration. If there is inefficient data ingestion from various sources, consider implementing efficient data connectors, using message queues for streaming data, and optimizing data serialization formats. If you experience slow and inefficient data transformation processes, use distributed processing frameworks, optimize data transformation logic, and leverage cloud-based data transformation services. In the case of inadequate storage capacity or slow query performance, consider choosing scalable storage solutions, optimizing database indexes, and implementing data partitioning. If there is inefficient scheduling and management of pipeline tasks, use data orchestration tools like Airflow, Prefect, or Dagster to automate and manage pipeline workflows.

Technical Debt Management

Startups often prioritize rapid development over long-term maintainability, leading to technical debt in their data pipelines. This debt can manifest as complex and poorly documented code, inconsistent formats, and brittle data pipeline architecture.

Some strategies to consider to avoid technical debt include modular design. Design pipelines with modular components that can be easily modified and replaced. Using IaC tools, such as Terraform or CloudFormation to automate infrastructure provisioning and data management at scale. And also consider implementing automated unit and integration tests to ensure code quality and prevent regressions.

Documentation and knowledge transfer practices should also be a part of your strategy. This includes both comprehensive documentation as well as comments. It's important to document pipeline architecture, data flow, data transformations, and code. And it's just as important to include clear and concise comments that explain functionality.

Keeping information in a central location (e.g. Confluence, Notion), and making a proactive effort to share is also important. Create thorough onboarding processes for new team members to familiarize them with the pipeline. Conduct regular knowledge sharing sessions to ensure that all team members understand the pipeline. And some teams are even using tools that automatically generate documentation from code.

By addressing these challenges proactively, startups can build robust and scalable data pipelines that support their growth and enable them to extract valuable insights from their data.

Cost Optimization

Resource Planning

Cost optimization is a critical consideration when designing and managing data pipelines, especially for startups and organizations with budget constraints.

In some cases this means choosing between server less and dedicated infrastructure. Serverless provides a pay-as-you-go pricing structure that auto-scales and generally reduces operational overhead. Some of the downsides include potential cold start delays, vendor lock-in, and less control over infrastructure. Common use cases for server less include event-driven pipelines, variable workloads, prototyping.

On the other hand, dedicated infrastructure provides predictable costs, greater control over infrastructure, and consistent performance. Some of the downsides include higher upfront costs, manual scaling, increased operational overhead. Ideal use cases for dedicated infrastructure include predictable and consistent workloads, performance-critical pipelines, compliance requirements.

Some teams use a hybrid approach, combining serverless and dedicated infrastructure to optimize costs and performance. For example, use serverless for event-driven tasks and dedicated instances for resource-intensive data transformations.

When handling variable workloads, options include:

• Auto-Scaling: Implement auto-scaling for compute and storage resources to adapt to fluctuating workloads.
• Serverless Computing: Leverage serverless functions or container services that scale automatically based on demand.
• Queuing Systems: Use message queues (e.g. Kafka, RabbitMQ) to decouple pipeline components and handle spikes in data volume.
• Scheduling: Schedule pipeline tasks during off-peak hours to reduce costs.
• Elastic Data Warehouses: Utilize data warehouses that scale compute and storage independently.

Build vs. Buy Decision Framework

As you build your strategy, there may be elements that you consider building, whereas others you consider buying. It's important to consider the total cost of ownership, features and functionality (does the tool meet your requirements?), scalability and performance, integration (can it seamlessly integrate with your existing systems?), maintenance and support, time to market, and whether anyone on your team has some level of expertise to build and maintain.

While commercial solutions come at a literal cost, including license fees, subscription costs, support fees, integration costs--development costs can amount to more. Custom builds include development costs, infrastructure costs, maintenance costs, personnel costs. And don't forget to factor in long term costs, not just the initial start up investment.

When calculating the ROI for building vs buying, some important metrics to consider include:

• Cost Savings: Calculate the cost savings achieved by automating data workflows and improving data quality.
• Revenue Growth: Measure the increase in revenue generated by improved data-driven decision-making.
• Time Savings: Quantify the time saved by data analysts and engineers due to automated data pipelines.
• Improved Efficiency: Measure the increase in operational efficiency resulting from streamlined data workflows.
• Reduced Risk: Calculate the reduction in risk associated with data errors and inconsistencies.
• Qualitative benefits: Improved customer satisfaction, and better business insights. Use metrics that are relevant to your business goals.

By carefully considering these factors, organizations can optimize their data pipeline costs and maximize their return on investment.

Data Pipelines in 2025

Data pipelines are absolutely essential in today's data-driven world. They automate the process of extracting data from various sources, transforming it into a usable format, and loading it into a destination for analysis. This is fundamental for any data-driven operation. They ensure that data is clean, consistent, and reliable, which is crucial for accurate analysis and decision-making.

By automating data workflows, pipelines save time and resources, reducing the need for manual data handling. This allows organizations to focus on analyzing data and deriving insights, rather than spending time on data preparation.

Data pipelines can handle large volumes of data, making them essential for organizations that need to process and analyze massive datasets. They can be scaled up or down as needed, ensuring that organizations can adapt to changing data needs.

Furthermore, data pipelines are the foundation upon which successful AI initiatives are built. They provide the necessary infrastructure to ensure that AI models have access to the high-quality data they need to perform effectively.

More Resources:

• Article: The Data Pipeline is the New Secret Sauce
• Article: AI Inference: A Guide for Founders and Developers
• Article: Data Privacy, Security, and Identity in the Age of GenAI

The post How to Create Data Pipelines appeared first on Heavybit.

Heavybit is thrilled to be sponsoring Data Council 2025, and we invite you to join us in Oakland from Apr 22-24 to experience 3 days of cutting-edge technical talks from the brightest minds in AI & data, learn more at datacouncil.ai and use the code HEAVYBIT20 for 20% off.

Introduction and Sai’s Journey into Databases

Craig Kerstiens: So I'm Craig Kerstiens, Chief Product Officer at Crunchy Data. I've got with me Sai who I'm actually really excited to talk to because I've known Sai for quite a while, since he was first in the industry.

Sai's seen and done a lot, and now is part of ClickHouse, helping people scale data warehouse loads.

But I'm really excited with the track that he's created, but I wanted to kind of, you know...

Sai, don't let me take away your introduction and, you know, give your own background if I missed a bunch of things.

But once you kind of give some of that, I'm really curious to drill into, first, something I've ever asked you, what drove you to databases in the first place?

Like, how did you end up in the data world? What brought you into it? So first, if you want to add anything to your own background and then, like, why database is for you personally.

Sai Krishna Srirampur: Thanks, Craig. So just to start with my background, right?

So yeah, hi folks, I'm very excited to be here, and thanks Heavybit for hosting this great podcast.

And I'm Sai and currently, I work at ClickHouse and I lead database integration efforts at ClickHouse.

So what my team does is we are responsible for providing a magical experience for customers to integrate external data stores with ClickHouse.

Prior to ClickHouse, I was the CEO and co-founder at PeerDB. And at PeerDB, we are building a data replication tool with laser focus on Postgres.

So our unique insight was we were really focused on Postgres and we were making it very easy for customers to move data from Postgres to external data stores.

And one of the data store was ClickHouse, and we added that connector and that took off. And that is why the acquisition happened.

And prior to that, Citus and Microsoft were really transformative in my life because that's where I got introduced to Postgres and that's what instilled passion in me of databases, right? Like, I got to work with a bunch of customers, right?

And in Citus, it was mostly SMBs, like, startups, right? And myself and Craig, like I would say that we sold Citus Cloud together, right?

And at Microsoft, it was a counterpart, we were working with Fortune 400 companies who were running Postgres at scale, right? They were running Postgres at hundreds of terabytes.

I feel that like Citus and Microsoft were transformative in my career because that's where I could see how customers are using Postgres and what are the problems that they were running into, right? And that's where PeerDB came from.

So what I observed was that moving data in and out of Postgres was a big problem, right?

Let it be related to migrations, right? Or let it be moving data to data warehouses, right? So that experience was not very performant.

It was not very scalable because existing data movement and ETL tools were all generalized, right?

So they were not laser-focused on Postgres, right?

So that was the unique insight that we identified and that's the reason we started PeerDB with this vision of building a laser-focused, like, data movement tool for Postgres.

But that's my journey in databases and, like, very excited to be here and be hosting the databases track for Data Council.

Craig: I love that the, like, the Unix philosophy of small sharp tools, right? Instead of being generalized, right, really focused on replication and solving it perfectly from one system to another.

I love that. But, like, how did you even get into databases in the first place?

Sai: Yeah, that's a great question. So the way I got into Citus-- that was my first introduction to databases and Postgres, it was just by chance.

I did not think of getting into databases. I t was my second job out of school.

It was just that one of my great, like, friends who was working there, so I thought it's okay and apply.

Craig: So it was a completely, like, a happy accident or unhappy accident, which...

Sai: Absolutely happy accident, right? And I'm not leaving Postgres and databases since then.

So that's how I got into databases. And my passion for databases continues because I believe that, like, number one, databases are very critical and important to this world, right?

Every application on the internet requires a database. So that is number one. And number two is, as application needs evolve, right, like, database also needs to evolve, and the problems and challenges that it surfaces are going to get only harder. And that's where you need solutions, right? Like, and that's what, like, drives me to be so passionate about databases.

Craig: Yeah, so I'd love... Like, I always say, like, working with databases, it took me personally a long time to get used to this that it's a very thankless job.

It's kind of, like, you don't show up to your bank and say thank you for not, like, losing my money or, like, thank you for, like, not having, like, my social security number stolen, right?

Like, when you hear from your bank, it's like, "Uh-oh, here's a year free of, like, free credit protection," right? Because there was a security breach.

Like, it's really a thankless jobs 'cause you store the data, you do it well, and your customers don't talk to you, right?

Like, it's critical, but it's a very thankless behind the scenes. Like, if you're doing your job well as a database, right, you kind of fade away into the background in what application developers build.

So it took me, personally, a long time to get to that. So in a minute, I want to get more into the track itself, but I'm kind of curious from your journey.

Before we get into that, any more, like, personal insights and experiences and kind of, you know, aha moments that you've had from your time in the data world.

Philosophy and Key Insights on Databases

Sai: I think I have, like, two insights that I wanted to share, right? Like, and maybe you'll agree with one of them, but, like, let's see.

So the first insight that I wanted to share is that across my decade of experience in databases, I believe that purpose built databases are more impactful than generalized databases.

And this comes from my experience at Citus, right? Like, what we observed was we were, like, trying to do, like, multiple use cases, right, like, it'd be multi-tenant SaaS, like, which was more OLTP and transactional heavy, then you had, like, you know, real time analytics, right?

So what I felt was because of our focus on multiple use cases, we spread thin, right? Like, so sure Citus was a great success, right, but it did not become a Snowflake, right?

Craig: I'm curious if this is the one I agree with or the other one because I mean, being a Postgres guy, right, like, Postgres can do a lot, but sorry, keep going.

Sai: Yeah, I think this is the one I'm not sure whether you'd agree with, but I am a strong believer of purpose built databases.

And that's the reason probably I'm working for ClickHouse now and the acquisition went through because ClickHouse is laser-focused on, like, analytics, right?

So that is one of my, like, insights. The second insight, I'm sure you'll agree, is users buy databases not for just the technology and features, they buy databases for the team, the thought leadership, and the support, right?

Like, I think, that is very, very important. And I would say that is equally important as the product and the features, right? So those are my two insights.

Craig: Yeah, no, I think, like, you, imply this, that the roadmap, the direction, the vision, the ethos, like, what a database is and becoming, right?

So when you say to the team, right, where is it driving, where is it headed to, right? What's the vision, what's the direction? So yeah, I'm highly aligned to that.

The first one we might have to dispute a little bit as I think, like, personally, right, I am in the Postgres camp. I've been in that world for a long time and think it's an amazing database.

And you know, I remember, the way, way back the Postgres gray-beards were, like, they remember when XML databases were going to take over the world.

Like, legit this was a thing of XML databases were going to take over the world and Postgres just added an XML, you know, data type.

It went and added JSON, et cetera. So, you know, I'll get off my Postgres bandwagon. I do think there is something to very specialized, you know?

I like the Unix tool philosophy, but I also see, you know, both expansion and compression within the data world.

Vision Behind the Databases Track

Shifting a little onto the track itself, 'cause when I saw the lineup, I was super excited.

It's like, you know, there's one track, but there's a lot going on in here. What was kind of your vision and goal with creating this track?

Sai: Yeah, so the team and vision of the databases track revolves around three main aspects.

First is we wanted to capture most of the current and latest innovations that are happening in the space. That is number one. Number two is we wanted to bring a holistic view and a diverse perspective to the databases track, right? And that is reflected via other speakers and the talks.

Like, and the database track is going to get started with, like, Andy Pavlo's talk, right? Like, and I feel that Andy Pavlo made databases cool, basically, right?

Craig: Yeah, I mean, can we say, like, databases aren't supposed to be sexy. Like, is it fair... Like, sorry, they're not a sexy technology and yet Andy's done an amazing job of that.

So yeah, I mean, right there, you lead it off with kind of the pinnacle of now databases are cool again.

Sai: Absolutely, absolutely. And then we have, like, a bunch of talks, like, starting with, like, transactional workloads, right?

Like, you have talks about Postgres and MongoDB. In the analytical side, you have talks about, like, InfluxDB, ClickHouse, DuckDB, and how the open data format and data lakes are affecting the data warehouse landscape.

There is a very interesting talk by Marco from Crunchy, you would know him a lot, on converging database architectures, right?

Where he talks about how they're extending Postgres for analytics by plugging in, like, DuckDB, right?

There is also, like, a talk by Etienne who is, like, the co-founder of Weaviate, which is a very famous vector database.

And then this talk is about agentic databases, right, and how they are going to be the next big thing, right? Like, this is more related to AI, right?

So that is the second thing where we wanted to bring diverse and holistic perspectives to the track.

Last, but not the least, this is something that is very personal to me, is we wanted to make it really customer-focused, right? Like, so that is reflected by the speakers, right?

Like, most of the speakers are customer-focused, like, practitioners. If you see, like, Paul Dix from InfluxDB and InfluxDB has very large adoption in the community.

If you see, like, Tanya who is the head of product at ClickHouse and ClickHouse has, like, you know, thousands of, like, businesses using it, right?

You have, like, Frank from, like, MongoDB and Frank has, like, a implementations background, right?

So those are the three main, like, things. One is, like, capturing, like, current innovations in the database space.

Next is bringing, like, holistic and diverse perspective to it. And the third is keeping it very grounded and, like, customer-focused, Craig.

Craig: Yeah, no, I love how it's kind of weaved the mix of...

Often you find academic conferences, you find, you know, industry conferences, you find, you know, vendor conferences and it's kind of a gorgeous mix between where you kind of run the gambit all in a single track.

What Makes This Track (and Data Council) Unique

Backing up, like, data conferences are out there. There's a lot of conferences, like, how is this unique from others?

Because I look at it and it jumps out to me. But I want to see, like, from your perspective as the track organizer, how is this unique from other data conferences out there that you've been to or attended or, you know, seen?

How is it different, 'cause to me it feels different looking at it. Like, as part of creating it, can you talk a little bit about that?

Sai: Yeah, that's a great question, right?

So the one thing what I've observed about, like, database conferences is that most of the database conferences are, like, tied to a technology, right?

You have, like, Postgres conferences, you have, like, MongoDB conferences, right?

So most of the times they are tied to a technology, but then the beauty of Data Council is it lets multiple database technologies talk about what is going on in their space, right?

So that is the theme also of the databases track where we, as I said, we wanted to keep it very diverse and bring in holistic views across these technologies. So that is one big difference, I would say, where it is not tied to a single technology, but it's more tied to what is going on in this space.

So that's the big difference.

Craig: The entire conference looks super exciting to me. But I also have to say, I might just camp out in your tracks room the entire time.

Like, are you sort of competing with, like, a... " No, I want this room to be packed versus all the others."

Like, are you camped in this room the entire time and you think it's the best room?

Can you talk a little bit of, like, you know, the other spread of the talks and tracks at the conference?

Sai: Absolutely, there are a bunch of, like, tracks, right? Like, there is the data engineering track, which is what, like, Data Council is known for, right?

I mean that's where it started from, right? Like, and this is the first time we added the databases track because this was a conversation with me and Pete last time.

Like, I told him that, like, "We should talk more about Postgres." I mean that's what I told because I'm a big Postgres fan.

Craig: All right, you've won me back over. Keep going, keep going.

Sai: I am a Postgres fan. So I told him that we should talk more about Postgres because people want to learn about Postgres, which is where Pete, like, had this brilliant idea that we'll start the databases track, but then that you have the data engineering track, there is foundation models track, right?

Like, there's an AI track, right? Like, so there are a bunch of tracks, and I think all of them are going to be, like, amazing, right?

And if you look at the speakers, right? Like, in one of the tracks, we also have, like, you know, the creator of, like, DuckDB who's going to be speaking.

Craig: Hannes, yeah.

Sai: Exactly, so I think all of the tracks are going to be fun, but I would say that, like, you know, the databases track probably would be the best because we have a legendary lineup, right?

Craig: I'm really excited, I may stop out to catch Hannes' talk. I love Hannes' talks and what they're doing with DuckDB.

So there's a lot of, you know, data conferences out there. We've covered how your're different, and in particular, this track among databases, right, from academic to practitioner to, you know, interesting things happening and where the future is.

Audience, Flow, and Accessibility

Can you talk a little bit about the themes and kind of how this flows from one to the next? Because it seems like, it really does flow from a...

You can start your day in this track and end your day in it, and there's a lovely journey five-hour movie that happens, right?

Sai: Absolutely. I think the way I look at it, right, like, so far we spoke about, like, you know, the speakers and the talks, right?

Like, getting into the audience, right, like, who would benefit from this track, right? Like, I did give, like, a thought for this.

And I would say that I see three types of audiences benefit from this track.

Number one is basically database users who want to figure out, like, their data stack and you know, they learn about all the innovation that's going on from legendary experts, right?

Like, and then through that, like, they fuel how their data stack is going to look like. So that is number one. The second type of audience I see is, like, database developers, right?

Like, if you see two to three of the talks in the track are related to, like, database development, right? Like, these are, like, builders, like, who are giving talks.

So these database developers can learn how, you know, actual development is done where these are builders who are building large-scale databases, right, databases that are supporting thousands of customers.

So that is the second audience. The third is, it's also for students, right, like, who want to learn about databases, right?

So I see these three type of audiences, database users, database developers, and students, who would benefit from this track. What I expect is because of the diverse set of audience and the diverse set of speakers, there would be a lot of learning and great discussions. And that's the goal of the track.

Craig: So if you don't want to accidentally stumble into databases, but be intentional about it, you jump right into this.

Like, I agree as a student, right, like, you can go to a lot of academic, just the spread of academic to protect practitioner here is truly unique.

Now, I guess shifting a little bit, was there anything, like, intimidating about creating this track, right?

Like, to me there's a lovely cohesion, but is there any part where, like, "Hey, I'm a Mongo person, this part's going to scare me."

Or, like, should anyone be intimidated by, you know, "Well, this talk isn't for me, that's talk's not for me."

Like, if it's foreign to me, how should I hang out or think about it? Or do you have thoughts on that cohesion versus, you know, how you struggled with creating the track?

Sai: I think that's a great question. So firstly, like, I think it was a struggle to create that track.

The reason is, like, curating all the hundreds of talks and choosing just six of them was very difficult, right?

Like, and we did it, like, more top to bottom, right? Like, where we decided, like, what's the vision of the track?

And then, like, you know, we created these buckets of different topics and then we got the best speakers in each of them.

Second is, like, I don't think that, like, it's going to be a challenge for someone who is foreign to something, right?

Like, for example, agentic databases are not my territory, right?

Like, it's completely new for myself, but then I'm so excited about it because I'm expecting, like, all the speakers in the track would give context on, like, you know, each of their, like, territory, right?

Like, I mean their topics. So I don't see it to be, like, you know, hard for, like, folks who are foreign to a topic because all of these are, like, super experienced speakers, right?

And I'm sure that they would do a great job. So, yeah.

Craig: So we've talked a lot about the track and the flow and the technology.

My favorite part of conferences is actually, usually the hallway track, right? And while I'm really excited to kind of camp myself in the track room and go from, like, Andy to time series to analytical, to transactional, like, just that whole flow, right?

My favorite part is usually the hallway track and connecting with people. Like, can you tell me a little bit of, like, the community you want to kind of want to bring together as part of that?

Sai: Yeah, so that's a great question.

I would say that one important goal of the track from the hallway and from the audience perspective is to keep it full of questions and keep it very conversational in the sense that the audience can ask questions, let it be during the talk or let it be off their talk, right?

Like, because finally, like, even though there is a legendary lineup, right, like, all of these speakers are just, like, people, right?

Like, I'm sure that they will also be learning a lot, like, from, like, the questions, right? Like, so I think it's going to be very important to keep it very conversational and, like, keep it very questions heavy.

And also not to forget, the audience can be the next speaker, right? So I would really recommend folks to submit talks.

Like, this time, they were, like, hundreds of talks, right? Like, and it was so hard for, like, the...

Craig: So you want your job to be harder next. You want more talks submitted next year and a harder job is what you're saying.

Sai: I agree, yes.

Craig: No, I love it. No, I think it's spot on. Like, you know, having spoken at conferences, right, like, there's this aura about you, but, like, no, like, many people in the audience...

I actually once gave a talk and in the back of the audience, and it was the most intimidating talk I ever gave, was Michael Stonebraker, the creator of Postgres.

And I was giving a talk on Postgres, and there were questions that came up and I was just, like, sweating bullets. 'cause here's the man that created Postgres that's won a Turing award, right?

And I'm giving a talk about Postgres, about the thing that he created and I had no idea. I just, the last minute, saw him walk in the back.

And so your audience can be full of experts. And so I think, you know, a big takeaway is engage, right?

Like, that's how you get so much out of it. Like, the speaker lineup, I've looked at it, it's amazing, but also, you know, you get out what you put into it.

And so definitely go engage with the speakers, engage with the people around you. Yeah, it's super exciting.

Emerging Trends and the Role of AI

So you've got to track about databases. There's a lot of emerging trends out there. Like, do we have to even say the word AI, right?

Like, it's a incredibly hot topic. Where do databases fit in there, right? Like, are databases dead and boring?

First year of the track, I'm super excited about it. Apparently I'm a weird nerd.

Yet what's the role of all the other things that we should be talking about? How does your track fit in with that?

How should we be thinking about broader trends and all of that, and when it comes to, you know, what's happening in the broader landscape?

Sai: Absolutely.

I think we should talk about AI because that is the emerging trend, and I strongly believe that for AI to succeed, databases are going to be very critical.

The reason I say that, right, like, I mean, AI would actually lead to demanding workloads for which databases need to live up to, right?

Like, so I'm expecting an orders of magnitude increase in requirements and demand for data.

Craig: So an "order of magnitude," are you thinking in terms of scale or user experience or both, like, 'cause I think, like, how people are working with things are different, right? So how does that impact databases?

Sai: Yeah, so orders of magnitude in respect to multiple things, right? Let it be performance, right?

AI would need databases which are 10 times faster, let it be with respect to latency, throughput, and concurrency, right?

That is one, second is, like, databases need to handle orders of magnitude data, right?

I mean, it's not going to be petabytes now, it's going to be zettabytes, right?

So that is number two, and number three, it's also going to be reliability, right?

So I see that in every dimension. Databases need to evolve to be able to support, like, AI, right?

That is the reason I think that AI cannot be successful without evolving databases. I think that's where the next set of innovation and companies, new companies probably will come up, where databases are being able to serve AI with these orders of magnitude, increase in demands and requirements.

Craig: All right, so we've talked a whole bunch about your track. I'm excited for it. Yes, it's your favorite child.

You're not supposed to have a favorite child, but your track is your favorite child.

But, like, if there's a second favorite, like, is there another track or also, and or, something else that you're incredibly excited about, in regards to the Data Council conference.

Sai: So another track that I love is going to be the data engineering track, like, right?

Like, this is probably because, like, PeerDB was a data replication tool, like, we moved data, right?

Like, but that is one track, which is, like, the root of why Data Council, like... I mean that's, like, the...

Craig: It's the origin, right? It's really the foundation.

Sai: Yeah, exactly.

Craig: And, you know, you can't get away from that and it does it so well every year, right?

Sai: Exactly, right, so that track is going to be a good one to attend.

And then there is a foundation models track as well, right, like, which goes into, like, AI, LLMs, and, like, you know, more, like, the guts of, like, LLMs, right?

Like, I think that is going to be, like, a fun one, right? Very, like, engineering and, like, technical focus, I would say.

And then there's also, like, a Lightning Talks track, right? It's going to be a lot of fun.

So last year, I gave a talk at, like, the Lightning Talks track and it was amazing. It was, like, 15 minutes of you know...

There was a talk about, like, Rust programming and, like, that impacting, like, databases, right? Like, and then I spoke about, like, you know, data movement, right?

Like, so it's, like, very, like, interesting, like, you know, quick bits across, like, the entire data space, right? Like, so I think...

Craig: I always highly recommend the lightning, I remember actually at PyCon years ago when Docker was first...

Like, Docker was first unveiled at a lightning talk at PyCon, and it's, you know, like, no spoilers, I don't know what's coming in the lightning talks.

But you never know, it's rapid fire, and the ability to convey a message in that time and you see so much so quickly that it's always a, you know, kind of a must see.

Like, any other kind of pieces or nugget, so the lightning talks, the data engineering track, the foundational model ones, like, any other kind of highlights of the conference?

Sai: Another thing that I would like to note is there are these hands-on sessions where there are speaker office hours, where audience can go and interact with speakers right away, and these office hours are right after the speaker gives the talk, right?

Like, so you can actually go spend time with the speaker, like, have a discussion, right?

So that is something that is very unique I saw in the Data Council conference where you get a chance to personally meet the speaker, have a discussion, right, like, and learn both ways, right?

So that is something that I'm very excited about as well.

Craig: Yeah, that's really, like, I... You know, usually, there's a swarm after a speaker, you know, talks and, like, it feels intimidating to go up there and you know, just even say hi or ask a question.

Like, the ability to kind of break out in a smaller format with them, that's actually...

I don't know if any other conference I can recall that's done that and that's actually really unique and actually probably takes some of the pressure off as a newbie attending a conference to go up and talk to these speakers.

Sai: Absolutely, and I think it also eases some pressure on the speaker as well because they can really hang out and talk to folks and they learn, right?

Like, because last time, I was a part of this and it was a lot of fun because, like, the talk was for 15 minutes, right, but then the hands-on, office hours went on for, like, 45 minutes, where we had so many amazing discussions.

Craig: Cool, well, I just want to say, you know, thanks so much for sitting down, Sai.

I'm super excited when you sent over kind of the rundown of the track. I was excited, I'm going to be there. You're going to be there.

Come up to us, say hi. If you're at Data Council, we'd love to see you. I'd love that databases are kind of becoming sexy again, right?

Thanks a lot to Andy Pavlo who's going to kind of kick things off for the day, you know, in the track, and then it's going to be a whirlwind tour.

It's kind of like this databases courses, all compressed in one day, one amazing track, a lot of speakers, so I'm really excited about it.

Look forward to kind of seeing folks there. I'll be there if anyone wants to talk Postgres.

Sai will be there if anyone wants to talk ClickHouse and you know, databases broadly.

We'll geek out with anyone on it. Sai, anything, you know, that you want to leave to folks?

Sai: No, that's it, very excited for the conference, right?

We started effort around six to seven months ago and the place where we are at now, right, like, with the kind of speakers.

The venue is also going to be amazing, right? Like, I mean I was speaking to Pete and he was saying that, like, it's going to be in Oakland and it's a very raw island, like, a very...

You know, it's an amazing venue so you should check that out, right? Like, so I'm sure that it's going to be a lot of fun and as Craig mentioned, right, just reach out to us, right?

We'd would love to have some great conversations, and very excited to see you there.

The post Data Council 2025: The Databases Track with Sai Krishna Srirampur and Craig Kerstiens appeared first on Heavybit.

In episode 10 of Open Source Ready, Brian and John chat with Taylor Dolezal, former CNCF Head of Ecosystem and current Chief of Staff at Merly AI, to discuss the latest trends in open source, AI, and Kubernetes. From the challenges of AI adoption to the evolving role of cloud-native technologies, Taylor shares insights on where the industry is headed and how developers and companies can navigate these rapid changes.

The post Ep. #10, The Whirlwind Pace of AI with Taylor Dolezal appeared first on Heavybit.

In this inaugural episode of Platform Builders, hosts Christine Spang and Isaac Nassimi of Nylas welcome Justin Belobaba, Founder and CEO of Nowsite. Justin shares his journey from finance to tech entrepreneurship and offers insights on the importance of failing fast, why startups must be willing to pivot, and how AI is shaping the future of solopreneurship. If you’re building a business—or just fascinated by the power of AI—you won’t want to miss this conversation.

The post Ep. #1, The Future of CRM is No CRM with Justin Belobaba appeared first on Heavybit.

In episode 32 of Generationship, Rachel speaks with Marcel Kornacker, creator of Pixeltable and a pioneer in database technology. They discuss how AI engineers can streamline data management, the challenges of founding a startup, and the future of generative AI in software development. Marcel also shares insights from his career journey and how AI is reshaping the way we work with multimodal data.

The post Ep. #32, Structuring Data with Marcel Kornacker appeared first on Heavybit.

What to Know About Synthetic Data for AI Programs

For software developers, large language models (LLMs) like ChatGPT can help modern coders write, test, and deploy with unprecedented speed and accuracy. Realizing the full potential for artificial intelligence use cases, particularly in highly regulated spaces, depends on access to massive amounts of relevant data of high quality, ideally managed by performant data pipelines.

And developers at startups may not have access to data at the scale they need. For those developers, there’s a solution: Synthetic data.

Synthetic data could soon outpace actual data as an input for AI. In 2023, Gartner identified synthetic data as one of its top five trends shaping the future of machine learning. By 2028, the firm projected, 80% of data for AI will be synthetically generated. Experts in highly regulated industries like healthcare suggest that “more and more, synthetic data is going to overtake and be the way people do AI in the future.”

>> Click to zoom in on this infographic
Synthetic data solves important data-related problems for devs.

What is Synthetic Data?

Synthetic data is artificially generated to meet a specific user need. Transformers, neural network architectures that transform inputs to outputs in ML processes, generally create synthetic data. The actual values in artificial data may need to go through anonymization, but the structure often mimics the shape of real datasets, using the same relationships, plot distributions, and statistical properties.

Synthetic data is not new. Physicists were creating synthetic audio a century ago, while major advances in autonomous driving over the last 20 years would have been impossible without synthetic data, which data science teams have used to test and refine algorithms without being constrained to a limited supply of data from real-world locations.

How Developers Use Synthetic Data in AI

In software development, the rapid emergence of AI and LLMs has made synthetic data an essential tool for developers throughout the ML lifecycle, particularly in regulated verticals including healthcare, finance, insurance, and manufacturing that handle sensitive data for customers or medical patients.

Synthetic data can speed up developer workflows in a number of ways:

Shoring Up Data Shortages

Data scarcity occurs for many reasons. Some organizations can’t attain real-world data at the scale their use case demands, including processes like recommendation engines, predictive systems, or natural language processing tasks. In highly regulated industries, data privacy requirements may limit developers’ access to production data or restrict them from dumping personal data directly into GPT.

For example, using publicly available data and simple Python scripting, a team of researchers at the Berlin Institute of Health built an accurate synthetic dataset modeling occurrence and outcomes from three rare diseases in the United States. The team built a data generation tool that addresses "the need to develop specific privacy-enhancing technologies for sharing rare disease data," the researchers wrote in a 2024 paper.

Augmentation of Incomplete or Inaccurate Data

Synthetic data can address other data deficiencies that developers face, such as simulating unusual real-world circumstances (rare diseases, for instance), swapping missing data points into incomplete datasets, and correcting inaccuracies in existing data to ensure ML algorithms produce the best results.

Augmenting existing data is essential to developing models that can identify counterfeit IT documents. Privacy regulations limit the amount of usable real-world data, and variations in the layout and content of ID documents create inconsistencies in the data that do exist. With fine-tuned generative language and vision models, you can create large synthetic ID datasets for training fraud detection algorithms.

Improving ML Model Performance

Synthetic data can help developers ensure their models are actually learning to recognize patterns, rather than just overfitting to limited training data. Introducing synthetic variations to ‘round out’ datasets prevents models from incorrectly attaining biases, such as drawing incorrect correlations from limited data, improving model performance on future data jobs.

One useful approach to improving model accuracy is the Synthetic Minority Oversampling Technique (SMOTE). It's a way of correcting for imbalance in original data by using nearest-neighbor algorithms to intentionally over-represent the minority class as you build training data. The result is a well-balanced synthetic dataset and, as a result, an ML model that's seen enough examples of minority results in training to identify them in production, which is vital for fraud detection.

Reducing Data Toil

Labeling data is a time-consuming and expensive process, especially for large datasets. Synthetic data often comes with automatic labels, reducing the need for manual annotation. For processes that require data in a particular format, such as tabular data or data arranged in a specific time series, automated tools that generate properly formatted synthetic data can also be powerful time savers.

In 2023, synthetic time-series data was key for the winners of a data-themed hackathon. The winning team used OpenAI's GPT-3.5 Turbo model to generate synthetic data projecting how friction would affect different materials over 31 points in time, based on a small set of existing historical data. Selective fine tuning ensured their model produced tabular data with the correct structure.

Generating Synthetic Data

There are several ways to go about the synthesis of synthetic data:

Model-Based Generation

One common way to create synthetic data is training machine learning models on your existing dataset to generate additional synthetic data. With this method, you can create a hybrid dataset that retains the statistical relationships and properties of the original dataset but adds synthetic elements.

This method’s particularly useful for filling gaps in existing data, extrapolating a large dataset from a small one, and scaling general-purpose generative AI models for more-specific uses.

Deep Learning Methods

Two types of deep learning models are widely used to create synthetic data: Generative adversarial networks (GANs) and variational autoencoders (VAEs), which can be used to generate either structured or unstructured data.

In a GAN, two network models collaborate: The generator creates new data while the discriminator assesses it. The generator applies the discriminator’s feedback to the next dataset it produces; that process repeats, persistently improving data quality until the user is satisfied.

VAEs are models in which encoders compress and compact actual data, which a decoder analyzes to generate synthetic data that recreates key characteristics of the original data. VAEs deliver datasets that are very similar to the existing data they’re mimicking.

Rules-Based Generation

If you have some certainty about how your synthetic dataset should diverge from the original, you can write it into rules and use Python to create a new dataset based on those rules. This method offers a great deal of control, but it scales poorly for complex datasets and cases where there are many intersecting rules.

How Can Synthetic Data Be Used for AI?

Synthetic data has become a crucial component in training AI models across multiple industries. Some key applications include:

• Protecting Privacy in Healthcare: Patient data is full of sensitive information, and healthcare providers are required by laws such as HIPAA to protect their privacy. Developers need access to data that’s as close as possible to actual real-world data in order to create new tools for identifying and treating illnesses. However, they are generally forbidden by law to feed unredacted electronic personal health information (ePHI) of patients into AI models for the purposes of testing. Synthetic data can give developers the data they need to innovate without exposing patients’ personally identifiable information (PII).
• Finding Fraud in Financial Services: Financial fraud detection requires vast amounts of transaction data. Like healthcare, the banking and payments industries are highly regulated and demand confidentiality. Synthetic data can replicate key elements of customer datasets without retaining identifiable information. ML models trained on such data are ideally suited to surface irregularities that often indicate fraudulent behavior.
• Simulating Complex Environments for Autonomous Vehicles: Long before they reach the road, AVs learn to drive in simulated road situations created by ML models. Automotive firms have access to massive amounts of real-world sensor data. Adding synthetic data lets team simulate unusual scenarios that AVs must experience to safely navigate the road. Emerging use cases include replicating crashes to evaluate safety features. 

Benefits of Synthetic Data for AI

Synthetic data doesn’t just have to be an option when real datasets are unavailable or inaccessible. It has several advantages over authentic data, making it an attractive option for startups and software developers:

1. No Bias

Bias in your underlying data can nullify whatever benefits you stand to gain from machine learning. And bias is pretty common in actual datasets: over- or under-representation of demographic groups in sampling, mislabeling of data elements, and the conscious or unconscious biases of the people who originally collected the data all occur.

Synthetic data can account for biases up front. Important optimization work like demographic balancing can offset inequities in the dataset you’re augmenting or imitating, and building bias mitigation into your GAN or VAE setup can ensure your synthetic dataset is free from disparities.

2. Lower Cost

Collecting, cleaning, and labeling real-world data is expensive, time-consuming, and resource-intensive. Manual data collection requires hiring data scientists, setting up infrastructure, and ensuring compliance with data regulations. In contrast, synthetic data can be generated programmatically, reducing costs and allowing startups to focus on innovation rather than data acquisition.

While there are commercial products like Amazon Bedrock, Microsoft’s AI Azure Foundry, and specialist startups like Tonic that create synthetic data via API, for cost-conscious organizations, there are also open-source tools that generate synthetic data.

3. Customizable

Synthetic data can be tailored to specific needs, making it more flexible than real-world data. Developers can generate datasets that include rare events, underrepresented scenarios, or domain-specific requirements, ensuring that AI models are trained on highly relevant data. Every element of a synthetic dataset is under your control.

Challenges in Using Synthetic Data for AI

Synthetic data is powerful, but it’s not perfect. But when you know what challenges to expect, you can build monitoring and mitigation into your plans. Here are a few of the complexities that come with using synthetic data for AI:

1. Technical Difficulty

Generating high-quality synthetic data requires advanced technical expertise to set and interpret benchmarks properly. Developers must be proficient in Python, machine learning frameworks, and generative modeling techniques like GANs or diffusion models. Additionally, ensuring that synthetic data maintains statistical fidelity with real-world data requires careful validation and monitoring.

2. Variety of Data

Synthetic data generation works well in structured environments but has limitations in multimodal scenarios. For example, AI models that rely on a combination of images, text, video, and audio may struggle to use purely synthetic data for computer vision or Internet of Things use cases. Developers need to carefully evaluate whether synthetic data alone is sufficient or if it should be combined with real-world data.

3. Data Quality

The quality of synthetic data depends on the sophistication of the generation process. Poorly generated synthetic data can be plagued by the same biases and inaccuracies as real-world datasets, leading to AI models that perform poorly in production. To ensure reliability, startups should consider investing in data quality checks, hiring professionals who can validate their synthetic training datasets before training AI models on them.

Final Thoughts

Synthetic data is a game-changer for AI development, offering scalability, privacy protection, and cost savings. While it comes with technical challenges, its benefits far outweigh the drawbacks, especially for startups looking to accelerate AI innovation. If you're a developer working on AI projects, now is the time to explore how synthetic data can help you build smarter and more efficient models.

More Resources

• Article: AI Inference: A Guide for Founders and Developers
• Article: Who Are the Developers Working on Generative AI Projects?
• Article: Data Privacy, Security, and Identity in the Age of GenAI

The post Synthetic Data for AI: Purpose and Use Cases appeared first on Heavybit.

In episode 17 of How It’s Tested, Eden sits down with Steve Tooke, Director of Product at Kosli and former co-founder of Cucumber, to explore the impact of Behavior-Driven Development (BDD) on software testing. They discuss how Cucumber evolved from an open-source tool into an industry staple, the challenges of test automation, and how AI is reshaping software quality.

The post Ep. #17, How BDD Changed Software Testing with Steve Tooke appeared first on Heavybit.

In episode 79 of o11ycast, Hamel Husain joins the o11ycast crew to discuss the challenges of monitoring AI systems, why off-the-shelf metrics can be misleading, and how error analysis is the key to making AI models more reliable. Plus, insights into how Honeycomb built its Query Assistant and what teams should prioritize when working with AI observability.

The post Ep. #79, AI and Otel: Look at your Data with Hamel Husain appeared first on Heavybit.

In episode 9 of Open Source Ready, Matt Trifiro joins Brian and John to discuss the evolving landscape of commercial open source, the challenges startups face in monetization, and how intellectual property law impacts the ecosystem. Tune in for insights on balancing community-driven development with sustainable business models.

The post Ep. #9, The Future of Commercial Open Source with Matt Trifiro appeared first on Heavybit.

In episode 31 of Generationship, Rachel Chalmers welcomes Deepti Srivastava, a trailblazer in enterprise technology and founder of Snow Leopard. Deepti discusses how her experiences with Oracle and Google Spanner inspired her mission to help businesses leverage live data for AI applications. Learn about the challenges of integrating AI, the myths of data unification, and the promise of generative AI.

The post Ep. #31, A Nursery for Stars with Deepti Srivastava appeared first on Heavybit.

In episode 46 of The Kubelist Podcast, Marc and Benjie chat with John Dietz, CEO of Konstruct, about Kubefirst, an open source platform that streamlines Kubernetes adoption. John shares the journey of building Kubefirst, overcoming Kubernetes complexity, and the challenges of commercializing open source software.

The post Ep. #46, Kubefirst with John Dietz of Konstruct appeared first on Heavybit.

In episode 8 of Open Source Ready, Brian and John sit down with Daniel Mangum, CTO of Golioth, to discuss his journey from distributed systems and Kubernetes to open source hardware. They explore the rise of RISC-V, the potential of decentralized social media, and how AI is shaping the future of computing. Plus, Daniel shares insights into FPGAs, the AT Protocol, and why open source innovation matters more than ever.

The post Ep. #8, Bridging Software & Hardware with Daniel Mangum of Golioth appeared first on Heavybit.

In episode 30 of Generationship, Rachel welcomes Ty Dunn to explore his journey and insights as the Co-founder and CEO of Continue, a company focused on creating AI tools to amplify developers rather than automate them out of their roles. Ty unpacks his background, the motivations behind founding Continue, the trajectory of AI-driven software development, and the philosophy of open source technology.

The post Ep. #30, Possibilities with Ty Dunn of Continue appeared first on Heavybit.

In episode 16 of How It’s Tested, Eden speaks with Jon Jagger, Director of Software at Kosli. The conversation dives into Jon's journey of creating Cyber-Dojo, his insights on test-driven development (TDD), and how software testing practices have evolved over the years. They also discuss Jon's current role at Kosli, the philosophy behind effective testing, and how regulated industries like banking can benefit from modern compliance practices.

The post Ep. #16, Test-Driven Development Demystified with Jon Jagger appeared first on Heavybit.

In episode 7 of Open Source Ready, Brian and John are joined by Kent C. Dodds to discuss the evolution of React, the rise of Remix, and the future of front-end development. From early struggles with Angular to React Router v7’s role in modern apps, Kent shares valuable insights for developers navigating the ever-changing JavaScript landscape.

The post Ep. #7, The Evolution of React with Kent C. Dodds appeared first on Heavybit.

In episode 29 of Generationship, Rachel chats with Leslie Fine, managing partner at Enjoy The Work. With a background in game theory, behavioral economics, and startup leadership, Leslie shares insights into transitioning from founder to CEO, aligning incentives for organizational success, and the transformative potential of AI. Don’t miss her practical advice for technical founders navigating leadership challenges.

The post Ep. #29, Game Theory with Leslie Fine appeared first on Heavybit.

Episode 78 of o11ycast examines the world of OpenTelemetry Transformation Language (OTTL) with Tyler Helmuth and Evan Bradley, the maintainers behind this innovative framework. Discover how OTTL enables powerful telemetry data transformations, its practical applications, and what lies ahead for OpenTelemetry's transformative ecosystem.

The post Ep. #78, Exploring OTTL with Tyler Helmuth and Evan Bradley appeared first on Heavybit.

What Local-First Dev Means for Startup Founders

If you’re a startup founder, you’re always looking for some kind of edge–a way to do things faster or more economically. Local first, which focuses on making an individual dev’s terminal (instead of a cloud-based dev environment) the center of gravity for front-end application development, offers real benefits in development speed and Cloud costs.

Thanks to pioneering research from organizations like Ink & Switch and ongoing interest from the Local First Dev Community, local-first dev has advanced by leaps and bounds, making software faster and simpler to create for many software teams. Prisma founder and startup advisor Johannes Schickling frequently covers the topic on the localfirst.fm podcast. Below, he explains why startup founders should look into local first:

• The Opportunity: Better Life for Devs, Users, and Everyone Else: Local first could mean not just a better life for devs, but better apps for end users
• Removing Data Bottlenecks Means Better UX: Streamlining data bottlenecks gives devs back time to focus on apps that provide users a better experience
• Removing Data Bottlenecks Means Successful, Disruptive Startups: Streamlined, ‘10X’ apps with better UX will have a meaningful chance of unseating incumbents
• How (Or If) to Apply Local First to Your Startup: Currently, local first seems like the best fit for startups building small, sharp tools
• Technical Benefits for Startups - Resourcing, Privacy: Local first can–potentially–mean improved, on-device privacy and being able to do less with more
• Business Benefits for Startups: Faster Dev, Leaner Budgets: Local first can–potentially–mean faster dev cycles and a smaller to nonexistent Cloud budget
• What a Local-First Future Looks Like: Cracking the code on data management could unlock a world where dev is as easy, and creative, as building Legos

The Opportunity: Better Life for Devs, Users, and Everyone

Schickling is bullish on local first’s benefits not just for developers, but for the end users of products that offer local first-like experiences. “A canonical example that's already a success story here is Linear, which is a next-generation product that clearly feels superior in terms of how fast it is and how well the product was executed.”

The project management tool has received praise thanks to its built-in sync engine that resolves changes and conflicts in real time under the hood, even as different users make changes and updates asynchronously. “And it was clearly made possible by a next-generation way to deal with data that was inspired by local first.”

Johannes Schickling explains how local first provides a more-responsive UX at the Local First Berlin conference. Image courtesy Local First Berlin

Removing Data Bottlenecks Means Better UX

While there’s an opportunity to improve app functionality for end users, there’s also room to improve app development. “The modern Web stack is notorious for its complexity–it’s comical how many different tools you have there. But in theory, it should all be very simple. We can run JavaScript on the client or on the server...yet we have this combinatorial explosion of different technologies.”

“A lot of that is constrained, or made terrible, by how complex data management is. Local first has this amazing opportunity to massively simplify the entire application stack, even though one of the golden rules in software development, and in computer science, is usually about conditionals. The answer to most questions is usually, ‘it depends.’”

“So far, we've been swinging this big hammer, no matter which app it is. Most applications we use on a daily basis are built with the same approach. However, local first gives you an opportunity to massively simplify everything and use a more-custom hammer. You can be more efficient, build simpler things, and therefore give application developers the time and affordance to work on what matters most: The app user experience.”

Removing Data Bottlenecks Means Successful, Disruptive Startups

“I’m currently developing a music app which is a very ambitious undertaking, as I'm designing this for myself as the customer, based on the typical ways I consume music. Rather than get too deep into the specific workflows, what’s important is the desire to do better. Application developers want to build better experiences that are credible replacements for the status quo.”

The founder points to Linear CTO Tuomas Artman’s contention that competitive MVPs are all about execution. “[Artman] argues that most products we could think of have already been built. Often, there is already a winner in the market right now. The way to replace that is to build something that's 10x better. And ‘better’ often means making it faster, more reliable, shipping features more quickly, and so on.”

Of course, there’s no magic wand to simply conjure up a 10x product. Building something better usually means making tough choices on what not to focus on during development. “Local first lets me spend my time on something else–making my app better–and spending less time on dealing with data problems.”

“Most products we could think of have already been built. The way to replace that is to build something that's 10x better...faster, more reliable, shipping features more quickly.” -Johannes Schickling, Founder / Overengineering Studios, Prisma

How (Or If) to Apply Local First to Your Startup

Schickling concedes that local-first development is still in the process of maturing through active research, and may not necessarily be a perfect fit for every startup–particularly those looking to actively manage data for thousands of users, or managing enormous, enterprise-grade data workloads with high operational requirements.

“If you think about the way data is managed with local first, it flips the gravity away from huge, monolithic cloud databases that are the source of truth for everything. We now have very capable devices in our pockets that can do a lot of processing and storage. If you think about the photo gallery on your phone, you might want to browse through your images–even if you’re on the subway or a plane.”

“Having challenges accessing your personal memories that are stored in some data center is a very different use case than what a typical enterprise needs. For now, local first might be a better fit for things that are focused around an individual or a small group with a high trust assumption–where it’s fine for someone to access a lot of that data, and you're not as densely connected.”

“Something that would not be easy to build with local-first technologies–a social network like LinkedIn, Twitter, or Facebook is a notorious counterexample. We’re even seeing advancements that help us move in that direction, but that's not where we are.”

“And I think it's hard to say what a ‘typical’ enterprise application would look like. Sometimes, they’re fairly small applications that deal with a fairly small amount of data, deployed in a particular setting on premise or with high trust assumptions. Local first might already be perfectly viable for those use cases.”

“But then, you have another kind of application shape: Something you want to roll out to millions of users right away, and be able to revoke data, and that sort of thing. And this is where it might not be as good of a fit yet. So I think it really depends. But overall, the developments are pointing in a direction where I think local first will become a viable replacement for many more application scenarios than just personal software.”

Technical Benefits for Startups - Resourcing, Privacy

Schickling points to Linear as a case study in how local first can reshape engineering organizations. “The typical software startup has a sizable engineering organization, split across front-end developers, back-end developers, DevOps, and so on. But Linear almost exclusively has front-end engineers because a lot of the other burden is taken care of by a sync engine.”

“If your startup’s situation allows for that, I think this can be a massive competitive advantage where you can keep the team smaller, make the application feel faster, and have many other benefits. However, if we’re not speaking about a hypothetical environment in 10 years, I think it really depends on the kind of application that you want to build.”

“Another interesting perspective on this is when you look at AI, particularly in the context of how to manage end users. This is where AI is meant to serve us, but in a way, it acts like a magnifier for a lot of the problems we're already experiencing. AI can only do a good job if it has great contextual data, but often, that contextual data is even more sensitive.”

“I might be biased because I’m European, but I believe users should really think twice about what kind of data they provide to online services. For example, Apple is a bit like one of the first local-first companies–most iPhone apps are local first, for instance. The company really prides itself on taking user privacy seriously and running AI locally wherever possible.”

“I think more people are picking up on that–that desire to run AI locally, particularly when it's dealing with latency or highly sensitive context.”

“I think [local first] can be a massive competitive advantage where you can keep the team smaller, make the application feel faster, and have many other benefits. However...I think it really depends on the kind of application that you want to build.”

Business Benefits for Startups: Faster Dev, Leaner Budgets

So, does local-first development the ideal technical and business foundation for every single tech startup going forward? “You’d need to do a viability study,” cautions the founder. “If you feel local first is a great fit for the application you’re building, you also need to consider how you will think about evolving the product going forward.”

For startup founders, Schickling emphasizes the importance of optionality–intentionally planning product and organizational roadmaps that don’t overly constrain your company too early in its life. “Local first certainly constrains your optionality in some regard. For example, if a user controls most of the data, and you would like to ship a feature that provides analytical reports across a cohort of users, you might no longer be able to do that.”

“So you need to think through things like: What does your product roadmap look like? Are there other constraints of local first that require you to modify your product? And how much of a problem would that be?”

“If you can determine that local first offers more advantages than disadvantages, making that choice can help you reap the benefits, such as a smaller engineering team, moving faster, and building a faster product–maybe even a 10x better product than what the market has currently to offer.”

“But you also need to see it in the context of where the technology is right now. There is a really interesting product, Muse, that Adam Wiggins built some years ago. At the time, they were very, very early [to implementing a successful version of local first]–maybe too early. They had to do a lot of engineering and a lot of pioneering research work to build this product experience.”

“Maybe if they were a few years later, they could have piggybacked on some of the data engineering work done by other companies–to help them get to market quicker and more efficiently. So if you feel like you'd like to do less pioneering work while you build your product to work around some of the limitations that are still sometimes appearing in those first data stacks, you might be better advised to wait.”

“In that case, you might want to hold off a few years and build it in a traditional way. It's a trade-off in terms of your timing assumptions. That said, [local first] can provide major economic benefits–where you can run much leaner when it comes to cloud infrastructure. In many cases, if you use local first, you simply don’t need cloud infra where you otherwise would have.”

“I think Linear has publicly talked about their [infrastructure] situation. Basically, when you consider the number of customers they’re serving and the impact they're having, they're running on a comically small footprint in terms of infrastructure. I’d guess that if you were to try to build the same sort of product experience in a more-traditional way, you’d probably be easily paying multiple orders of magnitude more in cloud spending every year, compared to what they do.”

“So, if you do the analysis, and you find that you can build your product in a local-first way, then you can expect that the amount of money you need to spend on cloud expenses could be ridiculously small.”

“I would also add that I think there will be a typical ‘Crossing the chasm’ evolution, with innovators, early adopters, and so on. And for all this to become enterprise ready, I think that is still probably a few years down the road.”

What a Local-First Future Looks Like

On the topic of a local-first future for software development, Schickling notes, “I'm personally interested in levers that bring down the effort it requires to bring something into existence. And I think we've seen some of that over the last few decades, like the first availability of the Cloud, or that we can run full-stack JavaScript with WebAssembly (Wasm). And most recently, how AI seems like it might empower entirely new demographics–people who don't do programming.”

“As a kid, I enjoyed playing with Lego, and I’m still nostalgic about the idea of Lego bricks that you can just put together. In theory, that should work for software, but we're not there yet. In my opinion, one of the huge barriers to making that happen is data management. Great applications are still bottlenecked by simple and easy data management.”

“I've studied this field for some time–how much effort it takes to build and continuously maintain something. Previously, projects would just take too much effort to build, and you couldn't justify this effort to the extent of raising VC money, for example. But now you can just do more with less, and that's why I'm excited about it.”

“We’re seeing a lot of promising AI tools–products that might, at least visually, look like the thing I want. But in some ways, some new projects seem very shallow–kind of like the “not hotdog” app from Silicon Valley. You want a real product that does real things, and to make a product real, you need data management that has no asterisks, works reliably, in a sensible way. And this is where we need good building blocks.”

More Resources:

• Article: How Local-First Is Changing Software
• Podcast: The Local First Podcast
• The Local First Developer Community

The post How Startup Founders Should Think About Local-First Dev appeared first on Heavybit.

What Local First Is, and Why It Matters

Local-first development is a development ethos that keeps data and code on your device first. It is inherently different from cloud-first development, which puts the central “source of truth” for code online.

While cloud-based development has offered a number of advantages, it has removed significant control and access from the hands of end-users. Because most cloud applications are offered as a service, in the event of an interruption of service, users lose access to their data.

Local-first development is about building products with the cloud as optional. Its goal is to provide the cross-device collaboration of cloud with the personal data ownership of pre-cloud software development.

The benefits of local first come with tradeoffs. Even though local-first devs work primarily with data and code on their own computers, they still need to manage authentications for sensitive customer data. They also need to manage data sync issues, though techniques like auto-merging and using conflict-free replicated data types (CRDT) have helped researchers and early adopters make significant strides forward.

To help us understand the potential impact of local first, we’ve assembled a group of experts, including Heroku co-founder Adam Wiggins, researcher Brooklyn Zelenka, Netlify co-founder Matt Biilmann, and Vella founder Yonatan Feleke to cover:

• The Benefits of Local First for Software Development
• The Potential Trade-Offs of Local First
• Why Local First Is Gaining Traction
• How Local First Will Change the Way We Build Software Products

Local First Will Make Small, Sharp Tools Sharper

Adam Wiggins is the Platform GM at The Browser Company. He also co-founded the research lab Ink & Switch; the pioneering Cloud infrastructure company Heroku; and Muse, the canvas for thinking. He envisions a future where local first lets developers experiment and innovate faster, as local-first development:

• Increases Velocity: CRDT-based sync engines are the foundation for local first. Building apps on top of such a sync engine takes away all the complexity of client-server network operations, error handling, optimistic UI updates, and more. Instead, developers can work faster by just operating on all the data as if it is local.
• Decreases Resource Requirements: Cloud hosting bills are a fraction of the size when the end user’s computer does a lot more of the basic work of the application.
• Lowers the Barrier of Entry to a Wider Variety of Projects: With local-first development, it’s fast and cheap to build an app that has multi-device and multi-user capability. That means many smaller-scale projects may be built that never would have been feasible in the cloud.

More Ownership, Speed, and Better Economics with Adam Wiggins

Wiggins is direct on the value of local-first development: “Local first offers the benefits of Cloud (shareable, collaborative docs by default) together with the benefits of traditional desktop software (on-device data is much faster and works offline). There are many reasons to prefer this architecture, but my personal feeling is the most important part is the data ownership and agency this gives to end users.

For commercial engineering teams, local first offers powerful, practical benefits that are difficult to ignore. “The biggest benefits we've seen from local-first development in commercial products at scale are improved developer productivity (and thus higher shipping velocity), and lower hosting costs due to needing much less server-side compute. This is on top of the user-facing benefits of more-responsive UIs and seamless offline capability, and the philosophical benefits of data ownership.”

The co-founder suggests that if local-first development faces any significant challenges, they aren’t technical so much as they are due to relative novelty. “Most developers assume that merge conflicts will be the hardest part, but in practice, this is essentially a non-issue thanks to over a decade of deep computer science algorithm work. There are some open technical questions in the space, schema migrations being one.”

“The real biggest challenge is just that this is such a new technology that (1) it's hard to find engineers with experience in it and (2) most of the libraries and vendors available today are still fairly young. Someone new to the community compared local-first development in 2024 as being comparable to React in 2013, which I think is about right. It's exciting to be part of such a dynamic and fast-moving technical community, but for building a product in this way it means you're way out on the frontier with fast-shifting infrastructure and fewer instructional resources to rely on.”

Heroku co-founder Adam Wiggins discusses the evolution of app development at the 2012 Waza Event. Image courtesy Heroku

Where Local First Goes From Here

While local first offers powerful benefits, Wiggins suggests its growing popularity may not be due to any specific economic or data privacy-related concern. “Honestly, I think it's more just a ‘tipping point’ kind of thing.” Since 2019, the co-founder and his team have been hard at work building out both the theory and principles behind local first development, and the practical frameworks developers need.

“A small set of people (I'm one of them) from the deep research world got excited about this direction for app development seven or eight years ago and laid some of the groundwork. But it's taken this long for the theoretical and philosophical basis to transform into a set of open-source libraries, commercial vendors, and real-world success stories to light the way.”

The co-founder sees an obvious future for traditional startups to successfully adopt local first. One example is the project-tracking app Linear, which offers a real-time sync engine under the hood that enables developers to work simultaneously with far less concern for conflicts or online data issues.

“Obviously, you can point to a success story like Linear and talk about how this will change large-scale, venture-backed software. Namely, faster iteration speed, more focus on interface development compared to the backend, and a higher-performance UI for end users.”

But Wiggins is hopeful that it will spur innovation from smaller groups or individuals who might otherwise have thought their projects didn’t merit the effort or resources. “My personal hope is that the biggest impact will be the change to the economics of small-scale apps.”

“Stuff like internal tools at a company, commercial products by indie app developers, or just home-cooked apps you might make for your friends and family. In the current world, the cost to host and manage (for example) a Postgres-backed database app remains high enough to stop some of these things from being built. A local-first app can live longer and at lower cost, and that may enable a whole new class of apps to become feasible.”

Reduced Complexity vs. Cultural and Business Pushback

Brooklyn Zelenka is a researcher and builder who has previously focused on Web development and blockchain. She has served as an engineer, technical lead, and founder at organizations like Fission, Kickstarter, and MetaLab. She explains that local first could have a profound impact on:

• Reducing Architectural Complexity: Local first will streamline the entire development stack, reducing dependence on tools that specifically enable building in the Cloud
• Building and Onboarding Developer Teams: By reducing architectural complexity, local first will also lower the barrier to entry on getting new developers ramped
• Cultural Expectations for “How We’ve Always Done It”: Local first may cause cultural clashes with conservative dev teams–and massive cloud providers with a vested interest in maintaining the status quo

Local First Might Be Like the Shift to Cloud With Brooklyn Zelenka

Zelenka suggests that local first is a response to the way software has become exponentially more complicated to make. “We’ve been building apps the same way for more than 30 years. And that's led to complexity. It's very difficult to onboard new developers. You have to learn front end, back end, managing containers–all of these things.”

“So you have to build a big team of experts in many different things. Which makes it very difficult for people to own their own data, to have a customized, malleable software interface, and so on. Local first is really about radically reducing the complexity of building applications.” Additional benefits like low latency and always-on access don’t hurt either.

The researcher’s frustration with modern development crystallized in 2019 when she noticed open repositories like GitHub offered thousands of projects–but hardly any seemed to offer the ability to bring infrastructure to the commons, leaving most to reinvent the wheel. “As a developer, I've written the same apps and modules over and over again–just to deploy them in different environments.”

Researcher Brooklyn Zelenka discusses authentication systems for local-first systems at the Local First Berlin conference. Image courtesy Local First Conference

Still Being Resolved: Big World vs. Small World, BigCo vs. Small Startup

Zelenka suggests that a key question to resolve is the “big world, small world” distinction. “If I want to collaborate with 10,000 people, no problem. But to try to ingest the entire data firehose for BlueSky? I’m not going to do that on my phone–that’s going to be out on a server somewhere. I can send an email on my phone, but I won’t run a mail server on it.”

Because the field is still relatively new, conventional tooling may not have caught up with the most recent developments. “The question is: How can we still respect local-first principles for ‘big world’ things? And can we have these autonomous, or ‘higher user agency’ architectures applied to things that have to run in a more-centralized way? For example, I’m currently working on a local-first access control project, as there is no drop-in solution for that today. In cases like these with no established option, people often fall back to using a cloud server.”

For smaller orgs, local first can represent the exact kind of path-less-traveled advantage you’d expect, though even larger organizations are starting to reap the benefits. “I’ve heard that in larger organizations, people are using things like auto-merge to sync database configurations, using local first and extremely high availability to maintain consistency. A lot of the focus, especially for CRDTs and in local first, is on collaborative software because that's where all the hard problems were a couple of years ago. There's always going to be problems in that space, but as time passes, they’re getting solved.”

How Local First Development Might Look in Practice

How will local first ultimately change the way people build software? The researcher suggests that a major difference will be making network infrastructure optional–which means managing cloud payloads across different regions could simply go away. “All [your infrastructure needs] to know is: I am able to upload some bytes, and somebody else is allowed to download those bytes.”

“And if the startup that you're relying on goes under, you can just switch to another one for your sync on the backend. There's a lot of talk in the community right now about sync engines–essentially a buffer between front end and back end. We used to describe the workflow needed for software as ‘shipping your box.’”

“As an example, let’s say you were using Rails to build a React app. Previously, you’d create some dummy data in the browser, mock up what it would look like, then build a backend and serialization on both sides, including an auth login system and all of those things. And then, to ‘ship your box,’ you’d have to deploy it somewhere and have it scale, as you’d expect.”

“With local first, you’d just build a front-end app. And for that data layer, just use this library. And if you want to back it up somewhere online, once it's at the server, you're done. You don't have to touch any special code. It feels like building an iOS or Android app–where you just have data ‘as an API.’”

“You don't need to become an expert in this towering stack of technologies anymore. Literally, what you’re working on in the ‘mockup’ phase is your production app. If you want to keep working on your app and try something out, you don't have to make a copy of the entire production database. You can just use your production data because it's your data. You have a copy that's ‘as good’ as anybody else's copy.”

You don't need to become an expert in this towering stack of technologies anymore. Literally, what you’re working on in the ‘mockup’ phase is your production app.” -Brooklyn Zelenka, Researcher/Ink & Switch

Cultural and Business Challenges

While local first offers tremendous potential for individual devs, the benefits might not seem obvious, or enticing, to stubborn system administrators who “have always done it this way,” or to massive providers who make billions by hosting cloud environments. Zelenka recalls pitching to skeptical audiences: “When I started working on this in 2019, I’d be on calls pitching to VCs, and they wouldn’t believe us–even when we’d show them a demo.”

“Is local first going to work for absolutely, completely every problem? Maybe not, but saying everything has to get shoved into a traditional architecture isn’t the only way we can ever possibly write software, either. It’s like when we saw the shift to the Cloud 10 or 15 years ago, there was absolutely nothing [established] at the time.”

“For people who have expertise in cloud scaling, it’s understandable if they wouldn’t be thrilled about part of their job potentially becoming obsolete due to a generic sync server. And from a business perspective, I would say there’s an open debate on whether local first is–and this isn’t quite the right framing, I admit–a ‘product’ versus a ‘feature.’”

Zelenka also concedes there are still questions about how local first itself can be a sustainable business model. “It's very difficult to build a moat around something that also doesn’t let you hold people's data for ransom–where they can just take their data somewhere else. It's very user-friendly, but maybe not compatible with the usual SaaS business model.”

“One version might be people making local first part of a larger SaaS application, or maybe we see people going back to selling licenses. While there are people who are excited about local-first features, some of the early pioneers in the space are still trying to figure out: What's the optimal business model to succeed financially in the long term?”

Local First Changes Development From Imperative to Declarative

Matt Biilmann is the co-founder of Netlify, a leading hosting platform that pioneered modern Web architecture with the Jamstack (JavaScript, APIs, and markup). He sees local-first development paving the way for a massively multiplayer future that entails:

• Sync Engines Handling the Gory Details: As robust sync engines evolve, conflicts between builds will approach zero, paving the way for massive adoption of AI agents
• Developers Working Harmoniously with AI Agents: Web apps could become joint projects between human devs and AI agents quietly resolving issues under the hood
• Developer Experience Becoming “Declarative”: Developers will worry less about conflicts and server issues, focusing on the code in front of them at this moment

Let Agents and Sync Engines Handle the Back End with Matt Biilmann

The co-founder reflects on the evolution of Web development–and how it hasn’t gotten much easier. “It feels like we've made our tooling more complex without really changing the kind of stuff we're building. That feels a little backwards.”

The co-founder suggests the real potential of local first could be in managing the state of a codebase, even as different parties work on it simultaneously. “The reason I’m looking forward to [advancements in] local first is a different abstraction of dealing with state and data. Every time you click a button, a developer has written some code that says, ‘Let me fetch this and show it to you,’ and every time you do an update, some other developer has code that says, ‘Let's change this piece.’”

“And you start having a local-first view of all the states that get synced–it persists, and it works across many different computers working on that state in some multiplayer manner. It feels like there’s an opportunity now for an ecosystem to make it really easy to build those kinds of experiences. And if that happens, we'll see a lot more of them, and the Web will be stronger and better.”

Matt Biilmann discusses the evolution of front-end development. Image courtesy Netlify

How Local First and Sync Engines Unlock a Multiplayer AI Agent Future

Biilmann confides, “What I like most about local first is that work feels very fast and collaborative. But there’s another part of it–the multiplayer. Once you start [using it], you have to deal with issues like conflict resolution. How do we let multiple different people work on something at the same time, and show feedback, and so on?”

The co-founder suggests that while today’s Web doesn’t always require concurrent development, that’ll change. “In the next five years, it's almost certain AI agents will play a much bigger role in how applications work and how we interact with computers. It will massively drive the number of websites and Web apps that actually need some form of multiplayer.”

“Suddenly, it won’t just be about real-time collaboration between humans working around the same underlying state. It might be you and a bunch of AI agents doing stuff that needs you to be the ‘human in the loop.’ It’s another area where I think local first and the multiplayer experience are going to be much more important. And the logical architecture to address that change is local first.”

The co-founder cautions that any serious local-first architecture will need to tackle the multiplayer issue. “When you’re using a browser, you don't get to run just one version of your application, even when there’s only one user. For example, how many versions of the same Google Sheet do I have open in my browser tabs? If I do some work on one tab, then accidentally open a new browser window with the same tab, I expect the new work will be in both of them–synced and accurate.”

“Given the way the Web works, if you start doing local first, you’d better have a sync story. Otherwise, you will quickly start seeing people lose their work, just within a single tab. Which is why all apps essentially become multiplayer apps.” The co-founder notes that the core of the experience will come down to a robust sync engine.

“Most apps don’t have any kind of sync engine. That's why a lot of this movement started around research interests, for things like CRDTs. That’s the big architectural difference. Without a sync engine, building something multiplayer is very manual and hard. On the other hand, the moment you do have that kind of sync engine, multiplayer is the logical next step.”

Without a sync engine, building something multiplayer is very manual and hard. On the other hand, the moment you do have that kind of sync engine, multiplayer is the logical next step.” -Matt Biilmann, CEO and Co-Founder/Netlify

Closed-Circuit Privacy vs. Figuring Out Where Data Goes

Biilmann muses, “When you think about having a sync engine behind your website–and loading a bunch of independent states in the browser–there's still a lot of challenges for very complex apps. If you have apps built–not on top of some simple data source–but on top of different APIs and services, then how do you make a viable sync engine?”

“If we’re worried about more than just syncing my Postgres database to this other user’s view of the database–when you suddenly have a set of different microservices, with the database and the APIs and so on, things change. No one has figured out how to simplify everything into a single engine without lots of complexity yet. And the tooling is still very new.”

“There’s no simple path to ‘Here’s how you build a local-first app with great synergy and the right infrastructure and the right front-end UI architecture. We're still figuring that out as a community, I think.”

As to why local first may be having a moment now, Biilmann points to Linear, whose real-time sync engine is an example of a real-world B2B SaaS app using the kind of tech that should underpin such systems. “I think that that has been one of the catalysts for getting people to believe in this kind of thing.”

“The other thing has been that Ink & Switch has done a lot of foundational work around CRDT to make this something we can actually think about seriously–on the technical underpinnings that need to be in place. And then, as I mentioned, as we move to a world that's much more AI-based, sync engines will become a much more natural way to drive it.”

The co-founder suggests that while on-device data privacy is a nice-to-have, it’s not a driving force for innovation. “I think you still send your data somewhere. Local first is more about the actual benefits to the experience and to the kind of applications that we now get to build. And maybe then data privacy, which is an interesting story, but I've rarely seen it change people’s behavior. Whereas giving developers tools that let them build better user experiences that are faster and more interactive and richer? That's what changes developers’ behavior.”

Local First + Sync Engines + AI Agents = The Declarative Web?

Biilmann suggests that local first could mean a massively simpler future for front-end developers that doesn’t involve writing stepwise code for repeated API queries. “The way we build apps today is very imperative. We write all these sequential calls to do this, then do that. In a local-first world, you have this idea of the UI representing the state that's relevant to the user at this time.”

“When you want to write a list, you just query that internally. When you want to update, you query that update, and you assume that behind the scenes, all the magic is happening. You move from an imperative to a declarative world in terms of how you deal with the state of your code.”

“It may matter less in a world where all the changes for that state are triggered by the current user viewing the browser. But it matters more if that state can also be changed by AI agents working in the background. The more we go in that direction, the harder it will be to work imperatively with the state of your code, and the more you will want a sync engine to solve that problem for you.”

“You will have agents running in the background, running tests and reporting back in the Cloud. That will be really common. Which means that as a developer, should you write code that asks every 10 seconds: ‘What are the AI agents doing? Should I update my UI?’ No. You’ll want some mechanism where, whenever the state of your app gets updated–whether by you or an AI agent–it just shows ‘the right thing.’ That’s the declarative approach to state management I think we’ll see.”

We Need Legos, Not Silos!

Yonatan Feleke is the founder of Vella.ai with experience across the stack at LinkedIn and Qualcomm. He started working on Vella during his Harvard MBA and would later found the thriving Local First community featuring 3,000+ developers to unlock the potential to:

• Dramatically Improve Individual Experience: What if total ownership of your data translated to outcomes like context-aware identity/auth management in any app?
• Radically Simplify Modern Architecture: What if your core infrastructure was mostly just your local database, a data model, and your business logic?
• Increase Interoperability Everywhere: With slimmer infrastructure and seamless integration of local data, orgs will see collaboration opportunities up and down the stack

The Future Is a Shared Data Substrate with Yonatan Feleke (Yonz)

Yonz suggests that anyone can see the need for local first by just looking at how people have to juggle their personal data today. “Everybody's fed up. Something is ‘off.’ Yes, our passwords are in one place. But you can’t use Notion to send an email or a calendar invite if you don’t have the LinkedIn profile of the person you’re trying to reach, for instance.”

“We were promised tools that would accelerate human ingenuity–Lego blocks that would let us build anything. But we’ve ended up with entrenched silos that prevent us from achieving our objectives. That’s my starting point. To me, local first is the path to tackling the central planning problem for our digital lives.”

The founder suggests two main challenges for pure local-first implementations – storage and compute. “For local first, there are questions about–if we completely adhere to having everything on the device with the Cloud to sync it–won’t we hit limits of inference and storage, especially with terabytes of photos and video?”

Yonz points out that other than all those stored videos and photos, people don’t necessarily have a ton of personal data–the remainder could be stored on-device–for most products. More importantly, making your device the ‘system of interface’ for cloud-based services unlocks many new possibilities for interoperability, like the filesystem in Linux.

“If my device is just a ‘warm cache’ for my emails, there is no world where I can have my emails read, modified, or even turned into a CRM by another app on my device because it will only have a limited seven-day download of all my information.”

“But if I downloaded my emails locally or with higher coverage for usability, then you can imagine a CRM running from my device and operating on my data. So, extensibility and connecting silos is a big part of why I get excited about local first prioritizing devices over the Cloud as the source of truth.”