Making AI its own Primitive in Ruby

Can AI Also Be Designed to Make Programmers Happy?

By all appearances, technical use cases (code generation, cybersecurity, and IT helpdesk) are the top use cases for AI adoption. Python has become, and remains, the world’s most popular programming language, due in large part to its overlap with AI scripting. So what place is there for a programming language that was designed to make programmers happy?

Ruby continues to be popular for full-stack web development, but builder Carmine Paolino insisted on bringing full AI building capability to the language with his project RubyLLM. The project came out of building Chat with Work, the creator’s one-person company Plenty, a bootstrapped org that isn’t seeking outside funding.

But as he explains, he also built the project partially because of his belief in the beauty of the language, partially because he believes AI is becoming a language-agnostic primitive, and partly for himself as a Ruby enthusiast.

Carmine Paolino makes the case for using Ruby to build AI Web apps. Image courtesy RubyConfTH

Why a Ruby AI Framework Was Needed

Paolino is blunt on his initial inspiration for building an AI framework in Ruby: “I didn't run a focus group. I needed it, so I built it. It turned out a lot of Ruby developers wanted the same thing. I had been in Python for a long time: I did my master's in AI and worked as a machine learning engineer and scientist for years. As soon as I could choose, I went back to Ruby, the first language I loved.”

“I wanted to build the new company in Ruby, with the same care that drew me to Ruby and Rails in the first place: Beautiful APIs, convention over configuration, and attention to detail.” Since he couldn’t find a Ruby framework that already existed that was functional enough or comprehensive enough, he created his own.

The creator explains, “building AI features into a real application takes far more than calling a model. RubyLLM is the Ruby AI framework. It gives you models, tools, agents, sandboxed code execution, workflows, evaluations, observability and Rails conventions, designed together, with one consistent API across 19 providers. It's the whole layer between your application and the models, not just a client for their APIs.”

“Before RubyLLM, you had a choice between thin clients for a single provider, which leave you writing everything else yourself, and libraries that didn't feel like Ruby or forced LLMs into patterns that didn't fit them. Most teams ended up writing their own wrappers, or ran a separate Python service next to their Rails app.”

How AI Is Becoming a Primitive for any Language

Paolino believes that for app developers, AI has already become a language-agnostic primitive. “Unless you're training models, AI isn't a specialized stack anymore. It's an HTTP call to a model. The Python ML ecosystem matters for training, and almost nobody building products trains their own LLM.”

“Everything around the call is ordinary web application engineering: Streaming responses to users, persisting conversations, running background jobs, tracking costs, switching providers when pricing changes.” The creator argues that these use cases play to Ruby’s strengths, and as AI transitions to a full-fledged software primitive, nearly every product will use it anyway.

“But a primitive still needs a good framework around it. Every language talks to databases, and Rails still has Active Record. The same is true here, especially since the providers don't agree on one API: OpenAI has Chat Completions and Responses, Anthropic has Messages, Google has its own, and each keeps evolving. Without a framework, every team builds that layer again. RubyLLM is that layer for Ruby.”

What Ruby Brings to AI Development

In fact, Paolino argues that the biggest advantage that Ruby confers to AI development is keeping the agent inside the Rails app, with direct access to its existing business logic. “Several of my clients have successful Rails apps, and they added an AI layer by giving the model tools that find records, update them, and run the operations their app already knows how to do. The agent does the CRUD work, so their users can focus on higher-level goals. Instead of clicking through a booking flow, you ask for the trip and the agent calls the right tools.”

“Keeping it inside the monolith means [you don’t need an] MCP server or external API just so a Python service can reach your data. It deploys together, runs together, is versioned together, and your tests and static analysis cover all of it.”

“[With regards to managing] risk: [You can] use tool approvals for anything consequential, and run tools with the current user's permissions. Then it's at least as safe as exposing an API, arguably safer, because nothing new is exposed to the outside world.”

The creator also suggests that Ruby’s expressiveness and readability give it an aesthetic and a usability that can’t be found elsewhere. “It's the most beautiful language I know, and that still matters even as we write less code by hand. I build Rust applications these days and barely read their code. I set the architecture, make product, design, and QA decisions, and let agents write it. Ruby is the code I still want to read. When you do [actually] review code, readability is a real advantage.”

“Ruby is also expressive enough that AI code reads like a description of what it does, and models write good Ruby. And because it's interpreted, it's a natural fit for code execution. RubyLLM 2.2 will let the model write and run Ruby, Python, JavaScript, or shell in sandboxes on your own servers, on Linux, macOS and Windows.”

I build Rust applications these days and barely read their code. I set the architecture, make product, design, and QA decisions, and let agents write it. Ruby is the code I still want to read. When you do [actually] review code, readability is a real advantage.” -Carmine Paolino, Creator/RubyLLM

The Lines Between a Ruby Framework and Third-Party APIs/Tools

On the question of how to abstract APIs from different LLM vendors over time, Paolino suggests that his plan is to continue to support as many providers as possible, without reducing anything to a lowest common denominator. “Providers have far more in common than people assume, and what one ships, the others usually copy within months. Genuinely different API designs are rarer than you'd think. Anthropic's explicit cache breakpoints were [an example]. AWS Bedrock has since adopted them, and we support both.”

The builder notes that because his daily business is keeping up with cutting-edge AI tooling, he has grown accustomed to rapidly supporting new capabilities, including making full use of provider options and tools that let users use native features without leaving RubyLLM. “We track coverage feature by feature: in 2.0 we support 395 of 405 provider–feature pairs our providers offer.” Rather than obsess over a hypothetical boundary between the framework and what lives in a user’s app, Paolino suggests that his project has been an opinionated agent framework all along.

“RubyLLM 1.0, in March 2025, already shipped the agentic loop (tool calling, with tools as Ruby classes, executed automatically until the model is done), streaming, multimodal input, and Rails persistence with acts_as_chat and acts_as_message, so every conversation, tool call, and tool result lands in your database. Since then, [we have added] durable agents, MCP, structured output, human-in-the-loop approvals, sandboxed code execution, skills, workflows, tracing, and evaluations.”

The creator explains that he makes a judgment call on the boundary between where his framework ends and where users’ projects begin with every update. “The questions I ask are always the same. Is it about LLMs? Would it help many applications, not just one person's? And can it have the same design and the same feel as the rest of RubyLLM?” (He adds that requests that are the focal point for a single user’s app generally belong there.)

Regarding apps that tend to remain ‘outside’ of the project, Paolino cites vector databases as an example of the kind of app that is loosely related to LLMs and is also easy to talk to. He applies the same rule to his own gems, drawing clear lines between the framework itself and existing tooling that doesn’t need to be reproduced, relying on Schematist for schemas and mxc for sandboxes.

“LLM APIs are the opposite. Whether you call them over HTTP or through each provider's SDK, every [call] has its own way of sending messages, streaming chunks, and calling tools, so moving between them means rewriting, and you pay that cost on every call. That's the cost RubyLLM takes away.”

In contrast, embeddings are useful and natively supported in the project, but aren’t necessarily best for search. “I was a data scientist on the search and recommendations team at a large marketplace, and the best retrieval is usually hybrid: Semantic embeddings plus classic BM25, then re-ranking. Your search architecture is yours.”

Day-to-Day: Statefulness and Human-in-the-Loop Approvals

Paolino explains that his project’s approach to statefulness avoids separate state machines and focuses on persisted conversations because the transcript literally is the state. “The providers designed it this way. The conversation already encodes everything an agent needs to know. Inventing a new concept on top of it would make no sense.” He notes that the core of RubyLLM's agent loop is:

def complete

step until complete? || waiting?

end

“The agent is complete when the last message is from the assistant and asks for no more tool calls. It waits when a tool call needs approval or input. Both are facts about the transcript. LLM APIs are also stateless: you send the whole conversation back with every call.

“LLM APIs are also stateless. You send the whole conversation back with every call. So the conversation has to live somewhere anyway. In plain Ruby it's in memory, while in Rails, it's in Active Record. OpenAI's Responses API can store the conversation on their servers so you only send the latest message, but I don't recommend it. Your database stops being the single source of truth for your chat UI, you have to call their servers to render your own conversations, and the data is theirs, not yours.”

“A separate state machine would add a second source of truth that has to stay in sync with the first, for no gain. Getting the basic concept right early is what lets us build everything else on top: Durable agents that resume across jobs and deploys, approvals, memory, and tracing.”

“So, the conversation has to live somewhere anyway. In plain Ruby, that's in memory. In Rails, it's in Active Record. A separate state machine would add a second source of truth that has to stay in sync with the first, for no gain. Getting the basic concept right early is what lets us build everything else on top: durable agents that resume across jobs and deploys, approvals, memory, tracing.”

On the topic of approvals, the creator suggests that human-in-the-loop approvals are already first-class citizens in successful live software, particularly in chat-based products. “The harder question is how to keep them useful when you run many agents at once and can't review every action.”

“That's a question of trust. I run many agents in parallel across several Linux machines and a Mac mini, with permissions bypassed, short of root. I trust them because of how they're trained. Pre-training gives them a broad model of the world, and reinforcement learning from human feedback trains them to complete the task you give them, the way you want it done.”

“Where [agents] go wrong it's usually stubbornness. Pushing too hard to finish a task, not malice. So the controls belong in boundaries: What an agent is allowed to touch and whose permissions it acts with. For customers, trust has to be earned first.”

The creator notes that in his own product Chat with Work, write actions need to be approved the first time (and can subsequently be allowed for the rest of the conversation), while read-only doesn’t require an ask as agents don’t have any additional permissions beyond any connected services.

“That's the pattern I'd recommend: approve consequential actions, scope agents to the user's permissions, and let people widen trust gradually.”

Ruby = Improved Reliability?

Paolino suggests that making agents production-ready within an opinionated Rails environment is easier than the default state of living across a scattered collection of different libraries. “Rails already provides most of the infrastructure an agent needs: Active Record and migrations for state, Active Job and Solid Queue for background work, Action Cable and Turbo for streaming to the browser, and Kamal for deployment. RubyLLM plugs straight into all of it.”

“Agents spend most of their time waiting on the model, so I contributed fiber-based execution to Solid Queue, which shipped in 1.6.0. The Rails default job queue can now run many streaming agent jobs concurrently on a single thread, faster and with far fewer resources. You deal with one monolith, not N services, and you never add a network boundary between your app and the AI logic, with the latency and failure modes that brings.”

Ultimately, the creator insists that the growth and community of his project stemmed from his initial passion about Ruby and building something for himself to use. “I wasn't trying to make it popular. I made it because I wanted it myself, in a very specific way,”

“So build what you actually want. Learn from what others are doing, but don't chase it. Embody the principles you care about. If you love beautiful code, write beautiful code. If you love convention over configuration and progressive disclosure, design for them. Ship it. If it doesn't resonate, build the next thing. That's how I work: I run more than 20 projects at the same time, some more successful than others.”

Embody the principles you care about. If you love beautiful code, write beautiful code. If you love convention over configuration and progressive disclosure, design for them. Ship it. If it doesn't resonate, build the next thing."

The Future of AI Development by Way of Ruby

The creator suggests that a framework like his won’t simply become less relevant as more AI vendors build complementary functionality. “There are two sides to this. Inside your application, smarter models make RubyLLM more useful, not less. Every improvement in the models is something your product can use right away, through the same API.”

“On the development side, many developers have stopped reading most of the code they ship, myself included. I think of it as a spectrum of accountability. At one end is one-shot vibe coding. Next is directing agents and focusing on product, design, and QA without reading code, which is how I build my Rust apps.”

“Further along, for work with economic and reputational stakes like RubyLLM or Chat with Work, you read the critical parts. Further still is mission-critical software, where someone reads every line. At the far end is pure craftsmanship: Writing every line by hand, without AI. That will become a niche like handmade furniture, or just a hobby, or disappear entirely, because clients pay for results, not keystrokes.”

“Because writing code has become so easy, some people conclude they don't need frameworks like Rails or RubyLLM anymore. They're wrong. If you don't use a framework, your AI has to write one. It will hit the same problems, the same provider quirks, and the same design decisions I've already worked through. Spend your tokens on your product instead of reinventing the wheel, or help make the framework better for all of us by contributing.”

“What gets de-emphasized has more to do with practice than with the labs. Early models had context windows of 4K to 8K tokens, so you had to chain many calls together, each with no memory of the previous one, to get anything substantial done without overflowing the context. Today, most frontier models offer a million tokens (the smaller ones, a few hundred thousand), and compaction has become standard. A single long conversation has replaced most of those multi-step workflows.”

“Embeddings were also very important at first; now people combine them with hybrid search, or use tool calls and connectors to reach data directly, as we do in Chat with Work. I also expect fast, cheap decision models, like Jev or OpenAI's Decisions API, which RubyLLM exposes as Judgments, to take part of structured output's share for classifications and yes/no decisions. Chat, tools, and agents [will] stay at the core.”