Can You Make AI Infrastructure Free Forever?

Can You Make AI Infrastructure Invisible?

Depending on which people you talk to, individual developers spend hundreds (to thousands) of dollars monthly on AI services, with some predicting that AI tool costs will simply exceed the cost of human developers soon.

What if the infrastructure and tooling needed to build with AI were free of charge, and wasn’t locked behind a dozen different vendors’ paid plans? That was the question that led developer Mike Kuykendall to create the open-source project Shimmy, a lightweight API replacement for local LLMs. Below, he explains his thought process and his vision for AI software built entirely on free infrastructure.

From Frustration to Lightweight Infrastructure

The start of Kuykendall’s journey was his own frustrations with working across the many different AI-powered software tools he had been using. “It honestly started as a simple shim, because I was getting sick of all the ways to connect with OpenAPI protocols.” The creator was surprised at the project’s popularity, which appears to have come from many other developers also being tired of having to log into dozens of different paid services.

“I just wanted something that was completely private and offline, where you can control your own inputs, outputs and everything else.” Since then, the creator has moved the project to his own Rust-based library Airframe to avoid having to depend on other vendors’ command-line interfaces. “Every time I turned around, I would try to do something else and immediately say, ‘Oh wait, this other thing is owned by someone else at that level.”

“I never really was looking for anything out of it other than just to make a good space for people. That's why I’m trying to make it a ‘free forever’ concept. If I can build products that use [this foundation] that would help people, I could sell [those products]. But the underlying service should be a place where we could all come together and just use AI for free. The other thing is, if you look at every service out there, everybody is just completely throttled on tokens. That's an enormous cost for people who want to do development now.”

“I think that companies want everyone to pay for inference, and you shouldn't have to. The thing is, everybody's already got bundled costs.” The creator points out that the cost of utilizing high-end foundation models continues to increase, while Chinese AI labs are turning out increasingly competitive open models...which aren’t always supported natively within mainstream AI development toolsets.

“Now the users are left with this mess. It was getting to a point where I was paying four different vendors and I wasn't even maximizing all the stuff I had. I asked myself, ‘Why can't we just have an agreed-upon substrate we just all go to?’”

“Why not make the substrate free? Shimmy is free, always will be free. I'm not going to charge you. You have it. You use your model, I make sure it's up to date and your model helps you succeed.” Kuykendall suggests his primary audience is software developers, who could eventually use tools built on a free infrastructure substrate.

If you look at every service out there, everybody is just completely throttled on tokens. That's an enormous cost for people who want to do development now.” -Mike Kuykendall, Creator/Shimmy

Freeing AI Infrastructure from Garbage Collection

Kuykendall makes the emphatic point that despite the domineering popularity of Python for AI use cases, he designed his project to be 100% Rust-based for a reason. “Garbage collection [in Python] is ridiculous. It’s bad engineering.”

The creator explains that he began learning Rust to challenge himself, but found it to be an excellent working language whose ownership and borrowing system inherently forces developers to manage memory safely at compile time, but also matched AI software development well by minimizing non-repeatable, nondeterministic outcomes.

“I also found that the garbage collection brought along with it a bunch of stuff you don't need. If you change your thinking around whether you need to retire some variables or let them gather in heaps, you can just use them and borrow them appropriately.” The creator notes that Rust’s static type system, which verifies every variable at compile time, was both more sensical and helped him create invariant pipelines that reduced unpredictable outcomes as he built.

How Free Infrastructure Cures Tokenmaxxing

Kuykendall suggests that a largely free AI infrastructure substrate should be the solution for headline-grabbing token budget problems from orgs like Uber and ServiceNow. “Just recently, almost every AI company cranked up their rates,” notes the creator, in reference to how major AI vendors recently repriced tokens that caused users everywhere to burn through their token budgets early.

“Last month, I ran out of tokens for every one of my top services on day 15. And I was staring down at the remaining two weeks, thinking to myself, ‘What am I supposed to do now?’ I was forced to go try to ‘do it cheap.’” For his own purposes, the creator went off and did what he imagined many other developers are doing: Trying a list of free alternatives, including OpenCode, Pi, Hermes Agent, and others.

“I think basically the corporations right now are trying to recoup their money. They realize that they're just going to have to take it from the people already using it. That's when I realized if I'm going to be a developer and I'm going to use this service, I need a more-economical way.”

Having already canceled many of his pricey AI subscription services, Kuykendall reflects on how other developers like him may be making the same choices...and ending up with the extra hundreds of dollars saved from those canceled services, perhaps ready to spend elsewhere. “I’m working on other products on top of shimmy. Because that's where the money should be, in my opinion. Application should be where we put the money, not the inference.”

An Inference Tipping Point for Engineering Orgs, Coming Soon?

Kuykendall foresees a not-to-distant future where engineering teams take a hard look at their AI bills, come to the same realization he did, and look to make a change. “I think the [largest AI vendors] understand that they overinvested.”

“In the meantime, if I had a young up-and-comer startup right now, I would use an inference stack that was free. I’d try to get the best compute I could get for my dollar. I think there is going to be an ‘API apocalypse,’ or whatever they’re calling it on social media, when everybody starts seeing viable alternatives they can use instead. And I think we're already getting there.”

The creator notes that open models like Alibaba’s Qwen family and Z.ai’s GLM models can already perform well on software development when coupled with high-end hardware, a stark contrast to some of his previous experiences. “I worked at the IRS for several years, with no AI available. We had to do everything ‘the hard way.’ There were times when I’d think: ‘I could save three hours if I could just use AI right now!’”

According to Kuykendall, we live in very different times now, with highly performant open models and high-end hardware both within reach of individual developers. “Long story short: What I wanted was for nerds like us to lay the groundwork so that it was just invisible infrastructure, for AI work, development, or anything else.”

“It was also important to build the project offline. There are capable builders in Asia, some of the biggest drivers of growth on my GitHub repos, because I think they’re under the Great Firewall.”

What I wanted was for nerds like us to lay the groundwork so that it was just invisible infrastructure, for AI work, development, or anything else.”

Lessons from Building Many Projects with AI

While AI can be a powerful assistant, Kuykendall warns other builders to avoid lulled into a false sense of security from sycophantic models that agree with you a little too much. “You have to separate yourself from this stuff, or at least watch for AI having an emotional impact on you. Try to get away from that and make it a source of data instead.”

“I've been in AI development pretty much from the moment it started. And I noticed while doing testing that tests would just happen to go the way I wanted them to, by a point or two. Always. The AI was quietly influencing things. So for founders, I would recommend that first of all, they should spread out their compute and find many points of view. Every one of these [models] is differently trained and they're going to give you a different answer. And a lot of times you have to watch your logic, because it'll tell you exactly what you want to hear.”

The creator recommends putting models through their paces, at minimum asking why ideas might be good and why they might be bad, and potentially pitting other models against each other. “You have to try to look at it as an empirical and scientific process, and try to take the human emotion out of it.”

What AI Infra Will Look Like in the Future

Kuykendall envisions a future AI infrastructure without the many layers of interface that block users from getting things done. “When I first started really spitballing it early on, I remember thinking: Why am I wasting my time with a computer, that's running a browser, that's running a window, that's running a thing that I connect to a server, that runs the AI? What's all this intervening stuff about?”

“Some of my early, kind of crazy experiments were running AI on hot metal and then having an operating system that's just an AI with tools. Because I found that I would rather have that than have to poke around for that one thing, or that one file. Why don't we just change the idea of an operating system to basically be an intention system? Why not take the AI, surround it with tools, and give it the ability to talk to me and others?”

“I think that's the next thing that would be really transformative. Taking out all the shell layers I don't need between myself and an AI with compute. I need to be able to talk to it. I need to have rewrite ability. I know this because I've already built more than a dozen AI tools. Then you put that in a stack and run on hot metal. I’d be saving a lot of money just doing that.”