A DIY agent fleet costs you a week, not a token budget.

Ryan Carson, a solo founder, spent $20,000 on Devin in one month while running 15 agents at once, and he managed them with “a folder system and a piece of paper.”

The token bill was the easy part. What limits a business with no engineering bench is the orchestration labor: the versioning, the escalation, the deciding what each agent should do next. That work shifts from engineering to supervision.

It either comes with the platform, or it comes out of your week.

What does a 15-agent fleet actually cost to run?

The money was legible. The attention was harder to price.

Carson’s agents worked across engineering, customer success and investor updates. That’s the kind of spread every operator recognizes: build the thing, answer customers, keep stakeholders informed, repeat until Friday stops meaning anything.

The operating system around the agents mattered more than the $20,000. A folder system and a piece of paper meant one person became dispatcher, reviewer, product manager and escalation path for 15 parallel workers.

That can work for a sprint. It’s harder to make it the way your business runs.

Carson’s own conclusion was that “more AI output is actually the wrong goal.” That’s an admission that the bottleneck included his own ability to aim, inspect and redirect the work fast enough for the extra output to matter.

Doesn’t Ramp prove you should build your own?

Ramp built for reasons that come with Ramp’s volume.

In The Pragmatic Engineer’s writeup on Ramp’s Inspect platform, the reasons were specific: local machines couldn’t support many parallel agents, the web team needed designer-facing frontend tooling third parties didn’t offer, and remote debugging got messy at the intersection of systems, backward compatibility and broken API contracts.

Inspect later supported 200+ additional agents without each team managing separate infrastructure. That’s a real payoff.

Ramp’s advantage came from connecting agents to proprietary telemetry, feature flags and internal databases. If you have that kind of internal system depth, building may be rational. If you don’t, you’re taking on the multi-quarter build without the reason it paid off.

Ramp is a volume precedent, not a template. The number of agents you can run is set by how much supervision you can absorb, not by how many you can afford.

One vendor build-vs-buy guide from Augment Code frames custom orchestration as a multi-quarter effort for most teams, which is vendor framing, but it matches the practical issue: the hard part is the system around the agents.

If agents are coworkers, who manages them?

Every persistent worker needs a manager, and the fleet doesn’t come with one.

Tara Seshan, an OpenAI product lead, has described the shift from “rowing” to “steering,” with human judgment becoming the differentiator as AI handles more execution, in Lenny’s Newsletter. That framing is useful. It’s also quietly expensive.

Steering one agent is a meeting. Steering 15 agents is a job.

Someone has to decide which tasks are safe, which answers need review, which failures should be retried, which prompts changed the result, and which agent is quietly doing a worse version of the work than last month. In a company without an engineering bench, that person is usually the founder or the most operationally competent person in the room.

That is how the founder becomes the manager of a team they never hired.

“Persistent coworker” sounds pleasant until the coworker never sleeps and keeps asking for judgment calls.

What should you require instead?

Require four capabilities from any option you consider, including the option where you build it yourself. If they’re not in the product, they’re in your calendar.

• Versioning: you can see which version of an agent produced a result, compare it to prior versions and roll back when a change makes the work worse.
• Escalation: the agent has a defined path for handing a decision to a human, and that human is named before the first production run.
• Limits: spend caps, permission scopes and a bounded blast radius define what an agent can damage when it goes wrong.
• Measurement: each agent has a number that tells you whether a change improved the outcome, tied to the iteration loop after launch.

Reliability is a property of the system around the model. The model matters, but production agents also need memory of what changed, boundaries on what they can touch, and a review loop that catches bad work before it becomes customer-visible work.

That’s the build-vs-buy question worth asking. Who runs that system every week?

At Levain, those four requirements are the default: each agent run is isolated in its own microVM, versioned so you can roll back, measured so you can see whether the agent is getting better, and priced as a platform fee on provider spend rather than as another hire. If you’re holding a list of jobs you want agents to do, we’ll walk through it with you and mark which ones need a human in the loop, which ones can run with limits, and which ones should wait.

Estimated reading time: 4 min