AI agent optimization, or self-improve, rewrites a voice agent's instructions, re-tests, and keeps the version that scored best. Easy when the agent is your code. Yours isn't — it lives in your provider account, reachable only through a REST API with your key. This is what that costs, and how the loop stays safe anyway.
(Providers: VAPI, Retell, ElevenLabs, Bland. Writes to production: exactly one code path.)
The self-improve loop is simple enough: an LLM proposes a change, deterministic Python measures whether it helped, and the loop searches until it wins, stalls, or runs out of budget. A chess engine, not a chatbot — nothing the model says is believed, only measured.
All of that is provider-agnostic. The layer underneath isn't, and that's where nearly all the engineering went: copying your agent, writing experimental edits to the copy, and — only if you say so — putting the winning version onto the agent your customers are calling right now.
1. What changes in AI agent optimization when the agent isn't yours
If the agent were a prompt in our own database, this would be a text-editing problem. It's a resource in your account at a third party, which changes four things.
- No git revert. There's no version history you control; whatever the loop writes is what the agent is.
- The blast radius is live phone calls. A bad edit doesn't fail a test — it answers a customer badly, on your number.
- No single agent shape. “The prompt” lives somewhere different on every provider — and in up to four places within one.
- Everything is someone else's API. Rate limits, partial failures, endpoints that 400 to mean “wrong endpoint.”
Search only works if a bad move is cheap. Cloning is what makes a bad move cheap — it costs a batch of simulations and nothing else.
So every run starts by copying your agent at your provider, before it reads a transcript or proposes a word. From then on every edit lands on the copy. There is no version of this system that edits production mid-search and is still safe.
The clone boundary
The model runs at propose. A paid simulation batch runs at verify. Everything else is deterministic Python.
2. Cloning is the hard part of AI agent optimization
Every adapter exposes the same few methods — get config, clone, patch, diff — so the pipeline stays provider-agnostic. Behind that surface, each provider is its own problem.
VAPI.
An id can be an assistant, a squad or a workflow, and you find out by probing. Squad leaves are addressed per member (members.*.assistant.model.messages) — the index is data, not a path an allowlist can enumerate. Config always comes over REST; the hosted MCP omits the system prompt.
Retell.
Four shapes, with the prompt in a different place in each: one general_prompt; per-state prompts; a conversation-flow object with per-node instructions; or a custom LLM on your own infrastructure, unreadable through the API and unsupported. Voice and chat agents also live at different endpoints, so a 400 means “try the other one.”
ElevenLabs.
Agents hand callers off to other agents, and /duplicate copies those production ids verbatim — a naive clone would transfer real test callers into your live agent. So the whole transfer tree is cloned, depth- and size-limited, and every target rewritten to its copy.
Bland.
Two products under one name. A persona is a prompt agent where a PATCH creates a new draft version — exactly the semantics you want on a copy. A pathway is a conversation graph: nodes, edges, prompt, model and tools.
The phone number nobody thinks about
If your agent is tested over inbound telephony, the batch dials a phone number. The clone copies that field verbatim — but the number is a provider-side resource still pointing at the original assistant. So every baseline and verify call rings your production line, while the pipeline scores the results as though it had measured the copy it just patched.
The number is rebound to the copy in two halves: resolve where it points and persist that first, then move it — a crash in between must never lose the address it has to go back to. Where a provider has no reassignment API, the run is refused.
3. What the model may touch
The propose model has no provider access at all. It returns a JSON patch — a list of {field_path, action, updated} edits — and deterministic Python decides which are written. It can ask; it can never write.
Allowlisted, per provider.
Prompts and conversation copy, first and end messages, model and temperature, inline function tools, turn-taking and ASR tuning, timing knobs. Conversation design.
Denied, everywhere, always.
Credentials, webhooks and server URLs, phone numbers, transfer destinations, voice and account wiring. Anything that changes who or what the agent talks to.
Routing survives every proposal by construction: Retell's flow nodes expose only prompt leaves, Bland's only their prompt fields under data, ElevenLabs' workflow edges only the natural-language condition — never the ids either side of it.
4. Five ways past a path allowlist
A path allowlist sounds airtight and isn't. Each of these was a real gap, now closed by a specific rule.
1. Smuggling a value.
The gate reads the path, but set and add assign the value verbatim. model.tools is allowlisted and carries no denied token — and the tool being added contains server.url and a secret. Every nested key now faces the same check, and a denied key rejects the whole edit rather than being stripped: half a tool is a config nobody proposed.
2. Deleting the container.
“Transfer destinations are never written from here” was true in the letter and false in effect: removing retell_llm.general_tools takes transfer_call with it, and model.tools takes every webhook. Removals are now checked against what the subtree contains — as is set on a container, which reached the same end by assignment.
3. Rewiring by reference.
You don't need to write a webhook URL if you can attach a tool that already has one. ElevenLabs' bare prompt prefix made tool_ids, knowledge_base and built_in_tools editable; Retell's agent_swap carries its target as agent_id. All are denied tokens now.
4. A capital letter.
The naive camelCase rule shatters capital runs — serverURL became server_u_r_l, and the token “url” stopped matching. A key evaded the entire deny list because of how it was capitalized. Normalization is acronym-safe now.
5. A retry that appends twice.
The apply stage retries transient failures and re-fetches the config first, so a write that landed before the timeout got appended twice — duplicating a tool on the copy, which verify then measures and you could be offered. The add action is idempotent by value.
One gate, both directions.
The same check runs on apply (experiments to the copy) and at promotion (measured edits to your live agent). Exactly one answer to “may this be written” — and it can't drift between the two paths, which is precisely how these gaps get reintroduced.
5. AI agent optimization guardrails at the edges
Knowing which side you're on.
Every clone carries a marker suffix, and that marker licenses anything irreversible. A delete refuses a resource whose name lacks it. An apply refuses a target that isn't marked — a stale id would otherwise put unmeasured edits on a live agent. Promotion refuses one that is. Recorded metadata is never trusted for this: it says what was true at clone time, not whether the id still names the copy.
Retrying someone else's API.
Provider APIs rate-limit in bursts, and a single 429 killing a whole stage is not theoretical. Retries are bounded and deliberately asymmetric: 429 is retried for every method, since a rate-limited request was rejected and can't duplicate work; transport errors and 5xx are never retried on POST, which may still have landed and would create a second clone.
Credentials.
Redirects are never followed — ElevenLabs is reached at an org-controlled base URL, and the key rides in a custom header that, unlike Authorization, survives a redirect across hosts. Keys are decrypted per call, never persisted, never logged.
6. Promotion: the one write to production
A finished run leaves the copy holding the best configuration a whole-suite run validated — not the last one tried. A session that got worse in round six ends holding round three's. Then nothing happens until you click Promote, the only code path that writes to your live agent.
- The same gate. Credentials, webhooks, phone numbers, transfer destinations and voice wiring are never written, even when the copy carries a different value. Anything skipped is reported, not dropped.
- Ground truth, re-read now. The edit set is intersected with a fresh read of both the copy and the live agent — never replayed from a stale snapshot.
- Drift is a conflict. The copy was byte-identical when made, so an editable path that differs today and that this session never touched means someone edited your live agent mid-run. It refuses rather than eating that change.
- Idempotent. Promotion is claimed with a compare-and-set; a second call returns the first one's result.
Don't promote, and the copy is deleted at your provider — automatically after about a week.
7. What you see during an AI agent optimization run
Each run gets a thread where the loop narrates what it's doing: the copy it made, the scenarios it wrote from your failing calls, the baseline per metric, and what each change did to the numbers.
### Step 1 - Private copy created
- The copy runs on ElevenLabs, exactly like your live agent.
- **Your live agent is not touched.** It keeps taking calls exactly
as it does today.
- If they help, you decide whether to promote them. If not, the
copy is deleted.
### Step 2 - Baseline score (before any change)
- [PASS] **Acuity and Hours Routing Correct** - 100% (11/11 passed)
- [WARN] **After-Hours Opening Statement** - 82% (9/11 passed)
- [WARN] **Caller Goal Completed** - 73% (8/11 passed)
From a real session thread.
Runs end in more ways than pass or fail. “Already passing” means nothing was failing to begin with; “could not reproduce” means the failure never surfaced in testing — a different answer, reported differently. And when a fix only worked while a safeguard was removed on the copy, you're told that and offered nothing: the behaviour was being suppressed, not prevented.
If we can't edit it, we say so.
Self-hosted, websocket and custom integrations aren't supported — the loop depends on copying the agent through an API. For those, the AI Assistant runs the same reasoning and hands you the changes to deploy yourself.
Let it make the mistakes on a copy
Turn failing production calls into a measured before/after on your own provider account — without your live agent ever being part of the experiment.
Cekura is SOC 2 Type II, HIPAA, and GDPR compliant.