Voice Design for Voice Agents: Prompt, Pick and Test the Right Voice

Voice design is a way to create a brand-new AI voice from a written description instead of recording or cloning a real person. You describe the speaker's age, accent, pitch, pace and job, get a few candidate voices back in seconds, and keep the one that fits your voice agent.

The hard part is not generating voices. It is choosing the right one and proving it works on real calls.

This guide covers how voice design works and how to write a prompt that holds up across languages. It then shows how to pick between candidates without relying on gut feel, and how to test the winner before callers hear it. It draws on research from Gradium's blog on how it builds and selects voices.

What is voice design?

Voice design generates a new synthetic speaker from text alone. No recording goes in. The model reads your description, samples voices that match it, and returns an ID you can use like any custom TTS voice on a normal text-to-speech endpoint.

There are three ways to get a voice for an agent, and each one fits a different situation.

Gradium's rule of thumb is simple. Clone when you need to sound like a specific person, and design when you have a brief but no speaker yet. Cloning also gets more expensive with every voice you add, because each new voice means another speaker to find, another consent and another licence.

Voice design and caller consent

Designing a voice removes the speaker consent question. It does not remove the caller consent one.

In a February 2024 declaratory ruling, FCC 24-17, the US Federal Communications Commission confirmed that the TCPA's rules on "artificial or prerecorded voice" cover AI technologies that generate human voices. The FCC treats an AI voice that imitates a real person and a wholly synthetic one the same way, because in both cases no person is speaking. So an outbound call that uses a designed voice still needs the called party's prior express consent, unless it is an emergency call or an exemption applies.

How does voice design work?

Here is how the AI voice design flow runs in Gradium Voice Design, with limits taken from its Voice Design documentation. Several other vendors' tools follow the same describe, generate, audition and save sequence with their own limits, such as three previews per request.

1. Describe the speaker. Write one or two sentences. Gradium accepts up to 500 characters in English, French, German, Spanish or Portuguese.

2. Generate candidates. The model expands your description into a fuller specification and samples one to five complete voices. Three candidates typically take three to five seconds.

3. Audition each one. Play every candidate on a short line. Gradium caps audition text at 100 characters and serves candidates on its REST endpoint only.

4. Keep the winner. Convert the candidate you want into a permanent voice ID. Converting is free but uses one custom-voice slot, and unconverted candidates are deleted after 30 days.

5. Use it like any other voice. A kept voice runs on the same streaming TTS endpoint as a catalog voice, with the same latency and output formats.

Two behaviours catch first-time users. First, every request samples fresh voices, so running the same prompt again will not bring back a candidate you liked. Keep it the moment you hear it.

Second, one request explores one character. The candidates are variations on a single description, so a different character needs a different description.

How do you write a voice design prompt?

The description is the only input the model gets, so write it like a casting brief. Name as many of these attributes as you can:

Put the intended use last. Gradium's documentation recommends ending with it, because the job shapes how the voice delivers a line, not only how it sounds. A prompt built this way reads like this:

A Colombian Spanish female voice, 35 to 45, for a bank's inbound support line: mid-low pitch, steady unhurried pacing, medium energy and warm, rounded resonance. Clear on numbers and calm with frustrated callers.

Agree on the target before the prompt

Before you write a single prompt, agree on who the voice is for. Gradium's research on how it selects voices lays out a five-step exercise it runs with enterprise customers:

1. Name the listener, not the buyer. For example, a customer calling about a blocked card is a target, while customers in general are not.

2. Make the persona specific. A caller who is driving and wants a one-sentence answer, for instance, tells you the pace and energy you need.

3. Write a "not us" list. The rejected words matter more than the chosen ones, since they keep a group from drifting toward a bland, safe voice.

4. Turn adjectives into acoustic terms. Words like "friendly" mean different things in different languages, while pitch, pace, energy and resonance can be checked by ear. Do this step with a native speaker of the target locale.

5. Write the scripts you will judge on. Judge on lines your agent will actually say, with tricky words and pronunciations included. Keep each script under 200 words and write it natively in each language rather than translating.

The cost is one workshop with the people who own the brand and the call flow. The payoff is a written target that ends the "I'll know it when I hear it" debate.

Does a voice design prompt work in every language?

Not reliably. Gradium found this out by reusing a prompt that produced good US English support voices, warm and enthusiastic with high energy, for French. Most of the resulting voices failed with French raters, and one French female category took around 400 candidates before a usable shortlist emerged.

The words themselves shifted meaning. Gradium measures a keep rate, the share of candidates raters choose to keep, and French raters were stricter overall at 0.59 against 0.69 in English. Candidates from prompts containing "warm" scored a 0.76 keep rate with US raters and 0.53 with French raters. "Energetic" held up in both, at 0.79 and 0.70.

Use case matters even more. Gradium reports that it moves the result more than any other axis of the target, including language and locale. In its tests, US English listeners favoured more effusive, higher-pitched female voices for customer support, and lower-pitched male voices with slower delivery for long-form narration. Gradium attributes the pattern to habits built by decades of phone menus, audiobook casting and broadcasting, rather than to anything in the sound itself.

The practical rule is to write one prompt per locale and per use case, with a native speaker, and to test each one separately. It costs more prompts and more listening sessions. It also saves you from shipping a voice that only works in the language you wrote it in.

How do you choose between voice design candidates?

"Choosing a voice is a casting decision," as Gradium puts it. Gradium says its customers tend to decide by gut feel, playing a few candidates to a room of stakeholders until someone declares a winner. Gradium estimates that roughly one candidate in a thousand has a quality it calls the hit factor. That means you need a large pool and a fair way to compare it.

Gradium's own selection funnel has four stages:

1. Generate a diverse pool. Hundreds of candidates per category, clustered so the survivors differ from each other rather than being near-copies.

2. Shortlist by listener vote. Crowd raters hear every candidate read the same in-language scripts and keep or discard each one, leaving about a dozen.

3. Rank head to head. Shortlisted voices compete against the current favourites on an Elo rating, the pairwise scoring system used to rank chess players.

4. Validate with native speakers. Native listeners confirm the accent is authentic and flag pronunciation problems.

Controls that keep a listening test honest

A listening test is only as honest as its controls. Gradium normalises loudness, randomises the order, caps each rater at 40 comparisons and two sessions in a row, and enforces a five-minute break to limit fatigue. The cost is rater time: with a cap of 40 comparisons per rater, a large pool needs a lot of raters.

If you run your own crowd test, borrow the controls in ITU-T P.808, the ITU recommendation for crowdsourced speech-quality tests. It recommends qualification and training steps before rating, two-eared headphones, a fresh random order for every rater, and one gold-standard and one trapping question for every 10 clips.

The promotion bar is statistical too. A candidate only earns a place if it ranks in the top five and the lower bound of its 95% confidence interval clears the median Elo of the voices already recommended. Statistical ties are cut. Matching the voices already recommended is not enough, because a tie only makes the recommended list longer.

Prefer blind pairwise comparisons to 1-to-5 ratings. Raters use rating scales differently, and a choice between two clips, with a tie allowed, sidesteps that problem. Score it as a win rate: wins plus half of ties, divided by comparisons, so 50% is par.

Here is the method at scale. In Gradium's September 2026 voice design test, native speakers made 7,627 blind pairwise comparisons across five languages and 50 accent prompts. Gradium's voices won 72.6% of comparisons against four other public voice design APIs, 13.6 points ahead of the next system, and led in every language. Treat it as what it is: a vendor-run test of its own product that measures how well a voice matches the prompt, not how it performs on a call.

Where an LLM judge helps and where it does not

Human panels are slow, so it is tempting to hand the job to a model. Gradium ran the same prompts past Gemini 3.1 Pro as a 1-to-5 judge. For the four systems it scored, the judge produced the same ranking as the human raters.

Agreement drops as the question gets more subjective. InstructTTSEval, an academic benchmark of 6,000 instruction-following test cases in English and Chinese, also uses Gemini as its judge. The authors checked it against human raters on 50 samples per task and language. Gemini matched the human majority on 86% of English items that set concrete acoustic parameters, but on only 66% of open-ended role-play items.

Use a model judge for concrete attributes such as gender, pitch, pace and accent presence, and for catching regressions between runs. Keep native listeners for the subjective call and for the final pick. Our guide to LLM as a judge covers how to calibrate one.

How do you test a designed voice inside your voice agent?

A listening test tells you which candidate fits the brief. It cannot tell you how that voice behaves across a whole call: reading back a card number, spelling an email address, keeping its pace when the caller cuts in, or holding up over a phone line.

Gradium's documentation already asks you to audition candidates with the exact model and settings you plan to ship, so that approval and production match. Take it one step further and audition over the channel your callers use.

The stack around the voice matters as well. Per Cekura's benchmarks, the mean Voice Tone + Clarity score ranged from 3.74 to 4.47, on the 5-point scale that study reports, across the seven voice agent platform configurations scored on that metric. The study ran eight configurations, and one has no score for it. The platforms chose their own models, speech components and settings, so the spread reflects whole stacks rather than voices alone.

Which metrics to compare across finalists

To compare finalists fairly, change only the voice and hold everything else constant. Cekura calls your agent the way a caller would, then records and scores the call. Every finalist runs through the same scenarios and the same metrics:

Metrics carry different costs, so spend them on finalists rather than on every candidate. In Cekura, Words Per Minute is free, Pronunciation Check costs 0.2 credits per call, and Humanness costs 0.3 credits per minute. Our guide to voice AI evaluation metrics explains how the wider set fits together.

Write scenarios that make the voice work hard. A support scenario should make the agent read back an amount, a date and the last four digits of a card. Gradium itself names spelling, acronyms, emails, phone numbers and codes as the cases that break voice agents in production. Its own customer support test sentence packs an amount, a date and card digits into a single line.

When to re-test a designed voice

When finalists tie on preference, let the call metrics break the tie. Then run the same evaluators again whenever the voice path changes, such as a new TTS model version, a new voice ID or new synthesis settings. Cekura can schedule evaluator runs with its Cronjobs feature or trigger them from your CI/CD pipeline. Once the voice is live, the same scoring engine runs on production calls.

How do you test your agent against every caller?

Voice design is not only for the agent's side of the call. Your callers bring their own accents, ages and speaking styles, and your speech-to-text model has to handle all of them.

Recognition accuracy moves with the speaker. A 2020 study in PNAS tested five commercial speech recognition systems and found an average word error rate of 0.35 for Black speakers against 0.19 for white speakers. Those systems have changed since, but the lesson holds: test with the callers you actually get.

Accents need targeted test lines. Gradium found that an accent shows up in specific vowels, consonant clusters and speech rhythms, not uniformly across a sentence. If a test line never calls for the sound that marks the accent, it cannot separate a convincing accent from a superficial one.

Automatic labels also struggle to tell an accent from its nearest neighbour, which is why Gradium signs off every accent with native speakers. Do the same before you trust accent-specific results.

Set up the callers you actually get

In Cekura, a personality owns the simulated caller's voice and accent, language, speaking speed, interruption timing, idle timing, background noise, volume and network conditions. The accent comes from the voice itself, so testing a new accent means choosing a voice that has it. When a catalog lacks the accent you need, Cekura can add voices on request. Cekura's personality override then runs every selected scenario once per personality: ten evaluators and three personalities produce 30 calls in one report.

Cekura's Transcription Accuracy metric shows what your speech-to-text got wrong for each caller. Because Cekura generates the caller's speech, it knows the exact words spoken, and it weights errors on names, numbers and negations more heavily than errors on filler words. Our guide to persona-based voice AI QA walks through building a caller set.

A voice design checklist before launch

1. Write the target. Listener, locale, age band, vocal traits, use case and delivery style, plus a "not us" list.

2. Write one prompt per locale and use case. Work with a native speaker and end each prompt with the intended use.

3. Generate widely and keep what you like. The model samples fresh voices on every request, so save a good candidate immediately.

4. Shortlist blind. Normalise loudness, randomise the order, compare in pairs and use native raters with capped sessions.

5. Audition with the shipping configuration. Use the model version and settings you will deploy, over a phone line.

6. Test finalists inside the agent. Run the same scenarios against each voice and compare task success, pronunciation, pace and clarity.

7. Re-test on every change. Re-run the suite when the voice or model changes, and keep scoring production calls.

Voice design makes it cheap to try many voices. Cekura makes it cheap to learn which one works on real calls.

Cekura tests voice agents before launch and monitors production calls with the same metrics. On supported platforms, its Improve Agent (beta) can change an agent's instructions on a private copy and re-test them when a metric fails. To see how your finalists score on your own scenarios, talk to a Cekura expert.

Frequently asked questions

What is voice design in AI?

Voice design is generating a new synthetic voice from a text description instead of a recording. You describe age, gender, accent, pitch, pace, energy and intended use, and the model returns candidate voices in seconds. You keep the one you want as a permanent voice ID for your text-to-speech.

How is voice design different from voice cloning?

Voice cloning copies a real person from a recording, so it needs their consent. Voice design invents a speaker from a description, so there is no recording and no speaker to clear. Clone when you must sound like a specific person. Design when you have a brief but no speaker.

Can I reuse a voice design prompt in another language?

Usually not without rewriting it. In Gradium's tests, descriptors such as "warm" pointed to a different sound for French raters than for US raters. A prompt that worked for US English support voices produced voices that French raters mostly rejected. Write each locale's prompt with a native speaker and test it on its own.

Does a designed voice work in a real-time voice agent?

Yes, once you keep it. A kept voice gets a normal voice ID that runs on the same streaming TTS endpoint as a catalog voice. Unconverted candidates are more limited: Gradium serves them on its REST endpoint only, so convert a candidate before you wire it into a live agent.

Can I use a designed voice commercially?

Usually, on a paid plan. Gradium says a designed voice is fully synthetic, so there is no voice actor licence or royalty, and you own the audio you generate. Its product page limits commercial use to paid plans, and keeps the free plan and beta services to internal, non-commercial use. Other vendors set their own terms, so check before you ship.

How do I test a designed voice before launch?

Test it twice. A blind listening test picks the voice that best fits the brief. Then run the same call scenarios against each finalist and compare pronunciation, pace, clarity and task success. Cekura runs those scenarios as simulated calls and scores every voice on the same metrics, before launch and on production calls.