A/B Testing Voice AI Campaigns in India 2026: Scripts, Voices, Call Windows and What Actually Moves Connect Rate

Rohan Mehta runs outbound campaigns for a mid-size NBFC out of a sixth-floor office in Bengaluru. It is the 8th of the month, and he is staring at a dashboard that should make him happy. Last week he ran a script test on his EMI-reminder voice AI campaign — Variant B opened with the borrower's first name and the due amount, Variant A opened with the lender's name. Variant B "won." Connect rate 51%, conversion on promise-to-pay up almost four points. He rolled B out to the full base.

This week, with B running everywhere, connect rate is back at 42% and promise-to-pay looks flat. Nothing changed in the script. He pulls the call logs and finds the answer in thirty seconds, and it has nothing to do with the opening line. Variant B ran its test sample mostly between the 3rd and the 6th — bounce week, when borrowers actually pick up. Variant A ran later. He did not test a script; he tested a calendar.

This is the most common failure in Indian outbound voice campaigns. It is also fixable.

The thesis

A/B testing voice AI campaigns in India is mostly an exercise in not fooling yourself. The mechanics of running a test — splitting traffic, swapping a script — are trivial. The hard part is that connect rate and conversion swing 10–15 points week to week for reasons that have nothing to do with what you changed: day-of-month, time-of-day, number freshness, festival calendars. If you do not control for those, every "win" is a coin flip you mistook for a signal. A disciplined testing program changes one variable at a time, randomizes a holdout, waits for a real sample, and reads a metric hierarchy — not a single headline number.

Why this matters more in 2026

Outbound voice AI is no longer a pilot curiosity in India. NBFCs run EMI reminders on it, D2C brands run abandoned-cart recovery, insurers run renewal nudges, and call volumes are large enough that a two-point connect-rate swing is real money. Dialing 80,000 numbers a month, the gap between a 44% and a 48% connect rate is roughly 3,200 extra conversations — and at typical funnel ratios, a few hundred extra payments.

The problem is that the people running these campaigns inherited their instincts from human call-centre management, where you A/B test by giving two teams two scripts. Voice AI changes the economics of testing — you can run twenty variants, swap them mid-campaign — but it does not change the statistics. More speed just means more ways to be confidently wrong, faster.

There is also a budget reason this matters now. Volumes are higher than two years ago, and finance expects cost-per-conversion to fall every quarter. A campaign lead who cannot tell a genuine lift from a calendar artifact will keep "improving" the script while the cost-per-payment drifts sideways. "We shipped four winning variants" is not an answer if the unit economics did not move.

Two things make 2026 specifically harder. First, TRAI DLT enforcement has tightened: every script variant you run as a registered template needs its own approved header, so you cannot freely swap promotional copy the way a marketer A/B tests an email subject line. Second, vendors now sell "voice selection" and "dynamic script optimization" as features — so campaign leads are asked to evaluate experimentation claims they have no framework to check. A demo that says "our AI picks the best-performing script automatically" is making a statistical claim, and most buyers cannot interrogate it. This post is that framework.

How to actually run a voice campaign A/B test

Start with the unit. You are not testing a script. You are testing a change to one variable while holding everything else fixed, measured against a randomized control drawn from the same population on the same days.

That last clause is the whole game. The single biggest source of fake wins in Indian outbound is letting the two arms run across different days or different call windows. EMI bounce calls cluster on the 3rd–7th of the month; festival weeks crater connect rates. If Variant A's sample is 60% bounce-week calls and Variant B's is 40%, you measured the calendar.

So the correct setup: every day, for every batch you dial, randomly assign each number to A or B at the moment of dialing. Same hours, same retry rules, same DPD (days-past-due) mix. The randomization has to happen at the contact level inside each dialing window — not "Monday is A, Tuesday is B."

Why contact-level and not list-level? Lists are never neutral. The list you dial on the 5th is heavier on fresh bounce cases; the list you dial on the 20th is heavier on chronic late-payers and aged numbers. If Variant A gets one list and Variant B gets another, the population caused the difference, not the variant. A platform that calls list-uploading an "A/B test" is selling you a confound generator.

One more discipline: write the test down before it starts. A single page — what you are changing, what you are holding constant, the one metric, the sample size, the stop date. Writing it stops you from quietly redefining "winning" halfway through when the numbers wobble. Most bad voice-campaign decisions are not analysis errors; they are the absence of a pre-commitment.

What is actually worth testing

Here is the menu, ordered roughly by how much it tends to move outcomes in Indian campaigns, with how to isolate each one.

Variable to test	Typical lift if it works	How to isolate it cleanly
Call window / time-of-day	8–15 pts on connect rate	Hold script, voice, cadence constant. Randomize numbers across 2–3 windows daily. Biggest lever, most confounded.
Retry cadence and gap between attempts	5–12 pts on cumulative connect	Same opening attempt for all; vary only gap (e.g. 4h vs 24h) and max attempts. Measure cumulative RPC, not single-attempt.
Opening line / first 8 seconds	3–8 pts on connect-to-completion	Connect rate is set before audio plays — so measure completion and drop-off in first 10s, not connect. Needs DLT headers for both.
Language: Hindi vs Hinglish vs regional	4–10 pts on completion in Tier-2/3	Segment by circle/pincode first; randomize within segment. Never compare a Delhi cohort to a Patna cohort.
Voice gender, age, pace	2–6 pts on completion	Hold script identical. Slower pace usually helps on older / Tier-3 cohorts. Small effect; needs large samples.
IVR-style confirm vs open question	3–7 pts on intent capture	Test the response-handling branch, not the opening. Measures whether the bot understood, not whether they answered.
Call length / how fast you get to the ask	2–5 pts on conversion	Shorter is usually better for reminders, worse for objection-heavy sales. Measure conversion, not completion.

Two things stand out from that table.

Connect rate is mostly decided before your script runs. Whether a number picks up depends on the call window, number freshness, caller ID, and the day. The audio your bot plays cannot influence connect rate — only what happens after connect. So if you test an opening line and report connect rate, you are reporting noise. Test opening lines on completion rate and first-10-second drop-off.

The call window is the highest-leverage variable and the most confounded. Indian outbound answer rates peak 11am–1pm and again 5–8pm IST. Hindi-belt borrowers rarely answer before 10:30am. If you have never deliberately tested call windows, that is almost certainly where your biggest unclaimed lift is — but it is also the variable most likely to contaminate every other test you run, because window and day-of-month interact.

A third point: retry cadence is undertested and run on gut feel. Most campaign leads have a fixed rule — three attempts, 24 hours apart — that nobody has tested. Does a second attempt four hours after a no-answer beat one a full day later? Cadence tests do not change registered content, so DLT does not gate them, and the cumulative-connect lift often beats any script tweak. Measure cadence on cumulative RPC across the full attempt sequence, and put the cost of the extra attempts into the decision.

The metric hierarchy

Do not optimize a number in the middle of the funnel. Read the whole chain, in order:

1. Attempts — numbers dialed. The denominator. If this differs between arms, your randomization is broken.
2. Connect rate — calls answered / attempts. Driven by window, number freshness, caller ID.
3. Right-party-contact (RPC) — the actual borrower/customer on the line, not a relative or a wrong number. In Tier-2/3 this gap is wide; numbers churn fast.
4. Conversation completion — RPC calls where the bot reached the ask without the person hanging up.
5. Intent captured — completion calls where the bot correctly understood the response (promise-to-pay, callback, dispute, not-interested).
6. Conversion — the actual outcome: payment made, cart recovered, renewal done.

A variant can win at level 2 and lose at level 6. A faster, pushier opening can lift completion while tanking promise-to-pay kept. The only metric that pays your salary is the bottom one; everything above it is diagnostic. Build dashboards around connect rate alone and you will ship variants that talk to more people and convert fewer. Our note on voice AI call analytics and QA goes deeper on instrumenting each stage.

What goes wrong

Five failure modes account for nearly every bad decision in voice campaign testing. Name them so you can catch yourself.

Testing five things at once. You change the opening line, the voice, the call window, and the retry gap, the variant wins, and you have no idea why — you cannot reproduce it, and cannot rule out that one change helped while three hurt. Fix: one variable per test. Testing combinations is a factorial design that needs far more volume; do not pretend a four-change "Variant B" is an A/B test.

Calling significance after 200 calls. A campaign lead sees Variant B at 54% and A at 47% after a day and declares a winner. At those sample sizes a 7-point gap is well inside normal noise — small samples on proportions are wild. Fix: decide your sample size before the test and do not look at the result until you hit it. We size this properly below.

Ignoring day-of-month and time-of-day confounds. This is Rohan's failure. The two arms ran across different days or different windows, and you measured the calendar. Fix: randomize at the contact level within each window, every day. Check that both arms have the same DPD mix and day-of-month distribution before you trust anything.

Optimizing connect rate while conversion drops. A variant that calls earlier connects more — but reaches groggy, irritated people who say no. Connect rate up, promise-to-pay down. The dashboard celebrates; collections suffers. Fix: never declare a winner on a mid-funnel metric. Tie every test to a bottom-funnel outcome and let it veto.

Vanity metrics and selective stopping. "Average call duration up 18%" — good or bad? For a reminder, probably bad; people are confused. "Sentiment score improved" — measured how? And the quiet one: you peek daily and stop the moment it looks good. Peeking inflates false positives badly. Fix: pre-register the metric and the stopping point, and hold to both.

A sixth, India-specific one: comparing cohorts that are not comparable. You run Hindi on one batch and Hinglish on another, but the Hindi batch happened to be a UP/Bihar list and the Hinglish batch was metro. You did not test language; you tested geography. Fix: segment by circle or pincode first, randomize the language test inside each segment, and read results per segment.

And a seventh that wrecks intent-capture tests: mistaking an ASR failure for a customer behaviour. When a Patna or Jodhpur accent pushes word error rate up, the bot mishears "haan, kar dunga" and logs no-intent. You then read that arm as "lower intent capture" and blame the script. Fix: before trusting any intent-capture comparison, pull a sample of transcripts and listen. If WER is worse in one arm's cohort, your intent metric is measuring transcription quality, not the customer.

The numbers

Realistic baselines for Indian outbound voice AI, so you know what a real lift looks like against the noise:

• Connect rate: 38–52% in the good 11am–1pm and 5–8pm windows; 22–34% outside them. Tier-1 numbers connect better than Tier-2/3, where numbers churn faster.
• RPC as a share of connects: 70–85% on fresh first-party data; below 60% on aged or third-party lists.
• Completion rate: 55–75% of RPC calls for a clean reminder script; lower for objection-heavy sales.
• Conversion: EMI promise-to-pay kept and abandoned-cart recovery both sit in low-double-digit percentages of conversations, varying by DPD and cart value.

Now the part most campaign leads skip: how big a sample you need.

You are testing a change to your opening, measured on completion rate. Baseline completion is 60%; you would consider a 5-point lift — to 65% — worth shipping. To detect a 5-point absolute difference on a ~60% proportion with normal confidence, you need roughly 1,400–1,600 completed conversations per arm — not per attempt. Completions, the level-4 metric.

Work backwards through the funnel. If connect rate is 45%, RPC is 78%, and completion is 60%, completions are about 45% x 78% x 60% = 21% of attempts. To get 1,500 completions per arm you need roughly 7,100 attempts per arm — about 14,200 total. For a campaign dialing 80,000 a month, that is around five to six days of volume.

But notice what just happened. To read this test honestly you must run it for five to six days, and those days will span different days-of-month. So you cannot let the calendar leak in — randomize A and B every day across the whole window, so both arms see the same mix of bounce-week and non-bounce-week days. Run it as "A this week, B next week" and the entire 14,200-call sample is worthless. The statistics need the days; the days need randomization.

Worked example. Rohan re-runs his opening-line test properly: two weeks, contact-level randomization every window, every day. Result: Variant A completion 60.4% (1,512 / 2,503 RPC), Variant B 63.1% (1,579 / 2,503 RPC). A 2.7-point gap. Is it real? At ~2,500 RPC calls per arm, the margin of error on each proportion is roughly ±1.9 points, so the difference comfortably excludes zero — B is genuinely ahead. Then he checks level 6: promise-to-pay kept is 11.8% for A, 11.6% for B. Flat. B holds more people on the line but collects no more. He keeps A. The "win" was real and worthless — exactly the outcome a disciplined test should surface before you ship. The same patience our 30-day voice AI pilot playbook builds in: measure long enough to be sure, short enough to act.

Contrast that with the original test. The first run skewed Variant B toward bounce week and showed 51% connect against A's 42%, promise-to-pay four points higher. Nobody questions a 9-point gap — but the gap was the calendar: bounce-week callers answer more and pay more regardless of the opening line. The first test did not exaggerate a real effect; it manufactured one. That is the difference between a confound and noise: noise scatters around the truth, a confound points you at a number unrelated to what you changed.

Do not eyeball significance. A 2.7-point gap on 2,500-per-arm samples is real; a 7-point gap on 200-per-arm samples is not. The reason is sample size, not the size of the gap. If your platform shows a "winner" badge that fires after a few hundred calls, ignore it.

Tooling: what to ask a platform about experimentation

Most voice AI platforms in India can technically run two scripts; very few make honest testing easy. When you evaluate a vendor — or audit your own build — push on these:

1. Contact-level randomized split. Can the platform assign each number to an arm at dial time, inside every window, automatically? Or does "A/B test" mean you upload two lists? List-based splitting is where calendar confounds enter. Insist on randomization at the contact level.
2. Holdout support. Can you keep a true control arm — old script, untouched — running alongside every test, indefinitely? A permanent holdout catches slow drift a one-off test misses.
3. Funnel-level reporting per arm. Can you see attempts, connect, RPC, completion, intent, and conversion broken out by arm — not just connect rate? If the dashboard only shows top-of-funnel by variant, you will optimize the wrong thing.
4. Confound controls in the cut. Can you filter results by day-of-month, DPD bucket, call window, and circle? Without these slices you cannot tell a real lift from a calendar.
5. Sample-size and significance built in. Does it tell you when a result is significant, or just show two numbers and let you guess? Be wary of tools that flash a "winner" badge after a few hundred calls.
6. DLT-aware variant management. Can it map each script variant to its registered template header and stop you dialing an unregistered variant?

On build-versus-platform: dialing and telephony you should almost never build yourself — number rotation, retry logic, and carrier connectivity are hard, regulated, and a solved problem. The experimentation layer is where larger NBFCs and D2C brands benefit from owning the analytics: their definition of "conversion" lives in their LMS or CRM, and only they can join a call outcome to a payment that cleared three days later. A workable middle path: platform for dialing, your own warehouse for the level-5 and level-6 truth. The platform tells you who completed; your data tells you who paid.

Compliance: the DLT constraint on testing script variants

Here is the constraint that catches campaign leads off guard. Under TRAI's DLT framework, the content of a registered voice/SMS campaign is tied to an approved template and header. You cannot treat script copy the way an email marketer treats a subject line. Each materially different script variant intended as a registered communication needs its own registered template and header, and registration takes time.

Practically, script-copy A/B tests have a lead time. To test "open with the lender name" against "open with the borrower name and amount" as registered promotional templates, you register both before the test starts and budget days, sometimes longer, for approval. You cannot decide on Monday to test new copy and dial it Tuesday. Build a small library of pre-registered variants so your roadmap is not gated on registration every cycle.

This is also why call-window, retry-cadence, and voice tests are operationally easier than script-copy tests — they do not change registered content. Starting an experimentation program, begin with window and cadence tests while your script variants sit in the registration queue.

Separately, DPDP 2023 governs the personal data in these campaigns. Your test framework is not exempt: randomization, holdouts, and analytics all process borrower data, so purpose limitation and consent records apply to test arms exactly as to production. Keep your holdout inside the same consent and retention rules as everyone else. A test cohort is not a loophole.

A six-to-eight-week implementation playbook

You do not need a data-science team to run this well. You need discipline and a calendar. Here is a program that takes a campaign-ops function from "we swap scripts and hope" to a real testing cadence.

1. Week 1 — Instrument the funnel. Make sure you can see all six levels per campaign: attempts, connect, RPC, completion, intent, conversion. If conversion lives in your LMS or CRM, build the join now. You cannot test what you cannot measure.
2. Week 1–2 — Establish baselines and confounds. Pull eight weeks of history. Chart connect and conversion by day-of-month, call window, circle, and DPD bucket. This is your noise map: now you know what a normal swing looks like, and what size of lift is worth chasing.
3. Week 2 — Run one window test. Easiest, highest-leverage, no DLT dependency. Randomize numbers across two or three call windows, hold everything else constant, run until you hit your pre-computed sample size. Read connect and conversion.
4. Week 3–4 — Run one cadence test. Vary only the retry gap and max attempts. Measure cumulative RPC and conversion across the full attempt sequence, not single-attempt connect.
5. Week 4 onward — Queue script variants. Register two opening-line templates with DLT now so they are approved by the time the earlier tests finish. Test on completion and conversion, never connect rate.
6. Week 5–6 — Add a permanent holdout. Carve out a small randomized slice that always runs your current best-known config, untouched. It is your drift detector and honest baseline.
7. Week 6–8 — Set the cadence and write it down. One variable at a time, pre-registered metric, pre-computed sample size, no peeking, decision tied to the level-6 outcome. Put it in a one-page protocol every analyst follows.

If you run abandoned-cart recovery, sequence the same way but read cart value as a segment — high-value carts behave nothing like low-value ones, and our breakdown of abandoned-cart recovery with a voice-AI-plus-human hybrid explains where the handoff threshold sits. The abandoned-cart recovery use case and EMI payment reminders use case pages show the funnel shapes you will test against.

What changes in the next 12 months

Three shifts are coming. First, adaptive routing — platforms that re-allocate dialing toward the winning arm automatically. Useful, but a bandit that shifts traffic mid-test breaks naive significance math, and most vendors will not warn you. Treat "auto-optimizing" features as something to audit, not trust.

Second, better accent handling. Today the "Hindi" demo is Delhi Hindi, and real-world WER runs 1.6–2.4x higher on Patna, Jodhpur, and Lucknow accents — which quietly contaminates intent-capture tests in Tier-2/3 cohorts. As models close that gap, language and voice tests in those circles get cleaner, and lifts you could not measure before become visible.

Third, regulatory tightening. DLT enforcement and DPDP rulemaking will keep maturing, and the gap between window/cadence testing (easy) and script-copy testing (registration-gated) will widen. Plan your roadmap around it. For feedback campaigns, the same discipline carries over to NPS and CSAT calls — see how AI voice agents perform on NPS and CSAT feedback calls in India.

Bottom line

A/B testing voice AI campaigns in India is not hard to do — it is hard to do honestly. The mechanics take an afternoon; the discipline takes a protocol: one variable at a time, contact-level randomization inside every window, a pre-computed sample size, a pre-registered metric, no peeking, and a decision anchored to the bottom of the funnel. Most "winning variants" in Indian outbound are time-of-day or day-of-month confounds wearing a costume. Build the noise map first, respect the DLT lead time on script copy, start with window and cadence tests, and judge every result by what it does to conversion — not connect rate.