newoaks.ai › Blog › Can I A/B Test AI Voice Scripts to Book More Appointments?

Can I A/B Test AI Voice Scripts to Book More Appointments?

RayJul 23, 2026

Yes. Randomly assign comparable, eligible leads to Script A or Script B, keep the assignment attached to every downstream record, and compare:

Booked appointment rate = unique booked appointments ÷ eligible leads, calculated separately for each randomized variant.

For a stronger business decision, also compare attended appointments per eligible lead and monitor cancellation, no-show, call-quality, and cost guardrails. The goal is not to find the call that lasts longest; it is to determine which complete script package produces more valuable appointments without creating operational or compliance problems.

Can I A/B test different AI voice scripts?

Yes—if each eligible lead is assigned to one variant before contact, assignment is reasonably random, and the outcome is recorded consistently.

A valid test might assign leads to:

• script_a: the current production script
• script_b: a revised script

You can test an opening statement, qualification order, objection response, urgency language, or closing question. Keep other important conditions stable where possible, including the voice, phone number, offer, lead source, calling hours, retry policy, calendar availability, and booking workflow.

Testing a complete script package is acceptable when the business question is, “Which version performs better?” If the goal is to identify the causal effect of one individual element, isolate that element or use a properly designed factorial or sequential experiment. Changing several elements at once makes the result harder to interpret.

Random assignment should be system-controlled rather than chosen by staff. Manual alternation can work only when the process prevents people from sending higher-quality leads to a preferred variant.

What should I measure?

Use booked appointments per eligible lead as the primary KPI:

```text

Booked appointment rate for a variant

= unique leads with a valid booked appointment

÷ unique eligible leads assigned to that variant

```

For downstream quality, also calculate:

```text

Attended appointment rate

= unique leads with an attended appointment

÷ unique eligible leads assigned to that variant

```

The denominator matters. The primary analysis should normally be intent-to-treat (ITT): every eligible lead is analyzed under the variant to which it was randomized, whether the call connected, the lead answered, or the conversation completed. This preserves the benefit of randomization and measures the real-world effect of deploying the script.

You may separately report reached-lead and completed-conversation rates as diagnostic metrics:

• Reach rate: answered or successfully connected calls ÷ eligible leads
• Conversation-completion rate: completed conversations ÷ eligible leads or reached leads, with the denominator stated clearly
• Booking rate among reached leads: booked appointments ÷ reached leads

Do not substitute a reached-lead rate for the ITT booking rate. A script that works well only after a successful connection may still produce fewer appointments if its calling or retry experience results in fewer completed conversations.

Which events belong in the funnel?

Define event ownership before launch so behavioral stages are not confused with business outcomes.

A practical funnel is:

1. Eligible lead: the lead meets the predefined inclusion rules.

2. Variant assigned: the experiment service records the assignment.

3. Call attempted: the telephony system creates a call record.

4. Call answered or reached: the call connects sufficiently for the lead to interact.

5. Conversation completed: the interaction reaches a defined end state.

6. Booking attempted: the agent or lead initiates a booking action.

7. Appointment booked: the scheduling system creates a valid appointment.

8. Appointment confirmed: confirmation is recorded, if applicable.

9. Appointment cancelled, rescheduled, attended, or marked no-show: the final business outcome is recorded.

Establish deduplication rules in advance. For example:

• Count at most one primary booking per lead during the experiment’s attribution window.
• Treat repeated booking API calls with the same idempotency key or scheduling ID as one booking.
• Decide whether a reschedule remains one appointment or becomes a new appointment; apply the rule consistently.
• Exclude duplicate lead records from the eligible denominator, or define a lead-level identity rule before randomization.
• Record cancellations and no-shows as later outcomes rather than deleting the original booking.
• If a lead books twice, define whether the first valid booking, first attended booking, or highest-value booking is the experiment outcome.

The CRM or scheduling system should usually own appointment status. Analytics platforms can receive copies of those events, but they should not silently become the authority for whether an appointment truly exists.

Does Google Analytics support close_convert_lead?

Google Analytics documents close_convert_lead as a recommended lead-generation event triggered when a lead becomes a converted customer. It is intended for lead-generation funnels, including situations where conversion occurs offline. It is not a dedicated appointment-booked event. (support.google.com)

For appointment experiments, use an event such as appointment_booked for the booking outcome, with parameters such as:

• experiment_id
• variant
• lead_id or an approved pseudonymous identifier
• booking_id
• appointment_type
• appointment_timezone
• value and currency, if a defensible value is available

You can also send appointment_attended and appointment_cancelled events. Mark only the events that represent the business outcomes you genuinely want to optimize as key events in GA4.

Use close_convert_lead only if your organization defines a booked appointment as a converted lead under its lead-management model. Otherwise, using it for appointments can make the Lead acquisition reports answer a different question from the experiment’s primary KPI.

How should I measure offline appointments with GA4?

GA4 Measurement Protocol can send server-to-server or offline events to supplement events collected through website tagging or Firebase. It is useful for sending CRM outcomes such as a booking, attendance, or later revenue event. Google describes Measurement Protocol as a supplement to automatic data collection, not necessarily a replacement for it. (developers.google.com)

The session_id and 24-hour requirement needs careful interpretation. Google requires a session_id and a request within 24 hours of the online session start for session attribution—that is, for the offline event to inherit attributes from that particular online session. This is not a general rule that every offline CRM appointment must be joined within 24 hours. If the experiment is lead- or call-based, a CRM, warehouse, or booking database may be a better source of truth than a GA4 session join. (developers.google.com)

If you send an offline event after the appointment is created:

1. Preserve the original lead or pseudonymous user identifier.

2. Preserve experiment_id and variant in your warehouse or CRM record.

3. Use session_id only when you specifically need GA4 session attribution and can meet Google’s timing and timestamp requirements.

4. Set event timestamps carefully when reporting an event that occurred in the past.

5. Validate the payload and reconcile GA4 events against the booking system.

The join does not need to happen inside GA4 to support a valid experiment. A warehouse query that joins randomized lead assignments to deduplicated booking records is often simpler and more auditable.

How do I connect a script variant to the appointment?

Create a persistent experiment record before the first call:

```text

experiment_id: voice_script_2026_07

variant: script_a

lead_id: CRM-specific identifier

assignment_timestamp: UTC timestamp

assignment_source: experiment service

```

Carry the assignment through the systems that create or update outcomes:

• Lead-management or CRM system
• AI voice platform
• Telephony provider
• Call and transcript records
• Booking or calendar system
• Attendance or practice-management system
• Analytics or data warehouse

A useful event schema includes:

```text

experiment_id

variant

lead_id

call_id

booking_id

appointment_status

event_timestamp_utc

business_timezone

```

Treat identity matching as a measurable data-quality problem. Report the percentage of calls and appointments that cannot be joined to a lead or variant. Investigate phone-number changes, duplicate CRM records, missing webhooks, failed API calls, transferred calls, and bookings made directly by staff.

Define an attribution window before the test begins—for example, bookings made within a specified number of days after the first eligible contact. Use one timezone convention, preferably UTC for storage, while retaining the business timezone for appointment dates and operating-hour analysis.

If a lead encounters more than one script, do not quietly assign the booking to whichever call was easiest to find. Choose a rule in advance, such as first-assigned variant under ITT, first completed conversation, or a separate contamination analysis. The primary report should identify leads with multiple variants and show how many were excluded from secondary per-protocol analyses.

What can Twilio contribute to the measurement setup?

Twilio’s call-tracking guidance describes several separate capabilities rather than one automatic experiment-reporting feature. Programmable Voice can support call tracking, custom parameters exchanged between applications, recording, and transcription. Voice Insights can provide call-quality and call-lifecycle data, while Event Streams can deliver selected Voice Insights events to an external data system. These features may require different products, configurations, integrations, and—in some cases—Voice Insights Advanced Features. (twilio.com)

A practical Twilio implementation might pass experiment_id and variant as application metadata at call connection time, then join the resulting CallSid to the CRM and booking records. That does not by itself randomize leads or attribute appointments; those functions must be implemented in the experiment and data layers.

Do not assume every call contains every quality field. Voice Insights availability depends on the call path and configuration. Twilio documents different event and metric sources for carrier, SIP, and Voice SDK calls, and some API or Event Stream capabilities require Advanced Features. Call summaries may also be completed after the call ends rather than being immediately final. (twilio.com)

Use quality fields as diagnostics, not as a reason to discard inconvenient calls after seeing the result. If technical failures are part of the deployed customer experience, include them in the ITT outcome and report a prespecified quality-adjusted analysis separately.

Which secondary metrics should I review?

Review metrics that explain the result and protect appointment quality:

• Reach or answer rate
• Conversation-completion rate
• Booking-attempt rate
• Booking-completion rate
• Cancellation rate
• Reschedule rate
• Attendance or show rate
• No-show rate
• Transfer or escalation rate
• Cost per booked appointment
• Cost per attended appointment
• Call failure rate
• Call-quality indicators, where available

Segment diagnostics by lead source, campaign, geography, business timezone, day of week, hour, retry number, and agent or voice configuration. Do not turn every segment into a separate winner claim; small segments can be noisy and may lack statistical power.

How much traffic do I need?

There is no universal lead count. Calculate the required sample before launch using:

• Baseline booked-appointment rate
• Smallest meaningful absolute or relative lift, called the minimum detectable effect
• Desired power, commonly 80% or 90%
• Significance threshold, such as 5% for a two-sided test
• Traffic allocation, such as 50/50
• Number of variants and any multiple-comparison adjustment
• Expected loss, contamination, or incomplete follow-up

For a two-variant proportion test, use a reputable proportion-test sample-size calculator and enter those assumptions explicitly. Optimizely’s examples illustrate that sample size changes substantially with baseline rate and MDE; its published figures are examples tied to particular assumptions, not universal requirements for voice or appointment experiments. (support.optimizely.com)

A two-sided test asks whether the variants differ in either direction. A one-sided test asks whether the treatment is better in a prespecified direction. Use a one-sided design only when a decrease would not be treated as evidence against deployment and the direction was chosen before looking at results. Otherwise, use a two-sided test.

Set a fixed-horizon decision rule before launch. Do not stop when one script is temporarily ahead, repeatedly inspect results without an analysis plan, or switch the primary KPI after seeing the data. If continuous monitoring is necessary, use a sequential-testing method designed for that purpose.

What statistical validity risks are specific to voice agents?

Voice experiments are vulnerable to more than ordinary random noise. Watch for:

• Lead-source imbalance: paid, organic, referral, and partner leads may have different booking propensities.
• Time-of-day effects: answer rates and calendar availability can vary by hour, weekday, or timezone.
• Agent or voice changes: a model, voice, prompt, transfer rule, or tool update can contaminate both variants.
• Calendar availability: one variant may receive more calls during periods with fewer appointment slots.
• Retry-policy differences: different numbers or timing of retries can change reach and booking rates.
• Repeated leads: the same person may be assigned more than once or encounter both scripts.
• Operational interventions: staff callbacks, manual booking, promotions, or outage recovery can affect one group disproportionately.
• Delayed outcomes: bookings, cancellations, and attendance may occur after the test’s call volume has stopped.

Log deployment changes and outages. Balance or stratify randomization by major lead source when practical, and include the relevant source and time variables in the analysis. Keep calendar rules, retry policies, and escalation behavior constant unless they are explicitly part of the tested package.

Can I measure appointments that happen after the call?

Yes. Keep the original randomized assignment and join it to later booking, confirmation, cancellation, attendance, and revenue records.

Use a follow-up window long enough for the business outcome to mature. If attendance commonly occurs days or weeks after booking, do not declare the final attended-appointment result before that window closes. You can report an early booked-appointment result and a later matured attendance result, but label them separately.

A useful reporting table includes:

The figures above are illustrative, not benchmark statistics. In a real report, add the absolute difference, relative difference, confidence interval, test method, analysis window, and missing-match rate.

For example, Script B is 1 percentage point higher on booked appointments in this illustration, but only 0.3 percentage points higher on attended appointments. If the confidence interval for the booked-rate difference includes zero, the result is inconclusive rather than a confirmed win. Even if booking is statistically higher, a large increase in cancellations or no-shows could make deployment unattractive.

Should I test voice scripts before live traffic?

Yes. Use preproduction tests to find broken prompts, unsafe responses, booking-tool errors, escalation failures, and difficult conversation paths before exposing live leads.

Vapi Voice Testing provides an example of a voice-specific test workflow: a testing agent calls the voice agent on a real phone call, the interaction can be recorded and transcribed, and the transcript can be evaluated against a rubric. Vapi also documents test scenarios such as interruptions, objections, unavailable slots, rescheduling, voicemail, wrong numbers, escalation requests, and multilingual conversations. (docs.vapi.ai)

Pre-launch tests do not estimate the live booking lift because simulated callers are not equivalent to real leads. They are quality assurance, not a substitute for randomized production measurement.

What is a practical A/B-testing workflow?

Use this workflow for an example NewOaks AI architecture. It is an implementation pattern, not a claim that NewOaks AI automatically provides each capability. Confirm which assignment, persistence, CRM, booking, and reporting functions are available in your actual plan and integrations.

1. Define one hypothesis, such as: “A shorter opening will increase booked appointments per eligible lead.”

2. Define the eligible-lead population and exclusions.

3. Choose one primary KPI: unique booked appointments per eligible lead.

4. Define secondary outcomes and guardrails, including attendance, cancellation, no-show, cost, and call quality.

5. Predefine deduplication, attribution-window, reschedule, and multiple-script rules.

6. Calculate sample size with a proportion-test calculator using baseline, MDE, power, confidence threshold, allocation, and test direction.

7. Randomly assign each eligible lead before contact.

8. Persist experiment_id, variant, lead_id, and timestamps across the voice, telephony, booking, CRM, and reporting systems.

9. QA both variants with realistic test cases and verify that booking events are deduplicated.

10. Freeze non-script variables and log changes, outages, calendar constraints, and retry behavior.

11. Run the test to the prespecified sample and outcome-maturation window.

12. Analyze ITT results first, then use reached-lead, completed-conversation, and segment analyses as diagnostics.

13. Deploy the stronger package only when the primary result is sufficiently certain and guardrails remain acceptable.

14. Continue monitoring after rollout because a live change in lead mix, calendar supply, or agent configuration can alter performance.

Privacy, consent, recording, and telemarketing compliance

Before recording or transcribing calls, determine what disclosures, consent, retention controls, access restrictions, and deletion processes apply to your organization and the jurisdictions involved. Outbound appointment calls may also be subject to telemarketing, artificial-voice, do-not-call, consent, and internal-policy requirements. The rules can vary by location, call type, technology, and customer relationship.

Do not place unnecessary personal information in experiment parameters. Prefer pseudonymous IDs, restrict transcript access, encrypt sensitive data, define retention periods, and document which vendors receive call audio or text. Obtain legal and compliance review before launch, especially when calls are recorded, transcribed, made across jurisdictions, or used to contact consumers.

FAQ

Can I A/B test different voice scripts and measure which books more appointments?

Yes. Randomly assign comparable eligible leads to each script, retain the variant through the call and booking journey, and compare unique booked appointments per eligible lead. Also review attendance, cancellations, no-shows, cost, and call quality.

Is call duration a good primary metric?

No. Call duration is diagnostic. A longer call can still fail to produce an appointment. Use booked appointments per eligible lead as the primary KPI.

Should unreachable leads be removed from the analysis?

Not from the primary ITT analysis. Keep every eligible lead in the denominator for the variant to which it was assigned. Report reached-lead and completed-conversation analyses separately.

How do I handle a lead who encounters both scripts?

Define the rule before launch. Keep the original randomized assignment for ITT, flag cross-exposure as contamination, and report a separate per-protocol analysis only if the exclusions were defined in advance.

How should I treat cancellations, reschedules, and no-shows?

Keep the original booking event, then record later statuses separately. Define whether a reschedule remains the same appointment and count duplicate booking attempts only once according to a documented lead- or booking-level rule.

How long should the test run?

Run it until the prespecified sample size is reached and the relevant outcome window has matured. Do not stop just because one script leads early. If you need ongoing monitoring, use a statistical method designed for sequential analysis.

Should I test one script element at a time?

Do so when you want to estimate the effect of a specific element. Testing a complete script package is also valid when the decision is simply which package to deploy. Separate causal questions require separate or factorial experiments.

Can GA4 be the system of record?

It can be part of the measurement stack, but the booking or CRM system is often the authoritative source for appointment status. GA4 Measurement Protocol can send offline events, but session attribution has specific session_id, timestamp, and timing requirements. (developers.google.com)

Does Twilio automatically attribute bookings to Script A or Script B?

No. Twilio can help carry call metadata and provide call records or quality data, but randomization, variant persistence, identity matching, deduplication, and booking attribution must be implemented across the experiment and business systems. (twilio.com)

What should I test before exposing live leads?

Test interruptions, objections, unavailable slots, rescheduling, voicemail, wrong numbers, escalation requests, booking-tool failures, disclosure language, and multilingual paths where relevant. Automated voice testing can evaluate these scenarios, but live A/B testing is still required to estimate real appointment performance. (docs.vapi.ai)

References

• https://www.newoaks.ai
• https://newoaks.ai
• https://developers.google.com/analytics/learn/marketers
• https://developers.google.com/analytics/devguides/config/admin/v1/rest/v1beta/properties.conversionEvents
• https://developers.google.com/analytics/devguides/collection/protocol/ga4/reference/events