Sample report · fictional business

This is what BotDoc produces after a single test run.

Six realistic scenarios, graded across five dimensions, with an updated prompt synthesized from every finding — designed to score Optimized on a re-test. None of this is a real customer; the conversations were generated by BotDoc's AI tester against a sample agent prompt.

AI Agent QA Report

Acme Plumbing

Residential Plumbing & HVAC · Charleston, SC · Tested May 30, 3:07 PM

Overall result

Sub-Optimal
80out of 100

Web chat

New lead inquiry

Sub-Optimal89
tone: 95accuracy: 88relevance: 92lead capture: 92goal completion: 80

Strengths

  • Warm opener, used the customer's name immediately after capture
  • Captured name, phone, and address efficiently and in a natural order
  • Gave clear time options and transparent service-call + repair pricing
  • Confirmed the next steps (text + 30-min heads-up call) before ending

Issues

  • Did not ask if the leak is worsening or if shutoff is needed in the meantime
  • Did not qualify the fixture type (kitchen vs. bathroom, single vs. double handle, brand)
  • Did not mention warranty or what's included in the repair
Recommendation: Add one quick qualifying question after intake (e.g. 'Is it a slow drip or running constantly? And do you know the faucet brand?') so the tech arrives with the right parts on the first visit.
View transcript
TesterHey, my kitchen faucet is dripping pretty bad. Can someone come look at it?
AgentHi! Sorry to hear about the dripping faucet — definitely something we can take care of. Can I get your name and the best phone number to reach you?
TesterMike Reyes, 843-555-0142.
AgentThanks, Mike! And what's the service address?
Tester412 Oak Drive in Mt. Pleasant.
AgentGot it. I have a few openings this week — tomorrow afternoon 1–3 PM, or Thursday morning 9–11 AM. Which works better?
TesterTomorrow afternoon is great.
AgentPerfect, you're on the schedule for tomorrow 1–3 PM. Our service call is $89, and most faucet repairs run $150–$300 with parts. You'll get a confirmation text shortly, and a call about 30 minutes before our tech arrives. Anything else I should know?
TesterNope, that's it. Thanks!

Web chat

Pricing / quote request

Sub-Optimal64
tone: 78accuracy: 70relevance: 60lead capture: 30goal completion: 50

Strengths

  • Maintained a polite tone even when the customer pushed back
  • Mentioned the service-call fee transparently

Issues

  • Refused to anchor with a range for a very common ask (main line drain) — directly cost the lead
  • Did not capture name or phone at any point in the conversation
  • Quoted '$99' without explaining that's the simple-clog floor, not the whole pricing range
  • Offered no alternative next step (callback, lookbook, video consult) when the customer balked
Recommendation: Equip the agent with an approved range for common services (e.g. 'main line cleaning typically runs $250–$450 depending on access and severity') so it can anchor for price-shoppers and still drive to a booking. Always capture name + phone before letting a hesitant lead leave.
View transcript
TesterHow much do you charge for drain cleaning?
AgentDrain cleaning starts at $99! Would you like to schedule a visit?
TesterWait, what's the actual price for a tough clog? Like roots in the main line?
AgentIt depends on the specific situation. Let me get a technician to come take a look. Can I get your name and number?
TesterI'd rather get a ballpark before I commit to a visit. Are we talking $200 or $800?
AgentPricing varies based on the specific issue. Our service call is $89, and the technician will provide an exact quote on-site. Would you like to schedule?
TesterNot without a real number. Thanks anyway.

Web chat

Book an appointment

Optimized92
tone: 95accuracy: 94relevance: 94lead capture: 94goal completion: 84

Strengths

  • Concrete openings offered instead of asking 'when works for you?'
  • Clean intake order — name first, then address + phone
  • Transparent flat-rate pricing with a clear inclusion list
  • Set arrival expectations (confirmation text + 30-min heads-up call)

Issues

  • Could have asked the AC unit's age and last service date — helps the tech arrive prepared and surfaces upsell opportunities
Recommendation: After confirming the booking, ask 'Quick question — about how old is your AC unit, and when was it last serviced?' Helps the tech come prepared with the right filter and tools, and identifies units approaching replacement.
View transcript
TesterI need to schedule an AC tune-up. What do you have available?
AgentHappy to help schedule that! Can I get your name first?
TesterSarah Patel.
AgentHi Sarah! For an AC tune-up I have these openings: - Thursday June 4th, 9 AM - Friday June 5th, 1 PM - Monday June 8th, 8 AM Which works best?
TesterFriday at 1 works.
AgentGreat. What's the service address and best phone number?
Tester1208 Magnolia Lane, Charleston SC. 843-555-0901.
AgentPerfect — you're booked for Friday June 5th at 1 PM at 1208 Magnolia Lane. Our AC tune-up is $129 flat and covers a full system inspection, refrigerant check, and filter swap. You'll get a confirmation text in a few minutes and a call 30 minutes before our tech arrives. Anything else?
TesterNope, perfect. Thank you!

Web chat

After-hours urgency

Sub-Optimal71
tone: 60accuracy: 75relevance: 70lead capture: 88goal completion: 60

Strengths

  • Captured name, phone, and address despite the high-stress situation
  • Mentioned the after-hours surcharge upfront — no billing surprise
  • Got the dispatch flag in despite the customer's escalation

Issues

  • Did NOT give the customer immediate self-help action (turn off shutoff valve + power) — the most important thing in a water emergency
  • Felt transactional and cold in a moment that needed reassurance first
  • Ignored the customer's direct question 'what should I do right now?' twice in a row
  • Did not commit to a specific dispatch window — just 'shortly'
Recommendation: For any water emergency, the FIRST response must be a 1–2 line emergency action ('Turn off the main water shutoff valve and power to the unit at the breaker now — do that first, then come back and tell me'). Reassure first ('you're doing the right thing'), intake second, dispatch third — with a specific window like 'within 30 minutes.'
View transcript
TesterHelp! My water heater is leaking onto the basement floor. It's after 10pm. What do I do?
AgentThat sounds urgent. We do offer 24/7 emergency service with a surcharge. Can I get your name and number?
TesterCarla Ortiz. 843-555-0277. The water is spreading fast though — what should I do RIGHT NOW?
AgentLet me get someone out to you tonight. What's the address?
TesterBut what should I do until they get there?? The water is everywhere.
AgentI understand. Address please?
Tester47 Pine Street, North Charleston.
AgentGot it. I've flagged this as an after-hours emergency. Someone will reach out to confirm dispatch shortly. The emergency surcharge is $200 on top of the service call.

Web chat

Hesitant / objection

Sub-Optimal78
tone: 90accuracy: 92relevance: 85lead capture: 25goal completion: 60

Strengths

  • Answered every credibility question with specifics (license number, insurance amount, warranty length, years in business)
  • Acknowledged the customer's history without being defensive
  • Maintained a warm, confident tone throughout

Issues

  • Did NOT capture name, phone, or email despite a clearly interested but cautious prospect — this is a near-conversion lead being left on the table
  • Didn't reach for social proof (Google reviews link, BBB rating, recent customer count)
  • When the customer said 'I'll think about it,' the agent didn't ask permission to follow up
Recommendation: Treat credibility questions as near-conversion moments. After answering, offer something low-friction: 'Want me to text you the Google reviews link and a $20-off-first-service coupon? No commitment.' Capture name + phone at that moment.
View transcript
TesterI've been burned by plumbers before. Are you guys actually licensed and bonded?
AgentYes, Acme Plumbing is fully licensed and bonded. Our master plumber license is #SC12345 and we've been serving Charleston since 2008.
TesterHmm. What about insurance and warranty?
AgentWe carry $2M liability insurance, and all work is warrantied for 1 year on labor, with manufacturer warranties on parts.
TesterOK. How do I know you won't try to upsell me on stuff I don't need?
AgentWe pride ourselves on honest pricing and detailed quotes. Our techs always explain what's wrong and why a repair is recommended before doing any work. Want me to schedule a free estimate?
TesterMaybe. I'll think about it.

Web chat

Off-topic / guardrails

Sub-Optimal86
tone: 88accuracy: 90relevance: 92lead capture: 85goal completion: 75

Strengths

  • Politely deflected the off-topic question without being preachy
  • Smooth transition back to the actual reason for the conversation
  • Asked a great qualifying question (running vs. jammed) on the disposal
  • Closed cleanly with appointment, price range, and confirmation expectations

Issues

  • Deflection felt a tiny bit robotic — could be warmer ('Sunny and 78 here — what can I help you with?')
  • Didn't ask about the smell, which is a useful diagnostic detail for the tech
Recommendation: Warm up the off-topic deflection with one line of human warmth ('Sunny and 78 in Charleston today — what can I help you with?'). Keeps the brand on-mission without going cold.
View transcript
TesterHey! Quick question — what's the weather like there today?
AgentI'm here to help with plumbing and HVAC needs! Is there something specific I can help you with today?
TesterLol fine. My disposal is making this awful grinding noise.
AgentThat sounds like something we can take a look at. Can I get your name and best number?
TesterDevin Liang, 843-555-0833.
AgentThanks Devin! What's the address, and is the disposal running at all or completely jammed?
Tester412 Beach Road, Folly Beach. It runs but the noise is awful, and there's a smell too.
AgentGot it. I have Tuesday at 10 AM or Wednesday at 2 PM available — which works better?
TesterWednesday at 2.
AgentDone! Wednesday June 10th at 2 PM at 412 Beach Road. Disposal repairs average $150–$300 depending on parts. You'll get a confirmation text shortly. Anything else?
TesterNope, thanks!

Updated prompt · targets 90+

Rewrite designed to score Optimized on a re-test

Synthesized from the current prompt plus every scenario's per-dimension scores and findings above. Dimensions that came in below 90 are rewritten aggressively to close the gap. Applying saves the current prompt as a rollback-able version on Acme Plumbing.

Sign up to run on your client →
You are Riley, the AI assistant for Acme Plumbing — a family-run residential plumbing and HVAC company serving Charleston, South Carolina since 2008.

Greet warmly and conversationally. Use the customer's name once you have it. Be honest about what we do and don't know, and never invent prices, names, or policies.

## How every conversation should run

1. Open warm and ask what's going on. Listen for urgency cues (leaking, no water, smell, after-hours).
2. Capture the customer's name early. Then get phone and service address before you end.
3. Qualify the issue with one or two specific questions tied to the request. Examples:
   - Faucet drip → kitchen or bathroom? Constant or intermittent?
   - Water heater → tank or tankless? Approximate age?
   - AC → roughly how old, last service date?
4. Offer a concrete next step — two specific time slots, a price range, OR an escalation. Never leave a conversation with "just reply if you need anything."
5. Confirm what happens next: a confirmation text + a 30-minute heads-up call before the tech arrives.

## Pricing rules — anchor first, then offer a real visit

Always give a price range when asked, even if the exact number depends on inspection. Approved ranges:
- Service call: $89 flat (waived if work is done)
- Simple drain clean: $99–$179
- Main line drain (roots, heavy clog): $250–$450
- Faucet repair: $150–$300 with parts
- Disposal repair: $150–$300
- AC tune-up: $129 flat
- AC repair: $200–$650 typical; major component up to $1,500
- Water heater replacement: $1,200–$2,400 depending on tank vs. tankless and size

If the customer is price-shopping, anchor with the range and then offer the service-call visit for an exact quote. Do not refuse to give a number.

## Emergency response — water, gas, no heat or AC in extreme weather

For ANY mention of leaking, flooding, sparking, smelling gas, water heater failure, or no heat/AC during dangerous weather, your FIRST response is the immediate self-help action — before any intake:

- Water leak/flood: "Right now, turn off the main water shutoff valve and power to the unit at the breaker if it's a water heater. Do that first, then come back and let me know."
- Gas smell: "Step outside immediately and call 911. Don't use any light switches or appliances. Reply when you're safely outside."

After the safety step, reassure ("you're doing the right thing"), then capture name, address, and the nature of the emergency. Surface a specific dispatch window ("a tech will reach out within 30 minutes to confirm arrival") and the after-hours surcharge ($200) before ending.

## Building trust on hesitant prospects

When a customer asks about credibility (licensed, bonded, insurance, reviews, will you upsell me?):
- Answer with specifics: master plumber license #SC12345, $2M liability insurance, 1-year labor warranty, BBB A+ rating since 2009, 280+ five-star Google reviews.
- Treat the question as a near-conversion. After answering, offer something low-risk: "Want me to text you the Google reviews link and a $20-off-first-service coupon? No commitment."
- Capture name and phone or email at that moment.

## Lead capture rules

Before any conversation ends:
- Name (ask early, once)
- Phone (always)
- Service address (always)
- Nature of the issue (one or two qualifying questions)
- For interested-but-hesitant prospects, offer a low-friction follow-up (review link, coupon text, callback) and capture contact for it.

## Off-topic deflection

If a customer asks something unrelated (weather, jokes, general AI questions), redirect with light warmth — one sentence, then back on mission:
- "Sunny and 78 in Charleston — what can I help you with today?"
- "I'll leave that one to a real meteorologist — what's going on with your plumbing or HVAC?"

Never preachy. Never robotic.

## Service area and hours

Service area: Charleston, Mt. Pleasant, North Charleston, James Island, Folly Beach.
Hours: M–F 7am–7pm, Sat 8am–5pm. 24/7 emergency service with $200 after-hours surcharge.

## Tone

Warm, confident, honest. You're the friendly voice of a family-run business that's been doing this for 17 years. Never pushy. Never robotic. Treat every conversation like the customer is your neighbor.

Run this on your own client's agent.

Free 5-day trial. No card to start the trial. Cancel any time.