When building voice-enabled chatbots using Amazon Lex, one of the biggest challenges is accurately capturing user voice input for slot values. For example, if users need to enter an account number or verification code, the accuracy of speech recognition is critical. This is where transcription reliability scores come in handy to ensure reliable slot filling.
What is a transcription confidence score?
The transcription confidence score indicates how confident Amazon Lex is in converting the audio to text for the slot value. These scores range from low to high and are separate from intent/entity recognition scores. Lex provides a confidence score for each voice slot value that can be used to:
- Verify whether the spoken slot value was understood correctly
- Decide whether to ask for confirmation or re-prompt
- Branch conversation flow based on recognition confidence
Here are some ways to leverage confidence scores to improve slot processing:
- Progress confirmation
- High confidence (>0.9): Accept slot value and continue.
- Medium confidence (0.6-0.9): Ask user for confirmation (“Did you say 12345?”).
- Low confidence (<0.6): Re-prompt for slot value
- Adaptive re-prompting
- Customize re-prompt messages based on trust level
- Provide more specific guidance for unreliable inputs
- Provide alternative input methods if needed
- branching logic
- Route to human agent after multiple unreliable attempts
- Skip checking for consistently reliable inputs
- Adjust validation rules based on confidence thresholds
This post includes AWS CloudFormation templates that demonstrate these patterns that you can deploy to your AWS account.
Using trust scores strategically can create a more robust slot filling experience, including:
- Reduce errors when retrieving critical information
- Improve self-service containment rates
- Allows you to better handle difficult audio conditions
- Achieve smarter conversation flows
Implementing these patterns can greatly improve the accuracy and reliability of Lex voicebot slot filling while maintaining a natural flow of conversation.
Solution overview
The following diagram shows the overall architecture of the solution.
Deliver conversational experiences using Amazon Lex bots integrated with Amazon Connect contact flows.
Amazon Lex asks callers, “In short, how can I help you today?” Amazon Lex reviews the slot value’s transcription confidence score to influence the verification routine.
interaction model
Below are the different customer experiences that callers can experience, depending on what they say and the transcription confidence score that Amazon Lex returns.
Virtual agent:
Thank you for calling Acme Travel.
Virtual agent:
…In a nutshell, why are you calling today?
customer:
I would like to confirm my reservation.
Virtual agent:
got it. 5 Please tell me the character’s reservation number. It can be found at the top of the information pack you sent.
customer:
1 9 AKB (recognized with low transcription confidence: 0.4)
Virtual agent:
Okay, let’s try again. What is my reservation number?
customer:
1 9 AAB (more clearly pronounced: recognized at 0.8)
Virtual agent:
1 9 AAB reservations are currently in progress.
Virtual agent:
Thank you for calling Acme Travel.
Virtual agent:
…In a nutshell, why are you calling today?
customer:
I would like to confirm my reservation for 1 9 AZ Y.
Virtual agent:
Reservations for 1 9 AZY are currently in progress.
Virtual agent:
Thank you for calling Acme Travel.
Virtual agent:
…In a nutshell, why are you calling today?
customer:
I would like to confirm my reservation.
Virtual agent:
got it. 5 Please tell me the character’s reservation number. It can be found at the top of the information pack you sent.
customer:
1 9 AZY (recognized with transcription reliability of 0.6)
Virtual agent:
1 9 Did you say AZY? Please answer yes or no.
customer:
yes
Virtual agent:
Reservations for 1 9 AZY are currently in progress.
In the example conversation, the IVR requests reservation reference information from the customer. The received transcription confidence score is evaluated by enabling conditional branching in Amazon Lex based on the audio confidence score. These conditions check values against certain thresholds. When the transcription confidence score exceeds a high threshold (for example, above 0.7), the conversation advances to the next state. If the score is in the medium confidence range (for example, 0.4 to 0.7), the user is prompted to confirm the interpreted input. Finally, if the score is below a minimum threshold (for example, less than 0.4), the user is asked to retry and provide the information again. This approach optimizes the conversation flow based on the quality of the captured input, preventing false or redundant slot captures and improving the user experience while increasing self-service containment rates.
Prerequisites
You need an AWS account, an AWS Identity and Access Management (IAM) role, and a user with permissions to create and manage the resources and components required by this application. If you don’t have an AWS account, see How do I create and activate a new Amazon Web Services account?
Additionally, you need an Amazon Connect instance. You will use the instance Amazon Resource Name (ARN) in a later step.
Deploy the Amazon Lex bot and Amazon Connect flow
To create a sample bot and configure runtime phrase hints, follow these steps: This example creates an Amazon Lex bot named disambiguation-bot with one intent (CheckBooking), and one slot type (BookingRef).
- Sign in to your AWS account and select startup stack To deploy the CloudFormation template:
- for stack nameenter your name, for example.
contact-center-transcription-confidence-scores. - choose Next.
- Specify the following parameters:
- for Bot nametype disambiguation-bot.
- for ConnectInstanceARNenter the ARN of your Amazon Connect instance.
- for Contact flow nameenter a name for your Amazon Connect contact flow (for example:
lex-check-booking-sample-flow). - for Log group nameenter the name of the Amazon CloudWatch log group where conversation logs will be saved.
- choose Next.
- Leave all remaining settings as default and select Next.
- choice I acknowledge that AWS CloudFormation may create IAM resources.
- choose submit.
- Wait until the CloudFormation stack is successfully deployed.
- Assign the contact flow to your Amazon Connect billing number in the Amazon Connect console.
Configure transcript confidence score logic
After creating the intent (CheckBooking), you can use the visual conversation builder to configure transcription confidence score logic.
The following diagram is an example of how to add logic to an intent. Highlighted in red are branching conditions that use transcription confidence scores to dynamically change the customer experience and improve accuracy.
Selecting a node displays the following configuration options where you can configure branch conditions:
Test the solution
To test your solution, look at conversations that include words that you may not understand clearly.
- Assign the Amazon Lex bot to your Amazon Connect workflow.
- I’ll make a call.
Amazon Connect asks, “Thank you for calling Acme travel. In short, why are you calling today?”
- “I would like to confirm your reservation,” I reply.
- When asked for your reservation number, say any two numbers and three letters (e.g. “1 9 AZY”).
This test checks your confidence score and either displays “Your reservation 1 9 AZY is currently in progress” or asks you to confirm “1 9 AZY.”
Restrictions
Audio transcription confidence scores are only available in English (GB) (en_GB) and English (United States) (en_US) language. Confidence score is only supported for 8 kHz audio input. The Amazon Lex V2 console test window uses 16 kHz audio input, so audio input does not provide a transcription confidence score.
cleaning
To delete the infrastructure created by a CloudFormation template, open the AWS CloudFormation console and delete the stack. This will remove the services and configurations installed as part of this deployment process.
conclusion
Optimizing the user experience is at the top of the Amazon Lex conversation designer’s priority list, and so is getting information accurately. This new feature allows designers to choose confirmation routines that foster more natural interactions between customers and bots. Verifying each input can degrade the user experience and cause frustration, but failing to verify when the transcription is unreliable can jeopardize accuracy. These improvements allow you to create a more natural and performant experience.
For more information about how to use intent confidence scores to build more effective conversations with Amazon Lex, see Build more effective conversations with Amazon Lex using confidence scores and improved accuracy. Please refer.
About the author
alex buckhurst He is a senior Amazon Connect consultant at Amazon Web Services, focused on building innovation and customer-centric design. In his free time, Alex enjoys playing squash, honing his barbecue skills, and spending time with his family.
Kai Lorek is a Senior Professional Services Consultant for Amazon Connect. He works on designing and implementing scalable customer experience solutions. In my free time, I enjoy playing sports, snowboarding, or hiking in the mountains.
neil kapadia He is a senior software engineer at AWS, working on designing and building scalable AI/ML services using large-scale language models and natural language processing. He has been with Amazon for over five years, working on Amazon Lex and Amazon Bedrock. In my free time, I enjoy cooking, reading, and traveling.
Anand Jumnani is a DevOps Consultant for Amazon Web Services based in the UK. Outside of work, he is passionate about club cricket and enjoys spending quality time with his family and friends.
