InGenious AI Case Study

InGenious AI, an agency that designs and builds AI-powered conversational interfaces, was an early adopter of the Facebook Messenger Customer Chat Plugin. Unfortunately, for one client's customer-support chatbot based on the plugin, the design's lack of support for free-text entry led to the default fallback message (AKA, the standard error response) being triggered 57% of the time, leading to a frustrating user experience and an unacceptable abandonment rate of 74%. According to co-founder Mark Chatterton, "This was a classic case of a bot doing more harm than good, and we had to act quickly to stop the bleeding."

Using Virtual Agent Analytics to quickly and easily analyze that client's chatbot performance, Ingenious AI was able to reduce not-handled messages (those with misunderstood, unsupported, or missed intents) by 72%, and subsequently the early-abandonment rate by 85%.

The chatbot in question was originally built for the Facebook Messenger mobile app to improve the customer-service experience. This bot was highly successful, with a conversion rate of 36%, default fallback triggered by only 4.7% of messages, and an early-abandonment rate of 3.8%.

Eventually, Ingenious AI helped the client deploy the chatbot to its website as well. But after completing the implementation, there was a surprising disparity in the number of users using free-text entry versus quick-reply buttons and image carousels: Whereas on the mobile app 93% of chatbot users preferred to interact using buttons and images, on the website that rate was only 39%. According to Chatterton, "This huge difference in the preferred method of interaction suggested that the bot's original quick-reply conversational flow was a poor fit for the website."

If working manually, it would take the team days, if not weeks, to comb through transcripts in the Facebook Pages Inbox for not-handled messages - not to mention verifying exactly when users had abandoned the chatbot. Fortunately, the bot was integrated with Chatbase Virtual Agent Analytics from the start. Says Chatterton, "Chatbase streamlined the analysis of our failing bot by grouping and showing a count of similar messages by intent. This report reduced the number of messages to read by over 66% - a great first step."

After determining that the top priority should be to tweak initial welcome messages and conversational-flow design to better support free-text entry, Ingenious AI used Chatbase's Session Flow to take a deeper dive into conversational-flow exits and their corresponding not-handled messages. "We discovered that critical information requested by customers was buried quite deep in the conversation flow," says Chatterton. In response, the team optimized the flow based on these learnings and new data coming in from the updated free-text conversational design. Those changes cut the not-handled rate to just under 16% and the abandonment rate to 11%, bringing the bot back to health.

Ingenious AI is continually analyzing customer messages and making small tweaks as needed and now has the chatbot performing at close to its original level. Using Chatbase, "We can quickly and easily pinpoint the source of poor experiences to fix failing bots, as well as prove to clients that our optimizations lead to measurably better customer satisfaction," says Chatterton. "Without it, it would take a huge effort to do those things manually."

chevron_left Back to Case Studies