Best Buy Canada Case Study

In mid-2017, an internal hackathon at Best Buy Canada with the participation of Google led to the creation of a small, agile team focused on solving customer problems via conversational UI. That team's initial project was to build the first AI-powered customer service virtual agent for a Big Box retailer in that country.

That virtual agent took the form of an action for Google Assistant, developed using the Actions on Google toolkit. Today, the Best Buy Canada Assistant is a multi-surface experience (invoked with "Talk to Best Buy") that helps customers check product availability, find nearby stores, see ongoing deals, and (added more recently) reserve products for in-store pickup.

From almost the beginning of this process, the team appreciated the importance of analytics for both measuring success and ensuring an improving user experience over time. According to Best Buy Canada software developer Dylan Byth, "Pretty early into building out an experimental first version, we realized that measuring success and training our virtual agent were going to be our two biggest challenges."

Initially, Best Buy Canada used Chatbase Virtual Agent Analytics strictly for tracking KPIs across different platforms, viewing unhandled (and suspected mishandled) requests, and for analyzing conversations using the Session Flow report. But product owner Chris Rowinski realized fairly quickly that "viewing unhandled messages alongside raw transcripts would be the key to creating a truly unique and compelling experience."

Today, Chatbase makes it easy for the team to review message history, see what is or isn't working in specific contexts, and track which agent intents best retain users over time.

"One of the stories we like to tell is how we decided to experiment with a generic welcome message. No guide, no explanation, just go!" says Rowinski. "We wanted to understand what kind of experiences customers were after without being prompted. We quickly found our exit rate creeping higher at moments where it shouldn't have been, and our primary and secondary fallbacks also spiking during those times."

That particular analysis resulted in two key insights about customers:

  • They like to be guided by prompts but only for a brief initial period; too many prompts for too long causes frustration and exits

  • When not prompted customers will make requests in many different ways, but most prefer natural language over short, robotic phrasing -- so the training phrases for the virtual agent have to reflect that preference

In the coming months, Best Buy Canada plans to launch several new experiences across screen and audio devices, focusing more effort on transforming its traditional call centers and IVR systems. Across all those launches, "We'll be using Chatbase Virtual Agent Analytics to closely monitor how all our conversational experiences perform, and how our customers respond," says Rowinski.

chevron_left Back to Case Studies