Problem
Chatbots did a good job in task-driven dialogues, but not yet be able to engage in open-domain conversations.
Alexa Prize is a worldwide competition on Conversational AI sponsored by Amazon. We are one of the ten university teams of Alexa Prize Grand Challenge 3.
Solution
‍We designed and developed an activity-centric chatbot to converse coherently and engagingly with people on a wide range of daily topics.
Outcome
From Jan to Apr 2020, we improved the Avg Session Time (min) from 1.01 to 1.41 and Avg Feedback Rating from 2.9/5.0 to 3.4/5.0.
Read our publication!
Overview

Develop a conversational AI for 100+ million Alexa users.

Team

Sabrina Lyu (Design Lead)
Han Xue (Research Lead)
Dakota Dong (Development Lead)
Ember Liu (Product Manager)
Neil Thawani (Project Coordinator)

Timeline

Jan - Aug 2020

Tools

Figma
Zeplin
ProtoPie
React

Role: Design Lead  Timeline: Jun 2019 - Aug 2020    Tools: Alexa Developer Console, Python, Rasa

Team:
Fanglin Chen (Team Lead)
Ta-Chung Chi (Algorithm)
Sabrina Lyu (Design)
Jiachen Gong (Data Analytics)
Tanmay Parekh (Development)
Rishabh Joshi (Development)
Anant Kaushik (Development)


We developed a social bot that engages users in sharing daily personal experiences in multiple domains. Our work contributes to Conversational AI in two aspects:

1) We extract common-sense knowledge expressed in large-scale user utterances in conversations, and find that more than 20% of the shared information is related to personal life, such as social relationships and individual activities.

2) Based on the underlying structure of daily life common sense knowledge, we decompose the task of open-domain social chat into a dialog management problem over a set of independent topical bots.

In addition to analysis of the effectiveness of the critical components in our design, we also present analysis on the breadth of common sense knowledge expressed in conversational language and the depth of conversations that can be grounded on common sense knowledge.

Manage multi-domain social chitchat through an activity-centric conversational backbone.

Team

Sabrina Lyu (Design Lead)
Han Xue (Research Lead)
Dakota Dong (Development Lead)
Ember Liu (Product Manager)
Neil Thawani (Project Coordinator)

Timeline

Jan - Aug 2020

Tools

Figma
Zeplin
ProtoPie
React

Nowadays, most of the chatbots did a good job in task-driven conversations - like, order a pizza, turn on the light, or request news. Conversational systems operate within restricted domains. However, these systems are not capable of having natural conversations of the quality that people expect from each other. Human-like conversational skills are difficult for machines to replicate. Key abilities include being able to engage on a wide range of subjects ("open-domain"), conversational intelligence (tracking context, monitoring progress) and social intelligence (building manage multiple goals in a conversation.

In our work, we distill human conversational knowledge from large-scale activity disclosure, and implement domain-specific chitchats in the resulting activity-centric conversational backbone.  

Our dialog framework also has the learning capability to predict appropriate topic switches, and self-evolve to pick up user feedback automatically.

The Process

Human-in-the-Loop Design

Team

Sabrina Lyu (Design Lead)
Han Xue (Research Lead)
Dakota Dong (Development Lead)
Ember Liu (Product Manager)
Neil Thawani (Project Coordinator)

Timeline

Jan - Aug 2020

Tools

Figma
Zeplin
ProtoPie
React

In the beginning, we hypothesis a conversational framework structured by the most frequent daily activities and build a Minimal Viable Product. Through this MVP, we were able to collect a large amount of data (over 200 conversations per day) and analyze the topic distribution in natural user activity disclosure.

Learning from the real data.

Our dialog framework also has the learning capability to predict appropriate topic switches, and self-evolve to pick up user feedback automatically. The data analysis result is being used to refine our framework as well as automatically training the chatbot.

Design Principles
Information Architecture

Topic Classes & Knowledge Structure

The structure of topic classes is the backbone for domain-specific storylines. After classified to one topic, the conversation will be hand to a topical bot. The goal of each topical bot is to develop a 3 to 10 turns conversation piece through slot-filling.

Design

Response Strategy: the [ACK] [Info] [Cont.] Template

[ACK] Always acknowledge user input.
- “That’s interesting.” “Sounds cool.”

[Info] Give info related to the extracted entity. Fetch external knowledge when needed.
- “Me too! I’m also busy working everyday”

[Cont.] Invite users to tell more with topic development.  
- “What are your thoughts?” “How was it?” Reduce the users’ cognitive load by suggesting the topic development.

A goal-oriented framework to develop the conversation within a topical bot.

We used a goal-oriented framework to develop the conversation within a topical bot. For example, the slots defined in Work bot is subject, job_title, and sentiment. The goal is defined as slot filling. The conversational development will be around these three slots.


Structure of the conversation flow (example of "work bot")


Conversation Flow (example of "music bot")

Implementing each topical bot in Rasa

● Key advantage: lower development cost by treating data as the first-class citizen.

● Each rasa bot is hosted as an ECS module for Cobot.

Dialog model

NLU (Natural Language Understanding) model

User Profiling

Technical Highlights

System Architecture

Learning capability

We intentionally included a "feedback bot" and a "topic transition bot" component in our system architecture to build the learning capability in our dialogue framework. The topic transition bot component aims to predict appropriate topic switches, and the feedback bot component is to pick up users' feedback and apply it to the revision of our system. With these components, our system can self-evolve automatically.

Learning to switch topic

Assuming that the coverage of topical bots is high enough, we counted the natural transition from past user data, and then normalize it to get the transition probability. To ensure the transition utterances are natural, we designed templates for the entry of each topical bot.

· Activate topic transition bot once there is low user engagement.
· The next topic is learned through past natural transitions given similar dialog context.

Learning from feedback

When conversational breakdown is detected (“you already asked me.” “I said I don’t want news.” “Alexa you lost your mind.”) We first acknowledge the error and then ask the user to give suggestions for us. According to the feedback we got, we mapped the entities extracted in user utterance to an existing topical bot.

· Activate feedback bot once breakdown is detected.
· Translate feedback to desirable bot response given similar dialog context.

Reflections

Embrace changes in a fast-paced environment

Within one year of this project, a lot of things have changed, from team members, goal and prioritization, to our system architecture. I keep being adaptive to embrace the changes in process and learned moving and iterating fast in the fast-paced environment.

The biggest challenge came with the COVID. After March, our information architecture based on an activity-centric framework suddenly failed since people had to stay at home and avoid activities like parties, hang out, etc. What people talk about in the conversation has changed a lot. I had to quickly come up with and test new options to navigate indoor activities. Luckily, facing this challenge, we discovered a new idea of learning from the shifting of entities (keywords) and developed a more adjustable framework.

I enjoy fast-paced environments which allow me to grow fast.

Communicate as explicit as possible

Working in a team with 7 developers, I communicated a lot and learned how important it is to be explicit. Design artifacts need to be simple, visual, and as explicit as possible so people can see for themselves and understand the flows. When it comes to specs, they need to be concise and specific so engineers won't be confused.

While evaluating the system and report issues to the team, I tried reduce the amount of work and frictions for engineers by prioritizing action items.