How a Slack bot is using analytics to better understand usage and improve the product offering.
Our team had been exploring Slack and Facebook Messenger bots and recently launched a Slack bot, Game Monk — it enables teams to play trivia amongst themselves.
On launch, the team wanted to better understand the usage of the bot to be able to improve the gameplay and user experience.
The issue is traditional analytics tools do not work well for bots
The tracking mechanisms are different.
Client side doesn’t work
Event based tracking overlooks rich information from the chat threads
The types of data captured are different
Users aren’t just clicking buttons or web links — they can post rich media into a bot like gifs and audio; pose free form, natural language questions; and even send in location information
The processing is different
With so many different types of data, the processing is inherently different
How do you handle sessions in Slack when you can have multiple users coming and going in a room, interacting with a bot?
The types of metrics you can get with bots are much richer, and more actionable
There’s sentiment analysis, conversational analytics, and AI response effectiveness — all which can be used to improve the bot experience
Dashbot to the rescue!
Realizing analytics were missing, the team switched focus to build a bot specific analytics platform — enabling other bot makers to have rich analytics as well.
With Dashbot, we’re able to see core usage metrics and more importantly, bot specific analytics — sentiment analysis, conversational analytics, AI response effectiveness, and even the fully recreated session transcripts.
Team Metrics
One of the basic metrics we wanted to see is how many Slack Teams installed the bot and how frequently they engaged.
In addition, we can drill down into the teams to see individual metrics around usage, sentiment, and messaging.
Session Handling
Session handling in Slack is quite interesting. Multiple users can interact with a bot in the same room and come and go — so what’s the session? This is quite a bit different from traditional web or mobile usage where one user is interacting one-on-one with the site or app.
With Dashbot, we’re able to calculate and report on true bot sessions. Notice how in some cases there are more users than sessions, and others more sessions than users, that’s because multiple users can be in the session at the same time.
At the same time, users can continue to interact in a chat room without communicating with the bot — there needed to be a way to ignore those non-bot messages, and we incorporated that into our platform.
Sentiment Analysis
What do users think of your bot? Do they like it? Are they ecstatic or are they getting frustrated?
The great thing about bots is — they’ll tell you!
Not only can you see the sentiment based on the session and individual messages, but you can look into the message threads and see what people are actually saying.
We’ve learned the trivia game can be quite challenging and people like to trash talk — both conditions can lead to negative sentiment, even though it’s not necessarily negative to the bot. Looking at the sentiment in the context of game play and the thread, one can see it’s often good natured ribbing amongst friends.
AI Response Effectiveness
How effective is your bot AI? How does it handle cases when it doesn’t understand the message? Does it prompt the user again, send a default response, or barrel along regardless of what was typed in?
With Dashbot you can see the messages sent in and out, and can transverse the flows to see what inputs triggered a default response, and where the AI broke down. In addition, you can view the full session transcripts to see what occurred prior to the break down. Using this information, can help improve the bot experience and increase overall engagement.
For example, in one session, when the question asked users to name as many NBA teams as they could, some of the users listed their responses inline separated by commas — the bot ignored those responses looking for single line answers instead. This is an opportunity to improve the bot handling as well as communicating to end users better.
The game has known failure responses when answers aren’t in the right format — we can look at those outgoing responses and see what preceded them.
For example, in the case of multiple choice questions, if the response isn’t recognized, the bot prompts: “Please answer A, B, C, or D.”
One might think a common cause for this error would be someone writing the choice out instead of the letter, but it turns out it’s a combination of laughter and not knowing the answer. This can be an opportunity to improve the bot with skip or hint functions, or even a better response to the happiness — i.e. “glad you liked it!”
At the same time, the bot prompts users for feedback after the game is over. We can see the messages that were sent in after the feedback message, as well as look at full threads to get a better idea of what was liked or disliked. Based on responses, while the game can be fun, we can also see the game can be too fast for some users.
Session Threads
With Dashbot, we’re able to see the top messages in and out, as well as the fully recreated session threads — with full multimedia — images, gifs, audio, locations, etc. This is useful for figuring out what leads to AI failures.
In one example, an international team played the trivia game and wasn’t able to answer many of the US specific questions — e.g. “Popular US TV Shows” and “State Capitals”.
Here’s a quick snippet of one of their threads. It quickly goes sideways with a series of obscenities in French.
The funny thing is, despite not knowing many of the categories, they keep playing:
Get Started — Integration is Easy!
Bot specific analytics can help improve your bot experience and increase user engagement.
Dashbot is simple to use — just add a few lines of code to your Slack or Facebook Messenger bot and you’re good to go. We handle the rest — all the data capture, processing, and reporting.
It’s free to use — you can sign up at www.dashbot.io. Simply integrate our sdk into your Slack or Facebook Messenger bot and you’ll see the data coming in immediately.
/blog
