6

Given the recent news about Microsoft's Tay chatbot being pulled off the internet after being exploited for its vulnerabilities, I was wondering if there is much research out there on the value of chatbots as part of a 'conversational interface' design trend, and whether there are specific applications that are more useful or popular compared to others. Slack and similar software applications have been using it for a while, but in so far as replacing actual humans online (because most of the time there doesn't seem to be anyone there) so that there is always someone/something you can talk to, there only seems to be a financial/resource gain for the company (minus the cost of implementation and maintenance).

Also, I was wondering if there are any documented processes or procedures for testing chatbots with actual end users, and if this testing is more quantitative due to the nature of the chatbots or if there are some qualitative assessments that are also done as part of the testing/commissioning process.

UPDATE: Facebook has just recently launched their Messenger Platform, which allows businesses and developers to access their chatbot API to build applications that incorporate this technology. No doubt Facebook has accumulated enough user data to come up with their design and concept, although how they used the data to create their design might remain a mystery.

Improve this question
5
  • Is your question "How do we test chat bots?" If so, wouldn't we test it like any other feature: user interviews; testing workshops; and analytics?
    – plainclothes
    Commented Apr 13, 2016 at 0:52
  • @plainclothes that would be my assumption, but having never had to work with something like a chatbot before as a UX designer, I know better than to make assumptions without being able to test them :p
    – Michael Lai ♦
    Commented Apr 13, 2016 at 1:00
  • So the first lightweight solution to testing your hypotheses is guerrilla testing and interviews. Then you prototype your idea and run more user tests. Is there something else you're looking for?
    – plainclothes
    Commented Apr 13, 2016 at 1:36
  • @plainclothes does that work when you are testing ideas on chatbots, which seems to require some interaction with end-users?
    – Michael Lai ♦
    Commented Apr 13, 2016 at 1:39
  • Like every other feature in a digital product? Of course. The fidelity of the initial prototype depends on the complexity of your concept, but the goal is to keep it as simple as possible.
    – plainclothes
    Commented Apr 13, 2016 at 1:43

1 Answer 1

Reset to default
3

Chat bots are not a new concept, but still in their infancy. Thanks to deep learning they start to mature. A lot of user research should be done so that they can be accepted by the society and avoid being called stupid bots.

Amelia is an interesting implementation: a virtual call centre agent that uses deep learning which can successfully handle 60% of the calls according to its developers. Also, maybe chat bots for visually impaired people will become famous.

About how to evaluate conversational interfaces. I think the best approach is more qualitative than quantitative. Diary studies with interviews will give great insight into what is happening. Quantitative metrics may be duration of conversations, number of conversations per day, turns taken, etc.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.