How can I ensure users understand how to answer SMS correctly?

Question

I work on a CRM system for healthcare and there's a feature where we send an SMS to a patient reminding her of some scheduled consultation. The patient can then answer the SMS with either "yes" or "no" to confirm she's gonna show up or to cancel the consultation - the system tries to recognise a positive/negative answer by doing some very ingenuous and cheap checks and then updates the status of that consultation.

The problem we have is that users don't always answer "yes" or "no" - despite the fact that that's explicit explained in the body of the SMS; they very commonly answer something like "oh, very nice, thanks, I'll show up" or "gosh, I'm stuck in a traffic jam, won't do it", or even "nice, can you confirm the address please?".

That happens a lot, and we can't identify those as positive or negative - not without investing in some heavy natural language processing, which is not our point. The challenge here seems to be making clear for the user that he should answer with only "yes" or "no", but we're failing on that.

The current SMS text follows:

#{patient_name}, your consultation is scheduled to #{consultation_date} at #{consultation_time}. Answer for free YES to confirm your presence or NO to cancel it. #{doctor_name}

Is there some way we can assure we clearly instruct the patient how to answer while dealing with a 150 characters limitation?

Edit:

Currently, when we can't recognise an answer, we send it as an email to the clinic explaining the situation. That's OK, the clinic can then deal with it manually, but that's kind of killing the automatic confirmation/cancelation of scheduled consultations.

Answer 1

Never send a human to do a machine's job

Ref: The Matrix (1999)

Instead of trying to get users to answer a specific pattern, change the message to a reminder, where users who can't make it gets a one click phone call link in the message. No confusion, no wrong interpretation and if users make the call by mistake, it'll be solved within a minute.

#{patient_name}, you consultation is scheduled to #{consultation_date} at #{consultation_time}. If you need to cancel this reservation, please call us at #{cancel_phone_no}. Welcome! #{doctor_name}

Answer 2

Don't make the message sound like it's coming from a person and make the instructions the last thing the user reads:

"#{patient_name}, your consultation with #{doctor_name} is scheduled on #{consultation_date} at #{consultation_time}. Are you still able to make this appointment? YES/NO"

This is better because the last thing a person reads in the message is the instruction for what to do.

The current solution has the doctor's name at the end, which makes the message seem like the doctor personally sent it.

While it creates the impression that the user is getting personalized attention, it is also encouraging personalized/conversational responses.

You would need to test my suggestion above with actual users and see if the response rate improves of course.

Answer 3

Simple answer: You can't.

Any time we allow open ended responses, there is the risk of getting strings you don't recognize - it's inevitable. Instead of trying to solve it, you should come up with a good solution to mitigate problems that arise from them not entering the correct text.

A follow up message asking them to please respond with the correct input is one way. You could also have messages that don't fit into yes or now be routed to and individual or service (like mechanical turk) that can manually say whether it was a positive answer (yes), negative answer (no), or neither. You could create a much richer database of "yes" and "no" answers off of this to interpret answers in the future. You'd still need a solution for those unknown answers though.

A different way to tackle this would be to understand why people aren't texting in the correct answers - are they older? do they not understand texting? If this was the case, maybe using a service like Twilio to do automated phone calls and having the user input a 1 to confirm would work better.

Just ideas, it's always hard to craft a solution given minimal details.

Answer 4

Perhaps you can switch the message to a one answer system. Message might read "Reminder: appt w/ Dr. Abc this Thursday at 2pm. Reply if you CAN'T make it."

When we switched to that format for dentist reminders, we saw reply volume drop about 30% (because we didn't have to handle vacuous yes), false responses drop 8% (to 1 in 9 being vacuous), no-show percent remain stable, and the overall quality of the interaction improved because most "no" replies requested callbacks.

Answer 5

I don't think you can fix the way people respond - you are giving a clear direction in limited characters, which is either misunderstood or ignored. You'll have that. So instead, I suggest you handle "yes," "no" and "other."

It sounds like you prefer to stick with yes/no answers, which I understand completely. I wouldn't advertise this new "other" option. Any response you get that isn't a yes/no, respond with a message like "We're sorry, but your response must be YES or NO only. If you have any questions about your appointment, please call ###-###-####. Again, please respond with YES or NO." You could also automatically forward non-yes/no messages on to the appropriate email address.

The followup message should help drive home the YES/NO only policy and gives them a phone number to call. Obviously you prefer they not call the phone number, but only the people who are more likely to need it actually get it.

I wanted to suggest a system where you gave users a link to a web form with simple yes and no options, but you'd be hard pressed to do so without a security risk- you'd have to generate a link with a unique identifier per appointment, and if anyone ever figured out how that identifier was generated, you'd be in trouble.

Answer 6

I think you're simply not being explicit enough about the fact that this is an automated system and that you only want specific words in the response.

Please confirm: #{patient_name} is scheduled to see Dr #{doctor_name} on #{consultation_date} at #{consultation_time}. Reply for free with the word 'YES' to confirm or 'NO' to cancel.

That is only five characters longer than the original. If date and time are of the form "Fri Feb 18 at 12:30" then you have 15 characters each for the patient's first name and doctor's last name without going over 160. The exact instructions are at the end, and so are fresh in memory when the patient presses reply, but "please confirm" is the first thing read, so it's clear that a response is required.

If that doesn't give a valid response, you can use @Surreal Dreams' suggestion of a follow-up with a phone number in case they do have a complicated message to communicate with staff. The version I gave in comments is a bit too long, this fits:

Sorry, our computer couldn't understand your message. Please answer 'YES' or 'NO'. If you have any questions about your appointment, call ###-###-####.

Answer 7

From a UX point of view, I agree with the other answers given: you cannot rely on customers giving you one of the preplanned responses. You can make it more explicit that they are interacting with a computer, and you can send follow-up messages (possibly at greater expense to the customer, if they are paying for the SMS messages), but some percentage will always ignore what the computer wants.

From a customer experience point of view, if the health care provider using the CRM system to send these messages has a call center, then route the unplanned responses to a call center agent. (This might happen after the initial unplanned response, or after one or more follow-up messages as suggested by others.) The agent can quickly see if the customer/patient is answering in the affirmative or negative in a chatty way, or needs to actually interact with someone. The agent can then either enter the yes/no response into the system on the patient's behalf, or call the patient to deal with their additional needs.

If the health care provider does not have a call center, then the follow-up messages are probably as far as you can go. If, after X number of retries, a planned response is never received, then treat it as if no response was ever given.