CASE STUDY
Bringing Science to the Art of Messaging — An Experiment on Testing Language
Unlike most companies that come to us for help, this single sign-on platform for educators wasn’t primarily seeking new users. Renowned for removing friction from using tech in the classroom, they were already integrated in over 60% of schools nationwide. This client needed to convince existing users to increase adoption. They also had a new paid product to introduce to the market and needed to shift perception from backend functionality into a powerful ally for helping students achieve more learning goals.
Our Challenge:
Create messaging that motivated teachers and administrators to become “hi-fidelity” users. How could we help people realize the full potential of their product? Which behavioral strategy would successfully motivate users to change key behaviors?
Our Approach:
Although messaging is an art form, we needed science to inform our strategy. We tested multiple Behavioral Science concepts to see which messaging got the most traction.
Create Hypotheses
Get Executive Feedback
Step Outside the Boardroom. Qualitative Interviews.
- Support for processing fluency and Social Proof.
- Relativity not resonating as strongly as we imagined.
- Evidence for an entirely new hypothesis. (see below)
Test All 4 Messages
- Open rate: Processing fluency (clarity) and endowment effect were both a modest 8% more effective at getting people to open the email.
- Click rate: Here we saw a significant difference. The endowment and the aspirational messaging increased open rates by just over 50%.
- Clarity and Benefits: Users rated messages that incorporated processing fluency (clarity) and endowment messages as being on average 11% better at conveying both attributes.
- Interestingly, only the processing fluency message was ranked by users as a message that would encourage them to explore the product more.
First we needed to unearth some relevant themes. We demoed their products, reviewed their research, and did separate interviews with the executive team. Each theme was then boosted with one of the following Behavioral Science concepts:
Processing Fluency:
What it is: Making it easier for people to process information by using simple, familiar language.
Why we used it: People said they weren’t sure exactly what the client did and craved more clarity.
Social Proof:
What it is: Helping people make decisions by showing them what their peers did in similar situations.
Why we used it: Many teachers and IT professionals look to other districts to discover and share successful strategies.
Relativity:
What it is: Using comparisons so people don’t have to evaluate something in a vacuum.
Why we used it: Users who added more apps to the platform had a much more positive experience using this solution.
We worked with our client to refine the phrasing. The team also requested messaging that was more visionary, so we added a 4th hypotheses: aspirational messaging.
To validate our proposed strategies and get real-world feedback, we interviewed teachers and IT professionals with different levels of satisfaction and adoption.
What we learned:
The Endowment Effect:
What it is: When people already own an object they tend to place more value on it than on an object they didn’t own.
Why we used it: Districts spend a LOT of money on Edtech.
It’s time to get our message in front of real users. We controlled for participants profession, usage, and years at their job.
Email Experiment:
We divided the client’s email list into four randomized groups and sent each group a unique subject line and body that featured each hypothesis. We measured both open and click rate to see which hypothesis trigger more participation.
Email Results:
Survey Experiment:
Surveys in the email randomly assigned users to evaluate 1 of each of the 4 messages we wished to test. We asked people to evaluate the message for clarity, the salience of benefits, and level of interest generated by each message.
