Changing the Message – How We Increased Our E-mail Open Rate by 14%

Surveytime is our fastly growing Market Research platform, where we provide survey samples from around the world to the biggest market research firms, using machine learning to match users with surveys they are more likely to complete.

Unfortunately, similar to ad-tech, the market research industry is ridden with fraud due to the nature of its business model where users are compensated for completing surveys. As a company, we put an emphasis on taking preventive measures against it and providing our demand partners with real human responders, as seen in the case study we published last year where we detected an elaborate fraud operation based in China and essentially brought it down. 

A huge part of Surveytime’s success and growth, past the fact that we employ multiple proprietary fraud prevention tools to ensure human-verified samples, are the different features we employ to reconnect and engage with our users. 

Identifying the Problem – Repetitiveness of the Single Title

Constantly striving to improve these features, we took a deeper look at our ‘New Survey’ emails in order to improve their open rates. These emails are sent out to existing users, as part of our various methods of reconnecting and engaging with users, to notify them that we have a new survey that matches their profile, providing them with another chance to earn more income on our platform. 

Initially, we would send out these emails to users using the same repeating title “We have a new survey for you”. We started off with good opening rates, but they gradually decreased, until they’ve plateaued. 

This title worked for a while because it was clear, simple, and to the point. The title matched the content – we had a new survey and the users had a chance to earn another dollar, so the message was coherent with the value proposition to the user, but it was repetitive. It might have felt personal once or twice, but over time our users became oblivious to it. 

Creating Variations – Changing the Sentiment Expressed in the Title

Since we’ve noticed this deterioration in our opening rates, we decided to test out other titles and see how we can bring our numbers back up, and hopefully keep them there.

Naturally, we decided to run an A/B test, and the optimal way to go about it was to test out a variety of titles without making any other changes to the emails. 

Thinking of different titles, we’ve figured we can play with it a bit, make the titles more personal, and use humor more freely. As long as we’re keeping our message clear, we’re good, or at least, we believe we should be. 

Here are the initial titles we tested: 

1. Hurry up, we have a limited quota on this survey!
2. Tommy has a new survey, just for you!
3. It’s Surveytime! We’re being literal, it’s time for a survey
4. Join [number of users] users that earned from this survey in the last few hours!
5. Hey [user name], we have a secret survey just for you! SSHHH!
6. Hey, we have a secret survey just for you! SSHHH!

We focused on conveying a different message on each one of the titles. The first one had a focus on scarcity, thus creating a sense of urgency. The second one focused on recency, emphasizing that it’s a new survey that we’re offering. With the third one, we went for humor, using wordplay relating our name to the content of the email. 

Our fourth title emphasized the communal aspect of our surveys, while the fifth title aimed to create a sense of personal contact, reaching out to users using their name. The sixth one, though very similar to the fifth one, changed its focus once we omitted the users’ name and created a sense of exclusivity. 

Analyzing the Data – How The Messaging Influenced Our Open-rates

The results were as follows: 

As you can see from the results, all titles showed an increase in open-rate, though some showed a very small increase, which meant that we were right about our assumption that the repetitiveness of our original title was, at least partially, the cause for our open-rate decrease – just the change in and of itself has improved our numbers. 

The titles that showed the lowest increase were ‘It’s Surveytime! We’re being literal, it’s time for a survey’ – our humorous title and ‘Hey [user name], we have a secret survey just for you! SSHHH!’ – the title that aimed to create a sense of a personal approach. 

Trying to understand the low numbers our humorous title yielded, we’ve concluded that the joke either didn’t work for our audience or that it got lost in translation – literally. Since we have a global user base we’ve translated our emails prior to sending them, which means that our sophisticated wordplay lost its meaning (or that we were just not funny, which is very unlikely). Nevertheless, we’ve decided we won’t continue using humor in our titles.  

The biggest surprise though was our personal approach title, which went against our previous assumptions and yielded our worst results with only a 0.24% increase. Trying to make sense of these results led us to believe that reaching out to our users using their first name might have been perceived as stalkery instead of being personal – as if approaching them while using their name was crossing a line into the ‘you know too much about me’ instead of the desired ‘you know me so well’. 

Seeing these results was even more confusing when compared with our exclusivity title ‘Hey, we have a secret survey just for you! SSHHH!’ which had a whopping 14.17% open-rate increase. The only difference between the titles is that we abandoned the personal touch, and as I’ve previously mentioned, it changed the focus of the title to convey a message of exclusivity. 

Moving Forward – Keeping Our Users Engaged 

The two titles with the lowest open-rate increase were stopped after 2 weeks of testing when we saw no improvement. The rest of the titles kept running for what has now been a few months. Over time, we saw a slow decrease in our open-rate numbers, but they are currently still higher from our starting point. 

Analyzing these results, we’ve concluded our steps moving forward should be as follows: increasing our titles stock, focus most of the added titles to convey a message of exclusivity, avoid humorous titles, and avoid ‘stalkery’ titles.

As always, these results are not universal – they are the conclusions we drew regarding our own audience at a certain point in time. Our best advice is to keep running A/B tests yourself and always try to find new messages to engage and reconnect with your audience.