The headlines have recently been flooded with the news that Facebook and OKCupid have been conducting experiments on their users in an effort to collect data and improve the overall user experience.
Analysts and commentators from the public at large have raised the same concerns: is this ethical? Does running certain types of testing violate user trust, especially when the impact is carried off your website and into the “real world?” Where do companies draw the line?
While you shouldn’t let fears dominate your testing agenda, it’s helpful to have a clear idea of where your company stands on these issues and how that impacts what you disclose to customers and website visitors. Here’s a closer look at some of the ethics of testing and what these recent case studies can teach us.
What can conversion testing teach us about dating? Image credit: Flickr user Sergey Sus
Privacy concerns and their impact on conversion testing
It’s impossible to have a discussion about data collection and testing without addressing the topic of user privacy.
Testing and data tracking, to one extent or another, raise ethical and privacy concerns about your customers’ information. The EU has already taken a strong stance by requiring websites that use tracking cookies to display a prominent notice for users.
In traditional market research, you’re asking a survey respondent to opt into your research. You might be trying to influence their response on some level, by using an incentive or a script designed to encourage their participation.
But today’s analytics programs and tracking programs usually measure everything from demographic data to behavioral information in an under-the-surface, unobtrusive way. Largely, this information is analyzed in the aggregate. Even if you’re looking at specific consumer segments, an individual user’s information is protected.
While a test for a different website layout or new copy might change smaller details or encourage a user to take action, an individual visitor’s experience isn’t drastically affected nor is her personal agency removed in any decision being made. But that doesn’t mean that you shouldn’t consider the ethics of conversion testing, and determine where your company’s policy stands on key issues. As we’ll explore below, the public backlash to testing can have an impact on your brand.
Deliberately changing the user experience
Facebook publicly announced in mid-June that it was testing outside its usual scope of conversion and user testing agenda. The social media giant is notorious for regularly testing out new models for its newsfeeds, including what ads users see and who on their friends list get displayed with regularity.
A certain amount of testing is, I think it’s fair to say, implied when you opt into the social media experience. Studies show that social media sites are designed to be increasingly addictive. In a sense, this demonstrates a highly effective testing program that is rapidly formulating hypotheses, designing tests, rolling them out, and incorporating findings into UX design.
But when news broke that Facebook had manipulated the emotional content of almost 700,000 people users to see whether emotions or moods were contagious, reactions were mixed.
The study question was simple: if the posts you saw were manipulated to all be positive or negative, would that impact the tone of the updates that you wrote? In other words, would depressing content make you depressed or joyful content make you happy? The study found that yes, there is a correlation between the tone of what users see and their own subsequent moods.
The study’s lead researcher published a public apology, and explained that the study was designed to answer questions about whether positive content made users feel bad — as we so often see portrayed in the media — or whether negative updates caused users to avoid Facebook. Many felt that the study crossed an ethical boundary, and both the FTC and the EU have launched investigations. The New York Times pointed out an important factor: typically academic research of this nature would require getting study participants’ consent. Instead, Facebook relied on users’ agreement to their terms and conditions when signing up for the site. Was that enough? Public and legislative reactions have been conflicted.
It raises important questions for sites and testers: do you have the minimum disclosures in place on your site? And does any of the testing that you’re conducting suggest that you need to disclose anything more clearly to users?
In large part, standard conversion testing doesn’t require you to take special steps. But if you’re running more in-depth projects, it’s worth at least examining the question.
Testing impacts brand perceptions
Not long ago, I was following a well-orchestrated launch of a new workout program from a major name in the fitness industry.
Part way through the launch, he published an interesting newsletter to his email list. He addressed the fact that customers were sharing feedback on social media and with his customer service team, expressing frustration with pricing. One customer bought the product for $49, while another paid $69. People wondered if it was a technical glitch or some other error. It turned out to be part of a price testing program, designed to see what price point got the most people to convert to ultimately determine the final pricing for the product.
Unfortunately, because of the way that it was handled, long-time customers felt deceived. People who had paid $20 more for a product than others on the same day were left with a sour taste. The fitness company offered refunds to people that paid the higher price, but stood behind its statement that this was a standard marketing practice.
If you’re conducting tests, particularly price testing, it’s important to structure your deployment carefully. The damage to your brand and customer trust can be a real byproduct if you don’t.
Off the screen, into your “real” life
OKCupid and other data sites rely on proprietary algorithms that use the data you supply in your profile, the answers to screening questions, and your behavioral history to help determine who you are likely to click with. Do you routinely go for blondes or dates over six feet tall? Whatever patterns emerge from your collective data, the idea is that the results sites display to you will become increasingly focused and on target over time.
Recently, OKCupid released the results of three user tests that it conducted. In the first, user profile images were hidden.
The second test hid selected profile details to see how it would impact user perceptions and personality ratings. In a third test, users were told that certain people were determined to be a better or worse potential date than its software had actually scored them.
In a blog post, the company’s CEO was unapologetic about the testing, “If you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.”
He goes on to explain the role of testing in building a great product, especially online:
“But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out.”
Largely, of course, he’s right. And interestingly, the OKCupid news didn’t engender the same fury that the Facebook revelations did. Yet one of the research questions driving the OKCupid experiment was to understand how effective their software really is. Typically, OKCupid measures effectiveness based on how often people respond to messages, how long a conversation goes, and the exchange of contact information. They decided to test out their effectiveness more systematically.
The company told “bad matches” (with about 30% compatibility) that they were actually 90% compatible.
Messaging increased and the results showed that the simple suggestion of compatibility moved the conversation forward. OKCupid based this on measures like minimum number of messages exchanged and whether contact information was given.
But what would happen if these meetings got taken offline?
I’m imagining the worst-case scenarios: a fervent vegan inadvertently goes to dinner with a trophy hunter; a Democrat on the market for another Democrat ends up in an awkward political debate with a diehard Libertarian. It’s just one date and, no doubt, some people were surprised to connect successfully with people they’d never have gone for otherwise. It was their time, expectations, and trust of the brand that ultimately may have been impacted.
Takeaways
Each of these case studies illustrates one major point: it’s possible that the testing you’re conducting could publicly impact your brand.
Your customers, the media, or the public at large could one day be scrutinizing the way that you collect and use data. It’s a useful thing to remember, when you’re developing and implementing tests.
One of the biggest takeaways is that while it’s imperative that your privacy policy and user terms clearly state that you may be tracking data or conducting testing, it may not be sufficient for the testing that you have in mind. Particularly in contexts where you plan to take your testing beyond basic analytics or into the realms of customers’ wallets and offline lives, it’s important to think about your disclosure policy.
Could your actions have implications to your customers’ lives or ongoing relationship with your brand that you need to consider? Are you comfortable that you could reasonably state to a customer that you took their privacy concerns seriously and had sufficiently disclosed your intentions?
Ultimately, you’re conducting testing to develop a better product and help connect with those people who most urgently need your information, products, or services. However, having a clear ethical stance on testing for your company and how that data will be handled is an important step for any company considering conversion testing.
Taking the time to clarify your position and thinking through the implications of a testing plan will help save you from embarrassing brand gaffes and customer concerns down the road.
