The Problem With Surveys

By | February 25, 2006

I love BBC World, the satellite news channel, and I love offering feedback (rarely welcome, as readers will know). In the hope of satisfying both passions I joined the BBC World Panel where “users are invited to register and record their comments online and to take part in regular surveys and questionnaires specifically on viewing and programming issues.”

The surveys are handled by a company called eDigital Research which claims to be

unique in our field as we combine an in-depth research background with a thorough understanding of developing and managing Internet websites. Our propriety research programmes are developed in-house by our experienced team of developers allowing us to develop bespoke client programmes and react to the immediacy of the Internet. We are able to translate complex market research data into concise management reports that highlight key business issues effecting the ROI. 

With all due respect to eDigital, which also trades as eMysteryShopper, eCustomerOpinions, eGlobalPanel and ePollingStation, all this language sounds tired and out of date. “The immediacy of the Internet”? What does that mean, exactly? The Internet is a huge bunch of people. That’s what the Internet is. For sure the Internet is “immediate” but it’s not just about being fast, it’s about connecting to customers, listeners, surveyees, whatever. (And what are “propriety programmes”? Do they mean proprietary, as in “something exclusively owned by someone, often with connotations that it is exclusive and cannot be used by other parties without negotiations” or propriety, a noun meaning “correct or appropriate behavior”. Both kind of make sense here, which perhaps illustrates the poor use of language here.)

Anyway, as I was filling out yet another BBC survey this morning I realised how old this kind of approach is. The survey in question was just like the other surveys I’ve done in this series: They are composed of questions and either multiple choice answers, or ordering selections from drop down menus, all of which are time-consuming for the user without ever really zooming in on the user’s real priorities. This is the kind of thing:


(The numbers go up to 10.) There’s nothing intrinsically wrong with this, but it also needs to reflect the way the Internet has changed the way consumers interact with companies, and the expectation (however unevenly reflected) that their voices be heard beyond choosing a few options that don’t necessarily capture the flavor of their attitude towards the product. This kind of multiple choice thing is fine in quick surveys for busy shoppers outside supermarkets, but not in a survey of volunteers who have deliberately set aside time to offer feedback.

Then, the equation should be quite different. The surveyor should be asking “How can I vacuum up as much of the rainbow of this user’s attitudes and thoughts about the service/product in question before they go to breakfast?. Sifting through their thoughts may take me longer, but the quality and usefulness of that freeform feedback is going to be much more valuable to the client than simply a few PowerPoint graphs.”

This lack of effort to gain access to the user’s real feelings towards the product/service is reflected in the final question, a broad one (without a question mark, oddly) but with only a small box for the user to type in their response:


As you can see, I went on at length about my passion for English soccer, and my ideas for how BBC World could expand its coverage without having to fork out big bucks for actual soccer footage. But I also, knowing from previous experience that as far as I know respondents to these BBC World surveys never hear back from the surveyors, however much extra feedback we type into the freeform text boxes. added this final comment:

One last thing: communicate with your respondents. This survey is too Old Thinking. Start a conversation with your viewers that doesn’t just involve us clicking multiple choice boxes. Email your respondents with follow-up questions, engage us as human beings. Some of us love BBC World and want to see it do well. But throwing our considered responses to your surveys into a deafening silence is not the best way to engage or keep your viewers. Nowadays the market is a conversation. Use it. And use us.

How should someone do this? I would say the BBC Viewer’s panel should be just that: a panel of selected viewers (chosen, perhaps, for the quality of their freeform feedback), overseen by a panel leader who maintains a blog, throwing out occasional gems from viewers’ responses and updating viewers on the progress of the changes being wrought in response to these surveys. This morning’s survey, for example, was initiated by an email with the intriguing paragraph:

Because of the costs involved, there are many problems showing clips of sports on news channels. Should BBC World show only what it can, or would you like to see a rolling results show on the channel? Whatever you think, and even if you aren’t a sports fan, we’d like to know what you think.

Off to a good start. But that’s just the beginning. Use blogs, use discussion forums, use blog comments, use a Wiki, use Skype, use whatever it takes to find out, to really find out, what your viewers want from you. Let them guide the discussion, not a market research company with spelling issues.

One thought on “The Problem With Surveys

  1. Pingback: Anonymous

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.