4th Mar 2009, 22:16
I am sorry if it sounded like my work was related to the car industry in some way. I am but a small cog in a giant big pharmaceutical firm. Our market research is only used internally to help us figure out what projects to pursue and what features to include. I have been very unimpressed with how well that works. In reality, all the real decisions are made by top level executives who ignore the market research. It is just like a Dilbert cartoon.
Like any other big company, my firm makes us sign papers that say we understand that we are subject to immediate termination if we say things that can reflect poorly on the firm or are outside the narrow confines of press releases from the PR department. We are told that as long we don't identify who we work for, we can say what we want. But as soon as we represent ourselves as belonging to the firm, we can't say anything interesting.
The thing I was trying to point out was how hard it is to get good results. I think CR does a great job on data collection and a pretty good job on reporting fairly raw results.
But I lost any notion that CR is "unbiased" when I found out that they gave the 2007 and 2008 Camry "better than average" reliability ratings when in fact they had not collected any data on those cars yet. Then they had to recant their ratings and report they were actually "worse than average" and remove the "recommended" status. Judging from my experience in a big bureaucracy, it looks to me like the same kinds of Dilbert cartoon behaviors happen at CR just like they do in any organization of any size. Little cogs like me collecting good data - executives above me ignoring it and following their internal biases.
4th Mar 2009, 14:54
Any one who is a regular reader of CR knows what the expected responses are. CR does not want that confounding factor in there data, but I guarantee you they know it is there. Any one who handles market research data struggles with this. I have been in numerous data review meetings where we look at the data, discuss the problems, then at the end say, you know, all we can do is analyze and report what we got.
This kind of thing is one of the very hardest things about doing good market research. Even though what a researcher can present to the subjects is very neutral and unbiased, like the form you mentioned, that form is really only a very small portion of what the subject is responding to. The responders carry with them a world of experience, conversations, TV ads they saw, remembrances of things they read in previous CR issues. And it affects how they respond.
I do not mean to disparage the fine work that CR does, and the positive contributions they make to our society. I am just trying to share a little knowledge about these issues so people can better understand how to interpret and use the results that CR gives.
Any one who has been involved in data collection in market research has seen a subject trying really hard and getting frustrated and saying something like "what is my response supposed to be?" One way we normally try to control for those kinds of overly involved people is to throw out the very most positive responses, and the very most negative responses. Maybe you don't know that if you ever filled out a survey and gave a product all 10's for very best of everything, that response probably got discarded and not factored into the final tally.