Thursday, May 04, 2006

How to read a survey

Just a few of the things I look for when I look at a survey/study/poll:

*I read the wording of the questions: are they leading questions?
*I look at the answer stems: are they adequate and cover most possibilities?
*I look for gaps: what helpful topics and questions were omitted?
*I look for an analysis of study limitations. I'm always astonished to find that many times, a public survey report does not include the limitations of the study, a requirement in academia.
*How was the polling conducted? Does it seem reasonably fair? What was the sample size? What were the demographics? Did they look at ethnicity/party affiliation/socio-economic status? Did they offer non English versions?
*Finally, does the report summary/press release match with my sense of the report findings? Any significant or interesting findings omitted?