Mark Atwood (fallenpegasus) wrote,
Mark Atwood
fallenpegasus

On base knowledge surveys

When I was in college, one of my Research Assistant jobs was to do clerical and basic number crunching work for base knowledge surveys. It takes scrupulous and expensive controls to prevent roughly a third of the surveys from being randomly answered for the lulz. Even with the most scrupulous controls and careful interview technique, there is still roughly 5% noise.

In other words, any newspaper headline of a newspaper article that is a restatement of the abstract of some random paper from some random academic journal based on some survey, especially if its a prepub paper or open access journal, that is of the form of "ONE THIRD OF LILPUTIANS ARE STUPID, ACCORDING TO SURVEY", is rank bullshit, and is anti-knowledge, as in anyone who reads it is less informed afterwards than before.

Show me the survey questions, the interview technique, the responder selection process, the population size, the population demographics, the pre-survey stats oversight board approval, the post-survey stats oversight board signoff, and the raw data, and THEN we will talk.

(My lead researcher when I was a RA sat on several of those Stats Oversight Approval Boards. I got well schooled in several of the ways that a researcher could lie to themselves, knowingly and unknowingly.)

This entry was originally posted at https://fallenpegasus.dreamwidth.org/857055.html. Please comment there using OpenID.
Subscribe

  • Razors

    I'm getting ads for I think five different "all metal" "get the best shave of your life" "throw away the plastic" razor startups. They all seem to be…

  • Doing what needs to be done

    On May 1st, one of my co-residents found one of the feral rabbits that live in the area cuddled up against a corner of the house. It was seriously…

  • The CTO of Visa, after listening to me present

    Some years ago, I was asked to travel to the corporate meeting center to present at a presentation-fest to the CxO staff of Visa. Yes, the one with…

  • Post a new comment

    Error

    Comments allowed for friends only

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 0 comments