也說問卷調查

這不是政治文, 雖然都是從政治新聞說起.


話說我先後在不開放的部落格寫了兩篇[1]最近看到幾篇問卷調查的結果的感想. 簡單摘要如下:

(1)關於問卷的信度(reliability)與效度(validity): 主要是從這篇新聞(搶救馬聲望 研考會民調加問支持度 )開始.

March 20
more about survey validity
Let's first read the following news article.

搶救馬聲望 研考會民調加問支持度 〔記者黃維助、王貝林／台北報導〕馬英九總統就職後，施政滿意度低檔徘徊，研考會已規劃參考國外的元首「政治聲望」民調模式，擬將總統的施政表現滿意度與政治聲望（支持度）進行切割調查，藉此營造馬總統個人的高聲望支持度。 擬與施政滿意度切割 (後略)

I want to briefly point out a couple of things, not from a political viewpoint, but from a research viewpoint.

First of all, whether it is necessary, or valid, to separate "satistication" and "support" into two constructs( or some may even argue that it is just an index) are worth study. It usually takes a lot of effort to verify the validity of the scale. Adding one or dropping one item from the survey is not as cheap as it goes.

Second, the time interval between survey is another thing worth mention. I have no idea about how the survey is administrated. However, whether some of the samples are repeated may or may not (it needs further verification) impact the result.

Third, when there is a big gap between two items that are supposed to correlate in a certain way (in this example, 表現滿意度僅四成三，支持度卻高達六成四的落差成績) , it should lead us to be caution about the survey results. In other words, the reliability of the scale is highly questionable.

Last but not least, it is a big step (and in right direction in my viewpoint) that the newly developed item (i.e. support) is added to facilitate cross samples (cross countries) analysis.



(2) 如何算是一個合理的解釋? 這是從同一個調查兩篇不同報導的對照而來.

其中一段提到:

"小朋友眼中父母親不幸福的原因，「不景氣失業、缺錢」，「常吵架、爸爸喝酒」是孩子最憂心的。有22%表示同住家人生活有困難，也有21%的小朋友表示，學校繳費有困難。
蔡文哲表示，全球性金融風暴的整體壓力，似乎還沒有很明顯壓迫到孩子，可能有些父母工作時間減少、放無薪假，陪伴孩子的時間增加，反而使得今年整體幸福感較去年提升。但已經有孩子開始反應「怕家裡沒錢」，尤其在東部、南部的孩子心中，沈重的經濟壓力也替孩子帶來煩惱，預估明年這種現象會更明顯。"

實證研究者最擔心的就是當資料分析結果1)不顯著,2)與原先預測相反, 那如何提出合理的解釋來自圓其說.

是的,關鍵就在於合理兩字. 上面那段解釋合理嗎? 我不認為.

In my opinion, it is not because

1) lacks internal consistency: It first argues that the financial crisis appears not hit the children yet (全球性金融風暴的整體壓力，似乎還沒有很明顯壓迫到孩子), but later it argues the opposite (但已經有孩子開始反應「怕家裡沒錢」), so which one should we believe? Another contradictory statement/example is also available--on the one hand, it said "就區域而言，小朋友感受到幸福的比例，南部地區小朋友占85.6%最高；最低為東部地區占78.6%；而北部地區占83.5%，" but on the other hand, it showed "尤其在東部、南部的孩子心中，沈重的經濟壓力也替孩子帶來煩惱". Again, which one should we believe?

2) lacks convincing evidence to prove why the explanation (i.e. 可能有些父母工作時間減少、放無薪假，陪伴孩子的時間增加，反而使得今年整體幸福感較去年提升) is plausible. To make a valid argument, one would have to show that the assumption is warranted, and the reasoning is correct. The assumption here is 有些父母工作時間減少、放無薪假，陪伴孩子的時間增加.

Is there any evidence to support this assumption? To support this assumption, one would have to show that when parents do not work and/or on leave, they spend all their extra time to company the children. If the extra time is spent on, say finding a job or being depressed or having fight or drinking problem (常吵架、爸爸喝酒」), then the assumption is not warranted. An invalid asssumption would not lead to a plausible argument.

3) mistaken "correlation" as "causality": Even above is a valid argument, it only proves that "父母工作時間減少、放無薪假，" appears to be positively related to "整體幸福感較去年提升". It did not prove that it is "父母工作時間減少、放無薪假" which leads to "整體幸福感較去年提升".

[1]其實有三篇,另一篇是更早的從台灣民眾統獨立場看統計: some thoughts about survey, 講到的是meta-analysis, temporal effect 還有measurement and analysis
--------
延伸閱讀:
統計是沒有立場的嗎?
也談民意調整的方法