Reducing postal survey nonresponse bias by sample selection incorporating noncontact propensity : a thesis presented in partial fulfilment of the requirements of the degree of Doctor of Philosophy at Massey University

Abstract

Noncontact, the failure of a postal survey sample member to receive a survey request, is a potential source of nonresponse bias that has largely been ignored. This is due to the difficulty of separating the components of nonresponse in postal surveys when nothing is heard from potential respondents. Yet, the need to understand postal nonresponse is increasing as more studies move to mixed mode designs incorporating a postal element, and technological, resource and societal changes increase the attractiveness of self-administered surveys. Thus, this research sought to estimate the level of noncontact in postal surveys, to identify the direction and magnitude of bias due to it, and to investigate targeted in-field mechanisms for reducing this bias. A series of empirical studies involving New Zealand postal surveys fielded between 2001 and 2006 were undertaken to meet these aims. Noncontact was found to relate to survey-independent demographic variables (e.g., age, household composition). Furthermore, its incidence was estimated to be as much as 400% higher than indicated by ‘gone, no address’ (GNA) returns, although an envelope message tested as part of the research was able to increase levels of GNA reporting significantly. Thus, noncontact was established as a nontrivial source of nonresponse in the surveys examined. As far as bias is concerned, noncontacts had a different profile compared to refusers and ineligibiles, and were estimated to account for up to 40% of net nonresponse error for some of the variables in the surveys examined. Accordingly, there appears to be a clear opportunity for methods targeted at reducing noncontact bias to improve final survey estimates for a range of items. A number of potential methods for reducing noncontact bias were explored, but only one had both a compelling theoretical foundation and potential for wide applicability; the noncontact propensity sampling (NPS) scheme. In a resampling simulation study a prototype of the scheme, which increases the selection probabilities for sample units with a higher predicted propensity for noncontact, consistently improved the demographic profile of valid postal survey returns compared to a simple random sample (SRS). Furthermore, the scheme reduced nonresponse bias by an average of 28% as measured against a range of frame variables (e.g., age, gender) and 17% as measured against survey variables for which census parameters were known (e.g., religiosity, household size, qualifications, income and marital status). Although the prototype NPS procedure increased the standard deviation of simulated point estimates for a given sample size (1,500 in this research), the effect was small; an average of 4% for frame variables and 2% for survey variables. Furthermore, the scheme had almost no impact on reported cooperation rates and is likely to be cost effective compared to other potential targeted in-field mechanisms, particularly in situations where researchers regularly survey a specific population. Pairing the scheme with three common post-survey adjustment methods (frame or census age/sex cell weighing, and response wave extrapolation) did not lead to consistently better estimates than an unweighted SRS. But this was largely due to the shortcomings of these methods because in many cases combining them with either sampling scheme (SRS or NPS) actually degraded estimates. This reinforces the idea that researchers should expend effort minimising bias during the field period rather than relying on post-survey weighting to deal with the issue. Finally, since the NPS scheme aims to reduce error due to noncontact but is not expected to affect error due to other components (e.g., refusal, ineligibility), it presents an opportunity for researchers to begin decomposing the various facets of postal survey nonresponse bias, an important precursor to the development of other targeted bias reduction interventions. Thus, as a methodological tool, the NPS scheme may serve a dual role as both a bias reduction and decomposition mechanism. In addition to their implications for postal survey research, the methods developed and insights into noncontact established in this research are likely to have applications in other domains. In particular, they will inform activities such as research into online survey nonresponse, organisational database management cost reduction and list procurement.