Reducing postal survey nonresponse bias by sample selection incorporating noncontact propensity : a thesis presented in partial fulfilment of the requirements of the degree of Doctor of Philosophy at Massey University
Loading...
Date
2008
Open Access Location
DOI
Authors
Editors
Journal Title
Journal ISSN
Volume Title
Publisher
Massey University
Abstract
Noncontact, the failure of a postal survey sample member to receive a survey
request, is a potential source of nonresponse bias that has largely been ignored.
This is due to the difficulty of separating the components of nonresponse in postal
surveys when nothing is heard from potential respondents. Yet, the need to
understand postal nonresponse is increasing as more studies move to mixed mode
designs incorporating a postal element, and technological, resource and societal
changes increase the attractiveness of self-administered surveys. Thus, this
research sought to estimate the level of noncontact in postal surveys, to identify the
direction and magnitude of bias due to it, and to investigate targeted in-field
mechanisms for reducing this bias. A series of empirical studies involving New
Zealand postal surveys fielded between 2001 and 2006 were undertaken to meet
these aims.
Noncontact was found to relate to survey-independent demographic variables (e.g.,
age, household composition). Furthermore, its incidence was estimated to be as
much as 400% higher than indicated by ‘gone, no address’ (GNA) returns, although
an envelope message tested as part of the research was able to increase levels of
GNA reporting significantly. Thus, noncontact was established as a nontrivial source
of nonresponse in the surveys examined.
As far as bias is concerned, noncontacts had a different profile compared to refusers
and ineligibiles, and were estimated to account for up to 40% of net nonresponse
error for some of the variables in the surveys examined. Accordingly, there appears
to be a clear opportunity for methods targeted at reducing noncontact bias to improve
final survey estimates for a range of items.
A number of potential methods for reducing noncontact bias were explored, but only
one had both a compelling theoretical foundation and potential for wide applicability;
the noncontact propensity sampling (NPS) scheme. In a resampling simulation study
a prototype of the scheme, which increases the selection probabilities for sample
units with a higher predicted propensity for noncontact, consistently improved the
demographic profile of valid postal survey returns compared to a simple random
sample (SRS). Furthermore, the scheme reduced nonresponse bias by an average
of 28% as measured against a range of frame variables (e.g., age, gender) and 17%
as measured against survey variables for which census parameters were known
(e.g., religiosity, household size, qualifications, income and marital status).
Although the prototype NPS procedure increased the standard deviation of simulated
point estimates for a given sample size (1,500 in this research), the effect was small;
an average of 4% for frame variables and 2% for survey variables. Furthermore, the
scheme had almost no impact on reported cooperation rates and is likely to be cost
effective compared to other potential targeted in-field mechanisms, particularly in
situations where researchers regularly survey a specific population.
Pairing the scheme with three common post-survey adjustment methods (frame or
census age/sex cell weighing, and response wave extrapolation) did not lead to
consistently better estimates than an unweighted SRS. But this was largely due to
the shortcomings of these methods because in many cases combining them with
either sampling scheme (SRS or NPS) actually degraded estimates. This reinforces
the idea that researchers should expend effort minimising bias during the field period
rather than relying on post-survey weighting to deal with the issue.
Finally, since the NPS scheme aims to reduce error due to noncontact but is not
expected to affect error due to other components (e.g., refusal, ineligibility), it
presents an opportunity for researchers to begin decomposing the various facets of
postal survey nonresponse bias, an important precursor to the development of other
targeted bias reduction interventions. Thus, as a methodological tool, the NPS
scheme may serve a dual role as both a bias reduction and decomposition
mechanism.
In addition to their implications for postal survey research, the methods developed
and insights into noncontact established in this research are likely to have
applications in other domains. In particular, they will inform activities such as
research into online survey nonresponse, organisational database management cost
reduction and list procurement.
Description
Keywords
Postal surveys, Non-response bias, Non-contact propensity, Marketing, Sample selection, Mail surveys