Random digit dialing and internet panel data collection methods in two Canadian provinces: Comparing costs,data missingness,straightlining,and sociodemographic characteristics of sample,and responses from a survey on nutrition policy support and causes of chronic disease

Abstract

Background:

There is little consensus on what public health survey administration methods are better (data generation and cost-wise) for collecting data on knowledge, attitudes, and beliefs (KAB). We compare random digit dialing (RDD) and internet panel sampling methods for gathering KAB data on chronic disease etiology and nutrition policy.

Design and methods:

We collected survey data from residents of Alberta and Manitoba in 2017, using population-based samples generated through: RDD and an internet panel. We calculated response rate and cost for each mode. To compare missing data and straightlining, we used linear regression. We used Chi-squared tests to compare sociodemographic characteristics between the two modes and to the 2016 Canadian Census data. KAB responses were also compared between modes using Chi-squared tests.

Results:

The internet panel was less expensive and had more missing data than the RDD. Straightlining was comparable across modes. Both modes tended to oversample specific population groups (e.g. older adults); while undersampling others (e.g. Indigenous people) compared to the Canadian Census. RDD had more females and older respondents than the internet panel respondents. Internet panel respondents were less supportive of nutrition policy options, and agreed more with individual responsibility and blame for obesity, compared to RDD respondents.

Conclusions:

Both modes present advantages and disadvantages. Differences in sociodemographics and KAB responses between modes indicate sampling methods for public health surveys may be considered in survey design and administration. Researchers should discuss their findings vis-a-vis the specific limitations of each method they employed and adopt strategies to mitigate them.

Keywords

survey random digit dial (RDD)internet panel public health chronic disease prevention healthy public policy

Significance for public health

Random digit dialing (RDD) and internet panels are common health survey administration methods used to collect data to inform public health interventions, practice, and policy. Our study demonstrated that internet panels cost less, had more missing data, and may be more representative of population gender, age, and household income; the RDD survey was more geographically representative, with a higher proportion of older respondents. RDD respondents also expressed more support for healthy public policy options and exhibited less individual blame for obesity. Differences in responses reflecting attitudes and beliefs related to health and policy may depend on administration mode and sample representativeness, thus identifying important equity and inclusion implications for public health research.

Introduction

In public health research, there is little consensus on best practices for collecting survey data.^1–4 In particular knowledge, attitudes, and beliefs (KAB) about healthy public policy and etiology and responsibility for chronic disease prevention^5–7 may be difficult to assess accurately because responses have been shown to differ by respondent characteristics like race, gender, and income – characteristics that are also related to the mode of survey delivery.^1,2 There is an enduring – if not growing – need for data from the general and specific population-based samples on pressing public health issues,^3,8 yet a corresponding difficulty in obtaining timely, cost-effective, and representative data from these diverse populations.^8,9 For instance, failing to include hard-to-reach populations (e.g. young adults) could compromise external validity and generalization and exclude diverse groups who likely would benefit the most from the public health research.^8,10 At worst, non-representative samples may provide a poor evidence base to fully inform the design and improvement of healthy public policies and health care services.¹⁰ Several methods for collecting survey data are commonly utilized (e.g. online panel surveys),^4,11 but the relative impact of data collection mode on the nature of data collected is unclear.^12,13

Internet surveys are gaining popularity given increased internet access, lower administration costs, ease of distribution, and overall convenience.^8,14 They may take less time to complete¹ and provide better response quality and reliability, particularly if respondents are experienced, motivated, and have minimal distractions.^2,15 Other benefits include consistent question delivery (e.g. minimizing unintended bias, inappropriate cues, or wording changes introduced by an interviewer), reduced cognitive recall burden on respondents, higher response rates,¹⁶ and capacity to present complex audio-visual material and skip patterns. Internet surveys, however, still pose challenges to researchers including systematic exclusion of those without internet access,^17,18 unwanted distractions and premature survey termination, more superficial cognitive processing, answering too quickly or falsely, repeating the same response (i.e. straightlining), or completing multiple surveys.^19,20 These concerns may be amplified in internet surveys administered to established respondent panels through survey firms, where financial incentives offered for survey completion may change the nature and representativeness of the respondent profile (e.g. a biased sample composition by attracting socioeconomically disadvantaged groups as the monetary payment could benefit their income, or increased fraudulent responses).²¹ However, research has also shown web-based surveys with incentives facilitate participant recruitment, reduce time for completion of data collection and improve responses.^22,23

Assuming broad ownership of landlines or cellular devices, surveys administered via telephone interviews may be more accessible because respondents do not need internet access, higher literacy, good vision, or computer skills.² Interviewers can increase engagement and response accuracy by establishing rapport and conveying enthusiasm to respondents. To access potential respondents by phone, one may call all phone numbers included in a list (e.g. a directory), or randomly sample numbers using the structure of telephone numbers in a given area, such as in random digit dialing (RDD). RDD garners the benefits of generating probability samples which reduce sampling bias and increase representativeness of the sample. However, telephone surveys may elicit more extreme answers compared to internet surveys,²⁴ and are prone to the recency effect when respondents are more likely to select the last answer they heard.²⁰ Responding to an interviewer may also trigger socially desirable responses. For example, Ansolabehere and Schaffner¹ found that telephone respondents agreed less that, “the government is almost always wasteful and inefficient” and were more likely to support affirmative action when compared to an internet sample.

Yet, that study found no differences in liberal or conservative responses to policy issues when comparing telephone, internet panel, and mail survey data.¹ Another study found that internet respondents answered more neutrally or negatively to attitudinal evaluations compared to telephone survey respondents, likely due to differences in the demographic makeup between internet and telephone samples.²⁵ While internet surveys had a better response rate, they yielded more item non-responses compared to telephone surveys.²⁵ Compared to internet data, telephone data were shown to have more random measurement error, more survey satisfaction, more bias from social desirability, and lower concurrent validity.^2,26 Although these limitations in both telephone and internet surveys have been previously discussed in the literature,^2,14,25,26 still, little is known about survey administration options for collecting information on public health policy issues to support future practice and intervention research.

In this methodological study, we use data from a survey our team conducted in 2017 to describe and compare data collection methods as they pertain to KAB about chronic disease etiology and nutrition policy. We aimed to:

Describe RDD and internet panel survey modes in terms of response rate, cost per participant, and response quality;

Describe sociodemographic characteristics of respondents to each survey administration mode and compare these to the provincial population;

Compare responses to the KAB questions about chronic disease etiology and support for nutrition policy options between the two modes.

Methods

Procedure

This is a methodological study examining two data collection methods for surveys. We collected and utilized data from the 2017 Chronic Disease Prevention Survey.²⁷ This survey was part of a larger program of research that examines KAB about chronic disease etiology and prevention as well as healthy public policy options among population-based samples of the Canadian general public and policy influencers.²⁷ For this current study, we used survey questions pertaining to cancer etiology, nutrition policy, and obesity etiology completed by the general public (community-dwelling residents aged 18 years and older) in Alberta and Manitoba. A research and practice-based team of experts from across Canada created and validated the survey questionnaire. The team worked with a survey administration firm to validate and field-test the survey to identify wording or saliency problems. The questionnaire is shown in the Supplemental File, File S.1. To investigate the relationships between the sampling method and cost, data quality, representativeness, data were collected using RDD (no incentive) and an incentivized internet panel option to increase response rates. For each province, the RDD aimed to recruit 1500 respondents in each province, whereas for the internet panel the target number of participants was 400 in each province. The eligibility criteria for participation were: age 18 years or older and residence in Manitoba or Alberta. During the recruitment process for RDD, the survey administration firm preferentially recruited males. This purposive within-household sample selection aimed to counter potential sex/gender bias by ensuring an equal distribution of sex/gender (as self-reported) among respondents, by balancing the higher proportion of single-parent female-headed households in North American urban samples. For both the RDD and internet panel modes, recruitment targeted equal proportions of respondents from urban and rural areas, and (self-identified) males and females. The participant recruitment for the RDD took 4 weeks. For the internet panel, recruitment lasted for 1 week. In the RDD, all participants provided a verbal informed consent prior to completing the survey which was recorded by the interviewer in the dataset. For the panel participants, after reading the study information, consent was implied by completing the questionnaire. The University of Alberta Research Ethics Board approved the study protocol (Pro00081566) for both RDD with verbal consent and internet panel with implied consent. The survey took, on average, 7 min to complete.

Participants and data collection

Random digit dialing (RDD)

Respondents (N = 3000; Alberta n = 1500, Manitoba n = 1500) were interviewed by professional survey administrators over the phone using RDD. In both provinces, 60% of the sample were interviewed using a landline, and 40% using mobile phones.

Internet panel

Respondents (N = 1267; Alberta n = 861, Manitoba n = 406) were sampled randomly from a pre-existing respondent pool affiliated with the survey administration firm contracted by the study team to collect data. While the this pool pre-existed this study, it was initially recruited by the survey firm using a hybrid method, where 75% were recruited via RDD and the other 25% were through other methods, including social media, word of mouth, and offline recruitment. Selected individuals within this pool were emailed an invitation to participate in the current survey. Those who responded received an incentive in the form of gift cards, pre-paid cards, or monetary transfers. Each section of the survey was presented per page, with the order of the questions randomized on each page. Although the online and telephone questionnaires were identical, the language was slightly tailored for the self-administered online survey. For instance, the respondents of the internet panel could read the “don’t know” option on their screen among the other response options. In turn, in the RDD, the survey firm did not read that response category to the respondents; however, that option could still be selected if respondents expressed they did not know how to answer that survey question.

Measures

Sociodemographic characteristics

To better describe respondents’ sociodemographic characteristics relative to each mode and determine the representativeness of the sample, we used all the survey questions as described below.

Age

Recorded in years as a number between 18 and 120.

Children

Dichotomous responses were collected by asking, “Are there children aged 17 or younger in your home?”

Education

Respondents were asked, “What is the highest level of education you have completed?” and selected a response option from: High school; College, or post-secondary trades/technical school; University; Undergraduate degree; or University graduate or professional degree.

Employment

We asked, “Which of the following best describes your employment status?” Respondents then selected from: Self-employed; Employed full time – more than 30 hours per week; Employed part time – less than 30 h per week; Unemployed; Student; Homemaker; Retired; or Other (specify).

Political alignment

We asked, “Regardless of any party affiliation or recent voting, how would you describe your political views?” Respondents selected from: Very liberal; Somewhat liberal; Neutral; Somewhat conservative; or Very conservative.

Gender

We asked, “I’ll now ask you about your gender identity, which is a person’s sense of being a woman, a man, both, neither, or anywhere in between. Which of the following gender identities do you identify with: Woman, Man, or Something else (please specify)?”

Health status

We assessed self-reported health by asking, “In general would you say your health is excellent, very good, good, fair or poor?”

Annual household income (Canadian dollar)

Respondents selected yearly income before taxes from the following categories: Under $20,000; $20,000 to just under $40,000; $40,000 to just under $70,000; $70,000 to just under $100,000; $100,000 to just under $125,000; or $125,000 or more.

Number of people in a household

We asked, “How many people, including yourself, live in your household?” recorded as a number between 1 and 10. Responses were categorized into: Single; Two; Three; or Four or More Person Households for analyses.

Immigrant, racial minority, and indigenous status

We assessed immigration history by asking, “Were you born in Canada, or did you move to Canada from somewhere else?” Respondents who moved to Canada from somewhere else were asked, “Do you consider yourself to be a member of a visible minority?” Respondents who were born in Canada were asked, “Do you identify yourself as Indigenous, Aboriginal, First Nations or Metis?” Binary responses were recorded.

Provincial area of residence

We asked respondents in Alberta [Manitoba], “Do you live in the greater Edmonton [Winnipeg] area, the greater Calgary area, or another city, town or place in Alberta [Manitoba]?” Responses were categorized into Urban (Edmonton, Calgary, or Winnipeg) and Rural for analyses.

Knowledge, attitude, and beliefs

Cancer etiology

Respondents were asked, “Please indicate how much you think each of the following items are linked to a person’s chances of getting cancer” on a four-point Likert-style scale from 1 “Definitely Linked,” 2 “Might be Linked,” 3 “Probably is Not Linked,” and 4 “Definitely is Not Linked.” These included individual factors (e.g. regular exercise), and environmental exposures (e.g. residing near industrial facilities).

Nutrition policy options

Respondents indicated their support for 15 evidence-informed nutrition policies targeting the food environment (e.g. “Tax sugary drinks and energy drinks on top of sales tax”; “Subsidize the purchase of healthy foods and beverages”) on a 4-point scale (1 = “Strongly Support”, 2 = “Support”, 3 = “Oppose”, and 4 = “Strongly Oppose”).

Obesity etiology

Respondents indicated their agreement on a 4-point scale: 1 = “Strongly Agree”, 2 = “Agree”, 3 = “Disagree”, and 4 = “Strongly Disagree”. These four items began with the stem, “When someone has a problem with obesity. . .” followed by: “it is their own fault,” “it is caused by circumstances beyond their control,” “it is their responsibility to deal with it,” and “it is society’s responsibility to deal with it.”

Data analyses

RDD response rate was first calculated as the number of complete interviews divided by the sum of complete and partial interviews, non-responding units (e.g. the number of refusals, break-offs, non-contacts, and other (e.g. lost records)), and the number of unknown eligibility (e.g. unknown if it is a housing unit, unknown if sampled housing unit contains an eligible respondent).²⁸ This was then multiplied by the estimated proportion of cases of unknown eligibility that could be eligible to participate.²⁸ Completion rate for the internet panel was calculated as the number of completed and partial questionnaires divided by the number of completed and partial questionnaires plus number of refusals, non-contacts, and other.²⁸ Survey cost was calculated as the amount billed by the survey firm divided by the final number of observations per modality. Straightlining was assessed using four methods: the mean root of pairs method (the mean of the root of the absolute differences between all pairs of items in a battery), the maximum identical rating method (the proportion of the maximum number of identical ratings in a battery), the standard deviation of battery method (the standard deviation of ratings for each respondent), and the scale point variation method (the probability of differentiation: Pd = 1 – ΣPi2).²⁹ Straightlining metrics were also used to complete a linear regression that controlled for sociodemographic differences between the RDD and internet panel samples that differed significantly between administration mode (i.e. urbanicity, migration status, racial minority status, political alignment, household size, household income, self-reported health-status, gender, employment, education, age, and province). The models were applied separately to the three blocks of questions: cancer etiology; support for nutrition policy; and etiology of obesity.

Missingness was assessed at the respondent level as the proportion of questions that were either skipped or recorded as “don’t know.” A linear regression was used to examine associations between administration mode and missingness. The model controlled for sociodemographic variables that differed significantly between administration modes (i.e. urbanicity, migration status, racial minority status, political alignment, household size, household income, self-reported health-status, gender, employment, education, age, and province).

Chi-squared test was used to compare respondents’ sociodemographic characteristics between RDD versus internet panel within each province; the results are presented alongside the 2016 Canadian Census population in the respondents’ respective province.³⁰ To assess the representativeness of the political spectrum captured by our survey in both provinces, the political alignment responses were compared with publicly-available data regarding voting outcomes from the 2015 federal general election (downloaded from Elections Canada),³¹ and data regarding the 2019 Alberta and Manitoba provincial general elections (downloaded from Elections Alberta and Elections Manitoba, respectively).^32,33 Chi-squared tests were applied to compare survey responses to the federal election data and to the provincial election data, separately. Parties from all elections were classified into three broad categories: Left (Very Liberal and Liberal), Center (Neutral), or Right (Very Conservative and Conservative), based on their policy stances.

Differences in responses by administration mode to questions regarding cancer etiology (linked vs not linked), nutrition policy options (support vs oppose) and obesity etiology (agree vs disagree) were compared using chi-squared tests within each province. Post-hoc comparisons of sociodemographic characteristics and response type (i.e. support vs oppose) for questions that demonstrated >5% differences in response type between administration modes (stratified by province) were also examined using Chi-squared tests (data not shown). All statistical analyses were completed in R, using the packages: tidyr, foreign, and dplyr. Including survey questionnaire development, participant recruitment, and data analysis, this study duration was 24 months.

Results

In Alberta, the RDD response rate was 10.5% (1500 completed interviews; 6831 refusals, break-offs, non-contacts, and others; 12,546 cases of unknown if a housing unit) and the completion rate for the internet panel was 20.0% (861 completed and partial questionnaires with 416 respondents completing the questionnaire before reaching the quota of 400 or end of survey date; 4311 valid emails). In Manitoba, the RDD response rate was 8.0% (1500 completed interviews; 9180 refusals, break-offs, non-contacts, and others; 33,445 cases of unknown if a housing unit), compared with the completion rate of 19.2% for the internet panel (406 completed and partial questionnaires with 400 respondents with valid responses before completing the quota of 400 or end of survey date; 2144 valid emails sent). Overall, 79% of the landline sample was over 55 years of age, and 79% of the mobile phone sample was between 18 and 34 years of age. The cost of the RDD survey was CA$52,500, or CA$17.50 per respondent, while the internet panel cost a total of CA$6500, or CA$5.13 per respondent.

Table 1 provides the straightlining measures for each province. For KAB about cancer etiology, the maximum identical rating method (B: −0.0247; 95% CI: −0.039, −0.010) and the scale point variation method (B: 0.020; 95% CI: 0.007, 0.033) showed that the RDD had more straightlining compared to the internet panel. For nutrition policy support, there were differences between modes with the mean root of pairs (B: 0.040; 95% CI: 0.015, 0.065) and standard deviation of battery methods (B: −0.089; 95% CI: −0.121, −0.057), whereby RDD had less straightlining compared to the internet panel. Lastly, for the etiology of obesity, there were differences using the maximum identical rating method (B: −0.074; 95% CI: −0.092, −0.055) and the scale point variation method (B: 0.055; 95% CI: 0.039, 0.072), whereby RDD had more straightlining compared to the internet panel. Overall, the differences between the two survey administration modes on straightlining were small.

Table 1.

Straightlining^a measures from population-based samples derived via random digit dialing and internet panel in the 2017 Chronic Disease Prevention Survey for Alberta and Manitoba.

Straightlining metric	Alberta		Manitoba
	RDD (N = 1500)	Internet panel (N = 416)	RDD (N = 1500)	Internet panel (N = 400)
	Mean (Q1, Q3)	Mean (Q1, Q3)	Mean (Q1, Q3)	Mean (Q1, Q3)
Mean root of pairs^a	0.33 (Q₁: 0.17; Q₃: 0.45)	0.32 (Q₁: 0.19; Q₃: 0.43)	0.33 (Q₁: 0.17; Q₃: 0.45)	0.32 (Q₁: 0.13; Q₃: 0.44)
Maximum identical rating^b	0.54 (Q₁: 0.38; Q₃: 0.62)	0.51 (Q₁: 0.38; Q₃: 0.62)	0.54 (Q₁: 0.46; Q₃: 0.62)	0.52 (Q₁: 0.38; Q₃: 0.62)
Standard deviation of battery^c	0.84 (Q₁: 0.65; Q₃: 1.04)	0.82 (Q₁: 0.65; Q₃: 0.97)	0.84 (Q₁: 0.65; Q₃: 1.04)	0.85 (Q₁: 0.65; Q₃: 1.11)
Scale point variation^c	0.58 (Q₁: 0.50; Q₃: 0.67)	0.58 (Q₁: 0.50; Q₃: 0.68)	0.58 (Q₁: 0.50; Q₃: 0.68)	0.59 (Q₁: 0.52; Q₃: 0.70)

RDD: random digit dialing.

Q₁: First quartile; 25% of observations are below this value and 75% are above it.

Q₃: Third quartile; 75% of observations are below this value and 25% are above it.

Straightlining refers to repeated response selection.

The final value of this index ranges from 0 (no straightlining) to 1 (completely straightlined).

The final value of this index is bounded from 0 to infinity, with smaller values indicating more straightlining, and larger values indicating less straightlining.

The internet panel had more missing data than the RDD survey. The average amount of missingness in the RDD survey across all questions was 2.5%, with a mode of 1.7%, and a range of 0.0 to 14.9% (Alberta: Mean, 2.6%; Mode, 1.7%; Range, 0.0%–14.9% | Manitoba: Mean, 2.4%; Mode, 1.9%; Range, 0.3%–14.5%). The average amount of missingness in the internet panel was 5.6%, with a mode of 3.6%, and a range of 1.0 to 16.1% (Alberta: Mean, 5.4%; Mode, 3.6%; Range, 1.0%–16.1% | Manitoba: Mean, 5.7%; Mode, 5.8%; Range, 1.5%–13.3%). Based on the linear regression model, the internet panel had 2.6% more missingness (B: 0.026; 95% CI: 0.020, 0.032; Supplemental Table S.1). For context, skipping one question would amount to 3% missing. Other variables associated with missingness were: being aged 65 and older (compared to 18–24; Age between 65 and 74, B: 0.015; 95% CI: 0.001, 0.030; Age 75 or older, B: 0.018; 95% CI: 0.002, 0.035) and having a household income between $100,000 and $125,000 (compared to making less than $20,000; B: −0.017; 95% CI: −0.033, −0.001; Supplemental Table S.1).

A comparison of sociodemographic characteristics between the RDD and internet panel surveys is presented in the Supplemental File, Table S.2. More RDD respondents were women, self-employed, older (75+ years), immigrants, had an income of $125,000 or more, and lived with children, compared with internet panel respondents in both provinces. More Alberta RDD respondents were from larger households; Indigenous, Aboriginal, First Nations or Métis (hereafter: Indigenous) respondents; and conservative or neutrally politically aligned (vs the Alberta internet panel). The internet panel in both provinces had fewer respondents who were self-employed, lived in large households (four or more people), immigrants, and those living in rural areas.

As shown in Table 2, compared to the Canadian Census, in general, the RDD survey and internet panel in both provinces had a larger number of respondents who were 55 years and older, those not living with children, those with a university undergraduate education or higher, and with household incomes between $100,000 and $125,000; and a smaller number of those aged 25–34, with high school education or less, with household incomes under $20,000, and people identifying as Indigenous. The RDD had a larger number of women and single person dwellings, compared to the Canadian Census. When compared to either federal or provincial voting data, liberal and central voters were overrepresented and conservative voters were underrepresented across both provinces. Internet panel respondents had smaller households and were more often from larger city centers compared to Census data and RDD respondents.

Table 2.

Sociodemographic characteristics between the 2016 Canadian Census and federal and provincial electoral polls, with the random digit dialing and internet panel respondents to the 2017 Chronic Disease Prevention Survey, stratified by Alberta and Manitoba.

Sociodemographic characteristics	Alberta census/ electoral data %	Alberta		Manitoba census/electoral data %	Manitoba
Sociodemographic characteristics	Alberta census/ electoral data %	RDD (N = 1500) %	Internet panel (N = 416) %	Manitoba census/electoral data %	RDD (N = 1500) %	Internet panel (N = 400) %
Age (years)
25–34	15.8	12.9^a	11.8^a	13.4	11.2^a	18.2^a
35–44	14.4	15.6	13.0	12.5	14.3	13.8
45–54	13.6	15.8^a	12.7	13.2	15.3^a	17.0^a
55–64	12.3	24.2^a	22.8^a	12.9	21.4^a	21.8^a
65–74	7.1	16.4^a	27.6^a	8.7	17.8^a	17.8^a
75+	5.2	11.4^a	6.7	6.9	13.5^a	4.8
Children in household
No	59.1	65.2^a	76.2^a	60.6	68.0^a	73.3^a
Yes	40.9	34.8^a	23.8^a	39.4	32.0^a	26.7^a
Education
High school incomplete	16.9	2.2^a	1.5^a	22.0	2.0^a	2.5^a
High school	27.9	21.2^a	17.8^a	29.6	26.8^a	33.1
College/technical/ trade	28.8	31.7^a	28.0	25.4	27.1	23.6
University undergraduate	19.5	33.0^a	36.5^a	17.2	32.0^a	32.1^a
University professional or graduate	6.9	11.9^a	16.3^a	5.7	12.1^a	8.8^a
Employment status
Self-employed	12.1	12.7	7.1^a	10.4	7.5^a	3.8^a
Unemployed	6.5	5.0^a	5.8	4.5	3.8	5.9
Gender
Men	50.1	40.3^a	47.5	49.4	42.0^a	49.4
Women	49.9	59.7^a	52.5	50.6	58.0^a	50.6
Gross household income
Under $20,000	9.7	4.5^a	5.8^a	9.6	5.2^a	12.1
$20,000 to < $40,000	16.7	11.6^a	15.3	17.0	17.0	21.0
$40,000 to < $70,000	23.3	20.1^a	23.1	24.7	25.8	31.7^a
$70,000 to < $100,000	17.8	19.6	21.7	19.0	19.1	18.2
$100,000 to < $125,000	10.4	14.0^a	15.3^a	10.7	13.0^a	8.4
$125,000 +	22.0	30.3^a	18.7	18.8	19.8	8.6^a
People in household
1	24.0	19.4^a	25.2	27.7	22.0^a	24.9
2	34.3	38.8^a	47.7^a	33.7	36.1	39.9^a
3	16.0	14.8	12.7	14.5	17.6^a	16.5
4+	25.7	27.0	14.4^a	24.0	24.3	18.6^a
Minority identity
No	76.5	86.3^a	88.2^a	82.5	83.1	84.5
Yes	23.5	13.7^a	11.8^a	17.5	16.9	15.5
Indigenous identity
No	93.5	94.9^a	97.4^a	82.0	90.3^a	88.2^a
Yes	6.5	5.1^a	2.6^a	18.0	9.7^a	11.8^a
Immigration status
Immigrated	23.0	18.8^a	14.5^a	19.7	19.0	11.8^a
Canadian-born	77.0	81.2^a	85.5^a	80.3	81.0	88.2^a
Area of residence
Edmonton/ Winnipeg	32.5	33.3	44.2^a	60.9	63.3	80.0^a
Calgary	34.2	33.3	36.5	-	-	-
Other	33.3	33.3	19.2^a	39.1	36.7	20.0^a
Political views – provincial
Liberal	33.3	31.6	38.3^a	47.3	40.3	37.8
Centrist	10.1	24.2^a	19.3^a	14.6	31.9^a	31.9^a
Conservative	56.7	44.2^a	42.4^a	47.3	27.8^a	30.3^a
Political Views – Federal
Liberal	14.5	31.6^a	38.3^a	26.0	40.3^a	37.8^a
Centrist	13.9	24.2^a	19.3^a	26.5	31.9^a	31.9^a
Conservative	71.6	44.2^a	42.4^a	47.5	27.8^a	30.3^a

RDD: random digit dialing.

p < 0.05; p-value refers to statistically significant differences between the provincial Census/electoral data (reference categories) and the given Chronic Disease Prevention Survey modality.

The proportion of respondents linking lifestyle behaviors to cancer, who supported a given nutrition policy, or agreed with statements on responsibility for obesity per survey modality and stratified by province are given in Table 3. In Manitoba, there was more agreement on behavioral and environmental links to cancer in the RDD sample compared to the internet panel sample, whereas in Alberta there were few differences. Internet panel respondents were generally less supportive of nutrition policy options, and had higher agreement for individual responsibility and blame for obesity compared to RDD respondents. RDD respondents, in contrast, agreed more that obesity is society’s responsibility to deal with.

Table 3.

Knowledge, attitudes, and beliefs about cancer etiology, nutrition policy options, and obesity etiology between the random digit dialing and internet panel respondents in the 2017 Chronic Disease Prevention Survey, stratified by Alberta and Manitoba.

Item	Alberta		Manitoba
Item	RDD (N = 1500)%	Internet panel (N = 416)%	RDD (N = 1500)%	Internet panel (N = 400)%
Links to likelihood of getting cancer
Participating in regular exercise	56.3	57.9	55.2^a	47.8
Maintaining a healthy body weight	62.6	60.1	62.1^a	50.5
Eating a healthy balanced diet	61.6	62.5	61.5^a	53.5
Eating sufficient servings of fruits and vegetables	59.9	61.1	59.7^a	51.7
Smoking cigarettes	97.6	96.6	97.7^a	95.3
Using other tobacco products	95.8	95.0	95.5	93.0
Exposure to tobacco smoke	94.5^b	91.1	95.8^a	92.0
Smoking marijuana	61.0	56.7	59.7	57.8^c
Drinking excessive alcohol	77.6^a	72.6	79.1	77.5
Where a person works	79.8^b	74.0	82.3^a	70.8
Where a person goes to school	24.7	20.4	29.2^a	21.3
Residing near industrial facilities	86.0	84.9	88.1^a	82.8
The neighborhood, town or city where a person lives	60.5^a	50.7	62.1^a	53.8
Support for nutrition policy options
Fund government media campaigns that encourage healthy food and beverage choices	85.9^a	78.8	89.3^a	77.8
Provide programs to educate the general public about how to make healthy food choices	94.1	92.8	96.2^a	91.0
Ensure breastfeeding is permitted and adequate facilities exist in all public buildings	91.7^a	85.3	92.4^a	84.8
Mandate priority space for healthy foods and beverages in all recreation facilities	86.2^a	76.2	88.7^a	80.0
Mandate policies for healthy foods and beverages at schools – including preschools	90.4^a	82.9	91.7^a	84.3
Prohibit advertising/promoting unhealthy foods and beverages to children under age 16	75.9^a	68.3	78.1^a	66.8
Subsidize the purchase of healthy foods and beverages	77.7^a	66.3	83.3^a	70.8
Tax the purchase of unhealthy foods and beverages	58.1^a	48.6	64.9^a	53.0
Tax sugary drinks and energy drinks on top of sales tax	58.1^b	51.9	62.3^a	53.0
Restrict sugary drink sales in all public buildings	57.7^a	51.4	60.7^a	54.8
Restrict sugary drink sales in all public recreation facilities	62.5^b	57.0	65.5^a	59.0
Restrict unhealthy food sales in all public recreation facilities	62.1^b	56.3	65.3^a	59.0
Permit zoning to restrict the supply of junk food near schools	69.4^a	58.4	70.5^a	61.3
Permit zoning to limit the number of fast food restaurants per square kilometer	57.2^a	45.7	57.4^a	48.3
Restrict or ban new fast food restaurant drive-through facilities	36.5^a	26.2	39.9^a	31.8
Agreement with Etiology of Obesity
When someone has a problem with obesity it is their own fault	32.3^a	38.7	32.7^a	43.8
When someone has a problem with obesity it is caused by circumstances outside of their control	35.9^a	20.4	37.8^a	26.0
When someone has a problem with obesity it’s their responsibility to deal with it	80.5^a	85.1	78.1^c	81.3
When someone has a problem with obesity it’s society’s responsibility to deal with it	26.6^a	19.7	32.7^a	22.3

RDD: random digit dialing.

p < 0.01; ^b p < 0.05; p-value refers to statistically significantly differences between RDD and internet panel. Comparisons use the “somewhat support or strongly support” ratings in RDD and internet panel.

p-values were significantly different on “somewhat oppose or strongly oppose” ratings between RDD and internet panel.

In the post-hoc comparisons of sociodemographic characteristics and response type, those who had higher self-rated health, identified as a racial minority, or an immigrant were supportive of nutrition policies. For nutrition policies related to children, breastfeeding, or food subsidies, those without children were less supportive and women were more supportive. For policies related to limiting fast food availability, immigrants were particularly supportive. Results for age, employment, income, and region were mixed for nutrition policy support. Overall, those who were more educated, racial minorities, and immigrants were more likely to agree that society has a responsibility to address obesity, and that obesity is caused by external factors. Right leaning voters tended to disagree with both of these sentiments. Women and left-leaning voters tended to disagree that obesity is the fault of the individual.

Discussion

Our study shows that the internet panel had higher response rates and cost far less (approximately one-third of the RDD survey) per respondent. The higher costs of the telephone surveys are due to the labor costs of live interviewers (including number of dialing attempts for each sampled phone number and the interviewer time to have a questionnaire completed) and telephone expenses. The RDD survey, however, had fewer missing responses than the internet panel, by about one question on average. More missingness among the internet panel may be explained by the accessibility of the “don’t know” option online, or could indicate superficial cognitive processing.^20,25 Both survey modalities had similar levels of straightlining, contrasting suggestions by Hays et al.¹⁹ that internet panel respondents may repeat responses to expedite survey completion.

The higher female representation in our RDD sample differed from Roster et al. ’s²⁵ findings that telephone surveys had fewer female respondents than internet respondents. Fewer internet panel respondents [Alberta] identified as Indigenous or immigrants (compared to RDD data) supporting research suggesting telephone data may recruit more racial minorities and older respondents.²⁵ Importantly, the variability between provinces on items such as education, where Alberta’s RDD sample was less highly educated (university undergraduate or higher) and Manitoba’s RDD sample was slightly more highly educated compared to their respective internet panel samples, demonstrates the importance of examining and accounting for regional differences in study design and evaluation.

We investigated mode effects on responses to items about cancer etiology, responsibility for obesity, and support for nutrition policy options. Our findings showed that internet panel respondents, compared to RDD respondents, agreed more with statements around individual responsibility and blame for obesity, and were less supportive of nutrition policy options. These findings, in addition to lower self-rated health and less reported income in the internet panel sample, may suggest more “honest” responding, and/or less socially desirable self-presentation, compared to the RDD respondents. Health and wealth are often associated with virtue or social value,³⁴ causing inflated self-reporting of these characteristics when being interviewed. Other research also found RDD interviews produced more socially desirable responses.^1,2,25

These findings are concordant with qualitative research examining ideas to reduce obesity in which liberal (left leaning) focus groups discussed junk food and gasoline taxes while conservative (right leaning) focus groups rejected these ideas and instead discussed more individual strategies like labeling food.³⁵ Other research found that older people, women, those who were non-White, liberal, had less education, lower income, and poorer health were more likely to recognize the social determinants of health and see social policy as health policy.³⁶ Some of these findings are consistent with our results, but critically, we have also demonstrated that these associations may depend on survey administration mode and sample representativeness.

Sample representativeness is a common issue in public health research. Our population-based samples were more educated and had fewer immigrants and racial minorities compared to Census data. These findings have important equity and inclusion implications for public health research. Data that do not meaningfully include diverse perspectives are limited in their ability to inform public health practice and healthy public policy, and at worst, may inadvertently perpetuate systemic biases and institutional discrimination. Weighting and adjustment for population estimates is a common strategy to address non-representativeness in surveys.^1,37 Representativeness may also be improved by employing a stratified sampling that aims to oversample specific population groups, including offline households in internet probability-based panel research,³⁸ using probability-based samples compared to non-probability samples, mixed-mode surveys compared to single-mode surveys,²⁰ and surveys that are not conducted on the internet (vs internet surveys).³⁹

Study strengths included using multiple methods to make descriptive comparisons of various modalities and geographies. Furthermore, we used the response rate and completion rate formulas published by the American Association for Public Opinion Research (AAPOR) – an association of public opinion and surveys.²⁸ These formulas reflect a standardized approach that allows for international comparability and replicability.²⁸ This study also presented several limitations. Firstly, the utilization of a survey that has been previously administered in 2017 limited our ability to implement sampling, recruitment, and data collection methods that were comparable. Therefore, the differences in results seen in this study may have been attributed to the technologies applied to collect data in each sampling method: Computer Assisted Telephone Interviewing (CATI) for RDD and Computer Assisted Web Interviewing (CAWI) for the internet panel. Future studies could compare surveys of the same sampling method (e.g. RDD or internet panel) with different data collection methods (CATI and CAWI). Secondly, despite repeated follow-up telephone calls or emails (respective to survey mode) to ensure a large sample size, our response rates were low in both survey administration modes and provinces, which reflects an ongoing trend in research using survey methodology.⁴⁰ While we recognize high response rates do not necessarily translate into representativeness (albeit it may address non-response bias)⁴¹ our low response rates are a limitation of this study and may have compromised generalizability. However, we weighted and adjusted for population estimates to address the low responses.

Thirdly, the branching used to identify racial minorities (i.e. only asked if respondents indicated they immigrated to Canada) may not capture Canadians who are racial minorities, but not immigrants. Election data should be considered in the context of low election turn-out rates, and when comparing our measure of intended vote to actual votes counted. Also, given the sensitive nature of specific survey questions on KAB about chronic diseases and nutrition policy support, the 4-point Likert scale was chosen to ensure survey respondents would choose the answer they lean more toward. Additionally, there are no differences in response rates, data quality, and internal consistency between even and odd number of response options.⁴² However, we recognize 4-point Likert scales, compared to the odd number of response options with a neutral zone, may increase missingness and reduce reliability, variance in scores, and responder acceptability.⁴² It is worth mentioning that the values are not corrected for multiple corrections across the analyses. Given the large number of comparisons used in this analysis, many of the correction methods would impose such a small alpha as to render nothing of significance. We wished to present the comparisons and show what factors may be associated with some of the differences identified, and have reported different values alongside each other so that readers may parse whether the difference is meaningfully large, not just of statistical significance.

Conclusions

Our research highlights practical components of RDD and internet panel sampling methods for a public health survey related to cost, data quality, and more nuanced differences including sample characteristics compared to Census data and provincial and federal election data. Differences in knowledge, attitudes, and beliefs about chronic disease prevention between the RDD and internet panel respondents suggests social desirability bias may be present when the respondents have to interact with interviewers, particularly for sensitive topics such as responsibility for obesity. We recommend public health researchers to reflect on such potential bias when choosing sampling methods. While the relative ease and much lower cost of internet panels are appealing and appear to be less prone to social desirability bias, we suggest that researchers carefully discuss data collection strategies and statistical methods in light of potential issues concerning sample composition (e.g. lower participation of older, immigrant respondents), geographic representativeness, and data missingness relative to their specific research aims. While our findings suggest some considerations for public health researchers to make when planning health surveys, we acknowledge that further investigation is needed to confirm the findings of our exploratory study. In particular, we recommend comparisons of these two survey administration modes (i.e. RDD and internet panel) using larger sample sizes and samples with specific population groups (e.g. only youth participants or rural-resident participants).

Supplemental Material

sj-docx-1-phj-10.1177_22799036251388566 – Supplemental material for Random digit dialing and internet panel data collection methods in two Canadian provinces: Comparing costs, data missingness, straightlining, and sociodemographic characteristics of sample, and responses from a survey on nutrition policy support and causes of chronic disease

Supplemental material, sj-docx-1-phj-10.1177_22799036251388566 for Random digit dialing and internet panel data collection methods in two Canadian provinces: Comparing costs, data missingness, straightlining, and sociodemographic characteristics of sample, and responses from a survey on nutrition policy support and causes of chronic disease by Kimberley D. Curtin, Mathew Thomson, Jo Lin Chew, Ana Paula Belon, Katerina Maximova and Candace I. J. Nykiforuk in Journal of Public Health Research

Footnotes

Acknowledgements

We are grateful to all participants who took part in the survey.

ORCID iDs

Kimberley Curtin

Ana Paula Belon

Candace I. J. Nykiforuk

Ethical considerations

This study received ethical approval from the University of Alberta Research Ethics Board (Pro00081566).

Consent to Participate

Informed consent was obtained from all participants (verbal for the RDD and implied for the internet panel) prior to responding to the survey questionnaire.

Author contributions

All authors were involved in conceptualization of the manuscript and all authors reviewed, edited, and approved the final manuscript. KC: Methodology, Investigation, Writing – original draft. MT, JC: Data Curation, Methodology, Formal analysis, Writing – original draft. APB: Investigation, Project Administration, Writing – review and editing. KM: Conceptualization, Methodology, Writing – review and editing. CN: Conceptualization, Methodology, Investigation, Supervision, Writing – review and editing, Funding acquisition.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: CIJN received support as an Applied Public Health Research Chair with funding from the Canadian Institutes of Health Research in partnership with the Public Health Agency of Canada and Alberta Innovates - Health Solutions) for work supporting the data collection from the survey panel (2014–2019; CPP 137909). The funding bodies were not involved in the design of the study or collection, analysis, and interpretation of data, or in writing the manuscript.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability statement

The datasets generated and/or analyzed during the current study are not publicly available due to requirements of our research ethics approval, but are available from the corresponding author on reasonable request.

Supplemental material

Supplemental material for this article is available online.

References

Ansolabehere

Schaffner

BF.

Does survey mode still matter? Findings from a 2010 multi-mode comparison. Polit Anal 2014; 22: 285–303.

Chang

Krosnick

JA.

National Surveys via RDD telephone interviewing versus the Internet: comparing sample representativeness and response. Public Opin Q 2009; 73: 641–678.

Hensen

Mackworth-Young

CRS

Simwinga

, et al. Remote data collection for public health research in a COVID-19 era: ethical implications, challenges and opportunities. Health Policy Plan 2021; 36(3): 360–368.

Lin

Chen

SI.

Efficiency and quality of data collection among public mental health surveys conducted during the COVID-19 pandemic: systematic review. J Med Internet Res 2021; 23(2): e25118.

Ul Haq

Mariyam

, et al. A comparative study of nutritional status, knowledge attitude and practices (KAP) and dietary intake between international and Chinese students in Nanjing, China. Int J Environ Res Public Health 2018; 15(9): 1910.

Curtin

Thomson

Nykiforuk

CIJ

. Who or what is to blame? Examining sociodemographic relationships to beliefs about causes, control, and responsibility for cancer and chronic disease prevention in Alberta, Canada. BMC Public Health 2021; 21(1): 1047.

Girois

Kumanyika

Morabia

, et al. A comparison of knowledge and attitudes about diet and health among 35- to 75-year-old adults in the United States and Geneva, Switzerland. Am J Public Health 2001; 91(3): 418–424.

Hlatshwako

Shah

Kosana

, et al. Online health survey research during COVID-19. Lancet Digit Health 2021; 3(2): e76–e77.

Zhao

Fils-Aime

Response rates of online surveys in published research: a meta-analysis. Comput Hum Behav Rep 2022; 7: 100206.

10.

Bonevski

Randell

Paul

, et al. Reaching the hard-to-reach: a systematic review of strategies for improving health and medical research with socially disadvantaged groups. BMC Med Res Methodol 2014; 14(1): 42.

11.

Reardon

Perlman

Dumas

, et al. Creating a probability survey panel for population health research: the experience of the NYC Health Panel. J Public Health Manag Pract 2025; 31: 828–835.

12.

Yang

Tan

EC.

Web-based and telephone surveys to assess public perception toward the national health insurance in Taiwan: a comparison of cost and results. Interact J Med Res 2015; 4(2): e9.

13.

Gideon

(ed.). Handbook of survey methodology for the social sciences. Springer, 2012.

14.

Blumenberg

Barros

AJD

. Response rate differences between web and alternative data collection methods for public health research: a systematic review of the literature. Int J Public Health 2018; 63: 765–773.

15.

Jagodzinski

KÜHnel

Schmidt

Is there a “Socratic Effect” in nonexperimental panel studies?: consistency of an attitude toward guestworkers. Sociol Methods Res 1987; 15: 259–302.

16.

Pollard

. Use of consumer panel survey data for public health communication planning: an evaluation of survey results. In: Joint Statistical Meetings - Section on Health Policy Statistics, https://pdfs.semanticscholar.org/c8a2/ce0c31bdd01800b0b82ca851d788b6e7c233.pdf (2002, accessed 28 May 2019).

17.

Rohde

Shapiro

. Falling through the net: toward digital inclusion. Report for the U.S Department of Commerce, October 2000, https://www.ntia.gov/report/2000/falling-through-net-toward-digital-inclusion

18.

Craig

Hays

Pickard

, et al. Comparison of US panel vendors for online surveys. J Med Internet Res 2013; 15(11): e260.

19.

Hays

Liu

Kapteyn

Use of Internet panels to conduct surveys. Behav Res Methods 2015; 47: 685–690.

20.

de Leeuw

. To mix or not to mix data collection modes in surveys. J Off Stat 2005; 21: 233–255.

21.

Singer

The use and effects of incentives in surveys. Ann Am Acad Pol Soc Sci 2012; 645(1): 112–141.

22.

Sobolewski

Rothschild

Freeman

The impact of incentives on data collection for online surveys: social media recruitment study. JMIR Form Res 2024; 8: e50240.

23.

Abdelazeem

Abbas

Amin

, et al. The effectiveness of incentives for research participation: a systematic review and meta-analysis of randomized controlled trials. PLoS One 2022; 17(4): e0267534.

24.

de Leeuw

. Data quality in mail, telephone and face to face surveys. T. T. Publicaties, 1992.

25.

Roster

Rogers

Albaum

, et al. A comparison of response characteristics from web and telephone surveys. Int J Mark Res 2004; 46: 359–373.

26.

Yeager

Krosnick

Chang

, et al. Comparing the accuracy of RDD telephone surveys and internet surveys conducted with probability and non-probability samples. Public Opin Q 2011; 75: 709–747.

27.

PLACE Research Lab. Chronic disease prevention survey, https://placeresearchlab.com/place-projects/chronic-disease-prevention-survey/#1489418973859-7611fa5a-1540 (2025, accessed 7 July 2025).

28.

American Association for Public Opinion Research (AAPOR). 2023 Standard definitions: final dispositions of case codes and outcome rates for surveys. 10th ed. Report, 2023, https://aapor.org/standards-and-ethics/standard-definitions/

29.

Kim

Dykema

Stevenson

, et al. Straightlining: overview of measurement, comparison of indicators, and effects in Mail–Web mixed-mode surveys. Soc Sci Comput Rev 2019; 37: 214–233.

30.

Statistics Canada. Download, Census Profile, 2016 Census, https://www12.statcan.gc.Ca/census-recensement/2016/dp-pd/prof/details/download-telecharger/comp/page_dl-tc.cfm?Lang=E (2017, accessed 26 October 2020).

31.

Elections Canada. 42nd General election: official voting results (raw data), https://www.elections.ca/content.aspx?section=res&dir=rep/off/42gedata&document=bypro⟨=e (2018, accessed 26 October 2020).

32.

Elections Alberta. 2019 General election, https://www.elections.ab.ca/news-reports/reports/general-elections/2019-general-election/ (, accessed 11 September 2020).

33.

Elections Manitoba. Results of the 42nd General Election, September 10th, 2019, https://www.electionsmanitoba.ca/en/Results/Elections/2019 (2019, accessed 11 September 2020).

34.

Saguy

Gruys

Morality and health: news media constructions of overweight and eating disorders. Soc Probl 2010; 57: 231–250.

35.

Niederdeppe

Robert

Kindig

DA.

Qualitative research about attributions, narratives, and support for obesity policy, 2008. Prev Chronic Dis 2011; 8: A39.

36.

Robert

Booske

BC.

US opinions on health determinants and social policy as health policy. Am J Public Health 2011; 101: 1655–1663.

37.

Gorman

Leyland

McCartney

, et al. Adjustment for survey non-representativeness using record-linkage: refined estimates of alcohol consumption by deprivation in Scotland. Addiction 2017; 112: 1270–1280.

38.

Blom

Herzing

JME

Cornesse

, et al. Does the recruitment of offline households increase the sample representativeness of probability-based online panels? Evidence from the German Internet panel. Soc Sci Comput Rev 2017; 35: 498–520.

39.

Cornesse

Bosnjak

Is there an association between survey characteristics and representativeness? A meta-analysis. Surv Res Methods 2018; 12: 1–13.

40.

Manzo

Burke

. Increasing response rate in web-based/Internet surveys. In: Gideon

(ed.) Handbook of survey methodology for the social sciences. Springer, 2012, pp. 327–343.

41.

Schouten

Cobben

Bethlehem

Indicators for the representativeness of survey response. Surv Methodol 2009; 35(1): 101–113.

42.

Østerås

Gulbrandsen

Garratt

, et al. A randomised comparison of a four- and a five-point scale version of the Norwegian Function Assessment Scale. Health Qual Life Outcomes 2008; 6: 1–9.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB