Exploring the external validity of survey data with triangulation: A case study from the Norwegian Offender Mental Health and Addiction (NorMA) Study

Abstract

Objectives

This paper demonstrates how to investigate the external validity of a study sample by triangulating survey and registry data, using data from the Norwegian Offender Mental Health and Addiction (NorMA) Study as a case.

Methods

We use survey data from the NorMA study (n = 1495), including the NorMA cohort (n = 733), and data from the Norwegian Prison Registry on all people imprisoned on 1 September 2013 (n = 3386). Triangulation was performed by (1) comparing the NorMA cohort to those lost to follow-up (n = 762), using survey data from the NorMA study. Secondly, we compared the NorMA cohort to the one-day population, using data from the Norwegian Prison Registry. We also stratified the one-day sample by possession of a Norwegian personal identification number (PIN).

Results

We found differences in birthplace, imprisonment and drug use between the NorMA cohort, lost to follow-up and the one-day population. Twenty-three percent of the one-day population did not have a Norwegian PIN. The NorMA cohort was more similar to those with a Norwegian PIN in the one-day population. Our triangulation demonstrates that 56–62% of the Norwegian prison population had an indication of drug use before imprisonment.

Conclusions

The NorMA cohort was overall representative of the one-day prison population holding a Norwegian PIN and less representative of prisoners without a Norwegian PIN. Using this method provides tangible inputs on the strengths and limitations of a study sample and can be a feasible method to investigate the external validity of survey data.

Keywords

NorMA Cohort External validity Triangulation Registry data Prison population Selection bias Drug use Cohort studies

Background

External validity rarely receives quantitative analysis, perhaps due to lack of comparison data.^1,2 A frequent threat to external validity is selection bias; bias caused by non-representative sampling, which consequently may hamper the study results. A representative study sample is especially important in descriptive studies that wish to describe a specific population at a point in time.³

In this article, we will explore how external validity and selection bias in survey data can be assessed with data triangulation with alternative data sources. To illustrate the triangulation method, we use the Norwegian Offender Mental Health and Addiction (NorMA) Study and the Norwegian Prison Registry.

Globally, as well as in Norway, epidemiological research on drug use in the prison population has been based on a wide variety of research designs, sampling strategies and measurements of drug use.^4–6 However, cross-sectional surveys, which are more sensitive to selection bias, are most common,^4,7 while national registries and databases provide untapped potential for research on drug use and the prison population.^8,9

The Nordic countries have developed advanced national registries and databases based on personal identification numbers (PINs) assigned to all residents. The unique PIN enables the linkage of information on an individual level between several registries for the purpose of official statistics and research. The advantage of these registries is their almost complete inclusion of the population. Registry data can therefore be used to triangulate research data^10,11 in order to investigate whether a study sample is representative of the target population. One important condition for validating with triangulation is that the study sample and the more complete data are drawn from the same target population. Triangulation can therefore be conducted using data from national surveys, insurance data or administrative records as alternatives to national registries, as long as it is possible to identify the study sample by a PIN.

The NorMA study was conducted in 2013–2014 in Norway and aimed to describe substance use and mental health among people in Norwegian prisons. A total of 1495 inmates responded to the survey, and 733 (49%) provided a Norwegian PIN and informed consent for its use in future research, thus constituting the NorMA cohort. Anonymous participation was also encouraged, and 762 (51%) inmates responded without providing a PIN, thereby constituting the lost to follow-up cohort.

In this study, we triangulated data from the NorMA study and the Norwegian Prison Registry to assess the representativeness and external validity of the NorMA cohort. We identified two selection processes potentially introducing bias into the NorMA cohort. Firstly, selection into the NorMA cohort among respondents in the NorMA study; those who provided their PIN’s at baseline and consent to link it in future research. Secondly, selection into the NorMA study from the target population; those who participated in the NorMA study among all prisoners in Norway at the time of data collection.

In this article, we will investigate these two selections with triangulation and discuss the representativeness of the NorMA cohort to the general Norwegian prison population. The specific objectives of the study are the following:

1. To compare the individuals in the NorMA cohort to those lost to follow-up using baseline survey data from the NorMA study, and

2. To compare the NorMA cohort to the general prison population using registry data from the Norwegian Prison Registry.

Methods

Setting

Norway has one of the lowest prison population rates in the world, with an average of 3850 individuals imprisoned in 2016, equal to a prison population rate of 73 per 100, 000 of national population, compared to a global average of 145 per 100, 000 and 655 per 100, 000 in the United States.^12,13 As 85% of prisoners serve sentences of less than 1 year and 60% serve less than 3 months, the yearly turn-over is high.¹⁴ Women constitute a minority in Norwegian prisons, with an annual proportion of approximately six percent. Almost two-thirds of prisons are high security prisons. All Norwegian prisons are publicly funded, and all inmates have access to universal health care coverage, like all other individuals in Norway.¹⁵ The criminal justice system is characterized by having a rehabilitative focus on incarceration, and some prisons have separate drug treatment units to achieve the goals of rehabilitation.¹⁶

Patients and/or the public were not involved in the design, conduct, reporting or dissemination plans of this research.

Study design and data sources

Triangulation requires two or more sources of data. We triangulated the NorMA cohort with data from the NorMA study (Objective 1) and registry data from the Norwegian Prison Registry (Objective 2), as illustrated in Figure 1.

Figure 1.

Triangulation of survey and registry data, made possible by the personal identification numbers (PIN) in the NorMA cohort.

The NorMA study

The NorMA study includes a cross-sectional study sample with baseline survey data. The survey contains a 116-item questionnaire, including questions on mental health and substance use, as well as demographics and other background information. The questionnaire took 30–60 min to complete and was available in five languages, including Norwegian, English, Russian, French and German. Data was collected by study investigators who visited 57 prison units (of 63 eligible) during the time of data collection from June 2013 until July 2014. Six prisons were not visited due to limited staff capacity and geographical inconvenience. Those prison units had a total capacity of 179 inmates and did not differ from the prisons included in the data collection. The questionnaires were administered by the study investigators and distributed to the prisoners on the day of the visit. The respondents filled out the questionnaire themselves. They could complete the questionnaire in a common room with others or alone in their cell, depending on their prison situation and preferences. All inmates in the prison on the day of data collection were considered eligible to participate in the study. The questionnaire included a consent form that explained the purpose of the study and sought permission for linking the survey responses with registry data. In addition to a signature field, an 11-digit field was provided to capture the PINs. For a more thorough description of the methodology and study design, please see Bukten et al.¹⁶

The Norwegian Prison Registry

The Norwegian Prison Registry was established in 1992 to serve a range of administrative and statistical purposes. The registry includes data from all Norwegian prisons on sentences, imprisonment status, security level, participation in programmes and other variables on the activities related to imprisonment. The registry is administered by the Correctional Service of Norway, who oversees execution of all remand and prison sentences in Norway.¹⁶

From the Norwegian Prison Registry, we retrieved data on all prisoners imprisoned on 1 day within the data collection period of the NorMA study. To choose a comparison date, we drew 10 separate days within the data collection period, excluding public holidays. The 10 days had an average prison population of 3995 individuals with deviation of less than ±2.5% from this average. For the methodological purpose of this study, we randomly selected the 1^st of September 2013 among the 10 days, on which 3908 individuals were imprisoned.

Among the 3908, 522 (13%) were also included in the NorMA cohort and therefore excluded from the one-day population. We retrieved data on all imprisonments from 1 January 1992 until 31 December 2019 for all persons in both the NorMA cohort and the one-day population. All the data from the Norwegian Prison Registry, including information referred to as ‘lifetime events’, therefore refers to imprisonments within the period from 1992 till 2019.

Measures

The two data sources included different variables and we therefore have different sets of measures in objective 1 and objective 2 of our analysis. The measures used in each objective and how they are defined are described in Table 1. The analysis of each data source and their measurements was performed separately, and we did not compare or test somewhat similar measures from both datasets against each other. We chose not to perform this comparison and verification of the survey data, to protect the integrity of the study participants, in accordance with our ethical approval from REK.

Table 1.

Measures.

Objective 1 NorMA cohort compared to lost to follow-up	Objective 2 NorMA cohort compared to the general prison population
Demography	Demography
Demography	Sex: % male
Sex: % male	Imprisonment
Country of birth: % born in Norway	Age: Average age at first imprisonment
History of family problems: % who had a childhood with substance use or mental health problems in the family	Re-offenders: % of prisoners with more than one lifetime imprisonment
Education: % with a formal education beyond primary school	Days per imprisonment: Average length of each imprisonments
Employment: % with any kind of employment: Including part-time employment or school: Before imprisonment	Number of imprisonments: Average imprisonments recorded in the Prison Registry
Imprisonment	Total days in prison: Average sum of days spend in prison (lifetime)
Re-offenders: % with any previous imprisonments	‘Use and possession’: % individuals who had a conviction for using and/or possessing drugs (own use) defined Medical Act section 24 and 31, Norwegian Law
Number of imprisonments: Average lifetime imprisonments: Including the sentence they served when answering the questionnaire
Type of crime: % with this crime as the main type the individual was sentenced or charged for
Drug usage and mental health
Respondents had a list of 16 drugs to choose from, including non-prescribed use of medications
Lifetime drug use: % who had used any kind of drugs (except alcohol) ever
Daily drug use: % with daily use of drugs 6 months before imprisonment
Harmful drug use: % with harmful drug use, measured with the Drug Use Disorder Identification Test (DUDIT).¹⁷ Harmful drug use was indicated by a score of ≥6 for men and ≥2 for women on the DUDIT. Individuals who indicated no lifetime drug use were coded into the ‘no harmful use’ group
Harmful alcohol use: % with harmful alcohol use measured with Alcohol Use Disorder Identification Test (AUDIT).^18,19 Harmful alcohol use was defined as a score of ≥8 for men and ≥6 for women on the AUDIT. Individuals who indicated no lifetime alcohol use were coded into the ‘no harmful use’ group
Current mental health and distress: % with ‘severe mental stress’ (score ≥185),²⁰ assessed with the Hopkins symptom check list (HSCL-10)

Analysis

We analyzed representativeness of the NorMA cohort according to our two objectives:

Objective 1:

The NorMA cohort compared to lost to follow-up.

For objective 1, we compared the individuals in the NorMA cohort to the lost to follow-up group using baseline survey data from the NorMA study, and focusing on measures related to demographics, imprisonment and drug use.

Objective 2:

NorMA cohort compared to the general prison population.

For objective 2, we compared the NorMA cohort to the one-day population (n = 3386), retrieved from the Norwegian Prison Registry. Objective 2 only included measures retrieved from the Norwegian Prison Registry on age, sex and imprisonment. We analyzed the one-day population in total and stratified by possession of a Norwegian PIN. An overview of the objectives, data and study samples is summarized in Table 2.

Table 2.

Overview of objectives, data sources and samples.

	1	2
Objective	To compare the individuals in the NorMA cohort to the lost to follow-up group	To compare the NorMA cohort to the general prison population
Data source	The NorMA baseline data	Norwegian Prison Registry
Data type	Survey (self-reported)	Registry data
Sample size	NorMA cohort (n = 733) vs. lost to follow-up (n = 762)	NorMA cohort (n = 733) vs. general prison population (n = 3386)

Medians and the interquartile ranges (Q1–Q3) were reported for skewed data, such as age at first imprisonment, number of imprisonments and days imprisoned. Test of difference between the NorMA cohort, the lost to follow-up group and the one-day population was done using chi-squared tests, T-test of means on normally distributed numerical variables and the Mann–Whitney-test for ordinal and categorical variables. We did all statistical analyses in SPSS Statistics version 26.

Missing data

The percentage of missing data ranged from 0 to 28.5% and was more frequent among the self-reported variables used to answer objective 1. Detailed information is described in Table 3. The registry data we used to answer objective 2 had no or very little missing data on all variables (not reported in table).

Table 3.

Baseline data on demographics, imprisonment and health by NorMA cohort (n = 733) and persons lost to follow-up (n = 762), with test of difference between groups.

	NorMA cohort n (%)	Missing n (%)	Lost to follow-up n (%)	Missing n (%)	p-value
Demographics
Male	682 (93.0)	0 (0.0)	710 (94.0)	7 (1.0)	0.434^a
Age, mean (SD)	35.5 (11.6)	0 (0.0)	33.6 (10.8)	131 (17.2)	0.001^b
Born in Norway	602 (84.2)	18 (2.5)	402 (55.0)	31 (4.1)	<0.001^a
Primary school or less	300 (41.4)	9 (1.2)	255 (34.1)	14 (1.8)	0.004^a
Not working or studying before incarceration	419 (58.4)	16 (2.2)	299 (41.1)	34 (4.5)	<0.001^a
History of family problems	265 (37.5)	26 (3.5)	191 (26.3)	35 (4.6)	<0.001^a
Imprisonment
Re-offenders	480 (68.3)	30 (4.1)	352 (50.4)	63 (8.3)	<0.001^a
Number of imprisonments, median (Q1–Q3)	3 (1–7)	0 (0.0)	1 (1–4)	1 (0.1)	<0.001^c
Type of crime in current imprisonment^d	716	17 (2.3)	691	71 (9.3)
Acquisitive crime	264 (36.9)		184 (26.6)		<0.001^a
Drug-related^e	386 (53.9)		309 (44.7)		<0.001^a
Violence, sexual violence and murder	328 (45.8)		277 (40.1)		0.001^a
Other	110 (15.4)		102 (14.8)		0.369^a
Mental health
Severe mental distress^f	297 (47.1)	103 (14.1)	200 (36.7)	217 (28.5)	<0.001^a
Drug usage
Lifetime drug use	541 (76.0)	21 (2.9)	444 (62.2)	48 (6.3)	<0.001^a
Daily use	411 (56.1)	0 (0.0)	289 (37.9)	0 (0.0)	<0.001^a
Harmful drug use	463 (65.4)	25 (3.4)	334 (47.9)	64 (8.4)	<0.001^a
Harmful alcohol use	413 (57.0)	9 (1.2)	390 (54.2)	43 (5.6)	0.284^a

^aX²-test.

^bT-test.

^cMann–Whitney test.

^dAs the respondents could have been convicted for more than one type of crime in one imprisonment, the percentage across the different types of crimes can add up to more than 100%.

^eDrug-related crimes incl. driving while intoxicated.

^fHSCL-10 score ≥1.85.

The standardized measures AUDIT, DUDIT and HSCL-10 were imputed according to standard methods. Missing values were replaced with a mean of the completed answers for individuals with at least five items answered for DUDIT and AUDIT and seven items answered for HSCL-10.^21,22

We performed multiple imputation by chained equations (MICE), with 10 imputations, on all missing values for the remaining variables. We tested the imputations with Chi²-test and T-test, but did not influence the statistical significance in difference between the groups. Due to the descriptive scope of this paper, we therefore chose to report only on data analyzed using complete case analysis.

We used the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) cohort checklist when writing our report.²³

Results

Objective 1: NorMA cohort compared to persons lost to follow-up

The NorMA cohort had more previous imprisonments compared to the persons lost to follow-up. Sixty-eight percent of the NorMA cohort had a previous sentence, and their median number of imprisonments was three (Table 3).

A larger proportion of the NorMA cohort reported involvement in all types of crimes compared to the lost to follow-up group (except for ‘other’ with 15% in both groups). The most common type of conviction in both groups was drug-related crime. Fifty-four percent of the NorMA cohort had this type of crime as one of their convictions or charges, while this was the case for 45% of the lost to follow-up group.

The NorMA cohort reported more drug use and mental distress, compared to the lost to follow-up. This included higher lifetime drug use (76 vs 62%), daily drug use (56 vs 38%) and as measured with a DUDIT score above cutoff (65 vs 48%) compared to the lost to follow-up group (See Table 3).

Objective 2: NorMA cohort compared to the general prison population

The NorMA cohort was younger when they entered prison the first time, compared to the one-day population (24 vs 27 years) (Table 4). Both groups had more than 90% male participants. Except for days per imprisonment, the NorMA cohort had more imprisonments (median four vs two) and had served more days in prison (902 vs 792 days) compared to the one-day population (Table 4). Three in four individuals in the NorMA cohort had more than one lifetime imprisonment, compared to approximately half of the individuals in the one-day population. Sixty-two percent of the NorMA cohort had been convicted of ‘use and possession’ at least once, compared to 54% in the one-day population.

Table 4.

Background characteristics and life-time imprisonment by NorMA cohort (n = 733) and one-day population (n = 3386), stratified by PIN and no PIN, respectively, with test of difference between NorMA cohort and the one-day populations.

	NorMA cohort (n = 733)	One-day, all (n = 3386)	p	One-day, PIN (n = 2479)	p	One-day, no PIN (n = 907)	p
Demography
Age at first imprisonment, median (Q1–Q3)^a	24 (20–33)	27 (21–36)	<0.001	25 (21–35)	0.017	31 (25–37)	<0.001
Male (%)^b	93	95	0.097	95	0.128	95	0.072
Imprisonment
Re-offender (%)^b	76	56	<0.001	73	0.259	9	<0.001
Number of imprisonments, median (Q1–Q3)^a	4 (2–7)	2 (1–5)	<0.001	3 (1–7)	0.003	1 (1–1)	<0.001
Days per imprisonment, median (Q1–Q3)^a	93 (29–240)	108 (35–304)	<0.001	99 (31–269)	0.116	394 (105–897)	<0.001
Total days in prison^a	902 (376–1928)	792 (279–1806)	0.036	962 (343–2129)	0.449	519 (175–987)	<0.001
«Use and possession» (%)^b	62	54	<0.001	57	0.015	46	<0.001

^aIndependent-Samples Mann–Whitney U Test.

^bX²-test.

In the one-day population (n = 3386), 907 individuals (27%) did not have a PIN, equivalent to 23% of the total prison population on this day (including the NorMA cohort, who all had PIN). When stratifying the one-day population by PIN/no PIN, those without Norwegian PIN where older and had fewer imprisonments but with more days per imprisonment compared to the NorMA cohort. Their total days of life-time imprisonments were less than the NorMA cohort’s (519 compared to 902 days) and fewer had a conviction for use and possession (46%).

Discussion

Studying the representativeness of a study sample is key in epidemiological health research. This includes considering potential sources of selection bias, related to the condition of the study setting and research design. Assessing this can require new perspectives, including alternative sources of data.

Rothman has argued that representativeness of a study sample does not in itself improve external validity; rather, it is the knowledge of specific conditions in the study setting and an understanding of causal mechanism that makes for proper generalization.²⁴ We argue that triangulation is a feasible method to assess these specific conditions and mechanisms of potential selection, and improves our knowledge of our sample’s external validity.

The NorMA cohort was different from the lost to follow-up group and the one-day population on most variables. However, when we stratified the one-day population on PIN, the one-day population with PIN and the NorMA cohort were more similar. This indicates that some of the differences between the NorMA cohort and the lost to follow-up group could be explained by the presence of participants without PIN in the lost to follow-up group.

The NorMA study had 51% lost to follow-up: Those who participated in the survey without providing a PIN and/or informed consent. We do not have information on why they did not provide a PIN. We do not know, for example, whether it was conscious choice, or if they simply could not remember their PIN, or if they did not have a Norwegian PIN. However, since we have the information provided in the NorMA survey, we are able to describe the lost to follow-up group with a broad range of characteristics. When comparing the NorMA cohort to the lost to follow-up group, we saw a difference in drug use and birth country. More than half of the NorMA cohort had used drugs in the previous 6 months before imprisonment, compared to 38% in the lost to follow-up. The NorMA cohort was mainly born in Norway, while a larger proportion of the lost to follow-up were born elsewhere (84% within NorMA cohort vs. 55% among lost to follow-up).

All Norwegian residents are assigned a PIN. As having a PIN was a criterion for participation in the NorMA cohort, immigrants without a visitor’s permit and tourists were not eligible for follow-up in Norwegian registries. The NorMA cohort is vulnerable to selection bias based on country of birth and citizenship, and it was therefore important to stratify the one-day population by PIN possession. One in four of the one-day population from the general prison population did not have a PIN, and this group was different from those with a PIN. The one-day population with a PIN was similar to the NorMA cohort when looking at sex, re-offending and total days in prison. The tendency was the same for all variables. The stratification thus indicated that the NorMA cohort is largely representative of the general prison population in possession of a Norwegian PIN, but not to those without.

Drug use

Fifty-six percent of the NorMA cohort reported daily drug use before imprisonment. From the registry data, we have information on convictions related to ‘use and possession’, which we use as a proxy for drug use prior to imprisonment. When looking at this proxy, 62% of the NorMA cohort and 57% of the general prison population with a Norwegian PIN had this conviction. Our triangulation can therefore demonstrate that 56–62% of the individuals in the Norwegian prison population had an indication of drug use prior to their imprisonment. However, the analysis also showed that this estimate will probably only be valid for the prison population holding a Norwegian PIN.

Limitations and strengths of the study

Both survey and registry data introduce some general strengths and limitations to our study. The survey data we used from the NorMA study included a very broad set of questions and provided rich baseline data. However, some variables had higher degrees of missing, which was more prevalent in the lost to follow-up group, with most missing values on the variables related to mental health status. On the other hand, the data we retrieved from the Norwegian Prison Registry had complete follow-up, but a limited and pre-defined set of variables chosen for administrational purposes. By combining the two, we gained valuable insight into the representativeness and external validity of the NorMA cohort, which is important for future research based on this cohort. Another strength of the Norwegian Prison Registry is the available data on those who did not have a Norwegian PIN and therefore are lost to follow-up in other registries. We do not know how many in the lost to follow-up group did not have a PIN, nor why those who had a PIN chose not to provide theirs. Though we cannot follow these individuals in future linkages with registries, the information provided to us in the Prison Registry data provides us with valuable insight into their characteristics, in comparison to the prison population with a PIN.

Implications

Studying selection bias is important in all sample-based research, because the selection itself, and the underlying mechanisms of selection, can affect the research results.

By triangulating different data source, as we have demonstrated in this article with survey data and registry data, we can investigate these selections and its effect on our results.

If PINs (or another linkable identification code) and informed consent are obtained, registry data can be used to investigate the representativeness of a study sample. For the triangulation method to be applicable, one should be able to identify the study sample within the alternative data source, which requires a linkable identification code. If possible, pay particular attention to groups that do not hold linkable identification codes, such as foreign citizens, and consider how they would appear in the study sample and how they should be treated in the analysis.

In our case, we linked our cohort to Norwegian registry data by using the Norwegian PIN. However, other potential data sources could be national surveys, insurance data or institutional records from institutions such as hospitals, education systems or prisons. Even with a limited number of variables available, for example, on demography or imprisonment, registry data can be valuable when investigating representativeness and evaluating the external validity of data and results. Including this information can therefore be relevant, even if linkage with registry data is not the main purpose of the study. When designing surveys with the intention to collect PIN, remember to emphasize to potential participants that anonymous participation is also encouraged and valuable to the survey.

Conclusion

Though all studies are at risk of selection bias, the characteristics and implications of these biases are often difficult to assess and measure. As we have shown in this study, triangulation of data sources can shed light on the specific conditions between the study sample and target population. With this method, we can improve our understanding of the mechanisms affecting selection bias and clarify the external validity of the study results.

Footnotes

Acknowledgements

The authors wish to thank the inmates who participated in the study, and the prison management and the staff members at the local prison units for assisting in collecting data. We also thank our colleague Ann Oldervoll for proofreading the manuscript.

Author Contributions

NTL performed the data analysis and drafted the paper. MRS supervised NTL, particularly on the statistical analysis. AB is the principal investigator of the NorMA study and in charge of data acquisition and ethical approvals of the study. All authors designed the study and interpreted the data. All authors read, revised and approved the final manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the South-Eastern Norway Regional Health Authority [grant number HSØ 2019091].

Ethics approval and consent to participate

The NorMA study was approved by the Norwegian Committee of Research Ethics (REK 2012/297). It was also approved by the Ministry of Justice and Public Security, and by the Directorate of the Norwegian Correctional Services, the national prison authorities in Norway. All methods were carried out in accordance with relevant guidelines and regulations. The PINs were provided by the participants following written informed consent. Furthermore, the use of Prison Registry data for comparison, including the identification of the NorMA cohort using PINs, was approved by REK in a supplementary approval in 2020. A precondition of the ethical approval from REK was that we refrain from testing internal validity of the NorMA study, as this could question the credibility and integrity of the participants in the study.

Consent for publication

Not applicable

Availability of data and materials

The datasets generated and/or analyzed during the current study are highly sensitive data and not publicly available due to the limitations of the ethical approval from the Norwegian Committee of Research Ethics. The data can be made available from the corresponding author on reasonable request.

ORCID iD

Nicoline Toresen Lokdam

References

Lesko

Buchanan

Westreich

, et al. Generalizing study results - a potential outcomes perspective. Epidemiology 2017; 28: 553–561.

Rothman

Greenland

Lash

. Modern epidemiology. 3rd ed. Philadelphia, USA: Lippincott Williams & Wilkins, 2008.

Rothman

Gallacher

Hatch

. Why representativeness should be avoided. Int J Epidemiol 2013; 42: 1012–1014.

Carpentier

Royuela

Montanari

, et al. The global epidemiology of drug use in prison. In: Kinner

Rich

(eds) Drug use in prisoners. Oxford University Press, 2018, pp.17–43.

Revold

. Innsattes levekår 2014. Før, under og etter soning (“Living conditions of inmates 2014 - Before, during and after imprisonment”). 2015.

Cramer

. Forekomst av psykiske lidelser hos domfelte i norske fengsler (“Prevalence of psychiatric illness among inmates in Norwegian prisons”). Oslo: Oslo University Hospital, 2014.

Carpentier

Royuela

Noor

, et al. Ten years of monitoring illicit drug use in prison populations in Europe: issues and challenges. Howard J Crim Justice 2012; 51: 37–66. DOI: 10.1111/j.1468-2311.2011.00677.x.

Lyngstad

Skardhamar

. Nordic register data and their untapped potential for criminological knowledge. Crime Justice 2011; 40: 613–645.

Lund

Bukten

. Harm to others from substance use and abuse: the underused potential in nationwide registers. Subst Abuse 2015; 9: 33–38. 2015/11/13. DOI: 10.4137/sart.s23545.

10.

Rutherford

McFarland

Spindler

, et al. Public health triangulation: approach and application to synthesizing data to understand national and local HIV epidemics. BMC Public Health 2010; 10: 447.

11.

Lawlor

Tilling

Smith

. Triangulation in aetiological epidemiology. Int J Epidemiol 2016; 45: 1866–1886.

12.

Institute of Crime & Justice Policy Research . World Prison Brief Data, https://www.prisonstudies.org/country/norway (2020, accessed 30 June 2020).

13.

Directorate of the Norwegian Correctional Service . Kriminalomsorgens årsstatistikk - 2016 (“Yearly statistics from the Norwegian correctional services -2016”). 2017.

14.

Directorate of Norwegian Correctional Service . Kriminalomsorgens årsstatistikk - 2018 (“Yearly statistics from the Norwegian correctional services -2018”). Norway, 2019.

15.

Directorate of Norwegian Correctional Service . About the Norwegian correctional service, https://www.kriminalomsorgen.no/index.php?cat=265199 (2020, accessed 30 June 2020).

16.

Bukten

Lund

Lobmaier

, et al. The Norwegian Offender Mental Health and Addiction Study – design and implementation of a national survey and prospective cohort study. Subst Abuse 2015; 9: 59–66. DOI: 10.4137/SART.S23546.

17.

Berman

Bergman

Palmstierna

, et al. Evaluation of the Drug Use Disorders Identification Test (DUDIT) in criminal justice and detoxification settings and in a Swedish population sample. Eur Addict Res 2005; 11: 22–31. 2004/12/21. DOI: 10.1159/000081413.

18.

Babor

Higgins-Biddle

Saunders

, et al. AUDIT: the alcohol use disorders identification test. In: Guidelines for use in primary health care. 2nd ed. Geneva: World Health Organization, 2001, p. 41.

19.

Saunders

Aasland

Babor

, et al. Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO Collaborative Project on early detection of persons with harmful alcohol consumption—II. Addiction 1993; 88: 791–804.

20.

Derogatis

Lipman

Rickels

, et al. The Hopkins Symptom Checklist (HSCL): a self‐report symptom inventory. Behav Science 1974; 19: 1–15.

21.

Hawthorne

Elliott

. Imputing cross-sectional missing data: comparison of common techniques. Aust N Z J Psychiatry 2005; 39: 583–590. 2005/07/06. DOI: 10.1080/j.1440-1614.2005.01630.x.

22.

Muller

Clausen

. Group exercise to improve quality of life among substance use disorder patients. Scand J Public Health 2015; 43: 146–152.

23.

von Elm

Altman

DGP

Egger

, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet (London, England) 2007; 370: 1453–1457. DOI: 10.1016/S0140-6736(07)61602-X.

24.

Rothman

Gallacher

JEJ

Hatch

. Why representativeness should be avoided. Int J Epidemiol 2013; 42: 1012–1014. DOI: 10.1093/ije/dys223.