Sage Journals: Discover world-class research

Abstract

Background:

The Better Outcomes Registry and Network Ontario Information System (BIS) has captured data on births in Ontario since 2012. Data and information quality is a foundational pillar of Ontario’s birth registry.

Objective:

To evaluate data quality and reliability, we compared birth data in the BIS with like data elements in the Canadian Institute for Health Information-Discharge-Abstract-Database (CIHI-DAD) which captures administrative, clinical, and demographic data on all hospital discharges.

Methods:

We used unique pregnancy identifiers to deterministically link maternal records in the BIS to the CIHI-DAD in the fiscal years 2016–2017 to 2020–2021. Percent agreement and Cohen Kappa Coefficients (simple or weighted) with 95% confidence intervals (CI) assessed agreement on selected elements in both databases. Sensitivity analyses explored the impact of the COVID-19 pandemic on data entry and quality processes.

Results:

There was excellent percentage agreement (⩾90%) between the two databases for all maternal elements assessed. Fourteen out of the twenty elements assessed indicated substantial (κ = 0.61–0.80) or almost perfect agreement (κ = 0.81–0.99) on Kappa tests. Sensitivity analyses restricting the linked cohort to data entered before (2016/2017–2019/2020) and during (2020/2021) the COVID-19 pandemic demonstrated no significant changes in agreement across all elements.

Conclusion:

Overall, the BIS and CIHI-DAD databases had high agreement on most maternal data elements; however, further examination is necessary to explore discrepancies identified.

Implications for health information management practice:

As the BIS is newer than the CIHI-DAD and uses a different method of data abstraction, routinely evaluating and enhancing data quality is crucial for providing accurate and valid evidence for health policy, surveillance, and research.

Keywords

data quality data linkage evaluation registries data analysis hospital information systems health information management agreement Kappa test validation

Introduction

Data and information quality is a foundational pillar of Ontario’s birth registry, the Better Outcomes Registry and Network (BORN). Since 2012, BORN Ontario has collected pregnancy, birth, and newborn information from Ontario’s hospitals, birth centres, midwifery practice groups, prenatal and newborn screening laboratories, and fertility clinics for the purposes of facilitating and improving maternal-newborn care (Murphy et al., 2021). Ontario has over 140,000 births per year, which represents about 40% of births in Canada and makes BORN Ontario the largest perinatal registry in the country (Murphy et al., 2021). Data collected in the BORN Ontario Information System (BIS) is routinely used to monitor clinical programs, benchmark and evaluate outcomes, support quality improvement and facilitate research (BORN Ontario, 2025). High-quality data are essential to meet BORN’s mission of being an authoritative source of accurate, trusted, and timely information to monitor, evaluate, and plan for the best possible beginnings for life-long health (BORN Ontario, 2023a). There have been four previous validation studies performed assessing data from the BIS (Bacal et al., 2020; Darling et al., 2024; Dunn et al., 2019; Miao et al., 2019). Detailed information on each study and what data they assessed can be found in online Supplement Box S1. To summarize, Miao et al. (2019) assessed key maternal-newborn data elements by comparing the information entered into the BIS to data from the Canadian Institute for Health Information-Discharge-Abstract-Database (CIHI-DAD). They observed excellent percentage agreement (⩾90%) and a Kappa coefficient range of substantial (0.61–0.80) to almost perfect agreement (0.81–0.99) for all variables of interest (Miao et al., 2019). They recommended further investigating data elements that met only substantial agreement and exploring additional data elements in future validation studies. In chart re-abstraction studies, Dunn et al. (2019) assessed 29 data elements, comparing BIS information to re-abstracted patient records. Percent agreement ranged from 56.9%–99.8% and recommended BORN clarify some definitions and reassess and refine problematic data elements. Bacal et al. (2020) evaluated data entered into the fertility database held at BORN (The Canadian Assisted Reproductive Technologies Register (CARTR) Plus) to re-abstracted patient records from fertility clinics across Canada. They found an overall high quality of information for selected elements from CARTR Plus database and corroborated the need for changes to data entry processes and further validation work into discrepancies identified. Darling et al. (2024) assessed the agreement between the BIS and CIHI-DAD on maternal and fetal data elements used to define and identify a low-risk birth cohort (Darling et al., 2024). While there is limited overlap between the elements examined by Miao et al. (2019), or with our study, Darling et al. (2024) similarly reported high agreement between the two databases and echoed the need to investigate sources of discordance. Since Darling et al. (2024), the BIS has undergone changes in variable definitions and to the data collection process in 2019 and 2024 that may have implications on the agreement of data.

The BIS undergoes routine enhancement cycles to improve data and information quality which includes adding new data elements or pick-list values, updating definitions to align with clinical practice guidelines, retiring variables/pick-list values and introducing new functionality to the data coming into the system. Following these validation studies, BORN underwent enhancements to the system that may have improved data capture and collection, but this has not been formally assessed. The objective of this study was to assess agreement on select maternal data elements that have not been previously validated before, in a BIS and CIHI-DAD linked dataset. Additionally, we examined recommendations made from previous quality studies and re-assessed agreement to determine if any changes had occurred in subsequent years of BIS data collection and use.

Method

Data sources

The CIHI-DAD contains pregnancy, birth, and disposition data for every hospital birth in Ontario. Extensive quality assessments are routinely conducted on these data (CIHI, 2017, 2023). The BIS is an Internet-based birth record of BORN Ontario. The BIS has a near-complete capture of all hospital, home and birth centre births in the province and is a reliable, high quality and comprehensive source of information on maternal demographics and health behaviours, pre-existing maternal health conditions, obstetric complications, intrapartum interventions and birth outcomes (Murphy et al., 2021). The BIS contains extensive data across the continuum of care including fertility treatment, prenatal screening, newborn screening and labour and birth outcomes. There are different methods of entering data into the BIS including manual data entry where hospital or midwifery practice group staff enter data directly into the BIS, automated extraction and upload from electronic health record systems and Health Level Seven (HL7) automated data feeds which allow real time clinical data documented in the electronic health record be sent to the BIS (Murphy et al., 2021). HL7 defines a set of international standards for the exchange, integration, sharing and retrieval of electronic health information (Health Level Seven International, 2025). Integrating HL7 feeds from a health record to the BIS involves sending data based on triggers which are then processed by an interface engine that translates and routes the data to match the BIS format. An ongoing program of data verifications, quality checks and formal training sessions for individuals collecting and entering data into the BIS assures a high level of data quality is maintained (Murphy et al., 2021).

Data linkage

Using all pregnancy records in the BIS, we excluded maternal records with incomplete birth records, missing or invalid pregnancy outcomes, multiple pregnancies or pregnancies ending in stillbirth/terminations and non-hospital births. We extracted pregnancies with delivery dates for five fiscal years: 1 April 2016 to 31 March 2021.

In the CIHI-DAD, maternal and infant delivery records are obtained from separate discharge abstracts. To address this, BORN has created a crosswalk file that uses deterministic and probabilistic linkage methods to connect infant and maternal records (Miao et al., 2019). We refined the crosswalk file to only include valid maternal records in the BIS and CIHI-DAD with consistent infant birth dates and singleton livebirth deliveries. We then matched these records with the CIHI maternal hospital discharge records to retrieve maternal data elements. Finally, we linked the BIS data and the CIHI-DAD using a unique maternal identifier assigned by BORN, retaining only mother–infant dyads. Maternal elements were selected based on clinical relevance and to re-evaluate and compare against previous validation studies. We compared the following maternal elements: induction of labour, postpartum hemorrhage (PPH), augmentation of labour, hypertensive disorders of pregnancy (HDP composite including eclampsia, preeclampsia, hypertension with superimposed preeclampsia), gestational hypertension, pre-existing hypertension, pre-existing diabetes, gestational diabetes, breech presentation, premature rupture of membranes (PROM)/preterm premature rupture of membranes (PPROM), placenta previa, placental abruption, 1st to 4th degree perineal laceration, vaginal birth after cesarean (VBAC), cesarean delivery and assisted vaginal delivery (forceps, vacuum or forceps plus vacuum).

Statistical analysis

Descriptive analyses were used to present summary statistics on BIS records matched to the CIHI-DAD dataset. Maternal records with missing data for selected elements were evaluated in internal bias analyses and excluded from element-specific denominators. Percent agreement and Cohen Kappa Coefficients (simple or weighted) with 95% confidence intervals (CI) were used to assess agreement on selected elements in both data sources (Cohen, 1960). The strength of agreement was evaluated using the following criteria: Kappa coefficient < 0: less than chance agreement; 0.01–0.20: slight agreement; 0.21–0.40: fair agreement; 0.41–0.60: moderate agreement; 0.61–0.80: substantial agreement; and 0.81–0.99: almost perfect agreement (Cohen, 1960; Viera and Garrett, 2005). Elements with kappa coefficients below almost perfect agreement were evaluated further to determine potential reasons for discrepancies. We performed sensitivity analyses to assess the impact of the COVID-19 pandemic on data entry and quality processes. We repeated our main model analyses separately, analysing data entered before the pandemic (2016/2017–2019/2020) and data entered during the pandemic (2020/2021). SAS 9.4 (SAS Institute Inc., Cary, NC, USA) was used to perform all data linkages and analyses. As a quality assurance project, this data quality assessment was exempt from Research Ethics Board review under article 2.5 of the Tri-Council Policy Statement (TCPS2), the overarching ethical framework for research involving human participants in Canada.

Results

In total, across the BIS and CIHI-DAD databases, 650,353 records were linked in five fiscal years of integrated maternal records. Figure 1 presents a study flow diagram of the cohort refinement and linkage process. The linkage rates between the two databases were 97.9% (650,353 of 663,997) in the BIS database and 99.8% (650,353 of 651,344) in the CIHI-DAD. Table 1 contains the descriptive statistics between the BIS and CIHI-DAD databases on selected maternal elements among the linked cohort. Internal data quality analyses identified no data elements with missing values exceeding 5%.

Figure 1.

Study flowchart of cohort refinement and linkage (fiscal year 2016–2017 to fiscal year 2020–2021).

Table 1.

Descriptive characteristics of selected elements between the BORN Database and CIHI-DAD in the linked dataset (fiscal year 2016–2017 to fiscal year 2020–2021).

Variables	N ^a	BIS		CIHI-DAD
Variables	N ^a	n	%	n	%
HDP	647,017	8896	1.38	13,387	2.08
Gestational hypertension	645,017	25,253	3.92	25,065	3.89
Pre-existing hypertension	646,663	6301	0.97	4655	0.72
Pre-existing diabetes	646,597	7157	1.11	6577	1.02
Gestational diabetes	646,870	55,857	8.63	55,532	8.58
PROM/PPROM	645,392	43,089	6.68	83,317	12.91
Placenta previa	643,809	4941	0.77	4778	0.74
Placental abruption	648,703	5363	0.83	6599	1.02
Induction	650,326	191,246	29.4	198,678	30.6
Augmentation	347,128	199,802	57.56	179,713	51.77
Breech presentation	625,092	22,284	3.56	23,221	3.71
Forceps	638,688	19,782	3.10	13,540	2.12
Vacuum	638,688	53,440	8.37	41,671	6.52
Forceps and vacuum	638,688	2286	0.36	2179	0.34
Perineal laceration: 1st degree	573,004	87,662	15.30	89,397	15.60
Perineal laceration: 2nd degree	573,004	183,807	32.08	184,903	32.27
Perineal laceration: 3rd or 4th degree	573,004	17,228	3.01	18,222	3.18
Cesarean birth	650,353	192,177	29.55	191,959	29.52
VBAC	646,429	18,529	2.87	16,268	2.52
Postpartum hemorrhage	649,562	18,096	2.79	33,193	5.11

HDP: hypertensive disorders of pregnancy (eclampsia, preeclampsia, hypertension with superimposed preeclampsia); PROM: premature rupture of membranes; PPROM: preterm premature rupture of membranes; VBAC: vaginal birth after cesarean.

Denominator excludes any records missing data for that variable.

Table 2 presents the agreement between the two databases among the linked cohort. All 20 maternal elements assessed indicated excellent percentage agreement (⩾90%) between the BORN and CIHI-DAD databases. Almost perfect agreement (κ = 0.81–0.99) was noted for gestational diabetes, induction of labour, breech presentation, vacuum-assisted delivery, 2nd degree perineal laceration, 3rd or 4th degree perineal laceration, cesarean birth and VBAC. Elements with substantial agreement (κ = 0.61–0.80) included pre-existing diabetes, placenta previa, placental abruption, augmentation of labour, forceps delivery and 1st degree perineal laceration. Finally, moderate agreement on Kappa tests (κ = 0.41–0.60) was noted for HDP, gestational hypertension, pre-existing hypertension, PROM/PPROM, forceps and vacuum-assisted delivery and PPH. Table 3 presents the sensitivity analysis that examined pre-pandemic data separately from pandemic data. We observed no significant changes in agreement after restricting the linked cohort to data entered before the COVID-19 pandemic (2016/2017–2019/2020) and data entered during the COVID-19 pandemic (2020/2021).

Table 2.

Agreement between the BORN database and CIHI-DAD in the linked dataset (fiscal year 2016–2017 to fiscal year 2020–2021).

Variables	N ^a	% agreement	Kappa test
Variables	N ^a	% agreement	Kappa coefficient^b	95% CI	p-Value
HDP	647,017	98.18	0.4655	0.4573–0.4737	<0.0001
Gestational hypertension	645,017	96.88	0.5842	0.5790–0.5894	<0.0001
Pre-existing hypertension	646,663	99.08	0.4522	0.4406–0.4638	<0.0001
Pre-existing diabetes	646,597	99.38	0.7042	0.6954–0.7129	<0.0001
Gestational diabetes	646,870	97.69	0.8530	0.8507–0.8553	<0.0001
PROM/PPROM	645,392	90.86	0.4884	0.4849–0.4918	<0.0001
Placenta previa	643,809	99.65	0.7652	0.7558–0.7746	<0.0001
Placental abruption	648,703	99.39	0.6652	0.6554–0.6750	<0.0001
Induction	650,326	91.89	0.8068	0.8052–0.8084	<0.0001
Augmentation	347,128	81.00	0.6180	0.6154–0.6206	<0.0001
Breech presentation	625,092	99.16	0.8799	0.8767–0.8831	<0.0001
Forceps	638,688	98.74	0.7532	0.7480–0.7584	<0.0001
Vacuum	638,688	97.54	0.8216	0.8189–0.8243	<0.0001
Forceps and vacuum	638,688	99.72	0.6011	0.5841- 0.6181	<0.0001
Perineal laceration: 1st degree	573,004	93.35	0.7453	0.7429–0.7477	<0.0001
Perineal laceration: 2nd degree	573,004	93.99	0.8623	0.8609–0.8637	<0.0001
Perineal laceration: 3rd or 4th degree	573,004	99.38	0.8973	0.8939–0.9007	<0.0001
Cesarean birth	650,353	99.87	0.9969	0.9967–0.9972	<0.0001
VBAC	646,429	99.38	0.8820	0.8783–0.8856	<0.0001
Postpartum hemorrhage	649,562	95.90	0.4615	0.4560–0.4669	<0.0001

CI: confidence interval; HDP: hypertensive disorders of pregnancy (eclampsia, preeclampsia, hypertension with superimposed preeclampsia); PROM: premature rupture of membranes; PPROM: preterm premature rupture of membranes; VBAC: vaginal birth after cesarean.

Denominator excludes any records missing data for that variable.

Kappa coefficient < 0: less than chance agreement; 0.01–0.20: slight agreement; 0.21–0.40: fair agreement; 0.41–0.60: moderate agreement; 0.61–0.80: substantial agreement; and 0.81–0.99: almost perfect agreement.

Table 3.

Sensitivity analysis between pre-pandemic (fiscal year 2016–2017 to 2019–2020) and pandemic data (fiscal year 2020–2021).

Variables	Pre-pandemic (2016–2017 to 2019–2020)				Pandemic (2020–2021)
	N ^a	% agreement	Kappa test		N ^a	% agreement	Kappa test
	N ^a	% agreement	Kappa coefficient^b	95% CI	N ^a	% agreement	Kappa coefficient^b	95% CI
HDP	519,759	98.21	0.4716	0.4625–0.4807	125,258	98.06	0.4407	0.4223–0.4592
Gestational hypertension	519,759	96.97	0.5770	0.5710–0.5829	125,258	96.53	0.6078	0.5971–0.6185
Pre-existing hypertension	521,119	99.11	0.4541	0.4409–0.4672	125,544	98.96	0.4454	0.4204–0.4704
Pre-existing diabetes	521,077	99.38	0.6979	0.6880–0.7078	125,520	99.38	0.7274	0.7090–0.7457
Gestational diabetes	521,336	97.68	0.8502	0.8476–0.8528	125,534	97.73	0.8637	0.8588–0.8686
PROM/PPROM	520,166	90.80	0.4863	0.4824–0.4902	125,226	91.13	0.4970	0.4891–0.5050
Placenta previa	518,919	99.65	0.7667	0.7563–0.7771	124,890	99.65	0.7587	0.7368–0.7805
Placental abruption	522,853	99.38	0.6624	0.6515–0.6733	125,138	99.43	0.6776	0.6552–0.7000
Induction	524,154	91.95	0.8052	0.8034–0.8070	126,172	91.64	0.8118	0.8083–0.8152
Augmentation	284,628	81.08	0.6195	0.6167–0.6224	62,500	80.66	0.6111	0.6049–0.6173
Breech presentation	503,801	99.19	0.8849	0.8814–0.8884	121,291	99.02	0.8591	0.8512–0.8670
Forceps	514,685	98.72	0.7481	0.7423–0.7540	124,003	98.85	0.7744	0.7630–0.7857
Vacuum	514,685	97.54	0.8214	0.8184–0.8244	124,003	97.54	0.8224	0.8163–0.8286
Forceps and vacuum	514,685	99.72	0.5973	0.5783–0.6163	124,003	99.73	0.6165	0.5785–0.6545
Perineal laceration: 1st degree	461,349	93.23	0.7425	0.7398–0.7452	111,655	93.83	0.7571	0.7516–0.7625
Perineal laceration: 2nd degree	461,349	93.92	0.8603	0.8587–0.8619	111,655	94.29	0.8703	0.8673–0.8734
Perineal laceration: 3rd or 4th degree	461,349	99.36	0.8923	0.8885–0.8962	111,655	99.48	0.9167	0.9100–0.9235
Cesarean birth	524,172	99.88	0.9971	0.9968–0.9973	125,987	99.85	0.9964	0.9959–0.9969
VBAC	520,408	99.39	0.8845	0.8806–0.8885	126,021	99.34	0.8710	0.8624–0.8797

Abbreviations: CI, Confidence Interval; HDP, Hypertensive Disorders of Pregnancy (Eclampsia, Preeclampsia, Hypertension with superimposed preeclampsia); PROM, Premature Rupture of Membranes; PPROM, Preterm Premature Rupture of Membranes; VBAC, Vaginal Birth After Cesarean. ^aDenominator excludes any records missing data for that variable.

Kappa coefficient <0: less than chance agreement; 0.01-0.20: slight agreement; 0.21-0.40: fair agreement; 0.41-0.60: moderate agreement; 0.61-0.80: substantial agreement; and 0.81-0.99: almost perfect agreement.

All p-values were <.0001.

Discussion

We observed excellent percentage agreement (⩾90%), and moderate to almost perfect agreement on Kappa tests for all assessed maternal data elements when linking the BORN CIHI-DAD databases. Of the elements assessed, 14 out of 20 indicated substantial or near-perfect agreement on Kappa tests; however, potential sources of discrepancies or discordance between six elements with moderate agreement (HDP, gestational hypertension, pre-existing hypertension, PROM/PPROM, forceps and vacuum-assisted delivery, PPH) warrant further exploration. Our findings of strong agreement or concordance on assessed maternal elements is consistent with previous validation studies between the BORN and CIHI-DAD databases (Darling et al., 2024; Miao et al., 2019), validation studies using legacy data from the Niday Perinatal database (Dunn et al., 2011), and other provincial perinatal database validation studies from British Columbia and Nova Scotia (Frosst et al., 2015; Joseph and Fahey, 2009).

Several data elements included in this study were also evaluated as part of other perinatal validation studies by Joseph and Fahey (2009) and Dunn et al. (2011) including forceps and vacuum-assisted delivery, induction of labour, cesarean birth/delivery and perineal lacerations. Compared to the Niday Perinatal Database Audit (Dunn et al., 2011), a historical precursor to the BIS and held within the BORN registry, our study demonstrated considerable improvement in agreement for the forceps and vacuum data element. The data element forceps and vacuum, which had 86.5% agreement (k = 0.56) in the Niday audit (Dunn et al., 2011) had an improved agreement of 99.7% (k = 0.60) in the BORN database. This could be due to the validation rules set in the BIS that require a certain value for the data element dependent on other variables in the system. For example, if “Failed forceps/vacuum” is selected for All indications for cesarean birth data element then forceps/vacuum cannot equal “None.” Validation rules are created in the BIS to help reduce the discrepancies between data elements which can therefore improve the overall data quality of a variable. Additionally, BORN has a regional coordinator team spread throughout the province who routinely provide feedback to sites when problems with data are found, with the goal to improve the quality of the data element.

Joseph and Fahey (2009) compared the detection of labour induction, cesarean delivery and perineal lacerations (1st–4th degree) between the CIHI-DAD and the Nova Scotia Atlee Perinatal Database (NSAPD; Joseph and Fahey, 2009). Based on the assumption that the NSAPD was a gold standard database, they observed high sensitivity (89.2%) and specificity (96.9%) values for labour induction, indicating the condition was coded with reasonable accuracy in the CIHI-DAD (Joseph and Fahey, 2009). Their findings were supported by a perinatal validation study from British Columbia also reporting high sensitivity (93.9%) and specificity (98.7%) values for this data element (Frosst et al., 2015). For the data element induction of labour, we observed similar percent agreement and an improved Kappa score (91.9% agreement, k = 0.81) compared to Miao et al.’s (2019) validation study of the BORN database and CIHI-DAD (92.1% agreement, k = 0.78). These findings are consistent with another BORN data quality study that evaluated labour type in a chart re-abstraction (Dunn et al., 2019). Regarding the induction of labour data element, improvements in accuracy are likely reflective of BORN introducing a clearer definition of this data element in 2015 to their data dictionary tool (BORN Ontario, 2023b; Miao et al., 2019). The definition of induced labour in the BORN database was updated to assist data entry providers and address identified coding issues as the previous definition failed to clarify that induction involves artificial intervention before spontaneous labour begins. Based on the Society of Obstetricians and Gynaecologists of Canada guideline, induced labour was updated to be defined as “the artificial initiation of labour by any intervention/s before the spontaneous onset of the latent phase of labour to help achieve a vaginal birth in 24–48 hours.” (BORN Ontario, 2023b; Miao et al., 2019).

For the cesarean birth data element, we observed consistent percent agreements (99.9%) and Kappa tests (k = 0.996) compared to the validation study by Miao et al. (2019; 99.8% agreement, k = 0.995). In the previously mentioned validation study between the CIHI-DAD and NSAPD, maternal indicators for cesarean delivery and perineal laceration demonstrated a high degree of accuracy (Joseph and Fahey, 2009). Compared to the Niday audit, we observed significant improvements in percent agreement and Kappa test scores for perineal laceration (Dunn et al., 2011). In our linked study cohort, the high percent agreement and Kappa tests suggest that these conditions are also accurately captured and documented in the BORN database.

Prevalence estimates for maternal data elements were similar between the two databases except for PPH and PROM/PPROM, which were markedly lower in the BORN database. Both PPH and PROM/PPROM demonstrated high percent agreement and moderate agreement on Kappa tests. With respect to PPH, potential reasons for discrepancy may be due to differences in the data sources submitting to the BORN and CIHI-DAD databases as well as differences in definitions and coding standards. Individuals with a severe PPH are more likely to be admitted to an intensive care unit (ICU), which is not captured by BORN (BORN Ontario, 2023c; Murphy et al., 2021). The CIHI-DAD receives data from all acute care settings, including the ICU, which would likely improve ascertainment of this condition. In addition, postpartum data holdings at BORN only document the maternal course of care experienced from 1-hour post birth and up to the birthing individual’s discharge or transfer from the hospital’s postpartum unit (BORN Ontario, 2023c; Murphy et al., 2021). This restricted period of documenting postpartum data challenges the collection of PPH subtypes (primary/early and secondary/late/delayed; CIHI, 2022; Wormer et al., 2023). Although the CIHI-DAD identifies more cases of PPH (33,193 cases compared to 18,096 in the BORN database), discrepancies in the database have been identified. A re-abstraction study of the 2015–2016 CIHI-DAD observed high agreement rates (89%) for PPH but identified two sources of discordance: incorrectly coding the timing of the haemorrhage as “intrapartum” which would erroneously exclude it from obstetric haemorrhage clinical groups and second, disagreements in chart documentation by physicians on the amount of blood loss and associated factors (CIHI, 2016). While PPH has been defined in published literature, variations in clinical definitions, including subjective assessments beyond standard norms, have persisted such that PPH remains a significantly underestimated obstetric problem (CIHI, 2022; Francois, 2014; Rath, 2011; World Health Organization, 2017). In 2023, BORN began a comprehensive review of all data element and pick-list value definitions with the goal to create high-quality definitions to promote consistency in interpretation both by those entering data into the BIS and those using BORN data. This included an enhancement in 2024 to PPH data to distinguish early PPH (within the first 24 hours after birth) to late PPH (occurring 24 hours to 6 weeks after birth). It is hoped that the updates to definitions based on clinical guidelines will help improve capture of data in the BIS. These enhancements will be evaluated over time as data enters into the system and PPH will be revisited in future quality analyses. Furthermore, because PPH diagnosis is based on a subjective clinical assessment that includes any amount of blood loss that may endanger a birthing individual’s haemodynamic stability, physicians are likely to document PPH even if observed blood loss does not satisfy clinical criteria stated in the coding guideline (CIHI, 2022; Francois, 2014). In the CIHI-DAD, this means that individuals are included in obstetric haemorrhage clinical groups if PPH is documented by the physician but clinical criteria may not be satisfied (CIHI, 2022).

Discordance was also noted for HDP, though this is also likely reflective of the varying subtypes of HDP and the differences in the labelling and classification systems used by the BORN database and the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision, Canada (ICD-10-CA) system used by the CIHI-DAD. This discrepancy is echoed in the re-abstraction study between the CIHI-DAD and the NSAPD (Joseph and Fahey, 2009). Though we examined hypertension in pregnancy as a composite outcome, consistent with a previous BORN re-abstraction study, we did not include gestational hypertension or pre-existing hypertension in this composite and examined these conditions separately. Compared to Dunn et al. (2019), we noted higher percent agreement HDP, gestational hypertension and pre-existing hypertension. Observed agreement on Kappa tests was similar, though slightly lower for HDP, likely due to our exclusion of gestational hypertension and pre-existing hypertension from the composite outcome.

Finally, as we had concerns about the effect of the COVID-19 pandemic on time for non-clinical activities such as data entry, sensitivity analyses restricting the linked cohort to data entered before (2016/2017–2019/2020) and during (2020/2021) the COVID-19 pandemic demonstrated no significant changes in agreement across all elements, including no improvement among elements with moderate agreement. To our knowledge, there has not been a validation study, to date, that has examined the potential impact of the COVID-19 pandemic on the accuracy of data. Though we did not observe significant changes in Kappa tests, the stability of our findings from these sensitivity analyses is reassuring, even for elements with moderate agreement.

Limitations

Our study has some limitations. First, our assessment of the agreement between both databases and exploration into the accuracy of the BORN database did not include a hospital chart audit. The introduction of random and systematic errors due to unclear or differing clinical definitions, inadequately trained personnel documenting or submitting data, incomplete or ineligible documentation in health records, and human error during manual data entry and upload from electronic health records means that the accuracy of data must be evaluated. A complete hospital chart audit provides the optimal methodological vehicle for this assessment (Dunn et al., 2011, 2019). A data abstraction study based on hospital chart review is planned to explore potential sources and reasons behind observed discrepancies. Second, while both BORN and CIHI possess guidelines and standardised protocols for data entry, there is the possibility that a small number of errors may be introduced from manual data entry and upload of electronic health records. We also cannot disregard potential errors related to manual data entry including the incorrect notation of both identifiers (e.g. Ontario Health Insurance Plan) and diagnoses codes. Incorrect identifiers may link to the wrong records and, depending on whether a mother–infant dyad is captured, also impacts whether these incorrectly linked records are retained or excluded from the final linked cohort. However, BORN has a linking and matching clerk who investigates discrepant records, verifies information with the data provider and corrects discrepancies as they arise. Incorrect notations of diagnoses codes could have introduced inaccuracies into our estimates. Third, discrepancies can also be due to aggregation errors resulting from multiple sources of potentially conflicting information. At BORN, multiple data sources, including hospitals, midwifery practice groups, and birth centres submit data on the maternal course of care and when populating data elements, the prioritisation of submitted responses typically relies on the last submitted encounter (analogous to abstracts submitted to CIHI). If these organisations submit conflicting information (e.g. hospital encounter indicates augmentation of labour but midwifery practice group contradicts augmentation) when the record level data are combined into one aggregate labour/birth encounter these issues of discrepant data are “resolved” by considering the most recent record submitted as the source of truth. These aggregation errors may be, in part, due to differences in data entry time points for submitting organisations – midwifery practice groups submit data to BORN after hospitals. The submission of conflicting information and issues determining the chronology of submitted abstracts is also a concern within the DAD (CIHI, 2016).

Conclusion

Overall, the BIS and CIHI-DAD databases had high agreement on most maternal data elements; however, further examination is necessary to explore discrepancies identified. These discrepancies may be due to differences in data element definitions, varying data entry time points, different submitting organisations or sources (i.e. BORN does not collect/receive maternal ICU data), and differences between clinical diagnosis in the BORN database and ICD-10-CA classifications used in the CIHI-DAD (Miao et al., 2019, 2020). As the BORN database is newer than the CIHI-DAD and uses a different method of data abstraction, routinely evaluating and enhancing data quality is crucial for providing accurate and valid evidence for health policy, surveillance and research.

Supplemental Material

sj-docx-1-him-10.1177_18333583251375127 – Supplemental material for BORN to be validated: Assessing agreement between Ontario’s birth registry and CIHI-DAD

Supplemental material, sj-docx-1-him-10.1177_18333583251375127 for BORN to be validated: Assessing agreement between Ontario’s birth registry and CIHI-DAD by Tavleen Dhinsa, Nicole F Roberts, Qun Miao, Carolina Lavin Venegas, Catherine Ménard, Kaamel Hafizi and Ann E Sprague in Health Information Management Journal

Footnotes

Acknowledgements

We would like to thank CIHI for providing the CIHI-DAD data to BORN Ontario.

Accepted for publication August 19, 2025.

Availability of data and materials

The datasets generated and/or analysed during this study are held securely by BORN Ontario and cannot be made publicly available due to the personal health information contained in them. To request BORN data, please complete the following form: .

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The authors received no financial support for the research, authorship and/or publication of this article.

ORCID iDs

Tavleen Dhinsa, MSc

Nicole F Roberts, MSc

Carolina Lavin Venegas, BScN, MSc

Ann E Sprague, RN, PhD

Supplemental material

Supplemental material for this article is available online.

References

Bacal

Fell

Shapiro

, et al (2020) The Canadian Assisted Reproductive Technologies Register (CARTR) Plus database: A validation study. Human Reproduction Open 2020(2): hoaa005.

BORN Ontario (2023a) BORN Ontario - About BORN. Available at: https://www.bornontario.ca/en/about-born/about-born.aspx#Mission (accessed 13 February 2024).

BORN Ontario (2023b) BORN Ontario - Data dictionary and library. Available at: https://www.bornontario.ca/en/data/data-dictionary-and-library.aspx (accessed 14 February 2024).

BORN Ontario (2023c) BORN Ontario - Data holdings. Available at: https://www.bornontario.ca/en/data/data-holdings.aspx (accessed 14 February 2024).

BORN Ontario (2025) Research that uses BORN data. Available at: https://www.bornontario.ca/en/publications/research-that-uses-born-data.aspx (accessed 22 June 2025).

Canadian Institute for Health Information (CIHI) (2016) Data quality study of the 2015–2016 discharge abstract database: A focus on hospital harm. Ottawa, ON: Canadian Institute for Health Information.

Canadian Institute for Health Information (CIHI) (2017) CIHI’s information quality framework. Available at: https://www.cihi.ca/sites/default/files/document/dad_1516_reab_report_en.pdf (accessed 13 February 2024).

Canadian Institute for Health Information (CIHI) (2022) A guide to obstetrical coding. Available at: https://www.cihi.ca/sites/default/files/document/obstetrical-coding-guide-en.pdf (accessed 14 February 2024).

Canadian Institute for Health Information (CIHI) (2023) Data quality documentation, Discharge Abstract Database — Current-year information, 2022–2023. Available at: https://www.cihi.ca/sites/default/files/document/dad-data-quality-current-year-information-2022-2023-report-en.pdf (accessed 13 February 2024).

10.

Cohen

(1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1): 37–46.

11.

Darling

Marquez

Park

(2024) Defining a low-risk birth cohort: A cohort study comparing two perinatal data sets in Ontario, Canada. International Journal of Population Data Science 9(1): 2364.

12.

Dunn

Bottomley

Ali

, et al (2011) 2008 Niday perinatal database quality audit: Report of a quality assurance project. Chronic Diseases in Canada 32(1): 32–42.

13.

Dunn

Lanes

Sprague

, et al (2019) Data accuracy in the Ontario birth Registry: A chart re-abstraction study. BMC Health Services Research 19(1): 1001.

14.

Francois

(2014) Postpartum hemorrhage. In: Foley

Strong

T Jr

Garite

(eds.) Obstetric Intensive Care Manual, 4e. Columbus, OH: McGraw Hill Medical.

15.

Frosst

Hutcheon

Joseph

, et al (2015) Validating the British Columbia Perinatal Data Registry: A chart re-abstraction study. BMC Pregnancy and Childbirth 15(1): 123.

16.

Health Level Seven International (2025) About Health Level Seven International | HL7 International. Available at: https://www.hl7.org/about/index.cfm?ref=nav (accessed 23 July 2025).

17.

Joseph

Fahey

(2009) Validation of perinatal data in the Discharge Abstract Database of the Canadian Institute for Health Information. Chronic Diseases in Canada 29(3): 96–100.

18.

Miao

Fell

Dunn

, et al (2019) Agreement assessment of key maternal and newborn data elements between birth registry and clinical administrative hospital databases in Ontario, Canada. Archives of Gynecology and Obstetrics 300(1): 135–143.

19.

Miao

Moore

Dougan

(2020) Data quality assessment on congenital anomalies in Ontario, Canada. Frontiers in Pediatrics 8: 573090.

20.

Murphy

MSQ

Fell

Sprague

, et al (2021) Data resource profile: Better Outcomes Registry & Network (BORN) Ontario. International Journal of Epidemiology 50(5): 1416.

21.

Rath

(2011) Postpartum hemorrhage – Update on problems of definitions and diagnosis. Acta Obstetricia et Gynecologica Scandinavica 90(5): 421–428.

22.

Viera

Garrett

(2005) Understanding interobserver agreement: The kappa statistic. Family Medicine 37(5): 360–363.

23.

World Health Organization, United Nations Population Fund & United Nations Children’s Fund (UNICEF) (2017) Managing complications in pregnancy and childbirth: a guide for midwives and doctors, 2nd ed. Geneva: World Health Organization.

24.

Wormer

Jamil

Bryant

(2023) Acute postpartum hemorrhage. In: StatPearls. Treasure Island, FL: StatPearls Publishing.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB