Factors affecting the manual linking of clinical progress notes to problems in daily clinical practice: A retrospective quantitative analysis and cross sectional survey

Abstract

This cross sectional study examines how patient characteristics, doctor characteristics, and doctors’ education and attitudes affect the extent to which doctors link progress notes to clinical problems. The independent effects of patient characteristics on the linking of notes was examined with a mixed model logistic regression. The effects of doctor characteristics and doctors’ education and attitudes on the link ratio was analyzed with univariate analysis of variance. A survey was used to obtain arguments and attitudes on linking notes. For “patient characteristics”, the odds of linking increased with an increase in the number of problems or hospital days, decreased, with an increase in the number of involved doctors, medical specialties or the number of notes. For “doctor characteristics”, the link ratio increased with more work experience. For “doctors’ education and attitudes”, the link ratio increased with more familiarity in linking notes and belief in the added value of problem oriented charting. “Overview” was the most cited reason for linking; “I don’t know how” the most cited reason for not linking. There is a huge variation within and between all disciplines. Important arguments, for and against, are found. Recommendations for policymakers and medical leadership are given to maximize the benefits.

Keywords

education electronic health records health information management policies problem-oriented medical records

Introduction

In recent years, Electronic Health Records (EHRs) have been implemented by an ever increasing number of hospitals. EHR systems have various forms and their implementation is a complex undertaking.¹ Fifty years ago, Weed² proposed a problem-oriented approach to improve the structure and content of clinical documentation. He also suggested to document the course of treatment according to the subjective, objective, assessment, and plan (SOAP) format.^2,3 The problem-oriented medical record (POMR) is a concept to organize data around medical problems. Although the POMR is widely adopted in EHR systems, only 15 original research studies have been published in the past 40 years.⁴

An accurate problem list is a core element of the POMR and provides many advantages for delivering good clinical care.⁵ It is intended to serve as a multi-disciplinary index for all health professionals. An accurate problem list can support a meaningful patient-clinician dialogue that fits with the trend of care becoming more patient-centered.^6,7 Although the term “problem list” seems to be a fairly common term in the literature^4,8 there still is discussion about many aspects of the problem list.^9,10 Moreover, there are several factors that are critical for the successful use of the problem list, such as structure, multi-disciplinary use, management, and policy.^11,12 The problem section, according to Health Level Seven International (HL7), lists and describes clinical problems or conditions currently being monitored for the patient.¹³ The concept of “problem” therefore has a broader scope within this definition than diagnosis in health and clinical management and can be used for all matters that require attention. Unfortunately it is still one of the great failures of clinical informatics to effectively integrate the management of the problem list into the clinician’s workflow.¹⁴

Linking notes to problems, as defined by HL7, is of great importance to gain a better understanding of the patient’s story and the clinical reasoning of the healthcare professionals involved. Furthermore, linking notes is useful for other aspects, such as accountability, continuity of care, education, and supervision. Nowadays, almost every EHR, including the systems from the dominant EHR vendors such as Epic and Cerner, use a patient-centered problem list where data can be linked to problems. Despite the benefits of linking, the multi-disciplinary use of the POMR, and in particular linking notes, is not a simple and straightforward matter.^15–19 The problem list can be used as an index, especially with increasing multi-disciplinary chart complexity or where patients transfer across settings. Ideally, a patient-centered problem-oriented overview could be created with data from different sources.²⁰ This also fits integrated practice units accountable for the total care for a medical condition and its complications, whereby value for the patient is created by providers’ combined efforts across the entire care cycle, rather than “focused factories” concentrating on narrow sets of interventions.²¹

Although many articles state that linkage between problems and data is essential for successful use of the problem list, linkage itself has hardly been studied.¹² Linking notes to problems is mentioned in articles, but it is mostly subjective. We are not aware of any study that has examined factors that influence the linking of notes to problems. Better understanding of these factors can contribute to further improvement of information quality and the continuity of care.

Set of factors under review

Figure 1 provides an overview of the characteristics to be analyzed about patients and doctors, as well as the education/attitudes of doctors, that may affect the linking of progress notes to items on a problem list.

Figure 1.

Set of factors under review that may affect the linkage of progress notes.

Patient characteristics include the number of problems a patient has in the chart, the number of doctors and medical specialties involved with a patient, the number of progress notes per patient, and the number of days that a patient has stayed in the hospital. Doctor characteristics include age, sex, medical specialty, and number of years of work experience. A third set of factors consists of doctors’ education and attitudes. These include education and experience with linking notes, personal and team attitudes toward linking notes, and the use of a directive for management of the problem list.

We expect a positive influence with an increase in any of the patient characteristics given the advantage of maintaining an overview of the patient’s case. For doctor characteristics, we do not expect any difference in age or sex. We expect that non-surgical specialties link more notes than surgical specialties because the latter often only concentrates to a specific problem to deal with, whereby linking notes may be of less value to them. We expect more linked notes with an increase in work experience because there will then be more experience with long-term treatment of patients where linking notes contributes to maintaining an overview of the patient’s case. We hypothesized that more education and experience with linking notes, a positive personal and team attitude toward linking notes, and the existence and use of directives for the problem list, would be positively related with the linking of notes because of the belief in the added value of this method.

Methods

Setting

In a Dutch university medical center, the linkage of progress notes over a 4-year period was studied.

Progress notes module and link functionality

An EHR progress notes module built upon a home grown EHR has been used with link functionality that is generic for all medical specialties²² and is based on the POMR and SOAP principles²³ which are now ubiquitous.²⁴ The starting point for adding and managing items on the problem list is a progress note. Problems are automatically added by what is entered in the Assessment-Diag/Hyp field (see screenshot in Figure 2). If a diagnosis is entered as part of a note, this note is automatically linked to this problem. One can write multiple diagnosis in one progress note. Allergies and active issues from the medical history are also visible on the problem list. A progress note, with or without a diagnosis, can be manually linked to one or more problems on the list by clicking the corresponding items. Based on the linked notes one can generate a problem-oriented overview that is, an overview of all notes linked to one or more problems. The problem list can then act as an index for all linked notes.

Figure 2.

Screenshot of the progress notes section of the clinical notes application.

Transaction log file and EHR database

A separate log file automatically recorded all user-system interactions, such as mouse clicks and application events. Each record of the log file contained items, such as patient identification, user identification, user role, performed action, and a date/time stamp, but no medical data. Two external professional testers validated the log file functionality. Querying the EHR database provided more detailed information for analysis of the progress notes.

Survey

To obtain doctor perceptions about the linking of notes, they received an on-line survey questionnaire. To link answers from the survey to data from the log file, the survey was not anonymous. Permission for this was granted by the privacy officer as a representative of the board of directors. In total, there were 58 questions involving various topics, such as user education, work experience, viewpoints about POMR, and use and usability of the system. More than half of the questions were about the POMR. Two open-ended questions about the linking of progress notes were not mandatory. We pilot tested the questionnaire with several doctors to assess intelligibility and clarity. For the actual survey, we selected doctors who had written at least 200 progress notes in 2012. After excluding doctors who were not able to respond because of extended leave or retirement, the total population was 700. These doctors were invited to take part in the survey by e-mail at the end of April 2013. A first reminder was sent 2 weeks later, followed by a second reminder at the end of May.

Analysis

Data has been extracted from the log file and EHR database using MS SQL Server Management. Statistical analyses were performed with IBM SPSS Statistics 25.0 (SPSS Inc., Chicago, IL). Firstly, the distribution of all notes over the days of the week and over the hours of a day was investigated. The calculated link ratio is the number of manually linked notes divided by the total number of notes that could be linked manually. Therefore we excluded automatically linked notes and notes from patients with an empty problem list. Although linking a note to a single problem or linking a general note may seem unnecessary, a note can always be linked to one or more problems. This makes a link ratio of 100% the theoretical, but not necessarily the ideal, maximum. An automatically linked note can also be linked manually. This is the case if a note is written with a diagnosis, and is also linked to an already listed problem. Overall analysis of the data was done using descriptive statistics. Significance for all analyses was set at p < 0.05.

Three sets of factors have been examined in accordance with the set of factors under review. For the set of “patient characteristics”, we counted the number of problems, regardless of the status of a problem, number of doctors involved, number of grouped specialties involved, number of progress notes, and number of hospital days, whereby each unique day where at least one progress note was written counts as a hospital day. Taking into account the nesting of notes within patients within doctors, we used a generalized linear mixed model with random intercepts for a mixed model logistic regression to study the independent effects of patient characteristics on the linking of notes expressed as an odds ratio.

For the set of “doctor characteristics”, doctors were categorized by their specialty in the following eight groups: (1) anesthesiology, (2) cardio-lung, (3) internists, (4) neurology, (5) pediatrics, (6) supporting (radiology, radiotherapy, and nuclear medicine), (7) surgeons, and (8) others. Univariate analysis of variance was used to study the effects of age, sex, medical specialty, and work experience on the link ratio. Also, the link ratio of the doctors was analyzed in the course of time. For variation between medical specialties and variation within a medical specialty, link ratios were calculated for all doctors who had written at least 200 progress notes from 2010 to 2013. A box plot was used to show the variation between doctors and medical specialties. We performed a non-respondent analysis of the survey, using chi-squared tests to study differences between responders and non-responders with respect to the medical specialty and the number of written progress notes, which were extracted from the log file.

For the set of “doctors’ education and attitudes”, the answers from the questionnaire were used in relation to the respondents’ link ratios. Univariate analysis of variance was used to study the effects in relation to the given answers. The information gathered from two open-ended questions from the questionnaire, asking the main reason to link or not to link notes, were first split into separate arguments and then categorized. Results from the other questions were grouped together and displayed in a table.

Results

In total, 1765 doctors and 1,878,705 notes referring to 193,703 patients were initially included. The extracted data from the transactional log file and EHR database with the numbers of patients, doctors and progress notes are summarized and represented in a data selection diagram (Figure 3).

Figure 3.

Data selection diagram that shows which data has been used for the individual results.

Figure 4 plots the distribution of all included notes over the days of the week. Of all notes, 202,534 were automatically linked and 435,008 were manually linked.

Figure 4.

Distribution of the unlinked, automatically linked, and manually linked notes over the days of the week.

The average percentage of manually linked notes on weekdays is 23.7% and on weekends 16.5%. The distribution of all included notes over the hours of the day shows a similar picture where the average percentage of manually linked notes from 8 am until 6 pm is 24.2% and from 6 pm until 8 am is 12.7%. Monday to Friday from 8 am to 6 pm the average percentage of manually linked notes is 24.6%.

Patient characteristics

Table 1 shows the effect of patient characteristics on the linking of notes. The table displays the percentage of notes and corresponding odds ratio for each patient characteristic.

Table 1.

Independent association between patient characteristics of notes and the linking of notes expressed in odds ratios (with 95% confidence intervals) for 1,540,084 notes on 135,899 patients.

Patient characteristics	p Value	n	% of notes	Odds ratio (95% CI)
Problems	<0.001	1	13.9	1.00
		2–3	20.4	1.04 (1.02 to 1.06)
		4–5	15.3	1.14 (1.11 to 1.16)
		6–10	25.1	1.24 (1.22 to 1.27)
		11–max	25.3	1.30 (1.27 to 1.33)
Doctors	<0.001	1–2	12.7	1.00
		3–4	12.9	0.87 (0.85 to 0.89)
		5–10	25.7	0.81 (0.79 to 0.83)
		11–20	21.3	0.73 (0.70 to 0.75)
		21–max	27.4	0.60 (0.58 to 0.62)
Medical specialty group	<0.001	1	27.1	1.00
		2	24.7	0.86 (0.85 to 0.88)
		3	19.0	0.79 (0.77 to 0.80)
		4	13.4	0.74 (0.72 to 0.75)
		5–max	15.8	0.67 (0.65 to 0.69)
Progress notes	<0.001	1–5	9.4	1.00
		6–10	12.7	0.80 (0.78 to 0.83)
		11–20	18.5	0.77 (0.74 to 0.80)
		21–50	26.4	0.76 (0.72 to 0.79)
		51–max	33.0	0.77 (0.73 to 0.82)
Hospital days	<0.001	1–5	12.3	1.00
		6–10	15.7	1.15 (1.12 to 1.19)
		11–20	21.7	1.24 (1.19 to 1.29)
		21–40	23.0	1.34 (1.28 to 1.39)
		41–max	27.3	1.36 (1.30 to 1.43)

An increase in the number of doctors, medical specialties or progress notes was independently associated with a smaller chance of linking. In contrast, an increase in hospital days or the number of problems was independently associated with a greater chance of linking.

Doctor characteristics

Table 2 shows the effects of doctor characteristics on the linking of notes. The table displays the link ratios for each doctor characteristic and categories with 95% confidence intervals. Age was not used because of the correlation between age and work experience (Cramer’s V = 0.643).

Table 2.

Link ratios (with 95% confidence intervals) of the doctors (n = 258) categorized by sex, medical specialty group, and work experience.

Doctor characteristics	p Value	Subdivision	Doctors	Link ratio (95% CI)
Sex	0.807	Male	119	0.37 (0.30 to 0.44)
Sex	0.807	Female	139	0.38 (0.32 to 0.44)
Medical specialty group	0.003	Anesthesiology	8	0.50 (0.29 to 0.71)
		Cardio-lung	22	0.27 (0.14 to 0.40)
		Surgeons	38	0.36 (0.26 to 0.46)
		Internists	58	0.29 (0.21 to 0.37)
		Pediatrics	37	0.20 (0.10 to 0.30)
		Neurology	22	0.44 (0.31 to 0.57)
		Others	62	0.41 (0.33 to 0.49)
		Supporting	11	0.53 (0.35 to 0.71)
Work experience in years	0.007	<5	57	0.28 (0.19 to 0.36)
		5–10	77	0.29 (0.21 to 0.36)
		10–15	40	0.39 (0.29 to 0.49)
		15–20	23	0.48 (0.35 to 0.62)
		>20	61	0.43 (0.35 to 0.51)

Link ratio: estimated marginal mean.

Significant variation was observed between medical specialties. Pediatrics had the lowest link ratio, while supporting had the highest link ratio. Also, significantly higher link ratios were found in doctors with more than 10 years of work experience. We found no significant differences in the link ratio between male and female doctors.

Figure 5 shows the variation between all doctors who wrote 200 or more notes, as well as the variation within a medical specialty. Within each specialty, there were doctors who did not link at all and doctors who linked very extensively. The same outcome was observed with regard to the respondents for the lowest (pediatrics) and highest (supporting) link ratios.

Figure 5.

Box plot of the link ratio of doctors who wrote 200 or more notes, grouped by medical specialty. n = 1161.

Figure 6 shows the link ratio of 1161 doctors who wrote 200 or more notes on 151,356 patients. About a quarter (24%) of the 1161 doctors (n = 274) manually linked 50% or more of their notes, representing 70% (302,967 out of 435,008) of all manually linked notes. Eleven percent of the 1161 doctors (n = 123) did not link any notes (link ratio = 0.00).

Figure 6.

Doctors’ link ratios (n = 1161).

Doctors’ education and attitudes

Table 3 shows the effect of doctors’ education and attitudes on the linking of notes. The table displays the link ratios for answer options to various aspects of doctors’ education and attitudes.

Table 3.

Link ratios (with 95% confidence intervals) of the doctors (n = 258) categorized by, familiarity with linking, training hours, medical training and POMR, problem list attention, believing that linking increases the comprehensibility, and favoring a hospital-wide directive.

Doctors’ education and attitudes	p Value	Answer	Doctors	Link ratio (95% CI)
I am familiar with linking notes	<0.001	TD	13	0.21 (0.03 to 0.39)
		D	36	0.22 (0.08 to 0.37)
		N	27	0.20 (0.04 to 0.36)
		A	120	0.40 (0.28 to 0.52)
		TA	62	0.46 (0.33 to 0.58)
I have spent . . . hours on preparation/training	0.852	<1	110	0.27 (0.19 to 0.35)
		1–4	115	0.27 (0.19 to 0.35)
		4–8	26	0.33 (0.21 to 0.46)
		8–12	5	0.29 (0.03 to 0.55)
		>12	2	0.33 (−0.07 to 0.73)
In my medical training, the POMR has been extensively discussed	0.004	TD	99	0.40 (0.28 to 0.52)
		D	88	0.34 (0.22 to 0.46)
		N	29	0.41 (0.26 to 0.56)
		A	33	0.29 (0.14 to 0.44)
		TA	9	0.05 (−0.17 to 0.27)
I pay a lot of attention to an accurate and complete problem list	0.051	TD	60	0.20 (0.06 to 0.33)
		D	94	0.25 (0.12 to 0.39)
		N	57	0.30 (0.17 to 0.44)
		A	41	0.40 (0.27 to 0.53)
		TA	6	0.34 (0.05 to 0.62)
I believe that the contents of the record become more comprehensible by linking notes to problems	0.011	TD	31	0.25 (0.10 to 0.41)
		D	45	0.21 (0.06 to 0.36)
		N	94	0.22 (0.08 to 0.35)
		A	77	0.36 (0.23 to 0.50)
		TA	11	0.45 (0.23 to 0.66)
I am in favor of a hospital-wide directive for using the problem list	0.380	TD	20	0.40 (0.22 to 0.57)
		D	27	0.29 (0.14 to 0.45)
		N	69	0.28 (0.16 to 0.41)
		A	107	0.24 (0.11 to 0.37)
		TA	35	0.28 (0.13 to 0.42)

A: agree; D: disagree; N: neutral; TA: totally agree; TD: totally disagree.

Link ratio is the estimated marginal mean.

Significantly higher link ratios were found in doctors who were familiar with linking notes and who believed that linking notes made the record more comprehensible. Attention to the POMR during medical training was significantly related to a lower link ratio. Doctors who paid a lot of attention to the problem list had higher link ratios, but the difference was not significant. We found no significant association between link ratios and training hours or favoring a hospital-wide directive.

The link ratio of the doctors over time showed that the percentage of doctors who link >50% of the notes remained stable, while the link ratio of doctors who linked <50% continued to decline.

Survey results

The results of the questionnaire were based on the response of 258 respondents (response rate = 37%). We found no differences between respondents (258) and non-respondents (442) with respect to specialty (p = 0.635) or the number of written notes (p = 0.329). In the questionnaire, we asked the doctors about the most important reason to link or not to link progress notes to problems. From the 258 doctors, there were 115 answers with one to three reasons on why to link, and 179 answers with one to three reasons on why not to link (Figure 7). “Overview” was by far the most mentioned reason to link notes, while the most mentioned reason not to link was “I don’t know how.”

Figure 7.

Doctors’ arguments in percentages about why (n = 126) or why not (n = 259) to link notes to medical problems. Reasons that scored less than 2% are grouped into “Other reasons.”

Table 4 shows the responses to the questions regarding education, experience, personal and team attitudes toward linking notes, and directives for the problem list. The results showed that many doctors had little training, most doctors were experienced and familiar with the functionality, splitting notes or linking them to multiple problems was rarely used, and a majority of the doctors were for a hospital-wide directive. Noteworthy is the question, “In daily practice I link the following average percentage of notes to problems?.” Doctors generally made a good estimate of the number of self-linked notes. The deviation of the response to the questionnaire and the measured score based on the EHR database for 71% of the doctors were less than 20%. In 9% of cases, there was an underestimation, while an overestimation was observed in 20%.

Table 4.

Answers to the survey questions concerning doctors’ education, experience, personal and team attitudes toward linking clinical notes (CN), and directives for the problem list.

	Totally disagree	Disagree	Neutral	Agree	Totally agree
Education
I have gained enough knowledge during my training/preparation for the use of CN to use the problem list in daily practice	2	18	32	40	8
I have gained enough knowledge during my training/preparation for the use of linking progress notes to problems in daily practice	2	16	28	45	10
Prior to the introduction of CN, I was informed of training opportunities for the use of CN		No (48)		Yes (52)
I have spent . . . hours on preparation/training for the use of CN	<1 (43)	1–4 (45)	4–8 (10)	8–12 (2)	>12 (1)
During my training/preparation for the use of CN, I have gained knowledge about linking notes to problems		No (43)		Yes (57)
Experience
I am experienced in using CN	2	3	10	48	36
I am experienced in using the possibilities of writing progress notes	2	3	7	52	36
I am familiar with the possibilities of the problem list	3	18	17	43	19
I am familiar with linking notes to problems	5	14	10	47	24
The number of years I have been active as a practicing doctor	<5 (22)	5–10 (30)	10–15 (16)	15–20 (9)	>20 (24)
Personal attitudes toward linking notes
In my medical training the POMR has been extensively discussed	38	34	11	13	3
I pay a lot of attention to an accurate and complete problem list	23	36	22	16	2
I make extensive use of the chronological overview	3	9	10	37	40
I make extensive use of the problem oriented overview	43	40	9	5	3
I believe that the contents of the record become more comprehensible by linking notes to problems	12	17	36	30	4
I regularly link notes to more than one problem	29	39	17	12	2
I regularly split notes to link it to one specific problem	55	38	5	1	1
I think it’s important to follow up diagnosis to make the problem list more understandable (e.g. admitting diagnosis, intermediate diagnosis and discharge diagnosis)	7	16	37	32	8
In daily practice, I link the following average percentage of notes to problems	<20 (45)	20–40 (9)	40–60 (12)	60–80 (14)	>80 (21)
Team attitudes toward linking notes
Within my medical discipline, there are colleagues who actively propagate linking notes to problems	19	33	24	22	3
I try myself to encourage colleagues to link notes to problems	29	34	21	14	2
I am sometimes approached by a colleagues about the fact that my notes are not properly linked	42	44	9	4	0
I sometimes approach colleagues on the fact that they do not properly link their notes	38	38	12	11	1
Directives for the problem list
There is, within my specialty, a directive for the management of the problem list		No (43)	Don’t know (43)	Yes (14)
There is, within my specialty, a directive for linking notes to problems		No (40)	Don’t know (43)	Yes (17)
These directives are aligned with the directives of other specialties		No (25)	Don’t know (56)	Yes (19)
Generally, I stick to these directives	0	8	29	58	4
Generally, my colleagues conform to this directive	2	15	42	40	2
It irritates me if colleagues are using the problem list wrongly	16	21	32	27	4
When colleagues use or build a problem list in the wrong way, I tend also to be less accurate	21	39	24	14	2
I am in favor of a hospital-wide directive for using the problem list	8	10	27	41	14

Values are frequencies with percentages in brackets unless stated otherwise.

Discussion

Principal findings

For “patient characteristics”, the linking clearly increased with an increase in the number of hospital days or the number of problems. Contrary to our hypothesis, an increase in the number of involved doctors, medical specialties or the number of notes clearly decreased the linking. For “doctor characteristics”, the linking increased with work experience of more than 10–15 years, which was in agreement with our hypothesis. The variation of linking notes between medical specialties, but also between doctors within a specialty, was enormous. Some doctors did not link at all, while others linked very extensively. In contrast to our hypothesis, internists had a slightly lower link ratio than surgeons. For “doctors’ education and attitudes”, doctors who were familiar with linking notes, paid attention to problem list accuracy, and believed in the added value of linking notes had a higher link ratio. Remarkably, the attention to the POMR during medical training seemed to have a negative effect on the linking of notes, possibly due to the expected additional administrative actions. Over time, the link ratio decreased within the group of doctors who linked 50% or less, while the group who linked 50% or more of their notes remained stable. During the evening and at night, significantly fewer notes are linked. By far, “Overview” was the most commonly given reason to link progress notes to problems, while the most common reason not to link was “I don’t know how” followed by “Forget”, “Too complicated”, “Time consuming”, and “No added value”.

Strengths of the study

The main strength of this study is that this is the first study that has measured the linking of notes from a generic multidisciplinary application and that provides comprehensive insight into factors that affect the manual linking of notes. Another strength of this study is the combination of data sources, a detailed transactional log file, the EHR database, and survey data. For the first two sources, data were collected over a relatively long period. The log file itself was validated twice by independent professional testers. Because the survey was not anonymous, we were able to investigate specific characteristics. Furthermore, during the almost 4 years of data collection, this module has hardly been changed.

Weaknesses of the study

The response rate to the survey (37%) was relatively low; however, our analysis of non-responders did not reveal any difference between respondents and non-respondents with respect to specialty or the number of written notes. Furthermore, the questionnaire was not anonymous. While this allowed the possibility to link the results to the log file, the lack of anonymity may have biased respondents. However, this effect is probably small, because there were no right, wrong, or socially desirable answers in our survey. We also had no reason to believe that questions were answered dishonestly.

The measurements were done in only one university hospital, which may limit the generalizability. Apart from that, comparison with other hospitals is difficult, because they use different systems that are highly configurable, what applies to both homegrown and commercial systems. Many variables play a role in the way in which doctors use the system.²⁵ Therefore, we mainly focused to compare different groups of doctors using the same system for a number of years in which the system has hardly changed.

Additionally, patient characteristics were measured at the last date of the log file, as well as the measurements for the other factors, such as the number of linked and unlinked notes. If, during the addition of new notes, a change occurs in these factors that could also influence linking, it is averaged out.

The fact that automatically linked notes that were also linked manually have been excluded, could lead to a small underestimate of the link ratio.

There may be other factors that we have not explored, which could explain the variation in linking notes to problems. Factors such as the nationality of doctors or patients, factors that affect physician professional satisfaction due to poor EHR usability,²⁶ working conditions for clinicians,^27,28 the location or level of care such as an intensive care unit, combined with operational policies, that may affect the department’s behavior and performance.²⁹ In addition, rewarding or financial incentives could be latent variables that might influence care provided by doctors,^30,31 and also the linking of notes so that, for example, all transactions for a specific problem are billable. Also, if one is in doubt but is responsible, the choice may be made not to link to a specific problem. So often cited arguments for not linking notes (Figure 7) can also be explained by other factors.

Relation to other studies

We are not aware of any other study that has analyzed the effect of patient- and doctor-related factors on linking notes. According to Chowdhry et al.,⁴ only one study explored the usability of a fully electronic POMR in daily clinical practice. Important reasons that are mentioned in this study from Bossen³² for not linking notes are “too complicated” and “time consuming”. Although manually applying these relationships takes time, doctors who are familiar with linking notes seems to manually link notes to problems relatively easily given the higher link ratio. In general, limitations of the POMR, such as complexity of maintenance and missing guidelines, especially multi-disciplinary, have been described by Salmon et al.³³ In our study, factors like too complicated, time consuming, and no policy are often mentioned as similar reasons not to link. Although all doctors are trained before and after implementation, usability and functionality remain a point for improvement. A possible approach for improvement is the use of problem-oriented templates that are associated with increased note quality without an increase of total charting time.³⁴

Linking notes to problems is often mentioned in articles, but these are often opinions or expectations. There are several articles in which alternatives have been suggested where the linking of data to problems can take place automatically because of the integration and application of medical knowledge.^35–37 Suggestions for linking data, or the automatic linking of data in the case of evidenced-based relationships, can also give an improvement.

Although the problem list is a fundamental item of patient-centered care, different clinicians often have different ways of working; therefore, significant variation is observed in the use of the problem list.^10,38–40 “Overview” was the most frequently reported reason to link progress notes to problems, which is also an important success factor mentioned in other studies.^11,12 Our survey shows several diagnosis-related reasons on why not to link progress notes, such as diagnosis from other doctors”, “no definitive diagnosis”, or “more than one diagnosis”. The notion of serving clinicians, patients, and other stakeholders in a broad sense, yet lacking a standard for using problem lists and for the linking of notes, is incompatible at the expense of the information quality.

Another important aspect of linking notes to problems is the human-computer interaction, particularly the usability of computer interfaces. Usability is especially important when performing complex tasks given that determining a health problem is a complex, collaborative activity that involves information gathering and clinical reasoning.⁴¹ Other research claims that improvements in EHR usability are essential in accurate recording diagnosis in EHRs.⁴² Authors recognize and endorse the results of the literature review conducted by Zahabi et al.⁴³ to usability guidelines for EHR interfaces focused on the diagnosis process and documentation. Despite existing research and further development of EHRs, improvement of usability cannot be taken for granted.⁴⁴ A recent large study of US physicians measuring the usability of the EHR systems shows that the usability is perceived as poor.⁴⁵ In a previous study from Cillessen et al.,²³ the overall usability of the clinical notes application, which contains the progress notes module for linking notes, was investigated. The results show that the overall physician usability satisfaction score was acceptable given the score of 3.2 on a scale from 1 (highly dissatisfied) to 5 (highly satisfied). Others argue that the functionality of adding relationships between different data, such as linking notes to problems, is an important factor that can further improve the usability of EHRs, increasing physician satisfaction with EHRs.^46,47

Finally, we discuss the problem list and the linking of notes in relation to the application of standard terminologies such as ICD and SNOMED CT. ICD is used primarily by professional coders working with clinical records for billing, research, statistics or other purposes. In the progress notes module, diagnoses can be entered as free text or as ICD-10 codes, but coding was rarely used. ICD is not intended or suitable for use directly by clinicians at the point of care and is not detailed enough to meet the needs of hospital specialists.⁴⁸ SNOMED CT is the most comprehensive healthcare terminology available used in electronic health record systems to facilitate clinical documentation and reporting and to retrieve and analyse clinical data.⁴⁸ Although there are organizations that have successfully implemented SNOMED CT, there is still much work ahead to bring SNOMED CT into routine clinical use.⁴⁹ Despite this work, the importance of using standards such as ICD, SNOMED, and others, cannot be understated, especially for the benefit of clinical decision support systems.⁵⁰ The increase in the use of standard terminologies, which are also becoming increasingly mandatory, may improve the quality of the problem list, which as a result, may have a positive effect on linking notes to problems.

Meaning of the study

The more complex the medical record, measured in the number of doctors, medical specialties, and notes, the more the link ratio decreases where we would actually expect the opposite. Linking notes to problems is appreciated in daily clinical practice to create an overview, but takes time, which is a frequently cited reason for not to do so, which may partly explain the low link ratio between 6 pm until 8 am. Accuracy of the problem list and belief in the added value seem to be important factors for linking notes. A minority of doctors, represented in all disciplines, link their notes very persistently. It is discouraging for them when colleagues, especially in multi-disciplinary use, affect the integrity of the problem list or do not link their notes, which in turn discourages others from linking notes and thus can deteriorate further over time. Better and regular training could increase the link ratio given the main reasons not to link notes, “I don’t know how” and “Forget.”

There are several recommendations to maximize the benefits of the linking of notes. First, policymakers must ensure a multi-disciplinary guideline for linking notes, which should at least pay attention to the content and management of the problem list and when to link which data. Second, medical leadership should emphasize the added value of linking notes. Third, doctors must be (repeatedly) trained on how to use the linking of notes. Fourth, policymakers and informaticians should optimize the desired way of working with, for example, easy to use templates or mappings. Finally, medical staff should conduct open and closed record reviews periodically for feedback.

Future research

More research is needed to define the required functionality and to improve the usability to maintain a multi-disciplinary problem list. In particular when a patient’s health condition (expressed as one of more complaints, issues, findings, diagnoses, risks, barriers, etc.) and its course are identified and monitored by healthcare provider(s) and/or the patient and patient family.¹³ This also fits with the call for a fundamental paradigm shift in the way health services are delivered, managed and funded from the World Health Organization (WHO).⁵¹ Patient access to the EHR may improve communication and increase engagement and self-management.⁵² Especially if more people or organizations are involved in the care of a patient the linking of notes is useful because it contributes to transparency. It would be useful to conduct observational studies in those circumstances where the link ratio is low.

Conclusions

Linking notes to clinical problems is seen as added value to keep an overview. Linking notes is not self-evident and varies from no use at all to extensive use. We found important arguments from doctors for and against the use of linking notes. Our key recommendations for policymakers and medical leadership to maximize the benefits of the linking of notes; ensure a multi-disciplinary policy, emphasize the added value, train doctors sufficiently and repeatedly, improve functionality and optimize usability to optimize the desired way of working, conduct record reviews periodically for feedback.

Footnotes

Acknowledgements

We want to thank all the doctors who participated in this study.

Details of contributors

FC, PdVR, and MB designed the study. FC collected the data and conducted the study. FC performed the statistical analyses with the support of HB, while PdVR, MB, and HB advised on the statistical analyses. FC drafted the manuscript, and all authors were involved in interpreting the data and revising the manuscript. All authors had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis. FC is the guarantor for this paper and affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: All authors have completed the ICMJE uniform disclosure form at . They also declare no support from any organization for the submitted work, no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years, and no other relationships or activities that could appear to have influenced the submitted work.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

License for publication statement

The Corresponding Author has the right to grant on behalf of all authors and does grant on behalf of all authors, a worldwide license to the Publishers and its licensees in perpetuity, in all forms, formats and media (whether known now or created in the future), to (i) publish, reproduce, distribute, display, and store the Contribution, (ii) translate the Contribution into other languages, create adaptations, reprints, include within collections and create summaries, extracts and/or, abstracts of the Contribution, (iii) create any other derivative work(s) based on the Contribution, (iv) to exploit all subsidiary rights in the Contribution, (v) the inclusion of electronic links from the Contribution to third party material where-ever it may be located; and, (vi) license any third party to do any or all of the above.

Ethical approval

Not applicable.

Transparency declaration

FC affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.

ORCID iD

Felix Cillessen

References

Boonstra

Versluis

Vos

JF.

Implementing electronic health records in hospitals: a systematic literature review. BMC Health Serv Res 2014; 14: 370.

Weed

LL.

Medical records that guide and teach. N Engl J Med 1968; 278: 593–600.

Weed

LL.

Medical records, medical education, and patient care; the problem-oriented record as a basic tool. Cleveland: Press of Case Western Reserve University, 1969.

Chowdhry

Mishuris

Mann

Problem-oriented charting: a review. Int J Med Inform 2017; 103: 95–102.

Wright

Pang

Feblowitz

, et al. A method and knowledge base for automated inference of patient problems from structured data in an electronic medical record. J Am Med Inform Assoc 2011; 18: 859–867.

Scichilone

RA.

Standardizing problem lists in physician practices. J AHIMA 2011; 82: 50–51.

Wright

Feblowitz

Maloney

, et al. Increasing patient engagement: patients’ responses to viewing problem lists online. Appl Clin Inform 2014; 5: 930–942.

Klappe

de Keizer

Cornet

Factors influencing problem list use in electronic health records-application of the unified theory of acceptance and use of technology. Appl Clin Inform 2020; 11: 415–426.

Holmes

The problem list beyond meaningful use. Part I: the problems with problem lists. J AHIMA 2011; 82: 30–33.

10.

Wright

Maloney

Feblowitz

JC.

Clinician attitudes toward and use of electronic problem lists: a thematic analysis. BMC Med Inform Decis Mak 2011; 11: 36–36.

11.

Simons

Cillessen

Hazelzet

JA.

Determinants of a successful problem list to support the implementation of the problem-oriented medical record according to recent literature. BMC Med Inform Decis Mak 2016; 16: 102–102.

12.

Hodge

Narus

SP.

Electronic problem lists: a thematic analysis of a systematic literature review to identify aspects critical to success. J Am Med Inform Assoc 2018; 25: 603–613.

13.

Health Level Seven International. International patient summary, http://international-patient-summary.net/mediawiki/index.php?title=Main_Page (2018, accessed 22 November 2019).

14.

Schiff

Bates

DW.

Can electronic clinical documentation help prevent diagnostic errors?

N Engl J Med 2010; 362: 1066–1069.

15.

Bayegan

The helpful patient record system: problem oriented and knowledge based. Proc AMIA Symp 2002: 36–40.

16.

Bossen

Evaluation of a computerized problem-oriented medical record in a hospital department: does it support daily clinical practice?

Int J Med Inform 2007; 76: 592–600.

17.

Zhou

Zheng

Ackerman

, et al. Cooperative documentation: the patient problem list as a nexus in electronic health records. In: Proceedings of the ACM 2012 conference on computer supported cooperative work, Seattle, WA, 11–15 February 2012, pp.911–920. New York, NY: ACM.

18.

Collins

Tsivkin

Hongsermeier

, et al. A continuum of sociotechnical requirements for patient-centered problem lists. Stud Health Technol Inform 2013; 192: 889–893.

19.

Mehta

Vakharia

Wright

EHRs in a web 2.0 world: time to embrace a problem-list Wiki. J Gen Intern Med 2014; 29: 434–436.

20.

Cillessen

Hofdijk

Transition requirements from problem list to an overarching care plan for the support of person-centered care. Stud Health Technol Inform 2020; 272: 292–295.

21.

Porter

ME.

What is value in health care?

N Engl J Med 2010; 363: 2477–2481.

22.

Cillessen

de Vries Robbé

PF.

Modeling problem-oriented clinical notes. Methods Inf Med 2012; 51: 507–515.

23.

Cillessen

de Vries Robbé

Biermans

MC.

A hospital-wide transition from paper to digital problem-oriented clinical notes. Appl Clin Inform 2017; 8: 502–514.

24.

Wright

Sittig

McGowan

, et al. Bringing science to medicine: an interview with Larry Weed, inventor of the problem-oriented medical record. J Am Med Inform Assoc 2014; 21: 964–968.

25.

Boonstra

Broekhuis

Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Serv Res 2010; 10: 231.

26.

Friedberg

Chen

Van Busum

, et al. Factors affecting physician professional satisfaction and their implications for patient care, health systems, and health policy. Rand Health Q 2014; 3: 1.

27.

Kroth

Morioka-Douglas

Veres

, et al. Association of electronic health record design and use factors with clinician stress and burnout. JAMA Netw Open 2019; 2: e199609.

28.

West

Dyrbye

Shanafelt

TD.

Physician burnout: contributors, consequences and solutions. J Intern Med 2018; 283: 516–529.

29.

Hasan

Bahalkeh

Yih

Evaluating intensive care unit admission and discharge policies using a discrete event simulation model. Simulation 2020; 96: 501–518.

30.

Clemens

Gottlieb

JD.

Do physicians’ financial incentives affect medical treatment and patient health?

Am Econ Rev 2014; 104: 1320–1349.

31.

Hellinger

FJ.

The impact of financial incentives on physician behavior in managed care plans: a review of the evidence. Med Care Res Rev 1996; 53: 294–314.

32.

Bossen

Accounting and co-constructing: the development of a standard for electronic health records. Comput Support Coop Work 2011; 20: 473.

33.

Salmon

Rappaport

Bainbridge

, et al. Taking the problem oriented medical record forward. Proc AMIA Annu Fall Symp 1996: 463–467.

34.

Mehta

Radhakrishnan

Warring

, et al. The Use of evidence-based, problem-oriented templates as a clinical decision support in an inpatient electronic health record system. Appl Clin Inform 2016; 7: 790–802.

35.

Weed

LL.

New connections between medical knowledge and patient care. BMJ 1997; 315: 231–235.

36.

Zeng

Cimino

Zou

KH.

Providing concept-oriented views for clinical data using a knowledge-based system: an evaluation. J Am Med Inform Assoc 2002; 9: 294–305.

37.

Buchanan

Accelerating the benefits of the problem oriented medical record. Appl Clin Inform 2017; 8: 180–190.

38.

Ancker

Kern

Edwards

, et al. How is the electronic health record being used? Use of EHR data to assess physician-level variability in technology use. J Am Med Inform Assoc 2014; 21: 1001–1008.

39.

Shea

Malone

Weinberger

, et al. Assessing organizational capacity for achieving meaningful use of electronic health records. Health Care Manage Rev 2014; 39: 124–133.

40.

Martin

Sbaffi

Electronic health record and problem lists in Leeds, United Kingdom: variability of general practitioners’ views. Health Informatics J 2020; 26: 1898–1911.

41.

National Academies of Sciences, Engineering, and Medicine. Improving diagnosis in health care. Washington, DC: The National Academies Press, 2015.

42.

Shah

Quinn

Chaudhry

, et al. Recording problems and diagnoses in clinical care: developing guidance for healthcare professionals and system designers. BMJ Health Care Inform 2019; 26: e100106.

43.

Zahabi

Kaber

Swangnetr

Usability and safety in electronic medical records interface design: a review of recent literature and guideline formulation. Hum Factors 2015; 57: 805–834.

44.

Kaipio

Hyppönen

Lääveri

Physicians’ experiences on EHR usability: a time series from 2010, 2014 and 2017. Stud Health Technol Inform 2019; 257: 194–199.

45.

Melnick

Dyrbye

Sinsky

, et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020; 95: 476–487.

46.

Sutton

Ash

Al Makki

, et al. A daily hospital progress note that increases physician usability of the electronic health record by facilitating a problem-oriented approach to the patient and reducing physician clerical burden. Perm J 2019; 23: 18–221.

47.

Cimino

JJ.

Putting the “why” in “EHR”: capturing and coding clinical cognition. J Am Med Inform Assoc 2019; 26: 1379–1384.

48.

Benson

Principles of health Interoperability: SNOMED CT, HL7 and FHIR. 3rd ed. London: Springer, 2016, p.451.

49.

Lee

Cornet

Lau

, et al. A survey of SNOMED CT implementations. J Biomed Inform 2013; 46: 87–96.

50.

Sutton

Pincock

Baumgart

, et al. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020; 3: 17.

51.

WHO. Global strategy on people-centred and integrated health services, https://www.who.int/servicedeliverysafety/areas/people-centred-care/global-strategy/en/ (2015, accessed 22 November 2019).

52.

Rathert

Mittler

Banerjee

, et al. Patient-centered communication in the era of electronic health records: what does the evidence say? Patient Educ Couns 2017; 100: 50–64.