Abstract
Most of our authors and readers know this journal encourages submission of research reports that employ observational designs. By observational, we mean quantitative studies that are not randomized experiments. Retrospective studies of electronic medical records, for example, might be designed as cohort studies, case-control studies, cross-sectional studies, or interrupted time series.
Our reason for being interested in such studies is simple: They provide valuable information and they are underappreciated in medical and public health journals. The randomized experiment is the “gold standard” design because of its strong internal validity. However, the external validity of experiments often is weak, as evidenced by the discovery of unexpected side effects of new medicines that were rigorously tested. Observational studies can test what happens in real life, which is different from what happens in the artificial circumstances of a rigorous clinical trial. Ideally, both kinds of studies, observational and randomized experiments, would be performed on any given topic, to give us a more complete understanding of whether an innovation works as expected.
Observational studies should meet certain standards. These include the following:
Adequate sample size. The study must have enough power to test the hypothesis. Typically, this means having as many cases as were studied in previously published studies on the same topic. Extracting fewer cases than necessary from medical records is an amateur’s mistake. Calling the result a pilot study does not justify publication.
Control variables. A fair test of a hypothesis in a observational study attempts to control for known major causes of the outcome in question. Failure to extract such variables when requesting data is shortsighted.
Statistical methods. All statistical methods make assumptions and, typically, those assumptions are imperfectly met in any given study. Experienced readers know that findings may have been different had different methods been used. As general rules, considering the following: (a) Do it the way previously published studies did it, and cite them. (b) Include control variables—usually this means multiple linear or logistic regression analysis is best. (c) Do not try to make up for weak data by using a complex statistical method—the best method might be the simpler and more familiar approach. (d) Break continuous variables into ordinal categories unless the situation clearly does not require it—and please do not pretend that a 5-level variable is a continuous variable. (e) Consider whether repeated-measures analysis of variance might be appropriate for your study—this design has been underutilized in public health and medicine compared to other fields.
Background literature. The literature review need not be exhaustive but capturing previously published similar studies is essential. A new study is not a contribution to knowledge if done in ignorance of the literature. Besides, similar published reports provide templates for the investigator to follow. They show which variables to include, how to measure them, and what statistical methods are standard for the research question.
Comparison group. A study may be described as “controlled” even when random assignment is not employed, if a comparison group is included. Comparison groups are plentiful; other sites or other time periods may serve to set up natural experiments. The single group, pre–post design is very weak and likely to be questioned by reviewers.
Mixed methods. Qualitative data may enrich a study dramatically. However, most studies are stronger if they include quantitative data as well, even if the quantitative results are limited to a few measures.
Multiple sites. Multisite studies are more convincing than single site studies. When multiple sites are included, stratification by site may be indicated. Published reports in medical journals frequently fail to recognize that a significant portion of the variation in performance variables across site is likely to be left unexplained by the measures available. Organizational culture, staff attitudes, patient perspectives—all might be important. Just because an association between 2 variables is found in one site does not mean it will hold true in others. Multisite studies are more convincing, for good reasons.
Provider surveys. Providers seem to be fascinated with provider surveys. These can be useful adjuncts to stronger evidence but often do not stand alone. For example, asking providers if they intend to change their practices after receiving an educational intervention is much less persuasive than demonstrating actual change with pre and post-intervention data.
Figures. Figures frequently are forgotten by authors. This is surprising since most will prepare figures summarizing their research for public presentations. However, it is apparent to editors that many authors lack experience in preparing figures and are not very good at designing them. Here, again, is where following the examples from the literature review will be helpful.
Writing quality. Most authors, myself included, benefit from using a software program to check for errors. Authors for whom English is a second language (ESL) also need this kind of assistance. Most ESL authors have employed someone for writing assistance, but, unfortunately, quite often that person is not as skilled as he or she thinks he or she is. Poor writing usually results in rejection prior to review. (Tip: After making corrections as suggested by the software, run the paper back through to see if the corrections were adequate. Repeat as necessary.)
Adhering to the 10 standards described above should help investigators who are planning an observational study in primary care or community health. Failure to do so may prevent the paper from being published anywhere. Publication in this journal will most definitely be affected.