Introduction
Reporting quality in clinical research is critical for evidence-based medicine and the reproducibility of studies. Previous work has mostly focused on the reporting quality of clinical trials, including those in neuro-oncology, urology, cardiology, nephrology, pharmaceutics, and infectious diseases.1–7 Although the quality of clinical trials has been improving,8,9 that of observational longitudinal studies remains low.10–12 However, few studies have addressed the reporting quality of trend analyses.
Trend analyses are critical for assessing changes in, and predicting the future of, epidemiological parameters.13,14 Recent trend-analysis guidelines recommend reporting slopes or beta/coefficients whenever possible.15 Additionally, the American Statistical Association and others recommend reporting effect sizes.16,17 Nevertheless, the reporting of these recommended statistical metrics in trend analyses remains largely unclear. Therefore, we examined the reporting quality of trend analyses in leading medicine and oncology journals, using the reporting of p-values, effect sizes, or beta/coefficients/annual percent change (APC) as the quality metrics. We also identified factors associated with reporting quality in these trend analyses.
Materials and methods
We systematically searched PubMed for the articles published over an 11-year period, from January 1, 2008, to December 31, 2018, whose titles included “trend” or “trends” in the following medicine and oncology journals: Ann Intern Med, Ann Oncol, BMJ, J Clin Oncol, J Natl Cancer Inst, JAMA Oncol, JAMA, Lancet, Lancet Oncol, and N Engl J Med. We considered including articles published after 2019; however, surges in publication numbers during and immediately after the COVID-19 pandemic (also known as SARS-CoV-2) could have influenced reporting quality.18,19 Therefore, we focused on the 11-year period from 2008 to 2018.
We included only original articles, research letters, and meta-analyses/systematic reviews that performed trend analyses. To ensure that articles focused primarily on trend analysis, we limited our search to title words, acknowledging that this approach might miss some relevant studies—a limitation we discuss. Three authors independently reviewed the full texts using a standardized data extraction form, recording publication year, journal specialty (medicine/oncology), model type, reporting of p-values, effect sizes (defined as quartiles/confidence/credible/uncertainty intervals), beta/coefficient/slope/APC, senior author location, and the presence of any authors affiliated with the School of Public Health, statistics department, or epidemiology department. Discrepancies were resolved through discussion, and in rare cases where consensus could not be reached, Dr. Zhang made the final decision.
According to guidelines,15 beta, coefficients, slopes, or APC should be reported in (piecewise) linear models. We assessed whether these metrics were reported in articles using linear models. Reporting quality was scored by assigning 1 point for reporting a p-value or effect size and another point for reporting a beta, coefficient, slope, or APC. For articles reporting the same analysis, each article was assessed independently. The sum of an article’s points represented its reporting-quality score, with a maximum of 2 points. Points were unweighted because the metrics cover different statistical aspects. To our knowledge, clinical significance among the various metrics has not been differentiated or compared in existing recommendations,15–17 although all metrics are clinically important and useful.
We used Chi-square tests, Fisher’s exact tests, and (ordinal) logistic regression to examine potential associations (Stata, version 15). Only factors with p < 0.10 in univariate analyses were included in multivariable logistic regression models. Two-sided p-values were reported, and statistical significance was defined as p < 0.05.
Results and discussion
Among the 398 identified reports of trend analyses published between 2008 and 2018, 297 met our inclusion criteria (Fig. S1). These included 38 (12.8%) analyses using non-parametric models, 226 (76.1%) using (piecewise) linear models, 32 (10.8%) using non-linear parametric models, and one (0.3%) using a semi-parametric model (Cox regression). Among these analyses, 193 (66.0%) and 216 (72.7%) reported p-values and effect sizes, respectively. Subgroup analyses showed that U.S.-based senior authors were more likely to report p-values or effect sizes than non-U.S. senior authors (Fig. 1), while reporting of these parameters was not associated with any other factors.
Among the 226 trend analyses using (piecewise) linear models (Table 1), 169 (74.8%) reported p-values, 183 (81.0%) reported effect sizes, 94 (41.6%) reported APC, and 34 (15.0%) reported beta coefficients/slopes. No multiple articles reported the same or similar analysis. Only 13 (5.8%) analyses reported neither p-values/effect sizes nor beta coefficients/slopes/APC. Ordinal logistic regression showed that author affiliation with the School of Public Health was associated with higher reporting-quality scores (odds ratio = 7.44, 95% confidence interval: 3.22–31.17). In multivariable regression models (Table 2), author affiliation with an epidemiology or statistics department was associated with reporting effect sizes, and U.S.-based senior authors (versus non-U.S.) were more likely to report p-values. No factors were independently associated with reporting APC (Table 2).
Table 1Reporting quality of trend analyses with linear models published in leading medicine and oncology journals, 2008–2018
| Subgroup | Reporting p-value | Proportion of reporting quality metrics, n/total by strata (%)
| Score 0 | Score 1 | Score 2 | p-value* |
|---|
| p-value | Reporting effect size | p-value | Reporting APC | p-value | Reporting Beta/coefficient | p-value |
|---|
| Overall | 169/226 (74.8) | | 183/226 (81.0) | | 94/226 (41.6) | | 34/226 (15.0) | 13/226 (5.8) | 119/226 (52.7) | 94/226 (41.6) | |
| Publication year | 0.352 | | 0.203 | | 0.082 | | 0.823 | | | | 0.801 |
| 2008–2011 | 41/56 (73.2) | | 41/56 (73.2) | | 30/56 (53.6) | | 8/56 (14.3) | | 2/56 (3.6) | 28/56 (50.0) | 26/56 (46.4) | |
| 2012–2015 | 71/89 (79.8) | | 73/89 (82.0) | | 31/89 (34.8) | | 15/89 (16.9) | 5/89 (5.6) | 50/89 (56.2) | 34/89 (38.2) | |
| 2016–2018 | 57/81 (70.4) | | 69/81 (85.2) | | 33/81 (40.7) | | 11/81 (13.6) | 6/81 (7.4) | 41/81 (50.6) | 34/81 (42.0) | |
| Journal specialty | 0.724 | | 0.326 | | 0.025 | | 0.911 | | | | 0.108 |
| Medicine | 114/151 (75.5) | | 125/151 (82.8) | | 55/151 (36.4) | | 23/151 (15.2) | 12/151 (7.9) | 80/151 (53.0) | 59/151 (39.1) | |
| Oncology | 55/75 (73.3) | | 58/75 (77.3) | | 39/75 (52.0) | | 11/75 (14.7) | 1/75 (1.3) | 39/75 (52.0) | 35/75 (46.7) | |
| Senior-author location | 0.012 | | 0.016 | | 0.058 | | 0.428 | | | | 0.827 |
| Non-U.S. | 52/80 (65.0) | | 58/80 (72.5) | | 40/80 (50.0) | | 10/80 (12.5) | 5/80 (6.3) | 40/80 (50.0) | 35/80 (43.8) | |
| U.S. | 117/146 (80.1) | | 125/146 (85.6) | | 54/146 (37.0) | | 24/146 (16.4) | 8/146 (5.5) | 79/146 (54.1) | 59/146 (40.4) |
| Any author in School of Public Health | 0.89 | | 0.682 | | 0.064 | | 0.217 | | | | 0.004 |
| No | 105/141 (74.5) | | 113/141 (80.1) | | 52/141 (36.9) | | 18/141 (12.8) | 7/141 (5.0) | 86/141 (61.0) | 48/141 (34.0) |
| Yes | 64/85 (75.3) | | 70/85 (82.4) | | 42/85 (49.4) | | 16/85 (18.8) | 6/85 (7.1) | 33/85 (38.8) | 46/85 (54.1) | |
| Any author in epidemiology department | 0.163 | | 0.028 | | 0.015 | | 0.497 | | | | 0.684 |
| No | 115/148 (77.7) | | 126/148 (85.1) | | 53/148 (35.8) | | 24/148 (16.2) | 8/148 (5.4) | 81/148 (54.7) | 59/148 (39.9) |
| Yes | 54/78 (69.2) | | 57/78 (73.1) | | 41/78 (52.6) | | 10/78 (12.8) | 5/78 (6.4) | 38/78 (48.7) | 35/78 (44.9) | |
| Any author in statistics department | 0.044 | | 0.089 | | 0.151 | | 0.413 | | | | 0.226 |
| No | 123/172 (71.5) | | 135/172 (78.5) | | 67/172 (39.0) | | 24/172 (14.0) | 11/172 (6.4) | 95/172 (55.2) | 66/172 (38.4) | |
| Yes | 46/54 (85.2) | | 48/54 (88.9) | | 27/54 (50.0) | | 10/54 (18.5) | 2/54 (3.7) | 24/54 (44.4) | 28/54 (51.9) | |
Table 2Multivariable models showing factors associated with reporting quality metrics of trend analyses with linear models published in leading medicine and oncology journals, 2008–2018
| Subgroup | Reporting p-value
| Reporting effect size
| Reporting APC
| p-value |
|---|
| OR (95% CI) | p-value | OR (95% CI) | p-value | OR (95% CI) |
|---|
| Publication year | | | | | |
| 2008–2011 | | | | reference | |
| 2012–2015 | | | | 0.51 (0.25 - 1.05) | 0.066 |
| 2016–2018 | | | | 0.72 (0.35 - 1.51) | 0.390 |
| Journal specialty | | | | | |
| Medicine | | | | reference | |
| Oncology | | | | 1.77 (0.97 - 3.22) | 0.063 |
| Senior author affiliation | | | | |
| Non-U.S. | reference | | reference | | reference | |
| U.S. | 2.17 (1.17 - 4.03) | 0.014 | 1.93 (0.96 - 3.88) | 0.065 | 0.80 (0.44 - 1.47) | 0.475 |
| Any author in School of Public Health | | | |
| No | | | | | reference | |
| Yes | | | | | 1.67 (0.92 - 3.02) | 0.090 |
| Any author in epidemiology department | | | |
| No | | | reference | | reference | |
| Yes | | | 0.38 (0.18 - 0.80) | 0.011 | 1.62 (0.90 - 2.91) | 0.108 |
| Any author in statistics department | | | |
| No | reference | | reference | | | |
| Yes | 2.29 (1.00 - 5.24) | 0.051 | 3.28 (1.21 - 8.90) | 0.020 | | |
Overall, the reporting quality of the included trend analyses was moderate to good, consistent with the reported increasing quality in clinical trials.9 However, several reporting-quality issues remain concerning. Reporting of p-values or effect sizes did not change over the publication years, despite recommendations advocating the use of effect sizes.16,17 The reasons underlying this are worth further investigation.
Non-U.S. senior authors reported p-values and effect sizes less frequently than their U.S. counterparts, highlighting a need for additional research and training. Furthermore, more than half of the trend analyses using linear models did not report p-values/effect sizes, slopes/beta/APC, or either, which is inconsistent with existing recommendations.15 Without these metrics, quantification and comparison of linear models are difficult or impossible, reducing the scientific rigor of these studies.
We also found that involvement of a statistics department was associated with more frequent reporting of effect sizes in oncological trend analyses, whereas involvement of an epidemiology department was associated with less frequent reporting. This may reflect the rigorous statistical training provided in statistics departments. Indeed, the participation of biostatisticians or epidemiologists has been associated with higher methodological quality,7 higher acceptance rates,20 and shorter times to publication. Accordingly, it has been recommended to include statisticians in more clinical research and trials.10,21,22
The paradoxical finding that involvement of an epidemiology department was associated with less effect-size reporting (odds ratio = 0.38, 95% confidence interval: 0.18–0.80, p = 0.011) warrants careful interpretation. One possible explanation is that the emphasis on effect-size reporting is relatively recent (2016–2019),16,17 whereas epidemiologists may historically have preferred other metrics. Given our focus solely on trend analyses, this finding may not generalize to other types of epidemiological or clinical studies, although it should be examined in future work. Notably, few studies on reporting quality of clinical trials or longitudinal studies have distinguished between biostatisticians and epidemiologists.7,20 Future research is warranted to fill this knowledge gap.
Few studies focus on the reporting quality of trend analyses, whereas many studies have examined that of clinical trials or epidemiological studies.7,9,11,12,23 Some experiences and data from clinical trials and trend analyses may be mutually informative. First, the development and implementation of reporting guidelines appear to improve the reporting quality of randomized clinical trials.23 However, no official reporting guidelines exist for trend analyses, which may be needed. We therefore recommend developing and implementing formal reporting guidelines for trend analyses, beyond current recommendations.15 Second, while librarians and information specialists did not significantly impact the reporting quality of systematic reviews, they have contributed to the journal review process.24 It would be interesting to explore whether their involvement could similarly improve reporting quality in trend analyses. Third, our findings highlight the need for journal editors and peer reviewers to more rigorously enforce reporting standards and improve reporting quality. Indeed, one recommendation has already been published by a leading medical journal.17 Publishers and professional societies could also play a more active role in enforcing reporting standards for trend analyses. Finally, our data suggest that increased involvement of statisticians in trend analyses, particularly in oncology research, may be beneficial and could extend to medical research more broadly. It is concerning that the involvement of epidemiology departments was associated with less frequent reporting of effect sizes. Further research is needed to confirm our findings and to understand the underlying reasons. Notably, involvement of either department was not associated with higher overall reporting-quality scores.
This study has several limitations. First, our search strategy, which relied on the presence of ‘trend/trends’ in titles, may have missed relevant analyses using alternative terminology (e.g., ‘temporal changes,’ ‘secular patterns’). Second, we focused on high-impact journals, limiting the generalizability of our findings to the broader literature. Journals of intermediate or low impact may publish trend analyses of lower quality than high-impact journals, and future research is warranted to test this hypothesis. Third, we did not include trend analyses published after 2018. These recent works are particularly interesting given the surges in publication numbers during and after the COVID-19 pandemic,18,19 but a longer timeframe and more data are likely needed to reliably examine biases and impacts associated with the pandemic; otherwise, conclusions may be inaccurate or misleading. Fourth, our title-based search may have missed articles that mention trend analyses only in the abstract or main text. Although such omissions likely represent a small proportion of relevant studies, this limitation may affect the generalizability of our findings. Finally, the reporting-quality metrics we used may not be applicable to all reports. Some trends are difficult to accurately model with a single algorithm and therefore may not report all recommended metrics.
Conclusions
The reporting quality of trend analyses in leading medicine and oncology journals appears moderate and should be further improved. We call for increased research and awareness regarding reporting quality in trend analyses in oncology research and beyond. The author’s affiliation with an epidemiology department was associated with less frequent reporting of effect sizes, whereas affiliation with a statistics department was associated with more frequent reporting. Interestingly, U.S.-based senior authors (versus non-U.S.) were more likely to report p-values. Additional studies are needed to validate our findings across other types of journals and future publications.
Declarations
Funding
This work was supported by the National Cancer Institute, National Institutes of Health (grant number R37CA277812 to LZ). The funder of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.
Conflict of interest
Dr. Lanjing Zhang is a co-editor-in-chief of Exploratory Research and Hypothesis in Medicine. The authors declare no other conflicts of interest.
Authors’ contributions
Concept and design (XY, YW, LZ), acquisition, analysis, and interpretation of data (XY, FD, YW, LZ), drafting of the manuscript (XY), critical revision of the manuscript for important intellectual content (XY, FD, YW, LZ), statistical analysis (XY, FD, LZ), and supervision (LZ, YW). Dr. Lanjing Zhang and Dr. Yating Wang have full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. There was no personal assistance or providers of special reagents from sources that do not fulfill the requirements of authorship.