Current use of routinely collected health data to complement randomized controlled trials: a meta-epidemiological survey

Lars G. Hemkens; Despina G. Contopoulos-Ioannidis; John P.A. Ioannidis

doi:10.9778/cmajo.20150036

Article Figures & Tables

Figures

Tables

Download figure
Open in new tab
Download powerpoint
Figure 1
Selection of RCD studies for the analysis. RCD = routinely collected health data.
Download figure
Open in new tab
Download powerpoint
Figure 2
The most frequent motivations for performing observational studies using routinely collected health data (RCD studies) claimed by authors who were aware of existing RCT evidence. Each circular area corresponds to one research motivation category; numbers of studies with multiple motivations are depicted in the corresponding overlapping areas (e.g., 11 RCD studies [5% of 213] reported both the assessment of a specific patient population and the study of "real world" effects as motivations). The diagram is schematic; percentages do not correspond to the size of circular areas.

Tables

Figures

View popup

Table 1: Characteristics of studies using routinely collected health data (RCD studies)

Characteristic	No. (%) of RCD studies*			p value
Characteristic	All studies n = 337	Studies with existing RCTs† n = 231	Studies without existing RCTs n = 106	p value
Publication year, median (IQR)	2008 (2006-2009)	2008 (2007-2009)	2008 (2005-2009)	0.4
Type of condition/disease/specialty				0.06
Chronic kidney disease	10 (3.0)	4 (1.7)	6 (5.7)
Cardiovascular disease	213 (63.2)	155 (67.1)	58 (54.7)
Cancer	39 (11.6)	26 (11.3)	13 (12.3)
Diabetes mellitus	8 (2.4)	6 (2.6)	2 (1.9)
Pediatrics	3 (0.9)	3 (1.3)	0
Pregnancy	1 (0.3)	1 (0.4)	0
Psychiatry	11 (3.3)	8 (3.5)	3 (2.8)
Pulmonology	10 (3.0)	6 (2.6)	4 (3.8)
Surgery	10 (3.0)	4 (1.7)	6 (5.7)
Transplantation	15 (4.5)	6 (2.6)	9 (8.5)
Other	17 (5.0)	12 (5.2)	5 (4.7)
Type of treatment				< 0.001
Coronary revascularization	94 (27.9)	74 (32.0)	20 (18.9)
Devices	9 (2.7)	3 (1.3)	6 (5.7)
Drugs	162 (48.1)	124 (53.7)	38 (35.8)
Radiation	10 (3.0)	5 (2.2)	5 (4.7)
Surgery‡	35 (10.4)	15 (6.5)	20 (18.9)
Different types§	8 (2.4)	3 (1.3)	5 (4.7)
Other	19 (5.6)	7 (3.0)	12 (11.3)
Type of comparator				0.2
Active intervention	175 (51.9)	125 (54.1)	50 (47.2)
Usual care	162 (48.1)	106 (45.9)	56 (52.8)
Type of routine data				0.6
Registry data	217 (64.4)	147 (63.6)	70 (66.0)
Administrative data	46 (13.6)	34 (14.7)	12 (11.3)
EMR/EHR	7 (2.1)	6 (2.6)	1 (0.9)
Other	67 (19.9)	44 (19.0)	23 (21.7)

Note: EHR = electronic health record, EMR = electronic medical record, IQR = interquartile range, RCT = randomized controlled trial.

*Unless stated otherwise.

†Studies in which RCTs (or meta-analyses or systematic reviews of RCTs) on the same research question were mentioned or cited, or cases where at least 1 previous RCT was identified in the literature searches.

‡Excludes coronary artery bypass graft surgery, which is included under coronary revascularization.

§Comparison of different types of interventions (e.g., drug therapy v. radiation).

View popup

Table 2: Motivation for research efforts reported by authors aware of existing RCT evidence

RCT-related motivation	No. (%) studies
RCT-related motivation	Studies with only 1 RCT-related motivation n = 125	All studies* n = 213
Assess effects in the "real world"	41 (32.8)	80 (37.6)
Assess different outcomes	34 (27.2)	68 (31.9)
RCT evidence inconclusive or inconsistent	23 (18.4)	55 (25.8)
Compared with randomized evidence	6 (4.8)	15 (7.0)
Compared with randomized and nonrandomized evidence	17 (13.6)	40 (18.8)
Study specific patient population	20 (16.0)	50 (23.5)
Other	7 (5.6)	21 (9.9)
None	-	19 (8.9)

Note: RCT = randomized control trial.

*For study details, see Appendix 4 (available at www.cmajopen.ca/content/4/2/E132/suppl/DC1).

View popup

Table 3: Scientific impact of studies using routinely collected health data (RCD studies)

Variable	Citation impact metric; median (IQR)
Variable	Journal impact factor in 2012	No. of citations per RCD study per year	Total no. of citations per RCD study
All studies (n = 337)*	4.5 (3.2-11)	4.0 (1.5-8.0)	22 (9-55)
RCT evidence mentioned or cited
No (n = 124)*	4.5 (3.5-11)	4.6 (2.5-9.3)	29 (13-61)
Yes (n = 213)*	4.5 (3.2-11)	3.0 (1.3-7.5)	19 (8-49)
p value	0.2	0.01	0.01
RCTs existed before publication
No (n = 106)*	4.5 (3.5-11)	4.2 (2.3-8)	26 (12-58)
Yes (n = 231)*	4.5 (3.2-11)	3.5 (1.3-7.9)	20 (8-53)
p value	0.2	0.1	0.09
Studies with RCT-related research motivation (n = 213)†
Assess effects in the "real world"
Yes	3.9 (2.9-6.5)	1.9 (0.95-5.2)	11 (5-36)
No	4.7 (3.2-14)	4.4 (1.9-9.6)	24 (10-60)
p value	0.08	< 0.001	0.002
Assess different outcomes
Yes	5.0 (3.5-14)	4.7 (2-13)	24 (10-76)
No	4.1 (3.1-9.1)	2.3 (1.3-6)	17 (7-36)
p value	0.1	0.01	0.06
RCT evidence inconclusive or inconsistent
Yes	4.8 (2.9-9.1)	3.0 (1.3-6.6)	14 (6-38)
No	4.1 (3.2-14)	3.1 (1.3-8)	19 (8-54)
p value	0.9	0.7	0.3
Study specific patient population
Yes	5.0 (3.2-14)	2.3 (1.3-5.4)	18 (7-38)
No	4.5 (3.2-11)	3.1 (1.3-8)	19 (8-52)
p value	0.7	0.4	0.4
Other
Yes	4.5 (3.6-13)	3.7 (1.6-9.2)	22 (9-55)
No	4.5 (3.2-11)	2.8 (1.3-7.3)	18 (8-47)
p value	0.2	0.4	0.4
No RCT-related motivation
Yes	3.5 (3.2-6.2)	5.0 (2.2-11)	29 (18-48)
No	4.5 (3.2-14)	2.9 (1.2-7.3)	18 (7-52)
p value	0.6	0.09	0.06