Using physician billing claims from the Ontario Health Insurance Plan to determine individual influenza vaccination status: an updated validation study

Kevin L. Schwartz; Nathaniel Jembere; Michael A. Campitelli; Sarah A. Buchan; Hannah Chung; Jeffrey C. Kwong

doi:10.9778/cmajo.20160009

Article Figures & Tables

Figures

Tables

Download figure
Open in new tab
Download powerpoint
Figure 1
Influenza vaccine effectiveness estimates for the 2010/11 season, with 95% confidence intervals, from Kwong and associates (31) before and after adjustment for misclassification bias using the validation results from this study.

Tables

Figures

View popup

Table 1: Demographic characteristics of the study population and percentage receiving influenza vaccine within the previous 12 months

Characteristic	Patients, no. (%)n = 47 301	Patients who received vaccine, %n = 18 684
Canadian Community Health Survey cycle
2007/08	33 840 (71.5)	38.9
2009/10	13 461 (28.5)	41.1
Sex
Female	25 904 (54.8)	42.4
Male	21 397 (45.2)	36.1
Age group, yr
12-17	4071 (8.6)	24.8
18-49	20 247 (42.8)	23.6
50-64	11 815 (25.0)	44.6
≥ 65	11 168 (23.6)	68.4
Residence*
Urban	37 406 (79.1)	39.3
Rural	9802 (20.7)	40.4
Neighbourhood income quintile†
1 (lowest)	9342 (19.7)	40.3
2	9402 (19.9)	39.6
3	9467 (20.0)	39.5
4	9568 (20.2)	38.7
5 (highest)	9330 (19.7)	39.7
Has regular doctor
Yes	43 110 (91.1)	41.3
No	4191 (8.9)	21.7
Risk factors for serious influenza infections‡
Hypertension	13 826 (29.2)	61.7
Asthma	6225 (13.2)	44.0
Diabetes	4877 (10.3)	63.7
Cancer	2572 (5.4)	64.4
COPD	1596 (3.4)	70.5
Congestive heart failure	1235 (2.6)	74.0
Myocardial infarction	928 (2.0)	69.6
Chronic kidney disease	886 (1.9)	68.9
Morbid obesity§	866 (1.8)	43.7
Stroke	861 (1.8)	66.0
Immunosuppression	679 (1.4)	74.7
Pregnancy¶	512 (1.1)	18.0
Dementia	245 (0.5)	64.5
None of the above	25 924 (54.8)	27.3

Note: COPD = chronic obstructive pulmonary disease.

*Missing data on 95 patients.

†Missing data on 192 patients.

‡May add up to more than 100%; patients may have more than 1 risk factor.

§Body mass index > 40.

¶Date of delivery between Nov. 1 and June 1.

View popup

Table 2: Performance measures of Ontario Health Insurance Plan physician billing claims compared with self-reported influenza vaccination using Canadian Community Health Survey data

Characteristic	Immunity, no.				Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
Characteristic	TP	FP	FN	TN	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
Total	9303	1218	9395	27 385	49.8 (49.0-50.5)	95.7 (95.5-96.0)	88.4 (87.8-89.0)	74.5 (74.0-74.9)
Canadian Community Health Survey cycle
2007/08	6506	1023	6666	19 645	49.4 (48.5-50.2)	95.1 (94.8-95.3)	86.4 (85.6-87.2)	74.7 (74.1-75.2)
2009/10	2797	195	2729	7740	50.6 (49.3-51.9)	97.5 (97.2-97.9)	93.5 (92.6-94.4)	73.9 (73.1-74.8)
Age group, yr
12-17	205	71	803	2992	20.3 (17.9-22.8)	97.7 (97.1-98.2)	74.3 (69.1-79.4)	78.8 (77.5-80.1)
18-49	1510	275	3272	15 190	31.6 (30.3-32.9)	98.2 (98.0-98.4)	84.6 (82.9-86.3)	82.3 (81.7-82.8)
50-64	2319	263	2947	6285	44.0 (42.7-45.4)	96.0 (95.5-96.5)	89.8 (88.6-91.0)	68.1 (67.1-69.0)
≥ 65	5269	609	2373	2918	68.9 (67.9-70.0)	82.7 (81.5-84.0)	89.6 (88.9-90.4)	55.2 (53.8-56.5)
Sex
Female	5536	708	5448	14 211	50.4 (49.5-51.3)	95.3 (94.9-95.6)	88.7 (87.9-89.4)	72.3 (71.7-72.9)
Male	3767	510	3947	13 174	48.8 (47.7-49.9)	96.3 (96.0-96.6)	88.1 (87.1-89.0)	76.9 (76.3-77.6)
Residence
Urban	7474	1016	7229	21 687	50.8 (50.0-51.6)	95.5 (95.3-95.8)	88.0 (87.3-88.7)	75.0 (74.5-75.5)
Rural	1812	201	2152	5638	45.7 (44.2-47.3)	96.6 (96.1-97.0)	90.0 (88.7-91.3)	72.4 (71.4-73.4)
Has a regular doctor (patient age, yr)
Yes (< 65)	3924	585	6430	21 505	37.9 (37.0-38.8)	97.4 (97.1-97.6)	87.0 (86.0-88.0)	77.0 (76.5-77.5)
Yes (≥ 65)	5205	600	2229	2632	70.0 (69.0-71.1)	81.4 (80.1-82.8)	89.7 (88.9-90.4)	54.1 (52.7-55.5)
No (< 65)	110	24	592	2962	15.7 (13.0-18.4)	99.2 (98.9-99.5)	82.1 (75.6-88.6)	83.3 (82.1-84.6)
No (≥ 65)	64	9	144	286	30.8 (24.5-37.0)	96.9 (95.0-98.9)	87.7 (80.1-95.2)	66.5 (62.1-71.0)
Risk factors for serious influenza infections (age ≥ 65 yr)
Hypertension	4012	448	1589	1671	71.6 (70.4-72.8)	78.9 (77.1-80.6)	90.0 (89.1-90.8)	51.3 (49.5-53.0)
Asthma	651	63	270	244	70.7 (67.7-73.6)	79.5 (75.0-84.0)	91.2 (89.1-93.3)	47.5 (43.2-51.8)
Diabetes	1 40	159	544	554	71.1 (69.1-73.2)	77.7 (74.6-80.8)	89.4 (87.8-91.0)	50.5 (47.5-53.4)
Cancer	858	81	342	359	71.5 (68.9-74.1)	81.6 (78.0-85.2)	91.4 (89.6-93.2)	51.2 (47.5-54.9)
COPD	586	55	265	234	68.9 (65.7-72.0)	81.0 (76.4-85.5)	91.4 (89.3-93.6)	46.9 (42.5-51.3)
Congestive heart failure	571	51	202	178	73.9 (70.8-77.0)	77.7 (72.3-83.1)	91.8 (89.6-94.0)	46.8 (41.8-51.9)
Myocardial infarction	329	34	130	120	71.7 (67.6-75.8)	77.9 (71.4-84.5)	90.6 (87.6-93.6)	48.0 (41.8-54.2)
Chronic kidney disease	323	36	134	116	70.7 (66.5-74.9)	76.3 (69.6-83.1)	90.0 (86.9-93.1)	46.4 (40.2-52.6)
Morbid obesity*	67	< 6	< 30	45	72.0 (62.9-81.2)	90.0 (81.7-98.3)	93.1 (87.2-98.9)	63.4 (52.2-74.6)
Stroke	334	39	124	137	72.9 (68.9-77.0)	77.8 (71.7-84.0)	89.5 (86.4-92.6)	52.5 (46.4-58.5)
Immunosuppression	337	25	133	123	71.7 (67.6-75.8)	83.1 (77.1-89.1)	93.1 (90.5-95.7)	48.0 (41.9-54.2)
Dementia	107	21	37	59	74.3 (67.2-81.4)	73.8 (64.1-83.4)	83.6 (77.2-90.0)	61.5 (51.7-71.2)
No risk factors	733	95	478	856	60.5 (57.8-63.3)	90.0 (88.1-91.9)	88.5 (86.4-90.7)	64.2 (61.6-66.7)
Risk factors for serious influenza infections (age < 65 yr)
Hypertension	1460	176	1471	2999	49.8 (48.0-51.6)	94.5 (93.7-95.3)	89.2 (87.7-90.7)	67.1 (65.7-68.5)
Asthma	734	87	1084	3092	40.4 (38.1-42.6)	97.3 (96.7-97.8)	89.4 (87.3-91.5)	74.0 (72.7-75.4)
Diabetes	649	76	572	983	53.2 (50.4-56.0)	92.8 (91.3-94.4)	89.5 (87.3-91.7)	63.2 (60.8-65.6)
Cancer	215	19	243	459	46.9 (42.4-51.5)	96.0 (94.3-97.8)	91.9 (88.4-95.4)	65.4 (61.9-68.9)
COPD	152	21	122	161	55.5 (49.6-61.4)	88.5 (83.8-93.1)	87.9 (83.0-92.7)	56.9 (51.1-62.7)
Congestive heart failure	81	11	60	81	57.4 (49.3-65.6)	88.0 (81.4-94.7)	88.0 (81.4-94.7)	57.4 (49.3-65.6)
Myocardial infarction	94	10	93	118	50.3 (43.1-57.4)	92.2 (87.5-96.8)	90.4 (84.7-96.1)	55.9 (49.2-62.6)
Chronic kidney disease	81	< 6	< 75	120	52.9 (45.0-60.9)	96.8 (93.7-99.9)	95.3 (90.9-99.8)	62.5 (55.7-69.3)
Morbid obesity*	144	27	141	411	50.5 (44.7-56.3)	93.8 (91.6-96.1)	84.2 (78.7-89.7)	74.5 (70.8-78.1)
Stroke	63	< 6	< 50	113	57.3 (48.0-66.5)	96.6 (93.3-99.9)	94.0 (88.4-99.7)	70.6 (63.6-77.7)
Pregnancy†	67	13	93	565	41.9 (34.2-49.5)	97.8 (96.5-99.0)	83.8 (75.7-91.8)	85.9 (83.2-88.5)
No risk factors	1725	321	4209	17 447	29.1 (27.9-30.2)	98.2 (98.0-98.4)	84.3 (82.7-85.9)	80.6 (80.0-81.1)

Note: CI = confidence interval, COPD = chronic obstructive pulmonary disease, FN = false negative, FP = false positive, NPV = negative predictive value, PPV = positive predictive value, TN = true negative, TP = true positive.

*Body mass index > 40.

†Date of delivery between Nov. 1 and June 1.

View popup

Table 3: Performance measures of Ontario Health Insurance Plan physician billing claims compared with self-reported influenza vaccination using Canadian Community Health Survey data, restricted to patients surveyed between Feb. 1 and Aug. 31

Characteristic	Immunity, no.				Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
Characteristic	TP	FP	FN	TN	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
Total	5639	408	5744	16 017	49.5 (48.6-50.5)	97.5 (97.3-97.8)	93.2 (92.6-93.9)	73.6 (73.0-74.2)
Canadian Community Health Survey cycle
2007/08	3745	276	3831	10 414	49.4 (48.3-50.6)	97.4 (97.1-97.7)	93.1 (92.4-93.9)	73.1 (72.4-73.8)
2009/10	1894	132	1913	5603	49.8 (48.2-51.3)	97.7 (97.3-98.1)	93.5 (92.4-94.6)	74.5 (73.6-75.5)
Age group, yr
12-17	126	34	507	1745	19.9 (16.8-23.0)	98.1 (97.5-98.7)	78.8 (72.4-85.1)	77.5 (75.8-79.2)
18-49	960	112	2025	9060	32.2 (30.5-33.8)	98.8 (98.6-99.0)	89.6 (87.7-91.4)	81.7 (81.0-82.5)
50-64	1401	83	1770	3610	44.2 (42.5-45.9)	97.8 (97.3-98.2)	94.4 (93.2-95.6)	67.1 (65.8-68.4)
≥ 65	3152	179	1442	1602	68.6 (67.3-70.0)	89.9 (88.6-91.3)	94.6 (93.9-95.4)	52.6 (50.9-54.4)
Sex
Female	3345	221	3320	8226	50.2 (49.0-51.4)	97.4 (97.0-97.7)	93.8 (93.0-94.6)	71.2 (70.4-72.1)
Male	2294	187	2424	7791	48.6 (47.2-50.0)	97.7 (97.3-98.0)	92.5 (91.4-93.5)	76.3 (75.4-77.1)
Residence
Urban	4485	348	4360	12 628	50.7 (49.7-51.7)	97.3 (97.0-97.6)	92.8 (92.1-93.5)	74.3 (73.7-75.0)
Rural	1142	59	1376	3 357	45.4 (43.4-47.3)	98.3 (97.8-98.7)	95.1 (93.9-96.3)	70.9 (69.6-72.2)
Has a regular doctor (patient age, yr)
Yes (< 65)	2424	218	3941	12 705	38.1 (36.9-39.3)	98.3 (98.1-98.5)	91.7 (90.7-92.8)	76.3 (75.7-77.0)
Yes (≥ 65)	3119	176	1349	1441	69.8 (68.5-71.2)	89.1 (87.6-90.6)	94.7 (93.9-95.4)	51.6 (49.8-53.5)
No (< 65)	63	11	361	1710	14.9 (11.5-18.2)	99.4 (99.0-99.7)	85.1 (77.0-93.2)	82.6 (80.9-84.2)
No (≥ 65)	33	< 6	< 95	161	26.2 (18.5-33.9)	98.2 (96.1-100.0)	91.7 (82.6-100.0)	63.4 (57.5-69.3)
Risk factors for serious influenza infections (age ≥ 65 yr)
Hypertension	2403	121	977	896	71.0 (69.6-72.6)	88.1 (86.1-90.1)	95.2 (94.4-96.0)	47.8 (45.6-50.1)
Asthma	368	18	159	134	69.8 (65.9-73.7)	88.2 (83.0-93.3)	95.3 (93.2-97.4)	45.7 (40.0-51.4)
Diabetes	808	52	332	293	70.9 (68.2-73.5)	84.9 (81.2-88.7)	94.0 (92.4-95.5)	46.9 (43.0-50.8)
Cancer	494	19	196	186	71.6 (68.2-75.0)	90.7 (86.8-94.7)	96.3 (94.7-97.9)	48.7 (43.7-53.7)
COPD	348	11	164	128	68.0 (63.9-72.0)	92.1 (87.6-96.6)	96.9 (95.2-98.7)	43.8 (38.1-49.5)
Congestive heart failure	345	10	117	94	74.7 (70.7-78.6)	90.4 (84.7-96.1)	97.2 (95.5-98.9)	44.5 (37.8-51.3)
Myocardial infarction	204	12	78	66	72.3 (67.1-77.6)	84.6 (76.6-92.6)	94.4 (91.4-97.5)	45.8 (37.7-54.0)
Chronic kidney disease	187	10	81	69	69.8 (64.3-75.3)	87.3 (80.0-94.7)	94.9 (91.9-98.0)	46.0 (38.0-54.0)
Morbid obesity*	40	0	13	26	75.5 (63.9-87.1)	100.0 (100.0-100.0)	100.0 (100.0-100.0)	66.7 (51.9-81.5)
Stroke	191	14	79	70	70.7 (65.3-76.2)	83.3 (75.4-91.3)	93.2 (89.7-96.6)	47.0 (39.0-55.0)
Immunosuppression	194	7	84	63	69.8 (64.4-75.2)	90.0 (83.0-97.0)	96.5 (94.0-99.0)	42.9 (34.9-50.9)
Dementia	59	11	19	31	75.6 (66.1-85.2)	73.8 (60.5-87.1)	84.3 (75.8-92.8)	62.0 (48.5-75.5)
No risk factors	415	35	276	473	60.1 (56.4-63.7)	93.1 (90.9-95.3)	92.2 (89.7-94.7)	63.2 (59.7-66.6)