a b s t r a c t

Background: Visio-vestibular examination (VVE) deficits are common following pediatric concussion. Guidelines recommend assessing these deficits on all potentially concussed youth given their Diagnostic and prognostic value, however test psychometrics of the VVE in the emergency department (ED) setting are unknown. Our ob- jective was to determine the inter-rater (IRR) and test-retest reliability (TRR) of the VVE in a pediatric ED. Methods: We enrolled 155 patients (112 IRR; 43 TRR) age 6-18 years with head injury presenting to the ED of a tertiary care children’s hospital. Exams were performed by a group of 65 attending/fellow physicians, pediatri- cians, and advanced practice providers. The VVE consisted of 9 maneuvers (smooth pursuits, horizontal/vertical saccades and gaze stability, binocular convergence, left/right monocular accommodation, complex tandem gait). Cohen’s kappa was calculated for IRR and TRR for each element.

Results: For IRR, 5/9 kappas (saccades, gaze stability, monocular accommodation) were in the moderate agree- ment range (0.40 to 0.60); remaining kappas showed fair agreement. For TRR, 6/9 maneuvers (saccades, horizon- tal gaze stability, monocular accommodation, tandem gait) showed substantial agreement (0.60 to 0.80). Kappas of 7/9 elements for subjects age 15-18 showed improved IRR and TRR. Conclusions: The individual elements of the VVE show fair to moderate agreement between providers and mod- erate to substantial agreement among the same provider in the ED setting. These findings suggest a role in the VVE in evaluating concussion acutely, particularly given its previously demonstrated ability to assist in risk strat- ification of concussed youth and the importance of early diagnosis for improved outcomes.

(C) 2020

1. Introduction

Concussions are common childhood injuries. Nearly 2 million con- cussions occur per year in the pediatric and adolescent population, ac- counting for over 200,000 emergency department (ED) visits annually [1,2]. The diagnosis, however, remains difficult, particularly in the acute setting [3,4]. As the physiology of concussion is better understood, visio-vestibular deficits are recognized as key elements of its morbidity. These deficits are frequently present in concussed youth [5,6]. In the past 3 years, the Centers for Disease Control and Prevision, the American Academy of Pediatrics Council on Sports Medicine and Fit- ness, and the International Conference on Concussion in Sport have all

Abbreviations: ED, emergency department; IR, inter-rater; TR, test-retest; VOMS, vestibular/oculomotor screening; VVE, visio-vestibular examination.

* Corresponding author at: The Children’s Hospital of Philadelphia, Roberts Center for Pediatric Research, 2716 South St, Philadelphia, PA 19103, United States of America.

E-mail address: [email protected] (D.J. Corwin).

released consensus recommendations that include performing visio- vestibular testing when diagnosing concussion [7-9]. One initial battery of visio-vestibular testing, the Vestibular/Oculomotor Screening (VOMS) examination, has been validated in the pediatric population [10], however its widespread use outside of the specialty setting has not been demonstrated. A modified version of this testing, the visio- vestibular examination (VVE), has been shown to be readily adopted by non-specialty providers in both the primary care and emergency de- partment settings [11,12]. This testing is important for both diagnosis and risk stratification, as deficits are strongly associated with prolonged Concussion symptoms [13-15]. This is especially relevant for the acute care provider, as multiple prior studies have shown early evaluation of pediatric concussion by multidisciplinary specialty clinics is associated with improved Recovery times [16,17].

While some data exist showing internal consistency [18,19] and high test-retest reliability [20-22] of the VOMS examination, this data was collected exclusively among healthy athletes and by sports medi- cine providers, limiting exam generalizability. No data currently exists

https://doi.org/10.1016/j.ajem.2020.06.020

0735-6757/(C) 2020

evaluating the test psychometrics of vestibular testing in non-athletes by providers outside of the specialty setting. In order to enable all clini- cians to most accurately utilize the examination, and therefore maxi- mize their ability to accurately diagnose and prognosticate pediatric concussion, it is imperative to understand its reliability. The primary ob- jective of this study was to determine the inter-rater and test-retest re- liability of the individual elements of the VVE in children presenting to a pediatric ED with head injury. Secondary objectives included determin- ing the reliability of the overall examination findings, and evaluating how reliability differed by age and time between exams.

1. Methods
   1. Study design and patient population

This was an observational, cross-sectional study of patients age 6-18 years old presenting to a pediatric ED in a tertiary care children’s hospital with a complaint of head injury over a two-year period, from 11/1/2017-12/31/2019. As we desired to establish the psychometrics of the VVE in all patients presenting with head injury (who would re- ceive the exam in real practice), any patient with a head injury with pos- sible concussion was included (final diagnosis of concussion was recorded). Additional inclusion criteria included Glasgow Coma Scale score of 13-15 (indicative of a mild traumatic brain injury) and pres- ence of a parent or guardian (for patients age b18 years-old). Exclusion criteria included underlying neurologic disorders that may predispose a subject to an abnormal visio-vestibular examination (including cerebral palsy, benign paroxysmal positional vertigo, bilateral or unilateral ves- tibular hypofunction, strabismus, diplopia); patients who were visibly intoxicated with an illicit substance or who had positive drug toxicology testing, if performed; developmental delay precluding performance of the examination; lower extremity trauma preventing gait and balance testing; Ocular trauma preventing eye tracking testing; concern for cer- vical Spinal injury preventing vestibulo-ocular reflex and gait tasting; and non-English speaking patients or families. Subjects were screened for eligibility by trained research assistants and enrolled in a conve- nience sample based on the presence of study team members described in Section 2.2. Subjects were enrolled when research assistants were present in our ED, from 7 am until 12 am, 7 days per week. After written parental consent (or patient consent for those 18 years old) and verbal child assent (for subjects b18 years old) were obtained, a brief medical record review, as well as patient and parent interview, were completed to confirm eligibility. The study was approved by our institution’s Insti- tutional Review Board.

Data collection

Following screening, demographic data (including age, sex, concus- sion history, mechanism of injury, and any excluding chronic neurologic problems) were obtained from the patient’s electronic medical record and patient/parent interview. The visio-vestibular examination was then performed by a study investigator. There were 65 team members eligible to enroll patients, including pediatric emergency medicine at- tending physicians (31), fellow physicians (11), pediatricians practicing in the emergency department (5), and advanced practice providers

(18). All study investigators were trained by the principal investigator, a pediatric emergency medicine physician with prior experience in sports medicine and expertise in performing the examination, during a 20-minute training session that included observation of exam perfor- mance. In addition, electronic decision support was provided in the form of an electronic template in the patient’s medical record (which lists examination elements), and a video demonstration of the VVE with a listing of abnormal findings on the institution’s acute head trauma clinical pathway [23]. After initial performance of the examina- tion, a second examination was performed, either by a different pro- vider for those patients in the inter-rater reliability arm, or by the

same provider for those patients in the test-retest reliability arm. To prevent undue patient burden, only two total examinations were per- formed (the test-retest reliability arm was initiated after completion of the inter-rater reliability arm). Examinations were performed at least 15 min apart to allow any provoked symptoms to resolve. Study data were collected and managed using Research Electronic Data Cap- ture (REDCap) tools hosted at our institution [24].

The visio-vestibular examination

The visio-vestibular examination performed in this study is a modi- fied version of the validated Vestibular/Ocular Motor Screening Assess- ment (VOMS) [10], and consists of 9 total elements. At our institution, we have standardized its performance across specialties, including pro- viders in primary care, emergency medicine, and sports medicine, and have demonstrated feasibility of performing the VVE in these settings [11,12]. The examination takes 3-5 min in total to perform. Abnormal- ities on each exam element are listed below, as previously described in studies evaluating the VVE in non-concussed youth [25]. We have previously shown that this examination can be performed developmen- tally in children as young as 6 years old [25]. Providers were instructed to perform the elements in the same order for each evaluation. Exam el- ements are described in full in Table 1, and are also demonstrated in video form via the following hyperlink: https://www.youtube.com/ watch?v=lGUDZnZOieM. Exam elements include:

(1) Smooth pursuits, where the patient follows the provider’s finger as it moves horizontally, progressing more rapidly, for 5 total repeti- tions. An abnormality on this portion of testing was defined as patient-reported symptom provocation (including headache, dizziness, eye fatigue, eye pain, or nausea), jerky or jumpy eye movements, or N3 beats of nystagmus.

(2,3) Fast saccades, where the patient rapidly moves his or her eyes between the examiner’s fingers (held either shoulder-width apart for horizontal saccades or between the mid-forehead and sternal notch for vertical saccades). Abnormalities included patient-reported symp- tom provocation (including worsening headache, dizziness, eye fatigue, eye pain, or nausea) after 20 or fewer repetitions. The number of repe- titions was chosen based on data showing that a significant proportion of non-concussed children can experience symptoms after 20 repeti- tions [25].

(4, 5) Gaze stability testing (the vestibular-ocular reflex), where the patient shakes his or her head “yes” (for vertical gaze stability) or “no” (for horizontal gaze stability) while fixing his or her eyes on the provider’s finger. Abnormalities included patient-reported symptom provocation (including worsening headache, dizziness, eye fatigue, eye pain, or nausea) after 20 or fewer repetitions. The number of repe- titions was chosen based on data showing that a significant proportion of non-concussed children can experience symptoms after 20 repeti- tions [25].

(6) Near-point of convergence testing, where the patient is asked to identify the distance where small print letters (generally assessed using either a pen cap or the examiner’s identification badge) become double. An abnormal exam was defined as doubling of letters at N6 cm from the patient’s forehead [26]. Providers were asked to record dis- tance as they would in usual practice, either via a tape measure or a general estimate.

(7, 8) Left and right monocular accommodation, where the patient is asked to determine the distance where small print letters (generally assessed using either a pen cap or the examiner’s identification card) become blurry. An abnormal examination was defined utilizing Hofstetter’s formula, where the cut-off for an abnormal distance in- creases as a child ages (>=8.7 cm for a child 6 years old, >=8.9 cm for a child 7 years old, etc. up to >=11.7 cm for a child 18 years old) [27]. Pro- viders were asked to record distance as they would in usual practice, ei- ther via a tape measure or a general estimate.

Table 1

Visio-vestibular examination elements.

Exam element Description Abnormalities

• Patient follows the

movement off a vertical line extending from the crown of the subject’s head to the midline between his or her feet) during any of the four ma- neuvers [28].

In addition to an assessment of abnormality for each individual mea-

(1) Smooth pursuits

(2,3) Fast saccades

(4,5) Gaze stability testing (the vestibular-ocular reflex)

(6) Near-point of convergence testing

(7, 8) Left and right monocular accommodation

(9) Complex tandem gait

provider’s finger as it moves horizontally, progressing more rapidly, between the subject’s shoulders

• Examiner stops in the middle of the patient’s visual field
• Complete 5 total

repetitions

• Patient rapidly moves his or her eyes between the examiner’s fingers (held either shoulder-width apart for horizontal sac- cades or between the mid-forehead and sternal

notch for vertical saccades)

• Patient is instructed to stop if significant symptom provocation occurs
• Complete 20 total repeti- tions if able
• Patient shakes his or her head “yes” (for vertical gaze stability) or “no” (for horizontal gaze stability) while fixing his or her eyes on the provider’s finger
• Patient is instructed to stop if any symptom prov- ocation occurs
• Complete 20 total repeti- tions if able
• The patient is asked to identify the distance where small print letters (generally assessed using either a pen cap or the examiner’s identification badge) become double.
• Providers record distance either via a tape measure or a general estimate
• The patient is asked to determine the distance where small print letters (generally assessed using either a pen cap or the examiner’s identification card) become blurry with one eye closed
• Providers record distance either via a tape measure or a general estimate
• Patient walks both for- wards and backwards with his or her eyes open and closed for 5 steps each (generally in order of for- ward eyes open, forward eyes closed, backward eyes open, backward eyes closed)
• Patient-reported symptom provocation (including headache, dizziness, eye fatigue, eye pain, or nau- sea)
• Jerky or jumpy eye move- ments
• Greater than 3 beats of nystagmus
• Patient-reported symptom provocation (including worsening headache, dizziness, eye fatigue, eye pain, or nausea) at less than or equal to 20 repetitions
• Patient-reported symptom provocation (including worsening headache, dizziness, eye fatigue, eye pain, or nausea) at less than or equal to 20 repetitions
• Doubling of letters at

N6 cm from the patient’s forehead [24].

• Defined utilizing Hofstetter’s formula, where the cut-off for an abnormal distance increases as a child ages (>=8.7 cm for a child

6 years old, >=8.9 cm for a child 7 years old, etc. up to

>=11.7 cm for a child 18 years old)

• Errors = steps off the straight line with any of the 4 maneuvers
• Sway = raising of arms for stability or any truncal movement off a vertical line extending from the crown of the subject’s head to the midline between his or her feet during any of the 5 maneuver

sure, an overall abnormal determination was made for each subject uti- lizing the total number of abnormal elements; based on prior data from non-concussed youth, 3 or more abnormal test elements (out of the 9 total) led to a designation of an abnormal visio-vestibular test battery [25].

Statistical analysis

Demographic and baseline statistics were summarized using stan- dard descriptive statistics. Both inter-rater and test-retest reliability were assessed using Cohen’s kappa for each individual element of the examination, as well as the overall test assessment. Kappas were interpreted with the following scale: b0 = less than chance agreement; 0 = change agreement; N0-0.20 = slight agreement; N0.20-0.40 = fair agreement; N0.40-0.60 = moderate agreement; N0.60-0.80 = substan- tial agreement; and N0.80-0.99 = near-perfect agreement, 1.00 = per- fect agreement [29]. A Kappa statistic was chosen for test-retest reliability as determinations were made on a categorical scale for each individual test (rather than an intraclass correlation coefficient [ICC], which would rely on continuous data). Secondary analyses of kappas for both inter-rater and test-retest reliability were performed to deter- mine if age or time between exams affected the measurements of reli- ability. A univariate logistic regression was utilized to determine the odds of a final diagnosis of concussion based on the total number of ab- normal examination elements, reported on a continuous scale from 0 to

9. The odds ratio determined the increased likelihood of a final concus- sion diagnosis based on the presence of an additional abnormal individ- ual examination element. Additionally, unadjusted odds of a final diagnosis of concussion based on the designation of an abnormal com- posite examination (3 or more abnormal individual elements) were cal- culated. The analysis was conducted using Stata version 14.2 (StataCorp, College Station, TX).

1. Results

Overall, 155 patients were recruited, 112 for the inter-rater reliabil- ity arm and 43 for the test-retest reliability arm. A flowchart of cohort derivation is shown in Fig. 1. Demographics, including age, gender, con- cussion history, and mechanism of injury are shown in Table 2. The me- dian time from injury to initial examination was 5.2 h (interquartile range 3.3 h to 20.0 h) and the first examination was performed with 48 h of injury in 85% of our sample. The median time between examina- tions was 35 min (interquartile range 27.0 min to 50.0 min). Overall, 63% of our sample received a diagnosis of concussion by the treating physician.

For inter-rater reliability, overall, the kappas for 5 of the 9 exam ele- ments fell between 0.40 and 0.60 (Table 3). Two of the elements with kappa between 0.20 and 0.40, smooth pursuits and complex tandem gait, had higher kappas when evaluating individual sub-components of the element testing (smooth pursuits kappa increased from 0.331 to 0.563 when evaluating for symptom provocation only rather than ab- normal signs and symptom provocation together; complex tandem gait kappa increased from 0.386 to 0.569 when considering only errors or sway with forward eyes open, forward eyes closed, and backward eyes open as abnormal elements). Several of the elements (smooth pur- suits, horizontal and vertical saccades, convergence, monocular accom- modation, and complex tandem gait) had kappas N0.60 when considering only adolescents age 15-18 years old. In addition, several

(9) Complex tandem gait, assessed by having a patient walk both

forwards and backwards with his or her eyes open and closed for 5 steps each. An abnormal examination was defined as errors (steps off the straight line) or sway (raising of arms for stability or any truncal

exam elements (horizontal and vertical saccades, horizontal and verti- cal gaze stability, convergence, and right monocular accommodation) had higher kappas when only considering those exams performed b30 min apart compared to the overall sample.

Fig. 1. Flow diagram of patients screened, approached, and enrolled into the current study.

Table 2

Patient demographics.

Characteristic	Overall, N (%)	Inter-rater reliability arm, N (%)	Test-retest reliability arm, N (%)
Patients	155	112	43
Age categories 6-8 years old	22 (14.2%)	18 (16.1%)	4 (9.3%)
9-11 years old	51 (32.9%)	36 (32.1%)	15 (34.9%)
12-14 years old	47 (30.3%)	37 (33.0%)	10 (23.3%)
15-18 years old	35 (22.6%)	21 (18.8%)	14 (32.6%)
Female gender	68 (43.9%)	55 (49.1%)	13 (30.2%)
Sports mechanism (vs. non-sports)	54 (34.8%)	41 (36.6%)	13 (30.2%)
Prior Concussion history	23 (14.8%)	17 (15.2%)	6 (14.0%)
Median hours between injury and first exam (IQR)	5.2 (3.3, 20.0)	4.6 (3.0,18.0)	5.5 (4.0, 24.9)
Time between exams Median minutes between exams (IQR)	35.0 (27.0, 50.0)	35.0 (27.0, 50.0)	35.0 (25.0, 53.0)
Exams within 30 min	51 (32.9%)	36 (32.1%)	15 (34.9%)
Final diagnosis of concussion	97 (62.6%)	68 (60.7%)	29 (67.4%)
Other final diagnoses Superficial head injury	21 (13.5%)	17 (15.2%)	4 (9.3%)
Face/scalp contusion	10 (6.5%)	5 (4.5%)	5 (11.6%)
Face/scalp laceration	12 (7.7%)	9 (8.0%)	3 (7.0%)
Facial bone fracture	4 (2.6%)	3 (2.7%)	1 (2.3%)
Other	11 (7.1%)	10 (8.9%)	1 (2.3%)

IQR = interquartile range.

When considering test-retest reliability, the majority of kappas (6 of the 9 exam elements) were between 0.60 and 0.80 (Table 4). Similar to inter-rater reliability, the two lowest kappa exam elements were smooth pursuits (kappa = 0.584) and binocular convergence (kappa = 0.588). Many of the individual exam elements (including smooth pursuits, verti- cal saccades, horizontal and vertical gaze stability, convergence, and left and right monocular accommodation) showed higher kappas, several N0.80, among adolescents age 15-18 years old. Also similar to the inter- rater reliability arm, for 5 of 9 exam elements (horizontal and vertical sac- cades, horizontal and vertical gaze stability, and tandem gait), those with

exams performed within 30 min of one another showed higher kappas when compared with the overall sample.

Finally, we analyzed the number of abnormal individual examina- tion elements, stratified by final diagnosis (Table 5). When evaluating the overall examination in children with and without a final diagnosis of concussion, in a graded analysis of a unadjusted regression, a trend emerged with each additional abnormal examination element increas- ing the likelihood of a final diagnosis of concussion being assigned (odds ratio = 2.1 of a concussion diagnosis [95% confidence interval 1.6 to 2.7] for each additional abnormal element). We additionally

Table 3

Inter-rater reliability for each exam element.

Inter-rater reliability	% Agreement	Overall Kappa	Kappa age 15-18	Kappa exams within 30 min
	(N = 113)	(N = 113)	(n = 21)	(n = 37)
Smooth pursuits	74%	0.331	0.767	0.235
Smooth pursuits (symptoms only)a	94%	0.563	0.632	0.767
Horizontal saccades	78%	0.512	0.798	0.643
Vertical saccades	75%	0.472	0.806	0.634
Horizontal gaze stability	81%	0.535	0.483	0.679
Vertical gaze stability	79%	0.478	0.284	0.532
Convergence	68%	0.347	0.500	0.423
L monocular accommodation	85%	0.521	0.773	0.492
R monocular accommodation	80%	0.367	0.615	0.492
Tandem gait (including backward eyes closed)	70%	0.386	0.529	0.235
Tandem gait (excluding backward eyes closed)b	80%	0.569	0.586	0.729

Smooth Pursuits (Symptoms Only): Evaluates reliability for smooth pursuits when only considering symptom provocation as an abnormality. Tandem gait (excluding backwards eyes closed): Evaluates reliability for tandem gait when only considering errors or sway with forward eyes open, forward eyes closed, and backward eyes open as abnormalities.

^a Evaluates reliability for smooth pursuits when only considering symptom provocation as an abnormality.

^b Evaluates reliability for tandem gait when only considering errors or sway with forward eyes open, forward eyes closed, and backward eyes open as abnormalities.

Table 4

Test-retest reliability for each exam element.

Test-retest reliability	% Agreement	Overall Kappa	Kappa Age 15-18	Kappa exams within 30 min
	(n = 43)	(n = 43)	(n = 14)	(n = 15)
Smooth pursuits	83%	0.584	0.689	0.588
Smooth pursuits (symptoms only)a	87%	0.492	0.435	0.755
Horizontal saccades	88%	0.777	0.625	0.875
Vertical saccades	88%	0.763	0.857	0.866
Horizontal gaze stability	80%	0.539	0.581	0.722
Vertical gaze stability	83%	0.634	0.720	0.865
Convergence	79%	0.588	0.683	0.359
L monocular accommodation	90%	0.732	0.831	0.512
R monocular accommodation	92%	0.773	0.824	0.432
Tandem gait (including backward eyes closed)	88%	0.721	0.400	0.837
Tandem gait (excluding backward eyes closed)b	91%	0.784	0.744	1.00

Smooth Pursuits (Symptoms Only): Evaluates reliability for smooth pursuits when only considering symptom provocation as an abnormality. Tandem gait (excluding backwards eyes closed): Evaluates reliability for tandem gait when only considering errors or sway with forward eyes open, forward eyes closed, and backward eyes open as abnormalities.

^a Evaluates reliability for smooth pursuits when only considering symptom provocation as an abnormality.

^b Evaluates reliability for tandem gait when only considering errors or sway with forward eyes open, forward eyes closed, and backward eyes open as abnormalities.

Table 5

Total number of abnormal examination elements (first examination only) by final diagnosis

of the examination had fair to moderate agreement between two raters and moderate to substantial agreement among the same rater. A reliable bedside tool is of utmost importance in concussion management from

Number of abnormalities No concussion diagnosis

(N, %)

0	34 (58.6%)	19 (19.6%)
1	12 (20.7%)	10 (10.3%)
2	7 (12.1%)	8 (8.3%)
3	5 (8.6%)	13 (13.4%)
4	0 (0%)	11 (11.3%)

Concussion diagnosis (N, %)

the acute setting. Previous studies have shown this testing can improve diagnostic accuracy in both the acute and subacute timeframe [12,30]. This finding is highlighted in the current study by the increased odds of a concussion diagnosis with each abnormal individual element and significantly increased odds of a concussion diagnosis with an overall abnormal examination. As important, if not more so, is the prognostic

5	0 (0%)	10 (10.3%)	value of the exam; abnormalities on individual elements of the exami-
6	0 (0%)	9 (9.3%)	nation have been shown to correlate strongly with prolonged concus-
7	0 (0%)	9 (9.3%)	sion symptoms [5,13-15]. Allowing the acute care provider to

8 0 (0%) 5 (5.2%)

9 0 (0%) 3 (3.1%)

found that, when classifying an abnormal overall examination as 3 or more abnormal individual elements, 61.9% of those ultimately diag- nosed with a concussion had an abnormal overall examination, com- pared to 8.6% of those without a concussion diagnosis (OR 17.2, 95% CI 6.3 to 46.9).

1. Discussion

This study, evaluating reliability of the visio-vestibular examination (VVE) for concussion in an acute care setting in children and adoles- cents presenting with head injury, found that the individual elements

recognize those likely to have prolonged recoveries shortly after injury is particular salient; two recently published studies have shown that in those seen in multidisciplinary concussion clinics, initial evaluation within 1 week was associated with improved recovery times [16,17].

Previous studies have evaluated the test-retest reliability of the Ves- tibular/Oculomotor Screening (VOMS) examination, a similar test of oc- ulomotor function, in healthy athletes in the non-acute setting. The VOMS examination relies on symptom provocation with 6 maneuvers (smooth pursuits, horizontal and vertical saccades, horizontal and verti- cal gaze stability, and visual motion sensitivity), as well as a measure- ment of near-point of convergence [10]. Worts et al. evaluated 45 high school athletes (age 13-18 years old) and found levels of agreement be- tween 49% and 89% on the individual VOMS elements compared over three time points when performed among a group of 6 testers (4 ath- letic trainers, a neuropsychologist, and a student) [20]. Our overall

percent agreements for the VVE are all within the upper range of these values (79% to 92% for our 9 exam elements). Yorke et al. evaluated the performance of 105 non-concussed adolescents on the VOMS exam, with 21 completing the examination twice. They found that there was perfect agreement in symptom changes for 64% of the examinations [21]. Finally, Anderson, et al. evaluated the performance of 58 healthy athletes on the VOMS examination approximately 3 months apart, and found ICC ranging from 0.29 to 0.71 [22]. Our results (kappas of in- dividual elements ranging from 0.539 to 0.773), among a much larger group of non-sports medicine providers, show similar, if not improved, test-retest reliability when compared with these studies. Of note, none of these studies evaluated inter-rater reliability of the VOMS examination.

In evaluating inter-rater reliability, we found several sub-analyses yielded improved kappa scores: evaluating only a portion of smooth pursuits and tandem gait, and considering only teenagers age 15-18 years old; these findings have implications for further exam eval- uation. Our smooth pursuit testing evaluated both symptom provoca- tion and abnormal signs, including jerky or jumpy eye movements and N3 beats of nystagmus during testing. When we examined smooth pursuits for symptom provocation alone, the inter-rater reliability im- proved into the moderate agreement range. Similarly, our tandem gait testing showed substantial agreement on test-retest reliability, but only fair agreement on inter-rater reliability. Our complex tandem gait maneuver is unique in evaluating a patient walking forwards and back- wards with eyes open and closed. The tandem gait performed in the Sport Concussion Assessment Tool, 5th Edition (a validated sideline evaluation tool) involves a timed test of the patient walking forward with eyes open only [31]. The latter task has been shown to have high test-retest reliability, as Howell et al. found an ICC of 0.86 in a sample of 32 healthy athletes age 9-18 years old [32]. Our added components have been shown to improve sensitivity of the testing, as having the pa- tient walk backward eyes closed has the highest sensitivity of any mea- sure of gait or balance in evaluating concussion [28]. However, in the current study, we found that this element in particular reduced inter- rater reliability, as removing backward eyes closed from the battery as an abnormal finding increased inter-rater reliability from fair to moder- ate (kappa increased from 0.386 to 0.569). In our previous work evalu- ating complex tandem gait, we found a significant number of false positives (over 50% of the non-concussed sample had abnormalities) on the backward eyes closed component [28]. Therefore, the high prev- alence of abnormalities in non-concussed patients may make identify- ing a true error more difficult between two examiners.

We found kappas were higher among older adolescents than youn- ger children for the vast majority of the exam elements (7 of the 9 ele- ments for both inter-rater and test-retest reliability). While limited previous data regarding the performance of Younger children on the VOMS examination exists, Iverson et al. evaluated 387 male non- concussed athletes 8-17 years old, and found no difference in the pro- portion of non-concussed athletes with symptom provocation on the testing elements when stratified by age [33]. Contrary to these findings, in an evaluation of non-concussed children presenting to an emergency department setting and performing the VVE, more abnormalities were present in 9-11 and 12-14 year-old children than 15-18 year old chil- dren [25]. Other studies have evaluated symptom reporting in concussed children of various ages, and found internal consistency of symptom scales to be higher in teenagers; for example, Sady et al. found a higher internal consistency on the Post-Concussion Symptom Inventory among children age 13-18 when compared to children age 8-12 and children age 5-7 [33]. The adolescent examination may be more reliable given the dependence on self-report of symptom provoca- tion for most examination elements. The fact that this relationship existed for both inter-rater and test-retest reliability supports this hypothesis.

Finally, we found only fair inter-rater reliability for binocular convergence (as well as right monocular accommodation), leading to

a need for further adaption of this exam maneuver. There is a wide variation in the literature of the incidence of abnormal conver- gence when performing vestibular and oculomotor testing among non-concussed youth. In a sample of emergency depart- ment patients similar to the current study, 2% of non-concussed youth age 6-18 years old had abnormalities [25]; Vernau et al. found 11.5% of youth athletes age 6-18 years old had abnormali- ties [34]; and Iverson et al. found 22% of male athletes age 8-17 years old had an abnormal binocular convergence [35]. It has been hypothesized that examiner-related differences contrib- ute to this variability [35]. Among these studies, there is variabil- ity in how distances are assessed; some studies utilize a single letter on an oversized tongue depressor, with distances measured using a tape measure [21], whereas others use an accommodative ruler with sliding card [25]. In this study, we encouraged pro- viders to measure as they would in real life practice, and many re- ported they would estimate distances rather than measure using a ruler or tape measure. Previous literature has shown that emer- gency department personnel are often inaccurate when estimating Physical exam findings, such as laceration length [36]. Taken to- gether, these results suggest that a more standardized approach to evaluating binocular convergence as part of vestibular and ocu- lomotor testing is needed.

There are several limitations to this study. Our study was performed in a single center, an academic tertiary care institution. As such, this may lower the generalizability of our results. In our center, all providers re- ceived electronic decision support and had access to a clinical pathway detailing the examination (in addition to specific training received for this study) [23]. As providers at an academic center, they may be more likely to incorporate new examination techniques into their prac- tice. However, we highlight the diversity of provider types and the large number of providers participating in this study. Since all our patients were enrolled from the emergency department setting, they likely rep- resent a specific cohort of concussed children. However, this sample likely includes both athletes and non-athletes given the high proportion of non-sports injuries in our sample, increasing our overall generaliz- ability when compared to studies focusing solely on athletes. The vast majority (85%) of our subjects were evaluated within 48 h of their in- jury, therefore we cannot comment on the reliability of our examination in the subacute phase of injury. However, this is representative of the population of children who seek care in the acute setting [37]. Finally, we recognize that a large number of patients evaluated for head injury during enrollment period were not approached due to missed screening or lack of availability of research assistants/study team members; as the primary objective of our study was evaluating the examiners performing the test, we do not believe this significantly biased our results.

1. Conclusions

The individual elements of visio-vestibular examination (VVE) for concussion show fair to moderate agreement between providers and moderate to substantial agreement among the same provider in the pediatric emergency department setting. Elements provoking symptoms (saccades, gaze stability) have higher reliability than those evaluating physical signs (smooth pursuits) or utilizing esti- mated measurements (binocular convergence), and overall the ex- amination was more reliable in older adolescents. These findings suggest a role for the VVE in evaluating concussion in the acute set- ting, particularly given its previously demonstrated ability to assist in risk stratification of concussed youth. Future work is needed to assess the reliability of a VVE with modifications to those elements assessing physical signs, and determine those VVE elements, along with other bedside evaluation tools, most discriminatory in diag- nosing and prognosticating youth concussion.

Presentations

Presented in abstract form at the Pediatric Academic Societies An- nual Meeting, Baltimore, MD, May 2019.

Financial disclosure

The authors have no relevant financial relationships to disclose. No honorarium, grant, or other form of payment was given to anyone to produce the manuscript.

Funding

Research reported in this publication was also supported by National Institute of Neurological Disorders and Stroke of the National Institutes of Health under award number R01NS097549. The content is solely the responsibility of the authors and does not necessarily represent the of- ficial views of the National Institutes of Health. This study was also sup- ported by an internal Nicholas Crognale Chair Divisional Research Grant from the Division of Emergency Medicine of Children’s Hospital of Philadelphia.

CRediT authorship contribution statement

Daniel J. Corwin:Conceptualization, Methodology, Investigation, Software, Data curation, Formal analysis, Writing – original draft.Kristy

B. Arbogast:Conceptualization, Investigation, Resources, Writing – re- view & editing, Supervision.Casey Swann:Methodology, Software, Data curation, Investigation, Formal analysis, Writing – review & editing.Rebecca Haber:Methodology, Software, Data curation, Investi- gation, Formal analysis, Writing – review & editing.Matthew F. Grady: Conceptualization, Investigation, Resources, Writing – review & editing.Christina L. Master:Conceptualization, Investigation, Resources, Writing – review & editing, Supervision.

Declaration of competing interest

The authors have no conflicts of interest relevant to this article to disclose.

Acknowledgements

We would like to thank the attending physicians, fellow physicians, pediatricians, and advanced practice providers of the Children’s Hospi- tal of Philadelphia Division of Emergency Medicine for their participa- tion in this study.

References

1. Bryan MA, Rowhani-Rahbar A, Comstock RD, Rivara F. Seattle sports concussion re- search collaborative. Sports- and recreation-related concussions in US youth. Pediat- rics. 2016;138:1-8.
2. Zonfrillo MR, Kim KH, Arbogast KB. Emergency department visits and head com- puted tomography utilization for concussion patients from 2006 to 2011. Acad Emerg Med. 2015;22:872-7.
3. Boutis K, Weerdenburg K, Koo E, Schneeweiss S, Zemek R. The diagnosis of concus- sion in a pediatric emergency department. J Pediatr. 2015;166:1214-20.
4. Bazarian JJ, Veazie P, Mookerjee S, Lerner EB. Accuracy of mild traumatic brain injury case ascertainment using ICD-9 codes. Acad Emerg Med. 2006;13:31-8.
5. Corwin DJ, Wiebe DJ, Zonfrillo MR, Grady MF, Robinson RL, Goodman AM, et al. Ves- tibular deficits following youth concussion. J Pediatr. 2015;166:1221-5.
6. Master CL, Scheiman M, Gallaway M, et al. Vision diagnoses are common after con-

cussion in adolescents. Clin Pediatr (Phila). 2016;53:260-7.

1. McCrory P, Meeuwisse W, Dvorak J, et al. Consensus statement on concussion in sport-the 5th international conference on concussion in sport held in Berlin, October 2016. Br J Sports Med. 2017;51:838-47.
2. Lumba-Brown A, Yeates KO, Sarmiento K. Centers for Disease Control and Preven- tion guideline on the diagnosis and management of mild traumatic brain injury among children. JAMA Pediatr. 2018;172:e182853.
3. Halstead ME, Walter KD. Moffatt K; council on sports medicine and fitness. Sport- related concussion in children and adolescents. Pediatrics. 2018;142:e20184074.
4. Mucha A, Collins MW, Elbin RJ, et al. A Brief Vestibular/Ocular Motor Screening (VOMS) assessment to evaluate concussions: preliminary findings. Am J Sports Med. 2014;42:2479-86.
5. Arbogast KB, Curry AE, Pfeiffer MR, Zonfrillo MR, Haarbauer-Krupa J, Breiding MJ. Point of health care entry for youth with concussion within a large pediatric care network. JAMA Pediatr. 2016;170:e160294.
6. Corwin DJ, Propert KJ, Zorc JJ, Zonfrillo MR, Wiebe DJ. Use of the vestibular and ocu- lomotor examination for concussion in a pediatric emergency department. Am J Emerg Med. 2019;37:1219-23.
7. Master CL, Master SR, Wiebe DJ, et al. Vision and vestibular system dysfunction pre- dicts prolonged concussion recovery in children. Clin J Sport Med. 2018;28:139-45.
8. Ellis MJ, Cordingley D, Vis S, Reimer K, Leiter J, Russell K. Vestibulo-ocular dysfunc- tion in pediatric sports-related concussion. J Neurosurg Pediatr. 2015;16:248-55.
9. Ellis MJ, Cordingley DM, Vis S, Reimer KM, Leiter J, Russell K. Clinical predictors of vestibulo-ocular dysfunction in pediatric sports-related concussion. J Neurosurg Pediatr. 2017;19:38-45.
10. Desai N, Wiebe DJ, Corwin DJ, Lockyer JE, Grady MF, Master CL. Factors affecting re- covery trajectories in pediatric female concussion. Clin J Sports Med. 2019;29:361-7.
11. Kontos AP, Jorgensen-Wagers K, Trbovich AM, et al. Association of Time since Injury to the first clinic visit with recovery following concussion. JAMA Neurol. 2020;77: 435-40.
12. Kontos AP, Sufrinko A, Elbin RJ, Puskar A, Collins MW. Reliability and associated risk factors for performance on the vestibular/ocular motor screening (VOMS) tool in healthy collegiate athletes. Am J Sports Med. 2016;44:1400-6.
13. Moran RN, Covassin T, Elbin RJ, Gould D, Nogle S. Reliability and normative reference values for the vestibular/ocular motor screening (VOMS) tool in youth athletes. Am J Sports Med. 2018;46:1475-80.
14. Worts PR, Schatz P, Burkhart SO. Test performance and test-retest reliability of the vestibular/ocular motor screening and King-Devick test in adolescent athletes dur- ing a competitive sport season. Am J Sports Med. 2018;46:2004-10.
15. Yorke AM, Smith L, Babcock M, Alsalaheen B. Validity and reliability of the vestibu- lar/ocular motor screening and associations with common concussion screening tools. Sports Health. 2017;9:174-80.
16. Anderson MN, Elbin RJ, Anderson MN, Dobbs EM, Schatz P. Baseline normative data and test-retest reliability of the vestibular/ocular motor screening (VOMS) assess- ment for high school athletes. Int J Exerc Sci Conf Proc. 2015;11:30.
17. Zonfrillo M, Nadel F, Corwin D, et al. ED pathway for evaluation/treatment of acute head trauma. The Children’s Hospital of Philadelphia Clinical Pathways. http:// www.chop.edu/clinical-pathway/head-trauma-acute-clinical-pathway.
18. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap) – a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009; 42:377-81.
19. Corwin DJ, Zonfrillo MR, Wiebe DJ, Master CL, Grady MF, Arbogast KB. Vestibular and oculomotor findings in neurologically-normal, non-concussed children. Brain Inj. 2018;32:794-9.
20. Lavrich JB. Convergence insufficiency and its current treatment. Curr Opin Ophthalmol. 2010;21:356-60.
21. Nunes AF, Monteiro PML, Ferreira FBP, Nunes AS. Convergence insufficiency and ac- commodative insufficiency in children. BMC Ophthalmol. 2019;19:58.
22. Corwin DJ, McDonald CC, Arbogast KB, et al. Clinical and device-basd metrics of gait and balance in diagnosing youth concussion. Med Si Sports Exerc. 2020;52:542-8 [Sep 12 [Epub ahead of print].
23. Viera AJ, Garrett JM. Understanding Interobserver agreement: the kappa statistic. Fam Med. 2005;37:360-3.
24. McDevitt J, Appiah-Kubi KO, Tierney R, Weight WG. Vestibular and oculomotor as- sessments may increase accuracy of subacute concussion assessment. Int J Sports Med. 2016;37:738-47.
25. Concussion in Sport Group. Sport concussion assessment tool – 5^th edition. Br J Sports Med. 2017;51:851-8.
26. Howell DR, Brilliant AN, Meehan 3rd WP. Tandem gait test-retest reliability among healthy child and adolescent athletes. J Athl Train. 2019;54:1254-9.
27. Sady MD, Vaughan CH, Gioia GA. Psychometric characteristics of the postconcussion symptom inventory in children and adolescents. Arch Clin Neuropsychol. 2014;29: 348-63.
28. Vernau BT, Grady MF, Goodman A, Wiebe DJ, Basta L, Park Y, et al. Oculomotor and neurocognitive assessment of youth ice hockey players: baseline associations and observations after concussion. Dev Neurpsychol. 2015;40:7-11.
29. Iverson GL, Cook NE, Howell DR, et al. Preseason vestibular ocular motor screening in children and adolescents. Clin J Sports Med. 2019 [Jun 19 [Epub ahead of print].
30. Bourne CL, Jenkins MA, Brewer KL. Estimation of laceration length by emergency de- partment personnel. West J Emerg Med. 2014;15:889-91.
31. Boutis K, Gravel J, Freedman SB, et al. The diagnosis of concussion in pediatric emer- gency departments: a prospective multicenter study. J Emerg Med. 2018;54:757-65.