a b s t r a c t

Background: acute otitis media is often misdiagnosed. Pediatric trainees learn otoscopy from supervisors who cannot concurrently view the eardrum. Digital, smartphone otoscopes show promise to improve the visibility and learning due to a concurrent view by trainees and supervisors. We aimed to determine whether use of digital otoscopes improved accuracy of the ear exams between medical trainees and their supervisors, compared to using traditional otoscopes. Secondarily, we evaluated whether the use of digital otoscopes reduced the number of repeat ear examinations by supervisors, changed the trainee’s confidence in their exam findings, and led to dif- ferences in the rate of antibiotics prescribed.

Methods: This study was a randomized controlled trial comparing use of a digital otoscope to a traditional oto- scope, in a pediatric emergency department and Primary care clinic in an academic tertiary care children’s center. We used a modified validated image-based grading scale to compare accuracy of the ear exam between trainees and supervisors. Surveys documented modified OMgrade scores, frequency of supervisor exams, trainee confi- dence on a 5-point Likert scale, and Antibiotic prescriptions. inter-rater agreement of trainees and supervisors, the number of supervisor confirmatory examinations performed, trainee confidence, and antibiotic Prescription rates were evaluated.

Results: Amongst 188 children, 375 ears were examined by 85 trainees and 22 supervisors. The digital otoscope was utilized in 92 (48.9%) exams and 96 (51.1%) used the traditional otoscope. Accuracy of ear exam findings between trainees and supervisors improved by 11.2% (95% CI: 1.5, 21.8%, p = 0.033) using the CellScope Oto (74.8%, 95% CI: 67.3, 82.1%) compared to the traditional otoscope (63.5%, 95% CI: 56.7, 70.4%). Fewer repeat su- pervisor exams were performed in the digital otoscope group (27.2%) vs. the traditional otoscope group (97.9%) (p < 0.001). There was no difference in mean trainee confidence in their examination (p = 0.955) or antibiotic prescription rates when using digital versus traditional otoscopes (p = 0.071).

Conclusions: Utilization of a digital otoscope resulted in increased accuracy of the ear exam between trainees and supervisors, and fewer total number of examinations performed on a given child. Compared to a traditional oto- scope, a digital otoscope may be a more efficient and effective diagnostic tool.

(C) 2021

1. Introduction

Acute otitis media (AOM) accounts for large numbers of outpatient visits and antibiotics prescribed for children every year. By age four, 60-80% of all children have had at least one episode of AOM [1,2]. Cur- rent clinical practice guidelines recommend examination by otoscope to visualize the tympanic membrane and middle ear for diagnosis of AOM

* Corresponding author at: Johns Hopkins Children’s Center, Pediatric Emergency Department, Suite G1509, 1800 Orleans Street, Baltimore, MD 21287, United States of America.

E-mail address: [email protected] (K. Kleinman).

[3]. However, otoscopic examination of young children presents several challenges including uncooperative patients, small ear canals, potential obstruction by cerumen and clinician discomfort utilizing the otoscope which likely result in over-diagnosis and unnecessarily prescribed anti- biotics [4,5].

In Clinical training settings, misdiagnosis of AOM is more common amongst novice otoscope users (medical students and residents) than in experienced practitioners (pediatric attendings and fellows) [6-8]. As the traditional otoscope does not allow for concurrent middle ear vi- sualization, the initial trainee examination is often verified by a repeat exam from overseeing faculty. These additional examinations may cause child discomfort, child and parental anxiety, and lengthen clinical

https://doi.org/10.1016/j.ajem.2021.04.030

0735-6757/(C) 2021

visits. Digital otoscopes may provide an alternative to optimize the otoscopic exam for their ability to improve tympanic membrane visual- ization and patient satisfaction. In a teaching setting, use of digital otoscopes may also aide mastery of the ear exam as images can be replayed as video or individual frame by frame images. In practice, im- proved diagnostic accuracy of tympanic membrane pathology amongst general practitioners and improved identification of important ana- tomic landmarks by medical students have been demonstrated with digital otoscopes, while patients and parents preferred digital otoscopes in a teaching hospital setting [9-12].

Previous studies suggest similar diagnostic accuracy of AOM be- tween pediatric otolaryngologists and pediatric residents who used dig- ital otoscopes [11,13]. Importantly, these studies do not mimic standard of care in teaching hospitals, in which confirmation of trainees’ ear exams are typically performed by board-certified pediatricians and pe- diatric emergency medicine physicians. To our knowledge, there have been no published studies comparing the performance of digital otoscopes between trainees and their pediatrics/pediatric emergency medicine attending supervisors using an objective, validated scale. To fill this existing knowledge gap, we conducted the present study with the primary aim to determine whether accuracy of the ear exam be- tween trainees and supervising physicians is superior for a digital oto- scope compared to traditional using a validated scale. The secondary aims were to determine whether, compared to the traditional otoscope, digital otoscopy increased inter-rater agreement, decreased repeat con- firmatory otoscopic examinations and antibiotic prescription rates, and whether trainee confidence in examination findings was higher.

1. Methods
   1. Study design and setting

We conducted an unblinded randomized controlled trial (Clinical Trials.gov Identifier: NCT04296448) to determine whether Cellscope Oto (Cellscope Inc., San Francisco, CA), a smartphone based digital oto- scope, was superior to a traditional Welch Allyn MacroView Otoscope (Hillrom Holdings Inc., Chicago, IL) for pediatric ear examinations. The study was conducted from July 2019 to February 2020 at the Johns Hop- kins Children’s Center Pediatric Emergency Department (PED) and Harriet Lane Clinic, two urban, academic clinical settings in Baltimore, MD. This study was approved by the Johns Hopkins Medicine Institu- tional Review Board.

• 1. Study procedure

All healthcare provider subjects were trained to use a digital oto- scope, the Cellscope Oto, with an emailed video demonstrating its functionality (Supplemental Video - A), or taught in-person by the study team. The Cellscope Oto’s magnifying otoscopic lens is a periph- eral attachment that lines up over an iPhone 7’s or iPod’s (Apple Inc., Cu- pertino, Ca) built-in camera and light source, and produces a live image in the Cellscope Application.

Each week of their 2 or 4-week clinical rotations in the PED and clinic, trainees were randomized to use either a traditional otoscope or the Cellscope Oto. Trainees examined their patients’ ears with either otoscope, completed the exam survey (Supplemental Fig. A), then discussed the case with their supervisor. When the trainee used the tra- ditional otoscope, the attending performed a confirmatory examination, however the attending used their discretion to either repeat the exam- ination for verification, or rely on the Cellscope Oto video, when the trainee used the Cellscope Oto. Next, the supervisor completed the exam survey (Supplemental Fig. B). All survey data was input into a REDCap database by study investigators [14,15].

• 1. Randomization

Each week was assigned for use of either the Cellscope Oto (A) or the traditional otoscope (B) using a two-week block randomization scheme, with a sequence of A-B or B-A, using a Microsoft Excel randomization function performed by study investigators. Randomizing the sequence of the two otoscopes throughout a trainee’s rotation was done to reduce the influence of the modified validated OMgrade scale on trainees’ mas- tery of the ear exam. Cellscope Oto devices remained locked in boxes during traditional otoscope weeks. At any time, subjects were allowed to revert to traditional otoscopes, during Cellscope Oto weeks.

• 1. Study subjects, inclusion and exclusion criteria

A convenience sample of children who had an ambulatory or PED visit during the study period and who had a clinical indication for an otoscopic examination at the discretion of the healthcare trainee or su- pervisor were included in the study. Healthcare provider subjects in- cluded both trainees and supervisors. Trainees consisted of third- and fourth-year medical students on their core pediatrics clerkship or sub- internship, residents (Pediatrics, Family Medicine, and Emergency Med- icine) of all post-graduate levels, and physician assistant emergency medicine residents. Supervisors included fellows and faculty who were all American Board of Pediatrics Certified/Eligible in Pediatrics and/or Pediatric Emergency Medicine. All trainees and supervisors who had clinical shifts in the PED and primary care clinic during the study period were included. There were no patient or provider exclusion criteria. Ver- bal consent was obtained for all trainees and supervisors by study collab- orators, and subjects by trainees or supervisors prior to participation.

• 1. Outcomes

The primary outcome was overall accuracy of the ear exams between trainees and supervisors, using a modified OMgrade Scale, a validated image-based grading scale (Fig. 1) [16,17,23]. In addition to the validated OMgrade Scale, trainees could select ‘obstructed view’ (cerumen impaction, foreign body, discharge), or ‘unsure’. The second- ary outcomes included need for repeat confirmatory otoscopic exami- nation by supervisors, antibiotic prescription, providers’ self-reported confidence in their examination findings, based on a 5-point Likert Scale (1- high confidence, 5- low confidence), and the use of a tradi- tional otoscope when a digital otoscope was available.

• 1. Sample size

A sample size of 264 ear examinations was required to detect a clin- ically significant 15% difference in overall accuracy between digital and traditional otoscopy based on a two-sample test of proportions while assuming an accuracy of 67% in the traditional otoscopy group. How- ever, to account for the non-independence of multiple ear examinations (right and left) within each child, we applied a variance inflation factor (VIF) which assumed a correlation of 0.30 amongst measurements within each child, resulting in a total of 344 ears (172 children). Based on our Clinical sites‘ patient volume we planned to conduct this study for 28 weeks.

• 1. Statistical analysis

Overall accuracy of ear examinations, based on the modified OMgrade scale, between trainee and supervisor was summarized for the traditional and digital otoscope and compared using logistic regres- sion based on generalized estimating equations with robust standard errors to account for the multiple observations (ears) per child. Inter- rater agreement between trainees and supervisors for the traditional and digital otoscopes and between the two methods was evaluated using Fleiss’ kappa (?) for dependent data with squared weights for

Fig. 1. Otitis media grade scale [23].

responses [18]. Repeat confirmatory supervisor examinations was com- pared between the two groups using a Fisher exact test, while antibiotic prescriptions were compared using a Chi square test and trainee confi- dence was compared using a Student t-test with unequal variances. For all analyses, responses were pooled for weeks during which the tradi- tional or digital otoscope was available. A P value <0.05 was considered statistically significant for all comparisons. Statistical analyses were per- formed using Stata Version 14.1 (StataCorp, College Station, Tx) and R Statistical Software (Version 4.0.2, R Foundation for Statistical Comput- ing, Vienna, Austria).

1. Results

A total of 197 children and 394 total ears were examined, of which 99 children (198 ears) were examined using Cellscope Oto and 98 chil- dren (196 ears) were examined by traditional otoscope. Of these, 19 ear examinations were excluded due to incomplete data, including in- stances in which a supervisor examination was not performed when traditional otoscope was used (4 in the traditional otoscope and 15 from the Cellscope Oto), resulting in a final analytic sample of 188 pa- tients (375 ears) examinations (Fig. 2) for the primary outcome. These examinations were performed by 85 trainees and 22 supervisors. Due to incomplete survey data for the secondary outcomes 9 ear examina- tions were excluded from the number of repeat confirmatory examina- tion analysis (2 from the traditional subset and 7 from the Cellscope Oto subset) and 8 were excluded from the antibiotic prescription rate anal- ysis (2 from the traditional subset and 6 from the Cellscope Oto subset). Characteristics of the examiners are provided in Table 1.

• 1. Trainee-supervisor overall accuracy and inter-rater agreement

The overall accuracy in the otoscopic findings between trainees and supervisors was 69.0% (Table 2). There was a statistically significant

11.2% (p = 0.033) improved accuracy using the digital otoscope (74.7%; 95% CI: 67.3, 82.1) compared to the traditional otoscope

(63.5%; 95% CI: 55.3, 71.8). The inter-rater level of agreement, based on Fleiss’ ?, for all otoscopic findings was 0.59. A statistically significant (p = 0.022) higher Fleiss’ ? was observed in examinations with the Cellscope Oto (Fleiss’ ? = 0.69) compared to traditional otoscope (Fleiss’ ? = 0.40). Results of both overall and inter-rater agreement were similar when analyses were performed for either right or left ears.

• 1. Secondary outcomes
     1. Number of repeat confirmatory examinations

When using the traditional otoscope a significantly greater propor- tion of subjects [97.9% (94/96)] underwent a repeat confirmatory exam- ination by supervisors, while only 27.2% (25/92) of subjects had a repeat exam with Cellscope Oto (p < 0.001).

• • 1. Trainee confidence

The trainee mean confidence in examination findings score for the traditional otoscope was 2.24 (Standard Deviation [SD] 1.00) and 2.23 (SD 1.14) when using the Cellscope Oto (p = 0.955).

• • 1. Antibiotic prescription rate

Overall, there were 32 of 188 (17.0%) subjects that were prescribed oral antibiotics. The overall rate of antibiotic prescription using the traditional otoscope was 21.9% (21/96 examination) and 12.0% (11/92 examination) when using the Cellscope Oto (p = 0.071).

• 1. Other findings

There was no difference in the rate of unsure view being reported by trainees when using the traditional otoscope (7/192, 3.7%) or the Cellscope Oto (3/183, 1.6%) (p = 0.339). More trainees reported

Fig. 2. CONSORT Flow diagram of subject inclusion and exclusion.

obstructed view (cerumen, discharge) when using the Cellscope Oto (35/183) as compared to those who used the traditional otoscope (14/ 192) (p = 0.001). A total of 15 trainees reverted back to the traditional otoscope when they were assigned to and first attempted to examine with the Cellscope Oto (Table 3).

1. Discussion
   1. Primary outcome: agreement

This study demonstrated that there was improved inter-rater agree- ment between trainees and supervisors when using the Cellscope Oto on pediatric ear exams, as compared to the traditional otoscope. To our knowledge this is the first study of a digital otoscope comparing agreement on an objective otitis media scale between real-world clini- cal supervising attendings and trainees, the standard of care at training institutions. These data suggest that a digital otoscope, such as Cellscope Oto, can improve visualization of the pediatric ear exam by trainees in a teaching hospital.

Table 1

Baseline characteristics of trainee and supervisor study participants.

These findings are in contrast to other studies on digital otoscopes [11,13]. Mousseau et al. showed the same diagnostic ac- curacy between pediatric otolaryngologists and pediatric residents using Cellscope Oto and demonstrated that the trainee-supervisor agreement using Cellscope Oto was similar to the traditional oto- scope [13]. However, our study demonstrated a statistically signif- icant improvement in diagnostic agreement using real life standard of care supervisors in pediatric emergency department settings and a validated assessment tool. Our study showed a statistically sig- nificant 11.2% higher agreement when using the Cellscope Oto (74.7%), as compared to the traditional otoscope (63.5%), and higher Fleiss ? when using the Cellscope Oto (Fleiss’ ? = 0.69, sub- stantial agreement) as compared to traditional otoscope (Fleiss’ ?

= 0.40, fair agreement). This suggests that the digital otoscope en- ables more consistent visualization of the tympanic membrane be- tween trainees and supervisors. The reasons for this are likely multifactorial. One explanation is that the digital otoscope better enabled trainees to evaluate the imagery and synthesize an assess- ment of the tympanic membrane. Anecdotally, trainees and super- visors reported that the digital otoscope enabled review of the same image repeatedly without the need for repeat examination, more complete visualization of the tympanic membrane over the

course of a recorded video, and review of the image against the

Characteristics Number (%)

Total Providers Trainees 85

Supervisors 22

Trainee Distribution

Resident 73 (86)

Medical Student - Core Clerkship 5 (6) Physician Assistant Resident 4 (5)

Medical Student - Sub-internship 3 (3)

Residency Program

Pediatrics 47 (64)

Non-Pediatrics 26 (36)

Residency Level

PGY1 30 (41)

PGY2 22 (30)

PGY3-5 21 (29)

Number of Children (by age)

0-2 94 (48)

3-7 63 (32)

8-11 15 (7)

12-21 25 (13)

Total 197

modified OMgrade scale side-by-side instead of from memory. Analysis of the video, frame-by-frame, plus improved magnifica- tion may have improved the ability for concordant analysis by the trainee and supervisor. The contributing factors of a digital oto- scope that improve trainee-supervisor agreement warrants further study, as these may be important to incorporate into medical trainee education.

The demonstration of 63.5% accuracy of diagnosis with a traditional otoscope when combined with the existing literature of 67% in Richards et al., and other similar findings shows that there is room for improve- ment in the accuracy of diagnosis in otitis media at academic ins- titutions [7,8,11,19,20]. This disparity between trainee-supervisor agreement may contribute to inaccurate diagnoses, which is an issue that has been well described amongst many genreal practioners [21,22]. Our study, demonstrating an 11.2% improvement in accuracy, offers a possible way forward in improving academic center patient care as it pertains to otoscopic examinations, yet 74.7% still leaves room for improvement.

Table 2

Trainee-supervisor overall and inter-rater agreement.

	Combined Digital and Traditional Otoscopes (CI-95%)	Digital Otoscope (CI-95%)	Traditional Otoscope (CI-95%)	P value
Agreement (%)	69.0 (63.4-74.6)	74.7 (67.3-82.1)	63.5 (55.3-71.8)	0.033
Fleiss Kappa	0.59 (0.46-0.72)	0.69 (0.54-0.84)	0.40 (0.20-0.60)	0.022

• 1. Secondary outcome: repeat confirmatory examinations

Another important finding of this study was the significant 70% de- crease in repeat confirmatory examinations by supervisors when their trainee used the Cellscope Oto with a recorded examination. These find- ings are supported by Mousseau et al. [13]. This decrease in repeat con- firmatory examination potentially allows for fewer times that a child needs to be held down for an otoscopic examination, decreased amount of time needed to completely evaluate a patient in the academic clinical setting, and potentially an increase in the opportunity for teaching given the shared evaluation of the images/video of the otoscopic examination. If technology such as a digital otoscope expedites the pediatric exam, evaluation of the length of stay during ambulatory/ED visits in an aca- demic medical setting may be an appropriate next step.

• 1. Secondary outcome: trainee confidence

This study demonstrated no difference between the trainee confi- dence in their assessment of the tympanic membrane when using the Cellscope Oto as compared to the traditional otoscope. Interestingly, Mousseau et al. showed that trainees reported higher confidence in their ear exam when using a traditional otoscope, as compared to the Cellscope Oto [13]. These data seem to indicate that a brief tutorial on the Cellscope Oto’s basic operation, encouragement by supervisors to use, and individual repeated use, quickly results in most providers being as confident in their otoscopic findings as compared to when they use traditional otoscope. As such, there seems to be little need for detailed formal training and a change in provider behavior should be scalable, however this should be further evaluated. These data bode well for a larger scale roll out or transition to a digital otoscope in aca- demic clinical settings.

It is also interesting that trainees reported relatively high confidence in their examinations with both devices, however the inter-rater agree- ment with the traditional otoscope was rather low.

• 1. Secondary outcome: antibiotic prescription rate

Our study did not show a statistically significant difference in the rate of antibiotic prescriptions when comparing the traditional otoscope and the Cellscope Oto, however there was a lower proportion of chil- dren prescribed antibiotics when using the Cellscope Oto. While we are unable to definitively confirm any subject’s otitis media diagnosis, this potential difference may be investigated in future studies. The pos- sibility that the otoscope tool may influence Antibiotic prescribing

Table 3 Number of providers who reverted back to traditional otoscope when digital otoscope was available, stratified by reason for reversion.

Reported Reason	Number of Providers
Imperfect Clarity	3
Imperfect Lighting	4
Imperfect Visualization	4
Not having device at time of examination	2
Discomfort with device	2
Total	15

practices could have significant implications in the antibiotic prescrip- tion rates, associated side effects, and Antimicrobial stewardship.

• 1. Implementation challenges

Introduction of the Cellscope Oto into clinical practice had several implementation obstacles and likely accounted for the 15 instances where trainees or supervisors chose to use the traditional otoscope in lieu of the Cellscope Oto. The Cellscope Oto light color temperature is cooler (bluer) as compared to many common traditional otoscopes which have a warmer (more yellow) light temperature, in addition to variations in apparent brightness may have contributed to some sub- jects noting imperfect lighting when surveyed. Furthermore, intermit- tent imperfect alignment of the Cellscope Oto’s attachment bracket onto the iPhone was noted. While easily adjusted, this may have con- tributed to some subjects noting imperfect view or clarity. Overall, we felt that the rate of these issues hindering Cellscope Oto’s use was an acceptable degree of technical difficulty with the rollout of a new device, is in line with prior studies, however usability testing is essential to en- sure functionality of a new device at scale [11].

• 1. Study limitations

The generalizability of the findings to other types of digital otos- copes is unclear. As of this writing, Cellscope Oto is no longer in produc- tion by the manufacturer. Despite this, other digital otoscope devices (e.g. Welch Allyn Digital MacroView Otoscope (Hillrom Holdings Inc., Chicago, IL), the WischMed Wispr (WiscMed LLC, Middleton, WI)) offer similar features such as a magnified view and digital image/video recording and playback and may potentially perform similarly to Cellscope Oto in a teaching setting, though these devices have not yet been studied. Importantly, this study was performed at a single center academic center and may not be generalizable to all settings.

1. Conclusion

This study demonstrates that a digital otoscope increased accuracy in pediatric otoscopic findings between medical trainees and their pedi- atric supervisors in an urban, high volume, academic clinical setting. Digital otoscopes may improve trainee visualization of the eardrum, which has implications in medical education as trainees strive to master the ear exam. This study also demonstrated that use of a digital otoscope substantially decreased the number of repeat confirmatory examina- tions by the supervising physician. By reducing the frequency of ear exams, digital otoscopes may mitigate the anxiety provoking experi- ence of ear exams for pediatric patients in training institutions.

Funding

Johns Hopkins Children’s Center Innovation Grant.

Declaration of Competing Interest

Cellscope, Inc. provided a portion of the Cellscope Oto devices at no cost after they were no longer available on the market. CellScope, Inc. was not involved in study design, data collection, analysis, conclusions,

or publication decisions. None of the study contributors have any finan- cial or other interests in Cellscope, Inc.

Acknowledgement

We would like to thank Erik Douglas for assistance in acquiring the Cellscope Oto.

Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi. org/10.1016/j.ajem.2021.04.030.

References

1. Kaur R, Casey JR, Pichichero ME. Emerging Streptococcus pneumoniae strains colo- nizing the nasopharynx in children after 13-valent pneumococcal conjugate vacci- nation in comparison to the 7-valent era, 2006-2015. Pediatr Infect Dis J. 2016;35

(8):901-6. https://doi.org/10.1097/INF.0000000000001206.

1. Tong S, et al. Trends in healthcare utilization and costs associated with acute otitis media in the United States during 2008-2014. BMC Health Serv Res. 2018;18(1):

318. https://doi.org/10.1186/s12913-018-3139-1.

1. Lieberthal AS, et al. The diagnosis and Management of Acute Otitis Media. Pediatrics. 2013;131(3):e964-99. https://doi.org/10.1542/peds.2012-3488.
2. McGrath LJ, et al. Trends in antibiotic treatment of acute otitis media and treatment failure in children, 2000-2011. PLOS ONE. 2013;8(12):e81210. https://doi.org/10. 1371/journal.pone.0081210.
3. Soares C, Clifton W, Freeman WD. Use of handheld video otoscopy for the diagnosis of acute otitis media: technical note. Cureus. 2019;11(9):e5547. https://doi.org/10. 7759/cureus.5547.
4. Paul CR, et al. Teaching the pediatric ear exam and diagnosis of acute otitis media: a teaching and assessment model in three groups. BMC Med Educ. 2017;17(1):146. https://doi.org/10.1186/s12909-017-0988-y.
5. Paul CR, et al. Teaching pediatric Otoscopy skills to pediatric and emergency medi- cine residents: a cross-institutional study. Acad Pediatr. 2018;18(6):692-7. https:// doi.org/10.1016/j.acap.2018.02.009.
6. Steinbach WJ, et al. Pediatric Residents’ clinical diagnostic accuracy of otitis media. Pediatrics. 2002;109(6):993-8. https://doi.org/10.1542/peds.109.6.993.
7. Damery L, et al. Interest of video-otoscopy for the general practitioner. Eur Ann Otorhinolaryngol Head Neck Dis. 2019;136(1):13-7. https://doi.org/10.1016/j. anorl.2018.10.016.
8. Hakimi A, et al. Utility of a smartphone-enabled otoscope in the instruction of otoscopy and middle ear anatomy. Eur Arch Otorhinolaryngol. 2019;276. https:// doi.org/10.1007/s00405-019-05559-6.
9. Richards JR, Gaylor KA, Pilgrim AJ. Comparison of traditional otoscope to iPhone oto- scope in the pediatric ED. Am J Emerg Med. 2015;33(8):1089-92. https://doi.org/10. 1016/j.ajem.2015.04.063.
10. Rappaport KM, et al. Assessment of a smartphone otoscope device for the diagnosis and management of Otitis Media. Clin Pediatr. 2015;55(9):800-10. https://doi.org/ 10.1177/0009922815593909.
11. Mousseau S, Lapointe A, Gravel J. Diagnosing acute otitis media using a smartphone otoscope; a randomized controlled trial. Am J Emerg Med. 2018. https://doi.org/10. 1016/j.ajem.2018.01.093.
12. Harris PA, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. 2019;95:103208. https://doi.org/10. 1016/j.jbi.2019.103208.
13. Harris PA, et al. Research electronic data capture (REDCap)-a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377-81. https://doi.org/10.1016/j.jbi.2008.08. 010.
14. Lundberg T, et al. Diagnostic accuracy of a general practitioner with video-otoscopy collected by a health care facilitator compared to traditional otoscopy. Int J Pediatr Otorhinolaryngol. 2017;99:49-53. https://doi.org/10.1016/j.ijporl.2017.04.045.
15. Lundberg T, et al. Remote evaluation of video-otoscopy recordings in an unselected pediatric population with an otitis media scale. Int J Pediatr Otorhinolaryngol. 2014; 78(9):1489-95. https://doi.org/10.1016/j.ijporl.2014.06.018.
16. Vanbelle S. Comparing dependent kappa coefficients obtained on multilevel data. Biom J. 2017;59(5):1016-34. https://doi.org/10.1002/bimj.201600093.
17. Davies J, et al. Otoscopy simulation training in a classroom setting: a novel approach to teaching otoscopy to medical students. Laryngoscope. 2014;124(11):2594-7. https://doi.org/10.1002/lary.24682.
18. Cooper S, et al. Otitis Media Module in the Pediatric Preclerkship Educational Exer- cises (PRECEDE) Curriculum. MedEdPORTAL. 2020;16. https://doi.org/10.15766/ mep_2374-8265.10920.
19. Pichichero ME, Poole MD. Assessing diagnostic accuracy and tympanocentesis skills in the management of Otitis Media. Arch Pediatr Adolesc Med. 2001;155(10): 1137-42. https://doi.org/10.1001/archpedi.155.10.1137.
20. Pichichero ME, Poole MD. Comparison of performance by otolaryngologists, pedia- tricians, and general practioners on an otoendoscopic diagnostic video examination. Int J Pediatr Otorhinolaryngol. 2005;69(3):361-6. https://doi.org/10.1016/j.ijporl. 2004.10.013.
21. Lundberg T, Hellstrom S, Sandstrom H. Development and validation of a new grad- ing scale for otitis media. Pediatr Infect Dis J. 2013;32(4):341-5. https://doi.org/10. 1097/INF.0b013e3182869397 In press.