Inter-Rater Agreement in Digital Otoscopy: What It Measures and What It Does Not

What Does “Inter-Rater Agreement” Actually Measure?

Inter-rater agreement measures how often two or more reviewers reach the same conclusion when they look at the same otoscopy image or video. It does not measure whether that shared conclusion is correct, nor does it say whether a device or exam is right for a particular person. A study can report excellent agreement while saying nothing about accuracy at all.

What Do the Key Terms in an Agreement Study Mean?

A few terms show up in almost every agreement study, and mixing them up is the most common way readers misread the findings.

• Inter-rater agreement: How often two or more separate reviewers, looking at the same image or video, mark the same finding.
• Intra-rater agreement: How often the same reviewer, or the same case examined by two different methods, reaches the same conclusion.
• Reference standard: The exam a study treats as the closest available answer for comparison — for example, an in-person exam by a specialist.
• Diagnostic accuracy: How often a method’s conclusion matches the reference standard. This is a different question from whether reviewers agree with each other.
• Kappa or percent agreement: The statistic researchers use to report how much two reviewers’ findings overlap.

Is High Agreement Between Reviewers the Same as High Accuracy?

No. Agreement and accuracy answer two different questions, and a real pair of otoscopy studies clearly shows the difference.

In a study of a smartphone-attachment otoscope used for at-home ear-tube monitoring, two independent ear, nose, and throat physicians separately reviewed the same recorded videos. Their findings matched closely for nearly all the features they checked. The same study also compared the video recordings to a traditional in-office exam and found strong overlap between the two methods (at least 80% agreement, P<.05, for tube-status findings). (Don et al., 2021)

A separate study asked a different question entirely. Medical residents examined children’s ears with either a traditional otoscope or a smartphone otoscope attachment, and their diagnoses were compared with those of a pediatric ear, nose, and throat specialist using a microscope — the reference standard. Diagnostic accuracy was 0.69 with the traditional otoscope and 0.74 with the smartphone attachment, a gap the researchers did not consider statistically meaningful. (Mousseau et al., 2018)

Line these up and the distinction is clear: the first study asked whether two reviewers see the same thing. The second asked whether a diagnosis matched an expert reference exam. A method could show very consistent readings between reviewers that are still consistently off the reference standard, or somewhat variable readings between reviewers that still land close to it on average. A study built to test one does not automatically answer the other.

How Do You Read an Agreement Claim in an Otoscopy Study?

Use these questions whenever a study, product page, or summary mentions “agreement” between reviewers or exam methods.

1. Agreement between what, exactly? Between two reviewers looking at the same material, or between a method and a reference standard? If the source doesn’t say which, treat the claim as incomplete.
2. Who were the reviewers? Specialists, trainees, or a mix. Agreement among specialists doesn’t tell you how a less-experienced reviewer or a parent would perform on the same images.
3. What was the reference standard? An in-person exam by a specialist is a stronger basis for comparison than a single reviewer’s opinion.
4. Was accuracy reported separately from agreement? If a summary only reports agreement, accuracy against a reference standard may not have been measured at all.
5. How many cases, and what population? A study of a few dozen children in one clinic doesn’t automatically apply to a different age group or setting.

As a working rule: if a source reports agreement between reviewers but never mentions a reference standard, treat it as a consistency finding, not an accuracy finding — and look for a separate accuracy study before drawing conclusions about correctness. Our How We Research page walks through this distinction in more detail, and our Editorial Policy covers how we vet sources before citing them.

Does This Mean a Smartphone Otoscope Is Safe to Rely On?

Agreement and accuracy statistics describe how a study performed under specific conditions. They are not a guarantee about any single exam, device, or image, and they are not a diagnosis. If you or your child has ear pain, drainage, hearing changes, or a concerning fever, contact a clinician for an in-person evaluation; if symptoms are severe or rapidly worsening, contact local emergency services. Our Start Here page outlines how to use consumer health images as a conversation aid without delaying care, and our Medical Disclaimer explains the limits of everything published here.

Frequently Asked Questions

Does high inter-rater agreement mean a smartphone otoscope is accurate?

Not by itself. High agreement means reviewers tend to see the same thing when looking at the same image or video. Whether that shared finding matches a reference-standard exam is a separate measurement, reported in different studies.

What is a reference standard in an otoscopy study?

It’s the exam researchers treat as the closest available answer for comparison — commonly an in-person exam by a specialist, sometimes performed with a microscope. Diagnostic accuracy is measured against this standard, not against another reviewer’s opinion.

Can I trust a device based solely on an agreement study?

An agreement study alone only tells you reviewers were consistent with each other. Look for a companion accuracy study comparing the device’s findings to a reference-standard exam before drawing conclusions about correctness.

Who reviewed the otoscopy images in the studies cited here?

In the home tympanostomy-tube study, two independent ear, nose, and throat physicians reviewed the recorded videos. In the emergency department study, medical residents’ diagnoses were compared with those of a pediatric ear, nose, and throat specialist using a microscope.

Sources: Don DM, Koempel JA, Fisher LM, Wee CP, Osterbauer B. Prospective Evaluation of a Smartphone Otoscope for Home Tympanostomy Tube Surveillance: A Pilot Study. Ann Otol Rhinol Laryngol. 2021. PubMed. Mousseau S, Lapointe A, Gravel J. Diagnosing acute otitis media using a smartphone otoscope: a randomized controlled trial. Am J Emerg Med. 2018. PubMed.

This article is for general education only and is not a diagnosis or a substitute for professional medical advice. Everyday Imaging Evidence is an independent editorial publication and is not affiliated with, and does not represent, the former CellScope company or its products.

By Everyday Imaging Evidence Editorial Team. Updated September 8, 2026.