According to a new study, support from an AI algorithm with high diagnostic accuracy has enhanced radiologists' performance detecting lung cancers on chest X-Rays and increased acceptance of human AI suggestions.
(A–C) Chest radiographs obtained in a 71-year-old male patient during a health checkup show reader susceptibility to high diagnostic accuracy artificial intelligence (AI). In the first session without AI, a thoracic radiologist with 16 years of experience read the chest radiograph as normal (A). High diagnostic accuracy AI observed potential lung cancer in the radiograph with an 89% CI as indicated by the nodule localization map (B) (as the color changes from blue to red, the probability of the presence of a nodule increases). When presented with the AI suggestion at the second reading session, the radiologist changed the decision and annotated lung cancer in the area that overlapped with the right hemidiaphragm (box annotation) (C). (D, E) Contrast-enhanced chest CT scans show a 6.8-cm lung mass (arrow) with an air bronchogram in the right lower lobe in the axial (D) and coronal (E) planes. This mass was pathologically proven to be an invasive mucinous adenocarcinoma. Therefore, the reader’s decision was incorrect in the first session but correct in the second session after following the AI suggestion. Image Credit: https://doi.org/10.1148/radiol.222976 ©RSNA 2023.
The study has been reported in the journal Radiology.
In this retrospective study, 30 readers, comprising 20 thoracic radiologists with around five to 18 years of experience and 10 radiology residents with only two to three years of experience, evaluated 120 chest X-Rays without AI. Among the 120 chest radiographs assessed, 60 were controls (36 males) and 60 were from lung cancer patients (32 males). Patients had an average age of 67 years.
In a second session, every group reinterpreted the X-Rays, helped by either a high- or low-accuracy AI. The readers were blind to the fact that two various AIs were utilized.
Utilizing the high-accuracy AI-enhanced readers’ detection performance to a greater extent than low-accuracy AI. Also, using high-accuracy AI resulted in highly frequent alterations in reader determinations—an idea called susceptibility.
It is possible that the relatively large sample size in this study bolstered readers’ confidence in the AI’s suggestions. We think this issue of human trust in AI is what we observed in the susceptibility in this study: humans are more susceptible to AI when using high diagnostic performance AI.
Chang Min Park, MD, PhD, Study Lead Author, Department of Radiology and Institute of Radiation Medicine, College of Medicine, Seoul National University
Task, Clinical Context Affect Perception of AI Performance
In comparison to the first reading session, readers helped by the high diagnostic accuracy AI at the second reading session displayed higher specificity (0.94 versus 0.88) and per-lesion sensitivity (0.63 versus 0.53).
Instead, readers supported by the low diagnostic precision AI at the second reading session did not display enhanced between the two reading sessions for any of such measurements.
Our study suggests that AI can help radiologists, but only when the AI’s diagnostic performance meets or exceeds that of the human reader.
Chang Min Park, MD, PhD, Study Lead Author, Department of Radiology and Institute of Radiation Medicine, College of Medicine, Seoul National University
The outcomes emphasize the significance of utilizing high-diagnostic-performance AI. But Dr. Park noted that the definition of “high diagnostic performance AI” could vary based on the task and the clinical context in which it will be utilized.
For instance, an AI model that has the potential to detect all abnormalities on a chest X-Ray might look perfect. However, such a model would have restricted value in decreasing the workload in a pulmonary tuberculosis mass screening setting.
Therefore, our study suggests that clinically appropriate use of AI requires both the development of high-performance AI models for given tasks and considerations about the relevant clinical setting to which that AI will be applied.
Chang Min Park, MD, PhD, Study Lead Author, Department of Radiology and Institute of Radiation Medicine, College of Medicine, Seoul National University
Scientists wish to expand their work on human-AI collaboration to other abnormalities on CT images and chest X-Rays.
Journal Reference:
Lee, J. H., et al. (2023) Effect of Human-AI Interaction on Detection of Malignant Lung Nodules on Chest Radiographs. Radiology. doi.org/10.1148/radiol.222976.
Source: https://www.rsna.org/