Trained on breast ultrasound images (BUSI), the hybrid model achieved 99.24% classification accuracy while also improving content-based image retrieval performance.
Background
Breast cancer is the second most commonly diagnosed cancer worldwide, and ultrasound remains a safe and accessible imaging modality for detection. As medical image databases continue to grow, content-based medical image retrieval (CBMIR) systems are becoming increasingly valuable. These systems enable clinicians to identify similar cases using visual features, supporting both diagnosis and treatment planning.
Previous deep learning approaches, particularly those using CNNs and attention mechanisms, have demonstrated strong performance. However, several limitations remain. Many studies rely on relatively small datasets, which can restrict generalizability. Cross-dataset validation is often limited, and explainability is not always integrated into model design. In addition, computational complexity can make real-world clinical deployment more challenging.
To address these gaps, the researchers of this study proposed a hybrid CNN-RNN architecture integrated with XAI. In this approach, CNNs handle spatial feature extraction, while RNNs model relationships within image representations. The addition of explainability techniques helps make predictions more transparent, improving both performance and clinical usability.
Model Architecture and Implementation
The study used the BUSI dataset, which includes 830 grayscale ultrasound images across three categories: normal (10.7 %), benign (62.4 %), and malignant (26.9 %). Originally stored in DICOM format, the images underwent extensive preprocessing. The dataset was organized into clearly labeled categories, and mask files were created to distinguish cancerous and non-cancerous regions. An 80:20 split was used for training and testing.
To evaluate performance, the researchers implemented four deep learning architectures.
The CNN model consisted of four convolutional layers with increasing filter sizes, ReLU activation, and max-pooling, followed by fully connected layers with dropout and a softmax output. The RNN model treated image data as sequences, using a long short-term memory (LSTM) layer to capture dependencies across pixel rows. The XAI model applied transfer learning with MobileNetV2 and used Grad-CAM to generate heatmaps highlighting regions that influenced predictions, alongside standard data augmentation techniques.
The proposed hybrid model integrates VGG16 for spatial feature extraction with an RNN to capture relationships within those features, while also incorporating Grad-CAM for interpretability. It also enables content-based image retrieval by using features from the penultimate layer as image signatures and comparing them using Euclidean distance to identify clinically similar cases.
Data augmentation techniques, including rotation, flipping, scaling, and contrast adjustment, expanded the training set significantly. Training was optimized using the Adam optimizer, categorical cross-entropy loss, early stopping, and adaptive learning rate adjustments.
Results and Performance Analysis
Performance was evaluated using accuracy, precision, recall, and F1-score, along with confusion matrix analysis to better understand misclassification patterns. The CNN model achieved 98.85 % accuracy with strong overall metrics and minimal errors.
The RNN model reached 94.58 % accuracy but showed some confusion between benign and normal cases. The XAI-based MobileNetV2 model achieved 89.68 % accuracy, offering interpretability at the cost of slightly lower classification performance.
The hybrid model delivered the strongest results, achieving 99.24 % accuracy along with the lowest training loss. Its confusion matrix showed near-perfect classification, with only minimal misclassifications across categories. Cross-validation further confirmed consistency, with a mean accuracy of 98.92 % and low variance. Grad-CAM visualizations aligned closely with clinically relevant regions, reinforcing confidence in the model’s predictions.
Comparative analysis showed that the hybrid model outperformed standalone approaches (CNN, RNN, and XAI) and compared favorably with previously reported methods such as DDA-Net and attention U-Net. The combination of spatial feature extraction, sequential modeling, and explainability appears to drive these improvements, while the retrieval component adds practical value by enabling clinicians to reference similar cases during diagnosis.
Conclusion
Overall, this study presents a hybrid deep learning model that combines CNNs, RNNs, and XAI for breast ultrasound image classification and retrieval.
Tested on the BUSI dataset, the model achieved 99.24 % accuracy, outperforming individual architectures and several comparable approaches. By integrating high performance with interpretability, the system moves closer to practical use in clinical workflows.
That said, the study remains limited to a single dataset and does not yet include external validation or clinical trial evaluation. Further work will need to expand testing across datasets and imaging modalities, incorporate patient-level data, explore newer architectures such as transformers, and validate performance through clinical studies involving radiologists.
Journal Reference
Fawzy, A. E., Almandouh, M. E., Herajy, M., & Eisa, M. (2026). Ensemble-based high-performance deep learning models for medical image retrieval in breast cancer detection. Scientific Reports, 16(1). DOI:10.1038/s41598-026-38218-y. https://www.nature.com/articles/s41598-026-38218-y
Disclaimer: The views expressed here are those of the author expressed in their private capacity and do not necessarily represent the views of AZoM.com Limited T/A AZoNetwork the owner and operator of this website. This disclaimer forms part of the Terms and conditions of use of this website.