Validation of Deep Learning Based AI Algorithm in Pattern Recognition of the Bilateral Mandibular Canals Using Cone Beam Computed Tomography

Abstract

AI has been proposed as a promising solution to improve diagnosis and reporting of findings in radiology. The first step in artificial intelligence (AI) deep learning (DL) network-based development is retrospective technical validation, which requires the evaluation of four crucial properties: accuracy, robustness, generalisability and reproducibility. Our research group introduced the first DL-based AI solution, the Deep Learning System (DLS 2020), for mandibular canal segmentation in 2020. DLS 2020 demonstrated good accuracy and robustness. To our knowledge, there have been no previous studies on the reproducibility and generalisability of automated AI/DL mandibular canal segmentation. There are 2 main objectives of this study. The focus of Objective 1 was to develop the DLS 2020 into the DLS 2022 and to evaluate the generalisability of the DLS 2020 and the accuracy and robustness of the DLS 2022. The results were analysed qualitatively and quantitatively. The dataset consisted of 982 CBCT scans as a development set and 150 scans for a test set. Three CBCT scanners were used in Finland and two in Thailand. Mandibular canal annotations in the test set were performed by four radiologists and compared with DLS prediction. The main results were reported in terms of a median of the symmetric mean curve distance (SMCD), which is the difference in distance between the radiologist’s annotation and the DLS prediction in 3D space calculated in millimetres at each voxel, with 0 mm representing perfect agreement. Quantitative analysis showed that the DLS 2022 had significantly lower variability than radiologists (0.74 vs 0.77 mm, p<0.001). The DLS 2022 also had a significantly lower variability than the DLS 2020 (0.74 vs. 0.78 mm, p<0.001). The DLS 2020 had a significant generalisability compared to the three new devices with 0.63 mm, 0.67 mm and 0.87 mm respectively (p < 0.001). Using radiologist consensus segmentation as the gold standard, the DLS 2022 showed a median SMCD of 0.39 mm, which was statistically significantly different (p<0.001) compared to individual radiologists with values of 0.62 mm, 0.55 mm, 0.47 mm and 0.42 mm. In the qualitative analysis, the estimated error rate for the DLS 2022 was 1% while the radiologist error rate was 2.4%. The focus of Objective 2 was to evaluate the reproducibility and generalisability of the DLS 2022 and to re-assess for the accuracy and robustness of the DLS 2022. Heterogeneous165 clinically repeated scans from 72 patients divided into four subgroups: Normal, Prosthetic TMJ, Orthognathic Surgery and Pathological groups were included in this study. Overall, the DLS 2022 had a median SMCD of 0.643 mm. The reproducibility of the DLS 2022 was assessed as a within-subject repeatability coefficient (RC). The RC was 0.329 mm, 0.574 mm, 1.707 mm, and 0.648 mm for normal, TMJ prosthetic, orthognathic, and pathological heterogeneities. In the qualitative analysis, three other radiologists rated the performance of the radiologist and the DLS 2022 twice for diagnostic validity using a Likert scale from 0: not usable for diagnostics to 4: fully diagnostically usable. The mean (standard deviation) Likert scores for Normal, TMJ Prosthetic, Orthognathic and Pathological heterogeneities were 3.97 (0.19), 3.88 (0.42), 3.97 (0.22), 3.94 (0.23) for the radiologist and 3.86 (0.62), 3.92 (0.38), 3.71 (0.90), 3.90 (0.39) for the DLS 2022. For Objective 2, the error rate for mandibular canal segmentation was 0.6% for the radiologist and 4.1% for the DLS 2022, showing better human performance compared to the DLS 2022. However, the DLS 2022 errors caused by outliers did not affect the clinical significance. In conclusion, the DLS 2022, developed from the DLS 2020, had improved performance in mandibular canal segmentation. All the crucial properties, including accuracy, robustness, generalisability, and reproducibility, were completely evaluated and showed efficient results. In the clinical evaluation, the efficiency of the DLS 2022 was comparable to that of radiologists. Therefore, the technical validation is complete and provides insight into the potential value of using the DLS 2022 in a realistic clinical setting.