Leveraging the Generalization Ability of Deep Convolutional Neural Networks for Improving Classifiers for Color Fundus Photographs

Original Paper

Leveraging the Generalization Ability of Deep Convolutional Neural Networks for Improving Classifiers for Color Fundus Photographs

Author Contributions

Author Contributions: Conceptualization, J.S., J.K. and S.T.K.;Formal Analysis, J.S. and J.K.; Methodology, J.S., J.K., S.T.K. and K.-H.J.; Writing—Original draft, J.S., J.K. and S.T.K.; Writing—Review & Editing, J.S. and K.-H.J.; Supervision, K.-H.J. All authors have read and agreed to the published version of the manuscript.

Intro

When trying to solve problems using deep learning, data is often built using methods like "crowdsourcing" to save human resources and costs, but the performance of deep learning models often depends heavily on the quality and quantity of the data. However, data built by non-experts or such real-world datasets can be inaccurate due to various human errors, which can degrade the generalization performance of the model. In addition, in many cases, obtaining labeled data for minor classes is practically impossible or requires a lot of time and effort, so performance degradation due to a "long-tail distribution" must also be considered. Therefore, when designing and training deep learning models in the field, we need to consider various situations and focus on building more robust models.

The team I was part of researched techniques to solve the aforementioned problems and participated in the pathological myopia classification challenge (PALM) held at MICCAI, a top medical conference, eventually winning 17th place. In this post, we introduce various practical techniques used in the competition. For more detailed explanations, please refer to the paper.

Although the data used in this paper appears to differ from the modality handled by Genesis Lab, we would like to introduce this technology because it can be applied not only to Genesis Lab's deep learning models but also to various other deep learning-based applications.

Dataset

To solve the task of predicting pathological myopia (PM), fundus images were used, and the image above is an example of a fundus image with PM. In the case of PM, in addition to having a tigroid pattern characteristic, PM can also be diagnosed based on other clinical evidence, such as the presence of retinal detachment. Since pathological myopia (PM) usually affects 0.9–3.1% of Asians and 1.2% of Australians, the training dataset also had a low proportion of positive cases, consisting of 400 labeled datasets.

Method

The team used two main approaches to boost the generalization performance of the deep learning model during the competition.

First, using semi-supervised learning to utilize unlabeled samples for additional training

Second, designing filtration networks to detect noise-positive data in order to reduce the impact of noisy datasets, thereby excluding the noise data detected by the filtration networks from the target task's training process.

Leveraging the Generalization Ability of CNNs for pseudo-labeling

One approach that can be used when there is insufficient training data is semi-supervised learning (SSL) utilizing unlabeled data. Among SSL methods, pseudo-labeling has proven effective in many studies. However, while traditional pseudo-labeling methods primarily address balanced class distributions, applying pseudo-labeling to models trained on "long-tail distribution" datasets can result in sub-par performance, and it is difficult to trust pseudo-labels from models trained on scarce datasets. Therefore, to obtain more accurate pseudo-labels, we formulated the following hypotheses considering prior domain knowledge, data imbalance, and size:

1. Since pathological myopia (PM) typically occurs at a rate of 0.9–3.1%, the proportion of PM in public fundus images would not overwhelm the Normal cases unless it is a specific PM dataset.

2. Even when deep neural networks (DNNs) are trained on noisy datasets, they initially learn the general features of the data in the early stages of training, and memorization occurs as training progresses.

Using these hypotheses, our team labeled all the collected unlabeled public datasets as "normal" beforehand and proceeded with training alongside the labeled dataset. Since we assumed all public datasets were normal, noise was naturally included, so we adopted a method that generates pseudo-leveling for the public dataset using the early-stage predictions of the model. Afterwards, we built the model by retraining it, using the public dataset as training data for the competition duration and the provided competition training dataset as the evaluation dataset. To secure additional data, we collected 91,509 unlabeled public images, using Kaggle [1], Messidor [2], IDRiD [3], REFUGE [4], and RIGA [5] datasets.

Filtering Suspicious Data

To minimize the impact of noisy datasets, our team designed filtration networks that detect noise-positive data and excluded the noise data detected by the filtration network $b$ from training the PM classification model $f \rightarrow \hat{p}\in\Delta(\mathbb{y})$.

$b$ detects noise-positive data using the uncorrupted validation dataset $D_{val}$ and the posterior distribution of $f$. The training of filtration networks $b$ occurs at each epoch of $f$, takes the posterior distribution $\hat{p}$ of $f$ on $D_{val}$ as input, and proceeds in a direction that reduces the error with the true label $y_{val}$. Therefore, a well-trained filtration network $b^*$ shows higher confidence on clean positive data while showing low confidence on suspiciously negative data.
Since $b^*$ is trained at each epoch of $f$, we use a logistic regression model ($b(\hat{p})=w^T\hat{p}$) for training time efficiency. As a result, if $b^*$ determines a given train sample from $D_{train}$ as noise data with a confidence level greater than or equal to $\tau$, that data does not contribute to the training of $f$:

$\theta_f = \theta_f - \eta \nabla L(\theta_f;x,y)\mathbf{1}\{b^*(\hat{p})>\tau\}$

To evaluate the validity of the filtration network, we compared quantitative results with existing state-of-the-art methods using the Kaggle 2015 dataset with 35,126 images (17,563 eyes) for training and 53,576 images (26,788 eyes), and the proposed method achieved the highest accuracy at p < 0.5 (Table 1).

Conclusion

In this post, we introduced a method for filtering noisy labels using classifier confidence and an SSL approach using publicly available unlabeled data. The noisy label filtration method we proposed showed higher performance compared to existing approaches on noise-added datasets. The pseudo-labeling method leveraging the generalize ability of DNNs was effective in detecting rare positive cases and enabled the learning of unfamiliar patterns that are difficult to learn from the original small training set, thereby yielding high performance. Ultimately, we concluded the competition with an AUROC performance of 0.9993 on the off-site validation set.

While we applied the proposed method to medical data through the competition, we anticipate that it can be widely applied to real-world settings that require a large volume of labeled data.

References

[1] Kaggle, “Diabetic Retinopathy Detection Competition Report,” Kaggle 2015.

[2] Decencière et al., “Feedback on a Publicly Distributed Database: The Messidor Database,” Image Anal. Stereol. 2014.

[3] Porwal et al., “Indian Diabetic Retinopathy Image Dataset (IDRiD): A Database for Diabetic Retinopathy Screening Research,” Data 2018.

[4] “Retinal Fundus Glaucoma Challenge,” REFUGE Grand Challenge 2019.

[5] Almazroa et al., “Retinal Fundus Images for Glaucoma Analysis: The RIGA Dataset,” SPIE Medical Imaging 2018.