Breast Tumor Cell Nuclei Segmentation in Histopathology Images using EfficientUnet++ and Multi-organ Transfer Learning

  • Dinh, Tuan Le (Dept. of Artificial Intelligence Convergence, Pukyong National University) ;
  • Kwon, Seong-Geun (Dept. of Electronics Engineering, Kyungil University) ;
  • Lee, Suk-Hwan (Dept. of Computer Engineering, Dong A University) ;
  • Kwon, Ki-Ryong (Dept. of Artificial Intelligence Convergence, Pukyong National University)
  • Received : 2021.08.04
  • Accepted : 2021.08.12
  • Published : 2021.08.30


In recent years, using Deep Learning methods to apply for medical and biomedical image analysis has seen many advancements. In clinical, using Deep Learning-based approaches for cancer image analysis is one of the key applications for cancer detection and treatment. However, the scarcity and shortage of labeling images make the task of cancer detection and analysis difficult to reach high accuracy. In 2015, the Unet model was introduced and gained much attention from researchers in the field. The success of Unet model is the ability to produce high accuracy with very few input images. Since the development of Unet, there are many variants and modifications of Unet related architecture. This paper proposes a new approach of using Unet++ with pretrained EfficientNet as backbone architecture for breast tumor cell nuclei segmentation and uses the multi-organ transfer learning approach to segment nuclei of breast tumor cells. We attempt to experiment and evaluate the performance of the network on the MonuSeg training dataset and Triple Negative Breast Cancer (TNBC) testing dataset, both are Hematoxylin and Eosin (H & E)-stained images. The results have shown that EfficientUnet++ architecture and the multi-organ transfer learning approach had outperformed other techniques and produced notable accuracy for breast tumor cell nuclei segmentation.



  1. R.L. Siegel, K.D. Miller, H.E. Fuchs, and A. Jemal, "Cancer statistics," A Cancer Journal for Clinicians 71, Vol. 71, pp. 7-33, 2021.
  2. S.V. Sree, E.Y. Ng, R.U. Acharya, and O. Faust, "Breast imaging: A survey," World Journal of Clinical Oncology, Vol. 2, pp. 171-178, 2011.
  3. S.Y. Kwon, Y.J. Kim, and G.G. Kim, "A deep learning-based automatic segmentation of breast mass in breast imaging," Journal of the Korean Society for Multimedia, Vol. 21, No. 12, pp. 1363-1369, Dec. 2018.
  4. M. Lekka, "Atomic force microscopy: A tip for diagnosing cancer," Nature Nanotechnology, Vol. 7, pp. 691-692, 2012.
  5. J.D. Bancroft and C. Layton, "The hematoxylins and eosin," in Bancroft's Theory and Practice of Histological Techniques (Eighth Edition), Elsevier, 2019, pp. 126-138.
  6. M.N. Gurcan, L.E. Boucheron, A. Can, A. Madabhushi, N.M. Rajpoot, and B. Yener, "Histopathological image analysis: A Review," IEEE Reviews in Biomedical Engineering, Vol. 2, pp. 147-171, 2009.
  7. G. Litjens, C. Sanchez, N. Timofeeva et al., "Deep Learning as a Tool for Increased Accuracy and Efficiency of Histopathological Diagnosis," Scientific Reports, Vol. 6, No. 26286, 2016.
  8. S. Bhattacharjee, D. Prakash, C.H. Kim, and H.K. Choi, "Multichannel Convolution Neural Network Classification for the Detection of Histological Pattern in Prostate Biopsy Images," Journal of the Korean Society for Multimedia, Vol. 23, No. 12, pp. 1486-1495, Dec. 2020.
  9. P. Naylor, M. Lae, F. Reyal, and T. Walter, "Segmentation of Nuclei in Histopathology Images by Deep Regression of the Distance Map," IEEE Transactions on Medical Imaging, Vol. 38, No. 2, pp. 448-459, Feb. 2018.
  10. A. Janowczyk and A. Madabhushi, "Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases," Journal of Pathology Informatics, Vol. 7, pp. 29, Jul. 2016.
  11. Z. Zhao, H. Lin, H. Chen, and P.A. Heng, "PFA-ScanNet: Pyramidal Feature Aggrega-Tion with Synergistic Learning for Breast Cancer Metastasis Analysis," Medical Image Computing and Computer Assisted Intervention-MICCAI 2019, pp. 586-594, 2019.
  12. S. Graham et al., "MILD-Net: Minimal Information Loss Dilated Network for Gland Instance Segmentation in Colon Histology Images," Medical Image Analysis, Vol. 52, pp. 199-211, 2019.
  13. D.J. Ho et al., "Deep Multi-Magnification Networks for Multi-Class Breast Cancer Image Segmentation," Computerized Medical Imaging and Graphics, Vol. 88, No. 101866, 2021.
  14. S. Minaee, Y.Y. Boykov, F. Porikli, A.J. Plaza, N. Kehtarnavaz, and D. Terzopoulos, "Image Segmentation Using Deep Learning: A Survey," arXiv preprint, arXiv:2001.05566, 2020.
  15. H. Noh, S. Hong, and B. Han, "Learning Deconvolution Network for Semantic Segmentation," 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1520-1528, 2015.
  16. V. Badrinarayanan, A. Kendall, and R. Cipolla, "SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, No. 12, pp. 2481-2495, 2017.
  17. O. Ronneberger, P. Fischer, and T. Brox, "UNet: Convolutional Networks for Biomedical Image Segmentation," Medical Image Computting and Computer-Assisted Intervention-MICCAI 2015, vol. 9351, 2015.
  18. N. Siddique, S. Paheding, C. P. Elkin, and V. Devabhaktuni, "U-Net and its Variants for Medical Image Segmentation: A Review of Theory and Applications," IEEE Access, Vol. 9, pp. 82031-82057, 2021.
  19. B. Baheti, S. Innani, S. Gajre, and S. Talbar, "Eff-UNet: A Novel Architecture for Semantic Segmentation in Unstructured Environment," in IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020.
  20. Z. Zhou, M.M.R Siddiquee, N. Tajbakhsh, and J. Liang, "UNet++: A Nested U-Net Architecture for Medical Image Segmentation," Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. DLMIA 2018, ML-CDS 2018. Lecture Notes in Computer Science, Vol. 11045, pp. 3-11, 2018.
  21. M. Tan and Q.V. Le, "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks," in Proceedings of International Conference on Machine Learning, 2019.
  22. L.D. Huynh and N. Boutry, "A U-Net++ with Pre-trained EfficientNet Backbone for Segmentation of Diseases and Artifacts in Endoscopy Images and Videos," in EndoCV@ISBI, 2020.
  23. A. Lagree, M. Mohebpour, N. Meti et al., "A Review and Comparison of Breast Tumor Cell Nuclei Segmentation Performances Using Deep Convolutional Neural Networks," Scientific Reports, Vol. 11, No. 8025, 2021.
  24. N. Kumar et al., "A Multi-Organ Nucleus Segmentation Challenge," IEEE Transactions on Medical Imaging, Vol. 39, No. 5, pp. 1380- 1391, May 2020.
  25. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen, "MobileNetV2: Inverted Residuals and Linear Bottlenecks," in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
  26. A. Buslaev, V.I. Iglovikov, E. Khvedchenya, A. Parinov, M. Druzhinin, and A.A. Kalinin, "Albumentations: Fast and Flexible Image Augmentations," Information, Vol. 11, No. 2, pp. 125, 2020.
  27. M. Amgad, L.A. Atteya, H. Hussein, K.H. Mohammed, E. Hafiz, M.A. Elsebaie, A.M. Alhusseiny, M.A. AlMoslemany, A.M. Elmatboly, P.A. Pappalardo et al., "NuCLS: A Scalable Crowdsourcing, Deep Learning Approach and Dataset for Nucleus Classification, Localization and Segmentation," arXiv preprint, arXiv:2102. 09099, 2021.
  28. X. Yi, E. Walia and P. Babyn, "Generative Adversarial Network in Medical Imaging: A Review," Medical Image Analysis, Vol. 58, No. 101552, 2019.