A review of current systems for annotation of cell and tissue images in digital pathology

• Autorzy: Korzynska Anna , Roszkowiak Lukasz , Zak Jakub , Siemion Krzysztof

Identyfikatory

DOI

10.1016/j.bbe.2021.04.012

• Języki publikacji: EN

Abstrakty

EN

With the advent and great advances of methods based on deep learning in image analysis, it appears that they can be effective in digital pathology to support the work of pathologists. However, a major limitation in the development of computer-aided diagnostic systems for pathology is the cost of data annotation. Evaluation of tissue (histopathological) and cellular (cytological) specimens seems to be a complex challenge. To simplify the laborious process of obtaining a sufficiently large set of data, a number of different systems could be used for image annotation. Some of these systems are reviewed in this paper with a comparison of their capabilities.

Słowa kluczowe

EN

digital pathology; deep learning; image processing; image analysis; data annotation

PL

patologia cyfrowa; uczenie głębokie; przetwarzanie obrazu; analiza obrazu

• Wydawca: Nałęcz Institute of Biocybernetics and Biomedical Engineering of the Polish Academy of Sciences ; Elsevier
• Czasopismo: Biocybernetics and Biomedical Engineering
• Rocznik: 2021
• Tom: Vol. 41, no. 4
• Strony: 1436--1453
• Opis fizyczny: Bibliogr. 97 poz., rys., tab.

Twórcy

autor

Korzynska Anna

• Laboratory of Processing and Analysis of Microscopic Images, Nalecz Institute of Biocybernetics and Biomedical Engineering Polish Academy of Sciences, Warsaw, Poland

autor

Roszkowiak Lukasz

• Laboratory of Processing and Analysis of Microscopic Images, Nalecz Institute of Biocybernetics and Biomedical Engineering Polish Academy of Sciences, Warsaw, Poland

autor

Zak Jakub

• Laboratory of Processing and Analysis of Microscopic Images, Nalecz Institute of Biocybernetics and Biomedical Engineering Polish Academy of Sciences, Warsaw, Poland

autor

Siemion Krzysztof

• Laboratory of Processing and Analysis of Microscopic Images, Nalecz Institute of Biocybernetics and Biomedical Engineering Polish Academy of Sciences, Warsaw, Poland; Medical Pathomorphology Department, Medical University of Bialystok, Bialystok, Poland

Bibliografia

• [1] W.C. of Tumours Editorial, E. Campo, N.L. Harris, WHO Classification of Tumours of Haematopoietic and Lymphoid Tissues, WORLD HEALTH ORGN; 2017. URL: https://www. ebook.de/de/product/29978648/who_classification_of_tumours_editorial_e_campo_n_l_harris_who_classification_of_tumours_of_haematopoietic_and_lymphoid_tissues.html.
• [2] Meijer GA, Belien JA, van Diest PJ, Baak JP. Origins of...image analysis in clinical pathology. J Clin Pathol 1997;50(5):365–70. https://doi.org/10.1136/jcp.50.5.365.
• [3] Hamilton PW, Van Diest PJ, Williams R, Gallagher AG. Do we see what we think we see? The complexities of morphological assessment. J Pathol J Pathol Soc Great Britain Ireland 2009;218(3):285–91.
• [4] Rubin R. Rubin’s pathology: clinicopathologic foundations of medicine. Lippincott Williams & Wilkins; 2007.
• [5] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521 (7553):436–44. https://doi.org/10.1038/nature14539.
• [6] Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JA, van Ginneken B, Sa´nchez CI. A survey on deep learning in medical image analysis. Med Image Anal 2017;42:60–88. https://doi.org/10.1016/j.media.2017.07.005.
• [7] Hollandi R, Szkalisity A, Toth T, Tasnadi E, Molnar C, Mathe B, et al. A deep learning framework for nucleus segmentation using image style transfer. Doi:10.1101/580605.
• [8] Moshkov N, Mathe B, Kertesz-Farkas A, Hollandi R, Horvath P. Test-time augmentation for deep learning-based cell segmentation on microscopy images. Sci Rep 10. doi:10.1101/814962.
• [9] Gurcan M, Boucheron L, Can A, Madabhushi A, Rajpoot N, Yener B. Histopathological image analysis: A review. IEEE Rev Biomed Eng 2009;2:147–71. https://doi.org/10.1109/rbme.2009.2034865.
• [10] Zarella MD, Bowman D, Aeffner F, Farahani N, Xthona A, Absar SF, Parwani A, Bui M, Hartman DJ. A practical guide to whole slide imaging: A white paper from the digital pathology association. Arch Pathol Lab Med 2018;143 (2):222–34.https://doi.org/10.5858/arpa.2018-0343-ra.
• [11] He L, Long R, Antani S, Thoma G. Computer-assisted diagnosis in cervical histopathology. Spie Newsroom.
• [12] Janowczyk A, Zuo R, Gilmore H, Feldman M, Madabhushi A, HistoQC. An open-source quality control tool for digital pathology slides. JCO Clin Cancer Inf 2019(3):1–7. https://doi.org/10.1200/cci.18.00157.
• [13] Fondón I, Sarmiento A, García AI, Silvestre M, Eloy C, Polónia A, Aguiar P. Automatic classification of tissue malignancy for breast carcinoma diagnosis. Comput Biol Med 2018;96:41–51. https://doi.org/10.1016/j.compbiomed.2018.03.003.
• [14] G.F.J.P. of Economics, F.J.P. of Economics, Smith PC, Cordes JDS. The 9 Pitfalls of Data Science, Oxford University Press; 2019. https://www.ebook.de/de/product/37110686/gary_fletcher_jones_professor_of_economics_fletcher_jones_professor_of_economics_pomona_college_smith_jay_data_scientist_cordes_the_9_pitfalls_of_data_science. Html.
• [15] Kyriazis AD, Noroozizadeh S, Refaee A, Choi W, Chu L-T, Bashir A, Cheng WH, Zhao R, Namjoshi DR, Salcudean SE, Wellington CL, Nir G. An end-to-end system for automatic characterization of iba1 immunopositive microglia in whole slide imaging. Neuroinformatics 2018;17(3):373–89. https://doi.org/10.1007/s12021-018-9405-x.
• [16] Wang S, Rong R, Yang DM, Fujimoto J, Yan S, Cai L, Yang L, Luo D, Behrens C, Parra ER, Yao B, Xu L, Wang T, Zhan X, Wistuba II, Minna J, Xie Y, Xiao G. Computational staining of pathology images to study the tumor microenvironment in lung cancer. Cancer Res 2020;80(10):2056–66. https://doi.org/10.1158/0008-5472.can-19-1629.
• [17] Chen C-M, Huang Y-S, Fang P-W, Liang C-W, Chang R-F. A computer-aided diagnosis system for differentiation and delineation of malignant regions on whole-slide prostate histopathology image using spatial statistics and multidimensional DenseNet. Med Phys 2020;47(3):1021–33. https://doi.org/10.1002/mp.13964.
• [18] Das A, Nair MS, Peter SD. Computer-aided histopathological image analysis techniques for automated nuclear atypia scoring of breast cancer: a review. J Digital Imag 2020;33 (5):1091–121. https://doi.org/10.1007/s10278-019-00295-z.
• [19] Keenan SJ, Diamond J, McCluggage WG, Bharucha H, Thompson D, Bartels PH, Hamilton PW. An automated machine vision system for the histological grading of cervical intraepithelial neoplasia (CIN). J Pathol 2000;192(3):351–62. https://doi.org/10.1002/1096-9896(2000)9999:9999<::aidpath708>3.0.co;2-i.
• [20] Demir C, Yener B. Automated cancer diagnosis based on histopathological images: a systematic survey; 2005.
• [21] May M. A better lens on disease. Sci Am 2010;302:74–7. https://doi.org/10.1038/scientificamerican0510-74.
• [22] Sejnowski TJ. The deep learning revolution. The MIT Press; 2018. URL: https://www.ebook.de/de/product/30073240/terrence_j_sejnowski_the_deep_learning_revolution.html.
• [23] Goodfellow I, Bengio J, Courville A, Bach F. Deep learning. MIT Press Ltd; 2016. URL: https://www.ebook.de/de/product/26337726/ian_goodfellow_joshua_ bengio_aaron_courville_francis_bach_deep_learning.html.
• [24] Dimitriou N, Arandjelović O, Caie PD. Deep learning for whole slide image analysis: An overview. Front Med 6. doi:10.3389/ fmed.2019.00264.
• [25] Acs B, Hartman J. Next generation pathology: artificial intelligence enhances histopathology practice. J Pathol 2019;250(1):7–8. https://doi.org/10.1002/path.5343.
• [26] Parwani AV. Next generation diagnostic pathology: use of digital pathology and artificial intelligence tools to augment a pathological diagnosis. Diagnos Pathol 14(1). doi:10.1186/s13000-019-0921-2.
• [27] Evans JL, Seo W, Macheski-Preston M, Fritz M, Puvanesarajah S, Hodge J, et al. Abstract 1635: A scalable, cloud-based, unsupervised deep learning system for identification, extraction, and summarization of potentially imperceptible patterns in whole-slide images of breast cancer tissue. In: Bioinformatics, Convergence Science, and Systems Biology, American Association for Cancer Research; 2019. doi:10.1158/1538-7445.am2019-1635.
• [28] Bulten W, Pinckaers H, van Boven H, Vink R, de Bel T, van Ginneken B, van der Laak J, van de Kaa CH, Litjens G. Automated deep-learning system for gleason grading of prostate cancer using biopsies: a diagnostic study. Lancet Oncol 2020;21(2):233–41. https://doi.org/10.1016/s1470-2045 (19)30739-9.
• [29] Caie PD, Dimitriou N, Arandjelović O. Precision medicine in digital pathology via image analysis and machine learning. In: Artificial Intelligence and Deep Learning in Pathology. Elsevier; 2021. pp. 149–173. doi:10.1016/b978-0-323-67538- 3.00008-7.
• [30] Girolami I, Pantanowitz L, Marletta S, Brunelli M, Mescoli C, Parisi A, Barresi V, Parwani A, Neil D, Scarpa A, Rossi ED, Eccher A. Diagnostic concordance between whole slide imaging and conventional light microscopy in cytopathology: A systematic review. Cancer Cytopathol 2019;128(1):17–28. https://doi.org/10.1002/cncy.22195.
• [31] Ginsburg G. Personalized medicine: revolutionizing drug discovery and patient care. Trends Biotechnol 2001;19 (12):491–6. https://doi.org/10.1016/s0167-7799(01)01814-5.
• [32] Garrison LP, Towse A. A strategy to support efficient development and use of innovations in personalized medicine and precision medicine. J Manage Care Special Pharm 2019;25(10):1082–7. https://doi.org/10.18553/jmcp.2019.25.10.1082.
• [33] Buettner R, Klenk F, Ebert M. A systematic literature review of machine learning-based disease profiling and personalized treatment. In: IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE; 2020. https://doi.org/10.1109/compsac48688.2020.00-15.
• [34] Samuel M, Gabrielsson S. Personalized medicine and back– allogeneic exosomes for cancer immunotherapy. J Internal Med. doi:10.1111/joim.12963.
• [35] Sirinukunwattana K, Domingo E, Richman SD, Redmond KL, Blake A, Verrill C, et al. Image-based consensus molecular subtype (imCMS) classification of colorectal cancer using deep learning. Gut (2020) gutjnl–2019–319866doi:10.1136/gutjnl-2019-319866.
• [36] Harder N, Schönmeyer R, Nekolla K, Meier A, Brieu N, Vanegas C, et al. Automatic discovery of image-based signatures for ipilimumab response prediction in malignant melanoma. Sci Rep 9(1). doi:10.1038/s41598-019-43525-8.
• [37] Fernandez G, Madduri AS, Marami B, Prastawa M, Scott R, Zeineh J, Donovan M. Artificial intelligence methods for predictive image-based grading of human cancers. In: Artificial Intelligence and Deep Learning in Pathology. Elsevier; 2021. pp. 175–210. doi:10.1016/b978-0-323-67538-3.00009-9.
• [38] Bardou D, Zhang K, Ahmad SM. Classification of breast cancer based on histology images using convolutional neural networks. IEEE Access 2018;6:24680–93. https://doi.org/10.1109/access.2018.2831280.
• [39] Arvaniti E, Fricker KS, Moret M, Rupp N, Hermanns T, Fankhauser C, et al. Automated gleason grading of prostate cancer tissue microarrays via deep learning. Sci Rep 8(1). doi:10.1038/s41598-018-30535-1.
• [40] Couture HD, Williams LA, Geradts J, Nyante SJ, Butler EN, Marron JS, et al. Image analysis with deep learning to predict breast cancer grade, ER status, histologic subtype, and intrinsic subtype. NPJ Breast Cancer 4(1). doi:10.1038/s41523- 018-0079-1.
• [41] Lin H, Chen H, Graham S, Dou Q, Rajpoot N, Heng P-A. Fast ScanNet: Fast and dense analysis of multi-gigapixel wholeslide images for cancer metastasis detection. IEEE Trans Med Imag 2019;38(8):1948–58. https://doi.org/10.1109/tmi.2019.2891305.
• [42] Sun H, Zeng X, Xu T, Peng G, Ma Y. Computer-aided diagnosis in histopathological images of the endometrium using a convolutional neural network and attention mechanisms. IEEE J Biomed Health Inf 2020;24(6):1664–76. https://doi.org/10.1109/jbhi.2019.2944977.
• [43] Yue X, Dimitriou N, Caie P, Harrison D, Arandjelovic O. Colorectal cancer outcome prediction from h&e whole slide images using machine learning and automatically inferred phenotype profiles. EasyChair. doi:10.29007/n912.
• [44] Dimitriou N, Arandjelović O, Harrison DJ, Caie PD. A principled machine learning framework improves accuracy of stage II colorectal cancer prognosis. NPJ Digit Med 1(1). doi:10.1038/s41746-018-0057-x.
• [45] Bychkov D, Linder N, Turkki R, Nordling S, Kovanen PE, Verrill C, et al. Deep learning based tissue analysis predicts outcome in colorectal cancer. Sci Rep 8(1). doi:10.1038/s41598-018-21758-3.
• [46] Nagpal K, Foote D, Liu Y, Chen P-HC, Wulczyn E, Tan F, et al. Development and validation of a deep learning algorithm for improving gleason scoring of prostate cancer. NPJ Digital Med 2(1). doi:10.1038/s41746-019-0112-2.
• [47] Irshad H, Veillard A, Roux L, Racoceanu D. Methods for nuclei detection, segmentation, and classification in digital histopathology: A review—current status and future potential. IEEE Rev Biomed Eng 2014;7:97–114. https://doi.org/10.1109/rbme.2013.2295804.
• [48] Aeffner F, Wilson K, Martin NT, Black JC, Hendriks CLL, Bolon B, Rudmann DG, Gianani R, Koegler SR, Krueger J, Young GD. The gold standard paradox in digital image analysis: Manual versus automated scoring as ground truth. Arch Pathol Lab Med 2017;141(9):1267–75. https://doi.org/10.5858/arpa.2016- 0386-ra.
• [49] Abels E, Pantanowitz L, Aeffner F, Zarella MD, Laak J, Bui MM, Vemuri VN, Parwani AV, Gibbs J, Agosto-Arroyo E, Beck AH, Kozlowski C. Computational pathology definitions, best practices, and recommendations for regulatory guidance: a white paper from the digital pathology association. J Pathol 2019;249(3):286–94. https://doi.org/10.1002/path.5331.
• [50] Polley M-YC, Leung SC, McShane LM, Gao D, Hugh JC, Mastropasqua MG, Viale G, Zabaglo LA, Penault-Llorca F, Bartlett JM, et al. An international ki67 reproducibility study. J Natl Cancer Inst 2013;105(24):1897–906.
• [51] Ljosa V, Sokolnicki KL, Carpenter AE. Annotated highthroughput microscopy image sets for validation. Nat Methods 2012;9(7):637. https://doi.org/10.1038/nmeth.2083.
• [52] Naylor P, Laé M, Reyal F, Walter T. Segmentation of nuclei in histopathology images by deep regression of the distance map. IEEE Trans Med Imag 2019;38(2):448–59.
• [53] Kumar N, Verma R, Sharma S, Bhargava S, Vahadane A, Sethi A. A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Trans Med Imag 2017;36(7):1550–60.
• [54] Coelho LP, Shariff A, Murphy RF. Nuclear segmentation in microscope cell images: A hand-segmented dataset and comparison of algorithms. In: IEEE International Symposium on Biomedical Imaging: From Nano to Macro. IEEE; 2009. p. 518–21. https://doi.org/10.1109/ISBI.2009.5193098. URL: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5193098.
• [55] Kaggle. https://www.kaggle.com/.
• [56] Caicedo JC, Goodman A, Karhohs KW, Cimini BA, Ackerman J, Haghighi M, Heng C, Becker T, Doan M, McQuin C, Rohban M, Singh S, Carpenter AE. Nucleus segmentation across imaging experiments: the 2018 data science bowl. Nat Methods 2019;16(12):1247–53. https://doi.org/10.1038/s41592-019-0612-7.
• [57] Aresta G, Araújo T, Kwok S, Chennamsetty SS, Safwan M, Alex V, Marami B, Prastawa M, Chan M, Donovan M, Fernandez G, Zeineh J, Kohl M, Walz C, Ludwig F, Braunewell S, Baust M, Vu QD, To MNN, Kim E, Kwak JT, Galal S, Sanchez-Freire V, Brancati N, Frucci M, Riccio D, Wang Y, Sun L, Ma K, Fang J, Kone I, Boulmane L, Campilho A, Eloy C, Polónia A, Aguiar P. BACH: Grand challenge on breast cancer histology images. Med Image Anal 2019;56:122–39. https://doi.org/10.1016/j.media.2019.05.010.
• [58] Borovec J, Kybic J, Arganda-Carreras I, Sorokin DV, Bueno G, Khvostikov AV, Bakas S, Chang EIC, Heldmann S, Kartasalo K, Latonen L, Lotz J, Noga M, Pati S, Punithakumar K, Ruusuvuori P, Skalski A, Tahmasebi N, Valkonen M, Venet L, Wang Y, Weiss N, Wodzinski M, Xiang Y, Xu Y, Yan Y, Yushkevich P, Zhao S, Muñoz-Barrutia A. Anhir: Automatic non-rigid histological image registration challenge. IEEE Trans Med Imag 2020;39(10):3042–52. https://doi.org/10.1109/TMI.2020.2986331.
• [59] Hou L, Gupta R, Arnam JSV, Zhang Y, Sivalenka K, Samaras D, Kurc TM, Saltz JH. Dataset of segmented nuclei in hematoxylin and eosin stained histopathology images of ten cancer types. Sci Data 7(1). doi:10.1038/s41597-020-0528-1.
• [60] Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L. ImageNet: A Large-Scale Hierarchical Image Database. In: CVPR09.
• [61] Everingham M, Eslami SMA, Van Gool L, Williams CKI, Winn J, Zisserman A. The pascal visual object classes challenge: A retrospective. Int J Comput Vis 2015;111(1):98–136.
• [62] Lin T-Y, Maire M, Belongie S, Bourdev L, Girshick R, Hays J, et al. Microsoft coco: Common objects in context. arXiv:1405.0312v3.
• [63] Hollandi R, Diósdi Á, Hollandi G, Moshkov N, Horváth P. AnnotatorJ: an ImageJ plugin to ease hand-annotation of cellular compartments. Doi:10.1101/2020.02.27.968362.
• [64] Humans in the Loop, The best image annotation platforms for computer vision (+ an honest review of each); 2018. URL: https://hackernoon.com/the-best-image-annotationplatforms?-for-computer-vision-an-honest-review-of-eachdac7f565fea.
• [65] Abramoff M, Magalháes P, Ram S. Image processing with imagej. Biophoton Int 2003;11:36–42.
• [66] Schneider CA, Rasband WS, Eliceiri KW. NIH image to ImageJ: 25 years of image analysis. Nat Methods 2012;9(7):671–5.
• [67] Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation; 2015. arXiv:1505.04597.
• [68] Adams R, Bischof L. Seeded region growing. IEEE Trans Pattern Anal Mach Intell 1994;16(6):641–7. https://doi.org/10.1109/34.295913.
• [69] Haralick RM, Shapiro LG. Image segmentation techniques. In: J.F. Gilmore, editor, Applications of Artificial Intelligence II, SPIE; 1985. doi:10.1117/12.948400.
• [70] Zakrzewska KE, Samluk A, Pluta KD, Pijanowska DG. Evaluation of the effects of antibiotics on cytotoxicity of EGFP and DsRed2 fluorescent proteins used for stable cell labeling. Acta Biochim Polon 61(4). doi:10.18388/abp.2014_1850.
• [71] Samluk A, Zakrzewska KE, Pluta KD. Generation of fluorescently labeled cell lines, c3a hepatoma cells, and human adult skin fibroblasts to study coculture models. Artif Organs 2013;37(7):E123–30. https://doi.org/10.1111/aor.12064.
• [72] Gomolka RS, Korzynska A, Siemion K, Gabor-Siatkowska K, Klonowski W. Automatic method for assessment of proliferation index in digital images of DLBCL tissue section. Biocybernet Biomed Eng 2019;39(1):30–7. https://doi.org/10.1016/j.bbe.2018.09.004.
• [73] Klonowski W, Korzynska A, Gomolka R. Computer analysis of histopathological images for tumor grading. Physiol Meas 2018;39(3) . https://doi.org/10.1088/1361-6579/aaa82c 034002.
• [74] Klonowski W, Korzynska A, Chwala A. Computer analysis of histopathological images for tumor grading. 2. Physiol Meas 2019;40(7) . https://doi.org/10.1088/1361-6579/ab267e 075010.
• [75] Markiewicz T, Korzynska A, Kowalski A, Swiderska-Chadaj Z, Murawski P, Grala B, Lorent M, Wdowiak M, Zak J, Roszkowiak L, Kozlowski W, Pijanowska D. MIAP – web-based platform for the computer analysis of microscopic images to support the pathological diagnosis. Biocybernet Biomed Eng 2016;36 (4):597–609. https://doi.org/10.1016/j.bbe.2016.06.006.
• [76] Litjens G, Bandi P, Bejnordi BE, Geessink O, Balkenhol M, Bult P, et al. 1399 h&e-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset. GigaScience 7 (6). doi:10.1093/gigascience/giy065.
• [77] Koohbanani NA, Jahanifar M, Tajadin NZ, Rajpoot N. Nuclick: A deep learning framework for interactive segmentation of microscopy images; 2020. arXiv:2005.14511.
• [78] Iwanowski M, Korzyńska A. Segmentation of moving cells in bright field and epi-fluorescent microscopic image sequences. In: Computer Vision and Graphics. Springer Berlin Heidelberg; 2010. p. 401–10. https://doi.org/10.1007/978-3-642-15910-7_46.
• [79] Korzynska A, Jurga M, Domanska-Janik K, Strojny W, Wloskowicz D. Analysis of stem cell clonal growth. In: Advances in Soft Computing. Springer Berlin Heidelberg; 2005. p. 577–84. https://doi.org/10.1007/3-540-32390-2_68.
• [80] Zychowicz M, Mehn D, Ruiz A, Colpo P, Rossi F, Frontczak-Baniewicz M, Domanska-Janik K, Buzanska L. Proliferation capacity of cord blood derived neural stem cell line on different micro-scale biofunctional domains. Acta neurobiologiae experimentalis 2011;71:12–23.
• [81] Bankhead P, Loughrey MB, Fernández JA, Dombrowski Y, McArt DG, Dunne PD, et al. QuPath: Open source software for digital pathology image analysis. Sci Rep 7(1). doi:10.1038/s41598-017-17204-5.
• [82] Korzynska A, Zak J, Siemion K, Roszkowiak L, Pijanowska D. CNN support to diagnostics in sjögren’s syndrome. In: Advances in Intelligent Systems and Computing. Springer International Publishing; 2019. p. 72–81. https://doi.org/10.1007/978-3-030-29885-2_7.
• [83] Hughes AJ, Mornin JD, Biswas SK, Beck LE, Bauer DP, Raj A, Bianco S, Gartner ZJ. Quanti.us: a tool for rapid, flexible, crowd-based annotation of images. Nat Methods 2018;15 (8):587–90. https://doi.org/10.1038/s41592-018-0069-0.
• [84] Ipeirotis PG, Provost F, Wang J. Quality management on amazon mechanical turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation - HCOMP10. ACM Press; 2010. doi:10.1145/1837885.1837906.
• [85] Iakovidis DK, Goudas T, Smailis C, Maglogiannis I. Ratsnake: A versatile image annotation tool with application to computer-aided diagnosis. Sci World J 2014;2014:1–12. https://doi.org/10.1155/2014/286856.
• [86] Russell BC, Torralba A, Murphy KP, Freeman WT. LabelMe: A database and web-based tool for image annotation. Int J Comput Vis 2007;77(1–3):157–73. https://doi.org/10.1007/ s11263-007-0090-8.
• [87] Torralba A, Russell BC, Yuen J, LabelMe. Online image annotation and applications. Proc IEEE 2010;98(8):1467–84. https://doi.org/10.1109/jproc.2010.2050290.
• [88] Neuman U, Korzyńska A, Lopez C, Łejeune M, Roszkowiak Lstrok;, Bosch R. Equalisation of archival microscopic images from immunohistochemically stained tissue sections. Biocybernet Biomed Eng 2013;33(1):63–76. https://doi.org/ 10.1016/s0208-5216(13)70056-1.
• [89] Korzynska A, Roszkowiak L, Siemion K, Zak J, Zakrzewska K, Samluk A, Wencel A, Pluta K, Pijanowska D. The analysis of the shape of the genetically modified human skin fibroblasts in culture. In: Recent Developments and Achievements in Biocybernetics and Biomedical Engineering. Springer International Publishing; 2017. p. 98–109. https://doi.org/10.1007/978-3-319-66905-2_8.
• [90] Korzynska A, Siemion K, Zak J, Roszkowiak L, Wencel A, Pluta K, Pijanowska D. The analysis of the movement of the genetically modified human skin fibroblasts in culture. In: 2018 IEEE International Conference on Imaging Systems and Techniques (IST). IEEE; 2018. https://doi.org/10.1109/ist.2018.8577087.
• [91] Computational Pathology Group, Asap. https://github.com/computationalpathologygroup/ASAP.
• [92] Ruifrok A, Johnston D. Quantification of histochemical staining by color deconvolution. Anal Quant Cytol Histol 23.
• [93] The GIMP Development Team, Gimp. https://www.gimp.org.
• [94] Eagleman DM. Visual illusions and neurobiology. Nat Rev Neurosci 2001;2(12):920–6. https://doi.org/10.1038/35104092.
• [95] Korzynska A. Neutrophils movementin vitro. Ann New York Acad Sci 2002;972(1):139–43. https://doi.org/10.1111/j.1749-6632.2002.tb04564.x.
• [96] Janowczyk A, Doyle S, Gilmore H, Madabhushi A. A resolution adaptive deep hierarchical (RADHicaL) learning scheme applied to nuclear segmentation of digital pathology images. Comput Methods Biomech Biomed Eng Imag Visual 2016;6 (3):270–6. https://doi.org/10.1080/21681163.2016.1141063.
• [97] Siemion K, Zak J, Roszkowiak L, Korzynska A. The proliferation index estimation with metpiki software in meningioma tissue samples. In: Virchows Archiv, vol. 473. 2018; p. 107.