Bone marrow examination is an essential step in both diagnosing and managing hematologic disorders. However, there are several challenges to perform bone marrow analysis on whole-slide images (WSIs), such as large dimensions of data to process and the complexity of the nature of such analysis, like numerous cell types to identify and small inter-class difference within a maturation stage among various cell types. To the authors' best knowledge, this is the first study on fully automatic bone marrow analysis using WSIs. In this study, we develop an efficient and fully automatic hierarchical deep learning framework for bone marrow WSI analysis in seconds. The proposed hierarchical framework consists of (1) a deep learning model for rapid localization of bone marrow particles and cellular trails generating regions of interest for further analysis, (2) a patch-based deep learning model for cell identification of 15 cell types and (3) a fast stitching model for integrating patch-based results and producing final outputs. In evaluation, the proposed method achieves high recall, accuracy and the area under the precision-recall curve (PR-AUC) metrics of 0.959, 0.992 and 0.972, respectively, and takes only 43 seconds to perform bone marrow analysis for a WSI. In comparison with the small-image-based benchmark methods, the proposed method demonstrates superior performance in recall, accuracy, PR-AUC and computational time. The proposed fully automatic method is demonstrated to be effective and efficient in bone marrow cell analysis of WSIs for routine clinical usage.

Bone marrow examination is an essential step in both diagnosing and managing hematologic disorders. However, there are several challenges to perform bone marrow analysis on whole-slide images (WSIs), such as large dimensions of data to process and the complexity of the nature of such analysis, like numerous cell types to identify and small inter-class difference within a maturation stage among various cell types. To the authors' best knowledge, this is the first study on fully automatic bone marrow analysis using WSIs. In this study, we develop an efficient and fully automatic hierarchical deep learning framework for bone marrow WSI analysis in seconds. The proposed hierarchical framework consists of (1) a deep learning model for rapid localization of bone marrow particles and cellular trails generating regions of interest for further analysis, (2) a patch-based deep learning model for cell identification of 15 cell types and (3) a fast stitching model for integrating patch-based results and producing final outputs. In evaluation, the proposed method achieves high recall, accuracy and the area under the precision-recall curve (PR-AUC) metrics of 0.959, 0.992 and 0.972, respectively, and takes only 43 seconds to perform bone marrow analysis for a WSI. In comparison with the small-image-based benchmark methods, the proposed method demonstrates superior performance in recall, accuracy, PR-AUC and computational time. The proposed fully automatic method is demonstrated to be effective and efficient in bone marrow cell analysis of WSIs for routine clinical usage.

[1] C. Reta, L. Altamirano, J. A. Gonzalez, R. Diaz-Hernandez, H. Peregrina,
I. Olmos, J. E. Alonso, and R. Lobato, “Segmentation and classification of
bone marrow cells images using contextual information for medical diagnosis of
acute leukemias,” PloS one, vol. 10, no. 6, 2015.
[2] P. Ghosh, D. Bhattacharjee, and M. Nasipuri, “Blood smear analyzer for white
blood cell counting: a hybrid microscopic image analyzing technique,” Applied
Soft Computing, vol. 46, pp. 629–638, 2016.
[3] T. Karthikeyan and N. Poornima, “Microscopic image segmentation using fuzzy
c means for leukemia diagnosis,” Leukemia, vol. 4, no. 1, pp. 3136–3142, 2017.
[4] S. Mishra, B. Majhi, P. K. Sa, and L. Sharma, “Gray level co-occurrence matrix
and random forest based acute lymphoblastic leukemia detection,” Biomedical
Signal Processing and Control, vol. 33, pp. 272–280, 2017.
[5] J. W. Choi, Y. Ku, B. W. Yoo, J.-A. Kim, D. S. Lee, Y. J. Chai, H.-J. Kong,
and H. C. Kim, “White blood cell differential count of maturation stages in
bone marrow smear using dual-stage convolutional neural networks,” PloS one,
vol. 12, no. 12, 2017.
[6] K. Kimura, Y. Tabe, T. Ai, I. Takehara, H. Fukuda, H. Takahashi, T. Naito,
N. Komatsu, K. Uchihashi, and A. Ohsaka, “A novel automated image analysis
system using deep convolutional neural networks can assist to differentiate mds
and aa,” Scientific reports, vol. 9, no. 1, pp. 1–9, 2019.
[7] S. Xie, R. Girshick, P. Dollr, Z. Tu, and K. He, “Aggregated residual trans-
formations for deep neural networks,” in 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 5987–5995, July 2017.
[8] A. T. Sahlol, P. Kollmannsberger, and A. A. Ewees, “Efficient classification of
white blood cell leukemia with improved swarm optimization of deep features,”
Scientific Reports, vol. 10, no. 1, pp. 1–11, 2020.
[9] Y. Xie, F. Xing, X. Shi, X. Kong, H. Su, and L. Yang, “Efficient and robust cell
detection: A structured regression approach,” Medical Image Analysis, vol. 44,
pp. 245 – 254, 2018.
[10] R. Chandradevan, A. Aljudi, B. Drumheller, N. Kunananthaseelan, M. Amgad,
D. Gutman, L. Cooper, and D. Jaye, “Machine-based detection and classifica-
tion for bone marrow aspirate differential counts: initial development focusing
on nonneoplastic cells,” Laboratory Investigation, vol. 100, 09 2019.
[11] S.-H. Lee, W. Erber, A. Porwit, M. Tomonaga, L. Peterson, and I. C. S. I.
Hematology, “Icsh guidelines for the standardization of bone marrow specimens
and reports,” International journal of laboratory hematology, vol. 30, no. 5,
pp. 349–364, 2008.
[12] H. Döhner, E. Estey, D. Grimwade, S. Amadori, F. R. Appelbaum, T. Büchner,
H. Dombret, B. L. Ebert, P. Fenaux, R. A. Larson, et al., “Diagnosis and
management of aml in adults: 2017 eln recommendations from an international
expert panel,” Blood, vol. 129, no. 4, pp. 424–447, 2017.
[13] S. Swerdlow, E. Campo, N. L. Harris, E. Jaffe, S. Pileri, H. Stein, J. Thiele,
D. Arber, R. Hasserjian, M. Le Beau, et al., “Who classification of tumours
of haematopoietic and lymphoid tissues (revised 4th edition),” IARC: Lyon,
vol. 421, 2017.
[14] P. L. Greenberg, H. Tuechler, J. Schanz, G. Sanz, G. Garcia-Manero, F. Sol,
J. M. Bennett, D. Bowen, P. Fenaux, F. Dreyfus, H. Kantarjian, A. Kuendgen,
A. Levis, L. Malcovati, M. Cazzola, J. Cermak, C. Fonatsch, M. M. Le Beau,
M. L. Slovak, O. Krieger, M. Luebbert, J. Maciejewski, S. M. M. Magalhaes,
Y. Miyazaki, M. Pfeilstcker, M. Sekeres, W. R. Sperr, R. Stauder, S. Tauro,
P. Valent, T. Vallespi, A. A. van de Loosdrecht, U. Germing, and D. Haase, “Re-
vised International Prognostic Scoring System for Myelodysplastic Syndromes,”
Blood, vol. 120, pp. 2454–2465, 09 2012.
[15] S. Kumar, B. Paiva, K. C. Anderson, B. Durie, O. Landgren, P. Moreau,
N. Munshi, S. Lonial, J. Bladé, M.-V. Mateos, et al., “International myeloma
working group consensus criteria for response and minimal residual disease as-
sessment in multiple myeloma,” The lancet oncology, vol. 17, no. 8, pp. e328–
e346, 2016.
[16] J. M. Bennett, D. Catovsky, M.-T. Daniel, G. Flandrin, D. A. Galton, H. R.
Gralnick, and C. Sultan, “Proposals for the classification of the acute leukaemias
french-american-british (fab) co-operative group,” British journal of haematol-
ogy, vol. 33, no. 4, pp. 451–458, 1976.
[17] L. Meintker, J. Ringwald, M. Rauh, and S. W. Krause, “Comparison of auto-
mated differential blood cell counts from abbott sapphire, siemens advia 120,
beckman coulter dxh 800, and sysmex xe-2100 in normal and pathologic sam-
ples,” American journal of clinical pathology, vol. 139, no. 5, pp. 641–650, 2013.
[18] X. Fuentes-Arderiu and D. Dot-Bach, “Measurement uncertainty in manual
differential leukocyte counting,” Clinical Chemistry and Laboratory Medicine
(CCLM), vol. 47, no. 1, pp. 112–115, 2009.
[19] C. Matek, S. Schwarz, K. Spiekermann, and C. Marr, “Human-level recognition
of blast cells in acute myeloid leukaemia with convolutional neural networks,”
Nature Machine Intelligence, vol. 1, no. 11, pp. 538–544, 2019.
[20] B. S. Wilkins, W. N. Erber, D. Bareford, G. Buck, K. Wheatley, C. L. East,
B. Paul, C. N. Harrison, A. R. Green, and P. J. Campbell, “Bone marrow
pathology in essential thrombocythemia: interobserver reliability and utility
for identifying disease subtypes,” Blood, vol. 111, pp. 60–70, 01 2008.
[21] T.-C. Yu, W.-C. Chou, C.-Y. Yeh, C.-K. Yang, S.-C. Huang, F. M. Tien, C.-Y.
Yao, C.-L. Cheng, M.-K. Chuang, H.-F. Tien, Q.-Y. Zhang, W.-H. Hsu, and
S.-C. Chou, “Automatic bone marrow cell identification and classification by
deep neural network,” Blood, vol. 134, pp. 2084–2084, 11 2019.
[22] D. Bychkov, N. Linder, R. Turkki, S. Nordling, P. E. Kovanen, C. Verrill,
M. Walliander, M. Lundin, C. Haglund, and J. Lundin, “Deep learning based
tissue analysis predicts outcome in colorectal cancer,” Scientific reports, vol. 8,
no. 1, pp. 1–11, 2018.
[23] G. Campanella, M. G. Hanna, L. Geneslaw, A. Miraflor, V. W. K. Silva, K. J.
Busam, E. Brogi, V. E. Reuter, D. S. Klimstra, and T. J. Fuchs, “Clinical-grade
computational pathology using weakly supervised deep learning on whole slide
images,” Nature medicine, vol. 25, no. 8, pp. 1301–1309, 2019.
[24] J. D. Ianni, R. E. Soans, S. Sankarapandian, R. V. Chamarthi, D. Ayyagari,
T. G. Olsen, M. J. Bonham, C. C. Stavish, K. Motaparthi, C. J. Cockerell, et al.,
“Tailored for real-world: A whole slide image classification system validated
on uncurated multi-site data emulating the prospective pathology workload,”
Scientific reports, vol. 10, no. 1, pp. 1–12, 2020.
[25] O. Iizuka, F. Kanavati, K. Kato, M. Rambeau, K. Arihiro, and M. Tsuneki,
“Deep learning models for histopathological classification of gastric and colonic
epithelial tumours,” Scientific Reports, vol. 10, no. 1, pp. 1–11, 2020.
[26] H. Lin, H. Chen, S. Graham, Q. Dou, N. Rajpoot, and P. Heng, “Fast scannet:
Fast and dense analysis of multi-gigapixel whole-slide images for cancer metas-
tasis detection,” IEEE Transactions on Medical Imaging, vol. 38, pp. 1948–
1958, Aug 2019.
[27] M. Peikari, M. J. Gangeh, J. Zubovits, G. Clarke, and A. L. Martel, “Triaging
diagnostically relevant regions from pathology whole slides of breast cancer:
A texture based approach,” IEEE Transactions on Medical Imaging, vol. 35,
pp. 307–315, Jan 2016.
[28] Z. Zhang, P. Chen, M. McGough, F. Xing, C. Wang, M. Bui, Y. Xie, M. Sap-
kota, L. Cui, J. Dhillon, et al., “Pathologist-level interpretable whole-slide can-
cer diagnosis with deep learning,” Nature Machine Intelligence, vol. 1, no. 5,
pp. 236–245, 2019.
[29] G. Chlebus, A. Schenk, J. H. Moltz, B. van Ginneken, H. K. Hahn, and
H. Meine, “Automatic liver tumor segmentation in ct with fully convolutional
neural networks and object-based postprocessing,” Scientific reports, vol. 8,
no. 1, pp. 1–7, 2018.
[30] J. De Fauw, J. R. Ledsam, B. Romera-Paredes, S. Nikolov, N. Tomasev,
S. Blackwell, H. Askham, X. Glorot, B. O’Donoghue, D. Visentin, et al.,
“Clinically applicable deep learning for diagnosis and referral in retinal dis-
ease,” Nature medicine, vol. 24, no. 9, pp. 1342–1350, 2018.
[31] J. Liu, H. Liu, Z. Tang, W. Gui, T. Ma, S. Gong, Q. Gao, Y. Xie, and J. P.
Niyoyita, “Iouc-3dsfcnn: Segmentation of brain tumors via iou constraint 3d
symmetric full convolution network with multimodal auto-context,” Scientific
Reports, vol. 10, no. 1, pp. 1–15, 2020.
[32] Y. Sun, S. Zhu, K. Ma, W. Liu, Y. Yue, G. Hu, H. Lu, and W. Chen, “Iden-
tification of 12 cancer types through genome deep learning,” Scientific reports,
vol. 9, no. 1, pp. 1–9, 2019.
[33] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521,
no. 7553, pp. 436–444, 2015.
[34] B. Hu, Y. Tang, E. I. Chang, Y. Fan, M. Lai, and Y. Xu, “Unsupervised learn-
ing for cell-level visual representation in histopathology images with generative
adversarial networks,” IEEE Journal of Biomedical and Health Informatics,
vol. 23, pp. 1316–1328, May 2019.
[35] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair,
A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neu-
ral Information Processing Systems 27 (Z. Ghahramani, M. Welling, C. Cortes,
N. D. Lawrence, and K. Q. Weinberger, eds.), pp. 2672–2680, Curran Asso-
ciates, Inc., 2014.
[36] Z. Cai and N. Vasconcelos, “Cascade r-cnn: High quality object detection and
instance segmentation,” IEEE Transactions on Pattern Analysis and Machine
Intelligence, pp. 1–1, 2019.
[37] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time
object detection with region proposal networks,” IEEE Transactions on Pattern
Analysis and Machine Intelligence, vol. 39, pp. 1137–1149, June 2017.
[38] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recogni-
tion,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 770–778, June 2016.
[39] E. F. Glassy, Color Atlas of Hematology: An Illustrated Field Guide Based on
Proficiency Testing. College of American Patholgists, 1998.
[40] C. F. Buckens, P. A. de Jong, C. Mol, E. Bakker, H. P. Stallman, W. P. Mali,
Y. van der Graaf, and H. M. Verkooijen, “Intra and interobserver reliability and
agreement of semiquantitative vertebral fracture assessment on chest computed
tomography,” PloS one, vol. 8, no. 8, p. e71204, 2013.
[41] U. Gianelli, A. Bossi, I. Cortinovis, E. Sabattini, C. Tripodo, E. Boveri,
A. Moro, R. Valli, M. Ponzoni, A. M. Florena, et al., “Reproducibility of the
who histological criteria for the diagnosis of philadelphia chromosome-negative
myeloproliferative neoplasms,” Modern Pathology, vol. 27, no. 6, pp. 814–822,
2014.
[42] B. Birkhead, D. Salt, and R. Jackson, “Reliability and consistency of bone
marrow assessments of complete remission in acute myelogenous leukaemia,”
Hematological oncology, vol. 5, no. 2, pp. 139–145, 1987.