Deep learning algorithms have proven to be extremely successful in image analysis. However, such algorithms require immense amounts of training data that are rarely available in healthcare. Lack of data sets, privacy concerns, legal restrictions, and inconsistent data collection methods in medical imaging have necessitated image synthesis. Nevertheless, synthesizing realistic images is a challenging task because medical images are often three-dimensional (3D) and contain a large number of voxels compared to two-dimensional images. Generative Adversarial Networks (GAN) are a commonly used approach in medical image synthesis. However, they have some limitations, e.g., mode collapse and unstable training are some of their main challenges. Recently, denoising diffusion probabilistic models (DDPMs) have emerged as a powerful family of generative models without adversarial training, which show superior performance and are widely studied in various image processing tasks. In this work, we propose a novel 3D DDPM - a conditional denoising diffusion probabilistic model for three-dimensional map-to-image synthesis. Through simple modifications using the conditioning image as a prior to the training and sampling process, we show that DDPM is capable of translating 3D images and generating high-quality images covering a wide variety. We also propose a new hybrid loss function that completely removes background noise and provides sharp images. In addition, this study presents an extended version of evaluation metrics that is more suitable for assessing image quality and highly correlated with human evaluation. In both qualitative and quantitative aspects, the experiments show that the proposed 3D-DDPM produces realistic images of high quality and outperforms state-of-the-art methods for medical image translation tasks.

[1] M. M. Kalan Farmanfarma et al., "Brain cancer in the world: an epidemiological review," World Cancer Research Journal, vol. 6: e1356, 2019, doi: 10.32113/wcrj_20197_1356.
[2] Z. Khazaei et al., "Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide stomach cancers and their relationship with the human development index (HDI)," World Cancer Research Journal, vol. 6, 08/03 2019, doi: 10.32113/wcrj_20194_1257.
[3] H. Sung et al., "Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries," CA: A Cancer Journal for Clinicians, 02/04 2021, doi: 10.3322/caac.21660.
[4] M. Angulakshmi and L. P. G G, "Automated brain tumour segmentation techniques- A review," International Journal of Imaging Systems and Technology, vol. 27, pp. 66-77, 03/01 2017, doi: 10.1002/ima.22211.
[5] N. Micallef et al., "A nested U-Net approach for brain tumour segmentation," in 2020 IEEE 20th Mediterranean Electrotechnical Conference ( MELECON), 16-18 June 2020 2020, pp. 376-381, doi: 10.1109/MELECON48756.2020.9140550.
[6] L. Gao et al., "Handling imbalanced medical image data: A deep-learning-based one-class classification approach," (in eng), Artif Intell Med, vol. 108, p. 101935, Aug 2020, doi:10.1016/j.artmed.2020.101935.
[7] H.-C. Shin et al., "Medical image synthesis for data augmentation and anonymization using generative adversarial networks," ArXiv, vol. abs/1807.10225, 2018.
[8] D. Patel et al., "Probabilistic recovery of missing images in contrast-enhanced CT," in NeuroIPS, 2020.
[9] C. Bowles et al., "Modelling the progression of Alzheimer's disease in MRI using generative adversarial networks" (SPIE Medical Imaging), SPIE, 2018.
[10] L. Sun, J. Wang, Y. Huang, X. Ding, H. Greenspan, and J. Paisley, "An Adversarial Learning Approach to Medical Image Synthesis for Lesion Detection," IEEE journal of biomedical and health informatics, vol. PP, 01/06 2020, doi: 10.1109/JBHI.2020.2964016.
[11] W. Li et al., "Magnetic resonance image (MRI) synthesis from brain computed tomography (CT) images based on deep learning methods for magnetic resonance (MR)-guided radiotherapy," (in eng), Quant Imaging Med Surg, vol. 10, no. 6, pp. 1223-1236, 2020, doi: 10.21037/qims-19-885.
[12] Y. Skandarani, P.-M. Jodoin, and A. Lalande, GANs for Medical Image Synthesis: An Empirical Study. 2021.
[13] T. Wang et al., "A review on medical imaging synthesis using deep learning and its clinical applications," (in eng), J Appl Clin Med Phys, vol. 22, no. 1, pp. 11-36, Jan 2021, doi: 10.1002/acm2.13121.
[14] I. Goodfellow et al., "Generative Adversarial Networks," Advances in Neural Information Processing Systems, vol. 3, 06/10 2014, doi: 10.1145/3422622.
[15] D. Saxena and J. Cao, Generative Adversarial Networks (GANs): Challenges, Solutions, and Future Directions. 2020.
[16] L. Wang, W. Chen, W. Yang, F. Bi, and F. R. Yu, "A State-of-the-Art Review on Image Synthesis With Generative Adversarial Networks," IEEE Access, vol. 8, pp. 63514-63537, 2020, doi: 10.1109/ACCESS.2020.2982224.
[17] P. Dhariwal and A. Nichol, Diffusion Models Beat GANs on Image Synthesis. 2021.
[18] J. Ho, A. Jain, and P. Abbeel, Denoising Diffusion Probabilistic Models. 2020.
[19] S. Bond-Taylor, A. Leach, Y. Long, and C. G. Willcocks, "Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models," IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021, doi: 10.1109/TPAMI.2021.3116668.
[20] Q. Wu, R. Gao, and H. Zha, "Bridging Explicit and Implicit Deep Generative Models via Neural Stein Estimators," in NeurIPS, 2021.
[21] D. Kingma and M. Welling, Auto-Encoding Variational Bayes. 2014.
[22] H. Larochelle and I. Murray, "The Neural Autoregressive Distribution Estimator," Journal of Machine Learning Research - Proceedings Track, vol. 15, pp. 29-37, 01/01 2011.
[23] G. Alain et al., "GSNs : Generative Stochastic Networks," Information and Inference, vol. 5, 03/18 2015, doi: 10.1093/imaiai/iaw003.
[24] R. van de Schoot et al., "Bayesian statistics and modelling," Nature Reviews Methods Primers, vol. 1, no. 1, p. 1, 2021/01/14 2021, doi: 10.1038/s43586-020-00001-2.
[25] P. Baldi, "Autoencoders, unsupervised learning and deep architectures," presented at the Proceedings of the 2011 International Conference on Unsupervised and Transfer Learning workshop - Volume 27, Washington, USA, 2011.
[26] A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum Likelihood from Incomplete Data via the EM Algorithm," Journal of the Royal Statistical Society. Series B (Methodological), vol. 39, no. 1, pp. 1-38, 1977. [Online]. Available: http://www.jstor.org/stable/2984875.
[27] J. M. Joyce, "Kullback-Leibler Divergence," in International Encyclopedia of Statistical Science, M. Lovric Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 720-722.
[28] A. Oord, O. Vinyals, and K. Kavukcuoglu, "Neural Discrete Representation Learning," 11/02 2017.
[29] I. Higgins et al., "beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework," in ICLR, 2017.
[30] H. Akrami, A. Joshi, J. Li, and R. Leahy, Robust Variational Autoencoder. 2019.
[31] A. Pagnoni, K. Liu, and S. Li, Conditional Variational Autoencoder for Neural Machine Translation. 2018.
[32] R. Wei, C. Garcia, A. ElSayed, V. Peterson, and A. Mahmood, "Variations in Variational Autoencoders - A Comparative Evaluation," IEEE Access, vol. PP, pp. 1-1, 08/20 2020, doi: 10.1109/ACCESS.2020.3018151.
[33] Z. Liu, P. Luo, X. Wang, and X. Tang, "Deep Learning Face Attributes in the Wild," 11/28 2014, doi: 10.1109/ICCV.2015.425.
[34] K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition Supplementary Materials," 2016.
[35] L. Cai, H. Gao, and S. Ji, "Multi-Stage Variational Auto-Encoders for Coarse-to-Fine Image Generation," 2019, pp. 630-638.
[36] K. Kurach, M. Lucic, X. Zhai, M. Michalski, and S. Gelly, The GAN Landscape: Losses, Architectures, Regularization, and Normalization. 2018.
[37] A. Abu-Srhan, M. A. M. Abushariah, and O. S. Al-Kadi, "The effect of loss function on conditional generative adversarial networks," Journal of King Saud University - Computer and Information Sciences, 2022/03/04/ 2022, doi: https://doi.org/10.1016/j.jksuci.2022.02.018.
[38] M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein GAN," 01/26 2017.
[39] T. Karras, T. Aila, S. Laine, and J. Lehtinen, "Progressive Growing of GANs for Improved Quality, Stability, and Variation," 10/27 2017.
[40] E. Fetaya, J.-H. Jacobsen, W. Grathwohl, and R. S. Zemel, "Understanding the Limitations of Conditional Generative Models," arXiv: Learning, 2020.
[41] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics," 03/12 2015.
[42] J. Song, C. Meng, and S. Ermon, "Denoising Diffusion Implicit Models," ArXiv, vol. abs/2010.02502, 2021.
[43] A. Nichol and P. Dhariwal, "Improved Denoising Diffusion Probabilistic Models," ArXiv, vol. abs/2102.09672, 2021.
[44] Z. Xiao, K. Kreis, and A. Vahdat, "Tackling the Generative Learning Trilemma with Denoising Diffusion GANs," ArXiv, vol. abs/2112.07804, 2021.
[45] Z. Xiao, K. Kreis, and A. Vahdat, Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. 2021.
[46] J. Wolleb, R. Sandkuehler, F. Bieder, P. Valmaggia, and P. Cattin, Diffusion Models for Implicit Image Segmentation Ensembles. 2021.
[47] Y. Xie and Q. Li, "Measurement-conditioned Denoising Diffusion Probabilistic Model for Under-sampled Medical Image Reconstruction," 2022, doi: 10.48550/ARXIV.2203.03623.
[48] J. Wyatt, A. Leach, S. M. Schmon, and C. G. Willcocks, "AnoDDPM: Anomaly Detection With Denoising Diffusion Probabilistic Models Using Simplex Noise," presented at the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2022/6, 2022. [Online]. Available: http://dro.dur.ac.uk/36134/.
[49] H.-C. Shin et al., "Medical Image Synthesis for Data Augmentation and Anonymization Using Generative Adversarial Networks," in Simulation and Synthesis in Medical Imaging, Cham, A. Gooya, O. Goksel, I. Oguz, and N. Burgos, Eds., 2018// 2018: Springer International Publishing, pp. 1-11.
[50] H. Zhang, Z. Huang, and Z. Lv, "Medical Image Synthetic Data Augmentation Using GAN," presented at the Proceedings of the 4th International Conference on Computer Science and Application Engineering, Sanya, China, 2020. [Online]. Available: https://doi.org/10.1145/3424978.3425118.
[51] M. Moreno López, J. M. Frederick, and J. Ventura, "Evaluation of MRI Denoising Methods Using Unsupervised Learning," (in English), Frontiers in Artificial Intelligence, Original Research vol. 4, 2021-June-04 2021, doi: 10.3389/frai.2021.642731.
[52] Y. Zhang, W. Zhang, Q. Zhang, J.-J. Yang, X. Chen, and S. Zhao, CMR motion artifact correction using generative adversarial nets. 2019.
[53] X. Zhang, C. Feng, A. Wang, L. Yang, and Y. Hao, "CT super-resolution using multiple dense residual block based GAN," Signal, Image and Video Processing, vol. 15, no. 4, pp. 725-733, 2021/06/01 2021, doi: 10.1007/s11760-020-01790-5.
[54] W. Li et al., "Magnetic resonance image (MRI) synthesis from brain computed tomography (CT) images based on deep learning methods for magnetic resonance (MR)-guided radiotherapy," (in eng), Quant Imaging Med Surg, vol. 10, no. 6, pp. 1223-1236, Jun 2020, doi: 10.21037/qims-19-885.
[55] A. Sharma and G. Hamarneh, "Missing MRI Pulse Sequence Synthesis Using Multi-Modal Generative Adversarial Network," IEEE Transactions on Medical Imaging, vol. PP, pp. 1-1, 10/04 2019, doi: 10.1109/TMI.2019.2945521.
[56] X. Yi, E. Walia, and P. Babyn, "Generative adversarial network in medical imaging: A review," Medical Image Analysis, vol. 58, p. 101552, 2019/12/01/ 2019, doi: https://doi.org/10.1016/j.media.2019.101552.
[57] A. Radford, L. Metz, and S. Chintala, "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks," CoRR, vol. abs/1511.06434, 2016.
[58] M. M. Saad, M. H. Rehmani, and R. O'Reilly, "Addressing the Intra-class Mode Collapse Problem using Adaptive Input Image Normalization in GAN-based X-ray Images," arXiv preprint arXiv:2201.10324, 2022.
[59] J.-Y. Zhu, T. Park, P. Isola, and A. Efros, Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. 2017, pp. 2242-2251.
[60] A. Almahairi, S. Rajeshwar, A. Sordoni, P. Bachman, and A. Courville, "Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data," presented at the Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2018. [Online]. Available: https://proceedings.mlr.press/v80/almahairi18a.html.
[61] Y. Lei et al., "MRI-only based synthetic CT generation using dense cycle consistent generative adversarial networks," (in eng), Med Phys, vol. 46, no. 8, pp. 3565-3581, Aug 2019, doi: 10.1002/mp.13617.
[62] Y. Pan, M. Liu, C. Lian, T. Zhou, and Y. Xia, "Synthesizing Missing PET from MRI with Cycle-consistent Generative Adversarial Networks for Alzheimer’s Disease Diagnosis: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part III," vol. 11072, 2018, pp. 455-463.
[63] K. Gong, J. Yang, K. Kim, G. El Fakhri, Y. Seo, and Q. Li, "<strong>Attenuation Correction of PET/MR Using Cycle-Consistent Adversarial Network</strong>," Journal of Nuclear Medicine, vol. 60, no. supplement 1, pp. 171-171, 2019.
[64] Z. Shen, Y. Chen, S. K. Zhou, B. Georgescu, and X. Liu, One-to-one Mapping for Unpaired Image-to-image Translation. 2020, pp. 1159-1168.
[65] H. Yang et al., "Unsupervised MR-to-CT Synthesis Using Structure-Constrained CycleGAN," IEEE Transactions on Medical Imaging, vol. 39, no. 12, pp. 4249-4261, 2020, doi: 10.1109/TMI.2020.3015379.
[66] P. Isola, J. Zhu, T. Zhou, and A. A. Efros, "Image-to-Image Translation with Conditional Adversarial Networks," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 21-26 July 2017 2017, pp. 5967-5976, doi: 10.1109/CVPR.2017.632.
[67] F. Lau, T. Hendriks, J. Lieman-Sifry, B. Norman, S. Sall, and D. Golden, ScarGAN: Chained Generative Adversarial Networks to Simulate Pathological Tissue on Cardiovascular MR Scans. 2018.
[68] H.-C. Shin et al., "Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks," in SASHIMI@MICCAI, 2018.
[69] A. Qasim et al., "Red-GAN: Attacking class imbalance via conditioned generation. Yet another perspective on medical image synthesis for skin lesion dermoscopy and brain tumor MRI," Machine Learning Research, vol. 121, pp. 655-668, 07/06 2020.
[70] A. Vaswani et al., "Attention is all you need," presented at the Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, California, USA, 2017.
[71] S. Zagoruyko and N. Komodakis, "Wide Residual Networks," 05/23 2016.
[72] Y. Wu and K. He, "Group Normalization," 03/22 2018.
[73] S.-B. Hong et al., "3D-StyleGAN: A Style-Based Generative Adversarial Network for Generative Modeling of Three-Dimensional Medical Images," ArXiv, vol. abs/2107.09700, 2021.
[74] S.-L. Lu, H.-C. Liao, F.-M. Hsu, C.-C. Liao, F. Lai, and F. Xiao, "The intracranial tumor segmentation challenge: Contour tumors on brain MRI for radiosurgery," NeuroImage, vol. 244, p. 118585, 2021/12/01/ 2021, doi: https://doi.org/10.1016/j.neuroimage.2021.118585.
[75] U. Baid et al., "The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification," ArXiv, vol. abs/2107.02314, 2021.
[76] A. Dagley et al., "Harvard Aging Brain Study: Dataset and accessibility," (in eng), Neuroimage, vol. 144, no. Pt B, pp. 255-258, Jan 2017, doi: 10.1016/j.neuroimage.2015.03.069.
[77] A. V. Dalca, J. V. Guttag, and M. R. Sabuncu, "Anatomical Priors in Convolutional Networks for Unsupervised Biomedical Segmentation," 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9290-9299, 2018.
[78] A. di Martino et al., "The Autism Brain Imaging Data Exchange: Towards Large-Scale Evaluation of the Intrinsic Brain Architecture in Autism," Molecular psychiatry, vol. 19, pp. 659 - 667, 2014.
[79] R. L. Gollub et al., "The MCIC Collection: A Shared Repository of Multi-Modal, Multi-Site Brain Image Data from a Clinical Investigation of Schizophrenia," Neuroinformatics, vol. 11, pp. 367-388, 2013.
[80] A. J. Holmes et al., "Brain Genomics Superstruct Project initial data release with structural, functional, and behavioral measures," Scientific Data, vol. 2, 2015.
[81] D. S. Marcus, T. H. Wang, J. Parker, J. G. Csernansky, J. C. Morris, and R. L. Buckner, "Open Access Series of Imaging Studies (OASIS): Cross-sectional MRI Data in Young, Middle Aged, Nondemented, and Demented Older Adults," Journal of Cognitive Neuroscience, vol. 19, pp. 1498-1507, 2007.
[82] K. Marek, D. Jennings, G. Tamagnan, and J. Seibyl, "Biomarkers for Parkinson's [corrected] disease: tools to assess Parkinson's disease onset and progression," (in eng), Ann Neurol, vol. 64 Suppl 2, pp. S111-21, Dec 2008, doi: 10.1002/ana.21602.
[83] "The ADHD-200 Consortium: A Model to Advance the Translational Potential of Neuroimaging in Clinical Neuroscience," (in eng), Front Syst Neurosci, vol. 6, p. 62, 2012, doi: 10.3389/fnsys.2012.00062.
[84] S. G. Mueller et al., "Ways toward an early diagnosis in Alzheimer's disease: the Alzheimer's Disease Neuroimaging Initiative (ADNI)," (in eng), Alzheimers Dement, vol. 1, no. 1, pp. 55-66, Jul 2005, doi: 10.1016/j.jalz.2005.06.003.
[85] S. Sara, S. Bara, A. Hammouch, and B. Cherradi, A robust comparative study of five brain extraction algorithms: (BET; BSE; McStrip; SPM2; TMBE). 2014.
[86] F. Oliveira and J. Tavares, "Medical image registration: A review," Computer methods in biomechanics and biomedical engineering, vol. 17, pp. 73-93, 01/25 2014, doi: 10.1080/10255842.2012.670855.
[87] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. 2017.
[88] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. B. Wojna, Rethinking the Inception Architecture for Computer Vision. 2016.
[89] J. Deng, W. Dong, R. Socher, L. J. Li, L. Kai, and F.-F. Li, "ImageNet: A large-scale hierarchical image database," in 2009 IEEE Conference on Computer Vision and Pattern Recognition, 20-25 June 2009 2009, pp. 248-255, doi: 10.1109/CVPR.2009.5206848.
[90] P. Subramaniam et al., "Generating 3D TOF-MRA volumes and segmentation labels using generative adversarial networks," Medical Image Analysis, vol. 78, p. 102396, 2022/05/01/ 2022, doi: https://doi.org/10.1016/j.media.2022.102396.
[91] L. Sun, J. Chen, Y. Xu, M. Gong, K. Yu, and K. Batmanghelich, Hierarchical Amortized Training for Memory-efficient High Resolution 3D GAN. 2020.
[92] S. Chen, K. Ma, and Y. Zheng, "Med3D: Transfer Learning for 3D Medical Image Analysis," ArXiv, vol. abs/1904.00625, 2019.
[93] A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. J. Smola, "A Kernel Two-Sample Test," The Journal of Machine Learning Research, vol. 13, pp. 723-773, 03/01 2012.
[94] Z. Wang, E. P. Simoncelli, and A. C. Bovik, "Multiscale structural similarity for image quality assessment," in The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, 9-12 Nov. 2003 2003, vol. 2, pp. 1398-1402 Vol.2, doi: 10.1109/ACSSC.2003.1292216.
[95] O. Ronneberger, P. Fischer, and T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation. 2015, pp. 234-241.
[96] D. C. Dowson and B. V. Landau, "The Fréchet distance between multivariate normal distributions," Journal of Multivariate Analysis, vol. 12, no. 3, pp. 450-455, 1982/09/01/ 1982, doi: https://doi.org/10.1016/0047-259X(82)90077-X.
[97] G. Kwon, C. Han, and D.-S. Kim, "Generation of 3D Brain MRI Using Auto-Encoding Generative Adversarial Networks," in MICCAI, 2019.
[98] S. Pasha, P. Babu, and Z. Vakil, Enhancement of MRI Brain Images with Histogram Equalization Techniques. 2019, pp. 1-4.
[99] G. Modanwal, A. Vellal, and M. A. Mazurowski, "Normalization of breast MRIs using cycle-consistent generative adversarial networks," (in eng), Comput Methods Programs Biomed, vol. 208, p. 106225, Sep 2021, doi: 10.1016/j.cmpb.2021.106225.
[100] X. Sun et al., "Histogram-based normalization technique on human brain magnetic resonance images from different acquisitions," BioMedical Engineering OnLine, vol. 14, no. 1, p. 73, 2015/07/28 2015, doi: 10.1186/s12938-015-0064-y.
[101] L. v. d. Maaten and G. E. Hinton, "Visualizing Data using t-SNE," Journal of Machine Learning Research, vol. 9, pp. 2579-2605, 2008.
[102] P. Chlap, H. Min, N. Vandenberg, J. Dowling, L. Holloway, and A. Haworth, "A review of medical image data augmentation techniques for deep learning applications," (in eng), J Med Imaging Radiat Oncol, vol. 65, no. 5, pp. 545-563, Aug 2021, doi: 10.1111/1754-9485.13261.
[103] L. Liu, Y. Ren, Z. Lin, and Z. Zhao, Pseudo Numerical Methods for Diffusion Models on Manifolds. 2022.
[104] Q. Zhang and Y. Chen, Fast Sampling of Diffusion Models with Exponential Integrator. 2022.