近年來，深度神經網路 (Deep Neural Network, DNN) 快速得發展，至今已應用在許多領域，例如智慧家電、人臉辨識與自動駕駛等等。深度神經網路模型透過大量的訓練資料，使正確率 (Accuracy) 達到一定標準，而訓練完成後會產生大量的權重資料 (Weight)，這些權重資料必須被儲存下來。快閃記憶體為適合之儲存裝置來儲存這些權重資料，快閃記憶體為具有低功耗、可擴充性、高效能等優點的非揮發性記憶體，隨著製程發展，雖然快閃記憶體有較高的儲存密度與較低的成本，但同時也造成了可靠度 (Reliability) 與耐久度 (Endurance) 降低的問題。
若是儲存之權重資料發生錯誤，在運算過程中將會產生誤差，導致正確率下降，因此本篇論文提出位址重映射容錯技術保護儲存於快閃記憶體之權重資料，本篇分析了權重資料之位元敏感度，定義出較重要之位元，並透過轉置電路將重要位元集中，本篇也提出位址重映射演算法進行重映射分析，透過位址重映射演算法將重要位元位址重映射至較安全位址，以提升深度神經網路模型之可靠度，使錯誤對正確率影響降低。
本篇論文實現了位址重映射技術電路，並且以深度學習框架開發模擬器，模擬不同深度神經網路模型應用位址重映射技術之實驗，實驗結果顯示當位元錯誤率 (Bit Error Rate, BER) 達到 0.05 % 時，深度神經網路模型 VGG16 在無修復技術之正確率下降約 30 %，本篇提出之技術仍可維持深度神經網路模型之正確率下降在 1 % 之內，而本篇也分析位址重映射技術之硬體成本，實驗結果顯示本篇提出之位址重映射技術使用的額外硬體成本不超過 0.2 %。

Deep Neural Network (DNN) has been widely used in smart appliances, face recognition and autonomous driving. After trained by training data, there will be large number of model weights. And weights data should be stored, flash memory can store large number of weights data. Flash memory is a non-volatile memory with the advantage of low power consumption, good scalability, and high performance. Due to the advance of process technology, the storage density of flash memory continues increase, but it also makes reliability and endurance decrease.
If flash memory suffered from finite endurance that lead high bit error rate (BER) in stored data, the DNN model of recognition accuracy would be affected. In order to improve reliability of DNN model, this thesis proposed the address remapping technique to protect weights data stored in flash memory. We analyze bit significant of weights data when faults are injected into weights data. Based on the analysis, we find the significant bit of weights data and propose transposer. Weights data will be stored according to their significance. The address remapping technique will change address of transposed weight data that stored in faulty word, and remapping address to more reliable word address.
The architecture of address remapping technique is also proposed. We used the deep learning framework for evaluating the accuracy of different DNN model. Experimental results show that based on 0.01 % BER in weights data, the DNN model of accuracy loss with proposed technique is less than 1 %. The hardware overhead for implementing technique is less than 0.2%.

[1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep Learning,” Nature, vol. 521, no. 7553, pp. 436–444, May 2015.
[2] R. Bez, E. Camerlenghi, A. Modelli, and A. Visconti, “Introduction to Flash Memory,” in Proc. IEEE, vol. 91, no. 4, pp. 489-502, Apr. 2003.
[3] Y. Li and K. N. Quader, “NAND Flash Memory: Challenges and Opportunities,” Computers, vol. 46, pp.23-29, Aug. 2013.
[4] A. S. Spinelli, C. M. Compagnoni, and A. L. Lacaita, “Reliability of NAND Flash Memories: Planar Cells and Emerging Issues in 3D Devices,” Computers, vol. 6, no. 2, pp. 16, Apr. 2017, [Online]. Available: https://www.mdpi.com/2073-431X/6/2/16.
[5] G. Mayuga, Y. Yamato, T. Yoneda, M. Inoue, and Y. Sato, “An ECC-based Memory Architecture with Online Self-repair Capabilities for Reliability Enhancement,” in Proc. IEEE European Test Symposium (ETS), pp. 1-6, May 2015.
[6] L. Yuan, H. Liu, P. Jia, and Y. Yang, “Reliability-based ECC System for Adaptive Protection of NAND Flash Memories,” in Proc. IEEE International Conference on Communication Systems and Network Technologies (CSNT), pp. 897-902, Apr. 2015.
[7] C. Kim, C. Park, S. Yoo, and S. Lee, “Extending Lifetime of Flash Memory Using Strong Error Correction Coding,” in Proc IEEE Transactions on Consumer Electronics, pp. 206-214, May. 2015.
[8] W. Kang, L. Zhang, W. Zhao, J. O. Klein, Y. Zhang, D. Ravelosona, and C. Chappert, “Yield and Reliability Improvement Techniques for Emerging Nonvolatile STT-MRAM,” in Proc IEEE Journal on Emerging and Selected Topics in Circuits and Systems, pp. 28–39, Mar. 2015.
[9] K. Mizoguchi, K. Maeda, and K. Takeuchi, “Automatic Data Repair Overwrite Pulse for 3D-TLC NAND Flash Memories with 38x Data-retention Lifetime Extension,” in Proc IEEE International Reliability Physics Symposium (IRPS), pp. 1-5, Mar. 2019.
[10] M. Mehedi and B. Ray, “Tolerance of Deep Neural Network Against the Bit Error Rate of NAND Flash Memory,” in Proc. IEEE International Reliability Physics Symposium (IRPS), pp. 1-4, Mar. 2019.
[11] J. S. Kim and J.-S. Yang, “DRIS-3: Deep Neural Network Reliability Improvement Scheme in 3D Die-Stacked Memory based on Fault Analysis,” in Proc IEEE Design Automation Conference (DAC), pp. 1-6, Jun. 2019.
[12] Y. Deguchi and K. Takeuchi, “3D-NAND Flash Solid-State Drive (SSD) for Deep Neural Network Weight Storage of IoT Edge Devices with 700x Data-retention Lifetime Extention,” in Proc. IEEE International Memory Workshop (IMW), pp. 1-4, May 2018.
[13] Y. Deguchi, T. Nakamura, A. Kobayashi, and K. Takeuchi, “12x Bit-Error Acceptable, 300x Extended Data-Retention Time, Value-Aware SSD with Vertical 3D-TLC NAND Flash Memories for Image Recognition,” in Proc. IEEE Custom Integrated Circuits Conference (CICC), pp. 1-4, May 2017.
[14] D. T. Nguyen, H. Kim, H. Lee, and I. Chang, “An Approximate Memory Architecture for a Reduction of Refresh Power Consumption in Deep Learning Applications,” in Proc. IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1-5, May 2018.
[15] T. Marukame, K. Ueyoshi, T. Asai, M. Motomura, A. Schmid, M. Suzuki, Y. Higashi, and Y. Mitani, “Error Tolerance Analysis of Deep Learning Hardware Using Restricted Boltzmann Machine Towards Low-Power Memory Implementation,” in Proc IEEE Trans. on Circuits and Systems II, pp. 462-466, Apr. 2017.
[16] G. Srinivasan, P. Wijesinghe, S. S. Sarwar, A. Jaiswal, and K. Roy, “Significance Driven Hybrid 8T-6T SRAM for Energy-Efficient Synaptic Storage in Artificial Neural Networks,” in Proc Design, Automation and Test in Europe Conference(DATE), pp. 151-156, Mar. 2016.
[17] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic Differentiation in Pytorch,” in Proc. Conf. Neural Info. Process. Syst. Workshop, pp. 1–4, Oct. 2017.
[18] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based Learning Applied to Document Recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, Nov. 1998.
[19] K. Simonyan and A. Zisserman. “Very Deep Convolutional Networks for Large-Scale Image Recognition,” in Proc. International Conference on Learning Representations(ICLR), 2015.
[20] D. Kahng and S. M. Sze, “A Floating Gate and its Application to Memory Device,” Bell Syst. Tech. J., 46, 1288, 1967.
[21] F. Masuoka, M. Momodomi, Y. Iwata and R. Shirota, “New Ultra High Density EPROM and Flash EEPROM with NAND Structured Cell,” in IEDM Tech. Dig., pp. 552-555, Dec. 1987.
[22] R. Dekker, F. Beenker, and L. Thijssen, "Fault Modeling and Test Algorithm Development for Static Random-access Memories," in Proc. IEEE Int’l Test Conf., pp. 343-352, Sep. 1988.
[23] IEEE 1005 Standard Definitions and Characterization of Floating Gate Semiconductor Arrays, Piscataway, NJ: IEEE Standards Sep. 1999.
[24] J. C. Yeh, K. L. Cheng, Y. F. Chou, and C. W. Wu, “Flash Memory Testing and Built-in Self-diagnosis with March-like Test Algorithms,” IEEE Trans. on Computer-Aided Design of Integrated Circuits and Systems, vol. 26, no. 6, pp. 1101–1113, Jun. 2007.
[25] M. G. Mohammad and L. Terkawi, “Fault Collapsing for Flash Memory Disturb Faults,” in Proc. IEEE European Symposium on Test (ETS), pp. 142–147, May 2005.
[26] Stefano D. C., Fabiano M., Piazza R., and Prinetto P, “Exploring Modeling and Testing of NAND Flash Memories,” in Proc. IEEE Design & Test Symp. (EWDTS), pp. 47-50, Sep. 2010.
[27] C. T. Huang, J. R. Huang, C. F. Wu, C. W. Wu, and T. Y. Chang, “A Programmable BIST Core for Embedded DRAM,” IEEE Design & Test of Computers, vol. 16, no. 1, pp. 59-70, Jan. 1999.
[28] A. J. V. D. Goor, “Using March Tests to Test SRAMs,” IEEE Design & Test of Computers, vol. 10, no. 1, pp. 8-14, Mar. 1993.
[29] R. Nair, S. M. Thatte, and J. A. Abraham, “Efficient Algorithms for Testing Semiconductor Random-access Memories,” IEEE Trans. Computers, vol. C-27, no. 6, pp. 572-576, Jun. 1978.
[30] K. L. Cheng, J. C. Yeh, C. W. Wang, C. T. Huang, and C. W. Wu, “RAMSES-FT: A Fault Simulator for Flash Memory Testing and Diagnostics,” in Proc. IEEE VLSI Test Symp. (VTS), pp. 281-286, Apr. 2002.
[31] C. T. Huang, J. C. Yeh, Y. Y. Shih, R. F. Huang, and C. W. Wu, “On Test and Diagnostics of Flash Memories,” in Proc. IEEE Asian Test Symp. (ATS), pp. 260-265, Jan. 2005
[32] J. Y. Hu, K. W. Hou, C. Y. Lo, Y. F. Chou, and C. W. Wu, “RRAM-Based Neuromorphic Hardware Reliability Improvement by Self-Healing and Error Correction,” in Proc. IEEE International Test Conference in Asia (ITC-Asia), Aug. 2018.
[33] X. Glorot, A. Bordes, and Y. Bengio, “Deep Sparse Rectifier Neural Networks,” in Proc. Conf. Artificial Intelligence and Statistics, vol. 15, pp.315-323, Apr. 2011.
[34] F. F. Li, R. Krishna, D. Xu, A. Byun, W. Shen, J. Braatz, D. Cai, J. Gwak, De-An Huang, A. Kondrich, F. Yu Lin, D. Mrowca, B. Pan, N. Rai, L. P. Tchapmi, C. Waites, R. Wang, Yi Wen, K. Yang, B. Yi, C. Yuan, K. Zakka, and Y. Zhang, “CS231n Convolutional Neural Networks for Visual Recognition,” Jan. 2015. [Online]. Available: https://cs231n.github.io/convolutional-networks/
[35] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” in NIPS, 2012.
[36] S. Teerapittayanon, B. McDanel, and H. T. Kung,“Distributed Deep Neural Networks Over the Cloud, the Edge and End Devices,” in Proc. IEEE 37th Int. Conf. Distrib. Comput. Syst. (ICDCS), pp. 328–339, Jun. 2017.
[37] L. Chen, J. Li, Y. Chen, Q. Deng, J. Shen, X. Liang, and L. Jiang, “Accelerator-Friendly Neural-network Training: Learning Variations and Defects in RRAM Crossbar,” in Proc. of the Conference on Design, Automation & Test in Europe (DATE), pp. 19–24, Mar. 2017.
[38] B. Liu, H. Li, Y. Chen, X. Li, Q. Wu, and T. Huang, “Vortex: Variation-aware Training for Memristor X-bar,” in Proc. IEEE Design Automation Conference (DAC), pp. 1–6, June. 2015.
[39] Y. LeCun, C. Cortes, C. Burges, “The MNIST Database of Handwritten Digits,” 2012. Available online: http://yann.lecun.com/exdb/mnist/.
[40] S. Weidman, “Deep Learning with Pytorch: a 60 Minute Blitz Training a Classifier,” Oct. 2019. [Online]. Available: https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html.