持續性學習領域的發展目的是為了解決災難性遺忘，使機器的學習模式跟人類一樣可以按照任務的時間序列來做學習。我們使用變分自編碼器的架構，來處理持續性學習的分類問題。在持續型學習的分類問題中生成模型解決的手段往往是生成先前任務的資料，但若生成出來的圖片品質參差不齊也會導致模型無法訓練。我們提出了一種新觀點是生成模型不一定要生成圖片，而是可以生成很多潛變量空間中的點讓這些點來限制潛變量空間的變化。再將知識轉移實作在潛變量的空間中，由於潛變量的空間所帶來的資訊量往往是很足夠的，所以我們只要讓潛變量空間變化不要太過劇烈，我們就能夠將先前任務的知識保存下來已減緩災難性遺忘的發生。在實驗中我們把相同定義域中的資料集 CIFAR10、CIFAR100 切分成很多任務，讓我們的模型以及 GEM、EWC、MAS、Riemannian walk、LWF、Packnet 來互相比較，來證明我們模型能在任務越多的資料集中保存知識的效果越好，成為目前最厲害的方法之一。並且透過不同定義域的資料集 MNIST、SVHN、CIFAR10 來展現我們模型的穩定性。

The purpose of developing continual learning is to solve catastrophic forgetting. We use structure of autoencoder to settle classification of continual learning. To author's knowledge, structure of variational autoencoder is used to deal with classification and generation task of continual learning. In classification of continual learning, methods of generative model usually are generating previous data. However, if quality of generated image is uneven, the model would fail to be trained. We proposed a new viewpoint that instead of images, generative model should generate latent space. First, constrain the change of latent. Then, implement knowledge transfer on latent space. Owing to the fact that information from latent space is sufficient, as long as the change of latent space is not too drastic, it is possible to preserve previous knowledge to alleviate catastrophic forgetting. In the experiment, we divided same domain data sets CIFAR10 and CIFAR100 into multiple tasks to compare our model with GEM、EWC、Riemannian walk、MAS、LWF、Packnet. We proved that the more tasks in data sets, the better performance of preserving knowledge our model have. In other words, our model is one of the most ideal models nowadays. We also showed the stability of our model through cross domain data sets MNIST, SVHN and CIFAR10.

[1] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,” Neural Networks, vol. 113, pp. 54–71, 2019.
[2] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska­Barwinska, et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521–3526, 2017.
[3] M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in Psychology of learning and motivation, vol. 24, pp. 109–165, Elsevier, 1989.
[4] Z. Li and D. Hoiem, “Learning without forgetting,” IEEE transactions on pattern analysis and ma­ chine intelligence, vol. 40, no. 12, pp. 2935–2947, 2017.
[5] W.­Y. Kang, C.­H. Han, and B.­T. Zhang, “Discriminative variational autoencoder for continual learn­ ing with generative replay,” 2019.
[6] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He, “A comprehensive survey on transfer learning,” Proceedings of the IEEE, vol. 109, no. 1, pp. 43–76, 2020.
[7] M. Delange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuyte­ laars, “A continual learning survey: Defying forgetting in classification tasks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[8] F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” in Interna­ tional Conference on Machine Learning, pp. 3987–3995, PMLR, 2017.
[9] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 139–154, 2018.
[10] A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. Torr, “Riemannian walk for incremental learn­ ing: Understanding forgetting and intransigence,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 532–547, 2018.
[11] Y. Cheng, D. Wang, P. Zhou, and T. Zhang, “A survey of model compression and acceleration for deep neural networks,” arXiv preprint arXiv:1710.09282, 2017.
[12] A. Mallya and S. Lazebnik, “Packnet: Adding multiple tasks to a single network by iterative pruning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7765–7773, 2018.
[13] C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner, “Variational continual learning,” arXiv preprint arXiv:1710.10628, 2017.
32
[14] D. Lopez­Paz and M. Ranzato, “Gradient episodic memory for continual learning,” arXiv preprint arXiv:1706.08840, 2017.
[15] H. Shin, J. K. Lee, J. Kim, and J. Kim, “Continual learning with deep generative replay,” arXiv preprint arXiv:1705.08690, 2017.
[16] G. M. Van de Ven and A. S. Tolias, “Generative replay with feedback connections as a general strategy for continual learning,” arXiv preprint arXiv:1809.10635, 2018.
[17] D. P. Kingma and M. Welling, “Auto­encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013.
[18] K. Sohn, H. Lee, and X. Yan, “Learning structured output representation using deep conditional gen­ erative models,” Advances in neural information processing systems, vol. 28, pp. 3483–3491, 2015.
[19] A. Rannen, R. Aljundi, M. B. Blaschko, and T. Tuytelaars, “Encoder based lifelong learning,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 1320–1328, 2017.
[20] M. J. Wainwright and M. I. Jordan, Graphical models, exponential families, and variational inference. Now Publishers Inc, 2008.
[21] M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul, “An introduction to variational methods for graphical models,” Machine learning, vol. 37, no. 2, pp. 183–233, 1999.
[22] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, “Variational inference: A review for statisticians,” Journal of the American statistical Association, vol. 112, no. 518, pp. 859–877, 2017.
[23] C. M. Bishop, Pattern recognition and machine learning. springer, 2006.
[24] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint
arXiv:1503.02531, 2015.
[25] J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,” International Journal
of Computer Vision, pp. 1–31, 2021.
[26] J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,”
arXiv preprint arXiv:1803.03635, 2018.
[27] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: Hints for thin
deep nets,” arXiv preprint arXiv:1412.6550, 2014.
[28] S. Zagoruyko and N. Komodakis, “Paying more attention to attention: Improving the performance of
convolutional neural networks via attention transfer,” arXiv preprint arXiv:1612.03928, 2016.