時間序列預測是使用歷史數據預測給定序列的未來值的任務。最近，這項任務引起了機器學習領域研究人員的注意，隨著大量歷史數據可用性的增加以及強大的預測技術推斷過去和未來值之間的隨機依賴性，以改善既費時又複雜的傳統預測方法。使用長短期記憶（Long Short-Term Memory；LSTM）網路為一種特殊類型的遞歸神經網絡，其優點是能夠學習所提供的網絡輸入和輸出之間的長期依賴性。在本文中，我們提出的方法是差分長短期記憶（Differencing Long Short -Term Memory；D-LSTM）網路架構，作為遞歸神經網絡的擴展。差分即是後值減去前值，可以使原資料減少雜訊使其變的平穩，並且提高預測準確度。我們設計了三維度非線性渾沌系統，藉由相圖、平衡點、李亞普諾夫指數、頻譜熵值…等技術，探討其運動行為，針對自行設計的渾沌系統做改變初始值及係數來進行預測研究。我們將所提方法的性能與自適應神經模糊推論系統(Adaptive Network based Fuzzy Inference System；ANFIS)及LSTM進行比較。使用均方根誤差(Root Mean Square Error；RMSE)測量標準，實證結果表明，所提出的D-LSTM模型幾乎優於其他方法。

Time series prediction is the task of using historical data to predict future values for a given sequence. Recently, this task has attracted the attention of researchers in the field of machine learning, with the increasing availability of a large amount of historical data and the strong predictive technology inferring random dependence between past and future values to improve time-consuming and complex traditional predictions method. Using a Long Short-Term Memory (LSTM), this is a special type of recurrent neural network that has the advantage of being able to learn the long term dependencies between the provided network inputs and outputs. In this thesis, we propose a Differencing Long Short-Term Memory (D-LSTM) architecture as an extension of recurrent neural networks. The differential is the latter value minus the previous value, which can reduce the noise of the original data to make it smooth and improve the prediction accuracy. We design a 3D nonlinear chaotic system and analyze its properties and dynamic behaviors by phase portraits, equilibrium points, Lyapunov exponents, spectral entropy etc. We study prediction result by change the initial value and the coefficient for our chaotic system. We compare D-LSTM with Adaptive Neuro Fuzzy Inference system (ANFIS) and LSTM, using Root Mean Square Error (RMSE) to measure their performance. The result shows that our model is almost better than others.

Reference
[1] Lorenz, E. N. (1963). Deterministic nonperiodic flow. Journal of the Atmospheric Sciences, 20:130–141.
[2] TienYien, L. & James, A.Y. (1975). Period three implies chaos. The American Mathematical Monthly, 82(10):985–992.
[3] Sun, K. (2016). Chaotic Secure Communication: Principles and Technologies. De Gruyter.
[4] Box, G. E. P. & Pierce, D. A. (1970). Distribution of Residual Autocorrelations in Autoregressive-Integrated Moving Average Time Series Models. Journal of the American Statistical Association, 65(332):1509-1526.
[5] Hinton, G. E. & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507.
[6] Kuremoto, T., Kimura, S., Kobayashi, K., & Obayashi, M. (2014). Time series forecasting using a deep belief network with restricted Boltzmann machines. Neurocomputing, 137(5):47–56.
[7] Kuremoto, T., Hirata, T., Obayashi, M., Mabu, S. & Kobayashi, K. (2014, Oct). Forecast Chaotic Time Series Data by DBNs. 2014 7th International Congress on Image and Signal Processing, Dalian, China, 1304-1309.
[8] JyhShing, R. J. (1993), ANFIS: Adaptive-Network-Based Fuzzy Inference System. IEEE Transactions on Systems, Man, and Cybernetics, 23(3):665-685.
[9] Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8):1735-1780.
[10] Hasim, S., Andrew, S., & Franoise, B. (2014). Long short term memory based recurrent neural network architectures for large vocabulary speech recognition. ArXiv:1402.1128.
[11] Graves, A., Mohamed, A., & Hinton G. (2013, January). Speech Recognition with Deep Recurrent Neural Networks. IEEE Conference on Acoustics, Speech and Signal Processing, Vancouver, Canada, 6645-6649.
[12] Ljun, L. (1987). System Identification Theory for User. Prentice Hall, Englewood Cliffs, New Jersey 07632, United States of America.
[13] Box, G. E. P. & Pierce, D. A. (1970). Distribution of Residual Autocorrelations in Autoregressive-Integrated Moving Average Time Series Models. Journal of the American Statistical Association, 65(332):1509-1526.
[14] Keizhevsky, A., Sutskever, I., & Hinton, G. E. (2012, December). ImageNet Classification with Deep Convolutional Neural Networks. NIPS'12 Proceedings of the 25th International Conference on Neural Information Processing Systems 1:1097-1105.
[15] Giles, C. L., Kuhn, G. M., & Williams, R. J. (1994). Dynamic recurrent neural networks: theory and applications. IEEE Transactions on Neural Networks, 5(2): 153-156.
[16] Jang, S. R. (1993). ANFIS: adaptive-network-based fuzzy inference system. IEEE Transactions on Systems, Man, and Cybernetics, 23(3):665-685.
[17] Odom, M. D., & Sharda, R. (1990, June). A neural network model for bankruptcy prediction. 1990 International Joint Conference on Neural Networks, San Diego.
[18] Erdi, T., Kadir, A., & Mehmet B. (2016). Comparison of linear regression and artificial neural network model of a diesel engine fueled with biodiesel-alcohol mixtures. Alexandria Engineering Journal, 55(4): 3081-3089.
[19] Shouliang, H., Zhuoshi, H., Jing, S. & Beidou, X. (2013). Using Artificial Neural Network Models for Eutrophication Prediction. Procedia Environmental Sciences, 18:310-316.
[20] Hirata, T., Takashi, K., & Masanao, O. (2015, August). Time Series Prediction using DBN and ARIMA. 2015 International Conference on Computer Application Technologies, Japan.
[21] Narendrababu, C. & Eswarareddy, B. (2014). A moving-average filter based hybrid ARIMA–ANN model forforecasting time series data. Applied Soft Computing, 23:27-38.
[22] Mehdi, K. & Mehdi, B. (2011). A novel hybridization of artificial neural networks and ARIMA models for time series forecasting. Applied Soft Computing, 11(2):2664-2675.
[23] Wenguan, W., Jianbing, S. (2018). Deep Visual Attention Prediction. IEEE Transactions on Image Processing, 27(5):2368-2378.
[24] Shuihua, W., YiDing, Lv. & Yuxiu, Sui. (2018). Alcoholism Detection by Data Augmentation and Convolutional Neural Network with Stochastic. Pooling Journal of Medical Systems January, 42(1):1-11.
[25] Jiuxiang, G. & Zhenhua, W. (2018). Recent advances in convolutional neural networks. Pattern Recognition, 77:354-377.
[26] Edward. C, Schuetz, A. (2017). Using recurrent neural network models for early detection of heart failure onset. Journal of the American Medical Informatics Association, 24(2):361-370.
[27] Mou, L., Ghamisi, P. & Zhu, X. X (2017). Deep Recurrent Neural Networks for Hyperspectral Image Classification. IEEE Transactions on Geoscience and Remote Sensing, 55(7):3639-3655.
[28] Liang, G., Naipeng, L., Feng, J., Yaguo, L. & Jing, L. (2017). A recurrent neural network based health indicator for remaining useful life prediction of bearings. Neurocomputing, 240(31):98-109.
[29] Sagheer, A. & Kotb, M. (2019). Time series forecasting of petroleum production using deep LSTM recurrent networks. Neurocomputing, 323(5):203–213.
[30] Petersen, N.C., Rodrigues, F. & Pereira, F.C. (2019). Multi-output bus travel time prediction with convolutional LSTM neural network. Expert Systems With Applications, 120:426–435.
[31] Gonzalez, J. & Yu, W. (2018). Non-linear system modeling using LSTM neural networks. IFAC-PapersOnLine, 51(13):485–489.
[32] Pedrycz, W. (1993). Fuzzy Control and Fuzzy Systems. New York: Wiley.
[33] JyhShing, R. J. & ChuenTsai, S. (1993), Functional equivalence between radial basis function networks and fuzzy inference systems. IEEE Transactions on Neural Networks, 4(1):156-159.
[34] Chuen, C. L. (1990) “Fuzzy logic in control systems: Fuzzy logic controller-Part I. IEEE Transactions on Systems, Man, and Cybernetics, 20(2):404-418.
[35] Kehui, S. (2016). Chaotic Secure Communication: Principles and Technologies. De Gruyter.
[36] Baranger, M. (2001). Chaos, Complexity, and Entropy: A physics talk for non-physicists. New England Complex Systems Institute.
[37] Benoit B. M. (1982). The Fractal Geometry of Nature. United States: W. H. Freeman and Company.
[38] Wolf, A., Swift, J. B., Swinney, H. L. & Vastano, J. A. (1986). Determining Lyapunov exponents from a time series, Physica D: Nonlinear Phenomena, 16(3):285-317.
[39] Powell, G. E. & Percival, I. C. (1979). A spectral entropy method for distinguishing regular and irregular motion of Hamiltonian systems, Journal of Physics A: Mathematical and General, 12(11):2053-2071.
[40] Qingfang, M. & Yuhua, P. (2007). A new local linear prediction model for chaotic time series. Physical Letters A, 370(5-6):465-470.
[41] Packard, N. H., Crutchfield, J. P., & Farmer, J. D. (1989). Geometry from a time series. Physical Review Letter, 45(9):712-716.
[42] Takens, F. (1981). Detecting strange attractors in turbulence. Dynamic Systems and Turbulence, 898:366-381.
[43] Kim, H. S., Eykhlt, R., & Salas, J. D. (1999). Nonlinear dynamics, delay times, and embedding windows. Physica D: Non- linear Phenomena, 127(1-2):48-60
[44] Fraser, A. M. & Swinney, H. L. (1986). Independent coordinates for strange attractors from mutual information. Physical Review A, 33(2):1134-1140.
[45] Kraskov, A., Stogbauer, H., & Grassberger, P. (2004). Estimating mutual information. Physical Review E, 69(9):019903.
[46] Rhodes, C.& Morari, M. (1997). The false nearest neighbors algorithm: An overview. Computers & Chemical Engineering, 21:1149-1154.
[47] Kennel, M., Brown, R., & Abarbanel, H. (1992). Determining embedding dimension for phase-space reconstruction using a geometrical construction. Physical Review A. 45 (6):3403–3411.
[48] Cao, L. (1997). Practical method for determining the minimum embedding dimension of a scalar time series. Physica D: Non- linear Phenomena, 110(1-2):43-50.
[49] Cao, L., Mees, A. & Judd, K. (1998). Dynamics from multivariate time series. Physica D: Nonlinear Phenomena, 121(1-2):75-88.
[50] Takagi, T. & Sugeno, M. (1983). Derivation of fuzzy control rules from human operator’s control actions. IFAC Proceedings Volumes, 16(13):55-60.
[51] Abraham, A. (2005). Adaptation of Fuzzy Inference System Using Neural Learning. StudFuzz 181:53–83.
[52] Hurst, H.E. (1951). Long-term storage capacity of reservoirs. Transactions of American Society of Civil Engineers, 116(1):770-799.
[53] Mandelbrot, Benoit B. & Wallis, James R. (1969). Robustness of the rescaled range R/S in the measurement of noncyclic long run statistical dependence. Water Resources Research, 5(5):967–988.
[54] Mackey, M. C. & Glass, L. (1977). Oscillation and chaos in physiological control systems. Science, 197(4300):287-289.
[55] Farmer, J.D. (1982). Chaotic attractors of an infinite-dimensional dynamical system. Physica D, 4(3):366-393.
[56] 張芸甄 (2018)。不同階數渾沌系統之特性分析與控制及其FPGA實現。國立台灣科技大學自動化及控制研究所碩士論文，未出版，臺北市大安區基隆路四段43號。