聯邦學習(Federated Learning, FL)為分布式機器學習(Distributed machine learning)的一類，其主要的焦點之一為通訊效率(communication efficiency)，因為大量參與訓練模型的邊緣設備(edge device)在每輪迭代中向參數伺服器(Parameter Server, PS)發送他們的更新。當參數伺服器進行深度學習(Deep Learning)時，神經網路(neural network)中的大量權重(weight)需要更新，將所有的權重從邊緣設備發送到參數伺服器將花費太多的通訊帶寬(communication bandwidth)。在實踐中，更新是權重的梯度(gradient)，因此應該在傳輸前將其量化(quantize)為有限大小的版本。在本篇論文中我們提出對於聯邦學習上行鏈路(up-link)的量化器方案，其中包含標量量化(scalar quantization)及格架式編碼量化(Trellis coded quantization)，經過模擬對比模型準確度(accuracy)與均方誤差(Mean Square error, MSE)，我們提出的量化器可以在速率(Rate)大幅降低後仍保有相當良好的準確度。

Federated Learning (FL) is a class of distributed machine learning, and one of its main focuses is communication efficiency. since a large number of participating edge devices, will send their updates to the Parameter Server(PS) at each round of the model training. When the PS is performing deep learning where lots of weights in a neural network need to be updated, sending all weights from the edge devices to the server will cost too much communication bandwidth. In practice, the updates are the gradients of weights, which should be quantized into finite-size versions before transmission. In this thesis we propose quantizer schemes for federated learning up-link , which include the scalar quantization and Trellis coded quantization(TCQ). After simulating the model accuracy and Mean Square error(MSE), our proposed quantizer can maintain good accuracy even after a significant reduction in Rate.

[1] McMahan, H. Brendan, et al. "Federated learning of deep networks using model averaging." arXiv preprint arXiv:1602.05629 2 (2016).

[2] Gray, Robert M., and David L. Neuhoff. "Quantization." IEEE transactions on information theory 44.6 (1998): 2325-2383.

[3] Zhang, Naifu, Meixia Tao, and Jia Wang. "Sum-rate-distortion function for indirect multiterminal source coding in federated learning." 2021 IEEE International Symposium on Information Theory (ISIT). IEEE, 2021.

[4] Singh, Navjot, et al. "Squarm-sgd: Communication-efficient momentum sgd for decentralized optimization." IEEE Journal on Selected Areas in Information Theory 2.3 (2021): 954-969.

[5] Lin, Chung-Yi, Victoria Kostina, and Babak Hassibi. "Differentially Quantized Gradient Methods." IEEE Transactions on Information Theory (2022).

[6] Chen, Zhong-Jing, et al. "DNN gradient lossless compression: Can GenNorm be the answer?." arXiv preprint arXiv:2111.07599 (2021).

[7] Zhang, Zhilu, and Mert Sabuncu. "Generalized cross entropy loss for training deep neural networks with noisy labels." Advances in neural information processing systems 31 (2018).

[8] Ungerboeck, Gottfried. "Channel coding with multilevel/phase signals." IEEE transactions on Information Theory 28.1 (1982): 55-67.

[9] Viterbi, Andrew. "Error bounds for convolutional codes and an asymptotically optimum decoding algorithm." IEEE transactions on Information Theory 13.2 (1967): 260-269.

[10] Cover, Thomas M. "Thomas. Elements of information theory." Wiley Series in Telecommunications (1991).

[11] Eriksson, Tomas, John B. Anderson, and Norbert Goertz. "Linear congruential trellis source codes: Design and analysis." IEEE transactions on communications 55.9 (2007): 1693-1701.

[12]Kirac, Ahmet, and P. P. Vaidyanathan. "Results on lattice vector quantization with dithering." IEEE Transactions On Circuits and Systems II: Analog and Digital Signal Processing 43.12 (1996): 811-826.

[13]O'Shea, Keiron, and Ryan Nash. "An introduction to convolutional neural networks." arXiv preprint arXiv:1511.08458 (2015).

[14]Gulli, Antonio, and Sujit Pal. Deep learning with Keras. Packt Publishing Ltd, 2017.

[15]Agarap, Abien Fred. "Deep learning using rectified linear units (relu)." arXiv preprint arXiv:1803.08375 (2018).

[16] Jang, Eric, Shixiang Gu, and Ben Poole. "Categorical reparameterization with gumbel-softmax." arXiv preprint arXiv:1611.01144 (2016).

[17] Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980 (2014).

[18] Virtanen, Pauli, et al. "SciPy 1.0: fundamental algorithms for scientific computing in Python." Nature methods 17.3 (2020): 261-272.

[19] Lin, Shih-Chun, and Hsuan-Jung Su. "Practical vector dirty paper coding for MIMO Gaussian broadcast channels." IEEE Journal on Selected Areas in Communications 25.7 (2007): 1345-1357.