簡易檢索 / 詳目顯示

回結果列表
研究生： 翁子謙
Tze-Qian Eng
論文名稱： Dealing with Non-IID data via the Self-supervision Engaged Split Federated Learning
Dealing with Non-IID data via the Self-supervision Engaged Split Federated Learning
指導教授： 鮑興國
Hsing-Kuo Pao
口試委員: 項天瑞
Tien-Ruey Hsiang
陳怡伶
Yi-Ling Chen
學位類別： 碩士
Master
系所名稱： 電資學院 - 資訊工程系
Department of Computer Science and Information Engineering
論文出版年： 2023
畢業學年度： 111
語文別： 英文
論文頁數： 53
中文關鍵詞： 聯邦學習 、拆分學習 、自監督學習 、非獨立同分佈
外文關鍵詞： federated learning, split learning, self-supervised learning, non-iid
相關次數： 點閱：328  下載：17 
分享至:
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報
  •  上一筆

    • Federated learning and split learning are two widely recognized distributed machine learning frameworks that aim to mitigate privacy concerns during the training process. These frameworks have gained attention from researchers in recent years, due to the increasing of privacy preserving awareness. A key challenge in federated learning is the presence of data heterogeneity among the clients. Specifically, each participating client possesses non-iid data. Numerous studies have proposed solutions to address this challenge in noniid data settings. However, previous works primarily focused on supervised settings, assuming that the data is fully labeled. In real-world scenarios, the collected data is often unlabeled, making the task of annotating such vast amounts of data both expensive and labor-intensive. Consequently, we propose an integrated framework that combines federated learning, split learning, and self-supervised learning to address the issue of unlabeled non-iid data within federated settings. Through extensive experimentation on both iid and non-iid data, we demonstrate that our proposed framework achieves comparable performance. Moreover, in several settings, our framework even surpasses the performance of previous state-of-the-art methods.

    Recommendation Letter Approval Letter Abstract Acknowledgments Contents List of Figures List of Tables List of Algorithms 1 Introduction 2 Related Work 2.1 Federated Learning on Non-IID Data 2.2 Split Federated Learning 2.3 Federated Self-supervised Learning 3 Methodology 3.1 Methods 3.1.1 Federated Learning 3.1.2 Split Learning with Federated Learning 3.1.3 Self-supervised Learning with Federated Learning 3.2 Proposed method 3.2.1 Local Client Training 3.2.2 Latent Representation Concatenation 3.2.3 Model Divergence 3.2.4 Periodic Local Synchronization 3.3 Methods Comparison 4 Experiments 4.1 Datasets 4.1.1 CIFAR-10 4.1.2 CIFAR-100 4.1.3 HDFS 4.1.4 IID and Non-IID Partitioning Strategy 4.2 Implementation Details 4.2.1 Log Anomaly Detection Implementations 4.2.2 SSL settings 4.2.3 Hyperparameter Settings 4.2.4 Evaluation Methods 4.3 Feasibility Study: Image Classification 4.3.1 Component Comparisons 4.3.2 Semi-supervised Evaluation 4.3.3 Transfer Learning 4.3.4 Impact of Training Rounds 4.3.5 Impact of Periodic Local Synchronization 4.3.6 Non-IID Analysis: Quantity-based Label Imbalance 4.3.7 Non-IID Analysis: Distribution-based Label Imbalance 4.3.8 Comparison of Different Numbers of Clients 4.3.9 Benchmark: Linear Evaluation 4.4 Feasibility Study: Log Anomaly Detection 4.4.1 Benchmark: Temporal-based Partition 4.4.2 Benchmark: Event-based Partition 5 Conclusion References Letter of Authority

    [1] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas,
    “Communication-efficient learning of deep networks from decentralized data,” in
    Artificial intelligence and statistics, pp. 1273–1282, PMLR, 2017.
    [2] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, “Federated learning with
    non-iid data,” arXiv preprint arXiv:1806.00582, 2018.
    [3] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated
    optimization in heterogeneous networks,” Proceedings of Machine learning and systems,
    vol. 2, pp. 429–450, 2020.
    [4] S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečnỳ, S. Kumar,
    and H. B. McMahan, “Adaptive federated optimization,” arXiv preprint
    arXiv:2003.00295, 2020.
    [5] C. Thapa, P. C. M. Arachchige, S. Camtepe, and L. Sun, “Splitfed: When federated
    learning meets split learning,” in Proceedings of the AAAI Conference on Artificial
    Intelligence, vol. 36, pp. 8485–8493, 2022.
    [6] C. Dwork, A. Roth, et al., “The algorithmic foundations of differential privacy,”
    Foundations and Trends in Theoretical Computer Science, vol. 9, no. 3–4, pp. 211–
    407, 2014.
    [7] M. Lecuyer, V. Atlidakis, R. Geambasu, D. Hsu, and S. Jana, “Certified robustness to
    adversarial examples with differential privacy,” in 2019 IEEE symposium on security
    and privacy (SP), pp. 656–672, IEEE, 2019.
    [8] A. Abedi and S. S. Khan, “Fedsl: Federated split learning on distributed sequential
    data in recurrent neural networks,” arXiv preprint arXiv:2011.03180, 2020.
    [9] F. Zhang, K. Kuang, Z. You, T. Shen, J. Xiao, Y. Zhang, C. Wu, Y. Zhuang, and X. Li,
    “Federated unsupervised representation learning,” arXiv preprint arXiv:2010.08982,
    2020.
    [10] W. Zhuang, X. Gan, Y. Wen, S. Zhang, and S. Yi, “Collaborative unsupervised visual
    representation learning from decentralized data,” in Proceedings of the IEEE/CVF
    international conference on computer vision, pp. 4912–4921, 2021.
    [11] W. Zhuang, Y. Wen, and S. Zhang, “Divergence-aware federated self-supervised
    learning,” in The Tenth International Conference on Learning Representations, ICLR
    2022, Virtual Event, April 25-29, 2022, OpenReview.net, 2022.
    [12] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive
    learning of visual representations,” in International conference on machine
    learning, pp. 1597–1607, PMLR, 2020.
    [13] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised
    visual representation learning,” in Proceedings of the IEEE/CVF conference
    on computer vision and pattern recognition, pp. 9729–9738, 2020.
    [14] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch,
    B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al., “Bootstrap your own latent-a new
    approach to self-supervised learning,” Advances in neural information processing
    systems, vol. 33, pp. 21271–21284, 2020.
    [15] X. Chen and K. He, “Exploring simple siamese representation learning,” in Proceedings
    of the IEEE/CVF conference on computer vision and pattern recognition,
    pp. 15750–15758, 2021.
    [16] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,”
    in Proceedings of the IEEE conference on computer vision and pattern recognition,
    pp. 770–778, 2016.
    [17] W. Xu, L. Huang, A. Fox, D. Patterson, and M. Jordan, “Largescale system problem
    detection by mining console logs,” Proceedings of SOSP’09, 2009.
    [18] V.-H. Le and H. Zhang, “Log-based anomaly detection with deep learning: How far
    are we?,” in Proceedings of the 44th international conference on software engineering,
    pp. 1356–1367, 2022.
    [19] Q. Li, Y. Diao, Q. Chen, and B. He, “Federated learning on non-iid data silos: An
    experimental study,” in 2022 IEEE 38th International Conference on Data Engineering
    (ICDE), pp. 965–978, IEEE, 2022.
    [20] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin,
    N. Gimelshein, L. Antiga, et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32,
    2019.
    [21] M. Du, F. Li, G. Zheng, and V. Srikumar, “Deeplog: Anomaly detection and diagnosis
    from system logs through deep learning,” in Proceedings of the 2017 ACM
    SIGSAC conference on computer and communications security, pp. 1285–1298,
    2017.
    [22] W. Meng, Y. Liu, Y. Zhu, S. Zhang, D. Pei, Y. Liu, Y. Chen, R. Zhang, S. Tao,
    P. Sun, et al., “Loganomaly: Unsupervised detection of sequential and quantitative
    anomalies in unstructured logs.,” in IJCAI, vol. 19, pp. 4739–4745, 2019.
    [23] G. Zerveas, S. Jayaraman, D. Patel, A. Bhamidipaty, and C. Eickhoff, “A
    transformer-based framework for multivariate time series representation learning,”
    in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &
    Data Mining, pp. 2114–2124, 2021.
    [24] A. Kolesnikov, X. Zhai, and L. Beyer, “Revisiting self-supervised visual representation
    learning,” in Proceedings of the IEEE/CVF conference on computer vision and
    pattern recognition, pp. 1920–1929, 2019.
    [25] X. Zhai, A. Oliver, A. Kolesnikov, and L. Beyer, “S4l: Self-supervised semisupervised
    learning,” in Proceedings of the IEEE/CVF international conference on
    computer vision, pp. 1476–1485, 2019.

    QR CODE