Introduction
ML Fundamentals
Classical Machine Learning
- Basics
- Unsupervised Learning
Neural Networks
Loss
Computer Vision
NLP
Parallel Computing
Anomaly Detection
- DBSCAN
- Autoencoder
Visualization

Powered by GitBook

On this page

Motivating Examples
What is federated learning
Let us recall parallel gradient descent
Federated Averaging Algorithm
Computation vs. Communication
References

Was this helpful?

Parallel Computing

Federated Learning

PreviousDecentralized And Ring All Reduce NextModel Parallelism: GPipe

Last updated 3 years ago

Motivating Examples

What is federated learning

Federated learning , [1] is [2] a kind of distributed learning.

How does federated learning differ from traditional distributed learning?

Users have control over their device and data.
Worker nodes are unstable.
Communication cost is higher than computation cost.
Data stored on worker nodes are not IID.
The amount of data is severely imbalanced.

Let us recall parallel gradient descent

Federated Averaging Algorithm

Computation vs. Communication

References

[1] McMahan and others: Communication-efficient learning of deep networks from decentralized data. In AISTATS, 2017. .
[2] Konevcny, McMahan, and Ramage: Federated optimization: distributed optimization beyond the datacenter. arXiv:1511.03575, 2015