# Parameter Server

## Parameter Server Architecture

![parallel\_computing\_client\_server](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-65b6d08734a709a487bbb5bd730fbeab9643b96f%2Fparallel_computing_client_server.png?alt=media)

## The Parameter Server

* The parameter server was proposed by for scalable machine learning.
* Characters: client-server architecture, message-passing communication, and asynchronous.
* (Note that MapReduce is bulk synchronous.)
* Ray , an open-source software system, supports parameter server.

## Synchronous Algorithm

![parallel\_synchronous\_algorithm\_1](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-12fa765768438cc60daa36625350be93f024abb2%2Fparallel_synchronous_algorithm_1.png?alt=media)

## Asynchronous Algorithm

![parameter\_server\_architecture\_1](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-45435a023edb046b65a6bbe39852fae50d52333d%2Fparameter_server_architecture_1.png?alt=media)

## Parallel Asynchronous Gradient Descent

![parallel\_asynchronous\_gradient\_descent\_1](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-869abeda935a3d9cfedc5a7a73e6212b5cc1e07f%2Fparallel_asynchronous_gradient_descent_1.png?alt=media)

## Pro and Con of Asynchronous Algorithms

* In practice, asynchronous algorithms are faster than the synchronous.
* In theory, asynchronous algorithms has slower convergence rate.
* Asynchronous algorithms have restrictions, e.g., a worker cannot be much slower than the others.

![Pro\_and\_Con\_of\_Asynchronous\_Algorithms](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-e41eb85f7611e483b103c0244ca5b6d8f40cabb6%2FPro_and_Con_of_Asynchronous_Algorithms.png?alt=media)

## Footnote