Parameter Server
Parameter Server Architecture
The Parameter Server
The parameter server was proposed by for scalable machine learning.
Characters: client-server architecture, message-passing communication, and asynchronous.
(Note that MapReduce is bulk synchronous.)
Ray , an open-source software system, supports parameter server.
Synchronous Algorithm
Asynchronous Algorithm
Parallel Asynchronous Gradient Descent
Pro and Con of Asynchronous Algorithms
In practice, asynchronous algorithms are faster than the synchronous.
In theory, asynchronous algorithms has slower convergence rate.
Asynchronous algorithms have restrictions, e.g., a worker cannot be much slower than the others.
Footnote
Last updated
