Submitted by scrawford on
| Title | Asynchronous SGD for DNN Training on Shared-Memory Parallel Architectures |
| Publication Type | Tech Report |
| Year of Publication | 2020 |
| Authors | Lopez, F., E. Chow, S. Tomov, and J. Dongarra |
| Technical Report Series Title | Innovative Computing Laboratory Technical Report |
| Number | ICL-UT-20-04 |
| Date Published | 2020-03 |
| Institution | University of Tennessee, Knoxville |
| Keywords | Asynchronous iterative methods, Deep learning, gpu, multicore CPU, Stochastic Gradient Descent |
| Abstract | We present a parallel asynchronous Stochastic Gradient Descent algorithm for shared memory architectures. Different from previous asynchronous algorithms, we consider the case where the gradient updates are not particularly sparse. In the context of the MagmaDNN framework, we compare the parallel efficiency of the asynchronous implementation with that of the traditional synchronous implementation. Tests are performed for training deep neural networks on multicore CPUs and GPU devices. |
File:
External Publication Flag:
