Type RMSprop
Namespace tensorflow.keras.optimizers
Parent Optimizer
Interfaces IRMSprop
Optimizer that implements the RMSprop algorithm. A detailed description of rmsprop. - maintain a moving (discounted) average of the square of gradients
- divide gradient by the root of this average $$mean_square_t = rho * mean_square{t-1} + (1-rho) * gradient ** 2$$
$$mom_t = momentum * mom_{t-1} + learning_rate * gradient / \sqrt{ /
mean_square_t + \epsilon}$$
$$variable_t := variable_{t-1} - mom_t$$ This implementation of RMSprop uses plain momentum, not Nesterov momentum. The centered version additionally maintains a moving average of the
gradients, and uses that average to estimate the variance: $$mean_grad_t = rho * mean_grad_{t-1} + (1-rho) * gradient$$
$$mean_square_t = rho * mean_square_{t-1} + (1-rho) * gradient ** 2$$
$$mom_t = momentum * mom_{t-1} + learning_rate * gradient /
sqrt(mean_square_t - mean_grad_t**2 + epsilon)$$
$$variable_t := variable_{t-1} - mom_t$$ References
See ([pdf]
http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf).