Type PolynomialDecay
Namespace tensorflow.keras.optimizers.schedules
Parent LearningRateSchedule
Interfaces IPolynomialDecay
A LearningRateSchedule that uses a polynomial decay schedule.
Methods
Properties
Public static methods
PolynomialDecay NewDyn(object initial_learning_rate, object decay_steps, ImplicitContainer<T> end_learning_rate, ImplicitContainer<T> power, ImplicitContainer<T> cycle, object name)
Applies a polynomial decay to the learning rate. It is commonly observed that a monotonically decreasing learning rate, whose
degree of change is carefully chosen, results in a better performing model.
This schedule applies a polynomial decay function to an optimizer step,
given a provided `initial_learning_rate`, to reach an `end_learning_rate`
in the given `decay_steps`. It requires a `step` value to compute the decayed learning rate. You
can just pass a TensorFlow variable that you increment at each training
step. The schedule is a 1-arg callable that produces a decayed learning rate
when passed the current optimizer step. This can be useful for changing the
learning rate value across different invocations of optimizer functions.
It is computed as:
If `cycle` is True then a multiple of `decay_steps` is used, the first one
that is bigger than `step`.
You can pass this schedule directly into a
tf.keras.optimizers.Optimizer
as the learning rate.
Example: Fit a model while decaying from 0.1 to 0.01 in 10000 steps using
sqrt (i.e. power=0.5):
The learning rate schedule is also serializable and deserializable using
tf.keras.optimizers.schedules.serialize and
tf.keras.optimizers.schedules.deserialize.
Parameters
-
objectinitial_learning_rate - A scalar `float32` or `float64` `Tensor` or a Python number. The initial learning rate.
-
objectdecay_steps - A scalar `int32` or `int64` `Tensor` or a Python number. Must be positive. See the decay computation above.
-
ImplicitContainer<T>end_learning_rate - A scalar `float32` or `float64` `Tensor` or a Python number. The minimal end learning rate.
-
ImplicitContainer<T>power - A scalar `float32` or `float64` `Tensor` or a Python number. The power of the polynomial. Defaults to linear, 1.0.
-
ImplicitContainer<T>cycle - A boolean, whether or not it should cycle beyond decay_steps.
-
objectname - String. Optional name of the operation. Defaults to 'PolynomialDecay'.
Returns
-
PolynomialDecay - A 1-arg callable learning rate schedule that takes the current optimizer step and outputs the decayed learning rate, a scalar `Tensor` of the same type as `initial_learning_rate`.
Show Example
def decayed_learning_rate(step):
step = min(step, decay_steps)
return ((initial_learning_rate - end_learning_rate) *
(1 - step / decay_steps) ^ (power)
) + end_learning_rate