# npdl.optimizers¶

Functions to generate Theano update dictionaries for training.

The update functions implement different methods to control the learning rate for use with stochastic gradient descent.

Update functions take a loss expression or a list of gradient expressions and a list of parameters as input and return an ordered dictionary of updates:

## Examples¶

Using SGD to define an update dictionary for a toy example network:

>>> import npdl
>>> from npdl.activations import ReLU
>>> from npdl.activations import Softmax
>>> from npdl.objectives import SCCE
>>> model = npdl.model.Model()
>>> model.compile(loss=SCCE(), optimizer=npdl.optimizers.SGD(lr=0.005))


## Optimizers¶

 SGD Stochastic Gradient Descent (SGD) updates Momentum Stochastic Gradient Descent (SGD) updates with momentum NesterovMomentum Stochastic Gradient Descent (SGD) updates with Nesterov momentum Adagrad Adagrad updates RMSprop RMSProp updates Adadelta Adadelta updates Adam Adam updates Adamax Adamax updates

## Detailed Description¶

class npdl.optimizers.SGD(*args, **kwargs)[source][source]

Generates update expressions of the form:

• param := param - learning_rate * gradient
update(params, grads)[source][source]

Update parameters.

Parameters: params : list A list of parameters in model. grads : list A list of gradients in model.
class npdl.optimizers.Momentum(momentum=0.9, *args, **kwargs)[source][source]

Generates update expressions of the form:

• velocity := momentum * velocity - learning_rate * gradient
• param := param + velocity
Parameters: momentum : float The amount of momentum to apply. Higher momentum results in smoothing over more update steps. Defaults to 0.9.

Notes

Higher momentum also results in larger update steps. To counter that, you can optionally scale your learning rate by 1 - momentum.

update(params, grads)[source][source]

Update parameters.

Parameters: params : list A list of parameters in model. grads : list A list of gradients in model.
class npdl.optimizers.NesterovMomentum(momentum=0.9, *args, **kwargs)[source][source]

Generates update expressions of the form:

• velocity := momentum * velocity - learning_rate * gradient
• param := param + momentum * velocity - learning_rate * gradient
Parameters: momentum : float The amount of momentum to apply. Higher momentum results in smoothing over more update steps. Defaults to 0.9.

Notes

Higher momentum also results in larger update steps. To counter that, you can optionally scale your learning rate by 1 - momentum.

The classic formulation of Nesterov momentum (or Nesterov accelerated gradient) requires the gradient to be evaluated at the predicted next position in parameter space. Here, we use the formulation described at https://github.com/lisa-lab/pylearn2/pull/136#issuecomment-10381617, which allows the gradient to be evaluated at the current parameters.

update(params, grads)[source][source]

Update parameters.

Parameters: params : list A list of parameters in model. grads : list A list of gradients in model.
class npdl.optimizers.Adagrad(epsilon=1e-06, *args, **kwargs)[source][source]

Scale learning rates by dividing with the square root of accumulated squared gradients. See [1] for further description.

Parameters: epsilon : float Small value added for numerical stability.

Notes

Using step size eta Adagrad calculates the learning rate for feature i at time step t as:

$\eta_{t,i} = \frac{\eta} {\sqrt{\sum^t_{t^\prime} g^2_{t^\prime,i}+\epsilon}} g_{t,i}$

as such the learning rate is monotonically decreasing.

Epsilon is not included in the typical formula, see [2].

References

 [1] Duchi, J., Hazan, E., & Singer, Y. (2011): Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12:2121-2159.
update(params, grads)[source][source]

Update parameters.

Parameters: params : list A list of parameters in model. grads : list A list of gradients in model.
class npdl.optimizers.RMSprop(rho=0.9, epsilon=1e-06, *args, **kwargs)[source][source]

Scale learning rates by dividing with the moving average of the root mean squared (RMS) gradients. See [1] for further description.

Parameters: rho : float Gradient moving average decay factor. epsilon : float Small value added for numerical stability.

Notes

rho should be between 0 and 1. A value of rho close to 1 will decay the moving average slowly and a value close to 0 will decay the moving average fast.

Using the step size $$\eta$$ and a decay factor $$\rho$$ the learning rate $$\eta_t$$ is calculated as:

$\begin{split}r_t &= \rho r_{t-1} + (1-\rho)*g^2\\ \eta_t &= \frac{\eta}{\sqrt{r_t + \epsilon}}\end{split}$

References

 [1] Tieleman, T. and Hinton, G. (2012): Neural Networks for Machine Learning, Lecture 6.5 - rmsprop. Coursera. http://www.youtube.com/watch?v=O3sxAc4hxZU (formula @5:20)
update(params, grads)[source][source]

Update parameters.

Parameters: params : list A list of parameters in model. grads : list A list of gradients in model.
class npdl.optimizers.Adadelta(rho=0.9, epsilon=1e-06, *args, **kwargs)[source][source]

Scale learning rates by the ratio of accumulated gradients to accumulated updates, see [1] and notes for further description.

Parameters: rho : float Gradient moving average decay factor. epsilon : float Small value added for numerical stability. decay : float Decay parameter for the moving average.

Notes

rho should be between 0 and 1. A value of rho close to 1 will decay the moving average slowly and a value close to 0 will decay the moving average fast.

rho = 0.95 and epsilon=1e-6 are suggested in the paper and reported to work for multiple datasets (MNIST, speech).

In the paper, no learning rate is considered (so learning_rate=1.0). Probably best to keep it at this value. epsilon is important for the very first update (so the numerator does not become 0).

Using the step size eta and a decay factor rho the learning rate is calculated as:

$\begin{split}r_t &= \rho r_{t-1} + (1-\rho)*g^2\\ \eta_t &= \eta \frac{\sqrt{s_{t-1} + \epsilon}} {\sqrt{r_t + \epsilon}}\\ s_t &= \rho s_{t-1} + (1-\rho)*(\eta_t*g)^2\end{split}$

References

update(params, grads)[source][source]

Update parameters.

Parameters: params : list A list of parameters in model. grads : list A list of gradients in model.
class npdl.optimizers.Adam(beta1=0.9, beta2=0.999, epsilon=1e-08, *args, **kwargs)[source][source]

Parameters: beta1 : float Exponential decay rate for the first moment estimates. beta2 : float Exponential decay rate for the second moment estimates. epsilon : float Constant for numerical stability.

Notes

The paper [1] includes an additional hyperparameter lambda. This is only needed to prove convergence of the algorithm and has no practical use (personal communication with the authors), it is therefore omitted here.

References

 [1] (1, 2) Kingma, Diederik, and Jimmy Ba (2014): Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980.
update(params, grads)[source][source]

Update parameters.

Parameters: params : list A list of parameters in model. grads : list A list of gradients in model.
class npdl.optimizers.Adamax(beta1=0.9, beta2=0.999, epsilon=1e-08, *args, **kwargs)[source][source]

update(params, grads)[source][source]