Loss functions

Gradient descent (batch, stochastic, mini-batch)

Learning rate

Momentum

Adaptive methods (AdaGrad, RMSprop, Adam)

Convex vs. non-convex optimization

Regularization (L1 / Lasso, L2 / Ridge)

Lagrange multipliers

Constrained optimization