Lecture 14: Review Models and Loss

  • Response var you want to estimate

  • Model, summarizes with parameter w

  • w is an estimator

Loss

  • \( L(w)=\frac{1}{n}\) # sum n i
  • sum of each \( y_i \)

  • We compare the red value vs the purple value. The green value is the best w, it minimizes loss.

Minimizing Loss

  • L(w*) ?

  • What is your 1. data, 2. model, 3. parameters, 4. loss, (also optimization method)

  • From 1-d (best avg) to 2-d, best func

  • \( w* \) is w that minimizes L(w). The best estimator is \( \hat{y}(w^*) \)

  • Can generalize our optimization to 3d! (3 weight params)
  • Can't plot our loss in 3d (cause it's 4d with 3 dim and loss dim)

  • One option is calculus set deriv of loss to 0
  • Other is grad descent/SGD

Gradient Descent

  • Can actually do brute force. do np.linspace and try all values. O(N^2)?

  • Find optimal weights of our sin model

  • Derivatives

  • Gradients, the opposite of which way to walk

  • The gradient descent algorithm visualized