Lecture 15: Least Squares Linear Regression

  • Linear Model

  • Vector form \( x ^ T \theta \)

  • Matrix is for each prediction

  • note the one on the left and \( Y = X \theta \)

  • Linear model of carat, depth, and table
# adds one to left. Horizontal stack
X = np.hstack([np.ones((n,1)), data[['carat', 'depth', 'table']].to_numpy()])
X

def linear_model(theta, xt): 
    return xt @ theta # The @ symbol is matrix multiply

Least Squares Linear Regression

  • Loss
    • By Calculus or Geometric Reasoning

  • You have a span
    • Make the residual perpendicular (orthogonal)

  • Find Theta that minimizes residual
    • Orthogonal is that equation
    • \( X^T X \) is a matrix. The optimal value is full rank

  • Our Loss is the average squared loss
def squared_loss(theta):
    return ((Y - X @ theta).T @ (Y - X @ theta)).item() / Y.shape[0]

theta_hat = inv(X.T @ X) @ X.T @ Y
theta_hat

Y_hat = X @ theta_hat

Geometry of Least Squares