Lecture 23: Logistic Regression

  • machine learning
  • when labeled you have supervised learning
    • when quantitative do regression
    • when categorical do classification
  • when unlabeled you have unsupervised learning
    • dimensionality reduction
    • clustering
  • finally reinforcement learning

Kinds of Classification

  • binary two classes
  • multiclass [cat, dog, car]
  • structured prediction ?

  • try least squares regression

  • two classes

  • truncated least square

Estimating the chance of success

  • two difference coins case by case

  • single expression \( p^y(1-p)^{1-y} \)

  • estimate probability find value that maximize the function
  • take a log

  • as a sum

  • you can minimize the average?

  • what is the function in the two hard cases?

  • as a loss function there is a penalty

Logistic Function

  • linear functions are not good for probabilities

  • t can be infinity and negative infinity
  • take e of both sides

  • model probability on the real line
  • sigmoid, defined on whole line, smooth, increasing, elongated S

  • derivative

Log Odds as a Linear Function

  • features
  • linear combo of features

  • log odds
  • probabiligty is of the log odds

The Steps of the Model

  • generalized linear model

  • linear regression continuous
  • categorical (probability Y is 1)

  • increase x by one unit

  • linearly seperable data

  • you need a little bit of uncertainty

  • with regularization term

Logistic Loss Function

Gradient Descent

Logistic Regression in Scikit Learn

Log Loss

  • or cross-entropy loss

  • log loss is convex!