My notes on Logistic Regression

posted Jun 9, 2014, 10:32 PM by Teng-Yok Lee [ updated Jun 10, 2014, 8:59 AM ]

REF: http://www.holehouse.org/mlclass/06_Logistic_Regression.html

Because it took me a while to finally derive it, I decide to put the detail here. Since I cannot type the equation nice, I simplify the notations.

Note 1: Gradients of logistic cost functions

Here the cost function is denoted as F(t) where t terms for theta. m is the number of samples. (x⁽ⁱ⁾, y⁽ⁱ⁾), i = 1 ... m, is the training set. h_t(x) =1/(1+exp(-t^Tx)) is the logistic function with parameter theta (t).

F(t) = 1/m sum_{i = 1 ... m} y⁽ⁱ⁾ log h_t(x⁽ⁱ⁾) + (1 - y⁽ⁱ⁾)log (1 - h_t(x⁽ⁱ⁾))

The partial gradient w.tr.t t_j is denoted as d/dt_j = d_j. Then the partial gradient at t_j, aka, d_j F(t) is derived as follows:

d_j F(t)
= 1/m sum_{i = 1 ... m} y⁽ⁱ⁾ / h_t(x⁽ⁱ⁾) d_j h_t(x⁽ⁱ⁾) + (1 - y⁽ⁱ⁾)/(1 - h_t(x⁽ⁱ⁾)) d_j (1 - h_t(x⁽ⁱ⁾))
= 1/m sum_{i = 1 ... m} y⁽ⁱ⁾ / h_t(x⁽ⁱ⁾) d_j h_t(x⁽ⁱ⁾) - (1 - y⁽ⁱ⁾)/(1 - h_t(x⁽ⁱ⁾)) d_j h_t(x⁽ⁱ⁾) <-- Remove the constant 1, which is in boldface above.
= 1/m sum_{i = 1 ... m} d_j h_t(x⁽ⁱ⁾) {y⁽ⁱ⁾ / h_t(x⁽ⁱ⁾) - (1 - y⁽ⁱ⁾)/(1 - h_t(x⁽ⁱ⁾))} <-- Separate d_j h_t(x⁽ⁱ⁾) from both terms.
= 1/m sum_{i = 1 ... m} d_j h_t(x⁽ⁱ⁾) {y⁽ⁱ⁾ (1 - h_t(x⁽ⁱ⁾)) - h_t(x⁽ⁱ⁾)(1 - y⁽ⁱ⁾)}/{h_t(x⁽ⁱ⁾)(1 - h_t(x⁽ⁱ⁾)}
= 1/m sum_{i = 1 ... m} d_j h_t(x⁽ⁱ⁾) (y⁽ⁱ⁾ - h_t(x⁽ⁱ⁾))/{h_t(x⁽ⁱ⁾)(1 - h_t(x⁽ⁱ⁾)}

where

d_j h_t(x⁽ⁱ⁾)
= d_j (1 + exp(-t^Tx⁽ⁱ⁾))^-1
= -(1 + exp(-t^Tx⁽ⁱ⁾))^-2 exp(-t^Tx⁽ⁱ⁾) x⁽ⁱ⁾_j

= -{1/(1 + exp(-t^Tx⁽ⁱ⁾)} {exp(-t^Tx⁽ⁱ⁾) / (1 + exp(-t^Tx⁽ⁱ⁾)} x⁽ⁱ⁾_j= -{h_t(x⁽ⁱ⁾)(1 - h_t(x⁽ⁱ⁾)} x⁽ⁱ⁾_j

Thus

d_j F(t)
= 1/m sum_{i = 1 ... m} d_j h_t(x⁽ⁱ⁾) (y⁽ⁱ⁾ - h_t(x⁽ⁱ⁾))/{h_t(x⁽ⁱ⁾)(1 - h_t(x⁽ⁱ⁾)}
= 1/m sum_{i = 1 ... m} (y⁽ⁱ⁾ - h_t(x⁽ⁱ⁾)) {-{h_t(x⁽ⁱ⁾)(1 - h_t(x⁽ⁱ⁾)} x⁽ⁱ⁾_j}/{h_t(x⁽ⁱ⁾)(1 - h_t(x⁽ⁱ⁾)}
= 1/m sum_{i = 1 ... m} (h_t(x⁽ⁱ⁾) - y⁽ⁱ⁾) x⁽ⁱ⁾_j

Note 2: How is logistic regression related to MLE?

Actually the logistic regression cost can be treated as the likelihood of a Bernoulli random variable. Here P[x; t] = h_t(x) is the probability that x belongs to class 0. Then the pdf of x is

f(x) = P[x; t]^y (1 - P[x; t])^{1 - y}

and its likelihood of theta t is:

L(t) = log f(x) = y log P[x; t]+ (1- y) log 1 - P[x; t] = y log h_t(x) + (1- y) log 1 - h_t(x)

That's why optimizing the cost function F is equivalent to find the Maximum Likelihood Estimator.

Comments