Logistic Regression

Logistic regression deals with a target variable that is categorical. Its goal is to predict the conditional probability that the target variable belongs to a specific class, given the values of the predictors.

Here we only consider the case with two classes. There exist an extension of logistic regression that deals with more classes, but the preferred method is linear discriminant analysis so the extension is not discussed here.

In the case where the target variable $Y$ can only belong to one of two categories. $Y$ can then be represented as such:
$$Y =
\begin{cases}
0 & \text{if in class } 1\\
1 & \text{if in class } 2
\end{cases}$$
The conditional probability that we are interested in is $\text{Pr}(Y = 1 | X)$, where $X$ represents the predictors. A nice property of logistic regression is that this conditional probability will always be greater than 0 and less than 1.

If we abbreviate $\text{Pr}(Y = 1 | X)$ as $p(X)$, the model that we are fitting is the following:
$$p(X) = \frac{e^{\beta_0 + \beta_{1}X_1 + \cdots + \beta_{p}X_p}}{1 + e^{\beta_0 + \beta_{1}X_1 + \cdots + \beta_{p}X_p}}$$
where $p$ is the number of predictors. Since this is equivalent to
$$\text{log}\left(\frac{p(X)}{1 - p(X)}\right) = \beta_0 + \beta_{1}X_1 + \cdots + \beta_{p}X_p,$$
we see that we are really modeling the log-odds of $p(X)$. The expression on the left-hand side is called the logit function (which explains the name of this method).

To estimate the coefficients $\beta_i$'s, we maximize the likelihood function
$$l(\beta_0, \beta_1, \cdots, \beta_p) = \prod_{i: y_i = 1}p(x_i) \prod_{j: y_j = 0}(1 - p(x_j)).$$

To assess the significance of each predictor in the model, we can perform z-tests. The null hypothesis is $H_0: \beta_i = 0$ and the z-statistic is $\frac{\hat{\beta_i}}{\text{SE}(\hat{\beta_i})}$.