K-Nearest Neighbors

KNN classifier
Here we want to estimate the conditional probability that a predictor vector $x_0$ belongs to a class $j$. To do so with the KNN classifier, first we identify the $K$ observations closest to $x_0$ and represent them by $N_0$. Then we take the proportion of points in $N_0$ with target class $j$:
$$\text{Pr}(Y = j | X = x_0) = \frac{1}{K} \sum_{i \in N_0} I(y_i = j).$$


KNN regression
Here we want to predict the target value for predictor vector $x_0$. To do so with KNN regression, we take the $K$ observations closest to $x_0$ (which we represent by $N_0$) and simply take the average of the target values:
$$\hat{f}(x_0) = \frac{1}{K} \sum_{i \in N_0} y_i.$$